dify 0.6.6 fails to deploy local xinference model

[lk1983823]

Self Checks

• This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.6

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I have docker built the update version of dify 0.6.6 . I follow the instructions to deploy a local qwen-vl-7b model under xinference. I have used my computer ip address and xinference's default ports 9997. But it failed.
My os version is Ubuntu 22.04

✔️ Expected Behavior

I am expecting to deploy the model successfully.

❌ Actual Behavior

It shows
Xinference credentials validate failed: get xinference model extra parameter failed, url: http://192.168.31.191:9997/v1/models/qwen-vl-chat, error: HTTPConnectionPool(host='192.168.31.191', port=9997): Max retries exceeded with url: /v1/models/qwen-vl-chat (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe983bb11e0>: Failed to establish a new connection: [Errno 111] Connection refused'))

[dosubot]

The error you're encountering indicates a connection issue between the Dify container and the Xinference service. To resolve this, follow these steps:

1. Verify Xinference Service: Ensure Xinference is actively running on your machine. If not, start it with the appropriate command, ensuring it listens on 0.0.0.0 to accept connections from any IP, which is crucial for Docker setups.

2. Correct IP Address in Dify Configuration: In your Dify configuration, use your machine's external IP address instead of localhost or 127.0.0.1. This ensures the Dify container can reach the Xinference service running on your host.

3. Check Firewall Settings: Make sure your firewall allows incoming connections on port 9997. Adjust your firewall settings if necessary to permit these connections.

4. Test Connectivity: Confirm there are no network issues preventing connectivity to the Xinference service. Testing can be done from another machine or within the Docker container using tools like curl.

By ensuring Xinference is running, using the correct IP address in your Dify configuration, adjusting firewall settings, and confirming network connectivity, you should be able to successfully deploy your model.

Details

Sources

• A Configuration Issue with Stable Diffusion in Dify
• Cannot connect to xinference
• dify cannot connect to xinference LLM
• en/tutorials/model-configuration/xinference.md
• zh_CN/getting-started/install-self-hosted/docker-compose.md

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[takatost]

You can directly curl the xinference API address inside the container to check if it's reachable. It might be a firewall issue.