LiteLLM Example Configs

[justinh-rahb]

Just wanted to dump some configs for common endpoints for future reference.

Note

Other than official OpenAI-endpoints, LiteLLM usually requires that an provider be specified before the model string in the Add a model field, example:

azure/gpt35turbo

Model Name can be whatever you want it to appear as in your list.
API Base URL typically only needs to be set for OpenAI-compatible APIs and Azure.

Warning

Gemini Endpoint:

ONLY Makersuite/AI Studio API keys are supported, VertexAI/GCP endpoints and authentication methods are NOT currently supported.

Anthropic

Model strings:

• claude-2
• claude-2.1
• claude-instant-1.2
• claude-3-sonnet-20240229
• claude-3-opus-20240229
• claude-3-haiku-20240307

Note

Anthropic's API requires that max_tokens parameter be sent in the payload, the maximum accepted value is 4096

Claude 2.1

Claude 3 "Sonnet"

Claude 3 "Opus"

Groq

Model strings:

• mixtral-8x7b-32768
• llama2-70b-4096

Mixtral 8x7B

Llama2 70B

Google Gemini

Model strings:

• gemini-pro

Gemini Pro

Mistral

Model strings:

• open-mistral-7b (aka mistral-tiny-2312)
• open-mixtral-8x7b (aka mistral-small-2312)
• mistral-small-latest (aka mistral-small-2402)
• mistral-medium-latest (aka mistral-medium-2312)
• mistral-large-latest (aka mistral-large-2402)

Open Mixtral 8x7B (formerly mistral-small)

Mistral Medium

Mistral Large

Azure OpenAI

Model strings:

• gpt35turbo
• gpt4

GPT 3.5 Turbo

---

Caution

OpenAI and Ollama Endpoints

Unless there is a specific reason why the Connections > Ollama or Connections > OpenAI methods are not working for your use-case, the following methods are NOT recommended to be used instead:

OpenAI

Model strings:

• gpt-3.5-turbo
• gpt-4
• gpt-4-turbo-preview
• gpt-4-vision-preview
• See more

GPT 4 Turbo

"OpenAI-compatible" endpoints

Ollama

Llama2

[HydeAndGeek]

ahhhh yes ty!

[PierrunoYT]

Hey it's not working

[PierrunoYT]

Sorry I'm just confused because Pinokio says it's up to date.

[krrishdholakia]

this is amazing @justinh-rahb - if you have any feedback for how we can improve our own docs for this, let me know - https://docs.litellm.ai/docs/

• Krrish, LiteLLM Maintainer

[justinh-rahb]

I get an error when I try to use claude-3-opus-20240229 & claude-3-sonnet-20240229:

AnthropicException - AnthropicException - {
"error": {
"type": "forbidden",
"message": "Request not allowed"
}
}

My guess is invalid API key.

[krrishdholakia]

@abdimussa87 if you run the proxy with --debug you should be able to see the request being sent to anthropic.

[abdimussa87]

It had to do with the region I was in. Now I'm getting this error:
AnthropicException - AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

[justinh-rahb]

It had to do with the region I was in. Now I'm getting this error: AnthropicException - AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

You're running an outdated version of Open WebUI container with an older version of LiteLLM prior to the release of Claude 3 API.

[abdimussa87]

Yup. That was it. Thank you.

[yelban]

No, in v0.1.115 (latest version) open-webui still cannot set LiteLLM to make any Claude 3 model work. Unless...

Upgrade LiteLLM to the latest version v1.34.12.

git clone https://github.com/open-webui/open-webui.git

cd open-webui

Replace litellm==1.30.7 with litellm==1.34.12 in the ./backend/requirements.txt file.

docker build -t ghcr.io/open-webui/open-webui:latest .

Then use the locally created docker image ghcr.io/open-webui/open-webui:latest.

You can use the Claude 3 models added by LiteLLM.

[justinh-rahb]

Look at this, all workin' and stuff:

[yelban]

May I ask if you are also using the Docker container to start open-webui?

When I used the official mirror: ghcr.io/open-webui/open-webui:latest, following the previous example, carefully checking and adding claude 3 litellm has never been successful until modifying the version of litellm in requirements.txt and using a self-built docker image to finally succeed.

[justinh-rahb]

May I ask if you are also using the Docker method to start open-webui?

Docker-installed main branch - v0.1.115, no modifications.

[yelban]

Delete the local open-webui images, After restarting using ghcr.io/open-webui/open-webui:main, I can confirm that the newly added claude 3 model can run correctly.

Docker-installed main branch - v0.1.115, no modifications It works perfectly.

Your response and reminder are greatly appreciated. Thank you. @justinh-rahb

[flefevre]

Thanks a lot for the elements you shared. I am facing a difficulty in setting the access of open webui to litellm.
I do not understand what i need to provide in the litellm-config.yaml file for the sk.
Should i connect to litellm to generate a specific key and put it in master_key?
Thanks for the explanoation.

openwebui | File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 372, in user_api_key_auth openwebui | assert api_key.startswith("sk-") # prevent token hashes from being used openwebui | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ openwebui | AssertionError openwebui | id='dccf746e-2efa-4248-923a-7bfddedea4e4' name='Iagen Innov' email='tot@email.com' role='admin' profile_image_url='/user.png' timestamp=1711742369 openwebui | INFO: 172.22.0.1:0 - "GET /litellm/api/model/info HTTP/1.1" 401 Unauthorized

[justinh-rahb]

Master key is irrelevant in the context of our internal LiteLLM proxy! You seem to be wanting to setup a separate external LiteLLM as it's own container. Use it as if it is an OpenAI endpoint, where Master key is your OPENAI_API_KEY you would use for the "OpenAI Connection", with base URL set to your LiteLLM container's address.

[flefevre]

Thanks for your answer.
You are right. I have installed in two separate docker compose files: 1) open webui and 2) litellm with redis.
We have had an exchange about it here with you and shared it here to help people.: #508 (reply in thread)