OWUI integration with Groq inference engine (not Elon Musk's Grok) - 500 tokens/sec

[tbendien]

@tjbck @justinh-rahb have y'all seen this?

Is there a way to integrate Groq API into OWUI?
https://wow.groq.com/

[justinh-rahb]

Yes. LiteLLM:

models groq/llama2-70b-4096 and groq/mixtral-8x7b-32768

[HydeAndGeek]

excuse me Justin. Kind sir. Would you be a gent and explain this in layman's terms? I have spun up docker, running ollama, i am on intel gpu, i7, 8gb ram, not doing too well here,things are superrrrrr slow, and i know groq is blazing fast, seeing if there's a way i can add her to owui. Thanks in advance seriously if you can help make this happen i will gladly cashapp like $5

[sutyrin]

How did it go?)

[rikhuijzer]

This is the configuration that I used to set up open-webui with llama3-70b running on Groq's servers. I've added three files:

config.yaml

model_list:
  - model_name: 'llama3-70b'
    litellm_params:
      model: 'groq/llama3-70b-8192'
      api_key: 'os.environ/GROQ_API_KEY'

docker-compose.yaml

version: '3.7'

services:
services:
  open-webui:
    container_name: 'open-webui'
    image: 'ghcr.io/open-webui/open-webui:main'
    volumes:
      - './open-webui:/app/backend/data:rw'
      - './config.yaml:/app/backend/data/litellm/config.yaml:ro'
    ports:
      - '3003:8080'
    environment:
      DEFAULT_MODELS: 'llama3-70b'
    env_file:
      - 'GROQ_API_KEY.env'
    extra_hosts:
      - 'host.docker.internal:host-gateway'
    restart: 'unless-stopped'

GROQ_API_KEY.env

GROQ_API_KEY=<YOUR API KEY>

After running this with

$ docker compose up

I had a working web UI with llama3 running at http://127.0.0.1:3003.

If you're saving this in Git, then don't forget to add **/*.env and **/open-webui/ to .gitignore. Also, set

    environment:
      ENABLE_SIGNUP: 'False'

if you put this configuration on a public server.

[yjrocks712]

have a look in this video the issue is easily explained in this . https://youtu.be/qOxsIlpIMt4

[gtasteve]

Adding the Groq API

1. GO to Settings | Connections
2. Enter https://api.groq.com/openai/v1 for the API link and add the API key you get from Groq

You will see all LLMs available on Groq

[basicallynabbo]

thank you sir

[yjrocks712]

did you watch this video ? https://youtu.be/qOxsIlpIMt4

…

On Fri, 24 May 2024 at 15:31, el_even ***@***.***> wrote: So, I'm following this tutorial, in the video, but open webui does not save my changes. Then, when I refresh the page, its blank (I know for a fact that the default OPEN AI URL is removed and as the groq url and api key are not changed, the OPEN AI URL is void). Any idea why (open webui is not saving my changes) ? I have also tried to set the OPEN AI URL directly in the docker env variables but I get the same result (blank page). — Reply to this email directly, view it on GitHub <#974 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BAQEM5CUNVJPKSVQ4XO6S2TZD4FYPAVCNFSM6AAAAABEBBR6K2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNBVGM3DG> . You are receiving this because you commented.Message ID: ***@***.***>