Langfuse integration

[flefevre]

langfuse integration
In order to monitor the open webui application (prompt usage, model sélection, metrics) , it would be à good idéa to intégrateur the langfuse api.
I am not part of it, so perhaps there is better alternative but it will be définitively a mandatory feature in the newt release of 'open webui'

[flefevre]

https://github.com/langfuse/langfuse

[Michelklingler]

Hello!
We've been talking about it in the discord for a few days and we found out that the connection is pretty straight forward using LiteLLM.

Of course it would only monitor the call made throught the LiteLLM integration.

We would just need to update 2 things in the code and it would work out of the box with a little of configuration.

1 - Add langfuse dependency to requierement.txt file

2 - Add pass the $user_id to LiteLLM to be able to have the user usage monitoring in the dashboard.

When this is added to the code, do the following to send your call to langfuse:

1 - When launching the Docker container pass the 2 following env. variable:

LANGFUSE_PUBLIC_KEY: "xxxxx"
LANGFUSE_SECRET_KEY: "xxxxx"

Those key are provided when you create a project in Langfuse.

2 - Edit the LiteLLM config.yaml file

And add under general setting the following:

general_settings: {}
litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]  

As mentionned in their doc https://litellm.vercel.app/docs/observability/langfuse_integration

Tested it works like a charm!
I just need someone to update the requierement file and pass the user variable with the API call to LiteLLM.

[jannikstdl]

Thanks for the documentation!

[flefevre]

@Michelklingler thanks a lot for your sharing.

i do not see the LANGFUSE_HOST variable , did you set it? in your docker-compose file?

did you sumit a PR to integrate ths in openwebui?

thanks again

[flefevre]

Additionnaly to the question on how you pass the LANGFUSE_HOST variable, I would like to know how to you trace the call that is made directly from Openwebui to Ollama ? since some models can be served by ollama and other from the litellm

[srd00]

About LANGFUSE_HOST I suppose he just forgot to mention it

About local ollama models all you gotta do is add these to litellm/config.yaml (sample data)

general_settings: {}
litellm_settings:
  failure_callback:
  - langfuse
  success_callback:
  - langfuse
model_list:
- litellm_params:
    api_base: http://127.0.0.1:11434
    model: ollama/mixtral:latest
  model_name: ollama/mixtral:latest   # Can be changed, this is what is actually shown on the ui
router_settings: {}

you can also use your own modelfile modelname and you can also add those through the ui.

[Michelklingler]

On the other side LiteLLM team added a flag I requested to be able to discard forwarding message content to Langfuse as it was too private for me to share actual message written to the chatbot.

Langfuse would only record Model / call / UserID

BerriAI/litellm#2586

[Michelklingler]

PR to add the Langfuse dependency to the requirements.txt file:

#1267

[flefevre]

@Michelklingler thanks a lot.
I didn't understand why i didn't see your answer.

I will test it before the end of the week.
i have à question, you made a PR with only the modification of the requirement, but you spoke also 'add Pass $user id', where is thèse filé ? Why you didn't subit in your PR ?

Sorry if i miss understand.

[Michelklingler]

Hi everyone!

Below is the config to run Langfuse locally.

The following code added to your docker-compose will create:

1 - A local Langfuse server that will monitor all request made through the LiteLLM integration.
2 - A local PostgreSQL database and a UI to save all request informations

Add the following below the docker-compose you use to launch the OpenWeb-UI (if you still run openWebUI with docker run, I highly suggest you move toward a docker-compose solution at this point, it will make it easier to maintain)

### SERVICE : Langfuse ###
  langfuse-server:
    image: ghcr.io/langfuse/langfuse:latest
    depends_on:
      - db
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/postgres
      - NEXTAUTH_SECRET=mysecret
      - SALT=mysalt
      - NEXTAUTH_URL=http://localhost:3000
      - TELEMETRY_ENABLED=${TELEMETRY_ENABLED:-true}
      - LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=${LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:-false}

 ### SERVICE : PostgreSQL Database for Langfuse ###
  db:
    image: postgres
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - 5432:5432
    volumes:
      - database_data:/var/lib/postgresql/data

### SERVICE : GUI for PostgreSQL Database ###
  adminer:
    image: adminer
    restart: always
    ports:
      - 8081:8080

[srd00]

Thanks! Tracking traces seem to work pretty well, did you have any luck on passing user_id and session_id to the langfuse callback?

[Michelklingler]

We would need to pass the User ID variable to LiteLLM as well.
For now the WebUI only pass the message ID, Model Name and content of the request + response.

Probably a minor adjustement to the code.

[ther3zz]

Do you know if this passes user feedback?
I've played around with it a bit and I can see traces but cannot see any feedback within langfuse

[zuberahmed1987]

I am also not able to see user_id and feedback in langfuse. Is there any work around/ configuration which I am missing.

[Michelklingler]

Langfuse is open source. The cloud version is only a hosted version, turnkey solution. You can install Langfuse locally using this repository: https://github.com/langfuse/langfuse

…

On Wed, Apr 17, 2024 at 1:51 AM flefevre ***@***.***> wrote: Hello @Michelklingler <https://github.com/Michelklingler> thanks a lot for the clearer documentation. You said: If data privacy is a concern for you, it is recommended to install the local version instead. But I didn't find any element on the Langfuse HOST configuration. It seems we can only use Langfuse official service. is it a environmental variable in the docker-compose.yml file OR in the LiteLLM Configuration File ? Could you provide such additional information for the community? Thanks again. François — Reply to this email directly, view it on GitHub <#1014 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APCHX3VXH7CXEDC52RCQNI3Y5YEWLAVCNFSM6AAAAABE6KTVVWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMZXHA2TC> . You are receiving this because you were mentioned.Message ID: ***@***.***>