Ollama token counts

[aiseei]

Describe the bug

Great product! easy to setup without the overheads.
We also use it to trace Ollama requests. Price info isnt the issue it is defn token count etc. Are there plans to support this ?

Thanks!

To reproduce

NA

Additional information

No response

[marcklingen]

Hi @aiseei,
Tracking token usage when using Ollama works well with Langfuse.

Ollama returns token counts at the end of the stream or together with the response when not streaming. Just tried this locally and it worked well.
Example from the api reference:

{
  "model": "llama2",
  "created_at": "2023-08-04T19:22:45.499127Z",
  "response": "",
  "done": true,
  "context": [1, 2, 3],
  "total_duration": 10706818083,
  "load_duration": 6338219291,
  "prompt_eval_count": 26, # <- input tokens
  "prompt_eval_duration": 130079000,
  "eval_count": 259, # <- output tokens
  "eval_duration": 4232710000
}

You can then add these token counts to the generation object in Langfuse to track them (docs).

[aiseei]

cool. will try that

[flefevre]

Could you share your ollama configuration in order to trace call with langfuse?
I am using ollama docker compose setup.
Thanks

[spicoflorin]

Hello!
I'm also interested on how to count the tokens when using the following models:

1. token counts used for the embeddings (I'm using sentence transformer)
2. token counts for prompt (I'm using llama2)
3. count completion tokens (I'm using llama2)
4. I'm using langchain with RAG to run the inference pipeline like this qa.run(query, callbacks=[stream_handler,langfuse_handler])

What configuration and code for Langfuse do I need for getting the above or at least the total input tokens and output_token?
Can you provide a full workable example on how to do it?
Suppose that I achieve these, where in the UI of the Langfuse server can I see them?

Thank you.

[marcklingen]

I assume that you use langchain @spicoflorin, this is different from the original question. When using langchain it depends on whether the token counts are correctly parsed from the llm response by the langchain class as langfuse gets them from there

[spicoflorin]

Hello, @marcklingen !

Thank you for your answer.
Yes, I'm using langchain with SenteceTransformer as embedding model and llama2 as generative model.
My prototype is based on genai-stack project where I have used langsmith as observaibility tool (that have incorporated the token counts feature)
Now, I would like to use langfuse for achieving (if it possible) the same functionalities.

Since I'm a beginner with these tools (langchain and langfuse), can you please share a code on how to:

1. get the tokens from the llm response
2. how to send them to the langfuse

Thank you in advance.

Regards,
Florin