Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: capture token counts from google vertex in langchain-python #2032

Open
marcklingen opened this issue May 10, 2024 · 1 comment · Fixed by langfuse/langfuse-python#683
Open

feat: capture token counts from google vertex in langchain-python #2032

marcklingen opened this issue May 10, 2024 · 1 comment · Fixed by langfuse/langfuse-python#683
Labels
integration-langchain prio-low sdk-python

Comments

@marcklingen
Copy link
Member

When using generate the GenerationChunk object contains usage_metadata. It'd be useful to capture the token counts from there.

Example:

[[GenerationChunk([...] generation_info={'usage_metadata': {'prompt_token_count': 15, 'candidates_token_count': 647, 'total_token_count': 662}})]]

Docs: https://python.langchain.com/v0.1/docs/integrations/llms/google_vertex_ai_palm/
@shenghann shenghann mentioned this issue May 14, 2024
Merged
@bell-steven
Copy link

bell-steven commented May 16, 2024 •
edited

Is this specific to vertex? Could this be the default for all models? I have noticed that the langchain integration does not generally use the token counts that come back with a response, and, the token counts in the response are different than what is displayed in the dashboard. Was digging into the python SDK code today to try and debug. It seems like the llamaIndex integration does this already.

More than happy to try and contribute towards this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
integration-langchain prio-low sdk-python
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add token usage parsing for Google Vertex AI models shenghann/langfuse-python
2 participants
@marcklingen @bell-steven