feat: Log for Function Calling in Langchain

[marcklingen]

Discussed in https://github.com/orgs/langfuse/discussions/1780

^{Originally posted by jaindees April 19, 2024}

Describe the feature or potential improvement

When can we have logs for function calling feature (agent tools), by default getting logged when a langchain based agent is invoked? It helps to understand what all functions/tools passed to LLM along with their descriptions and entity definitions. If it is already there then please help point me to the documentation. Thanks for the wonderful tool!!!

Additional information

No response

[marcklingen]

@jaindees can you add a quick snippet here which helps reproduce the issue?

EDIT: found code example in GH discussion post

[Konsti-s]

@marcklingen I would not mind working on this feature since my company also uses LangFuse and I was actually missing this feature a couple of times.

Since API changes are required I probably need some guidance on how to do it correctly if anybody is willing to spend some time to have me onboarded. Also I have no clue where you want to have this feature integrated in the UI and how to update the token counting if necessary.

I just changed a couple of lines here and made the Langchain Callbackhandler from your python SDK include the tools from the model_parameters. I now already got some basic information about the available tools per generation in my dev-server as you can see on the screenshot.

[marcklingen]

hi @Konsti-s, sorry for slow response here.

I'd probably rather add this as a first object to the input section instead of extending model params which are currently strictly of type Record<string, Any> to be able to render them as labels on an object in the ui.

What do you think about extending the chatml input to:
input=[{role:"tools", content:{}},{role:"system",content:""},...]