How to get generations and scores in the same trace when using OpenAI sdk or other tools

[databill86]

Hello,

Thank you for this great tool. We are using it with litellm and it's really good.

Actually, I'm trying to log scores into langfuse, I'm using Ragas for the evaluation. So far, it's a bit tricky to log both scores and generations of Ragas in the same place, at once.

For the scores, I have the trace_id for each completion, Now, the question is how to trace the prompts and generations used by ragas to get the scores and log them with the aforementioned completions.

I'm using the openAI sdk and not langfuse for the generations. I've configured litellm as the proxy.

from langfuse import Langfuse
from ragas import evaluate
...

# call a method to get the llm generations for the dataset: using openai sdk
....


# evaluate and log the scores
langfuse = Langfuse()

    for i, row in df_results.iterrows():
        # get trace using trace_id
        trace_id = row["trace_id"]

        langfuse.span(
            trace_id=trace_id,
            name="retrieval",
            input={"question": row["question"]},
            output={
                f"answer_{store}_hybrid": row[f"answer_{store}_hybrid"],
            },
        )

        score_row = {
            "context_precision": row["context_precision"],
            "context_recall": row["context_recall"],
            "context_relevancy": row["context_relevancy"],
            "faithfulness": row["faithfulness"],
        }

        langfuse.score(
            trace_id=trace_id,
            name="scores",
            input={
                "question": row["question"],
                "context": row[f"answer_{store}_hybrid"],
                "true_answer": row["true_answer"],
                f"answer_llm_{llm_model}": row[f"answer_llm_{llm_model}"],
            },
            output=score_row,
        )
        # iterate over scores and log them
        for k, v in score_row.items():
            langfuse.score(trace_id=trace_id, name=k, value=v)

    langfuse.flush()

Is there a way to log the generations done by ragas to get the scores in the corresponding traces.

[UPDATE]: I updated the message, as I couldn't tackle the second problem.

[marcklingen]

Hi @databill86, did you figure this out on your own? Happy to help

[databill86]

No I didn't, I figured i should just log traces using llmlite using some predefined ids, and then get the ids back with langfuse and score the results.
Then the only thing left is how to trace the ones with ragas.

[databill86]

@marcklingen sorry I updated the message, I hope there is a simple way to handle this using langfuse, because I really don't want to implement some other classes to wrap the ragas llm interface.