Evaluators can access all columns

[aakrem]

• Now evaluators have access to whatever correct answer column from the testset.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
agenta	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 31, 2024 7:46am

[mmabrouk]

I don’t understand why you specify correct_answer_key as a special argument for all evaluators. This makes the assumption that the evaluator would need access to one column in the test set that has the correct_answer. However, it could be that the evaluator requires multiple columns (which do not even need to be semantically correct answers), or it might not require anything other than the output of the LLM app (as for instance the case of a toxicity evaluator).

Instead, what I propose is to have the correct_answer_key as part of the configuration of the evaluators (i.e. in settings_values). This would simplify the code (less arguments in the calls), and make the logic more general.

The second comment is about the UI, I think that per default, the evaluators we have should use the correct_answer column. We can have this option hidden inside a collapse.

[mmabrouk]

Thanks for the PR @aakrem

I reviewed the migration. It makes sense to me from logical perspective. For the syntax itself (using .dict() and deleting __dict__) I trust you have tested it.

[mmabrouk]

Thanks for the PR, @aakrem! Great job. I have a few concerns (unrelated to the code changes).

I think the title of the PR and its description are a bit misleading. I have two questions:

* If an evaluator has access to all columns, why can we only access a single column in the test set?

This has been fixed. Now you configure in the evaluator which columns it has access to and it would be able to access them

* If the evaluators has access to any correct answer column from the test set, why are similarity match, JSON field match, webhook test, and Levenshtein distance the only evaluators that do?

It depends on the evaluator. Some of them, like JSON check, which checks whether the llm output is in json format does not require access to any ground truth obviously.

Responded to review and clarified issues in #1606 (comment)