Too few questions for senior category

[rodion-m]

Hi! Thanks for a useful benchmark. The separation of quants is great.

Am I right in thinking that we currently have only one question in the senior category? I'm afraid it might not be representative.

[rodion-m]

Also, if the question is only one, what does "Passed" column mean here?

[the-crypt-keeper]

@rodion-m There's 3 questions in total, the meat of this test is in vm.yaml which asks the LLM to build an assembler. The score is based on how well the answer performs on the Checks.. the model can get up to weight points for each check and partial marks are possible: for example if 3 of 8 returned list entries are correct the model will still get 3 points, otherwise the scores would all be too low.