How are models that use in-context examples handled?

[sachit-menon]

Hi, thanks for this great work! I was wondering how models which can use in-context examples are evaluated. Is everything in this benchmark kit zero-shot?

[kennymckormick]

Hi , @sachit-menon
Everything is zero-shot now. We do not officially support few-shot since only few models support that.

[ys-zong]

Hi, we developed an ICL benchmark for VLLMs here: https://github.com/ys-zong/VL-ICL. Welcome to try it out.

@kennymckormick I wonder if you have a plan to integrate VL-ICL into this very useful repo. I'm also happy to contribute:)

[sachit-menon]

Thanks!

…

________________________________ From: Yongshuo Zong ***@***.***> Sent: Friday, April 12, 2024 6:49:15 AM To: open-compass/VLMEvalKit ***@***.***> Cc: Sachit Menon ***@***.***>; Mention ***@***.***> Subject: Re: [open-compass/VLMEvalKit] How are models that use in-context examples handled? (Issue #90) Hi, we developed an ICL benchmark for VLLMs here: https://github.com/ys-zong/VL-ICL. Welcome to try it out. @kennymckormick<https://github.com/kennymckormick> I wonder if you have a plan to integrate VL-ICL into this very useful repo. I'm also happy to contribute:) — Reply to this email directly, view it on GitHub<#90 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFD4ZUJGPM3SXJHDLZRRUMLY4632XAVCNFSM6AAAAABC6RF2OWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRGUZDINBVGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kennymckormick]

Definitely Yes! Welcome to implement your new dataset VL-ICL in VLMEvalKit. After implemented, we can help with the evaluation of API models.
BTW, you can check our latest refactor PR #140 , and make sure you are developing based on the new interface.

Hi, we developed an ICL benchmark for VLLMs here: https://github.com/ys-zong/VL-ICL. Welcome to try it out.

@kennymckormick I wonder if you have a plan to integrate VL-ICL into this very useful repo. I'm also happy to contribute:)