Xtuner和VLMEvalKit针对MMBench的结果不对齐

[thomas-yanxin]

基于Xtuner训练LLaVA-Phi3

使用Xtuner评估MMBench-Dev-EN：0.7096
使用VLMEvalKit评估MMBench-Dev-EN：0.549828

[LZHgrla]

@thomas-yanxin
请问 vlmevalkit 的 model config 是怎么配置的？

[thomas-yanxin]

xtuner_series = {
'llava-internlm2-7b': partial(LLaVA_XTuner, llm_path='internlm/internlm2-chat-7b', llava_path='xtuner/llava-internlm2-7b', visual_select_layer=-2, prompt_template='internlm2_chat'),
'llava-internlm2-20b': partial(LLaVA_XTuner, llm_path='internlm/internlm2-chat-20b', llava_path='xtuner/llava-internlm2-20b', visual_select_layer=-2, prompt_template='internlm2_chat'),
'llava-internlm-7b': partial(LLaVA_XTuner, llm_path='internlm/internlm-chat-7b', llava_path='xtuner/llava-internlm-7b', visual_select_layer=-2, prompt_template='internlm_chat'),
'llava-v1.5-7b-xtuner': partial(LLaVA_XTuner, llm_path='lmsys/vicuna-7b-v1.5', llava_path='xtuner/llava-v1.5-7b-xtuner', visual_select_layer=-2, prompt_template='vicuna'),
'llava-v1.5-13b-xtuner': partial(LLaVA_XTuner, llm_path='lmsys/vicuna-13b-v1.5', llava_path='xtuner/llava-v1.5-13b-xtuner', visual_select_layer=-2, prompt_template='vicuna'),
'llava-phi-3': partial(LLaVA_XTuner, llm_path='microsoft/Phi-3-mini-4k-instruct', llava_path='xtuner/llava-phi-3-mini-xtuner, visual_select_layer=-2, prompt_template='phi3_chat'),
}

[LZHgrla]

@thomas-yanxin
因为llava-phi-3的llm也微调了，所以llm_path不应是原llm

'llava-phi-3': partial(
    LLaVA_XTuner,
-   llm_path='microsoft/Phi-3-mini-4k-instruct',
+   llm_path='xtuner/llava-phi-3-mini-xtuner',
    llava_path='xtuner/llava-phi-3-mini-xtuner',
    visual_select_layer=-2,
    prompt_template='phi3_chat'),

[thomas-yanxin]

solved，tks