hf_Bert_large failed

[jinsong-mao]

Hi

I can't run the hf_Bert_large model on my local site, I was running it from pytorch repo and my command is:
./benchmarks/dynamo/torchbench.py --performance --inference --bfloat16 --backend=inductor --only=hf_Bert_large --output=torchbench_inference.csv

the error message is:

looks something wrong with the input.

I found that this model is good to run on dashboard,
https://hud.pytorch.org/benchmark/torchbench/inductor_max_autotune?startTime=Mon,%2006%20May%202024%2010:12:38%20GMT&stopTime=Mon,%2013%20May%202024%2010:12:38%20GMT&granularity=hour&mode=training&model=hf_Bert_large&dtype=amp&lBranch=main&lCommit=02093b6c6ae1046368e2500881d0bb5880873386&rBranch=main&rCommit=68a1f787c8ce0783f39c35a91fa1e6614b974050

anybody can help to resolve this issue? many thanks.

[FindHao]

Can you try this version of transformers? transformers==4.38.1

[jinsong-mao]

@FindHao I checked my environment and make sure that the version of transformers is 4.38.1, and the issue is still exists.

[jinsong-mao]

@FindHao BTW, I tried to build a docker with the dockerfile from the repo and ran the same test, the model hf_Bert_large still failed with the same issue, looks the environment to run the dashboard is different?

[FindHao]

The torchbench in dynamo is slightly different from this repo. You can try the above PR I submitted to solve this issue. Or, you can run with this repo's run.py in your tests.

[jinsong-mao]

@FindHao Thanks for the fixing, it works for inference, there is another similar bug in pytorch benchmark repo: https://github.com/pytorch/pytorch/blob/main/benchmarks/dynamo/torchbench.py#L435

I think it should be "if isinstance(cloned_inputs, dict):" , which looks like a typo, and the hf model training will fail because of this.

thanks

[FindHao]

Thanks for your reply! It is a typo. Fixed in pytorch/pytorch#126981