You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I duplicate the llama model and rename it into llama_7b, changed the model parameters according to llama_7b specification, looks like this:
skiped the CPU eager mode, only run the cuda model.
it reports the following issue when running with this command:
python userbenchmark/dynamo/dynamobench/torchbench.py -dcuda --float16 -n1 --inductor --performance --inference --filter "llama" --batch_size 1 --in_slen 32 --out_slen 3 --output-dir=torchbench_llama_test_logs
If I want to run this model, how should I fix it? my hardware is A100-40G
thanks
The text was updated successfully, but these errors were encountered:
We only guarantee the runability of models on PT eager mode on A100 40GB in our CI. It is possible that inductor uses more GPU memory than eager mode, causing OOM. Optimizing GPU memory usage with inductor is an open question.
cc @msaroufim
@xuzhao9 I tried to use 4xA100-40G to avoid the OOM issue, looks torchbench.py only use one GPU's memory, I used options like --device-index or --multiprocess, both failed. do you have any advice on multi GPU support?
Hi
I duplicate the llama model and rename it into llama_7b, changed the model parameters according to llama_7b specification, looks like this:
skiped the CPU eager mode, only run the cuda model.
it reports the following issue when running with this command:
python userbenchmark/dynamo/dynamobench/torchbench.py -dcuda --float16 -n1 --inductor --performance --inference --filter "llama" --batch_size 1 --in_slen 32 --out_slen 3 --output-dir=torchbench_llama_test_logs
If I want to run this model, how should I fix it? my hardware is A100-40G
thanks
The text was updated successfully, but these errors were encountered: