New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

推理时候显存分配 #163

Open

xxcoco763 opened this issue Dec 28, 2023 · 1 comment

xxcoco763 commented Dec 28, 2023

模型推理的时候使用device_map="auto" 把模型分到了各张卡上，但是文本加载的显存直接全部放到0号卡了，可以将这部分显存也平均分配吗？

Author

xxcoco763 commented Dec 29, 2023

运行的是 webui
代码如下：
python3 demo.py
--base_model /home/models/LongAlpaca-7B
--context_size 32768
--max_gen_len 1024 \

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment