New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请问lmdeploy如何在多卡部署 #555

Open

ChingKwanCheung opened this issue Feb 26, 2024 · 1 comment

ChingKwanCheung commented Feb 26, 2024

输入命令lmdeploy serve api_server后，默认是在单卡（0号卡）上部署，请问怎么实现在多卡上部署？

zhulinJulia24 commented Apr 9, 2024

CUDA_VISIBLE_DEVICES=0,2 使用前缀或者环境变量指定可以使用的卡
对于命令lmdeploy serve api_server 来说可以--help查看使用说明，其中 --tp 2表示用2卡部署

因而多卡部署可以结合1,2，如
CUDA_VISIBLE_DEVICES=0,2 lmdeploy serve api_server $model_path --tp 2 表示使用0卡和2卡进行2卡部署

具体也可见 https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/serving/api_server.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment