-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
如何支持百川13B #2
Comments
试下这个分支 https://github.com/gameofdimension/vllm/tree/13b from vllm import LLM, SamplingParams
llm = LLM(model="baichuan-inc/baichuan-13B", trust_remote_code=True)
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
"希望这篇文章能",
"给六岁小朋友解释一下万有引",
]
sampling_params = SamplingParams(temperature=0.1, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") |
谢谢,这个可以直接用在baichuan-13B-chat版本吧,我试过了,回答都很简短,几个字了 |
跟非 vllm 的比较一下呢 |
差很多, 回答都很简短 |
那就是这个模型本身的问题了,跟 vllm 适配无关哈 |
你好,想问下这个repo进行张量并行的时候会报错:ray.exceptions.RaySystemError: System error: No module named 'transformers_modules' 该怎么解决呢~ |
据说 7B 的 tp 也有问题,13B tp 部分是沿用 7B 的。可以参考下这里的讨论 |
Baichuan-13B-Chat 模型是没有问题,直接使用 |
这个需要设置一个参数就行了,export HF_MODULES_CACHE=安装的 Python 库的地址 |
@gameofdimension 在 tp=2 时,有没有发现什么问题? |
我们已经解决了,W_pack 的问题,切错了导致的,基于你的代码改的,https://github.com/Mr-Nineteen/vllm/tree/dxwang |
你可以给我的 fork 提个 pr 吗,我再提到官方去。或者你直接提到官方去也行 |
@gameofdimension PR 给你了,代码还可以优化一下的,之前我没有进一步优化一下逻辑;发现社区也有 MR 改了 https://github.com/vllm-project/vllm/pull/598/files ,不过实现不一样 |
请问如何加载自己微调的baichuan13B呢?我这边报错:ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported. |
请问如何支持百川13B,怎么修改代码呢
The text was updated successfully, but these errors were encountered: