-
Notifications
You must be signed in to change notification settings - Fork 255
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Unrecognized configuration class when quantizing llava #1601
Comments
The quantization of vl models is not supported until #1553 get merged. May try the PR directly if you are in a hurry. |
@AllentDan Thanks for your reply. I have two follow-up questions and would appreciate further confirmation.
|
2 tasks
|
Thank you very much. Will try. If it's ok I'd like to keep this issue open just for now. |
Supported in the latest main. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
Describe the bug
When running the w4a16 quantization for llava models, the
transformers==4.42.0
library will raise theUnrecognized configuration class
error, i.e., the llava model class is not registered in transformers and thus cannot be found. I know this is not really a bug with lmdeploy itself. I've seen the exact same issue reported in other repos (e.g. here), where the suggestion was to usetransformers==4.31.0
; yet it didn't help.I was also surprised that no one else raised this issue and there seem to be plenty people succeeded in quantizing llava models. Thus by opening this issue I want to see if there's anything wrong on my side.
Note that below I was trying to quantize
lmms-lab/llama3-llava-next-8b
, but the same error was also there if changing it toliuhaotian/llava-v1.5-7b
.What I've tried
I tried switching transformers version between
4.31.0
, the latest4.42.0
, and the one specified by llava authorstransformers@ git+https://github.com/huggingface/transformers.git@1c39974a4c4036fd641bc1191cc32799f85715a4
; yet none of them worked. This is kind of expected because regardless of the transformers version I'd expect some manual registration performed like here?Reproduction
Environment
Error traceback
The text was updated successfully, but these errors were encountered: