New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to deploy trained llava model? #586
Comments
@zodiacg At the same time, we will provide a script ASAP to convert the xtuner trained model (such as llava-internlm2 models) to the llava official format model. |
It would be very helpful since we have trained some llava models. We hope we can test them in an interactive way. |
From your replies do I understand correctly that the merge doesn't add the llava features to the model ? Here are the steps I followed: I tried to convert my finetuned result to HF using above guide, then merged it like this to existing xtuner llava:
However writting this I suppose the second parameter is a LLM Qlora and unrelated to the Llava adapter probably ? |
@zodiacg We also release the related LLaVA-Llama-3-8B models, which can be found on above docs. |
Hi, thanks for your reply. I have tried to follow the steps, but my folders does not match the ones from the examples, using the Qlora finetune config. Indeed, in my pth to LLava in Xtuner format, I have two folders, llm_adapter, and projector, as well as a xtuner_config.py. No other files, as shown in the README with "visual_encoder_adapter". Thus, when trying to convert to HF, I did Which did not work, with the following error : I haven't done again the training since my comment two weeks ago, maybe there was an update to the library also which should now include the folder ? Also, when trying to replace the --vision_model_id by |
The scripts introduced are specifically tailored for LLaMA as the LLM. The primary appeal of xtuner, at least from my perspective, is the flexibility it offers to use other LLMs as the base. I hope that the xtuner-llava structure will also be supported. |
@zodiacg Yes, we are developing this feature in other PRs; |
Hi! @zodiacg
Then, please use the above saved llm as the value of For the value of |
Thanks, this worked well for me.
Why, if the llm is frozen, do I need to merge a qlora to the base LLM ? Shouldn't it train only the projection layer here ? Thanks for the help above and your reactivity in previous questions 🙏 |
@flotos As for the |
@flotos So, do not forget to merge your lora. |
Thanks very much for your time, this is very clear. |
Currently the trained llava model can only be used by CLI (without the ability to use new images) or tested using benchmark tools.
How can we deploy it using API or WebUI as a more user-friendly interface?
The text was updated successfully, but these errors were encountered: