Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Error deploying HuggingFace model llava-v1.6-mistral-7b with lmdeploy: Unrecognized model type llava_mistral #1573

Closed
1 of 2 tasks
enogalaxies opened this issue May 10, 2024 · 11 comments · Fixed by #1579
Closed
1 of 2 tasks

[Bug] Error deploying HuggingFace model llava-v1.6-mistral-7b with lmdeploy: Unrecognized model type llava_mistral #1573

enogalaxies opened this issue May 10, 2024 · 11 comments · Fixed by #1579
Labels
awaiting response Stale

Comments

@enogalaxies
Copy link

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.

Describe the bug

When attempting to deploy the llava-v1.6-mistral-7b model from HuggingFace using lmdeploy, I encountered an error indicating that the model type llava_mistral is not recognized by the Transformers library. This issue occurred despite the lmdeploy documentation stating that any model in HuggingFace format should be supported for inference. The same process worked previously for a model of type vicuna, but is failing for mistral.Local inference with the model works correctly.

Reproduction

Attempt to deploy the llava-v1.6-mistral-7b model using lmdeploy.
Observe the error message in the logs.

Environment

sys.platform: linux
Python: 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: Tesla T4
CUDA_HOME: /usr/local/cuda-11.7:/usr/local/cuda-11.6
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 2.1.2+cu121
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.6  (built against CUDA 11.8)
    - Built with CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.16.2+cu121
LMDeploy: 0.3.0+
transformers: 4.40.1
gradio: 4.16.0
fastapi: 0.110.1
pydantic: 2.6.4
triton: 2.1.0

Error traceback

2024-05-09 11:58:07,078 - lmdeploy - ERROR - ValueError: The checkpoint you are trying to load has model type `llava_mistral` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
2024-05-09 11:58:07,078 - lmdeploy - ERROR - <transformers> test failed!
Load model config with transformers==4.40.1 failed. Please make sure model can be loaded with transformers API.
@irexyc
Copy link
Collaborator

irexyc commented May 10, 2024

Currently, only turbomind backend support vision language model, and turbomind backend doesn't support moe architecture, so you can't deploy mistral model.

https://github.com/InternLM/lmdeploy/blob/main/docs/en/supported_models/supported_models.md#models-supported-by-turbomind

@enogalaxies
Copy link
Author

Sorry, we are not using moe architecture mistral，we use llava-v1.6-mistral-7b in hugging face link：
https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b

@lvhan028 lvhan028 mentioned this issue May 10, 2024
Merged
1 task
@enogalaxies
Copy link
Author

when i use pytorch backend the same error occur blew:

lmdeploy serve api_server /data/kai.qiao/model_repo/llava/liuhaotian/llava-v1.6-mistral-7b --server-port 3333 --backend pytorch --model-name mistral

ValueError: The checkpoint you are trying to load has model type llava_mistral but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
2024-05-10 17:58:49,668 - lmdeploy - ERROR - test failed!
Load model config with transformers==4.40.1 failed. Please make sure model can be loaded with transformers API.

@irexyc
Copy link
Collaborator

irexyc commented May 10, 2024 •
edited

Currently, we only support llava llama. To support llava-mistral, the LlavaMistralConfig and LlavaMistralForCausalLM should be used here, but I am not sure the if modification is enough. (including the #1579 modification)

We will check and support the model later if it doesn't use the moe or windows attention that turbomind backend doesn't support currently.

@enogalaxies
Copy link
Author

Thank you very much for your patience

@lvhan028
Copy link
Collaborator

#1579 works for llava_mistral now. May give it a try.

@enogalaxies
Copy link
Author

i compile src and execute always occur error ,like blew

python mistra_7b.py
Fetching 15 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 194180.74it/s]
2024-05-13 18:55:03,879 - lmdeploy - INFO - Using turbomind engine
Traceback (most recent call last):
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/model/llava.py", line 21, in check_llava_install
import llava # noqa: F401
File "/home/kai.qiao/.local/lib/python3.8/site-packages/llava/init.py", line 1, in
from .model import LlavaLlamaForCausalLM
ImportError: cannot import name 'LlavaLlamaForCausalLM' from 'llava.model' (/home/kai.qiao/.local/lib/python3.8/site-packages/llava/model/init.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "mistra_7b.py", line 4, in
pipe = pipeline('liuhaotian/llava-v1.6-mistral-7b',model_name='llava_v1.6_mistral_7b',
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/api.py", line 94, in pipeline
return pipeline_class(model_path,
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/serve/vl_async_engine.py", line 16, in init
self.vl_encoder = ImageEncoder(model_path)
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/engine.py", line 68, in init
self.model = load_vl_model(model_path)
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/model/builder.py", line 36, in load_vl_model
return LlavaVisionModel(model_path, arch=arch)
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/model/llava.py", line 66, in init
self.build_model()
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/model/llava.py", line 71, in build_model
check_llava_install()
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/model/llava.py", line 23, in check_llava_install
raise ImportError(
ImportError: To use LlavaVLModel, please install llava by pip install git+https://github.com/haotian-liu/LLaVA.git --no-deps

@lvhan028
Copy link
Collaborator

ImportError: To use LlavaVLModel, please install llava by pip install git+https://github.com/haotian-liu/LLaVA.git --no-deps
As guided by the log, pip install llava please

@enogalaxies
Copy link
Author

after install the error the same

@github-actions GitHub Actions
Copy link

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

@github-actions github-actions bot added the Stale label May 22, 2024
@github-actions GitHub Actions
Copy link

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting response Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

support mistral and llava_mistral in turbomind lvhan028/lmdeploy
3 participants
@lvhan028 @irexyc @enogalaxies