Add docs of support new vl model #1332
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lvhan028
reviewed
Mar 25, 2024
| @@ -0,0 +1,131 @@ | |||
| # lmdeploy.vl 新模型支持 | |||
lvhan028
reviewed
Mar 25, 2024
| @@ -0,0 +1,131 @@ | |||
| # lmdeploy.vl 新模型支持 | |||
|
|
|||
| 目前，有一批 VLM 模型采用如下图所示的架构。图片经过 Vison Encoder 得到图片特征，之后经过 Projection 映射到文本的特征空间。最后，将图片特征与文本特征拼接后送入 LLM 进行推理。这类 VLM 模型有一个特点，即拼接后的特征送入 LLM 推理时并不区分特征的类型，两种特征之间没有交互计算。 | |||
There was a problem hiding this comment.
目前，LMDeploy 支持类似 LLaVA 架构的多模态视觉模型。如下图所示，在这种架构中，图片经过 ...
lvhan028
reviewed
Mar 25, 2024
|
|
||
| 对于此类架构的模型，使用 LMDeploy 可以很方便的添加新模型的支持。 | ||
|
|
||
| ## 模型支持 |
There was a problem hiding this comment.
我感觉 紧跟着 "模型支持" 的文字部分，可以融入到前文。这样，VisonModel, VLChatTemplateWrapper 可以作为 H2 标题。
lvhan028
reviewed
Mar 25, 2024
|
|
||
| > \[!NOTE\] | ||
| > | ||
| > 一般 VLM 模型有一个对应的不带图片输入的LLM 模型，如 Qwen-VL-Chat 和 Qwen-7B-Chat，请先确保这个 LLM 模型可以被 TurboMind 引擎推理，或者他的模型结构和 TurboMind 已支持的模型结构相同。 |
lvhan028
reviewed
Mar 25, 2024
|
|
||
| def build_model(self): | ||
| # init an empty model | ||
| with init_empty_weights(): |
There was a problem hiding this comment.
init_empty_weights() 源自哪里？
lvhan028
reviewed
Mar 25, 2024
|
|
||
| # move model to cpu and load weight | ||
| model.to_empty(device='cpu') | ||
| load_model_from_weight_files(model, self.model_path) |
There was a problem hiding this comment.
load_model_from_weight_files 是可以通用的吗
lvhan028
reviewed
Mar 25, 2024
|
|
||
| 下面以 Qwen/Qwen-VL-Chat 模型为例，展示如何使用 LMDeploy 添加这类模型的支持。 | ||
|
|
||
| ### VisonModel |
lvhan028
reviewed
Mar 25, 2024
| 添加新的视觉模型，主要需要修改两个地方: | ||
|
|
||
| 1. 抽取 VLM 模型对应的 vision 模型，并实现 `forward` 特征抽取函数 | ||
| 2. 修改 `load_vl_model` 函数, 使 VLM 模型在加载时可以找到对应的 VisionModel 模型 |
There was a problem hiding this comment.
下面的例子并没有 load_vl_model 函数
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
add docs of support-new-vl-model