How to finetune and serve the finetuned model using VLLM like tools

[monksgoyal]

Hi I want to finetune it on my custom OCR tasks, can you share finetuning script and also help in telling the best way to serve it very very fast.

[xs818818]

同问

[brucethemoose]

SGlang support would be excellent, as it doesn't seem to work out of the box.

[brucethemoose]

Made an issue for it here as well: sgl-project/sglang#297

SGlang already supports llava, so perhaps adding deepseek would not be as huge of a jump as vllm (which currently does not support vision models).