How can i update to vLLM v0.4.1 for llama3 support ?

[Lhemamou]

Hello everyone,

I would like to update the vLLM version to v0.4.1 in order to get access to LLAMA3 but i don't know how modify the fork runpod/vllm-fork-for-sls-worker. Could you please guide me ? Happy to help in some way!

[nuckcrews]

+1

[nerdylive123]

+1, looking to figure this out soon

[houmie]

Same issue here. There is a blocking bug on Llama3 that has been fixed with v0.4.1.

[arthrod]

Pretty please

[alpayariyak]

Hi all, thank you for raising this issue! I have just merged the vLLM 0.4.2 update into main, you can use it by changing your Docker image in your endpoint from runpod/worker-vllm:stable-cudaX.X.X to runpod/worker-vllm:dev-cudaX.X.X. From my testing thus far, everything seems in order, but if you notice any issues, please let me know. After an initial test period, I'll release the update officially to replace the default stable images. Thanks all!