Prepare model for deployment to Private Vertex AI endpoint

[BriianPowell]

Hello, I have a use-case where I'd like to deploy this model to a private Vertex AI endpoint, is there any documentation/literature around how to do that?

[pkgoogle]

Hi @BriianPowell, welcome, does this answer your question? https://cloud.google.com/vertex-ai/generative-ai/docs/open-models/use-gemma Let us know if it does not.

[BriianPowell]

Hey there, @pkgoogle thanks for getting back to me. Currently there is a no way to deploy the Gemma version from the Model Garden to a private Vertex AI endpoint. I have some constraints on my project where the Vertex AI Endpoint needs to be attached to a VPC.

I am thinking about following this guide from one of the links located in that article.

Would this work in conjunction with the image that's being created in this repo or are they totally different things?

[pkgoogle]

Hi @BriianPowell, I believe it will work -- can you give it a try and see if you run into any issues? Thanks.

[BriianPowell]

@pkgoogle Just thinking here, but my understanding is that the current state of this project doesn't allow hosting the image as a api_server? I think I may have to go the vllm or hex-llm route