Ideal Infrastructure

[jordanparker6]

What would be an ideal infrastructure set up in the cloud?

Is this more designed for K8?

GPUs seem to complicate hosting as we can't just use Cloud Run.

It would be nice to understand the minimum infrastructure commitment required to get something performant and which pathways would provide scalable compute. Again, GPUs complicate things it seems.

[SweatyCrayfish]

Your "ideal" infrastructure will very with respect to your needs and your current infrastructure. For example, AWS and hugigngface have been working together for quite some time. In this case you would not even need to download your LLM to your environment. On the other hands, if you already use OpenAI than Azure is a better choice. Overall, if you work at Biotech or Bio related industries you most likely want to use AWS since most Bio related companies already using AWS and have solid infrastructure attached to that. If you work for a Finance than you probably using Azure. Finally, if your company is consumer related than you probably using Google Cloud. My suggestion is that whatever you use, you want to work on Linux OS because there are many tools attached to that.

[euroblaze]

And which self-hosted options would you recommend?

Of similar interest are private clouds for LLM inferencing.

We are over here in Europe and most companies don't default to clouds due to GDPR _and sovereignty reasons.