Autoscaling Llama.cpp server clusters in GPU spot instances.

[jboero]

I wrote up a Terraform module PoC for autoscaling clusters of Llama.cpp server in GCP spot instances with GPUs. Autoscaling is a bit overzealous but it tends to work pretty well. Anyone curious to try I'd love to hear feedback.

https://github.com/jboero/terraform-google-llama-autoscale

[phymbert]

Great, please have a look to:

• kubernetes example #6546

I believe helm approach is more flexible, but we can also introduce a terraform example.

[jboero]

How does the K8s sample work with GPUs though? Each autoscale VM here adds another GPU (or set of GPUs) to the cluster. As I understand most cloud managed Kubernetes options don't support GPUs or customizing kernel/drivers either.

[phymbert]

It works really fine with GCP Kubernetes offer, just add a GPU pool, and classical autoscalling works pretty fine.

[jboero]

Oh cool that's good to know. Do you know what the cost roughly comes out to be?

[phymbert]

Add a GPU pool:

gcloud container node-pools create xx-gpu \
    --cluster xxx \
    --accelerator type=nvidia-l4,count=2,gpu-driver-version=latest \
    --num-nodes=1 \
    --enable-autoscaling  \
    --min-nodes 0 --max-nodes 2 \
    --zone=europe-west3-b \
    --machine-type g2-standard-24

No idea regarding particular pricing override from the machine type pricing.

[jboero]

Oh nice this supports spot instances also (--spot). It's been a while since I've played with GKE but all checks out at the same rates of $0.08/node for Skylake spot instances in europe-west4. Also it seems the GKE layer doesn't add any management costs. This screenshot from a bare minimal 3 node cluster.