GPU Scheduling & Autoscaling

Two Problems Unique to Serving Models

The previous lesson got your service running in a cluster. An LLM service adds two requirements a normal web service doesn't have: it needs a GPU, and it needs to scale on a signal CPU can't give you. This lesson covers both, and the mechanism behind each turns out to be the same reconcile loop you already understand.

A GPU is not like CPU and memory. CPU is divisible and over-committable — ten Pods can share a core and the kernel time-slices them. A GPU, in the standard Kubernetes model, is integer and non-shareable: a Pod that requests one GPU holds the entire device for its lifetime. You don't get half a GPU, and you can't over-commit it. That single fact shapes how you schedule, how you budget, and why an unrequested GPU silently becomes no GPU at all.

How a Pod Gets a GPU

Four things have to line up for a Pod to land on a GPU and use it.

First, the NVIDIA device plugin — a DaemonSet running on every GPU node — discovers the physical GPUs and advertises nvidia.com/gpu to the kubelet as a schedulable resource. Now the node's capacity says nvidia.com/gpu: 1. Second, your Pod requests it in the spec; with no request, you get the CPU-node outcome above. Third, the scheduler filters to nodes with a free GPU and binds the Pod — and because a GPU is integer and non-shareable, that node's GPU is now fully consumed. Fourth, node pools and taints keep the economics sane: GPU nodes carry a taint, so only Pods that tolerate it land there, which stops cheap CPU-only workloads from squatting on expensive GPU hardware.

The subtle correctness detail: put the GPU under limits. Kubernetes copies an extension-resource limit to the request automatically, and since a GPU cannot be over-committed, request and limit must be equal anyway. Specifying it under limits is the canonical form.

Autoscaling: The HPA Control Loop

Now the second problem. Load isn't constant, and you don't want a human resizing the Deployment at 3am. The Horizontal Pod Autoscaler does it — and it is, once again, a reconcile loop.

The loop runs every 15 seconds: observe the metric, compare it to your target, decide the desired replica count with desired = ceil(currentReplicas × currentMetric / targetMetric), and act by patching the Deployment's replica count. The ReplicaSet then spins up or tears down Pods, and the scheduler places the new ones. ACT feeds back into OBSERVE; the loop settles when desired equals current. Three guardrails keep it sane: minReplicas (a floor so a latency-sensitive service never scales to zero), maxReplicas (a budget ceiling), and a stabilizationWindow (a cooldown so noisy metrics don't make it flap).

The single most important decision is which metric. The default examples all use CPU utilization, and for a GPU-bound LLM service CPU is the wrong choice — it lags. The GPU saturates, requests pile up in the queue, time-to-first-token climbs, and CPU sits at 40% the whole time. Scale on the signal that leads the outage: queue depth or time-to-first-token, exposed as a custom metric through a Prometheus adapter.

Now write both. The challenge below gives you a stub manifest. Add the GPU resource request to the Deployment and define an HPA that scales on a custom metric with sane bounds.

What's Next

Your service now claims the GPU it needs and scales itself on a metric that predicts pain. The next lesson puts a real LLM behind it with vLLM — continuous batching, serving profiles, and the architecture that turns one GPU into many concurrent generations, so the replicas your HPA spins up are each pulling their full weight.