Serving an LLM with vLLM

From "It Loads" to "It Serves"

Loading a model into memory and getting one completion back is a solved problem — transformers does it in ten lines. Serving that model to many concurrent callers, on one GPU, at a cost that does not bankrupt you, is the actual job. This lesson is about the engine that closes that gap: vLLM.

When Mission Control needs an answer, it does not care how the model is loaded. It POSTs to an OpenAI-compatible endpoint and expects a completion. The interesting engineering is entirely on the serving side — and the two ideas that make it work are PagedAttention and continuous batching.

PagedAttention — The KV Cache as Virtual Memory

Every token a model generates produces key and value tensors that must be kept around for the rest of the sequence. That is the KV cache, and it is the dominant consumer of GPU memory during inference. The naive approach reserves one contiguous block per request, sized for the maximum possible length. The result is brutal: a request that asks for 50 tokens but might ask for 4,000 reserves space for 4,000, and 98% of it sits empty.

PagedAttention borrows the operating-system trick of paging. The KV cache becomes a pool of fixed-size blocks. A sequence is allocated blocks on demand as it generates, and blocks return to the pool when it finishes. There is no per-request contiguous reservation, so fragmentation drops from 60–80% to under 4%.

Continuous Batching — Stop Waiting for the Slowest

The second idea is about when the GPU does its work. Static batching collects N requests, runs them together until the last one finishes, then starts the next batch. The problem is obvious once you see it: a request that finishes early leaves its GPU slot idle while the batch waits on the slowest sequence, and a request that arrives mid-batch waits for the whole batch to clear.

Continuous batching operates at the iteration level instead. Every decode step, the scheduler admits any waiting requests into freed slots and evicts any that finished. The GPU stays saturated.

On mixed-length conversational traffic this is typically a 2–4x throughput gain on identical hardware. It is the single highest-leverage knob you turn before you ever consider adding a second GPU — and on a single-stack setup with one GPU to its name, there is no second GPU to add, so squeezing the one you have is the whole game. (One caveat for Apple-silicon single stacks like Knox's Mac Mini: vLLM itself targets NVIDIA/AMD GPUs and has no Metal backend, so the engine there would be llama.cpp or MLX — but the continuous-batching principle is the same, and a single NVIDIA GPU node is the cleanest place to run vLLM as described.)

Picking a Serving Profile

There is no single best configuration — there is the right one for your workload. The three knobs that matter most:

• max_num_seqs — the ceiling on concurrent sequences. Higher means more throughput, until the KV cache runs out and the scheduler starts preempting.
• max_model_len — the maximum context window you will admit. Long contexts eat KV cache, so this trades directly against concurrency.
• gpu_memory_utilization — how much of the card vLLM may claim for the KV-cache pool (typically 0.85–0.95).

Three archetypes fall out of these knobs. A throughput profile (batch summarization, embedding-style bulk jobs) maxes max_num_seqs and accepts higher per-request latency. A latency profile (an interactive chat surface) caps concurrency lower so each token comes back fast. A long-context profile (32K-token RAG) raises max_model_len and deliberately lowers max_num_seqs because each request is cache-hungry.

What's Next

You can now serve a model efficiently. The next lesson builds the layer on top: RAG in production — the retrieve → rerank → cache → generate path, grounded in how Semantic Memory Layer serves knowledge with local MiniLM embeddings and ChromaDB. The serving engine you tuned here is the final hop of that pipeline.