Inference Economics
Four levers that decide your inference bill — quantization, prefix caching, spot, and scale-to-zero — ranked by dollar impact against quality risk.
The Bill Is a Design Decision
Inference cost is not a line item you discover at the end of the month — it is the sum of architectural choices you made on the way. Four levers dominate that sum. The discipline is knowing which ones are free wins to turn on immediately, and which are deliberate trades you make only after measuring.
This matters more, not less, on a single-stack setup. On a single-GPU node, squeezing utilization is the priority before adding hardware — one machine serves Semantic Memory Layer, Mission Control, and the content pipelines. Every wasted cycle is a cycle stolen from something else running on the same box.
Free Wins First
Two of the four levers cost you nothing in quality. Turn them on before you touch anything that trades against output.
Prefix caching is the first. Any workload with a shared leading prompt — a system prompt that defines an agent's tools, a RAG template, a few-shot preamble — recomputes that prefix's KV cache on every request unless you cache it. With a 2,000-token shared prefix across thousands of requests, you are paying for the same compute thousands of times. Prefix caching reuses the KV blocks for the shared prefix so each request only computes its unique suffix.
Scale-to-zero is the second. Bursty traffic means idle capacity, and idle GPU is pure waste. Scaling the service to zero replicas when no traffic arrives means the idle hours cost nothing; the only price is a cold-start latency on the first request after idle — acceptable for internal, bursty workloads.
The Quantization Trade — Made on Purpose
Quantization is the lever everyone reaches for first and should reach for nearly last. It works by storing model weights in fewer bits — FP16 → INT8 → INT4 — which shrinks GPU memory and lets a bigger model fit on the same card. The catch is that fewer bits means less precision, and less precision means some quality loss.
The curves are not symmetric, and that asymmetry is the entire opportunity. FP16 → INT8 roughly halves memory for around 2% quality loss — usually imperceptible, and the default-safe trade. INT8 → INT4 halves memory again but the quality drop is steeper and measurable. Reach for INT4 only when fitting the model on the GPU you have is the binding constraint, and always evaluate on your own traffic before shipping it. Blind-quantizing a model you have not measured is shipping a worse product to save memory you might not even need.
Spot — Cheap Hardware With a Catch
Spot or preemptible GPU capacity runs 60–80% below on-demand rates because the provider can reclaim it with little notice. That makes it excellent for interruption-tolerant work — batch inference, embedding jobs, anything that can checkpoint and resume — and risky for a latency-sensitive interactive endpoint that cannot afford a mid-request eviction. The trade is real money against operational complexity: you must build drain-and-restart handling to use it safely.
The Order That Saves the Most
Put the four levers in sequence and a playbook falls out. Start with prefix caching and scale-to-zero — both free, both pure waste correction. Then, if you still need to cut cost or fit a larger model, quantize to INT8 and evaluate. Only if fitting on the GPU is still the binding constraint do you go to INT4, and only with traffic evaluation behind it. Spot enters the moment you have interruption-tolerant batch work to run on it. The savings compound, but only if you take them in the order of least quality risk first.
What's Next
You can serve, ground, and cost-control an inference service. The next lesson makes the whole thing reproducible: infrastructure as code — the write → plan → apply → state loop, remote state with locking, and a Terraform module that provisions the GPU node pool your service runs on.