Shipping Safely — Canary, Blue-Green & Drift

The Two Halves of Shipping

Shipping an AI service safely is two problems, not one. The first is the cutover: how do you get a new version into production without taking the old one down with it? The second is the slow problem nobody plans for: how do you notice when a model that was perfect at deploy time has quietly rotted three weeks later? The first is a deployment-strategy question. The second is a drift question. An operator who solves only the first ships a broken model carefully.

Choosing a Deployment Strategy

There is no universally best deployment strategy. There is only a trade between blast radius (how many users a bad version touches) and capacity cost (how much extra infrastructure you pay for during the rollout) — with rollback speed as the tiebreaker.

Rolling is the Kubernetes default: replace pods N-at-a-time until the whole fleet runs the new version. It costs no extra capacity, which is why it is the default — but a bad rollout drains across the fleet before you notice, and rolling back is just as slow as rolling forward. Medium blast radius, slow rollback.

Canary routes a small, increasing fraction of traffic to the new version — 1%, then 5%, 25%, 100% — gating each promotion on SLO and burn-rate metrics. A bad version only ever touches the canary slice, and because promotion is metric-gated, a degrading version auto-aborts before it spreads. Small blast radius, fast rollback, moderate cost. For GPU-backed LLM serving this is usually the right answer.

Blue-green stands up a complete second environment (green) running the new version, then flips the router from blue to green in a single cut. Rollback is instant — flip back to blue — but you pay for 2× capacity during the cut, which is brutal when each node is a GPU. All-or-none blast radius, instant rollback, high cost.

Shadow mirrors live traffic to the new version and discards its responses. Users never see shadow output, so the blast radius is zero — but it is a validation tool, not a promotion mechanism. It is how you prove a new model's latency and quality on real production prompts before you ever risk a cutover.

For Knox's single-stack topology — the Mac Mini running Semantic Memory Layer, Mission Control, and the content pipelines — full blue-green is overkill, but the principle holds: never replace the only running instance in place. The deploy pattern of record (docker compose build <service> && docker compose up -d <service>) is effectively a fast rolling swap, and the right hardening is a canary-of-one: bring the new container up, verify its L3 health endpoint proves the artifact is fresh, and only then cut traffic over. Make the rollback a single docker compose up -d of the previous image and the whole maneuver stays cheap.

Now write that strategy as config: a canary rollout that steps traffic up gradually and gates each promotion on an SLO analysis, so a degrading version auto-aborts before it ever reaches the fleet.

The Slow Problem: Drift

You shipped carefully. The canary passed. Three weeks later, quality complaints rise — and nothing changed. No deploy, no alert, no code diff. This is drift, and it is the failure mode unique to ML systems: the model is static, but the world it serves is not.

The detection loop closes around the running service:

1. Serve. The model handles live traffic. Log every (input, prediction) pair with a timestamp — the gateway tees prompt features and outputs to a drift sink.
2. Monitor. Compare the live distribution to the training/reference window. Input drift shows up as a shift in prompt length, token mix, or embedding centroid (measured with PSI or KL divergence). Prediction drift shows up as a shift in refusal rate, output length, or confidence/entropy.
3. Threshold. Is the divergence past the alarm line and sustained? A single weird hour is noise; require something like PSI > 0.2 for > 6h so you do not page on a blip.
4. Trigger. Act. Gradual drift schedules a retrain on fresh data. A sudden cliff rolls back to last-known-good now, and you go find the upstream change that caused it.

Then the loop closes: the retrained or rolled-back version ships back into Serve, and the reference window resets. Drift detection is continuous, not a one-time gate.

The single most useful fact here: input drift precedes prediction drift. The input distribution shifts first; the degraded predictions are the downstream consequence. Watch the inputs and you get a head start on the failure — you see the cause forming before the effect accumulates. This is exactly the philosophy behind tesseractintelligence.io: detect the distribution shift early, before it becomes a visible failure.

The Operator's Takeaway

Pick a deployment strategy by what rollback costs you: canary for GPU-backed serving because the blast radius is small and a bad version auto-aborts; blue-green when you can afford 2× capacity for an instant flip; shadow to validate a new model on real traffic before risking anything. And then remember that the deploy is not the finish line — it is where the clock starts. Models drift because the world moves and the weights do not. Wire the serve-monitor-threshold-trigger loop, watch input drift as your early-warning signal, and treat retrain-or-rollback as a continuous reflex, not a one-time event.

What's Next

You can now deploy safely and detect drift before it becomes a silent outage. The next lesson is the track's capstone: assemble every layer — containerized, orchestrated, served, observed, and SLO-defined — into a single graded end-to-end build. Every box must be demonstrable on a running system, not just described in a file.