Ask Knox

The Checklist No One Gives You

Every item on this list represents a real incident that happened to a real AI SaaS product. Not theoretical risks — actual production failures, actual churned users, actual unexpected bills.

Most developers learn these lessons after launch, when the cost of the lesson is measured in user trust and emergency engineering time. The point of building this checklist before launch is to pay those tuition costs in preparation time instead.

Read each item. If you cannot honestly say it is done, you are not ready to charge money.

The Pre-Launch Gate: All Five Sections Green

COST PROTECTION

Two-block prompt caching with a hit rate above 60% (the “Prompt Caching” lesson). If you have not measured your cache hit rate, it is probably low. The two-block structure — stable system context in block one, variable user content in block two — is the mechanism that gets you above 60%. Check it before launch, not after your first unexpected bill.

Token-aware rate limiting with daily budgets per user (the “Auth and Rate Limiting for AI SaaS” lesson). A flat request rate limit (10 requests per minute) does not protect against users who send extremely long inputs. A token budget does. Both should be present.

Per-user daily cost alert set at 10× the average session cost (the “Observability for AI Systems” lesson). This is the automated circuit breaker for billing: one user running up 10× the normal cost triggers an alert before it becomes a significant charge. Set it before launch, not after the first anomalous invoice.

AUTH LAYER

JWT validation happens before the AI call, not after (the “Auth and Rate Limiting for AI SaaS” lesson). This is the ordering that matters. An endpoint that accepts a request, calls the AI, and then validates the JWT has already paid for the AI call by the time it rejects the user. The AI call is expensive. Validate first.

API keys are server-side only. If your Anthropic API key is in your frontend JavaScript bundle, it is compromised. Every browser that loads your page can see it. Check this explicitly.

Subscription tier enforcement is active. A free tier user should not be able to call the same AI endpoint as a Pro tier user with no throttling difference. The tier check happens at the auth layer, not as an afterthought in the AI call path.

OBSERVABILITY

Structured AI call logging (the “Observability for AI Systems” lesson). Every call logs: user ID, session ID, tokens used, cost, latency, mode (from classifier), cache hit status, trace ID. If you cannot answer "what did user X's session last Tuesday cost and why?" you do not have sufficient observability.

Cost dashboard with daily burn visible. You need to see the previous 24 hours of cost at a glance, broken down by user tier. If costs spike, you need to know within an hour, not when the invoice arrives.

Quality metrics tracked: refusal rate, session completion rate. A spike in the refusal rate means your prompt changed something unexpectedly or the underlying model changed. Session completion is the behavioral signal that users are getting what they need.

RESILIENCE

Fallback chain tested manually (the “Error Handling and Graceful Degradation” lesson). Before launch, simulate a 503 by pointing your AI client at a nonexistent endpoint. Confirm the fallback chain activates: retries run, cache is checked, static fallback is served. Do not assume it works — verify it works.

Circuit breaker configured with a threshold and cooldown (the “Error Handling and Graceful Degradation” lesson). The threshold (50% failure rate in 60 seconds) and cooldown (30 seconds before HALF-OPEN) are defaults. Verify they are active, not commented out.

15-second timeout on every AI call. asyncio.wait_for(coro, 15) in Python. A timeout in your HTTP client config in TypeScript/Go. Verify this exists. Users should never wait 45 seconds for an error.

ITERATION READY

Prompt version file in git, regression suite passing (the “System Prompt Versioning and Regression Testing” lesson). The suite takes 30 seconds to run. If it is not passing before launch, you are shipping a known regression.

Feature flag for instant rollback active and tested (the previous lesson). Toggle the flag. Confirm 100% of traffic routes to the control prompt. Toggle it back. This test should take two minutes and should happen before launch day.

The Launch Checklist

Launch day is too late to implement any of the above. Launch day is for monitoring.

Before traffic arrives: Set your cost dashboard to update every 15 minutes. Confirm your alert thresholds will send a notification to the right person at the right number. Send a test alert to verify the notification path works. Know who gets paged at 2am if the circuit breaker opens.

Write the incident runbook. Four scenarios, four documented responses:

API key exposed: revoke immediately in the provider console, rotate to new key, audit recent calls for abuse
Cost spike: identify the user_id or session driving it, apply emergency rate limit, investigate root cause
Quality regression: toggle the prompt feature flag to control, examine recent prompt change log, identify the commit
Provider outage: circuit breaker should have opened automatically; verify fallback chain is serving; check provider status page for ETA

Canary at 1% first. If you have the infrastructure for staged rollout (the previous lesson), launch to 1% of your user base for the first hour. This is not a test — this is real launch traffic. Watch the cost dashboard. Watch the quality metrics. Expand to 100% when the first hour shows nothing unexpected.

Post-Launch Week 1

The work is not done when the product is deployed. The first week of real user traffic is your most important data source.

Daily: Check cache hit rate. If it drops below 50%, something changed about incoming request patterns or your prompt structure. Investigate.

Daily: Check the quality metrics dashboard. Refusal rate, session completion, average response length. Establish baselines in Week 1 — you need them to detect regressions in Week 2 and beyond.

Every other day: Read 10 real user sessions end-to-end. Not metrics — actual conversations. This is how you find failure modes that numbers do not surface: the tone that is slightly off, the format that does not match user expectations, the edge case your test suite did not cover.

The Architecture You Built in Course 2

Step back and look at what this course constructed. From the “Single Agent vs. Multi-Agent” lesson to 523, the architecture that emerged:

Classifier → Router → Specialist Agents — incoming requests are typed, routed to the appropriate prompt and configuration, and handled by a model tuned for that use case. Cheap classification, expensive specialization only where needed.

Three-layer cache stack — response cache for exact matches, semantic cache for near-matches, prefix cache for shared prompt context. The combination achieves 60%+ hit rates that make the economics work.

Token-aware rate limiting — per-user daily budgets that prevent any single user from consuming a disproportionate share of cost, regardless of how they structure their requests.

Structured observability — every AI call is logged with the fields needed to answer "what happened, when, for whom, at what cost." The observability layer is what makes debugging a 5-minute investigation instead of a multi-day mystery.

Circuit breaker + fallback chain — the resilience layer that converts provider outages from user-visible failures into invisible degradations. Users get responses during outages; those responses are just slightly degraded.

Versioned prompts + regression suite — the engineering discipline that makes prompt changes as safe as code changes. Version history, automated testing, rollback in under 5 minutes.

Staged rollout + A/B testing — the deployment rigor that prevents prompt changes from being "ship to everyone and hope for the best."

What Comes Next

For anyone who has not yet validated the business idea behind the product — that work is Course 1: AI as SaaS: Build What People Pay For. The technical architecture in this course is the engine. Course 1 is the destination — the validated demand that tells you the engine is pointed somewhere worth going.

The full picture is on jeremyknox.ai. The article "The AI Gold Rush Is Real — You're Just Mining the Wrong Thing" covers the operator mindset in depth: why demand validation precedes architecture, and why the builders who win start with a painful problem rather than a powerful tool.