The Minimum Viable AI Architecture
The four non-negotiable layers every AI product needs before charging money — with working TypeScript code you can drop into production today.
What You Actually Need Before Charging Money
The previous lesson covered the five failure modes that break AI products in production. This lesson gives you the four layers that address four of them — runaway costs, redundant spend, ungraceful failures, and prompt drift — the minimum viable AI architecture you should have in place before you charge a single user.
The fifth failure mode, auth hardening, is deliberately not one of these four layers — not because it is optional, but because it is delegated. Lesson 501's priority order still holds: auth comes first. The assumption here is that you are using a managed auth provider (Supabase, Clerk, Auth0) rather than rolling your own, which handles sessions, password storage, and token security for you. Treat it as Layer 0, with two non-negotiable checks before launch: your AI provider API key never appears in client-side code, and every AI endpoint verifies the authenticated user server-side before doing any work — the rate limiter in Layer 1 is only as good as the identity it keys on.
Note the word "minimum." This is not the complete architecture for a scaled AI product. It is the baseline below which you are not production-ready. Everything beyond these four layers — semantic caching, vector databases, multi-agent orchestration, fine-tuned models — is a Phase 2 investment that pays off once you have paying users and real usage data to optimize against.
The Four Non-Negotiables
Layer 1: Per-user rate limiting. Before any request reaches your AI API, check whether the requesting user has exceeded their daily (or hourly) cap. If yes, return a 429 immediately — no API call is made, no cost is incurred. The rate limit protects you from two scenarios: external abuse (bots, scrapers) and internal abuse (your own generous free tier being exploited by power users who will never convert to paid).
The implementation is simpler than most developers expect. At minimum, a Map<string, number> tracking requests per userId is sufficient for a single-server app. Upstash Redis is the right upgrade path for production — a managed, serverless Redis with a free tier that handles the multi-instance case cleanly.
Layer 2: Response caching. After the rate limiter passes, check whether you have already answered this request. Use the user's message as the cache key. On a cache hit, return the stored response with no API call. On a miss, continue to the AI call and store the result before returning it.
In most AI applications, 30–50% of requests are repeat or near-repeat questions. This layer eliminates that entire cost fraction. An in-memory Map<string, string> is the minimum viable implementation. It does not survive server restarts and does not work across multiple instances — both acceptable tradeoffs for a single-server MVP.
Layer 3: Error fallback. Wrap every AI API call in a try/catch. When the call fails, do two things: return a graceful message to the user ("Service temporarily unavailable — please try again"), and log the failure with structured JSON that includes the userId, the error, and the timestamp. The graceful message keeps the user experience intact. The log gives you the data to investigate later.
This is the layer most developers skip because it feels like defensive coding for an unlikely scenario. The AI API call fails more often than you expect. Rate limits, brief outages, timeout events — these happen. Without a fallback, users see unhandled exceptions. With one, they see a polite message and you see a log entry.
Layer 4: Basic observability. After every successful AI response, log a structured JSON record: userId, message length, response length, cost estimate, latency in milliseconds, and whether the request was served from cache. This is how you catch prompt drift (outputs change in size or structure), runaway costs (cost per request increases), and abuse (a single userId dominates the logs).
This is not an analytics platform. It is a console.log call with structured fields. Searchable logs in Vercel, Railway, or any standard hosting platform are sufficient for the minimum viable version.
Why Not a Vector Database or Fine-Tuning?
Both are powerful. Neither is part of the minimum viable architecture.
A vector database enables semantic search, retrieval-augmented generation, and similarity matching across large document sets. You need it when your product requires retrieval from custom knowledge — not as baseline infrastructure for every AI product.
Fine-tuning produces a model specifically optimized for your use case and data. You need it when prompt engineering has reached its limits and you have enough high-quality training examples to justify the cost and complexity. That is a Phase 2 or Phase 3 decision.
Multi-agent orchestration — chains of AI calls where each agent hands off to the next — is architecturally powerful and right for specific products. It multiplies the cost and complexity of every layer above. You want these four layers working first.
Building It: The Code
The BuildChallenge below asks you to wire all four layers into a realistic Next.js API route. The starter code gives you the bare endpoint. Your task is to add the rate limiter, the cache, the structured logging, and the graceful error fallback.
The solution is about 50 lines. Every line is doing something specific. When you understand why each line is there, you understand why the minimum viable architecture is the shape it is.
For deeper reading on production AI product patterns, jeremyknox.ai covers the next layers — semantic caching, multi-tier rate limiting, and structured observability pipelines.