Request Design and Token Budgeting
Every token costs money. Request design is cost control. Here is the anatomy of a real request, three strategies for managing conversation history, and the context compression pattern that cuts your bill by two-thirds before you touch the AI call.
The Cost Is in the Context, Not the Question
Most engineers think about token costs at the wrong time — after they see the bill. By then, the architecture is set, the prompts are deployed, and every optimization requires rework.
The engineers who build profitable AI SaaS products think about tokens before the first line of code. They understand that a token is a unit of money, and that most of the tokens in any AI request are not the user's question — they are everything else you send along with it.
Here is the anatomy of a real request.
The numbers above are not hypothetical. A typical AI SaaS product sends: a system prompt that explains who the AI is and what it should do (2,000 tokens), a conversation history that preserves context across turns (1,500 tokens), retrieved content from a knowledge base or user documents (1,000 tokens), and finally the user's actual message (100 tokens). The output might be 900 tokens.
Total: 5,500 tokens. The user's question is less than 2% of it.
At $0.003 per 1,000 input tokens and $0.015 per 1,000 output tokens, that is $0.0138 in input costs plus $0.0135 in output costs — roughly $0.027 per call. Scale that to 10,000 daily requests and you are spending $270 per day, $8,100 per month.
The system prompt and conversation history together are 63% of your input tokens — and because output is priced at 5x the input rate, that works out to roughly 38% of the total per-call cost. Both are compressible. Most engineers never touch them.
Input Token Anatomy
The four components of your input context each have different compression characteristics.
System prompt is fixed overhead. It is the same on every request. If you are paying full price for it every time, you are overpaying. The right move is provider-side prefix caching. On Anthropic, you opt in by marking the prompt with explicit cache_control breakpoints and pay roughly 10% of the normal rate on cache reads; OpenAI applies automatic prefix caching with roughly a 50% discount on cached input. On both, the requirement is the same: your prefix must be stable. If your system prompt changes on every request, you cannot cache it. Design it to be stable.
Conversation history is the fastest-growing component. It starts at zero and grows with every user turn. At 10 turns it is a moderate cost. At 50 turns it is the dominant cost. Unmanaged conversation history is the most common cause of AI SaaS margin problems.
Retrieved context is the most controllable. When you pull from a knowledge base, you control how much you retrieve. Chunking your documents and retrieving only the most relevant chunks (top-3 out of 100) is not an optimization — it is correct architecture. Retrieving everything and letting the AI sort it out is expensive and less accurate.
User message is the one thing you genuinely cannot compress. It is also, ironically, the smallest component.
Output is priced at 5x the input rate on most providers. This makes max_tokens a financial decision, not just a UX one.
Conversation History Management
There are three strategies for managing conversation history. Each is appropriate in different contexts.
Sliding window is the simplest. Keep the last N turns verbatim — typically 5 to 10. Everything older is dropped. Implementation is 3 lines of code. The tradeoff: early context is lost. For most support and assistant use cases, the last 5 turns contain everything the AI needs. Users rarely reference something they said 20 messages ago.
Summarization is the semantic middle ground. When history exceeds a threshold (say, 15 turns), summarize the oldest 10 turns into a paragraph. Keep that paragraph plus the recent turns verbatim. The AI retains the gist of early decisions and facts without the full token cost. The cost of the summarization call is recovered in 2 subsequent requests.
Selective retention is the most efficient and the hardest to implement. Classify each turn: does it contain a decision, a user preference, a fact, or a key constraint? If yes, retain it. If not, drop it. A 50-turn conversation might compress to 8 turns of genuinely load-bearing context. This requires a classifier — either a lightweight AI call or a rules-based system.
For a new product, start with sliding window. You can always upgrade to summarization once you have data showing that early context matters for your users.
The Context Compression Decision Flow
Before sending any AI request, run this decision tree. It takes less than a millisecond in code and can cut your token cost by 60–70%.
The three-node decision flow does the following. First, it checks whether conversation history exceeds 10 turns. If yes, it summarizes the older turns before sending. Second, it checks whether total estimated input exceeds 3,000 tokens. If yes, it trims retrieved context to the top-3 most relevant chunks. Third, for support and QA tasks (not code generation), it sets max_tokens to 300 — users in a support context do not read 4,096-token responses, and you are paying for every token whether they read it or not.
The cost difference at 10,000 daily calls is $90/day optimized versus $270/day unoptimized — $5,400/month in savings from three decision nodes.
Per-User Token Budgets
Standard rate limiters count requests. A user who sends 10 messages with 100-token histories and a user who sends 10 messages with 5,000-token histories both count as "10 requests." They do not cost the same — the second user costs 50x more.
The correct primitive is a daily token budget per user, not a daily request count. Your free tier might allow 10,000 tokens per day. Your pro tier might allow 500,000. Enforce these budgets by tracking tokens_in + tokens_out against a daily counter per user_id with a 24-hour TTL.
Knox's Mission Control does this across all its AI-connected services. When the daily token budget for a service spikes unexpectedly, it fires an alert within 2 hours — not at month-end when the bill arrives.
The Numbers That Matter
At 10,000 daily requests with no optimization: $270/day, $8,100/month. At 60% reduction via the three-node decision flow: $90/day, $2,700/month. The difference pays for two full-time engineers.
The math changes based on your pricing, your volume, and your provider — but the principle does not. Every token you send is a cost decision. The question is whether you are making it deliberately or accidentally.
For a deeper look at how caching interacts with token design, Mission Control's per-service cost dashboard lives at jeremyknox.ai — it shows the real numbers from a production AI system with 40+ connected services.