Model Tier Routing: Cost-Efficient Agent Architecture
The three-tier routing framework, prompt caching leverage, and the silent failure mode that lets quota errors erase entire job runs without a trace.
Every dollar your agent fleet spends on model calls is a routing decision. Most of those decisions are made implicitly — the agent was built with one model hardcoded, and that model runs everything. Scoring, analysis, generation, orchestration: same model, same rate, every call.
The three-tier routing framework replaces that default with an explicit policy. Each task type gets the cheapest model that can handle it correctly. The result is a 5-10x cost reduction on the same workload, with no degradation in output quality where it matters.
The Three-Tier Framework
Tier 1 — Haiku (approximately $1/MTok input)
Haiku handles observation, scoring, classification, and high-frequency binary decisions. These are tasks where the answer space is constrained and the model's job is pattern recognition, not reasoning.
Examples from Knox's fleet:
- Signal scoring: given a news item and a watch list, return a relevance score from 0 to 1
- Category classification: tag an event as macro, micro, or noise
- Binary gate decisions: does this content match the alert threshold? Yes or no.
The key property of a Tier 1 task is that getting it wrong is cheap. A misclassified signal gets filtered out. The agent makes another decision in five minutes. There is no compounding consequence from a single error.
Tier 2 — Sonnet (approximately $3/MTok input)
Sonnet handles analysis, reasoning, directive routing, and content generation. These are tasks where the model needs to synthesize multiple inputs into a structured output, and errors have downstream consequences.
Examples:
- Strategy briefs: synthesize 10 signals into a 200-word brief with a recommended action
- Directive routing: given a new task, classify it, assign priority, and route to the correct agent
- Content generation: write a structured report from structured data
Tier 3 — Opus (approximately $5/MTok input)
Opus is for genuine complexity: orchestration-level decisions that involve conditional branching across multiple agents, one-off deep analysis where reasoning depth matters, and situations where Sonnet has demonstrably failed.
The key word is "reserve." Opus should see a small fraction of your total call volume. If Opus is handling more than 10-15% of your fleet's calls, the routing policy needs to be tighter.
The Routing Matrix
The practical implementation is a lookup table keyed on task type:
| Task type | Tier | Rationale |
|---|---|---|
| Signal scoring | Tier 1 | High frequency, constrained output |
| Classification | Tier 1 | Pattern recognition, not reasoning |
| Strategy brief | Tier 2 | Synthesis required, downstream impact |
| Directive generation | Tier 2 | Structured output, moderate complexity |
| Deep analysis | Tier 3 | Reasoning depth required |
| Orchestration | Tier 3 | Multi-agent branching logic |
The observation vs. action rule is the fastest heuristic: if the agent is watching something (reading a feed, scoring an item, flagging an event), use Haiku. If the agent is deciding what to do about what it watched (routing a directive, generating a response, changing system state), use Sonnet.
This single rule correctly routes the majority of fleet calls. The routing matrix handles the edge cases.
Prompt Caching: The Biggest Single Lever
Model cost gets most of the attention. Prompt caching gets far less. This is a mistake.
A fleet agent with a stable 5,000-token system prompt makes that call dozens of times per day. (Mind the minimum cacheable prefix — roughly 2,048 tokens on Sonnet and 4,096 on Haiku and Opus; a prompt below the threshold silently does not cache at all.) Without caching, every call pays the full input price. With caching, the first call after a 5-minute idle period is a cache write, billed at a 1.25x premium over the base input price (2x for the 1-hour TTL). Every subsequent call within the TTL is a cache read, costing approximately 10% of the base input price — about 8% of what the write cost.
For a fleet running agents 5-10 times per day with the same system prompt, the savings from caching alone are significant. A 5,000-token system prompt on Sonnet costs $0.015 in base input per call; cached, you pay roughly $0.019 for the occasional write and $0.0015 for every read. Scale that across 50 agents running 8 times per day for 30 days.
The implementation requirement is minimal: use the Anthropic SDK's cache control parameter on the system prompt. The standard cache TTL is 5 minutes after the last call. For agents that run less frequently (hourly summaries, daily reports), use the extended cache option — a 1-hour TTL that keeps the system prompt cached across longer idle periods and preserves the savings even for low-cadence agents. As long as your agents run on a cadence shorter than their cache TTL, you maintain near-continuous cache hits.
The Silent Failure Mode
The most dangerous cost routing failure is not overspending. It is silent non-spending — the job that does not run and leaves no trace.
When a Tier 3 model call fails with a quota error (429, insufficient credits, or rate limit), the default agent behavior is often to log an exception and exit. The job record shows as completed. The downstream consumer sees no output and waits. Nobody fires an alert because no alert was wired.
The correct fallback pattern has three steps:
- Opus call fails → catch the exception, log the model ID and error code
- Retry immediately with Sonnet — for most tasks, Sonnet produces an acceptable output
- If Sonnet also fails → fire an alert and write the job to a retry queue, do not mark it as complete
The retry queue matters. "Alert and stop" without a retry queue means the task is silently dropped once the on-call engineer clears the alert. "Alert and queue" means the task runs successfully once quota resets.
When NOT to Route Down
Tier routing is not always about routing down. Some tasks genuinely require reasoning capability, and routing them to Haiku produces confident-sounding wrong answers.
The failure signature is subtle: Haiku will complete the task, return a well-formatted output, and be wrong in ways that are hard to detect without ground truth. On scoring tasks this is fine — the score is verifiable. On strategic decisions, a confident but incorrect Haiku output can propagate through five downstream agents before the error surfaces.
Route down for: observation, scoring, classification, binary gates.
Do not route down for: multi-step reasoning, cross-source synthesis, decisions with significant downstream consequences, tasks where the error cost exceeds the routing savings.
Building the Routing Layer
The routing layer sits between the orchestrator and the model API. The orchestrator labels each task with a type. The router maps type to model. Neither the orchestrator nor the model knows about the other's details.
This separation matters because routing policies change. When Haiku's capabilities improve, you update the routing table, not every orchestrator that uses Haiku. When a new task type is added, you add one entry to the table. The routing layer is the single place where cost policy is enforced.
Start with a conservative policy — Sonnet as the default, Haiku for clearly bounded tasks, Opus only when you have explicit evidence that Sonnet is insufficient. Measure the cost delta after two weeks. Then tighten the policy based on what you observe.
The goal is a routing hit rate that keeps Tier 3 below 15% of call volume and Tier 1 above 40%. Those numbers are not targets — they are diagnostic signals. If Tier 3 is at 40%, you do not have a routing policy; you have a default.