Semantic Cache vs. Response Cache vs. Prefix Cache
Three caches. Each catches a different class of repeat. Most AI products add one and wonder why costs are still high. Here is how all three work, what each one catches, and how to stack them for maximum savings.
Most Engineers Add One Cache and Wonder Why Costs Are Still High
When a developer adds "caching" to an AI product, they usually mean: hash the input, store the output, return it on a match. This is a response cache. It is the right first step. It is also incomplete.
There are three distinct caches that catch three different classes of repeat requests. Each costs different amounts to implement. Each pays off differently depending on your traffic pattern. A system with all three runs at a fraction of the cost of a system with only one — and the difference compounds as traffic grows.
The Response Cache: Exact Match
The response cache is the simplest. Hash the input, store the output. On an identical input, return the stored output. Zero AI cost, sub-millisecond latency.
This cache excels on FAQ-style traffic: "What are your refund terms?", "How do I reset my password?", "What is the difference between the free and pro plan?" These questions get asked verbatim, over and over. A response cache catches all of them after the first.
The miss rate on conversational traffic is high — roughly 80%. Most users do not ask the exact same question twice. But the 20% who do, they are free. The implementation is 5 lines:
const cache = new Map<string, string>()
const key = input
if (cache.has(key)) return cache.get(key)
const result = await callAI(input)
cache.set(key, result)
return result
In production, replace the Map with Redis for multi-instance support and TTL expiration. The in-memory version works fine for a single-server MVP.
The Semantic Cache: Similarity Match
First, the one concept this section rests on: an embedding. An embedding is just a way of turning a piece of text into a list of numbers (a vector) so that text with similar meaning lands at nearby points, while unrelated text lands far apart. You compare two embeddings with cosine similarity, a score from 0 (unrelated) to 1 (identical meaning) — so 0.95 means "almost the same thing said differently." A vector store is the database that holds these vectors and finds the nearest ones quickly.
The semantic cache is the response cache's smarter sibling. Instead of checking for exact string equality, it embeds the input query and searches for nearest neighbors in a vector store. If the nearest neighbor exceeds a similarity threshold (typically 0.95 cosine similarity), return the stored answer.
This catches paraphrases. "How do I reset my password", "forgot my password", and "can't log in, what should I do" are three different strings. They embed to vectors that are within 0.95 similarity of each other. The semantic cache serves all three from a single stored answer.
The implementation requires two components that most AI SaaS products already need: an embedding model and a vector store. Semantic Memory Layer — Knox's knowledge OS across all connected services — runs a local MiniLM embedding model at zero marginal cost. Every query to Semantic Memory Layer goes through semantic search before hitting the LLM. The cache hit for "clear the stale trades.db.lock" returns the same answer as "how do I reset the trade lock" — one embed call saves one full inference call.
The threshold is tunable. At 0.95 (the default), you get accurate matches with few false positives. Lowering to 0.85 increases hit rate at the cost of occasionally returning a slightly wrong cached answer. For support tools, 0.90-0.95 is the right range. For anything where precision matters more than cost, stay at 0.95 or higher.
The Prefix Cache: Provider-Side, Automatic
The prefix cache runs provider-side — you never store anything yourself — but how you turn it on differs by provider. On Anthropic, it is explicit opt-in: you mark the stable prompt blocks with cache_control: { type: 'ephemeral' } breakpoints, exactly as lesson 515 walked through. On OpenAI, prefix caching is automatic for qualifying prompts. Do not confuse the two — an Anthropic integration without cache_control markers gets no prefix caching at all.
The mechanism is the same everywhere: the first time you send a request with a given system prompt prefix, the provider caches the KV state for that prefix in their infrastructure. On subsequent requests with the same prefix, they skip recomputing that KV state and charge a discounted rate for the cached portion. The discount differs by provider: Anthropic charges roughly 10% of the normal token rate on cache reads; OpenAI's automatic caching discounts cached input by roughly 50%; Google's context caching has its own pricing model with separate storage costs.
The savings are substantial. A 2,000-token system prompt at $0.003/1K costs $0.006 per call at full price. With Anthropic's 90% cache-read discount, it costs $0.0006 per call. At 20,000 calls/day, that is $108/day in savings — from one cache_control marker on the system prompt block. One detail to budget for: the request that first writes the prefix to the cache is billed at roughly 1.25x the base input rate (a 25% surcharge on cache_creation_input_tokens), so the discount applies to reads, not the initial write. Across steady traffic this is noise — you break even after a single hit — but it explains the small cache_creation line on your first invoice.
The single requirement: your prefix must be stable. Every character change invalidates the cache. Design your system prompt to have a stable, immutable header (the part that defines the AI's role and constraints) and a dynamic tail (the part that changes per request, like the user's context). Only the stable header qualifies for prefix caching.
How to Stack Them
The correct check order is: response cache first (sub-millisecond, free), semantic cache second (~10ms, embed cost only), LLM call third (500-2,000ms, full inference cost). Prefix cache fires automatically inside the LLM call — no check required.
On a cache miss, populate both caches on the way back up. The response cache stores the exact input-output pair. The semantic cache stores the embedding vector and the output. Future queries that match either the exact string or the semantic neighborhood will be served without hitting the LLM.
At 10,000 calls/day, the two caches apply in sequence. The response cache fires first at a 20% hit rate — 2,000 calls served, 8,000 fall through. The semantic cache then catches 40% of those remaining 8,000 — another 3,200 served. That leaves roughly 4,800 calls reaching the LLM and about 5,200 served from cache at near-zero cost. Add prefix caching and the ~4,800 calls that do reach the LLM cost 90% less on the system prompt portion. The compounding effect is significant.
For implementation patterns and the Semantic Memory Layer semantic search architecture that powers Knox's production systems, jeremyknox.ai covers the full stack. The InDecision platform uses all three cache layers in its AI recommendation engine.