Most AI agents are not expensive because the model is expensive. They are expensive because they ask the same question over and over again.
An agent that re-reads the same tool schema on every loop, re-fetches the same document for every one of a hundred parallel branches, and re-embeds the same sentence each time a retry fires is not thinking — it is paying rent on its own amnesia. The model bill is the symptom. Redundancy is the disease.
This is not a “save 15% on your OpenAI invoice” article. It is a map of every seam in an agent’s request path where a token is spent twice, and the concrete patterns — exact-match caching, semantic caching, request coalescing, and idempotency-keyed deduplication — that close those seams. If you already know what an idempotency key is, you are the intended reader. We are going to talk about where each mechanism actually lives, when each one lies to you, and the failure modes that quietly reintroduce the cost you thought you eliminated.
1. The Real Cost Isn’t the Model — It’s the Redundancy
Do the arithmetic before you touch a single cache, because the number is what justifies the work.
Take a retrieval agent handling a support queue. Each turn it re-reads the system prompt (say 1,200 tokens), re-embeds the user’s question, and re-fetches the top-k chunks. Across a 500-request day, that is roughly 600,000 tokens of input that are byte-for-byte identical to the previous request. At current frontier-model input pricing, that identical, repeated context is typically a third to half of the total input spend — before a single output token is generated.
Output is worse, because agents retry. A tool call that times out is retried; an agent that re-plans after a failed step regenerates a near-identical preamble. In a multi-step agent, a single user turn routinely produces 4–8 model calls, and a meaningful share of them are the same call with a different timestamp.
The rule that follows: the cheapest token is the one you never send. Every mechanism in this article is a way of moving a token from the “sent again” column to the “sent once” column. Cache hit rate is not a vanity metric — it is the difference between an agent that scales linearly with usage and one that scales linearly with uniqueness.
2. Where Caching Actually Lives in an Agent’s Request Path
Caching in an agent is not one cache. It is several, and they sit at different seams. Treating them as interchangeable is how you build a cache that costs more than it saves.
Map the request lifecycle and you find six distinct seams:
- Tool schema and system prompt. The largest single chunk of repeated input in most agents. It changes rarely, so it should be cached aggressively — but it is also mutated (injected with retrieved context), so you cache the stable prefix, not the assembled whole.
- Embedding generation. If you embed the same text twice — once on ingest, once on query, or once per branch — you are paying for a deterministic computation. Embeddings are a pure function of their input; cache them by content hash.
- Retrieval / tool output. A document fetched for branch A is the same document fetched for branch B. Cache tool results keyed by the tool’s arguments.
- The completion itself. Exact-match and semantic caches live here, keyed on the prompt or its embedding.
- In-flight requests. Two concurrent branches asking the same question should coalesce into one upstream call (Section 4).
- The side effect. The thing that actually changes the world — the charge, the write, the send. This is where idempotency keys live, and where “deduplication” means something different from “caching.”
The key insight: caching is safe at the seams where the data is deterministic, and dangerous at the seams where the data is derived. A cached tool result is safe if the underlying record hasn’t changed. A cached completion is safe if the prompt and the world haven’t changed. Every invalidation rule in Section 5 is just a precise statement of which of those two things changed.
3. Exact-Match vs. Semantic Caching — and When Each Lies to You
There are two ways to decide whether a new request is “the same” as an old one, and they make opposite kinds of mistakes.
Exact-match caching hashes the raw prompt and reuses the response only on a byte-for-byte match. It is safe, deterministic, and nearly free to operate — a key-value lookup. Its failure mode is misses: natural-language users almost never phrase the same intent the same way twice, so an exact cache on a conversational agent idles at a single-digit hit rate while looking correct. It lies by under-performing, quietly, while you assume it is working.
Semantic caching embeds the query and returns the nearest cached response above a similarity threshold — typically cosine similarity in the 0.85–0.95 range. It catches paraphrases and looks spectacular on a hit-rate chart. Its failure mode is false hits: a query that is numerically close but semantically wrong gets served confidently. That is not a small bug. Poorly tuned semantic caches have been measured with false-positive rates high enough that the cache is actively serving wrong answers. A semantic cache lies by over-performing, which is worse, because you trust it.
The guidance that survives contact with production:
- Exact-match is the right default for deterministic seams — tool results, schemas, embeddings, anything keyed on a content hash.
- Semantic is only for the conversational seam, where phrasing varies but intent is stable — and only where a wrong answer is cheap and detectable.
- Threshold is a risk dial, not a tuning knob to optimize for hit rate. Set it high (0.92+) for anything where a false hit has consequences — finance, medical, legal. You can go lower only where the cost of a wrong answer approaches zero. Anyone who tells you to “just set it to 0.85 for more hits” is optimizing the wrong variable.
- Semantic caching is a strict superset of exact matching — cosine similarity 1.0 is an exact match. If your stack lets you run semantic as a single layer that also catches exact hits, do that rather than running two layers that duplicate infrastructure and complicate invalidation.
One more lie to watch for: the embedding call itself. If every request embeds the query before checking the cache, you have merely moved the cost from the LLM to the embedder. The embedding model is the dial that decides whether semantic caching is a latency win or a wash — a small, fast embedding model keeps the savings; a heavy one cancels them out.
4. Request Deduplication and Coalescing: Stop Sending the Same Call Twice
Caching answers the question “have I seen this before?” Deduplication answers a different question: “am I about to send this right now, more than once?” They are complementary, and people who implement one without the other leave half the money on the table.
In-flight coalescing (single-flight). When ten parallel branches of an agent all need the same document at the same moment, a naive implementation fires ten identical upstream calls. Coalescing — the single-flight pattern — lets exactly one request proceed while the other nine await its result. Within a single process this is a map of in-flight promises; across instances it requires a distributed lock or a SET NX claim so that only one worker fetches and the rest subscribe to the result. This is also your primary defense against the cache stampede — the thundering herd that forms when a hot key expires and every concurrent request misses at once.
Idempotency keys. The subtlety here is where you put the key. An idempotency key should cover the unit of work you cannot afford to duplicate — the action with an external side effect — not the inference call that produced it. Key the charge, not the thought that led to it. If you key on the LLM call, you will happily deduplicate two identical inferences while still firing two charges, which is exactly backwards.
The harder problem is that an agent paraphrases its own request on retry, which breaks the byte-for-byte assumption idempotency keys rely on. The fix is to make the key an orchestration contract: generate it once at the start of the unit of work, thread it through every retry and every downstream tool call, and have the server record the key with the result so a replay returns the original response instead of re-executing. Stripe’s 24-hour key window is the canonical reference model — the retry inside that window is a no-op.
The failure mode nobody budgets for: the stale, half-written result. A retry that replays a cached “success” for an operation that never actually committed leaves you with a confident agent and a missing side effect. Or the async tool call the agent fired, received an ID for, marked complete — and the work never landed. Deduplication without a completion check turns “at-most-once” into “maybe-once.” The ledger that records a key as done must be written only after the side effect is confirmed, not when the response is generated.
5. Cache Invalidation Is the Only Hard Part
Every cache that saves money is also a cache that can serve a stale answer. The invalidation rules are what separate a cost-saving cache from a correctness incident, and they are simpler than most teams fear — because they all reduce to one question: what changed?
- Prompt-prefix / schema cache: invalidate on deploy. The system prompt and tool schema change only when you ship code, so a version or content hash in the key makes invalidation automatic and free.
- Embedding cache: invalidate on embedding-model change. Different models produce different vector spaces; swapping models without rotating the cache key silently mixes incompatible vectors.
- Tool-result cache: invalidate on the data changing, not on a timer. A TTL is a blunt instrument — it either evicts while the data is still valid (wasted money) or serves stale data until the TTL expires (wrong answers). Prefer invalidating on writes: when the underlying record updates, purge the keys that depended on it.
- Semantic completion cache: the hardest seam, because “the world changed” is not observable from the prompt alone. Time-bounded TTLs are the honest floor here — short enough that staleness is bounded, and paired with the awareness that a semantic cache on a fast-moving domain is borrowing correctness from the future.
The trap is time-based invalidation applied uniformly. A TTL is not an invalidation strategy; it is a staleness budget. Use it where you cannot observe the change event, and use explicit invalidation everywhere you can. The cache that pays for itself is the one whose invalidation is a side effect of your write path, not a separate cron job someone forgets to run.
6. Is It Worth Engineering? A Decision Framework
Not every agent should have all of this. The value of each mechanism scales with your redundancy, not your volume — a low-traffic agent with a single linear flow has almost nothing to deduplicate, while a branching, retrying, parallel agent leaks money regardless of how few users it has.
Work down this list and stop at the first “no”:
- Do you have deterministic repeated input? (schemas, prompts, embeddings, tool results) → exact-match cache keyed on content hash. This is the highest ROI, lowest-risk layer, and it is nearly always worth doing.
- Do you have concurrent or retried requests? → in-flight coalescing plus idempotency keys on side-effecting operations. If your agent retries anything, this is non-negotiable, because a retry without a key is a duplicate charge waiting to happen.
- Do you have conversational paraphrasing? → semantic caching, with the threshold set as a risk dial and the understanding that you are now responsible for false-hit correctness.
- Do you have a fast-moving domain? → reconsider step 3. A semantic cache on data that changes hourly is a liability wearing a savings disguise.
The honest summary: the first two layers pay for themselves in a week; the third is where the real judgment lives. Most agents are over-paying on deterministic redundancy they can eliminate with a hash map and a key — and under-thinking the semantic layer that actually requires care.
Want to design AI employee roles from scratch rather than deploy templates? The AI Agent Architects bootcamp opens soon — the waitlist gets first access: aitokenlabs.com/ai-agent-architects/waitlist
About the Author
Anthony Odole is a former IBM Senior Managing Consultant, where he served as Enterprise Architect on Fortune 500 engagements, and the founder of AIToken Labs. He helps business owners cut through AI hype by focusing on practical systems that solve real operational problems.
His flagship platform, EmployAIQ, is an AI Workforce platform that enables businesses to design, train, and deploy AI Employees — AI agents that function as digital workforce members — that perform real work without adding headcount.
