Most token-cost advice is aimed at people who don’t understand how the bill is computed. It tells you to “write shorter prompts” and “use smaller models” as if those were the two dials that matter. They aren’t. If you’re designing production AI employees, the cost of an agent is decided by architecture long before it’s decided by prompt wording — and most of the money leaks out through a handful of structural choices you can measure and fix directly.
This guide is the engineering playbook. It assumes you already know what a token is and that you’re past the point of being impressed by a long context window. What you want is the how: where the cost actually comes from, and which levers are worth pulling versus which are theater.
Where Token Costs Actually Come From
Before optimizing anything, you have to be able to read the invoice correctly. Two structural facts explain almost every agent bill.
Output tokens are drastically more expensive than input tokens. Across the major providers the output-to-input price ratio is roughly 4× to 8×, and it’s not an accident of pricing — it’s physics. Generating output is autoregressive: the model runs once per token, with a growing KV cache, and the GPU becomes memory-bandwidth bound. Input prefill, by contrast, is a parallel, compute-dense operation. The ratio clusters around 5× for Anthropic and 6× for OpenAI, drops near 2× for DeepSeek, and widens past 8× for Gemini. There is no universal multiplier, but the direction is consistent everywhere: output is where money goes.
The immediate consequence is that your input-to-output ratio matters more than your raw token count. A documentation generator that emits thousands of output tokens per call has a completely different cost profile from a RAG assistant that stuffs 50,000 input tokens into context and emits fifty back. Sorting your traffic by that ratio tells you where the money actually is before you write a single line of optimization code.
Reasoning tokens silently inflate output. Thinking models bill their hidden reasoning tokens as output, even though you never see them. On hard problems a model with a modest sticker ratio can bill like a far more expensive one. If you’re routing a “simple” task to a reasoning model, you may be paying a 10× premium for deliberation the task didn’t need. This is the single most common silent price increase in agent systems.
There’s a second, less comfortable fact: any specific price you memorize is already wrong. LLM API prices fell roughly 80% between early 2025 and early 2026, and that rate hasn’t slowed. A cost model built on last quarter’s numbers is stale. Treat pricing as a moving target: the principle “model selection sets your ceiling; routing and caching decide how close to the floor you operate” holds even as every individual number changes.
Model Routing
Model routing is the single highest-leverage cost reduction available, and it’s almost never done well. Most teams either send everything to a frontier model (overspending on trivia) or send everything to a cheap model (degrading the hard cases). Neither is engineering.
The pattern is task classification treated as a first-class problem, not an afterthought. A typical enterprise distribution routes roughly 70% of queries to a budget model, 20% to a mid-tier model, and 10% to a premium model for genuinely demanding work. Compared to a single flagship model, this tiered approach routinely cuts average per-query cost by 60–80%. The spread between the cheapest and most expensive model on identical traffic can exceed 40× — which means routing is the budget.
The router itself is the interesting part. Three approaches, in increasing order of sophistication:
- Rule-based routing. Classify by explicit signals — query length, presence of code, intent tags, tool type. Cheap, deterministic, and predictable. Good enough as a first pass, and it gives you a baseline to beat.
- A classifier model. A small, fast model scores each request and assigns it to a tier. This is the common production answer because it generalizes beyond hand-written rules.
- Cascading with fallback. Send to the cheap model first; if confidence is low or a quality check fails, escalate to the mid or premium tier. This optimizes for the minimum model that can do the job, which is exactly the right objective.
The trap is routing on vibes. If you can’t measure whether the cheap model actually did the job, you’ll route badly in both directions — overpaying out of fear, or silently shipping degraded output. Route on evaluations, not on intuition, and re-run the math after every price change. A routing decision older than a quarter is running on expired prices; in a price war, loyalty to last quarter’s winner is just a donation to your vendor.
Prompt Compression and Message Pruning
Your instinct is probably to rewrite prompts. That’s the wrong place to start — in most real systems, the prompt itself is a small fraction of the tokens. The money is in the context around the prompt: tool outputs, retrieved documents, logs, and accumulated conversation history. That’s where context explosion happens.
Prompt compression shrinks the context you send while preserving what the model needs to act. The principled versions don’t just truncate — they score token importance and drop the low-value remainder. Techniques like LLMLingua-style importance scoring can achieve dramatic compression with minimal performance loss. The honest caveat: compression itself is an LLM call, so it has its own cost. You only win if the tokens you save on the main call exceed the tokens you spend compressing. Measure both sides.
Message pruning is the simpler, higher-certainty move. The rule: don’t send everything you ever saw; send what the current request actually needs. Concretely:
- Send only the retrieval chunks relevant to this query, not the full document set.
- Truncate tool outputs aggressively — a 4,000-token JSON blob from an API often contains 200 tokens of signal.
- Keep the last N turns of conversation, and summarize rather than retain older history.
The deeper principle is input-heavy, output-light. Because output is so expensive relative to cached input, the winning architectures are biased toward feeding a lot of cheap, well-organized context in and demanding very little expensive output back. Design your agent to emit a decision, not an essay, and put the burden on the input side where the price is lower.
Context Window Management Over Long Conversations
Long-running agents are where token costs silently compound, because every turn re-sends the entire accumulated history. A conversation that spans dozens of tool calls can balloon to tens of thousands of tokens per request — and you pay for all of it on every subsequent call.
The core skill is context pruning and eviction — deciding what stays in the window and what gets turned into something smaller. Three techniques carry most of the weight:
- Summarization. Periodically compress the conversation into a running summary, then drop the raw turns. This is the classic “rolling summary” pattern and it works, with one caveat: summaries lose detail, and for tasks that depend on exact prior values (a quoted number, a specific error message) you need to preserve those verbatim rather than paraphrase them away.
- Message eviction. Drop or collapse low-value turns — tool calls that returned nothing useful, retries, verbose logs. Treat the window as a cache with a budget, not an append-only log.
- Structured memory. Move durable facts (user preferences, entities, decisions) into a store the agent retrieves rather than a transcript it replays. A compact state object of “what we know” is far cheaper to carry than the full narrative of “how we learned it.”
The mental model that keeps all of this honest: the context window is a budget, not a feature. A 1M-token window is not an invitation to send 1M tokens — it’s headroom you should use sparingly, because every token you retain is a token you re-pay on every subsequent turn. Long-horizon agents that don’t prune don’t just get slower; they get quadratically more expensive.
Caching and Deduplication
If routing decides your ceiling, caching decides how close to the floor you get. The economics are stark: cached input can cost 75–90% less than uncached input, and on some models a single output token can cost up to 80× a cached input token. That 80× ratio is the design constraint most teams ignore.
Prompt caching works because most of an agent’s context is stable across turns — the system prompt, the tool definitions, the knowledge base, the codebase context. When a request shares a prefix with a cached one, the provider skips most of the re-prefill work. The savings are real, but they’re conditional:
- Order matters. Cache hits depend on prefix matching. Put your stable content (system prompt, tool schemas, retrieved docs) first, and keep dynamic content (the new user turn) at the end. If you interleave volatile data into the stable prefix, you break the cache and pay full price.
- Stability matters. Switching models mid-conversation, or reordering your system prompt, invalidates the cache. Standardize your prefix and guard it.
- Cache expires. Provider caches have TTLs (often minutes to an hour). High-frequency traffic with a stable prefix benefits most; sporadic traffic may see the cache expire between calls.
Semantic caching is the second layer. When your agent repeatedly asks the model the same kind of thing — “classify this support ticket,” “extract the entities from this invoice” — you can cache responses keyed by an embedding of the input, and replay the stored answer when a sufficiently similar request arrives. This is deduplication at the semantic level, and for high-volume, repetitive workloads it can remove entire classes of calls from the bill.
The honest corollary: caching is most effective in high-volume systems. A single-user prototype won’t feel it. A production agent serving thousands of conversations a day will — which is exactly when the cost matters.
What You Shouldn’t Optimize (The Engineering Judgment)
Not every cost is worth chasing. Part of being the architect is knowing which optimizations are real and which are theater — and which are actively harmful.
Don’t optimize prompt wording first. It’s the most visible lever and usually the least consequential. If your context is exploding from tool outputs and history, shaving ten tokens off the system prompt is noise. Fix the structure first, then the prose.
Don’t optimize latency against cost blindly. Smaller models are cheaper and faster, but if they force you into more retries, more escalation, or more turns to reach the same answer, the total cost can rise. Measure cost per completed task, not cost per call. A cheap model that needs three attempts to do what a mid-tier model does in one isn’t cheap.
Don’t chase caching in low-volume systems. If traffic is sporadic, the cache expires before it pays, and the engineering effort is wasted. Caching is a scaling investment, not a default.
Don’t let price cuts lull you into architectural laziness. The Jevons paradox is real: when tokens get cheaper, people use more of them, and total spend flatlines or rises. The discipline that matters — route on evaluations, prune aggressively, cache stable context — is valuable because it’s structural, not because it’s tied to any particular price. Build it once, and it compounds as prices move in either direction.
The thread through all of this: optimize the architecture, not the adjectives. The teams that keep agent costs down are the ones who treat routing, pruning, and caching as core design decisions made up front — not as a cleanup pass after the bill arrives.
Summary Checklist
- Measure the input-to-output ratio before optimizing anything; output is where the money is.
- Route on evaluations, not intuition — tier your traffic (roughly budget/mid/premium) and re-run the math after every price change.
- Watch reasoning-token inflation — don’t send simple tasks to thinking models that bill deliberation as output.
- Prune tool outputs and retrieval aggressively — send the 200 tokens of signal, not the 4,000 tokens of blob.
- Bias the architecture input-heavy, output-light — demand a decision, not an essay.
- Treat the context window as a budget — summarize, evict, and move durable facts to structured memory.
- Cache stable prefixes and keep them stable — stable content first, dynamic content last, standardized and guarded.
- Add semantic caching for repetitive classification/extraction workloads.
- Optimize cost per completed task, not cost per call — a cheap model that retries isn’t cheap.
- Skip caching in low-volume systems — it’s a scaling investment, not a default.
Want to design AI employee roles from scratch rather than deploy templates? The AI Agent Architects bootcamp opens soon — the waitlist gets first access: aitokenlabs.com/ai-agent-architects/waitlist
About the Author
Anthony Odole is a former IBM Senior Managing Consultant, where he served as Enterprise Architect on Fortune 500 engagements, and the founder of AIToken Labs. He helps business owners cut through AI hype by focusing on practical systems that solve real operational problems.
His flagship platform, EmployAIQ, is an AI Workforce platform that enables businesses to design, train, and deploy AI Employees — AI agents that function as digital workforce members — that perform real work without adding headcount.
