Every agent you ship has a quiet tax built into it. It isn’t the model’s per-token price and it isn’t the inference latency you already budgeted for. It’s the fact that every single message you send re-plays everything that came before it. A conversation that started as a 400-token exchange is, by turn 40, a 40,000-token prompt — and you are paying full input price on all of it, every time, whether the model actually uses that history or not.
This is not a bug you can wait for a vendor to fix. It’s an architectural decision, and it’s yours to make. The moment you decide how an agent remembers a long conversation, you have decided its cost curve, its latency profile, and — more subtly — the quality ceiling of everything it says after the first ten minutes of talking.
TL;DR: Long conversations get expensive because every turn re-sends the whole history, so cost scales with the square of the conversation length. Context window compression fixes this with a layered strategy: keep the recent turns verbatim, summarize the middle, and retrieve anything older from durable memory. No single technique wins — a hybrid of truncation, summarization, and retrieval is what makes long conversations both affordable and coherent.
Why does my AI agent’s context window blow up in long conversations?
Because most agents use history replay: the full transcript is appended to every new prompt. Cost and latency scale roughly quadratically, and even a 1M-token window degrades in quality once relevant details sink into the middle where models attend poorly.
Every naive agent framework does the same thing under the hood. It maintains a list of messages, concatenates them into the prompt, and ships the whole thing to the model on every turn. Turn n sends n messages. Total tokens sent across a conversation of length n is therefore O(n²) — the quadratic blow-up. A 100-turn support conversation isn’t 100× the cost of one turn; it’s on the order of 5,000× when you sum the cumulative replay.
Worse, the bigger window you buy to paper over this, the more you run into the “lost in the middle” problem: Stanford and UC Santa Barbara research shows LLMs perform measurably worse when the information they need sits in the middle of a long context, even when it’s technically present. You are paying more for a model that is getting dumber at using what you gave it.
And the economics are brutal at the top end. Gemini 1.5 Pro charges roughly twice per token once you cross 128K tokens. Claude’s long-context tiers cost more per million tokens than its standard tier. The vendors are explicitly pricing against the “just buy a bigger window” instinct. Context compression is one lever among several — reducing AI agent token costs end to end also means routing cheaper models to easy requests, which we cover separately.
What is context window compression, actually?
Context window compression is the practice of reducing the token count of what an agent sends to the model each turn — by truncating, summarizing, or retrieving instead of replaying — so that cost and latency stay flat while the information the model actually needs survives.
Notice what the definition does not say. It does not say “make the prompt smaller at any cost.” Compression is a selection problem, not a size problem. The goal is to maximize the signal-to-noise ratio inside the window, not to minimize bytes. A 4,000-token prompt that contains the three facts the model needs will beat a 200,000-token prompt that buries them, on every axis you care about: cost, latency, and answer quality.
The three compression families
There are only three real mechanisms for keeping a conversation’s context bounded. Everything else in the ecosystem is a variation or a hybrid of these.
1. Truncation (the sliding window). Keep the most recent k turns verbatim, drop everything older. Cheap, deterministic, zero extra inference. It’s the default in most chat frameworks precisely because it costs nothing to implement. The cost is paid in memory: anything that fell off the window is gone, and the agent will contradict itself or re-ask questions you already answered.
2. Summarization. When the history crosses a threshold, run a compression pass that rewrites the older turns into a compact block, then keep that summary plus the recent turns. This preserves the gist of early context at a fraction of the token cost. The cost is paid in latency and fidelity: every summary is an extra inference call, and details that didn’t make the summary are lost forever — a subtle form of drift that compounds as summaries get re-summarized.
3. Retrieval (memory hierarchy). Write facts, decisions, and user preferences to durable storage (a vector store, a database, or structured key-value memory) as the conversation unfolds. Each turn, retrieve only the entries relevant to the current query. This is the only approach that scales to genuinely long-lived agents — ones that need to remember you across days, not just turns. The cost is paid in complexity and miss rate: retrieval can fail to surface a fact the model genuinely needs.
| Strategy | Cost | Recall | Complexity | Best for |
|---|---|---|---|---|
| Sliding window (truncation) | Lowest | Low (old turns lost) | Low | Short bursts, stateless Q&A |
| Summarization | Medium (extra inference) | Medium (gist only) | Medium | Medium-length sessions, 20–100 turns |
| Retrieval / memory hierarchy | Medium (embedding + search) | High (if retrieved) | High | Long-lived agents, cross-session memory |
| Hybrid (buffer + summary + retrieval) | Controlled | Highest | Highest | Production agents |
How do I choose between truncation, summarization, and retrieval?
You don’t — not in production. Choose a layered hybrid and let each layer handle the timescale it’s best at: sliding window for the last few turns, a rolling summary for the session so far, and retrieval for anything that must survive across sessions.
The heuristic that actually holds up: recency, then session, then lifetime. The last few turns need to be verbatim, because that’s where pronouns, half-finished thoughts, and immediate intent live — a summary of “the user is asking about the second option” is useless when you need the exact wording. The middle of the session can be summarized, because the model needs the gist, not the transcript. Everything older than the session — user preferences, past decisions, stored facts — belongs in durable memory and comes back via retrieval.
There’s a tempting shortcut that fails: pick one technique and apply it everywhere. A pure sliding window is what makes agents forget your name on turn 30. A pure summary approach is what makes an agent confidently paraphrase a number wrong. A pure retrieval approach is what makes an agent sound amnesiac about the sentence you just typed. The three failure modes are different, and layering is the only way to cancel them out.
Implementation depth
Here’s the shape of a layered memory manager in pseudocode. Don’t copy it verbatim — the point is the structure of the decision, not the syntax.
class LayeredContext:
def __init__(self, window_k=8, summary_budget=800, max_total=4000):
self.recent = [] # verbatim turns (sliding window)
self.summary = "" # rolling session summary
self.long_term = VectorStore() # durable, cross-session facts
def build_context(self, user_input):
# 1. Always keep the last k turns verbatim
prompt = format(self.recent[-self.window_k:])
# 2. Summarize the session-so-far when it outgrows budget
if token_count(self.recent) > self.summary_budget:
self.summary = summarize(self.summary, self.recent[:-self.window_k])
self.recent = self.recent[-self.window_k:]
# 3. Retrieve durable facts relevant to this turn only
facts = self.long_term.search(user_input, top_k=5)
prompt = self.summary + facts + prompt
# 4. After the turn, persist durable facts
if is_durable_fact(user_input):
self.long_term.upsert(user_input)
return clip_to_budget(prompt, self.max_total)
Three details here matter more than the code itself.
Summarize on a budget, not a timer. The mistake is summarizing every N turns. The right trigger is a token budget: when the verbatim buffer crosses a threshold, compress only the part that’s older than the window. This is exactly what LangChain’s ConversationSummaryBufferMemory does — it keeps recent messages raw and folds the rest into a rolling summary, flushing on a max_token_limit rather than a turn count.
Retrieve on the current query, not the whole history. A common failure is searching your memory store with the entire transcript as the query. You end up retrieving whatever was talked about most recently, not what’s relevant now. The user’s latest message — possibly plus the last turn of the model’s reply for context — is the right retrieval query.
Persist asynchronously, after the turn. Writing to durable memory should never sit on the critical path of the response. Upsert facts after you’ve returned the answer, or the “compression” you added is now adding latency to every single exchange.
If you’re building this yourself, the pieces are all available: LangChain ships ConversationBufferWindowMemory (truncation), ConversationSummaryMemory (summarization), and ConversationSummaryBufferMemory (the hybrid), and any vector store — Pinecone, Qdrant, pgvector, Weaviate — gives you the retrieval tier. LlamaIndex exposes a similar trio of memory abstractions. What none of them give you is the judgment call about where to draw each boundary, which is the part that actually determines whether your agent is good or just cheap.
Compression is also just one slice of the bill. Once your context is lean, routing requests to cheaper models when the task is easy and caching repeated requests instead of re-computing them are the next two levers that compound with it.
The hidden tradeoffs nobody puts in the blog post
Compression is not free, and pretending it is will bite you in production. Three costs hide in the fine print.
Summarization has latency and drift. Every compression pass is a full inference call, and it runs synchronously if you’re not careful — your user waits for the summary before they get their answer. Then there’s drift: a summary of a summary loses fidelity each round, and small errors compound. A number that was “roughly $4,000” in turn 15 can quietly become “about $4,500” by turn 60, and the model will state it with total confidence because it’s reading its own summary, not the source.
Retrieval has a miss rate. The cleanest memory architecture in the world is useless if the embedding search doesn’t surface the fact the model needs. This is the failure mode that earns the harshest user verdicts: the agent had the information, stored it correctly, and still answered wrong because retrieval returned the top-5 semantically similar chunks instead of the one relevant line. Mitigate with hybrid search (lexical + semantic), explicit key-value fact extraction for things like names and numbers, and — for regulated domains — exact retrieval instead of summarization, because a paraphrased legal clause is worse than no clause at all.
The compression step itself costs tokens. LangChain’s own engineering notes surface this honestly: in some workloads, the token savings from compression were offset by the extra inference calls doing the compressing. What they got was better quality at roughly the same cost — the right trade for a production agent, but not the free lunch the “cut costs 90%” headlines imply. If your goal is purely cost, measure the summarization calls you’re adding before you declare victory. When retrieval is your third tier, optimizing embedding spend matters too — the memory hierarchy has its own bill.
FAQ
What’s the difference between truncation and summarization?
Truncation (a sliding window) keeps the last k turns verbatim and deletes the rest — free but lossy. Summarization rewrites older turns into a compact block, preserving the gist at the cost of an extra inference call. Truncation loses detail; summarization loses fidelity. Both lose something; they lose different things.
Do I really need retrieval if I have a 1M-token window?
Yes. A bigger window doesn’t fix the “lost in the middle” problem — models attend poorly to mid-context information — and long-context tiers often cost more per token. Retrieval keeps the window small and relevant, which improves quality and cost simultaneously. Window size is not a substitute for selection.
How much can context compression actually save?
Real-world measurements range from 40–80% token reduction on conversational workloads, with layered approaches reaching 90%+ in document-heavy cases. But the honest answer is “measure it”: summarization adds inference calls, so your net savings depend on how much you compress versus how much you spend compressing.
What’s the simplest thing I can do today?
Switch from full-history replay to a sliding window plus a rolling summary. That’s ConversationSummaryBufferMemory in LangChain or its equivalent elsewhere — keep the last few turns raw, fold the rest into a summary on a token budget. It’s a one-line change that flattens most of the quadratic cost curve.
How do I test whether compression is hurting quality?
Build an eval set of conversations with known “gotcha” facts planted early — a name, a number, a preference — then check whether the agent still recalls them after 50 turns. Run it against each compression strategy and watch for drift (summarization) and miss rate (retrieval). Observability tools like Opik or LangSmith will show you exactly what’s in context at each step.
Long conversations are where agents either prove themselves useful or quietly become too expensive to keep. The teams that get this right treat the context window as a budget to allocate, not a capacity to fill — and they allocate it with a layered strategy: recent turns verbatim, the session summarized, the lifetime retrieved.
The teams that get it wrong do the same thing every time: they buy a bigger window and keep replaying everything, then wonder why a support agent that’s brilliant for ten minutes costs like a full-time employee by the end of the month.
If this is the kind of problem you’re solving in code — memory hierarchies, retrieval miss rates, summarization drift — you’re past templates and into architecture. The AI Agent Architects runs as a small five-week cohort for engineers designing these systems from scratch, and the waitlist hears every enrolment date first: Join the AI Agent Architects waitlist
About the Author
Anthony Odole is a former IBM Senior Managing Consultant, where he served as Enterprise Architect on Fortune 500 engagements, and the founder of AIToken Labs. He helps business owners cut through AI hype by focusing on practical systems that solve real operational problems.
His flagship platform, EmployAIQ, is an AI Workforce platform that enables businesses to design, train, and deploy AI Employees — AI agents that function as digital workforce members — that perform real work without adding headcount.
