Most people watch the model bill. Your vector database is the cost that grows while you’re not looking — because embedding spend scales with data volume, not request volume. Every chunk you ingest, every re-embed of a barely-changed document, every index that holds ten million 1,536-dimensional floats in RAM: those are fixed costs that compound silently as your corpus grows. The model bill goes up when usage goes up. The vector bill goes up when time passes.
This is a builder’s guide to the four levers that actually move the number — dimension reduction, quantization, index selection, and tiered storage — plus the honest question of when you should re-embed at all, and when embeddings are the wrong primitive entirely.
TL;DR: Cut vector spend in four moves: shrink dimensions (Matryoshka or PCA), quantize (int8 or PQ), pick an index that matches your scale (HNSW under ~10M vectors, IVF-PQ beyond), and tier cold vectors to disk. Cache embeddings instead of recomputing them, and re-embed only when meaning actually changes. Roughly 60–90% of the spend is recoverable before recall meaningfully degrades.
Why Does Embedding Cost Keep Climbing?
Embedding cost is a function of three things: how many vectors you store, how wide each vector is, and how often you regenerate them. None of those is proportional to how often your agent is queried.
A 100 million-vector corpus at 1,536 dimensions is roughly 614 GB of raw floats. Put it in an HNSW index with its graph overhead and you’re past 1.2 TB of RAM. That’s not a spike you’ll notice in a dashboard — it’s rent you start paying the day your corpus crosses a threshold, and it renews every month.
The three root causes to name before you touch anything:
- Dimensional blow-up. You’re paying for 1,536 dimensions even when 384 would hold the same concepts.
- Index memory. The graph that makes search fast is itself the most expensive thing in the system.
- Re-embedding churn. Re-embedding a document because one sentence changed wastes a fixed cost that should be cached.
What Actually Drives the Bill Up?
The mechanism that surprises people is that vector cost is storage-bound, not throughput-bound. A request that hits your database costs almost nothing at the margin; the cost is the memory and disk those vectors occupy while idle. So optimization isn’t about making queries faster — it’s about making the resident footprint smaller.
That distinction matters because it changes where you look. If spend tracked throughput, you’d optimize the query path — but it tracks footprint, so you optimize the data and the index. It’s the same category of thinking as AI agent model selection for cost efficiency: the biggest win usually comes from choosing the right underlying component, not from tuning the one you already have.
How Do I Reduce Embedding Dimensions Without Losing Recall?
Dimension reduction is the highest-leverage, lowest-risk move, and there are two genuinely different tools for it.
Matryoshka Representation Learning (MRL) is the better answer when you control the embedding model. An MRL-trained model produces embeddings whose first N dimensions are themselves a valid N-dimensional embedding — so you can truncate a single 1,024-dim vector down to 256 and keep a coherent representation, rather than training or recomputing at the smaller size. OpenAI’s text-embedding-3-small uses exactly this, which is why you can pass a dimensions parameter and shorten embeddings while retaining most of the concept. The honest caveat: quality doesn’t degrade evenly. A truncation that holds up for recall may underperform on ranking precision or clustering, so validate against your retrieval metric before shipping it.
PCA is the fallback when you’re locked to a model that doesn’t support MRL. You project existing high-dimensional vectors onto their top principal components. It’s lossy in a different way — PCA finds the directions of maximum variance in your data, which may not be the directions that matter for your queries — but it’s model-agnostic and cheap to apply post-hoc.
A common pattern is two-stage: shortlist with truncated vectors, rerank the top candidates with full-size ones. The shortlist gets you most of the speed and storage win; the rerank recovers the precision.
Which Index Should I Choose for Cost?
Index choice is where memory and recall trade off directly, and the right answer changes with scale.
| Index | Memory (1M, 128-dim) | Recall | Best when |
|---|---|---|---|
| Flat (exact) | ~512 MB | 100% | < 50K vectors, correctness > cost |
| HNSW | ~523 MB (graph-heavy) | ~90%+ | Real-time RAG under ~10M vectors |
| IVF-PQ | ~40 MB | ~70–90% | Large corpora, memory-constrained |
| Scalar (int8) | ~256 MB | ~95%+ | Middle ground, cheap win |
HNSW is the standard for real-time retrieval under roughly ten million vectors — fast, high recall, but the most memory-hungry because the navigation graph lives fully in RAM. IVF-PQ is the answer at scale: it clusters vectors (IVF) then compresses residuals with product quantization (PQ), shrinking a corpus by up to 100x versus HNSW at the cost of recall dropping into the 70–90% band depending on your compression ratio.
The lesson from the numbers: the same 100-million-vector corpus that needs 1.2 TB in HNSW fits in under 20 GB with IVF-PQ. If you’re not at that scale, don’t pay for the complexity — but know the line you’ll cross.
How Do I Quantize to Cut Storage?
Quantization is compressing the precision of each vector component, and it’s the single biggest storage lever per vector.
- Scalar quantization (int8) maps 32-bit floats to 8-bit integers — a 4x storage cut with recall loss typically in the low single digits. This is the “free” win most teams skip.
- Product quantization (PQ) goes further, splitting each vector into sub-vectors and approximating each against a learned codebook. It’s how IVF-PQ hits those 62–100x compression numbers, but recall falls to 50–74% at aggressive settings, so treat the compression ratio as a recall dial, not a free lunch.
The meta-point: quantization and index choice are one decision, not two. You pick a compression target, then choose the index that meets your recall floor at that target.
Should I Cache Embeddings Instead of Recomputing Them?
Yes, and this is the most underused lever in the whole stack. An embedding is a pure function of its input — the same text produces the same vector every time. If you re-embed a document because one clause changed, or re-embed the same query across parallel branches, you’re paying for a deterministic computation twice.
Two rules make this concrete. First, cache embeddings by content hash, invalidating only when the embedding model changes — a different model produces a different vector space, so mixing them silently corrupts retrieval. Second, re-embed only on a semantic change threshold: skip re-embedding a document when the edit is a typo fix or a date stamp, and only regenerate when the meaning genuinely shifted.
This pairs directly with the broader request-side work — AI agent caching and request deduplication covers the full map of where tokens and embeddings get spent twice and how to close those seams. If you want the deeper treatment of the retrieval half, AI agent semantic caching walks through the implementation guide for cutting token costs on the query side.
How Do I Tier Storage for Cold Vectors?
Not all vectors deserve RAM. Tier them by access pattern:
- Hot tier (RAM): vectors your agent queries constantly — recent documents, active users’ context, current product catalog.
- Warm tier (SSD): vectors queried occasionally, where tens of milliseconds of latency is acceptable.
- Cold tier (object storage): archive vectors you must retain for compliance or long-tail but almost never retrieve.
The cost gradient between RAM and object storage is an order of magnitude or more, so moving even a third of your corpus to warm or cold storage is where the big architectural savings live. The catch is retrieval latency and recall — tiering works when your query patterns are genuinely skewed, and it’s a waste of engineering when access is uniform.
When Are Embeddings the Wrong Primitive?
Be honest about this one, because the cheapest embedding is the one you never generate. Embeddings are the right tool for semantic similarity — “find me chunks about this idea.” They’re the wrong tool for exact matches, structured filters, and any query where a keyword or a boolean is the correct answer.
If your retrieval is mostly “find the document with this order number” or “filter tickets by status and owner,” full-text search or a key-value lookup does it faster, cheaper, and more correctly than a vector search ever will. The hybrid pattern — keyword match for the precise stuff, embeddings for the fuzzy stuff — is usually the honest architecture, and it means you only pay for vectors where they earn their keep.
That “right tool for the job” instinct extends across the whole cost surface. Once retrieval isn’t the bottleneck, the next question is whether the model you’re routing to is oversized for the task — AI agent hybrid model routing is the same tradeoff applied to inference spend, and it composes cleanly with everything here.
FAQ
Do I need embeddings at all for my agent?
Only if your retrieval is semantic. For exact-match and structured queries, full-text search or key-value lookups are faster, cheaper, and more correct — use embeddings just for the fuzzy part.
How much can I save with quantization?
Scalar (int8) cuts storage ~4x with single-digit recall loss; product quantization reaches 60–100x compression but drops recall to 50–90% depending on the ratio you choose.
Does Matryoshka truncation hurt quality?
Not equally across tasks. Truncated embeddings often hold up for recall but can underperform on ranking precision or clustering — validate against your own retrieval metric before shipping.
Should I re-embed on every document edit?
No. Cache embeddings by content hash and re-embed only when meaning changes past a threshold — a typo fix or timestamp update shouldn’t trigger a fresh vector.
When should I move to IVF-PQ from HNSW?
When your corpus crosses roughly ten million vectors and HNSW’s RAM footprint is the dominant cost; below that, HNSW’s recall is usually worth the memory.
Want to design AI employee roles from scratch rather than deploy templates? The AI Agent Architects runs as a small five-week cohort, and the waitlist hears every enrolment date first: Join the AI Agent Architects waitlist
About the Author
Anthony Odole is a former IBM Senior Managing Consultant, where he served as Enterprise Architect on Fortune 500 engagements, and the founder of AIToken Labs. He helps business owners cut through AI hype by focusing on practical systems that solve real operational problems.
His flagship platform, EmployAIQ, is an AI Workforce platform that enables businesses to design, train, and deploy AI Employees — AI agents that function as digital workforce members — that perform real work without adding headcount.
