I Put a Million Chunks in One Small Postgres. Vector Search Took Under a Millisecond.

Part 3 of a 3-part series on building retrieval-augmented generation for voice agents. Part 1 was about why voice RAG is its own problem. Part 2 was about finding the right paragraph. This one is about what happens when there are a lot of paragraphs, and a lot of callers.

"Does it scale?"

It's the question that arrives right after "does it work?", usually from someone holding a calendar.

And it's a slippery question, because "scale" isn't one thing.

Once I started trying to answer it, I found four different questions hiding inside it:

1. More customers. What happens when the database holds ten times, then a hundred times, more knowledge?
2. Bigger knowledge bases. What happens when one customer uploads something enormous?
3. More calls. What happens when a lot of people phone at once?
4. More uploads. How fast can new knowledge become searchable?

So I went and measured. Here's what I found.

#The setup (and what it can't tell you)

Before the numbers, here's exactly what I did, because scaling claims without a setup are just vibes with units.

• A throwaway Postgres with a vector extension, in a container capped at 4 CPUs and 8 GB of RAM. Think "one modest server", not a cluster.
• The same table shape and the same kind of vector index we use, built at 10 thousand, 100 thousand and 1 million rows.
• Synthetic vectors, generated in clusters so the index isn't just chewing on noise. They're not real embeddings, so these results are about speed, not accuracy.
• Knowledge-base sizes drawn to match what we actually see: in our development environment the median knowledge base has about 75 chunks, the 90th percentile about 580, and the largest about 1,300. In other words, small. Most voice agents need a few policies, not a library.
• Everything on localhost, one request at a time.

What I didn't do: a full concurrent-callers load test. You'll see napkin math for that part, and I'll label it as napkin math.

#Question 1: What happens when the database gets 100× bigger?

Here's the thing about vector search on a voice call: every search is scoped to one customer's knowledge base. The database never searches the whole table. It finds that customer's rows and searches those.

So making the table 100× bigger should barely matter.

Let me check.

Vector search for one question:

• 10 thousand rows → about 0.3 ms
• 100 thousand rows → 0.2 to 0.4 ms
• 1 million rows → 0.2 to 0.6 ms

The table got a hundred times bigger and the search got a fraction of a millisecond slower. Even the slowest cell on that chart is well under one millisecond.

Other numbers from the million-row run, in case you're sizing a box:

• Loading 1 million rows: 14 seconds
• Building the vector index: 105 seconds (on 4 CPUs)
• Total database size: 3.1 GB

That's one small server and a few minutes of build time for a million paragraphs. At our typical knowledge base size, that's on the order of ten thousand customers.

#Question 2: What if one customer uploads something enormous?

Okay. The table can be huge. But what about a single knowledge base that's huge? Somebody will upload a whole product catalog. They always do.

I made single knowledge bases of 5 thousand, 20 thousand, 100 thousand and 300 thousand chunks inside the million-row table and timed the vector search again.

From a 75-chunk knowledge base to a 300,000-chunk one, a 4,000× increase, the vector search went from 0.2 ms to 1.7 ms.

That's about 8× slower for 4,000× more data. The index is doing exactly what it was built to do.

To be precise about what I'm claiming: that's the vector search itself. A real retrieval has other steps around it, and I'm not making claims about those here.

#The storage bill is a vector-size bill

How much RAM does an index of a million chunks want?

• 256-dimension vectors: 1.4 GB (measured at 1 million rows)
• 768 dimensions: about 4.1 GB
• 1536 dimensions: about 8.2 GB

(The last two are scaled up from a 20,000-row build, so they're estimates. The shape is linear: three times the dimensions, three times the bytes.)

Remember Part 2, where I found that smaller vectors cost almost nothing in accuracy? This is the payoff. Shrinking the vectors is nearly free on the accuracy side, and it's a 3× to 6× discount on the index you have to keep in memory.

If you've ever picked an embedding size because "bigger is better", this is the invoice.

#Question 3: What happens when a lot of people call at once?

This is the one I did not load-test. So what follows is napkin math, and I'm going to show my napkin.

On a call, the part of retrieval that eats the most CPU is turning the caller's question into a vector: about 20 ms on 4 CPU threads. The vector search itself is a rounding error next to that.

If one question at a time could use all four cores for 20 ms, that's about 50 questions a second.

Assume each live call needs a knowledge lookup every 10 seconds or so. That's an upper bound of roughly 500 simultaneous calls on one 4-core box.

It's an upper bound, because whatever else sits in your path (network hops, connection handling, anything you add) comes out of that.

Why I'm not calling it a result:

• Real thread scaling on small inputs is rarely perfect, in either direction
• Real callers don't ask on a metronome
• Concurrent queries interact (queueing, caches, database connections)
• I assumed a lookup every 10 seconds, which is a guess

What I'm comfortable saying: the live path is stateless. Questions go in, vectors and paragraphs come out, nothing is held between them. When one box runs out of headroom, you add another box. That's the property you want from something that has to survive Monday morning.

#Question 4: How fast can new knowledge become searchable?

This is where the real work is, and it's not in the live path.

I timed the offline embedding step on the same small box: about 31 chunks per second on 4 threads.

That's with short chunks (about 120 characters on average), and real paragraphs are longer, so real throughput will be lower. But on those numbers, a million chunks would take about 9 hours on one 4-core worker.

That sounds slow until you remember what it is: an offline job. Nobody is on the phone waiting for it. And because it's offline, you can split it across many cheap workers and finish in minutes.

That's exactly why Part 1 said to do the expensive work before the call. Offline is where you can throw parallel hardware at the problem. A live phone call is the one place you can't.

#The summary

#Why this matters more than it sounds

When people ask whether retrieval scales, they picture the vector search: a big index, a clever algorithm, a graph of nearest neighbours.

That's the part that scales. It's the best-engineered, most-benchmarked piece in the whole path, and on one small server it handled a million chunks in under a millisecond.

Which means the things worth your time are the ones around it: how small you can make your vectors, how you split ingestion across workers, and how you measure the whole path rather than just the famous part.

#The takeaway

A million chunks. One small Postgres. Under a millisecond for the search.

A 4,000× bigger knowledge base cost 8× more time.

Smaller vectors cut the memory bill 3× to 6×.

Ingestion is the slow part, and it's the part nobody is waiting on.

And that's the series.

Part 1: a voice agent isn't enterprise search. Keep the live path boring.

Part 2: the right paragraph, first, with a handful of small, checkable fixes.

Part 3: the boring part scales just fine.

The goal was never to build a retrieval system that impressed anyone.

The goal was a caller who never noticed it was there.