How I Got a Voice Agent to Find the Right Paragraph 86% of the Time, Even When the Caller Says "Uh... and What About Sundays?"

Part 2 of a 3-part series on building retrieval-augmented generation for voice agents. Part 1 was about why voice RAG is a different problem. This one is about finding the right paragraph.

Your policy document says:

"Account deletion requests are processed after a 30-day waiting period."

Your caller says:

"How do I get rid of my account completely?"

Not one word in common. Not "delete". Not "account deletion". Not "waiting period".

And yet a human agent would find the right paragraph in about two seconds.

This is Part 2 of the series, and it's about teaching a machine to do the same thing, in about 20 milliseconds, on audio that went through a speech-to-text model first.

I'll cover what "accurate" means on a call, how I measured it without fooling myself, and the four small changes that moved the number the most. None of them is a new model.

#What "accurate" means when someone is on the phone

In Part 1 we talked about what a wrong answer costs on a call. The AI doesn't show a list of results. It speaks.

That changes which number you care about.

In a search UI, you care about the top 10. The human scans. Result #4 is fine.

On a call, the model gets a handful of paragraphs and picks one to talk from. If the right paragraph isn't first, you're relying on the model to rescue you.

So I tracked two numbers:

• Top-1 → how often the correct paragraph is the very first result
• Top-5 → how often it's anywhere in the handful we'd actually hand to the model

Top-1 is the one that keeps me awake.

#The test (and its limits, up front)

I built a fake retailer. Let's call it Nimbus Mart. It has 80 paragraphs of policy across ten topics: returns, delivery, memberships, payments, warranty, accounts, stores, cancellations, support and offers.

The paragraphs are deliberately full of near-twins:

• returns for sale items vs. regular items vs. electronics vs. gifts
• Sunday delivery vs. weekday delivery
• Nimbus Plus vs. Nimbus Pro vs. Nimbus Family

Because real documents are like that. Every policy has a cousin that says almost the same thing with one number changed.

Then I wrote 140 questions the way callers actually ask them:

The noisy ones are the same questions run through a speech-to-text-style mangler: no punctuation, filler words, dropped words, "X R 440" instead of "XR-440".

Caveats, because I'd want them if I were reading this:

• I wrote this test myself, on a made-up company. It is not customer data.
• It's small. Treat gaps under about 5 points as noise. Where it matters I ran a bootstrap and I'll quote the 95% interval.
• Your documents and your callers will give you different numbers. That's the point of building your own test (more on that below).

#The result

The retrieval step is plain vector search: turn the question into a vector, find the nearest paragraphs.

Across all 140 questions: 86% top-1 and 95% top-5.

A few things I like about that chart:

• Paraphrases: 88%. "Get rid of my account" finds "account deletion". That's the whole point of vectors: they match meaning, not words.
• Codes and product names: 100%. "OD-30" and "PAY-402" and "XR-440" are exactly the kind of thing people assume vector search will fumble.
• Noisy transcripts: 91%. More on this below, because it's the one that surprised me most.
• Hinglish: 93%. Callers who code-switch mid-sentence still land on the right paragraph.
• Hindi in Devanagari: 100%. (Six questions, so take that as encouraging rather than conclusive.)

Now the four things that moved the numbers.

#Lever 1: The 30-character prefix

Embedding models like the one I tested are trained with small task prefixes: a few words that tell the model "this text is a question" or "this text is a document".

If you skip them, nothing crashes. Nothing logs a warning. Everything still returns vectors.

It just quietly gets worse.

Top-1 went from 77% without the prefixes to 86% with them. A gain of about 9 points (95% interval: 4 to 14).

That's a bigger jump than most "advanced retrieval" techniques promise, for zero extra milliseconds.

I only knew to check because we'd been burned once already: switching an embedding runtime had once changed our results slightly, and the cause was a missing prefix. If you take one thing from this post, take this: delete your prefixes once in a test and see what happens.

#Lever 2: Smaller vectors are almost free

Using the model's full-size vectors: 87% top-1.

Using shrunk-down ones: 86%.

That difference is well inside the noise (95% interval: -2 to +5 points). So you can store far smaller vectors and keep essentially all the accuracy.

Why do I care? Because in Part 3 you'll see exactly what vector size does to your memory bill. Spoiler: it's a straight line, and this is how you get to the cheap end of it.

#Lever 3: Hand the retriever a whole question

This is the biggest one.

On a call, people don't speak in standalone questions. They say:

"Do you deliver on Sundays?" "And what's the cutoff for that?"

The second one, on its own, is not a question a retriever can answer. "That" is doing a lot of work.

On 12 follow-up questions, vector search got 25% when I gave it the words as spoken, and 100% when I gave it the same question with its context filled in.

Same retriever. Same documents. Same everything. A 75-point swing.

(It's only 12 questions, so don't tattoo the exact number on your arm. But the size of the gap tells you where the real opportunity lives.)

The lesson: the biggest "retrieval upgrade" in this whole post isn't a retrieval upgrade. It's making sure the retriever receives a whole question. How you do that without adding dead air is worth a post of its own.

#Lever 4: Speech-to-text noise didn't hurt

I was braced for this one to go badly.

Real calls aren't clean text. They're transcripts: no punctuation, "uh" and "okay so" at the front, a word dropped here and there, and product codes spelled out letter by letter.

So I ran the paraphrases and code questions through that mangler and compared.

Clean: 91% top-1. Noisy: 91%.

Exactly the same. Vectors care about meaning, and meaning survives "uh okay so" just fine. That's a pretty comforting property for a system whose input is always a transcript.

#What if the answer isn't there?

Part 1 promised the live path should stay boring. That includes the "I don't know" path.

A caller asks something your documents don't cover. The retriever still returns something, because nearest-neighbour search always returns the nearest. The question is whether you can tell when "nearest" is actually "nowhere close".

The cheap signal works well. The similarity score of the top result:

• Questions with an answer: median 0.62
• Questions without one: median 0.40

That one number separates the two groups with an AUROC of 0.92. (1.0 would be perfect, 0.5 is a coin flip.)

At a cutoff that keeps 90% of answerable questions, 80% of the unanswerable ones were correctly turned away, with no extra model and no extra latency.

The four that slipped through were:

• "How much is the Nimbus Ultra plan?"
• "How do I become a seller on Nimbus?"
• "Do you have a store in Hyderabad?"
• "How do I track my refund status in the app?"

Spot the pattern. All near-misses. Each one is almost a question the knowledge base does answer. Those are the ones where your agent's fallback instructions earn their keep. (Only 20 out-of-scope questions, so: direction, not decimals.)

#A recipe for a boring voice-RAG pipeline

I'm deliberately not going to list our exact models, sizes or thresholds. They're tuned to our data, and yours will want different ones. What I will give you is the order of operations I'd defend:

1. Hand the retriever a complete question. Fix fragments before retrieval.
2. Do the expensive work offline. (Part 1.) Clean, chunk and embed ahead of time.
3. Use the same model, with the right prefixes, on both sides. Test them by deleting them once.
4. Store compact vectors. Check on your own data that accuracy holds.
5. Use the top score as a tripwire. Below it, route to your "I don't know" path.
6. Send few, whole paragraphs to the model. (Remember the truncation bug.)
7. Add a new stage only when you can show it earns its milliseconds on your own questions, in the languages your callers actually use.

And the most important one: build a golden set. One hundred real questions with known answers is enough to detect a 10-point swing. It's a day of work, not a quarter, and it will settle more architecture arguments than any blog post, including this one.

#Why this matters more than it sounds

Voice retrieval is not enterprise search with a time limit.

The tricks that win on a search page assume a human is reading the results and compensating for them. On a call, nobody is compensating. The retriever's top pick is the answer.

So the work isn't about the most sophisticated pipeline. It's about small, checkable things: the right prefix, a whole question, a sensible cutoff, a test set you trust.

#The takeaway

Vector search found the right paragraph first 86% of the time on questions with typos, filler words, Hinglish and half-finished follow-ups.

The prefix added about 9 points.

A complete question added 75 on follow-ups.

Smaller vectors cost nothing.

Noisy transcripts cost nothing.

The goal isn't a more impressive retrieval pipeline.

The goal is the right paragraph, first, every time, in 20 milliseconds.

In Part 3, I'll try to break it: a million paragraphs, one small database, and how far it goes.