Level 2 · Application · 4 min
How do you evaluate a retriever without labels?
Everyone agrees retrieval should be measured, and almost nobody has a labelled set to measure against. Twenty questions written by hand beat two thousand generated ones — a synthetic question is written from the passage it is meant to find, and shares its vocabulary in a way a real one never does.
Every team building retrieval agrees it should be measured. Almost none of them have a labelled set to measure against, so what actually happens is this: someone changes the chunk size, tries three queries by hand, decides the results "feel better", and ships.
The reason is never disagreement. It is that building a labelled set sounds like a project, and there is always something more urgent.
It is an afternoon.
Twenty real questions beat two thousand generated ones
The tempting shortcut is to generate the set: feed each document to a model, ask for questions it answers, and collect thousands of pairs. It scales, it costs nothing, and it is close to worthless.
A generated question is written from the passage it is meant to find. It inherits that passage's vocabulary, its phrasing, its framing. So it retrieves the passage easily — not because your retriever is good, but because the question is a paraphrase of the answer. You have measured your embedding model's ability to match a text against a rewrite of itself.
Real questions arrive in a stranger's words. Somebody who has just hit a problem does not know your headings. They type what they are feeling.
The set behind this site is eighteen questions written by hand. It took an afternoon and it has caught real regressions.
What a labelled case actually is
Two fields: the question, and the source that should answer it.
question: 'why did my retrieval get worse when I used bigger chunks?'
expect: 'why-recall-drops-when-chunks-get-bigger'
Write the question first, in the vocabulary of the problem rather than the
article. can prompt injection reach the DOM? is how somebody asks after being
shown the attack; it is not a phrase the article uses.
Then score. For each question, retrieve, and record where the correct source landed: recall@1 is how often it came first, recall@3 how often it appeared in the top three. Those two numbers are the whole instrument.
Three things that will bite
One gold answer per question stops being true as the corpus grows. This set assumed exactly one correct source until a new document made a second one genuinely right, and the case started failing. That failure measured the label going stale, not retrieval getting worse. The fix is to accept several — but only when a reader asking that question would honestly be served by either, and decided before looking at where anything ranked. Adding an answer because a case started failing is how a test becomes a rubber stamp.
Record absolute counts, not ratios. A percentage lets a batch of easy new cases hide an existing one that broke. Absolute numbers mean adding cases forces you to re-measure and re-record by hand, which is the point.
Measure the thing you are about to build for. A case here ranked its target seventh, and the obvious diagnosis was that dense retrieval was missing a lexical match — a textbook argument for hybrid search. Measured first: the word in question appeared in zero chunks of the corpus. Full-text search had nothing to contribute; the machinery would have been built and changed nothing. The document simply never stated the fact in prose. Writing one sentence moved it to rank 1.
The gate that filtered nothing
The same set caught a second, quieter failure.
This site drops passages below a relevance floor before paying a model to read them. The floor was set at 0.72, which sounded appropriately cautious. Measured across the labelled questions and a set of deliberately unrelated ones, the two bands are: relevant questions score 0.825 and up, unrelated ones 0.800 and below.
0.72 sits below the entire unrelated band. It filtered nothing. It had never filtered anything, and it read like a safeguard in every code review it survived.
That is the failure mode worth naming: not a gate that rejects too much, but a gate that rejects nothing while looking exactly like a gate. No error, no alert, no test — the only thing that finds it is a number measured on both sides of the line.
What twenty questions buy you
Not certainty. The set is small and hand-written and it knows nothing about questions nobody has asked yet.
What it buys is a baseline: a number the next change gets compared to. Without one, "retrieval feels worse" is unfalsifiable and every argument about it is taste. With one, a change either moves the number or it does not, and you find out in the seconds it takes to run.
The afternoon is not the cost. Not having spent it is.
Practical
You can measure whether a retrieval change helped or hurt, instead of deciding by how the results feel when you try three queries by hand.
Write twenty questions the way a real user would ask them, without looking at your documents. Record which source should answer each one. Score your retriever against them and write the number down. That number, not your impression, is what the next change gets compared to.
In this article
The terms above, defined. New to this? Start here — nothing in the article assumes you already knew them.
- chunk Definition ↩
One slice of a longer document, embedded on its own so retrieval can return the relevant part instead of the whole file.
Not to be confused with document: A document is the source. A chunk is the unit actually searched — and one embedding has to speak for everything inside it.
- embedding Definition ↩
A list of numbers that encodes a piece of text’s meaning, so that closeness between two lists stands in for closeness of meaning.
Not to be confused with vector index: An embedding is the number list for one piece of text. A vector index is the structure that makes searching millions of them fast.