Level 2 · Application · 1 min
Why does recall drop when you make your chunks bigger?
Bigger chunks carry more context, so retrieval should improve. It usually gets worse. The reason is that embedding a long passage averages away the specific thing your query was looking for.
It is the most reasonable-sounding intuition in retrieval, and it is wrong more often than it is right: if the model keeps missing context, give it bigger chunks.
Then recall goes down, and nobody can explain why.
One vector has to speak for the whole passage
An embedding is a fixed-size summary. A 1,536-dimension vector represents a fifty-word passage and a five-hundred-word passage with exactly the same budget. The longer the passage, the more different ideas are competing for the same coordinates.
A short chunk about connection pooling is about connection pooling, and its vector sits where connection-pooling questions look. Put that same paragraph in a two-thousand-word chunk that also covers deployment, logging and testing, and the resulting vector lands somewhere in the middle of all four topics — near none of them in particular.
The specific signal did not disappear. It got averaged.
The shape of the curve
This produces a curve with a peak rather than a slope. Chunks that are too small fragment the answer across several of them, so no single chunk contains enough to be useful. Chunks that are too large dilute the signal until the right one stops ranking. Somewhere between is a maximum, and its position depends on your documents and your queries — not on a number in someone's blog post.
Which is why the only useful move is to measure it on your own corpus.
Why this gets misdiagnosed
The failure looks like a model problem. Retrieval quality drops, the top results look vaguely on-topic instead of correct, and the obvious suspect is the embedding model. Teams switch models, see no improvement, and conclude retrieval is just hard.
The tell is the kind of wrong. A model mismatch returns results from the wrong subject area. Over-chunking returns results from the right area that do not contain the answer — plausible, adjacent, useless. If your failures look plausible, suspect your chunk size before your model.
Practical
You can diagnose a recall regression caused by a chunking change, rather than blaming the embedding model.
Build a set of twenty questions you know the answers to, and the passage each answer lives in. Index the same corpus at three chunk sizes. Measure how often the correct passage appears in the top five. Plot recall against chunk size and find your own curve, because it will not match anyone else's.
In this article
The terms above, defined. New to this? Start here — nothing in the article assumes you already knew them.
- chunk Definition ↩
One slice of a longer document, embedded on its own so retrieval can return the relevant part instead of the whole file.
Not to be confused with document: A document is the source. A chunk is the unit actually searched — and one embedding has to speak for everything inside it.
- embedding Definition ↩
A list of numbers that encodes a piece of text’s meaning, so that closeness between two lists stands in for closeness of meaning.
Not to be confused with vector index: An embedding is the number list for one piece of text. A vector index is the structure that makes searching millions of them fast.