Level 1 · Orientation · 1 min
Why does similarity search need an index at all?
Comparing a query against every vector is exact and simple, and it stops being viable at a few hundred thousand rows. Here is where the wall is, and what an index trades away to get past it.
The honest starting point for vector search is a loop. You have a query vector, you have N stored vectors, and you compare the query against every one of them. This is called a flat or brute-force index, and it has a property no approximate method can offer: it is exactly right, every time.
It is also linear. Ten times the documents, ten times the work.
Where the wall actually is
The useful question is not "is brute force slow" but "at what size does it stop clearing my latency budget". That depends on three things: how many vectors you have, how many dimensions each one has, and how much time you are willing to spend.
A single comparison is a dot product over the dimensions. At 1,536 dimensions that is around 1,536 multiply-adds. Modern hardware does this fast, and a well-vectorised scan over a hundred thousand embeddings finishes in a few milliseconds. Over ten million, on one core, you are into seconds — and seconds is not a budget, it is an outage.
Most teams discover this boundary in production, because it does not exist in development. A prototype with two thousand chunks will never show you the problem.
What an index trades
Approximate nearest neighbour indexes buy sublinear search by giving up the guarantee. They do not promise the true nearest neighbours; they promise something close, most of the time. The parameter you tune is how close, and how often.
That trade is usually worth it, and it is a trade, not an upgrade. Some proportion of queries will now miss a document a brute-force scan would have found. If you have no way to measure that proportion, you have no way to know whether your retrieval got worse when you added the index — which is a far more common failure than anyone admits.
The rule
Stay exact until the numbers force you off it. Then measure what you lost.
Practical
You can estimate whether a brute-force scan is still fine for your corpus, and say what an index would cost you in recall.
Take a corpus you already have. Time an exact scan over all of it, then over a tenth of it, and confirm the relationship is linear. Estimate the row count at which you cross your latency budget. That number, not a blog post, decides when you need an index.
In this article
The terms above, defined. New to this? Start here — nothing in the article assumes you already knew them.
- embedding Definition ↩
A list of numbers that encodes a piece of text’s meaning, so that closeness between two lists stands in for closeness of meaning.
Not to be confused with vector index: An embedding is the number list for one piece of text. A vector index is the structure that makes searching millions of them fast.
- chunk Definition ↩
One slice of a longer document, embedded on its own so retrieval can return the relevant part instead of the whole file.
Not to be confused with document: A document is the source. A chunk is the unit actually searched — and one embedding has to speak for everything inside it.