Level 2 · Application · 4 min
What similarity threshold should I use to group things by meaning?
Five support cases about the identical problem scored between 0.49 and 0.82 against each other, while unrelated pairs reached 0.66. The distributions overlap, so no single threshold separates them — and the system that got it wrong reported no error at all.
We built a system that reads support cases and groups the ones describing the same underlying problem. Group enough of them and a pattern appears — thirty customers hit this one bug — which is worth far more than thirty separate conversations.
To test it, I wrote five cases that are unmistakably about one thing:
- Exported CSV totals do not match the dashboard
- Report download shows different numbers than the screen
- Discrepancy between exported spreadsheet and web totals
- Why is my CSV export missing revenue?
- Export sum does not reconcile with dashboard sum
You read those in four seconds and knew. Same complaint, five ways.
The machine put four of them in separate groups.
What the machine actually sees
It never reads the words. Each case is converted into a long list of numbers — 1,536 of them — that together encode its meaning. That list is an embedding. Think of it as an address: cases about similar things end up at similar addresses.
Comparing two cases means measuring how close their addresses are, which produces one score between 0 and 1. Around 0 means unrelated; around 1 means near-identical.
Then comes the only real decision: how close is close enough to call it the same problem? That number is the threshold. We shipped 0.80, which sounds appropriately strict for "these are the same thing".
What the five cases actually scored
The best pair reached 0.82. The worst pair — two cases any human would file together without hesitating — scored 0.49.
At a threshold of 0.80, exactly one of the ten pairs qualified. That is why four cases landed in groups of their own. The system did precisely what it was told. What it was told was wrong.
Why lowering the number does not fix it
The obvious response is to lower the threshold, so I measured every comparison on the test workspace: 17 cases, 136 pairs, each labelled by hand as the same problem or a different one.
| Threshold | Correct groupings | Wrong groupings |
|---|---|---|
| 0.50 | 9 of 10 | 13 |
| 0.55 | 9 of 10 | 5 |
| 0.60 | 6 of 10 | 1 |
| 0.70 | 3 of 10 | 0 |
| 0.80 | 1 of 10 | 0 |
There is no row where both columns are good, and the reason is in the raw numbers rather than the tuning. Pairs about different problems ranged from 0.11 to 0.66. Pairs about the same problem ranged from 0.49 to 0.82. Those two ranges overlap between 0.49 and 0.66 — the weakest true match scores lower than the strongest false one.
Any threshold inside that band cuts through both distributions at once. Any threshold outside it sacrifices one of them entirely. This is not a bug to patch; it is a property of the data.
It is like guessing whether two people are siblings by comparing their heights. Usually siblings are closer. But some siblings differ wildly and some strangers match exactly, so no single height difference tells you who is related.
The failure mode worth naming
Nothing errored. Cases embedded correctly, the grouping step obeyed its rule exactly, and the summarising step correctly declined to invent a headline for a group of one. Every log was clean. Every test passed.
An untuned system and a broken one look identical from the outside, and this one would have stayed invisible until customers started seeing support cases filed apart that obviously belonged together. The measurement is what makes the difference visible — not the monitoring, which had nothing to report.
What to do instead
Do not quietly lower the number. Tuning to five hand-written cases would bake a guess — derived from ten sentences I wrote myself — into the most trust-critical behaviour in the product. A threshold chosen to fit a demo is a threshold that will be wrong on real data in a way nobody can trace.
Expect real data to score higher. These test cases are a title and two sentences. Actual support cases carry whole conversations, which gives the model far more to measure. The scores here are a floor rather than a forecast — but a floor you have measured beats a ceiling you have assumed.
One global number is the actual limit. A single threshold has to serve every topic in every workspace, and the distributions differ per topic. The approaches that work find the boundary separately for each cluster instead of demanding one line fit all of them. That experiment is only runnable now because the distribution is measured; before, there was nothing to compare a change against.
The system was never broken. It was untuned, in a test workspace, before a single customer saw it — which is the best possible place to discover that your one number was a guess wearing the costume of a decision.
Practical
You can tell whether a similarity threshold in your system is a measured boundary or a guess, and you know what to measure to find out which.
Take any place your system compares embeddings against a cut-off. Label fifty pairs by hand as same or different, score them all, and plot both distributions. If they overlap, no value of that threshold is correct — and you now know the size of the problem instead of the shape of a hope.
In this article
The terms above, defined. New to this? Start here — nothing in the article assumes you already knew them.
- threshold Definition ↩
The cut-off score above which a system treats two things as the same.
Not to be confused with relevance floor: A threshold decides whether two things are the same. A floor decides whether anything is worth acting on at all — usually to avoid spending money.
- distribution Concept ↩
The full spread of values a measurement produces, rather than its average — the shape that tells you whether a single cut-off can separate two groups.
Not to be confused with an average: Two groups can have averages far apart and still overlap heavily. The average hides the overlap; the distribution is where the overlap is visible.
- embedding Definition ↩
A list of numbers that encodes a piece of text’s meaning, so that closeness between two lists stands in for closeness of meaning.
Not to be confused with vector index: An embedding is the number list for one piece of text. A vector index is the structure that makes searching millions of them fast.