Level 3 · Internals · 7 min
How do I tune a threshold when I only logged the matches?
A similarity score was recorded every time two support cases were grouped and discarded every time they were not, so every number available for tuning was, by construction, above the threshold. The three near-misses that moved it were invisible until one column kept them.
A threshold decides which comparisons count. Every system with one records its acceptances for free, because an acceptance produces something — a group, a match, a link. Almost none of them record the refusals, because a refusal produces nothing.
The refusals are where the entire tuning signal lives.
I hit this building a support-intelligence platform whose job is to notice when forty people are writing in about the same broken login. It reads every case, groups the ones describing one underlying problem, and writes each group up as a sentence a leader can act on. An earlier piece measured why that grouping was failing. This is what came after — what it took to stop guessing at the number, and what happened when the fix appeared to do nothing.
The shape of the bias
Cases are compared by embedding each one and measuring how close it lands to the centre point of each existing group. Above the threshold it joins that group; below it, it founds a new group of one.
The system recorded the score every time a case joined. When a case failed to join, the score it failed with was computed, compared, and thrown away.
Nothing about that is an error, and it makes the data useless for the one question anyone was asking. Every recorded number was above the threshold because that is what being recorded meant. The log could describe the population it admitted in as much detail as you liked, and could say nothing whatsoever about where the line should have been — it is a university studying its admissions policy by surveying the students it accepted.
The database was full. The dashboards were clean. There was no measurement of the thing under discussion anywhere in the system, and no error anywhere either.
One column
So when a case founds its own group, record how close it came to joining an existing one. We called it the near-miss. One migration, a few lines in the worker.
To probe it we wrote four cases about a single root cause, phrased the way duplicated reports actually arrive — same feature, same symptom, four customer voices:
SSO login loop after password reset · Cannot sign in with SSO following a password change · SSO redirect keeps looping once the password is reset · Stuck in an SSO redirect loop after resetting my password
At the shipped threshold of 0.80, all four founded their own group. The next run said why:
| Case | Near-miss | Missed by |
|---|---|---|
| Stuck in an SSO redirect loop… | 0.787 | 0.013 |
| SSO redirect keeps looping… | 0.773 | 0.027 |
| Cannot sign in with SSO… | 0.759 | 0.041 |
| the same case, unrelated groups | 0.295 | — |
Three near-duplicates missed by hairs. The fourth row is the reassuring half: measured against genuinely unrelated content, the same case scored 0.295. Same theme lands near 0.77; different theme lands near 0.30; in this data there is a wide empty valley between them.
That table took one column and one run. The argument it replaced had been going on for weeks.
Where the line went, and why not lower
We moved the threshold to 0.72: roughly 0.04 below the weakest true match we had observed, and roughly 0.055 above the strongest false match we had ever measured, which was 0.665.
Deliberately not lower. The earlier measurement also contained a true match at 0.493 — two cases any human files together without hesitating — and chasing it would have dragged the line underneath the false-match ceiling. That trade is a missed insight for a wrong one, and those two costs are not equal. A missing insight costs a discovery. A wrong insight is a confident sentence about a problem that does not exist, in front of someone who may act on it.
Then something that felt pedantic and paid for itself immediately: the measured numbers became a test. Raise the threshold back above 0.787 and a test named after the measurement fails. A constant justified by numbers should fail something when the numbers stop justifying it — otherwise the justification lives in a comment, and a comment cannot fail.
The four cases fused into one group, cleared the three-member minimum the summariser insists on, and the system wrote its first sentence.
The family that stayed apart
The same workspace held a second family: five cases about exported spreadsheet totals disagreeing with the dashboard, written earlier and worded more loosely. After re-clustering at 0.72, one pair grouped and three stayed solo. Their near-misses were 0.714, 0.698 and 0.665.
Below the line by hairs again, and this time lowering it was not available. That last number is exactly the strongest false match on record. A line low enough to catch this family sits on the ceiling where unrelated things start fusing, with no margin at all. There is no clean cut here — only a best available place to stand, which is the finding of the article this one follows.
But there was a second explanation worth building for. This grouping strategy is order-dependent: each case is compared only against the groups that exist when it arrives, and nothing ever revisited a placement. Five same-theme cases arriving in an unlucky order can found three groups whose centres only drift toward each other as members accumulate. Two groups could end up genuinely closer than the threshold and still be filed as strangers, purely because of what arrived first.
So we built a merge pass: after each complete run, compare every pair of group centres, and merge any pair closer than the threshold.
Three design choices carry it, and each is the conservative direction:
- Centre against centre, not member against member. A centre is an average, and an average sits closer to the middle than the things it averages — so comparing centres understates how similar the members are. The pass can repair a split; it can never lower the bar.
- A merged group's summary goes stale, not deleted. Membership only grew, so nothing the old summary said became false. It stays on screen until rewritten. When members leave a group we delete instead — the direction of the change decides.
- A heal, not a gate. If the pass fails, it logs and the run still succeeds. A corpus with fragments is degraded; a crashed worker is broken.
Review caught a real bug in it before it shipped. I had gated the pass on the worker not wanting to continue — which is also true when the worker fails or stalls halfway, states where the corpus is half-placed and the centres are garbage. "Not continuing" is not "finished"; it includes "gave up". The fix was a first-class completion flag, with tests pinning all three states.
It merged nothing, and that was the proof
We ran it. merged: 0. Ran it again. merged: 0. The export family stayed
apart.
My first instinct was that the pass was broken. The data said otherwise, and this is the observation the whole exercise turns on.
A group of one has a centre equal to its only member. So for singletons, centre-against-centre is case-against-case — and those numbers were already on the table: 0.665 to 0.714, below the threshold. The merge pass was not failing. It was correct. This family is not an ordering artefact; these cases are measurably sub-threshold to one another, and a pass that merged them would have been quietly lowering the line its conservatism exists to protect.
A mechanism that does nothing, for the right reason, on evidence you can check, is a mechanism you can trust the first time it does something.
And it will. The designed path is a bridging case: the next export complaint joins whichever fragment it is nearest, pulls that centre toward the others, and then a pair of centres clears the line. The pass is not dead weight; it is waiting for the corpus to grow, which is what corpora do. The alternative is per-workspace calibration — if one company's genuine duplicates habitually score 0.66 to 0.71 while another's score 0.78, one global line was always going to be a compromise — and this table is the first concrete evidence for building it.
What generalises
Instrument the refusals. Any system with a threshold records its acceptances for free and discards its rejections, and the rejections carry every bit of information about where the threshold belongs. One column turned taste into arithmetic twice here.
Fear the confident error more than the missing answer, and put that asymmetry into every choice rather than into a principle. Here it is the margin above the false-match ceiling, the summariser's refusal to write about small groups, the merge pass comparing centres, and stale beating deleted.
A number is not a finding. This threshold is 0.72. So was the relevance floor in this site's own retrieval evaluation — until it was measured and turned out to sit below every score an unrelated question produced, filtering nothing at all while reading like a safeguard. It is 0.78 now. One 0.72 was a measured boundary and the other was a guess wearing the same digits, and only the bands they were measured against can tell you which is which.
When a mechanism does nothing, check whether nothing was correct before you fix it. The most useful run the merge pass ever had was the one where it refused to act, because the refusal could be checked against measured data and found right. That is when a system earns trust — and it is indistinguishable from being broken until you have the numbers to tell the difference.
Practical
You can tell whether your threshold has any measurement behind it, and you know the one column to add that turns the argument about where it belongs into arithmetic.
Find where your system compares a score against a cut-off. When the comparison fails, record what it scored rather than discarding it. Come back in a week and sort those numbers. The gap between the ones that missed by a hair and the ones that missed by a mile is the only evidence about where the line belongs — and until you kept them, you had none of it.
In this article
The terms above, defined. New to this? Start here — nothing in the article assumes you already knew them.
- threshold Definition ↩
The cut-off score above which a system treats two things as the same.
Not to be confused with relevance floor: A threshold decides whether two things are the same. A floor decides whether anything is worth acting on at all — usually to avoid spending money.
- embedding Definition ↩
A list of numbers that encodes a piece of text’s meaning, so that closeness between two lists stands in for closeness of meaning.
Not to be confused with vector index: An embedding is the number list for one piece of text. A vector index is the structure that makes searching millions of them fast.