The Score That Earns the Merge

Part 6 ended with an endpoint I had to build before I was allowed to build anything else.

POST /identity/links/{id}/unlink. It exists because somebody asked what the platform is actually for if it can never approve a merge on its own, and going to answer that properly I found that a merge could be created through the API and undone only by someone writing SQL against production. A decision you cannot reverse through the API is a decision nothing should make unattended. That was the floor, and it went in.

This part is the thing that was waiting on it.

The Number Platforms Get Bought On

This series opened with forty-seven people out of three hundred and twelve failing to map. An auto-processing rate of 84.9%, though nobody called it that at the time — it was just a bad Tuesday.

That number is the sales magnet. Data is the product — the series opened on that claim and nothing since has weakened it — and the auto-processing rate is what gets the product bought. When somebody evaluates a platform in this space, the question underneath every other question is what fraction of a file lands without a human touching it, and what it costs when the platform is wrong. Part 6 made the number computable by writing a gate_results row for every evaluation, pass and fail alike. It did not move it.

Identity resolution controls roughly half of it. Two records arrive that might be the same person, the deterministic tier can't answer, and today the answer is: ask the model, store a proposal, wait for a human.

The obvious way to move the number is to threshold the model's confidence. Part 6 built the harness to check whether that confidence means anything and never got to run it, so the honest state is that nobody knows. But there is a second problem with thresholding it, and that one doesn't go away with a calibration run: a model's self-reported confidence is not reproducible. Prompt version drifts, the model behind the endpoint changes, and a 0.87 from March cannot be re-derived in December. A decision you cannot re-derive is a decision you cannot audit, and audit is the whole argument of the previous part.

So this part builds the tier the Part 3 design named and then deliberately parked:

deterministic first, probabilistic second (Fellegi-Sunter weighted), policy third — auto-link, discard, steward queue

LinkMethod.PROBABILISTIC has been sitting in the enum since migration 002 with a docstring saying the engine was deliberately not built. identity_links already stored method, score, actor and a snapshot of the field values at the moment of decision. Part 3 recorded the prediction in as many words — adding probabilistic matching later would be a new writer rather than a migration.

That prediction held. This part is that writer.

Almost None of This Runs

Before any of what follows, one thing has to be said plainly, because the rest of this article is about the exception rather than the rule.

Identifiers match first, and they match deterministically. resolve_person has been tier 1 since Part 3. It reads a spoke table where a person holds many identifiers, each qualified by who issued it, and it matches on exact equality of the full triple — assigning authority, type, value:

resolve_person(pool, assigning_authority="CMS", id_type="mbi",
               identifier_value="1EG4-TE5-MK73")

An MBI is the strongest of these, because CMS issues it once and never reuses it. A payer's member ID is strong inside that payer. An MRN is unique inside one hospital's walls and meaningless outside them — which is why the authority is part of the key and never dropped. An identifier without its issuer is a number rather than an identity claim.

Most volume resolves right there and never reaches a score.

Fellegi-Sunter exists for what falls through: two records with no identifier in common. A member arriving from a new payer, a patient whose MRN belongs to a hospital this system has never seen, an 834 whose subscriber ID changed when the employer switched carriers. That is the population this whole part is about, and it is the residue rather than the main flow.

What a Score Is Made Of

Fellegi and Sunter published the framework in 1969, and the reason it has outlived every fashion since is that it asks a question with an answer.

For each field you compare, two probabilities:

• m — how often this field agrees, given the two records are the same person. It is a measure of data quality. Names get typed wrong, so first_name has an m below 1. Dates of birth are copied from forms, so theirs is higher.
• u — how often this field agrees, given the two records are different people. It is a measure of coincidence. Two random strangers share a gender about half the time, so gender has a u near 0.5. They share an SSN-last-four about once in ten thousand.

Agreement on a field is worth the log of the ratio between them:

@property
def agree_weight(self) -> float:
    """`log2(m / u)`. Positive when agreement is evidence of a match."""
    return math.log2(self.m / self.u)
 
@property
def disagree_weight(self) -> float:
    """`log2((1 - m) / (1 - u))`. Negative for any informative field."""
    return math.log2((1.0 - self.m) / (1.0 - self.u))

Sum across the fields and you have a score in bits of evidence. Agreeing on gender is worth about one bit, because half the world agrees with you. Agreeing on an SSN-last-four is worth around thirteen, because almost nobody does.

Two details in that code matter more than the formula.

A missing field contributes exactly zero. Not a small penalty, not an average — zero. A field nobody sent must move the score in neither direction, which is why MISSING is a separate comparison outcome rather than a flavour of disagreement. Half the 834s in the world omit gender. Treating silence as evidence would make the score a measure of how chatty the payer's HR system is.

Frequency-specific u is most of what separates this from naive scoring. A single u per field says agreement on a surname is worth a fixed amount. But agreement on Nakagawa is far stronger evidence than agreement on Smith, and collapsing them throws away the best signal in the vector. When the comparison carries the agreed value and the weight set holds a u for it, that value's own rarity is what gets used.

Hold that last thought.

Nobody Chooses the Weights

Here is where most identity matching goes quietly wrong, including every version of it I have written before this one.

You need an m and a u for every field. The tempting move is to pick them. Date of birth feels reliable, call it 0.95. Surnames change on marriage, call it 0.85. SSN-last-four is basically unique, call u 0.0001. The numbers look like measurements, the scorer runs, the thresholds get tuned until the results look right, and the whole apparatus is a set of hunches wearing scientific notation.

The weights have to be measured rather than chosen, and the data to measure them is already in the database.

The positive class comes free: every pair the deterministic tier already linked is a pair known to be the same person. Compare those and count how often each field agrees, and that count is m. The negative class comes from random pairs — draw two people at random and they are, with overwhelming probability, different people. Count agreement there and that is u.

make estimate-weights does exactly that and writes docs/weight-estimation-report.md, which is committed to the repository for the same reason Part 6 commits its calibration report: a threshold whose evidence lives only in somebody's terminal history is a threshold nobody can re-examine.

The report carries the parts that are easy to leave out.

Confidence intervals, per field. An m of 0.94 from 12,000 pairs and an m of 0.94 from nine are different claims, and only one of them survives contact with a Wilson interval.

Clamp counts. An m of exactly 1.0 makes the disagreement weight negative infinity, and one disagreeing field then vetoes everything. A u of exactly 0.0 makes the agreement weight positive infinity, and one agreeing field carries the entire decision. Both are what an estimator returns when a field never varied in the sample. Both are catastrophic, both are silent, and both are clamped strictly inside (0, 1) at estimation time — then refused again in the scorer, because a constraint enforced in one place holds only until somebody adds a second caller.

A payer-skew assessment. If the true-match pairs come overwhelmingly from one payer, then m is that payer's data-entry consistency rather than a general truth about human names, and the report has to say so in those words.

The conditional independence caveat, named rather than assumed. Fellegi-Sunter treats field agreements as independent given match status, and this vector plainly violates that. first_name and gender are correlated. ssn_last_four is shared within families, so a parent and child agree on it while being different people. Neither is corrected for, and both push the score the same way — up, on exactly the pairs that should not merge.

That last one is why the design says the auto-link threshold ships narrower than the numbers justify rather than at them.

The Constraint That Would Have Rejected Every Row

identity_links.score has had a CHECK constraint since migration 002:

score FLOAT CHECK (score IS NULL OR (score >= 0 AND score <= 1))

Written in Part 3, when the only thing that could produce a score was an agent reporting a confidence, and a confidence between zero and one is exactly right.

A Fellegi-Sunter score lands in the twenties.

Every row this part writes would have been rejected by a constraint written five parts ago, and the build plan did not catch it — despite specifying the scorer and the storage in the same document — because it never once wrote the number down next to the constraint. The plan described a log-odds sum in prose and described the column in DDL, and the two descriptions never met on the same line.

The fix that suggests itself is to drop the bound. That is the wrong fix, and working out why was the most useful ten minutes of the part.

A constraint that has been correct for five parts can be correct for the wrong reason the whole time. This one was, and it took a value it had never seen to reveal which half of it was load-bearing.

A Fact Beats a Score

The tier produces three outcomes, and the order the checks run in is the safety property.

The veto sits above the bands. Two different Medicare Beneficiary Identifiers issued by the same assigning authority are two people, whatever the demographics say — the same judgement Part 6's reject_conflicting_authoritative_id makes about proposals, arriving now at a different layer. A fact beats a score, and a score is not evidence against a fact.

The clause I want to defend is the other one.

"Exactly one candidate in that band" is a refusal, not a tie-break. When two candidates both score above the threshold, the obvious behaviour is to take the higher and move on. That behaviour is precisely how a scorer merges a pair of twins.

- id: twins_same_address
  truth: different_people
  incoming:  {first_name: Marcus, last_name: Reyes,
              date_of_birth: 2011-09-02, ssn_last_four: "8830"}
  candidate: {first_name: Marco,  last_name: Reyes,
              date_of_birth: 2011-09-02, ssn_last_four: "8831"}

Two candidates above the threshold means the evidence cannot separate them. Ambiguity escalates, and it escalates to the layer that can ask a different kind of question. This is the part of the design I would defend hardest: the model stops being a scorer and becomes an adjudicator of the genuinely ambiguous. A language model is a poor instrument for "do these two SSN-4s agree" and a good one for "is this a married name or a different person."

That is also where the cost saving lives. The response from ?resolve=true now carries an agent_calls_avoided count, because a tier that resolves the easy pairs arithmetically and hands over only the hard ones is cheaper per file and better per decision at the same time.

Five Thousand Draws From Thirty-Six Pairs

The estimation harness had a floor: at least 5,000 random pairs for the negative class, because u is a small number and small numbers need volume.

I ran make estimate-weights against the real database. It reported 5,000 random pairs drawn from a population of nine people.

Nine people admit thirty-six distinct pairs.

The sampler had been drawing with replacement and counting draws. A sample size of 5,000 with the statistical power of 36 is exactly the mistake MIN_RANDOM_PAIRS was written to prevent, arriving through the back door with the floor's own approval.

It was make estimate-weights that found it, on the real database, in about four seconds — not the 1,077 tests that run without one. The suite has grown to be very good at proving the code does what it says. Running the command is still what proves the code says the right thing.

The fix dedupes, stops when the population runs out, reports how many distinct pairs were actually available, and refuses with a sentence that names the real problem: no seed and no sample size can reach the floor.

There was a second one hiding underneath. The sampler was told to draw the negative class across payers, to stop u measuring one payer's data-entry habits. The first version treated that as an exclusion rather than a preference, and one of the "payer groups" it was excluding across was the set of people carrying no assigning authority at all. Twenty-two of the thirty-six available pairs became undrawable, and the sample came out at fourteen.

A person carrying no assigning authority makes a pair whose payer relationship is unknown, which belongs in a different bucket from one that genuinely spans two payers. Cross-payer is now a preference at 0.8 rather than a rule at 1.0, because a sample containing no within-payer pairs is a different bias rather than the absence of one.

The Run That Refused

With the sampler honest, the run produced a real answer.

Nine synthetic people, and none of them had ever been deterministically linked to another, so the positive class was empty. With no true-match pairs there is no denominator for m on any field, and without m there is no weight for anything.

make estimate-weights refused to publish a weight set, and the report is committed saying so:

Estimating m from a handful of pairs produces a number with the shape of a measurement and the content of a guess, and a threshold resting on it would be a threshold nobody could defend.

So Part 8 ships a complete probabilistic tier that has never scored a real pair, and a per-field weights table with no rows in it.

Tier 2 is therefore switched off. With no active weight set, every candidate escalates to the agent and ingest behaves exactly as it did in Part 6 — same path, same proposals, same human doing the disposing.

The thing I want to point at is how it is off. Nothing auto-links without an active weight set carrying an explicit auto_link_above, and both halves fail safe independently: no weight set means everything escalates, and a weight set without a threshold means nothing auto-links at any score. The feature is absent rather than disabled — there is no flag in the wrong position to find, because there is no flag.

That is the same by-construction safety Part 6 used for auto-approve, and it is becoming the house style. A safe state you get by default is worth more than a safe state everybody remembers to maintain.

And the refusal is a property of the write path rather than of my restraint. publish_weight_set will not attach a threshold to a report that has not earned one. I could not have shipped the auto-link band from this run by changing my mind about it; I would have had to change the code that refuses.

The Guard That Fired Right and Reported Wrong

One more, found while smoke-testing make weights-diff, and it is small enough to miss and general enough to be worth the paragraph.

A report came through whose two sample floors had both passed and whose per-field estimates were empty. publish_weight_set refused it, correctly. The message it printed quoted the two sample counts — the two numbers that were completely fine.

A guard that fires for the right reason and reports the wrong one is worse than a guard with no message at all, because it sends the reader off to fix something that isn't broken. The refusal now carries every disqualification the report found, named individually, so the reason on screen is the reason that fired.

Every test asserted only that the refusal happened.

That is twice in one part — the sampler, and now this — where a test proved a thing occurred and proved nothing about whether it occurred for the stated reason. I have started treating "assert it fired" as half a test.

What Ships Switched Off

Three of those seven modules are pure — comparison, scoring, resolver — and the purity is enforced by walking their import graphs rather than by convention, the same way Part 6 holds its gate layer. The other four are impure, and the test asserts that too, because a purity check that cannot fail proves nothing about the modules it passes.

Which means the same vector and the same weight set produce the same score forever. A merge decided today can be re-judged in three years against the weights that actually made it, rather than against whatever the model says in three years. That reproducibility is the entire reason the score earns the merge and the confidence doesn't.

And now the awkward part.

This is the second consecutive part to build a measuring instrument and have nothing to measure with it. Part 6 shipped make calibrate, a labelled corpus of sixty-three pairs, and no API key to run it. Part 8 shipped an estimator, a report, a publish lifecycle that refuses unearned thresholds — and a database holding nine people.

Both refusals were right. Both leave the identical hole: machinery that works, and no evidence it works on anything.

I could fold that into the retrospective and call it a lesson learned, which would make it an apology. It's more interesting than that, so it has become its own part instead. Getting real volume in — Synthea at scale, enough deterministically linked pairs that m has a denominator worth the name — then running both harnesses for real and publishing what comes out. Including if what comes out says the design was wrong. A confidence score that turns out to be uncalibrated, or a weight set whose auto-link band never clears the false-merge bar, is the most useful finding either harness could produce.

Next, though, is the one that has been carried since Part 1 and keeps getting deferred by things that seemed more urgent: lineage, the catalog, the audit trail. Every record traceable to the file it came from, and the story of debugging a real failure by following it backwards.

The scorer is finished, tested, reproducible, and has never seen a pair worth scoring. Shipping it switched off was the only honest thing to do with it — and finding that out cost less than shipping a threshold I hadn't earned.