Part 9 closed on a promise about what came next: the whole thing end to end under an orchestrator, with lineage and contributions carrying through rather than bolted on afterward.
So I went looking for the pipeline to put under the orchestrator, and found it had been running end to end since Part 4. It was living inside an HTTP request handler.
The Thing Hiding in Plain Sight
One file, 1,069 lines. Two functions in it — _process and _process_834 — doing hash, dedupe, store, detect, parse, map, gate, write, gate again, resolve, finish. Every stage the platform has, in order, correct, and covered by tests. Reachable only by a person with curl.
Four things follow from that, and none of them is hypothetical.
Nothing retries. A transient database error seven hundred lines in fails the whole file. The bytes are on disk, the row says failed, and recovery means a human running it again from the top.
Nothing resumes. Parse succeeded, map succeeded, the write failed. All of that work is discarded and redone, because there is nowhere to put a half-finished file except in the handler's local variables.
No stage runs on its own. Identity resolution is a query parameter — ?resolve=true — so running it over members already committed to the database means re-parsing the file to reach it. The stage was designed to stand alone. Its own docstring promises it "cannot fail the ingest". There has never been a way to invoke it by itself.
Nothing starts by itself. Partners drop files. The inbox directory was watched by nobody.
The pipeline was never missing. Everything around it was.
Three Guarantees Written for One Process
The handler makes three promises that took real work to earn, and an orchestrator attacks all three at once.
The canonical write is one transaction. Every person, identifier and coverage is written alongside the lineage row explaining where it came from, in a single transaction on a single connection, so no window exists in which a record sits in the database without its provenance.
The contribution is threaded, never ambient. Part 9 gave every write a change set to belong to, and passes it down as an argument on purpose. The module that defines it refuses a context variable in so many words: ambient state is one concurrent task away from attributing this file's rows to whichever other file happened to be parsing at the time.
A failed run leaves the change set open. The contribution row commits before the work begins and is marked complete after. A crash in between leaves it unfinished, permanently, which is the truthful record of a unit of work that died partway.
Asset, as an orchestrator uses the word: a named thing that exists in the world — a parsed file, a table of validated records — rather than a task that ran. You declare what each asset depends on, and the orchestrator works out what to run, in what order, and what can be rebuilt without rebuilding everything upstream of it.
Dagster — the orchestrator Part 1 chose before there was anything to orchestrate — runs each asset step in its own operating-system process by default. A pooled connection does not cross that boundary. An open transaction does not cross it. A Python object threaded down a call stack does not cross it.
Getting a file through an asset graph is an afternoon. Getting it through with those three promises intact is the part.
Three Hundred and Forty-Nine Lines, and Not One Test
The first step was a pure lift: six stages out of the handler into their own module, importing neither the web framework nor the orchestrator, each one taking what it needs as an argument. Then the route rewired to call them. The handler went from 1,069 lines to 720.
The plan set one condition on that step. Every existing route test passes unmodified. If a test needs editing to go green, the extraction was wrong and I stop and fix the extraction.
Four tests broke, and the reason is worth a paragraph. They reach into the pipeline by patching through the route — and Python's patching binds a name where it was imported to, not where the function is defined. Move a function into another module and the patch target moves with it, however faithfully the behaviour was preserved.
Two of the four turned into real seams on the way past. Where a stored file lands became an argument instead of a module global the route read directly; the resolver classes became parameters. Both are better than what they replaced, for a reason that has nothing to do with tests: a caller can only substitute what it is handed. The other two needed a two-line re-export and a comment naming the coupling honestly.
Then the extraction was clean, and the interesting question was why.
Every stage already had its own tracing span, its own status transition on the file's row, its own early return. Part 4 built it that way to get useful traces. Part 6 kept it that way so the gate could commit its verdicts before the write. Nobody was designing for an orchestrator, and the stage boundaries came out where an orchestrator would have drawn them anyway.
The file status column tells the same story. received → detected → parsing → parsed → mapped has been an asset materialization log since Part 4, and I had been reading it as a progress bar.
The orchestrator was already there, six parts early, spelled as a status column.
Which settled a design question before it could become an argument. There is no second materialization log. The assets read that column to decide whether they should run, and write it through the same function the route has always used. One state machine, two callers.
What the Process Boundary Charged
Three guarantees, written for one event loop and one call stack, carried across a boundary that offers neither. The bill came to two items.
| Guarantee | Before | After |
|---|---|---|
| The contribution is threaded, never ambient | An argument down a call stack | A run tag, reloaded from the database by each step |
| The canonical write is one transaction | One transaction on one connection | Unchanged — it lives inside a single asset |
| A failed run leaves the change set open | The context manager never reaches its exit | A success hook, and deliberately no failure hook |
Plus one thing that table does not show: a file's accumulated errors and warnings were three local variables in the handler, and are now fields on the value each asset returns.
The run tag is the threading argument moved one level up. Only the contribution's id crosses the boundary, as a string; each step reloads the row rather than trusting a cached copy, because a change set whose actor was read in one process and edited in another is a class of bug this platform does not need to invent.
The failure guarantee took the strangest shape. Orchestrators offer failure hooks precisely so you can tidy up after a crash, and tidying up here would be a lie — an unfinished contribution is how the audit trail says a change set was abandoned. So there is a success hook on the last asset, no failure hook anywhere, and a test that walks every module in the package and fails if one appears. "We didn't write one" is a claim that decays the first time somebody adds one for a good reason.
The headline criterion was that the same file through both callers produces identical rows — persons, identifiers, coverage, lineage, gate results, and exactly one contribution each. I had written in the plan that this was the check most likely to fail first, on the theory that six parts of a single entry point leaves every assumption untested and a second caller is the cheapest bug-finder available.
It passed on the first run. That is the weaker story and the truer one. The stages were genuinely separate before anybody tried to separate them, and the bugs this part found were all in the new code.
The Pool That Would Have Worked
The connection pool holds connections bound to the event loop that created them. Each orchestrator step runs its stage on a loop it owns.
Borrowing the existing module-level pool works perfectly under the multiprocess executor. Every step is a fresh process, the global starts empty, each one builds its own. The bug never fires.
Switch to the in-process executor — for a test, for a small file, because the UI offers it as a checkbox — and two steps running at once find the pool already created on the first step's loop. Two files' work, quietly sharing connections.
A guarantee that holds only under one executor setting was never a guarantee.
So the pool became something each step constructs for itself. And the test that settles the whole threading argument runs two files with their canonical writes forced to overlap by a quarter of a second, then checks that every lineage row landed on its own change set. It passed, which is what makes the run tag the threading argument one level up rather than ambient state wearing a badge.
Teaching It Not to Re-Run
Part 9 left reprocessing restricted to files that produced nothing — rejected, failed, detected, parsed. A file that reached mapped has canonical rows behind it, and running it again writes a second person for the same human being. Doing that safely needs supersession of the first pass's rows, which is temporal versioning, which Part 9 looked at and deliberately did not build.
It is worth being concrete about what that guard is guarding, because "writes a second person" is engineering language for something that happens to someone.
A member is in the database once, with a coverage span, a subscriber relationship, and an identifier a provider's office has already checked against. Run that file a second time and the member is in there twice. Neither row is marked as the duplicate, because nothing in the system knows that one of them is — what it has is two records that match closely, which is the exact condition Part 8's scorer exists to escalate rather than settle. The next eligibility check reaches whichever one it reaches. If that is the new row, the coverage history begins on the afternoon somebody clicked a button, and the twelve months before it belong to a record nobody is looking at.
An orchestrator's entire proposition is a re-materialize button.
So the platform acquired one and immediately had to teach it not to do the thing orchestrators are for. The guard lives inside the canonical write itself rather than at either entry point, which means an operator clicking re-materialize hits exactly the same refusal a direct API call hits. One guard, in the place both callers pass through.
I had budgeted for this going badly — for the refusal needing to be repeated at three separate doors, each one a place to forget it. It needed the opposite: the check moved down into the write, where neither caller can route around it, and both doors got simpler.
Thirty-Six Seconds
The decision to defer event streaming has been a paper claim since Part 1. File-based work, a directory watcher, and the database for lightweight internal signalling — revisit when real-time eligibility needs sub-second answers. Nine parts of asserting that and no measurement behind it.
This is the part that runs it. A file dropped into the inbox, against the real daemon:
| Step | Elapsed |
|---|---|
| Drop → the watcher notices | 24 s |
| Notice → run launched | under 1 s |
| Launch → first step begins | 4 s |
| Six-step graph, one member, separate processes | 8 s |
| Drop → rows committed | ≈ 36 s |
Nothing in that argues for a message broker. The dominant term is a polling interval, which is a constant in one file, and the case the deferral names — sub-second eligibility responses — would not be reached by shortening it. The decision holds, and for the first time it holds on a number.
The File the Daemon Lost
The first time I started the daemon and dropped a file, the file vanished.
Claiming the file's row and requesting the run are two separate steps, and they cannot be collapsed into one: the row has to exist before the partition that names it exists. A watcher tick that dies between those two writes leaves a registered file, an open change set, and no run. On the next scan the content hash reports the file as already ingested — so it is stranded, permanently, by the very check that makes the watcher safe to re-run.
The deduplication that keeps a watcher from processing a file twice is the same mechanism that can keep it from processing a file at all.
The fix is one line of judgement: a duplicate whose file never left received gets picked up rather than skipped. Finding it took starting the daemon, which makes this the fourth part running where running the thing caught something the test suite had agreed was fine. The suite could not have found this one. The stranding requires a tick to die between two database writes, and that first run produced exactly that, because the watcher had been wired to a job that did not exist yet.
There is a detail in the wreckage I have been turning over since. The open, unfinished contribution that the crash left behind is still sitting in the database, exactly as designed. The failure guarantee — the one written in Part 9 for a failure I imagined, and defended again in this part against an orchestrator that wanted to clean it up — did its job on the first unplanned failure it ever met.
A Test That Passed for the Wrong Reason
The test for the one-transaction guarantee raised an error after the write function returned, then checked that no rows survived. It was green from the moment it was written.
By the time that function returns, the transaction has committed. The test was asserting that a database which had already durably stored three rows still contained them. It proved nothing, and it proved nothing in a confident green colour.
The failure has to land inside the transaction. So the coverage insert is what fails now, and the person and the identifier written before it in the same transaction have to disappear along with all three lineage rows.
A transaction test that passes is not evidence until you have watched it fail for the right reason.
What Ships, and What Is Still Loose
Six assets, a watcher that runs by default, the change set travelling as a run tag, and an end-to-end graph that produces the same rows as the endpoint it was lifted out of. Three things are loose, and they are worth naming while they are still small.
Neither change-set kind fits a standalone resolution run. Resolution can now be materialized on its own, over members already committed — a genuinely new capability. But the five kinds were closed and constrained in Part 9, and a resolution that is not part of any file's ingest fits manual when a person triggered it and reprocess when the graph did. Both are a stretch. Neither word was coined for "resolution, again, by itself". The plan said in advance that a forced mapping would be a finding rather than a licence to add a sixth kind, so there is no sixth kind and no migration, and the looseness is written down where the next part will trip over it.
One module in the project lacks a convention every other module follows. The orchestrator resolves an asset's configuration annotation at decoration time, against the class object. Deferred annotations hand it a string it cannot resolve, and the failure arrives at import. The house style loses to the framework in exactly one file, with the reason in the file rather than buried in a commit message.
The actor is still trusted input. Unchanged from Part 9, and restating it costs a sentence. There is no authentication layer; the trail is tamper-evident about what the system did and takes the caller's word for who asked.
Part 11 is where all of this finally meets enough volume to say whether any of it is calibrated. Two parts have now built a measuring instrument and had nothing to point it at — a confidence-scoring harness that has never run for want of an API key, and a weight estimator that found zero true-match pairs against a floor of five hundred and refused, correctly, to publish a number.
The pipeline now runs on its own, retries, resumes, and records who asked. What it still cannot tell me is whether the numbers it produces mean anything.
