Parsing the Unparseable

The 834 file from Part 1 is still sitting there. Three hundred and twelve member records, forty-seven of which failed because a companion guide variation didn't match the mapping table. The file that started this entire series.

In Part 1, I described the problem. In Part 2, I designed the architecture that data flows through — five layers, five patterns, each independent of the others. In Part 3, I built the canonical model that sits at the bottom — persons not members, a spoke table for identity, three clocks for bitemporality, and a test suite that catches regressions structurally.

The canonical model is ready to hold what a parser produces. This part builds the parser.

The first force from our framework — constraints — dominates everything in this article. An X12 834 file is constrained by a specification written in 1978 and amended through HIPAA 5010. The parser is constrained by what the file actually contains versus what the spec says it should contain. The mapping layer is constrained by the canonical model's design decisions — half-open intervals, spoke-table identity, typed columns versus JSONB. Every constraint in Part 3 becomes an implementation challenge in Part 4.

One Hundred and Six Characters

Every X12 file begins the same way. Not approximately the same way — exactly. The ISA segment is a fixed-width header, always 106 characters, and it has been 106 characters since before most developers alive today were born.

ISA*00*          *00*          *ZZ*SENDER         *ZZ*RECEIVER       *260101*1200*^*00501*000000001*0*P*:~

Position 3 is the element separator. Usually *, but the spec allows any character — I've seen pipes, carets, and one memorable file that used | because someone at a clearinghouse configured their translator wrong in 2004 and twenty years of downstream systems adapted to the mistake rather than fixing it.

Position 104 is the sub-element separator. Position 105 is the segment terminator. Usually ~, but again — any character is valid.

The tokenizer reads those three positions, then splits the rest of the file accordingly. No hardcoded delimiters. No assumptions. The file tells you how to read it.

After the ISA, the file is a flat stream of segments — each one a segment ID followed by element values, separated by the element separator, terminated by the segment terminator. INS*Y*18*021**A is an INS segment with five elements. The structure is uniform. What each element means changes entirely based on which segment it belongs to, which loop it sits inside, and which companion guide the sender follows.

That's where parsing stops being mechanical.

Reading the File, Not the Extension

Before parsing begins, the platform has to know what it's looking at. A file uploaded to POST /ingest might be an 834, a 270, an HL7v2 ADT message, a FHIR Patient resource, a CSV export from a legacy HR system, or a JPEG someone uploaded by accident.

The detector reads the content. Never the extension.

This is the Gateway pattern from Part 2 — content-based routing to format-specific parsers. An .edi file might contain HL7v2. A .txt file might be X12. A .json file without resourceType is just JSON, and claiming it's FHIR would be a lie the gate layer would have to catch.

The State Machine

An 834 file is a nested structure flattened into a stream. The parser's job is to reconstruct the nesting.

The parser walks the segment stream and tracks where it is. An INS segment opens a new member loop. An HD segment opens a coverage line within that member. An N1*75 opens a reporting category loop. An SE closes the transaction.

Each segment handler reads the elements it cares about and ignores the rest. An unknown segment — a ZZZ that no spec defines — is logged as a warning and carried in metadata. Unknown segments are data, not errors. Dropping them silently would be worse than crashing on them, because a dropped segment is a fact you can never recover.

The critical design decision: error isolation. When a member loop fails — a bad date, a missing required field, an enum value that doesn't match any known code — the parser records the error, skips that member, and continues to the next INS segment. One bad record in a three-hundred-member file should not block the other two hundred and ninety-nine.

# Simplified — the real parser has forty segment handlers
for segment in segments:
    try:
        handler = self._handlers.get(segment.segment_id)
        if handler:
            handler(segment)
        else:
            self._warn(f"unrecognized segment: {segment.segment_id}")
    except ParseError as exc:
        self._fail_member(exc)
        # Parser continues — next INS opens a fresh member

What the Parser Extracts

Every segment handler maps elements to a field on the intermediate model. The element positions are fixed by the X12 specification — INS01 is always the subscriber indicator, INS02 is always the relationship code, INS03 is always the maintenance type. What changes between files is which values appear in those positions, and that's where companion guides make the spec a starting point rather than an answer.

The Layer Between

The parser emits an intermediate model. Not the canonical model from Part 3 — a raw representation of what the file said, with every field exactly as the file expressed it.

This layer exists because the translation from "what the file says" to "what the canonical model holds" involves decisions the parser shouldn't make. The parser's job is extraction. The mapper's job is translation.

Three examples of why the distinction matters:

Identity resolution. An 834 doesn't tell you whether this member already exists in your system. The file carries a subscriber ID, maybe a member ID, maybe an SSN. The mapper is where you check: does this identifier already resolve to a known person? The parser can't answer that question because the parser doesn't know anything about the database. Keeping it side-effect-free — no database reads, no database writes — makes it testable with nothing but strings.

Date conventions. The 834 sends DTP*349 (benefit end date) as the last covered day. December 31st means "covered through December 31st, inclusive." The canonical model uses half-open intervals: [valid_from, valid_to). Coverage through December 31st becomes valid_to = 2026-01-01 — the first day the coverage is not active.

Enum translation. The X12 sends gender as M, F, or U. The canonical model stores male, female, unknown. The X12 sends benefit status as a single character — A for active — but the canonical model derives termination from the maintenance type code, not from the benefit status. These are mapping rules, not parsing rules.

The Plus-One Rule

The date convention deserves its own section because it's the single most bug-prone mapping in healthcare enrollment data.

An employer HR system sends an 834 with DTP349 = 20261231. The HR administrator means "covered through the end of the year." The 834 specification says DTP349 is the benefit end date — the last day of coverage. The canonical model stores dates as half-open intervals: the start is included, the end is excluded.

The +1 rule: every DTP*349 value gets one day added. The mapper does this. The parser does not — the parser stores 2026-12-31 as a date, exactly as the file said. The intermediate model is the file's truth. The canonical model is the platform's truth. They disagree about what "December 31st" means, and the mapper is where that disagreement is resolved.

The Bug That Almost Wasn't

The mapper was working. Tests were passing. Canonical entities were landing in Postgres. I built a three-member fixture — a subscriber, a spouse, and a child — and ran it through the full pipeline.

Three persons created. Three PersonIdentifier rows created. Everything looked correct.

Then I looked at the identifier data.

All three identifiers had the same value: SUB002. The subscriber's REF*0F — their subscriber ID from the enrollment file. The spouse had it. The child had it.

This bug would not have surfaced in a single-member test. It required a family — a subscriber and at least one dependent sharing the same subscriber ID. The test that pins it creates a three-member family and asserts that exactly one PersonIdentifier row is created across all three members. The subscriber's.

The bugs that survive the longest are the ones that require two records to exist before they appear. Single-record tests are necessary. Multi-record tests are where the design actually gets tested.

The Benefit Status Trap

The build plan said to map INS05 values S and T to SUSPENDED and TERMINATED. The X12 spec says otherwise.

INS05 is "Benefit Status Code" — but the values don't mean what the labels suggest. A is Active. C is COBRA. S is "Surviving Insured" — a widow or widower who continues coverage after the subscriber's death. T is TEFRA — coverage that continues under specific Medicare regulations.

Both S and T mean coverage continues. Neither means terminated. Neither means suspended.

Termination in an 834 is expressed through INS03 — the maintenance type code. 024 is "Cancellation or Termination." When the parser sees INS03 = 024, the mapper derives BenefitStatus.TERMINATED, regardless of what INS05 says. A terminating transaction typically arrives with INS05 still set to A (Active) because the benefit status code reflects the member's category, not the transaction's intent.

Nothing in the codebase maps to BenefitStatus.SUSPENDED. A test asserts that explicitly — it walks every combination of INS03 and INS05 values and verifies that SUSPENDED never appears. The enum value exists in the canonical model because a future source (a claims status update, a manual override) might need it. But the 834 parser doesn't produce it, and a test makes sure of that.

The raw INS05 value is preserved in payer_extensions whenever it's anything other than A or empty. The mapping collapses S and T into ACTIVE for the canonical model, but the original wire value survives for payers who need it for their own reporting.

The Pipeline Comes Alive

POST /ingest is where everything connects. A file arrives. The platform reads it, stores it, identifies it, parses it, maps it, writes it, and records every step.

Every canonical write — persons, identifiers, coverages — happens inside a single database transaction. If the third member's coverage insert fails because of an overlap constraint violation, none of the three members are committed. Partial canonical state is worse than no canonical state: a person without their coverage looks like an uninsured member to anyone downstream.

The lineage rows go in the same transaction. A canonical record without a lineage record is an orphan — you know the person exists, but you can't trace them back to the file that created them. The lineage table records the raw_file_id, the entity_type, the entity_id, and the transformation that produced it. agent_confidence is NULL because this is deterministic parsing, not AI inference. That column earns its values in Part 5.

The raw_files record lives outside the canonical transaction. Status updates — detecting, detected, parsing, parsed, mapped, failed — are written as they happen, not inside the final commit. If the canonical write fails, the file is still marked as failed with an error_summary explaining what went wrong. The file's lifecycle is separate from the data's lifecycle, and a failure in one doesn't corrupt the record of the other.

Correlation IDs

The moment a raw_files record is created, bind_correlation_id(raw_file_id) runs. From that point forward, every log line — detection, parsing, mapping, writing, lineage — carries that file's UUID. When a member fails to parse at 3 AM, the correlation ID connects the error to the file, the file to the sender, and the sender to the employer. clear_context() runs in a finally block so the next request starts clean.

This is the "Correlation ID" pattern from Part 2 — trivial to implement, transformative to operate. The difference between "something failed" and "file abc-123 from ACME Corp failed on member 47 because DTP*349 contained 00000000 which is not a valid date."

What the Pipeline Holds

The ingest layer is the first part of the platform that touches real data. Everything before it — the canonical model, the query helpers, the migration scripts — was designed in isolation, tested against synthetic data, and validated against rules that existed on paper. The parser tested those rules against what files actually contain. Two bugs surfaced that no amount of schema design would have caught: the shared dependent identifier, the NUL bytes in metadata. Both were found by running code against data and reading the output, not by writing tests first. The tests came after, to pin the fix.

The platform now processes an 834 from raw bytes to canonical entities in Postgres:

One format down. Five to go.

The 834 parser handles the base spec — the HIPAA 5010 transaction set as defined by 005010X220A1. Companion guide variations — the payer-specific tweaks that make every real 834 slightly different from the spec — are noted and carried — validation belongs to the gate layer (Part 6). The parser's contract is: if the file is structurally valid X12, every record that can be extracted will be extracted, and every record that can't will be reported with the segment and position that caused the failure.

In Part 5, the AI agent layer arrives. The parser extracted fields mechanically — INS01 is always the subscriber indicator, NM1*IL always carries the name. The agent will handle what the parser can't: a flat CSV from a legacy HR system with columns labeled EMP_NAME and COV_START and no X12 structure at all. The parser reads formats that follow a specification. The agent reads formats that follow nothing but the intent of whoever exported them.

The forty-seven failed records from Part 1 haven't been fixed yet. The parser handles the mechanics. The gate layer (Part 6) will enforce the business rules. But between them — in Part 5 — sits the layer that can look at a companion guide variation and understand what the sender meant, even when what they sent doesn't match the mapping table. That's the gap this platform was built to close.

The parser reads the file. The agent reads the intent. The gate verifies the result. Three layers, three questions, each one blind to the others. That independence is the safety property.