The file arrived on a Tuesday afternoon. An 834 enrollment transaction from a mid-size employer group — three hundred and twelve member records. New enrollments, a few plan changes, a handful of terminations. I'd processed thousands of files like this. Routine work.
The file passed X12 syntax validation. Segments, terminators, delimiters — all clean. The parser read every record without throwing an exception. By every technical measure, this file was valid.
Forty-seven member records failed to map.
The employer had switched HR systems six weeks earlier. Their new system generated X12 files that were structurally correct but semantically different from their old ones. The dependent loops were nested differently. The maintenance type codes used a companion guide variation that the mapping table didn't recognize. The coverage effective dates were technically valid but followed a convention the rules engine had never seen — effective dates set to the first of the month, termination dates set to the last day of the same month, which in the system calculated to zero days of coverage.
Forty-seven people whose health insurance enrollment didn't process. Forty-seven people who, if they walked into a doctor's office that week, would show up as uninsured.
The fix took three engineers two days. Custom mapping rules for one employer, one HR system, one companion guide variation. Rules that would break the next time that HR system updated. Rules that existed nowhere else in the codebase because no other employer generated files this way. The team shipped the fix, moved on to the next fire, and filed it under "this is just how it works."
It is how it works. That's the problem.
The 834 file was the easy part. The syntax was solved decades ago. What wasn't solved — what still isn't solved — is the gap between structurally valid and operationally useful. The difference between a file that parses and a file that means what it says.
That gap is where people lose their health insurance.
The Lifecycle Nobody Designed
Healthcare data doesn't live in one system. A single patient — one person walking into a doctor's office with a sore throat — generates data across at least five different formats, in five different systems, none of which were designed to talk to each other.
Follow this patient through a single care episode:
| Stage | What happens | Format | System |
|---|---|---|---|
| Enrollment | The patient's employer enrolls them in a health plan | X12 834 | Payer enrollment system |
| Eligibility | The provider checks if coverage is active | X12 270/271 | Payer eligibility gateway |
| Admission | The patient is admitted to a facility | HL7v2 ADT | Hospital EHR |
| Claim | The provider bills for services rendered | X12 837P | Claims processing system |
| Payment | The payer sends remittance to the provider | X12 835 | Payment/remittance system |
| Interoperability | The standard everything is converging on | FHIR R4 | The future |
Six formats across six systems for one patient's journey — and none of them were designed to talk to each other.
Each of these formats was designed independently. The X12 transaction sets were built in the 1970s for electronic data interchange — batch files transmitted over dial-up connections. HL7v2 was built in the 1980s for hospital systems that needed to share lab results and patient movements. FHIR was built in the 2010s as the modern answer — RESTful, JSON-based, designed for the internet.
None of them share a common data model. A member ID in an 834 is not the same field as a subscriber ID in a 270. A patient identifier in an HL7v2 ADT message uses a completely different format from a FHIR Patient resource. The same person — same name, same date of birth, same Social Security number — exists as a different data structure in every system they touch.
The hardest problem in healthcare data isn't parsing. Parsing is solved. The hardest problem is identity — knowing that the "member" in your enrollment system, the "subscriber" in your eligibility system, and the "patient" in your clinical system are the same human being.
And this is just one patient, one episode of care. A mid-size health plan processes hundreds of thousands of enrollment transactions a month. A hospital system runs fifty thousand eligibility checks a day. The data volume isn't the challenge — it's the fact that every file arrives in a format that was designed for a different era, a different system, and a different purpose.
What Exists — and Why It Falls Short
This isn't a new problem. Entire companies exist to solve healthcare data integration. I've worked with most of them over the past two decades.
Microsoft BizTalk was the standard enterprise middleware for EDI processing in the early 2000s. It could parse an 834 and route it to a database. It could transform an HL7v2 message into a flat file. But BizTalk is XML-heavy, infrastructure-dense, and in maintenance mode. Microsoft is pushing everyone to Azure Integration Services. The architecture is legacy. The talent pool is shrinking.
Informatica is the enterprise data integration platform — best-in-class for data quality, governance, and master data management. But it was built for batch ETL, not real-time healthcare transactions. Adding AI capabilities has been retrofit, not native. And the licensing cost alone would consume a startup's runway.
Boomi is the iPaaS answer — cloud-native, pre-built connectors for HL7 and X12, fast to deploy. Good for getting data from point A to point B. But the connectors handle format parsing, not clinical or business rule validation. When your employer sends an 834 with a companion guide variation that doesn't match the connector's expectations, you're back to writing custom code.
Stedi is the closest to what I'm thinking about. API-first EDI processing with free interactive X12 guides that are genuinely excellent. But Stedi focuses on EDI broadly — not healthcare-specific semantics. It doesn't understand that a member ID in an 834 represents the same person as a subscriber in a 270. It parses transactions. It doesn't model patient journeys.
Every platform I've used solves the same problem — data movement. Getting bytes from one system to another. Transforming format A into format B. They're plumbing. Excellent plumbing, in some cases. But plumbing doesn't understand what's flowing through the pipes. When a new employer sends a file that doesn't match your expectations, someone — a developer, an analyst, an engineer on the enrollment team — sits down and writes custom mapping logic by hand. That logic lives in a config file, or a stored procedure, or a Python script that one person maintains. And when that person leaves, the knowledge walks out the door with them.
The gap isn't parsing. The gap is understanding.
These platforms can tell you that segment INS*Y*18*021*28*A*E**AC is a valid X12 834 insured benefit record. What they can't tell you — what no existing platform can tell you without a human writing custom rules — is that this specific combination of qualifiers means "new enrollment of a dependent child under COBRA continuation coverage." That semantic understanding requires domain knowledge, not just format awareness.
The existing platforms solve data movement. The unsolved problem is data understanding — knowing what the data means, not just how it's structured.
What I'm Building
I'm building a healthcare data platform that closes the gap between parsing and understanding. A system that can ingest any healthcare data format, use AI agents to figure out what the data means, and then validate everything through deterministic rules before it touches a production system.
The core principle is four words: AI proposes, rules dispose.
The AI agent does the heavy lifting. When a new file arrives — an 834 from an unfamiliar employer, a flat CSV export from a legacy HR system, an HL7v2 message with non-standard segments — the agent parses it, infers field mappings, and proposes how each record maps to the canonical data model.
But nothing moves past a gate without deterministic validation. Schema checks, CMS format compliance, referential integrity, business rules — all code, all testable, all blocking. The AI makes it fast. The gates make it safe.
In healthcare, trust without verification is a compliance violation. AI can infer that a column labeled "DOB" maps to a date of birth field. But only a deterministic rule can verify that the date format is valid, the member is between 0 and 120 years old, and the value matches the enrollment record already in the system. AI without guardrails is a liability.
The platform has five layers:
| Layer | What it does | Why it matters |
|---|---|---|
| Ingest | Accepts any format — X12, HL7v2, FHIR, CSV, flat files. Detects format automatically. | You can't control what employers, providers, and payers send you. The platform has to meet them where they are. |
| AI Agent | Parses files, infers field mappings, proposes canonical model transformations. Uses sub-agents for format-specific logic. | This is the intelligence layer — the part that turns format awareness into semantic understanding. |
| Deterministic Gate | Validates every AI-proposed transformation against schema rules, CMS compliance checks, and business logic. Blocks anything that fails. | The safety layer. AI proposes, rules dispose. Nothing reaches the canonical model without passing every gate. |
| Canonical Model | A unified representation of members, providers, coverage, claims, payments, and encounters. One truth, regardless of source format. | The single source of truth. Every query, every report, every downstream system reads from this model. |
| Observability | Lineage tracking, data catalog, audit trail, anomaly detection. Every record traceable to its source, every transformation auditable. | Healthcare compliance demands full traceability. But beyond compliance — you can't debug what you can't see. |
This architecture is opinionated. The AI layer is not optional — it's the core differentiator. The gate layer is not a nice-to-have — it's what makes the AI trustworthy. The canonical model is not a theoretical exercise — it's the product. Everything else is plumbing to get data into the model and confidence that the data in the model is correct.
The Decisions Before the Code
Every article in this series will cover a major engineering decision. Before I write a line of platform code, there are decisions that frame everything that follows. Here's what I've locked down and why.
Python
The language decision came down to four candidates: Python, Go, C#/.NET, and Java.
Go would give us performance and concurrency. C# would give us a strong type system and enterprise credibility. Java would give us the largest ecosystem of enterprise healthcare libraries and the deepest talent pool. All three are serious choices for serious infrastructure.
I chose Python. Here's why.
The AI agent layer is the platform's differentiator. Every major AI framework — the Anthropic SDK, LangChain, LlamaIndex, the tooling around prompt engineering and agent orchestration — is Python-first. The healthcare data libraries I need are Python-native: hl7apy for HL7v2 parsing, fhir.resources for FHIR R4 with Pydantic validation, pyx12 and TigerShark for X12 transaction sets. Pydantic — the runtime validation library — is practically built for this use case. Define a canonical model as Pydantic models, get schema validation, JSON serialization, and type safety that catches healthcare data errors at runtime.
I'm using Python with discipline: strict type hints everywhere, Pydantic models for all data structures, and mypy in CI. This isn't a Jupyter notebook. It's infrastructure code with the type safety guardrails that Python is often criticized for lacking.
The audience matters too. This series shares architecture decisions publicly. Python code is readable in a blog post. C# generics, Java boilerplate, and Go interfaces need more context to follow. For a series about decision-making, the language that lets me show my thinking most clearly is the right one.
PostgreSQL
Not much to deliberate here. Healthcare data has relational integrity requirements — members belong to coverage plans, claims reference providers and members, payments reference claims. Foreign keys matter. Referential integrity matters. Postgres gives us that, plus JSONB for the variable fields that every payer handles differently. A FHIR R4 resource is a JSON document — JSONB columns handle them natively.
The raw ingest layer stores original files — the actual 834s, the HL7v2 messages, the CSVs — in object storage (local filesystem for now, S3 when it goes to cloud). Postgres holds the canonical model. Raw files are immutable artifacts. Parsed, validated, canonical data is relational.
Dagster
The orchestration framework for data pipelines. I chose Dagster over Airflow, Prefect, and plain async queues.
Airflow is the industry standard, but it's a workflow scheduler — it thinks in tasks and DAGs. Dagster thinks in assets. An asset is a piece of data that your pipeline produces — a parsed enrollment file, a validated member record, a canonical coverage entity. Dagster tracks which assets exist, when they were last computed, and what upstream assets they depend on. That's data lineage built into the orchestration layer, not bolted on after the fact.
For a healthcare data platform where every transformation needs to be traceable and every record needs to be auditable, having lineage as a first-class concept in the orchestration framework is the difference between building observability and retrofitting it.
FastAPI
The API layer. The platform needs endpoints for eligibility queries, template management, and eventually a UI for the enrollment team to review AI-proposed mappings before they hit production. FastAPI gives me async request handling, automatic OpenAPI documentation, and native Pydantic integration — the same models that define the canonical schema also define the API contracts.
Docker from Day One
Local development runs in Docker Compose — Postgres, the application, and eventually Dagster's scheduler all in containers. Reproducible environments, no "works on my machine" conversations. When the cloud decision comes (and it will — that's a later article), the containerized architecture means the migration is deployment configuration, not application rewrite.
I'm staying local-only for the MVP. No cloud provider decision yet. That choice is better made after I've felt the pain of needing to deploy — and the article about it will be more honest for having lived through the decision instead of making it abstractly.
Testing with Real Formats, Fake Patients
The test data strategy matters. I'm using:
- Synthea — generates synthetic patients in FHIR R4, C-CDA, and CSV. Over a hundred clinical conditions modeled. Realistic patient journeys with encounters, conditions, medications, and claims.
- CMS SynPUF — synthetic Medicare claims data that matches real CMS file layouts. The closest you can get to production claims data without touching PHI.
- X12.org sample transactions — official examples for 834, 835, 837, and 270/271. The reference implementations.
- Stedi's EDI Inspector — paste any X12 file, get validation and JSON conversion. Invaluable for debugging parser output.
Real formats, fake patients. The test data looks like production data because it follows the same specifications. But no real person's health information is involved, ever.
The Series
This article is Part 1 of nine. Each part covers a major engineering decision — the kind of decision that shapes everything downstream. I'm building the platform with Claude Code in a private repository. The articles share the thinking, the architecture, and the tradeoffs. The code stays private.
Two projects, one story.
In Part 2, I go deeper into the architecture — the enterprise patterns that shape the platform's five-layer design, the dependency rules that keep each layer honest, and a new pattern for the AI era that Patterns & Practices never wrote.
Part 3 designs the canonical data model — the unified representation that maps five incompatible formats into one truth. Part 4 builds the ingest layer — the X12 834 parser, format detection, and the pipeline skeleton that processes incoming files. Part 5 introduces the AI agent layer — how I use Claude to infer field mappings from unknown file formats. Part 6 is the gate layer — deterministic validation that makes AI output trustworthy. Part 7 adds observability and lineage. Part 8 wires everything end-to-end: an 834 enrollment file arrives, gets parsed, mapped, validated, and stored with full traceability.
Part 9 is the honest retrospective. What I got wrong, what I'd do differently, and what the real numbers looked like versus what I planned.
I designed it on paper in the Solution Architecture series. Now I'm building it for real.
Why Build in Public
The best engineering blogs — Netflix, Stripe, Uber — share decisions, not implementations. They tell you why they chose Cassandra over DynamoDB, not how to configure a Cassandra cluster. They show you the architecture diagram and the tradeoff matrix, not the repository. The value is in the reasoning, not the code.
That's what this series is. The private repository is where I build. These articles are where I think out loud. And thinking out loud invites the most valuable kind of feedback — the engineer in Bangalore who's solved this problem differently, the enrollment operations lead in Minneapolis who knows the 834 companion guide edge cases I haven't hit yet, the architect who sees a flaw in my approach before I discover it the hard way.
If you work with healthcare data, you've felt this pain. If you've built data platforms, you've made these tradeoffs. If you've used AI in production systems where trust and compliance are non-negotiable, you know why "AI proposes, rules dispose" isn't a slogan — it's a survival strategy.
The platform starts with enrollment. An 834 file, a canonical model, and the forty-seven member records that shouldn't have failed.
I'm building the system that makes sure they don't.
