When I started building this platform, I didn't set out to use patterns from 2003. I wasn't thinking about Microsoft's Patterns & Practices guides when I designed the five-layer pipeline. I wasn't consciously implementing the Strategy pattern when I defined the parser interface. Nobody opens a Gang of Four book and says, "I'll use a Chain of Responsibility here."
But when I stepped back and looked at what I'd built — the architecture diagram, the dependency rules, the way data flows through the system — every major structural decision maps to a pattern that was codified before half of today's developers started writing code.
Either these patterns are so deeply embedded in how experienced architects think that we use them without realizing it, or the problems they solve haven't changed in twenty years.
Both are true.
In Part 1, I laid out what I'm building: a healthcare data platform that ingests five incompatible data formats, uses AI to understand what the data means, and validates everything through deterministic rules before it touches a production system. I picked the stack — Python, PostgreSQL, Dagster, FastAPI, the Anthropic SDK — and explained why.
This article is about the layer beneath the stack. The architectural decisions that would be the same whether I wrote this in Python, C#, Go, or Java. The patterns that persist even when everything around them changes.
The Patterns & Practices Legacy
For architects who came up through the Microsoft ecosystem in the 2000s, Patterns & Practices was more than a book series. It was a way of thinking about systems.
The Enterprise Library. The Application Architecture Guide. The integration patterns that shipped alongside .NET 2.0. They taught you to see software as compositions of named, repeatable solutions — not as clever code, but as structural decisions with known consequences. Every pattern came with an intent, a motivation, a set of trade-offs, and a warning about when it would break.
The frameworks those patterns described are gone. BizTalk is in maintenance mode. WCF is a memory. The Azure equivalents have been renamed at least twice. But the intellectual discipline — name your architectural decisions, understand their consequences, document why you chose them over the alternatives — that never aged.
When I designed this healthcare data platform, five patterns from that era kept showing up. Not because I went looking for them. Because the problems demanded them.
The Pipeline That Patterns Built
The platform's architecture is five layers. Data enters at the top and flows through each layer in order. No layer skips ahead. No layer reaches back.
This is Pipes and Filters. One of the oldest architectural patterns in software — documented in the original POSA book in 1996, adopted by Microsoft's integration guidance, and still the right answer thirty years later for any system where data passes through a series of independent transformations.
Each layer is a filter. Data flows through the pipe between them. No filter knows or cares about the internal workings of the filters before or after it. The ingest layer doesn't know about AI agents. The gate layer doesn't know where the data came from. The canonical model doesn't know how many validation rules exist.
Pipes and Filters gives you something specific in healthcare: auditability at every stage. When a regulator asks "what happened to this enrollment record between receipt and storage," the answer is a walk through the pipeline — each filter's input and output, each transformation logged, each decision recorded.
In healthcare, this independence is more than good design. When an AI agent proposes a field mapping and a gate rejects it, the gate's decision has to stand on its own. It can't reason about what the agent was trying to do. It only knows the rules and the data in front of it. That independence is the safety property.
Five Layers, Five Patterns
Each layer in the pipeline maps to a pattern that was codified decades before this platform existed.
| Layer | Pattern | What it means here |
|---|---|---|
| Ingest | Gateway + Strategy | One entry point accepts any format. Content-based detection routes to format-specific parsers — one interface, many implementations. |
| AI Agent | Proposal (new) | The agent's output is always a proposal with a confidence score. Never a decision. |
| Gate | Chain of Responsibility | Schema → compliance → referential → business rules. Each check runs in sequence. Any error-severity failure stops the chain. |
| Canonical | Canonical Data Model | A unified representation that every source format maps into. One truth. The product. |
| Observability | Health Endpoint Monitoring + Correlation ID | Liveness and readiness probes with distinct semantics. A correlation ID bound at ingest rides through every layer. |
The Gateway in the ingest layer does something specific: detection is content-based, never extension-based. An .txt file containing an ISA segment is X12. A .json file containing a resourceType field is FHIR. The parser doesn't trust the filename — it reads the content and decides. Without calling it a Gateway, it's just a thing the ingest code does. With the pattern name, it's a structural commitment that every future developer understands.
The Strategy pattern is what makes the parser layer extensible without modification. When Part 4 adds the X12 834 parser and Part 5 adds HL7v2, neither changes the ingest orchestration code. The interface stays the same. The detection routes to a different implementation. Adding a new format is a new file, not a rewrite.
The Chain of Responsibility in the gate layer runs four categories of validation in sequence: schema, CMS compliance, referential integrity, business rules. Any error-severity failure stops the chain. The check that catches the problem is the check that reports the problem. The gate engine doesn't need to know which specific rule failed — it just knows the chain didn't complete.
The patterns persist because the problems persist. The stack changed — .NET to Python, SOAP to REST, stored procedures to AI agents. The problems didn't.
What Runs Today
The five-layer diagram is the target architecture. After Part 1, only the spine exists — the FastAPI app, the database, structured logging, and distributed tracing. Everything else is a documented placeholder waiting for its part in the series.
This is the honest picture. A health endpoint that separates liveness from readiness. A database pool that lets the app start even if Postgres is temporarily down. Structured logs and distributed traces from day one. The architecture is real. The pipeline it's built for is coming.
Request Lifecycle
A single traced request through the system shows how the patterns compose at runtime — the Gateway routes it, OpenTelemetry wraps it in a span, and every database call nests under that parent span:
The parent-child relationship between the HTTP span and the database span is what makes "which query was slow in this request" answerable. Health probes skip the span entirely — they're excluded at the instrumentor level, and the custom ProbeFilteringSpanProcessor catches the database noise that the exclusion misses.
The Composition Root
There's a file at the heart of the platform that does nothing interesting. app.py creates the FastAPI application, initializes the database pool, sets up tracing, and wires the route handlers. No business logic. No data transformation. No clever code.
It's the most important file in the repository.
In Dependency Injection terminology — straight from Mark Seemann's work, which grew alongside Microsoft's Patterns & Practices — app.py is the Composition Root: the single place in the application where all the pieces are wired together. Initialization order matters and is explicit. Logging configures at import time. The app creates after that. Tracing attaches after the app exists, because the OpenTelemetry FastAPI instrumentor wraps the ASGI app in middleware and needs a real app instance to wrap.
The Composition Root gives you a dependency graph you can reason about:
These import rules are architecture, enforced at the code level. The canonical model is the shared vocabulary — every layer can read it, none can modify it from the outside. Gates never import agents, which means the validation layer has no knowledge of how the AI works. If a gate could see the agent's reasoning, there'd be a temptation — in code or in thinking — to give the agent the benefit of the doubt. The gate doesn't give anyone the benefit of the doubt. That's its job.
The Composition Root also governs startup. From docker compose up to serving traffic, every step has a reason for its position in the sequence:
Logging configures at import time so that startup logs are structured. The app creates after that. Tracing attaches after the app exists because the OpenTelemetry FastAPI instrumentor wraps the ASGI app in middleware. The database pool initializes inside the lifespan because it's async and needs a running event loop. Every position is a consequence of a technical constraint, made explicit by the Composition Root.
AI Proposes, Rules Dispose — A Pattern for the AI Era
Patterns & Practices never wrote this one. They couldn't have. The technology didn't exist.
But the design thinking is the same: identify a recurring architectural problem, name the solution, describe its structure, and document its consequences. If the P&P team were writing this pattern entry today, it might look like this:
Pattern: Proposal-Gate
Intent: Separate fast, probabilistic AI inference from slow, deterministic validation in systems where correctness is non-negotiable.
Motivation: AI agents can parse unfamiliar data formats, infer field mappings, and propose transformations faster than any human. But they hallucinate. They make confident mistakes. In healthcare — where a wrong field mapping can cancel someone's insurance — speed without verification is a liability.
Structure: The AI agent produces a proposal: a structured output with a confidence score attached to every inference. The deterministic gate layer validates the proposal against schema rules, compliance checks, referential integrity, and business logic. Only proposals that pass every gate reach the canonical model.
Key constraint: The gate layer is pure code — no model calls, no network, no non-deterministic operations. Gates are testable, reproducible, and auditable. They produce the same result every time on the same input.
Schema encoding: In the database, lineage.agent_confidence is a nullable float. When it's 0.92, an AI agent inferred the mapping with 92% confidence. When it's NULL, deterministic code produced the record. That null is the pattern made visible in the data model — it distinguishes what a machine inferred from what a rule proved.
The consequences of this pattern are specific.
Every AI-produced record carries provenance. You can query the database for every record where the agent confidence was below 0.8 and review those cases. The gate layer is independently testable — you write tests against the gates without running the AI, without calling any API, without spending tokens. Gates are pure functions of their inputs. And when a gate rejects an agent's proposal, both the proposal and the rejection are stored. The audit trail doesn't just show what succeeded — it shows what was attempted and why it failed.
This is the platform's core architectural insight. The AI makes it fast. The gates make it safe. The pattern is the boundary between inference and proof.
Two things this diagram asserts. Rejection is recorded, not discarded — a rejected record produces gate_results rows explaining exactly which gate refused it and why. And lineage is written twice with different meaning: the agent's mapping carries a confidence score, while the deterministic store carries NULL. That single nullable column is how you later separate what was inferred from what was proved.
The confidence score in the schema is the boundary between what a machine inferred and what a rule proved — made queryable.
When the Textbook Meets the Database Driver
Patterns give you a starting point. Reality gives you the final answer.
I set up OpenTelemetry tracing — spans for every request, exported to Jaeger for visualization. Standard setup from the observability textbook. Then I noticed the health probes. Kubernetes-style liveness and readiness checks, hitting the endpoints every few seconds. The traces were drowning in probe noise.
The textbook solution: exclude the health routes from the FastAPI instrumentor. I did. The HTTP spans disappeared. But two database spans per probe kept showing up — parentless, cluttering the trace view. The readiness probe runs SELECT 1 to check if Postgres is alive, and asyncpg fires a pool-reset statement on connection release.
I tried OpenTelemetry's suppress_instrumentation() context manager. It did nothing. I read the asyncpg instrumentation source code. The package ignores OpenTelemetry's suppression key entirely — verified by tracing through the actual library code.
The fix: a custom ProbeFilteringSpanProcessor that drops root-only spans matching the known probe queries. Root-only is the critical qualifier — an identical SELECT 1 inside a real request keeps its span. Measured: sixty probes produce zero noise spans. Down from forty leaked spans before the filter.
The lesson isn't about OpenTelemetry or asyncpg. The lesson is about the relationship between patterns and the real world. The Health Endpoint Monitoring pattern told me to separate liveness from readiness. Pipes and Filters told me to keep the observability layer independent. Both were right. Both were necessary. Neither warned me that a third-party database driver would ignore the standard suppression mechanism.
Patterns are the architecture. Code is the negotiation between the architecture and the world as it actually is. When a pattern doesn't account for a library's behavior, you don't abandon the pattern. You build the bridge — a span processor, a workaround, a filter — and you document why it exists so the next engineer doesn't rip it out thinking it's unnecessary.
Documentation Is Architecture
The platform has a developer guide. Thirteen documents. Getting started, domain primer, architecture, codebase map, data model, runtime flows, observability, testing, workflow standards, roadmap, glossary, troubleshooting. The sequential reading path takes two hours. The structure is progressive disclosure — start with getting started, end with the roadmap, dip into the glossary when a term isn't clear.
This isn't documentation in the way most projects use the word. This is architecture made legible.
Patterns & Practices understood something that most engineering teams still don't: if a decision isn't written down, it isn't architecture — it's tribal knowledge that walks out the door when someone leaves. The developer guide documents not just what the code does but why the initialization order in app.py matters, why gates never import agents, why agent_confidence being nullable is a design decision rather than an oversight.
There's a rule in the guide that captures the whole philosophy: if a document and the code disagree, the code is right and the document is a bug — fix both in the same commit. Documentation that lives alongside the code, reviewed in the same pull requests, enforced by the same definition of done — that's what separates architecture from aspiration.
Every troubleshooting entry starts with a command to confirm the diagnosis before applying the fix. The guide doesn't just say "restart the container." It says "run this command to verify this is actually your problem, then restart." Diagnostic discipline is an architectural choice too.
The developer guide follows the MSDN model: sequential reading path for onboarding, reference sections for daily work, troubleshooting organized by symptom. Two hours to read sequentially. Seconds to find what you need when something breaks at 3am.
The Physical Architecture
All of this runs in Docker Compose on a single machine. No cloud provider. No Kubernetes. Three containers, two bind mounts, one volume.
The src bind mount gives uvicorn hot-reload without rebuilds. The data bind mount persists raw files across container replacements — raw healthcare files are immutable artifacts that must outlive the process that wrote them. The cloud decision is deferred by design. The article about deploying to production will be more honest for having lived through the need instead of making the choice abstractly.
What Persists
Twenty years ago, Microsoft's Patterns & Practices team sat down and codified how experienced architects solve recurring problems. They gave the solutions names. They documented the trade-offs. They wrote everything down so the next architect could build on that thinking instead of rediscovering it from scratch.
The frameworks they wrapped those patterns in are gone. The languages changed. The deployment targets changed. The fact that this platform has an AI agent layer — a concept that would have been science fiction when P&P published — changes the solution space entirely.
The patterns survived all of it.
Pipes and Filters structures the pipeline because data flowing through independent transformation stages is still the right architecture for traceable, auditable processing. The Composition Root organizes the initialization because understanding what depends on what is still the right way to manage complexity. Strategy shapes the parser layer because separating interface from implementation is still the right way to add capability without rewriting existing code. Chain of Responsibility governs the gates because running validations in sequence with early termination is still the right way to enforce mandatory checks.
And Proposal-Gate — a pattern that didn't exist twenty years ago — follows the same design instinct that produced every pattern in the P&P catalog: identify the recurring problem, name the solution, describe the boundary, make the consequences visible in the code.
The patterns persist because the problems do. Technologies arrive, peak, and fade. Frameworks get rewritten, renamed, and retired. The need for validated data, traceable transformations, and auditable decisions doesn't go anywhere. And maybe the most enduring thing isn't any single pattern — it's the discipline of naming your decisions and holding yourself to them. The habit of thinking in architecture rather than in code.
That discipline is what Patterns & Practices really taught. The rest was just examples.
In Part 3, I stop thinking about how the system is built and start thinking about what the system holds. The canonical data model — the unified representation that maps five incompatible formats into one truth. The foundation everything else depends on, and the decision that will make or break the platform.
