A multi-agent pipeline for technical diagnosis
A script holds the control flow. The agents do the thinking. Nothing gets written until an independent run has confirmed it.
This page is an architecture write-up, not a code release. The tool runs inside my employer's environment against internal systems, so the source, the system names and anything it has ever processed cannot appear on a public site. What follows is the design and the reasoning, which are mine to discuss. Numbers describe the shape of the codebase, not any customer or any outcome.
The problem
Deep technical investigation starts cold every single time. Each new problem begins with the same unpaid work: assemble the context, find out what has already been tried, work out which of a hundred diagnostic tools is the relevant one, and only then start thinking. The thinking is the valuable part and it is the smallest part of the elapsed time.
The obvious move is to hand the whole thing to a language model and ask it to investigate. I tried that. It fails in a specific and instructive way: a model holding its own control flow will declare victory early. It finds a plausible cause, stops looking, and writes a confident answer. For diagnosis work, confidently wrong is materially worse than slow.
The approach
So I inverted it. A plain script owns the control flow; the agents are called into fixed slots. The step order is not a decision the model gets to make. This one choice is where most of the reliability comes from — the pipeline cannot skip verification to save effort, because skipping is not something it is able to express.
Four ideas do the work:
- A rule-based depth gate. Not every problem deserves the full pipeline. A deterministic check decides whether to hold, draft only, or investigate properly. Deciding this in code rather than by asking a model keeps cost proportional to difficulty and makes the decision auditable.
- Redundant parallel investigation, then reconciliation. Three investigators run over the same problem independently and without seeing each other's work, and a fourth pass reconciles them. Where they agree, confidence is earned rather than asserted. Where they disagree, the disagreement is itself the most useful signal in the whole run — it points precisely at the part that is genuinely ambiguous.
- Build, then independently verify. A reproduction is constructed in my own sandbox, and then a second, separate run has to reproduce it again from scratch before the finding is allowed through. A claim that only survives the run that produced it has not been tested.
- Contract-driven steps. Every step declares the artifacts it must produce. Coordination between agents happens entirely through files in a shared directory — no message bus, no shared memory, no live protocol between agents. Any step can be inspected, re-run, or resumed on its own, and the whole run leaves a readable audit trail on disk as a side effect rather than as a feature someone had to build.
Architecture
Four layers, each depending only on the one below. Backends wrap the systems that hold the data. Tools are self-contained scripts, each doing one thing, each independently runnable and testable. A shared core holds authentication and the common data handling. Orchestration sits on top and is deliberately the thinnest layer.
Files are the integration surface throughout. It is unfashionable and it is the reason the thing is debuggable: when a run goes wrong, the intermediate state is sitting on disk in a format I can read.
Tradeoffs
- Redundant investigation costs three times as much.
- Three parallel investigators mean three times the compute for one answer. I judged that worthwhile because the failure mode being bought off — a confident wrong conclusion reaching a person who then acts on it — is far more expensive than the compute. That trade would not hold for a low-stakes task.
- A fixed step order cannot adapt.
- A model-driven orchestrator could shortcut an obvious case or invent a step I never anticipated. Mine cannot. I gave up that flexibility for predictability, and the depth gate recovers some of it in a way I can inspect. If the shape of the work changed substantially, this would be the first constraint to bite.
- File-based coordination does not scale out.
- It assumes a shared filesystem and a single host. That is fine for what this is and would be wrong for a real distributed service, which would need a proper state store and an actual workflow engine.
- Some of it is only exercised on synthetic input.
- Parts of the pipeline have run end to end on real work; other paths have only been proven against test fixtures. I try to be precise about which is which, because a demo that quietly conflates the two is how people end up trusting something that has never actually run.
Outcome
The part I did not predict is the most useful: the reconciliation step turned out to be the highest-value component, not the investigation. Three independent passes disagreeing is a far better signal about where the real uncertainty lies than any single confident answer, and it is information that a single-agent design structurally cannot produce.
The second surprise was how much of the value came from writing things down in a fixed structure. Once every run produced the same artifacts in the same shape, patterns across separate problems became visible — a large family of superficially different cases collapsed into a much smaller set of recurring root causes. That was a consequence of the contract, not a feature anyone designed.
What I would do differently
Build the verification gate first. I built investigation first because it is the interesting part, and spent longer than I should have trusting outputs that had not been independently confirmed. Everything that made this reliable is downstream of "prove it twice", and that should have been step one rather than an improvement I arrived at.
I would also keep the documentation honest from the start. The write-ups drifted from the code faster than I expected, to the point where the docs described an earlier design. Now I verify claims against the source before repeating them, including on this page.