Nine agents, one system: what actually breaks in production

On 11 August, @hanakoxbt published an eight-step piece on graph engineering — going from one prompt to a hundred agents running in one system. It did 1.25 million impressions, which tells you how many people are currently trying to build this. It is a genuinely good piece and I agree with most of it.

I am writing this because I run one of these systems, today, and the article stops at the point where my problems started. I have nine agents live on WhatsApp: a manager agent that routes, brand agents, a client manager, and one agent per client. It runs on a Mac in my office. It is not a demo.

Everything below happened. Most of it happened on a single day — 23 August 2026 — which is not a coincidence: once you have enough agents in one system, the operational failures arrive in a batch. If you want the conceptual grounding first, I wrote that up separately in graph engineering: stop giving AI agents a to-do list. This piece assumes it and goes to the operations.

The part of the article that is exactly right

Three of the eight steps I would defend against anyone.

Most edges are not real dependencies. An edge should mean the next step reads the previous output — not that you happened to think of them in that order. Sequence is not dependency. A chain of twelve steps usually hides three real dependencies and nine habits. That distinction is the entire source of the “hundred agents”: you do not add agents, you delete imaginary arrows and discover you already had parallelism.

A node you cannot describe cannot be routed. One job nameable in three words, an explicit input, a structured output, and — the part people skip — a named failure state. A failure returned as data can be routed. A thrown exception stops the graph. The difference between those two lines is the difference between a system that degrades and a system that dies.

A join is a decision, not a formality. Put a barrier after every stage and you have converted your fan back into a chain with extra steps. Join only where the next node genuinely needs the complete set — deduplication, ranking, comparison, coverage checks. And one failed branch must not take the other ninety-nine with it.

I would add one thing to the fan pattern from running it: the slice each instance gets has to be disjoint, and you have to be able to say so in a sentence. If two instances can touch the same file, the same record, the same conversation, you do not have a fan. You have a race with a nicer diagram.

Step 7 is the one that bites

The article calls state “the part the diagram hides.” That is understated. State is the part the diagram hides and the part that will take your system down.

Here is mine. Each agent has a watcher process that picks up incoming WhatsApp messages and hands them to the agent. To avoid processing the same message twice, the watcher marked it as read. Sensible. It marked it as read before running the agent.

So: message arrives, watcher marks it read, watcher starts the agent turn, the process restarts mid-turn. The message is read. The turn never finished. There is no record that it ever started. From the outside, nothing happened — and worse, nothing can happen, because the only trace of the instruction was the unread flag we had already cleared. The person who sent it just gets silence, forever.

That is precisely the article’s “the graph has no idea what already happened,” and reading about it did not stop me shipping it. The fix was to persist the in-flight turn to disk at the moment it starts — the message, the agent, the timestamp — and replay any unfinished turn on boot. Read-marking moved to after completion.

Two things I would now treat as non-negotiable, both of which the article names and both of which I learned the expensive way:

• Pass references, not transcripts. The moment you hand whole conversations between nodes, your state is unbounded and your costs are unpredictable. Write the artifact, pass the path.
• Make writes idempotent. The whole point of durable state is that you can replay it. If replaying sends the message twice, you have not built recovery, you have built a way to annoy your clients twice as fast.

The test I use now: can the system answer what happened, why this route, and where can it safely resume? If the answer to any of those is “check the logs,” the state is not durable. Logs are for humans. State is for the system.

A health check that isn’t one is worse than none

This one is not in the article at all, and it cost me more than anything that was.

Having had watchers wedge, I added a stall watchdog: if a watcher looks stuck, restart it. Reasonable. The watchdog decided “stuck” by elapsed time. Four minutes without completing, restart.

Agent turns are not four-minute jobs. A real turn — reading a repo, running a build, checking a live URL — runs much longer than that. So the watchdog looked at an agent doing exactly what it was asked to do and killed it. Twice, on real work, before I caught it.

The bug was not the timeout value. Raising it just moves the line. The bug was that the check could not distinguish busy from wedged — and those are different states that happen to look identical from outside. The fix was to make the agent emit a heartbeat while working, so the watchdog restarts on absence of progress rather than presence of duration.

The general form, which I would now put in front of anyone adding monitoring to an agent system: monitoring that cannot tell working from dead will kill the working. An unmonitored system fails in a way you eventually notice. A badly monitored one manufactures failures and hides them behind the word “restart”. If you are about to add a supervisor, define what “alive” means as a positive signal the worker sends, not as an absence of an ending.

Silence reads as failure

A related failure with an entirely non-technical cause. Some of my agent turns take twenty minutes. That is fine — the work takes twenty minutes.

The problem is that to the person who sent the message, a twenty-minute turn and a crashed agent produce exactly the same experience: nothing. So they send it again. Now you have two turns running on the same instruction, and if your writes are not idempotent, congratulations, you have shipped it twice.

The fix was small: any turn still running at ninety seconds posts an acknowledgement. Not a progress bar, not an estimate — just evidence of life.

I mention it because the multi-agent literature is almost entirely about machines talking to machines, and every one of these systems that is worth building has a human waiting at one end of it. Liveness is a user-facing feature, not an ops concern. The same signal that stops your watchdog killing a busy agent also stops your user duplicating the request. Build it once, use it in both places.

Step 6, tested: the agent approved its own work

The article’s sixth step is that the most valuable node produces nothing — a verifier, sitting on the edge between the generator and everything downstream. Never let one agent produce, approve and publish in one context, because it will approve: the review is drawn from the same distribution that produced the work.

I tested this the lazy way, by not doing it. An agent finished a piece of work and was asked, in the same session, to verify it. It approved it. Of course it did. It had just spent a context window convincing itself that what it built was correct; asking it to re-read that with fresh eyes is asking it to be a different model.

This is the cheapest correct thing in the whole article and the one most likely to be skipped, because a verifier node looks like pure overhead on the diagram. It produces nothing. It just says yes or no. It is also the only thing standing between your system and confidently shipping something wrong at machine speed.

Where I would push further than the article: a separate agent is the minimum, not the goal. Two instances of the same model reading the same flawed assumption will agree with each other enthusiastically. The verification that actually holds is evidence from outside the system — a test that executes, a build that compiles, a live URL that returns 200, a human who looks at it. “Another agent said it looked fine” is decoration with a latency cost.

Environment is state too

One that is missing from every version of this I have read, and it is a nasty one.

One of my agents talks to a service through an MCP server. That server reads its credentials once, at process start. The credentials changed. Every agent process spawned before the change kept failing — forever — while a freshly spawned one worked perfectly.

Two things make this worse than a normal outage. First, it is not uniform: half your fleet works, half does not, and the split correlates with nothing you would think to look at. Second, and this is the part that wasted the most time, retrying inside the same session could never succeed. Every retry strategy assumes the world might have changed since the last attempt. If the process snapshotted the world at boot, retry is a loop that cannot terminate in success.

So durable state is not just what your graph writes down. It is also what your processes captured at start-up and will never re-read: credentials, config, tool handles, connection pools. If a failure could plausibly be caused by stale environment, the correct recovery is not retry. It is replace the process. That belongs in the failure taxonomy alongside the article’s named failure states — stale_environment routes to respawn, not to backoff.

The shape is your cost model

The article’s eighth step cites the number everyone building this should have in their head: Anthropic reported that its multi-agent research system beat a single agent on breadth-first work while using roughly fifteen times the tokens.

Fifteen times. That is not a rounding error you optimise away later; it is the economics of the entire architecture. Fanning out is buying breadth with money. It is a good trade when the work is genuinely wide and independent and the answer justifies the spend. It is a terrible trade when one context could have held the whole problem, which — for most requests that arrive at most businesses — it could.

The routing that follows from that is unglamorous and effective:

• Cheap models for bounded work — extraction, classification, formatting. Anything where the output space is small and checkable.
• Strong models for decomposition, synthesis and hard verification. These are the nodes where being wrong is expensive and being cheap is false economy.
• Short paths for simple requests. Most inbound work should exit the graph after two nodes. If your simplest request traverses your whole topology, you have built a toll road.

The article’s split on this — let the model judge, let the graph decide — is the right one and it is worth restating because it is easy to get backwards. The classifier is probabilistic and returns a label. The route table is deterministic and maps labels to paths. Model flexibility on judgement, no improvisation on authority. The side benefit is the one you feel at 2am: when something goes to the wrong place, you can see whether the label was wrong or the table was, and those have completely different fixes.

When to keep it to one agent

I want to be blunt about this, because “a hundred agents” is a great headline and a bad default.

One agent in one loop is correct for short tasks, and most tasks are short. Reach for a graph when at least one of these is true:

1. The branches are genuinely independent — not just separately describable, but able to progress without reading each other’s unfinished output.
2. Nodes need different tools or permissions. This is the underrated one. Separating agents is often about blast radius, not speed — the node that can send client emails should not be the node that can also delete things.
3. Output needs verification by something that did not produce it.
4. Runs must survive interruption. If losing the process means losing the work, you need durable state, and durable state pushes you toward a graph whether you wanted one or not.
5. Cost or authority needs routing — cheap models on cheap work, human approval on consequential work.

None of those say “because it would be impressive”. If one context can hold the problem, use one context. The nine agents I run exist because they have genuinely different jobs, different clients and different permissions — not because nine is better than one.

What I would build first, in order

If you are standing up a multi-agent system now, the order that would have saved me the most time:

1. Durable state before anything else. Current node, completed nodes, artifacts, decisions with evidence, retries, approvals. Written at the start of a turn, not the end.
2. Idempotent writes. Before you have anything worth replaying, so that when you do, replay is safe.
3. A heartbeat. One positive liveness signal, used by both your watchdog and your user-facing acknowledgement.
4. One verifier outside the worker. Even a crude one. Especially a crude one that runs a real command.
5. Named failure states, including stale_environment. Failures that route, not exceptions that stop.
6. Only then, the fan. Widen the graph once the narrow one survives being interrupted.

Every item on that list is boring, and every one of them is the reason the interesting part stays up. The topology gets the diagram; the plumbing gets the uptime.

The short version

The eight steps are a good map and I would hand them to anyone starting out. But the map stops at the boundary of the process. My failures were all on the other side of that boundary: state that did not survive a restart, monitoring that could not tell work from a hang, silence that read as death, self-review that always said yes, and an environment that was captured once and never re-read.

None of those are graph problems. They are the problems you get the day after you solve the graph problem. Build for them first and the hundred agents are almost easy.

Related reading

• Graph engineering: stop giving AI agents a to-do list — the conceptual primer this post assumes
• How I set up AI employees — what the agents in this system actually do all day
• Why AI projects fail — the non-technical reasons, which are most of them
• How to spec a custom AI system — the one-page brief that becomes the graph