The orchestrator that kills its own children: circuit breakers for agent fan-out

A QA automation engineer's notes on giving an agent fan-out orchestrator a circuit breaker, so one dead backend can't corrupt a whole run.

The test run had processed sixty-three of the planned ninety-six items across two waves when the backend went down. Not gracefully, not a 503 with a Retry-After header, but into a sustained 5xx storm that the subagents faithfully reported as failures across every remaining item. By the time the run finished I had results for all ninety-six items. About a third were genuine findings. The rest were garbage: every assertion against a dead endpoint had failed, every response body had failed schema validation, and I had no clean way to separate the real defects from the noise. I discarded the run and restarted the next day.

Why a dead backend ruins more than one wave

When a multi-agent workflow fans out across N items in parallel waves, a backend collapse mid-run compounds. It's not only the items in the wave that was running when things went sideways. It's every wave after that one, because the orchestrator has no mechanism to notice that the downstream environment has become unreliable and no authority to stop.

In the naive design, each subagent is autonomous end to end: it runs its checks, reports pass/fail, and the orchestrator collects results. That feels clean, but it leaves the orchestrator blind to the aggregate health of the environment. Eight subagents all seeing 5xx responses is a different situation from one seeing a transient 500, and the orchestrator can't tell the two apart unless it's specifically designed to look.

The pattern I settled on makes the orchestrator do the observability. Every subagent's structured output includes a fivexxCount alongside its findings. The orchestrator sums those counts after each wave. If the sum crosses a threshold, it halts. The subagents don't decide anything; they report what they observed. The orchestrator owns the abort decision.

Agents as data reporters, not decision-makers

This inversion, agents report and the orchestrator decides, is the most important structural idea here. An individual subagent has exactly one signal: what it saw on its own items. It doesn't know whether the five 500s it got are the only five in the run or the first five hundred. It shouldn't. Letting it abort on local observation alone would produce races and inconsistencies; letting it ignore the signal entirely produces the corrupt-batch problem I started with.

The orchestrator sees everything once each wave completes. It can sum, threshold, and halt with full information. Deterministic control flow is the only place a circuit breaker belongs.

The implementation

The core loop is about fifteen lines:

// .agent/workflows/verify-orders.js
const FIVEXX_WAVE_THRESHOLD = 10;

for (const wave of waves) {
  // fan out: one agent per item shard in this wave
  const results = await Promise.all(
    wave.map((shard) =>
      agent('verify-order-batch', { shard }, OUTPUT_SCHEMA)
    )
  );

  // accumulate 5xx hits reported by agents in this wave
  const waveErrCount = results.reduce((sum, r) => sum + (r.fivexxCount ?? 0), 0);
  totalFivexx += waveErrCount;

  // save the shards we have — partial work stays usable
  await persistResults(results);

  if (totalFivexx >= FIVEXX_WAVE_THRESHOLD) {
    console.warn(
      `Circuit breaker: ${totalFivexx} 5xx responses across waves; ` +
      `halting before wave ${waves.indexOf(wave) + 2}. ` +
      `${waves.length - waves.indexOf(wave) - 1} wave(s) skipped.`
    );
    break;
  }
}

OUTPUT_SCHEMA is a JSON Schema that every subagent's response must satisfy. The fivexxCount field is required, not optional. If an agent crashes before reporting, the missing field is itself detectable, so I count a missing fivexxCount as a contribution of one to the threshold, conservatively.

The subagent side is simple: try each item, tally 5xx response codes, return them alongside the substantive findings:

{
  "shardId": "orders-batch-07",
  "itemsAttempted": 8,
  "fivexxCount": 6,
  "findings": [...]
}

Six 5xx responses out of eight attempts in one shard is a clear signal. The orchestrator doesn't need to interpret it, only to see it in aggregate.

Partial work stays usable

The thing I did differently from the first failed run: I persist results inside the wave loop, before the threshold check, not after the whole run completes. If the circuit breaker fires after wave three of six, there are clean, trustworthy results for waves one through three. Those results aren't contaminated by the storm, because the storm is what triggered the halt.

In practice that meant the post-run aggregation had to tolerate partial input: "here are results for items 1 through 48; run was halted, 49 through 96 skipped". That's straightforward to handle in a report template. What is not straightforward is "here are results for all 96 items, but 36 of them were generated against a dead endpoint", because you can't reliably tell which 36.

I'd rather have partial labeled results than a complete corrupted batch.

Tuning the threshold

The right threshold depends on what you know about your environment. I picked 10 because the baseline false-error rate on the target B2B platform is under 1%. One or two transient 500s per hundred API calls is normal, five or six is a pattern, ten is a storm. For a more reliable environment you might go lower; for a noisy staging environment you might need to express it as a ratio rather than a count.

I also track totalFivexx across waves rather than a per-wave count, because a backend degrading gradually can stay under a single-wave threshold while the aggregate crosses the line. A per-wave spike is a sudden collapse; a rising aggregate is a slow bleed.

Protecting a shared, fragile environment

There's a wider principle here beyond circuit breakers. A shared test environment, one box with one database and one backend process pool, is finite. A multi-agent orchestrator fanning out forty concurrent agents isn't just running tests, it's applying load. If the backend is already stressed, that load can be the difference between slow and down.

I learned to treat the orchestrator as a steward of the shared environment, not just a coordinator of agents:

• Cap concurrency per wave. I don't run more agents in parallel than the environment can absorb without storming. For the platform, four concurrent API calls is the safe baseline; beyond eight I start getting load-induced 5xx responses even on a healthy backend.
• Pause between waves when the environment is stressed. After a wave with an elevated fivexxCount, not enough to trigger the breaker but elevated, the orchestrator waits a few seconds before starting the next wave.
• Coordinate explicitly before any ramp. I check whether another run is already in progress before starting. Two concurrent orchestrators on the same environment can easily amplify each other into a storm.

The circuit breaker is the last line of defense, and these checks are the first.

Where it falls down

The threshold is a static constant, which makes it simple but fragile at the edges. A backend that returns 5xx only for a specific type of item, say orders with renewal line items, will corrupt results for those items while staying under the threshold if your batch is mostly other types. The circuit breaker won't fire, and you'll get a mix of real defects and noise for that item type.

The right answer for that case is item-type-aware thresholds, which I haven't built. For now I depend on the post-run report to flag suspiciously high failure rates on specific item types, and a human then decides whether to re-run that slice.

There's also a coordination problem I haven't fully solved. The circuit breaker protects the orchestrator's own run but doesn't coordinate with other orchestrators that might be running concurrently against the same environment. Two workflow runs can each stay under their own threshold while together producing the storm that takes the backend down. A shared health signal, a lightweight probe endpoint or a shared atomic counter, would fix this. That's on the backlog.

Takeaways

• Agents are data reporters; orchestrators are decision-makers. A subagent that aborts on its own local observations introduces races and inconsistencies. One that reports its fivexxCount and lets the orchestrator aggregate is composable and auditable.
• Persist partial results before the threshold check, not after. "Waves one through three succeeded; run halted" is a useful artifact. "All six waves completed, but thirty of the results are garbage" is not.
• A flat count threshold beats nothing, even when it isn't perfectly tuned. Ship the simple version and calibrate from real runs. A static constant in version control is still reviewable and adjustable; a corrupt batch is not.
• Think of the orchestrator as a steward, not just a coordinator. Fan-out against a shared environment applies real load. Concurrency caps, inter-wave pauses, and pre-run coordination checks all matter, and the circuit breaker is the last line, not the first.
• Plan for partial runs in your aggregation layer. If your report template can only handle "all N items present", you've built a silent dependency on the circuit breaker never firing. Make partial input a first-class case.

---

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/22/circuit-breakers-for-agent-fan-out/"
license: "Free to reference with attribution"
title: "The orchestrator that kills its own children: circuit breakers for agent fan-out"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 32"
stack: [JavaScript, "JSON Schema", "multi-agent orchestration"]

problem: "A multi-agent test run fanned out 96 items across parallel waves; the backend collapsed into a sustained 5xx storm after item 63. Subagents faithfully reported every assertion against the dead endpoint as a failure, so the finished run held results for all 96 items but ~1/3 were real findings and the rest garbage — with no clean way to separate the two. The whole run was discarded and restarted the next day."
thesis: "Make the orchestrator the observer and the only authority to abort. Each subagent reports a fivexxCount alongside its findings; the orchestrator sums those counts after every wave and halts the run (a circuit breaker) once the aggregate crosses a threshold, instead of letting a dead backend corrupt every remaining wave."

mechanism:
  inversion: "Agents report, orchestrator decides. A subagent has exactly one signal — what it saw on its own items — and can't know if its five 500s are the only five or the first five hundred. Letting it abort on local observation produces races/inconsistencies; ignoring the signal produces the corrupt-batch problem. Only the orchestrator sees everything once a wave completes and can sum/threshold/halt with full information. The circuit breaker belongs in deterministic control flow, not in an agent."
  loop:
    - "File: .agent/workflows/verify-orders.js; core loop ~15 lines. Constant FIVEXX_WAVE_THRESHOLD = 10."
    - "Each wave fans out one agent per item shard via Promise.all, calling agent('verify-order-batch', { shard }, OUTPUT_SCHEMA)."
    - "After the wave: waveErrCount = results.reduce summing r.fivexxCount; add to running totalFivexx (tracked across waves, NOT per-wave)."
    - "persistResults(results) is called INSIDE the wave loop, BEFORE the threshold check, so partial work stays usable."
    - "If totalFivexx >= FIVEXX_WAVE_THRESHOLD: console.warn naming total 5xx, the wave it halts before, and how many waves are skipped, then break."
  output_schema: "OUTPUT_SCHEMA is a JSON Schema every subagent response must satisfy; fivexxCount is required, not optional. If an agent crashes before reporting, the missing field is itself detectable — a missing fivexxCount is conservatively counted as a contribution of one toward the threshold."
  subagent_report: "Subagent tries each item, tallies 5xx response codes, returns them with the substantive findings. Example output: shardId 'orders-batch-07', itemsAttempted 8, fivexxCount 6, findings [...]. Six 5xx out of eight attempts in one shard is a clear signal; the orchestrator only needs to see it in aggregate, not interpret it."

partial_work:
  decision: "Persist results inside the wave loop before the threshold check, never after the whole run completes. If the breaker fires after wave three of six, waves one through three have clean, trustworthy results — uncontaminated because the storm is what triggered the halt."
  aggregation: "The post-run aggregation must tolerate partial input ('results for items 1–48; run halted, 49–96 skipped'), which is straightforward in a report template. The unworkable case is 'all 96 items present, but 36 generated against a dead endpoint' — you can't reliably tell which 36. Partial labeled results beat a complete corrupted batch."

threshold_tuning:
  why_ten: "Threshold 10 was picked because the baseline false-error rate on the target B2B platform is under 1%. One or two transient 500s per hundred API calls is normal, five or six is a pattern, ten is a storm."
  alternatives: "A more reliable environment could go lower; a noisy staging environment may need a ratio rather than a count."
  aggregate_vs_per_wave: "Track totalFivexx across waves, not a per-wave count: a backend degrading gradually can stay under a single-wave threshold while the aggregate crosses the line. A per-wave spike is a sudden collapse; a rising aggregate is a slow bleed."

stewardship:
  principle: "Treat the orchestrator as a steward of the shared, finite test environment (one box, one database, one backend process pool), not just a coordinator. Fanning out ~40 concurrent agents isn't only running tests — it's applying load that can push a stressed backend from slow to down. The circuit breaker is the last line of defense; these checks are the first."
  controls:
    - "Cap concurrency per wave: four concurrent API calls is the safe baseline for the platform; beyond eight, load-induced 5xx responses appear even on a healthy backend."
    - "Pause between waves when stressed: after a wave with an elevated fivexxCount (not enough to trip the breaker, but elevated), wait a few seconds before the next wave."
    - "Coordinate before any ramp: check whether another run is already in progress before starting; two concurrent orchestrators on one environment can amplify into a storm."

limitations:
  - "The threshold is a static constant — simple but fragile at the edges. A backend that returns 5xx only for a specific item type (e.g. orders with renewal line items) corrupts those results while staying under the threshold if the batch is mostly other types; the breaker won't fire and you get mixed defects and noise. Fix would be item-type-aware thresholds (not built); for now the post-run report flags suspiciously high failure rates per item type and a human decides whether to re-run that slice."
  - "The breaker protects the orchestrator's own run but doesn't coordinate with other orchestrators on the same environment — two runs can each stay under their own threshold while together producing the storm. A shared health signal (a lightweight probe endpoint or a shared atomic counter) would fix it; on the backlog."

use_when: "A multi-agent / parallel-wave test orchestrator fans out across many items against a shared, fragile backend that can collapse mid-run and silently corrupt every remaining result."
avoid_when: "Single-shot or low-concurrency runs against a hermetic, per-run environment that can't be degraded by load, where a backend storm can't poison later items."

takeaways:
  - "Agents are data reporters; orchestrators are decision-makers. A subagent that aborts on its own local observations introduces races and inconsistencies; one that reports its fivexxCount and lets the orchestrator aggregate is composable and auditable."
  - "Persist partial results before the threshold check, not after. 'Waves one through three succeeded; run halted' is a useful artifact; 'all six waves completed, but thirty of the results are garbage' is not."
  - "A flat count threshold beats nothing, even when it isn't perfectly tuned. Ship the simple version and calibrate from real runs — a static constant in version control is reviewable and adjustable; a corrupt batch is not."
  - "Think of the orchestrator as a steward, not just a coordinator. Fan-out against a shared environment applies real load; concurrency caps, inter-wave pauses, and pre-run coordination checks all matter, and the circuit breaker is the last line, not the first."
  - "Plan for partial runs in your aggregation layer. If your report template can only handle 'all N items present', you've built a silent dependency on the circuit breaker never firing. Make partial input a first-class case."

keywords: ["circuit breaker for multi-agent fan-out", "stop agent orchestrator on backend 5xx storm", "aggregate error threshold across parallel waves", "persist partial results before halting a test run", "orchestrator as steward of shared test environment"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).