A QA automation engineer's notes on deleting a run broker, a discovery deadline and a circuit breaker once the real fix had landed somewhere else.
Every mature test suite has a layer like this one. Something went badly wrong once. You built machinery to survive it. The machinery is still there years later, wrapped around a hot path nobody profiles, exercised by unit tests that pass, and quietly taxing every reader who has to understand the code underneath it.
Mine was a run-scoped discovery broker, plus the discovery deadline and the circuit breaker that sat under it. Roughly 5,000 lines. A few weeks later I deleted all of it in one commit. The deletion is not the interesting part. The interesting part is how I proved it was safe, and what I deliberately kept.
The storm that justified it
The suite does not hardcode entity IDs. Tests resolve real records out of a live shared environment at fixture-setup time. That design is right for an environment where the data drifts daily. It also means the suite issues a lot of discovery queries in parallel, against a box that other people are using at the same time.
For a long time the cheapest thing a finder could reach for was free-text search. On a quiet environment that query shape returned in 1.6 to 2.4 seconds. Under load it took 12 to 20. In July, parallel workers all fired that shape at once. The result was a 502 storm that took the run down, and briefly took the shared environment with it. The worker-side half of that fight, lazy worker-scoped discovery plus shard rotation, is in Parallel Playwright workers vs a 502 storm. Bounding the finders themselves is in Discovery without a meltdown.
Then I armoured the hot path, in the order you would expect.
- A discovery deadline. Every finder fire got a 26 second wall and a 90 second in-flight lease. One hung query could no longer hold a worker forever, and no second dispatch could pile in behind it.
- A circuit breaker. Two tiers. The fine tier tripped after three consecutive transient failures on one query fingerprint. The emergency tier tripped after eight transients inside a 60 second window on a coarser endpoint key. Workers would stop hammering an endpoint that had already told them it was unwell.
- A run broker. The ambitious one. A client, a protocol and a server, with launcher-side lifecycle hooks. A discovery query fired by four workers at once became one upstream round trip, dispatched centrally and fanned back out. Each domain barrel grew a hand-cloned coalesce seam of about 70 lines to talk to it.
None of that was unreasonable. Given the failure I had just watched, all three were defensible.
What actually fixed the problem
The run broker was built to make an expensive query safe to fire many times. What actually fixed things was making the query cheap.
That work is a separate story. It replaced blind searching with an estate model: a small set of verified capability pins resolved by GET-by-id, plus a title-prefix filter that classifies a whole pool family in one shaped server-side query. The numbers are the point. A pin verify runs about 200ms at the median and 350ms at p95. A shaped filter runs 300 to 630ms. Free-text was 1.6 to 2.4 seconds quiet and 12 to 20 under load, and a static guard now bans it from every hot path.
Once the hot path was pin verifies and bounded reads, the armour was defending against a load profile the suite no longer generated. The discovery deadline was wrapping queries that finished in a fifth of a second. The circuit breaker was watching for a consecutive-transient pattern that had stopped happening. The run broker was coalescing round trips that were no longer expensive enough to be worth coalescing.
This is the ordinary lifecycle of crisis infrastructure, and it is worth naming. The fix that makes your armour unnecessary usually lands somewhere else entirely, and nobody thinks to go back and check.
Proving it, rather than asserting it
"I probably do not need this any more" is not a reason to delete 5,000 lines. So I commissioned a gate run instead. Keep the armour in place, instrument every seam, and run the full API tier until one question has a data answer. Does any of it ever fire?
The gate run got lucky. It rode a live 502 storm on the shared environment, which is exactly the weather the armour was built for. It recorded:
- zero deadline fires
- zero breaker trips
- zero cooldown suppressions
That is the whole argument. Not "the code looks unused", and not "nobody remembers why it is there", but a counter at each seam reading zero through the specific weather the code exists to survive. If any of those three counters had been non-zero, the right move would have been to keep the module and write down why.
One nuance mattered here. The storm proved that the armour never engaged. It did not prove that nothing protects the suite. Something still does. The memo layer underneath does resolve-once deduplication when two tests race the same key, and it evicts a transient failure behind a 45 second anti-hammer cooldown. Above that sits a flat 120 second per-test hang wall in the runner config. When you retire a guard, say explicitly what is left holding the floor. Otherwise the next incident review concludes that you removed the only thing standing between the suite and the storm.
Unwired is not deleted, and dormancy has a price
At the gate run I did the cautious thing. I unwired the run broker from the config and left the modules in the tree. One revert of one config hunk would re-arm the pilot. That felt like free optionality.
It was not free. For the next twenty-six days:
- Two unit spec files kept executing on every
npm run check. They tested a code path that nothing could reach. They passed. They cost time on every check, and worse, they made the module look maintained. - Four domain barrels kept their coalesce seams, about 70 lines each, wrapped around getters that now had exactly one code path.
- Every reader who came to the discovery layer to learn how an entity gets resolved had to first work out that half of what they were reading was dormant.
- The four launcher-side lifecycle modules sat in the tree. After the gate run, nothing ever wired them into the runner config again.
Dormant code is not neutral. It is documentation that lies, weighted by how sophisticated it looks.
The deletion
The retirement pass removed everything in one commit. The run broker client, protocol and server, plus their two unit specs. The four launcher-side lifecycle modules. The discovery deadline and the circuit breaker, plus their specs. The four hand-cloned coalesce seams, so each bag getter collapsed back to a plain memoized fetch. And one leftover descriptor that existed only to feed the run broker's dispatch path. The commit reads 4,975 lines removed against 166 added.
The seams were the part that made the deletion worth doing rather than merely tidy. The four blocks were near-identical. The accounts copy carried a comment describing itself as a mechanical clone of the orders one, and a diff showed that the classify closures were byte-identical across all four. That is exactly the kind of thing somebody copies into a fifth domain on the assumption that it is the house pattern.
What deliberately stayed, and why that matters more
Two things were flagged for deletion and survived the pass. The reasoning is the reusable part.
A query fingerprint helper. It was born as the run broker's dedup key. It looked like broker scaffolding, so it went straight onto the deletion list. Then a grep for consumers found four live ones: the discovery event ledger, the skip renderer in the fixtures barrel, the latency lane in the run-history tooling, and a cardinality audit job. The run broker was one consumer of that helper, and not the most important one. The helper stayed.
A served-by provenance tag on the memo layer. The run broker used the tag to mark whether a result came from the broker or from the local worker. Only one value is possible now. The tag stayed anyway, because the discovery ledger schema carries the field and the tag is genuinely generic.
The lesson I took from that pass: origin is not ownership. A module built for one feature can outlive that feature. The only way to find out is to enumerate the call sites, not to reason from the file's history. My own first pass got this wrong, and a grep caught it. Do the grep across the entire source tree, not just the folder you are working in, and do it before you write the deletion commit rather than after.
Rewrite the escape hatch when you delete the thing
The pilot had a documented re-arm story: revert one config hunk. That story died with the seams. Reverting the config now points at modules and call sites that no longer exist.
So the decision record replaces it. The escape hatch is a revert of the deletion commit, which brings back the modules, the specs and all four seams together and consistently. The design history, the timing drills and the gate-run evidence stay in the record even though the code does not.
An escape hatch that describes a tree shape you no longer have is worse than no escape hatch, because somebody will try it during an incident.
Takeaways
- Crisis armour outlives its crisis. Write down what would make each piece unnecessary on the day you build it, then go back and check.
- Prove disuse with a counter, not a conviction. Instrument each seam and run it through the weather it was built for. A zero through a live storm is an argument. "It looks unused" is not.
- Say what is still holding the floor. Retiring a guard is safe only if you can name what replaced it.
- Unwired code costs you. Specs that pass against unreachable modules, seams that read as house style, and a re-arm story that quietly rots. Pick deletion or wiring, not the middle.
- Origin is not ownership. Enumerate the consumers before you delete anything that merely looks like scaffolding. My fingerprint helper had four.
- A pilot that proves it was unnecessary succeeded. That was the framing on the day I unwired the run broker, and it is why the deletion was uncontroversial four weeks later. Treat "I did not need it" as a failure and you end up carrying every experiment you ever ran.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/02/delete-the-armour-you-no-longer-need/"
license: "Free to reference with attribution"
title: "5,000 lines of armour I never needed: retiring the infrastructure you built for a storm"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 42"
stack: [TypeScript, Playwright, "Node.js", git, "REST API"]
problem: "A discovery-first Playwright suite resolves entities from a live shared environment. Free-text search (1.6-2.4s quiet, 12-20s under load) fired by parallel workers caused a July 502 storm, so three layers of armour were built on the discovery hot path. A later change made the query cheap, and the armour then defended against a load profile the suite no longer produced."
thesis: "Crisis infrastructure outlives its crisis because the fix that makes it unnecessary usually lands somewhere else. Prove disuse with instrumented counters read through the original weather, then delete rather than leave the code dormant, and name what still holds the floor."
armour_built:
discovery_deadline: "26s wall per finder fire plus a 90s in-flight lease, so one hung query could not hold a worker and no second dispatch piled in behind it."
circuit_breaker: "Two tiers. Fine tier trips after 3 consecutive transient failures on one query fingerprint. Emergency tier trips after 8 transients inside a 60s window on a coarser endpoint key."
run_broker: "Client, protocol and server plus launcher-side lifecycle hooks. One query fired by four workers became one upstream round trip, dispatched centrally and fanned back out. Each domain barrel carried a hand-cloned coalesce seam of about 70 lines."
what_actually_fixed_it:
estate_model: "Verified capability pins resolved by GET-by-id, plus a title-prefix filter that classifies a whole pool family in one shaped server-side query."
measured_costs: "Pin verify about 200ms median, 350ms p95. Shaped filter 300-630ms. Free-text 1.6-2.4s quiet, 12-20s under load, now banned from every hot path by a static guard."
consequence: "The deadline wrapped queries finishing in a fifth of a second, the breaker watched for a pattern that had stopped, and the broker coalesced round trips no longer worth coalescing."
evidence_gate:
design: "Keep the armour wired, instrument every seam, run the full API tier and answer one question with data: does any of it ever fire?"
result: "The run rode a live 502 storm on the shared environment and recorded zero deadline fires, zero breaker trips, zero cooldown suppressions."
decision_rule: "A non-zero counter at any seam would have meant keep the module and record why."
what_still_holds_the_floor: "The memo layer does resolve-once deduplication for racing tests and evicts a transient failure behind a 45s anti-hammer cooldown. A flat 120s per-test hang wall sits in the runner config."
dormancy_cost:
duration: "26 days between unwiring the run broker from the config and deleting it."
costs: "Two unit specs kept executing on every npm run check against an unreachable code path, and made the module look maintained. Four domain barrels kept 70-line coalesce seams around single-path getters. Readers of the discovery layer had to work out which half was dormant. The four launcher-side lifecycle modules were never wired into the runner config again."
verdict: "Dormant code is documentation that lies, weighted by how sophisticated it looks. Pick deletion or wiring, not the middle."
deletion_commit:
removed: "Run broker client, protocol and server plus two unit specs; four launcher-side lifecycle modules; discovery deadline and circuit breaker plus their specs; four hand-cloned coalesce seams; one descriptor that only fed the broker dispatch path."
size: "4,975 lines removed against 166 added, in one commit."
why_the_seams_mattered: "The four seam blocks were near-identical. The accounts copy called itself a mechanical clone of the orders one, and a diff showed byte-identical classify closures. A fifth domain would have copied it as the house pattern."
deliberately_kept:
query_fingerprint_helper: "Born as the broker dedup key and first listed for deletion. A grep found four live consumers: the discovery event ledger, the skip renderer in the fixtures barrel, the latency lane in the run-history tooling, and a cardinality audit job."
served_by_tag: "The memo layer provenance tag the broker used to mark broker-served versus locally served results. Only one value is possible now, but the discovery ledger schema carries the field and the tag is generic."
rule: "Origin is not ownership. Enumerate call sites across the whole source tree before the deletion commit, not after."
escape_hatch:
old: "Revert one config hunk to re-arm the pilot. That story died with the seams, because the config now points at modules and call sites that no longer exist."
new: "Revert the deletion commit, which restores modules, specs and all four seams consistently. The decision record keeps the design history, the timing drills and the gate evidence."
principle: "An escape hatch describing a tree shape you no longer have is worse than none, because somebody will try it during an incident."
use_when: "A test suite or service carries load-shedding, coalescing or timeout machinery built after a specific incident, and a later change altered the load profile that justified it."
avoid_when: "The original load profile still occurs, or no seam is instrumented, so disuse can only be asserted rather than measured."
takeaways:
- "Crisis armour outlives its crisis. Write down what would make each piece unnecessary on the day you build it, then go back and check."
- "Prove disuse with a counter, not a conviction. Instrument each seam and run it through the weather it was built for."
- "Say what is still holding the floor. Retiring a guard is safe only if you can name what replaced it."
- "Unwired code costs you: specs passing against unreachable modules, seams that read as house style, and a re-arm story that rots. Pick deletion or wiring, not the middle."
- "Origin is not ownership. Enumerate the consumers before deleting anything that merely looks like scaffolding."
- "A pilot that proves it was unnecessary succeeded. Teams that read that as failure carry every experiment they ever ran."
keywords: ["delete unused infrastructure safely", "retire circuit breaker and timeout wrapper", "prove dead code with instrumentation counters", "dormant code carrying cost in a test suite", "escape hatch for a deletion commit", "coalescing broker no longer needed"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).