From four guards to twenty-eight: running a convention gate at scale

A QA automation engineer's notes on what changes when a handful of AST checks grows into a standing policy layer.

An earlier post described four small AST guards enforcing three conventions: one import barrel, one annotation triad, one defect fence. A shared parsing helper let them coexist. A planning note had pencilled in "about twenty" guards. I counted instead. package.json lists twenty-eight guard:* scripts, and the bundle that runs them in one process imports exactly twenty-eight main() functions. Seven landed over about three weeks.

Name the unit first: one guard is one guard:<name> npm script backed by one check-*.ts file. Several of those scripts carry more than one rule, so the rule count is higher.

Four guards is a convenience. Twenty-eight is a policy layer that other people's changes pass through, applied to code I did not always write, faster than I can hand-review every hit. That shift is the subject here.

What the gate checks now

A representative handful:

• Import and annotation hygiene (the original set): one fixtures barrel for test and expect, and a traceability triad on every runnable case.
• A false-pass catalogue, enforced structurally. Five assertion shapes that can never fail for the reason the test claims to check: a verification loop that drops the null rows it should compare, a bare "the counts differ" oracle, an "any 4xx is a pass" rejection check, a length check that is non-negative by definition, and a skip computed from the response under test. The taxonomy is a separate post; this guard fences the class out.
• A skip laundered through a catch block. A test.skip(...) inside a catch is flagged unless the catch narrows to a typed miss and rethrows everything else. About 98 sites across 23 spec files carried the blanket form before the codemod that landed with the guard.
• Fixture health, not just fixture existence. A minted estate of thirteen hand-built records gives one test account an activity trail that discriminates on every filter the search endpoint exposes. Nothing in the live data substitutes for it, so a spec must resolve it through its named health check, not a bare id constant.
• Comment debt. A survey measured 455 functional specs carrying 50,269 comment lines: 27.5 percent of every line, and 239 headers over 40 lines long. A slimming campaign paid that down; this guard stops it growing back, with a per-file cap on header length and comment density.
• Document debris. Automated edit tooling leaves scaffolding inside checked-in markdown. Six files carried it, one in a worked example that had stopped being valid TypeScript.

The mechanical rules are not the interesting ones. Each interesting one encodes an incident.

A guard born from an outage

The suite resolves real records out of a live shared environment instead of hardcoding ids, and the cheapest finder available was free-text search. In July, parallel workers all fired that shape at once and produced a 502 storm that took the run down and briefly took the environment with it.

I armoured the hot path afterwards: a discovery deadline, a circuit breaker, a run-scoped coalescing broker. Most of it got deleted a month later, once the underlying query got cheap and a gate run proved none of it ever fired. One piece stayed: the guard banning the toxic query shape.

The deadline, the breaker and the broker were runtime machinery, built to survive a bad query fired anyway. The guard stops it being written at all. "The fix landed" and "nobody writes this shape again" are separate claims, and only a static check backs the second one.

A neighbouring budget guard runs on the same logic. The bounded finder helper takes a want (hits to return) and an enrichMax (candidates it will ever collect). A call with enrichMax below want is unreachable by construction, so the finder reports a miss on plentiful data, while typechecking and running green. Only a guard notices, because the conflicting values are call arguments, not types.

A policy needs an escape hatch

At twenty-eight guards, refusing every violation stops being viable. Some hits are real bugs; some are the ambiguous case the author never anticipated. A gate with no way to say "I have read this, and it is fine" gets bypassed the first time it blocks something legitimate, and it stays bypassed.

Every guard ships with the same escape hatch:

// ALLOW-RULE: caught-skip - narrow catch around an eventual-consistency read. The skip
// condition is a regex against the one known 409, and every other error rethrows, so auth
// death and a 5xx still go red.

Two rules keep the hatch honest.

The reason is mandatory. An empty reason is itself a violation, and it fails the gate regardless of grandfathering. That stops the hatch becoming a checkbox nobody fills in.

And the guard reads the whole contiguous comment run above the flagged line. The scan walks up through // lines to the first line of code, bounded at twelve lines, and accepts the tag anywhere in that run. It cannot manufacture a false exemption: the tag and a non-empty reason must still be written, attached to the site with no code in between. The first version read only the line directly above, so a four-line justification exempted nothing. That broke the first three hatches anyone wrote.

The ratchet: adopting a new rule without a big bang

A brand-new rule, run against a live corpus of close to five hundred specs, never finds zero violations. It finds every instance accumulated before the rule existed. One census, taken ahead of a launch, counted 994 sites of one pattern, 702 of a second and 358 of a third. Blocking the merge on all of that at once is not calibration; it is a wall.

The fix is a ratchet baseline: a committed JSON file keyed on (file, rule, count). Everything already there is grandfathered as a warning; anything above its recorded floor fails the gate; a count below its floor is a stale entry, also a warning, because the floor may only ever shrink.

The key is deliberately not a line number, which churns on every unrelated edit above the site, and not the bare filename, which would let a flagged file pick up a second violation for free:

{ "file": "src/domains/orders/tests/search/orderSearch.ui.spec.ts",
  "rule": "skip-from-response", "count": 2 }

A new rule is load-bearing for new code on day one, and the backlog drains on its own schedule.

The guard that flagged its own remedy

I built a false-pass rule to catch skips computed from the response under test: a test that skips itself on a status code it should instead be asserting against. The first cut flagged a few dozen sites, including this:

test.skip(seenStatuses.size < 2, 'only one status value in this environment, so the ordering claim would be vacuous');

That is not the bug. It is the prescribed fix for a different finding from an earlier audit: an honest, pre-request check that a distinct-value gate is even meaningful. My rule matched any read of a status-shaped property, and this domain's rows carry a business field called status, on an order and on a contact, unrelated to any HTTP response.

A guard that flags its own prescribed remedy is worse than no guard, because the fastest way to quiet a noisy check is to weaken whatever it complains about. The real invariant was never "the condition mentions status" but "the skip reads a response the same test also asserts on". The rule now resolves both sides to the response binding they came from, and fires only when the two meet.

A second failure in the same guard was not a domain-knowledge gap but a regex I got wrong. I assumed a word-boundary pattern would match whole identifiers. \b does not treat a lowercase-to-uppercase transition as a boundary, so \bcount\b never matched inside allRowsCount or matchedRowsCount. That camelCase family is the exact shape the audit's headline finding was written in. Review did not catch it; a unit test did, pinning the case that must fire and the one that must stay clean. The fix was a second, case-sensitive pattern requiring a preceding lowercase letter, which keeps accountLabel out.

Both times the remedy was the same. Run a new rule over the whole live corpus first and read every hit. Bucket them into real violations, prescribed idioms the rule must not touch, and legitimate patterns that merely look similar. That middle bucket is always a rule bug.

Two rosters, one bundle

npm run check used to invoke each guard as its own tsx process: one cold start, one fresh load of the compiler module and one walk of the spec tree per guard. At nine guards, in early July, I collapsed them into a bundle.

The fix touched no guard's detection logic. Every guard already exported its rule as a main() behind an entry-point check, an idiom from the four-guard post adopted so one guard's exit call could not take down the process when another imported its helpers. So one new script could import all the main() functions and run them back to back in one warm process, aggregating the worst exit code. Every standalone npm run guard:<name> still runs unchanged.

The bundle did add a failure mode: two rosters of the same guards, the bundle's array and the serial chain in package.json that a fallback flag still runs. Nothing asserted the two agreed, and for weeks the serial chain ran one guard fewer. A guard missing from one roster still passes every gate; it just stops being enforced on that path. The fix is a unit test comparing both rosters by name and order.

Tidiness at guard four is the reason bundling was an option at guard twenty-eight. Duplicating the roster is the reason it needed a test of its own.

Takeaways

• Growth changes the design problem, not just the count. A convenience for your own specs becomes a policy once other people's changes have to pass it, and a policy needs calibration, an escape hatch and gradual adoption.
• Say what your unit is before you quote a number. Twenty-eight npm scripts is not twenty-eight rules.
• Calibrate every new rule against the live corpus before it blocks anything. A bucket that reads "this is the prescribed fix for a different finding" means the rule is wrong, not the code.
• Ratchet on (file, rule, count), never on a line number or a bare filename. Counts survive unrelated edits above the site and still catch the next violation in an already-flagged file.
• A mandatory, multi-line escape hatch keeps a hard gate honest. An empty justification is itself a violation, and a real justification is rarely one line.
• The entry-point discipline adopted at guard four is what lets guard twenty-eight exist cheaply, and it was written for a different reason entirely.
• Two lists of the same thing will drift. If you duplicate a roster for a fallback path, pin the pair with a test.

---

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/05/from-four-guards-to-twenty-eight/"
license: "Free to reference with attribution"
title: "From four guards to twenty-eight: running a convention gate at scale"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 45"
stack: [TypeScript, "TypeScript compiler API", Playwright, Node.js, npm, JSON]

problem: "A set of four AST convention guards grew to twenty-eight. At four, a guard is a convenience for your own specs; at twenty-eight it is a policy layer other people's changes must pass, applied to code the author did not write, faster than anyone can review every hit. A new rule run against a live corpus of ~500 specs never finds zero violations, so blocking the merge on the whole backlog is a wall, not a gate."
thesis: "Scaling a convention gate is a design problem, not a counting problem. Three mechanisms make it survivable: a mandatory multi-line escape hatch, a committed (file, rule, count) ratchet baseline that grandfathers the backlog as warnings while failing anything new, and corpus calibration of every rule before it blocks anything."

inventory:
  unit: "One guard is one guard:<name> npm script backed by one check-*.ts file. Several scripts carry multiple rules, so the rule count is higher. Twenty-eight is the count of scripts, verified three ways: package.json lists 28 guard:* scripts (29 with the guard:all bundle stand-in), and the bundle imports 28 main() functions into a 28-entry array. Seven landed over about three weeks."
  representative_rules: "Import barrel + traceability triad (the original set). Five blind-oracle shapes that cannot fail for the stated reason: a verification loop dropping the null rows it should compare, a bare 'the counts differ' oracle, an 'any 4xx is a pass' rejection check, a length check non-negative by definition, and a skip computed from the response under test. A test.skip inside a catch, unless the catch narrows to a typed miss and rethrows (~98 sites across 23 spec files before the codemod). One banned free-text lookup shape. Fixture health: a minted estate of thirteen hand-built records backing one test account's activity trail must be resolved through its named health check, never a bare id constant. Comment debt: 455 functional specs carried 50,269 comment lines, 27.5% of all lines, 239 headers over 40 lines. Document debris left by automated edit tooling: six markdown files, one holding a worked example that had stopped being valid TypeScript."
  guard_vs_runtime_armour: "After a 502 storm from a free-text discovery query, the author built a discovery deadline, a circuit breaker and a coalescing broker, then deleted almost all of it once the query got cheap. The guard banning the toxic shape stayed. Armour survives a bad query being fired; a guard stops it being written. 'The fix landed' and 'nobody writes this shape again' are separate claims, and only a static check backs the second."
  budget_guard: "The bounded finder helper takes want (hits to return) and enrichMax (candidates ever collected). enrichMax below want is unreachable by construction, so the finder reports a miss on plentiful data while typechecking and running green. Only a guard notices, because the conflicting values are call arguments, not types."

adoption:
  escape_hatch: "// ALLOW-RULE: <guard-name> - <reason>, in a comment above the flagged site. The reason is mandatory: an empty one is itself a violation and fails the gate regardless of grandfathering, so the hatch cannot become an unfilled checkbox."
  hatch_lookback: "The scan walks the contiguous // comment run up to the first line of code, bounded at twelve lines, accepting the tag anywhere in that run. It cannot manufacture a false exemption: the tag and a non-empty reason must still be written, attached with no code in between. The first version read only the line directly above, so a four-line justification exempted nothing. It broke the first three hatches anyone wrote."
  ratchet_key: "(file, rule, count), committed as JSON. Not a line number, which churns on unrelated edits above the site; not a bare filename, which would let a flagged file pick up a second violation free."
  ratchet_verdicts: "Above the floor FAIL; at the floor WARN (grandfathered); below the floor WARN (stale, the floor may only shrink); an empty escape hatch FAILs regardless. A new rule is load-bearing for new code on day one while the backlog drains on its own schedule."
  why_ratchet: "A pre-launch census counted 994 sites of one pattern, 702 of a second, 358 of a third, across a corpus of close to five hundred specs."

war_story:
  self_flagging_rule: "The skip-from-response rule matched any read of a status-shaped property, so it flagged test.skip(seenStatuses.size < 2, ...), the honest pre-request distinct-value gate an earlier audit had PRESCRIBED as the fix for a different finding. The domain's own rows carry a business field named status, unrelated to HTTP."
  correction: "The invariant is not 'the condition mentions status' but 'the skip reads a response the same test also asserts on'. The rule now resolves both sides to their response binding and fires only when the two meet."
  regex_bug: "A word-boundary pattern was assumed to match whole identifiers. \\b does not treat a lowercase-to-uppercase transition as a boundary, so \\bcount\\b never matched inside allRowsCount or matchedRowsCount, the exact camelCase family the audit's headline finding was written in. Review missed it; a unit test pinning a must-fire and a must-stay-clean case caught it. Fixed with a second case-sensitive pattern requiring a preceding lowercase letter, which keeps accountLabel out."
  method: "Run every new rule over the whole live corpus first, dump every hit, read every one, and bucket into real violations, prescribed idioms the rule must not touch, and a similar-looking legitimate pattern. The middle bucket is always a rule bug."

bundling:
  problem: "npm run check invoked each guard as its own tsx process: one cold start, one fresh load of the compiler module and one walk of the spec tree per guard."
  fix: "At nine guards, in early July, one script imported every guard's exported main() and ran them back to back in a warm process, aggregating the worst exit code. No detection logic changed; standalone npm run guard:<name> still works."
  precondition: "Every guard already gated its process.exit(main()) behind an entry-point check, adopted at four guards so one guard's exit could not kill the process when another imported its helpers. That tidiness is what made bundling cheap later."
  roster_drift: "The bundle created a second roster beside the serial chain in package.json, and nothing asserted they agreed. The serial chain ran one guard fewer for weeks. A guard missing from one roster still passes every gate; it just stops being enforced on that path. Fixed by a unit test comparing both rosters by name and order."

use_when: "A repository has enough project-specific structural conventions that review cannot hold the line, and enough contributors that a new rule will land on code its author never wrote."
avoid_when: "A handful of rules over a small corpus you fully own, where a plain failing check and a hand-fixed backlog cost less than a baseline file, a hatch convention and a roster test."

takeaways:
  - "Growth changes the design problem, not just the count. A convenience for your own specs becomes a policy once other people's changes have to pass it, and a policy needs calibration, an escape hatch and gradual adoption."
  - "Say what your unit is before you quote a number. Twenty-eight npm scripts is not twenty-eight rules."
  - "Calibrate every new rule against the live corpus before it blocks anything. A bucket that reads 'this is the prescribed fix for a different finding' means the rule is wrong, not the code."
  - "Ratchet on (file, rule, count), never on a line number or a bare filename."
  - "A mandatory, multi-line escape hatch keeps a hard gate honest. An empty justification is itself a violation."
  - "The entry-point discipline adopted at guard four is what lets guard twenty-eight exist cheaply, and it was written for a different reason entirely."
  - "Two lists of the same thing will drift. If you duplicate a roster for a fallback path, pin the pair with a test."

keywords: ["scaling AST lint guards in a test repo", "ratchet baseline file rule count", "grandfather existing lint violations without blocking merges", "ALLOW-RULE escape hatch mandatory reason", "guard that flags its own prescribed fix", "word boundary regex misses camelCase identifiers"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).