A QA automation engineer's notes on cutting 5,880 comment lines to 3,820, proving the diff touched nothing but prose, and ratcheting it so it cannot grow back.
A comment gets written once, when somebody understands the code best, and almost nobody touches it again. The code around it keeps moving: refactors, renames, a fix that narrows a condition. No type checker re-reads a comment the way one re-reads a signature, so it drifts quietly and never fails a build.
Across 412 functional spec files the suite carried 44,180 comment lines, 26% of every line in them. The median file was 28% comment, the 90th percentile 43%, the worst 61%. Worse, some of those comments were flatly wrong about the code beneath them, in a way that sends the next reader hunting for a bug in the wrong file.
Proving a slimming pass touched nothing but prose
Cutting comments hits a credibility problem: how do you know you did not also cut a line of logic?
The first wave covered fourteen spec files: 5,880 comment lines down to 3,820, a drop of 2,060 lines, or 35%. Four were over 38% comment before the pass, the worst at 52.8%. One header block ran 149 lines before the file's first import. Fourteen LLM subagents did the work, one file each, in about 22 minutes of wall clock.
The proof has to be mechanical. Run an unfiltered diff, then re-filter to lines that are neither comments nor blank:
git diff -U0 -- '*.spec.ts' \
| grep -E '^[+-]' \
| grep -vE '^(\+\+\+|---)' \
| grep -vE '^[+-][[:space:]]*(//|/\*|\*)' \
| grep -vE '^[+-][[:space:]]*$'
Zero lines survived that filter across all fourteen files. Not "this looks like a comments-only change" but a grep-provable statement that every changed line was a comment or blank.
A second proof came from another angle: the script that generates the test-to-ticket index parses these same files for header metadata. Regenerating it gave a byte-identical result, which a trim touching a tracked field would not have produced.
A third hazard: comment lines get addressed from outside the file. The repo held 2,208 spec.ts:NNN line-anchored citations, 173 of them pointing into the fourteen wave files. Deleting a comment line above an anchor silently rots a citation elsewhere. About 162 of the 173 sat in archives, audits and append-only decision logs, which are point-in-time records and stay frozen. Four were live anchors in standards docs, repaired by hand.
Rationales that were confidently, specifically wrong
Proving the diff safe is the easy half. Reading the surviving comments against the code, line by line, was harder. Five stated something false about the code beneath them, each specific enough to sound authoritative.
- A contract narrower than the code. A page-object helper's doc said its
rootparameter had to be the exact custom element wrapping one lookup field's controls. The code accepted any enclosing container, and a real caller passed a whole search panel. The true contract is weaker: the root must enclose exactly one lookup, or sibling testids resolve across fields. - A wrong date. A guard's header cited the commit that deleted a file and dated it eight days early.
- A hardcoded config claim that had moved. A retry-containment guard opened with "the
apiproject runsretries: 2anduirunsretries: 1". By thenapipinned retries to 0. The comment now points at the config as the source of truth, with the live set derived and never restated. - A clause exactly backwards. A shared ratchet library gave two reasons why a sibling guard is not one of its consumers. The second said the sibling escalates a missing baseline to exit 2 while the library degrades to a warning. The truth ran the other way round.
- A duplicated, orphaned header block. Two file-level doc comments sat on the same module, written at different times, one of them stale.
A sixth was a number that stopped being true: a barrel comment said the split-out file holds 60 descriptor objects. It held 66.
None of this was sabotage. Each comment was true when written and became false through an unrelated change elsewhere. A false rationale costs more than a missing one: a gap sends the next reader off to find out, a wrong explanation sends them somewhere else and they trust it.
The same pass harvested 44 suspected defects as unverified leads. Reading comments against code is a slow, cheap oracle that finds what no assertion does.
A budget, because a one-time cleanup regrows
Slimming the files once just resets the clock. Comments accumulate for the same reason they go stale: writing one is cheap, and nobody's workflow forces a review of whether it should still exist.
So the cleanup shipped with a guard. src/tooling/prose-budget.ts checks two rules per spec file: oversized-header fires when the leading comment block runs past 35 lines, comment-share when countable comment lines pass 30% of the file.
The stored magnitude is a count, not a ratio. A ratio moves when unrelated code lands, so a percentage floor churns for reasons unrelated to comments.
Never-touch comment classes do not count toward density. Skip reasons, allow-rule rationales, schema-gap markers and compiler directives are machine-read, so counting them would let the guard flag its own remedy.
The guard scans text rather than parsing an AST, the opposite of the choice in the convention guards. Those hunt a token their own headers quote in prose, so a text scan would flag their own documentation. This one counts comment lines, and it is not a spec file, so it never scans itself.
Every current violator is grandfathered at a (file, rule, count) floor: 318 entries, 199 header and 119 density.
{ "file": "src/reporting/tests/invoiceLedgerView.report.ui.spec.ts",
"rule": "comment-share",
"count": 19 }
A grandfathered file warns. An unlisted violator fails. A count above its floor fails. A count below it is a stale entry that gets shrunk. The guard can never be satisfied by adding comments, which makes it a ratchet, not a linter you route around. A later refactor added header lines to one spec and the gate refused the push until the debt was paid down.
Where a rationale actually belongs
A comment budget raises a tension: sometimes you need to explain why a test is parked, and that is exactly the prose the ratchet keeps out. The ratchet turned out to push that prose somewhere better.
I re-triaged 76 parked tests against the classifier that decides whether a skip counts as parked coverage or lost coverage. Only 18 classified correctly. In one of the worst files the rationale was excellent: a per-group deployment probe naming the blocking ticket, its status, and a live count of the missing priority marker across 361 rows. It sat in a header comment, which never reaches the results JSON.
The destination is the annotation argument, a runtime string rather than a comment, so it costs nothing against the budget:
test.fixme(
'TC1: each result row shows a priority marker @MT-8267',
{ annotation: { type: 'fixme', description:
'NOT-DEPLOYED (PROJ-7030 cancelled): priority now ships as row tinting plus a static ' +
'legend dialog, never the per-row marker this case looks for. Latest probe this month ' +
'found zero markers across 361 live rows. Needs re-authoring; no deploy un-parks it.' } },
async ({ page }) => { /* ... */ },
);
48 of the 76 had nowhere to put a reason. The other 10 carried one and still failed to classify. One wrote not_deployed with an underscore against a pattern expecting not[- ]deployed, plus a ticket prefix with a trailing letter the pattern did not match. It reads as a guarded skip to a human and as silent lost coverage to the classifier. The verdict vocabulary behind the exercise is "Not deployed" is a first-class verdict.
The parallel problem: comments that name the tool that wrote them
A different scrub ran alongside the density work. The tree had accumulated references to the coding agent that helped write it, and the tree ships to an external mirror, so they were a leak.
The naive way to find them is a case-insensitive grep for the short tokens: an agent-name fragment, a two-letter model-family abbreviation. That grep adds about 71 hits which are not references: camelCase identifiers containing the fragment, plus binaries the scan should never have opened. With word boundaries and binaries skipped, the real count was 64, all genuine.
Loosening the boundary over-matches on short tokens; tightening it under-matches. Both look like a clean scan until you check the denominator.
Fixing the 64 was not a bulk delete. Zero sat in a domain spec file. All 64 lived in nine infrastructure files and split three ways. 34 belonged to a canary script whose job is to verify the outer wrapper's agent-hook copies, so the references are its subject matter; it moved to the outer tree instead. 15 were one environment variable the harness sets to identify a session, load-bearing for run attribution on a shared box, so it became one neutrally named constant every call site imports. The last 15 were genuine comment mentions to reword.
Two hits were traps. A guard's default-roots array names the agent config directory as logic, not as a mention. Delete that entry and the guard scans nothing, hits its mirror carve-out and returns 0: silently green, permanently.
What both passes have in common
Neither pass was about deleting text. Both share a root cause: text that nobody is on the hook to revisit will eventually say something false. Both got the same fix: prove the change mechanically, ratchet it, and enforce it where it will actually run. It is the instinct I applied to an agent context file in Your context file is code, pointed at a different target.
Takeaways
- A comments-only diff needs a mechanical proof, not a careful read. Filter the diff to non-comment, non-blank lines. If anything survives, it is not a comments-only change, however it looked in review.
- A false rationale costs more than a missing one. A gap sends the next reader to find out. A confident, wrong explanation sends them somewhere else, and they trust it.
- Comment lines are addressed from outside the file. Snapshot the line anchors that cite them before you delete anything, then repair the live ones and freeze the dated records.
- A ratchet has to be a floor, not a target. Grandfather what is already there, store an integer count rather than a ratio, and fail only on growth.
- A parked test's rationale belongs in the annotation argument, not the comment above it. A comment is not in the results JSON, so no classifier can ever see it.
- A naive grep is not a census. Word-boundary the short tokens, skip the binaries, and check whether a hit is a mention or load-bearing logic before you edit it.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/15/comments-are-a-maintained-artifact/"
license: "Free to reference with attribution"
title: "Comments are a maintained artifact, not a one-time deposit"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 55"
stack: [TypeScript, Playwright, git, grep, Node, JSON]
problem: "Comment prose in a Playwright suite rots invisibly: no type checker re-reads it and no build fails on it. A census of 412 functional spec files found 44,180 comment lines, 26% of all lines, median 28%, p90 43%, worst 61%. Several comments stated something false about the code beneath them."
thesis: "Treat comments as a maintained artifact. Slim them in a pass whose safety is proven by grep rather than by review, correct the rationales that became false, then hold the level with a ratchet guard so the debt cannot regrow."
slimming_pass:
scale: "Wave 1 covered 14 spec files: 5,880 comment lines to 3,820, a drop of 2,060 lines (-35%). Four files were over 38% comment before the pass; the worst was 52.8%. One leading header ran 149 lines. Executed by 14 LLM subagents, one file each, about 22 minutes of wall clock."
proof_1: "git diff -U0 over the trimmed specs, piped through greps that drop the +++/--- headers, any +/- line starting with // or /* or *, and any blank +/- line. Zero lines survived across all 14 files."
proof_2: "Regenerating the test-to-ticket index, which parses the same files for header metadata, produced a byte-identical result. Two independent proofs cover each other's blind spot."
anchor_hazard: "Comment lines are addressed from outside the file. The repo held 2,208 spec.ts:NNN line-anchored citations; 173 pointed into the 14 wave files. Ruling: ~162 in archives, audits and append-only decision logs stay FROZEN (point-in-time records; rewriting them falsifies history), 4 live anchors in standards and skill docs repaired by hand, 2 describing a byte absent at HEAD left standing as history."
side_harvest: "44 suspected defects surfaced as unverified leads. Reading comments against code is a slow, cheap oracle that finds what no assertion does."
false_rationales:
scoping_contract: "A page-object helper's doc said its root parameter had to be the exact custom element wrapping one lookup field. The code accepted any enclosing container and a real caller passed a whole search panel. The true contract: the root must enclose exactly one lookup, or sibling testids resolve across fields."
wrong_date: "A guard header dated the commit that deleted a file eight days early."
stale_config_restatement: "A retry-containment guard asserted 'the api project runs retries: 2 and ui runs retries: 1' as fact; api had since pinned retries to 0. Fix: point the comment at the config as source of truth and state the live set is derived, never restated."
inverted_clause: "A shared ratchet library gave two reasons why a sibling guard is not a consumer and got one backwards: it claimed the sibling escalates a missing baseline to exit 2 while the library warns. The reverse was true."
orphan_header: "Two file-level doc blocks on one module, written at different times, one stale; nobody deleted the first when the second arrived."
stale_count: "A barrel comment said the split-out file holds 60 descriptor objects; it held 66."
mechanism: "None were sabotage. Each was true when written and became false through an unrelated change elsewhere."
comment_budget_ratchet:
artifact: "src/tooling/prose-budget.ts, shipped in the same commit as the slimming pass."
rules: "oversized-header fires when the leading comment block exceeds 35 lines. comment-share fires when countable comment lines exceed 30% of the file."
count_not_ratio: "The stored magnitude is an integer count, not a percentage. A ratio moves when unrelated code lands, so a percentage floor churns for reasons unrelated to comments."
never_touch_exclusion: "Machine-read classes (skip reasons, allow-rule rationales, schema-gap markers, compiler directives) do not count toward density, so the guard cannot flag its own remedy."
text_scan_not_ast: "It measures comment lines rather than matching a token, and it is not a spec file, so it never scans itself. The AST convention guards hunt a token their own headers quote in prose, which is why they needed a parser."
baseline: "318 grandfathered entries at (file, rule, count) floors: 199 header, 119 density. A grandfathered file WARNs; an unlisted violator FAILs; a count above its floor FAILs; a count below its floor is a stale entry that gets shrunk. It can never be satisfied by adding comments."
proven_in_anger: "A later refactor added header lines to one spec and the gate refused the push until the debt was paid back down."
skip_rationale_placement:
finding: "Of 76 parked tests re-triaged against the parked-vs-lost coverage classifier, only 18 already classified correctly. In the worst file the rationale was excellent (a per-group deployment probe naming the blocking ticket, its status, and a live count of the missing priority marker across 361 rows) and sat in a header comment, which never reaches the results JSON."
no_slot: "48 of the 76 used a declaration-form fixme, which has no reason slot at all."
fix: "Pass the annotation argument: test.fixme(title, { annotation: { type: 'fixme', description: '...' } }, fn). A runtime string costs nothing against the comment budget and lands in the run report."
near_miss: "The other 10 carried a reason that still failed to classify: 'not_deployed' with an underscore against a pattern expecting 'not[- ]deployed', plus a ticket prefix with a trailing letter the pattern did not match. Reads as a guarded skip to a human and as silent lost coverage to the classifier."
ai_reference_scrub:
motive: "The tree accumulated references to the coding agent that helped write it and ships to an external mirror, so the references were a leak."
denominator: "A case-insensitive grep for the short tokens adds about 71 hits that are not references: camelCase identifiers containing the fragment as a substring, plus binary documents. With word boundaries and binaries skipped the real count was 64, all genuine."
distribution: "Zero hits in any domain spec file. All 64 sat in 9 infrastructure files, split three ways: 34 in a canary script whose job is to verify the outer wrapper's agent-hook copies (relocated to the outer tree rather than stripped of its vocabulary); 15 in one env var the harness sets to identify a session, load-bearing for run-lease attribution on a shared box (wrapped in one neutrally named exported constant every call site imports); 15 genuine comment mentions to reword."
traps: "Two hits name the agent config directory inside a guard's default-roots array as LOGIC. Delete that entry and the guard scans nothing, hits its mirror carve-out and returns 0: silently green, permanently."
enforcement_placement: "The canary moved to the push gate rather than the everyday check. A check that stops regrowth must run on every change that could cause it, or it is a memory rather than a check."
use_when: "A long-lived test suite carries heavy explanatory prose, several people or agents write it, and nothing in the workflow forces a review of whether a comment is still true."
avoid_when: "A young or small codebase where the comment corpus is small enough to read whole, or a codebase whose comments are generated from the code and cannot drift independently."
takeaways:
- "A comments-only diff needs a mechanical proof, not a careful read. Filter the diff to non-comment, non-blank lines. If anything survives, it is not a comments-only change."
- "A false rationale costs more than a missing one. A gap sends the next reader to find out; a confident wrong explanation sends them somewhere else and they trust it."
- "Comment lines are addressed from outside the file. Snapshot the line anchors that cite them before deleting anything, repair the live ones, freeze the dated records."
- "A ratchet has to be a floor, not a target. Grandfather what is already there, store an integer count rather than a ratio, and fail only on growth."
- "A parked test's rationale belongs in the annotation argument, not the comment above it. A comment is not in the results JSON, so no classifier can see it."
- "A naive grep is not a census. Word-boundary the short tokens, skip the binaries, and check whether a hit is a mention or load-bearing logic before editing it."
keywords: ["stale code comments that contradict the code", "prove a diff is comments only with git diff and grep", "comment density ratchet guard", "comment to code ratio budget in a test suite", "test.fixme annotation description in Playwright results JSON", "scrub AI tool references from a codebase"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).