Your suite has too many tests

A QA automation engineer's notes on a consolidation campaign that removed 246 tests from a Playwright suite without dropping one traceability reference.

Test count is a cost, not an asset. Every test is a login, a navigation, a seed round trip, and a line in a report someone reads when it goes red. A suite that grew for two years by accretion is not thorough. It is a tax on every run and every reviewer.

I recently closed a consolidation campaign over a Playwright suite of 580 spec files and 4,911 tests. It removed 246 runtime tests, deleted two spec files, cut roughly 75 page loads and dialog opens, and dropped 16 serial restore calls. Every retired case kept its traceability tag. The interesting part is not the count. Finding those 246 took two different tools, and neither one alone would have found them.

What accumulates in a mature suite

Give a suite two years and several authors and it grows the same way every time. A form gets four negative-value tests where one table-driven test would do. A second file checks one more column on a screen the first file already opened. A defect pin outlives its defect, and nobody notices that its twin already covers the fixed behaviour.

None of this is anyone's fault in the moment. Each test was the right size for the change it shipped with. The redundancy is only visible in aggregate, when somebody reads a whole folder at once. Combining at import time is a different job, covered in Don't photocopy your manual tests into Playwright. This post is about the debt a suite accrues after the port.

The honest fix is not deleting tests to hit a number. It is folding the homogeneous cases into table-driven walks: one test() per genuinely distinct behaviour, with the varying inputs as typed rows, and every retired case's traceability tag alive on the test that now covers it. Get that wrong and you either lose coverage silently, or hide the bloat better.

A similarity floor, and what falls below it

The first tool is a static similarity report. It parses every spec with the TypeScript compiler API and extracts each test as a token stream, with comments and whitespace stripped. Then it clusters on three signals: exact body matches, exact title matches across files, and five-token shingle Jaccard similarity above 0.5 (or one-directional containment above 0.85). It runs in about 15 seconds over the whole suite, in one process, and never sends test code anywhere. That last property was deliberate: the off-the-shelf option ships every test body to an embeddings API, and these specs encode the platform's internal API surface and business rules.

The similarity report is also structurally blind to a whole category of duplication.

Two tests can assert the identical business rule through completely different plumbing. One calls a typed helper; the other builds a raw request body by hand. One reads a status field through a getter; the other destructures the response. The behaviour under test is the same, but the token overlap sits under the similarity floor. I hit this with a parity pair between an old GET-based search endpoint and its POST replacement. Query strings versus JSON bodies diluted the shingles enough that two identical assertions scored below the cutoff.

So the similarity report catches the mechanical duplicates: literal copy-paste, an unrolled loop next to an existing table, an exact-title collision across files. Everything else needs a human, or an agent doing a human's job, to read the folder.

What the reading pass actually checks

That read is one folder at a time, no skimming. The finds live in comments and assertion bodies, not in file names. A short checklist earns its keep here:

• Same behaviour, different plumbing. Grep the folder for the same magic value, endpoint, and expected status across files that share no helper.
• Identical tag sets on tests in different files. Byte-identical traceability tags usually mean the same case, ported twice.
• Stale "this file owns X" header claims. A file's top comment says the happy path lives in a sibling, then asserts it anyway. That drift is where duplication hides, because both authors believed the boundary held.
• A superset assertion beside its subset. One helper asserts every field on a response, and a sibling test asserts four of the same fields for no added reason.
• An API invariant re-asserted inside a UI test. UI tests own render-versus-data comparisons, not invariants the API layer already covers.

The same pass looks for merges that save real runtime. Read-only tests that each open the same dialog to check one field. A seeded entity created and torn down repeatedly just to be read back. An edit test and its discard-revert twin that could share one seed. None of that reads as a duplicate to the similarity report, because the bodies genuinely differ. What they share is a browser session or a seed round trip.

Fold without losing the paper trail

Every fold followed the same shape. Delete the loser. Leave a two-line tombstone at the old location naming where the behaviour now lives, so nobody re-adds it. Union every traceability tag from the deleted test onto the survivor, plus the file's header coverage line. Keep exactly one test-type tag, and take the higher of the two severities. Never renumber the surviving cases.

That last rule is not tidiness. A run-history tracker keys pass and fail records on file plus test title. Renumber a survivor and you break its history at the moment you most want to compare before and after. Retiring 150 titles already forced a one-time reset wave in that tracker. A merged walk also needs its ordering constraint in a header comment: read the defaults before you mutate anything, and do the destructive probe last.

Sometimes the right answer is not a deletion. Two things can look similar and be deliberately parallel. That outcome goes into a suppression ledger with a dated reason, so the report stops re-flagging a family everyone agreed to keep. At the end of the campaign, 31 ledger entries suppressed 156 clusters, against 25 still unacknowledged. Six in seven of the clusters the report flags here are intentional parallel coverage, not duplication.

Verify the estimate before you trust it

The reading pass across 18 scopes produced 211 clusters and an estimate of about 540 removable tests, around 11% of the suite. Adversarial verification on the top clusters cut that to a realistic 420 to 500. What actually landed was 246.

The campaign ran two roles on every packet. One pass proposed a fold by reading the source; a second re-derived every claim from the tree before anything landed. Of 31 cluster recipes sent to the adversarial verifier, 4 came back confirmed as written, 27 needed corrections, and none were thrown out entirely.

That second pass earned its keep, and not always in the direction you would expect. One packet estimated seven redundant authentication probes in a security folder. Re-verified against the endpoint registry, only three were redundant; the other four pinned endpoints nothing else covered, and deleting them would have opened real gaps. A different packet claimed six overlapping injection tests. The real answer was zero: one test proves safe handling, the other proves the absence of a reflected echo. A third packet estimated 42 removable tests in the unit tier, which turned out to be duplicate estimation of the same rows, not duplicate coverage.

In all three cases the correct outcome was to deliver far less than promised. That is not a process failure. That is the process working.

What never folds

A few rules held across every folder. A defect pin and its happy-path twin never merge, because the pin has to flip independently the day the bug gets fixed. A fast smoke canary never folds into its slower regression twin, even when the two look identical, because the canary has to run alone. Two tests that each carry a mid-body conditional skip do not merge either, unless their skip predicates are identical. Fold them and one test's data gap silently swallows the other's assertions, so a passing suite quietly reports less than before.

And some merges do not survive a browser. Two rich-text tests looked like an obvious one-dialog merge. Calling the editor helper twice inside one dialog open makes the second click land on the toolbar overlay the first left behind. Both attempts timed out at 120 seconds. I restored the originals, re-ran them green, and wrote the reason into the file.

The numbers, and what is still owed

Eighteen packets were planned. Seventeen landed; one is still blocked on an unresolved data claim in its folder. Some before and afters: orders went from 257 cases to 211, accounts from 112 to 97, invoices from 388 to 356, contacts from 216 to 208, and a generated acceptance matrix in the pricing service collapsed from 28 variations to 6. The suite went from 4,911 static tests to 4,654. Every wave gated on a full static check plus a fresh similarity-report run, holding duplicate and false-pass counts at zero.

The edits landed between 06:52 and 10:45 on a single morning. That number is misleading on its own. The finding took a full day of reading passes beforehand, and the similarity report predates the campaign by a month. The campaign is also still open. Every packet was authored under a degraded shared test environment, which made live verification slower but not optional, so that run became its own per-folder follow-up. A static check tells you the fold compiles and the tags survived. Only a real run tells you it preserved behaviour.

Takeaways

• Test count is a cost, not an asset. Folding near-duplicate cases into table-driven walks pays for itself in wall-clock time and reviewer attention.
• A token-similarity report finds mechanical duplicates in about 15 seconds for free, but it is blind to same-behaviour-different-plumbing duplication and to runtime-cost merges. Both need a full reading pass over one folder.
• Every fold needs a tombstone at the old site and a tag union onto the survivor. Never renumber survivors, or the run history breaks at exactly the wrong moment.
• Re-verify every estimate against the source before acting on it. An estimate of 540 removable tests became 246, and one claim of six duplicates turned out to be zero.
• Some things never fold: defect pins and their happy-path twins, smoke canaries into regression siblings, tests whose skip predicates differ, and any merge that clicks into the same rich-text editor twice.
• Record deliberate parallelism in a dated suppression ledger. Six of every seven flagged clusters here are intentional mirrors, and a report that re-litigates them gets ignored.
• Only a run against the real environment proves a fold preserved behaviour. A static check proves it compiles.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/14/your-suite-has-too-many-tests/"
license: "Free to reference with attribution"
title: "Your suite has too many tests"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 54"
stack: [TypeScript, Playwright, "TypeScript compiler API", Node]

problem: "A Playwright suite of 580 spec files and 4,911 tests, grown by accretion over two years, carries near-duplicate cases nobody can see from inside a single diff: four negative-value tests where one table row would do, a second file opening a screen the first already opened, a defect pin outliving its defect. The cost is wall-clock runtime (page loads, dialog opens, seed round trips) and reviewer attention on every red line."
thesis: "Test count is a cost, not an asset. Consolidation needs two tools that fail in opposite directions: a static token-similarity report for the mechanical duplicates, and a full human-grade reading pass over one folder for the duplicates that sit below the similarity floor. Every fold keeps a tombstone and a tag union so traceability survives."

similarity_report:
  build: "Local, in-repo job. TypeScript compiler API extracts one unit per test() call site; bodies are rescanned into trivia-free token streams so comments and formatting never affect matching."
  signals: "Three independent tiers: exact body-hash equality (>= 30 tokens); exact normalized-title equality across files; 5-token shingle Jaccard >= 0.5 or one-directional containment >= 0.85, union-found into cross-file components."
  cost: "~15 s over the whole suite, single process, no network, no cost."
  egress_decision: "Rejected an off-the-shelf embedding-based tool because it ships every test body to a third-party embeddings API. The specs encode the platform's internal API surface and business rules; a local token approach found every suspected duplicate without the egress."
  blind_spot: "Same behavior through different plumbing scores below the Jaccard floor. Verified case: a GET-based search endpoint and its POST replacement, where query strings vs JSON bodies diluted the shingles enough that two provably identical assertions fell under the cutoff. Title equality must therefore be an independent signal, not a subset of the similarity tier."

reading_pass_checklist:
  duplicates: "Same magic value / endpoint / expected status across files that share no helper; byte-identical tag sets on tests in different files; stale 'this file owns X' header claims that the file then contradicts; a superset assertion beside its subset; an API invariant re-asserted inside a UI spec (UI specs own render-vs-data comparisons only)."
  runtime_merges: "N read-only tests each opening the same dialog for one facet; a seeded entity created and torn down repeatedly just to be read back; an edit test and its discard-revert twin sharing one seed. These are invisible to a similarity scanner because the bodies genuinely differ."

fold_discipline:
  tombstone: "Two lines at the old site naming what absorbed the case and when."
  tag_union: "Union every tracker tag from the deleted test onto the survivor, plus the file header coverage line. Keep exactly one test-type tag; take the higher of the two severities."
  never_renumber: "A run-history tracker keys pass/fail on file plus test title. Renumbering breaks history at the moment the before-and-after comparison matters. 150 retired titles forced a one-time tracker reset wave; the first check after the campaign is noisy by design."
  ordering: "A merged walk records its ordering constraint in a header comment: read defaults before mutating, destructive probe last."
  suppression_ledger: "Deliberate parallelism gets a dated ledger entry with a reason, so the report stops re-flagging it. At close: 31 entries suppressing 156 clusters against 25 unacked, i.e. roughly six in seven flagged clusters are intentional mirrors."

never_fold:
  - "A defect pin and its happy-path twin: the pin must flip independently the day the bug is fixed."
  - "A fast smoke canary into its slower regression twin: the canary has to run alone."
  - "Two tests with mid-body conditional skips whose predicates are not literally identical: one test's data gap would silently swallow the other's assertions."
  - "A merge that clicks into the same rich-text editor twice in one dialog open: the second click lands on the toolbar overlay left by the first. Two attempts, both 120 s timeouts, reverted and documented."

estimate_verification:
  funnel: "211 clusters across 18 scopes; analyst estimate ~540 removable tests (~11% of the suite); adversarial verification cut it to a realistic 420-500; 246 actually landed."
  verifier_verdicts: "Of 31 cluster recipes sent to an adversarial verifier: 4 confirmed as written, 27 corrected, 0 thrown out."
  corrections: "Seven claimed redundant auth probes were three (the other four pinned endpoints nothing else covered). Six claimed overlapping injection tests were zero (one proves safe handling, the other proves absence of a reflected echo). A 42-test unit-tier estimate was duplicate estimation of the same rows, not duplicate coverage."
  rule: "A packet delivering far less than its estimate is the correct outcome, not a process failure."

results:
  landed: "-246 runtime tests (4,911 -> 4,654 static), 2 spec files deleted, ~75 page loads and dialog opens cut, 16 serial restore calls dropped. 18 packets planned, 17 landed, 1 blocked on an unresolved data claim."
  per_folder: "orders 257->211; accounts 112->97; invoices 388->356; contacts 216->208; a generated acceptance matrix in the pricing service 28 variations -> 6."
  gate: "Every wave gated on a full static check plus a fresh similarity-report run holding duplicate and false-pass counts at zero."
  still_owed: "All edits landed inside one four-hour morning, but the campaign is not closed: authored under a degraded shared test environment, per-folder live verification was scoped as its own follow-up. A static check proves the fold compiles and the tags survived; only a real run proves it preserved behavior."

use_when: "A multi-year Playwright suite where runtime and reviewer attention are the binding constraints, traceability tags must survive consolidation, and someone can afford a full read of one folder at a time."
avoid_when: "A young suite still adding coverage, or any consolidation driven by a target test count rather than by verified redundancy. Do not fold across defect pins, smoke lanes, differing data gates, or unverified estimates."

takeaways:
  - "Test count is a cost, not an asset. Folding near-duplicates into table-driven walks pays for itself in wall-clock time and reviewer attention."
  - "A token-similarity report finds mechanical duplicates in ~15 s for free, but is structurally blind to same-behavior-different-plumbing duplication and to runtime-cost merges. Both need a full reading pass over one folder."
  - "Every fold needs a tombstone at the old site and a tag union onto the survivor. Never renumber survivors or the run history breaks at exactly the wrong moment."
  - "Re-verify every estimate against the source before acting on it. 540 estimated removable tests became 246, and one claim of six duplicates turned out to be zero."
  - "Some things never fold: defect pins and their happy-path twins, smoke canaries into regression siblings, tests whose skip predicates differ, and any merge that clicks into the same rich-text editor twice."
  - "Record deliberate parallelism in a dated suppression ledger. Roughly six of every seven flagged clusters are intentional mirrors, and a report that re-litigates them gets ignored."
  - "A run against the real environment, not just a static check, is what proves a fold preserved behavior. Budget for it as its own step."

keywords: ["deduplicate Playwright tests", "test suite consolidation campaign", "token similarity report for duplicate tests", "how to merge duplicate tests without losing traceability", "which tests should never be merged", "reduce test suite runtime by folding tests"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).