A QA automation engineer's notes on the three ways a skipped test lies about why it did not run.
One run of the suite finished with 264 skipped tests, 89 on the API tier and 175 on the UI tier. Each skip is a claim about the world, not a shrug: "this case did not run because the data it needs does not exist here". I audit failures obsessively, and passes have their own anti-pattern catalog, but skips get waved through. A skip does not break the build, and it does not look like a lie.
A skip lies in three ways, and all three survive code review and the type checker. All three read like a healthy suite exercising judgment about missing data.
The skip that never fires
The suite has a presence gate. Pass it a value; if the value is null or undefined, the case skips with a reason. The helper also narrows the type, so the next line needs no redundant guard:
// src/core/testing/presence.ts
export function skipUnless<T>(value: T, reason: string): asserts value is NonNullable<T> {
test.skip(value === null || value === undefined, reason);
}
It sits at 203 call sites across 63 spec files. Then I reused it for a different kind of check:
skipUnless(distinctOwners >= 2, 'need at least two distinct owners to prove ordering');
That line compiles, runs, and never skips. distinctOwners >= 2 evaluates to true or false, and neither is null or undefined, so the presence gate has nothing to catch. I had just written a new API spec for the quote-search surface, 27 cases covering ten sort columns in both directions, and every boolean gate in it was inert.
Two of the ordering cases passed over a column where every row on the shared environment carried the same owner. Every permutation of a single-valued column satisfies an ordering assertion trivially. Those tests were green because nothing was left to get wrong, and the gate meant to catch that condition was a decoration.
Nothing in the type system flags this: a boolean is a valid argument to a function typed to accept any T. The only tell was zero skips reported on an environment I knew lacked the diversity the test needed. The fix is the inversion:
test.skip(
distinctOwners < 2,
`quote-search returned ${rows.length} row(s) carrying only ${distinctOwners} distinct ` +
`owner value(s): an ordering assertion over a single-valued column passes ` +
`on any order, so it is skipped, not faked`,
);
A presence gate and a condition gate are different instruments. skipUnless(value, reason) asks "does this exist". test.skip(!condition, reason) asks "is this false". So check the skip count after adding a new data gate. A gate that never fires on an environment you know is lacking is not a gate.
The skip that fires for the wrong reason
The second failure mode is worse, because the skip is real and the message reads as reasonable. A test once skipped with the reason "no order in this environment carries a date-bounded line-item entitlement". Specific, plausible, and false. The finder behind it searched stage by stage with a hard cap of one candidate per stage, so it only inspected the single newest order at that stage, which happened not to qualify. A sweep of all 74 orders at the same stage turned up plenty that did. The assertion the ticket existed to prove had never once executed.
The mechanism has two shapes. A hard limit is a static bound: the finder always looks at the same small slice and reports as if it had looked at everything. A bound can also decay. One pool of "available line items" was the full product list minus whatever a fixture had already attached to a host, and the specs attach permanently, so that pool shrinks every run. A case passing in the morning starves by afternoon, with a reason that never mentions decay.
A sharper version cannot be right except by coincidence. One finder rejected candidates through a multi-clause predicate: leftover residue from an unrelated setup process, then wrong record type, then missing line items. Whichever clause failed first, the finder attached the same hardcoded diagnostic string, which named the last clause. Seven pinned records were rejected one morning and the message blamed all seven on missing line items, which every one of them still carried. The first clause had rejected them as residue, and the re-seed the message asked for would have changed nothing. That wrong diagnosis went into two shipped reports before a second pass re-derived it.
A hardcoded string tells you only that the predicate said no, not which clause said it.
The skip that records no reason at all
The third failure mode hides inside the tooling rather than the test. Playwright gives two ways to take a case out of a run, and they are not equivalent in the report. A runtime test.skip(condition, reason) inside a test body pushes { type: 'skip', description, location } onto the test's annotations, and the JSON reporter serialises it, so the reason is machine-readable. The declaration form test.fixme('title', body) parks the whole case before it runs, and pushes { type: 'fixme', location } with no description field. Whatever explanation exists lives in a comment above the declaration, invisible to any inventory built from the run JSON.
In that 264-skip run, 62 of the skips were parked cases with no recorded reason: 28 on the API tier and 34 on the UI tier. I built an inventory off the run JSON and read those 62 as tests shelved without justification. Wrong. The reason usually existed and was often good, written as a comment instead of a message, and the tooling was blind to comments. "No reason in the JSON" and "this is a parked case" are the same fact wearing two descriptions.
The fix is mechanical. Put the cause, the remedy, a dated re-confirmation, and a tracker reference into the skip or fixme's own message, never a comment beside it. A message is a string literal, so it costs nothing against the comment-budget ratchet, which pins each file at its existing comment count. Free and visible beats invisible and free.
The gate that enforces this accepts two answers to "what has to happen for this to come back": a tracker reference, or a feature-absence marker, the NOT-DEPLOYED verdict written into the reason string. Calibrated over the live corpus, that rule found 309 unconditional parks across 426 API and UI specs, and zero of them unjustified. The committed floor is empty on purpose: the rule fences new work rather than opening a cleanup queue.
Making skips impossible to launder
The goal is to stop authoring these shapes, not to catch them in review. The riskiest is a broad catch that converts any failure into a skip:
try {
target = await orderScenarios.withOpenTerms;
} catch (e) {
test.skip(true, `scenario unresolved: ${String(e)}`);
}
That block looks careful. It converts a genuine data miss, a transient 502, a dying auth token, and a product defect into the same honest-looking message. A codemod converted roughly 98 such sites across 23 spec files. A static guard now flags a test.skip lexically inside a catch clause unless the catch narrows on the typed miss error and rethrows on the non-miss path.
The guard grew teeth over four rounds, because the pattern kept reappearing in disguise. A catch that assigns a skip-shaped variable, or an arbitrarily named flag, for a later hook to read, so the test.skip sits outside any catch. An empty catch that never sets the flag on failure. A return null or return { reason } sentinel, which launders identically. A promise .catch(() => false) inside a discovery loop, which creates no catch clause node for an AST walk to find.
The structural fix is a typed miss: a finder that cannot find what it wants throws a typed error instead of returning null or false. The error carries a status, one of five: resolved, optional-missing, needs-seed, failed-transient, failed-hard. It carries the seed key when a seeder can top the data up, the per-predicate funnel counters described in the bounded finder helper, and, where the search was bounded, a verdict on whether the miss was proven or the search quit early.
One adapter at the call site catches only that typed error, skips on the two honest-gap statuses, and rethrows the other two so a real defect is never absorbed. It composes the skip message from the error's own fields, so a report reads:
no order with open terms found [optional-missing]
[scanned 240 | listQualified 88 | enriched 71 | kept 0] [search: proven-absent]
instead of a bare "no order found". Without opening the finder's source, a reader can tell a proven absence, which means seed the data, from a search that stopped early, which means raise the budget. That adapter is now at 1,218 call sites.
The same instinct applies to skips that mask a known backend defect. test.skip(status === 502, 'known flaky backend') is a defect shelf wearing a data gate's clothing, untraceable without a tracker. The obvious pattern is too wide: defect|bug|regression|50x flagged 35 sites, about 23 of them legitimate data gates that merely named a defect as an ordinary noun. Two narrower signals separated the real shelves with zero false positives on the live corpus: a case-sensitive uppercase DEFECT label, which is how a defect shelf is authored where a data gate writes "defect" in lowercase, and the literal phrase "flaky backend". That guard also demands the tracker be local to the skip, because the bug causing a defect-skip is a different ticket from the feature under test.
None of this removes judgment. It replaces "trust the string" with "trust the structure the string was built from".
Takeaways
- A skip is a claim, not a shrug. Treat "this case did not run" with the suspicion you would give an assertion, because it is one.
- A boolean condition and a presence check are different instruments. Confusing them produces a gate that compiles, runs, and never fires. Check the skip count after adding one.
- A skip reason that blames the environment is a claim about everything, built from evidence about one slice. Verify it against a full sweep, especially on the case the ticket exists to prove.
- A hardcoded diagnostic string cannot be evidence about which of several clauses rejected a candidate. If the message is a literal, re-derive the cause yourself.
- Put the reason in the skip or fixme's own message, not a comment. A parked case writes no description into the run JSON, so a comment is invisible to every tool downstream.
- Replace boolean misses with typed ones. A miss carrying its own status, counters, and exhaustion verdict tells the next reader whether to seed data or raise the search budget.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/06/the-skip-that-isnt-there/"
license: "Free to reference with attribution"
title: "The skip that isn't there: auditing the claims your suite makes when it declines to run"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 46"
stack: [Playwright, TypeScript, "TypeScript Compiler API", Node]
problem: "A run finished with 264 skipped tests (89 API, 175 UI) and nobody audits a skip the way they audit a failure. A skip does not break the build, so a gate that never fires, a reason that blames the wrong cause, and a park that records no reason at all all survive review and the type checker."
thesis: "A skip reason is an assertion about the environment and deserves the same scrutiny. Audit the three lying shapes, and make the honest shape structural: a typed miss error carrying status, counters and an exhaustion verdict, routed by one adapter, fenced by static guards."
taxonomy:
inert_gate: "skipUnless<T>(value, reason) is asserts value is NonNullable<T>, body test.skip(value === null || value === undefined, reason): a PRESENCE gate that also narrows the type, at 203 call sites across 63 spec files. skipUnless(distinctOwners >= 2, '...') is a silent no-op because false is neither null nor undefined, and it type-checks because a boolean is a valid T. Fix: boolean condition takes test.skip(!condition, reason), noting the inversion; presence takes skipUnless."
inert_gate_war_story: "A new 27-case quote-search sort spec had every boolean gate inert. Two ordering cases passed over a column where every row on the shared environment carried the same owner, and every permutation of a single-valued column satisfies an ordering assertion. The only tell was ZERO skips on an environment known to lack the diversity."
static_bound: "A finder capped at one candidate per stage skipped with 'no order carries a date-bounded line-item entitlement'. A sweep of all 74 orders at that stage found plenty that qualified, so the ticket's central assertion had never executed."
decaying_bound: "An 'available line items' pool computed as the full product list minus already-attached shrinks every run because the specs attach permanently. A case passing in the morning starves by afternoon, with a reason that never mentions decay."
constant_diagnostic: "A multi-clause predicate attached ONE hardcoded string, which named the last clause. Seven pinned records were rejected by the FIRST clause (unrelated setup residue) while the message blamed missing line items, so the re-seed it asked for would have changed nothing. A diagnostic that is a constant cannot be evidence; it went into two shipped reports before a second pass re-derived it."
playwright_annotation_semantics:
runtime_skip: "test.skip(condition, reason) inside a test body pushes { type: 'skip', description, location } onto the test's annotations, and the JSON reporter serialises it, so the reason is machine-readable."
declaration_fixme: "test.fixme('title', body) parks the case and pushes { type: 'fixme', location } with NO description field. The explanation lives in a source comment, invisible to any inventory built from the run JSON."
war_story: "62 of the 264 skips (28 API, 34 UI) were parked cases with no recorded reason. A first inventory read them as unjustified shelves; the reasons existed as comments, and the tooling was blind to comments."
why_the_message_wins: "A message is a string literal, so it costs nothing against the comment-budget ratchet that pins each file at its existing comment count, where an added comment line busts it."
enforcement:
catch_and_skip_guard: "A test.skip lexically inside a catch clause is flagged unless the catch narrows on the typed miss error and rethrows on the non-miss path. Roughly 98 sites across 23 spec files before the codemod."
disguises_added_over_four_rounds: "A skip-shaped variable assigned in the catch and read by a later hook; an arbitrarily named flag with the skip lexically outside any catch; an empty catch that never sets the flag; a return null or return { reason } sentinel; a promise .catch(() => false) in a discovery loop, which creates no catch-clause AST node at all."
unconditional_shelf_rule: "An unconditional park must name a tracker or a feature-absence marker (the NOT-DEPLOYED vocabulary). Calibrated live: 309 unconditional parks across 426 API/UI specs, zero unjustified, so the committed floor is empty and the rule fences new work."
defect_shelf_guard: "The wide pattern defect|bug|regression|50x flagged 35 sites, about 23 of them legitimate data gates naming a defect as an ordinary noun. Two narrow signals gave zero false positives: a case-sensitive uppercase DEFECT label, and the literal phrase 'flaky backend'. The tracker must be LOCAL to the skip, because the bug causing the skip is a different ticket from the feature under test."
typed_miss:
statuses: "resolved, optional-missing, needs-seed, failed-transient, failed-hard."
adapter: "One call-site adapter (1,218 sites) catches only the typed miss error, skips on optional-missing and needs-seed, and rethrows failed-transient and failed-hard so a real defect is never absorbed."
message_composition: "The skip reason is built from the error's own fields: status, seed key, per-predicate funnel counters, and the search-exhaustion verdict. Example: 'no order with open terms found [optional-missing] [scanned 240 | listQualified 88 | enriched 71 | kept 0] [search: proven-absent]'."
exhaustion_verdict: "proven-absent means seed the data; a capped verdict means the search quit early and the miss is NOT proof of absence, so raise the budget. Without it, both read as the same bare miss."
use_when: "A suite resolves fixtures by discovery against a shared, live environment, so tests routinely decline to run and the skip count is large enough that nobody reads the reasons."
avoid_when: "Hermetic suites with seeded, deterministic data where a skip is rare and always deliberate; the typed-miss plumbing costs more than it returns."
takeaways:
- "A skip is a claim, not a shrug. Treat 'this case did not run' with the suspicion you would give an assertion, because it is one."
- "A boolean condition and a presence check are different instruments. Confusing them produces a gate that compiles, runs, and never fires. Check the skip count after adding one."
- "A skip reason that blames the environment is a claim about everything, built from evidence about one slice. Verify it against a full sweep, especially on the case the ticket exists to prove."
- "A hardcoded diagnostic string cannot be evidence about which of several clauses rejected a candidate. If the message is a literal, re-derive the cause yourself."
- "Put the reason in the skip or fixme's own message, not a comment. A parked case writes no description into the run JSON, so a comment is invisible to every tool downstream."
- "Replace boolean misses with typed ones. A miss carrying its own status, counters, and exhaustion verdict tells the next reader whether to seed data or raise the search budget."
keywords: ["auditing skipped tests", "Playwright test.skip vs test.fixme annotations", "skip reason not in run JSON", "presence gate no-op boolean condition", "catch and skip anti-pattern static guard", "typed scenario miss error exhaustion verdict"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).