A QA automation engineer's notes on running a folder of tests twice, once with every assertion inverted, to find the tests that cannot fail.
Every quality gate in the suite answered a version of the same question: does this test look right. Right imports, right annotations, a status code check, a schema validator. None answered the question that decides whether the suite is worth anything. If the feature broke tonight, would this test, on this run, turn red?
I already had a catalog of ways a test can pass while proving nothing: "False passes: the anti-pattern catalog that keeps the green suite honest". What I lacked was a mechanical way to ask each test directly: prove to me you can fail. So I built the falsifier.
An audit that reads code is not an audit that runs it
The existing quality tools are static. They read a spec file and reason about its shape: is there an assertion outside every conditional, does a mutation re-read from the server, is the response validated before anything is asserted. That catches a lot. It misses a test that looks idiomatic and still cannot fail, because the branch carrying its only assertion never executes at runtime.
The falsifier can. Run a folder of tests twice: once normally, once with every assertion inverted. A test that stays green when it should be failing has an oracle problem, whatever its source code looks like.
Two neighbouring tools already existed, and neither answers it. A static blind-oracle guard flags five known shapes of dead assertion, and it keeps owning tautologies, since a tautology inverts into a failure that the falsifier credits as real. A per-field perturbation harness measures assertion strength, at one child run per field. The falsifier answers only presence and reach, at two runs per folder.
Two runs, one inversion
The mechanism is simple, thanks to a convention that predates it. Every spec imports expect from one shared fixtures barrel, and a lint guard bans direct imports from the test runner package: 564 of 567 imports go through the barrel, and the three exceptions are unit tests of the guards. One export line is the whole seam.
Under inversion, every positive matcher access reroutes through Playwright's own .not, so a check that required toBe now requires not.toBe. Playwright computes the negation, so a failure carries a real assertion error with a real diff.
// src/core/testing/falsifyExpect.ts, condensed
function flipMatchers(matchers: MatchersLike): MatchersLike {
return new Proxy(matchers, {
get(target, prop) {
if (prop === 'not') return target; // double negation
if (prop === 'toPass') return Reflect.get(target, prop, target);
if (prop === 'resolves' || prop === 'rejects') {
return flipMatchers(Reflect.get(target, prop, target)); // one layer down
}
const negated = Reflect.get(target, 'not', target);
return Reflect.get(negated, prop, negated); // positive to negated
},
});
}
// and the seam itself, in the barrel:
export const expect = process.env.FALSIFY === '1'
? falsifyExpect(countedExpect)
: countedExpect;
Unset, the path is byte-identical to today's.
An assertion that already reads .not double-negates back to positive, and 761 call sites depend on that. The resolves and rejects layers nest a second set of matchers, so the inversion recurses one layer down. expect.soft is an Expect itself, so the same factory wraps it recursively. expect.poll inverts at its matcher layer: a truthful poll never satisfies the inverted matcher, so it times out and fails, correctly. .toPass has no negation in Playwright, so it stays truthful and gets named in the report's limitations. Every other static passes through untouched: asymmetric matchers such as objectContaining are arguments, not oracles.
The one oracle that deliberately stays truthful
The schema validator that gates every response was the biggest design decision here, and I got it backwards at first. The instinct is to invert everything.
A recount changed the question. The feasibility estimate assumed roughly 1,070 call sites into the shared validator helpers. The census found 267: 239 for assertValid and 28 for assertValidExcept. That reframed the problem from "how do I invert a thousand call sites cheaply" into "what would inverting this validator prove".
Not much, it turned out. The response schemas mostly lack additionalProperties: false: of 114 component schemas across the three contract files, 21 declare it. The validator's real weak spot is not "does it reject a wrong type", which inversion would exercise. It is "does it notice a field the API silently added or renamed", which inversion cannot exercise.
So the validator stays truthful, and every call into it gets counted instead. A per-test tally records how many times each oracle kind fired on the executed path, and Playwright carries it out as an annotation in the JSON report. A live validator call plus zero invertible assertions elsewhere is not a failure of this tool. It is schema validation working as designed, and the count says so.
The fences around a very sharp seam
An environment variable that inverts every assertion must never leak into an ordinary run, so it ships with fences. The inversion fires only alongside a second handshake variable that the falsifier's CLI sets, which kills ad-hoc invocations, and a startup script in every pretest lane refuses to run on a stray inversion variable left from a previous shell. Sharded runs refuse to participate: a partial inversion across shards produces nonsense. The regression-history recorder is force-disabled: an inverted run's outcomes are not real regressions. Defect-pin specs drop out of selection, along with every project other than the API and UI tiers, and the assertion timeout drops from five seconds to two.
What the first audit caught was a bug in the falsifier
The calibration pass covered two search folders and 119 tests. 105 came back proven able to fail, with zero swallowed inversions. Not the interesting part.
The interesting part looked like a flake and was not. One row was reported as failing in both clean children at four workers, and the brief that reached me named a parallel race as established fact. There was no race. That row is an inline pinned-failure test for a known backend defect, the test.fail pattern I use as a living bug tracker. Both clean runs recorded it as "status": "expected", "ok": true, which Playwright reports as green.
A Playwright JSON report gives every test two statuses, and they disagree on every held pin. test.status is the outcome (expected, unexpected, flaky, skipped); results[n].status is the raw attempt (passed, failed, timedOut). A held pin records a raw failed inside an outcome of expected. My classifier read the raw attempt, so it labelled a green pin "not clean, fix this first" and dropped the row.
The reproduction took ten minutes. I re-ran the folder live at four workers twice: 33 passed, 3 skipped, 0 unexpected each time, and green solo. The fix was to key the classifier on the expected outcome. That mattered more than the 105 provens, because it separates a tool you can trust from one that manufactures false alarms. The lane excluded defect pins by filename, so this pin slipped through: it is inline in an ordinary spec. A census then found 19 inline test.fail call sites across 10 spec files in the lane.
The survivor queue is a worklist, not a verdict
Every test green under inversion lands in a review queue. Six days after calibration I swept 39 scopes across every domain folder. Of 1,894 scoreable tests, 1,787 were proven able to fail (94.4%) and 107 survived, across 44 spec files. All 107 were worked the same day: 102 re-audited as proven, 5 justified as expected survivors. Not one root cause matched the label the tool assigned first.
56 were tagged "zero oracle", and almost none were oracle-free. They were UI tests whose assertions live inside a page object, and page objects bind the assertion library directly rather than through the barrel. The fix is not to crack open the page object's internals, but to add one invertible check on its public surface: an exact sentinel-row count, or a boolean state pin.
34 were schema-gated: a helper checks both the status and the error shape internally, through the raw library, so nothing is left in the test body to invert. One barrel-level status pin beside the helper call closes that gap.
All 10 tagged "swallowed" were supposed to be the catch-and-launder shape. None were. Every one was negation asymmetry inside the inversion itself. An inverted "not visible" check returns as soon as the condition is momentarily true, which happens while the page is still loading. The fix is a wait gate that inversion cannot touch, then a re-assert against the settled state.
A fourth class did not appear in the labels: sort verifiers pass vacuously on a page with zero or one row, so every search that feeds a sort proof now carries a floor of two visible rows. The four classes are a fix recipe, and the order matters: guessing the wrong class first wastes a debugging pass on the wrong file.
A lane, not a gate
The falsifier never runs by default and never runs across the full suite uninvited. It is folder-scoped, run on demand, kept out of every merge gate. It tells you where to look, not that the code is broken, and "go investigate this test" makes a poor gate that gets muted within a month.
Report first, gate later, once the false-positive rate has earned the weight. After calibrating the oracle counters across five folders and roughly 300 classified tests, the split was clean: the API tier had zero zero-oracle hits, and in the UI tier 42 of 42 non-setup hits were page-object delegation. So the ratchet shipped for the API tier only, where the barrel is the asserting path by convention. A blanket ratchet would have fired almost entirely on a convention I introduced.
Takeaways
- A test that reads correctly is not the same claim as a test that can fail. Only an inverted run answers the second question.
- Do not invert an oracle whose real weakness inversion cannot see. The schema validator's blind spot is additive drift, so it stays truthful and gets counted.
- Get the real call-site count before you choose the mechanism. A recount from 1,070 assumed sites to 267 real ones changed which design was worth building.
- The tool's own classifier is a bug surface. My first real finding was the falsifier misreading a held defect pin's raw attempt status as a failure.
- A survivor label describes what the lane could see, not what the test lacks. Diagnose the blindness before you touch the file.
- Report before you gate. The zero-oracle ratchet shipped for the API tier only, because in the UI tier it would have flagged my own convention.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/03/can-this-test-fail-at-all/"
license: "Free to reference with attribution"
title: "Can this test fail at all? A falsifier that inverts every assertion"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 43"
stack: [Playwright, TypeScript, "JavaScript Proxy", "JSON Schema", Node]
problem: "Static quality gates only check that a spec looks right: imports, annotations, status checks, schema validation. None answer whether a test would turn red if the feature broke, because a spec can be idiomatic and still carry its only assertion on a branch that never executes at runtime."
thesis: "Run a folder twice, once clean and once with every assertion inverted through the test runner's own negation. A test green in both runs proved nothing on its executed path. Ship it as an on-demand, folder-scoped lane, never a merge gate."
mechanism:
seam: "Every spec imports expect from one shared fixtures barrel (564 of 567 imports; the 3 exceptions are unit meta-tests of the guards) and a lint guard bans direct test-runner imports. One env var swaps that single export; unset, the path is byte-identical."
proxy: "A Proxy get trap reroutes every positive matcher access through Playwright's own .not, bound to the negated receiver, so failures carry real assertion errors and diffs rather than an invented negation."
edge_cases: "A spec-level .not double-negates back to positive (761 call sites). resolves/rejects recurse one matcher layer down. expect.soft is an Expect, so the factory wraps it recursively. expect.poll inverts at its matcher layer and a truthful poll then times out, which is correct. .toPass has no Playwright negation and stays truthful (2 sites, named in report limitations). Every other static passes through untouched because asymmetric matchers are arguments, not oracles."
counters: "Helper oracles are counted, not inverted: a per-test tally of oracle kinds ships out as a Playwright annotation in the JSON report, so the classifier joins on it with no extra files. The counters now run in every normal run."
key_decisions:
validator_stays_truthful: "A recount cut the assumed 1,070 shared-validator call sites to 267 (assertValid 239, assertValidExcept 28). Of 114 component schemas across 3 contract files, only 21 declare additionalProperties: false, so the validator's real weakness is additive drift, which inversion cannot exercise."
scope_vs_neighbours: "A static blind-oracle guard keeps five known dead assertion shapes and keeps tautologies, which the falsifier wrongly credits as proven. A per-field perturbation harness (one child run per field-and-operation case, 9 curated read-only API targets) keeps assertion strength. The falsifier answers only presence and reach, at two runs per folder."
lane_not_gate: "Opt-in, folder-scoped, on-demand, dev environment only, no CI, never in the full-run wrappers. A verdict of 'go investigate this test' makes a gate that gets muted within a month."
fences:
handshake: "The inversion fires only when a second CLI-set variable is present, killing ad-hoc invocations; a pretest startup script refuses to run on a stray ambient inversion variable."
isolation: "Sharded runs refuse to participate (partial inversion is nonsense). The regression-history recorder is force-disabled. Defect-pin specs and all non-API/UI projects drop out of selection. Assertion timeout drops from 5s to 2s."
war_story:
classifier_bug: "Calibration on two search folders, 119 tests, 105 proven, 0 swallowed. The one apparent finding looked like a four-worker parallel race and was not. A Playwright JSON report carries two statuses per test: test.status is the outcome (expected/unexpected/flaky/skipped) and results[n].status is the raw attempt (passed/failed/timedOut). A held test.fail pin records a raw 'failed' inside an outcome of 'expected'. The classifier read the raw attempt and dropped a green pin as 'not clean'."
resolution: "Re-run live at 4 workers twice: 33 passed / 3 skipped / 0 unexpected each time, and green solo. Fixed by keying on the expected outcome. The lane excluded defect pins by filename infix, which missed 19 inline test.fail call sites across 10 in-lane spec files."
sweep: "39 scopes across every domain folder: 1,894 scoreable tests, 1,787 proven (94.4%), 107 survivors in 44 files. All 107 worked the same day: 102 re-audited proven, 5 justified as expected survivors."
survivor_fix_recipes:
zero_oracle_56: "Not oracle-free. UI tests whose assertions live in page objects that bind the assertion library directly. Fix: add one invertible check on the page object's public surface (exact sentinel-row count, boolean state pin), never crack open protected internals."
schema_gated_34: "A helper checks status and error shape internally through the raw library, leaving nothing in the test body to invert. Fix: one barrel-level status pin beside the helper call."
swallowed_10: "Not catch-and-launder, as labelled. Negation asymmetry: an inverted 'not visible' or 'not at URL' check returns while the page is still loading. Fix: a wait gate inversion cannot touch, then a re-assert against the settled state."
vacuous_sort: "Sort verifiers pass vacuously on a 0-row or 1-row page. Fix: a floor of at least 2 visible rows after every search that feeds a sort proof."
limits: "Cannot see weak-but-real assertions (perturbation territory), tautologies (credited as proven), data-symmetric oracles where both sort directions hold, oracles bound outside the barrel, or defect pins, which invert perversely and are excluded."
use_when: "A suite has one import choke point for its assertion library, keeps page-object and helper assertions bound separately, and needs a cheap runtime answer to whether a passing test could fail at all."
avoid_when: "Assertion imports are scattered with no single seam, the target environment is one where re-running arrange and act phases is unsafe, or you want a merge gate before any false-positive history exists."
takeaways:
- "A test that reads correctly is not the same claim as a test that can fail. Only an inverted run answers the second question."
- "Do not invert an oracle whose real weakness inversion cannot see. The schema validator's blind spot is additive drift, so it stays truthful and gets counted."
- "Get the real call-site count before you choose the mechanism. A recount from 1,070 assumed sites to 267 real ones changed which design was worth building."
- "The tool's own classifier is a bug surface. The first real finding was the falsifier misreading a held defect pin's raw attempt status as a failure."
- "A survivor label describes what the lane could see, not what the test lacks. Diagnose the blindness before you touch the file."
- "Report before you gate. The zero-oracle ratchet shipped for the API tier only, because in the UI tier it would have flagged the author's own convention."
keywords: ["can this test fail at all", "assertion inversion test falsifier", "mutation testing alternative for Playwright", "tests that pass while asserting nothing", "Playwright expect not proxy", "oracle execution counters per test"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).