Counting your oracles instead of mutating them

A QA automation engineer's notes on a runtime counter that finds the tests which pass without executing a single assertion.

The test-the-tests lane started with an ambitious idea. Invert every assertion in the suite, run it again, and see which tests still pass. That is mutation testing's oldest trick, and it works, but it is expensive to run and expensive to read: roughly 9,400 assertion call sites have to go through a fake failure, and then you have to separate real gaps from schema helpers behaving exactly as designed. Partway through building that falsifier I shipped a much cheaper cousin, almost as an afterthought. It turned out to be the more useful of the two. It just counts.

The question that does not need inversion

Here is a question much smaller than "can this test fail at all". Did this test, on the path it actually executed, run any assertion at all?

That needs no second run, no flipped matcher, and no diff. It needs a counter that increments every time an oracle fires, attached to the test currently running and read back when that test finishes. I use "oracle" for anything that can turn a test red. Nothing here simulates failure. It is bookkeeping on the happy path the suite already ran, at a wall-clock cost that stayed inside run-to-run noise on the canary.

A green test with zero executed oracles is a specific, nameable failure mode: an early return, a swallowed catch, or a branch that never fires against real data skipped every assertion on the way through. Until the counter existed, the only way to find one was to read every spec by hand.

Where the seam has to live

The trick is to find the one place every assertion already passes through. Here that place is the fixtures barrel: a convention guard bans a direct test-framework import inside a spec, so specs take expect from one module instead. Of 566 spec files, 564 do, and the three that import it raw are meta-tests of the guards themselves. That import point is the seam.

const countedExpect = new Proxy(baseExpect, {
  apply(target, thisArg, args) {
    recordOracle('expect');
    return Reflect.apply(target, thisArg, args);
  },
});

Every expect(value) call reaches the real implementation through the apply trap. The count happens once per call, and behaviour stays untouched: same matcher, same error class, same message, same typing.

The first version stopped there, and that was a bug. expect.soft and expect.poll are oracle entry points in their own right: a failed soft assertion reds the test at the end, and a polled one rejects on timeout. The suite has 11 soft sites and 8 polled ones, and one spec's entire oracle set is soft. Under the falsifier, a soft-only test that correctly failed read as "died before any oracle ran". So the proxy grew a get trap over those two properties, counting at the call and never at the property read: soft is a getter that mints a fresh matcher object on every access, so a read-time counter would tally accesses instead of assertions.

What the seam leaves alone matters as much as what it catches. An asymmetric matcher used as an argument, as in expect(x).toEqual(expect.objectContaining({...})), is a value being constructed rather than an oracle firing. Count where something could fail the test, not at every touch of the expect namespace.

Widening past the one keyword

The proxy does not catch the helper oracles: the suite's wrappers for schema validation, Ajv reason pinning, rejection assertions, and ordering checks. The reason is not the obvious one. Avoiding expect is only what the schema validator does, and it throws a plain Error. The other three call expect, but import it from the test framework instead of from the barrel. The load-bearing property is the import source.

The fix is not a cleverer proxy. It is instrumentation on entry:

export function assertValid<T>(payload: unknown, schema: JsonSchema): asserts payload is T {
  recordOracle('schema'); // ON ENTRY: a failing schema gate still counts as an executed oracle
  // ajv validation, throws on failure
}

That brings the tally to seven kinds: three from the barrel (expect, soft, poll) and four helper families instrumented by hand (schema, ajv-reason, rejection, ordering). Every kind is zero-filled and never sparse, so a report written before a kind existed still parses, reading zero for what it never had.

For the first five days that seam was undefended. A tidy-up could move one helper onto the barrel's counted expect, the meaning of the tally would shift underneath it, and nothing would notice. I recorded the fragility and shipped anyway. Then a review pushed back, and the fence turned out to be small: an AST guard pinning the import source of five named helper files, namespace imports and re-exports included. A missing file counts as a violation of its own, so a rename cannot silently empty the fence. It lands at zero hits on the live suite.

Where the count goes to survive

None of this matters if the count evaporates at the end of the test. The counter lives in worker-process memory, keyed by the test's own id. A worker runs one test at a time, so a single "current test" slot is unambiguous.

The counting module deliberately does not import the test framework. Bare tsx jobs load the schema validator outside the runner, and a transitive framework import there would drag the runner into a plain Node process. So an auto fixture pushes the test identity in, zeroes a tally before the body runs, drains it afterwards, and writes the result into the test's annotations as a small JSON blob. Annotations, not attachments: they land inline in the JSON report the runner already produces.

Two details earn their keep. recordOracle sits on roughly 9,400 assertion paths, so every mutation is inside a swallowing try/catch: a counter that turned a passing assertion into a failure would be a worse bug than the one it was built to find. And starting a capture re-zeros rather than resumes: a retry re-runs with the same test id in the same worker, and must report its own oracles.

The annotation then gets pushed unconditionally, all-zero tallies included. Skipping the write when the count is zero is tempting, because zero is the boring case. But that ambiguity is the distinction the whole feature exists to preserve: a missing annotation has to mean "never instrumented", not "ran and asserted nothing".

Zero and missing are not the same finding

The report keeps them as two separate buckets. Conflating them was my first mistake in an early draft.

ZERO means the test passed, its annotation is present, and every kind in it reads zero. The instrumented path ran and asserted nothing: the test had every opportunity to fail and never took it.

MISSING means there is no oracle annotation at all. That is evidence the test never went through the counted seam, not evidence that it asserted nothing. A whole legitimate population lives in that bucket: page-object-bound UI specs keep their assertions inside page object methods, and those methods bind the raw framework expect. Across 83 page object files there are 444 expect( sites, and not one imports from the barrel. Lumping those in with ZERO would flood the report with expected members, and everyone would learn to ignore it inside a week.

What counting can never tell you

Counting oracles proves an assertion executed. It proves nothing about whether that assertion could ever have failed. I had been burned by exactly that gap twice in one afternoon of draining the blind-oracle backlog.

The first case was a non-vacuity guard on a results grid, written as "the row locator does not have a count of zero". It runs, it increments the counter, and it passes. Angular Material renders an empty result set as a row: a single-cell empty-state placeholder. So the guard passed on a search that matched nothing, and the spec reported green for weeks while proving the opposite of its claim.

The second case sorted a grid by order stage and used "more than one row" as its non-vacuity floor. That grid rendered seven rows, all carrying the same stage. Ties are legitimately allowed, so both directions were satisfied and a deliberate inversion of the sort still passed. The floor was sized to the row count when the claim needed distinct values.

Both assertions executed. Both are vacuous.

Counting catches "nothing ran", which a patient reviewer would eventually catch too, just more slowly. It does not catch "something ran but could not have told you anything". That needs an inversion, or a distinct-value floor sized to the actual claim. Counting is the cheap, always-on smoke detector; inversion is the expensive, targeted check on whether the detector is wired to anything. The false-pass catalog covers the shapes themselves.

Why I shipped a report and not a gate

The obvious next move for a metric like this is to fail the build on it. I calibrated first, over five test folders and about 300 classified tests. Every non-setup zero-oracle row was page-object delegation, 42 times out of 42, and across roughly 250 API-tier rows there were zero hits.

So a blanket ratchet would have fired almost entirely on page-object delegation, a documented convention rather than a bug, and a gate that mostly flags its own remedy trains people to ignore it. MISSING stays observational for the same reason: it fires on every report predating the counters and on every setup-only project by design, and a gate that fires on history gets muted, not fixed. The one exception is a stricter API-tier mode, where a present, all-zero tally is calibrated to zero hits and exits non-zero.

The counter therefore earns its keep as a review queue with receipts, not a verdict. It names which tests to look at, and why each one qualified. What you do with that list is the judgement call a careful reviewer makes reading the file directly, except the list arrives automatically, on every run.

Takeaways

• Counting which assertions executed is cheaper than mutating them: one proxy at the seam every assertion already passes through, plus entry-point instrumentation on the helpers that bypass it.
• Count at the call, not at the property read. A getter that mints a fresh object per access would tally accesses instead of assertions.
• Keep ZERO and MISSING as separate findings: "ran and asserted nothing" is not "never went through the counted path at all".
• Push the annotation unconditionally, all-zero tallies included. An optional annotation reintroduces the ambiguity the counter was built to remove.
• Calibrate before you gate. A blanket ratchet on a metric with a legitimate zero population teaches people to ignore the gate; a narrower, calibrated exception earns trust.
• An executed oracle is not a meaningful one. A row-count floor passes on an empty-state placeholder, and a sort assertion passes on a page with one repeated value.
• Fence any assertion path that bypasses the counting seam. Nothing else catches a tally that quietly changes meaning.

---

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/04/counting-oracles/"
license: "Free to reference with attribution"
title: "Counting your oracles instead of mutating them"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 44"
stack: [TypeScript, Playwright, "JavaScript Proxy", Ajv, Node]

problem: "A test can pass having executed no assertion at all: an early return, a swallowed catch, or a branch that never fires against real data skips every oracle on the path actually taken. Static guards cannot see it, because the assertion is written in the file but never reached. Reading every spec by hand was the only detection method."
thesis: "Count executed oracles instead of inverting them. A counting proxy at the single import seam every assertion already passes through, plus entry-point instrumentation on the helpers that bypass that seam, makes 'passed with zero executed oracles' visible in every ordinary run, with no second run and a wall-clock cost inside run-to-run noise."

mechanism:
  seam: "The fixtures barrel. A convention guard bans a direct test-framework import in a spec, so 564 of 566 spec files take expect from one module (the 3 raw importers are meta-tests of the guards). That import point carries the counting Proxy, whose apply trap records one 'expect' oracle then delegates via Reflect.apply, leaving matcher, error class, message and typing byte-for-byte the framework's."
  expect_family: "A get trap covers exactly expect.soft (11 sites) and expect.poll (8 sites), added after a review: both are real oracle entry points, since a failed soft reds the test at the end and a polled one rejects on timeout. Counting is at the CALL, never the property read, because soft is a getter minting a fresh matcher per access and a stored `const p = expect.poll` would count an oracle that never ran. Wrappers are memoised against the value read."
  deliberately_uncounted: "Every other static passes through. An asymmetric matcher used as an ARGUMENT (expect(x).toEqual(expect.objectContaining({…}))) is a value being constructed, not an oracle firing; likewise extend, configure and internal-state reads. Counting them would inflate every tally."
  helper_oracles: "Four helper families are invisible to the proxy and record their own kind ON ENTRY, so a failing oracle still counts as executed: schema (assertValid/assertValidExcept), ajv-reason, rejection, ordering. They are invisible not because they avoid expect (only the schema validator does, throwing a plain Error) but because they import expect raw from the test framework rather than from the barrel: the IMPORT SOURCE is load-bearing. Seven kinds total, always zero-filled and never sparse, so an older report still parses with a new kind reading 0, while a payload carrying none of the known kinds reads as no evidence rather than as evidence of zero."
  persistence: "Worker-process memory keyed by the test id, since a worker runs one test at a time. The counting module must not import the test framework, because bare tsx jobs load the schema validator outside the runner, so an auto fixture pushes the test identity in rather than resolving it from the runner. That fixture zeroes a tally before the body, drains it after, and writes it as a JSON annotation (not an attachment) so it rides inline in the runner's JSON report."

non_obvious_decisions:
  - "recordOracle must never throw: it sits on roughly 9,400 API and UI assertion paths, so every mutation is inside a swallowing try/catch and it is a silent no-op with no active capture (worker-scoped discovery, tsx jobs, module load, setup-only projects)."
  - "Beginning a capture re-zeros rather than resumes, because a retry re-runs with the same test id in the same worker and must report its own oracles. The fixture is also declared FIRST so its drain is the LAST teardown step, since auto fixtures tear down in reverse declaration order and an annotation on every test would otherwise make the suite's annotation-gated result logger print for every test."
  - "The annotation is pushed unconditionally, all-zero included. A missing annotation must mean 'never instrumented', never 'ran and asserted nothing'."
  - "The helper import seam shipped undefended for five days, then got an AST guard pinning the import source of five named helper files. It bans the barrel import including namespace imports and re-exports, treats a missing file as its own violation so a rename cannot empty the fence, and calibrated at zero hits."

report_semantics:
  zero: "Passed, annotation present, every kind zero. The instrumented path ran and asserted nothing."
  missing: "No oracle annotation at all. Evidence the test never went through the counted seam, not evidence it asserted nothing. Legitimate population: page-object-bound UI specs, 83 page object files with 444 expect( sites and zero barrel imports."
  gating: "Calibrated over five test folders and about 300 classified tests: every non-setup zero-oracle row was page-object delegation (42 of 42), and roughly 250 API-tier rows produced zero hits. So the report is observational by default and always exits 0, with one exception: a stricter API-tier mode where a present all-zero tally exits non-zero. A blanket ratchet would fire almost entirely on the page-object delegation convention, making the gate flag its own remedy, and MISSING stays observational because it fires on pre-counter reports and setup-only projects, where a gate that fires on history gets muted rather than fixed."

limits:
  ceiling: "Counting proves an oracle executed, never that it could have failed. It catches 'nothing ran'; it cannot catch 'something ran but could not have told you anything'. The complement is assertion inversion, or a distinct-value floor sized to the claim: counting is the cheap always-on smoke detector, inversion the expensive targeted check on whether the detector is wired to anything."
  vacuity_war_stories:
    - "A non-vacuity guard asserting the row locator does not have a count of zero, on a grid where Angular Material renders an empty result set as a single-cell empty-state placeholder row. Green for weeks against a table holding only the placeholder."
    - "An ordering assertion with a floor of more than one row, on a grid rendering seven rows all carrying the same order stage. Ties are allowed, so both directions were satisfied and a deliberate inversion of the sort still passed. The floor was sized to row count when the claim needed distinct values."

use_when: "A suite has one import choke point for its assertion library (or can be given one), runs against a live shared environment where branches silently do not fire, and needs a per-run signal for tests that go green without executing an oracle."
avoid_when: "Assertions arrive through many uncontrolled import paths, or the real question is whether an executed assertion is strong enough, which needs mutation or inversion rather than counting."
takeaways:
  - "Counting which assertions executed is cheaper than mutating them: one proxy at the seam every assertion already passes through, plus entry-point instrumentation on the helpers that bypass it."
  - "Count at the call, not at the property read. A getter that mints a fresh object per access would tally accesses instead of assertions."
  - "Keep ZERO and MISSING as separate findings: 'ran and asserted nothing' is not 'never went through the counted path at all'."
  - "Push the annotation unconditionally, all-zero tallies included. An optional annotation reintroduces the ambiguity the counter was built to remove."
  - "Calibrate before you gate. A blanket ratchet on a metric with a legitimate zero population teaches people to ignore the gate; a narrower, calibrated exception earns trust."
  - "An executed oracle is not a meaningful one. A row-count floor passes on an empty-state placeholder, and a sort assertion passes on a page with one repeated value."
  - "Fence any assertion path that bypasses the counting seam. Nothing else catches a tally that quietly changes meaning."
keywords: ["detect tests that pass without asserting anything", "count executed assertions in Playwright", "JavaScript Proxy around expect to count oracles", "zero-oracle passing test report", "vacuous test detection without mutation testing", "instrumenting expect.soft and expect.poll"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).