A QA automation engineer's notes on the six checks a failing assertion must clear before it becomes a filed defect.
A test goes red. That is a fact. "The product is broken" is a claim, and a much bigger one than the red test justifies. I learned the difference the expensive way. I sent a report with six defect candidates. The owner pushed back hard: the report was not trustworthy, and I should work it out properly. The owner was right on four of the six. Those four were my own test's assumptions, not the product's fault. Two survived.
A failing assertion proves the app disagrees with the test. It does not prove the app disagrees with the intended behaviour. Closing that gap is a repeatable discipline. Here is the ladder I now climb before a red test becomes a filed defect. Each rung is cheaper than the one after it, and each kills a specific story about the cause. The triage taxonomy sorts a big run into buckets; this ladder is what happens after a candidate lands in the "probably a product defect" bucket.
Rung one: reproduce it outside the suite
A red assertion tells you that test code expected one thing and observed another. Separate the two failure modes first. Is the test's expectation wrong, or is the product's behaviour wrong? The cheapest way to find out is to repeat the exact call by hand, outside the test framework, and read the raw response instead of the assertion's diff.
Of my six original candidates, one was a title-matching heuristic. It assumed duplicate records always carry a "(Cloned)" suffix. They do not, on the data I had drawn that day. Another was an equivalence that only holds when an order carries a single invoice; multi-invoice orders break it by design. A third was a generated document, which read the correct customer name all along. Each needed me to read what the test asserted, then ask whether that assertion was ever a fair description of the product. The reflex worth building is to ask what data shape makes the test's assumption false, before asking what product defect makes it fail.
Rung two: probe the other environment
If the behaviour survives rung one, run the identical operation against a second, independently seeded environment. If it repeats there, you have killed a large class of "broken on X" claims.
This is where most of my false positives died. I was ready to call one rendering quirk an environment-specific display defect. It rendered identically on both environments. Only one held multi-invoice data, and that data shape exposed it. Same code, same behaviour, different data. Before you write "broken on X", attempt the call on environment Y and record what came back.
Rung three: find the control that has the trait but lacks the symptom
Here is the trap I fell into on a genuine defect candidate. Seven rows were missing from a search response for one line item. All seven were migrated records, and all seven sat under a single parent status. I wrote up the defect and named migration as the cause. The owner asked a fair question: does every migrated record fail? I had not checked.
The same search endpoint, on the same line item, returned 106 rows in total, 81 of them also migrated. They came back fine. The real criterion was the parent status alone. Migration was just something the failing rows had in common, because the seed data over-represents it.
Run both controls before you name a cause: the trait without the symptom (migrated and visible), and the symptom without the trait (missing and not migrated). If either control is non-empty, your one-variable story is wrong. Where the endpoint filters on the suspected variable, prefer a full sweep over a sample argument. I swept every value of the parent-status field and summed the counts: the fourteen per-status totals added up to 61,204, matching the endpoint's own unfiltered count.
Rung four: name the field on the wire, not the multiplier
Some defects show up as a wrong number: a total that opens 7% too high, a computed value close but not exact. The instinct is to describe the fault as a multiplier, then look for a second host that isolates the arithmetic. A ratio is derived, so many mechanisms fit it: double-applying a rate and reading the wrong rate land on almost identical percentage differences. The cheaper move is to capture the response the surface received, and look for the disputed number verbatim, as a named field.
On one pricing defect I was ready to argue that double-taxing and wrong-rate-selection were indistinguishable, and I proposed two orders in different tax jurisdictions to separate them. I needed neither. The response the form received already carried the inflated figure, 856, in a field named for the tax-inclusive total, beside the correct tax-exclusive figure of 800. The prefill was reading the wrong field, not computing the wrong number. Name that field in the report, because it is the first objection a developer raises. Keep the second-host comparison as a fallback for when nothing on the wire matches the number.
Rung five: check whether the fault reproduces one at a time
Some faults do not exist under the conditions your investigation runs in. I probed an invoicing endpoint by hand, one call at a time, six times each across three related routes: eighteen clean responses out of eighteen. About an hour later the full suite hit the same endpoints under its normal concurrency and produced 38 gateway failures, the first inside the opening six minutes.
A serial probe cannot create the condition a load-dependent fault needs, so a clean serial probe is consistent with a badly broken endpoint. The cheap tell is disjointness. If the same spec file fails on a different set of cases each run, the cause is probabilistic (contention, pool exhaustion, a gateway timeout), not a defect tied to one input. Across the two runs either side of that probe, one file's failing cases were almost fully disjoint, and the report booked a dozen recoveries beside nine regressions in that file.
Never write "recovered" or "fixed" off a serial recheck. The honest claim is "not currently failing under serial load". Report the load dependence in those terms: does not reproduce under a single serial probe, reproduces under concurrent load. Omit that and the fix team closes the ticket as "cannot reproduce".
Rung six: measure one hop upstream before you claim impact
A reproduced defect proves the endpoint is wrong, not that a real caller reaches the broken branch, and severity depends on that second question. I once filed a truncating-limit defect as high severity, reasoning that the default page size was small enough for ordinary traffic to hit it. The owner pushed back: that depends on what the real caller sends, and I had not measured it.
Measured end to end, the upstream step that feeds this endpoint returns 20 ids, resolving to only 9 rows. Nine rows cannot overflow a page of 20, because the response holds one row per resolved id. Truncation was impossible. My test fixture pinned 40 ids, sized for coverage and twice anything a real caller sends. The defect was reachable only because my own probe used an unrealistically large input.
A second lesson sits inside it. When the real caller finally shipped, I captured its request off the wire: it sends a page size of 50, not the default 20 I had assumed. The verdict held; the premise was wrong. An inferred caller value is still an inference inside a correct conclusion, so say which numbers you measured and which you assumed. Keep the pin, because a latent defect goes red the moment its trigger lands, and label it with the change that would make it live.
What a report looks like once it clears the ladder
The ladder is not only a filter. Each rung leaves an artifact behind, so it also dictates the shape of the write-up. Filed defects now carry the same headings:
TITLE one sentence, naming the criterion I proved, not the one I suspected
SUMMARY what happens, and what it costs
STEPS every value captured, with the date and the environment
CONTROL the near-identical case that behaves correctly (rungs two and three)
DISCRIMINATOR the sibling paths that already do the right thing
IMPACT measured rather than inferred (rung six), with the measurement shown
CAVEAT the reading of the spec under which this is not a defect
WHY NOT DESIGN why the documented behaviour does not cover this
GUARD the pinned spec that flips green the day it is fixed
Two headings do disproportionate work. The caveat states, in my own words, the reading of the spec under which the fix team is right and I am wrong. Finding that reading is the work; offering it first is what makes the rest credible. The guard ties the claim to a pinned failing spec, so the defect stays observable instead of becoming a paragraph nobody re-runs.
Most candidates die on the ladder, and that is the point
Six candidates went up the first two rungs in one batch. Four died there, all test-side assumptions. Two survived and shipped as filed defects. The upper rungs came later, and each retired a claim I had been confident about: migration as the cause, double-taxing as the mechanism, "recovered" as a verdict, and high severity on a defect no caller could reach.
That is not a bad hit rate. It is the ladder working as designed. A report that clears all six rungs is a claim I can defend line by line, because at each step I asked what would make my own assumption false, then checked.
Takeaways
- A red assertion is evidence of a mismatch with the test, not proof of a product defect. Reproduce by hand and read the raw response first.
- Probe the second environment before you write "broken on X". The difference is usually the data shape, not the code.
- A trait shared by every failing row is a correlate of your sample. Find the control that has the trait but lacks the symptom, then sweep until the parts sum to the unfiltered whole.
- A ratio fits several mechanisms; a named field on the wire fits one. Capture what the surface received before proposing a second host.
- Serial probes cannot see load-dependent faults. If failing cases trade places across runs of one file, report "does not reproduce serially, reproduces under concurrency" instead of "fixed".
- A reproduced defect proves behaviour, not impact. Measure the upstream volume and the real caller payload before assigning severity, and say which numbers you measured.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/13/sure-enough-to-file-a-bug/"
license: "Free to reference with attribution"
title: "Sure enough to file a bug: the evidence ladder before you blame the product"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 53"
stack: [Playwright, TypeScript, REST API, HTTP wire capture]
problem: "A batch of six defect candidates was promoted straight from failing spec assertions to a filed report. The owner rejected it and was right on four of the six: a title heuristic that assumed a '(Cloned)' suffix, an equivalence that only held on single-invoice orders, a generated document that in fact carried the correct customer name, and a worker-isolation assumption that only held on one environment's data."
thesis: "A failing assertion proves the app disagrees with the test, not with the intended behaviour. Climb a six-rung evidence ladder, cheapest rung first, before a red test becomes a filed defect. Each rung eliminates one specific story about the cause."
ladder:
rung_1_reproduce_outside_the_suite: "Repeat the exact call by hand and read the raw response, not the assertion diff. Ask 'what data shape makes my test's assumption false?' before 'what product defect makes this fail?'."
rung_2_second_environment: "Run the identical operation against an independently seeded second environment. Same behaviour on both kills the 'broken on X' claim. A rendering quirk read as environment-specific rendered identically on both; only one environment held multi-invoice data."
rung_3_controls_and_sweep: "Seven rows missing from a search response were all migrated and all under one parent status; the same endpoint returned 106 rows for that line item, 81 of them migrated and visible. The criterion was the parent status alone. Run both controls (trait without symptom, symptom without trait) and prefer a population sweep: the 14 per-status counts summed to 61,204, exactly the unfiltered total."
rung_4_field_not_ratio: "A ratio is derived and fits many mechanisms; a named field on the wire fits one. A total opening 7% high was not arithmetic: the response the form received carried 856 in the tax-inclusive field beside the correct 800 tax-exclusive figure. The prefill selected the wrong field."
rung_5_serial_vs_concurrent: "18 of 18 clean serial responses across three routes probed 6 times each, then 38 gateway failures under the full suite an hour later, starting inside the first six minutes. Disjointness is the cheap proof of intermittency: one file's failing cases were almost fully disjoint across two runs, booking 12 recoveries beside 9 regressions."
rung_6_measure_the_caller: "An endpoint probe proves behaviour, never impact. A truncating-limit defect filed as high severity was unreachable: the upstream step returns 20 ids resolving to 9 rows against a page of 20. The 40-id test fixture was twice any real payload. When the caller shipped it sent a page size of 50, not the assumed default of 20."
report_shape:
headings: "TITLE (the criterion proved, not suspected), SUMMARY, STEPS (every value, dated, per environment), CONTROL, DISCRIMINATOR, IMPACT measured rather than inferred, CAVEAT, WHY NOT DESIGN, GUARD."
caveat_section: "States the spec reading under which the fix team is right and the reporter is wrong. Finding it is the work; offering it first is what makes the rest credible."
guard_section: "Ties the claim to a pinned failing spec so the defect stays observable and flips green on the day it is fixed."
precision_limits: "A filter value returning zero rows is only proven dropped if another route shows it holds data. Label a latent defect with the trigger that would make it live."
outcomes:
first_batch: "6 candidates, 4 retracted on rungs one and two, 2 filed."
later_retractions: "Rungs three to six each retired a separate later claim: migration as the cause, double-taxing as the mechanism, 'recovered' as a verdict, and high severity on an unreachable defect."
use_when: "Promoting a failing spec assertion, a triage bucket entry, or an agent-generated finding into a filed defect against a product team you do not own, especially on a shared multi-environment platform where test data shape varies by environment."
avoid_when: "Hermetic unit tests over code you own, where the assertion and the implementation share one author and one commit, and the cost of an incorrect defect report is zero."
takeaways:
- "A red assertion is evidence of a mismatch with the test, not proof of a product defect. Reproduce by hand and read the raw response first."
- "Probe the second environment before writing 'broken on X'. The difference is usually the data shape, not the code."
- "A trait shared by every failing row is a correlate of the sample. Find the control that has the trait but lacks the symptom, and sweep until the parts sum to the whole."
- "A ratio fits several mechanisms; a named field on the wire fits one. Capture what the surface received before proposing a second host."
- "Serial probes cannot see load-dependent faults. Report 'does not reproduce serially, reproduces under concurrency' rather than 'fixed'."
- "A reproduced defect proves behaviour, not impact. Measure the upstream volume and the real caller payload before assigning severity, and say which numbers were measured and which assumed."
keywords: ["evidence before filing a bug report", "is it a test bug or a product bug", "control and discriminator in a defect report", "serial probe versus load-dependent 5xx", "measure the caller before claiming severity", "correlation in the failing sample is not the criterion"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).