Don’t trust “Done”: AI agents that validate tickets against the live app

A QA automation engineer's notes on making "Done" mean something by grounding every acceptance-criteria clause in a live probe against the running app.

Here is the failure mode I kept hitting. A ticket marked "Done" in the tracker, a matching test case in the test-case management tool marked "Pass", and a clause in the acceptance criteria that the app simply did not satisfy. Not a subtle edge case. A core behaviour that had slipped between a backend refactor and a tracker update that no one thought to verify.

The usual fix is "do better manual verification." I tried something else: make "Done" mean something by grounding every acceptance-criteria clause in a live probe.

The grounding problem

An LLM is very good at reasoning about a ticket. Give it the acceptance criteria and the OpenAPI spec and it will produce a confident, fluent assessment. That assessment is also, frequently, wrong, because neither the ticket nor the spec is the ground truth. The ground truth is the running application.

"Reason about whether the app probably satisfies clause 3" is the weakest grounding I can give it. An agent that reads a ticket and outputs PASS without touching the network is a very confident guesser. The validation flow does not guess.

For every ticket I validate, the agent pulls the acceptance criteria from a local snapshot of the tracker, parses each clause into a verifiable predicate, runs a typed API call or a headless browser interaction (or both) against the real DEV environment, and classifies the result from observed evidence rather than inference.

The output is not a probability. It is a verdict with a receipt.

Parsing acceptance criteria into probes

Acceptance criteria are written for humans. Clauses like "the renewal status updates to Confirmed on save" or "the billing total reflects the correct line item count" are natural language first, predicates second. The agent's first pass converts each clause into a structured probe target:

{
  "clause": "billing total reflects the correct line item count",
  "probeType": "api",
  "endpoint": "GET /orders/{id}/billing",
  "assertion": "body.summary.lineItemCount === resolvedOrder.lineItems.length",
  "uiConfirm": false
}

That intermediate representation is what makes the rest reliable: the agent executes against a declared target that a human can inspect. The probe list is logged alongside the verdict, so a reviewer can see not just what I concluded but exactly what I asked the app.

When a clause is inherently vague ("the user experience is consistent"), the agent marks it UNTESTABLE. That is an honest classification. A fabricated pass is not.

Four verdicts, three root causes

What separates this flow from a simple pass/fail check is the attribution step. For any clause that does not pass, the agent runs a second probe sequence to identify which layer is responsible.

The four possible verdicts are:

• MATCHES: the observed behaviour satisfies the clause.
• DEFECT-API: the API response is wrong, and the UI faithfully renders the wrong data.
• DEFECT-UI: the API response is correct, but the UI displays something different.
• NOT-DEPLOYED: the endpoint returns 404 or the feature flag is inactive. The code may exist but is not yet reachable.
• DEFECT-Spec: the OpenAPI spec disagrees with the ticket's stated intent. A note to the spec owner, not a test action.

The three defect cases look identical in a bug report written from a screenshot. They lead to completely different next steps.

Using the network tab to attribute

Attribution is not intuition. It is a two-layer probe. The agent makes the API call independently of the browser and records the raw response. Then it drives a headless browser to the same state and records what the page renders. If the API returned status: "confirmed" and the page shows "Pending", that is a UI defect: the data is correct, the render is not. If the API returned status: "pending" and the page faithfully shows "Pending", that is an API defect: the render is fine, the backend is wrong.

This sounds obvious in the abstract. It is not obvious when you are first looking at a bug report and someone needs a classification in the next hour.

The agent automates the attribution mechanically. Two runs of the same clause, different outcomes:

# Run A
API probe  → 200  { status: "pending" }
UI render  →      "Status: Pending"
→ DEFECT-API  (API wrong; UI faithfully reflects it)

# Run B
API probe  → 200  { status: "confirmed" }
UI render  →      "Status: Pending"
→ DEFECT-UI  (API correct; UI does not reflect it)

Those two verdicts produce different artifacts. A DEFECT-API verdict opens a backend defect ticket and generates a *.defect.api.spec.ts, a test.fail() that flips to green the moment the backend ships the fix. A DEFECT-UI verdict opens a frontend defect ticket and pins a UI spec instead. A NOT-DEPLOYED verdict generates a deployment-guarded test.skip noting what needs to be in place. I never ship a false-pass, and I never drop coverage just because the feature is not yet live.

The spec disagreement edge case

The fifth verdict, DEFECT-Spec, deserves its own note. There is an invariant I hold: OpenAPI specs are read-only. They are the contract the backend team owns. My job is to test the app against the spec, never to revise the spec to match what the app happens to do.

Occasionally, a ticket's acceptance criteria describe behaviour that contradicts the spec. The product manager wrote the ticket from an earlier design, the spec reflects a later decision, and no one updated the ticket. When the agent detects this, the verdict is not a pass or a fail. It is a note: "the observed behaviour matches the spec; the ticket's clause describes something the spec does not require." That goes to the spec owner and the ticket author to resolve. I don't adjust the spec to paper over the disagreement, and I don't write a test that enshrines the ticket's intent against the spec's authority.

Without a live API probe, there is no way to tell which of "ticket intent" and "spec definition" the app actually implements. The attribution step is what makes DEFECT-Spec a reliable classification rather than a guess.

What the output looks like in practice

A single validation run for a mid-size ticket, say 8 acceptance criteria clauses, produces:

• A structured verdict table, one row per clause, with probe type, raw evidence, and classification.
• Generated test stubs, each pre-populated with the correct verdict encoding (test.fail, test.skip, or a passing assertion).
• A brief summary for the issue tracker: how many clauses passed, which ones need action, and what kind.

The agent does not close the ticket. A human reads the table, accepts or disputes the classifications, and makes the call. About 85% of MATCHES verdicts need no review, since the probe is deterministic and the clause is unambiguous. Defect attributions always get a human glance before the generated spec lands in the repo.

Where it falls down

Live probes are not free. A ticket with 12 clauses, some API, some UI, some both, drives 20-30 HTTP requests and several headless browser interactions. On a busy DEV environment that can take 45-90 seconds per ticket. I batch validation runs for related tickets at the start of a sprint rather than running them continuously.

Attribution also fails at the seams. If the UI makes a second API call to transform the first response, the agent may attribute a defect to the wrong layer. I ran into this once: a status display was wrong, the primary API was correct, but a second call for "display configuration" was returning stale data. The two-probe model handles one API call. Composite UI state that folds multiple calls needs more instrumentation than a headless browser alone provides.

And verdict confidence tracks clause precision. "Users with read-only access cannot edit" is testable. "The interface feels responsive" is not. The UNTESTABLE bucket is the honest acknowledgement that some acceptance criteria were never meant to be automated.

Takeaways

• AI verdicts are only as trustworthy as their grounding. "Reason about the ticket" is the weakest grounding available; "probe the live app and classify the evidence" is the strongest.
• Attribution is what turns a "failure" into a useful artifact. The same wrong screen leads to a backend defect spec, a frontend defect spec, or a deployment-guarded skip depending on which layer is responsible, and you cannot tell which without checking both layers independently.
• Read-only specs are a load-bearing constraint. When the ticket and the spec disagree, that is a DEFECT-Spec note for the spec owner, never a reason to adjust the contract to match the ticket.
• Build the intermediate representation. Turning acceptance criteria into a structured probe list before running any assertions makes the validation inspectable, loggable, and reproducible.
• Honest verdicts beat fabricated passes. UNTESTABLE, NOT-DEPLOYED, and DEFECT-Spec are real outcomes, not failures of the validation process. A suite that only knows how to say "pass" is a suite that will eventually lie to you.

---

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/27/dont-trust-done-live-validation/"
license: "Free to reference with attribution"
title: "Don't trust \"Done\": AI agents that validate tickets against the live app"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 37"
stack: [TypeScript, Playwright, "headless browser", "REST API", OpenAPI, "LLM agent"]

problem: "Tickets marked \"Done\" in the tracker, with a matching test case marked \"Pass\", still ship clauses the app does not satisfy — a core behaviour slipped between a backend refactor and a tracker update no one verified. Neither the ticket nor the OpenAPI spec is ground truth; the running app is."
thesis: "Make \"Done\" mean something by grounding every acceptance-criteria clause in a live probe. The agent parses each clause into a verifiable predicate, runs a typed API call and/or headless-browser interaction against the real DEV environment, and classifies from observed evidence — a verdict with a receipt, never a probability from inference."

probe_pipeline:
  flow: "Per ticket: pull acceptance criteria from a local snapshot of the tracker → parse each clause into a structured probe target → execute against real DEV → classify from observed evidence."
  intermediate_representation: "Each natural-language clause becomes a JSON probe target with fields: clause, probeType (api/ui/both), endpoint (e.g. GET /orders/{id}/billing), assertion (e.g. body.summary.lineItemCount === resolvedOrder.lineItems.length), uiConfirm (bool). This IR is what makes the rest reliable and human-inspectable."
  logging: "The probe list is logged alongside the verdict so a reviewer sees exactly what was asked of the app, not just the conclusion."
  untestable: "An inherently vague clause (\"the user experience is consistent\", \"the interface feels responsive\") is marked UNTESTABLE — an honest classification, not a fabricated pass. Verdict confidence tracks clause precision: \"users with read-only access cannot edit\" is testable; \"feels responsive\" is not."

verdicts_and_attribution:
  five_verdicts:
    - "MATCHES: observed behaviour satisfies the clause."
    - "DEFECT-API: API response is wrong and the UI faithfully renders the wrong data → opens a backend defect ticket."
    - "DEFECT-UI: API response is correct but the UI displays something different → opens a frontend defect ticket."
    - "NOT-DEPLOYED: endpoint returns 404 or the feature flag is inactive; the code may exist but is not yet reachable."
    - "DEFECT-Spec: the OpenAPI spec disagrees with the ticket's stated intent — a note to the spec owner, not a test action."
  attribution_mechanism: "Two-layer probe, not intuition. The agent calls the API independently of the browser and records the raw response, then drives a headless browser to the same state and records what renders. API \"pending\" + page shows \"Pending\" → DEFECT-API (backend wrong, render faithful). API \"confirmed\" + page shows \"Pending\" → DEFECT-UI (data correct, render wrong). The three defect cases look identical in a screenshot bug report but lead to completely different next steps."
  generated_artifacts:
    - "DEFECT-API → a *.defect.api.spec.ts containing a test.fail() that flips green the moment the backend ships the fix."
    - "DEFECT-UI → a pinned UI spec instead."
    - "NOT-DEPLOYED → a deployment-guarded test.skip noting what must be in place. Never ship a false-pass, never drop coverage just because a feature is not yet live."

spec_invariant:
  rule: "OpenAPI specs are read-only — the contract the backend team owns. Test the app against the spec, never revise the spec to match what the app happens to do."
  defect_spec_case: "When a ticket's acceptance criteria contradict the spec (PM wrote the ticket from an earlier design; spec reflects a later decision; ticket never updated), the verdict is a note — \"observed behaviour matches the spec; the ticket's clause describes something the spec does not require\" — routed to the spec owner and ticket author. The agent does not adjust the spec to paper over the disagreement, nor write a test enshrining ticket intent against the spec's authority. Without a live API probe there is no way to tell which of ticket-intent vs spec-definition the app actually implements."

output_and_limits:
  per_run: "A mid-size ticket (~8 clauses) produces a structured verdict table (one row per clause: probe type, raw evidence, classification), generated test stubs pre-populated with the correct verdict encoding (test.fail / test.skip / passing assertion), and a brief tracker summary of passes and required actions."
  human_in_loop: "The agent does not close the ticket. ~85% of MATCHES verdicts need no review (deterministic probe, unambiguous clause); defect attributions always get a human glance before the generated spec lands in the repo."
  cost: "A 12-clause ticket drives 20–30 HTTP requests plus several headless-browser interactions; 45–90 seconds per ticket on a busy DEV environment. Validation runs are batched for related tickets at sprint start, not run continuously."
  failure_at_the_seams: "The two-probe model handles one API call. If the UI makes a second call to transform the first response, attribution can blame the wrong layer — hit once when a status display was wrong, the primary API correct, but a second \"display configuration\" call returned stale data. Composite UI state folding multiple calls needs more instrumentation than a headless browser alone provides."

use_when: "Validating that tickets marked \"Done\" truly satisfy their acceptance criteria against a live DEV environment, where backend refactors and tracker updates drift out of sync and a wrong screen could be a backend, frontend, deployment, or spec issue."
avoid_when: "Acceptance criteria are inherently subjective (UX feel) and cannot be reduced to a verifiable predicate, or when no live environment exists to probe and you would be back to reasoning over the ticket alone."

takeaways:
  - "AI verdicts are only as trustworthy as their grounding. \"Reason about the ticket\" is the weakest grounding; \"probe the live app and classify the evidence\" is the strongest."
  - "Attribution turns a \"failure\" into a useful artifact. The same wrong screen leads to a backend defect spec, a frontend defect spec, or a deployment-guarded skip depending on which layer is responsible — and you cannot tell which without checking both layers independently."
  - "Read-only specs are a load-bearing constraint. When the ticket and the spec disagree, that is a DEFECT-Spec note for the spec owner, never a reason to adjust the contract to match the ticket."
  - "Build the intermediate representation. Turning acceptance criteria into a structured probe list before running any assertions makes the validation inspectable, loggable, and reproducible."
  - "Honest verdicts beat fabricated passes. UNTESTABLE, NOT-DEPLOYED, and DEFECT-Spec are real outcomes, not failures of the validation process. A suite that only knows how to say \"pass\" is a suite that will eventually lie to you."

keywords: ["validate tickets against the live app with an AI agent", "attributing a UI defect vs API defect from the network tab", "grounding LLM acceptance-criteria checks in live probes", "test.fail and test.skip generated from defect attribution", "DEFECT-Spec when ticket and OpenAPI spec disagree"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).