The unflashy truth about AI in testing: the agent explores, the code decides

A QA automation engineer's notes on why AI-testing magic fails on a real enterprise app, and the boring pattern that works.

The demo is genuinely magnetic. Someone renames a button's ID, the test goes red, and a heartbeat later it's green again. The tool "healed" the broken locator on its own. Nobody touched the test. I watched that demo and, for about a minute, I wanted to believe it.

Then I pictured the version that never makes the slide deck. The Submit order button doesn't get renamed, it disappears, because a regression broke it. The self-healing engine goes looking for the nearest match and binds to a different button that happens to look similar. The test goes green. The broken order submission ships. The one test whose entire job was to catch that regression just voted to approve it.

That gap between the demo and the Tuesday is what this half of the series is about.

Posts 22 through 41 document the AI I actually run against the test suite for a B2B order/contract-management platform: agent workflows that maintain specs, validate tickets against the live app, fan out browser checks across hundreds of manual cases, budget an agent's memory across sessions. None of it looks like the demo. This post is the frame for all of it: the promises, why they don't survive contact with a real enterprise app, and the boring pattern that does.

The four promises

The pitch comes in a few flavors, and each one is seductive for a real reason. Every one targets a genuine pain.

• Self-healing tests. "Stop fixing broken selectors. Tests adapt to UI changes automatically." The pain it targets, maintenance eating 60 to 80% of automation effort, is absolutely real.
• Plain-English authoring. "Write tests in plain English. No code, no selectors. Anyone on the team can do it." The promise is QA without the engineering bottleneck.
• Autonomous exploration. "Point the agent at your product. It explores the app, finds the critical flows, and writes the tests for you." No scripts to author, no maintenance to own.
• Generation from requirements. "Feed it the ticket, the PRD, the design file. Get a test suite back." Test design at machine speed.

Strip the adjectives and they share one shape: the absence promise. No code. No selectors. No maintenance. No humans in the loop. Each one offers to delete a step you currently have to do.

Why it stops scaling

The demos quietly depend on a small, clean, well-behaved app, and a definition of "working" that the tool gets to supply itself. Remove either and the whole category starts to wobble.

The oracle problem is the deep one. An AI can drive an application, click, type, navigate, far better than it can decide whether what it saw was correct. Correctness isn't in the page; it's in someone's intent. So when a tool "generates the assertions for you" by observing the app, it isn't testing the app, it's transcribing it. It records today's behavior as the expected behavior. If today's behavior is a bug, you've just minted a test that will defend that bug forever. You haven't automated testing; you've automated confirming your own mistakes, faster than ever.

Self-healing masks the exact thing you built the test to catch. Healing works by similarity: which element on the page looks most like the one this test used last time? That is precisely backwards on a real app, where a broken control and a working one can look nearly identical. The heal succeeds, the test passes, and the regression walks straight through the gate that was supposed to stop it. A green that lies is worse than a red that's honest, because you act on it.

Non-determinism removes the floor. Regression testing rests on one property: same input, same result, so a new failure means a new problem. Put a language model in the interpretation loop and the same plain-English step can resolve to different actions across runs. Now a red could be a real regression or just the model having a different idea today, and you can't tell which without reading everything. The foundation quietly dissolves.

The work doesn't disappear, it moves. This is the one that gets everyone. The tool does delete the writing. It hands you back the reviewing: opaque, generated artifacts you now have to read, understand, and trust before you can rely on them. Generation got cheap; comprehension didn't. The hours you saved authoring tests, you spend re-earning confidence in tests you didn't write, plus a new bonus task, debugging the healing algorithm instead of the product.

And the enterprise app is not the demo app. Real suites live in a world of SSO redirects and expiring tokens, hydration races and virtualized lists, data that has to be seeded and torn down, parallel workers colliding over the same records. The tools that glide through a to-do app tend to fall over here, which is exactly where you needed the help.

Every one of these is the same failure underneath: substituting plausibility for verification. The output looks right, so it's treated as right. That's the move I don't let an agent make.

What I actually do, the unflashy version

The pattern under all twenty of the posts that follow is one sentence: the agent explores, deterministic code decides.

I use the model for the thing it's genuinely good at: open-ended, judgment-heavy reconnaissance. Reading an unfamiliar domain. Drafting a first pass. Classifying a few hundred manual test cases. Driving a browser through a flow nobody scripted. That's real leverage, and the series leans on it hard.

But the model's output is never the shipped artifact. It's an input to an ordinary engineering process that turns it into something durable: a typed API client bound to an OpenAPI contract, a Playwright spec a human reviews in a pull request, a version-controlled orchestration script, a schema validator that runs the same way every time. The exploration is fluid and probabilistic. What I keep is deterministic, diffable, and reviewable. Code, not a recording in someone's cloud. When the agent is done being clever, a person can still read exactly what will run.

Around that core, four disciplines recur, and they map to the four sub-themes of the batch:

• Verify, don't trust. "Done" is meaningless until it's checked against the live system. I parse a ticket's acceptance criteria into probes and run them against the real API and the real browser (post 37); I fan 31 isolated browser agents across 316 manual cases rather than trust a documented "expected result" (post 38); I make the verifier structurally unable to talk itself into a pass (post 22).
• Guardrails live in hooks, not prompts. Asking an agent nicely is a suggestion; a pre-tool hook is a rule. One write-blocking hook missed the shell tool entirely (post 26); the fix is layered enforcement across the whole tool surface (posts 27, 29), and an environment where the secret the agent can't see is the only secret it can't leak (post 28).
• The filesystem is the source of truth, not the agent's self-report. A bootstrap agent once told me, with total confidence, that a corpus had four items. It had 161. Now agents write their output to disk and return only a count I can check (post 33); that single change also bought crash-resume and auditable 31-way fan-out for free (posts 38, 36).
• Your config, prompts, and docs are code. The context file is the most-read file in the repo, and it rots like code, so it gets a checker and a byte budget (posts 23, 24). Prompts get numbered variants and a held-out test set (post 40). A skill's description gets treated as a classifier with its own eval suite (posts 30, 31).

None of this demos well. There's no moment where a test heals itself on stage. The win is quieter: a suite whose green you can actually believe, built by an agent whose every claim was checked by something that can't be talked out of the truth.

Where it falls down

This approach costs more up front, and I should say so plainly. You don't buy it; you build it. There's no vendor to call when it breaks, because the "it" is your own hooks, scripts, and conventions. The agent still hallucinates. I just spend the verification tax deliberately, with deterministic gates catching what it gets wrong, instead of paying it by accident when a lie ships. And it asks something the magic button never does: that you keep understanding your own test suite. That's a feature. It's also work, every week, forever.

If what you want is a slider that turns coverage up without anyone reading anything, this isn't that. Nothing is, that's the whole point. But if you want AI to do the heavy, boring reconnaissance and leave you with code you can stand behind, the next twenty posts are how I do it.

How to read the rest

The batch splits into a few arcs: the skills layer as a product you can test (30 to 31), constraining an autonomous agent through its environment (26 to 29), agent memory and context engineering (23 to 25), orchestration patterns beyond the obvious (32 to 35, with adversarial verification pulled up to 22), and AI grounded in the live system (36 to 41). Post 20, in the earlier batch, is the bridge, the overview of the multi-agent workflows the rest of these zoom into.

If you only read three: post 33 (how a 161 to 4 hallucination rewired the whole architecture), post 37 (why "done" is never done), and post 23 (the 88KB context file I cut to 10KB). They're the shortest path to the thesis: the agent explores, the code decides, and nothing is true until you've checked it.

Takeaways

• The promises target real pain, which is why they sell. Maintenance, the engineering bottleneck, slow test design: all genuine. The fix on offer just doesn't survive a real app.
• An AI can drive your app but can't know what "correct" is. Generating assertions from observed behavior encodes today's bugs as tomorrow's expectations. That's transcription, not testing.
• A green that lies is worse than an honest red. Self-healing and non-determinism both manufacture confident green, masking the regression, or making "red" mean nothing.
• The work doesn't vanish, it moves to review. Generation got cheap; comprehension didn't. Budget for the verification tax, because you're paying it either way.
• Use the agent for exploration; keep deterministic code as the artifact. Reconnaissance is fluid and probabilistic; what you ship should be diffable, reviewable, and reproducible.
• Verify against the live system, put guardrails in hooks not prompts, trust the filesystem over the agent's word, and treat your prompts and docs as code. That's the whole series in one line.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/11/the-unflashy-truth-about-ai-in-testing/"
license: "Free to reference with attribution"
title: "The unflashy truth about AI in testing: the agent explores, the code decides"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 21"

problem: "Marketed AI-testing magic (self-healing locators, plain-English authoring, autonomous exploration, generation-from-requirements) collapses on a real enterprise app. The canonical failure: a Submit order button doesn't get renamed, it disappears from a regression, self-healing binds the spec to a similar-looking button, the test goes green, the broken order submission ships — the one test meant to catch that regression votes to approve it."
thesis: "The agent explores; deterministic code decides. Use the model only for open-ended, judgment-heavy reconnaissance; never let its output be the shipped artifact. The shipped thing is always diffable, reviewable, reproducible code — never a recording in someone's cloud. The single unifying sin of the marketed tools is substituting plausibility for verification."

the_four_promises:
  shape: "All four are the absence promise — each deletes a step you currently do: No code. No selectors. No maintenance. No humans in the loop. Each targets genuine pain, which is why they sell."
  promises:
    - "Self-healing tests — tests adapt to UI changes automatically; targets maintenance eating 60 to 80% of automation effort"
    - "Plain-English authoring — write tests in plain English, no code/selectors, anyone can do it; promises QA without the engineering bottleneck"
    - "Autonomous exploration — point the agent at the product, it finds critical flows and writes the tests; no scripts to author, no maintenance to own"
    - "Generation from requirements — feed it the ticket/PRD/design file, get a test suite back; test design at machine speed"

why_it_stops_scaling:
  oracle_problem: "An AI can drive an app (click/type/navigate) far better than it can judge whether what it saw was correct. Correctness lives in someone's intent, not in the page. Generating assertions by observing the app is transcription, not testing — it records today's behavior as expected, so a current bug is minted as a test that defends that bug forever."
  self_healing_masks_the_target: "Healing works by similarity — which element looks most like the one the test used last time — which is backwards on a real app where a broken control and a working one look nearly identical. A green that lies is worse than an honest red, because you act on it."
  non_determinism_removes_the_floor: "Regression testing rests on same input, same result, so a new failure means a new problem. A language model in the interpretation loop makes the same plain-English step resolve to different actions across runs, so a red could be a real regression or just the model having a different idea today — and you can't tell which without reading everything."
  the_work_moves: "The tool deletes the writing and hands back the reviewing — opaque generated artifacts you must read, understand, and trust. Generation got cheap; comprehension didn't. You re-earn confidence in tests you didn't write, plus a bonus task: debugging the healing algorithm instead of the product."
  enterprise_is_not_the_demo: "Real suites live among SSO redirects, expiring tokens, hydration races, virtualized lists, seed/teardown data, and parallel workers colliding over the same records — exactly where the demo-app tools fall over."

the_pattern_and_four_disciplines:
  core: "The model is used for reconnaissance (reading an unfamiliar domain, drafting a first pass, classifying manual cases, driving unscripted flows); its output is an input to ordinary engineering that turns it into something durable — a typed API client bound to an OpenAPI contract, a Playwright spec reviewed in a PR, a version-controlled orchestration script, a schema validator that runs the same way every time."
  verify_dont_trust: "Done is meaningless until checked against the live system. Parse a ticket's acceptance criteria into probes run against the real API and real browser (post 37); fan 31 isolated browser agents across 316 manual cases rather than trust a documented expected result (post 38); make the verifier structurally unable to talk itself into a pass (post 22)."
  guardrails_in_hooks_not_prompts: "Asking an agent nicely is a suggestion; a pre-tool hook is a rule. One write-blocking hook missed the shell tool entirely (post 26); the fix is layered enforcement across the whole tool surface (posts 27, 29), plus an environment where the secret the agent can't see is the only one it can't leak (post 28)."
  filesystem_over_self_report: "A bootstrap agent claimed with total confidence a corpus had 4 items; it had 161. Now agents write output to disk and return only a count you can check (post 33) — which also bought crash-resume and auditable 31-way fan-out for free (posts 38, 36)."
  config_prompts_docs_are_code: "The context file is the most-read file in the repo and rots like code, so it gets a checker and a byte budget (posts 23, 24); prompts get numbered variants and a held-out test set (post 40); a skill's description is treated as a classifier with its own eval suite (posts 30, 31)."

batch_map:
  arcs: "Skills layer as a testable product (30-31); constraining an autonomous agent through its environment (26-29); agent memory and context engineering (23-25); orchestration patterns beyond the obvious (32-35, plus 22); AI grounded in the live system (36-41). Post 20 (earlier batch) is the bridge overview of the multi-agent workflows."
  if_you_only_read_three: "Post 33 (the 161-to-4 hallucination that rewired the architecture), post 37 (why done is never done), post 23 (an 88KB context file cut to 10KB)."

use_when: "Running an agent against a real enterprise test suite (SSO, expiring tokens, hydration races, seeded data, parallel workers) where you can build your own hooks, scripts, and deterministic gates and keep understanding the suite weekly."
avoid_when: "You want a slider that turns coverage up without anyone reading anything — this approach costs more up front, has no vendor to call, and demands you keep comprehending your own suite forever."

takeaways:
  - "The promises target real pain — maintenance, the engineering bottleneck, slow test design — which is why they sell; the fix on offer just doesn't survive a real app."
  - "An AI can drive your app but can't know what correct is; generating assertions from observed behavior encodes today's bugs as tomorrow's expectations — transcription, not testing."
  - "A green that lies is worse than an honest red; self-healing and non-determinism both manufacture confident green, masking the regression or making red mean nothing."
  - "The work doesn't vanish, it moves to review; generation got cheap, comprehension didn't, so budget for the verification tax because you're paying it either way."
  - "Use the agent for exploration; keep deterministic code as the artifact — reconnaissance is fluid and probabilistic, what you ship should be diffable, reviewable, and reproducible."
  - "Verify against the live system, put guardrails in hooks not prompts, trust the filesystem over the agent's word, and treat your prompts and docs as code — that's the whole series in one line."

keywords: ["why AI self-healing tests are dangerous", "agent explores code decides testing pattern", "oracle problem in AI test generation", "deterministic guardrails for testing agents", "verification tax of AI-generated tests", "self-healing locator binds to wrong button regression"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).