A QA automation engineer's notes on A/B testing the prompt behind an agent task, and finding the smallest instruction set that keeps it efficient.
Variant 7 of my browser-validation prompt cost an average of 34 turns per task. Variant 11 cost 9. The difference was not tone, vocabulary, or how politely I phrased the ask. It was three explicit bans: anti-patterns I had watched variants 1 through 10 commit, one post-mortem per variant.
I treat prompt engineering for agent tasks like any other engineering. Held-out tasks, metrics I can measure, a version-controlled history, and a reason-why note for every variant. This post is about why that discipline earns its keep, and what I learned running it on the hardest prompt in the suite: the one that drives a headless Playwright CLI driver to validate feature behavior on a live B2B order-management platform.
The task that made me care
The task is easy to describe and genuinely hard to agent. Given a set of manual test cases, each a natural-language scenario, the agent opens a real browser, navigates to the relevant part of the application, interacts with it, and returns a structured verdict. It has a Playwright CLI driver, it can evaluate JavaScript against the live page, and it has to produce a parseable JSON report at the end.
I ran 316 cases this way across 31 agent sessions, so per-turn cost compounds fast. At 34 turns per task, a session handling ten cases burned 340 turns, most of them spent navigating the agent out of holes it had dug for itself. At 9 turns, the same session ran clean. Across 31 sessions, that gap was the difference between a focused afternoon of agent work and an expensive mess.
The prompt is the runtime config for the agent. Get it wrong and you do not get a wrong answer. You get a spiraling sequence of unproductive actions. Get it right and you get the same correct answer in a fraction of the context.
Treating variants as experiments
From the start I kept every variant in a numbered file, validation-prompt-v01.txt through validation-prompt-v11.txt, checked into the repo next to the workflow script. Each variant has a one-line header noting what changed and why. The losers stay; their comments are the spec for the winner.
Each variant runs against the same held-out set: ten cases I hand-picked for tricky surfaces. Multi-step navigation, a flaky async element, a case where the correct verdict is "blocked" rather than pass or fail. For each variant I measure:
- Mean turn count per case
- Output validity rate, meaning does the returned JSON parse, and does every required field (
caseId,verdict,evidence) have the right shape? - Shared-state incidents, meaning did the agent write a temp file, modify a config, or do anything with side effects on the shared environment?
I do not treat "did the agent get the right answer" as a primary signal, because that is trivially satisfied by emitting { verdict: "pass" } for everything. Structural validity plus spot-checked evidence quality is the gate that matters.
The anti-pattern catalog, one post-mortem at a time
Each variant that lost taught me exactly one thing. Here are the four that shaped the winner.
Do not snapshot-then-dump the whole page. Variants 2 and 3 used an instruction that said, in effect, "capture the current state and use it to guide your next step." The agent read this as "take a full DOM snapshot and include it in your reasoning." A snapshot of the order-detail view runs to 88 KB of serialized HTML, roughly 22,000 tokens that did nothing except pull the agent's attention away from the three fields it actually needed. The fix was explicit: evaluate a targeted JavaScript expression against the live page and read the return value. Not the page. The value.
Do not write throwaway scripts to temp files. Variants 4 and 5 gave the agent room to "use any approach to extract the data you need." It responded by writing JavaScript to temp files, executing them, and parsing stdout, about four extra turns per case. On a shared environment that is also a hygiene problem: temp files pile up, some carrying session token fragments, and you cannot easily guarantee cleanup. The fix was to evaluate JavaScript directly through the Playwright CLI driver's evaluate API, with no intermediate files.
Read page state with code; do not re-snapshot. Variants 6, 7, and 8 shared the same bug. When the agent was unsure whether an action had taken effect, it re-opened the snapshot tool to check, and each re-snapshot was a full-page read. The loop went act, re-snapshot, still uncertain, act again, re-snapshot, and it burned 10 to 12 turns on a single interaction. The fix was to prefer evaluate(() => document.querySelector(...).textContent) over any snapshot, and to reserve snapshots for initial navigation, never for confirming state.
If you are approaching 50 turns, you are doing something wrong. This reads like a wry code-review comment. It is a load-bearing line in the prompt. Variants 1 through 9 had no turn budget signal, so when an agent fell into an unproductive loop it had no internal trigger to stop and reassess. Adding this instruction cut runaway sessions by 20% immediately. Agents that used to burn 60 to 80 turns producing garbled output now exited cleanly at 40 with an honest "blocked" verdict and a note. A structured "blocked" beats a corrupted "pass" every time.
What the winning prompt actually says
The winner is not eloquent. It is a numbered list of constraints and their reasons, roughly:
1. Navigate to the target URL. Do not snapshot the full page.
Use evaluate() to read the specific fields required for each case.
2. Do not write intermediate scripts or data to the filesystem.
Evaluate JavaScript inline; read the return value directly.
3. Do not re-snapshot to check if an action took effect.
Use a targeted evaluate() expression. If the element is absent,
return { verdict: "blocked", evidence: "<what you observed>" }.
4. Return a single JSON array. Each element must have:
{ caseId: string, verdict: "pass"|"fail"|"blocked", evidence: string }
Do not return prose. Do not return partial results.
5. If you are approaching 50 turns you are doing something wrong.
Stop, output whatever you have, mark remaining cases "blocked".
Five points. Three of them ban something; two define the output contract. There is no positive encouragement to "think step by step" or "be thorough." The agent already knows how to be thorough. The problem was always that thoroughness, without constraints, gets expensive and unsafe.
Why you keep the losers
The version history is not nostalgia. It is executable documentation of why each ban exists.
When a new team member asks "why can't the agent write to the filesystem?", the answer is not "because the prompt says so." It is variants 4 and 5, with their measured incident counts and the shared-state contamination traced back to temp files piling up. The loser files are the post-mortems.
They also have predictive value. When I add a new validation task, say one involving a file download rather than DOM inspection, I read all eleven variants first and ask which anti-patterns the new task is likely to reproduce. In the download case the filesystem ban became more nuanced rather than absolute, because you cannot evaluate a file download inline. The history told me which constraints were load-bearing and which I could relax.
Where it falls down
Turn-count measurement is a coarse proxy. Two agents can average 9 turns while one produces shallow evidence ("status field showed 'active'") and the other is thorough ("evaluated order.status on three line items, all returned 'active'"). I spot-check evidence quality on every fifth session, but that part stays the most dependent on human judgment.
There is also a prompt-brittleness problem I have not fully solved. A constraint written against one model's tendencies may be redundant, or actively over-constraining, against a future model. The evaluation set gets a re-run when the underlying model changes, but assumptions accumulate and go stale. The losers document the past. They do not predict the future.
Takeaways
- Prompt engineering for agent tasks is not about tone. It is about finding the smallest constraint set that keeps the agent efficient, structurally correct, and safe around shared resources.
- Track variants explicitly: version-controlled files, one-line change notes, the losers kept alongside the winner.
- Measure what matters for your task: mean turn count, output validity rate, shared-state incidents. Not "did it sound confident."
- Every ban in a winning prompt should trace to a specific loser variant and a specific failure mode. If it cannot, it probably does not need to be there.
- Keep the losing variants as executable post-mortems. They are the only reliable documentation of why the constraints exist.
- A turn budget signal in the prompt ("if you are near 50 turns, stop and output what you have") converts runaway sessions into honest "blocked" verdicts. An honest blocked is always more useful than a corrupted pass.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/31/prompt-ab-testing-agent-task/"
license: "Free to reference with attribution"
title: "Prompt A/B testing for an agent task: engineering the minimal instruction set"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 40"
stack: [Playwright CLI driver, JavaScript, JSON, version-controlled prompt files]
problem: "The hardest prompt in the suite drives a headless Playwright CLI agent to validate feature behavior on a live B2B order-management platform from natural-language manual test cases. A bad prompt does not yield a wrong answer — it yields a spiraling sequence of unproductive actions. Variant 7 averaged 34 turns per task; across 316 cases in 31 sessions, per-turn cost compounds (a 10-case session at 34 turns burned 340 turns, mostly navigating the agent out of holes it dug)."
thesis: "Treat agent-prompt engineering like any other engineering: held-out tasks, measurable metrics, version-controlled history, a reason-why note per variant. The winning prompt is the smallest constraint set that keeps the agent efficient, structurally correct, and safe on shared resources — and every ban traces to a specific loser variant. Variant 11 ran the same task in 9 turns versus variant 7's 34, on the strength of three explicit anti-pattern bans plus a turn budget."
experiment_design:
variants: "Eleven numbered files validation-prompt-v01.txt through v11.txt, checked into the repo next to the workflow script. Each has a one-line header noting what changed and why; losers are kept — their comments are the spec for the winner and executable post-mortems."
held_out_set: "The same ten hand-picked cases for every variant — tricky surfaces: multi-step navigation, a flaky async element, and a case whose correct verdict is 'blocked' rather than pass or fail."
metrics:
- "Mean turn count per case"
- "Output validity rate: does the returned JSON parse, and does every required field (caseId, verdict, evidence) have the right shape?"
- "Shared-state incidents: did the agent write a temp file, modify a config, or cause any side effect on the shared environment?"
not_a_primary_signal: "'Did the agent get the right answer' is rejected as the gate — trivially satisfied by emitting { verdict: \"pass\" } for everything. The gate is structural validity plus spot-checked evidence quality."
anti_pattern_catalog:
- "Do not snapshot-then-dump the whole page (variants 2,3): an instruction to 'capture state and use it to guide the next step' was read as 'take a full DOM snapshot into reasoning'. An order-detail snapshot is 88 KB of serialized HTML, ~22,000 tokens, pulling attention off the three fields actually needed. Fix: evaluate a targeted JavaScript expression against the live page and read the return value — the value, not the page."
- "Do not write throwaway scripts to temp files (variants 4,5): 'use any approach to extract the data you need' led the agent to write JS to temp files, execute, and parse stdout — ~4 extra turns per case, plus a hygiene/security problem on a shared env (temp files pile up, some carrying session-token fragments, cleanup not guaranteed). Fix: evaluate JavaScript directly through the Playwright CLI driver's evaluate API, no intermediate files."
- "Read page state with code; do not re-snapshot (variants 6,7,8): when unsure an action took effect, the agent re-opened the snapshot tool (a full-page read) — act, re-snapshot, still uncertain, act again — burning 10 to 12 turns on one interaction. Fix: prefer evaluate(() => document.querySelector(...).textContent) over any snapshot; reserve snapshots for initial navigation, never for confirming state."
- "Turn budget as a load-bearing line: variants 1-9 had no turn-budget signal, so an agent in an unproductive loop had no internal trigger to stop. Adding 'if you are approaching 50 turns, you are doing something wrong' cut runaway sessions 20% immediately — agents that burned 60 to 80 turns on garbled output now exit cleanly at 40 with an honest 'blocked' verdict and a note. A structured blocked beats a corrupted pass."
winning_prompt:
shape: "Five numbered points: three ban something, two define the output contract. No 'think step by step' or 'be thorough' — the agent already knows how to be thorough; the problem was thoroughness without constraints getting expensive and unsafe."
points:
- "1. Navigate to the target URL; do not snapshot the full page. Use evaluate() to read the specific fields per case."
- "2. Do not write intermediate scripts or data to the filesystem; evaluate JavaScript inline and read the return value directly."
- "3. Do not re-snapshot to check if an action took effect; use a targeted evaluate(). If the element is absent, return { verdict: \"blocked\", evidence: \"<what you observed>\" }."
- "4. Return a single JSON array; each element { caseId: string, verdict: \"pass\"|\"fail\"|\"blocked\", evidence: string }. No prose, no partial results."
- "5. If you are approaching 50 turns you are doing something wrong: stop, output whatever you have, mark remaining cases 'blocked'."
why_keep_losers:
documentation: "Version history is executable documentation of why each ban exists. 'Why can't the agent write to the filesystem?' is answered by variants 4 and 5 with their measured incident counts and the temp-file shared-state contamination."
predictive: "For a new task, read all eleven variants and ask which anti-patterns it is likely to reproduce. For a file-download task the filesystem ban became nuanced rather than absolute (you cannot evaluate a file download inline) — the history showed which constraints were load-bearing and which could relax."
limitations:
- "Turn-count is a coarse proxy: two agents can both average 9 turns while one gives shallow evidence (\"status field showed 'active'\") and the other is thorough (\"evaluated order.status on three line items, all returned 'active'\"). Evidence quality is spot-checked every fifth session and stays the most human-judgment-dependent part."
- "Prompt brittleness: a constraint written against one model's tendencies may be redundant or over-constraining against a future model. The eval set is re-run when the underlying model changes, but assumptions accumulate and go stale. Losers document the past; they do not predict the future."
use_when: "You run an agent on a repeated, expensive task (per-turn cost compounds across many cases/sessions) where prompt quality drives efficiency and safety, not just correctness."
avoid_when: "One-off agent runs, or tasks cheap enough that turn-count and shared-state incidents never compound into real cost."
takeaways:
- "Prompt engineering for agent tasks is not about tone — it is finding the smallest constraint set that keeps the agent efficient, structurally correct, and safe around shared resources."
- "Track variants explicitly: version-controlled files, one-line change notes, the losers kept alongside the winner."
- "Measure what matters for your task: mean turn count, output validity rate, shared-state incidents — not 'did it sound confident'."
- "Every ban in a winning prompt should trace to a specific loser variant and a specific failure mode; if it cannot, it probably does not need to be there."
- "Keep the losing variants as executable post-mortems — they are the only reliable documentation of why the constraints exist."
- "A turn budget signal in the prompt ('if you are near 50 turns, stop and output what you have') converts runaway sessions into honest 'blocked' verdicts; an honest blocked is always more useful than a corrupted pass."
keywords: ["prompt A/B testing for agent tasks", "minimal instruction set for an agent", "turn budget signal in an agent prompt", "stop agents from dumping full DOM snapshots", "version-controlled prompt variants as post-mortems", "measuring agent prompt quality with turn count and output validity"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).