Agentic QA in practice: multi-agent workflows that maintain the test suite

A QA automation engineer's notes on multi-agent workflows that maintain a test suite — orchestration in code, models only at the leaves.

Most "AI in testing" content stops at the demo: prompt a model, get a test file, applause. That was never my bottleneck. My bottleneck is the one every mature suite has, which is maintenance. Docs drift, specs rot into false-passes, and approved refactor plans sit unimplemented because nobody has three free days.

So instead of one chat window and a lot of vibes, I keep a registry of about twenty workflow scripts: plain JavaScript files, checked into the repo next to the tests, versioned and code-reviewed like everything else. Each one orchestrates multiple LLM subagents against the Playwright suite for a B2B order-management platform. The loops, fan-out, phases, and aggregation are ordinary deterministic code. Only the leaf tasks ("audit this one file", "review this one diff") are model calls.

That inversion is the whole trick. Here are three of the workflows, and the principles that fell out of building them.

Workflows are code; only the leaves are models

A workflow script exports metadata (name, phases, argument contract) and then runs orchestration primitives. agent() spawns a subagent with a prompt and an optional structured-output schema, parallel() fans out, pipeline() streams items through stages. Everything else is plain JS you can read in review:

// .agent/workflows/doc-drift.js
export const meta = {
  name: 'doc-drift',
  description: 'Audit docs against the code; classify drift by truth ' +
    'direction; adversarially confirm; report-only by default.',
  phases: [
    { title: 'Inventory', detail: 'enumerate the doc corpus' },
    { title: 'Verify',    detail: 'one auditor per doc, read-only' },
    { title: 'Confirm',   detail: 'adversarial re-check of every finding' },
    { title: 'Repair',    detail: 'apply mode only: fix confirmed stale claims' },
  ],
}

const MODE = args.mode === 'apply' ? 'apply' : 'report'

Because the script is code, a re-run with the same args does the same thing, a reviewer can diff a behavior change, and "the agent decided to do something weird" is mostly off the table. The agent never decides the control flow at all.

Example 1: doc-drift, drift detection with a truth direction

Documentation lies harmlessly until someone follows it. The standards docs cite path:line anchors, symbol names, and counts ("five domains", "47 specs"). All of those go stale.

doc-drift fans out one read-only auditor per doc. Each auditor extracts every verifiable claim (anchors, symbols, counts, recipe snippets, cross-references) and checks it against the actual code with grep and file reads. The design idea that took the longest to get right is truth-direction classification: when a doc and the code disagree, you have to decide which side is wrong before you "fix" anything.

const FINDINGS_SCHEMA = {
  type: 'object', required: ['findings'],
  properties: {
    findings: { type: 'array', items: {
      type: 'object', required: ['quote', 'status', 'evidence'],
      properties: {
        quote:    { type: 'string' },  // exact drifted text in the doc
        status:   { enum: ['DOC_BEHIND',       // code moved on -> fix the doc
                           'CODE_VIOLATION', // doc states an invariant the code now violates -> FLAG, never edit
                           'RUNTIME_ONLY',   // only checkable by a live run -> leave it
                           'UNDECIDABLE'] },    // can't tell -> a human decides
        evidence: { type: 'string' },  // what the code shows, with file:line
        suggestedEdit: { type: 'string' },
      } } } },
}

The dangerous failure mode is the second one. If a standard says "scenario resolvers MUST throw on miss, never fabricate an entity" and the code stopped doing that, a naive doc-fixer would happily rewrite the doc to bless the bug. Truth direction makes that an explicit classification with an explicit rule: a regression gets flagged, never papered over.

Every finding then goes through an adversarial second agent that re-derives the evidence from scratch, with confirmed: false as the default, because a wrong "fix" to a standards doc costs more than a missed one. Only confirmed DOC_BEHIND findings are ever editable, only in explicit apply mode, with one editor per doc so writes never collide. The default mode produces a report and touches nothing.

Example 2: batch spec grading, one grader per file, machine-aggregatable verdicts

I have a written rubric for what a good spec looks like: convention criteria (single fixtures import, traceability annotations, no hardcoded entity IDs) and test-quality dimensions, with explicit false-pass anti-patterns. The assertion wrapped in if (rows.length > 0) that silently passes on an empty result set. The status check that accepts both 2xx and 4xx. The expect(count).toBeGreaterThanOrEqual(0) that cannot fail.

The batch-audit workflow fans out one auditor per spec file. Each reads the rubric, reads the spec in full, writes a detailed markdown shard, and returns one machine-parseable verdict:

{"path": "orders/tests/search/orderSearch.ui.spec.ts",
 "verdict": "Needs-rework", "quality": 2,
 "critical": 1, "major": 3, "vacuous": 4,
 "criticalFindings": ["orderSearch.ui.spec.ts:84 — assertions inside if (rows.length > 0); green on empty results"],
 "headline": "search coverage that cannot fail when the search returns nothing"}

Because the verdicts are structured, the aggregation is deterministic code, not another model call: a verdict distribution, a per-domain breakdown, and a severity-sorted table (Production-ready / Needs-fixes / Needs-rework) with every Critical finding carrying a real file:line. One run graded 120+ UI specs and surfaced 17 needing rework, almost all for false-pass patterns rather than style nits. A human then decided which fixes to schedule. The workflow grades; it does not silently rewrite tests.

Example 3: gated implementation, cheap builder, strong skeptic

For an approved refactor plan, I run a build/gate split. A cheaper model implements each phase inside an isolated git worktree (it physically cannot touch the shared tree), and a stronger model gates the phase before anything is committed.

for (const ph of plan.phases) {
  await build(ph)                       // cheaper model implements one phase
  let v = await verify(ph)              // stronger model: full static check suite
                                        // + adversarial diff review vs the plan
  if (!passed(v)) {
    await fix(ph, v.findings)           // one surgical fix pass, findings only
    v = await verify(ph)
  }
  if (!passed(v)) return halt(ph)       // red after retry = stop; a human reviews
  await commit(ph)                      // green = commit, move on
}

The gate is not "looks good to me". It runs the repo's entire static gate (typecheck, zero-warning lint, the AST convention guards) and then reads the full diff against the plan section, checking phase-specific invariants ("selection logic unchanged, telemetry only", "no new dependency", "this library layer still imports no test runner"). Pass means commit with a descriptive message; fail means one bounded fix pass, then halt. A six-phase data-layer refactor landed this way as six reviewed, individually green commits on a branch, which a human then inspected and merged.

The principles that carried over

Across all twenty-odd workflows, the same patterns kept earning their keep:

• Structured output schemas everywhere. An agent that returns prose needs another agent to read it. An agent that returns {verdict, critical, findings[]} feeds a sort function.
• Adversarial verification as a standing pattern. Generator and skeptic are different agents with different prompts, and the skeptic's default answer is "not confirmed". This is the single biggest reducer of confidently wrong output I've found.
• Guardrails in code, not prompts. Read-only phases run agents that literally lack write access. The OpenAPI contract files are protected by write-time hooks I added, so a mismatch with the spec is a defect to record, never a spec to "fix". Asking nicely in a prompt is a suggestion; a hook is a rule.
• Idempotency. Re-runs skip work already done (already-covered items, already-committed phases) so a crashed run resumes instead of duplicating.
• The human stays the router. Workflows produce reports, diffs, and branch commits. Humans decide what lands. Nothing merges itself.

Where it falls down

Honesty section, because the genre needs one. Truth-direction calls are sometimes genuinely ambiguous, and the model occasionally classifies a regression as stale docs. The adversarial pass catches most of these, not all, which is exactly why apply mode is opt-in. Adversarial verification roughly doubles the token bill for audit-type workflows; I pay it for anything that edits, and skip it for throwaway reports. And the workflows themselves rot like any code. They hardcode file lists and path assumptions that drift, so I now run audits on the workflow and skill registry itself, same pattern, one level up. Turtles, but versioned ones.

Takeaways

• Put the orchestration in versioned, reviewable code; let models do only the leaf tasks. Determinism is a feature you can't prompt for.
• Classify disagreements by truth direction before "fixing" anything, and never let an agent rewrite the documented invariant to match a regression.
• Demand structured outputs so results aggregate with a sort function, not another model call.
• Pair every generator with a skeptic whose default verdict is "not confirmed", and enforce hard limits with hooks and missing write access, not prompt etiquette.
• Keep humans as the router: agents propose reports, diffs, and gated commits; people decide what lands.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/09/agentic-qa-in-practice-multi-agent-workflows/"
license: "Free to reference with attribution"
title: "Agentic QA in practice: multi-agent workflows that maintain the test suite"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 20"
stack: [JavaScript, Playwright, LLM subagents, git worktrees, JSON Schema]

problem: "The real bottleneck in a mature suite is maintenance, not authoring: docs drift (path:line anchors, symbol names, counts go stale), specs rot into false-passes, and approved refactor plans sit unimplemented because nobody has three free days. \"Prompt a model, get a test file\" demos never touch that."
thesis: "Invert the usual agent design: keep about twenty workflow scripts — plain checked-in JavaScript, versioned and code-reviewed — where loops, fan-out, phases, and aggregation are deterministic code, and only the leaf tasks (audit one file, review one diff) are LLM subagent calls. The agent never decides control flow."

orchestration_model:
  primitives: "Each workflow script (.agent/workflows/*.js) exports meta (name, description, phases, argument contract) then runs primitives: agent() spawns a subagent with a prompt and optional structured-output schema; parallel() fans out; pipeline() streams items through stages. Everything else is readable plain JS."
  why_code: "A re-run with the same args does the same thing, a reviewer can diff a behavior change, and \"the agent decided to do something weird\" is off the table because the agent never controls flow."
  registry: "~20 workflow scripts live next to the tests; audits are also run on the workflow and skill registry itself — same pattern, one level up."

workflow_1_doc_sync:
  purpose: "Audit standards/docs against the code, classify drift, adversarially confirm, report-only by default (apply mode opt-in). Fans out one read-only auditor per doc; each extracts every verifiable claim (anchors, symbols, counts, recipe snippets, cross-references) and checks it against code via grep and file reads."
  truth_direction: "The hardest design idea: when a doc and the code disagree, decide which side is wrong before \"fixing\". A FINDINGS_SCHEMA tags each finding's status: DOC_BEHIND (code moved on → fix the doc), CODE_VIOLATION (doc states an invariant the code now violates → FLAG, never edit), RUNTIME_ONLY (only checkable by a live run → leave it), UNDECIDABLE (a human decides). Each finding carries quote, evidence (file:line), and suggestedEdit."
  danger: "Without truth direction, a naive doc-fixer rewrites the doc to bless a bug — e.g. blessing code that stopped throwing on a scenario-resolver miss and now fabricates an entity. Regressions get flagged, never papered over."
  adversarial_gate: "Every finding goes through a second agent that re-derives evidence from scratch with confirmed:false as default, because a wrong fix to a standards doc costs more than a missed one. Only confirmed DOC_BEHIND findings are editable, only in apply mode, one editor per doc so writes never collide."

workflow_2_batch_spec_audit:
  purpose: "Grade specs against a written rubric — convention criteria (single fixtures import, traceability annotations, no hardcoded entity IDs) and test-quality dimensions, with explicit false-pass anti-patterns."
  false_pass_anti_patterns:
    - "Assertions wrapped in if (rows.length > 0) — silently green on an empty result set"
    - "Status check that accepts both 2xx and 4xx"
    - "expect(count).toBeGreaterThanOrEqual(0) — cannot fail"
  mechanism: "One auditor per spec file reads the rubric, reads the spec in full, writes a markdown shard, and returns one machine-parseable verdict: {path, verdict, quality, critical, major, vacuous, criticalFindings[] (each with real file:line), headline}."
  aggregation: "Because verdicts are structured, aggregation is deterministic code, not another model call: a verdict distribution, per-domain breakdown, and a severity-sorted table (Production-ready / Needs-fixes / Needs-rework)."
  result: "One run graded 120+ UI specs and surfaced 17 needing rework, almost all for false-pass patterns rather than style nits. A human decides which fixes to schedule; the workflow grades, never silently rewrites tests."

workflow_3_gated_implementation:
  purpose: "Implement an approved refactor plan with a build/gate split: a cheaper model implements each phase inside an isolated git worktree (it physically cannot touch the shared tree); a stronger model gates the phase before any commit."
  loop: "For each plan phase: build (cheap model) → verify (strong model: full static gate + adversarial diff review vs the plan). If not passed, one surgical findings-only fix pass, then verify again. Still red after retry → halt for a human. Green → commit with a descriptive message and move on."
  gate_contents: "Not \"looks good to me\": runs the repo's entire static gate (typecheck, zero-warning lint, the AST convention guards), then reads the full diff against the plan section checking phase-specific invariants — e.g. \"selection logic unchanged, telemetry only\", \"no new dependency\", \"this library layer still imports no test runner\"."
  result: "A six-phase data-layer refactor landed as six reviewed, individually green commits on a branch, which a human then inspected and merged."

principles:
  - "Structured output schemas everywhere: an agent that returns prose needs another agent to read it; {verdict, critical, findings[]} feeds a sort function."
  - "Adversarial verification as a standing pattern: generator and skeptic are different agents with different prompts; the skeptic defaults to \"not confirmed\". Biggest reducer of confidently-wrong output."
  - "Guardrails in code, not prompts: read-only phases run agents that literally lack write access; OpenAPI contract files are protected by write-time hooks so a mismatch is a defect to record, never a spec to fix. A hook is a rule; a prompt is a suggestion."
  - "Idempotency: re-runs skip already-done work (already-covered items, already-committed phases) so a crashed run resumes instead of duplicating."
  - "The human stays the router: workflows produce reports, diffs, and branch commits; nothing merges itself."

limitations:
  - "Truth-direction calls are sometimes genuinely ambiguous; the model occasionally classifies a regression as stale docs. The adversarial pass catches most, not all — which is why apply mode is opt-in."
  - "Adversarial verification roughly doubles the token bill for audit-type workflows; paid for anything that edits, skipped for throwaway reports."
  - "The workflows rot like any code — they hardcode file lists and path assumptions that drift — hence auditing the workflow/skill registry itself."

use_when: "A mature, code-reviewed test suite where maintenance (drift, false-passes, unimplemented refactor plans) is the bottleneck, and you want repeatable, diffable, human-routed automation rather than a chat window."
avoid_when: "One-off test generation or a small suite without standing docs/conventions to maintain — the orchestration scaffolding and doubled token cost outweigh the payoff."

takeaways:
  - "Put the orchestration in versioned, reviewable code; let models do only the leaf tasks. Determinism is a feature you can't prompt for."
  - "Classify disagreements by truth direction before fixing anything, and never let an agent rewrite the documented invariant to match a regression."
  - "Demand structured outputs so results aggregate with a sort function, not another model call."
  - "Pair every generator with a skeptic whose default verdict is \"not confirmed\", and enforce hard limits with hooks and missing write access, not prompt etiquette."
  - "Keep humans as the router: agents propose reports, diffs, and gated commits; people decide what lands."

keywords: ["multi-agent QA workflows", "agentic test suite maintenance", "truth-direction doc drift classification", "adversarial verification generator and skeptic", "deterministic orchestration with LLM leaf tasks", "structured-output verdicts for batch spec grading", "gated refactor implementation in git worktrees"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).