A QA automation engineer's notes on fanning out parallel browser agents over a legacy manual-test corpus without losing trust in the results.
The number that stopped me was 316. That was the count of legacy manual test cases I needed to reconcile against the live application before retiring a folder in the test-case management tool. For each one I had to verify the app still matched the documented expectations, classify the case, and capture evidence. The naive path was a tester clicking through them for several days. The slightly smarter path was one LLM subagent working through them serially. I did neither.
I fanned out 31 parallel browser agents. Each one drove its own isolated Playwright session, got handed a precise case slice, and emitted structured verdicts and per-step screenshots into its own output directory. The whole 316 resolved in a single run. Here is what made that trustworthy rather than just fast.
The structure of the problem
Each manual test case was a short document: a title, 3-8 acceptance-criteria steps, a feature-area reference, and links to tracker tickets like PROJ-6272. Some were purely API-level ("creating an order with a missing required field returns a 422"). Others required a real browser. Others were gated on preconditions I could not set up, like infrastructure states or third-party integrations in a specific mode. And a meaningful chunk were simply not applicable: cases written for a workflow that exists only in a different deployment tier.
That spread (pass, fail, blocked, partial, not-applicable) matters enormously. Collapsing it to pass/fail loses real information. "Not-applicable" is not a cop-out. It is the only honest verdict for an API-only case a browser agent cannot exercise, and conflating it with "fail" adds systematic false failures to your results.
I built that taxonomy into the schema from the start:
{
"caseId": "MT-5681",
"verdict": "pass",
"steps": [
{ "step": 1, "action": "Navigate to order list", "outcome": "pass", "evidence": "screenshot-mt5681-step1.png" },
{ "step": 2, "action": "Filter by status = pending", "outcome": "pass", "evidence": "screenshot-mt5681-step2.png" }
],
"notes": ""
}
Verdicts were exactly one of: pass, fail, blocked, partial, not-applicable. The schema was fixed before any agent started. That meant aggregation was deterministic code, a sort and count across a flat array, rather than another model call trying to summarise prose.
Pre-assigning everything before the first agent starts
The thing that makes parallel fan-out trustworthy is not the fan-out itself. It is that the orchestrator pre-assigns every agent's work before any of them start, so no two agents can collide.
I pre-assigned three things per agent, deterministically.
The case slice. I sorted the 316 cases by ID, split into 31 contiguous slices of roughly 10 cases, and wrote each slice to a JSON file on disk before invoking the agent. The agent received a file path, not a dynamic query. It could not accidentally pick up someone else's slice.
The output directory. Each agent wrote its verdict file and screenshots to a unique path: output/batch-01/ through output/batch-31/. No two agents shared a directory, and a crashed agent left a partial directory that was trivially detectable rather than a corrupted shared file.
The browser session name. This caught me out in an early draft. Playwright lets you name browser contexts, which is useful for auth routing but also a source of collisions if two agents try to open the same named context against the same environment at once. I solved it by assigning a unique session name per agent at pre-assignment time, derived from the batch index. Agents never chose their own names.
A sketch of the orchestrator's pre-assignment loop:
const BATCH_COUNT = 31;
const batches = chunk(cases, Math.ceil(cases.length / BATCH_COUNT));
const assignments = batches.map((slice, i) => ({
batchId: String(i + 1).padStart(2, '0'),
outputDir: path.join(OUTPUT_ROOT, `batch-${String(i + 1).padStart(2, '0')}`),
sessionName: `verify-session-${String(i + 1).padStart(2, '0')}`,
caseFile: writeCaseSlice(slice, i), // written to disk before any agent starts
}));
await fs.mkdir(OUTPUT_ROOT, { recursive: true });
for (const a of assignments) await fs.mkdir(a.outputDir, { recursive: true });
// only NOW do we fan out
await Promise.all(assignments.map(a => agent(buildPrompt(a))));
The ordering matters. Everything is written, every directory exists, and every assignment is immutable before the fan-out happens. Fan-out without isolation turns into chaos at scale, and isolation without pre-assignment leaves you hoping nothing collides.
The read-only-DB constraint
The source of truth for the manual test cases was a SQLite snapshot, a point-in-time export of the test-case management tool, ingested into a local database at the start of the run. That database was the input. It was never the output.
This was not optional. With 31 concurrent agents, any design that had them writing back to a shared database would either need a serialisation layer (a bottleneck) or risk write contention (silent data loss). Instead, each agent wrote its verdict to its own directory, and a final aggregation step, which was sequential, cheap, and deterministic, read all 31 directories and merged them into one results file.
The provably-untouched source DB also served as audit evidence: no case had been silently mutated, reclassified retroactively, or dropped. That matters for a reconciliation that decides which cases get retired and which get ported into automation.
What each agent actually did
Each agent received:
- The JSON case slice (10-ish cases, with acceptance criteria and ticket references)
- A locator crib, a compact description of the selectors and navigation helpers for that slice's feature area, derived from the actual page objects in the codebase
- The verdict schema
- An explicit instruction to emit
not-applicablerather thanfailfor any case it could not meaningfully exercise, with a requirednotesfield
The locator crib was the piece of the prompt I iterated on most. Without it, agents either hallucinated selectors or exhausted their context budget exploring the DOM. Handing them the relevant POM surface, just the named locators and their semantic descriptions rather than the full implementation, cut both failure modes substantially.
Each agent opened a real browser session, navigated the application, and emitted a screenshot at each step. Screenshots were named deterministically (<session-name>-mt<id>-step<n>.png) and written to the pre-assigned output directory. Step-level evidence was essential for "partial" verdicts: without per-step screenshots, a partial is unactionable.
The aggregation
With all 31 agents complete, the aggregation ran in a few seconds:
const verdicts = [];
for (const dir of batchDirs) {
const raw = await fs.readFile(path.join(dir, 'verdicts.json'), 'utf8');
verdicts.push(...JSON.parse(raw));
}
const byVerdict = Object.groupBy(verdicts, v => v.verdict);
const summary = {
total: verdicts.length,
pass: byVerdict.pass?.length ?? 0,
fail: byVerdict.fail?.length ?? 0,
blocked: byVerdict.blocked?.length ?? 0,
partial: byVerdict.partial?.length ?? 0,
notApplicable: byVerdict['not-applicable']?.length ?? 0,
};
I verified summary.total === 316 before trusting the rest. There were zero identity mismatches: every MT-#### from the source appeared exactly once in the merged output.
The spread across the 316 cases was not a clean win column. A meaningful portion came back not-applicable, which were API-only cases the browser agent correctly declined to fake. A real portion were blocked or partial, each with evidence and a notes field pointing to the specific step or unmet precondition. Those became the actionable tail, things to investigate or file against the relevant tracker ticket.
Where it falls down
The locator crib needs maintenance. If page objects change and the crib is not updated, agents start guessing and verdict quality drops. I caught this mid-run: one batch's verdicts were clustered with notes fields saying "could not locate the filter panel", because a locator had been renamed in the POM the week before. The fix was quick. The detection took longer.
The 31-agent ceiling was practical, not principled. It was the number where the total browser-session count stayed below what the environment could sustain without degrading page-load times enough to cause spurious failures. A smaller or larger fleet is a configuration change, and the isolation guarantees hold at any size.
Parallelism also does not help with the cases it cannot reach. The blocked bucket, cases requiring out-of-band infrastructure, remains a human problem. Thirty-one agents in parallel cannot manufacture a third-party integration in a specific error state. They can at least report honestly that they cannot.
Takeaways
- Pre-assign every agent's slice, output directory, and session name before spawning. Fan-out without pre-assignment is shared mutable state pretending to be parallelism.
- "Not-applicable" is a real verdict. Build it into the schema. Conflating it with "fail" adds systematic false failures for cases the agent genuinely cannot exercise.
- Keep the source database read-only and aggregate by merging files. A shared writable database under 31 concurrent writers needs a concurrency layer. One file per agent and one merge pass at the end is simpler and auditable.
- Hand agents a locator crib, not raw acceptance criteria. It is the single highest-leverage prompt improvement for browser-driving agents. If the POM changes, update the crib.
- Per-step evidence (screenshots) is what makes partial verdicts actionable. A "partial" without per-step evidence is just a polite "something went wrong".
- The aggregation is deterministic code, not a model call. Structured verdicts let a sort function produce the final report. The agents do the browser work, the code does the counting.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/28/parallel-browser-agents-at-scale/"
license: "Free to reference with attribution"
title: "31 browser agents, hundreds of manual tests, one run: scaling live verification without losing trust"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 38"
stack: [Node.js, Playwright, SQLite, "LLM browser agents", JSON]
problem: "316 legacy manual test cases had to be reconciled against the live app before retiring a folder in the test-case management tool — each verified, classified, and evidenced. A tester clicking through takes days; one LLM subagent working serially is still slow; both lose trust at scale."
thesis: "Fan out 31 parallel browser agents, each driving an isolated Playwright session over a pre-assigned case slice and emitting structured verdicts plus per-step screenshots into its own output directory. The whole 316 resolves in one run, and isolation + pre-assignment + a fixed verdict schema make it trustworthy, not just fast."
verdict_taxonomy:
values: ["pass", "fail", "blocked", "partial", "not-applicable"]
why_five: "Collapsing to pass/fail loses real information. 'not-applicable' is the only honest verdict for an API-only case a browser agent cannot exercise; conflating it with 'fail' injects systematic false failures."
case_shape: "Each manual case is a short doc — title, 3-8 acceptance-criteria steps, a feature-area reference, and tracker links like PROJ-6272. Mix of API-level (e.g. missing required field returns 422), browser-required, precondition-gated (infra states / third-party integrations in a specific mode), and not-applicable cases written for a different deployment tier."
schema: "Fixed before any agent started: per-case caseId (MT-####), verdict, a steps[] array (step, action, outcome, evidence screenshot filename), and a notes field. Fixed schema makes aggregation deterministic code — a sort/count over a flat array — not another model call summarising prose."
pre_assignment:
principle: "What makes fan-out trustworthy is not the fan-out — it is that the orchestrator pre-assigns every agent's work before any start, so no two agents collide. Everything written, every directory created, every assignment immutable before fan-out."
three_things_per_agent:
case_slice: "316 cases sorted by ID, split into 31 contiguous slices of ~10, each written to a JSON file on disk before the agent is invoked. The agent gets a file path, not a dynamic query, so it cannot grab someone else's slice."
output_directory: "Unique path per agent, output/batch-01/ through output/batch-31/. No shared directory; a crashed agent leaves a trivially detectable partial directory rather than corrupting a shared file."
session_name: "Unique Playwright browser-context name per agent (verify-session-NN), derived from the batch index at pre-assignment time. Named contexts are useful for auth routing but collide if two agents open the same name against one environment — agents never choose their own names."
ordering: "mkdir the output root and every batch dir, writeCaseSlice for all batches, build immutable assignments, and only THEN Promise.all the agent fan-out. Fan-out without isolation is chaos at scale; isolation without pre-assignment leaves you hoping nothing collides."
read_only_source:
source_of_truth: "A SQLite snapshot — point-in-time export of the test-case management tool — ingested into a local DB at run start. It is the input, never the output."
why: "31 concurrent agents writing back to a shared DB would need a serialisation bottleneck or risk write contention / silent data loss. Instead each agent writes its own directory; a final sequential, cheap, deterministic aggregation reads all 31 and merges into one results file."
audit_value: "A provably-untouched source DB is audit evidence that no case was silently mutated, retroactively reclassified, or dropped — which matters when the reconciliation decides what gets retired vs ported into automation."
agent_prompt:
inputs: "Each agent received its JSON case slice (~10 cases with acceptance criteria and ticket refs), a locator crib, the verdict schema, and an explicit instruction to emit not-applicable (not fail) with a required notes field for any case it could not meaningfully exercise."
locator_crib: "The most-iterated prompt piece: a compact description of selectors and navigation helpers for the slice's feature area, derived from the actual page objects (POM) in the codebase — just named locators and their semantic descriptions, not full implementation. Without it agents hallucinated selectors or burned their context budget exploring the DOM; with it both failure modes dropped substantially."
evidence: "Each agent opened a real browser, navigated, and emitted a screenshot per step, named deterministically <session-name>-mt<id>-step<n>.png into the pre-assigned dir. Per-step screenshots are what makes a 'partial' verdict actionable rather than a polite 'something went wrong'."
aggregation:
mechanism: "After all 31 agents finish, a sequential pass reads verdicts.json from each batch dir, concatenates into one array, and Object.groupBy by verdict yields total/pass/fail/blocked/partial/notApplicable counts in a few seconds."
validation: "Verified summary.total === 316 before trusting the rest. Zero identity mismatches — every MT-#### from the source appeared exactly once in the merged output."
outcome_spread: "Not a clean win column. A meaningful portion came back not-applicable (API-only cases the browser agent correctly declined to fake); a real portion were blocked or partial, each with evidence and a notes field pointing to the specific step or unmet precondition — the actionable tail to investigate or file against a tracker ticket."
limitations:
crib_maintenance: "If page objects change and the crib is not updated, agents start guessing and verdict quality drops. Caught mid-run: one batch's notes clustered on 'could not locate the filter panel' because a POM locator had been renamed the week before. The fix was quick; the detection took longer."
agent_ceiling: "31 was practical, not principled — the count where total concurrent browser sessions stayed below what the environment could sustain without degrading page-load times into spurious failures. Fleet size is a config change; isolation guarantees hold at any size."
blocked_bucket: "Parallelism does not reach cases needing out-of-band infrastructure (e.g. a third-party integration in a specific error state). That stays a human problem; agents can at least report honestly that they cannot."
use_when: "Reconciling a large corpus of manual/legacy test cases against a live app, or any batch browser-verification job where the work partitions cleanly and you need trustworthy, auditable, parallel throughput."
avoid_when: "Small case counts where a single serial run is fine, cases gated on out-of-band infrastructure agents cannot manufacture, or environments that cannot sustain many concurrent browser sessions without spurious failures."
takeaways:
- "Pre-assign every agent's slice, output directory, and session name before spawning. Fan-out without pre-assignment is shared mutable state pretending to be parallelism."
- "Not-applicable is a real verdict. Build it into the schema. Conflating it with fail adds systematic false failures for cases the agent genuinely cannot exercise."
- "Keep the source database read-only and aggregate by merging files. A shared writable database under 31 concurrent writers needs a concurrency layer; one file per agent and one merge pass at the end is simpler and auditable."
- "Hand agents a locator crib, not raw acceptance criteria. It is the single highest-leverage prompt improvement for browser-driving agents. If the POM changes, update the crib."
- "Per-step evidence (screenshots) is what makes partial verdicts actionable. A partial without per-step evidence is just a polite 'something went wrong'."
- "The aggregation is deterministic code, not a model call. Structured verdicts let a sort function produce the final report. The agents do the browser work, the code does the counting."
keywords: ["parallel browser agents at scale", "fan-out LLM agents for test verification", "pre-assigning agent work to avoid collisions", "not-applicable verdict in test taxonomy", "read-only source DB with file-per-agent aggregation", "locator crib for browser agents", "reconciling manual test cases against a live app"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).