Cosmin Pop

Why a binary pass/fail suite has no honest way to describe a spec that correctly documents a feature that has not shipped yet, and how a deployment-guarded skip (NOT-DEPLOYED) becomes queued work instead of red noise.

How 31 parallel, isolated Playwright agents reconciled 316 legacy manual test cases against a live app in a single run, and the pre-assignment discipline (case slice, output directory, session name) that kept it trustworthy, not just fast.

A ticket marked 'Done', a test case marked 'Pass', and a clause the app simply does not satisfy: the failure mode I kept hitting. Neither the ticket nor the OpenAPI spec is ground truth; the running app is. So every acceptance-criteria clause gets grounded in a live probe: a typed API call and a headless-browser render, classified from observed evidence into MATCHES, DEFECT-API, DEFECT-UI, NOT-DEPLOYED, or DEFECT-Spec. A verdict with a receipt, not a probability.

Answering 'does this ticket cover the feature?' from the live tracker API costs four round-trips, auth, and rate limits: thirty seconds and a slice of context, for a read-only lookup that agents repeat constantly. So the whole tracker becomes a nightly SQLite snapshot, queried offline in milliseconds. The real unlock is baking institutional knowledge (like never trusting the raw status column) into the schema, so any agent queries it correctly without knowing the history.

A code-review pass produced 69 mechanical fixes: the kind of list that sits in a ticket for months. Doing them serially takes weeks; bundling them into one agent prompt yields a 185-file diff nobody can review. The fourth option is a wave-gated campaign of parallel LLM subagents, with a read-only census first, a self-healing static-check gate between every wave, and folder-scoped write access. 61 of 69 landed in one branch, zero merge conflicts.

Not every task belongs to the LLM. Plotting ambiguity against blast radius shows where to reach for deterministic orchestration with assertions instead, and reduce the agent to a shell runner behind a precondition gate.

An agent handed back four plausible, entirely made-up items for a 161-item corpus, with no error signal. Routing worker output through files instead of agent chatter makes a multi-agent pipeline resistant to that failure.

When the backend died mid-run, the subagents dutifully logged every remaining item as a failure and buried the real findings in noise. The fix: a circuit breaker in the orchestrator that halts fan-out once failures cluster.

A skill that aces softball prompts proves nothing. Eval-driven development for AI skills: harvest real failure modes from unguided runs, then build discriminating evals that separate a working skill from an impressive one.

An agent picks which skill to run by matching the prompt against each skill's description. That makes the description a classifier, not a summary: write it to fire on the right prompts and stay silent on the wrong ones.