All Posts

A row-count non-vacuity check passes on a grid whose only row is the empty-state placeholder. Ordering, filtering and deduplication claims each need a floor sized to what they actually assert, not to how many rows came back, and every oracle deserves an inversion test before it is trusted.

A skipped test is a claim about the world, not a shrug, and it lies in three ways that survive code review and the type checker: a gate that never fires, a reason that blames the wrong cause, and a park that records no reason at all. Replacing boolean misses with typed ones makes the honest shape structural.

Four AST convention guards grew to twenty-eight, and that changed the design problem, not just the count. A policy layer other people's code must pass through needs a ratchet baseline, a mandatory escape hatch, and corpus calibration before a new rule can block anything.

A green test that executed zero assertions is a real failure mode, an early return or a swallowed catch skipping every oracle on the path it took. A runtime counter at the one seam every assertion passes through finds it on every ordinary run, cheaper than inverting every assertion to prove the same thing.

Static quality gates check that a test looks right, not whether it can actually turn red. Running a folder twice, once clean and once with every assertion inverted through the runner's own negation, answers that directly, and finds oracle gaps a code review would never catch.

A discovery hot path got a deadline, a circuit breaker and a run broker after a 502 storm. Once a cheaper query made the load profile disappear, the armour was proven unused with a live-storm gate run, not a hunch, then deleted along with the seams a fifth domain would have copied.

A generator asks what should exist and regenerates everything, silently overwriting human edits and breaking downstream links. A reconciler asks what already exists, what changed, and what is the minimum mutation to stay current, using a stable ID hashed from meaning, not a counter or a content hash.

How treating an agent's driving prompt as an experiment (held-out cases, measured turn count, version-controlled losers) took a browser-validation task from 34 turns per case down to 9, on the strength of three explicit anti-pattern bans.

Why a binary pass/fail suite has no honest way to describe a spec that correctly documents a feature that has not shipped yet, and how a deployment-guarded skip (NOT-DEPLOYED) becomes queued work instead of red noise.

How 31 parallel, isolated Playwright agents reconciled 316 legacy manual test cases against a live app in a single run, and the pre-assignment discipline (case slice, output directory, session name) that kept it trustworthy, not just fast.

A ticket marked 'Done', a test case marked 'Pass', and a clause the app simply does not satisfy: the failure mode I kept hitting. Neither the ticket nor the OpenAPI spec is ground truth; the running app is. So every acceptance-criteria clause gets grounded in a live probe: a typed API call and a headless-browser render, classified from observed evidence into MATCHES, DEFECT-API, DEFECT-UI, NOT-DEPLOYED, or DEFECT-Spec. A verdict with a receipt, not a probability.

Answering 'does this ticket cover the feature?' from the live tracker API costs four round-trips, auth, and rate limits: thirty seconds and a slice of context, for a read-only lookup that agents repeat constantly. So the whole tracker becomes a nightly SQLite snapshot, queried offline in milliseconds. The real unlock is baking institutional knowledge (like never trusting the raw status column) into the schema, so any agent queries it correctly without knowing the history.

A code-review pass produced 69 mechanical fixes: the kind of list that sits in a ticket for months. Doing them serially takes weeks; bundling them into one agent prompt yields a 185-file diff nobody can review. The fourth option is a wave-gated campaign of parallel LLM subagents, with a read-only census first, a self-healing static-check gate between every wave, and folder-scoped write access. 61 of 69 landed in one branch, zero merge conflicts.

Not every task belongs to the LLM. Plotting ambiguity against blast radius shows where to reach for deterministic orchestration with assertions instead, and reduce the agent to a shell runner behind a precondition gate.

An agent handed back four plausible, entirely made-up items for a 161-item corpus, with no error signal. Routing worker output through files instead of agent chatter makes a multi-agent pipeline resistant to that failure.

When the backend died mid-run, the subagents dutifully logged every remaining item as a failure and buried the real findings in noise. The fix: a circuit breaker in the orchestrator that halts fan-out once failures cluster.

A skill that aces softball prompts proves nothing. Eval-driven development for AI skills: harvest real failure modes from unguided runs, then build discriminating evals that separate a working skill from an impressive one.

An agent picks which skill to run by matching the prompt against each skill's description. That makes the description a classifier, not a summary: write it to fire on the right prompts and stay silent on the wrong ones.

One agent's git add -A swept in another agent's half-written edits and committed them under the wrong name. The fix was not more trust but strict file ownership: partition the repo, stage by name.

Spawning an AI agent with env: process.env hands it every secret you hold, including the tracker token it must never see. An allowlist-first subprocess env makes leakage impossible by default.

Layered guardrails keep an autonomous coding agent from editing the OpenAPI contract specs it should treat as read-only. Why I kept a third layer commented out instead of deleting it.

A pre-tool hook that guards the Edit and Write tools leaves a bash-shaped hole: the agent just writes the file with a shell command instead. Why one guardrail is never enough.

How a coding agent remembers a project across sessions: a three-tier memory split of session-start context, durable feedback rules, and a queryable knowledge graph.

An agent's context file is code the model executes every session. Here's the linter, CI gate, and size budget I built to stop it drifting silently.

Cutting an 88KB context dump (injected at every session start and every pre-compaction pump) down to a 10KB evergreen working set with a hard byte budget.

Pair every generator with a skeptic, the advice goes. But a skeptic that shares the generator's output schema is just a generator in a hat, able to reach 'COVERED' by the same optimistic path. Classifying 316 legacy manual cases against a Playwright suite, the fix wasn't a harsher prompt but a structural constraint: make…

The self-healing demo is magnetic: rename a button's ID, the test goes red, then heals green. Nobody touched it. Now picture the version that never makes the slide deck: the Submit button doesn't get renamed, it disappears in a regression, healing binds to a similar-looking button, the test goes green, and the broken order…

Most 'AI in testing' content stops at the demo: prompt a model, get a test file, applause. The real bottleneck is maintenance (drift, false-passes, unimplemented refactor plans). So I keep ~20 workflow scripts, plain checked-in JavaScript where loops, fan-out, and aggregation are deterministic code and only the leaf tasks are LLM calls: doc-drift with truth-direction drift classification, a batch spec grading pass with machine-aggregatable verdicts, and gated refactor implementation in isolated git worktrees. The agent never decides control flow; the human stays the router.

A manual test case is a unit of manual labor, not of coverage, and automation inverts every economic assumption that shaped it. Porting several hundred manual cases 1:1 into Playwright buries the real coverage under hundreds of tiny specs that each re-pay full login and navigation. The rule I enforced above all: never default…

Two aging Playwright suites for the same platform (a browser E2E suite and a separate API-only suite) had diverged into two repos with their own auth and data hacks, more expensive to touch every quarter. Not a big-bang rewrite and not an in-place refactor: a strangler-fig migration into one repo where both tiers…

Bumping the worker count made the suite's discovery phase fan a few concurrent GET /orders calls and the shared dev environment sprayed 502s, under modest load. Nobody could answer with a number: at what concurrency does it break? I answered it with ~500 lines of TypeScript inside the Playwright repo I already had…

A handful of test conventions lived in a standards doc enforced by code review, so they decayed. Grep-based pre-commit checks failed too: template-literal titles, multi-line calls, and JSDoc prose force regex to choose between missing violations and crying wolf, and either kills trust in the gate. What worked: four small scripts that parse every…

Which requirement does this test cover, and which tests cover PROJ-7251? On a regulated-adjacent suite, 'let me check the spreadsheet' isn't an answer. I encode traceability as typed imperative calls inside each test body (trackingRef/testType/severity), mirror them as title tags for grep, report them automatically via an auto-fixture, generate a ticket-to-tests index, and enforce the whole convention with an AST guard in the merge gate. About four lines per test, no test-management plugin.

For a Playwright suite running 24/7 against a shared Cognito+SSO environment, auth, not the product, was the top cause of red runs: tokens expiring mid-run into waves of opaque 401s, or two processes refreshing at once and leaving everyone with a dead token. The fix is a three-layer architecture: validate, refresh, recover…

The shared dev order list opened to three pages of garbage: one leaked entity per write-path test per run, piling up since the suite first learned to POST, and tripping the discovery code that finds real entities. The fix wasn't a cleanup sprint but a one-page written policy: net-zero by default with a verified…

On a shared dev environment where orders can't be deleted, test data drifts under other teams' deploys and curious humans. Creating fresh entities floods the box; reusing one by name breaks the moment another spec edits it. The pattern that held: find-or-create seed pools keyed on a deterministic title, resolving each slot into one of four outcomes (REUSED, REPAIRED, DEGRADED, CREATED) and repairing drift with targeted idempotent writes. Repair beats recreate.

Fifty page objects across five domains is exactly the size where the layer usually rots: every entity form a 400-line fork, a utils/ folder nobody owns, specs reaching two layers down. Three structural decisions kept mine from rotting: one abstract base class owning entity-form mechanics so a new form costs ~40 lines not 400,…

Angular Material abstractions make a Playwright suite go green while the app quietly does the wrong thing. The root cause every time is that the visible DOM is not the state the form will submit. fill() leaves a ReactiveForms control pristine so the PATCH drops the field and the backend still returns 200. A…

A freshly re-architected suite came back with 143 failures, a number everyone discounted, because a raw count says only 'the suite is broken.' Classify first, fix second: force every failure into exactly one of DEFECT-API / DEFECT-SPEC / ENV / TEST / REGRESSION, each with its own owner and SLA, pin product bugs with…

A red test gets looked at within the hour; a test that passes while asserting nothing never does, and 34% of one legacy search suite were exactly that. This is the catalog I now audit every spec against: empty guards, mixed-status acceptance, echo-not-persistence, the bodyless 200, soft-assertion drift, over-broad error checks; each with…

test.skip, softened assertions, and stale ticket comments all share one fatal flaw: when the backend finally fixes the bug, nothing tells you. Instead, pin each known defect with a co-located *.defect.api.spec.ts that asserts the spec-correct behavior under Playwright's test.fail(): the run stays green while the bug lives, and the day the fix ships an…

Add nullable: true every time a test reddens and, eighteen months on, your OpenAPI spec is no longer a contract. It's a changelog of every backend bug you ever shipped. The fix: the vendored specs are read-only. The suite checks the API against the spec, never the reverse. A mismatch is a bad test…

Strict Ajv validation has no concept of 'yes, we know': one ticketed backend bug turned 76 happy-path tests red overnight and buried every new regression in the noise. Loosening the schema or skipping validation both lie invisibly. The fix is a central ledger of named drift exceptions, each carrying an owner, a ticket, and…

A backend deploy regressed an ID field from integer to string, every response still returned 200 OK, and every test stayed green, because casting res.json() to a type is a wish, not a check. Here's the one gate that closes the hole: assertValid(), a thin Ajv wrapper every API response passes through, with one…

Four parallel Playwright workers turned a discovery-first test suite into a denial-of-service tool. GET /orders returned 502s within a minute. The fix used no lock server or reservation service: lazy worker-scoped fixtures to stagger the load, and one line of Euclidean modulo to fan workers out across distinct entities.

A naive test-data finder is a load test in disguise: it pages the whole list and fires a per-entity GET for every row, all at once. pagedFind makes every cost dimension of the search an explicit, named bound, and its counters tell you where the funnel died when it misses.

Hardcoded entity IDs go stale the moment someone reshapes your shared test environment. I banned them: every spec names the shape of data it needs, and a ScenarioResolver finds a real entity at fixture time.

Security said no to TOTP, no to service accounts, no to disabling MFA. So I built a solution that uses OAuth2 refresh tokens to maintain Playwright browser sessions in CI/CD, without ever triggering push notification MFA after the initial setup. Here's the COSMIC Auth Pattern.