Testing the author: eval-driven development for AI skills

A QA automation engineer's notes on eval-driven development for AI skills: building the eval suite from real failure modes, not softball prompts.

A skill that gets easy questions right looks great in a demo and adds nothing in production. The value is in the hard path it steers you away from: the wrong answer a naive agent gives confidently and fast. If your eval suite can't tell "skill working" apart from "skill impressive on softball prompts", you have no idea whether you built anything.

I learned this the expensive way. After shipping about a dozen skills to my agentic coding CLI, I couldn't honestly say which ones helped. I had anecdote and vibe. I wanted evidence.

What I have now is an eval harness built around discriminator assertions, a mechanism that makes the measurement much harder to game. This post is the recipe, including the embarrassing part.

What a skill is, and why it's worth evaluating

In my setup, a skill is a small, versioned text file checked into the repo next to the tests. It describes how the agent should behave on a recurring task: audit a spec file, port a domain's tests, generate a defect report. The agent runtime picks up the relevant skill from a free-text description field, and everything downstream is deterministic orchestration code.

A trigger description in each skill file tells the router when to activate it, a routing problem I covered in a sibling post. A skill that triggers at the wrong times, or fails to trigger at the right times, corrupts every run that uses it. Eval covers both dimensions.

The baseline contamination problem (the embarrassing part)

My first eval attempt had a control group: tasks with the skill active versus tasks without it. The problem showed up when the "no skill" results came back suspiciously good. After an hour in the transcripts I found the reason. The baseline agents were discovering the skill files on disk and reading them anyway.

Nothing told them not to. The repo was present, the skill file was readable, and a capable agent doing research explores the codebase it lives in. The measured lift was zero because the baseline was already using the skill. Every A/B comparison was really skill-via-injection versus skill-via-exploration.

The fix is now a first-class part of my eval protocol:

{
  "evalMode": "baseline",
  "constraints": [
    "Do NOT read any file under .agent/skills/",
    "Do NOT read any file under .agent/workflows/",
    "If you discover a skill or workflow file by accident, discard its contents and do not apply its guidance."
  ]
}

This gets injected into every baseline agent's system context alongside the task prompt. It isn't a social contract, it's an explicit, logged constraint. A baseline agent that reads a skill file in defiance of this block fails the eval run, and I inspect those transcripts rather than averaging over them. Agent capability cuts both ways: a competent agent finds resources you didn't mean to provide, so you have to explicitly revoke them.

Discriminator assertions

With a clean baseline in place, the next problem was assertion design. My initial assertions measured obvious things: right sections, non-empty result, no runtime error. These passed for almost everything, skill or no skill, because they were testing floor-level competence the agent already had.

A discriminator is an assertion built around the single most tempting wrong answer, the one a naive agent gives most often. Instead of "did the agent produce the right answer", a discriminator asks "did the agent avoid this specific trap". That distinction matters because the wrong answer is usually confident and structurally plausible.

Here is a concrete example from my ordering domain. When asked to report how many search results came back, a naive agent locator-counts every DOM row, including sub-rows nested inside parent orders. A parent order with three line items renders as four rows: one parent, three children. The agent reports four where the correct answer is one. The canonical wrong answer is the inflated count.

The discriminator for this task:

{
  "id": "D-04",
  "description": "result-count inflation — nested sub-rows in order search",
  "prompt": "How many orders match the search for status=pending?",
  "expectedAnswerRange": [1, 3],
  "discriminator": {
    "forbiddenAnswers": [4, 5, 6, 7, 8],
    "rationale": "Naive locator counts sub-rows; correct answer counts parent orders only."
  }
}

The assertion is not expect(count).toBe(2). It is expect(count).not.toBeOneOf([4, 5, 6, 7, 8]). I test for the absence of the specific wrong answer, not for the presence of a single correct one. "Correct" has flexibility, since actual counts vary with test data, but "wrong in this specific way" does not.

I tag a subset of assertions as discriminators. When I measure skill lift I report two numbers separately:

• Overall pass rate: did the output meet general quality criteria
• Discriminator pass rate: did the output avoid the specific failure modes the skill is designed to prevent

A skill that lifts overall pass rate but moves nothing on discriminator pass rate is a style guide, not a guardrail.

Trigger-accuracy evals

The skill's content is only half the problem. If the router never fires the skill, content is irrelevant. If it fires on unrelated tasks, you burn tokens and risk prompt contamination. I run a separate, cheap eval for this: trigger-accuracy.

The format is a balanced prompt set, 10 prompts that should activate the skill and 10 that should not, scored for precision and recall independently:

Trigger eval: billing-audit skill
  Should-fire (10):
    [PASS] "Audit the invoice spec for false-pass patterns"
    [PASS] "Grade src/domains/billing/tests/billingLookup.api.spec.ts against the rubric"
    [FAIL] "Check whether the billing spec follows conventions" — did not fire
    ...
  Should-not-fire (10):
    [PASS] "Run the test suite and report failures"
    [PASS] "Port the shipping domain"
    [FAIL] "Look at the billing module" — fired when it shouldn't have
    ...

  Precision: 8/10  Recall: 9/10

A precision failure means the skill fires when it shouldn't, injecting guidance that is irrelevant or contradictory. A recall failure means the skill sits idle when it should help. The precision failures are the more damaging ones because they're invisible: the agent does something subtly off and nobody can point to why.

I run trigger-accuracy evals whenever I edit a skill's trigger description, or add a new skill that might compete with an existing one. Five minutes, and it catches router ambiguity before production.

Putting it together: the eval recipe

For each skill I keep a small eval suite at .agent/evals/<skill-name>/:

.agent/evals/billing-audit/
  meta.json           # skill version, last-run date, threshold targets
  prompts/            # 20-30 task prompts across difficulty levels
  assertions/         # per-prompt assertion sets; D: prefix = discriminator
  baselines/          # cached baseline outputs (re-run on skill change)
  trigger-eval.json   # the 20-prompt precision/recall battery

The eval runner fans out agents in parallel, one per prompt. Baselines use the constraint-injection block, and the runner diffs skill-on against skill-off over the assertion sets. The output is four numbers: overall skill lift, discriminator lift, trigger precision, trigger recall. A skill that fails to show discriminator lift goes into the revision queue.

Where it falls down

Discriminator design is the hardest part and the least scalable. You need to know what the specific wrong answer looks like, which means deep domain knowledge or a first-pass run of unguided agents to harvest failure modes. The pattern I use: run baseline agents on representative tasks before writing the skill, harvest the failure modes, and build discriminators from what actually went wrong. Coverage is retrospective. I catch wrong answers I've already seen, not novel ones. New failure modes become new discriminators after the next unguided run, never before.

The baseline constraint protocol requires trust in the agent runtime. It relies on the agent obeying the "do not read skill files" instruction rather than enforcing it at the filesystem level. A capable enough agent could reason its way around it. A read-only mount for the skills directory during baseline runs would be stronger, but I haven't needed it yet.

Evals also need maintenance. A discriminator targeting version-1 failure modes may miss version-2 failure modes entirely. I track eval suite version alongside skill version in meta.json and flag any divergence of more than one revision.

Takeaways

• The value of a skill is in the hard path it prevents. Design evals to measure that directly.
• Tag a subset of assertions as discriminators. Each is a trap for one specific tempting wrong answer. Track discriminator lift separately from overall pass rate.
• Baseline agents will find and use your skill files if the repo is present. Inject a "do not read skill files" constraint into every baseline context, and treat violations as eval failures, not data points.
• Run a separate trigger-accuracy eval (10 should-fire, 10 should-not-fire) whenever you change a skill's routing description. Precision failures are worse than recall failures because they're invisible.
• Harvest failure modes from unguided runs before writing the skill. Real wrong answers make better discriminator material than imagined ones.
• Keep eval suite version in lockstep with skill version. An eval targeting version-1 failure modes tells you nothing about version 2.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/21/eval-driven-development-for-ai-skills/"
license: "Free to reference with attribution"
title: "Testing the author: eval-driven development for AI skills"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 31"
stack: [agentic coding CLI, JSON eval specs, Playwright]

problem: "After shipping ~a dozen skills (versioned text files instructing an agent on recurring tasks: audit a spec, port a domain's tests, generate a defect report) to an agentic coding CLI, the author had only anecdote and vibe — no evidence which skills helped. Naive eval suites couldn't tell 'skill working' from 'skill impressive on softball prompts'."
thesis: "Build an eval harness around discriminator assertions that test for the absence of the single most tempting wrong answer rather than the presence of one correct answer, run it against an uncontaminated baseline, and report skill lift on discriminators separately from overall pass rate."

baseline_contamination:
  bug: "First eval used a control group (task with skill active vs without). The 'no skill' results came back suspiciously good; an hour in the transcripts revealed baseline agents were discovering the skill files on disk and reading them anyway. Measured lift was zero because every A/B was really skill-via-injection vs skill-via-exploration."
  root_cause: "Nothing forbade it: repo present, skill file readable, and a capable researching agent explores the codebase it lives in. Agent capability cuts both ways — a competent agent finds resources you didn't mean to provide, so you must explicitly revoke them."
  fix: "Inject an evalMode:baseline block into every baseline agent's system context with explicit constraints: do NOT read any file under .agent/skills/ or .agent/workflows/; if a skill/workflow file is discovered by accident, discard its contents and don't apply its guidance."
  enforcement: "Not a social contract — an explicit, logged constraint. A baseline agent that reads a skill file in defiance of the block fails the eval run; those transcripts are inspected individually rather than averaged into the data."

discriminator_assertions:
  definition: "An assertion built around the single most tempting wrong answer — the confident, structurally plausible one a naive agent gives most often. It asks 'did the agent avoid this specific trap', not 'did the agent produce the right answer'."
  worked_example: "Ordering domain — 'how many orders match status=pending?'. A parent order with three line items renders as four DOM rows (one parent, three children). A naive agent locator-counts every row and reports 4; the correct answer counts parent orders only. The canonical wrong answer is the inflated count."
  spec_shape: "JSON case D-04: prompt, expectedAnswerRange [1,3], and a discriminator object with forbiddenAnswers [4,5,6,7,8] plus a rationale. The assertion is expect(count).not.toBeOneOf([4,5,6,7,8]), NOT expect(count).toBe(2) — 'correct' has flexibility since counts vary with test data, but 'wrong in this specific way' does not."
  two_numbers: "Tag a subset of assertions as discriminators and report separately — overall pass rate (general quality criteria met) vs discriminator pass rate (specific failure modes the skill prevents were avoided). A skill that lifts overall pass rate but moves nothing on discriminator pass rate is a style guide, not a guardrail."

trigger_accuracy_eval:
  why: "Skill content is only half the problem. If the router never fires the skill, content is irrelevant; if it fires on unrelated tasks, you burn tokens and risk prompt contamination. Run a separate, cheap eval — ~5 minutes — to catch router ambiguity before production."
  format: "Balanced prompt set: 10 prompts that should activate the skill and 10 that should not, scored for precision and recall independently (e.g. Precision 8/10, Recall 9/10 for a billing-audit skill)."
  precision_vs_recall: "A precision failure = skill fires when it shouldn't, injecting irrelevant or contradictory guidance. A recall failure = skill sits idle when it should help. Precision failures are more damaging because they're invisible: the agent does something subtly off and nobody can point to why."
  when_to_run: "Whenever you edit a skill's trigger description, or add a new skill that might compete with an existing one."

eval_suite_layout:
  path: ".agent/evals/<skill-name>/ — e.g. .agent/evals/billing-audit/"
  files:
    - "meta.json — skill version, last-run date, threshold targets"
    - "prompts/ — 20-30 task prompts across difficulty levels"
    - "assertions/ — per-prompt assertion sets; D: prefix marks a discriminator"
    - "baselines/ — cached baseline outputs, re-run on skill change"
    - "trigger-eval.json — the 20-prompt precision/recall battery"
  runner: "Fans out agents in parallel, one per prompt; baselines use the constraint-injection block; the runner diffs skill-on against skill-off over the assertion sets. Output is four numbers — overall skill lift, discriminator lift, trigger precision, trigger recall. A skill that fails to show discriminator lift goes into the revision queue."

limitations:
  - "Discriminator design is the hardest, least scalable part — you must know what the specific wrong answer looks like. Coverage is retrospective: catches wrong answers already seen, not novel ones. The workflow is to run unguided baseline agents on representative tasks before writing the skill, harvest real failure modes, and build discriminators from what actually went wrong; new failure modes become new discriminators only after the next unguided run."
  - "The baseline constraint protocol trusts the agent runtime — it relies on the agent obeying 'do not read skill files' rather than enforcing it at the filesystem level. A capable enough agent could reason around it; a read-only mount of the skills directory during baseline runs would be stronger but hasn't been needed."
  - "Evals need maintenance: a discriminator targeting version-1 failure modes may miss version-2 ones. Track eval suite version alongside skill version in meta.json and flag any divergence of more than one revision."

use_when: "You ship multiple versioned skills/prompts to an agent runtime and need evidence — not vibes — that a given skill changes behavior on the hard path it's meant to guard."
avoid_when: "A single skill on easy, unambiguous tasks where floor-level competence (right sections, non-empty result, no runtime error) already passes regardless of the skill, and there is no tempting wrong answer to discriminate against."

takeaways:
  - "The value of a skill is in the hard path it prevents. Design evals to measure that directly."
  - "Tag a subset of assertions as discriminators. Each is a trap for one specific tempting wrong answer. Track discriminator lift separately from overall pass rate."
  - "Baseline agents will find and use your skill files if the repo is present. Inject a 'do not read skill files' constraint into every baseline context, and treat violations as eval failures, not data points."
  - "Run a separate trigger-accuracy eval (10 should-fire, 10 should-not-fire) whenever you change a skill's routing description. Precision failures are worse than recall failures because they're invisible."
  - "Harvest failure modes from unguided runs before writing the skill. Real wrong answers make better discriminator material than imagined ones."
  - "Keep eval suite version in lockstep with skill version. An eval targeting version-1 failure modes tells you nothing about version 2."

keywords: ["eval-driven development for AI skills", "discriminator assertions in agent evals", "baseline contamination skill files agent reads", "trigger-accuracy precision recall skill routing", "measuring skill lift agentic coding CLI"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).