Lint rules regex can’t write: enforcing test conventions with AST guards

A QA automation engineer's notes on enforcing test conventions the type checker and regex can't, with the TypeScript AST.

The suite has a handful of conventions that every spec must follow, and for months they lived where conventions go to die: in a standards doc, enforced by code review. Things got missed. New specs copied old mistakes. So I tried the obvious next step, grep-based pre-commit checks, and they failed in interesting ways. What finally worked was a set of four small standalone scripts that parse every spec with the TypeScript compiler API and check the rules structurally, on the AST. They run as part of npm run check, right next to tsc and ESLint, and a violation blocks the merge.

Here is why text matching wasn't enough, how the guards work, and the one implementation idiom that let four guards share a helper module without stepping on each other.

The conventions I needed to enforce

Three rules, all of them invisible to the type checker:

1. One import barrel. Specs import test and expect only from @core/fixtures, the barrel I maintain that binds the auto-fixtures and re-exports custom annotation functions, never from @playwright/test directly. This includes type-only imports: the barrel re-exports Page, Locator, APIRequestContext, and APIResponse, so even import type { Page } from '@playwright/test' is banned. A spec that bypasses the barrel silently loses the fixtures and reporting hooks the barrel wires in.

2. The annotation triad. Every runnable case, test(...), test.fail(...), or test.only(...) with a literal title and a function body, must call trackingRef(), testType(), and severity() so each result is traceable to a ticket like PROJ-6354. A @needs-ticket tag in the title is the escape hatch for cases with no sourceable ticket, and it waives only the trackingRef() call, never the other two.

3. The defect fence. A plain test() in a *.defect.* file is forbidden unless it's positively classified as a pass-today case. Known-defect cases must be test.fail(). This is what makes bug-tracker conversions stick: once a fixed defect's tracker quietly wears a plain test() again, the gate goes red.

Why grep can't do this

I genuinely tried regex first. Three things killed it:

• Template-literal titles. Half our titles are back-ticked with interpolation: test(`TC3: ${label} rejects a stale token @PROJ-8907`, …). A line regex looking for test(' misses these entirely, and one loose enough to catch them starts matching things that aren't test declarations.
• Multi-line calls. Prettier happily puts the title on one line and the body on the next. A line-anchored regex never sees the callee and the title together, so it can't tell test.skip('title') (a bodiless placeholder, exempt) from test('title', async () => { (runnable, must carry the triad).
• JSDoc prose. The spec headers talk about the conventions: comments contain the literal strings test.fail() and trackingRef(. A text scan false-positives on every one of them. The AST never has this problem, because comments aren't CallExpression nodes.

Regex gives you a choice between missing violations and crying wolf. Either one kills trust in the gate.

Parsing specs with the compiler API

You don't need a full ts.Program for this. ts.createSourceFile parses a single file into an AST with no project context, in milliseconds. The shared helpers are small:

// src/testing/guard-ast.ts — pure helpers, zero side effects
import ts from 'typescript';
import { readFileSync } from 'node:fs';

export function parseSpec(file: string): ts.SourceFile {
  return ts.createSourceFile(file, readFileSync(file, 'utf8'), ts.ScriptTarget.Latest, true);
}

export type CalleeKind = 'test' | 'test.fail' | 'test.fixme' | 'test.skip' | 'test.only' | 'other';

export function calleeKind(e: ts.Expression): CalleeKind {
  if (ts.isIdentifier(e) && e.text === 'test') return 'test';
  if (ts.isPropertyAccessExpression(e) && ts.isIdentifier(e.expression) && e.expression.text === 'test') {
    return `test.${e.name.text}` as CalleeKind;
  }
  return 'other';
}

/** A call whose first arg is a string/template title is a case declaration. */
export function isCaseDecl(call: ts.CallExpression): boolean {
  const a = call.arguments[0];
  return !!a && (ts.isStringLiteralLike(a) || ts.isTemplateExpression(a));
}

/** Reconstruct a title from a string OR template literal — static tags survive interpolation. */
export function titleOf(a: ts.Expression): string {
  if (ts.isStringLiteralLike(a)) return a.text;
  if (ts.isTemplateExpression(a)) return a.head.text + a.templateSpans.map((s) => s.literal.text).join('');
  return '';
}

Look at titleOf: for a template literal it concatenates the head and the span tails, so `TC3: ${label} … @needs-ticket` still yields a string containing @needs-ticket. The static tags survive interpolation, which is exactly what regex couldn't give me.

Classifying a test call

The annotation-triad guard walks every node, and when it finds a runnable case declaration, it collects the identifier calls inside the body and diffs them against the required set:

const REQUIRED = ['testType', 'severity', 'trackingRef'] as const;

function scanTriad(sf: ts.SourceFile): Violation[] {
  const violations: Violation[] = [];
  const visit = (n: ts.Node): void => {
    if (ts.isCallExpression(n) && isCaseDecl(n)) {
      const kind = calleeKind(n.expression);
      const runnable = kind === 'test' || kind === 'test.fail' || kind === 'test.only';
      const body = caseBody(n); // the arrow/function arg, if any
      if (runnable && body) {
        const title = titleOf(n.arguments[0]);
        const calls = identifierCallsIn(body); // Set of fn names called in the body
        const missing = REQUIRED.filter(
          (c) => !calls.has(c) && !(c === 'trackingRef' && /@needs-ticket\b/.test(title)),
        );
        if (missing.length) violations.push(at(sf, n, title, missing));
      }
    }
    ts.forEachChild(n, visit);
  };
  visit(sf);
  return violations;
}

The exemptions are as deliberate as the rule. test.skip and test.fixme declarations are out of scope, since shelved coverage carries provenance in its reason string instead. So is a case with a non-literal title, like test(tc.name, …) in a parametrized loop. That last one is an intentional under-approximation, because a false positive in a merge gate is worse than a miss, and the triad usually lives inside the loop anyway.

When the guard fires, the message tells you exactly what to do:

✗ src/domains/billing/tests/invoiceCreate.api.spec.ts:48:3
    case is missing required annotation call(s): severity(), trackingRef() (or a @needs-ticket title tag)
    title: "TC4: rejects an invoice with a negative line total @regression"
    FIX: add the missing call(s) as first lines of the test body, order trackingRef → testType → severity.

check-spec-standards: 0 import + 1 annotation violation(s) in 1 file(s).

Exit codes matter too: 0 clean, 1 violation, and 2 for an internal error. A crash in the guard itself must never read as "clean".

Four guards, one helper module, no import side effects

The first guard I wrote ended with process.exit(main()) at the top level. That worked fine until guard number two wanted to reuse its AST helpers, and importing the file ran the first guard and exited the process. The fix is a five-line idiom:

import { fileURLToPath, pathToFileURL } from 'node:url';

export function isMainModule(moduleUrl: string): boolean {
  const entry = process.argv[1];
  if (!entry) return false;
  return moduleUrl === pathToFileURL(entry).href || fileURLToPath(moduleUrl) === entry;
}

// at the bottom of every guard:
if (isMainModule(import.meta.url)) process.exit(main());

It's the ESM equivalent of Python's if __name__ == '__main__'. With every guard gated this way, the duplicated helpers collapsed into one side-effect-free guard-ast.ts that all four import, and any guard can now be imported by a unit test without detonating.

Why not a custom ESLint rule?

I considered it, and it's a legitimate choice. ESLint rules get you editor squiggles at typing time and a familiar suppression story. But they also cost you: a plugin package to build and version, ESLint's visitor and context API to learn, and rule logic squeezed into a framework designed for per-file style checks.

Standalone scripts gave me the opposite trade. They're trivially debuggable with npx tsx src/testing/check-spec-standards.ts, they hand me the full compiler API, they do multi-file rules when I need them, and they let me control the failure messages completely. Since the scripts run in the same npm run check gate as ESLint,

"check": "npm run typecheck && npm run lint && npm run guard:defect-tests && npm run guard:spec-standards && ..."

a violation blocks the merge either way. I gave up squiggles and kept simplicity. For three or four project-specific rules, that's the right side of the trade. If the rule count keeps growing, it would be worth revisiting.

Takeaways

• If a convention matters, a machine should enforce it. Review-only rules decay; a gate that goes red doesn't.
• Regex can't see structure. Template-literal titles, multi-line calls, and prose in comments break text matching in both directions. The AST sees real CallExpressions and nothing else.
• ts.createSourceFile is cheap. You don't need a type checker or a full program, since single-file parsing is fast enough to walk a few hundred specs on every check.
• Prefer under-approximation in a merge gate. Exempt the genuinely ambiguous cases (dynamic titles, bodiless placeholders); a gate that false-positives gets bypassed.
• Gate every script entry point with isMainModule(import.meta.url). It's what lets guards share helpers and be unit-tested without import-time side effects.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/04/lint-rules-regex-cant-write-ast-convention-guards/"
license: "Free to reference with attribution"
title: "Lint rules regex can't write: enforcing test conventions with AST guards"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 16"
stack: [TypeScript, "TypeScript compiler API", Playwright, ESLint, Node.js, ESM]

problem: "A handful of suite-wide spec conventions lived in a standards doc enforced only by code review — so they decayed: things got missed and new specs copied old mistakes. Grep/regex pre-commit checks failed too, forced to choose between missing violations and crying wolf, either of which kills trust in the gate."
thesis: "Enforce structural test conventions with four small standalone scripts that parse every spec via ts.createSourceFile and check rules on the AST (not text). They run in npm run check beside tsc and ESLint, and a violation blocks the merge."

conventions_enforced:
  one_import_barrel: "Specs import test and expect only from @core/fixtures (the barrel that binds auto-fixtures and re-exports custom annotation functions), never from @playwright/test directly. Bans type-only imports too — the barrel re-exports Page, Locator, APIRequestContext, APIResponse — so even import type { Page } from '@playwright/test' is forbidden. Bypassing the barrel silently loses fixtures and reporting hooks."
  annotation_triad: "Every runnable case — test(...), test.fail(...), or test.only(...) with a literal title and a function body — must call trackingRef(), testType(), and severity() so each result traces to a ticket like PROJ-6354. A @needs-ticket tag in the title is the escape hatch waiving ONLY the trackingRef() call, never the other two."
  defect_fence: "A plain test() in a *.defect.* file is forbidden unless positively classified as a pass-today case; known-defect cases must be test.fail(). This makes bug-tracker conversions stick — when a fixed defect's tracker quietly wears a plain test() again, the gate goes red."

why_regex_fails:
  - "Template-literal titles: half the titles are back-ticked with interpolation; a line regex for test(' misses them, and a looser one matches non-test declarations."
  - "Multi-line calls: Prettier puts the title on one line and the body on the next, so a line-anchored regex never sees callee and title together — can't distinguish test.skip('title') (bodiless, exempt) from test('title', async () => { (runnable, must carry the triad)."
  - "JSDoc prose: spec header comments contain the literal strings test.fail() and trackingRef(, so a text scan false-positives on each — the AST never has this problem because comments aren't CallExpression nodes."

mechanism:
  parser: "ts.createSourceFile parses a single file into an AST in milliseconds with no project context — no full ts.Program or type checker needed; fast enough to walk a few hundred specs every check."
  shared_helpers: "src/testing/guard-ast.ts — pure, zero-side-effect helpers all four guards import: parseSpec, calleeKind (returns 'test' | 'test.fail' | 'test.fixme' | 'test.skip' | 'test.only' | 'other'), isCaseDecl (first arg is a string/template title), titleOf."
  title_reconstruction: "titleOf concatenates a template literal's head text and span literal tails, so `TC3: ${label} … @needs-ticket` still yields a string containing @needs-ticket — static tags survive interpolation, which is what regex couldn't give."
  triad_scan: "scanTriad walks every node; on a runnable case (kind test, test.fail, or test.only) with a body it diffs REQUIRED = ['testType','severity','trackingRef'] against identifierCallsIn(body), exempting trackingRef when /@needs-ticket\\b/ matches the title."
  exemptions: "test.skip and test.fixme are out of scope (provenance lives in their reason string); non-literal titles like test(tc.name, …) in parametrized loops are an intentional under-approximation — a false positive in a merge gate is worse than a miss, and the triad usually lives inside the loop anyway."
  output: "Failure message names the file:line:col, the missing call(s), the offending title, and a FIX line giving call order trackingRef → testType → severity, plus a summary like 'check-spec-standards: 0 import + 1 annotation violation(s) in 1 file(s).'"
  exit_codes: "0 clean, 1 violation, 2 internal error — a crash in the guard itself must never read as clean."

isMainModule_idiom:
  bug: "The first guard ended with top-level process.exit(main()); when a second guard imported its AST helpers, the import RAN the first guard and exited the process."
  fix: "A five-line isMainModule(import.meta.url) helper (compares against process.argv[1] via pathToFileURL/fileURLToPath) — the ESM equivalent of Python's if __name__ == '__main__'. Gating every guard's process.exit(main()) behind it let the duplicated helpers collapse into one side-effect-free guard-ast.ts that any guard or unit test can import without detonating."

signatures:
  - "export function parseSpec(file: string): ts.SourceFile"
  - "export function calleeKind(e: ts.Expression): CalleeKind"
  - "export function isCaseDecl(call: ts.CallExpression): boolean"
  - "export function titleOf(a: ts.Expression): string"
  - "export function isMainModule(moduleUrl: string): boolean"

rejected_alternative: "A custom ESLint rule — legitimate, and it buys editor squiggles at typing time plus a familiar suppression story — but costs a plugin package to build and version, ESLint's visitor/context API to learn, and rule logic squeezed into a per-file-style framework. Standalone scripts are trivially debuggable (npx tsx src/testing/check-spec-standards.ts), hand you the full compiler API, do multi-file rules, and fully control failure messages. The trade: gave up squiggles, kept simplicity — right for three or four project-specific rules; revisit if the rule count keeps growing."

use_when: "A test suite has a few project-specific structural conventions (import barrels, mandatory annotation calls, defect-file classification) that the type checker and review can't reliably enforce."
avoid_when: "The rule count grows large enough that a versioned ESLint plugin's editor integration and suppression story outweigh standalone-script simplicity."

takeaways:
  - "If a convention matters, a machine should enforce it. Review-only rules decay; a gate that goes red doesn't."
  - "Regex can't see structure. Template-literal titles, multi-line calls, and prose in comments break text matching in both directions; the AST sees real CallExpressions and nothing else."
  - "ts.createSourceFile is cheap. You don't need a type checker or a full program — single-file parsing is fast enough to walk a few hundred specs on every check."
  - "Prefer under-approximation in a merge gate. Exempt genuinely ambiguous cases (dynamic titles, bodiless placeholders); a gate that false-positives gets bypassed."
  - "Gate every script entry point with isMainModule(import.meta.url). It lets guards share helpers and be unit-tested without import-time side effects."

keywords: ["enforce test conventions with TypeScript AST", "ts.createSourceFile lint rule", "why regex can't lint test files", "AST guard merge gate Playwright", "isMainModule ESM main module check", "custom ESLint rule vs standalone script"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).