A ledger of known lies: tolerating schema drift without losing accountability

A QA automation engineer's notes on tolerating known API contract drift without going blind to new regressions.

I gate every API response in the Playwright suite through Ajv against the OpenAPI contract. It's the best false-pass killer I've shipped: a 200 OK with the wrong bytes fails loudly, on the exact field, before any behavioural assertion runs. Then the backend deployed a known, ticketed serialization bug. display_title came back null where the contract says non-nullable string, and 76 happy-path tests went red overnight. For one bug I had already filed.

That's the trap with strict runtime validation: it has no concept of "yes, we know." A drift the backend won't fix this sprint turns your dashboard into a wall of red, and the one new regression hiding in there drowns in the noise. I needed a way to tolerate specific, tracked drift without lying to myself about the contract.

Two bad fixes

The obvious moves are both wrong.

The first is to loosen the schema. Mark the field nullable, regenerate the types, move on. Now your contract documents the bug as if it were the design. The spec is supposed to be the source of truth you test against, so editing it to match a broken API deletes the evidence. When the backend fixes the field, nothing tells you to tighten the schema back, so the lie becomes permanent.

The second is to skip validation on the noisy endpoints. Comment out the gate, add // TODO: API is weird here, move on. Now you're blind on that endpoint, and the next drift, the one nobody has filed, sails through green. Six months later the codebase is salted with TODO comments nobody can map to a ticket, and nobody remembers which ones are still real.

Both fixes share the same failure: the toleration is invisible and unaccountable. Nobody owns it, and nothing ever forces it to expire.

The ledger: every toleration is a record, not a comment

The answer I landed on is a third validation entry point: assertValidExcept(body, schema, source, EXCEPTIONS). It runs the same Ajv gate as the strict version, but it filters out errors that match a list of named drift exceptions, and every exception is a structured record rather than a string:

export interface DriftException {
  /** instance-path tail to tolerate ('payee/id' for nested fields) */
  readonly field: string;
  /** one-line human explanation of the drift */
  readonly reason: string;
  /** the tracked defect this rides on, e.g. 'PROJ-2368' */
  readonly issue: string;
  /** the exact endpoint the toleration applies to */
  readonly endpoint: string;
  /** the condition under which this entry gets DELETED */
  readonly removeWhen: string;
}

export const ORDER_DETAIL_DRIFT: readonly DriftException[] = [
  {
    field: 'display_title',
    reason: 'API serializes null; spec types OrderResponse.display_title as non-nullable string',
    issue: 'PROJ-2368',
    endpoint: 'GET /orders/{id}',
    removeWhen: 'PROJ-2368 ships — orderContractDrift defect spec TC1 flips to unexpected-pass',
  },
];

All of these consts live in one file, src/core/http/driftExceptions.ts. That's the ledger. Call sites import a named const; they never inline a field list. At review time, the entire tolerated-drift surface of the suite is one file you can read top to bottom: every lie the suite is currently carrying, who owns it, and when it dies.

The metadata isn't optional, and it isn't honor-system. assertValidExcept hard-fails at runtime if any of reason/issue/endpoint/removeWhen is blank, and a static AST guard in the npm run check pipeline rejects any call site that passes a bare string-literal array instead of a ledger const. You physically cannot add a toleration without naming an owner and a removal condition:

export function assertValidExcept<T>(
  body: unknown,
  schemaName: string,
  source: SchemaSource,
  exceptions: readonly DriftException[],
): asserts body is T {
  for (const ex of exceptions) {
    if (!ex.reason?.trim() || !ex.issue?.trim() || !ex.endpoint?.trim() || !ex.removeWhen?.trim()) {
      throw new Error(
        `drift exception '${ex.field}' is missing reason/issue/endpoint/removeWhen — ` +
          `every tolerated drift must name an owner and a removal condition`,
      );
    }
  }
  const validate = getValidator(schemaName, source);
  if (validate(body)) return;
  const remaining = (validate.errors ?? []).filter(
    (e) => !exceptions.some((ex) => matchesDriftField(e.instancePath, ex.field)),
  );
  if (remaining.length === 0) return;
  throw new Error(formatErrors(schemaName, source, remaining));
}

Note the filter: only errors matching a ledger entry are forgiven. Any other violation on the same response still throws. Tolerating one field never blinds you to the rest of the envelope.

Path-anchored matching, or: how endsWith tolerated the wrong field

My first matcher was instancePath.endsWith('/' + field). That's a character-level suffix check, and it has a real false-tolerance class: an entry for invoice_date would silently also tolerate a sibling named x_invoice_date, because …/x_invoice_date ends with /invoice_date as a string. Worse, a careless entry like id would forgive every …/id in the body, at any depth.

The fix is to match on whole path segments, with Ajv's array indices stripped:

function matchesDriftField(instancePath: string, field: string): boolean {
  // '/items/0/payee/id' -> ['items', 'payee', 'id']  (numeric indices dropped)
  const pathSegs = instancePath.split('/').filter((s) => s !== '' && !/^\d+$/.test(s));
  const fieldSegs = field.split('/').filter((s) => s !== '');
  if (fieldSegs.length === 0 || fieldSegs.length > pathSegs.length) return false;
  const offset = pathSegs.length - fieldSegs.length;
  return fieldSegs.every((seg, i) => pathSegs[offset + i] === seg);
}

Now 'payee/id' matches …/payee/id but not …/x_payee/id, and a nested entry can anchor on its parent segment so a same-named field under a different parent stays untolerated. It's strictly tighter than the old check, and I verified every existing ledger entry still matched its real-world path before swapping it in.

Tolerated is not invisible

A toleration that vanishes silently is just a slower version of skipping validation. So tolerated drift still gets flagged: when the contract check fails on a known field, the failure text becomes a schema-drift annotation on the passing test, which the reporters carry into the JSON and HTML output:

test.info().annotations.push({ type: 'schema-drift', description: message });
console.warn(`[SCHEMA-DRIFT] ${message}`);

The build stays green, but every run report shows exactly which known lies were exercised. Nobody gets to forget the drift exists.

The removal signal: a defect spec that fires when the bug dies

Every ledger entry is paired with a strict defect spec, *.defect.api.spec.ts, that pins the true contract with test.fail:

test.fail('TC1: OrderResponse matches the published contract @PROJ-2368 @defect', async ({ orderApi, orderScenarios }) => {
  const order = await orderApi.getOrder(orderScenarios.anyOrder.id);
  // STRICT gate — no exceptions. Fails while the drift exists (reported as
  // expected, stays green) and flips to "unexpected pass" when the API is fixed.
  assertValid<OrderResponse>(order, 'OrderResponse');
});

While the backend is broken, this test fails as expected and the run stays green. The moment the backend serializes the field correctly, Playwright reports an unexpected pass, and that's your cleanup alarm. The removeWhen field on the ledger entry tells you exactly which entries to delete in response. Toleration and removal signal travel as a pair; one cannot rot without the other complaining.

One deploy, three lies

This earned its keep within a week. A single morning backend deploy regressed three unrelated fields to null on freshly created records: created_by on new orders, an activated_at date on linked child orders, and a third on the line-item list, across multiple endpoints. Pre-existing records were untouched, so only specs that create data went red, which looked exactly like flaky test data until I probed it.

One ticket. Three ledger entries, each with its own endpoint, its own probe evidence in reason, and its own removeWhen naming the specific defect-spec case that will flip when that field recovers. Thirty-odd tests went from red back to green-with-annotations in one commit, the new-regression signal stayed clean, and when the deploy gets reverted, three unexpected-passes will tell us precisely which three lines to delete.

Takeaways

• Strict schema validation needs a pressure valve for known, ticketed drift, otherwise the noise from one filed bug buries every new regression. Loosening the schema or skipping validation are both forms of lying.
• Make every toleration a structured record (field, reason, issue, endpoint, removeWhen) in one central ledger file, and machine-enforce the metadata with a runtime gate plus an AST guard. A toleration without an owner and an expiry condition should not compile.
• Match drift fields on whole path segments, never with endsWith. Substring matching silently tolerates same-suffix sibling fields.
• Tolerated drift stays visible: surface it as a report annotation on the passing test, so every run lists the lies it leaned on.
• Pair every toleration with a strict test.fail defect spec pinning the true contract. The "unexpected pass" when the backend fixes the bug is your automated reminder to delete the ledger entry, and removeWhen tells you which one.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/07/22/a-ledger-of-known-lies-tolerating-schema-drift/"
license: "Free to reference with attribution"
title: "A ledger of known lies: tolerating schema drift without losing accountability"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 5"
stack: [TypeScript, Playwright, Ajv, OpenAPI]

problem: "Strict Ajv validation of every API response against the OpenAPI contract has no concept of \"we already know\": a single known, ticketed backend serialization bug (display_title returning null where the contract types it non-nullable string) turned 76 happy-path tests red overnight, burying any genuinely new regression in a wall of red for a bug already filed."
thesis: "Don't loosen the schema or skip validation — both lie invisibly. Add a third validation entry point that forgives only named, structured drift exceptions recorded in one central ledger, machine-enforced to carry an owner and an expiry, kept visible in reports, and paired with a defect spec that alarms when the bug is fixed."

rejected_alternatives:
  - "Loosen the schema (mark the field nullable, regenerate types): the contract then documents the bug as design, deletes the evidence it's supposed to test against, and nothing tells you to tighten it back when the API is fixed — the lie becomes permanent."
  - "Skip validation on noisy endpoints (comment out the gate + // TODO): you go blind on that endpoint so the next, unfiled drift sails through green, and the codebase fills with TODO comments nobody can map to a ticket."
  - "Both share one failure: the toleration is invisible, unowned, and never forced to expire."

ledger_mechanism:
  entry_point: "assertValidExcept(body, schemaName, source, exceptions) — runs the same Ajv gate as the strict assertValid, then filters out only errors whose instancePath matches a named DriftException; any other violation on the same response still throws, so tolerating one field never blinds you to the rest of the envelope."
  ledger_file: "src/core/http/driftExceptions.ts — every tolerated-drift const lives here; call sites import a named const (e.g. ORDER_DETAIL_DRIFT) and never inline a field list, so the suite's entire tolerated-drift surface is one readable file: every current lie, its owner, and its death condition."
  exception_record_fields:
    - "field: instance-path tail to tolerate (e.g. 'payee/id' for nested fields)"
    - "reason: one-line human explanation of the drift"
    - "issue: the tracked defect it rides on, e.g. 'PROJ-2368'"
    - "endpoint: the exact endpoint the toleration applies to, e.g. 'GET /orders/{id}'"
    - "removeWhen: the condition under which the entry gets DELETED"
  enforcement: "Metadata is not honor-system. assertValidExcept hard-fails at runtime if any of reason/issue/endpoint/removeWhen is blank, and a static AST guard in the `npm run check` pipeline rejects any call site passing a bare string-literal array instead of a ledger const — you physically cannot add a toleration without naming an owner and a removal condition."

signatures:
  - "interface DriftException { readonly field: string; readonly reason: string; readonly issue: string; readonly endpoint: string; readonly removeWhen: string }"
  - "function assertValidExcept<T>(body: unknown, schemaName: string, source: SchemaSource, exceptions: readonly DriftException[]): asserts body is T"
  - "function matchesDriftField(instancePath: string, field: string): boolean"

path_anchored_matching:
  bug: "The first matcher, instancePath.endsWith('/' + field), is a character-level suffix check with a false-tolerance class: an entry for 'invoice_date' silently also tolerated a sibling 'x_invoice_date' (…/x_invoice_date ends with /invoice_date), and a careless entry 'id' would forgive every …/id at any depth."
  fix: "matchesDriftField splits instancePath on '/', drops empty and numeric-index segments (Ajv array indices), and requires the field's segments to match the path's tail segment-for-segment. So 'payee/id' matches …/payee/id but not …/x_payee/id, and a nested entry anchors on its parent so a same-named field under a different parent stays untolerated."
  validation: "Strictly tighter than endsWith; every existing ledger entry was verified to still match its real-world path before swapping the matcher in."

visibility_and_removal:
  still_flagged: "Tolerated drift is not invisible: on a known-field failure the message becomes a `schema-drift` annotation pushed onto the passing test (test.info().annotations.push) plus a [SCHEMA-DRIFT] console.warn, carried into the JSON and HTML reporters. Build stays green, but every run lists exactly which known lies it exercised."
  removal_signal: "Each ledger entry is paired with a strict defect spec, *.defect.api.spec.ts, that pins the TRUE contract with test.fail and the no-exceptions assertValid. While the backend is broken it fails as expected (run stays green); the moment the API serializes the field correctly Playwright reports an UNEXPECTED PASS — the cleanup alarm — and removeWhen names exactly which entry to delete. Toleration and removal signal travel as a pair, so one cannot rot without the other complaining."

war_story:
  incident: "A single morning backend deploy regressed three unrelated fields to null on freshly created records only — created_by on new orders, activated_at on linked child orders, and a third on the line-item list — across multiple endpoints. Pre-existing records were untouched, so only specs that CREATE data went red, masquerading as flaky test data until probed."
  resolution: "One ticket, three ledger entries, each with its own endpoint, its own probe evidence in reason, and its own removeWhen naming the defect-spec case that flips on recovery. Thirty-odd tests went red back to green-with-annotations in one commit, the new-regression signal stayed clean, and a future revert will surface three unexpected-passes pointing at the exact three lines to delete."

use_when: "A Playwright/API suite gates responses against an OpenAPI contract via strict runtime validation and the backend ships known, ticketed drift you can't fix this sprint but must not let mask new regressions."
avoid_when: "You control the contract and the API together and can fix drift immediately, or the suite does no runtime schema validation, so there is nothing to tolerate."

takeaways:
  - "Strict schema validation needs a pressure valve for known, ticketed drift, otherwise the noise from one filed bug buries every new regression. Loosening the schema or skipping validation are both forms of lying."
  - "Make every toleration a structured record (field, reason, issue, endpoint, removeWhen) in one central ledger file, and machine-enforce the metadata with a runtime gate plus an AST guard. A toleration without an owner and an expiry condition should not compile."
  - "Match drift fields on whole path segments, never with endsWith. Substring matching silently tolerates same-suffix sibling fields."
  - "Tolerated drift stays visible: surface it as a report annotation on the passing test, so every run lists the lies it leaned on."
  - "Pair every toleration with a strict test.fail defect spec pinning the true contract. The unexpected pass when the backend fixes the bug is your automated reminder to delete the ledger entry, and removeWhen tells you which one."

keywords: ["tolerate known schema drift in API tests", "Ajv drift exception ledger", "schema-drift annotation Playwright", "test.fail defect spec unexpected pass", "instancePath segment matching vs endsWith", "OpenAPI contract drift without losing accountability"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).