Reconcile, don’t regenerate: stable IDs for AI-generated test docs

A QA automation engineer's notes on giving AI-generated test docs a stable identity so each run reconciles the diff instead of regenerating everything.

The first version of my AI test-doc generator failed in the worst way: by working. Feed it a changed ticket, get a fresh set of test cases back, push them to the test-case management tool. Within two weeks I had three problems. Human edits to step wording were silently overwritten on the next ticket update. The review queue showed 40-plus "changed" tests that were structurally identical to the previous version. And a downstream link between a test case and an automated spec broke because the generated doc's internal identifier had been reassigned. I had built a generator. What I needed was a reconciler.

The difference is not subtle. A generator asks "what should exist?" and produces it. A reconciler asks "what already exists, what changed, and what is the minimum mutation to stay current?" Reconciliation needs one thing a generator can never give it: a stable identity for each generated test that survives ticket edits.

Why counters fail and content hashes aren't enough

The obvious identity scheme is a counter: test 1, test 2. Reorder the output, add a scenario in the middle, or regenerate from scratch and the counter reassigns. Every downstream link, every manual annotation, every audit trail breaks.

The next obvious scheme is a content hash. Hash the step text, use that as the ID. Change one word and the identity is gone. The record that a human approved this test, that it maps to an automated spec, that it has been executed against the last three releases is gone with it. A content hash is a change detector, not an identity.

What works is an identity derived from meaning, from the conceptual slot a test occupies in the coverage space, not from its exact wording or list position. I call these stable IDs.

Computing a stable ID

A stable ID is a short deterministic hash of the semantic inputs that define what a test covers, independent of how step wording evolves. For this pipeline, those inputs are:

• The target folder, the feature area the test lives in (e.g., billing/invoices)
• The tickets it covers, sorted and deduplicated (e.g., PROJ-7974,PROJ-7179)
• The normalized title, lowercased, punctuation stripped, whitespace collapsed

The hash is computed once at generation time and never recomputed from step text. Steps can be reworded, added, or removed and the stable ID does not move. The only way to get a new stable ID is to move the test to a different folder, change which tickets it covers, or fundamentally rename it.

import { createHash } from 'node:crypto';

/** Inputs that define a test's conceptual identity — never the step text. */
interface StableIdInputs {
  folder: string;          // e.g. "billing/invoices"
  ticketKeys: string[];    // e.g. ["PROJ-7974", "PROJ-7179"]
  normalizedTitle: string; // lowercased, punctuation stripped, whitespace collapsed
}

function normalizeTitle(raw: string): string {
  return raw.toLowerCase().replace(/[^\w\s]/g, '').replace(/\s+/g, ' ').trim();
}

function computeStableId(inputs: StableIdInputs): string {
  const sorted = [...inputs.ticketKeys].sort().join(',');
  const seed = `${inputs.folder}|${sorted}|${normalizeTitle(inputs.normalizedTitle)}`;
  return createHash('sha256').update(seed).digest('hex').slice(0, 16);
}

/** Content hash is separate — it detects whether step text has actually changed. */
function computeContentHash(steps: string[]): string {
  return createHash('sha256').update(steps.join('\n')).digest('hex').slice(0, 16);
}

The two hashes do different jobs. The stable ID is the identity. It follows the test across edits and must be returned verbatim by every prompt. The content hash is the change detector. It tells the planner whether step text has actually moved, which gates whether a re-push is needed at all.

The planner's five verdicts

Before any model call, a deterministic planner reconciles the generated-doc database against the current tracker state. It emits one verdict per test:

• keep: stable ID present, content hash unchanged, ticket unchanged. Nothing to do.
• stale: stable ID present, but the ticket has been updated since the content hash was last set. Queue for a diff-aware prompt.
• orphaned: stable ID present, but the ticket it covers has been closed, superseded, or removed from scope. Flag for human review, do not auto-delete.
• gap: a ticket in scope has no generated tests at all, or fewer than the minimum. Queue for initial generation.
• bootstrap: stable ID exists in the pipeline's records but cannot be found in the test-case management tool (deleted externally, or the tool was reset). Re-push the last known content.

type Verdict = 'keep' | 'stale' | 'orphaned' | 'gap' | 'bootstrap';

interface PlanEntry {
  stableId: string;
  verdict: Verdict;
  reason: string;
}

The planner is pure deterministic logic with no model calls. On about 300 generated tests covering 161 tracker items, a typical sprint re-plan classified 241 as keep, 31 as stale, 4 as orphaned, 18 as gap, and 6 as bootstrap. Only the 49 non-keep entries went on to a model call.

The diff-aware prompt

The stale entries are the bulk of the work in an ongoing pipeline, and they go through a diff-aware prompt with strict constraints:

1. Preserve unchanged steps verbatim. If a step's assertion is unaffected by the ticket change, the wording must not change, not "improved", not "clarified". This is what makes human annotations survive.
2. Keep the step count stable unless the ticket change explicitly adds or removes a scenario. A wording-fix ticket must not silently expand a test from 6 steps to 9.
3. Return the same stable ID. Models like to "helpfully" regenerate a cleaner identifier. In the schema the stable ID field is read-only, so the prompt cannot omit it or change it.
4. Tag every step's change disposition. Each step carries one of unchanged | updated | added | removed. This makes the diff human-readable and makes the planner's next run exact: unchanged steps carry their previous content hash forward.

{
  "stableId": "a3f9c21b4e8d07f1",
  "contentHash": "9b2e5a7c1d4f83a0",
  "title": "Invoice total recalculates when a line item quantity is updated",
  "steps": [
    { "seq": 1, "text": "Navigate to an open order with at least two line items.", "change": "unchanged" },
    { "seq": 2, "text": "Edit the quantity of the first line item to a value that changes the subtotal.", "change": "updated" },
    { "seq": 3, "text": "Confirm the invoice total reflects the new subtotal within the same session.", "change": "unchanged" }
  ]
}

The content hash in the output is computed from the new step texts and compared to the stored one. If they match, meaning the ticket changed but the steps still correctly describe the behavior, nothing is pushed to the test-case management tool. That suppresses the noisiest class of phantom updates: ticket typo fixes that do not change what the test checks.

Where it falls down

The stable-ID scheme needs the identity inputs to be stable themselves, and sometimes they aren't. A folder restructure, say splitting billing into billing/invoices and billing/payments, invalidates a whole family of stable IDs. I handle that with an explicit migration script that repoints them before the next plan run. Manual effort, not automatic.

The "preserve unchanged steps verbatim" rule is harder to enforce than to state. Models occasionally paraphrase a step that was already unambiguous, not wrong, just different, which trips the content hash for no reason. I run a post-processing pass that normalizes whitespace and punctuation and promotes a candidate step back to unchanged when the semantic content matches. A small reconciler sitting on top of the reconciler.

The planner's orphaned verdict also demands a policy decision a machine cannot make. A closed ticket often still has test cases worth keeping, because the behavior didn't disappear, the ticket just reached resolution. I flag orphans, route them to a human review queue, and never auto-delete.

Takeaways

• Identity must be derived from meaning, not assigned by a counter or taken from content. A stable ID hashed from folder, tickets, and normalized title survives step rewording and survives regeneration runs. Counters break on reorder, content hashes break on any edit.
• Keep two hashes separate. The stable ID is the identity and never changes with step text. The content hash is the change detector and updates with every step edit. Conflate them and you get either everything re-pushing or history silently breaking.
• A deterministic planner before any model call cuts the noise sharply. On about 300 tests, planning classified over 80% as keep in a typical sprint. The model call is expensive, the planner is cheap arithmetic.
• Diff-aware prompts must name the preservation constraint explicitly. "Minimize changes" is not enough. Unchanged steps come back verbatim, step count is stable unless the ticket explicitly adds a scenario, and the stable ID field is read-only.
• Suppress re-pushes by comparing content hashes after generation. A ticket typo fix that doesn't change what the test checks should not flood the test-case management tool. Gate on the hash, not on whether the source ticket changed.
• Human annotations survive because the identity does. Approval records, automation links, and execution history attach to a stable ID that predates any individual run. Without it, every regeneration is a clean slate and the curation work was for nothing.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/01/reconcile-dont-regenerate/"
license: "Free to reference with attribution"
title: "Reconcile, don't regenerate: stable IDs for AI-generated test docs"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 41"
stack: [TypeScript, "Node crypto (SHA-256)", JSON, "test-case management tool", "issue tracker"]

problem: "An AI test-doc generator that regenerates fresh test cases from a changed ticket silently overwrote human edits to step wording, flooded the review queue with 40+ structurally-identical 'changed' tests, and broke a link between a test case and an automated spec because the generated doc's internal identifier was reassigned. A generator answers 'what should exist?'; what was needed was a reconciler answering 'what exists, what changed, what is the minimum mutation?'"
thesis: "Give every generated test a stable identity derived from its conceptual coverage slot (not its wording or list position), then reconcile against existing state instead of regenerating. A deterministic planner emits a per-test verdict and only non-keep entries reach an expensive model call."

stable_id_scheme:
  definition: "A short deterministic hash of the semantic inputs that define WHAT a test covers, independent of how step wording evolves. Computed once at generation time, never recomputed from step text."
  inputs:
    - "folder: the feature area the test lives in (e.g. billing/invoices)"
    - "ticketKeys: the tickets it covers, sorted and deduplicated (e.g. PROJ-7974,PROJ-7179)"
    - "normalizedTitle: lowercased, punctuation stripped (regex [^\\w\\s]), whitespace collapsed, trimmed"
  hash: "seed = folder|sortedTickets|normalizedTitle, then SHA-256, first 16 hex chars"
  invariance: "Steps can be reworded, added, or removed and the stable ID does not move. A new stable ID only results from moving folder, changing which tickets are covered, or fundamentally renaming the test."
  rejected_schemes:
    - "Counter (test 1, test 2): reassigns on reorder, mid-list insertion, or scratch regeneration; breaks every downstream link, annotation, and audit trail."
    - "Content hash of step text: a change DETECTOR, not an identity — one reworded word destroys the identity and with it the approval record, spec mapping, and execution history."
  two_hashes: "stableId is the identity (returned verbatim by every prompt, never recomputed from steps). A separate computeContentHash(steps) over steps joined by newline is the change detector that gates whether a re-push is needed."

signatures:
  - "interface StableIdInputs { folder: string; ticketKeys: string[]; normalizedTitle: string }"
  - "function computeStableId(inputs: StableIdInputs): string"
  - "function computeContentHash(steps: string[]): string"
  - "type Verdict = 'keep' | 'stale' | 'orphaned' | 'gap' | 'bootstrap'"
  - "interface PlanEntry { stableId: string; verdict: Verdict; reason: string }"

planner:
  nature: "Pure deterministic logic, no model calls; runs before any model call to reconcile the generated-doc database against current tracker state, emitting one verdict per test."
  verdicts:
    keep: "stable ID present, content hash unchanged, ticket unchanged — nothing to do."
    stale: "stable ID present but ticket updated since content hash last set — queue for diff-aware prompt; the bulk of ongoing work."
    orphaned: "stable ID present but its ticket closed, superseded, or out of scope — flag for human review, never auto-delete."
    gap: "a ticket in scope has no generated tests, or fewer than the minimum — queue for initial generation."
    bootstrap: "stable ID in pipeline records but not found in the test-case management tool (deleted externally or tool reset) — re-push last known content."
  worked_numbers: "~300 generated tests covering 161 tracker items; a typical sprint re-plan classified 241 keep, 31 stale, 4 orphaned, 18 gap, 6 bootstrap — only the 49 non-keep entries reached a model call (>80% keep)."

diff_aware_prompt:
  constraints:
    - "Preserve unchanged steps verbatim — no 'improving' or 'clarifying'; this is what makes human annotations survive."
    - "Keep step count stable unless the ticket explicitly adds or removes a scenario (a wording-fix ticket must not expand 6 steps to 9)."
    - "Return the same stable ID — the field is read-only in the schema so the model cannot omit or 'helpfully' regenerate it."
    - "Tag every step's change disposition as unchanged | updated | added | removed, making the diff human-readable and the next plan exact (unchanged steps carry their previous content hash forward)."
  output_shape: "JSON with stableId, contentHash, title, and steps[] each carrying seq, text, and change disposition."
  re_push_suppression: "The output content hash (from new step texts) is compared to the stored one; on a match — ticket changed but steps still correct — nothing is pushed, suppressing the noisiest phantom updates (ticket typo fixes that don't change what the test checks)."

limitations:
  - "Stable IDs require stable identity inputs. A folder restructure (splitting billing into billing/invoices and billing/payments) invalidates a whole family of IDs; handled by an explicit manual migration script that repoints them before the next plan run — not automatic."
  - "'Preserve verbatim' is hard to enforce: models paraphrase already-unambiguous steps, tripping the content hash for nothing. A post-processing pass normalizes whitespace/punctuation and promotes a candidate back to unchanged on semantic match — a reconciler on top of the reconciler."
  - "The orphaned verdict needs a human policy call: a closed ticket often still has tests worth keeping because the behavior persists; orphans are routed to a human review queue, never auto-deleted."

use_when: "An LLM pipeline repeatedly regenerates documents (test cases, specs) that accumulate human curation, downstream links, and audit history across edits to a changing source."
avoid_when: "One-shot generation with no human edits, no downstream links, and no need to track change over time — there is nothing to reconcile against."

takeaways:
  - "Identity must be derived from meaning, not assigned by a counter or taken from content. A stable ID hashed from folder, tickets, and normalized title survives step rewording and regeneration runs; counters break on reorder, content hashes break on any edit."
  - "Keep two hashes separate. The stable ID is the identity and never changes with step text; the content hash is the change detector and updates with every step edit. Conflate them and you get either everything re-pushing or history silently breaking."
  - "A deterministic planner before any model call cuts the noise sharply. On ~300 tests, planning classified over 80% as keep in a typical sprint; the model call is expensive, the planner is cheap arithmetic."
  - "Diff-aware prompts must name the preservation constraint explicitly. 'Minimize changes' is not enough: unchanged steps come back verbatim, step count is stable unless the ticket explicitly adds a scenario, and the stable ID field is read-only."
  - "Suppress re-pushes by comparing content hashes after generation. A ticket typo fix that doesn't change what the test checks should not flood the test-case management tool. Gate on the hash, not on whether the source ticket changed."
  - "Human annotations survive because the identity does. Approval records, automation links, and execution history attach to a stable ID that predates any individual run; without it, every regeneration is a clean slate and the curation work was for nothing."

keywords: ["stable IDs for AI-generated test cases", "reconcile vs regenerate LLM documents", "deterministic planner before model call", "diff-aware prompt preserve unchanged steps", "content hash vs identity hash", "suppress phantom test-case updates"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).