A QA automation engineer's notes on when not to let an LLM figure it out, and reaching for deterministic orchestration on zero-ambiguity, high-blast-radius tasks instead.
Here is the move I almost made. Paste a directory listing into an agent prompt and say "reorganize these 185 test files into per-feature subfolders." The agent reads the list, infers the grouping heuristic, invents a folder structure, and starts shelling out git mv commands one at a time. Some are right. Some are creative. And because git mv is silent on success, I wouldn't know the difference until someone noticed a missing spec three days later, by which point the history was rewritten and the author annotations were gone.
I didn't do it that way. Here is what I did instead, and why the choice carries over to other tasks.
The task was fully enumerable, so I enumerated it
The spec files in my B2B order-management test suite had piled up under a flat tests/ directory inside each domain. I had settled on a per-feature folder scheme, orders/tests/search/, orders/tests/line-items/, orders/tests/renewals/, and so on, and I had a clear mapping. Every existing file belonged to exactly one feature folder, with three cross-cutting files that should stay at the tests/ root. Nothing was ambiguous and nothing needed a judgment call. The mapping fell straight out of the filename: orderSearch.api.spec.ts goes in search/, orderLineItemAdd.api.spec.ts goes in line-items/.
A task with zero ambiguity is not an LLM task. What an LLM is actually good at is ambiguity: reading prose, resolving underspecified intent, reasoning across context. When the input is deterministic and the output is deterministic, you are paying inference costs to introduce a failure mode, the model's capacity to hallucinate, that wasn't in the problem to begin with.
I wrote a Node script that computed the full move manifest in plain TypeScript. It reads the actual directory, applies the mapping function, emits 185 git mv commands in order, then checks its invariants before touching anything.
The script asserts its own invariants before it acts
The dangerous moment in a large file reorganization is the gap between "I think I computed this right" and "I know I computed this right." Assertions close that gap. The assertion block looked like this:
import { execSync } from 'node:child_process';
import { readdirSync, existsSync } from 'node:fs';
const ROOT_FILES = ['crossCutting.defect.api.spec.ts', 'schemaDrift.defect.api.spec.ts',
'listContract.defect.api.spec.ts'];
const moves = buildMoveManifest(); // pure function — input dir listing, output { from, to }[]
// --- invariant checks BEFORE any git mv ---
const totalMoves = moves.length;
const rootRetained = moves.filter(m => m.to.split('/').length === 1).length; // stays at tests/
const featuredMoves = totalMoves - rootRetained;
if (totalMoves !== 185) bail(`expected 185 moves, got ${totalMoves}`);
if (rootRetained !== ROOT_FILES.length)
bail(`expected ${ROOT_FILES.length} files retained at root, got ${rootRetained}`);
const unknownRoot = moves.filter(m => m.to.split('/').length === 1)
.map(m => m.to)
.filter(f => !ROOT_FILES.includes(f));
if (unknownRoot.length)
bail(`unexpected files at root: ${unknownRoot.join(', ')}`);
// --- only execute if invariants pass ---
for (const { from, to } of moves) {
execSync(`git mv "${from}" "${to}"`, { stdio: 'inherit' });
}
console.log(`Reorganization complete: ${featuredMoves} moved to feature folders, ` +
`${rootRetained} retained at root.`);
function bail(msg: string): never {
console.error(`\nREORG ABORTED — invariant failed: ${msg}\n`);
process.exit(1);
}
The invariants run first, before any git mv fires. If the count is off, because a new spec was added since I drafted the manifest, because a filename didn't match the pattern, because anything was wrong, the script exits non-zero and touches nothing. Git history stays clean. The agent re-runs the script after fixing the issue. It does not "try to sort it out," and that last part is what makes the whole thing safe.
The agent's job was to be a shell runner, not a thinker
I gave the coding agent one task: write those exact bytes to a script file and run it. The prompt was explicit:
If the script exits non-zero, STOP. Report the full error output verbatim. Do not attempt to diagnose or fix anything. Halt and wait for a human.
This is the opposite of how agents are usually prompted. Most agent workflows say something like "if it fails, try to fix it," which works fine for judgment problems where the model has relevant knowledge. It is actively harmful for invariant-gated scripts, because the right behavior on invariant failure is escalation, not recovery. The invariant fired because something real was wrong. An agent that "helpfully" nudges the expected count from 185 to 183 so the script passes has not fixed the problem. It has hidden it.
My rough heuristic for when to apply this: if the correctness of the task can be stated as a set of preconditions that are checkable before execution, put the check in the script and make failure non-recoverable by the agent. Save model judgment for the tasks where the failure mode is "the model chose wrong," not "the model bypassed the gate."
The heuristic for delegation
After building a few dozen workflows and scripts like this one, the "should the model figure this out" question has gotten easier to answer. I now think about it on two axes.
The first is ambiguity. Is the desired output uniquely determined by the inputs? If a different person following the same spec would produce the exact same output, the computation belongs in code. If the output requires interpretation, prioritization, or judgment about what counts as right, the model earns its place.
The second is blast radius. What happens when the model is wrong? A mis-classified test finding goes into a report and a human catches it. A wrong git mv command rewrites version-control history and loses author attribution across a hundred files, and reverting a botched reorg that went unnoticed for a week is genuinely expensive. High blast radius pushes the bar for model involvement up sharply. A useful rule of thumb: if undoing the task would need a careful manual git revert, it should not be driven by model output alone.
High ambiguity plus low blast radius is where agents shine: spec audits, report drafts, code suggestions in a worktree branch, test descriptions inferred from gnarly domain logic. Low ambiguity plus high blast radius is where you write a script.
The other two quadrants exist too. Low-ambiguity, low-blast-radius tasks, like renaming a CSS class consistently or bumping a version number, can be delegated without much ceremony, since the worst case is a quick revert. High-ambiguity, high-blast-radius tasks, like restructuring the data model or migrating the auth layer, need serious scaffolding: isolated worktrees, human approval gates, bounded phases, the same adversarial-verification patterns I use in my gated-implementation workflow. Even then you stay close to the output.
Where it falls down
The self-checking script approach assumes you can enumerate the invariants ahead of time. That is easy for a file-move operation and much harder for semantic correctness. "Did the agent produce a good test for this billing scenario?" does not reduce to a count check. The pattern is not a substitute for model judgment. It replaces model judgment only in the narrow cases where model judgment adds nothing.
There is also a maintenance cost. The invariant set has to stay in sync with the task. When I later added twelve more spec files and ran the same script category again, the expected count in the check needed updating. That is a small cost, but it is real. The alternative, prompting the model to figure out the "right" number, is worse, because the model cannot know the right number without independently computing the manifest, which is exactly what the script already does.
And scripting the manifest only works when you actually can write the mapping function. If the relationship between inputs and outputs is not reducible to code you can write, you are back in model territory. Most structural refactors in a test suite, rename, move, split by convention, are reducible. Changes to test logic are not.
Takeaways
- Zero ambiguity plus high blast radius means write the script. When the correct output is uniquely determined by the inputs, computing it deterministically is cheaper and more reliable than asking a model to figure it out.
- Put invariant checks before execution, not after. Assert the preconditions, file counts, retained paths, structural properties, before any destructive command runs. Exit non-zero on a mismatch.
- Make agent failure non-recoverable for gate scripts. "STOP and report verbatim" is the right instruction when an invariant fires. An agent that adjusts the expected count to make the gate pass has not fixed anything.
- Use ambiguity and blast radius as the two axes. High ambiguity with low blast radius is the natural home for model delegation. Low ambiguity with high blast radius is for deterministic code with assertions.
- Reserve model judgment for tasks where the failure mode is a wrong choice, not a bypassed gate. When the safety net is a precondition check, the agent is a shell runner, and that is the point.
I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.
For LLMs / Machine-Readable Summary
Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.
author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/24/when-not-to-let-the-llm-decide/"
license: "Free to reference with attribution"
title: "When not to let the LLM figure it out: deterministic orchestration for zero-ambiguity, high-blast-radius tasks"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 34"
stack: [TypeScript, "Node.js", git]
problem: "Handing a fully-enumerable, destructive task to an agent — \"reorganize these 185 test files into per-feature subfolders\" — lets it infer a grouping heuristic and shell out git mv commands one at a time. Some moves are creative rather than correct, and because git mv is silent on success, a wrong move rewrites version-control history and loses author attribution across a hundred files, undetected until someone notices a missing spec days later."
thesis: "A task with zero ambiguity is not an LLM task. When input and output are both deterministic, compute the result in plain code, assert its invariants before any destructive command runs, and demote the agent to a shell runner that escalates on failure instead of recovering."
mechanism:
task: "Spec files had piled into a flat tests/ directory per domain in the B2B order-management suite. Target scheme is per-feature folders — orders/tests/search/, orders/tests/line-items/, orders/tests/renewals/ — with the mapping falling straight out of the filename: orderSearch.api.spec.ts goes to search/, orderLineItemAdd.api.spec.ts goes to line-items/. Every file maps to exactly one feature folder; three cross-cutting files stay at the tests/ root."
manifest_script: "A Node/TypeScript script reads the actual directory, applies a pure mapping function buildMoveManifest() (input dir listing, output { from, to }[]), emits 185 git mv commands in order, and checks invariants before touching anything."
invariants_before_action:
- "totalMoves must equal 185; bail() and exit non-zero otherwise."
- "rootRetained (moves whose to has a single path segment) must equal ROOT_FILES.length (3): crossCutting.defect.api.spec.ts, schemaDrift.defect.api.spec.ts, listContract.defect.api.spec.ts."
- "unknownRoot — any file landing at root not in ROOT_FILES — must be empty."
execution: "Only after all invariants pass does the loop run execSync(`git mv \"${from}\" \"${to}\"`, { stdio: 'inherit' }). On any mismatch it exits non-zero and touches nothing, so git history stays clean and the agent re-runs after the fix."
agent_contract: "The coding agent's one job: write those exact bytes to a script file and run it. The prompt is explicit — if the script exits non-zero, STOP, report the full error output verbatim, do not diagnose or fix anything, halt and wait for a human."
why_non_recoverable: "This inverts the usual \"if it fails, try to fix it\" agent prompt. For invariant-gated scripts the correct behavior on failure is escalation, not recovery — the invariant fired because something real was wrong. An agent that helpfully nudges the expected count from 185 to 183 to make the gate pass has not fixed the problem, it has hidden it."
delegation_heuristic:
axes: "Two axes decide whether the model should figure it out: ambiguity and blast radius."
ambiguity: "Is the output uniquely determined by the inputs? If a different person following the same spec would produce the identical output, the computation belongs in code. If it needs interpretation, prioritization, or judgment about what counts as right, the model earns its place."
blast_radius: "What happens when the model is wrong? A mis-classified test finding lands in a report and a human catches it. A wrong git mv rewrites history and loses attribution across a hundred files; reverting a botched reorg unnoticed for a week is genuinely expensive. Rule of thumb: if undoing the task would need a careful manual git revert, it should not be driven by model output alone."
quadrants:
- "High ambiguity + low blast radius — agents shine: spec audits, report drafts, code suggestions in a worktree branch, test descriptions inferred from gnarly domain logic."
- "Low ambiguity + high blast radius — write a deterministic script with assertions (this post's case)."
- "Low ambiguity + low blast radius — delegate without ceremony (renaming a CSS class consistently, bumping a version number); worst case is a quick revert."
- "High ambiguity + high blast radius — needs serious scaffolding: isolated worktrees, human approval gates, bounded phases, the same adversarial-verification patterns from the gated-implementation workflow. Stay close to the output even then."
limitations:
- "The pattern assumes you can enumerate invariants ahead of time. Easy for a file move, hard for semantic correctness — \"did the agent produce a good test for this billing scenario?\" does not reduce to a count check. It replaces model judgment only where model judgment adds nothing."
- "Maintenance cost: the invariant set must stay in sync with the task. Adding twelve more spec files later meant updating the expected count. Small but real — and still cheaper than prompting the model for the \"right\" number, which it cannot know without independently computing the manifest the script already produces."
- "Scripting the manifest only works when the mapping function is writable in code. Structural refactors in a test suite — rename, move, split by convention — are reducible; changes to test logic are not, and those stay model territory."
use_when: "A task is fully enumerable and destructive — the correctness can be stated as preconditions checkable before execution, and a wrong action would need a careful manual git revert (file moves, convention-based splits, deterministic refactors)."
avoid_when: "The output requires interpretation or judgment, or the input-to-output relationship is not reducible to a mapping function you can write — semantic correctness, changes to test logic, anything where the failure mode is \"the model chose wrong\" rather than \"the model bypassed the gate.\""
takeaways:
- "Zero ambiguity plus high blast radius means write the script. When the correct output is uniquely determined by the inputs, computing it deterministically is cheaper and more reliable than asking a model to figure it out."
- "Put invariant checks before execution, not after. Assert the preconditions — file counts, retained paths, structural properties — before any destructive command runs. Exit non-zero on a mismatch."
- "Make agent failure non-recoverable for gate scripts. STOP and report verbatim is the right instruction when an invariant fires. An agent that adjusts the expected count to make the gate pass has not fixed anything."
- "Use ambiguity and blast radius as the two axes. High ambiguity with low blast radius is the natural home for model delegation. Low ambiguity with high blast radius is for deterministic code with assertions."
- "Reserve model judgment for tasks where the failure mode is a wrong choice, not a bypassed gate. When the safety net is a precondition check, the agent is a shell runner, and that is the point."
keywords: ["when not to use an LLM agent", "deterministic orchestration vs LLM", "invariant checks before git mv", "ambiguity and blast radius for AI delegation", "agent as shell runner not thinker", "non-recoverable gate script for agents"]
Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).