“Not deployed” is a first-class verdict: giving your test suite a richer vocabulary than green and red

A QA automation engineer's notes on giving deployment-guarded skips a first-class verdict instead of treating them as red noise.

The pull request is merged, the ticket is marked done, the agent has validated the expected behavior and written a solid spec. Then it runs, and it fails. Nothing is broken. The feature simply is not deployed yet, and the build pipeline is still weeks out. What do you do?

If your suite only knows "pass" and "fail", you have two options: delete the test, or keep watching it fail on every run until release ships. Both are wrong in different ways. There is a third option, and it costs you a small vocabulary expansion. I call it NOT-DEPLOYED.

Green and red is not a complete vocabulary

We build test suites around a binary because the binary is easy to act on. Green means ship, red means stop. That clarity is real, and I do not want to erode it. But it smuggles in an assumption: that every check is either verifying behavior that exists today or catching a regression in behavior that existed yesterday. There is a third state that neither covers well. The test correctly describes future behavior that has simply not shipped yet.

When an agentic coding flow validates documented acceptance criteria against the live application and the feature is not there, "fail" is the wrong label. The test is not broken. The application is not broken either, because it has not claimed to support this feature. The mismatch is an honest state of affairs: the documentation is ahead of the deployment. Calling it "fail" produces red noise that trains reviewers to ignore failing tests, which is exactly the trust erosion that makes a mature suite worthless.

Deleting the test is wrong too. You lose the coverage at the moment you need it most, when the feature eventually ships. Every deploy of a new feature is when bugs are most likely to creep in, and you want your spec waiting on the other side of that merge, not re-created from scratch.

NOT-DEPLOYED is a positive finding

When my agentic validation flow hits a documented feature that is absent from the application, I do not file a red. I classify the result as NOT-DEPLOYED, a positive finding that means the test correctly describes future behavior and the feature just has not shipped.

The distinction matters. A failure implies something is wrong and needs fixing. A NOT-DEPLOYED verdict implies something is right and needs patience. Both are actionable, but they are different actions, and conflating them corrupts your issue queue.

In practice, NOT-DEPLOYED becomes a deployment-guarded skip. The spec looks for the feature in the interface, finds it absent, and skips with a reason string that describes what it is waiting for. It does not fake a pass by loosening assertions, and it does not delete itself. It steps aside, explains why, and activates automatically when the feature ships.

The deployment guard in practice

The pattern is intentionally small. Here is an adapted version of what it looks like in a Playwright spec:

// src/domains/orders/tests/renewals/renewalEarlyExit.api.spec.ts

test('TC1: early-exit renewal reduces billed total by pro-rata amount @PROJ-8620', async ({ ordersApi }) => {
  testType('api'); severity('critical'); trackingRef('PROJ-8620');

  const featureFlag = await ordersApi.getFeatureFlags();
  if (!featureFlag.earlyExitRenewal) {
    test.skip(
      true,
      'NOT-DEPLOYED: early-exit renewal not yet deployed — ' +
      'skipped, not faked; auto-activates on deploy (PROJ-8620)',
    );
    return;
  }

  const order = await ordersApi.findOrderWithActiveRenewal();
  const before = await ordersApi.getBilledTotal(order.id);

  await ordersApi.triggerEarlyExit(order.id, { effectiveDate: today() });

  const after = await ordersApi.getBilledTotal(order.id);
  expect(after.amount).toBeLessThan(before.amount);
  expect(after.adjustmentReason).toBe('early_exit_proration');
});

Three things are happening here that are easy to miss.

The detection is behavioral, not config-file-based. The guard queries the actual API (or the DOM, or a feature-flag endpoint) rather than reading a local flag from a .env file. That way the guard activates and deactivates in response to what is real in the environment, not in response to somebody remembering to update a file.

The skip reason is a prose sentence, not a ticket number. "NOT-DEPLOYED: early-exit renewal not yet deployed, skipped, not faked; auto-activates on deploy (PROJ-8620)" tells the next engineer exactly what is being waited for and confirms the skip is intentional rather than a lazy workaround. The ticket reference anchors it to traceability.

The assertions inside the guard are real. When the feature flag flips, the spec does not need to be revisited before it becomes useful. It drops straight into the full assertion set, field-level expectations included. Zero human intervention on deploy day.

The two bad alternatives

It is worth being explicit about why the obvious options fail.

Delete the test. You lose coverage at the worst possible moment. A new feature lands with no automated validation until someone remembers to re-port it. In a fast-moving B2B platform where renewals, billing adjustments, and order status transitions are high-stakes, that gap can go undetected through several deploys. This happens. I have watched a manual test case get removed "until the feature ships" and then quietly never re-created.

Leave the test in and let it fail. This trains every engineer on the team to look at failing tests and think "probably not deployed yet." That habit is where a gate starts to die. The signal gets corrupted in both directions: real regressions start hiding inside the NOT-DEPLOYED noise, and the NOT-DEPLOYED tests start silently passing if the implementation lands in a broken state on a day nobody is paying attention.

Vocabulary, not just mechanics

The mechanics of test.skip are not new. What is new, and what makes the difference, is treating NOT-DEPLOYED as a first-class verdict in the reporting vocabulary rather than a technical artifact.

In my run reports, NOT-DEPLOYED skips appear in their own column, separate from coverage gaps and from feature-complete skips. A run summary that says "204 pass / 3 fail / 12 NOT-DEPLOYED" is readable. The 12 are not failures. They are queued work, each one a test that will activate automatically when its flag flips. The engineers who own the deploy know exactly what is waiting for them.

I keep a lightweight registry, a comment block at the top of each guarded spec, that records the tracking ticket, the expected deploy window if known, and the detection method. A spike in NOT-DEPLOYED count in a weekly report prompts the question of whether a deploy is slipping, not a panic about test quality.

This is the broader principle. A binary pass/fail output forces you to treat every non-pass the same way. A vocabulary that includes NOT-DEPLOYED, KNOWN-DEFECT (my test.fail pattern), and ENVIRONMENT-UNAVAILABLE lets engineers read a run report the way they read code, with the context baked in.

Where it falls down

Deployment guards only work if the detection is honest. The tempting shortcut is to write the guard as if (process.env.FEATURE_X !== 'true') test.skip(...) and then forget to set the flag in CI. Now you have a spec that reads as NOT-DEPLOYED in every environment, including the one where the feature shipped two months ago. I consider this a false-skip anti-pattern on par with the false-pass catalog. It just fails in the opposite direction.

The discipline is this: the guard must detect the feature from the live system, not from a config file a human has to remember to flip. If the feature is a new API endpoint, probe for a 200. If it is a UI surface, look for the element in the DOM. If it is a behaviour change in an existing endpoint, probe for the distinguishing field in a response. The guard and the spec can share the same API client, so there is no reason not to.

Second, NOT-DEPLOYED verdict handling needs to be in your reporting pipeline from day one. If CI treats any skip as "ignore it", the mechanism still works but the observatory is broken. You will not notice when counts climb or when a guarded spec never activates after a deploy. Add the distinct column before the first guarded spec ships.

Takeaways

• Green and red is not a complete vocabulary. "The test is correct; the feature just has not shipped" is a third state that neither binary describes.
• NOT-DEPLOYED is a positive finding: the spec correctly describes future behavior. Treat it as queued work, not noise.
• Write deployment guards that detect the feature from the live system, an API probe, a DOM query, or a feature-flag endpoint, not from a local config file a human controls.
• Fill the skip reason with a prose sentence: what is being waited for, why it is intentional, and the ticket reference. The next engineer should never have to guess.
• Keep the real assertions inside the guard body. When the feature ships, the spec activates with full coverage and zero rework.
• Report NOT-DEPLOYED as its own column in run summaries. A climb in the count is a deploy-tracking signal, not a test-quality signal.

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/08/29/not-deployed-first-class-verdict/"
license: "Free to reference with attribution"
title: "\"Not deployed\" is a first-class verdict: giving your test suite a richer vocabulary than green and red"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 39"
stack: [TypeScript, Playwright, REST API]

problem: "An agentic validation flow writes a correct spec from documented acceptance criteria, but the feature is not deployed yet (build pipeline weeks out). A binary pass/fail suite leaves only two bad moves — delete the test (lose coverage at the moment a new feature ships and bugs are most likely) or let it fail forever (red noise that trains reviewers to ignore failing tests and erodes trust in the gate)."
thesis: "Add NOT-DEPLOYED as a first-class verdict: a positive finding meaning the spec correctly describes future behavior that just has not shipped. Implement it as a deployment-guarded skip that detects the feature from the live system, keeps the real assertions intact, and auto-activates on deploy."

verdict_semantics:
  not_deployed: "A positive finding — the test is right, the feature needs patience, not fixing. Distinct from a failure (something is wrong) so the two trigger different actions and never get conflated in the issue queue."
  honest_state: "The documentation is ahead of the deployment; neither the test nor the application is broken. Labeling it fail produces red noise and corrupts the gate."
  reporting_vocabulary: "Sits alongside two sibling verdicts the author uses: KNOWN-DEFECT (his test.fail pattern) and ENVIRONMENT-UNAVAILABLE. A vocabulary beyond green/red lets engineers read a run report the way they read code, context baked in."

deployment_guard:
  artifact: "Adapted Playwright spec at src/domains/orders/tests/renewals/renewalEarlyExit.api.spec.ts — test 'TC1: early-exit renewal reduces billed total by pro-rata amount @PROJ-8620', tagged testType('api'), severity('critical'), trackingRef('PROJ-8620')."
  mechanism: "Query the live feature state (ordersApi.getFeatureFlags().earlyExitRenewal); if absent, test.skip(true, reason) and return; otherwise run the full body."
  three_subtleties:
    - "Detection is behavioral, not config-file-based: probe the actual API / DOM / feature-flag endpoint so the guard tracks what is real in the environment, not someone remembering to update a .env file."
    - "Skip reason is a prose sentence, not a bare ticket number: 'NOT-DEPLOYED: early-exit renewal not yet deployed — skipped, not faked; auto-activates on deploy (PROJ-8620)' — states what is awaited, confirms intent, and the ticket ref anchors traceability."
    - "Assertions inside the guard are real (field-level expectations like after.amount toBeLessThan before.amount and after.adjustmentReason === 'early_exit_proration'): when the flag flips the spec drops into full coverage with zero human intervention on deploy day."
  detection_recipes: "New API endpoint → probe for a 200. UI surface → look for the element in the DOM. Behavior change in an existing endpoint → probe for the distinguishing field in a response. Guard and spec can share the same API client, so there is no excuse not to detect from the live system."

reporting_and_registry:
  distinct_column: "NOT-DEPLOYED skips get their own column in run summaries, separate from coverage gaps and feature-complete skips — e.g. '204 pass / 3 fail / 12 NOT-DEPLOYED', where the 12 are readable as queued work, not failures."
  registry: "A lightweight registry — a comment block at the top of each guarded spec — records the tracking ticket, expected deploy window if known, and detection method."
  signal: "A spike in NOT-DEPLOYED count in a weekly report is a deploy-tracking signal (is a deploy slipping?), not a test-quality panic. Wire the column into the reporting pipeline before the first guarded spec ships, or the mechanism works but the observatory is blind to climbing counts or specs that never activate post-deploy."

anti_patterns:
  false_skip: "if (process.env.FEATURE_X !== 'true') test.skip(...) then forgetting to set the flag in CI — the spec reads NOT-DEPLOYED in every environment, including one where the feature shipped two months ago. A false-skip on par with the false-pass catalog; it just fails in the opposite direction."
  delete_the_test: "Loses coverage at the worst possible moment; a new feature lands unvalidated until someone remembers to re-port it. War story: the author watched a manual test case get removed 'until the feature ships' and then quietly never re-created."
  let_it_fail: "Trains the whole team to read failing tests as 'probably not deployed yet' — where a gate starts to die. Signal corrupts both ways: real regressions hide inside NOT-DEPLOYED noise, and guarded specs silently pass even if the implementation lands broken on a day nobody is watching."

use_when: "An agentic or human suite validates documented acceptance criteria against a live, fast-moving platform where specs are written ahead of deploys and high-stakes flows (renewals, billing adjustments, order status transitions) must not lose coverage across the merge that introduces them."
avoid_when: "Hermetic suites built only against already-shipped behavior, or pipelines where CI collapses every skip to 'ignore it' and the distinct verdict column is not in the reporting from day one."

takeaways:
  - "Green and red is not a complete vocabulary. 'The test is correct; the feature just has not shipped' is a third state that neither binary describes."
  - "NOT-DEPLOYED is a positive finding: the spec correctly describes future behavior. Treat it as queued work, not noise."
  - "Write deployment guards that detect the feature from the live system — an API probe, a DOM query, or a feature-flag endpoint — not from a local config file a human controls."
  - "Fill the skip reason with a prose sentence: what is being waited for, why it is intentional, and the ticket reference. The next engineer should never have to guess."
  - "Keep the real assertions inside the guard body. When the feature ships, the spec activates with full coverage and zero rework."
  - "Report NOT-DEPLOYED as its own column in run summaries. A climb in the count is a deploy-tracking signal, not a test-quality signal."

keywords: ["not-deployed test verdict", "deployment-guarded skip Playwright", "richer test vocabulary beyond pass fail", "feature-flag detection in tests", "false-skip anti-pattern", "test.skip with prose reason auto-activate on deploy"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).