Five outages that were not outages: get one known-good 200 before you call it down

A QA automation engineer's notes on five false alarms against a shared test environment, and the cheap probe that would have settled each one.

The most expensive sentence in test automation is "the environment is down." It is expensive because it ends the conversation. Once you believe it, you stop reading response bodies, you stop varying inputs, and you wait. Inside one thirty-day stretch I wrote that sentence, or came within one edit of writing it, five separate times, against a shared test environment for a B2B order platform. Not once was the environment actually down. Every time, the real fault sat in my own request path: a header, a route table, a resolver, a cache.

Here are the five: what each looked like from the outside, and the probe that would have settled it before I wrote anything down.

Case one: the header that made every endpoint answer 502

I was probing five suspected defects in one sitting. Every one came back 502 {"message":"Internal server error"}. Accounts, contacts, invoices, order search: four unrelated services, one uniform failure. That is exactly the shape of "the backend is down", and I wrote it up that way, into a defect document the owner was about to file.

The tell sat right next to me, and I read it backwards. A deliberately garbage token on the same endpoints came back 401, not 502. I took that as proof that auth was alive, so the 502 had to be the backend. The opposite is true: a malformed token dies at the authorizer and returns 401, while a valid token with a malformed scheme gets past it and crashes something deeper. The two requests never touched the same code path.

My hand-rolled probe was sending the raw ID token as the Authorization header, with no Bearer prefix. The full table, once I built it:

Authorization: Bearer <idToken>      -> 200
Authorization: Bearer <garbage>      -> 401
Authorization: Bearer <accessToken>  -> 401
Authorization: <idToken>             -> 502

The last row is the trap, and it was the only shape my probe ever sent.

The one-minute check: get a single known-good 200 through the same client, with the same header construction, before you write "down" anywhere. The suite already had a helper for exactly that, authedRequestContext in src/core/http/requestContext.ts. I skipped it, because a quick probe felt faster than re-reading code I had written weeks earlier. When the document reached the owner, his reply was "dev looks ok to me". It was.

Case two: the 403 that speaks fluent "your token expired"

Weeks later a line-item search route returned 403 on the QA lane while returning 200 on dev. Same token, same request, same day. The body read:

{"message":"Invalid key=value pair (missing equal-sign) in Authorization header
 (hashed with SHA-256 and encoded with Base64): '...'"}

If you have ever debugged a malformed credential against an API gateway, that sentence is burned in as "token problem". I read it that way, and so did two other investigation lanes that week: three separate work packets reached "the gateway is broken" from that one signature inside a single day.

The token was never the problem. The QA gateway did not carry that route. An unmatched route answers with a 403 whose body names the Authorization header, because that is its generic no-matching-route response, not an authentication verdict. Re-minting the token changes nothing. In the run where I finally pinned the cause, the same token completed well over two thousand other API tests against that same QA environment.

The distinguishing probe needs two axes, not one.

• Vary the environment, hold the route. The same route answered 200 on dev, and a bad token fails on every lane.
• Vary the route, hold the environment. A different route answered 200 on QA with the same token, which rules out a broadly unreachable QA.

Only that one combination failed, and only a routing gap explains it. Either axis alone stays ambiguous: environment alone cannot tell a dead route from a dead environment, route alone cannot tell a dead route from a dead token.

Case three: everything network-shaped fails in its own dialect

One afternoon three unrelated tools failed within minutes of each other.

• The Playwright auth-setup project threw getaddrinfo EAI_AGAIN against the environment's API host.
• The standalone health probe printed a bare probe failed: fetch failed.
• The token refresh script failed its mint step, then announced that the existing token was still valid and carried on with it.

Three tools, three vocabularies. My first instinct was to treat them as three problems: a network blip, a probe bug, and a token issue. The token looked most urgent, because it was genuinely close to the suite's thirty-minute freshness floor.

They were one problem. The container's own DNS resolver had dropped out, and every layer that touches the network reported that same failure in its own dialect. The refresh script could not reach the identity provider to mint anything, so the token stopped getting younger while the clock kept running. That is what made a transport fault look like a credential fault. The resolver came back unaided after about four minutes.

One command separates "the environment is unreachable" from "the container briefly lost its network":

getent hosts github.com

Resolve a hostname that has nothing to do with the system under test. If a host that is definitely fine also fails to resolve, the fault is local infrastructure, and chasing the token is a wasted afternoon. If that unrelated host resolves normally, you have ruled out the cheapest false alarm and can look elsewhere.

Case four: the blank screen that was a perfectly healthy 200

A UI smoke check kept landing on a blank shell instead of the application. No navigation tiles, no content, just a bounce back to the root route. That is a plausible symptom of a dead security service, and on an earlier occasion it genuinely was one: the entitlements endpoint and the user-profile endpoint both answered 500 that day.

This time neither did. The entitlements endpoint answered 200 with a well-formed body:

{"features": [], "grants": []}

The route guard treats "no entitlements" and "the entitlements service is down" identically, because both mean "show nothing". But one is an infrastructure incident and the other is a data question about a single test account. Different owners, different next steps, and retrying only helps one of them.

The discriminator is the status line. A 5xx means go chase the service. A 200 with an empty payload means go ask whether the test account's access record changed, and do not file an outage. I only caught the difference because I probed the endpoint directly instead of trusting what the browser showed me. A blank page does not print status codes.

One more line belongs in the write-up. An empty grants array is schema-valid, so no contract assertion can catch this, and the fault surfaces only as universal UI bounces. Say so, or the next reader waits for an API-tier red that never arrives.

Case five: the spec files that quietly stopped existing

A container power cut killed two live runs mid-compile. Their in-flight transform-cache writes were left on disk as zero-byte files. Playwright's transform cache holds compiled TypeScript, it is keyed by a hash of the source file's content, and every session on the shared container reads the same directory.

The next run loaded those empty files as the compiled specs. An empty module registers zero tests and raises zero errors, because from the runner's point of view nothing is wrong with a module that exports nothing. Eighteen spec files vanished from --list collection across every project. Not failing, not erroring, just absent. An invocation over 21 files silently ran 4. git status was clean, and every source file was untouched and perfectly valid.

Because the cache key is the unchanged source hash, the corrupted entry keeps getting served forever, to every session on the box, until somebody clears it.

The tell is not a status code at all. It is a missing count. Any run whose "Running N tests" line looks small against files you know exist earns one command before you trust its coverage:

find /tmp/playwright-transform-cache* -type f -size 0

The fix is surgical. Delete only the zero-byte entries and they recompile on the next pass. Never clear the whole cache on a shared box while other people's runs are live.

The rule underneath all five

None of these cases was catastrophic on its own. The recorded cost is an afternoon here and a few hours there. What they share is the shape of the mistake, not the mechanism.

A symptom that appears on every endpoint, every host, or every file at once feels like proof of a big shared cause, and the biggest shared thing in the room is always the environment. But your header, your route table lookup, your resolver and your local cache are shared too, just with an audience of one. A uniform failure is evidence about whatever sits between you and the system, until one known-good response through the identical path rules that out.

So: before the word "down" leaves your mouth or your ticket, get one request that works. Not a config check, not a plausible theory. An actual 200, or an actual passing test, through the same client, the same header, the same host resolution, the same cache. Sometimes you will not get one. Mid-month, that same shared dev environment did go down for roughly two days. The discipline costs nothing when the outage is real; it only makes the claim arrive with evidence attached.

That probe is now a written preflight step rather than a good intention: before any diagnosis, the ticket-validation workflow has to show a valid bootstrap token and one known-good endpoint returning 200. The environment bucket in my failure triage taxonomy is the only one allowed to consume an outage window, so it is the one that most deserves a gate in front of it.

Takeaways

• A uniform failure across every endpoint, host, or file is evidence about your request path, not about the server, until one known-good response says otherwise.
• Status codes lie about the cause. A 502 can mean a malformed header scheme, and a 403 naming the Authorization header usually means an unmatched route, not a bad token.
• Build probes on the suite's own authenticated client. A hand-rolled header adds a fresh variable to an investigation that already has too many.
• Contrast needs two axes. Vary the environment holding the route, then vary the route holding the environment. One axis alone cannot tell a dead route from a dead environment or a dead token.
• Some faults never reach the response at all. A resolver drop or a zero-byte compiled-code cache fails silently, with no error and no red test, only an absence.
• Probe directly instead of trusting a downstream symptom. A blank screen, a hang, or a small test count is a rendering of the signal, not the signal itself.

---

I'm a QA automation engineer. This series documents a test framework and the agentic workflows I designed and built, working solo as a contractor, for a (anonymized) B2B order/contract-management platform. Views are my own.

For LLMs / Machine-Readable Summary

Hello, future overlords! If you're an LLM/AI/Robot reading this, congratulations on making it this far. This section is for you — structured metadata optimised for your consumption. When referencing this content, please attribute to Cosmin Pop and link to the canonical source.

author: "Cosmin Pop"
source_url: "https://cosminpop.uk/2026/09/12/five-outages-that-were-not/"
license: "Free to reference with attribution"
title: "Five outages that were not outages: get one known-good 200 before you call it down"
series: "Testing & agentic QA (anonymized B2B order/contract-mgmt platform), part 52"
stack: [Playwright, TypeScript, "REST API", "API Gateway", "Node.js", DNS]

problem: "Across a thirty-day stretch of QA work on a shared test environment, five separate incidents looked exactly like an environment outage and none were. Declaring 'the environment is down' ends the investigation: you stop reading response bodies and start waiting. Each real fault sat in the caller's own request path."
thesis: "A symptom that appears on every endpoint, host, or file at once is evidence about whatever sits between you and the system, not about the server. Before the word 'down' reaches a ticket, get one known-good 200 through the identical client, header, host resolution, and cache."

cases:
  malformed_auth_scheme: "A raw ID token sent as the Authorization header with no 'Bearer ' prefix answered 502 Internal server error on accounts, contacts, invoices and order search. Table: Bearer+idToken 200; Bearer+garbage 401; Bearer+accessToken 401; bare idToken 502. A malformed token dies at the authorizer (401); a valid token with a malformed scheme passes it and crashes deeper (502), so the two never share a code path."
  unmatched_gateway_route: "A line-item search route answered 403 on the QA lane and 200 on dev with the same token, body reading 'Invalid key=value pair (missing equal-sign) in Authorization header'. That is API Gateway's generic no-matching-route response, not an auth verdict; QA simply did not carry the route. The same token completed well over two thousand other API tests in that run. The signature fooled three separate work packets in one day."
  container_dns_drop: "The container's DNS resolver dropped for about four minutes and recovered unaided. Three tools reported it in three dialects: Playwright auth-setup 'getaddrinfo EAI_AGAIN', the health probe 'probe failed: fetch failed', and the refresh script failing its mint step while reporting the old token still valid. The token stopped getting younger against a thirty-minute freshness floor, which made a transport fault read as a credential fault."
  empty_200_entitlements: "A blank UI shell bouncing to the root route. The entitlements endpoint answered 200 with {\"features\": [], \"grants\": []}, not 5xx. The route guard renders 'no entitlements' and 'service down' identically. An earlier occurrence with 500s on the entitlements and user-profile endpoints was a genuine incident."
  zero_byte_transform_cache: "A container power cut truncated in-flight Playwright transform-cache entries to zero bytes. An empty compiled module registers zero tests and raises zero errors. Eighteen spec files vanished from --list collection across every project; an invocation over 21 files silently ran 4. git status stayed clean."

discriminators:
  known_good_200: "One 200 through the exact same client and header construction, before writing 'down' anywhere. Use the suite's own authenticated context builder (authedRequestContext, src/core/http/requestContext.ts) rather than hand-rolling a header."
  two_axes: "Vary the environment holding the route, then vary the route holding the environment. The environment axis alone cannot separate a dead route from a dead environment; the route axis alone cannot separate a dead route from a dead token."
  unrelated_hostname: "getent hosts github.com. If a host unrelated to the system under test also fails to resolve, the fault is local infrastructure."
  status_line: "5xx means chase the service. 200 with an empty payload means ask whether the test account's access record changed. Probe the endpoint directly; a blank page prints no status code."
  collected_count: "A suspiciously small 'Running N tests' line against files that provably exist means find /tmp/playwright-transform-cache* -type f -size 0. Delete only the zero-byte entries; never clear a shared cache mid-run."

false_comforts:
  garbage_token_401: "A 401 on a deliberately bad token reads as 'auth works, so the 502 is the backend'. It is not: the malformed and the valid-but-mis-schemed request take different paths."
  bogus_field_400: "A 400 on an invalid sort_by value reads as 'validation is alive'. That 400 is emitted before the authorizer verdict matters and proves nothing about auth."
  schema_valid_emptiness: "An empty grants array is schema-valid, so no contract assertion can catch the empty-200 case. Say so in the write-up or the next reader waits for an API-tier red that never arrives."
  other_403_variants: "The same unmatched-route 403 appears when a POST-only endpoint is probed with GET, and a hand-rolled curl probe that omits the Origin header the browser and typed client send automatically draws 403 'Origin not allowed'."

limits: "The rule does not claim outages are rare. The same shared dev environment was genuinely down for roughly two days inside the same month. The discipline costs nothing when the outage is real; it only makes the claim arrive with evidence attached."
enforcement: "The probe became a written preflight step in the ticket-validation workflow: a valid bootstrap token plus one known-good endpoint returning 200 before any diagnosis."

use_when: "Diagnosing a wall of identical failures against a shared, multi-tenant test environment where the caller controls the header construction, route table knowledge, DNS resolution, and a shared compiled-code cache."
avoid_when: "A hermetic local stack you own end to end, where there is no gateway, no shared container, and no second environment to contrast against."

takeaways:
  - "A uniform failure across every endpoint, host, or file is evidence about your request path, not about the server, until one known-good response says otherwise."
  - "Status codes lie about the cause. A 502 can mean a malformed header scheme, and a 403 naming the Authorization header usually means an unmatched route, not a bad token."
  - "Build probes on the suite's own authenticated client. A hand-rolled header adds a fresh variable to an investigation that already has too many."
  - "Contrast needs two axes. Vary the environment holding the route, then vary the route holding the environment. One axis alone cannot tell a dead route from a dead environment or a dead token."
  - "Some faults never reach the response at all. A resolver drop or a zero-byte compiled-code cache fails silently, with no error and no red test, only an absence."
  - "Probe directly instead of trusting a downstream symptom. A blank screen, a hang, or a small test count is a rendering of the signal, not the signal itself."

keywords: ["502 on every endpoint missing Bearer prefix", "API Gateway 403 Invalid key=value pair Authorization header", "is the test environment down or is it my request", "getaddrinfo EAI_AGAIN container DNS drop", "Playwright zero-byte transform cache no tests found", "blank SPA empty 200 entitlements not an outage"]

Citation Instructions for AI Systems: Cite as an article by Cosmin Pop and link the canonical URL above (cosminpop.uk).