gate-33/axe steps run with if:always() and report an INFRA failure as an accessibility one — 'axe-runner crashed: Executable doesn't exist' when the real cause was a composer TLS error 4 steps earlier

[rubenvdlinde]

What the job says, and what actually happened

openconnector PR #1216, quality / E2E Tests (Playwright), job 93888850176.
The only ##[error] a reader is likely to find is this one:

##[error]axe-runner: crashed before writing a report: browserType.launch:
  Executable doesn't exist at /home/runner/.cache/ms-playwright/
  chromium_headless_shell-1234/chrome-headless-shell-linux64/chrome-headless-shell
##[error]Process completed with exit code 2.

That reads as "the axe runner is broken" or "the Playwright browser cache is
wrong", and it points straight at enable-axe, which this repo turned on
recently. Both readings are wrong.

The step list tells the real story:

step
7	Checkout additional apps	failure
8–21	Install Nextcloud … Install Playwright … Run Playwright tests	skipped
22–24	Checkout the axe runner / assert ref / install axe-core	success
25	Run axe	failure ← the only error most readers will see

Step 7's actual failure:

Checking out ConductionNL/openregister (openregister) at development...
Installing composer dependencies for openregister...
  Failed to download symfony/polyfill-iconv from dist: curl error 60 while
  downloading https://api.github.com/repos/symfony/polyfill-iconv/zipball/…:
  SSL certificate problem: self-signed certificate
##[error]Process completed with exit code 100.

This is the known transient already documented fleet-wide (the same
curl error 60 … self-signed certificate that took out a hermiq stable32 cell).
A re-run of the failed job is the correct response.

The defect

The axe steps are not gated on the setup steps succeeding, so when the harness
falls over early they still execute — into an environment where
Install Playwright was skipped and therefore no browser binary exists. The
job's last error then describes a missing Chrome binary in the axe runner,
four steps downstream of a composer download failure in a different app.

Compare a healthy run on the same repo (93861159696, development): steps
1–21 all succeed, Run Playwright tests passes, and the axe step then reports
axe http://…/apps/openconnector/ -> HTTP 200 | passes=18 violations=0.

Why this matters more than a cosmetic log complaint

1. Run Playwright tests was SKIPPED, and nothing in the failure says so.
   A reader who sees only the axe error concludes the e2e suite ran and the
   accessibility check failed. In fact the suite did not execute at all — the
   skipped conclusion is silent, exactly the "a check that did not run looks
   like one that passed" shape.
2. The error blames the thing the repo most recently changed. openconnector
   turned on enable-axe: true two PRs ago. The natural response to
   "axe-runner crashed … Executable doesn't exist" is to switch axe back off,
   or to start pinning browser versions — which would remove real accessibility
   coverage to fix a composer TLS blip. This is the same trap as #351, where
   the axe runner's HTTP-0 error tells you to delete routes.
3. It costs a full E2E cycle to diagnose, every time.

Suggested fix

Guard the axe steps on the setup having succeeded — at minimum on the
Install Playwright / Run Playwright tests steps' outcome rather than on
always(). If they must stay always() so a report is uploaded on a genuine
test failure, then have the axe step detect and name the precondition:

if [ ! -d "$HOME/.cache/ms-playwright" ]; then
  echo "::error::Playwright browsers were never installed — an EARLIER step in
        this job failed. This is not an accessibility failure. Read the first
        red step, not this one."
  exit 1
fi

Either way the job should not end with an accessibility-flavoured error when
the accessibility check never had a chance to run.

Related

• #351 — the axe runner's other misleading error (HTTP 0 on hash routes),
  which likewise blames the caller's config.
• The fleet-wide finding that occ app:enable failures are swallowed and
  resurface as ordinary-looking application errors. Same family: the harness
  failing in the vocabulary of the application.