Gate 19 treats a scenario as covered when its @e2e tag names a Playwright file that exists and is collected. It does not ask whether any test inside that file exercises the scenario. So a tag can point at a real, running, green file that says nothing about the requirement, and the gate credits it.
What was measured, in dossiq
starter-content-and-templates REQ-TPL-02, "a result template presets the outcome text", carries an @e2e tag naming tests/e2e/starter-content-and-templates.spec.ts.
That file exists, is collected by the chromium project, and executes six tests. It contains no test for that scenario: zero references to TemplatePicker, to the outcome text, or to the fixture the scenario names. No other e2e file covers it either.
The requirement therefore read as covered by an executing suite, while nothing anywhere exercised it. The feature's only real evidence was a component test mounting a dialog that no user can open, so it was green about unreachable code.
Why this one is expensive
It is the citation-versus-scenario failure again, and it is worse than an uncovered requirement, because an uncovered requirement is visible. This one is invisible: the tag resolves, the file runs, the suite is green, and the gate agrees.
It also interacts badly with cleanup work. We nearly retired the dialog on the grounds that a spec-named feature was protected by a live scenario. It was not protected by anything.
The ask
Make gate 19 require evidence from inside the file, not merely its existence. Options, roughly in order of cost:
- Require the scenario id or tag string to appear in the file, not just the file to be named by the tag. Cheap, catches this case, still fools a stale comment.
- Require it to appear inside a
test(...) title or a tag list Playwright can filter on, so --grep can select it. Better, and makes the coverage claim executable.
- Report, per scenario, the test titles that claim it, so a reviewer can read what is being asserted rather than trusting a count.
Even option 1 would have failed this case honestly.
Related
A separate gate issue from the same night: #689, where gate 32's suggested remedy is itself an axe violation. Both share a shape worth naming in the gate guidance: a gate that reports on something adjacent to the thing it is asked about.
Gate 19 treats a scenario as covered when its
@e2etag names a Playwright file that exists and is collected. It does not ask whether any test inside that file exercises the scenario. So a tag can point at a real, running, green file that says nothing about the requirement, and the gate credits it.What was measured, in dossiq
starter-content-and-templatesREQ-TPL-02, "a result template presets the outcome text", carries an@e2etag namingtests/e2e/starter-content-and-templates.spec.ts.That file exists, is collected by the
chromiumproject, and executes six tests. It contains no test for that scenario: zero references toTemplatePicker, to the outcome text, or to the fixture the scenario names. No other e2e file covers it either.The requirement therefore read as covered by an executing suite, while nothing anywhere exercised it. The feature's only real evidence was a component test mounting a dialog that no user can open, so it was green about unreachable code.
Why this one is expensive
It is the citation-versus-scenario failure again, and it is worse than an uncovered requirement, because an uncovered requirement is visible. This one is invisible: the tag resolves, the file runs, the suite is green, and the gate agrees.
It also interacts badly with cleanup work. We nearly retired the dialog on the grounds that a spec-named feature was protected by a live scenario. It was not protected by anything.
The ask
Make gate 19 require evidence from inside the file, not merely its existence. Options, roughly in order of cost:
test(...)title or a tag list Playwright can filter on, so--grepcan select it. Better, and makes the coverage claim executable.Even option 1 would have failed this case honestly.
Related
A separate gate issue from the same night: #689, where gate 32's suggested remedy is itself an axe violation. Both share a shape worth naming in the gate guidance: a gate that reports on something adjacent to the thing it is asked about.