gate-19: an @e2e anchor that is unparseable or dangling is silently ignored — 22 real, running tests scored as zero coverage

[rubenvdlinde]

Summary

Gate-19 silently ignores two classes of broken @e2e anchor. Both look
exactly like coverage that is simply absent, and one of them hides real,
running, asserting tests.

Measured in ConductionNL/docudesk while taking gate-19 from 375 → 358.

1. An anchor under openspec/changes/ constructs no ref at all

scripts/lib/check_e2e_coverage.py:

_E2E_PATH_RE = re.compile(
    r"@e2e\s+openspec/specs/(?P<spec>[^/]+)/[^\s#]*#(?P<slug>[A-Za-z0-9_-]+)"
)
_E2E_SHORT_RE = re.compile(
    r"@e2e\s+(?P<spec>[A-Za-z0-9_-]+)::(?P<slug>[A-Za-z0-9_-]+)"
)

An anchor of the form

// @e2e openspec/changes/<change>/specs/<spec>/spec.md#<slug>

matches neither. collect_ref_status never sees it, so it is not reported
as dangling, not reported as dead, not reported at all.

This is the normal shape for any test authored while its change was still in
openspec/changes/ and then archived to openspec/specs/. In docudesk there
were 22 such anchors across 2 files, and both files contain real Playwright
tests that run and assert — 10 scenarios' worth of genuine coverage scored as
zero. Repointing them (and nothing else) moved the number by 5 immediately;
the rest needed the tests strengthened first.

Second half of the same trap: the archived slugs also carry a scenario-
prefix (#scenario-correspondence-opens-from-the-menu) that the canonical slug
does not have. Fixing only the directory yields a dangling anchor — which
brings us to:

2. A dangling anchor is not reported either

An anchor that parses fine but names a scenario that does not exist is silently
dropped. docudesk had 6, e.g.
document-comparison::the-ui-must-provide-a-side-by-side-comparison-view
(a requirement title, never a scenario) and
folder-batch-analysis::initiate-folder-analysis-on-a-folder-with-5-documents
(renamed to …-by-folder-path-existing-behavior). Each is a file claiming
traceability to something that is not there, and nothing says so.

Why this matters more than the count

The gate's whole contract is "the count is the honest number". These two
classes make it overstate debt (real coverage scored zero) while a third
known defect, #343 (file-level tags credited without checking the body), makes
it understate debt. An app's number is currently the sum of three unrelated
errors, which is why two agents can reasonably disagree about the same tree.

Suggested change (additive, non-blocking)

Report both classes as diagnostics on stdout without changing the verdict or
the count — a new blocking condition across 22 repos would be reckless, and
these are author mistakes that people can only fix once they can see them:

[gate-19] e2e-coverage: NOTE — 22 @e2e anchor(s) match no known form and were
          ignored (…/custom-dictionary-recognition.spec.ts:19 →
          openspec/changes/…). Anchors must name openspec/specs/<spec>/…#<slug>
          or <spec>::<slug>.
[gate-19] e2e-coverage: NOTE — 6 @e2e anchor(s) name a scenario that does not
          exist (…/folder-analysis.spec.ts:14 →
          folder-batch-analysis::initiate-folder-analysis-on-a-folder-with-5-documents).

Detection is cheap — the valid ref set is already built by
parse_spec_scenarios, and a catch-all @e2e\s+(?!exclude)(\S+) minus the two
matched spans gives the unparseable set.

A reproducer script and the docudesk measurements are in
/home/rubenlinde/fleet-board/findings/docudesk.md.

Not proposing

Auto-rewriting changes/ → specs/ anchors. The slug differs too (the
scenario- prefix), and silently repointing an author's anchor to a
similarly-named scenario is how a test ends up claiming something it does not
assert.