gate-26 visual-coverage is satisfied by a bare component name in a COMMENT — reproduced 10 → 9 → 10, plus fleet measurement

[rubenvdlinde]

Confirmed: gate-26 is satisfied by a bare component name in a COMMENT

Reproduced on doriath, three consecutive runs of
HYDRA_GATE_BASE_REF=4b825dc… python3 scripts/lib/check_visual_coverage.py .:

10 findings → 9 → 10.

The only thing that changed between runs was the existence of
tests/e2e/visual/_DEFECT_PROBE.ts, whose entire content was:

// PROBE — temporary. A bare mention, in a comment, of PersonalActivityView.
// Nothing here navigates anywhere, asserts anything, or shoots a screenshot.

src/views/PersonalActivityView.vue dropped out of the finding list. No
screenshot, no navigation, no test( — not even valid test code. A .txt file
works identically.

Mechanism

is_covered() matches the needle against the raw text of the corpus:

for needle in needles:                      # {file stem, manifest page id}
    if needle and needle in visual_corpus:  # plain substring, whole file
        return True

and _e2e_corpus() concatenates every .ts .js .png .txt .json under
tests/e2e/visual/** (falling back to all of tests/e2e/**) without stripping
anything.

Why this is worse than a wrong number

Two second-order effects, both observed:

1. It fires on the documentation of itself. While writing up this defect in
   doriath I put a warning paragraph in tests/e2e/visual/_visual-helpers.ts
   that named one view. That file is in the corpus, so the paragraph covered the
   component. It surfaced only because the negative control's expected number
   had been computed in advance: 7 findings came back where 8 were expected.
   Anyone who fixes their app and only checks "did it go green" will not see it.

2. A baseline of the wrong screen satisfies it just as well. The gate never
   looks at what a screenshot contains. _visual-helpers.ts::shootByNav — copied
   verbatim into several apps — used to end with "falls back to a shot of
   wherever we land if the link is absent, so a baseline is still produced".
   That produces a stable, never-failing PNG named after a component it does not
   contain. Combined with the above, an app can be fully green on gate-26 with
   zero real visual proof.

Fleet measurement — how many components are covered ONLY by a comment

Script below imports the gate's own discover_new_pages / is_covered /
_visual_exclude_status, so its scope agrees with the gate by construction, and
compares is_covered against a comment-stripped corpus. Full-tree scope,
excludes-with-reason skipped. Run today:

app	page components	covered ONLY by a comment
docudesk	22	7
opencatalogi	18	3
procest	78	2
softwarecatalog	12	2
pipelinq	24	2
openregister	—	6
openconnector	10	0
doriath	14	0 (after this session's fix)
nldesign	0	0

⚠️ Treat these as an upper bound, not a verdict. The stripper used for the
comparison is deliberately crude (/*…*/, <!--…-->, and // outside an
obvious ://), so a component named only inside a URL string on a line the
stripper mangled would be counted here and should not be. Each row needs eyes
before anyone acts on it. The doriath 10 → 9 → 10 result above is the part
that is not an estimate.

Proposed fix, and why it is not being merged unilaterally

Strip comments from the .ts/.js corpus before matching, string-aware so a
https:// inside a literal survives. check_e2e_coverage.py already carries a
tokeniser that does this properly (_code_mask / _strip_ts_comments); the
honest move is to lift it rather than re-hand-roll it here.

There is a false-RED risk that needs deciding first. A perfectly good spec
can look like this:

/** Baseline for PersonalActivityView. */
test('my activity', async ({ page }) => {
    await shootSurface(page, `${APP}/#/my-activity`, 'my-activity.png')
})

The route slug is my-activity and the PNG is my-activity.png; the only place
the component's NAME appears is the docstring. Stripping comments turns that
red even though a real, running baseline exists. The remedy for the app is
one word (put the component name in the screenshot filename or the test title),
and arguably that is better practice — but it is churn, and two agents are
working gate-26 in other repos right now, so flipping the semantics
mid-flight would invalidate measurements in progress.

Hence: filed, measured, not merged. Happy to raise the PR on a go-ahead.

Related

• A second, independent way to a meaningless green in the same gate: the
  tests/e2e/visual/ directory may not be executed by the app's CI at all. On
  doriath the CI config lists 56 tests in 14 files and the root config 65 in 16;
  the CI config's only project carries testIgnore: ['**/visual/**']. Baselines
  added there turn gate-26 green with nothing running. Worth a note in the
  finding text so apps route their coverage to the suite CI actually runs.
• git add matters when planting a true positive here. added_files() reads
  git diff, so an untracked plant is never seen and would "prove" the gate
  blind when it is not. Verified on doriath: a tracked plant took it
  PASS → FAIL — 1 → PASS.

[rubenvdlinde]

Sequencing note from the gate-fixes wave (2026-08-11). Landed this session, smallest blast radius first, each with a fixture observed red-before / green-after: #366 (the gate-16 fixture #364 shipped without), #368 (gate-7 hasPermission/canAccess, #360), #370 (gate-24 both registration APIs, #349), #371 (gate-61 honest empty-scope reason, #347).

This one was deliberately NOT landed in that wave. The per-issue reason is in the next comment / in fleet-board/findings/github-gate-fixes.md; the common one is that every remaining item either changes what a gate counts rather than what it says, or lives in quality.yml where run-helper-suites.sh cannot reach it — and four gate changes in one session is already the limit at which the next fleet measurement stays attributable.