gate-19: one requirement-level @e2e exclude silently exempts every sibling scenario — 471 of openbuild's 509 exclusions share a reason (42 reasons, one covers 32)

[rubenvdlinde]

Summary

A single @e2e exclude <reason> written at requirement level exempts every scenario under that requirement — including scenarios the stated reason does not describe. Combined with #345 (an exclude is read as positive coverage), one sentence can turn up to 32 scenarios green.

Measured on ConductionNL/openbuild@development, using the gate's own parser:

scenarios 767   excluded 509
distinct exclusion reasons: 80
reasons covering MORE THAN ONE scenario: 42
scenarios excluded by a SHARED reason: 471      <- 92.5% of all exclusions

Top fan-outs — one reason, N scenarios:

N	spec	reason (truncated)
32	application-versions	pure-backend schema/lifecycle/CRUD spec — all scenarios are OR REST contracts…
29	openbuild-runtime	(inherited from requirement)
29	version-promotion	mixed spec — backend promotion endpoint + strategy logic verified by Newman/PHPUnit; UI dialog scenarios…
26	version-routing	mixed spec — server-side RBAC gate, ManifestResolverService, and store/composable contracts…
25	openbuild-exporter	mixed spec — ExportJob schema/lifecycle, ZIP/GitHub pipeline, async background-job…
20	openbuild-application-register	pure-backend schema/data spec — all scenarios are OR REST + repair-step contracts…

Note how many begin with "mixed spec — …". The author is explicitly saying some of these scenarios are backend-only and some are not, and the tag then exempts all of them uniformly.

A worked example where the reason is provably wrong for the scenario it exempts

openspec/specs/save-as-template/spec.md, REQ-SAT-001:

### Requirement: REQ-SAT-001 "Save as template" action and dialog on the application-detail surface
@e2e exclude pure-logic capture contract — verifying the captured record carries schema
definitions only and no object rows is a property of the pure `captureTemplate` function,
covered by Vitest; no Playwright-observable UI surface for record internals.
...
#### Scenario: Viewer cannot save a template
- **GIVEN** a user with only viewer rights on the Application

The reason is about the internals of a captured record. The scenario it silently exempts is about whether a viewer sees a button — as Playwright-observable as anything in the suite, and writable today (tests/e2e/global-setup.ts mints .auth/rbac-viewer.json).

Same shape at REQ-SAT-005, whose reason is about the canManage Vitest tests and swallows Scenario: Seeded cards remain read-only.

Why this is hard to see from outside

1. Report mode prints excluded: 509 as a single number, so a reason doing the work of 32 looks exactly like 32 reasons doing the work of one.
2. coverage_pct is computed as covered / (scenarios − excluded) — openbuild reports 27.9%, which sounds like a coverage problem in the 258 remaining scenarios and says nothing about the 509 removed from the denominator. The unqualified figure is 72/767 = 9.4%.
3. Per gate-19: '@e2e spec::slug exclude <reason>' in a test file is recorded as POSITIVE coverage — the regex stops at the slug #345 the gate cannot tell an exclusion from real coverage anyway, so nothing downstream re-examines them.

Suggested fix — deliberately non-reddening

I have not implemented this, on purpose: any change to exclusion accounting reddens many repos at once, and #347 is already being held for that reason. Landing it needs to be sequenced.

Two changes, neither of which fails a build:

1. Report mode: emit per-reason fan-out — {reason, spec, requirement, scenario_count, scenarios[]} — so a reason covering 32 scenarios is visible as one line. Also emit the unqualified covered / scenarios alongside the exclusion-adjusted percentage.
2. Gate mode: print an advisory (not a failure) when one reason exempts more than N scenarios, naming the requirement — [gate-19] NOTE: 1 exclusion reason exempts 32 scenarios in application-versions.

A stricter follow-up worth discussing separately: require a requirement-level exclusion to be repeated per scenario once it would exempt more than one, forcing the author to re-read the reason against each scenario. That one would redden repos, so it should be staged behind a flag.

The general lesson

⚠️ An exemption's reason is a testable claim, and it is only tested against the scenario the author had in mind while writing it. Every sibling scenario inherits a justification nobody ever checked against it. In this repo that is 471 scenarios carrying a reason written for something else.

Related: #343 (file-level @e2e tag credited without checking the test body), #345 (an @e2e exclude read as POSITIVE coverage).
Concrete instance in an app repo: ConductionNL/openbuild#178.

[rubenvdlinde]

A correctness note for the proposed fix, from re-measuring openbuild (the repo this was filed from).

#356 proposes a per-reason fan-out in report mode. Implemented naively that
report would be misleading, because _make_scenario_entry does not store a
reason for requirement-level exclusions at all — it stores the literal
sentinel "<inherited from requirement>":

elif current_req_excluded:
    final_excluded = True
    final_reason = None if current_req_bare else "<inherited from requirement>"

In openbuild that sentinel is the exclude_reason of 238 of 509 exclusions
(46.8%). So a fan-out grouped on exclude_reason reports one enormous bucket
whose "reason" is a marker, and every distinct requirement justification behind
it is invisible — which is the opposite of what the report is for.

Measured both ways on one tree (ff25c1be5), grouping by (spec, reason):

	distinct	covering >1	scenarios on a shared reason	biggest
exclude_reason as stored	80	42	471 (92.5%)	32
requirement text recovered	172	100	437 (85.9%)	32

So the fix needs _make_scenario_entry to carry the requirement's actual
reason text through (and ideally the exclusion LAYER too) before any per-reason
report can be honest. Both are available at the call site and simply discarded.

The layer is worth emitting for its own sake — the three fail differently, and
no app's split predicts another's:

layer	openbuild	nldesign (from nldesign#263)
whole-spec	217 (42.6%)	61.5%
requirement-level	238 (46.8%)	6.8%
scenario-level	54 (10.6%)	31.6%

And a measurement caveat for anyone re-running this: I first grouped by
reason alone, got 58/21/472 against the original 80/42/471, and nearly filed the
original figure as an artifact. The original grouped by (spec, reason) and
reproduces exactly under its own key. Reproduce a predecessor's number under
their method before correcting it.

Finally, the size of what inheritance is holding, measured by re-running report
mode with both inherited layers disabled and nothing else changed:

	covered	excluded	uncovered
as shipped	98	509	160
scenario-level markers only	100	54	613

455 scenarios are excluded by an inherited marker only; 453 carry no anchor
at all. openbuild's reported e2e debt understates the scenario-level-justified
figure by 3.8×.

[rubenvdlinde]

Sequencing note from the gate-fixes wave (2026-08-11). Landed this session, smallest blast radius first, each with a fixture observed red-before / green-after: #366 (the gate-16 fixture #364 shipped without), #368 (gate-7 hasPermission/canAccess, #360), #370 (gate-24 both registration APIs, #349), #371 (gate-61 honest empty-scope reason, #347).

This one was deliberately NOT landed in that wave. The per-issue reason is in the next comment / in fleet-board/findings/github-gate-fixes.md; the common one is that every remaining item either changes what a gate counts rather than what it says, or lives in quality.yml where run-helper-suites.sh cannot reach it — and four gate changes in one session is already the limit at which the next fleet measurement stays attributable.