Skip to content

fix(gate-19): read the scenarios of an open change - #797

Merged
rubenvdlinde merged 1 commit into
mainfrom
fix/hydra-711-gate19-change-specs
Sep 28, 2026
Merged

rubenvdlinde merged 1 commit into
mainfrom
fix/hydra-711-gate19-change-specs

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

Gate 19 (e2e-coverage) never read the scenarios of an open change (ConductionNL/hydra#711). A change's scenarios live in openspec/changes/<change>/specs/<cap>/spec.md until it is archived, and the gate read openspec/specs/ only. So the PR that writes a change and the PRs that implement it came back EMPTY SCOPE, and the runner turned that into a skip inside a green summary. On 28 Sep that left 3,897 scenarios in six repos unread, including every @e2e exclude reason written inside a change.

What changes:

  • A touched delta spec is in scope in diff mode. So is every delta spec of an open change the diff touches any file in, because an implementation PR ticks the change's tasks.md and does not edit its specs.
  • A full sweep (no base) reads the deltas of open changes too.
  • A delta scenario's ref is <cap>::<slug>, the ref it keeps after archive, so an anchor written against the change keeps working.
  • A scenario that a MODIFIED block restates from the main spec is not new, so its existing exclusion still counts. REMOVED and RENAMED blocks owe no test. openspec/changes/archive/ is skipped.
  • The runner enters gate 19 when only openspec/changes/ exists. Report mode is unchanged and still measures the main specs.
  • ChangedSpecFilesShapeTest used to re-implement the filter it tested and asserted that a change delta is not a spec. It now calls the real select_spec_paths.

The gate stays advisory (.github#477), so this adds lines to the advisory channel and blocks nothing. One consequence to know: a PR that writes a change now reports each scenario it adds as "new scenario without a test", because the gate's existing rule says a scenario a change adds cannot be excluded.

Red tests (all red before the fix, green after): the 8 tests in the new OpenChangeScenarioTest, among them test_a_change_pr_that_adds_a_delta_scenario_is_checked and test_an_implementation_pr_that_ticks_tasks_reads_the_change_scenarios, plus ChangedSpecFilesShapeTest.test_selects_both_shapes_and_rejects_the_rest.

Live check on learniq development (055b9d9), the issue's recipe: a branch that only adds openspec/changes/zz-livecheck/specs/zz-cap/spec.md with one UI scenario. Before: EMPTY SCOPE, exit 3. After: zz-cap::the-banner-shows-on-the-dashboard — new scenario without a test, exit 1. A full sweep of the same tree went from 92 to 579 uncovered scenarios (809 scenarios sit in its open changes).

Exit codes: test_check_e2e_coverage.py 0 (199 tests), tests/test-hydra-gates-bin.sh 0 (70 passed), bash -n on the runner 0, shellcheck at error level 0. tests/run-helper-suites.sh is red on three suites that are red on main at fc3cfb3 too: test_check_connections_declaration.js (1), test_gate116_connections_declaration_scope.sh (1) and test_gate_45_to_55_acceptance.sh (2, ajv not installed locally). Every other suite passed.

This repo has no development branch, so the PR goes to main. The skill text in hydra (.claude/skills/hydra-gate-e2e-coverage/SKILL.md) follows in a hydra PR.

Fixes ConductionNL/hydra#711

Gate 19 read openspec/specs/ only. A change's scenarios live in
openspec/changes/<change>/specs/<cap>/spec.md until it is archived, so the
PR that writes a change and the PRs that implement it came back EMPTY SCOPE
and the scenarios reached the gate after the code had merged.

A touched delta spec is now in scope, and so is every delta spec of an open
change the diff touches any file in (an implementation PR ticks tasks.md).
A full sweep reads open changes too. A scenario a MODIFIED block restates
from the main spec is not new, REMOVED and RENAMED blocks owe no test, and
openspec/changes/archive/ is skipped. The shape test now calls the real
selector instead of a copy of it.

Fixes ConductionNL/hydra#711
@rubenvdlinde
rubenvdlinde merged commit 0aebe3f into main Sep 28, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant