fix(gate-19): read the scenarios of an open change - #797
Merged
Merged
Conversation
Gate 19 read openspec/specs/ only. A change's scenarios live in openspec/changes/<change>/specs/<cap>/spec.md until it is archived, so the PR that writes a change and the PRs that implement it came back EMPTY SCOPE and the scenarios reached the gate after the code had merged. A touched delta spec is now in scope, and so is every delta spec of an open change the diff touches any file in (an implementation PR ticks tasks.md). A full sweep reads open changes too. A scenario a MODIFIED block restates from the main spec is not new, REMOVED and RENAMED blocks owe no test, and openspec/changes/archive/ is skipped. The shape test now calls the real selector instead of a copy of it. Fixes ConductionNL/hydra#711
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gate 19 (e2e-coverage) never read the scenarios of an open change (ConductionNL/hydra#711). A change's scenarios live in
openspec/changes/<change>/specs/<cap>/spec.mduntil it is archived, and the gate readopenspec/specs/only. So the PR that writes a change and the PRs that implement it came back EMPTY SCOPE, and the runner turned that into a skip inside a green summary. On 28 Sep that left 3,897 scenarios in six repos unread, including every@e2e excludereason written inside a change.What changes:
tasks.mdand does not edit its specs.<cap>::<slug>, the ref it keeps after archive, so an anchor written against the change keeps working.MODIFIEDblock restates from the main spec is not new, so its existing exclusion still counts.REMOVEDandRENAMEDblocks owe no test.openspec/changes/archive/is skipped.openspec/changes/exists. Report mode is unchanged and still measures the main specs.ChangedSpecFilesShapeTestused to re-implement the filter it tested and asserted that a change delta is not a spec. It now calls the realselect_spec_paths.The gate stays advisory (.github#477), so this adds lines to the advisory channel and blocks nothing. One consequence to know: a PR that writes a change now reports each scenario it adds as "new scenario without a test", because the gate's existing rule says a scenario a change adds cannot be excluded.
Red tests (all red before the fix, green after): the 8 tests in the new
OpenChangeScenarioTest, among themtest_a_change_pr_that_adds_a_delta_scenario_is_checkedandtest_an_implementation_pr_that_ticks_tasks_reads_the_change_scenarios, plusChangedSpecFilesShapeTest.test_selects_both_shapes_and_rejects_the_rest.Live check on learniq development (055b9d9), the issue's recipe: a branch that only adds
openspec/changes/zz-livecheck/specs/zz-cap/spec.mdwith one UI scenario. Before: EMPTY SCOPE, exit 3. After:zz-cap::the-banner-shows-on-the-dashboard — new scenario without a test, exit 1. A full sweep of the same tree went from 92 to 579 uncovered scenarios (809 scenarios sit in its open changes).Exit codes:
test_check_e2e_coverage.py0 (199 tests),tests/test-hydra-gates-bin.sh0 (70 passed),bash -non the runner 0, shellcheck at error level 0.tests/run-helper-suites.shis red on three suites that are red on main at fc3cfb3 too:test_check_connections_declaration.js(1),test_gate116_connections_declaration_scope.sh(1) andtest_gate_45_to_55_acceptance.sh(2, ajv not installed locally). Every other suite passed.This repo has no development branch, so the PR goes to main. The skill text in hydra (
.claude/skills/hydra-gate-e2e-coverage/SKILL.md) follows in a hydra PR.Fixes ConductionNL/hydra#711