From 6043d46c461fcfe4886dd20582795e5cceb4de20 Mon Sep 17 00:00:00 2001 From: Yaraslau Tamashevich Date: Sun, 6 Sep 2026 23:09:19 +0200 Subject: [PATCH] ci: run the scenario corpus in ladder-tests, and let corpus changes reach it Nothing in CI executed scripts/scenario/run_scenarios.py. The corpus could therefore be refused wholesale by a real server with every workflow green, and was: sixteen of sixteen ledger scenarios refused for five days and through a merge (morph#460), while drift-guard.yml's scenario-coverage job reported "ledger actions 18/18 dispatched, workflows 16/16" the whole time. That job parses the corpus; only running it can see a refusal. Three changes. `ladder-tests` now runs the corpus after its ctest step. It is the right job: drift-guard.yml deferred this because running the corpus needs the ladder__server binaries and that workflow deliberately has no build, but `ladder-tests` already builds every one of them unconditionally (cmake/morph_add_rung.cmake). --build-dir names the build so no stale binary from another tree can be picked up, and --rung is passed once per corpus directory found on disk rather than left to the driver's default, so the set that runs is demonstrably the set that exists. A missing binary is not tolerated: run_scenarios.py exits 2, which is correct and is not softened. `scripts/scenario/` joins ci-path-regex. Without it a pull request touching only the corpus matched nothing and skipped `ladder-tests` altogether -- so the one gate that can catch a broken scenario would have been absent from exactly the changes most able to break one, which is morph#179's defect one directory over. check_rung_filters.sh now probes two corpus paths against the generated regex, behaviourally, and its self-test reintroduces the omission and requires the gate to catch it. A pre-flight check refuses a corpus this job could not have run. A scenarios// directory with no RungSpec drops out of the driver's default set in silence, and a corpus for a rung this job configures no server for is only reported after a full Qt + ladder build. Both are this issue's own failure one level down, so both are checked on a bare checkout in seconds. This is what resolves the bank interaction (morph#470): examples/bank is deliberately not in examples/rungs.txt and its server comes from a local add_executable(), so when its corpus lands the check fails by name rather than letting five of six directories run and calling that success. drift-guard.yml's comment saying this run "is its own change" now points at the change instead of deferring it. Fixes #462 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C6uB9zxdSG8qp3VNAAyFvF --- .github/workflows/ci.yml | 114 +++++++++++++++++++++++++++++ .github/workflows/drift-guard.yml | 9 ++- scripts/check_rung_filters.sh | 24 ++++++ scripts/ladder_rungs.sh | 11 +++ scripts/test_check_rung_filters.sh | 10 +++ 5 files changed, 165 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5068413de..c27a5ab4d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -856,6 +856,81 @@ jobs: echo "run=false" >> "$GITHUB_OUTPUT" fi + # ── Every scenario corpus must be one this job actually runs ────── + # The step at the end of this job runs scripts/scenario/run_scenarios.py + # over the corpus. A corpus can drop out of that run in two ways, and + # neither announces itself: + # + # * a scenarios// directory with no RungSpec in run_scenarios.py. + # The driver's default set is the intersection of the two, so an + # unknown directory is simply not in it -- and passing --rung + # for it is an argparse error, not a run. + # * a rung this job builds no server for. The driver exits 2 in that + # case (find_server), which is correct and deliberately not softened + # into a skip -- but it only says so after a full Qt + ladder build. + # + # Both are morph#462's own failure one level down: a gate that quietly + # runs five corpora out of six reports green exactly as loudly as one + # that ran them all. So they are checked here instead, on a bare + # checkout, in seconds, before anything is installed or compiled. + # + # This is what will fire when a corpus arrives for something that is not + # a ladder rung. examples/bank is the live case (morph#470): its server + # comes from a local add_executable() rather than morph_add_rung(), and + # bank is deliberately absent from examples/rungs.txt, so this job + # configures nothing for it. The job then goes red naming bank, and the + # fix is to make this job build that server and add it below -- not to + # let the corpus run four of six directories in silence. + - name: Check every scenario corpus has a server this job builds + if: steps.filter.outputs.run == 'true' + run: | + set -euo pipefail + corpora="$(find scripts/scenario/scenarios -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort)" + if [ -z "$corpora" ]; then + echo "::error::scripts/scenario/scenarios/ holds no corpus directories -- the scenario step at the end of this job would report success having run nothing" + exit 1 + fi + + # The rungs run_scenarios.py knows how to start a server for. Read + # out of the module rather than restated here: a second copy of this + # list is a second thing that can fall behind. + known="$(python3 - <<'PY' + import importlib.util + import sys + + spec = importlib.util.spec_from_file_location( + "run_scenarios", "scripts/scenario/run_scenarios.py") + module = importlib.util.module_from_spec(spec) + # Registered before exec_module: @dataclasses.dataclass looks its + # class's own module up in sys.modules while decorating, and raises + # if it is not there yet. + sys.modules[spec.name] = module + spec.loader.exec_module(module) + print("\n".join(sorted(module.RUNGS))) + PY + )" + + # The rungs this job configures: -DMORPH_LADDER_RUNGS=all below + # reaches exactly the names in examples/rungs.txt. + configured="$(bash scripts/ladder_rungs.sh list | sort)" + + status=0 + for rung in $corpora; do + if ! printf '%s\n' "$known" | grep -qxF "$rung"; then + echo "::error::scripts/scenario/scenarios/${rung}/ has no RungSpec in scripts/scenario/run_scenarios.py, so nothing can start a server for it and the driver would leave it out of its default set" + status=1 + elif ! printf '%s\n' "$configured" | grep -qxF "$rung"; then + echo "::error::scripts/scenario/scenarios/${rung}/ is a corpus for '${rung}', which is not listed in examples/rungs.txt -- this job configures no ladder_${rung}_server for it. Either list it there, or build its server in this job and extend this check to expect it." + status=1 + elif [ ! -d "examples/${rung}/src/server" ]; then + echo "::error::examples/${rung}/src/server/ does not exist, so cmake/morph_add_rung.cmake declares no ladder_${rung}_server to run scripts/scenario/scenarios/${rung}/ against" + status=1 + else + echo "ok: ${rung} -- corpus, RungSpec and ladder_${rung}_server all accounted for" + fi + done + exit "$status" + - name: Cache apt packages if: steps.filter.outputs.run == 'true' uses: actions/cache@v4 @@ -974,6 +1049,45 @@ jobs: QT_QPA_PLATFORM: offscreen run: ctest --preset gcc-debug -L ladder -LE stress --output-on-failure + # ── The scenario corpus, against the servers just built ──────────── + # Nothing in CI ran scripts/scenario/run_scenarios.py until this step + # (morph#462). The corpus could therefore be refused wholesale by a real + # server with every workflow still green -- and was: sixteen of sixteen + # ledger scenarios refused for five days and through a merge (morph#460), + # while drift-guard.yml's scenario-coverage job reported "ledger actions + # 18/18 dispatched, workflows 16/16" throughout. That job parses the + # corpus; only running it can see a refusal. + # + # Here rather than in drift-guard.yml, for the reason drift-guard.yml's + # own comment gives for deferring it: running the corpus needs the + # ladder__server binaries, which that workflow deliberately has no + # build to produce. This job already builds every one of them + # unconditionally (cmake/morph_add_rung.cmake), so the build this step + # needs is already paid for above. + # + # --rung is passed once per corpus directory found on disk rather than + # left to the driver's default: the set that runs is then demonstrably + # the set that exists, and the check near the top of this job has already + # refused any corpus this job could not have run. --build-dir names the + # build, so a stale binary in some other build tree cannot be picked up. + # + # A missing binary is not tolerated. run_scenarios.py exits 2 when a + # server is absent and that is the right answer; softening it into a skip + # would rebuild the blindness this step exists to end. + - name: Run the scenario corpus against the built servers + if: steps.filter.outputs.run == 'true' + env: + QT_QPA_PLATFORM: offscreen + run: | + set -euo pipefail + rungs=() + for dir in scripts/scenario/scenarios/*/; do + rungs+=(--rung "$(basename "$dir")") + done + echo "corpus rungs: ${rungs[*]}" + python3 scripts/scenario/run_scenarios.py \ + --build-dir build/gcc-debug "${rungs[@]}" + # ── Linux: every rung's tests under AddressSanitizer + UBSan ────────── # Before this job, no rung test ran under any sanitizer: ladder-tests above # builds plain gcc-debug, and the linux-sanitizers matrix deliberately skips diff --git a/.github/workflows/drift-guard.yml b/.github/workflows/drift-guard.yml index 4b6df3524..ec8fda757 100644 --- a/.github/workflows/drift-guard.yml +++ b/.github/workflows/drift-guard.yml @@ -256,9 +256,12 @@ jobs: # # What this job does NOT do is *run* the corpus: that needs the five # ladder__server binaries built, which is a build this workflow does - # not have and would stop being fast and dependency-free to acquire. Running - # scripts/scenario/run_scenarios.py in CI is its own change -- see the PR - # that added the corpus. + # not have and would stop being fast and dependency-free to acquire. That run + # lives in ci.yml's `ladder-tests` job, which already builds every one of + # those binaries -- see its "Run the scenario corpus against the built + # servers" step (morph#462). The two are complementary and neither replaces + # the other: this job sees a surface nothing covers, that one sees a covered + # surface a real server refuses. # # No path filter, on the same reasoning as rung-filter-lint above: the # surface this tracks is spread across include/, examples/ and diff --git a/scripts/check_rung_filters.sh b/scripts/check_rung_filters.sh index 9e3249ec2..506755dcb 100644 --- a/scripts/check_rung_filters.sh +++ b/scripts/check_rung_filters.sh @@ -149,6 +149,30 @@ else note "ci.yml contains no hand-written rung alternation" fi +# ── 1b. The scenario corpus must also trigger the ladder jobs ──────────────── +# ci.yml's `ladder-tests` runs scripts/scenario/run_scenarios.py against the +# rung servers it has just built, so scripts/scenario/ is now a ladder input in +# the same sense examples// is. It was not in the generated filter when +# that step was added (morph#462): a pull request that touched only the corpus +# matched nothing, skipped `ladder-tests` altogether, and so skipped the only +# job that can tell whether a scenario still passes -- on precisely the changes +# most able to break one. Same shape as morph#179, one directory over. +# +# Behavioural, like check 1: real paths are matched against the generated +# regex, so a pattern rewritten into something that matches nothing fails here +# rather than passing on the strength of containing the right words. +for probe in \ + "scripts/scenario/scenarios/ledger/probe.scenario" \ + "scripts/scenario/run_scenarios.py" +do + checks=$((checks + 1)) + if printf '%s\n' "$probe" | grep -qE "$ci_regex"; then + note "ci.yml ladder filter matches ${probe}" + else + fail "ci.yml ladder filter does NOT match ${probe} -- a change confined to the scenario corpus would skip ladder-tests, and with it the only job that runs the corpus" + fi +done + # ── 2. wasm-ladder.yml's literal path lists must cover every rung ──────────── for event in push pull_request; do globs="$(workflow_paths "$wasm_workflow" "$event")" diff --git a/scripts/ladder_rungs.sh b/scripts/ladder_rungs.sh index bf6f32169..f2af88385 100644 --- a/scripts/ladder_rungs.sh +++ b/scripts/ladder_rungs.sh @@ -43,6 +43,16 @@ readonly rung_file="${repo_root}/examples/rungs.txt" # this job's own definition, and this filter itself # examples/{LADDER,IMPLEMENTATION,TESTING}.md # the ladder's normative rules +# scripts/scenario/ the scenario corpus and its driver, which +# `ladder-tests` now *runs* against the servers it has +# just built (morph#462). Without this entry a pull +# request that touched only the corpus would match +# nothing and skip the job -- so the one gate that can +# catch a broken scenario would be absent from exactly +# the changes most able to break one. That is +# morph#179's defect in a second location, and +# drift-guard.yml's scenario-coverage job carries no +# path filter at all for the same reason. # Not `readonly`: this script has to stay runnable under the bash 3.2 that # macOS still ships, where `readonly` on an array assignment is a syntax error. _extra_patterns=( @@ -56,6 +66,7 @@ _extra_patterns=( 'CMakePresets\.json$' '\.github/workflows/ci\.yml$' 'scripts/ladder_rungs\.sh$' + 'scripts/scenario/' 'examples/LADDER\.md' 'examples/IMPLEMENTATION\.md' 'examples/TESTING\.md' diff --git a/scripts/test_check_rung_filters.sh b/scripts/test_check_rung_filters.sh index 7efa7a2df..f5072b327 100644 --- a/scripts/test_check_rung_filters.sh +++ b/scripts/test_check_rung_filters.sh @@ -118,6 +118,16 @@ expect_caught "the generated ci.yml regex silently omitting a rung" \ "edit scripts/ladder_rungs.sh -e 's@alternation+=\"examples/@[ \"\$rung\" = lims ] || alternation+=\"examples/@'" \ "ci.yml ladder filter does NOT match examples/lims/src/models/probe.cpp" +# The scenario corpus dropping out of the generated filter. This is the pin +# morph#462 asked for: ci.yml's `ladder-tests` runs the corpus, so a corpus-only +# pull request that skipped the job would skip the only run of it -- and a +# filter that stopped matching would say nothing at all, exactly as morph#179's +# did. Nothing about ci.yml changes here; the generator alone is made to drop +# the entry, and the gate must still notice. +expect_caught "the generated ci.yml regex silently omitting scripts/scenario/" \ + "edit scripts/ladder_rungs.sh -e \"/^ 'scripts\\/scenario\\/'\$/d\"" \ + "ci.yml ladder filter does NOT match scripts/scenario/scenarios/ledger/probe.scenario" + # The other direction: a rung directory that declares itself but is absent from # the authority, so nothing builds, tests or filters on it. expect_caught "a declared rung missing from examples/rungs.txt" \