Skip to content

ledger: spell the scenario corpus's enums the way the server reads them - #472

Merged
Yaraslaut merged 1 commit into
masterfrom
fix/460-ledger-scenario-enums
Sep 6, 2026
Merged

ledger: spell the scenario corpus's enums the way the server reads them#472
Yaraslaut merged 1 commit into
masterfrom
fix/460-ledger-scenario-enums

Conversation

@Yaraslaut

Copy link
Copy Markdown
Member

Fixes #460.

morph#444 gave every ledger enum a glz::meta/glz::enumerate, so glaze decodes and encodes them by enumerator name. The scenario corpus still spelled them as their underlying integers, so a real ladder_ledger_server refused every payload carrying one with expected_quote, and every reply assertion written against the integer form was wrong in the same way. This rewrites the corpus; #444 is correct and is untouched.

Measured before / after

Re-measured on the merge base 8bd4132b (post-#461), not taken from the issue. The issue's master column is stale in exactly the way it predicted it would be; its "master + #461" column is arithmetic, and it turns out to be right.

issue (master f6a4e464) measured here (8bd4132b, post-#461) after
--rung ledger 16 FAIL, 0 ok 15 FAIL, 1 ok 16 ok, 0 FAIL (twice)
whole corpus 73 files, 57 ok, 16 FAIL 73 files, 58 ok, 15 FAIL 73 files, 73 ok, 0 FAIL
files needing the rewrite 16 15 of 16
key=<int> send sites 105 101 0
escaped reply assertions 166 154 0
field <enum> <op> <int> 18 18 0
enum inside inline JSON 1 1 0
prose blocks asserting integers 7 8 (see below) 0

The one file that already passed is store-list-and-undo-an-entry.scenario, added by #461 and written in the enumerator-name form. Baseline failure, for the record:

FAIL step 5 (line 42): do OpenAccount ledgerId=1 name="Zero-sum EUR cash" kind=0 currency=1
  expected: @kind == ok
  actual:   @kind == err
  reply:    err message="1:49: expected_quote
   {"ledgerId":1,"name":"Zero-sum EUR cash","kind":0,"currency":1}
                                                   ^"

Prose is 8 blocks, not 7. The issue's list of seven omits store-list-and-undo-an-entry.scenario:52-55, whose note reads "Every other file in this directory still sends the pre-#444 integers and is refused by the running server for it" — true when written, false after this PR, so it is corrected too.

The rewrite is per-action

kind= names two enums. OpenAccount.kind is AccountKind; SubmitReport.kind is ReportKind. They collide ten lines apart in submit-a-report-and-poll-it.scenario (:56"Asset", :67"MonthlyStatement"), so a global kind=0 → "Asset" is wrong on the second. The rewrite resolves every site against the action of the do line it sits on or follows, and refuses to convert a site whose (action, field) pair it does not know — it reported zero such sites, which is how the 274 conversions are accounted for: 101 + 154 + 18 + 1.

The inline-JSON site is the trap a send-site grep misses: zero-sum-holds-per-currency.scenario:84's legs=[{…,"foreignCurrency":0},…], now "foreignCurrency":"USD".

The six vacuity-risk comparisons

Runner.compare (scripts/scenario/morph_scenario.py:952) is plain Python != on the parsed JSON value. Once status decodes as a string, expect ok field status != 2 reads "Pending" != 2 — trivially, permanently true. The file goes green while the check it exists for has stopped existing. All six are ReportStatus, all six asserted on a GetReportStatus reply, all six now compare against a quoted enumerator name:

site before after
submit-a-report-and-poll-it.scenario:70 field status != 2 field status != "Failed"
submit-a-report-and-poll-it.scenario:74 field status != 2 field status != "Failed"
submit-a-report-and-poll-it.scenario:81 field status != 2 field status != "Failed"
submit-a-report-and-poll-it.scenario:85 field status != 2 field status != "Failed"
submit-a-report-and-poll-it.scenario:100 field status != 2 field status != "Failed"
open-account-transact-report-close.scenario:117 field status != 2 field status != "Failed"

(The 12 == sites are safe by construction — "Asset" == 0 is false, so they fail loudly. two-books-are-isolated.scenario:106,109 went status == 0status == "Pending".)

Shown live, not asserted. Flipping :100 to != "Pending" and running it against a real server fails, which is only possible if the comparison reaches the real value:

FAIL step 16 (line 100): do GetReportStatus jobId=$statement
  expected: status != Pending
  actual:   status == Pending
  reply:    ok body={"status":"Pending"}

That reply body is also the direct confirmation that ReportStatus is a string on the wire now.

The static scan, and why --mutate is not the control

mutate_scenario.py cannot detect the vacuous class, and this was measured here rather than taken on trust. Seeding one dead status != 2 back into submit-a-report-and-poll-it.scenario, against a live seeded server:

$ python3 scripts/scenario/run_scenarios.py --rung ledger --build-dir <dir>
every scenario passed in: ledger                                        # EXIT 0

$ python3 scripts/scenario/morph_scenario.py .../submit-a-report-and-poll-it.scenario --server ws://…
submit-a-report-and-poll-it.scenario: 19 steps, 34 assertions, no failures

$ python3 scripts/scenario/mutate_scenario.py .../submit-a-report-and-poll-it.scenario --server ws://…
submit-a-report-and-poll-it.scenario: 28 mutants, 28 caught, 0 survived  # EXIT 0

Zero survivors on a file with a provably dead assertion. It is structural: mutation perturbs the scenario, not the server, and both mutants of a dead line die anyway — the kind flip dies on the reply's kind, and !=== dies because "Pending" == 2 is false.

The scan is what caught it. Seeded, caught, removed:

$ cd scripts/scenario/scenarios/ledger
$ grep -nE '(^|[^A-Za-z])(kind|currency|foreignCurrency|trigger|action|status)(\\")?"?[[:space:]]*(==|!=|=|:)[[:space:]]*-?[0-9]' *.scenario
submit-a-report-and-poll-it.scenario:100:expect ok field status != 2

Why this scan rather than the issue's three. It is one regex covering all four syntaxes at once — kind=0, "foreignCurrency":0, \"kind\":0, field status != 2 — where the issue's scan 1 requires a literal = and so is blind to the inline-JSON site (trap 1), which the issue has to cover by naming that one line by hand. It also picks up prose stating the integer form near one of those tokens, which is how three comment lines (submit-a-report-and-poll-it.scenario:29,33, open-account-transact-report-close.scenario:111) that no acceptance grep in the issue mentions were found and fixed. Checked as a superset: on the pre-change tree it matches 277 lines — the 274 code sites plus those 3 prose lines — in exactly the 15 files that needed rewriting, and zero lines in store-list-and-undo-an-entry.scenario.

All four scans, on the branch as it stands:

$ cd scripts/scenario/scenarios/ledger
$ grep -nE '\b(kind|currency|foreignCurrency|trigger|action)=[0-9]+' *.scenario
(exit 1)
$ grep -nE '\\"(kind|currency|status|trigger|action)\\":[0-9]+' *.scenario
(exit 1)
$ grep -nE 'field (kind|currency|status|trigger|action) (==|!=) [0-9]+' *.scenario
(exit 1)
$ grep -nE '(^|[^A-Za-z])(kind|currency|foreignCurrency|trigger|action|status)(\\")?"?[[:space:]]*(==|!=|=|:)[[:space:]]*-?[0-9]' *.scenario
(exit 1)

Verification

Release, GCC 16.2.1, own build dir configured -DMORPH_BUILD_LADDER=ON -DMORPH_LADDER_RUNGS=all -DMORPH_BUILD_NET=ON -DMORPH_BUILD_QT=ON.

$ python3 scripts/scenario/run_scenarios.py --rung ledger --build-dir <dir> --twice
ledger: 16 file(s) against ws://127.0.0.1:…
  ok   a-rule-and-the-category-it-sets.scenario
  ok   a-second-editor-on-the-same-book.scenario
  ok   a-signed-in-session-is-what-authorises-work.scenario
  ok   accounts-of-every-kind.scenario
  ok   an-unbalanced-entry-changes-nothing.scenario
  ok   bootstrap-a-book-over-the-wire.scenario
  ok   budget-limit-and-spend-report.scenario
  ok   categorise-an-account.scenario
  ok   exactly-once-store-transaction.scenario
  ok   import-a-statement-chunk.scenario
  ok   open-account-transact-report-close.scenario
  ok   store-list-and-undo-an-entry.scenario
  ok   submit-a-report-and-poll-it.scenario
  ok   two-books-are-isolated.scenario
  ok   update-rule-version-conflict.scenario
  ok   zero-sum-holds-per-currency.scenario
ledger: second pass, same database
  … all 16 ok again …

every scenario passed in: ledger                                 # EXIT 0
$ python3 scripts/scenario/run_scenarios.py --build-dir <dir>
pastebin: 12 …   bookmarks: 14 …   polls: 10 …   kanban: 21 …   ledger: 16 …
73 ok, 0 FAIL
every scenario passed in: pastebin, bookmarks, polls, kanban, ledger   # EXIT 0

$ python3 scripts/scenario/run_scenarios.py --rung ledger --build-dir <dir> --mutate
  ok   … (+ mutants)   × 16      0 survivors
every scenario passed in: ledger                                 # EXIT 0

$ python3 scripts/scenario/scenario_coverage.py                  # EXIT 0
  ledger     actions 19/19 dispatched (0 exempt), workflows 16/16
  total actions dispatched: 73/73

$ python3 scripts/scenario/test_morph_scenario.py                # EXIT 0
Ran 132 tests in 0.313s
OK

$ bash scripts/check_spec_citations.sh    # 0
$ bash scripts/check_rung_filters.sh      # 0
$ bash scripts/check_deprecated_markers.sh# 0

--mutate is reported as necessary but not sufficient — see the section above for why it is not the control for the defect this issue is about.

What is not verified here

  • scenario_coverage.py reports ledger actions 19/19 and total 73/73, not the 18/18 and 72/72 in the issue. Both numbers moved on master between the issue being written and this branch's merge base; nothing in this PR changes an action name, and the gate exits 0 either way.
  • Nothing was run on Windows or macOS, and no CI job executes this corpus — which is the whole of ci: nothing runs the scenario corpus, so 16 refused ledger scenarios stayed green for five days #462 and deliberately out of scope here.
  • test_morph_scenario.py was not edited. Its pins are on file names and counts; this PR changes file contents only, and the file set is unchanged, so no pin needed touching.
  • Only scripts/scenario/scenarios/ledger/** is modified — no tooling script, no examples/**.

🤖 Generated with Claude Code

morph#444 gave every ledger enum a `glz::meta`/`glz::enumerate`, so glaze
decodes and encodes them by enumerator name. The scenario corpus still spelled
them as their underlying integers, so a real `ladder_ledger_server` refused
every payload carrying one with `expected_quote` and every reply assertion
written against the integer form was wrong in the same way. Measured on
`8bd4132b`: `--rung ledger` was `15 FAIL, 1 ok` -- only
`store-list-and-undo-an-entry.scenario`, written after #444, passed -- and the
whole corpus `73 files, 58 ok, 15 FAIL`.

The rewrite is per-action, not a substitution, because `kind=` names two
different enums: `OpenAccount.kind` is `AccountKind` and `SubmitReport.kind` is
`ReportKind`, and they collide ten lines apart in
`submit-a-report-and-poll-it.scenario`. 274 sites in 15 files: 101 `key=<int>`
send sites, 154 escaped reply assertions (`\"kind\":0` -> `\"kind\":\"Asset\"`),
18 `field <enum> <op> <int>` comparisons, and one enum inside the inline
`legs=[...]` JSON (`zero-sum-holds-per-currency.scenario`'s
`"foreignCurrency":0`, which a send-site grep for `currency=` does not see).
Eight prose blocks that documented the integer wire form now document the
enumerator names, including the note in `store-list-and-undo-an-entry.scenario`
that said its siblings were still broken.

Six of the 18 comparisons were the dangerous ones. `Runner.compare`
(`morph_scenario.py:952`) is plain Python `!=`, so `expect ok field status != 2`
reads `"Pending" != 2` once `status` decodes as a string -- permanently true,
measuring nothing, and green. All six (`submit-a-report-and-poll-it.scenario`
70/74/81/85/100 and `open-account-transact-report-close.scenario:117`) are now
`!= "Failed"`. The two `== 0` sites in `two-books-are-isolated.scenario` are
`== "Pending"`.

Neither a green run nor a mutation run detects that class. Seeding one dead
`status != 2` back into `submit-a-report-and-poll-it.scenario` and running it
against a live server gave `19 steps, 34 assertions, no failures` and
`28 mutants, 28 caught, 0 survived` -- mutation perturbs the scenario, not the
server, and both mutants of a dead line die anyway. A static scan is the check
that catches it, and it did:

    $ grep -nE '(^|[^A-Za-z])(kind|currency|foreignCurrency|trigger|action|status)(\\")?"?[[:space:]]*(==|!=|=|:)[[:space:]]*-?[0-9]' *.scenario
    submit-a-report-and-poll-it.scenario:100:expect ok field status != 2

With the seed removed that scan returns nothing, as do the three narrower ones
in the issue. That the surviving comparisons are live rather than vacuous was
shown directly: flipping line 100 to `!= "Pending"` fails with
`reply: ok body={"status":"Pending"}`.

Verified: `--rung ledger --twice` green on both passes against one database;
the whole corpus `73 files, 73 ok, 0 FAIL` (bookmarks 14, kanban 21, pastebin
12, polls 10 unchanged); `--rung ledger --mutate` 0 survivors;
`scenario_coverage.py` exit 0 (`ledger actions 19/19 dispatched, workflows
16/16`); `test_morph_scenario.py` 132 tests, OK.

The missing CI gate that let this go unnoticed for five days is morph#462, and
stays out of scope here.

Fixes #460

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C6uB9zxdSG8qp3VNAAyFvF
@Yaraslaut
Yaraslaut merged commit 8a67ffe into master Sep 6, 2026
21 checks passed
@codecov

codecov Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

scenario: all 16 ledger scenarios are refused by a real server since morph#444 gave the enums glz::meta

1 participant