Skip to content

fix: bind render measurements to acquired input identity - #43

Merged
owjs3901 merged 1 commit into
mainfrom
integration/harness-determinism
Sep 13, 2026
Merged

fix: bind render measurements to acquired input identity#43
owjs3901 merged 1 commit into
mainfrom
integration/harness-determinism

Conversation

@owjs3901

Copy link
Copy Markdown
Contributor

The harness appeared nondeterministic: the same commit and binary produced about theme d3137cabf5a5/11.24% in one session and 67d6de70e679/7.44% in another. It was not nondeterminism. It was stale inputs being attributed to the wrong generator, and the harness had no way to notice.

Root cause

render.mjs never invokes the generator. It renders whatever already sits under themes/, src/screens/ and out/, and nothing recorded which binary produced those files - so pointing DEVUP_MCP_BIN at a current build proved nothing about what was actually being measured.

The old d3137cabf5a5 theme survives in the W11 worktree. Diffed against 67d6de70e679 it has exactly 25 changes, all typography lineHeight - unitless ratios like 1.6 becoming "26px". That is precisely what 757dc4a changed. Reversing only that conversion reproduces popup's f26ad027d6de exactly, final newline included.

Controlled reproduction:

About inputs Theme Divergence
current modules + current theme 67d6de70e679 7.46 / 4.06 / 2.41
current modules + saved W11 theme d3137cabf5a5 7.46 / 4.06 / 2.41
saved W11 modules + saved W11 theme d3137cabf5a5 11.24 / 6.92 / 4.37

The last row reproduces the bad session exactly. The asymmetry is explained too: about and popup use percentage typography and were affected by that conversion; grid and keyframes have no typography entries and landing already used pixel line heights, which is why those three always reproduced.

Every lead was checked and eliminated with specifics rather than assumption - CallCache::path_for computes one filename and never elects among candidates; node scope changes no used-token set; targets group by theme content hash with no leader race. The one that was real is accumulated artifacts.

The fix

Acquisition removes the outputs it is about to request, saves the full server response, checks completion status and required files, and records a failed frame as skipped while continuing instead of dying. Each target records the server identity, executable SHA-256, and hashes of its module, theme, reference PNG, snapshot and assets. Rendering validates those before building and refuses a missing explicit theme rather than falling back to another file.

Verification

Reproduced independently in the main checkout, two fresh acquire/render rounds, all 15 screens identical to the decimal, exit 0 both times.

The refusal also works: pointing the new render.mjs at inputs acquired by a different binary correctly failed with popup-422-5682: binary differs from acquisition; reacquire this screen - the exact failure mode that produced the bad measurement now gets caught instead of silently rendering.

python scripts/acquire.py with no argument previously died on report with a raw FileNotFoundError, aborting every screen after it. It now completes with 15 targets: notice x3 and report are measurable for the first time.

Thresholds

Updated only after both rounds matched, and now backed by repeated measurement. Some values increase - popup-422-5682 3.59 to 3.64, notice-422-7088 3.24 to 3.35. Those are corrected baselines, not regressions: the lower numbers came from inputs generated before 757dc4a and were never reproducible. popup-answer-* is untouched and explicitly not remeasured.

No generator, theme, provenance, golden or manifest file was touched. Korean keep-all is untouched. The four pre-existing pyright errors in acquire.py are fixed since that code was being edited anyway.

Gates: fmt 0, clippy 0, cargo test --workspace 1059 passed / 0 failed / 2 ignored, insta --check no drift.

@owjs3901
owjs3901 merged commit 9c27de0 into main Sep 13, 2026
8 checks passed
@owjs3901
owjs3901 deleted the integration/harness-determinism branch September 13, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant