[majinghe]

Why

The catalog's 85 heal cases never ran unattended: they need validation-tools artifacts (bitrot shard verifiers, heal metadata inspector, native regressions) compiled from the exact candidate source SHA, which only the interactive validation job wired up. This brings heal into the nightly fold like every other suite — closing the last coverage gap from #113's catalog.

What it does (nightly 07:00 UTC + manual dispatch)

1. Resolve the newest successful nightly candidate (same resolver as the sweep)
2. Dispatch rustfs-validation-tools.yml on rustfs/rustfs with candidate_sha + auto_testing_sha + all three build flags, poll to completion (~20-40 min)
3. Download the tools / native / shared artifacts and verify the inspector binding to the candidate (inspector.py --candidate — the same identity gate the interactive job uses)
4. Run 71 heal cases — every procedure-ready case minus the 13 operator-assisted environment scenarios — through the same run_validation_case.sh entry, with per-case 900 s timeout and fleet hygiene between cases
5. Report grouped by module; the 13 environment scenarios are listed as BLOCKED (they need forced-restart hosts / dedicated multi-set topologies by design — not silently skipped); FAIL/TIMEOUT keep the run red

Notes

• skip_tools dispatch input reuses the newest existing tools run for debugging
• Tool-run identity is verified twice: workflow path + success conclusion at build time, and the inspector↔candidate SHA binding at use time
• First run will establish real per-case durations; loop budget is 480 min (71 cases), job cap 600 min
• Diagnostic entrances (C11/F07/I05/J02) self-report BLOCKED via their own result writers when applicable

Verification plan

Dispatch from this branch with limit: 5 for a smoke pass, then a full run.

[majinghe]

Design correction after live verification

Dispatched a real validation-tools build for the current nightly candidate (run 41650347338):

• heal-metadata-inspector (tools + shared artifacts): success — builds for any candidate
• native-heal-regressions / manager-native-regressions: fail — both are pinned to reviewed reference candidates via hard-coded REVIEWED_CANDIDATES allowlists ("native regression contract requires the reviewed immutable candidate checkout")

Follow-up inspection of the executors showed the same allowlists gate the case runners: race.py source_contract() and the I05/F07 preflights raise on any non-reviewed source_sha — these cases cannot run against arbitrary nightlies by design.

Branch updated (d180d62):

1. Tools dispatch now build_shared_heal only (inspector + shared encoder proven to build per-candidate)
2. Downloads tools + shared artifacts only
3. Selection excludes, alongside the 13 environment scenarios, the 11 candidate-pinned executors (6 races, A09 nolock, I05/F07, C11/J02 diagnostics) — 60 cases run unattended
4. Case verdicts are now read from each case's result.json; self-reported BLOCKED (diagnostic gates) is informational, only FAIL / INFRA_ERROR / TIMEOUT keep the sweep red
5. Report lists both BLOCKED categories explicitly (environment-gated / candidate-pinned)