Repository navigation
test(validation): migrate canonical release checks to auto-testing - #113
Merged
Merged
Conversation
cxymds
changed the title
cxymds
marked this pull request as ready for review
October 28, 2026 10:09
This was referenced Oct 29, 2026
overtrue
pushed a commit
to rustfs/rustfs
that referenced
this pull request
Nov 1, 2026
…#9107) * ci(validation): sweep the migrated canonical validation cases nightly The canonical release-validation cases migrated into auto-testing by rustfs/auto-testing#113 (66 implemented, all non-heal) currently run only through the manual validation flow; the last fleet executions stopped on 2026-10-27 and nothing schedules them. Add a scheduled validation sweep that closes the gap: - nightly at 19:30 UTC (after the functional chain settles), rotating windows of the implemented catalog cases so the whole catalog is covered roughly weekly, with a bounded batch per night - workflow_dispatch for a single case (--case), one module (--module), or an explicit batch (--limit), mirroring how the lanes are testable standalone - fleet admission through functional_environment enter/finish with the same quarantine discipline as the lanes, inter-case node cleanup, per-case logs + reports retained as a sweep artifact - a markdown sweep report uploaded to rustfs/dashboard under functional-reports/validation/ so the cases appear next to the lane reports, plus an alert-on-failure job for scheduled runs - heal.* cases stay on the interactive validation flow: many need the reviewed native/tool collectors built by separate validation-tools runs before fleet execution The registry gains the workflow responsibility (monitor: workflow, max-age 60h so the watchdog notices a dead sweep) and the wiring test baseline count moves with it. Verified locally: check_wiring, check_test_wiring, test_validation_registry (9/9), test_functional_chain + test_functional_chain_health + test_security_workflow (99/99). * ci(validation): group the sweep report by catalog module The validation cases cover distinct areas (pool lifecycle, ILM expiration, bucket/site replication, kms, data services, identity, s3 compatibility), so a flat case table hides the category breakdown. Render one section per catalog module with its own case table and a Subtotal line, and keep the single grand - Total:/- PASS:/- FAIL: block for the dashboard overview parser. Subtotals deliberately use a different bullet prefix so the overview regexes cannot mistake a module subtotal for the grand total. * ci(validation): sweep the eight planned cases whose executors exist The planned catalog entries are not all unimplemented: all eight (kms.sse-kms-round-trip, tiering.transition-and-restore, events.delivery, audit.request-correlation, iam.policy-enforcement, oidc.login-and-session, s3.basic-object-round-trip and scanner.usage-convergence) have dedicated executors routed in run_validation_case.sh and live PASS evidence recorded in the migration map. Widen the sweep filter to include them: the rotating nightly batch now covers 74 cases (66 implemented + these 8), and the fresh evidence each sweep produces feeds their acceptance review. heal.* procedure-ready cases remain excluded - their executors need the reviewed native/tool collectors built by separate validation-tools runs before fleet execution. --------- Co-authored-by: Hauser Zhi <housemecn@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Migrate all 158 canonical release-validation cases into auto-testing's catalog, dispatchers, execution policies, immutable candidate binding, evidence contracts and cleanup. Preserve source IDs, fixtures, assertions and required topologies; reuse existing executors where the source semantics match. The frozen catalog is commit
524eb5b32c3236497fdd8a9271df0494fa3d9974, SHA-256463fdd02dee43e42fef4f3aacf7506709ce87af535e1a53d91c521bc2e361cad.The current implementation review records 145 implemented cases, 13 environment procedures disabled by default, and zero partial implementations. The 13 procedures preserve control flows, capability checks, original oracles and unmet reasons; execution still requires the stated resources and a reviewed resource adapter. This delivery does not claim all 158 cases can run on the current fleet. See the implementation review and
validation-migration-review.jsonfor per-case source coverage and code digests.Execution follows auto-testing conventions. Service deployment consumes an immutable package and pinned tools; ordinary live cases do not compile a local RustFS checkout. The companion Functional Chain driver checks out its own orchestration scripts, and candidate-source compilation remains isolated to hosted package/tool jobs. The old validation framework is not a runtime dependency. Companion workflow and existing product changes: rustfs/rustfs#8455.
Completed source execution paths
source-plan: timeout/H1, timeout/H2, queued-successor/H1 and queued-successor/H2 execute serially with independent namespaces. Preflight verifies owner and append/delete generation subsets from the same successful native artifact. The evidence join binds original request/ACK bytes, request and lock identities, candidate/run identities, and coordinator timing. Execution stops and retains the scene on the first execution, evidence-chain or cleanup failure.validation_report.pyproduct admission remain unchanged.Independent live acceptance
There are 43 cases without complete live acceptance, including the 13 disabled environment cases. The other 30 already have implementations. These counts aggregate immutable candidates and historical audits; they do not certify one candidate across all 158 cases. The 22 retained failures have not all been attributed to product defects. Product, environment and harness causes must be evaluated from the original evidence rather than inferred from FAIL alone. Diagnostic, native, recovery and partial results never increase canonical PASS.
G06's eight H1/H2 public branches are independently accepted from 40427815826/1, including deterministic ACK state, physical and peer S3 evidence, overlap and all 20 cleanup receipts. Its earlier failure and interrupted run remain unaccepted historical evidence. G01/G02's later live failures are retained; their earlier successful native prerequisites do not replace those results.
Verification
Implementation commit:
88fcc621cbaf42b3d8b4ae3c76426ff4f1c3d760.python3 -B scripts/validation_catalog.py checkandpython3 -B scripts/validation_migration_review.py --require-completepassed.bash -n scripts/validation_heal/run_case.sh, fullactionlint .github/workflows/validation-catalog-tests.ymlincluding ShellCheck, andgit diff --checkpassed.40092a3a35a7ea966a484662bdfd33207dfc99e2: 53 Functional Chain tests passed, including actual-shell failure propagation and environment export behavior for native preflight. Full actionlint including ShellCheck and whitespace checks passed.No new fleet/live workflow was dispatched for this implementation completion. The complete composed F07/I05 paths still require independent live audit; their canonical gates continue to refuse PASS. Local checks and source review establish the delivered test implementation, not a product certification.
Impact and rollback
Both existing PRs remain draft. No additional PR or merge is requested. Retain the 13 environment gates and all original failure/evidence records. Reverting the latest helper/workflow commits restores the previous orchestration while preserving historical results and private retained scenes.