Skip to content

fix(test): make VM failure reruns deterministic - #137

Merged
cxymds merged 3 commits into
mainfrom
cxymds/fix-vm-test-failures
Nov 4, 2026
Merged

cxymds merged 3 commits into
mainfrom
cxymds/fix-vm-test-failures

Conversation

@cxymds

@cxymds cxymds commented Nov 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Make the failed VM reruns deterministic without changing RustFS product code or the canonical assertions.
  • Add --log-file support to the security suite and preserve its full output in CI artifacts.
  • Stage S3 Tables packages on the data disk, retry transient downloads, and keep installation failures visible.
  • Propagate tier log-capture failures instead of silently accepting a crashed logger.
  • Make pool validation resets tolerate an uninstalled service and collect rolling RustFS log files as MPU-drain evidence.
  • Add a hard timeout and consecutive-failure cutoff to performance rounds so a stuck warp request cannot consume the whole job.
  • Allow pool expansion to run multiple bounded warp rounds until the required storage threshold is reached.
  • Fix the registry check so retired event names in comments are not treated as active workflow wiring.
  • Make the MRF permission fixture independent of the caller's umask.

Verification

  • python3 -m unittest -v test_rustfs_security test_rustfs_table test_sensitive_diagnostics test_rustfs_performance_testing
  • python3 -m unittest -v test_validation_pool
  • python3 -m unittest -v test_validation_migration.ImplementationReviewTest
  • python3 scripts/validation_registry.py check
  • cd scripts && python3 -m unittest -v test_functional_dispatch
  • bash test_rustfs_tier.sh
  • bash test_rustfs_pool_expand.sh
  • bash -n across all shell scripts

The VM live rerun is the next verification step. Product failures from the previous run remain out of scope for this PR.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Live setup aborts on an unbound remote RUN_ID, and storage monitoring can continue writing after a failed capacity measurement.

4 open issues
What changed in this PR

This PR improves VM validation rerun reliability in the RustFS test harness without changing product code or canonical assertions.

Changes:

  • Adds security log capture, data-disk package staging, and clearer logger failure propagation.
  • Adds performance timeouts and failure cutoffs, plus configurable pool-fill rounds and expanded diagnostics.
  • Updates regression tests, workflow registry checks, and permission fixtures.
File Description
validation-migration-review.json Refreshes hashes for modified validation files.
test_validation_pool.py Checks missing-service handling and file-log collection.
test_validation_heal_persistence_mrf.py Makes fixture permissions independent of umask.
test_rustfs_tier.sh Adjusts logger-crash regression tests.
test_rustfs_table.py Checks data-backed package staging and retries.
test_rustfs_security.py Verifies dry-run log capture.
test_rustfs_pool_expand.sh Tests service handling, error classification, and multiple rounds.
test_rustfs_performance_testing.py Adds timeout and failure-cutoff coverage.
scripts/​validation_registry.py Checks declared events rather than comment text.
scripts/​validation_pool/​run_case.sh Handles absent services and collects rolling logs.
scripts/​test_functional_dispatch.py Tests retired event names in comments.
rustfs-tier-test.sh Preserves logger signal-exit statuses.
rustfs-table-test.sh Stages packages on the data disk with download retries.
rustfs-security-test.sh Adds --log-file output capture.
rustfs_pool_expand.sh Adds configurable fill rounds and broader error detection.
rustfs_performance_testing.sh Adds round timeouts and consecutive-failure cutoffs.
rustfs_performance_test.sh Exposes and forwards benchmark limits.

Review effort: Balanced

Comment thread rustfs-table-test.sh Resolved
Comment thread rustfs_pool_expand.sh Resolved
Comment thread rustfs-security-test.sh Resolved
Comment thread rustfs_pool_expand.sh Resolved
@cxymds
cxymds merged commit 719d5c8 into main Nov 4, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants