Skip to content

fix(performance): 验证每轮清理并区分 HTTP 与逐对象删除错误 - #146

Merged
cxymds merged 2 commits into
mainfrom
cxymds/fix-performance-cleanup
Nov 10, 2026
Merged

cxymds merged 2 commits into
mainfrom
cxymds/fix-performance-cleanup

Conversation

@cxymds

@cxymds cxymds commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

压测准备和收尾的批量删除可能返回 HTTP 200,但 DeleteResult 内含逐对象 ServiceUnavailable。Warp 清理失败仍可能退出 0,固定桶的残留会在后续轮次重复清理,放大错误日志并混淆正式负载与清理结果。

本修改让每轮创建独立桶,使用 --noclear 将清理交给测试脚本,单独限制清理时间和重试次数,并复核对象残留、未完成 multipart 和桶删除。清理重试曾出现错误时仍判整轮 FAIL;清理未完成则停止矩阵,后续用例明确记录 NOT_RUN。删除错误在每次尝试后立即保存;清理曾失败或被中断的标记跨后续恢复保留,重新解析也不能抹掉失败历史。

  • 保留原生 benchdata、准备/清理日志、桶创建凭据和逐请求 JSONL,分别统计 HTTP 503、请求失败与受影响键的失败条目;记录 request-id、SDK 重试次数和服务端提供的错误原因。
  • 凭据通过环境变量传给 Warp,避免进入原生结果的命令行。中断时尝试退出清理,重新解析仍使用独立报告,旧结果的清理状态为 unknown。
  • 新增批量删除专项工具,比较 1/32/128/500/1000 对象在空闲及 32 并发 GET 下的删除行为,保存首次删除结果和恢复前后残留;CI 上传上述证据。

验证:相关兼容性测试 61 项通过(包含真实 boto3 对本地 HTTP 服务解析 HTTP 200 XML 逐键错误、HTTP 503 禁重试,以及中断清理);全部 27 个 Shell 脚本语法检查通过,缺少 HOME 的回归通过,三个修改的工作流 YAML 解析通过。

这份 PR 修复测试套件的清理责任、失败判定和诊断证据。RustFS 底层锁/RPC/仲裁触发点尚未确认,不能据此声称产品故障已经修复。VM 专项验证按约定等待现有全量复测结束、环境恢复通过后进行。

Copilot AI balanced review requested due to automatic review settings November 10, 2026 08:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cleanup failures can be lost across retries or resumptions, allowing false PASS certification, while interruption during preparation can leave newly created buckets behind.

4 open issues
What changed in this PR

This PR moves benchmark cleanup responsibility from Warp to the test harness, adding per-round bucket isolation, cleanup verification, and diagnostic evidence. It addresses test reliability without claiming to fix the underlying RustFS failure.

Changes:

  • Add bounded cleanup, residue checks, lifecycle receipts, and cleanup-aware round verdicts.
  • Add a bulk-delete probe comparing idle operation with concurrent GET load.
  • Expand regression coverage, documentation, and CI evidence uploads.
File Description
test_rustfs_performance_testing.py Adds lifecycle, interruption, and reparse regression tests.
test_performance_s3.py Tests cleanup accounting, recovery, and HTTP response handling.
scripts/​performance_s3.py Implements bucket ownership, verified cleanup, and request evidence.
scripts/​performance_delete_probe.py Adds idle and GET-loaded bulk-delete diagnostics.
rustfs_performance_testing.sh Integrates isolated buckets, cleanup, and lifecycle reporting.
rustfs_performance_test.sh Adds cleanup prerequisites and configuration forwarding.
README.md Documents lifecycle behavior, evidence, and probe usage.
.github/​workflows/​rustfs-performance-test.yml Checks prerequisites and uploads diagnostic artifacts.
.github/​workflows/​performance-script-tests.yml Expands triggers and runs S3 helper tests.
.github/​workflows/​environment-compatibility-tests.yml Includes cleanup helpers in compatibility coverage.

Review effort: Balanced

Comment thread rustfs_performance_testing.sh Resolved
Comment thread scripts/performance_s3.py Resolved
Comment thread scripts/performance_s3.py Resolved
Comment thread scripts/performance_delete_probe.py Resolved
@cxymds
cxymds merged commit e9bdf0f into main Nov 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants