disagg: adapt GC merge fan-in after S3 read failures (#10800)

[ti-chi-bot]

This is an automated cherry-pick of #10800

What problem does this PR solve?

Issue Number: close #10798, ref #10794

Problem Summary:

Background GC merges can still fan in too many segments after bounded S3 stream retries are exhausted. That keeps a single GC merge exposed to long remote-read windows, while medium forward seeks still prefer draining the existing stream instead of reopening from the target offset.

What is changed and how it works?

33e66a4a07 disagg: tighten S3 forward seek reopen threshold
bd070ca5fe disagg: classify retry-exhausted S3 reads
7cbe1147ae disagg: adapt GC merge fan-in after S3 errors
085cb158be disagg: clarify S3 retry exhaustion flow
1f87ade691 disagg: narrow GC cap adjustment triggers

• tighten the S3 forward-seek reopen threshold from 128 KiB to 64 KiB and update the boundary tests
• let S3RandomAccessFile::read and S3RandomAccessFile::seek throw ErrorCodes::S3_ERROR after the bounded stream retry budget is exhausted, with doc comments for the public APIs and concise comments around the retry-exhaustion branch
• add a table-level gc_mergeable_segments_cap so background GC merge fan-in is capped and can be reduced after S3_ERROR
• only reduce the cap when gcTrySegmentMerge fails with S3_ERROR, and recover the cap only after checkSegmentUpdate succeeds
• add failpoints and unit tests for cap limiting, S3-error-triggered reduction, recovery, and non-S3 failures

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• New Features

  • Background garbage collection now dynamically adjusts merge concurrency based on S3 error conditions, automatically recovering when operations succeed.
  • S3 operations now throw explicit error codes when retry budgets are exhausted, improving error handling clarity.

• Chores

  • Adjusted S3 stream forwarding threshold from 128KB to 64KB for optimized performance.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)

• release-8.5
• release-7.5
• release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 556e204b-b16c-460c-a55f-8a0253b71d49

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-04-14 05:22:23.949602794 +0000 UTC m=+1452149.154962851: ☑️ agreed by JaySon-Huang.
• 2026-04-14 05:36:36.460300747 +0000 UTC m=+1453001.665660794: ☑️ agreed by CalvinNeo.

[ti-chi-bot]

@kolafish: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, JaySon-Huang, JinheLin, kolafish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [CalvinNeo,JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment