Skip to content

[improvement](be) Apply trash sweep policy to shutdown tablets - #66395

Draft
wenzhenghu wants to merge 1 commit into
apache:masterfrom
wenzhenghu:wzh/master-shutdown-tablet-direct-delete
Draft

[improvement](be) Apply trash sweep policy to shutdown tablets#66395
wenzhenghu wants to merge 1 commit into
apache:masterfrom
wenzhenghu:wzh/master-shutdown-tablet-direct-delete

Conversation

@wenzhenghu

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: None

Related PR: HYDCP#75

Shutdown tablets are processed after the DataDir trash sweep. Previously, even when a manual cleanup or high disk usage made the current sweep delete trash immediately, newly resolved shutdown tablet paths could still be moved back into trash according to the global retention configuration. This delayed disk-space reclamation and recreated trash entries during an urgent cleanup.

This PR:

  • builds one immutable sweep policy per DataDir and shares it between trash cleanup and shutdown-tablet resolution;
  • deletes eligible shutdown tablet paths directly when trash retention is disabled, during manual cleanup, or above the existing high-watermark threshold;
  • preserves move-to-trash behavior for normal retention sweeps;
  • leaves shutdown tablets on unused DataDirs in the queue for a later sweep;
  • keeps the existing transition, active-tablet, path, UID, TabletMeta, failure requeue, reference-count, and resolved-budget semantics;
  • adds direct-delete result, timing, and reason metrics plus an unused-DataDir deferred gauge;
  • adds BE unit coverage for the policy matrix, explicit filesystem modes, mixed DataDir policies, direct-delete failure retry, path-success/meta-failure retry, and intentional-skip metric semantics.

No FE-BE protocol, persistent format, or new configuration is introduced.

Release note

Shutdown tablet paths now follow the current DataDir trash sweep policy. During immediate cleanup conditions they are deleted directly instead of being moved into trash again.

Check List

  • Formatting: ./build-support/clang-format.sh and ./build-support/check-format.sh passed
  • Static checks: target-file clang-format 16 dry-run and git diff --check passed
  • BE build: not completed; the local build was interrupted during third-party dependency preparation before source compilation
  • BE unit tests: coverage added but not executed per the requested validation scope
  • Regression tests: not run
  • Behavior changed: Yes, only for shutdown tablets during immediate trash cleanup conditions
  • Documentation required: No

### What problem does this PR solve?

Issue Number: None

Related PR: HYDCP#75

Problem Summary: Shutdown tablets were moved into trash even when the current sweep was deleting trash immediately for manual cleanup, disabled retention, or high disk usage. This delayed disk-space reclamation and could recreate trash entries during an urgent cleanup. Build one immutable policy per DataDir and use it for both trash expiration and shutdown-tablet path resolution. Eligible shutdown tablet paths are deleted directly during immediate cleanup conditions and continue to move to trash during normal retention sweeps. Shutdown tablets on unused DataDirs remain deferred, and existing transition, UID, path, metadata, reference-count, and failure-requeue safeguards are preserved.

### Release note

Shutdown tablet paths now follow the current per-DataDir trash sweep policy. During immediate cleanup conditions they are deleted directly instead of being moved into trash again.

### Check List (For Author)

- Test: Unit Test coverage added but not executed per requested validation scope
    - Static checks: clang-format 16, build-support/check-format.sh, and git diff --check passed
    - BE build: Not completed; interrupted during third-party dependency preparation before source compilation
- Behavior changed: Yes. Shutdown tablet paths may be deleted directly during immediate trash cleanup conditions.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wenzhenghu
wenzhenghu marked this pull request as draft August 3, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants