Skip to content

[improvement](cgroup) inactive_file should be treated as available memory to avoid query be cancelled (#64347)#64414

Open
yiguolei wants to merge 1 commit into
apache:branch-4.0from
yiguolei:branch-4.0
Open

[improvement](cgroup) inactive_file should be treated as available memory to avoid query be cancelled (#64347)#64414
yiguolei wants to merge 1 commit into
apache:branch-4.0from
yiguolei:branch-4.0

Conversation

@yiguolei

Copy link
Copy Markdown
Contributor

]Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Sometimes we may see this errors in cgroup or k8s environment. Allocator sys memory check failed: Cannot alloc:5343, ...,process memory used 85.41 GB exceed limit 108.00 GB or sys available memory 5.88 GB less than low water mark 6.00 GB.
The mem_limit term is false (85.41 < 108). The 5343-byte allocation is rejected only by sys available memory 5.88 GB < low water mark 6.00 GB. 5.88 GiB available implies cgroup_mem_usage of about 114 GiB, roughly 29 GiB above process memory used (85.41 GiB); that gap is unmapped read page cache. The kernel reclaims clean page cache before OOM, so the memory is available, but Doris cannot reclaim it and the rejection repeats on later allocations. (low water mark 6.00 GB is the default: min(120 - 108, 120 * 5%) = 6.)

Before this PR, cgroup_mem_usage = memory.current - inactive_file - slab_reclaimable. So some active files page cache is not treated as recycleable memory. So cgroup_mem_usage is a bit larger than RSS.

None

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
  • This is a refactor/code format and no logic has been changed.
    - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason

  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
  • Yes.

  • Confirm the release note

  • Confirm test cases

  • Confirm document

  • Add branch pick label

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…mory to avoid query be cancelled (apache#64347)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Sometimes we may see this errors in cgroup or k8s environment. Allocator
sys memory check failed: Cannot alloc:5343, ...,process memory used
85.41 GB exceed limit 108.00 GB or sys available memory 5.88 GB less
than low water mark 6.00 GB.
The mem_limit term is false (85.41 < 108). The 5343-byte allocation is
rejected only by sys available memory 5.88 GB < low water mark 6.00 GB.
5.88 GiB available implies cgroup_mem_usage of about 114 GiB, roughly 29
GiB above process memory used (85.41 GiB); that gap is unmapped read
page cache. The kernel reclaims clean page cache before OOM, so the
memory is available, but Doris cannot reclaim it and the rejection
repeats on later allocations. (low water mark 6.00 GB is the default:
min(120 - 108, 120 * 5%) = 6.)

Before this PR, cgroup_mem_usage = memory.current - inactive_file -
slab_reclaimable. So some active files page cache is not treated as
recycleable memory. So cgroup_mem_usage is a bit larger than RSS.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@yiguolei yiguolei requested a review from morningman as a code owner June 11, 2026 08:22
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 13.79% (4/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.44% (25513/35711)
Line Coverage 54.29% (270352/497975)
Region Coverage 51.84% (223885/431918)
Branch Coverage 53.25% (96373/180971)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants