Skip to content

fix(mem): honor explicit worker budget override - #1657

Open
atirna wants to merge 9 commits into
DeusData:mainfrom
atirna:fix/index-worker-explicit-memory-budget
Open

fix(mem): honor explicit worker budget override#1657
atirna wants to merge 9 commits into
DeusData:mainfrom
atirna:fix/index-worker-explicit-memory-budget

Conversation

@atirna

@atirna atirna commented Aug 15, 2026

Copy link
Copy Markdown

What does this PR do?

CBM_MEM_BUDGET_MB is resolved before a supervised worker starts, but a worker was then silently reduced to the daemon's equal-share cap. On a 376 GB host with the default four job slots, an explicit 300000 MiB override could still become about 48 GB and hit mem.backpressure.futile.

This keeps the aggregate cap for the fraction-derived default, while preserving a valid explicit override (which is still clamped to detected RAM).

Verification

  • scripts/test.sh --suites mem (51 passed)
  • Added a regression guard that fails before the change when a 64 MiB explicit override is reduced to a 16 MiB worker cap.
  • scripts/lint.sh could not run on this host because cppcheck, clang-tidy, and clang-format are not installed.

Related to #1654 (override path only; does not close the default-slice or crash-vs-degrade parts)

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci) — unavailable locally
  • New behavior is covered by a test (reproduce-first for bug fixes)

Signed-off-by: Atirna <288419661+atirna@users.noreply.github.com>
@atirna
atirna requested a review from DeusData as a code owner August 15, 2026 14:07
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@DeusData

Copy link
Copy Markdown
Owner

Policy decision on what an explicit CBM_MEM_BUDGET_MB should mean — we're taking a middle path that differs from the PR as written, and we'd love you to carry it: the override becomes an AGGREGATE budget divided across job slots at the parent, rather than per-worker absolute. Reasoning: per-worker absolute lets N slots × override oversubscribe the host, which is precisely the #1654 OOM axis we just closed; aggregate-divided makes the override finally DO something (today it's silently eaten by the cap — your diagnosis of that silent reduction was correct and your source-string discriminator is sound) while the user's number remains a true ceiling. That likely means: keep your discriminator, divide the explicit value by the slot count where the equal-share cap is computed today, and restore the renamed policy test to assert the aggregate ceiling. Your CI red was pure GitHub infra (CodeQL init 503) — a rerun clears it. Up for the rework?

atirna and others added 2 commits August 18, 2026 11:13
Keep the source-string discriminator and still apply the parent
equal-share cap so N workers cannot oversubscribe the host.

Signed-off-by: Atirna <288419661+atirna@users.noreply.github.com>
@atirna

atirna commented Aug 18, 2026

Copy link
Copy Markdown
Author

yes, took the middle path.
CBM_MEM_BUDGET_MB stays an aggregate, parent still divides it across the job slots, and the worker cap still applies so N workers cant oversubscribe. kept the source string as CBM_MEM_BUDGET_MB when the env discriminator fired, so a clip isnt logged as daemon_worker_cap.
also restored resolve_budget_worker_cap_preserves_lower_user_override: 8 MiB explicit stays 8 under a 16 MiB slot share, 64 MiB clips to 16, fraction path still daemon_worker_cap.
scripts/test.sh --suites mem 52 passed.

@DeusData DeusData added bug Something isn't working stability/performance Server crashes, OOM, hangs, high CPU/memory priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Aug 18, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thank you for carrying the requested middle path. The latest revision now treats CBM_MEM_BUDGET_MB as an aggregate ceiling, preserves a lower user override, and clips a higher value to the per-slot share, which matches the maintainer decision without reopening the host-oversubscription axis from #1654.

The next step is review of the final commit and CI on that exact SHA; no further scope expansion is needed. Thank you for turning the policy correction around cleanly and restoring the test that distinguishes the lower explicit override from the clipped case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. stability/performance Server crashes, OOM, hangs, high CPU/memory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants