Skip to content

fix(mem): use real /proc RSS on Linux (mimalloc current_rss undercounts, blinding backpressure)#861

Merged
DeusData merged 2 commits into
mainfrom
distill/776-linux-rss-undercount
Jul 4, 2026
Merged

fix(mem): use real /proc RSS on Linux (mimalloc current_rss undercounts, blinding backpressure)#861
DeusData merged 2 commits into
mainfrom
distill/776-linux-rss-undercount

Conversation

@DeusData

@DeusData DeusData commented Jul 4, 2026

Copy link
Copy Markdown
Owner

fix(mem): use real /proc RSS on Linux — mimalloc current_rss undercounts, blinding backpressure

Distilled from #776's commit 132460f5 (author @petercoxphoto), extracted as a
single atomic stability fix. The rest of #776 (index-time importance scoring,
the repo_map MCP tool, per-file cap / enforcing memory ceiling) is deferred as
separate features and is not included here.

Refs #832.

The bug

cbm_mem_rss() asks mimalloc for current_rss (mi_process_info(...)) and
returns it whenever it is > 0, only falling back to os_rss() (the
/proc/self/statm reader) when it is exactly 0.

On Linux that is wrong. mimalloc's _mi_prim_process_info()
(vendored/mimalloc/src/prim/unix/prim.c) never sets current_rss on Linux
— it only fills peak_rss from getrusage's ru_maxrss. So current_rss
keeps mi_process_info()'s default of pinfo.current_commit
(vendored/mimalloc/src/stats.c:555): mimalloc's own committed-page counter.
This project deliberately tunes that counter low in cbm_mem_init()
(mi_option_arena_eager_commit=0 + purge_decommits=1 + purge_delay=0, to
reduce upfront memory), so on Linux "current RSS" reads a few MB while the real
process RSS is multiple GB.

Consequence: cbm_mem_over_budget() and the parallel-parse backpressure spin
(pass_parallel.c) — and any memory ceiling — were blind to real memory
pressure on Linux
. The undercounted value is small-but-nonzero, so it even
defeats the existing current_rss > 0 ASan-only fallback guard.

macOS/Windows are unaffected: mimalloc sets current_rss correctly there via
task_info / GetProcessMemoryInfo.

The fix (per-platform, surgical)

In cbm_mem_rss(), on __linux__ prefer os_rss() (/proc/self/statm, already
implemented and unaffected by mimalloc's internal accounting) as the primary
source, falling back to mimalloc's value only if /proc is unavailable.
macOS/Windows behavior is unchanged (they still use mimalloc's accurate
current_rss). cbm_mem_peak_rss() is untouched — peak_rss is set correctly
on Linux via ru_maxrss.

This complements #752: it makes the RAM budget/backpressure tiers actually bite
on Linux instead of silently never firing.

Reproduce-first test — red/green

Added mem_rss_reflects_external_resident_memory to tests/test_mem.c. It makes
the two quantities diverge deterministically:

  1. mi_malloc() a small live block so mimalloc's committed counter is a small
    positive value (this both defeats the unfixed current_rss > 0 fallback
    guard and pins the reported value low). mi_malloc routes through mimalloc
    regardless of MI_OVERRIDE, so it works in the ASan test-runner too.
  2. Grow true process RSS by ~256 MB via a raw anonymous mmap — memory
    mimalloc's committed counter never sees, but /proc/self/statm does.
  3. Assert cbm_mem_rss() >= 128 MB.
  • Unfixed Linux: cbm_mem_rss() returns the ~few-MB committed counter →
    assertion FAILS → RED.
  • Fixed Linux: returns the /proc RSS (>= 256 MB) → GREEN.
  • macOS/Windows: current_rss already reflects the mapped region, so the
    test passes both before and after the fix — the RED manifests only on the
    Linux CI leg, which is exactly where the production undercount bit.

Which tier proves RED

The bug is Linux-only, so it cannot go RED on the macOS dev host (there
mi_process_info's current_rss is accurate via task_info). The mechanism was
verified locally with a standalone mimalloc probe replicating the production
allocator config and the test's alloc pattern: after mi_malloc(1MB) + a raw
256 MB mmap, mimalloc's current_commit = 7 MB while true OS resident =
258 MB. On Linux cbm_mem_rss() (unfixed) returns current_rss == that
current_commit (~7 MB) → far below the 128 MB threshold → RED. The Linux CI leg
of make test (test-runner mem) is the tier that turns this guard red on the
unfixed code and green on the fix.

Verification

  • make -f Makefile.cbm cbm — production binary (MI_OVERRIDE=1), -Werror clean.
  • make -f Makefile.cbm lint-ci — cppcheck + clang-format + NOLINT check clean.
  • CBM_INDEX_SUPERVISOR=0 ./build/c/test-runner mem — 34 passed (GREEN on macOS,
    as expected for a Linux-only bug).

…ts, blinding backpressure)

Distilled from #776's 132460f. This project sets arena_eager_commit=0 +
purge_decommits=1 + purge_delay=0 in cbm_mem_init to reduce upfront memory, so
mimalloc's committed-page counter reads low. On Linux mi_process_info() never
sets current_rss (vendored/mimalloc/src/prim/unix/prim.c only fills peak_rss),
so current_rss defaults to that low committed counter — cbm_mem_rss() returned a
few MB while true RSS was multiple GB, leaving cbm_mem_over_budget() backpressure,
the memory ceiling, and the host RAM tier blind on Linux.

Prefer os_rss() (/proc/self/statm) as the primary source on Linux; macOS and
Windows are unchanged (their mi_process_info current_rss is accurate via
task_info / GetProcessMemoryInfo). cbm_mem_peak_rss() is untouched. Complements
#752 by making the RAM tiers actually bite on Linux.

Reproduce-first: mem_rss_reflects_external_resident_memory pins mimalloc's
committed counter low with a live mi_malloc, then grows true RSS via a raw 256MB
mmap (invisible to mimalloc); unfixed Linux returns the ~few-MB committed counter
(RED), fixed Linux returns /proc RSS (GREEN). macOS/Windows pass either way, so
the RED manifests on the Linux CI leg.

Co-authored-by: petercoxphoto <info@petercox.ie>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData DeusData enabled auto-merge July 4, 2026 19:36
…-undercount

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData DeusData merged commit 610531a into main Jul 4, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant