Repository navigation
feat(orchestrator): per-sandbox host-pool working set and reclaim estimate (density observability, off by default) - #3766
Open
AdaAibaby wants to merge 1 commit into
Conversation
…aim estimate Density observability, off by default. Guest RAM is faulted into the hugepage pool lazily (UFFD + hugetlbfs), so a node's real ceiling is the working set actually faulted in, not the RAM sandboxes nominally declare — and an idle sandbox keeps holding pool pages it no longer uses, because nothing reclaims them at runtime. Today there is no per-sandbox visibility into either, so any reclaim/sharing/tiering work has no data to justify or tune against. This adds two per-sandbox gauges, gated behind the sandbox-density-metrics flag (default false → no new series until enabled): - orchestrator.sandbox.mem.working_set: host bytes faulted into the pool for the sandbox since resume, read from the UFFD handler's existing ServeStats (no new instrumentation on the fault path) via a new Sandbox accessor MemoryWorkingSetBytes. An upper bound on current pool residency. - orchestrator.sandbox.mem.reclaimable_estimate: pool memory the node holds that the guest is not actively using — max(0, workingSet - guestInUse), where guestInUse is the envd-reported used RAM. Combined with free-page reporting: when the sandbox's balloon returns freed pages itself (BalloonModeReporting), the pool is self-reclaiming and the estimate is 0, so it does not double-count pages the device already discards. The estimate feeds a later reclaim decision, so it is floored at zero and never over-reports; reclaimableBytes is a pure function with table tests (gap, floor-at-zero, no-guest-report upper bound, free-page-reporting zero). Scope: observability only — nothing is reclaimed, paused or shared here. It is the data floor for later density work (live balloon reclaim, same-template page sharing, pool-pressure auto-pause); see e2b-dev#3763. Off by default means zero behaviour and zero added metric cardinality until an operator turns it on. Refs e2b-dev#3763.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 29, 2026 08:20
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step toward #3763 (raising sandboxes-per-node). Observability only, off by default — nothing is reclaimed, paused, or shared here. It establishes the data floor that later density work (live balloon reclaim, same-template page sharing, pool-pressure auto-pause) needs to justify and tune itself.
Why
Guest RAM is faulted into the hugepage pool lazily (UFFD + hugetlbfs), so a node's real ceiling is the working set actually faulted in, not the RAM sandboxes nominally declare — and an idle sandbox keeps holding pool pages it no longer uses, because nothing reclaims them at runtime. Today there is no per-sandbox visibility into either number, so the density conversation is all estimates.
What
Two per-sandbox gauges, gated behind the
sandbox-density-metricsflag (default false → no new series until an operator enables it):orchestrator.sandbox.mem.working_set— host bytes faulted into the pool for the sandbox since resume. Read from the UFFD handler's existingServeStats().Bytes(no new fault-path instrumentation) via a newSandbox.MemoryWorkingSetBytes()accessor. Upper bound on current pool residency.orchestrator.sandbox.mem.reclaimable_estimate— pool memory the node holds that the guest is not actively using:max(0, workingSet − guestInUse), whereguestInUseis the envd-reported used RAM. Combined with free-page reporting: when the sandbox's balloon returns freed pages itself (BalloonModeReporting), the pool is self-reclaiming, so the estimate is reported as 0 rather than double-counting pages the device already discards.The estimate feeds a later reclaim decision, so it is floored at zero and never over-reports.
reclaimableBytesis a pure function with table tests: gap, floor-at-zero (never negative), no-guest-report upper bound, free-page-reporting → 0, zero footprint.Safety / scope
SandboxObserverregression suite pass;go build/go vetclean.Refs #3763. This is PR 1 of a planned sequence; it intentionally ships only the measurement so the subsequent reclaim/sharing/tiering PRs can cite real per-sandbox data.