Skip to content

feat(orchestrator): per-sandbox host-pool working set and reclaim estimate (density observability, off by default) - #3766

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/sandbox-density-metrics
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:feat/sandbox-density-metrics

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

First step toward #3763 (raising sandboxes-per-node). Observability only, off by default — nothing is reclaimed, paused, or shared here. It establishes the data floor that later density work (live balloon reclaim, same-template page sharing, pool-pressure auto-pause) needs to justify and tune itself.

Why

Guest RAM is faulted into the hugepage pool lazily (UFFD + hugetlbfs), so a node's real ceiling is the working set actually faulted in, not the RAM sandboxes nominally declare — and an idle sandbox keeps holding pool pages it no longer uses, because nothing reclaims them at runtime. Today there is no per-sandbox visibility into either number, so the density conversation is all estimates.

What

Two per-sandbox gauges, gated behind the sandbox-density-metrics flag (default false → no new series until an operator enables it):

  • orchestrator.sandbox.mem.working_set — host bytes faulted into the pool for the sandbox since resume. Read from the UFFD handler's existing ServeStats().Bytes (no new fault-path instrumentation) via a new Sandbox.MemoryWorkingSetBytes() accessor. Upper bound on current pool residency.
  • orchestrator.sandbox.mem.reclaimable_estimate — pool memory the node holds that the guest is not actively using: max(0, workingSet − guestInUse), where guestInUse is the envd-reported used RAM. Combined with free-page reporting: when the sandbox's balloon returns freed pages itself (BalloonModeReporting), the pool is self-reclaiming, so the estimate is reported as 0 rather than double-counting pages the device already discards.

The estimate feeds a later reclaim decision, so it is floored at zero and never over-reports. reclaimableBytes is a pure function with table tests: gap, floor-at-zero (never negative), no-guest-report upper bound, free-page-reporting → 0, zero footprint.

Safety / scope

  • Off by default: zero behaviour change and zero added metric cardinality until enabled.
  • No hot-path change: working-set comes from counters the UFFD handler already maintains; the metrics callback just reads them.
  • Pure-function test + existing SandboxObserver regression suite pass; go build/go vet clean.

Refs #3763. This is PR 1 of a planned sequence; it intentionally ships only the measurement so the subsequent reclaim/sharing/tiering PRs can cite real per-sandbox data.

…aim estimate

Density observability, off by default. Guest RAM is faulted into the hugepage
pool lazily (UFFD + hugetlbfs), so a node's real ceiling is the working set
actually faulted in, not the RAM sandboxes nominally declare — and an idle
sandbox keeps holding pool pages it no longer uses, because nothing reclaims
them at runtime. Today there is no per-sandbox visibility into either, so any
reclaim/sharing/tiering work has no data to justify or tune against.

This adds two per-sandbox gauges, gated behind the sandbox-density-metrics
flag (default false → no new series until enabled):

- orchestrator.sandbox.mem.working_set: host bytes faulted into the pool for
  the sandbox since resume, read from the UFFD handler's existing ServeStats
  (no new instrumentation on the fault path) via a new Sandbox accessor
  MemoryWorkingSetBytes. An upper bound on current pool residency.
- orchestrator.sandbox.mem.reclaimable_estimate: pool memory the node holds
  that the guest is not actively using — max(0, workingSet - guestInUse),
  where guestInUse is the envd-reported used RAM. Combined with free-page
  reporting: when the sandbox's balloon returns freed pages itself
  (BalloonModeReporting), the pool is self-reclaiming and the estimate is 0,
  so it does not double-count pages the device already discards.

The estimate feeds a later reclaim decision, so it is floored at zero and
never over-reports; reclaimableBytes is a pure function with table tests
(gap, floor-at-zero, no-guest-report upper bound, free-page-reporting zero).

Scope: observability only — nothing is reclaimed, paused or shared here. It is
the data floor for later density work (live balloon reclaim, same-template
page sharing, pool-pressure auto-pause); see e2b-dev#3763. Off by default means zero
behaviour and zero added metric cardinality until an operator turns it on.

Refs e2b-dev#3763.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants