What happened
During a ~70-sandbox benchmark run on 2026-07-05 (~23:00 UTC), staging provisioning collapsed mid-run:
Provision failed — (container) — Provision failed during container phase:
No registered hosts below pressure guard: staging-host-agent-01(disk=95.08%>=95%)
Every subsequent create failed — the run's holdout phase got zero cells (the consuming loop correctly refused to emit a verdict; see tangle-network/supervisor-lab#19 for the run report). We moved the workload to the prod endpoint, which handled ~280 further sandboxes fine.
Questions / asks
- Why doesn't the staging pool autoscale? One registered host (
staging-host-agent-01) appears to be the entire staging capacity. When it crosses the pressure guard, staging is hard-down for every consumer. Is a second host / autoscaling intentionally out of scope for staging, or just not wired yet?
- Is there disk reclamation on the host? Our workloads create sandboxes that
pip install ortools (~50 MB each) into per-box venvs and are deleted afterwards. If image layers / overlay dirs / volumes from deleted sandboxes aren't garbage-collected, any sustained benchmark fills the host — 95% may be mostly reclaimable.
- Failure UX is good, capacity headroom isn't: the pressure guard failing loud is the right behavior (it saved our experiment from silent corruption). But a single-host staging with no GC means every heavy consumer will hit this.
Context for sizing
That day's load from one benchmark project: ~160 sandboxes on staging (CAD + firmware-style OR workers, each ~1–10 min, opencode backend, pip installs in-box). If that's enough to fill the host, GC or a pressure-triggered scale-out would pay for itself immediately.
What happened
During a ~70-sandbox benchmark run on 2026-07-05 (~23:00 UTC), staging provisioning collapsed mid-run:
Every subsequent create failed — the run's holdout phase got zero cells (the consuming loop correctly refused to emit a verdict; see tangle-network/supervisor-lab#19 for the run report). We moved the workload to the prod endpoint, which handled ~280 further sandboxes fine.
Questions / asks
staging-host-agent-01) appears to be the entire staging capacity. When it crosses the pressure guard, staging is hard-down for every consumer. Is a second host / autoscaling intentionally out of scope for staging, or just not wired yet?pip install ortools(~50 MB each) into per-box venvs and are deleted afterwards. If image layers / overlay dirs / volumes from deleted sandboxes aren't garbage-collected, any sustained benchmark fills the host — 95% may be mostly reclaimable.Context for sizing
That day's load from one benchmark project: ~160 sandboxes on staging (CAD + firmware-style OR workers, each ~1–10 min, opencode backend, pip installs in-box). If that's enough to fill the host, GC or a pressure-triggered scale-out would pay for itself immediately.