[AdaAibaby]

What

Stops terminal sandbox lifecycle events (killed) from being silently dropped on their way to ClickHouse. Fixes #3770.

Today those events reach sandbox_events through a non-blocking, fixed-size queue that drops on overflow with no retry and no back-pressure. Under a burst of kills the orchestrator logs "Killing sandbox" but the row never lands in sandbox_events, so billing/audit/analytics disagree with the logs (and with the separate webhook/Redis path). We observed 253 such sandboxes in one production day, clustered in a single burst window — see #3770 for the full root-cause trace.

This is not the already-fixed drain-loss bug (publishEventAsync is correctly TrackWork()-guarded); the loss is one layer below, in the ClickHouse delivery target.

Scope — what this does and does not fix

This PR reduces the probability of terminal-event loss and adds back-pressure + bounded durability. It is a mitigation, not a correctness guarantee:

• ✅ Removes the silent-drop cliff for killed under queue overflow (A).
• ✅ Survives a transient ClickHouse blip via bounded retry (B).
• ✅ Bounds the shutdown drain so a stalled flush can't exceed the kill window (addresses a hazard the first revision introduced — see below).
• ❌ Does not fix crash loss: on SIGKILL/OOM/panic the in-memory queue (now up to ~20000 deep) is still lost. True billing/audit correctness needs a durable path (Redis-as-source-of-truth + a ClickHouse consumer, a WAL, or a reconciliation/backfill job). Tracked separately in Sandbox lifecycle events lost on ungraceful orchestrator exit: in-memory queue has no durable/reconciled path #3776.

Changes

A — bounded-blocking push for terminal events (the core fix)

• batcher: add PushWithContext(ctx, item) — blocks until the queue has room or ctx is done, instead of dropping the instant the queue is full. On timeout it drops and returns ErrBatcherQueueFull, matching Push's accounting. The RLock/Lock discipline means Stop cannot close the channel mid-send, so there's no send-on-closed panic.
• events delivery: route only terminal events through PushWithContext with a flag-controlled timeout. High-volume non-terminal events keep the non-blocking Push, so the hot path is unchanged. The publish already runs in the TrackWork-guarded goroutine, so bounded blocking here does not block the Delete RPC.
• Scoped to killed (not paused/checkpointed): those are recoverable and can be higher volume, so they stay on the non-blocking path for now; isTerminalEvent is a one-liner to widen later, ideally behind a per-type flag.

B — retry a failed flush before dropping the batch

• batcher: on flush error, retry with capped exponential backoff up to clickhouse-batcher-flush-max-retries before giving up, so a transient ClickHouse blip doesn't discard a whole batch. Each retry increments batcher.flushes{result="retry"}.
• Interaction note (honest): the batcher is single-goroutine, so while it retries it is not draining the queue. During a sustained outage B therefore adds intake pressure (A's next terminal events can still time out and drop). B is a net win for blips, not for prolonged outages — which is why retries are off by default (below) and the drain is bounded.

Bounded shutdown drain (fixes a hazard the first revision introduced)

• The first cut retried on the batcher's start ctx, which isn't cancelled until after shutdown, so a stalled flush at stop could retry past the scheduler's kill_timeout → SIGKILL → lose the whole queue (the opposite of the goal). Now batcher.StopWithContext(ctx) bounds the drain, cancelling an in-flight flush + its retries when the budget expires; events.Close derives that budget from the shutdown ctx and clickhouse-batcher-drain-timeout-ms (default 10s) and honors ForceStop. The flush goroutine is always awaited (never leaked).

Observability

• events delivery: add sandbox_events.delivery.dropped, a per-event_type / terminal drop counter, so terminal-event loss is directly alertable instead of hidden in the generic batcher.items.dropped.

Rollout — dark-launched

The two behavior-changing flags default to their legacy values, so merging is a no-op until enabled per-environment via LaunchDarkly:

flag	default	effect when set
clickhouse-batcher-terminal-push-timeout-ms	0 (legacy non-blocking drop)	max time a killed event blocks for queue room before dropping
clickhouse-batcher-flush-max-retries	0 (legacy single attempt)	extra flush attempts after the first failure
clickhouse-batcher-drain-timeout-ms	10000	bounds the shutdown drain (safety bound; only bites when there's a backlog/stall at stop)

Tests

• batcher: PushWithContext blocks-then-succeeds / times out / errors when not started; flush retries recover a transient failure; static batcher does not retry; StopWithContext bounds a 1,000,000-retry drain (returns promptly instead of hanging).
• events: isTerminalEvent (killed-only) / terminalPushTimeout units; routing test proving a killed event applies bounded back-pressure on a full queue while non-terminal events drop immediately.
• Full packages/clickhouse suite + go vet + golangci-lint clean.