Sandbox lifecycle events lost on ungraceful orchestrator exit: in-memory queue has no durable/reconciled path #3776

Summary

Sandbox lifecycle events reach ClickHouse through an in-memory, best-effort queue on each orchestrator. Even with the back-pressure + retry improvements in #3770 / #3774, events still sitting in that queue are lost on an ungraceful exit (SIGKILL, OOM, panic, or a graceful drain that exceeds the scheduler's kill_timeout). Because the terminal killed event is what billing/audit reconcile against, this is a durability gap, not just a throughput one: sandbox_events can silently under-count terminations, and nothing backfills the missing rows.

This issue tracks the durable/reconciled path that #3774 deliberately did not attempt (it's a mitigation, not a guarantee).

Why #3774 isn't enough

#3774 makes terminal-event delivery best-effort-harder (bounded-blocking push, bounded flush retry, bounded drain, per-type drop metric), but the event is still only in RAM until a batch flushes. The queue was recently widened in prod (queue-size 1000→20000, max-batch-size 1000→5000), which raises the number of events lost per ungraceful exit (up to ~20000 + an in-progress 5000 batch). Node drain/rebalance — exactly when hard-kills happen — is the same scenario that produced the original incident (#3770).

Concretely, loss still occurs when:

• the process is SIGKILLed/OOM-killed/panics with a non-empty queue;
• a graceful drain can't flush the backlog within kill_timeout (ClickHouse slow/down at shutdown);
• ClickHouse is down long enough that bounded retries give up and the batch is dropped.

Options (seeking maintainer steer on direction)

1. Redis Streams as the source of truth + a ClickHouse consumer.
Orchestrators already publish lifecycle events to a Redis stream (the webhook path) in addition to the direct ClickHouse batcher. Make the stream authoritative: a dedicated consumer reads it and writes sandbox_events, with consumer-group acks so an unacked event is redelivered after a crash. Removes the parallel best-effort writer entirely and gives at-least-once delivery. Biggest change; cleanest end state.

2. Durable local WAL for terminal events.
Terminal events are appended to an on-disk log on the node before/alongside the batcher; a flusher drains the WAL to ClickHouse and truncates on ack. Survives process crash (not disk loss). Smaller blast radius than (1), but adds a disk format + recovery path per node.

3. Reconciliation / backfill job.
A periodic job compares an authoritative kill signal (orchestrator logs / Redis stream / routing-record teardown) against sandbox_events and backfills missing killed rows. Doesn't prevent loss but makes the system self-healing across all loss modes including crash — and is the cheapest to ship. Could run first as detection-only (emit a sandbox_events.reconciliation.missing metric) to size the real gap before writing backfill.

Suggested sequencing

Ship 3 in detect-only mode first — it quantifies the residual loss after #3774 across all modes (including crash), with no risk — then decide between the durable paths (1 preferred long-term) based on the measured gap.

Context / related

• Terminal sandbox lifecycle events (killed) silently dropped under burst: non-blocking ClickHouse queue, no retry, no back-pressure #3770 — root cause of the log-vs-ClickHouse divergence (queue overflow / flush failure / ungraceful exit).
• fix(clickhouse): stop silently dropping terminal lifecycle events under burst #3774 — mitigations (bounded-blocking push, retry, bounded drain, per-type drop metric). Explicitly scoped to reducing loss, not guaranteeing durability.