Revert drill 2: restore AlitaBot

[Szer]

Reverts #303. Drill 2 complete — it found four defects, three of them in the monitor/deploy tooling built last night.

1. verify-deploy.sh has a blind window (most serious)

The deploy went GREEN while the new pod was comprehensively broken. notify-failure and sre both skipped.

Phase 3 queries Loki for the last 2 minutes (verify-deploy.sh:263), but it runs while ArgoCD is still mid-sync — so it samples the old, healthy pod's logs and passes. The drill's first error landed at 11:02:44, after the check had already run.

Any failure that manifests during rollout ships silently. Crashloops are caught by Phase 2; runtime bugs — the common case — are not.

Fix needed: sample Loki only after the new ReplicaSet is confirmed Ready, and scope the query to the new pod rather than the container label.

2. The dormant carve-out disables error detection

The monitor agent reported: "alita is traffic_class: dormant — rules 3 and 4 (series volume-based rules / emerged-from-zero) are disabled for this bot."

But the stats it was handed read:

log_errors_24h: current=13  median_28d=0  z_score_28d=33.59
emerged_from_zero=true  emerged_from_zero_significant=true

A z-score of 33.6 was suppressed by a carve-out intended to stop false positives on message-count series. Errors emerging from zero on a dormant bot is precisely the signal that matters.

Fix needed: the dormant carve-out must apply to traffic/volume series only, never to error series.

3. Change correlation suppresses real incidents

"Classification: attributed — the increased error count is plausibly explained by the recent intentional DRILL changes... that makes the observed error volume expected."

A deploy explaining an error burst does not make it benign — it makes it a bad deploy. Attribution currently converts "we know what caused this" into "this is fine".

Fix needed: attribution may explain an anomaly's cause; it must never suppress an error-rate anomaly. A deploy correlated with a new error burst raises severity.

4. The evidence bundle lacks per-group SourceContext

"per-group identifiers and z-scores were not present in the provided Loki grouping output (only counts-only lines)"

Rules 1 (new error group) and 2 (error-group z-score) were therefore unevaluable. §6.1 of the design specifies grouping by SourceContext + message prefix; the implementation emits counts only. Program.DrillRuntimeFailureLoop was a brand-new SourceContext unseen in 28 days — a guaranteed rule-1 hit that could not be seen.

Confirmed working

• Gatherer recorded the truth: errors: 13, 2 error groups
• Statistics were correct: z=33.59, emerged_from_zero_significant: true
• Change correlation retrieved both git merges and ArgoCD deploy history
• Owner sent 2 messages, both logged with full TraceIds, no reply — user-facing outage proven

🤖 Generated with Claude Code

https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU