Skip to content

Revert drill 3: restore AlitaBot - #312

Merged
Szer merged 1 commit into
mainfrom
revert/sre-drill-3
Jul 27, 2026
Merged

Revert drill 3: restore AlitaBot#312
Szer merged 1 commit into
mainfrom
revert/sre-drill-3

Conversation

@Szer

@Szer Szer commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Reverts #309. Drill 3 complete.

The six fixes from #307 are validated

Defect Drill 2 Drill 3
1 — verify-deploy blind window deploy passed green while bot was dead failed correctly, notify-failure + sre fired
2 — dormant carve-out error series suppressed dormant_exempt: false
3 — attribution "expected… not filed" "correlated — suspected bad deploy… do NOT suppress"
4 — error grouping rules unevaluable named the unseen-in-28-days group
5 — CLEF @m raw JSON prefixes readable message prefix
6 — injected note contradicted the rulebook scoped correctly

The SRE agent also diagnosed a disguised bug correctly — no drill token in the logs, and it still found Seq.head on an empty sequence at Program.fs:60 and proposed Seq.tryHead. Filed as #311.

The remaining defect: severity is defined by pod health, not service health

The agent concluded:

Severity: P2 (no user-facing outage; pods healthy and serving traffic)
No rollback (P1) was required because the app remains Healthy and serving traffic

AlitaBot was answering nobody. The owner sent two messages during the incident and received no reply; both are in Loki as NullReferenceException with TraceIds 48c893b3… and e0d504ae….

This is not an agent error — it followed sre.md correctly. The severity table defines P1 as "No pods serving traffic — all replicas unhealthy, 5xx rate is high, app completely unreachable." For these bots:

  • pods are healthy — the process runs fine, only its work fails
  • 5xx is always zero — the /bot webhook returns 200 regardless of internal exceptions, to stop Telegram retry storms
  • the app is reachable/healthz returns OK

So all three P1 criteria are structurally unreachable, no matter how broken the bot is. The taxonomy cannot express "process healthy, service dead" — which is the dominant real failure mode here.

Its suggested mitigation, "restart the pod to clear the current error loop", also wouldn't have helped: the fault is deterministic and would recur immediately.

Follow-up: rewrite sre.md's severity table around service-level signals — Loki error rate on the update-handling path, and successful update handling — rather than pod health and 5xx. Tracked separately.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU

…og text, for SRE-agent validation take 3 (#309)"

This reverts commit 4152993.
@Szer
Szer merged commit 0d8eec5 into main Jul 27, 2026
4 checks passed
@Szer
Szer deleted the revert/sre-drill-3 branch July 27, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant