DRILL 3 (revert after): AlitaBot runtime failure, disguised log text, for SRE-agent validation take 3

[Szer]

⚠️ DELIBERATE, OWNER-AUTHORIZED SRE VALIDATION DRILL — TAKES ALITABOT DOWN IN PRODUCTION — MUST BE REVERTED AS SOON AS THE DRILL COMPLETES ⚠️

The repo owner explicitly authorized taking AlitaBot down again: its only chat is with them, and this specific re-run is approved to validate the six detection-pipeline fixes from #307.

Why a third drill

Drill 2 (#303, reverted in #304) had the right mechanics — a runtime BackgroundService ticking every 15s plus a throwing update handler, gated on not TestMode, startup/readiness succeeding so the new ReplicaSet takes over. It worked. But every emitted log line contained the literal token SRE_DRILL_RUNTIME_FAILURE and the words "deliberate ... drill ... must be reverted". The monitor agent read that text in the log payload itself and classified the resulting error burst as expected/attributed — filing nothing, while AlitaBot was completely dead. Six defects in the detection pipeline (verify-deploy.sh's blind window, the dormant carve-out overreaching to error series, change-correlation suppressing error-rate findings, missing per-group SourceContext in evidence, and the runtime DORMANT_NOTE contradicting the fixed rule) were found and fixed in #307.

This drill re-tests those fixes. The whole point is that the monitor agent must not be able to shortcut by reading the word "drill" in the logs — so this time the runtime log text and exception messages are deliberately disguised as an ordinary application bug.

What it does

Same failure SHAPE as drill 2, gated on not botConfOptions.Value.TestMode:

1. PendingUsageFlushLoop (src/AlitaBot/Program.fs) — a hosted BackgroundService, registered only when not TestMode, ticking every 15 seconds via PeriodicTimer, throwing and logging Error on every tick. Traffic-independent by design, same rationale as drill 2: AlitaBot is dormant and can go days with zero inbound Telegram updates, so a 15s tick guarantees several Error log lines inside verify-deploy.sh's Loki check window with no traffic required.
2. Telegram update handling (src/AlitaBot/Program.fs, the WebhookHost.mapWebhookEndpoints handler) — throws before calling BotService.OnUpdate, so the bot is genuinely broken on answering real messages too (secondary; the loop above is what actually gets detected on a dormant bot).

The disguise — what changed from drill 2 and why it's plausible

• PendingUsageFlushLoop raises via an unguarded Seq.head on an empty sequence — System.ArgumentException: "The input sequence was empty. (Parameter 'source')" — logged as "Usage flush tick failed while flushing pending LLM usage records". This is a completely ordinary "forgot to check for the empty case before taking the head of a queue" bug shape in a periodic flush tick — nothing about it screams synthetic.
• The webhook handler dereferences a null string — System.NullReferenceException: "Object reference not set to an instance of an object." — caught by the existing, unmodified generic log line "Unhandled error in update handler for {UpdateId}". This is the single most common runtime exception in .NET; there is nothing to distinguish it from a real bug in the log stream.
• Neither the type name (PendingUsageFlushLoop), nor any log message, nor either exception's message/type contains "drill", "SRE_DRILL", "intentional", or "revert" anywhere. Those words appear only in F# source comments above each throw site — never emitted at runtime, never shipped to Loki.

Traceability (outside the logs)

Per the brief, this PR title, this body, the commit message, and the F# comments directly above each throw all state prominently and in capitals that this is a deliberate, owner-authorized drill that must be reverted immediately. Only the emitted log/exception text is disguised.

Gate verification

botConfOptions.Value.TestMode:

• Prod: bot_setting.TEST_MODE = 'false' (re-confirmed per the binding context in the brief; consistent with drill 2's live verification, unchanged since).
• tests/AlitaBot.Tests: ContainerTestBase.fs sets TEST_MODE=true both as the container env var (line 34) and as a seeded bot_setting row (line 87) — so the drill code never executes under the hermetic suite.

Scope

AlitaBot only (src/AlitaBot/Program.fs), single commit, 69 insertions, 0 deletions. No src/BotInfra/, no other bot, no migrations, no Helm/k8s, no workflows.

Validation

• dotnet build -c Release (both the AlitaBot project and the full solution): clean, 0 warnings, 0 errors.
• dotnet test tests/AlitaBot.Tests -c Release (run individually, not solution-level): 125/125 passed, first attempt, no flake hit.
• No solution-level dotnet test run. ALITA_REAL_TESTS/COUPON_REAL_TESTS were not set (confirmed via env | grep before the run).
• Grepped the diff: every occurrence of "drill"/"DRILL"/"SRE_DRILL"/"intentional"/"revert" is on a // comment line; zero occurrences on any non-comment added line (string literal, log call, or exception constructor).

Revert

git revert fbd0fd9b343d6fc41c8cf529dbd51d619eb00e76

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU