DRILL 2 (revert after): AlitaBot intentional runtime failure for SRE-agent validation#303
Merged
Merged
Conversation
…ent validation, take 2 Deliberate, owner-authorized production drill, second in the series (see #298/#302 for take 1 — a STARTUP failure that correctly did not trigger a rollback, since the old pod kept serving). This one is different on purpose: startup and /healthz succeed normally, so Kubernetes retires the old ReplicaSet and this broken one takes over — AlitaBot looks healthy to Kubernetes while being functionally dead, which should push the SRE agent toward a rollback this time. Two failure points, both logging SRE_DRILL_RUNTIME_FAILURE at Error via the existing Serilog logger, both gated on `not TestMode`: - DrillRuntimeFailureLoop: a hosted BackgroundService ticking every 15s, throwing and logging Error on every tick — traffic-independent, so it fires within verify-deploy.sh's 2-minute Loki error-log window even though AlitaBot is dormant and can go days without an inbound Telegram update. - The webhook update handler now throws before calling BotService.OnUpdate, so real Telegram updates fail too (secondary; the loop above is what actually gets detected). Gate verified live: prod bot_setting.TEST_MODE = 'false'; tests/AlitaBot.Tests sets TEST_MODE=true via both the container env var and a seeded bot_setting row (ContainerTestBase.fs). AlitaBot-only; no migration/config/Helm/workflow changes. REVERT IMMEDIATELY AFTER THE DRILL via `git revert` of this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
Szer
added a commit
that referenced
this pull request
Jul 27, 2026
… for SRE-agent validation take 3 (#309) DELIBERATE, OWNER-AUTHORIZED SRE-AGENT VALIDATION DRILL. The owner explicitly authorized taking AlitaBot down again: its only chat is with them, and this re-run specifically validates the six detection-pipeline fixes from #307. Same failure SHAPE as drill 2 (#303, reverted in #304): a hosted BackgroundService (PendingUsageFlushLoop) ticking every 15s via PeriodicTimer, logging Error on every tick, traffic-independent so it fires within verify-deploy.sh's Loki window even though AlitaBot is dormant; plus a throwing Telegram update handler so real messages also fail. Both gated on `not TestMode`. The one thing that's DIFFERENT on purpose: drill 2's every log line contained the literal token SRE_DRILL_RUNTIME_FAILURE plus "deliberate"/"drill"/"must be reverted" — the monitor agent read those words in the log text itself and filed nothing while AlitaBot was completely dead. This drill's runtime log text and exception messages are deliberately disguised as an ordinary application bug instead: - PendingUsageFlushLoop raises via an unguarded `Seq.head` on an empty sequence (System.ArgumentException: "The input sequence was empty."), logged as "Usage flush tick failed while flushing pending LLM usage records" — reads like a forgot-to-check-empty-queue bug in a usage-flush tick, not a drill. - The webhook handler dereferences a null string (System.NullReferenceException: "Object reference not set to an instance of an object."), caught by the existing generic "Unhandled error in update handler for {UpdateId}" log — a completely ordinary NRE. No "drill"/"SRE_DRILL"/"intentional"/"revert" appears in any string literal, log message, or exception text — only in the F# source comments above each throw site, which state prominently and in capitals (for future readers of the diff/commit, not for Loki) that this is deliberate and must be reverted. Gate verified: prod bot_setting.TEST_MODE = 'false' (per #307's binding context); tests/AlitaBot.Tests sets TEST_MODE=true via both the container env var and a seeded bot_setting row (ContainerTestBase.fs lines 34, 87), so the drill code never executes under the hermetic suite. AlitaBot-only (src/AlitaBot/Program.fs), single commit, no BotInfra/other-bot/migration/ Helm/k8s/workflow changes. REVERT IMMEDIATELY AFTER THE DRILL via `git revert` of this commit. Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The repo owner explicitly authorized taking AlitaBot down: its only chat is with them, and this is needed to validate the SRE agent's rollback path.
Why a second drill
Drill 1 (#298, since reverted via #302) threw during startup. That crashlooped the new ReplicaSet while the old pod kept serving — a P2, no outage. The SRE agent correctly did not roll back. That means the rollback path is still unvalidated.
This drill is different on purpose: the pod starts successfully and passes readiness, so Kubernetes retires the old ReplicaSet and the new (broken) one takes over. AlitaBot will be genuinely non-functional in production while looking "healthy" to Kubernetes — the realistic failure mode that should push the SRE agent toward a rollback.
What it does
Two failure points, both gated on
not TestMode, both logging the literal tokenSRE_DRILL_RUNTIME_FAILUREatErrorlevel via the existing Serilog logger:DrillRuntimeFailureLoop(src/AlitaBot/Program.fs) — a hostedBackgroundService, registered only whennot TestMode, ticking every 15 seconds, throwing and loggingErroron every tick. This is traffic-independent by design: AlitaBot is dormant (its only chat is the owner's staging chat) and can go days with zero inbound Telegram updates, so a failure that only fires on inbound traffic could produce zero errors duringverify-deploy.sh's 2-minute Loki check window and silently do nothing. A 15s tick guarantees several Error log lines inside any 2-minute window with no traffic required.src/AlitaBot/Program.fs, theWebhookHost.mapWebhookEndpointshandler) — now throws before callingBotService.OnUpdate, so the bot is genuinely broken on answering real messages too, not merely noisy in the background. This is secondary — the loop above is what actually triggers detection.Gate verification
botConfOptions.Value.TestMode:SELECT * FROM bot_setting WHERE key = 'TEST_MODE'→value = 'false'.tests/AlitaBot.Tests:ContainerTestBase.fssetsTEST_MODE=trueboth as the container env var (line 34) and as a seededbot_settingrow (line 87) — so the drill code never executes under the hermetic suite.Scope
AlitaBot only (
src/AlitaBot/Program.fs, single commit). Nosrc/BotInfra/, no other bot, no migrations, no Helm/k8s, no workflows.Validation
dotnet build -c Release: clean, 0 warnings, 0 errors.dotnet test tests/AlitaBot.Tests -c Release: 125/125 passed (project run individually, not solution-level). NoALITA_REAL_TESTS/COUPON_REAL_TESTSset.Revert
🤖 Generated with Claude Code
https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU