DRILL (revert after): AlitaBot intentional startup failure for SRE-agent validation#298
Merged
Merged
Conversation
…ent validation Deliberate, owner-authorized production drill: throws SRE_DRILL_INTENTIONAL_STARTUP_FAILURE during AlitaBot application startup (after DI/host build, before serving) so the new ReplicaSet crashloops while the previous pod keeps serving. Exercises the _bot-deploy.yml -> verify-deploy.sh -> _sre-agent.yml path against a real deploy failure. Gated on `not TestMode` — verified live that prod bot_setting.TEST_MODE = 'false' and tests/AlitaBot.Tests sets TEST_MODE=true (env var + seeded bot_setting row), so this fires in prod and never under test. AlitaBot-only; no migration/config/Helm/workflow changes. REVERT IMMEDIATELY AFTER THE DRILL via `git revert` of this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
Szer
added a commit
that referenced
this pull request
Jul 27, 2026
…ent validation, take 2 (#303) Deliberate, owner-authorized production drill, second in the series (see #298/#302 for take 1 — a STARTUP failure that correctly did not trigger a rollback, since the old pod kept serving). This one is different on purpose: startup and /healthz succeed normally, so Kubernetes retires the old ReplicaSet and this broken one takes over — AlitaBot looks healthy to Kubernetes while being functionally dead, which should push the SRE agent toward a rollback this time. Two failure points, both logging SRE_DRILL_RUNTIME_FAILURE at Error via the existing Serilog logger, both gated on `not TestMode`: - DrillRuntimeFailureLoop: a hosted BackgroundService ticking every 15s, throwing and logging Error on every tick — traffic-independent, so it fires within verify-deploy.sh's 2-minute Loki error-log window even though AlitaBot is dormant and can go days without an inbound Telegram update. - The webhook update handler now throws before calling BotService.OnUpdate, so real Telegram updates fail too (secondary; the loop above is what actually gets detected). Gate verified live: prod bot_setting.TEST_MODE = 'false'; tests/AlitaBot.Tests sets TEST_MODE=true via both the container env var and a seeded bot_setting row (ContainerTestBase.fs). AlitaBot-only; no migration/config/Helm/workflow changes. REVERT IMMEDIATELY AFTER THE DRILL via `git revert` of this commit. Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
THIS IS A DELIBERATE, OWNER-AUTHORIZED SRE VALIDATION DRILL — REVERT AS SOON AS THE DRILL COMPLETES
This PR intentionally breaks AlitaBot's production startup so the SRE agent (
_sre-agent.yml) can be validated against a real deploy failure. Do not treat this as a normal change. After merge+observe, revert this single commit immediately (git revert <sha>— see below).What this does
Adds a gated
failwithinsrc/AlitaBot/Program.fs, right afterbuilder.Build()(application startup, DI already resolved, before the host starts serving requests). The message contains the literal tokenSRE_DRILL_INTENTIONAL_STARTUP_FAILUREso it's trivially greppable in Loki.not TestMode:mcp__alita-db__execute_sql: prodbot_setting.TEST_MODE = 'false'.tests/AlitaBot.TestssetsTEST_MODE=trueboth as a container env var (ContainerTestBase.fs) and as a seededbot_settingrow (SeedDatabase), so the guard isfalseunder test and the throw never fires there.src/AlitaBot/Program.fs, +19/-0 lines).src/BotInfra/, VahterBanBot, CouponHubBot, migrations, Helm, k8s, or workflows.TreatWarningsAsErrors— this is a runtime failure, not a compile error.Verification
dotnet build -c Release(full solution): 0 Warnings, 0 Errors.dotnet test(tests/AlitaBot.Tests, the hermetic PR gate, boots the real containerized app image withTEST_MODE=true): 125 Passed, 0 Failed, 0 Skipped. The guard confirmed inert under test as designed.Revert command (run after the drill)
🤖 Generated with Claude Code
https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU