Require explicit opt-in before real-money tests run#299
Merged
Conversation
…eal-money tests run Credential presence in ~/.alita-test/env or ~/.coupon-test/env was the ONLY gate on tests/AlitaBot.RealTests and tests/CouponHubBot.RealTests, so a plain `dotnet test -c Release` at the solution level actually ran real Telegram MTProto + paid LLM calls whenever a developer's machine had working credentials (it did, twice, tonight — real cost incurred). Add a second, explicit gate in each project's RealAssemblyFixture: InitializeAsync now only does real setup when ALITA_REAL_TESTS=1 / COUPON_REAL_TESTS=1 is set (checked via process env only, never the credentials file, so it can't get "cached" alongside credentials), and SkipUnlessCore checks it before credentials so every test skips with a message naming the exact variable. The two workflow_dispatch-only workflows and the `make real-test`/`make coupon-real-test` dev-loop targets set the variable themselves; AGENTS.md now documents the rule and that automation must never set it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/AlitaBot.RealTestsandtests/CouponHubBot.RealTestsdrive a real Telegram MTProto session and real paid LLM/OCR backends. Their only gate was credential presence in~/.alita-test/env/~/.coupon-test/env— so a plaindotnet test -c Releaseat the solution level actually ran real, billed tests on any machine with working credentials. This happened tonight (twice):AlitaBot.RealTests 14 passed/18 skipped.ALITA_REAL_TESTS=1/COUPON_REAL_TESTS=1— checked via process env only (never the credentials file, so it can't accidentally get "cached" alongside credentials and silently re-enable future runs).RealAssemblyFixture.fs'sInitializeAsyncnow gates ALL real setup (DB, ngrok tunnel, bot process, webhook registration, MTProto login) on the opt-in, andSkipUnlessCorechecks it before credentials so every individual test skips with a message naming the exact variable. No test in either project can reach the fixture's real Telegram/LLM surface without going throughSkipUnlessCore/SkipUnlessUserClientfirst (verified: every[<Fact>]/[<Theory>]in both projects calls one of these as its first statement).alita-real-test.yml/coupon-real-test.yml(bothworkflow_dispatch-only, triggers unchanged) now set the opt-in var in their "Run real-Telegram suite" step.make real-test/make coupon-real-testset it too, so the intended manual dev loop is unaffected.AGENTS.md's Testing section now states plainly that solution-leveldotnet test -c Releaseis safe by default, states the cost rationale, and that agents/automation must never set the opt-in var.Validation
dotnet build -c Release— clean, 0 warnings, 0 errors.dotnet test -c Releaseat solution level, without the opt-in set, on a machine where both~/.alita-test/envand~/.coupon-test/envare fully populated with working credentials:CouponHubBot.RealTests: 32 total, 0 passed, 32 skipped — every skip message readsreal-money tests opted out: set COUPON_REAL_TESTS=1 ....AlitaBot.RealTests: 32 total, 3 passed, 29 skipped. The 3 passing tests (GeminiTransientDetectionTests) are pure, zero-I/O unit tests of a string classifier that live in this project but never touchRealAssemblyFixture/Telegram/LLM at all (pre-existing, by design — see the file's own doc comment). Every test that touches the fixture skips withreal-money tests opted out: set ALITA_REAL_TESTS=1 ....AlitaBot.Tests125/125 passed,VahterBanBot.Tests213/213 passed,CouponHubBot.Tests198/198 passed,VahterBanBot.Unit.Tests28/28 passed,SerializationCompat.Tests51/51 passed,CouponHubBot.Ocr.Tests19/19 passed.🤖 Generated with Claude Code
https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU