Skip to content

Require explicit opt-in before real-money tests run#299

Merged
Szer merged 1 commit into
mainfrom
fix/real-tests-explicit-optin
Jul 27, 2026
Merged

Require explicit opt-in before real-money tests run#299
Szer merged 1 commit into
mainfrom
fix/real-tests-explicit-optin

Conversation

@Szer

@Szer Szer commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • tests/AlitaBot.RealTests and tests/CouponHubBot.RealTests drive a real Telegram MTProto session and real paid LLM/OCR backends. Their only gate was credential presence in ~/.alita-test/env / ~/.coupon-test/env — so a plain dotnet test -c Release at the solution level actually ran real, billed tests on any machine with working credentials. This happened tonight (twice): AlitaBot.RealTests 14 passed/18 skipped.
  • Adds an explicit, additional opt-in per project — ALITA_REAL_TESTS=1 / COUPON_REAL_TESTS=1 — checked via process env only (never the credentials file, so it can't accidentally get "cached" alongside credentials and silently re-enable future runs).
  • Single choke point per project: RealAssemblyFixture.fs's InitializeAsync now gates ALL real setup (DB, ngrok tunnel, bot process, webhook registration, MTProto login) on the opt-in, and SkipUnlessCore checks it before credentials so every individual test skips with a message naming the exact variable. No test in either project can reach the fixture's real Telegram/LLM surface without going through SkipUnlessCore/SkipUnlessUserClient first (verified: every [<Fact>]/[<Theory>] in both projects calls one of these as its first statement).
  • alita-real-test.yml / coupon-real-test.yml (both workflow_dispatch-only, triggers unchanged) now set the opt-in var in their "Run real-Telegram suite" step. make real-test / make coupon-real-test set it too, so the intended manual dev loop is unaffected.
  • AGENTS.md's Testing section now states plainly that solution-level dotnet test -c Release is safe by default, states the cost rationale, and that agents/automation must never set the opt-in var.

Validation

  • dotnet build -c Release — clean, 0 warnings, 0 errors.
  • dotnet test -c Release at solution level, without the opt-in set, on a machine where both ~/.alita-test/env and ~/.coupon-test/env are fully populated with working credentials:
    • CouponHubBot.RealTests: 32 total, 0 passed, 32 skipped — every skip message reads real-money tests opted out: set COUPON_REAL_TESTS=1 ....
    • AlitaBot.RealTests: 32 total, 3 passed, 29 skipped. The 3 passing tests (GeminiTransientDetectionTests) are pure, zero-I/O unit tests of a string classifier that live in this project but never touch RealAssemblyFixture/Telegram/LLM at all (pre-existing, by design — see the file's own doc comment). Every test that touches the fixture skips with real-money tests opted out: set ALITA_REAL_TESTS=1 ....
  • Hermetic projects unaffected: AlitaBot.Tests 125/125 passed, VahterBanBot.Tests 213/213 passed, CouponHubBot.Tests 198/198 passed, VahterBanBot.Unit.Tests 28/28 passed, SerializationCompat.Tests 51/51 passed, CouponHubBot.Ocr.Tests 19/19 passed.
  • The opt-in-present path was not exercised (no real Telegram/LLM calls were made in this validation) per the brief's cost-safety instruction.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU

…eal-money tests run

Credential presence in ~/.alita-test/env or ~/.coupon-test/env was the ONLY gate
on tests/AlitaBot.RealTests and tests/CouponHubBot.RealTests, so a plain
`dotnet test -c Release` at the solution level actually ran real Telegram MTProto
+ paid LLM calls whenever a developer's machine had working credentials (it did,
twice, tonight — real cost incurred). Add a second, explicit gate in each
project's RealAssemblyFixture: InitializeAsync now only does real setup when
ALITA_REAL_TESTS=1 / COUPON_REAL_TESTS=1 is set (checked via process env only,
never the credentials file, so it can't get "cached" alongside credentials), and
SkipUnlessCore checks it before credentials so every test skips with a message
naming the exact variable. The two workflow_dispatch-only workflows and the
`make real-test`/`make coupon-real-test` dev-loop targets set the variable
themselves; AGENTS.md now documents the rule and that automation must never set it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
@Szer
Szer merged commit 3e1df29 into main Jul 27, 2026
5 checks passed
@Szer
Szer deleted the fix/real-tests-explicit-optin branch July 27, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant