fix(test): restore subscription prompt answers in provision validation stdin - #9313
Merged
Merged
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
…tion tests The nightly `azure-dev - cli` pipeline has failed every night since 07-24. Regression from #9125 ("Auto-enable no-prompt mode in CI environments"). Root cause: the provision-validation-quota tests answer a fixed stdin script, but the interactive prompt sequence differs between playback and live: - In playback the recording harness sets AZD_DEBUG_SYNTHETIC_SUBSCRIPTION, so no subscription prompt fires (only the validation confirm). - In live nightly there is no synthetic subscription, so azd prompts to select the subscription first, shifting every stdin answer by one line. #9125 also changed Confirm to return its default (Yes) on EOF/blank instead of the old blank=No behavior, removing the blank-line padding the original stdin relied on. The combination broke the tests in live: - Sub tests: the lone "n" was eaten by the subscription Select prompt -> `'n' is not an allowed choice`. - RG tests: answers shifted; the validation confirm hit EOF -> default Yes -> azd deployed the intentionally-invalid template -> `template contains errors`. Fix: persist AZURE_SUBSCRIPTION_ID to the azd environment (via cfgOrStoredSubscription(session), mirroring deploy_test.go) so the subscription prompt never fires in either mode. The prompt sequence is now identical across playback and live, so the stdin scripts answer exactly the remaining prompts (RG select/name for RG scope, then the explicit "n" for the validation confirm). Cassettes must be re-recorded because the recorded flow prompted for (and enumerated) the subscription, whereas the fixed flow resolves it non-interactively: cd cli/azd && mage record -filter=Test_CLI_ProvisionValidationQuota Fixes #9312 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e3af6cd6-bc77-45e7-a150-33390aa47bd2
vhvb1989
force-pushed
the
vhvb1989/fix-provision-validation-quota-stdin
branch
from
July 27, 2026 18:23
d4cb958 to
7326006
Compare
vhvb1989
marked this pull request as ready for review
July 28, 2026 05:16
vhvb1989
requested review from
JeffreyCA,
RickWinter,
richardpark-msft and
tg-msft
as code owners
July 28, 2026 05:16
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates quota-validation functional tests to avoid subscription prompts and keep stdin aligned with validation prompts.
Changes:
- Persists the recorded/configured subscription in six tests.
- Documents the resulting prompt sequences.
Comments suppressed due to low confidence (1)
cli/azd/test/functional/provision_validation_quota_test.go:285
- [azd-code-reviewer] This says both values are persisted in the azd environment, but the subscription-scoped tests persist only
AZURE_SUBSCRIPTION_ID; they supplyAZURE_LOCATIONthroughcli.Env. Keep the documented prompt contract aligned with the actual setup.
// The subscription and location are resolved non-interactively (the test persists
// AZURE_SUBSCRIPTION_ID / AZURE_LOCATION to the azd environment before provisioning), so no
// subscription/location prompt fires. The only prompt is the validation warning confirm
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
jongio
reviewed
Jul 28, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
Two things below. The description mismatch Copilot flagged on line 44 is also still open, I replied there with what I found.
- line 44:
cfg.SubscriptionIDcan be empty in a live run, and persisting an empty value shadows an exportedAZURE_SUBSCRIPTION_ID - line 284: the sub-scoped helper says AZURE_LOCATION gets persisted to the azd environment, but only the RG tests do that
jongio
reviewed
Jul 28, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
Still open from my earlier pass:
The title and the Fix section say this restores the leading blank lines in stdinForProvisionWithValidationNo / stdinForRGProvisionWithValidationNo. Neither helper's return value changes in this diff, so the PR text describes the opposite of what the code does. #9312 carries the same wording. The commit message on 7326006 is accurate, so only the PR title, the Fix section, and the issue write-up need updating. Worth correcting before merge, because the next person chasing a nightly regression here will search for a stdin change that isn't in the tree.
Also unanswered: line 44 (cfg.SubscriptionID can be empty in a live run) and line 284 (AZURE_LOCATION isn't persisted to the azd environment in the sub-scoped tests).
One new note inline.
jongio
approved these changes
Jul 28, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
Verified the fix against the code and the recordings, so my earlier comments are non-blocking.
Two things worth correcting before merge, neither of which needs a code change:
The Fix section in the description still says this restores the leading blank lines in the stdin helpers. It doesn't. It persists AZURE_SUBSCRIPTION_ID so the subscription prompt never fires, which is the right call because restoring the blank line would fix live and break playback. The #9312 write-up says the same thing and needs the same edit.
The commit message says the cassettes must be re-recorded. They don't need it. None of the six recordings contain a GET /subscriptions call, and all six already carry a subscription_id variable, so playback resolves the same id it did before. azd env set issues no HTTP, and the playback leg of this PR's CI is green. Following that instruction would churn roughly 5MB of testdata for nothing.
The two inline suggestions I left earlier (the empty-subscription guard and the comment block repeated across all six tests) are optional and shouldn't hold up the nightly fix.
JeffreyCA
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9312
Problem
The
azure-dev - clischeduled (nightly) pipeline has failed every night since 2026-07-24. Regression introduced by #9125 (0a8ef0e1b, "Auto-enable no-prompt mode in CI environments"), confirmed viagit merge-base(absent from last-good builde6f0357, present in first-failingc36b2fe).#9125 removed the subscription-selection line from the provision-validation-quota stdin helpers, assuming the subscription resolves non-interactively from
AZURE_SUBSCRIPTION_ID. In the live nightly, azd still fires the subscriptionSelectprompt, so the piped answers became misaligned by one line:Test_CLI_ProvisionValidationQuota_Sub_*): the lone"n"was consumed by the subscription Select prompt →'n' is not an allowed choice.Test_CLI_ProvisionValidationQuota_RG_*): answers shifted up one; the validation confirm hit EOF and — with Auto-enable no-prompt mode in CI environments #9125's newEOF=default=Yesbehavior — azd proceeded to deploy the intentionally-invalid template →The deployment template contains errors.Fix
Restore the leading blank line(s) that accept the default subscription (and location) in
stdinForProvisionWithValidationNo/stdinForRGProvisionWithValidationNo, so the piped stdin matches the real prompt sequence. The explicit"n"for the now-default-Yes validation confirm is preserved.Validation
gofmtclean,go vet ./test/functional/passes.Note
Test_DeploymentStacks/Subscription_Scope_Up_Downalso fails intermittently (A referenced resource was not found) but its stdin was not touched by #9125 — likely separate Azure-side flakiness, out of scope here.