Howie/provision on fly by howieleung · Pull Request #49750 · Azure/azure-sdk-for-python

Howie Leung (howieleung) · 2026-10-28T17:48:25Z

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

The pull request does not introduce [breaking changes]
CHANGELOG is updated for new features, bug fixes or other significant changes.
I have read the contribution guidelines.

General Guidelines and Best Practices

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

Pull request includes test coverage for the included changes.

…ng state - Introduced `GetEnvironmentFileVariables.Tests.ps1` to validate environment variable parsing from .env files. - Added `GetSampleRequiredEnvironmentVariables.Tests.ps1` to ensure required environment variables are correctly identified for samples. - Created `InvokeSampleTest.Integration.Tests.ps1` and `InvokeSampleTest.Tests.ps1` for testing sample execution in various modes, including playback and recording. - Implemented `ResetRecordingState.Tests.ps1` to verify cleanup of recording state and pytest cache. - Developed `ResolveSampleTest.Tests.ps1` to test sample resolution and error handling for ambiguous or missing files. - Added `TestRecordingGitConflicts.Tests.ps1` to check for recording conflicts in git repositories. - Created `TestHelpers.ps1` to provide reusable functions for test setup and synthetic package creation. - Updated `conftest.py` to support command-line options for environment variable configuration.

…rsion to 2.10.0 (#49650)

… values

…kill

#49613) * Add comprehensive tests for environment variable handling and recording state - Introduced `GetEnvironmentFileVariables.Tests.ps1` to validate environment variable parsing from .env files. - Added `GetSampleRequiredEnvironmentVariables.Tests.ps1` to ensure required environment variables are correctly identified for samples. - Created `InvokeSampleTest.Integration.Tests.ps1` and `InvokeSampleTest.Tests.ps1` for testing sample execution in various modes, including playback and recording. - Implemented `ResetRecordingState.Tests.ps1` to verify cleanup of recording state and pytest cache. - Developed `ResolveSampleTest.Tests.ps1` to test sample resolution and error handling for ambiguous or missing files. - Added `TestRecordingGitConflicts.Tests.ps1` to check for recording conflicts in git repositories. - Created `TestHelpers.ps1` to provide reusable functions for test setup and synthetic package creation. - Updated `conftest.py` to support command-line options for environment variable configuration. * Update cspell configuration and improve test assertions for multiline values

- Introduced SampleCiHelpers.ps1 for Azure CLI operations and manifest handling. - Created Test-BicepTemplates.ps1 to validate Bicep template compilation. - Added Bicep templates for resource group and sample resources. - Implemented sample-resources.bicepparam for parameter bindings. - Developed run_samples.py to execute pytest nodes against Bicep-created projects. - Added select_samples.py for resolving sample test queries. - Created test_run_samples.py for validating sample test invocation. - Implemented test_select_samples.py for testing sample selection logic. - Included standard-setup.lock.json for dependency management.

…pdate README; add tests for automatic provisioning flow

…tics

…viewToolboxTool; add VoiceAgentProfanityFilterMode; update apiHash and CrossLanguageVersion (#49708)

…Azure/azure-sdk-for-python into feature/azure-ai-projects/vnext

…uality parameters in API documentation

Copilot

🟡 Changes recommended

Numbered log entries can bypass credential redaction and expose secrets in published attachments, while reused artifact names cause publication failures when jobs rerun in the same build.

2 open issues

What changed in this PR

Adds on-demand live testing for azure-ai-projects samples, provisioning temporary Foundry environments around the existing synchronous and asynchronous test harness.

Changes:

Selects and distributes samples across up to four workers with identity, quota, provisioning, and cleanup checks.
Adds shared-identity bootstrap templates and temporary workload infrastructure.
Adds sanitized log attachments, validation-endpoint handling, documentation, and regression tests.

File	Description
`sdk/ai/azure-ai-projects/tests/samples/shared-identities/infra/main.bicep`	Creates the permanent identity group and role assignments.
`sdk/ai/azure-ai-projects/tests/samples/shared-identities/infra/IdentityConfiguration.json`	Defines shared identities and required roles.
`sdk/ai/azure-ai-projects/tests/samples/shared-identities/infra/identities.bicep`	Creates and tags managed identities.
`sdk/ai/azure-ai-projects/tests/samples/shared-identities/infra/Deploy-Identities.ps1`	Previews or deploys shared identity infrastructure.
`sdk/ai/azure-ai-projects/tests/samples/sample_log.py`	Centralizes credential redaction and bounded log writing.
`sdk/ai/azure-ai-projects/tests/samples/sample_executor.py`	Integrates sanitized logs and configurable capture directories.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/Test-BicepTemplates.ps1`	Checks compiled infrastructure contracts and source provenance.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/SamplePreflight.Tests.ps1`	Tests regional eligibility and quota checks.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/SampleCiHelpers.Tests.ps1`	Tests target, ownership, identity, and diagnostic safeguards.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/Remove-SampleResources.Tests.ps1`	Tests cleanup confirmation and ownership enforcement.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/New-SampleDeploymentPlan.Tests.ps1`	Tests offline planning and unsupported-input rejection.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/Invoke-SampleProvisioning.Tests.ps1`	Tests rejection before resource mutation.
`sdk/ai/azure-ai-projects/tests/samples/ci/validation/IdentityResolution.Tests.ps1`	Tests shared-identity resolution and metadata validation.
`sdk/ai/azure-ai-projects/tests/samples/ci/test_validation_environment.py`	Tests validation-endpoint precedence.
`sdk/ai/azure-ai-projects/tests/samples/ci/test_select_samples.py`	Tests wildcard selection and queue partitioning.
`sdk/ai/azure-ai-projects/tests/samples/ci/test_sample_ci_attachments.py`	Tests redaction and per-result JUnit attachments.
`sdk/ai/azure-ai-projects/tests/samples/ci/test_run_samples.py`	Tests runner arguments, readiness, and environment isolation.
`sdk/ai/azure-ai-projects/tests/samples/ci/test_pipeline.py`	Checks pipeline structure and execution safeguards.
`sdk/ai/azure-ai-projects/tests/samples/ci/select_samples.py`	Collects, selects, and partitions existing sample tests.
`sdk/ai/azure-ai-projects/tests/samples/ci/sample_ci_attachments.py`	Publishes sanitized executor logs through JUnit markers.
`sdk/ai/azure-ai-projects/tests/samples/ci/run_samples.py`	Checks readiness and executes selected live tests.
`sdk/ai/azure-ai-projects/tests/samples/ci/Remove-SampleResources.ps1`	Deletes verified run-owned temporary resources.
`sdk/ai/azure-ai-projects/tests/samples/ci/README.md`	Documents setup, execution, safety, and maintainer checks.
`sdk/ai/azure-ai-projects/tests/samples/ci/New-SampleDeploymentPlan.ps1`	Generates queue-bound deployment plans.
`sdk/ai/azure-ai-projects/tests/samples/ci/Invoke-SampleProvisioning.ps1`	Performs admission checks and provisions workloads.
`sdk/ai/azure-ai-projects/tests/samples/ci/infra/standard-shared-identities.bicep`	Defines Foundry dependencies using existing identities.
`sdk/ai/azure-ai-projects/tests/samples/ci/infra/standard-setup.lock.json`	Pins upstream reference sources and checksums.
`sdk/ai/azure-ai-projects/tests/samples/ci/infra/sample-resources.bicepparam`	Supplies environment-backed deployment parameters.
`sdk/ai/azure-ai-projects/tests/samples/ci/infra/sample-resources.bicep`	Deploys the sample model and GitHub MCP connection.
`sdk/ai/azure-ai-projects/tests/samples/ci/infra/resource-group.bicep`	Creates temporary groups with ownership and expiry tags.
`sdk/ai/azure-ai-projects/tests/conftest.py`	Preserves the environment-supplied validation endpoint.
`sdk/ai/azure-ai-projects/tests.yml`	Orchestrates planning, admission, live tests, cleanup, and results.

_{Review effort: Balanced}

…n-fly

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-10-29T22:20:04Z

[Pilot] PR Pipeline Failure Analysis

What failed

Azure DevOps pipeline python - pullrequest (build 7017292) failed in the CSpell spelling validation step for sdk/ai/azure-ai-projects. The job exited with PowerShell exited with code '1' after CSpell found 48 issues across 18 files. No test failures were reported (failed_pipeline_tests was empty) — this is purely a spell-check/lint failure, not a build or test regression.

Relevant pipeline output

CSpell: Files checked: 32, skipped: 2, Issues found: 48 in 18 files.
##[error]sdk/ai/azure-ai-projects/tests.yml:156:31 - Unknown word (pscore)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/Invoke-SampleProvisioning.ps1:59:102 - Unknown word (retagged)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/Invoke-SampleProvisioning.ps1:163:78 - Unknown word (bicepparam)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/New-SampleDeploymentPlan.ps1:15:69 - Unknown word (DEFINITIONID)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/SampleCiHelpers.ps1:181:42 - Unknown word (cnotin)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/SampleCiHelpers.ps1:616:67 - Unknown word (idtyp)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/infra/sample-resources.bicepparam:4:72 - Unknown word (unconfigured)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/infra/standard-shared-identities.bicep:22:31 - Unknown word (foundy)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/infra/standard-shared-identities.bicep:247:10 - Unknown word (caphostproj)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/sample_ci_attachments.py:71:97 - Unknown word (pytrace)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/test_sample_ci_attachments.py:54:34 - Unknown word (Nlcjpw)
##[error]sdk/ai/azure-ai-projects/tests/samples/ci/test_select_samples.py:88:32 - Unknown word (axpy)
##[error]sdk/ai/azure-ai-projects/tests/samples/sample_log.py:38:54 - Unknown word (opusr)
##[error]Spelling errors detected. To correct false positives or learn about spell checking see: https://aka.ms/azsdk/engsys/spellcheck

(additional repeated DEFINITIONID/idtyp occurrences in *.Tests.ps1 validation files omitted for brevity — same root cause.)

Root cause

Several new identifiers/technical terms introduced by this PR's sample-provisioning scripts and bicep infra files (e.g. bicepparam, caphostproj, DEFINITIONID, idtyp, pytrace, axpy) are not present in the repo's CSpell dictionary (.vscode/cspell.json). One occurrence, foundy in standard-shared-identities.bicep:22:31, looks like a likely typo (probably intended to be foundry, given the Azure AI Foundry context of this PR) rather than a legitimate new term, and should be reviewed/corrected rather than allowlisted.

Recommended next steps

Add the legitimate new technical terms/identifiers (bicepparam, caphostproj, DEFINITIONID, idtyp, cnotin, pytrace, axpy, pscore, retagged, unconfigured, opusr, Nlcjpw) to the CSpell word list in .vscode/cspell.json.
Review standard-shared-identities.bicep:22 for the foundy occurrence — correct it to foundry if it is a typo, otherwise add it to the dictionary if intentional.
Re-run the python - pullrequest CSpell/validation stage after the fix to confirm all 48 issues are resolved.
See https://aka.ms/ci-fix

Automated fix: Requested

Generated by Pipeline Analysis Next Steps · copilot · sonnet50 · 34.7 AIC · ⌖ 0.576 AIC · ⊞ 8.9K · ◷

…re-sdk-for-python into howie/provision-on-fly

The base branch was changed.

Copilot

🟡 Changes recommended

Live sample output validation targets the sanitized playback URL instead of the configured project because pytest CLI mode overwrites the endpoint passed through the environment.

1 open issue

CLI pytest config overrides the configured validation endpoint · New

2 resolved since last review

_{Review effort: Balanced}

+    environment = os.environ.copy()
+    environment.update(state["environment"])


Copilot

🔵 Needs a closer look

The pytest configuration hook overwrites the environment-provided validation endpoint with a sanitized default, causing both default samples to attempt validation against a placeholder after provisioning billable resources.

1 open issue

CLI pytest config overrides the configured validation endpoint

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The pytest configure hook overwrites the environment-supplied validation endpoint with a sanitized placeholder, causing both sample validators to target the wrong endpoint even in correctly configured live runs.

1 open issue

CLI pytest config overrides the configured validation endpoint

_{Review effort: Balanced}

Howie Leung (howieleung) added 30 commits October 19, 2026 09:47

Update version to 2.10.0 and add tsp-location.yaml configuration file

8fe4ea0

Add CloudComputerUsePreviewTool and related models; update package ve…

c54e6ce

…rsion to 2.10.0 (#49650)

Update cspell configuration and improve test assertions for multiline…

aa52ccf

… values

Merge branch 'feature/azure-ai-projects/vnext' into howie/recording-s…

8a6963e

…kill

Refactor pipeline configuration to remove manual approval stage and u…

a76a5cc

…pdate README; add tests for automatic provisioning flow

Check Foundry account quota before ephemeral sample provisioning

a488e2a

Capture secret-safe ARM failure codes before sample resource cleanup

8971a39

Preserve ARM codes from protected sample deployment preflight failures

d3c1fef

Protect secret-bearing CLI JSON parser failures

5338b28

Serialize sample CI model deployments to avoid account conflicts

b54fb3c

Bound new Standard project readiness and capture safe runtime diagnos…

71d7fec

…tics

Record successful on-demand MCP sample pipeline validation

a14d275

Attach sanitized sample API logs and print output to ADO test results

f480bcb

Reuse shared sanitized sample log files for ADO attachments

b529172

Record verified shared-log test-result attachments and cleanup

2c3ad48

Reuse existing validation project from secret pipeline environment

f012dae

Identify sample pipeline application before validation secret preflight

c1a1c40

Require explicit sample CI subscription and default to GPT-5.2

a0050f4

Require pipeline tenant environment and bind cleanup to both target IDs

9301c8b

Harden sample CI defaults, regional preflight and captured logs

b308fa6

Register PSGallery before installing sample CI safety tests

29ea175

Use approved tools feed for pinned Pester bootstrap

8868861

Remove deprecated CloudComputerUsePreviewTool and CloudComputerUsePre…

eb60461

…viewToolboxTool; add VoiceAgentProfanityFilterMode; update apiHash and CrossLanguageVersion (#49708)

Update CrossLanguageVersion and tsp-location.yaml commit reference

62e8273

Merge branch 'main' into feature/azure-ai-projects/vnext

12fdcce

Merge branch 'feature/azure-ai-projects/vnext' of https://github.com/…

a4c6564

…Azure/azure-sdk-for-python into feature/azure-ai-projects/vnext

Update openai-typespec to version 1.29.0; enhance model options and q…

a85414b

…uality parameters in API documentation

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Comment thread sdk/ai/azure-ai-projects/tests/samples/sample_log.py Outdated

Comment thread sdk/ai/azure-ai-projects/tests.yml

Howie Leung (howieleung) added 2 commits October 29, 2026 10:23

Merge branch 'main' into feature/azure-ai-projects/vnext

0d3cab4

Merge branch 'feature/azure-ai-projects/vnext' into howie/provision-o…

d90749f

…n-fly

This comment has been minimized.

Sign in to view

Update regex to redact secret names in logs

2dadfe9

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Darren Cohen (dargilco) previously approved these changes Oct 29, 2026

View reviewed changes

Howie Leung (howieleung) added 2 commits October 29, 2026 15:37

Add custom words for cspell in Azure AI Projects test files

2ffe3dc

Merge branch 'howie/provision-on-fly' of https://github.com/Azure/azu…

c8443db

…re-sdk-for-python into howie/provision-on-fly

Base automatically changed from feature/azure-ai-projects/vnext to main October 30, 2026 16:39

jenny (JennyPng) requested review from Chenxi Jiang (ChenxiJiang333), iscai-msft and Yuchao Yan (msyyc) as code owners October 30, 2026 16:39

Merge remote-tracking branch 'origin/main' into howie/provision-on-fly

75353f3

Copilot AI balanced review requested due to automatic review settings October 30, 2026 16:58

Copilot started reviewing on behalf of Howie Leung (howieleung) October 30, 2026 16:58

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Comment thread sdk/ai/azure-ai-projects/tests/samples/ci/run_samples.py

Comment on lines +172 to +173

environment = os.environ.copy()

environment.update(state["environment"])

Copilot AI balanced review requested due to automatic review settings October 30, 2026 17:07

Copilot started reviewing on behalf of Howie Leung (howieleung) October 30, 2026 17:07

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Howie Leung (howieleung) force-pushed the howie/provision-on-fly branch from 6350435 to 7e1f9b7 Compare October 30, 2026 17:24

Copilot AI balanced review requested due to automatic review settings October 30, 2026 17:24

Copilot started reviewing on behalf of Howie Leung (howieleung) October 30, 2026 17:25

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Howie Leung (howieleung) force-pushed the howie/provision-on-fly branch from 7e1f9b7 to b5410d6 Compare October 30, 2026 17:45

Copilot AI balanced review requested due to automatic review settings October 30, 2026 17:45

Copilot started reviewing on behalf of Howie Leung (howieleung) October 30, 2026 17:46

View session

fix: update README.md links to point to the correct GitHub paths

635b121

Howie Leung (howieleung) force-pushed the howie/provision-on-fly branch from b5410d6 to 635b121 Compare October 30, 2026 17:49

Navigation Menu

Repository navigation

Howie/provision on fly - #49750

Howie/provision on fly#49750
Howie Leung (howieleung) wants to merge 48 commits into
mainfrom
howie/provision-on-fly

Howie Leung (howieleung) commented Oct 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

github-actions Bot commented Oct 29, 2026

What failed

Root cause

Recommended next steps

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		environment = os.environ.copy()
		environment.update(state["environment"])

Conversation

Howie Leung (howieleung) commented Oct 28, 2026

Description

All SDK Contribution checklist:

General Guidelines and Best Practices

Testing Guidelines

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

github-actions Bot commented Oct 29, 2026

What failed

Root cause

Recommended next steps

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants