You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add PlatformValidation sync/async live and recorded SDK tests - #49303
Twenty recorded scenarios exercise all 25 SDK methods in both sync and async clients:
Operation group
Methods per client
CloudValidation
Create/update, GET, PATCH, DELETE, LIST by RG, LIST by subscription
Validation execution plans
Create/update, GET, PATCH, DELETE, LIST
Execution plan runs
Create a new run, GET, DELETE, LIST
Validation test runs
GET, LIST
Test catalog, versions and categories
GET and LIST for each
Operation status
GET
Provider operations
LIST
Existing-resource PUT is additionally exercised for CloudValidation and execution
plans. Assertions verify identity, persisted metadata/configuration, completed
workload results, ownership and post-delete 404.
The implementation follows the documented workflow:
Install dev_requirements.txt and the editable SDK in a virtual environment.
Use AzureMgmtRecordedTestCase, framework credentials and resource preparers.
Record real sync/async requests with the test proxy decorators.
Provision workload prerequisites through service-created MRGs, without manual
MRG creation or role assignments; verify child-first cleanup.
Sanitize credentials, image URLs, signed queries and unrelated list metadata;
verify nested JSON and replay the full included suite.
Publish through manage_recordings.py push, update assets.json, and verify
playback from a separate checkout with freshly downloaded published assets.
Current validation: package-root playback using CI's collection scope reports 236 passed, 2 failed, 50 skipped. All 218 mocked regressions and 18 non-catalog
recorded scenarios pass. The 50 skips are existing generated placeholders; no
new skips or xfails were introduced. Repository Black and whitespace checks pass.
The two failures are the sync/async catalog scenarios. Catalog tests and categories
are declared @subscriptionResource, and versions inherit their parent's scope in
the official TypeSpec.
Expected IDs now use the framework's configured subscription: /subscriptions/{subscriptionId}/providers/Microsoft.PlatformValidation/....
The deployed service and existing recordings return /providers/Microsoft.PlatformValidation/...
instead. Strict assertions expose this service/contract mismatch, matching Go and
JavaScript coverage. Focused regressions reject missing or wrong subscription IDs.
Ready for review, not merge-ready: current validation is playback, not a fresh
live run. Recordings and their published tag are unchanged; no responses were
rewritten to satisfy the assertions. After the service is corrected, rerun live
recording, publish refreshed assets, and verify playback/CI. The earlier
235-pass result used provider-global catalog expectations and does not establish
compliance with the subscription-scoped contract.
Published recordings:
20 recordings, 414 HTTP interactions. The package README documents setup,
prerequisites and the end-to-end live/record/playback/publication commands.
Includes a minimal shared test-tool SAS-regex fix: preserve JSON escape
characters while redacting signed queries. Regression cases cover all four SAS
fields across three JSON nesting levels and escaped values.
Test-only changes: no generated SDK code, public APIs, package versions, runtime
dependencies or pipelines changed. A product changelog entry is not applicable.
All SDK Contribution checklist:
The pull request does not introduce breaking changes.
CHANGELOG is updated for new features, bug fixes or other significant changes. Not applicable: test-only changes.
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.
shivangirana20
changed the title
(AI Generated) Test hardening: test_cloud_validation_lifecycle in TestCloudValidations
Test hardening: test_cloud_validation_lifecycle in TestCloudValidations
Oct 1, 2026
Add sync/async catalog, execution, update and result scenarios with service-managed prerequisites and ownership-checked cleanup. Publish sanitized recordings and document the official Python SDK test workflow.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2a042b96-9dc7-4414-9d1e-5bb57db74da4
Fix only the shared test-tool SAS expression and add regression coverage for escaped values at multiple JSON nesting depths. Verified clean-source live recording exposed the dependency.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2a042b96-9dc7-4414-9d1e-5bb57db74da4
shivangirana20
changed the title
Test hardening: test_cloud_validation_lifecycle in TestCloudValidations
Add PlatformValidation sync/async live and recorded SDK tests
Oct 27, 2026
Adds handwritten live and recorded tests for the PlatformValidation management SDK, covering synchronous and asynchronous clients without changing generated SDK code or public APIs.
Changes:
Adds lifecycle, update, catalog, operation-listing and execution-result scenarios.
Adds prerequisite provisioning, ownership-aware cleanup, sanitization and mocked regression tests.
Documents recording workflows, references published assets and updates shared SAS sanitization to preserve JSON escapes.
Move sanitizer configuration into a regular test helper so regression tests do not import an ambiguous conftest module when generated and handwritten tests are collected together. Preserve fixture scope, dependencies, sanitizer rules, and assertions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7a785b66-f182-4517-86a0-2775420a38dd
Fixed the CI test-collection import collision in commit 2952409. Sanitizer configuration now lives in a regular test helper; neither regression test imports conftest.py. Fixture scope/dependencies, sanitizer rules, assertions, recordings, generated SDK code, and CI configuration are unchanged.
Validation used the same package-root collection and ignore flags as CI, covering generated and handwritten tests together. Before: 97 failed, 138 passed, 50 skipped. After: 235 passed, 50 skipped. The 50 skips are the existing generated placeholders; no new skips were added. Black and git diff --check pass. The push triggers fresh CI; these local results do not claim remote CI is green.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Execution clients can leak signed VHD credentials through debug logs, while standalone live operation tests omit the documented authentication variables and may fail or use an unintended identity.
Derive catalog IDs from the configured subscription and reject missing or wrong scope in focused regressions. Document the two strict catalog playback failures against current service responses; retain published recordings unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7a785b66-f182-4517-86a0-2775420a38dd
This assertion still requires a provider-global catalog ID. The official ValidationTestRunProperties.testId contract identifies the catalog resource's ARM ID, and ValidationTest is subscription-scoped, as the new catalog tests correctly enforce. A service correction that satisfies those tests would therefore fail the lifecycle and read-only result scenarios here. Build this expected ID from the configured subscription as well.
Azure Pipelines build 7017108 failed on all 6 platform legs (macos311, ubuntu2404_311, ubuntu2404_311_coverage, Ubuntu2404_313, Ubuntu2404_314, windows2022_312; both whl and sdist installs), each with the identical failure:
A separate GitHub Actions job, Test Azure SDK Tools, also failed with test_rust_support_is_disabled_by_default_and_forwarded raising KeyError: '${{ if parameters.InstallMsRustToolchain }}'. This test file (eng/tools/azure-sdk-tools/tests/test_native_build_pipeline.py) is not touched by this PR's diff, so this failure is unrelated pre-existing/flaky repo-tooling test noise, not caused by this change.
Relevant pipeline output
Catalog ID assertion failure (identical on every platform)
tests/test_catalog.py:78: in test_public_catalog
self.assert_entry(test, test_id, name)
entry = {'id': '/providers/Microsoft.PlatformValidation/validationTests/compute-vmimagevalidation-vm-boot-test', 'name': 'compute-vmimagevalidation-vm-boot-test', ...}
resource_id = '/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.PlatformValidation/validationTests/compute-vmimagevalidation-vm-boot-test'
def assert_entry(self, entry, resource_id, name):
assert entry.name == name
actual_id = entry.id
> assert (
actual_id and actual_id.lower() == resource_id.lower()
), "Catalog ID differs from the canonical subscription-scoped identity."
E AssertionError: Catalog ID differs from the canonical subscription-scoped identity.
E - /subscriptions/00000000-0000-0000-0000-000000000000/providers/microsoft.platformvalidation/validationtests/compute-vmimagevalidation-vm-boot-test
E ? ---------------------------------------------------
E + /providers/microsoft.platformvalidation/validationtests/compute-vmimagevalidation-vm-boot-test
tests/test_catalog.py:48: AssertionError
Identical trace for test_public_catalog_async, same recorded response (tests/recordings/test_catalog.*), confirming one root cause reproduced across all platforms/install modes.
Diagnosis
This is a recorded-response / contract mismatch, not a logic bug in the test or production code. The PR author's own description confirms this: the test asserts a subscription-scoped catalog resource id (/subscriptions/{subscriptionId}/providers/Microsoft.PlatformValidation/...), matching the official TypeSpec's @subscriptionResource declaration for the catalog/category/version hierarchy. However, the committed recording (and the currently deployed PlatformValidation service) still returns a provider-global id (/providers/Microsoft.PlatformValidation/...), without the subscription segment.
No amount of client-side code change can make the committed recording return a different id value — the assertion is deliberately strict and correctly flags a genuine service/recording contract gap. The author explicitly marked the PR "Ready for review, not merge-ready" pending either:
A service-side fix so the catalog API returns subscription-scoped IDs, followed by re-recording, or
A decision to relax the assertion if provider-global IDs are the intended (TypeSpec-divergent) contract.
Recommended next steps
Do not patch the test assertion to silently accept the provider-global ID; that would mask a real API-contract divergence from the published TypeSpec.
Confirm with the PlatformValidation service team whether the catalog/version id should be subscription-scoped (per TypeSpec) or provider-global (current deployed behavior and current recording).
Once resolved, re-record test_public_catalog/test_public_catalog_async live and republish the recording per the package README's live/record/playback/publish workflow, then rerun CI.
The unrelated test_rust_support_is_disabled_by_default_and_forwarded failure in Test Azure SDK Tools is not caused by this PR's diff; re-run that job or investigate separately if it persists.
Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
shivangirana20
changed the title
Description
Adds handwritten PlatformValidation integration tests using the official
Python SDK testing guide
and management-plane testing guide.
Twenty recorded scenarios exercise all 25 SDK methods in both sync and async clients:
Existing-resource PUT is additionally exercised for CloudValidation and execution
plans. Assertions verify identity, persisted metadata/configuration, completed
workload results, ownership and post-delete 404.
The implementation follows the documented workflow:
dev_requirements.txtand the editable SDK in a virtual environment.AzureMgmtRecordedTestCase, framework credentials and resource preparers.MRG creation or role assignments; verify child-first cleanup.
verify nested JSON and replay the full included suite.
manage_recordings.py push, updateassets.json, and verifyplayback from a separate checkout with freshly downloaded published assets.
Current validation: package-root playback using CI's collection scope reports
236 passed, 2 failed, 50 skipped. All 218 mocked regressions and 18 non-catalog
recorded scenarios pass. The 50 skips are existing generated placeholders; no
new skips or xfails were introduced. Repository Black and whitespace checks pass.
The two failures are the sync/async catalog scenarios. Catalog tests and categories
are declared
@subscriptionResource, and versions inherit their parent's scope inthe official TypeSpec.
Expected IDs now use the framework's configured subscription:
/subscriptions/{subscriptionId}/providers/Microsoft.PlatformValidation/....The deployed service and existing recordings return
/providers/Microsoft.PlatformValidation/...instead. Strict assertions expose this service/contract mismatch, matching Go and
JavaScript coverage. Focused regressions reject missing or wrong subscription IDs.
Ready for review, not merge-ready: current validation is playback, not a fresh
live run. Recordings and their published tag are unchanged; no responses were
rewritten to satisfy the assertions. After the service is corrected, rerun live
recording, publish refreshed assets, and verify playback/CI. The earlier
235-pass result used provider-global catalog expectations and does not establish
compliance with the subscription-scoped contract.
Published recordings:
20 recordings, 414 HTTP interactions. The package README documents setup,
prerequisites and the end-to-end live/record/playback/publication commands.
Includes a minimal shared test-tool SAS-regex fix: preserve JSON escape
characters while redacting signed queries. Regression cases cover all four SAS
fields across three JSON nesting levels and escaped values.
Test-only changes: no generated SDK code, public APIs, package versions, runtime
dependencies or pipelines changed. A product changelog entry is not applicable.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines