[xitzhang (xitzhang)]

Summary

First PR in a two-part GPT Live stack, targeting feature/azure-ai-projects/vnext.

This PR contains TypeSpec-emitted SDK APIs and only the supporting changes needed to generate and consume them. It does not add a handwritten Live WebSocket client or new Live samples.

Generation

• Source: [AI Foundry] Add GPT Live voice subprotocol using OpenAI TypeSpec 1.29 azure-rest-api-specs#47196 at e30cf6f0dcfbca167edf731d4e48874a63639e8b.
• Existing workflow: tsp-client update --debug, followed by the repository's PostEmitter.ps1.
• TypeSpec Python emitter 0.63.10, compiler 1.17.0, OpenAI TypeSpec 1.29.0.
• Generated Live configuration/delegation/event models and persisted conversation/item/audio REST operations.
• Refreshed API Markdown, API metadata, and the public-method inventory for this emitted-only branch.

Necessary supporting changes

• Update only @azure-tools/openai-typespec from 1.28.0 to 1.29.0, as required by the service specification.
• The lockfile retains every other package entry unchanged; no unrelated dependency upgrades or runtime-requirement changes.
• Keep the emitted branch free of handwritten model-readback customization. The optional sub_protocol readback hook and its behavior tests belong to dependent custom-code PR [azure-ai-projects] Add GPT Live WebSocket clients and samples #49631.
• Keep generated constructor/default/type-annotation regressions in the existing Realtime test file and generated Live metadata-model tests in this emitted PR; move the six handwritten readback behavior cases to [azure-ai-projects] Add GPT Live WebSocket clients and samples #49631.

No emitted Python source was hand-edited; only the established automated post-emitter corrections are included.

Original implementation validation (before metadata regeneration)

• Standalone PR 1 source imports successfully; generated Live models and REST operations are available, and beta.voice_agents.live is intentionally absent.
• 265 offline Realtime, readback, transport, and VoiceAgents telemetry cases passed with runtime/unawaited-coroutine warnings treated as errors. These are pure unit tests, run with --noconftest to avoid the unrelated recorded-test proxy bootstrap in the sparse validation checkout.
• SDK Mypy: passed, 57 source files, Python 3.10 target.
• Repository-pinned Pylint: passed, 10.00/10.
• Black: passed for the handwritten compatibility changes/tests.
• Wheel build, APIView token generation, canonical API Markdown/metadata export, and 201-method inventory generation: passed.
• Minimal npm lockfile: npm ci --dry-run --ignore-scripts --no-audit --no-fund passed. Existing Node 22.17 / Babel 22.18 engine warnings remain non-blocking.

Scope / follow-up

• Handwritten synchronous/asynchronous Live WebSocket support, transport reuse, samples, and their tests are in dependent [azure-ai-projects] Add GPT Live WebSocket clients and samples #49631, based on this PR's branch.
• Live REST pagination remains the uncustomized generated implementation; no package-local paging override is introduced here.

Metadata regeneration update - 2026-10-23

Regenerated this existing emitted-only PR from the latest head of
Azure/azure-rest-api-specs#47196: e30cf6f0dcfbca167edf731d4e48874a63639e8b.

API changes

• Lifecycle snapshots now use VoiceAgentLiveSessionResource in session.started,
  session.updated, and session.closed.
• Delegation-trigger metadata now uses VoiceAgentLiveDelegationItem.
• These models preserve foundry_managed_agent ownership and the resolved agent version,
  rather than presenting the native client's internal delegation owner.
• The two old native metadata model names are replaced within this unreleased feature.
  No released 2.9.0 model/export is removed.
• No new REST operations, preview-header keys, runtime dependencies, or SDK-local model-casting patches.
• Existing parent samples do not reference the replaced metadata names; no new sample workflow is needed.

Regeneration and validation

• tsp-client update --debug: passed, pinned to the full commit above.
• Existing PostEmitter.ps1: passed, including paired API export and the public-method inventory.
• api.md / api.metadata.yml: regenerated together; exact metadata hash verified.
  APIView's canonical generated Markdown formatting is retained.
• 10 new enabled offline model tests cover all lifecycle snapshots, ownership targets,
  pinned versions, wire roundtrips, native/null delegation modes, and public exports.
• New model tests + existing parent Realtime guards: 54 passed.
• Standard SDK + all-samples Pyright scope: 0 errors, 0 warnings.
• SDK Mypy targeting Python 3.10: passed.
• SDK Pylint: 10.00/10. Generated-model/test Black and test syntax/collection checks passed.
• Source package editable installation and wheel/API build succeeded.

The automated optional-selector post-emitter typing corrections remain here. The handwritten sub_protocol runtime readback hook is owned by custom-code PR #49631, not this emitted PR.
This PR remains draft. The handwritten client consumer updates remain in #49631.

SDK base update - 2026-10-23

Merged the current feature/azure-ai-projects/vnext base at ba6c352b044d5705c3d654a1d35c5b8522aebeb2 into this emitted branch in commit ede03b49d28fdb682cd199946d83825c55e4f956.

• GitHub reports MERGEABLE (no source merge conflicts); CI check status is separate and is not being reported as all-green. The PR remains draft with the same target.
• Generated Python merged automatically: both the base's Cloud Computer Use tool models and the GPT Live metadata models are retained. No generated Python was manually edited.
• Resolved tsp-location.yaml and the APIView source-version conflict in favor of this PR's GPT Live source pin, while retaining the incoming Cloud Computer Use cross-language mappings.
• Regenerated api.md and api.metadata.yml together from the combined model surface and verified their exact hash.
• Preserved the incoming sample-recording helpers/tests and environment-handling changes without modification.
• Validation: 54 offline tests passed; SDK + all-samples Pyright 0 errors, 0 warnings; SDK Mypy targeting Python 3.10 passed; SDK Pylint 10.00/10; Black passed for generated model files and metadata regression tests. Public export/discriminator smoke checks verified both feature sets.

Source provenance / next emission

This was an SDK-base integration, not a fresh TypeSpec re-emission. The GPT Live source remains e30cf6f0dcfbca167edf731d4e48874a63639e8b; the base's Cloud Computer Use models originated from b2cf9e36ed967aae8c43ed56973e1d1ac99255b3. Azure/azure-rest-api-specs#47196 currently has independent conflicts with its own base. Before another full emission, the upstream TypeSpec source must include both feature sets so regeneration does not remove the inherited Cloud Computer Use models. No upstream specification branch was changed here.

The local WebSocket branch and #49631 remain at c69c028870ca5ad18cec77373dbba30e2a4be82d; no WebSocket files, commits, or pushes were made for this update.

Emitted-first base integration - 2026-10-26

Merged current feature/azure-ai-projects/vnext at 12fdcce0b41fb1689c6bae09fd13060ac720f38e into this emitted branch first; published commit 99c7cdce70854f21b54847040ce6e9fa2ac1f1d0.

• Preserved the new VoiceAgentProfanityFilterMode and optional VoiceAgentInputTranscription.profanity_filter property from the base.
• Kept the base's intentional removals of deprecated Cloud Computer Use models and enum values; those earlier models are no longer present.
• Preserved GPT Live models and delegation ownership metadata. Generated Python merged automatically, without manual SDK source edits.
• Resolved all TypeSpec/API-mapping/API-hash conflicts in this emitted checkout, then regenerated paired API artifacts. The exact parent API hash is b9dab8e10c9c0ea2ede30861977bf9d23332477f69e250c79e4a32de8b0531f8.
• Validation: 54 tests passed, seven new-enum wire roundtrip checks passed, SDK/all-samples Pyright reported 0 errors/warnings, SDK Mypy targeting Python 3.10 passed, Pylint 10/10, generated-model Black passed.
• The current base also includes its merge from main; all such inherited changes were preserved. Only six package files differ from this PR's prior head as part of this integration.
• The GPT Live source pin remains e30cf6f0dcfbca167edf731d4e48874a63639e8b; the base additions originate from 843aff73c2ace0d34082b485d72e0752e7849640. This was an SDK base merge, not a fresh single-source TypeSpec emission. A future full emission must first combine both upstream source feature sets.

The published emitted branch was then synchronized into the local WebSocket checkout using a non-committing merge. No WebSocket commit or push was made; that local sync is pending explicit user approval. CI completion is separate from source mergeability and is not claimed as all-green here.

Handwritten customization relocation - 2026-10-26

Addressed the scope of Howie's review comment in commit e8fdf78f48e082d235585999daa305407e4ba53d:

• Removed _deserialize_voice, its patch_sdk() registration, and its exclusive _models import from this emitted branch. models/_patch.py now matches the current feature-branch base exactly.
• Moved ownership of the six hook-dependent readback cases (four selector cases plus two non-voice/unknown-kind cases) and the readback bug-fix changelog entry to the custom WebSocket branch. These tests are retained there, not disabled.
• Kept the two generated constructor/default/annotation cases, ten generated lifecycle/delegation metadata cases, and automated post-emitter typing corrections here.
• No generated SDK file, public API report, runtime dependency, or recording asset changed for this relocation.
• Validation of the emitted-only branch: 48 tests passed, Black passed for both edited Python files, SDK/all-samples Pyright 0 errors/warnings, SDK Mypy targeting Python 3.10 passed, SDK Pylint 10/10.

The raw generated-only readback uses VoiceAgentBaseDefinition and preserves payloads/omission. Concrete Live/Realtime subtype preservation remains supplied by the dependent handwritten branch. The latest parent has been synced locally into that branch with the original hook and six tests retained; that WebSocket sync remains uncommitted/unpushed pending explicit approval. The review thread has not been resolved or replied to automatically.

Current base conflict fix - 2026-10-26

The feature base advanced again to a85414bb8a99d8e84b6737621639797152bb0fe1, creating new conflicts after the earlier integration. Merged that exact base and resolved all three generation-metadata conflicts in emitted commit d0f96d113e27ccbab63eba5f264c93395f8b0e2d.

• After publication, GitHub reports MERGEABLE, and the PR base SHA matches the current remote feature-branch tip.
• The base's OpenAI TypeSpec 1.29 updates were already present in this emitted SDK. The merge introduces no runtime/content delta after resolving the metadata; it updates branch ancestry so the current source merge is conflict-free.
• API Markdown/hash and unique mapping JSON verified; no remaining conflict markers; 48 emitted-only tests passed.
• The handwritten readback hook remains excluded from this PR.
• No local or remote WebSocket changes were made. The existing local WebSocket sync is still pending against parent e8fdf78f48e; this new emitted integration was not automatically merged there.

Source mergeability is separate from CI status; all-green CI is not claimed.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

[Pilot] PR Pipeline Failure Analysis

What failed

Azure Pipeline build 7017865 failed for azure-ai-projects across all platforms (macos311, Ubuntu2404_313, Ubuntu2404_314, ubuntu2404_311, ubuntu2404_311_coverage, windows2022_312), sdist and whl variants, plus the static-analysis job.

1. Test failures (15 unique sample tests × 12 artifact files, same root cause everywhere): tests/samples/test_samples.py/test_samples_async.py/test_samples_evaluations.py fail with ImportError because azure.ai.projects.models no longer exposes the agent-tool classes the samples import: CodeInterpreterTool, FileSearchTool, FunctionTool, ImageGenTool, WebSearchTool, WebSearchPreviewTool, ComputerUsePreviewTool.

2. Pyright check (30 errors) — same root cause as above: samples/agents/tools/*.py and samples/evaluations/sample_agent_response_evaluation_with_function_tool.py fail with reportAttributeAccessIssue ("X is unknown import symbol") for the same tool classes, plus reportArgumentType/reportAssignmentType errors where list[MCPTool] is passed where list[Tool] is expected (MCPTool is not recognized as a Tool subtype) in sample_agent_mcp*.py, sample_toolbox_with_shell*.py, sample_toolboxes_with_search*.py, and sample_workflow_multi_agent_with_mcp_approval.py.

3. CSpell check failure (unrelated to the above): sdk/ai/azure-ai-projects/api.md and azure/ai/projects/models/_models.py contain unknown words (brise, flitz, harema, juni, nira, noeul, nuri, shida) — these look like non-English/sanitized sample strings or enum values introduced by the new generated Live models, not recognized by the spell-check dictionary.

Relevant pipeline output

ImportError: cannot import name 'CodeInterpreterTool' from 'azure.ai.projects.models'
(.venv_sdist/lib/python3.13/site-packages/azure/ai/projects/models/__init__.py)

samples/agents/tools/sample_agent_code_interpreter.py:32: ImportError

[azure-ai-projects :: pyright]
samples/agents/tools/sample_agent_code_interpreter.py:32:61 - error: "CodeInterpreterTool" is unknown import symbol (reportAttributeAccessIssue)
samples/agents/sample_workflow_multi_agent_with_mcp_approval.py:68:20 - error: Argument of type "list[MCPTool]" cannot be assigned to parameter "tools" of type "list[Tool] | None"
  "MCPTool" is not assignable to "Tool" (reportArgumentType)
30 errors, 0 warnings, 0 informations

##[error]sdk/ai/azure-ai-projects/api.md:9674:135 - Unknown word (brise)
##[error]sdk/ai/azure-ai-projects/azure/ai/projects/models/_models.py:15281:33 - Unknown word (brise)
##[error]Spelling errors detected. To correct false positives or learn about spell checking see: https://aka.ms/azsdk/engsys/spellcheck

Recommended next steps

• Export CodeInterpreterTool, FileSearchTool, FunctionTool, ImageGenTool, WebSearchTool, WebSearchPreviewTool, and ComputerUsePreviewTool from azure.ai.projects.models (add to models/__init__.py and __all__), or update the samples' imports if these classes were intentionally renamed/relocated in this TypeSpec-emitted version.
• Confirm MCPTool is declared as a Tool subtype (or union member) in the generated type hierarchy so pyright accepts list[MCPTool] where list[Tool] is expected; otherwise widen the samples' type annotations/arguments.
• Add the flagged tokens (brise, flitz, harema, juni, nira, noeul, nuri, shida) to the repository's cspell allow-list/dictionary, or replace them if they are typos rather than intentional sanitized sample values.
• See https://aka.ms/ci-fix

Automated fix: Requested

Generated by Pipeline Analysis Next Steps · copilot · sonnet50 · 110.4 AIC · ⌖ 0.576 AIC · ⊞ 8.9K · ◷