[xitzhang (xitzhang)]

Summary

Second PR in the GPT Live stack. Depends on #49629; merge #49629 first.

This PR targets xitzhang/gpt-live-generated so its diff contains only handwritten WebSocket support and associated samples, tests, API reports, and documentation. The parent PR supplies the TypeSpec emission and required tooling pin. This custom-code PR owns model-readback compatibility.

Handwritten support

• Add beta.voice_agents.live alongside the existing Realtime client, with synchronous and asynchronous clients and matching context-manager behavior.
• Keep the new SDK implementation to three files: _live.py, _live_utils.py, and aio/_live.py.
• Reuse the existing authenticated WebSocket transport through private shared bases, preserving Realtime public signatures, credential and initial-URL TLS/same-origin checks, SDK identification, and diagnostic logging.
• Reuse emitted Live models and discriminator mappings for event serialization/deserialization; retain raw mappings for forward compatibility.
• Support session controls, text and PCM audio, Responses-delegation function results, cancellation propagation, error handling, and transport/HTTP-session cleanup.
• Foundry owns startup: wait for session.started, do not send another session.start, and do not negotiate the Realtime subprotocol. Live does not invent a response.cancel command.

Samples and tests

• Add connection/lifecycle, synchronous/asynchronous headless multi-turn text, actual WAV audio streaming, and Responses function-tool samples.
• Keep the original single-prompt text default. Multiple quoted prompts reuse one connection and wait for each backend completion.
• Function results use the canonical call_id from complete response.output_item.done items.
• Audio completion uses a bounded inactivity heuristic that ignores silent/very-low-level PCM, rejects silence-only replies, and preserves captured PCM in counts and optional WAV output.
• Add enabled deterministic coverage for typed/raw events, authentication and trust guards, session lifetime, failures, cancellation/concurrency, exact wire shapes, sample sequencing, PCM thresholds, and cleanup.

There are no generated SDK source, TypeSpec pin, or dependency changes in this PR. The handwritten voice-definition readback compatibility hook belongs to this custom-code PR. No runtime dependencies or audio-device tooling are added.

Original implementation validation (before metadata regeneration)

• 704 targeted offline cases passed, including the parent readback tests, existing Realtime/telemetry regressions, and 87 sample guards. Runtime/unawaited-coroutine warnings were treated as errors. These are pure unit tests run with --noconftest to avoid unrelated recorded-test proxy bootstrap in the sparse validation checkout.
• SDK Mypy: passed, 60 source files, Python 3.10 target.
• Live sample Mypy: passed, six modules, using the repository sample-check settings.
• Repository-pinned Pylint: passed, 10.00/10.
• Black: passed, 17 handwritten/sample/test files.
• Clean wheel build and source-content verification: passed.
• Public API reports match the validated runtime; no private unbound type variables leak into the public methods.

Historical service E2E (before this metadata regeneration)

The runtime and sample source bytes match the implementation exercised against the real configured Foundry project earlier in this session:

• 7/7 scenarios passed across all five scripts: connection, default sync/async text, two-turn sync/async text, function calling, and concurrent recorded-audio streaming.
• Connection and audio examples used the existing gpt-live-1 deployment in client-delegation mode. Text/tool examples additionally used an existing gpt-4.1 Responses backend; a second model is optional architecture, not a general Live requirement.
• Both two-turn examples retained the supplied name across one socket.
• The real model requested the demonstration weather function, the client returned its explicitly simulated Seattle/72 F result, and the backend continued.
• Sent 128,400 PCM16 bytes (80,400 spoken WAV bytes plus 48,000 silence bytes), verified by SHA-256.
• Captured 278,400 bytes of actual returned mono PCM16/24-kHz audio, 5.8 seconds. Transcript: “The largest lake in the world by surface area is the Caspian Sea.”
• All seven sample-created agent versions were deleted by their samples and independently verified absent with server 404 responses. WebSockets and async HTTP sessions closed; no server errors or recovery cleanup were needed.
• No recordings or raw audio are stored in this SDK repository. Test Proxy recordings are published separately through the asset pointer. No credentials, private deployment identifiers, pipeline definitions, or infrastructure are committed.

Known boundaries

• Text examples print Responses backend text, not a frontend speech-finished signal.
• Live has no speech-finished event; long speech pauses can still split the bounded WAV example's reply.
• The full existing sample Mypy run has 547 errors in 19 untouched evaluator/workflow sample files. The six Live sample modules pass; unrelated samples were not changed.
• Live conversation REST operations remain emitted-only, including the known generated last_id/has_more paging limitation.

Stack maintenance

Keep this PR based on xitzhang/gpt-live-generated while #49629 is open. After #49629 merges, retarget/rebase onto the updated feature/azure-ai-projects/vnext as needed and verify the diff before merging.

Metadata regeneration update - 2026-10-23

Merged the updated emitted branch from #49629 normally (no rebase/force push), using
Azure/azure-rest-api-specs#47196 at e30cf6f0dcfbca167edf731d4e48874a63639e8b.

Consumer updates

• Updated the lifecycle/delegation model references in the Live client and sample guard tests.
• Added sync/async public-client checks for Foundry-managed ownership across all three lifecycle
  snapshots and delegation creation, including the pinned agent version and exact wire preservation.
• Documented effective ownership metadata in the existing GPT Live README section.
• The handwritten parser already uses emitted discriminator mappings, so no additional runtime
  model-conversion patch or public WebSocket signature change was needed.
• Regenerated paired API reports to resolve the merge's metadata-hash conflict.
• Generated models, TypeSpec pin and cross-language mapping match the emitted parent exactly;
  they are not duplicated in this PR's diff against its parent branch.
• Existing microphone/WAV sample behavior and all nine dedicated REST operations are retained.
• No redirect-policy/security hardening, dependency changes, or live CI pipeline changes are included.

Current validation

• Combined ordinary pytest run: 679 passed in 26.34 seconds, with live mode disabled.
  This includes Live/Realtime clients, shared transport, 10 generated-model regressions,
  sample/recording-helper guards, and both recorded REST tests.
• The published Live REST recordings still replay successfully under
  python/ai/azure-ai-projects_6289cbf17e; no recording or asset-pointer update was needed.
• A separate playback run blocked live fixture creation/cleanup and outbound Python service
  connections while verifying the same recorded HTTP/audio responses.
• Standard SDK + all-samples Pyright: 0 errors, 0 warnings.
• SDK Mypy targeting Python 3.10: passed.
• All 8 Live sample/helper modules pass Mypy with the standard sample-check settings.
• SDK Pylint: 10.00/10. Edited-test Black checks passed.
• Paired API metadata hash and the source editable installation were verified.

No new live-service sample run was performed for this metadata-only regeneration; the historical
E2E results above are not being presented as new service validation of this refresh.
This PR keeps its existing ready/non-draft state and its xitzhang/gpt-live-generated target.

Handwritten customization relocation - 2026-10-26

The parent emitted PR #49629 has removed the non-emitted _deserialize_voice hook in commit e8fdf78f48e082d235585999daa305407e4ba53d. This custom-code branch retains the same hook in the existing models/_patch.py, plus its six enabled readback regression cases and bug-fix changelog entry.

The local branch has synced the updated emitted parent using a pending, non-committing merge and retained the customization byte-for-byte. Its source contents are unchanged from the pre-relocation combined SDK; the changed parent makes the customization a child-only diff. This local WebSocket sync has not been committed or pushed, so its publication is still awaiting explicit user approval. The existing remote head remains 069241f86859c5f7e562f0e0835cdab0869a1527.

Validation: 679 tests passed, including the six retained readback cases and both published REST playback tests; SDK/all-samples Pyright 0 errors/warnings, SDK Mypy Python 3.10 passed, Pylint 10/10, Black passed. Generated/public API artifacts and runtime dependencies are unchanged by the relocation. No live-service execution or recording update was needed.

[azure-pipelines]

Azure Pipelines:
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Input-recording overwrite protection, mapping serialization, and concurrent asynchronous cleanup require correction.

3 open issues

• Prevent overwriting recordings through hard-linked paths · New
• Normalize mapping inputs before Live API serialization · New
• Ensure concurrent close callers await connection cleanup · New

What changed in this PR

Adds handwritten GPT Live WebSocket support to Azure AI Projects, building on the generated APIs in #49629, which must merge first.

Changes:

• Adds synchronous and asynchronous Live clients using shared Realtime transport.
• Adds lifecycle, multi-turn text, WAV streaming, and function-tool samples with deterministic tests.
• Updates documentation, public exports, and API reports.

Paths below are relative to sdk/ai/azure-ai-projects/.

File	Description
tests/​samples/​test_live_samples.py	Adds offline sample guards.
tests/​agents/​test_realtime_client.py	Adapts shared-transport identification checks.
tests/​agents/​test_realtime_client_async.py	Adapts asynchronous identification checks.
tests/​agents/​test_live_client.py	Covers Live protocol and transport behavior.
samples/​agents/​voice/​sample_voice_agent_live_text_conversation.py	Demonstrates synchronous multi-turn text.
samples/​agents/​voice/​sample_voice_agent_live_text_conversation_async.py	Demonstrates asynchronous multi-turn text.
samples/​agents/​voice/​sample_voice_agent_live_function_tool.py	Demonstrates Responses function results.
samples/​agents/​voice/​sample_voice_agent_live_connection.py	Demonstrates session lifecycle.
samples/​agents/​voice/​sample_voice_agent_live_audio_conversation_async.py	Streams WAV audio and captures replies.
samples/​agents/​voice/​live_sample_utils.py	Shares lifecycle and event helpers.
README.md	Documents Live usage and limitations.
docs/​public-methods.md	Lists the Live entry point.
docs/​GeneratePublicMethodsDoc.ps1	Includes Live in method discovery.
CHANGELOG.md	Records Live support and samples.
azure/​ai/​projects/​operations/​_patch.py	Exposes synchronous Live operations.
azure/​ai/​projects/​aio/​operations/​_patch.py	Exposes asynchronous Live operations.
azure/​ai/​projects/​aio/​_realtime.py	Extracts shared asynchronous transport.
azure/​ai/​projects/​aio/​_live.py	Implements asynchronous Live support.
azure/​ai/​projects/​_realtime.py	Extracts shared synchronous transport.
azure/​ai/​projects/​_live.py	Implements synchronous Live support.
azure/​ai/​projects/​_live_utils.py	Defines shared Live protocol helpers.
api.metadata.yml	Refreshes the API hash.
api.md	Reports updated public APIs.

_{Review effort: Balanced}