Skip to content

feat(voicechat): add Windows RTX desktop example - #1239

Draft
yifeif-nv wants to merge 2 commits into
NVIDIA:mainfrom
yifeif-nv:windows/nemotron-voice-lab
Draft

yifeif-nv wants to merge 2 commits into
NVIDIA:mainfrom
yifeif-nv:windows/nemotron-voice-lab

Conversation

@yifeif-nv

Copy link
Copy Markdown
Collaborator

Background

Add a buildable Windows voice application that demonstrates Nemotron VoiceChat through TensorRT-RTX, with live transcripts, reactive visuals, and a presentation mode. The existing full-duplex example uses Linux/ALSA. This source example also addresses delayed playback and repeated replies observed during longer interactive conversations.

This draft depends on #1218. Its existing commit remains in the comparison to main; review the Windows example commit separately from that prerequisite.

Exit Criteria

  • Users can download dependencies and build the Windows desktop application, native bridge, and RTX model locally using documented PowerShell stages.
  • The example distributes source, scripts, tests, and documentation only. No third-party binaries, libraries, SDK headers, Python/npm environments, model weights, bundles, or audio/media assets are added.
  • Capture, playback, interruption, and context refresh remain bounded during continued conversation and recover without reinjecting abandoned assistant replies.
  • Source checks and supported-host validation are recorded separately from uncompleted clean-machine qualification.

Implementation

  • Add examples/windows_voicechat: sandboxed Electron UI, canvas/SVG visuals, bounded NDJSON C++ bridge, settings, local diagnostics, stream mode, and labeled model-free rehearsal.
  • Add PowerShell setup/run entry points. Setup downloads verified Python, Electron, NVIDIA SDK components, and build tools into an external workspace, installs pinned Python packages, downloads a pinned checkpoint, and builds locally. Relative settings and launchers support relocated local installations. Native setup requires administrator rights only when the Microsoft toolchain is missing.
  • Add a source-distribution audit and Windows CI for index contents, JavaScript units, and PowerShell parsing/planning. There is no binary release/archive publication step.
  • Make standard TensorRT optional in CMake, select only the Nemotron family for this Windows build, and support Windows DLL discovery and Unicode bundle/cache paths. The example builds and requires trt_rtx.
  • Use 20 ms microphone packets to avoid racing the native fallback clock. Refresh carries only the latest unanswered request, with bounded repetition detection and one retry. A public, family-owned live-control interface resets dialogue through a worker barrier while keeping microphone/audio frontend continuity; shared core remains model-agnostic.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Validation

Commands and Results

  • cmake --build build-windows-rtx --parallel 8, ctest --test-dir build-windows-rtx --output-on-failure, and cmake --install build-windows-rtx: passed after the public-header changes, with 10/10 CTests including real RTX inference. Installed public header and exported interface target verified.
  • python -m pytest examples/windows_voicechat/native/test_native_startup.py -q with the staged bridge: 4 passed.
  • python -m pytest -q -p no:cacheprovider tools/tests/test_architecture.py tools/tests/test_family_impact.py tools/tests/test_public_source_hygiene.py tools/tests/test_pr_metadata.py: 82 passed.
  • node --test examples/windows_voicechat/desktop/tests/*.test.js: 23 passed, including fresh-workspace discovery and relocation.
  • python examples/windows_voicechat/audit_source.py --staged: passed for all 40 example source/text files; negative tests in isolated Git repositories rejected force-added binaries, vendored source, oversized files, and encoded media.
  • Windows PowerShell 5.1 parser and Setup.ps1 -Plan: passed. An isolated Setup.ps1 -Stage Python downloaded and verified CPython, created a venv, and imported SSL. Spaces/apostrophes in paths, relative launcher relocation, Run.ps1 -ValidateOnly, and App-only rehearsal validation passed.
  • node desktop/tests/electron-audio.integration.cjs and node desktop/tests/electron-transcript.integration.cjs from the example directory: passed using downloaded Electron and separately installed Playwright; real WebAudio/IPC, 20 ms capture, interruption/reset controls, and 1,034 events with 300 retained rows were checked.
  • python -m pytest -q -p no:cacheprovider families/nemotron_voicechat/tests/test_build_policy.py families/nemotron_voicechat/tests/test_quantization.py: 18 passed.
  • Repository-pinned ruff check --config ruff.toml on changed Python source, clang-format --dry-run --Werror on changed C++ source, and git diff --check: passed.
  • Full RTX conversation evidence, before the source-packaging/path/public-header refinements: 540.028 seconds, 12 topic checks, six actual context refreshes, spoken and manual interruption, no runtime/renderer errors, and maximum playback lead 0.941333 seconds. Follow-up model-free checks cover the packaging/path refinements. See the example's VALIDATION.md for scope and reproduction.

Hardware, Environment, and Revisions

  • Published source head: e803846cfdfbd7dedcbcbb9d23649e5870fbd8ab; its tree is byte-identical to the locally validated source tree cb3410412ec14537403a0297af948e02f67ac56e.
  • Windows x64; RTX 5090 with 32 GB VRAM; 128 GB host RAM; NVIDIA driver 591.86.
  • MSVC 19.44, CUDA 13.4, TensorRT-RTX 1.6.1.120, CPython 3.12, Electron 44.3.0.
  • Prerequisite feat(voicechat): stabilize compressed full-duplex sessions #1218 revision: 7458623963a038fa1ae1f1bac28eee6a5c792514.
  • Checkpoint nvidia/NVIDIA-NemotronLabs-VoiceChat-11B revision 359ada7b1c60851e40ff08065f9b0340244f27e0; tokenizer revision 6533e8de2c68e4536bf7c411d7a3ce5734111476.
  • TensorRT-RTX bundle uses explicit Thinker W8A8 / higher-precision sensitive projections / FP16 TTS policy. The locally built bundle is approximately 18 GB and is not included.

Not Run / Remaining Gaps

  • Full five-stage setup on a freshly installed Windows machine, lower-memory GPUs, other GPU/driver combinations, acoustic echo cancellation, and heavy concurrent GPU workloads remain unqualified.
  • A full-model soak was not repeated after source-only packaging, path portability, and public-header relocation. Nine minutes is the measured run, not a claim of empirically proven unlimited operation.
  • The broader Linux CI orchestration tests were attempted locally on Windows. Their Bash/compiler/POSIX execution cases cannot establish a passing Linux gate here; remote CI remains authoritative. No test criteria were weakened.
  • Remote CI and the automated premerge gate are not claimed as passed by the local checks.

Contributor Self-Review

  • I have completed a self-review of this change.

Notes For Future Readers

  • Merge feat(voicechat): stabilize compressed full-duplex sessions #1218 first, then rebase this draft to remove its prerequisite commit from the comparison. Do not merge this draft as independent validation of feat(voicechat): stabilize compressed full-duplex sessions #1218.
  • Review setup/distribution boundaries and the README first, then the desktop/bridge protocol, shared Windows build/loading mechanics, and family-owned recovery/public live-control interface.
  • Third parties are downloaded from upstream on the user's machine. Local assembly includes installed runtimes and their notices; it must not be published as this example's source archive. Upstream repository files already present before this PR are unchanged by that policy.
  • Model graph/runtime-version changes require rebuilding the bundle. MSVC's signed bootstrapper and transitive pip resolution are disclosed; this is not a fully locked reproducible binary distribution.

Risk level

  • Low
  • Medium
  • High

The example is opt-in and keeps model logic family-local, but adds Windows native loading/build support and changes experimental VoiceChat conversation recovery. Fresh-machine qualification and prerequisite integration remain necessary before treating it as a turnkey release.

Add an opt-in W8A8 Thinker and FP16 TTS path for 24 GiB-class hardware. Bound recurrent and cache state across transparent context rollovers, preserve prompt conditioning, and harden ALSA playback, diagnostics, tests, and documentation.

Signed-off-by: Yifei Fang <277870278+yifeif-nv@users.noreply.github.com>
Add a source-only Windows voice application with reactive visuals, bounded audio transport, and continuous conversation recovery. PowerShell setup downloads dependencies locally and builds the TensorRT-RTX runtime and model; no third-party packages or model weights are distributed with the example.

Add relocation checks, source-artifact auditing, and Windows source CI. Build on the compressed VoiceChat runtime in PR NVIDIA#1218.

Signed-off-by: yifeif-nv <277870278+yifeif-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant