fix(tts): integrated GPU support for MagpieTTS/NanoCodec incl. CUDA local-transformer mirror (extends #60)

[femo23]

fix(tts): integrated GPU support for MagpieTTS / NanoCodec, including the CUDA local-transformer mirror (extends #60)

Problem

On unified-memory NVIDIA parts, ggml enumerates the GPU as GGML_BACKEND_DEVICE_TYPE_IGPU, not _GPU
(ggml-cuda.cu sets the type from cudaDeviceProp.integrated). nemo-speech doctor on a DGX Spark:

Features: asr backend_cuda ... tts
Devices:
  [0] integrated-gpu NVIDIA GB10 (121.7 GiB)
  [1] cpu            CPU (121.7 GiB)

The ASR runtime (src/runtime/ggml/backend.cpp) already accepts both types, but the TTS path only asks for the
discrete type in three places, so Magpie TTS silently runs on the CPU:

1. src/tts/magpietts/model.cpp — ggml_backend_init_by_type(GPU) returns null → CPU fallback (fixed by fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec #60)
2. src/tts/nanocodec/model.cpp — same (fixed by fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec #60)
3. src/tts/magpietts/lt.cpp — magpietts_model_init_local_transformer_copy() requires dev_type == GPU, so once
   1–2 are fixed, --tts.lt-fp32 with --tts.lt-backend cuda/auto fails on these devices with
   cannot create CUDA local-transformer mirror from a non-GPU model (new in this PR)

Symptoms on main (8642eaa) and the 0.2.0 release on a DGX Spark GB10 (CUDA 13.0, sm_121, JetPack-style L4T):

$ nemo-speech serve --tts-model magpie_tts_multilingual_357m.v2607.f16.gguf ... --device cuda:0
--lt-backend cuda requires a CUDA ggml backend; current backend is CPU
nemo-speech serve: MagpieTTS synthesis failed

and without --device, synthesis succeeds but takes ~6 s per sentence (CPU) with no warning.

Change

Two commits:

• 864559c by @hsuanguo, cherry-picked unchanged from fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec #60: try GGML_BACKEND_DEVICE_TYPE_IGPU after the discrete
  attempt fails, before the CPU fallback, in the MagpieTTS and NanoCodec loaders.
• This PR's commit: accept IGPU in magpietts_model_init_local_transformer_copy() as well, matching the ASR
  runtime's check.

#60 has been open since 2026-10-04 without maintainer review; this PR carries it forward with the missing third
site so the fix is complete for the fp32 local-transformer path. Happy to rebase onto #60 if it merges first (only
the lt.cpp commit would remain).

Validation (DGX Spark, GB10, CUDA 13.0, docker/Dockerfile with CUDA_ARCH=121)

Magpie v2607 (frame stacking, stacked_slots=16) + NanoCodec 22 kHz, served through riva_server:

MagpieTTS backend: CUDA0 - NVIDIA GB10
NanoCodec backend: CUDA0 - NVIDIA GB10
... lt-backend=auto effective=cuda lt-precision=native sampling-backend=auto effective=cuda ...

config	first audio	RTF
main (CPU fallback, silent)	~6 s / sentence	~0.7
#60 only, default LT precision	29–34 ms	0.05
#60 only, --tts.lt-fp32=true --tts.lt-backend=cuda	fails by inspection: lt.cpp L278 rejects any device whose type is not _GPU, and on this hardware it is _IGPU
this PR, --tts.lt-fp32=true --tts.lt-backend=cuda	37 ms	0.08 — log: MagpieTTS local transformer mirror: CUDA0, precision=fp32 (61 tensors)

Streaming through the Riva gRPC client (synthesize_online, LINEAR_PCM 22.05 kHz), 3 runs each, 8–9 s utterances.
Running in production for a home assistant on the Spark since 2026-10-11 (SIGKILL/restart tested).

Notes

• Re the automated review on fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec #60 (IGPU initialises but tensor allocation fails → no CPU retry): that behaviour is
  unchanged from how a discrete GPU OOM is handled today in both loaders, so I left it out of scope here rather than
  restructure the loaders; happy to follow up separately if wanted.
• Jetson Orin (JetPack 7.2) was the reporter's platform on fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec #60; DGX Spark is a second confirmation.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/NeMo-Speech.cpp/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 6377031a-df8a-4209-9be1-3cbc01660a1a

---

📥 Commits

Reviewing files that changed from the base of the PR and between 8642eaa and d3a5097.

---

📒 Files selected for processing (3)

• src/tts/magpietts/lt.cpp
• src/tts/magpietts/model.cpp
• src/tts/nanocodec/model.cpp

---

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

---

📝 Summary

Summary by CodeRabbit

• New Features
  • MagpieTTS and NanoCodec can now use integrated GPUs when available, providing an additional hardware option for audio generation.
  • When a dedicated GPU backend is unavailable, these models try an integrated GPU before falling back to CPU. Forced-CPU behavior remains unchanged.

Walkthrough

MagpieTTS and NanoCodec now try an integrated-GPU backend after GPU initialization fails, before falling back to CPU. MagpieTTS also accepts integrated-GPU devices when checking devices for CUDA local-transformer mirrors.

Changes

Integrated-GPU Backend Support

Layer / File(s)	Summary
Device acceptance and backend fallback src/tts/magpietts/model.cpp, src/tts/nanocodec/model.cpp, src/tts/magpietts/lt.cpp	MagpieTTS and NanoCodec try an integrated-GPU backend after GPU initialization fails, with CPU fallback retained. MagpieTTS accepts integrated-GPU devices for CUDA mirror initialization.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

---

Merge Risk: ⚪ Minimal · up to d3a50

The loaders now try an integrated GPU before CPU fallback, and MagpieTTS permits the selected device for its CUDA mirror. No concrete merge-blocking failure is established, though the pinned ggml predicate’s handling of IGPU remains unverified.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly summarizes the main change: integrated GPU support for MagpieTTS and NanoCodec, including the CUDA local-transformer mirror.
Description check		The description is directly related to the changeset. It explains the integrated-GPU issue, the affected code paths, the implementation, scope, and validation results.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Confidence Score: 5/5

[Medium impact] This PR appears safe to merge; no actionable defects were found.

Summary

Adds integrated GPU support to both TTS loaders and the CUDA local-transformer mirror.

• MagpieTTS and NanoCodec can now load on integrated GPUs.
• The Magpie CUDA mirror accepts integrated GPUs.

_{Reviews (1) · Last reviewed commit: "[pre-commit.ci] auto fixes from pre-comm..." · Reviewed by Greptile}

[anand-nv]

The CUDA backend in ggml-cuda.cu unconditionally returns GGML_BACKEND_DEVICE_TYPE_GPU from ggml_backend_cuda_device_get_type() (line 4639) — it never reports IGPU for any CUDA device. This is confirmed on both GB10/DGX Spark and Jetson Thor. I'm not sure why this fails for you.
Can you provide more details on this? like the firmware version etc?

[femo23]

Thanks for looking — I think we're reading two different ggml trees, which explains the disagreement.

The unconditional return GGML_BACKEND_DEVICE_TYPE_GPU is in the old vendored ggml submodule (ggml-org/ggml c03b4e2, 2026-05-21, ggml-cuda.cu ~L4988), i.e. what 0.1.0 and main before #57 built against. Upstream changed this a week later in ggml-org/llama.cpp#23007 (2026-05-28, "ggml: auto apply iGPU flag CUDA/HIP if integrated device"):

static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) {
    ...
    return prop.integrated ? GGML_BACKEND_DEVICE_TYPE_IGPU : GGML_BACKEND_DEVICE_TYPE_GPU;
}

#57 (2026-10-01) dropped the ggml submodule and moved the llama.cpp pin from 560445b (May 16) to bd4f514 (Sep 24), which includes that change — and v0.2.0 carries the same bd4f514 pin (ggml/src/ggml-cuda/ggml-cuda.cu L5079–5088 there). Since GB10 and Thor both report cudaDeviceProp.integrated = 1, every 0.2.0 build on those parts now gets _IGPU from ggml, and ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_GPU) returns null for the TTS loaders. The ASR runtime already accepts both types (src/runtime/ggml/backend.cpp), which is why only TTS regressed. I'd guess your GB10/Thor confirmation was against the pre-#57 tree.

Details from my box, built from main @ 8642eaa with docker/Dockerfile (CUDA_ARCH=121, CUDA 13.0.0 toolkit per the Dockerfile pin):

• NVIDIA DGX Spark (GB10), DGX OS build 2025-10-04, Ubuntu 24.04.5, kernel 7.0.0-1019-nvidia, driver 580.178.04
• nemo-speech doctor (0.2.0): [0] integrated-gpu NVIDIA GB10 (121.7 GiB) — that label comes straight from ggml_backend_dev_type()
• unpatched: nemo-speech serve ... --device cuda:0 → --lt-backend cuda requires a CUDA ggml backend; current backend is CPU; without --device, TTS silently runs on CPU (~6 s per sentence)
• patched: MagpieTTS backend: CUDA0 - NVIDIA GB10, NanoCodec backend: CUDA0 - NVIDIA GB10, first audio 29–34 ms

Easy to reproduce on any Spark/Thor with the 0.2.0 release binaries: run doctor and check whether the device line says integrated-gpu. Happy to add anything else you'd like to see.

[anand-nv]

Verified it works. Issue is due to the ggml version.