fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec by hsuanguo · Pull Request #60 · NVIDIA/NeMo-Speech.cpp

hsuanguo · 2026-10-04T23:41:47Z

Got this error:

--lt-backend cuda requires a CUDA ggml backend; current backend is CPU
nemo-speech serve: MagpieTTS synthesis failed

Add an IGPU initialization attempt after discrete-GPU initialization fails, before the existing CPU fallback, in both MagpieTTS and NanoCodec.

copy-pr-bot · 2026-10-04T23:41:51Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

coderabbitai · 2026-10-04T23:42:15Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/NeMo-Speech.cpp/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 70d0af72-ed24-4bc2-8ab9-f240547c7689

📥 Commits

Reviewing files that changed from the base of the PR and between b809bbb and 864559c.

📒 Files selected for processing (2)

src/tts/magpietts/model.cpp
src/tts/nanocodec/model.cpp

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Summary

Summary by CodeRabbit

Improvements
- MagpieTTS and NanoCodec now try an integrated GPU when the primary GPU is unavailable, before falling back to CPU.
- When CPU is forced, GPU initialization is skipped.

Walkthrough

MagpieTTS and NanoCodec now attempt integrated-GPU initialization after GPU initialization fails. The existing CPU fallback remains. The integrated-GPU attempt is skipped when CPU is forced.

Changes

TTS backend fallback

Layer / File(s)	Summary
Integrated GPU fallback `src/tts/magpietts/model.cpp`, `src/tts/nanocodec/model.cpp`	When CPU is not forced and GPU initialization fails, both model loaders try the integrated GPU backend before the existing CPU fallback.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 86455

The change adds an integrated-GPU attempt before the existing CPU fallback. No actionable merge-blocking risk is established by the available evidence.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the integrated-GPU backend support added for MagpieTTS and NanoCodec.
Description check	✅ Passed	The description explains the reported backend error and the integrated-GPU initialization attempt added before the CPU fallback in both models.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

Signed-off-by: hsuanguo <hsuan.guo@gmail.com>

greptile-apps · 2026-10-04T23:46:19Z

Confidence Score: 4/5

[Medium risk] Adds integrated GPU fallback to text-to-speech model loading.

The PR should not merge until an IGPU allocation failure can fall back to CPU without aborting synthesizer creation.

Findings

IGPU allocation prevents CPU fallback ▶

Summary

The PR adds integrated-GPU initialization between discrete-GPU initialization and CPU fallback for MagpieTTS and NanoCodec.

Model allocation failure after IGPU initialization does not fall back to CPU.

_{Reviews (1) · Last reviewed commit: "fixed tts loaders on igpu"}

greptile-apps · 2026-10-04T23:46:23Z

+        if (!model.backend) {
+            model.backend = ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_IGPU, nullptr);


IGPU allocation prevents CPU fallback

If an integrated GPU initializes but lacks enough memory for the model tensors, loading returns an error instead of trying the CPU backend. NanoCodec has the same path. Either failure stops synthesizer creation, even though the CPU fallback could have loaded the model. Retry loading on CPU when IGPU allocation fails, or check capacity before selecting the IGPU.

anand-nv · 2026-10-05T10:53:50Z

Can I know what is the hw platform you are trying to deploy this on?

hsuanguo · 2026-10-05T11:34:20Z

Can I know what is the hw platform you are trying to deploy this on?

Hi, it's Jetson Orin on Jetpack 7.2

femo23 · 2026-10-11T06:00:18Z

Confirming this on a second platform: DGX Spark (GB10, CUDA 13.0, sm_121). nemo-speech doctor lists the device as [0] integrated-gpu NVIDIA GB10, so ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_GPU) returns null and Magpie silently runs on CPU (~6 s per sentence); with this change Magpie v2607 + NanoCodec load on CUDA0 and first audio drops to 29–34 ms (RTF 0.05) over the Riva gRPC stream.

One more site has the same check: src/tts/magpietts/lt.cpp (magpietts_model_init_local_transformer_copy) requires dev_type == GGML_BACKEND_DEVICE_TYPE_GPU, so once the loaders are fixed, --tts.lt-fp32 with --tts.lt-backend cuda/auto still fails on these devices with "cannot create CUDA local-transformer mirror from a non-GPU model". Accepting _IGPU there too works on the Spark (MagpieTTS local transformer mirror: CUDA0, precision=fp32 (61 tensors), 37 ms first audio).

I've opened #65 which carries your commit unchanged plus that third hunk, so the fix is complete in one PR — happy to drop mine to just the lt.cpp commit if this one merges first, or if you'd rather fold the hunk into this PR directly:

const auto dev_type = device ? ggml_backend_dev_type(device) : GGML_BACKEND_DEVICE_TYPE_CPU;
if (!device || (dev_type != GGML_BACKEND_DEVICE_TYPE_GPU && dev_type != GGML_BACKEND_DEVICE_TYPE_IGPU)) {

@anand-nv — hardware here is DGX Spark; the Jetson Orin report above and this one are the same root cause (cudaDeviceProp.integrated → _IGPU).

…ocal-transformer mirror (extends #60) (#65) * fixed tts loaders on igpu Signed-off-by: hsuanguo <hsuan.guo@gmail.com> * fix(tts): accept integrated GPU devices for the CUDA local-transformer mirror ggml reports unified-memory NVIDIA parts (DGX Spark GB10, Jetson) as GGML_BACKEND_DEVICE_TYPE_IGPU. With the loader fix for MagpieTTS and NanoCodec (#60) the model now gets a CUDA backend on these devices, but magpietts_model_init_local_transformer_copy() still required the discrete GPU type, so --tts.lt-fp32 with --tts.lt-backend cuda (or auto) failed with "cannot create CUDA local-transformer mirror from a non-GPU model". Accept IGPU there too, matching src/runtime/ggml/backend.cpp. Verified on DGX Spark (GB10, CUDA 13.0, sm_121): Magpie v2607 + NanoCodec on CUDA0, local-transformer mirror created on CUDA0 at fp32 (61 tensors), first audio 29-37 ms, RTF 0.05-0.08; previously ~6 s per sentence on CPU. Signed-off-by: Femo <femoalaofin@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: hsuanguo <hsuan.guo@gmail.com> Signed-off-by: Femo <femoalaofin@gmail.com> Co-authored-by: hsuanguo <hsuan.guo@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

anand-nv · 2026-10-11T06:34:10Z

#65 includes your fixes. Closing this PR.

hsuanguo requested review from anand-nv, pskrunner14 and rmittal-github as code owners October 4, 2026 23:41

fixed tts loaders on igpu

864559c

Signed-off-by: hsuanguo <hsuan.guo@gmail.com>

hsuanguo force-pushed the fix/tts-loader-igpu branch from dbaab03 to 864559c Compare October 4, 2026 23:44

greptile-apps Bot reviewed Oct 4, 2026

View reviewed changes

femo23 mentioned this pull request Oct 11, 2026

fix(tts): integrated GPU support for MagpieTTS/NanoCodec incl. CUDA local-transformer mirror (extends #60) #65

Merged

anand-nv closed this Oct 11, 2026

Navigation Menu

Repository navigation

fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec - #60

fix(tts): support integrated GPU backends in MagpieTTS and NanoCodec#60
hsuanguo wants to merge 1 commit into
NVIDIA:mainfrom
hsuanguo:fix/tts-loader-igpu

hsuanguo commented Oct 4, 2026

Uh oh!

copy-pr-bot Bot commented Oct 4, 2026

Uh oh!

coderabbitai Bot commented Oct 4, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

greptile-apps Bot commented Oct 4, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Oct 4, 2026

Uh oh!

anand-nv commented Oct 5, 2026

Uh oh!

hsuanguo commented Oct 5, 2026

Uh oh!

femo23 commented Oct 11, 2026

Uh oh!

anand-nv commented Oct 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if (!model.backend) {
		model.backend = ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_IGPU, nullptr);

Conversation

hsuanguo commented Oct 4, 2026

Uh oh!

copy-pr-bot Bot commented Oct 4, 2026

Uh oh!

coderabbitai Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

greptile-apps Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

anand-nv commented Oct 5, 2026

Uh oh!

hsuanguo commented Oct 5, 2026

Uh oh!

femo23 commented Oct 11, 2026

Uh oh!

anand-nv commented Oct 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Oct 4, 2026 •

edited

Loading

greptile-apps Bot commented Oct 4, 2026 •

edited

Loading