support file-based ASR for nemotron streaming model#885
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
rui-ren
changed the title
There was a problem hiding this comment.
Pull request overview
Adds file-based WAV transcription for Nemotron streaming ASR models in the C++ SDK.
Changes:
- Routes Nemotron file requests through the streaming processor.
- Parses 16 kHz PCM16/float32 WAV files with channel downmixing.
- Adds language mapping, temperature handling, and OpenAI JSON output.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
audio_session.h |
Declares Nemotron transcription and WAV parsing helpers. |
audio_session.cc |
Implements Nemotron inference, decoding, language mapping, and WAV parsing. |
| int completion_tokens = 0; | ||
|
|
||
| auto decode_all_tokens = [&]() { | ||
| while (!generator->IsDone() && !original_request.canceled) { |
| size_t count = std::min(kNemotronSamplesPerChunk, samples.size() - offset); | ||
| run_one_pass(processor->Process(samples.data() + offset, count)); | ||
| } | ||
| run_one_pass(processor->Flush()); |
| } | ||
| } | ||
|
|
||
| auto samples = LoadPcmWavAsFloatSamples(req.filename); |
|
Why? Parakeet can execute faster and more accurate file-based transcription. I wouldn't want customers to rely on streaming models for non-streaming scenarios, and we're encouraging them with this. |
This is not a way to encourage them to do github/copilot-cli#4024 (comment) they are using streaming model, however they do file-based transcription and get error raised. IMO, we better to cover this edge case for them, instead of switch models? What do you think? @sylvanc |
Hmm, I haven't realized they're using file-based transcription. Ok, but there are better models for this going forward. |
Add file-based Nemotron ASR transcription support in C++ SDK v2
Summary
This PR completes file-based audio transcription support for Nemotron streaming ASR models in sdk_v2/cpp by finishing the C++ AudioSession implementation and aligning behavior with the existing OpenAI-JSON transcription flow.
For more details, please refer to github/copilot-cli#4024 (comment)
What changed
• Implemented/finished Nemotron file transcription path in:
• Added WAV file parsing for file-based transcription input:
• RIFF/WAVE validation
• fmt / data chunk handling with bounds checks
• 16kHz enforcement
• PCM16 and float32 decoding
• Added Nemotron-specific language runtime option mapping ( lang_id ) and temperature handling.
• Fixed decode flow to keep a single generator across incremental audio processing (100ms chunks + flush), which resolves truncated transcription output for file-based runs.
Behavior
• OpenAI-JSON file transcription for Nemotron models now returns full transcript text from WAV input in SDK v2.
• Existing streaming and non-Nemotron paths remain unchanged.
End-to-end scenario covered
• Model alias: nemotron-3.5-asr-streaming-0.6b
• Input: sample-speech-1m-16k.wav
• Path exercised: JS SDK v2 AudioClient.transcribe(...) → C++ SDK v2 audio transcription pipeline