feat: Add mu-law support for voicing. - #304
Open
KakaruHayate wants to merge 12 commits into
Open
Conversation
Introduce 'mulaw' voicing_domain and mu-law energy handling across configs and preprocessing. Configs updated to include voicing_domain='mulaw' (acoustic/variance templates and instances). Acoustic/variance binarizers now pass hparams['voicing_domain'] to get_voicing. Parameter adaptor clamps voicing max to 0 when domain is 'mulaw'. get_energy_librosa and get_voicing gain new domain/mu params and implement mu-law compression (default mu=255) to produce a dB-like range compatible with existing pipelines. These changes add support for mu-law representation of voicing/energy. revert
|
@codex review |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Author
|
Will be implemented in V3 |
…nvpi#305 Replace usage of onnxslim with onnx-simplifier (onnxsim) across exporters. Exporters now call onnxsim.simplify(..., include_subgraph=True) and assert the returned validation flag. Remove the simplify_onnx helper from utils/onnx_helper.py and update requirements to depend on onnxsim>=0.6.5. Adjust variance exporter prefix/ignored-pattern handling and remove an unused re import. Update GettingStarted.md to relax Python requirement to 3.8+ and clarify the PyTorch recommendation. Update GettingStarted.md Update GettingStarted.md
Add an optional `noise` input to acoustic/pitch/variance ONNX export so the initial sampling noise comes from outside the graph instead of an internal RandomNormalLike. Feeding identical noise makes inference fully deterministic and reproducible (enables seed-based retake upstream). Defaults to None -> internal torch.randn, so training / eager paths are unchanged. No retrain needed. See SEED_NOISE_CHANGES.md for the I/O contract and verification steps.
Port the pitch/variance retake mechanism to the acoustic model. In keep regions the previously generated mel is fed back as a condition so the model reproduces it; retake regions are regenerated from fresh noise. - acoustic_encoder: add retake_embed(2,H) + mel_cond_proj(mel_bins->H); condition += retake_embed(mask) + mel_cond_proj(norm(gt_mel) * keep). Use float keep = 1 - retake.float() instead of bool ~retake to avoid an ONNX Not op that fails on the DirectML EP. Gated by use_acoustic_retake. - toplevel: thread acoustic_retake + gt_mel through DiffSingerAcoustic.forward. - acoustic_task: sample a fresh retake mask per step; move shared random_retake_masks to utils (reused by variance_task). - deployment: feed retake/gt_mel into the fs2_aux graph, InputMetadata-gated; diffusion graph and externalized noise untouched (stock models unaffected). - configs/acoustic.yaml: use_acoustic_retake default false.
KakaruHayate
added a commit
to KakaruHayate/DiffSinger
that referenced
this pull request
Jul 2, 2026
Sync only mu-law related changes from openvpi/DiffSinger PR openvpi#304: - voicing_domain: 'mulaw' in configs (all 4 config files) - mu-law compression in binarizer_utils (get_energy/get_voicing) - voicing range clamp in param_adaptor - pass voicing_domain in acoustic/variance binarizers
KakaruHayate
added a commit
to KakaruHayate/DiffSinger
that referenced
this pull request
Jul 2, 2026
Sync mu-law changes from openvpi/DiffSinger PR openvpi#304: - voicing_domain: mulaw in configs - mu-law compression in binarizer_utils - voicing range clamp in param_adaptor - pass voicing_domain in binarizers
…ens_b + blend) P1-a phoneme mix experiment. Acoustic fs2_aux export gains two per-token inputs: - tokens_b (int64 [1, n_tokens]): secondary phoneme token sequence - blend (float32 [1, n_tokens]): per-token mix weight in [0, 1] txt_embed = (1 - blend) * emb(tokens) + blend * emb(tokens_b); blend all-zeros is bit-identical to the no-mix model. Mirrors the noise-externalization pattern (1e51d45): optional kwargs default to None (plain lookup in training), always fed at export.
…r (tokens_b + blend) Extends the P1-a phoneme mix to the variance-side linguistic encoder, shared by dur/pitch/variance roles. Both forward paths (word / phoneme) blend two txt_embed lookups via _blend_txt_embed; blend all-zeros => bit-identical to the no-mix encoder. Mix inputs passed as trailing kwargs so the optional 'languages' positional does not shift. Per-token axis (n_tokens), aligned to tokens.
P3 包络:blend 从逐音素 [1,n_tokens] 改为逐帧 [S,n_frames](S=目标槽数,动态)。 base + S 个目标音素流一次批量 encoder,共用 base 的 mel2ph/extra_embed/spk_embed 展开到帧, 再逐帧凸组合(base 权重=1-Σblend,clamp>=0)。blend 全 0 => 与原模型逐值一致。 tokens_b:[S,n_tokens] 目标; blend:[S,n_frames]; 槽轴(0)动态,trace 用 2 槽。 风险点=encoder 批量(batch>1)导出/运行是否正确,需 Python 侧先验证。variance 侧不变(仍 P1-a 逐音素)。
pitch/variance 的 pre 段在 forward_mel2x_gather 展开 encoder_out 得 condition 后, 用 _blend_condition 逐帧凸组合 base + S 个目标 encoder_out(各自 gather 到帧,共用 ph_dur): condition = base_w·cond_base + Σ blend_k·cond_target_k, base_w=1-Σblend(clamp>=0)。 仅 encoder_out 派生的 condition 参与混合,melody/retake/pitch/spk 等共享项在混合后叠加一次; 去噪(reflow/diffusion)在混合后的单条 condition 上只跑一次。新增 encoder_out_b[S,nTok,H]+ blend[S,nFrames](槽轴动态),默认 zeros => 逐值等价无混合。S=1 全程 batch1。variance_pre 的 混合输入走 trailing kwargs 避开可选 spk_embed 位移。
onnx.compose.merge_models 不前缀边(value)名,base_w 的 .clamp(min=0) 产生的 /Clip_output_0 与 post/diffusion 的 clamp_spec Clip 撞名 → 合并报 overlapping edge names。改为 base_w=1-Σblend (不 clamp):凸性由 C# 保证(单槽 blend∈[0,1]⇒Σ≤1;N槽归一),故 1-Σ≥0,clamp 本就多余。 acoustic(fastspeech2)+ pitch/variance role(toplevel._blend_condition)两处同修。
Raw log-mel 0.0 is not a neutral padding value: norm_spec maps it to +1.0 — the very top of the normalized range, i.e. maximum loudness. Every batch therefore filled the padded tail of shorter samples with full-loudness garbage. Two consequences: 1. The diffusion backbones (WaveNet / LYNXNet / LYNXNet2) receive no padding mask, so their receptive field (~181 frames for LYNXNet2 with kernel_size=31 x 6 layers) leaks the fake signal into the trailing valid frames. The loss mask (mel2ph > 0) hides this from the loss on padding frames, but the contaminated valid frames near the boundary are fully counted — a systematic bias on utterance tails, exactly where breathy endings and vibrato decay live. 2. The aux decoder loss is not masked at all: with zero-padding the aux decoder was actively trained to predict maximum loudness on padding frames from near-zero condition. Padding with spec_min (-12 by default) maps to -1.0 (silence) and sits next to the mel extractor's true silence floor log(1e-5) = -11.51, so padded regions now look like ordinary trailing silence — consistent with what the model sees at inference time. Note: this changes the training data distribution slightly; models trained before/after this fix are checkpoint-compatible but their padded-region behavior differs.
Replace zero-padding with spec_min for mel padding to avoid full-loudness garbage in shorter samples.
KakaruHayate
added a commit
to KakaruHayate/DiffSinger
that referenced
this pull request
Jul 31, 2026
Source: openvpi/DiffSinger PR openvpi#304 at a9b9be6. Existing 0.06 smoothing widths were preserved while adding the reviewed mulaw domain configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce 'mulaw' voicing_domain and mu-law energy handling across configs and preprocessing. Configs updated to include voicing_domain='mulaw' (acoustic/variance templates and instances). Acoustic/variance binarizers now pass hparams['voicing_domain'] to get_voicing. Parameter adaptor clamps voicing max to 0 when domain is 'mulaw'. get_energy_librosa and get_voicing gain new domain/mu params and implement mu-law compression (default mu=255) to produce a dB-like range compatible with existing pipelines. These changes add support for mu-law representation of voicing/energy.