fix: pad mel with spec_min instead of 0.0 in acoustic collater#313
Open
KakaruHayate wants to merge 2 commits into
Open
fix: pad mel with spec_min instead of 0.0 in acoustic collater#313KakaruHayate wants to merge 2 commits into
KakaruHayate wants to merge 2 commits into
Conversation
Raw log-mel 0.0 is not a neutral padding value: norm_spec maps it to +1.0 — the very top of the normalized range, i.e. maximum loudness. Every batch therefore filled the padded tail of shorter samples with full-loudness garbage. Two consequences: 1. The diffusion backbones (WaveNet / LYNXNet / LYNXNet2) receive no padding mask, so their receptive field (~181 frames for LYNXNet2 with kernel_size=31 x 6 layers) leaks the fake signal into the trailing valid frames. The loss mask (mel2ph > 0) hides this from the loss on padding frames, but the contaminated valid frames near the boundary are fully counted — a systematic bias on utterance tails, exactly where breathy endings and vibrato decay live. 2. The aux decoder loss is not masked at all: with zero-padding the aux decoder was actively trained to predict maximum loudness on padding frames from near-zero condition. Padding with spec_min (-12 by default) maps to -1.0 (silence) and sits next to the mel extractor's true silence floor log(1e-5) = -11.51, so padded regions now look like ordinary trailing silence — consistent with what the model sees at inference time. Note: this changes the training data distribution slightly; models trained before/after this fix are checkpoint-compatible but their padded-region behavior differs.
Replace zero-padding with spec_min for mel padding to avoid full-loudness garbage in shorter samples.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The acoustic collater pads mel with raw log-mel
0.0, which is not a neutral value:norm_specmaps it to +1.0 — the top of the normalized range, i.e. maximum loudness. Every batch fills the padded tail of shorter samples with full-loudness garbage.Two consequences:
Backbone receptive-field leakage. The diffusion backbones (WaveNet / LYNXNet / LYNXNet2) receive no padding mask. LYNXNet2's receptive field is ~181 frames (kernel_size=31 × 6 layers), so the fake full-loudness signal leaks into the trailing valid frames of every non-longest sample. The loss mask (
mel2ph > 0) only hides padding frames themselves — the contaminated valid frames near the boundary are fully counted. This is a systematic bias on utterance tails, exactly where breathy endings and vibrato decay live.Aux decoder trained on garbage. The aux decoder (shallow diffusion) loss is not masked at all: with zero-padding it was actively trained to predict maximum loudness on padding frames from near-zero condition.
Measured on an untrained LYNXNet2 (1024ch × 6L): valid-frame output shift up to ~1.3% within the receptive field of the padding boundary, zero beyond it.
Fix
Pad with
spec_min(-12 by default) instead. It maps to -1.0 (silence) after normalization and sits next to the mel extractor's true silence floorlog(1e-5) = -11.51, so padded regions now look like ordinary trailing silence — consistent with what the model sees at inference.One-line behavioral change + comment; falls back to -12.0 if
spec_minis absent from config.Compatibility