[Refactor] Drop non-global aux-loss averaging mode#1965
Closed
HAOCHENYE wants to merge 1 commit into
Closed
Conversation
BREAKING CONFIG CHANGE: remove BalancingLossConfig.balancing_loss_global_average and ZLossConfig.z_loss_global_average. These switches only selected the per-rank (non-global) averaging mode, which is incompatible with reduce-sum gradients: the router/gate is a replicated param whose gradient is SUM-aggregated across the replicate group, so a full per-rank aux loss would inflate that gradient by world_size (the old FSDP AVG hid this by mean-reducing). No config or test in the repo ever set either switch to False. The finalize/accumulate branch now keys purely on dist.is_initialized(): distributed uses the global-average statistics; the single-process branch is kept (reference / eval run with the process group uninitialized) and is numerically the world-size-1 special case of the global branch (tokens_per_expert_global == tokens_per_expert_local, seqlen_global == valid_tokens), so its values are unchanged. Design doc S5.3 updated to state aux loss has only one averaging mode. Verified: single-process balancing/z-loss values bit-identical before/after; distributed balancing A/B, z-loss A/B and EP=1/EP=2 token-mean parity all pass.
HAOCHENYE
force-pushed
the
reduce-sum-04-remove-world-size
branch
from
July 17, 2026 08:11
006ac3c to
4f4db55
Compare
HAOCHENYE
force-pushed
the
reduce-sum-05-drop-nonglobal
branch
from
July 17, 2026 08:11
853ebc1 to
433634c
Compare
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR (5/5) — breaking config change. Drops the
balancing_loss_global_average/z_loss_global_averageswitches and the non-global aux-loss mode, which is incompatible with reduce-sum (SUM-aggregated replicated router grad would inflate by world_size). Distributed = global-average; single-process is its W=1 special case.Full stack (merge bottom-up, under #1959):
reduce-sum-01-fsdp-helper— [FSDP] Add reduce-sum gradient reduction helperreduce-sum-02-split-logging— [Loss] Split logging loss from backward lossreduce-sum-03-switch-sum— [FSDP][Loss] Switch gradient reduction to SUMreduce-sum-04-remove-world-size— [Refactor] Remove unused world_size plumbingreduce-sum-05-drop-nonglobal— [Refactor] Drop non-global aux-loss averaging mode