Skip to content

Add sum_scatter to the ring backend - #4236

Open
erwinzhang7 wants to merge 1 commit into
ml-explore:mainfrom
erwinzhang7:ring-sum-scatter
Open

Add sum_scatter to the ring backend#4236
erwinzhang7 wants to merge 1 commit into
ml-explore:mainfrom
erwinzhang7:ring-sum-scatter

Conversation

@erwinzhang7

@erwinzhang7 erwinzhang7 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Fixes part of #4285.

all_reduce_impl already runs its send/recv plan twice, and the code says why:
"Two times the same send/recv operations, first scatter reduce and then gather."
The k == 0 half is a reduce-scatter, so sum_scatter is that phase with the local
segment kept instead of continuing into the gather.

Verified on two M4 Pro minis over a direct Thunderbolt link: the call returns
cleanly on both ranks where current main hangs indefinitely, and the ring
distributed test suite gains a sum_scatter case.

Note that this does not address the second half of the issue. An error raised out of
ReduceScatter::eval_cpu still does not reach the caller; this change means the
common path no longer raises at all.

@zcbenz zcbenz added the await verification This pull request is non-trivial and requires a human expert to verify its correctness. label Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants