Skip to content

[dist] XPU distributed backend selection (XCCL / oneCCL) is unverified — needs XPU hardware #3777

Open

Description

@li-lizhe

Context

#3771 binds distributed training to the active accelerator instead of assuming CUDA. Writing that change surfaced a second, independent defect on XPU.

get_default_dist_backend() returns "nccl" for every non-NPU device. Since get_accelerator_device_type() does select XPU, an XPU run reaches init_process_group(backend="nccl") and cannot start at all on a build without NCCL — the same class of failure #3771 fixes for Ascend NPU.

Why this is not in #3771

The correct backend for XPU is XCCL (oneCCL; the legacy name on older builds is "ccl"), so the fix is small:

_ACCELERATOR_DIST_BACKENDS = {"npu": ("hccl",), "xpu": ("xccl", "ccl")}

but it advertises a runtime path that nobody here can exercise. We have no Intel XPU hardware in our environment, so under the repository's hardware-validation requirement the XPU arm was the only path #3771 changed without real-device evidence. #3771 therefore keeps the historical "nccl" value for XPU — i.e. exactly the pre-PR behaviour, unchanged — and the selection is tracked here for whoever has the hardware.

What a fix needs

  1. Add the XPU entry above (keep "ccl" only for builds that positively confirm it through torch.distributed.is_ccl_available()).
  2. Run an exact-head 2-rank torchrun on Intel XPU with XCCL, showing per-rank xpu:0 / xpu:1 binding, an all_reduce, DDP forward/backward, the end-of-epoch cache release, plus the exact PyTorch / oneAPI / device versions.
  3. Update tests/test_distributed_device_setup.py accordingly — the current test pins the narrowed behaviour (XPU stays "nccl" even when a build reports is_xccl_available()), so it needs to be flipped.

Related

Activity

  1. li-lizhe commented on Oct 17, 2026

    @li-lizhe
    ContributorAuthor

    The mirror of this on the PR side: #3771 is narrowed so it keeps the pre-PR "nccl" value for XPU (no behaviour change there), which is what makes that PR's every changed runtime path verifiable on hardware. See the reply on #3771 for the exact-head Ascend evidence that remains in that PR.

  2. added a commit that references this issue on Oct 21, 2026
    564839b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions