[kevssim]

What does this PR do?

Adds delta (sparse) weight sync over HCCL for Ascend NPU to the delta_sharded
checkpoint engine, so NPU RL runs can skip full-weight resyncs between trainer and
rollout the same way CUDA hosts do.

Highlights

• One pipeline, two transports — the seed/steady state machine, bucketing and flush
  manifests live in a transport-agnostic DeltaSyncMixin shared by the NCCL and HCCL
  engines. The CUDA path is refactored but behaviourally unchanged.
• Direct broadcast into the vLLM workers — NPU has no device IPC, so trainer rank 0
  broadcasts each flush into a persistent PyHccl group whose members are the vLLM worker
  processes themselves: one zmq manifest subscription plus one or two broadcasts per flush,
  no IPC-handle negotiation.
• Zero new config — backend=delta_sharded auto-selects the HCCL engine on NPU and the
  NCCL engine on CUDA; the vLLM weight-transfer backend likewise defaults to
  twinkle_delta_hccl on NPU and twinkle_delta_ipc elsewhere.
• Tied-embedding fix on the full-sync path — with tie_word_embeddings=True the HF
  state_dict lists one tensor twice (lm_head.weight and *.embed_tokens.weight). vLLM
  loads only the name its traversal reaches first and then asserts the canonical one did
  load (AutoWeightsLoader._check_skipped_aliases), so sending both made that check fail.
  The duplicate head is now dropped; one write covers both.

Scope and limitations

vLLM rollout only; BF16 non-quantized dense models; trainer FSDP2 with TP=PP=EP=1; vLLM
internal data_parallel_size=1; no LoRA; rebuild_group=True is not supported (the HCCL
group is persistent). Multiple rollout replicas are supported (replica-major, TP-minor).

Test

Unit and multi-process suites under tests/checkpoint_engine/:

• test_delta_hccl.py — trainer-side engine: rendezvous, lazy communicator, close.
• test_delta_hccl_receiver.py — receiver side: wire decode, pump protocol, channel lifecycle.
• test_delta_hccl_loopback.py + _hccl_loopback.py — multi-process end-to-end protocol on
  CPU (gloo stands in for PyHccl, zmq is real), covering seed/steady rounds, zero-change
  rounds, bucket slicing, reseed, corrupted payloads and shuffled receiver start order.
• test_delta_npu_orchestration.py — manager rendezvous and the rollout RPC sequence, mocked.
• test_delta_vllm.py — vLLM integration: patch loaders, tied aliases, TP geometry.

Validated on a 2-node Ascend cluster (2-card FSDP2 trainer + 2-card vLLM rollout, TP=2)
with Qwen3.5-9B and Qwen3.5-4B: delta-synced rollout weights are byte-identical to a
full resync across change ratios from 0% to 3%, plus a real optimizer step.