Skip to content

fix(rl): avoid broken ROCm IPC event stream waits - #2533

Merged
TATP-233 merged 1 commit into
mainfrom
fix/issue-2530-rocm-ipc-event-wait
Oct 26, 2026
Merged

TATP-233 merged 1 commit into
mainfrom
fix/issue-2530-rocm-ipc-event-wait

Conversation

@TATP-233

Copy link
Copy Markdown
Collaborator

Summary

  • Add an explicit event_wait="stream" | "synchronize" mode to CudaRingTransport.
  • Select synchronize on HIP/ROCm and keep stream waits on NVIDIA CUDA.
  • Document the ROCm compatibility path as a performance-impacting fallback: unfixed ROCm CLR 7.2 emulated IPC event streamWaitEvent omits the 32-entry signal-ring modulo, so the 33rd reuse addresses signal[32] and fails with hipErrorInvalidValue.
  • Add unit coverage for runtime selection and host-event synchronization plus a real HIP slow test that crosses the 33rd event reuse.

This partially addresses #2530; the upstream runtime and benchmark-provenance items remain tracked there.

Validation

uv run --no-sync pytest -q tests/rl/runtime/ipc/test_inference_ring.py
# 19 passed, 4 deselected

uv run --no-sync mypy src/unilab
# passed

uv run --no-sync pyright
# 0 errors

make check
# passed

make test-all
# 2833 passed, 61 skipped, 375 deselected, 18 warnings
# benchmark import smoke: 35/35 modules, 36/36 scripts

Remote MI300X / ROCm 7.2 / Genesis 1.3.3:

ROCR_VISIBLE_DEVICES=0 uv run --no-sync pytest -q -m slow \
  tests/rl/runtime/ipc/test_inference_ring.py::test_hip_inference_ring_survives_rocm_ipc_event_slot_reuse
# 1 passed

Production-shape learner-coupled SAC run:

ROCR_VISIBLE_DEVICES=0 timeout 600 .venv/bin/train \
  --algo sac --task g1_walk_flat --sim genesis \
  algo.num_envs=2048 algo.batch_size=256 \
  algo.replay_buffer_n=4 algo.learning_starts=1 \
  algo.updates_per_step=1 algo.max_iterations=128 \
  algo.save_interval=1000 algo.algo_params.use_compile=false \
  training.no_play=true training.export_onnx=false \
  training.log_dir=/tmp/unilab-rocm-genesis-ipc-fix-2048
# exit 0; completed_iterations=128; total_env_steps=264,192
# tail_env_steps_per_sec=34,690.9; final_env_steps_per_sec=36,000.4
# normal_completion, no dropped replay batches, collector exit 0

Small 64-env run also crossed the former tick-32 failure:

completed_iterations=128
total_env_steps=8,448
tail_env_steps_per_sec=1,257.4
final_cycle_wall_ms=51.74
status=completed

@TATP-233
TATP-233 requested a review from caozx1110 October 26, 2026 15:35
@TATP-233
TATP-233 merged commit df65c22 into main Oct 26, 2026
12 checks passed
@TATP-233
TATP-233 deleted the fix/issue-2530-rocm-ipc-event-wait branch October 26, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant