fix(rollout): pause generation before releasing offload memory

[EazyReal]

What changed

RolloutServer.offload() is now a three-phase quiescence transition across all offloaded engine groups: pause every engine and wait, flush every engine and wait, then release memory. RolloutServer.onload_kv() restores KV-cache/CUDA-graph memory before issuing continue_generation(). The new ServerGroup helpers keep the existing non-blocking Ray ObjectRef shape and skip needs_offload=False groups, matching offload()/onload().

Why

offload() previously called release_memory_occupation() without first making the rollout engines quiescent. The default rollout function is mostly safe today because it aborts in-flight requests until the server reports idle before returning, but that guarantee lives in the rollout function rather than at the offload boundary. A custom --rollout-function-path or a multi-turn client paused between turns can return with pending work. In that state, the flush inside release_memory_occupation() can either spin until TimeoutError("Timeout while flushing cache."), or it can succeed during a transient idle window while a newly admitted request races with memory release.

Pausing first makes the offload boundary own the invariant: no new generation is admitted, all current generation is drained, and only then is memory released.

Coordination model

This PR uses Ray phase barriers at the rollout-manager boundary: all pause RPCs are acknowledged before any flush RPC is issued, and all flush RPCs are acknowledged before any release RPC is issued. That is the right layer for this transition because rollout offload/onload is driven by RolloutManager, not by all Megatron ranks participating in a collective.

This matches the current verl SGLang shape: SGLang sleep/wake is coordinated by awaiting the rollout server calls across replicas, while explicit torch.distributed barriers are used in backend paths such as TRT-LLM resume/update where all distributed training ranks participate and non-leader ranks may still be offloading. slime already uses that distributed-barrier pattern in the actor-side weight-update paths (update_weight_from_tensor.py, update_weight_from_distributed.py, update_weight_from_distributed_delta.py). This PR keeps those existing rank collectives in the weight-update layer and adds the missing rollout-engine phase barrier where the offload race occurs.

Validation

• uv run --no-project --with pytest --with requests --with pyyaml python -m pytest tests/test_rollout_offload_coordination.py -q
• python3 -m py_compile slime/ray/rollout.py tests/test_rollout_offload_coordination.py
• git diff --check upstream/main...HEAD
• uv run --no-project --with jinja2 python .github/workflows/generate_github_workflows.py leaves the generated workflow unchanged.

[EazyReal]

@zhuzilin could you review this one? It fixes a rollout offload race by making offload own the quiescence invariant: pause all engines and wait, flush all and wait, then release memory. I checked current verl: SGLang sleep/wake uses replica/server awaits; torch distributed barriers remain in backend weight/update paths where all ranks participate, which matches the layering here.

[EazyReal]

@zhuzilin refreshed on latest main and checks are green. This pauses generation and flushes cache before releasing offload memory, then resumes generation after KV/CUDA graph onload; it reduces unsafe offload while requests are active. Could you review when you have bandwidth?