Skip to content

fix: cancel started rollouts when trajectory_group_batches exits - #1141

Open
salma2vec wants to merge 1 commit into
OpenPipe:mainfrom
salma2vec:fix/cancel-batch-rollouts-on-exit
Open

salma2vec wants to merge 1 commit into
OpenPipe:mainfrom
salma2vec:fix/cancel-batch-rollouts-on-exit

Conversation

@salma2vec

@salma2vec salma2vec commented Oct 22, 2026 •

Copy link
Copy Markdown

Summary

Closes #1140

trajectory_group_batches started rollouts as tasks but never cancelled them when it exited early. Closing the iterator, raising a rollout error past max_batch_exceptions, or cancelling the consumer left those rollouts running with no owner. They kept hitting the inference server and environments after the training loop had stopped.

Changes

  • src/art/batches.py: wrap the batching loop in try/finally. On exit, cancel the tasks still in pending and asyncio.gather(..., return_exceptions=True) them together with the last done set. This also reads the sibling exceptions that were left unread when task.result() raised on the first failure, so asyncio no longer logs Task exception was never retrieved.
  • Most of the diff is re-indentation under the new try. Batching, ordering, skip_batches and exception-budget behaviour are unchanged. Coroutines that were never started stay owned by the caller, which test_batches_completion_bursts.py already assumes.
  • tests/unit/test_batches_cleanup.py: four regressions, one each for aclose(), an over-budget failure, consumer cancellation, and unread sibling exceptions. Each test releases the blocked rollout after the exit and checks that it was cancelled and never finished.

Test plan

  • The four new tests fail on main (5f0c1fb) and pass with this change.
  • The issue's reproduction prints ['finished after iterator close'] on main and [] with this change.
  • tests/unit/test_batches_completion_bursts.py (12 tests) passes unchanged.
  • test_metric_routing, test_metrics_builder, test_trajectory_copy, test_yield_trajectory: 84 passed in total with the batches suites. Three test_pipeline_trainer_batching tests fail identically on main in my environment because msgspec/nest_asyncio aren't installed. They don't touch batches.py.
  • ruff format / ruff check (v0.15.11, the pre-commit pin) are clean on the changed files.
  • I didn't run the full prek run --all-files or ty. A full uv sync needs the CUDA backend deps, which aren't available on macOS.

Found and fixed with Claude Code. The red/green runs above were executed locally.

🤖 Generated with Claude Code

Closing the iterator, propagating a rollout error, or cancelling the
consumer left the rollout tasks it had started running with no owner.
Cancel and await them on exit, and retrieve sibling exceptions left
unread when the first failure is raised.

Closes OpenPipe#1140

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

trajectory_group_batches leaves started rollouts running after close, error or cancellation

1 participant