Repository navigation
Conversation
Closing the iterator, propagating a rollout error, or cancelling the consumer left the rollout tasks it had started running with no owner. Cancel and await them on exit, and retrieve sibling exceptions left unread when the first failure is raised. Closes OpenPipe#1140 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1140
trajectory_group_batchesstarted rollouts as tasks but never cancelled them when it exited early. Closing the iterator, raising a rollout error pastmax_batch_exceptions, or cancelling the consumer left those rollouts running with no owner. They kept hitting the inference server and environments after the training loop had stopped.Changes
src/art/batches.py: wrap the batching loop intry/finally. On exit, cancel the tasks still inpendingandasyncio.gather(..., return_exceptions=True)them together with the lastdoneset. This also reads the sibling exceptions that were left unread whentask.result()raised on the first failure, so asyncio no longer logsTask exception was never retrieved.try. Batching, ordering,skip_batchesand exception-budget behaviour are unchanged. Coroutines that were never started stay owned by the caller, whichtest_batches_completion_bursts.pyalready assumes.tests/unit/test_batches_cleanup.py: four regressions, one each foraclose(), an over-budget failure, consumer cancellation, and unread sibling exceptions. Each test releases the blocked rollout after the exit and checks that it was cancelled and never finished.Test plan
main(5f0c1fb) and pass with this change.['finished after iterator close']onmainand[]with this change.tests/unit/test_batches_completion_bursts.py(12 tests) passes unchanged.test_metric_routing,test_metrics_builder,test_trajectory_copy,test_yield_trajectory: 84 passed in total with the batches suites. Threetest_pipeline_trainer_batchingtests fail identically onmainin my environment becausemsgspec/nest_asyncioaren't installed. They don't touchbatches.py.ruff format/ruff check(v0.15.11, the pre-commit pin) are clean on the changed files.prek run --all-filesorty. A fulluv syncneeds the CUDA backend deps, which aren't available on macOS.Found and fixed with Claude Code. The red/green runs above were executed locally.
🤖 Generated with Claude Code