Repository navigation
Conversation
gather_trajectory_groups and gather_trajectories awaited asyncio.gather, which re-raises the first over-budget failure without cancelling the other rollouts. They kept running against inference, environments and after_each judges with no owner. Cancel and await them before re-raising. Closes OpenPipe#1144 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1144
gather_trajectory_groupsandgather_trajectoriesusedasyncio.gather, which re-raises the first exception but leaves the other awaitables running. When a rollout failed pastmax_exceptions, the caller got the error while the sibling rollouts kept hitting inference, environments andafter_eachjudges with no owner. #1140 / #1141 cover the same gap intrajectory_group_batches.Changes
src/art/gather.py: a small_gather_or_cancelhelper wraps the rollouts as tasks and awaitsasyncio.gather. On any exception it cancels the remaining tasks, awaits them, then re-raises the original exception. The type is unchanged:TaskGroupwould wrap it in anExceptionGroup. Both gather functions use it.set_gather_context, asasyncio.gatherdid, so they see the same context. Results, ordering, the tolerated-exception path and cancellation of the awaiting task are unchanged.wrap_group_awaitable's existingBaseExceptionhandling, so it adds to the failed gather'sexceptionstally in the progress bar. That context is discarded when the gather raises.tests/unit/test_gather_cleanup.py: an over-budget failure cancels the sibling in each function, and a control checks that a tolerated failure still returns every other group.Test plan
main(5f0c1fb) and pass here. The tolerated-failure control passes on both.['finished after gather raised']onmainand[]here.test_batches_completion_bursts,test_ruler_metrics,test_auto_trajectory,test_metrics_builder,test_trajectory_copy,test_yield_trajectory: 83 passed.ruff format/ruff check(0.15.11) clean.prek/tynot run (no CUDA backend deps on macOS).Found and fixed with Claude Code. The runs above were executed locally.
🤖 Generated with Claude Code