Skip to content

fix: cancel sibling rollouts when a gather fails - #1145

Open
salma2vec wants to merge 1 commit into
OpenPipe:mainfrom
salma2vec:fix/gather-cancels-rollouts-on-failure
Open

salma2vec wants to merge 1 commit into
OpenPipe:mainfrom
salma2vec:fix/gather-cancels-rollouts-on-failure

Conversation

@salma2vec

Copy link
Copy Markdown

Summary

Closes #1144

gather_trajectory_groups and gather_trajectories used asyncio.gather, which re-raises the first exception but leaves the other awaitables running. When a rollout failed past max_exceptions, the caller got the error while the sibling rollouts kept hitting inference, environments and after_each judges with no owner. #1140 / #1141 cover the same gap in trajectory_group_batches.

Changes

  • src/art/gather.py: a small _gather_or_cancel helper wraps the rollouts as tasks and awaits asyncio.gather. On any exception it cancels the remaining tasks, awaits them, then re-raises the original exception. The type is unchanged: TaskGroup would wrap it in an ExceptionGroup. Both gather functions use it.
  • Tasks are still created inside set_gather_context, as asyncio.gather did, so they see the same context. Results, ordering, the tolerated-exception path and cancellation of the awaiting task are unchanged.
  • A cancelled sibling goes through wrap_group_awaitable's existing BaseException handling, so it adds to the failed gather's exceptions tally in the progress bar. That context is discarded when the gather raises.
  • tests/unit/test_gather_cleanup.py: an over-budget failure cancels the sibling in each function, and a control checks that a tolerated failure still returns every other group.

Test plan

  • The two cancellation tests fail on main (5f0c1fb) and pass here. The tolerated-failure control passes on both.
  • The gather_trajectory_groups / gather_trajectories leave sibling rollouts running after a failure #1144 repro prints ['finished after gather raised'] on main and [] here.
  • With test_batches_completion_bursts, test_ruler_metrics, test_auto_trajectory, test_metrics_builder, test_trajectory_copy, test_yield_trajectory: 83 passed.
  • ruff format / ruff check (0.15.11) clean.
  • Full prek / ty not run (no CUDA backend deps on macOS).

Found and fixed with Claude Code. The runs above were executed locally.

🤖 Generated with Claude Code

gather_trajectory_groups and gather_trajectories awaited asyncio.gather,
which re-raises the first over-budget failure without cancelling the
other rollouts. They kept running against inference, environments and
after_each judges with no owner. Cancel and await them before
re-raising.

Closes OpenPipe#1144

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gather_trajectory_groups / gather_trajectories leave sibling rollouts running after a failure

1 participant