Repository navigation
Revert "Enable deepagent background dispatch; set bloom68 working limit to 36h" - #81
Merged
Merged
Conversation
…it to 36…" This reverts commit 00ddabd.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #74: back to the deepagent default (
background=False) and the 72h bloom68 working limit that goes with the $300 per-sample cap.Why
working_limitcaps working time: wall time minus time spent blocked on shared resources (waiting for a connection, rate-limit backoff). This keeps contention from penalizing the agent. When there is one agent loop, "blocked" is unambiguous. With background subagents, at any moment some of the sample's tasks are queued while others are generating or running tools. There's no principled way to collapse that into a single clock:On a bloom68
epochs=4test run with background dispatch, these rules disagree by up to about 3x. At least one request was queued 58% of the time, a request or tool was running 78% of the time, and on average only about 30% of a sample's tasks were making progress.Inspect's current implementation also has bugs here. Working time can go negative. It is also computed one way while the run is going (the value that enforces the limit) and another way in the final results.
Separately, background subagents swallow arbitrary exceptions (UKGovernmentBEIS/inspect_ai#5955). For example, a lost sandbox fails the sample when it happens in the main agent or a foreground subagent, but is silently absorbed in a background one.
With
background=False, the existing working-time definition is well defined again, and a run's results don't depend on how much contention there was.This also restores
test_react_agent_gets_no_subagents_kwarg, which #74 removed.