Skip to content

Revert "Enable deepagent background dispatch; set bloom68 working limit to 36h" - #81

Merged
tadamcz merged 1 commit into
mainfrom
revert-74-deepagent-background-64
Oct 20, 2026
Merged

tadamcz merged 1 commit into
mainfrom
revert-74-deepagent-background-64

Conversation

@tadamcz

@tadamcz tadamcz commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Reverts #74: back to the deepagent default (background=False) and the 72h bloom68 working limit that goes with the $300 per-sample cap.

Why

working_limit caps working time: wall time minus time spent blocked on shared resources (waiting for a connection, rate-limit backoff). This keeps contention from penalizing the agent. When there is one agent loop, "blocked" is unambiguous. With background subagents, at any moment some of the sample's tasks are queued while others are generating or running tools. There's no principled way to collapse that into a single clock:

  • Waiting whenever any task is queued: this subtracts time when other subagents were doing real work. When the connection pool is full, some task is almost always queued, so working time drops to about 0.
  • Waiting only when all tasks are queued: partly-contended time counts in full. If 4 of 5 subagents are stuck in the queue, the sample is charged as if it ran at full speed, so contention still penalizes it.
  • Counting the fraction of tasks not queued: this is the closest to fair. But it depends on how many tasks the agent chooses to spawn (a sleeping subagent counts as working). It also still isn't the counterfactual we care about. Without queueing, subagents would have returned sooner and the agent would have made different decisions.

On a bloom68 epochs=4 test run with background dispatch, these rules disagree by up to about 3x. At least one request was queued 58% of the time, a request or tool was running 78% of the time, and on average only about 30% of a sample's tasks were making progress.

Inspect's current implementation also has bugs here. Working time can go negative. It is also computed one way while the run is going (the value that enforces the limit) and another way in the final results.

Separately, background subagents swallow arbitrary exceptions (UKGovernmentBEIS/inspect_ai#5955). For example, a lost sandbox fails the sample when it happens in the main agent or a foreground subagent, but is silently absorbed in a background one.

With background=False, the existing working-time definition is well defined again, and a run's results don't depend on how much contention there was.

This also restores test_react_agent_gets_no_subagents_kwarg, which #74 removed.

@tadamcz
tadamcz merged commit 700985f into main Oct 20, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant