Repository navigation
fix(bench): run the network canary on the host PATH - #605
Merged
Merged
Conversation
Under network 'off' the canary ran with the agent's PATH, where every network tool resolves to a blocker, so a canary calling one could never report a reachable network yet the run still recorded fetch-blocked+canary-checked. The canary now runs with BENCH_HOST_PATH, so it fails only when the network is actually unreachable. Fixes #590
e54-bot
commented
Oct 9, 2026
e54-bot
left a comment
Collaborator
Author
There was a problem hiding this comment.
LGTM — verified against issue #590's acceptance criteria.
- The canary now runs with PATH=BENCH_HOST_PATH while keeping the rest of the scrubbed env, so it resolves real network tools instead of the blocker shims; BENCH_HOST_PATH is always populated by build_env (agent_bench.py:259) and the only caller passes that env (agent_bench.py:488). Marker recording at agent_bench.py:510 and the docs (agent-benchmark.md, agent-benchmark-howto.md) now match the behavior.
- Both new tests fail under the previous implementation and are deterministic across hosts (the command -v case falls through to exit 0 whether or not the host has curl; the end-to-end test uses a local fake curl, no real network).
- The two local suite failures (test_scenarios_are_solvable_and_not_vacuous, test_matrix_...) reproduce identically on unmodified main and are unrelated.
Note: submitted as a comment because e54-bot authored this PR and GitHub does not allow self-approval; the verdict itself is an approve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under network
off,--canary-cmdran with the agent'sPATH, where every network tool resolves to a blocker shim. A canary calling a blocked tool (for examplecurl -sf <url>) exited non-zero whether or not the network was reachable, so the run recordedfetch-blocked+canary-checkedwithout any real network check.The canary now runs with
BENCH_HOST_PATH(the unscrubbed hostPATHthe harness already exposes), so it fails only when the network is actually unreachable, and a reachable network marks the runinvalidas documented. The marker,--canary-cmdhelp, and both benchmark docs now describe this accurately.Fixes #590
Test plan
test_network_canary_does_not_resolve_a_blocker: a canary that inspectscommand -v curlno longer resolves the blocker dir (fails on the old implementation, passes now)test_network_canary_calls_a_real_tool_on_the_host_path: a plantedcurlon the hostPATHthat succeeds marks the runinvalid(network reachable); one that fails recordsfetch-blocked+canary-checkedpython3 -m unittest test_agent_bench— new tests pass; the only failures (test_scenarios_are_solvable_and_not_vacuous,test_matrix_skips_...) reproduce identically on unmodifiedmainwith the local debug binary (suite calibration vs binary version), unrelated to this change