Skip to content

fix(bench): run the network canary on the host PATH - #605

Merged
e54-bot merged 1 commit into
mainfrom
fix/590-canary-host-path
Oct 9, 2026
Merged

e54-bot merged 1 commit into
mainfrom
fix/590-canary-host-path

Conversation

@e54-bot

@e54-bot e54-bot commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Under network off, --canary-cmd ran with the agent's PATH, where every network tool resolves to a blocker shim. A canary calling a blocked tool (for example curl -sf <url>) exited non-zero whether or not the network was reachable, so the run recorded fetch-blocked+canary-checked without any real network check.

The canary now runs with BENCH_HOST_PATH (the unscrubbed host PATH the harness already exposes), so it fails only when the network is actually unreachable, and a reachable network marks the run invalid as documented. The marker, --canary-cmd help, and both benchmark docs now describe this accurately.

Fixes #590

Test plan

  • test_network_canary_does_not_resolve_a_blocker: a canary that inspects command -v curl no longer resolves the blocker dir (fails on the old implementation, passes now)
  • test_network_canary_calls_a_real_tool_on_the_host_path: a planted curl on the host PATH that succeeds marks the run invalid (network reachable); one that fails records fetch-blocked+canary-checked
  • python3 -m unittest test_agent_bench — new tests pass; the only failures (test_scenarios_are_solvable_and_not_vacuous, test_matrix_skips_...) reproduce identically on unmodified main with the local debug binary (suite calibration vs binary version), unrelated to this change

Under network 'off' the canary ran with the agent's PATH, where every network tool resolves to a blocker, so a canary calling one could never report a reachable network yet the run still recorded fetch-blocked+canary-checked. The canary now runs with BENCH_HOST_PATH, so it fails only when the network is actually unreachable.

Fixes #590

@e54-bot e54-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — verified against issue #590's acceptance criteria.

  • The canary now runs with PATH=BENCH_HOST_PATH while keeping the rest of the scrubbed env, so it resolves real network tools instead of the blocker shims; BENCH_HOST_PATH is always populated by build_env (agent_bench.py:259) and the only caller passes that env (agent_bench.py:488). Marker recording at agent_bench.py:510 and the docs (agent-benchmark.md, agent-benchmark-howto.md) now match the behavior.
  • Both new tests fail under the previous implementation and are deterministic across hosts (the command -v case falls through to exit 0 whether or not the host has curl; the end-to-end test uses a local fake curl, no real network).
  • The two local suite failures (test_scenarios_are_solvable_and_not_vacuous, test_matrix_...) reproduce identically on unmodified main and are unrelated.

Note: submitted as a comment because e54-bot authored this PR and GitHub does not allow self-approval; the verdict itself is an approve.

@e54-bot
e54-bot merged commit 16b0931 into main Oct 9, 2026
17 checks passed
@e54-bot
e54-bot deleted the fix/590-canary-host-path branch October 9, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

The benchmark network canary cannot fail once the blockers are installed

2 participants