Skip to content

Pin HelmsDeep v0.5.0; use Shepherd infores names in the sweep examples - #65

Merged
maximusunc merged 2 commits into
mainfrom
claude/vibrant-pascal-ar89gt
Nov 1, 2026
Merged

maximusunc merged 2 commits into
mainfrom
claude/vibrant-pascal-ar89gt

Conversation

@maximusunc

@maximusunc maximusunc commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

What

  1. Bumps the HelmsDeep pin in requirements-runners.txt from v0.4.0 to v0.5.0.
  2. Renames the sweep example targets from aragorn / arax / bte to shepherd-aragorn / shepherd-arax / shepherd-bte, and documents why the name matters.

Why

HelmsDeep v0.5.0

v0.5.0 stops counting an ARA's refusal of a query type it doesn't support as a load failure. BTE answers every Pathfinder query with a 500 saying "BTE does not support Pathfinder type queries.", so the mixed (aras_mixed) run against BTE failed every checkpoint on Pathfinder errors.

On the sync path, HelmsDeep now treats such a refusal as unsupported instead of failed. That covers a 501, or a "does not support" / "not supported" / "not implemented" message in the response's status, description or error-level logs. An unsupported answer:

  • adds no error and no latency sample
  • isn't sent again for the rest of the run, so that class's users sit idle
  • isn't judged in that class's checkpoint
  • is reported in the run's summary output and summary.json

The ARS path is unaffected. A related Shepherd change, BioPack-team/shepherd#200, makes BTE return 501 for Pathfinder. HelmsDeep handles either form, so this pin doesn't depend on it.

Sweep target names

The sweep passes each target's NAME to --target. With the default --trapi_version auto, the version is looked up in SmartAPI as infores:<NAME>. The Shepherd-hosted ARAs are registered as infores:shepherd-aragorn, infores:shepherd-arax and infores:shepherd-bte. A sweep using the bare names read the wrong entries, or none: when nothing is found it falls back to 2.0.0 with only a log warning. As a result, Aragorn and BTE were sent TRAPI 2.0 while SmartAPI lists them at 1.6.0.

Updated:

  • README.md
  • the test_harness/sweep.py docstring and --help
  • deploy/cronjob.example.yaml
  • deploy/helm/test-harness/values.example.yaml
  • the chart's sweep.targets error message

The README and the --target / NAME help now say the name must be the infores the service is registered under in SmartAPI. They also say what else the name controls: each target's output directory and filename prefix, and the agent the Information Radiator reports under.

Testing

🤖 Generated with Claude Code

https://claude.ai/code/session_01S2yuRAujys2XXK6YPjSay3

v0.5.0 stops scoring an ARA's refusal of a query class it doesn't support
(e.g. BTE on Pathfinder, a 500 saying "does not support", or a 501) as a
load failure in the sync ARA runs, so the mixed profile against BTE no
longer fails every checkpoint on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAujys2XXK6YPjSay3
The sweep passes each target's NAME to --target, and --trapi_version auto
looks the TRAPI version up in SmartAPI by infores:<NAME>. The examples used
aragorn/arax/bte, but the Shepherd-hosted ARAs are registered as
shepherd-aragorn/shepherd-arax/shepherd-bte, so a sweep copied from them
read the wrong entries (or none, falling back to 2.0.0 with only a warning)
and sent Aragorn and BTE TRAPI 2.0 while SmartAPI lists them at 1.6.0.

Updates the README, the sweep docstring and --help, the CronJob and Helm
examples, and the chart's error message, and says in the README and the
--target / NAME help that the name must be the SmartAPI-registered infores
(it also labels the results and the Radiator agent).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRAujys2XXK6YPjSay3
@maximusunc maximusunc changed the title Pin HelmsDeep v0.5.0 Pin HelmsDeep v0.5.0; use Shepherd infores names in the sweep examples Nov 1, 2026
@maximusunc
maximusunc marked this pull request as ready for review November 1, 2026 14:17
@maximusunc
maximusunc merged commit 2caa427 into main Nov 1, 2026
4 checks passed
@maximusunc
maximusunc deleted the claude/vibrant-pascal-ar89gt branch November 1, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants