Skip to content

feat(harness): add toolsearch for deferred tools - #114

Merged
sanchitrk merged 4 commits into
canaryfrom
claude/tool-search-66dad0
Oct 30, 2026
Merged

sanchitrk merged 4 commits into
canaryfrom
claude/tool-search-66dad0

Conversation

@sanchitrk

@sanchitrk sanchitrk commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Built-in toolsearch tool: the model finds tools it isn't offered yet (exposure deferred) and loads them for its next request, without the prompt changing every time one is registered.

  • toolsearch (packages/harness/src/plugin/builtin/tool/toolsearch.ts, ranker in src/tool/search.ts): BM25 over each tool's name, description and parameter text; limit defaults to 8; fixed description; exposure: "model". Ships optIn like grep/find/ls.
  • Loading writes agent: the ToolSettled projector writes one agent revision adding the matches to modelTools.add at the settle event's own seq (no second event).
  • Offered set per request: declared (direct/model) tools, plus modelTools.add (deferred only), minus modelTools.remove; agent is read once per request. Registry.resolve(selection) exposes wire (declared), offered and deferred; the system prompt's tool index lists declared tools only, so a load never moves the prefix.
  • Tool changes in the transcript: new durable event session.thread.tools.changed appends a system entry (toolsAdded, toolsRemoved, declaration hash) when the offered set differs from what the transcript replays to; context assembly turns system entries into aikit SystemMessages.
  • ToolUnavailable: a call to a registered tool outside the selection (removed, or deferred and not loaded) fails with a tagged error result; unknown names keep Unknown tool.

Tests

End-to-end, each ending in an __artifacts__/ snapshot (test/tools.toolsearch.test.ts): a search loads a tool and the next request offers it through a system entry; empty/bad query and no matches; the limit and the default; a fork cut before the load does not offer the tool while one cut after does; calling an unloaded or removed tool returns ToolUnavailable. Plus one live run with openai-codex / gpt-6-luna (test/tools.toolsearch.live.test.ts), passing.

tools.exposure.json changed on purpose: calling a registered deferred tool is now ToolUnavailable instead of Unknown tool.

sanchitrk and others added 4 commits October 30, 2026 11:52
The model searches the deferred tools it is not offered yet with BM25 over
each tool's name, description and parameter text, and the matches are
loaded for its next request. A load writes one `agent` revision in the
settling event; the offered set per request is the declared tools plus
`modelTools.add` minus `modelTools.remove`. When it differs from what the
transcript declares, a `system` entry carries `toolsAdded`/`toolsRemoved`
and context assembly turns it into an aikit system message, so the request
prefix stays cached. A call to a registered tool outside the selection
fails with `ToolUnavailable`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`ContextCodec.decodeMessage` returns the stored message of a `user`, `assistant`
or `system` entry, so a thread's history decodes whole once a tool was loaded.
`ContextCodec.parts` reads a message's parts; a system message has none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`unknown-tool` and `invalid-arguments`, like the other error codes; the
unknown-tool text reads `Unknown Tool`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sanchitrk
sanchitrk merged commit c93760d into canary Oct 30, 2026
2 checks passed
@sanchitrk
sanchitrk deleted the claude/tool-search-66dad0 branch October 30, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant