Skip to content

feat(harness): rank toolsearch by name, coverage and namespace - #132

Merged
sanchitrk merged 2 commits into
canaryfrom
claude/cod-203-toolsearch-a22b2e
Nov 2, 2026
Merged

sanchitrk merged 2 commits into
canaryfrom
claude/cod-203-toolsearch-a22b2e

Conversation

@sanchitrk

Copy link
Copy Markdown
Contributor

Summary

  • toolsearch scores each tool's name, namespace text and description as separate fields. Name terms count 3× their IDF, namespace terms 0.5×, and BM25 runs on the description only.
  • Order: an exact name or __suffix match comes first (thread_read finds mcp__app__thread_read; dev-radius matches dev_radius). Then come tools that cover more query terms, then score. A tool whose last name segment is in the query gets a bonus.
  • Terms are NFC-normalized, and non-ASCII text is split with Intl.Segmenter, so CJK queries match CJK descriptions.
  • Deferred MCP tools now carry their server's name, description and instructions as namespace text (DeferredTool.namespace on ToolOffer.deferred). This text never reaches the model wire.
  • The tool name, its parameters and its output are unchanged.
  • Separately, pnpm-workspace.yaml permissions is reformatted so vp check passes.

Tests

  • New E2E toolsearch ranking: 15 queries over a fixed catalog of built-in-style and fake MCP tools, recorded in __artifacts__/toolsearch-ranking.json.
  • New E2E in mcp.thread: a server's configured description and its instructions decide which server's echo is loaded (mcp.toolsearch.namespace.json).
  • Regenerated artifacts:
    • tools.toolsearch.limit.json: same 8 tools, in a new order for the 13-word query.
    • mcp.section.json: the atlas server's instructions mention "echo", so echo now loads all 8 atlas tools.
  • Live: tools.toolsearch.live against Anthropic passes, and its artifact is unchanged. The openai-codex cases were skipped because the login token had expired.
  • vp check and vp run typecheck pass.

🤖 Generated with Claude Code

sanchitrk and others added 2 commits November 2, 2026 15:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Score a tool's name, its MCP server's namespace text and its description
as separate fields: name terms count three times their IDF, namespace
terms half, and BM25 runs on the description alone. Tools covering more
query terms rank first, an exact name or `__` suffix ranks before all,
and a tool whose last name segment is in the query gets a bonus.

Terms are NFC-normalized and non-ASCII text is split with
Intl.Segmenter, so CJK queries match CJK descriptions.

Deferred MCP tools carry their server's name, description and
instructions as namespace text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sanchitrk
sanchitrk merged commit aba2d65 into canary Nov 2, 2026
2 checks passed
@sanchitrk
sanchitrk deleted the claude/cod-203-toolsearch-a22b2e branch November 2, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant