Repository navigation
feat(harness): rank toolsearch by name, coverage and namespace - #132
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Score a tool's name, its MCP server's namespace text and its description as separate fields: name terms count three times their IDF, namespace terms half, and BM25 runs on the description alone. Tools covering more query terms rank first, an exact name or `__` suffix ranks before all, and a tool whose last name segment is in the query gets a bonus. Terms are NFC-normalized and non-ASCII text is split with Intl.Segmenter, so CJK queries match CJK descriptions. Deferred MCP tools carry their server's name, description and instructions as namespace text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
toolsearchscores each tool's name, namespace text and description as separate fields. Name terms count 3× their IDF, namespace terms 0.5×, and BM25 runs on the description only.__suffixmatch comes first (thread_readfindsmcp__app__thread_read;dev-radiusmatchesdev_radius). Then come tools that cover more query terms, then score. A tool whose last name segment is in the query gets a bonus.Intl.Segmenter, so CJK queries match CJK descriptions.DeferredTool.namespaceonToolOffer.deferred). This text never reaches the model wire.pnpm-workspace.yamlpermissionsis reformatted sovp checkpasses.Tests
toolsearch ranking: 15 queries over a fixed catalog of built-in-style and fake MCP tools, recorded in__artifacts__/toolsearch-ranking.json.mcp.thread: a server's configured description and its instructions decide which server'sechois loaded (mcp.toolsearch.namespace.json).tools.toolsearch.limit.json: same 8 tools, in a new order for the 13-word query.mcp.section.json: the atlas server's instructions mention "echo", soechonow loads all 8 atlas tools.tools.toolsearch.liveagainst Anthropic passes, and its artifact is unchanged. The openai-codex cases were skipped because the login token had expired.vp checkandvp run typecheckpass.🤖 Generated with Claude Code