Repository navigation
feat(compress): C0 — every file map measured + c3 map-eval harness (2.120.0) - #160
Merged
Merged
Conversation
…n compress/read rows, map_by_backend + map->read chain Claude-Session: https://claude.ai/code/session_01VPxmnq6oZegRJLcYMqTaFw
…ap (C0) Sixteen fixtures under tests/map_eval/fixtures (Python incl. malformed, TS, JS, Go, Rust, Markdown, YAML, JSON, CSS, HTML, R regex path, Java generic path) plus two seeded generators (20 KB single-line bundle, 250 KB / 400-function module), each annotated by hand in fixture_suite.jsonl with kind, qualified name, params, ret and cat -n verified ranges. services/bench/map_eval.py renders each through the callable the MCP tool uses (FileMemoryStore.get_or_build_map today, services.file_map.render_map when C1 ships it), parses both the canonical and the legacy grammar, and grades recall, precision, signature completeness, range accuracy, determinism, tokens, chrome share. Gates must_pass / xfail(phase) / info with floors, ceilings and the campaign's per-case token gate; `c3 map-eval` wired beside shell-eval; tests/test_map_eval.py is the CI gate; docs/map-eval.md. Baseline against the legacy renderer: 4 must_pass, 10 xfail(C1), 1 xfail(C4), 1 info; recall mean 0.72, signature completeness 0.20, chrome share 0.14. Findings for C1 and C4 are listed in the doc.
…ap-eval harness Claude-Session: https://claude.ai/code/session_01VPxmnq6oZegRJLcYMqTaFw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase C0 of the c3_compress remediation (campaign c3-compress-remediation, plan reviewed with Cod).
Measured on this repository's telemetry: c3_compress was 132 of 3,932 tool calls, 130 of them mode=map, and no row said which backend built the map, which parser extracted the symbols, whether the record was fresh, or whether the map led to a targeted read. This PR adds the instrument before the renderer changes in 2.121.0.
parser(tree_sitter | regex | generic); pre-2.120.0 records re-extract on next update.detail(requested mode, backend, parser, cache_hit, sections, symbols, fallback reason, ranges/lines served).aggregate_tool_telemetrygainsmap_by_backendandmap_read_chain(maps followed by a read of the same file within 5 calls; reads preceded by a map).c3 map-eval: hand-annotated gold harness for the file map (16 fixtures across 12 languages plus malformed, minified and 250 KB generated files), scoring tokens, symbol recall/precision, signature completeness, range accuracy and determinism. Baseline committed against the current renderer: signature completeness 0.20, recall 0.72, 10 cases xfail C1, 1 xfail C4.Tests: 3747 passed locally (tests/test_map_telemetry_c0.py, tests/test_map_eval.py new). ruff clean.
https://claude.ai/code/session_01VPxmnq6oZegRJLcYMqTaFw