Repository navigation
Bench: full-history no-retrieval control arm on LongMemEval-S #16
Description
A reviewer noted that Honcho's benchmark post (linked from our comparison table) reports Gemini 3 Pro alone — full history, no memory — at 92.0 on LongMemEval-S, i.e. 2.2 points below our 94.2 and plausibly within sampling error on 500 questions. Their reader is a version older and their judge was GPT-4o, so the numbers are not directly comparable.
Proposed control, credit to the reviewer: run a no-retrieval arm in our harness — same reader, same judge, full untruncated history instead of retrieval.
Interpretation:
- If it lands within noise of 94.2: on this benchmark retrieval saves tokens but adds no measurable accuracy; the 94.2 reflects the reader more than the memory.
- If it lands clearly below: we have a clean measurement of what retrieval adds, isolated from the reader.
Either way the result gets published on the dashboard alongside the existing arms, and the compaction result is unaffected (there the evidence has left the window; live window scores 0.0).
Related existing gap already on the dashboard: 'Net saving versus no memory — UNMEASURED.'
Activity
patricznr1 commentedon Oct 3, 2026
I'm the reviewer from the LinkedIn thread. Thanks for opening this so quickly, and for the credit.
One suggestion for the run. E9 already has a third point, on the dev half only: top 20 sessions scored 92.71 against 94.33 for top 10 (n = 247). That is four questions, so it is a hint and no more. But top 20 holds at least as much of the evidence as top 10, and it still scored lower. If top 20 is run on all 500 next to the new arm, the page shows three points on the same questions: 10, 20 and all sessions (about 48 per question). A steady drop would mean retrieval adds accuracy by removing distractors. A flat line would mean it saves tokens and nothing else.
Two separate accuracy figures cannot settle a gap of two points at n = 500. Both arms answer the same questions, so the paired sign-flip test you use in E3 fits here too.
For reference: this is the second of three control arms in rule 9 of MBRC, a reporting checklist for memory benchmarks that I published (no memory, full context, corrupted memory). E3 already has the first one at the retrieval level, the none arm. https://patric-zeller.de/MBRC, DOI 10.5281/zenodo.22878931. As on LinkedIn: I build a memory system myself, and no figure of mine appears in that document.
doronp commentedon Oct 3, 2026
Control arm implemented and run — see PR #17.
Result:
full_history(no retrieval, full untruncated history, same reader+judge as our 94.2 run) scores 94.60 vsrerank1294.20 — identical within sampling error (n=500). Per the pre-registered interpretation: retrieval saves ~95% of reader tokens (~125k → ~5k/question) with no measurable accuracy penalty. Published either way per this issue: docs/benchmarks/E16-full-history-control.md + Table 3 in E9 + the RESULTS.md dashboard gap is now closed with the empirical numbers.Independent verification: orchestrator reproduced the full table bit-for-bit from the warm response cache; dataset sha256 matches the pin.