Repository navigation
fix: parse Claude Code transcripts over 50 MB instead of skipping them - #370
Mark Warren (ZeSnarf) wants to merge 2 commits into
Conversation
readFileSafe returns null for a file over MAX_FILE_SIZE, so the longest Claude Code sessions, usually the most important ones, were left out whole. Most of their bytes are raw api-request records the parser never reads. Transcripts are now streamed line by line, those records are left out, and parsing stops with a warning only once the kept lines pass MAX_FILE_SIZE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mangesh Raut (mangeshraut712)
left a comment
There was a problem hiding this comment.
CLAUDE_API_REQUEST_LINE = /^\s*\{\s*"type"\s*:\s*"api-request/; strictly anchors to the start of the line, requiring "type" to be the very first key immediately following {.
If a transcript's serializer places any other property first (e.g. {"timestamp": ..., "type": "api-request", ...} or {"uuid": ..., "type": "api-request", ...}), CLAUDE_API_REQUEST_LINE.test(rawLine) returns false.
Because the size accumulation happens prior to JSON parsing:
keptBytes += rawLine.length;
if (keptBytes > MAX_FILE_SIZE) {
truncated = true;
return;
}any large API record with reordered keys will fail the regex filter, have its full multi-megabyte payload added to keptBytes, and prematurely trigger truncated = true (aborting further streaming via shouldStop).
Consider removing the opening brace anchor (e.g. /"type"\s*:\s*"api-request/ or checking rawLine.includes('"api-request"')) so API request records are bypassed before size accounting regardless of key order.
Matches "type": "api-request" anywhere in the raw line instead of only as the first key, so a serializer that writes another key first no longer counts the record towards the size limit. Quotes inside JSON strings are escaped, so a prompt that mentions the text is still parsed (covered by a new test).
|
Thanks Mangesh Raut (@mangeshraut712), good spot. I've changed it to look for "type": "api-request" anywhere in the line, so the key order doesn't matter anymore. Added a test for that too, plus one that makes sure a prompt that just mentions it still gets parsed, since quotes inside strings are escaped and can't match. |
Description
parseClaudeSessionFilereads a transcript withreadFileSafe, which returnsnullfor any file overMAX_FILE_SIZE(50 MB). Those sessions were left out completely, and they tend to be the longest and most important ones. Most of their size is rawapi-requestrecords, which the parser never reads.This PR:
forEachJsonlLineinstead of reading it whole;api-requestlines before parsing them as JSON;MAX_FILE_SIZEto the lines it keeps. Past that limit it stops with a warning and parses what it has, instead of dropping the session;CACHE_VERSION, so sessions that were skipped before are picked up.We found this by running the extension on real Claude Code logs: the two biggest sessions, over 50 MB each, were missing from every view.
Related Issues
Fixes #364.
None filed.
Checklist
npm run checkpasses (typecheck + lint + spellcheck + knip + tests)parser-claude.test.ts: a transcript over the limit, made of API request records, still parses its turn)