Skip to content

fix: parse Claude Code transcripts over 50 MB instead of skipping them - #370

Open
Mark Warren (ZeSnarf) wants to merge 2 commits into
microsoft:mainfrom
PharSideLabs:claude/upstream-claude-large-transcripts
Open

Mark Warren (ZeSnarf) wants to merge 2 commits into
microsoft:mainfrom
PharSideLabs:claude/upstream-claude-large-transcripts

Conversation

@ZeSnarf

@ZeSnarf Mark Warren (ZeSnarf) commented Nov 3, 2026 •

Copy link
Copy Markdown

Description

parseClaudeSessionFile reads a transcript with readFileSafe, which returns null for any file over MAX_FILE_SIZE (50 MB). Those sessions were left out completely, and they tend to be the longest and most important ones. Most of their size is raw api-request records, which the parser never reads.

This PR:

  • streams the transcript with forEachJsonlLine instead of reading it whole;
  • skips api-request lines before parsing them as JSON;
  • applies MAX_FILE_SIZE to the lines it keeps. Past that limit it stops with a warning and parses what it has, instead of dropping the session;
  • bumps CACHE_VERSION, so sessions that were skipped before are picked up.

We found this by running the extension on real Claude Code logs: the two biggest sessions, over 50 MB each, were missing from every view.

Related Issues

Fixes #364.
None filed.

Checklist

  • npm run check passes (typecheck + lint + spellcheck + knip + tests)
  • Changes are covered by tests (parser-claude.test.ts: a transcript over the limit, made of API request records, still parses its turn)
  • Documentation updated (if applicable): not needed

readFileSafe returns null for a file over MAX_FILE_SIZE, so the longest
Claude Code sessions, usually the most important ones, were left out
whole. Most of their bytes are raw api-request records the parser never
reads. Transcripts are now streamed line by line, those records are left
out, and parsing stops with a warning only once the kept lines pass
MAX_FILE_SIZE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLAUDE_API_REQUEST_LINE = /^\s*\{\s*"type"\s*:\s*"api-request/; strictly anchors to the start of the line, requiring "type" to be the very first key immediately following {.

If a transcript's serializer places any other property first (e.g. {"timestamp": ..., "type": "api-request", ...} or {"uuid": ..., "type": "api-request", ...}), CLAUDE_API_REQUEST_LINE.test(rawLine) returns false.

Because the size accumulation happens prior to JSON parsing:

keptBytes += rawLine.length;
if (keptBytes > MAX_FILE_SIZE) {
  truncated = true;
  return;
}

any large API record with reordered keys will fail the regex filter, have its full multi-megabyte payload added to keptBytes, and prematurely trigger truncated = true (aborting further streaming via shouldStop).

Consider removing the opening brace anchor (e.g. /"type"\s*:\s*"api-request/ or checking rawLine.includes('"api-request"')) so API request records are bypassed before size accounting regardless of key order.

Matches "type": "api-request" anywhere in the raw line instead of only as
the first key, so a serializer that writes another key first no longer counts
the record towards the size limit. Quotes inside JSON strings are escaped, so
a prompt that mentions the text is still parsed (covered by a new test).
@ZeSnarf

Mark Warren (ZeSnarf) commented Nov 3, 2026 •

Copy link
Copy Markdown
Author

Thanks Mangesh Raut (@mangeshraut712), good spot. I've changed it to look for "type": "api-request" anywhere in the line, so the key order doesn't matter anymore. Added a test for that too, plus one that makes sure a prompt that just mentions it still gets parsed, since quotes inside strings are escaped and can't match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claude transcripts over 50 MB are dropped entirely (31% of my prompts)

2 participants