Repository navigation
perf(storage): keep no raw copy of imported events that projected - #66
Merged
Merged
Conversation
…d write
Watcher (app/watch.py): with passive mode on, a watchdog observer (FSEvents
on macOS, inotify on Linux) watches the Claude, Cursor and Codex transcript
folders. A burst of .jsonl writes is debounced (1.5 s) and the changed,
enabled agents are imported in-process with live sessions included, so a
running session shows up in about two seconds instead of after the schedule
and the 10-minute hold-back. In-process passes now close only imported
sessions quiet for 10 minutes (complete-imported takes quiet_seconds; an
older bridge's {} still closes all). The schedule keeps running for analysis,
OpenCode/Hermes, and anything a dropped event missed. Without watchdog, or
with COT_PASSIVE_WATCH=0, nothing is watched.
Large files: new bytes are read 1 MB at a time with parser state carried
across chunks, so the bridge peaks at ~140 MB instead of ~720 MB, and
transcripts over 50 MB are no longer skipped. A line longer than a chunk is
read whole up to 64 MB; a longer one is skipped with a warning. Lines split
on "\n" only, offsets are cut on bytes (invalid UTF-8 no longer shifts them).
Fixes Cursor appends being dropped: import line numbers restarted at 0 on
every pass, so an appended line got the dedup key of the file's first line.
Line numbers are now file-wide (kept in the identity sidecar, counted once
for older entries); keys of already-imported lines are unchanged.
Overlap: the import poster sends batches from a background thread while the
next batch parses (SQLite and HTTP release the GIL).
0xCardinal
force-pushed
the
perf/watch-stream
branch
from
October 22, 2026 00:58
302f66b to
302a101
Compare
The raw ingest ledger stored every body twice over: raw_payload (cut at 64 KB) next to events.payload (uncapped). For an imported event that projected, the event row and the transcript on disk both hold the body, so the ledger row now keeps its metadata, hash and status but no payload. Hook bodies keep theirs (cleanup migrations can delete hook events while their ledger rows stay), as do failures, ignored rows and duplicates. Existing DBs are pruned once, in batches of 2,000 per write, at the end of the background startup work. Freed pages are reused by later writes; shrinking the file needs a VACUUM, left to the user. A fresh import of 611 MB of Claude history: 437 MB -> 320 MB, same speed.
0xCardinal
force-pushed
the
perf/storage
branch
from
October 22, 2026 00:59
a00c41c to
d4f2287
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 7 of the ingest performance plan. Stacked on #65 (which is stacked on #64).
Where the bytes go
Fresh import of a copy of real Claude history (611 MB), by
dbstat:events222 MB (payload104 MB,detail70 MB),raw_ingest_events138 MB (raw_payload103 MB), FTS 32 MB, indexes ~33 MB.raw_payloadis the same body asevents.payload, except capped at 64 KB, and nothing reads it back except the raw-ingest listing.What changed
raw_payload. The event row (uncapped) and the transcript on disk both still hold the body.prune_import_raw_payloads()runs once at the end of the background startup work, 2,000 rows per write so ingest gets in between, and records completion in settings. Freed pages are reused by later writes; shrinking the file on disk needs aVACUUM, which this PR doesn't run.Numbers
cot.db(read-only count): projected import rows hold 198 MB of raw payload, about 6%. Projected and duplicate hook rows hold another ~366 MB; left alone here for the reason above.Not done
A contentless FTS index: FTS is 32 MB of 320 MB and search reads from it, so not worth the risk here.
Tests
New tests in
test_raw_ingest_drift.py: which rows keep a raw copy, and the one-time batched prune. 628 passed.🤖 Generated with Claude Code