[drknowhow]

What broke

c3 showed up as failed to connect with no error anywhere. Claude Code's own log for the session was the whole story:

Successfully connected (transport: stdio) in 2830ms
Connection established with capabilities: ... "serverVersion":{"name":"C3 v2.125.0"}
Tool "c3_shell" description truncated ...
<nothing>

It connected, listed its tools, and died. Reproduced 3/3 by hand; under -X faulthandler the process exits 139 with:

Windows fatal exception: access violation

Thread 0x0000c5a0 [c3-embed-index] (most recent call first):
  chromadb/api/rust.py, line 421 in _get
  chromadb/api/models/Collection.py, line 159 in get
  services/embedding_index.py, line 475 in _remove_file_chunks
  services/embedding_index.py, line 368 in build
  cli/mcp_server.py, line 170 in _bg_embed

Root cause

The HNSW segment in .c3/embeddings/chromadb was corrupt. A damaged segment does not surface when the store is opened — chromadb hands back a client and a collection handle without complaint, then faults on the first real read. Driving that read directly gives the catchable form:

chromadb.errors.InternalError: Error executing plan: Error sending backfill request
to compactor: Failed to apply logs to the hnsw segment writer

But in the server that read happens on the background c3-embed-index thread, where it comes back as a native access violation instead, and takes the process with it. Nothing is logged, and the host can only report a dead server.

Second occurrence — .c3/embeddings/file_hashes_corrupt_20260717.json is the 2026-07-17 one, cleaned up by hand at the time. _init_backends had no corruption handling at all, so the store's failure mode was always "kill the server".

The fix

_open_chroma probes the store with count() on the init path, where the failure is still a catchable Python exception rather than a fault on a background thread. A failed probe is treated as corruption:

1. Close the client. Windows will not release its handle on chroma.sqlite3 otherwise — a rename fails with PermissionError while it is open, and succeeds after client.close() (verified; the global clear_system_cache() is deliberately not used, since other stores share the process).
2. Move the store aside to .c3/embeddings/quarantine_corrupt_<timestamp>/, damaged bytes preserved for post-mortem rather than deleted.
3. Take the hash file with it. It claims vectors the fresh store does not have, so leaving it behind would suppress the very rebuild the quarantine exists to force.
4. Reopen empty, and probe again. One retry, never a loop; a second failed probe propagates, and _init_backends degrades the index to unavailable instead of taking the server down.

The store is a rebuildable cache, so recovery costs one re-embed and no data.

Verification

• Against the real corrupt store captured on 2026-09-06 (318 MB, copied to a temp project): detected, quarantined, reopened, and the get() that used to segfault returns cleanly. Exit 0.
• Full MCP repro after clearing the store: initialize + tools/list + a second tools/list 45 s later, all answered, exit 0, empty stderr. Before: exit 139.
• 5 new tests in tests/test_embedding_index_corrupt_store.py covering healthy-store passthrough, quarantine-and-reopen, hash-file handling, retry bound, and degrade-not-raise.
• Full suite: 3960 passed, 7 skipped. Two pre-existing failures, both confirmed present at HEAD without this change: tests/test_ci_act.py::TestActIntegration::test_runs_a_linux_job_and_reports_a_clean_failure ('unparsed' != 'ruff') and tests/test_codex_mcp_transport.py::test_stdio_host_session (30 s handshake timeout — fails 2/2 at HEAD, 1/2 with this branch).

Note

The corrupt store on this box has already been quarantined by hand, so c3 connects again today. This change is what keeps the next corruption from presenting as a silently dead MCP server.

https://claude.ai/code/session_01TmRb4DvoqKCroerYKJ9poc

[yepgent]

Rebased onto main (was 3 days stale, CONFLICTING/DIRTY) during a post-merge sweep.

The only conflicts were version bookkeeping — main has moved to 2.129.0 since this branch was cut, so shipping it as 2.127.1 would have walked the version backwards. Resolved by retargeting the release rather than reverting main:

• pyproject.toml + cli/c3.py → 2.129.1
• the CHANGELOG entry moved from its historical [2.127.1] - 2026-09-07 slot to [2.129.1] - 2026-09-10 at the top; its text is unchanged.

No change to services/embedding_index.py or the new test. Verified locally on the rebased head: tests/test_embedding_index_corrupt_store.py 5 passed, tests/test_version_sync.py 1 passed.

New head 6b47ef9 (force-pushed with lease from 89d40e3). CI re-runs on the rebase.