Repository navigation
RFC: add member indexing role for one-writer shared snapshots - #275
hyein-cbio wants to merge 3 commits into
Conversation
|
Added an embedding compatibility check in bd4be7d. The previous read-only search path could accept a query embedding from a different model if its dimensions matched the stored vectors. This was reproduced with synthetic vectors against a real SQLite/vec0 table; it is not evidence of a production incident. The follow-up adds:
Existing unlabelled indices remain usable by leaders. Member searches reject them and require a leader-local rebuild and republication. An incremental update alone does not establish the provenance of existing vectors. No automatic reset or rebuild is performed, and members remain read-only. Validation: 87 focused tests and 3 existing member/readonly daemon tests passed, along with Ruff and mypy. An expanded run passed 293 tests and failed 5: four could not create Unix sockets in the execution environment, and one required the uninstalled optional The [design note](https://github.com/hyein-cbio/cocoindex-code/blob/bd4be7d26c05d086a359c2799008fa52673ce62b/docs/pr275-embedding-compatibility-rfc.md) describes the metadata format and migration policy. Feedback would be helpful on whether rejecting legacy snapshots is appropriate, and whether query-only parameter changes should allow explicit recertification rather than a rebuild. Model revision and endpoint identity are not currently included in the contract. |
Allow a per-process COCOINDEX_CODE_INDEXING_ROLE=member runtime to search a published target_sqlite.db without starting or refreshing indexes. Default remains leader so existing users are unchanged. Member MCP sessions skip background indexing, ignore refresh_index, and open the mapped snapshot read-only. Explicit ccc index, search --refresh, and reset are rejected in that process. A matching DB path mapping is required so members cannot fall back to a local index. This is an RFC for a one-writer team snapshot workflow. Incremental state stays on the leader; only a completed SQLite snapshot is published.
bd4be7d to
b607638
Compare
Summary
This is an RFC / draft for an experimental one-writer team snapshot workflow.
Some teams do not want every developer machine to incrementally re-index. The intended model is:
target_sqlite.dbThe new process-local setting is:
COCOINDEX_CODE_INDEXING_ROLE=member # default: leaderleaderremains the default, so existing users are unchanged.Why this is an RFC
CocoIndex does not recommend sharing live LMDB/SQLite databases over a mounted filesystem. This PR does not do that. Instead it keeps incremental state (
cocoindex.db) on the leader and publishes only a completed SQLite snapshot.Still, this is a new usage model:
I would like maintainer feedback on whether this direction is acceptable before treating it as merge-ready.
Behavior
When
COCOINDEX_CODE_INDEXING_ROLE=member:searchnever refreshes or triggers load-time indexing, even ifrefresh_index=trueccc index,ccc search --refresh, andccc resetare rejected in that processCOCOINDEX_CODE_DB_PATH_MAPPINGis required; local index fallback is refusedThe role is process-local. Setting it in
.mcp.jsonaffects the MCP process only. The bundledSessionStarthook skipsccc indexwhen the parent process already hasCOCOINDEX_CODE_INDEXING_ROLE=member.Example member MCP config
{ "mcpServers": { "cocoindex-code": { "command": "ccc", "args": ["mcp"], "env": { "COCOINDEX_CODE_INDEXING_ROLE": "member", "COCOINDEX_CODE_DB_PATH_MAPPING": "/Users/bob/src/myrepo=/Volumes/team/ccc-indexes/myrepo" } } } }Leader workflow: index locally, then
sqlite3.Connection.backup()the completedtarget_sqlite.dband atomicallyos.replace()it onto the share. Do not publishcocoindex.db.Non-goals
.cocoindex_code/settings.ymlTest plan
prek run --all-filesindexccc index/search --refresh/resetare rejectedFeedback especially welcome on:
COCOINDEX_CODE_INDEXING_ROLEis the right knobccccommand