Repository navigation
Type checks, MemoryStore hooks, reads beside writes - #30
Conversation
Documentation build overview
14 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
tobixen
left a comment
There was a problem hiding this comment.
Disclaimer: AI-generated review below the line, generated by Claude Opus 5.5, from the prompt:
Do a review on PR #30 and post a comment
Nice work, and the benchmark numbers are convincing. All 680 unit tests pass locally on 1081bff. A few things worth a look before merging, roughly most important first:
-
An in-process read can now make a big listing fail. Before, a read in another thread waited for the write. Now a read that's open when the writer reaches
COMMITmakes the writer wait up totimeout, and after that the whole listing is rolled back withStoreBusyError.test_a_read_held_open_past_the_timeout_makes_a_write_busypins down exactly this. Reads in this process already take turns on_read_turn, so_commit(src/calendaring_sync/store/sqlite.py:447) could take that lock, withtimeout, beforeCOMMIT. That way only other processes can make a commit busy, as before. -
cache_spill = OFFis also set in WAL mode (sqlite.py:381). In WAL, spilled pages go to the WAL file and don't lock readers out, so the extra memory buys nothing there. I checked a WAL file: the store still setscache_spillto 0. Suggest checkingPRAGMA journal_modeand turning spilling off only in rollback-journal mode. Even then the cost has no upper bound for scopes with large content such as attachments. That deserves a sentence in the news entry, not just the docstring. -
Exact
str/intrefusesStrEnum/IntEnum(_checks.py:27).replica.session(Scope.WORK, ...)with aStrEnumnow raisesscope_id must be str, not Scope. #27 argues for this, but an enum is a common way to name scope ids. Either acceptStrEnumand storestr(value), or say in the news entry that subclasses are refused. -
_read_turn.acquire()has no timeout (sqlite.py:443). A read queued behind another long read in this process waits forever, while the docs say a store waitstimeoutseconds for "another process or thread". -
The
MemoryStoredoctest contradicts the paragraph above it (memory.py:86). The doctest's_loadunpickles on every transaction. The text says a file-backed store reads once when it opens, leaves_loadalone and callssuper()._save()._PickledStorein the tests follows the advice, so it would make the better example. -
Minor:
check_argument_typesandtest_every_entry_point_checks_its_typesonly look atinspect.isfunction. A publicclassmethod,staticmethodor property setter added later would go unchecked, and the test wouldn't notice. None exist today. The test could at least fail on public attributes it skips.
… WAL mode Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
7bbf288 to
866376a
Compare
|
Fixed 1, 2, 5 and 6. |
Closes #24, closes #27, closes #28.
Argument types are checked from the annotations now, so a wrong type fails the same way on both stores. Earlier
delete_record("1")deleted record 1 on SQLite and did nothing in memory. Costs 1 to 8% on the big listing.MemoryStorehas_load()and_save()to override, for anyone who wants to pickle it or put it in an object database.Reads don't wait for another thread's write anymore. Different from the issue though: reads share one SQLite connection instead of one per thread, because per-thread scans were 4 to 10x slower under the GIL. I also had to turn
cache_spilloff, otherwise SQLite locked reads out partway through a big write anyway. Catch is a large listing holds its changes in memory till it commits. Not in WAL mode, which doesn't have the problem.Longest read during a 20,000 object listing went from ~2 s to 0.06 s. The listing itself is the same speed as main.