Skip to content

fix(lancedb): translate the spill failure on the write path too - #465

Merged
gloryfromca merged 1 commit into
mainfrom
fix/lance-transient-write-errors
Sep 24, 2026
Merged

gloryfromca merged 1 commit into
mainfrom
fix/lance-transient-write-errors

Conversation

@gloryfromca

Copy link
Copy Markdown
Member

Summary

#463 translated LanceError(IO): Execution error: Spill has sent an error into the retryable VectorStoreBusyError only inside the read deadline wrapper. The soak that ran with #463 shows that was the wrong place: 42 spill failures, 0 translated, 104 rows filed unrecoverable. The traceback frames (Windows soak, everos_soak_run3_integration883007c/server.log):

worker.py:800  outcome = await handler.handle_added_or_modified(...)
  -> _apply_lance_changes -> index_repo.upsert(to_upsert)
  -> router.py:48 -> backends/lancedb.py:115 -> repository.py:466 (merge_insert)
  -> lancedb/table.py:5269 execute_merge_insert
RuntimeError: lance error: LanceError(IO): Execution error: Spill has sent an error

So it is merge_insert under _locked. Same translation there; the write is idempotent by id, so the worker's retry (backoff, ExternalServiceError path) is the right answer. Marker docstring corrected (four runs: 187 / 180 / 34 / 42).

Area

  • architecture (storage)

Verification

  • New test: the phrase raised inside _locked becomes VectorStoreBusyError; against main's _locked it fails (bare RuntimeError propagates).
  • test_core/test_persistence/test_lancedb + test_cascade green.
  • Live check: the next 2 h soak on main; lancedb_transient_execution_error must be > 0 and cascade_worker_unrecoverable back to the fuzz baseline (ValidationError count).

🤖 Generated with Claude Code

#463 translated 'Execution error: Spill has sent an error' only inside the
read deadline wrapper, on the assumption the failure came from a query.
The integration-branch soak that ran with #463 says otherwise: 42 spill
failures, 0 translated, 104 unrecoverable rows, and the traceback frames
put it in upsert -> merge_insert -> execute_merge_insert, i.e. under
_locked. Apply the same translation there; the write is idempotent by id,
so the worker's retry is the right answer.

Test: the phrase raised inside _locked becomes VectorStoreBusyError; fails
against main's _locked.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gloryfromca
gloryfromca enabled auto-merge (squash) September 24, 2026 09:34
@gloryfromca
gloryfromca merged commit 8bb5c32 into main Sep 24, 2026
11 checks passed
@gloryfromca
gloryfromca deleted the fix/lance-transient-write-errors branch September 24, 2026 09:38
This was referenced Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant