Skip to content

libsql-server: fix deadlock between namespace fork and delete - #2326

Open
factory-kyle-mistele wants to merge 1 commit into
tursodatabase:mainfrom
factory-kyle-mistele:fix-metastore-deadlock
Open

factory-kyle-mistele wants to merge 1 commit into
tursodatabase:mainfrom
factory-kyle-mistele:fix-metastore-deadlock

Conversation

@factory-kyle-mistele

Copy link
Copy Markdown

Fixes #2325.

Problem

MetaStore::remove locks conn and then configs. The flush at the end of every namespace fork (process with no new config) locks configs and then conn. If a fork finishes while a namespace delete runs, the two threads deadlock. After that, every namespace call waits on configs forever, while /health still returns 200. The issue has a reproduction script.

Change

  • process now locks conn before configs when it flushes, the same order as remove.
  • try_process takes the locked connection from its caller instead of locking it itself. The flush path that has a new config holds no other lock, so it only moves the conn lock into the caller.

I kept both locks held during the write rather than dropping configs before it. Dropping it would also break the deadlock, but a delete of the same namespace could then run between reading the config and writing it. The write would then bring back the row of a namespace that was just deleted.

Test

namespace::meta_store::test::concurrent_flush_and_remove_do_not_deadlock runs 1,000 flushes of one namespace and 1,000 removes of another at the same time, with a 30 s timeout.

  • Without the fix, it failed 6 of 6 runs (deadlock, timed out after 30 s).
  • With the fix, it passed 21 of 21 runs, in about 0.15 s each.
cargo test -p libsql-server --lib namespace::meta_store

I also ran cargo fmt --check, the rest of the libsql-server unit tests (101 passed, 2 ignored) and cargo test -p libsql-server --test tests namespaces (24 passed). The two that fail there, namespaces::meta::meta_attach and namespaces::meta::meta_attach_shared_schema ("Namespace foo doesn't allow attach"), fail the same way on main without this change.

I also ran the reproduction script from the issue against release builds of sqld. Without this change, it deadlocked in 19 of 20 runs. With it, 50 of 50 runs finished all 20 rounds (8,000 forks) without a deadlock.

`MetaStore::remove` locks `conn` and then `configs`. The flush that ends
every namespace fork (`process` with no new config) locked `configs` and
then `conn`. When a fork finished during a namespace delete, each thread
held the lock the other needed, and every later namespace call hung on
`configs`.

Take `conn` before `configs` in `process` too, and pass the locked
connection to `try_process`.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sqld: deadlock when a namespace fork and a namespace delete run at the same time

1 participant