Repository navigation
libsql-server: fix deadlock between namespace fork and delete - #2326
Open
factory-kyle-mistele wants to merge 1 commit into
Open
factory-kyle-mistele wants to merge 1 commit into
factory-kyle-mistele wants to merge 1 commit into
Conversation
`MetaStore::remove` locks `conn` and then `configs`. The flush that ends every namespace fork (`process` with no new config) locked `configs` and then `conn`. When a fork finished during a namespace delete, each thread held the lock the other needed, and every later namespace call hung on `configs`. Take `conn` before `configs` in `process` too, and pass the locked connection to `try_process`. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2325.
Problem
MetaStore::removelocksconnand thenconfigs. The flush at the end of every namespace fork (processwith no new config) locksconfigsand thenconn. If a fork finishes while a namespace delete runs, the two threads deadlock. After that, every namespace call waits onconfigsforever, while/healthstill returns 200. The issue has a reproduction script.Change
processnow locksconnbeforeconfigswhen it flushes, the same order asremove.try_processtakes the locked connection from its caller instead of locking it itself. The flush path that has a new config holds no other lock, so it only moves theconnlock into the caller.I kept both locks held during the write rather than dropping
configsbefore it. Dropping it would also break the deadlock, but a delete of the same namespace could then run between reading the config and writing it. The write would then bring back the row of a namespace that was just deleted.Test
namespace::meta_store::test::concurrent_flush_and_remove_do_not_deadlockruns 1,000 flushes of one namespace and 1,000 removes of another at the same time, with a 30 s timeout.cargo test -p libsql-server --lib namespace::meta_storeI also ran
cargo fmt --check, the rest of the libsql-server unit tests (101 passed, 2 ignored) andcargo test -p libsql-server --test tests namespaces(24 passed). The two that fail there,namespaces::meta::meta_attachandnamespaces::meta::meta_attach_shared_schema("Namespacefoodoesn't allow attach"), fail the same way onmainwithout this change.I also ran the reproduction script from the issue against release builds of sqld. Without this change, it deadlocked in 19 of 20 runs. With it, 50 of 50 runs finished all 20 rounds (8,000 forks) without a deadlock.