NullPointerException in NitriteMVStore.removeMap when a non-unique index is first used from several threads

[matre]

Versions: nitrite 5.3.0, nitrite-mvstore-adapter 5.3.0, h2-mvstore 2.4.240, Java 17. Also seen on 4.4.2.

What happens

When a file database is opened and several threads use the same non-unique index for the first time at once, some of the finds fail:

java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
    at java.base/java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1111)
    at java.base/java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1102)
    at org.dizitart.no2.mvstore.NitriteMVStore.removeMap(NitriteMVStore.java:179)
    at org.dizitart.no2.mvstore.NitriteMVMap.drop(NitriteMVMap.java:265)
    at org.dizitart.no2.index.SingleFieldIndex.migrateLegacyIndex(SingleFieldIndex.java:242)
    at org.dizitart.no2.index.SingleFieldIndex.findCompositeMap(SingleFieldIndex.java:211)
    at org.dizitart.no2.index.SingleFieldIndex.findNitriteIds(SingleFieldIndex.java:131)
    at org.dizitart.no2.index.ComparableIndexer.findByFilter(ComparableIndexer.java:66)
    at org.dizitart.no2.collection.operation.ReadOperations.findSuitableStream(ReadOperations.java:236)
    at org.dizitart.no2.collection.operation.ReadOperations.createCursor(ReadOperations.java:116)
    at org.dizitart.no2.collection.operation.ReadOperations.find(ReadOperations.java:74)
    at org.dizitart.no2.collection.operation.CollectionOperations.find(CollectionOperations.java:114)
    at org.dizitart.no2.collection.DefaultNitriteCollection.find(DefaultNitriteCollection.java:175)

The next find on the same index works. We hit it in production when two environments in one JVM share a database file: both release it, it closes, and their first queries reopen it at the same moment.

Reproduction

Nitrite API only. With 4 threads and 100 openings, 92 of 400 finds fail on 5.3.0.

static Nitrite open(File file) {
    return Nitrite.builder()
            .loadModule(MVStoreModule.withConfig().filePath(file.getPath()).build())
            .openOrCreate();
}

@Test
public void firstUseOfANonUniqueIndexFromSeveralThreads() throws Exception {
    File file = File.createTempFile("nitrite-legacy-race", ".db");
    file.delete();

    Nitrite db = open(file);
    NitriteCollection items = db.getCollection("items");
    items.createIndex(IndexOptions.indexOptions(IndexType.NON_UNIQUE), "state");
    items.insert(Document.createDocument("state", "new"));
    db.close();

    int threads = 4, openings = 100;
    List<Throwable> failures = new CopyOnWriteArrayList<>();
    for (int opening = 0; opening < openings; opening++) {
        Nitrite reopened = open(file);
        NitriteCollection collection = reopened.getCollection("items");
        CyclicBarrier start = new CyclicBarrier(threads);
        Thread[] workers = new Thread[threads];
        for (int i = 0; i < threads; i++) {
            workers[i] = new Thread(() -> {
                try {
                    start.await();
                    collection.find(FluentFilter.where("state").eq("new")).toList();
                } catch (Throwable t) {
                    failures.add(t);
                }
            });
            workers[i].start();
        }
        for (Thread worker : workers) worker.join();
        reopened.close();
    }
    assertEquals(0, failures.size());
}

Cause

Three things combine. Line numbers are from 5.3.0.

1. Closing a collection recreates the legacy index map. IndexManager.close() opens every index map by indexMeta.getIndexMap(), which is the legacy name from deriveIndexMapName(). NitriteStore.openMap creates a map that does not exist, so each close leaves an empty legacy map behind, and migrateLegacyIndex() runs again after every open. A database that was never written by Nitrite < 4.4 still gets these maps.

2. Two index objects can exist for one descriptor. ComparableIndexer.findNitriteIndex() does containsKey, then new SingleFieldIndex(...), then put, which is not atomic. Two threads can each create an instance. migrateLegacyIndex() is guarded by synchronized (this) and a per-instance migrationChecked, so both instances migrate the same legacy map.

3. The second drop gets a null name. The first instance removes the map from the MVStore. The second opened the same map before that, and its MVMap.getName() now returns null, because the map.<id> meta entry is gone. NitriteMVMap.drop() passes that null to closeMap and removeMap, and nitriteMapRegistry.remove(null) throws.

Suggested fixes

• ComparableIndexer.findNitriteIndex(): use indexRegistry.computeIfAbsent(...), so there is one instance per descriptor.
• IndexManager.close(): close only maps that are already open, for example through the registry, instead of openMap by the legacy name.
• NitriteMVMap.drop(): read the name once before closing, and tolerate a map that is already removed.

Workaround

Use every single-field index once right after opening the database, before sharing it between threads, for example with collection.find(where(field).eq(null)) for each index in listIndices().