Skip to content

cache: replace read/dirty with a concurrent hash-trie #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
twmb merged 2 commits into from
Sep 27, 2026
Merged

cache: replace read/dirty with a concurrent hash-trie #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dc5b386
cache: back Cache with a concurrent hash-trie
twmb Sep 27, 2026
10ea4ee
README: describe the trie and benchmark against sync.Map
twmb Sep 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: ci

on:
push:
branches: ["main"]
pull_request:

jobs:
test:
# amd64 and arm64: the trie/cache core is lock-free, and arm64's weaker
# memory ordering surfaces reordering bugs that x86's TSO hides.
# 1.25 is the minimum in go.mod.
name: test (${{ matrix.os }}, go ${{ matrix.go }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm]
go: ["1.25.x", stable]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- run: go vet ./...
- run: go test -race ./...

test-386:
# 32-bit: uintptr hashes are 32 bits, so the trie is 8 levels deep
# instead of 16 and the test hash constants must fit the word size.
# The race detector is unsupported on 386, so this runs without it.
name: test (linux/386)
runs-on: ubuntu-latest
env:
GOARCH: "386"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: stable
- run: go vet ./...
- run: go test ./...
130 changes: 43 additions & 87 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ cache
Package cache provides a concurrency safe, mostly lock-free, singleflight
request collapsing generic cache with support for stale values.

The guts of this package are similar to `sync.Map`, with minor differences in
when the internal locked write-map is promoted an atomic read-only map, and
major differences to support singleflight request collapsing.
The storage is a concurrent hash-trie similar to Go's internal
`HashTrieMap`, which backs `sync.Map`: loads are lock-free, and inserts and
deletes lock only the trie node being changed.

Functions in this API that return a `KeyState` return the state last, rather
than the error last: the value and error are cached as a single internal unit
Expand Down Expand Up @@ -34,9 +34,12 @@ value can be kept and returned from `Get` during a refresh / if a refresh
fails. Keys can be manually expired with `Expire`. Internally expired values or
errors can be occasionally cleaned with `Clean`.

Out of an abundance of paranoia that this code is correct, there are unit tests
to hit 97% coverage of the `cache.Cache` type. As well, all tests against
`sync.Map` are copied into this library and used against `cache.Cache`.
Out of an abundance of paranoia that this code is correct, the tests include
all stdlib `sync.Map` tests run against `cache.Cache`, a property test
against a mutex guarded map, a linearizability checker for concurrent
operations on a single key, structural checks of the trie after concurrent
stress, and regression tests for specific races. CI runs the tests with the
race detector on amd64 and arm64, and without it on 386.

This package also provides a cached `Item` and a `Set`. `Item` can be used to
populate an expensive value once and expire or replace it when needed, similar
Expand All @@ -54,87 +57,40 @@ Documentation
Benchmarking
------------

Some of the APIs are actually faster or more memory efficient than `sync.Map`
because of generics. Some are slower and require more allocs because of the
request collapsing aspect. Some benchmarks below show more CPU, but this is due
to a few slight changes in how the write (dirty) map is internally promoted to
the read map.

The benchmarks are from the stdlib using a key and value of type int:
These are the stdlib `sync.Map` benchmarks, run against `sync.Map` and against
a `cache.Cache[int, any]` adapted to the same interface. Both are hash-tries
underneath. The numbers are medians of 8 runs on an Apple M2 with Go 1.27;
the last column is allocs per op, `sync.Map` / cache.

```
name old time/op new time/op delta
LoadMostlyHits 12.8ns ±44% 14.0ns ±50% ~ (p=0.393 n=10+10)
LoadMostlyMisses 10.7ns ±53% 4.6ns ±47% -57.31% (p=0.000 n=10+10)
LoadOrStoreBalanced 351ns ± 3% 705ns ±27% +100.68% (p=0.000 n=8+10)
LoadOrStoreUnique 663ns ±13% 1296ns ±31% +95.53% (p=0.000 n=10+10)
LoadOrStoreCollision 9.71ns ±60% 29.26ns ±46% +201.44% (p=0.000 n=10+10)
LoadAndDeleteBalanced 11.2ns ±42% 7.1ns ±48% -36.06% (p=0.035 n=10+10)
LoadAndDeleteUnique 9.19ns ±62% 4.17ns ±47% -54.59% (p=0.001 n=10+10)
LoadAndDeleteCollision 10.9ns ±41% 14.3ns ±35% +31.18% (p=0.023 n=10+10)
Range 3.73µs ±48% 5.40µs ±45% ~ (p=0.063 n=10+10)
AdversarialAlloc 298ns ±22% 259ns ±21% ~ (p=0.063 n=10+10)
AdversarialDelete 83.7ns ±14% 105.6ns ±20% ~ (p=0.052 n=10+10)
DeleteCollision 6.87ns ±71% 4.86ns ±100% ~ (p=0.424 n=10+10)
SwapCollision 201ns ±26% 266ns ±31% ~ (p=0.143 n=10+10)
SwapMostlyHits 40.3ns ±42% 59.7ns ±51% ~ (p=0.143 n=10+10)
SwapMostlyMisses 662ns ±18% 607ns ±24% -8.19% (p=0.023 n=10+10)
CompareAndSwapCollision 33.4ns ±41% 24.9ns ±65% -25.44% (p=0.023 n=10+10)
CompareAndSwapNoExistingKey 10.3ns ±46% 2.0ns ±53% -80.15% (p=0.000 n=10+10)
CompareAndSwapValueNotEqual 9.64ns ±51% 6.02ns ±95% -37.49% (p=0.019 n=10+10)
CompareAndSwapMostlyHits 46.9ns ±39% 56.8ns ±47% ~ (p=0.143 n=10+10)
CompareAndSwapMostlyMisses 22.9ns ±39% 13.4ns ±48% -41.48% (p=0.023 n=10+10)
CompareAndDeleteCollision 28.1ns ±47% 18.3ns ±37% -34.88% (p=0.019 n=10+10)
CompareAndDeleteMostlyHits 64.0ns ±38% 63.9ns ±52% ~ (p=0.165 n=10+10)
CompareAndDeleteMostlyMisses 18.3ns ±37% 8.7ns ±53% -52.65% (p=0.023 n=10+10)

name old alloc/op new alloc/op delta
LoadMostlyHits 6.00B ± 0% 0.00B -100.00% (p=0.000 n=10+10)
LoadMostlyMisses 6.00B ± 0% 0.00B -100.00% (p=0.000 n=10+10)
LoadOrStoreBalanced 74.7B ±16% 195.3B ± 1% +161.45% (p=0.000 n=10+10)
LoadOrStoreUnique 159B ± 7% 330B ± 7% +107.15% (p=0.000 n=9+10)
LoadOrStoreCollision 0.00B 32.00B ± 0% +Inf% (p=0.000 n=10+10)
LoadAndDeleteBalanced 4.00B ± 0% 0.00B -100.00% (p=0.000 n=10+10)
LoadAndDeleteUnique 8.00B ± 0% 0.00B -100.00% (p=0.000 n=10+10)
LoadAndDeleteCollision 1.00B ± 0% 1.00B ± 0% ~ (all equal)
Range 16.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
AdversarialAlloc 53.4B ± 3% 64.8B ± 3% +21.35% (p=0.000 n=10+10)
AdversarialDelete 22.6B ±12% 39.0B ± 0% +72.57% (p=0.000 n=10+10)
DeleteCollision 0.00B 0.00B ~ (all equal)
SwapCollision 16.0B ± 0% 80.0B ± 0% +400.00% (p=0.000 n=10+10)
SwapMostlyHits 28.0B ± 0% 86.0B ± 0% +207.14% (p=0.000 n=10+10)
SwapMostlyMisses 124B ± 0% 136B ± 1% +9.71% (p=0.000 n=10+10)
CompareAndSwapCollision 7.60B ± 8% 21.70B ±17% +185.53% (p=0.000 n=10+10)
CompareAndSwapNoExistingKey 8.00B ± 0% 0.00B -100.00% (p=0.000 n=10+10)
CompareAndSwapValueNotEqual 0.00B 0.00B ~ (all equal)
CompareAndSwapMostlyHits 33.0B ± 0% 91.0B ± 0% +175.76% (p=0.000 n=10+10)
CompareAndSwapMostlyMisses 23.0B ± 0% 16.0B ± 0% -30.43% (p=0.000 n=10+10)
CompareAndDeleteCollision 0.00B 2.00B ± 0% +Inf% (p=0.000 n=10+8)
CompareAndDeleteMostlyHits 39.0B ± 0% 90.6B ± 1% +132.31% (p=0.000 n=10+10)
CompareAndDeleteMostlyMisses 16.0B ± 0% 8.0B ± 0% -50.00% (p=0.002 n=8+10)

name old allocs/op new allocs/op delta
LoadMostlyHits 0.00 0.00 ~ (all equal)
LoadMostlyMisses 0.00 0.00 ~ (all equal)
LoadOrStoreBalanced 2.00 ± 0% 3.00 ± 0% +50.00% (p=0.000 n=10+10)
LoadOrStoreUnique 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10)
LoadOrStoreCollision 0.00 1.00 ± 0% +Inf% (p=0.000 n=10+10)
LoadAndDeleteBalanced 0.00 0.00 ~ (all equal)
LoadAndDeleteUnique 0.60 ±100% 0.00 -100.00% (p=0.011 n=10+10)
LoadAndDeleteCollision 0.00 0.00 ~ (all equal)
Range 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
AdversarialAlloc 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
AdversarialDelete 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
DeleteCollision 0.00 0.00 ~ (all equal)
SwapCollision 1.00 ± 0% 1.00 ± 0% ~ (all equal)
SwapMostlyHits 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
SwapMostlyMisses 6.00 ± 0% 4.00 ± 0% -33.33% (p=0.000 n=10+10)
CompareAndSwapCollision 0.00 0.00 ~ (all equal)
CompareAndSwapNoExistingKey 0.50 ±100% 0.00 -100.00% (p=0.033 n=10+10)
CompareAndSwapValueNotEqual 0.00 0.00 ~ (all equal)
CompareAndSwapMostlyHits 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=10+10)
CompareAndSwapMostlyMisses 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=10+10)
CompareAndDeleteCollision 0.00 0.00 ~ (all equal)
CompareAndDeleteMostlyHits 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=10+10)
CompareAndDeleteMostlyMisses 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
name sync.Map cache allocs
LoadMostlyHits 3.10ns 3.28ns 0 / 0
LoadMostlyMisses 1.66ns 1.81ns 0 / 0
LoadOrStoreBalanced 47.2ns 80.0ns 2 / 3
LoadOrStoreUnique 95.1ns 121ns 3 / 4
LoadOrStoreCollision 2.53ns 10.3ns 0 / 1
LoadAndDeleteBalanced 2.05ns 1.86ns 0 / 0
LoadAndDeleteUnique 2.13ns 0.65ns 0 / 0
LoadAndDeleteCollision 3.43ns 2.15ns 0 / 0
Range 1.45µs 2.25µs 0 / 0
AdversarialAlloc 5.61ns 4.59ns 0 / 0
AdversarialDelete 2.98ns 2.91ns 0 / 0
DeleteCollision 1.90ns 1.66ns 0 / 0
SwapCollision 122ns 258ns 1 / 1
SwapMostlyHits 35.9ns 30.2ns 2 / 1
SwapMostlyMisses 208ns 245ns 3 / 3
CompareAndSwapCollision 6.02ns 9.11ns 0 / 0
CompareAndSwapNoExistingKey 4.81ns 0.80ns 0 / 0
CompareAndSwapValueNotEqual 3.69ns 3.07ns 0 / 0
CompareAndSwapMostlyHits 38.3ns 27.7ns 3 / 2
CompareAndSwapMostlyMisses 11.6ns 7.93ns 2 / 1
CompareAndDeleteCollision 3.08ns 4.36ns 0 / 0
CompareAndDeleteMostlyHits 43.5ns 62.9ns 3 / 3
CompareAndDeleteMostlyMisses 5.25ns 5.31ns 0 / 1
Clear 84.8ns 128ns 2 / 3
```

Loads are within 10% of `sync.Map`, and deletes are faster. Stores cost more:
each stored value is a `loading` that also carries singleflight and expiry
state, and the adapter's `LoadOrStore` allocates a closure for its miss
function on every call.
108 changes: 108 additions & 0 deletions cache/autoclean_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
package cache

import (
"runtime"
"testing"
"time"
)

func TestAutoClean_RunsAndCleansExpired(t *testing.T) {
c := New[string, int](
MaxAge(time.Millisecond),
AutoCleanInterval(2*time.Millisecond),
)
defer c.StopAutoClean()

c.Set("a", 1)
waitEntries(t, c, 0)
}

// waitEntries waits until the trie holds n entries.
func waitEntries[K comparable, V any](t *testing.T, c *Cache[K, V], n int) {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
for trieEntries(c) != n {
if time.Now().After(deadline) {
t.Fatalf("trie has %d entries, want %d", trieEntries(c), n)
}
time.Sleep(time.Millisecond)
}
}

func TestAutoClean_DisabledWhenStaleAgeNegative(t *testing.T) {
// AutoCleanInterval is ignored when MaxStaleAge < 0 (stales kept forever).
// We verify no goroutine is spawned by confirming StopAutoClean is a no-op
// on a cache whose quitClean channel was never created.
c := New[string, int](
MaxAge(time.Millisecond),
MaxStaleAge(-1),
AutoCleanInterval(time.Millisecond),
)
if c.quitClean != nil {
t.Fatal("quitClean channel should not be allocated when MaxStaleAge<0")
}
c.StopAutoClean() // no-op must not panic
}

func TestAutoClean_StopIdempotent(t *testing.T) {
c := New[string, int](
MaxAge(time.Millisecond),
AutoCleanInterval(time.Millisecond),
)
c.StopAutoClean()
c.StopAutoClean() // sync.Once guards the close; second call must be safe.
}

func TestAutoClean_ZeroIntervalNoGoroutine(t *testing.T) {
// autoCleanInterval == 0 means disabled; no goroutine, no channel.
c := New[string, int](MaxAge(time.Second))
if c.quitClean != nil {
t.Fatal("quitClean channel should not be allocated with zero interval")
}
c.StopAutoClean()
}

// Once StopAutoClean returns, no Clean runs.
func TestAutoClean_StopHaltsCleaning(t *testing.T) {
c := New[string, int](
MaxAge(time.Millisecond),
AutoCleanInterval(time.Millisecond),
)
c.Set("a", 1)
waitEntries(t, c, 0)
c.StopAutoClean()

c.Set("b", 2)
time.Sleep(20 * time.Millisecond)
if n := trieEntries(c); n != 1 {
t.Fatalf("trie has %d entries after StopAutoClean, want 1", n)
}
c.Clean()
if n := trieEntries(c); n != 0 {
t.Fatalf("trie has %d entries after Clean, want 0", n)
}
}

// A cache that is never stopped is still garbage collected, and its clean
// goroutine exits.
func TestAutoClean_ExitsWhenCacheCollected(t *testing.T) {
for _, newDone := range []func() chan struct{}{
func() chan struct{} { return New[int, int](AutoCleanInterval(time.Millisecond)).cleanDone },
func() chan struct{} { return NewItem[int](AutoCleanInterval(time.Millisecond)).c.cleanDone },
func() chan struct{} { return NewSet[int](AutoCleanInterval(time.Millisecond)).c.cleanDone },
} {
done := newDone()
deadline := time.Now().Add(10 * time.Second)
for exited := false; !exited; {
runtime.GC()
select {
case <-done:
exited = true
case <-time.After(10 * time.Millisecond):
if time.Now().After(deadline) {
t.Fatal("clean goroutine did not exit after its cache became unreachable")
}
}
}
}
}
Loading
Loading