rigazilla · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -319,6 +319,18 @@ if(GTest_FOUND)
             # Note: NOT gtest_main - we provide our own main()
     )
 
+    # Multi-server seed list — runtime failover (issue #12) integration tests (multi-server)
+    add_executable(multiseed_integration_tests
+        tests/integration/MultiSeedIntegrationTest.cpp
+    )
+
+    target_link_libraries(multiseed_integration_tests
+        PRIVATE
+            hotrod-client
+            GTest::gtest
+            # Note: NOT gtest_main - we provide our own main()
+    )
+
     # SASL/SCRAM authentication integration tests (auth-enabled single server)
     add_executable(auth_integration_tests
         tests/integration/AuthIntegrationTest.cpp
@@ -356,6 +368,7 @@ if(GTest_FOUND)
     add_test(NAME ConcurrentMultiServerTests COMMAND concurrent_multiserver_tests)
     add_test(NAME RetryViewIntegrationTests COMMAND retryview_integration_tests)
     add_test(NAME PingRetryIntegrationTests COMMAND pingretry_integration_tests)
+    add_test(NAME MultiSeedIntegrationTests COMMAND multiseed_integration_tests)
     add_test(NAME AuthIntegrationTests COMMAND auth_integration_tests)
 
     # Set working directory so scripts can be found
@@ -431,6 +444,10 @@ if(GTest_FOUND)
         WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
         LABELS "integration;docker;topology;multi-server;retry"
     )
+    set_tests_properties(MultiSeedIntegrationTests PROPERTIES
+        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
+        LABELS "integration;docker;topology;multi-server;failover"
+    )
     set_tests_properties(AuthIntegrationTests PROPERTIES
         WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
         LABELS "integration;docker;auth"
@@ -445,7 +462,8 @@ if(GTest_FOUND)
         ContainsKeyIntegrationTests ErrorHandlingIntegrationTests FromUriIntegrationTests
         TopologyIntegrationTests
         ConcurrentClientsTests HashAwareRoutingIntegrationTests ConcurrentMultiServerTests
-        RetryViewIntegrationTests PingRetryIntegrationTests AuthIntegrationTests
+        RetryViewIntegrationTests PingRetryIntegrationTests MultiSeedIntegrationTests
+        AuthIntegrationTests
         PROPERTIES TIMEOUT 300
     )
     endif()  # NOT WIN32 (integration tests)

diff --git a/README.md b/README.md
@@ -64,7 +64,8 @@ See [`examples/quickstart/README.md`](examples/quickstart/README.md) for full do
 - **Google Test** (for testing)
 
 **Runtime Requirements**:
-- Infinispan Server 16.0+ (tested with 16.2)
+- Infinispan Server 16.0+ (all integration suites run against `:latest` —
+  currently 16.3; only the quickstart example pins 16.2)
 - Docker (for integration tests and quickstart example)
 
 ### Linux (Fedora/RHEL)
@@ -115,7 +116,7 @@ ctest -C Release --output-on-failure
 
 ## Testing
 
-### Unit Tests (250 tests)
+### Unit Tests
 ```bash
 cd build
 ./unit_tests
@@ -124,7 +125,7 @@ cd build
 ctest -R UnitTests --output-on-failure
 ```
 
-### Integration Tests (92 tests, 19 suites)
+### Integration Tests
 Requires Docker to run Infinispan server:
 ```bash
 cd build
@@ -153,8 +154,9 @@ ctest --output-on-failure
 **Current Progress**: Full CRUD (PING/GET/PUT/REMOVE) + SCRAM auth + topology
 awareness + hash-aware routing + async multiplexing + metadata/version-based
 operations (Step 10) + typed error handling and user-decided retry, keyed and
-keyless (Step 11) + Hot Rod URI support (`RemoteCache::fromUri`, issue #9) — all
-shipped. **Next:** bulk operations (Step 12) or multiplexing benchmarks. See
+keyless (Step 11) + Hot Rod URI support (`RemoteCache::fromUri`, issue #9) +
+multi-server seed list with startup/runtime failover (issue #12) — all shipped.
+**Next:** bulk operations (Step 12) or multiplexing benchmarks. See
 [docs/STATUS.md](docs/STATUS.md) for live status.
 Per-step milestone history (frozen at Step 9) is archived at
 [docs/archive/PROGRESS.md](docs/archive/PROGRESS.md).
@@ -163,9 +165,9 @@ _(Step numbers follow [`../hotrod-foundry/ROADMAP.md`](../hotrod-foundry/ROADMAP
 [`docs/STATUS.md`](docs/STATUS.md) is authoritative for which steps are done — not
 any headline count.)_
 
-**Test Results** (verified 2026-10-09):
-- Unit Tests: **250/250 passing** ✅
-- Integration Tests: **92/92 passing** ✅ across 19 suites (against live Infinispan via Docker)
+**Test Results** — unit and integration suites all pass (integration against live
+Infinispan via Docker). See [`docs/STATUS.md`](docs/STATUS.md) for the current
+coverage breakdown and counts.
 
 ## Features
 
@@ -197,6 +199,9 @@ any headline count.)_
 - ✅ **Hot Rod URI** (`RemoteCache::fromUri("hotrod://...")`) — build a client from a
   connection string (hosts, credentials, `sasl_mechanism`/`client_intelligence`/
   `protocol_version` params); `hotrod://` only for now (TLS and more in issue #10)
+- ✅ **Multi-server seed list** (issue #12) — all hosts in the URI become seed
+  servers with startup *and* runtime failover (ops fall back to the seed list when
+  the connected server dies or no topology is known, e.g. in BASIC mode)
 - ✅ **Integration test framework** (GoogleTest + Docker + multi-node clusters)
 
 ### Full CRUD with Smart Routing and Async Operations
@@ -244,7 +249,9 @@ auto cache = RemoteCache::fromUri(
     "my-cache");
 cache->connect();
 
-// v1 supports hotrod:// only; the first host seeds the topology, credentials
+// Supports hotrod:// only. All hosts become seed servers (issue #12): connect()
+// tries each in order, and ops fall back to the seed list when the connected
+// server dies or no topology is known (e.g. BASIC mode). Credentials
 // enable SCRAM, and sasl_mechanism/client_intelligence/protocol_version (Java
 // "4.0"/"4.1" form) are honoured. Userinfo and query values are percent-decoded
 // (RFC 3986), so a reserved character in a password is written encoded — e.g. a
@@ -289,8 +296,8 @@ cpp17-client/
 │   ├── hash/                    # MurmurHash3 + consistent hashing
 │   └── operations/              # Hot Rod operations (with failover)
 ├── tests/                       # Test suite
-│   ├── unit/                    # 223 unit tests
-│   └── integration/             # 92 integration tests, 19 suites (multi-node clusters)
+│   ├── unit/                    # unit tests (gtest)
+│   └── integration/             # integration tests (multi-node clusters, Docker)
 ├── documentation/               # AsciiDoc user guide (index.adoc + topics/)
 ├── examples/                    # Usage examples
 │   └── quickstart/              # Simple GET/PUT/REMOVE + retry + auth examples
@@ -364,11 +371,14 @@ In short, following the hotrod-foundry porting guidelines:
 - ✅ **v0.8.0** (2026-09-20): Metadata / version-based operations (Step 10)
 - ✅ **v0.9.0** (2026-09-25): Typed error handling + user-decided retry, keyed &
   keyless (Step 11)
+- ✅ **2026-10** (issue-based): Hot Rod URI `fromUri` (#9) + multi-server seed list
+  with startup/runtime failover (#12)
 - 🎯 **v0.10.0** (upcoming): Bulk operations (Step 12)
 
 ---
 
 **Maintained by**: rigazilla  
 **Status**: Active development  
-**Current Step**: Error handling + user-decided retry complete (Step 11, keyed &
-keyless). Next: bulk operations (Step 12). ✅
+**Current Step**: Step 11 (error handling + user-decided retry, keyed & keyless)
+complete; since then Hot Rod URI `fromUri` (#9) and multi-server seed list +
+failover (#12) shipped. Next: bulk operations (Step 12). ✅
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
@@ -75,7 +75,7 @@ again when true async landed.
 **Status:** ✅ Superseded — the real `MultiplexedConnection` replaced this shim
 (see next entry). No blocking shim remains in `RemoteCache.cpp`.
 
-## 2026 (multiplexing landing) — MultiplexedConnection design
+## 2026-07-02 — MultiplexedConnection design
 
 **Decision:** One connection per server multiplexes concurrent requests over a
 single socket. A dedicated read-loop thread reads responses; a
@@ -862,6 +862,41 @@ today, so decoding them is currently a no-op; it is done anyway for Java parity
 and to be ready for issue #10's `token=` (OAUTHBEARER), whose values can carry
 `=`/`+`/`/`.
 
+## 2026-10-09 — Stop repeating test counts across docs (CI badge considered, rejected)
+
+**Problem.** The unit/integration test counts were hand-typed in ~6 places across
+`README.md` and `docs/STATUS.md`. Every test change meant editing all of them, and
+nothing verified the numbers — one README spot had silently drifted to `223 unit
+tests` while the real count had moved on.
+
+**Considered: a CI-published shields.io badge.** Compute counts from the *built test
+binaries* in CI (`scripts/count_tests.sh`, counting via `--gtest_list_tests` so the
+number matches what actually runs and needs no Docker) and force-push a shields
+endpoint JSON to an orphan `badges` branch for the README to embed. Prototyped, then
+**rejected as overkill** for a solo, intermittently-worked repo: it added a CI step
+with `contents: write`, an orphan branch, a helper script, and a badge that 404s
+until the first post-merge run and lags behind CDN caching — a lot of moving parts
+to avoid typing a number.
+
+**Decision.** Just remove the duplication. `docs/STATUS.md` (the authoritative
+current-state doc) carries the totals **once**, in its "Test status" section;
+everywhere else describes *what* is covered rather than repeating the totals:
+- README drops all hard counts — the `### Unit Tests` / `### Integration Tests`
+  headings, the directory-tree comments, and the "Test Results" block (which now
+  points at STATUS.md for the breakdown).
+- Per-feature breadcrumbs ("shipped … N unit + M integration tests") stay, as
+  point-in-time slice notes.
+
+The one-place count is refreshed by hand when it drifts; `--gtest_list_tests` on the
+built binaries (`unit_tests` plus the ctest `integration`-labelled suites) is the
+quick way to get the real number without a `grep TEST(`, which overcounts
+parameterized/disabled macros.
+
+**Provenance.** This decision + convention originated in PR #14
+(`feat/test-count-badge`); that PR was closed unmerged and its changes were
+re-applied here (it had gone stale/conflicting against `main`). Counts as of this
+re-application: **250 unit / 96 integration / 20 suites**.
+
 ## 2026-10-12 — Integration tests hung on CI: Infinispan `:latest` CLI stopped exiting on EOF
 
 **Symptom.** Every CI run from Oct 9 on hung: `GetIntegrationTests` started and
@@ -903,3 +938,140 @@ was missing from the ctest `TIMEOUT 300` group — the exact hang the 2026-10-12
 fix addressed everywhere else. Added `-f -` to its CLI invocation and listed
 `FromUriIntegrationTests` in the timeout block so this suite can no longer stall
 the CI job on Infinispan `:latest`.
+
+## 2026-10-19 — Multi-server seed list: startup + runtime failover (issue #12)
+
+**What:** `fromUri` now carries *all* comma-separated hosts as seed servers
+(`RemoteCache::seeds_`, retained in URI order; mirrors the Java client's immutable
+`ClusterInfo.getInitialServers()`). Two failover tiers were added:
+- **Startup** — `connect()` tries each seed in order, reusing the pre-built
+  `connection_` for `seeds_[0]` and building fresh for the rest; the first success
+  becomes the bootstrap `connection_` (and `host_`/`port_`). If every seed is
+  unreachable it throws one `HotRodClientException(BeforeSend)` whose message lists
+  every seed tried (with its error).
+- **Runtime** — a new `tryConnectSeeds(exclude, tried)` helper (caller holds
+  `stateMutex_`) is wired into the fall-through points of both selection methods:
+  `selectAnyServer` (keyless), the hash-owners-exhausted path of
+  `selectServerForKey`, and that method's no-hash/no-owners/empty fallback (which
+  previously just `return connection_`). It prefers the live bootstrap `connection_`
+  (fast path, so the happy path and BASIC mode do no new I/O), else walks the seeds
+  via the existing pure `orderKeyCandidates({}, seeds_, exclude∪tried)` and adopts
+  the first reachable seed as the new bootstrap connection so the next op's fast
+  path reuses it instead of re-probing a dead one.
+
+**Why it matters:** v1 of the URI work (issue #9) used only the first host as the
+seed and relied on topology discovery for the rest, which loses startup failover
+and — in BASIC mode, where no topology is ever learned — any failover at all.
+
+**Design decisions (discussed before implementing, compared against the Java 16.3
+client):**
+- **Honor `proxyToNonOwner` for keyed seed fallback (D5 preserved).** When a key's
+  owners are all down/excluded *and* `proxyToNonOwner_ == false`, we do **not** fall
+  back to a seed — a seed serving a non-owned key is a proxy, exactly what that flag
+  forbids. So keyed seed fallback is gated on `proxyToNonOwner_`. (Java has no such
+  flag — it always proxies; our default `true` matches Java, the strict mode is a
+  C++-only opt-in.) Bootstrap cases (empty/no topology, no owners) ignore the flag,
+  since there are no owners to exhaust.
+- **Caller-driven exhaustion ≠ unreachability.** If the topology is non-empty but
+  the caller excluded *every* known server, we throw rather than rescue via a seed.
+  Found via `PingRetryIntegrationTest.AllServersExcludedThrowsBeforeSend`: the seed's
+  address (`localhost:11222`) differs from the topology's internal container address
+  (`172.18.0.x:11222`) for the *same* node, so an exclusion set built from topology
+  addresses does not contain the seed — the seed fast path would have wrongly
+  rescued the excluded ping. Fix: seed fallback fires only when we have no topology
+  at all, or when we actually attempted ≥1 server and it failed (not when the
+  candidate list was emptied purely by exclusion). Applied to both selection paths
+  (an explicit `candidates.empty()` throw in `selectAnyServer`; a `!tried.empty()`
+  gate on the keyed path).
+- **Reassign `connection_` on runtime failover (+ fast path), and lock its reads.**
+  Pointing `connection_` at the reached seed keeps `isConnected()` truthful and
+  avoids re-probing the dead bootstrap seed on every op during an outage. Because
+  `connection_` can now be reassigned from a user/op thread (under `stateMutex_`),
+  the previously-unlocked reads in `isConnected()`/`disconnect()` now take
+  `stateMutex_` to remove the shared_ptr assignment race. `disconnect()` copies the
+  pointer out under the lock and calls `close()` *outside* it (close joins the read
+  loop, which may be blocked in `handleTopologyUpdate` waiting on the same mutex).
+  The startup `connect()` reassignment is pre-read-loop/single-threaded.
+- **Runtime seed (re)connection is gated on a connect()..disconnect() lifecycle
+  flag (`active_`).** First cut of `tryConnectSeeds` reconnected whenever the
+  bootstrap connection was down — which silently re-established a connection for an
+  op issued *after* `disconnect()` (or before `connect()`), breaking the long-
+  standing contract that such ops throw. Caught by the full `ctest` sweep: all ten
+  `*AfterDisconnect` suite tests failed (they `EXPECT_THROW std::runtime_error`
+  after `disconnect()`; BASIC mode has empty topology, so they hit exactly the new
+  keyed seed-fallback path). Fix: `active_` is set true at the end of a successful
+  `connect()` and false in `disconnect()` (both under `stateMutex_`); when false,
+  `tryConnectSeeds` returns the bootstrap `connection_` unchanged so the op throws
+  "Not connected" as before. `ReconnectAndPing` (disconnect → connect again) still
+  works because `connect()` re-sets the flag.
+
+**Divergences from the Java client (intentional, cross-checked against
+`/home/rigazilla/git/infinispan/client/hotrod-client` @ 16.3):**
+- **connect() all-seeds-down throws; Java tolerates.** Java's `OperationDispatcher
+  .start()` eagerly connects to all initial servers but swallows every failure and
+  defers to the first op. We keep our existing "`connect()` establishes ≥1
+  connection or throws" contract — clearer for callers; the thrown message names
+  every seed for diagnosis.
+- **No failed-server timeout memo.** Java keeps a global `connectionFailedServers`
+  set with `serverFailureTimeout` expiry so it pre-emptively avoids known-dead
+  servers. We have no equivalent, so during an outage we re-probe dead *topology*
+  members per op (the dead *bootstrap* seed is avoided by the reassign+fast-path).
+  Latency-under-failure only, never a wrong result. Left as future work.
+- **Privileged `connection_` kept (not folded into the pool).** Java 16.x has one
+  equal channel per address and no distinguished connection; our bootstrap
+  `connection_` lives outside `connectionPool_`, which is why the fast-path/locking
+  dance exists and why `seeds_[0]` can hold a duplicate socket (one as `connection_`,
+  one as a pool entry) once it also appears in topology. Keeping it was a scope call:
+  folding it into the pool as just-another-entry (the Java-faithful model, which
+  would delete the fast-path/locking code) is a larger refactor tracked as future
+  work. Functionally the layered model matches Java's observable failover behavior.
+
+**Tests:** +2 startup-failover in `FromUriIntegrationTest` (recover via 2nd seed;
+all-seeds-unreachable throws naming each) and +2 `MultiSeedIntegrationTest`:
+`RuntimeFailoverAcrossSeedsBasic` (2-node cluster, BASIC client seeded with both,
+kill the connected node, next op recovers via the other seed) and
+`TopologyAwareStartupFailoverAndHashRouting` (HASH client from a multi-seed URI
+with a dead first seed: connect() fails over, then topology discovery + hash
+routing work). The pure "all topology members unreachable -> seed" corner is not
+staged as an integration test: the seeds are the cluster nodes, so killing every
+topology member also kills every seed — the BASIC test isolates the seed tier
+instead. All affected
+regressions re-run green: RetryView (incl. the D5 proxy-disabled throw), PingRetry
+(incl. the all-excluded throw fixed above), HashAwareRouting, ConcurrentMultiServer;
+unit 250/250; `-Werror`/Release build clean.
+
+## 2026-10-19 — Schema gotcha preserved from STATUS: replace_if_unmodified nibble flip
+
+**What:** Recording — in its proper home — a schema bug found 2026-09-20 that had
+only ever lived in STATUS.md, moved here during the STATUS backlog cleanup so it is
+not lost. `hotrod40.ksy` `replace_if_unmodified_request` packs the expiration
+`time_units` nibbles in the OPPOSITE order from `put_request`/`expiration_params`
+(lifespan and maxIdle swapped). Java uses one shared `writeExpirationParams` for
+both PUT and replaceWithVersion, so PUT's order — **high nibble = lifespan, low
+nibble = maxIdle** — is authoritative and is what this client implements
+(`replaceWithVersion` 0x09, shipped 2026-09-20). Cross-checked against this client +
+Java `ReplaceIfUnmodifiedOperation` + a corrected reading of the schema.
+**Why it matters:** if someone later "fixes" the client to match the `.ksy` here,
+replaceWithVersion's lifespan/maxIdle would silently swap. The `.ksy` is still wrong
+on this point; fix pending upstream in hotrod-dissector.
+
+## 2026-10-19 — Align all test fixtures to `:latest` (cluster compose was pinned 16.2)
+
+**What:** `test-configs/docker-compose-cluster.yml` (the multi-node fixture used by
+`start_cluster.sh`/`add_cluster_node.sh` for the topology/hash/retry/concurrent/
+multi-seed suites) pinned `quay.io/infinispan/server:16.2`, while the single-server
+and auth fixtures (`scripts/start_infinispan_noauth.sh`, `start_infinispan_auth.sh`)
+already used `:latest`. Changed all four cluster node images to `:latest` so every
+integration suite exercises the same (newest) server. The quickstart example
+(`examples/quickstart/start-server.sh`) intentionally stays pinned to 16.2 — it's a
+copy-paste demo, not a test.
+
+**Why:** the split meant different suites tested different server versions (so a
+multi-node regression on a new server could hide until a cluster run happened to use
+it). STATUS.md already claimed "integration tests pull `:latest`", which was an
+overclaim until now.
+
+**Risk + mitigation:** `:latest` can drift and break (see the 2026-10-12 CLI-hang
+entry) — the `TIMEOUT 300` guard per ctest suite plus `cli.sh -f -` batch mode keep
+a server-side change failing fast rather than hanging CI. Verified the full
+multi-node suite set green against `:latest` = 16.3.0 on 2026-10-19 after the switch.