[levkk]

Problem

The cluster connection manager currently reloads the config only once if it detects an offline connection pool. During fast, consecutive reloads, it's possible to hit two offline clusters consecutively, propagating the error to the client.

Fixes #1933

Solution

Loop until the cluster returns a valid connection or an error indicating there won't be one, ever, e.g., because the connection pool has been removed.

[codecov]

Codecov Report

❌ Patch coverage is 96.66667% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../src/backend/pool/connection/cluster_connection.rs	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Metal bench (arm64) passed. All 152 measures within bounds.

suite	throughput	cpu/txn	user cycles/txn	user instr/txn	kernel cycles/txn	kernel instr/txn	ctx/txn
pooler		20.06 (+0.2% ± 0.6%)	20,468 (-0.2% ± 0.3%)	35,233 (-0.1% ± 0.06%)	31,228 (+0.4% ± 1.2%)	51,802 (+0.1% ± 0.3%)
lb		20.61 (+0.1% ± 1.0%)	21,049 (-0.1% ± 0.2%)	36,151 (-0.1% ± 0.04%)	31,826 (+0.4% ± 1.8%)	52,976 (+0.1% ± 0.2%)
sharding-w2		38.43 (+0.5% ± 0.7%)	52,836 (+0.2% ± 1.1%)	102,561 (-0.0% ± 0.02%)	47,956 (+0.8% ± 1.2%)	54,438 (+0.0% ± 0.07%)	1.52 (+0.2% ± 0.1%)
sharding-w8	217,352 (-0.0% ± 0.4%)	33.7 (+0.0% ± 0.3%)	51,768 (+0.1% ± 0.5%)	57,905 (-0.1% ± 0.03%)	41,541 (-0.1% ± 0.5%)	42,953 (-0.0% ± 0.07%)	0.1433 (-0.3% ± 1.4%)

Counters per txn

suite	frontend stall	l1i refill	backend stall
pooler	8,610 (-0.3% ± 0.9%)	2,799 (-0.8% ± 0.4%)
lb	8,909 (-0.1% ± 0.6%)	2,849 (-0.7% ± 0.3%)
sharding-w2	24,444 (+0.5% ± 2.6%)	4,910 (+0.3% ± 1.6%)
sharding-w8	26,445 (+0.5% ± 0.6%)	4,796 (+0.3% ± 0.2%)

Microbench: all 135 measures within bounds.

lb flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

microbench flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

pooler flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w2 flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w8 flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 6e3aa15 with 678843a on r8g.metal-24xl (spot).

[github-actions]

Metal bench (x86) passed. All 152 measures within bounds.

suite	throughput	cpu/txn	user cycles/txn	user instr/txn	kernel cycles/txn	kernel instr/txn	ctx/txn
pooler		35.29 (+0.0% ± 0.1%)	24,682 (+0.1% ± 0.2%)	36,696 (-0.1% ± 0.01%)	57,653 (-0.1% ± 0.3%)	38,435 (+0.0% ± 0.1%)
lb		36.27 (+0.2% ± 0.08%)	25,801 (+0.8% ± 0.2%)	37,635 (-0.1% ± 0.01%)	58,551 (-0.1% ± 0.2%)	39,127 (-0.0% ± 0.1%)
sharding-w2		57.89 (-0.5% ± 0.1%)	56,709 (-1.5% ± 1.0%)	106,001 (-0.0% ± 0.03%)	73,020 (+0.1% ± 0.4%)	43,080 (+0.1% ± 0.1%)	1.382 (+0.2% ± 0.07%)
sharding-w8	144,853 (-0.7% ± 0.6%)	49.63 (-0.1% ± 0.6%)	61,721 (-0.5% ± 0.8%)	61,412 (-0.0% ± 0.04%)	64,462 (+0.3% ± 0.7%)	34,235 (+0.1% ± 0.1%)	0.1587 (+0.7% ± 1.1%)

Counters per txn

suite	frontend stall	l1i refill	backend stall
pooler	77,580 (+0.1% ± 0.4%)	3,809 (+0.2% ± 0.1%)
lb	79,962 (+0.5% ± 0.5%)	3,926 (+1.0% ± 0.2%)
sharding-w2	169,718 (-2.5% ± 1.7%)	6,418 (-0.9% ± 0.5%)
sharding-w8	132,655 (-2.1% ± 1.2%)	5,084 (-1.7% ± 0.2%)

Microbench: all 135 measures within bounds.

Microbench latency: 8 benches changed more than 3%

benchmark	latency	baseline	change	instructions change
extended_cold::delete_by_key	4,195 ns	4,363 ns	-3.84% ± 1.68%	+0.00% ± 0.00%
extended_cold::aggregate	8,684 ns	9,024 ns	-3.76% ± 0.80%	+0.00% ± 0.00%
simple_cold::delete_by_key	4,084 ns	4,236 ns	-3.57% ± 1.40%	+0.00% ± 0.00%
simple_cold::set	2,850 ns	2,950 ns	-3.36% ± 2.18%	+0.00% ± 0.00%
simple_cold::aggregate	8,676 ns	8,952 ns	-3.08% ± 0.61%	+0.00% ± 0.00%
extended_cold::update_by_key	4,777 ns	4,927 ns	-3.03% ± 1.69%	+0.00% ± 0.00%
simple_cold::select_in_list	6,167 ns	6,359 ns	-3.01% ± 1.58%	+0.00% ± 0.00%
extended_cold::select_in_list	6,256 ns	6,450 ns	-3.00% ± 0.84%	+0.00% ± 0.00%

lb flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

microbench flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

pooler flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w2 flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

sharding-w8 flamegraph vs 678843a

Open interactive for hover, zoom and search.

Open interactive.

Logs

lb, microbench, pooler, sharding-w2, sharding-w8, verdict.json

Compared 6e3aa15 with 678843a on c7a.metal-48xl (spot).