[juliannguyen4]

CLIENT-5530: pluggable discovery in the Python client, on the C client's CLIENT-5735-pluggable-discovery branch. This PR also includes its CI test layers (CLIENT-5534, previously #1300) and the L5 driver (CLIENT-5853, previously #1301), so they can be reviewed together.

C client

The submodule points at the C client's CLIENT-5735-pluggable-discovery branch (cceff7db). It has C stage merged in except for b95eb8546, which only changes when simple reads set INFO1_GET_ALL. Move it to a C stage commit once the C branch merges.

That branch builds without changes to the Python client. Since the previous submodule commit, it:

• exports as_cluster_translate_address on Windows and describes ca_path errors with strerror_s there.
• shares one collision check and message between the cluster and a new as_discovery_conformance_collision entry point for the C client's L1 conformance runner. The message is unchanged, so the collision tests still pass. The Python client does not call the C conformance entry points.
• makes the C client's own discovery tests less flaky.

Client config

Two new client config dicts map onto as_config (documented in a new "Discovery and address translation" section of aerospike.rst; validate_keys covers both):

• discovery: provider ("static", "dns", or a SeedCandidateProvider instance), merge_policy, execution_mode, refresh_interval_ms, refresh_timeout_ms, tend_refresh_deadline_ms, and dns (names, record_type, port, tls_name, timeout_ms, servers).
• translation: provider ("static-map", "port-map", "pass-through"), on_miss ("pass_through" or "strict"), static_map, port_map.

ip_map keeps working: it is the map of the static-map translator, as in the C client, and setting it together with translation.static_map or another translator is a ParamError. hosts may be empty or omitted when a seed provider is set. use_shared_connection is rejected with any provider other than static/static-map, since shared clients are looked up by their configured seeds.

Behaviour change from the C client: with use_services_alternate true, a node without an alternate-access-address is no longer added. A seed node without one raises InvalidNodeError (-8), "Node ... has no alternate-access-address and use_services_alternate is true. Configure alternate-access-address on the server or set use_services_alternate to false", and a peer without one is logged as an error. It is documented under use_services_alternate, and test_bool_settings[use_services_alternate] is replaced by test_use_services_alternate, which asks the server for service-clear-alt (or service-tls-alt with TLS) and expects the client to connect if it is set and this error if it is not.

Python seed provider

aerospike_helpers.discovery.SeedCandidateProvider (with SeedCandidateContext) is registered through as_config_set_binding_seed_provider, so it is accepted only with execution_mode "thread" (the C client raises ParamError otherwise).

• The refresh takes the GIL on the C client's discovery thread, so a slow or blocked refresh does not delay tend.
• close() is called once when the client object is destroyed (not by Client.close(), so the client can connect again). If a refresh is still running then, the C client releases the provider on the discovery thread when the refresh returns, and that thread takes the GIL to call close(); nothing is called during interpreter finalization.
• An exception or an invalid return value fails the refresh with its message; with no seed left, connecting raises SeedDiscoveryFailed carrying it.
• aerospike_close() in the destructor now releases the GIL, because close may wait up to refresh_timeout_ms for a refresh that needs it.

There is no Python address translator (the C client has a binding seed provider but no binding translator), and no aws-tags provider (P2; see C client). YAML config is unchanged.

Exceptions

TranslationMiss (-25), TranslationCollision (-26) and SeedDiscoveryFailed (-27), subclasses of ClientError, with stubs and docs. Their messages are the C client's.

Other changes

• error_from_raised_exception() (pending Python exception to as_error) moved to exception.c and replaces the serializer's copy. Deserialization failures from Vector and HyperLogLog now include the exception type, e.g. Unable to deserialize bytes as Vector: ValueError: Vector buffer too short: missing 6-byte header.

CI: Discovery tests workflow

Runs the pluggable discovery test layers from the client libraries Testing Strategy and Pluggable Discovery: Testing a Client Library in CI. Everything that does not run is reported as a visible hole rather than skipped silently.

Layer	Job	What runs
L0	L0 unit tests	test_l0_naming.py (U-11) and test_l0_discovery.py, no server
L2 CE	L2 discovery (ce, ...)	default and reachable (port-map translation, strict miss, DNS seeding, re-seed, a blocked provider refresh), port-map-range (pre-filled map, out-of-range nodes), pass-through-names (N-15), pass-through-ports on a released server and on a server that advertises access addresses verbatim (N-16 with its NFR-1 scale-out timing, N-18, and the L5 driver smoke test), and the L2c fault profiles idle-reset (F-1), low-mtu (F-2) and port-range (F-3), with F-4 on default
L2 EE	L2 discovery (ee, ...)	TLS through the proxy: default (N-6) and cn-only (N-17, a strict expected failure). Needs FEATURES_CONTENT
L3	L3 discovery (k8s)	K-2 and K-3 (scale up and down under a running client), K-9 (NodePort stability across pod rescheduling and recreation) and K-10 (T3 variant: each pod advertises as-endpoint.test with its own Service NodePort), on kind and AKO
L3 TLS	L3 discovery (k8s, TLS)	K-6: every pod serves TLS from one shared certificate for the cluster tls-name, and the client validates each node against that tls-name
—	Skipped layers	A report-skipped-layers row for every layer and profile that does not run, with the reason

L1 does not run. Its shared fixtures test translation, seed providers, the merge policy and TLS name matching, which are all C client built-ins, so the C client runs them (TG-4). The Python client has no entry point of its own for any of them; running the fixtures against the C client from Python would only test C code.

How it works:

• build reuses build-and-optionally-test-wheels-on-one-platform.yml for one manylinux_x86_64 wheel on Python 3.12, which every layer installs.
• L2 uses the conformance repo's discovery-l2-topology action. The tests run in a python:3.12-slim-bookworm container on the client network, which has no route to the node addresses, with the topology's env file and --add-host entries. The workspace is mounted at the same path, so the tests reach the node server's control directory.
• L2 talks to each node through its Toxiproxy listener with a raw info request, and asserts routing from the server's own statistics: each node serves exactly the keys it masters, with no proxied commands.
• L2c adds Toxiproxy toxics through the topology's TOXIPROXY_URL: reset_peer for F-1 and a timeout toxic that holds data with no reply for F-4. The low-mtu and port-range profiles set the client network's MTU and the proxy's source port range.
• L3 uses setup-kind-ako from shared-workflows v5.1.0 and runs on the runner, reading the cluster through kubectl with the action's kubeconfig (test/discovery/l3_cluster.py). The L3 TLS job sets the action's enable-tls input, which generates a CA and a certificate whose only SAN is the tls-name, and starts the AerospikeCluster serving TLS from it. That input is in aerospike/shared-workflows#405, so the TLS job is pinned to that PR's commit until a release carries it. Both jobs need FEATURES_CONTENT; without it they are skipped and reported.
• test/discovery/conftest.py deselects L2 tests written for another profile (L2_PROFILE comes from the topology's env file), skips L2 and L3 tests with no topology, and with DISCOVERY_FAIL_ON_SKIP=1, which every job sets, turns a skipped test into a failure, so a misconfigured job cannot pass silently.
• The conformance repo's actions are pinned to one commit.

The C client moves the node it seeds through the L3 ingress port to the address the node advertises, so K-10 asserts that every node, the seeded one included, is held at its advertised address. Java keeps the seeded node at the seed.

Not run, and reported by Skipped layers:

• L0 U-2, U-8, U-10, U-13 and U-15 (translator contracts) and U-12 (max_socket_idle default): C client built-ins, so the C client suite owns them.
• L1: the shared fixtures test C client built-ins, so the C client runs them.
• L2 N-13 (map refresh): deferred, since TranslationMapProvider is not in this release.
• L2 CE unresolved-access: it checks how the server starts and runs no client.
• L2b M-1 to M-4 and M-6 to M-12 (thread-model harness): the discovery scheduler is a C client built-in, so the C client suite owns them. M-5 runs in L2.
• TLS certificate matching rules: TLS name matching is a C client built-in, so the C client suite owns it. N-6, N-17 and K-6 cover the handshake through the Python client.
• L2 EE and L3, when FEATURES_CONTENT is absent.

Tests

• L0 (test_l0_discovery.py, no server), ordered by ID to match the testing strategy:
  • U-1: a Python provider that returns no hosts, raises or blocks past refresh_timeout_ms leaves the client on its configured seeds, shown by the client's warning for its configured seed; a periodic refresh keeps running after errors; needs_periodic_refresh() must return a bool. Also the provider context, invalid hosts, SeedDiscoveryFailed with no seed host left, tend-mode rejection, and close() running once (including after a refresh that outlives the client).
  • U-3: default construction starts no discovery thread (a periodic provider adds exactly one).
  • U-4: the binding accepts the dns provider's config, and record_type in either case.
  • U-5: every documented name of merge_policy, execution_mode, discovery.provider, translation.provider and on_miss is accepted.
  • U-7: ip_map and translation.static_map accept and reject the same input.
  • Without an ID: exception classes and config validation.
  • The translators themselves (U-2, U-7's legacy parity, U-8, U-9, U-10, U-13, U-15), the DNS resolver (U-4) and the merge itself (U-5) are covered by the C client's tests, so L0 only tests the Python client's own code.
• L2 (test_l2_translation.py, default, reachable and port-map-range profiles), ordered by ID:
  • N-1: without a translator, init fails with Peers not reachable naming every advertised endpoint.
  • N-2 and N-5: port-map by endpoint and by node name; every node joins at its listener and serves its own keys with no proxying.
  • N-4: the dns provider seeds from the endpoint's SRV records, served by setup-dns-server.
  • N-7: a node whose listener goes away drops out, commands for its keys fail, and it rejoins when the listener returns.
  • N-8: strict miss; the victim is never a node, the miss is logged every tend with C's message, and keys on mapped masters work.
  • N-10: an unmapped reachable node passes through at its advertised address with no miss logged.
  • N-11 and N-19 (port-map-range): a node started into the pre-filled range joins within two tend intervals of the later of the peers-generation bump and the node answering through its range listener; a node outside the range is decided by on_miss (strict logs the miss, pass-through tries the advertised address).
  • N-12: with every node gone, the client requests one urgent refresh from a thread-mode provider and re-seeds within two tend intervals plus the connect timeout. Tend mode rejects Python providers, so only the thread-mode variant runs.
  • N-14: use_services_alternate against a seed with no alternate-access-address fails init with InvalidNodeError.
  • N-18: two nodes mapped to the same address create no node, and the collision is logged.
  • M-5: commands keep working while a refresh is blocked, the next refresh sees every tend that ran meanwhile, and close() returns after refresh_timeout_ms while a refresh is blocked.
• L2 (test_l2_pass_through.py): N-15 on pass-through-names, and on pass-through-ports N-16 with translation: {"provider": "pass-through", "on_miss": "strict"} plus its scale-out step (active within two tend intervals of the later of the bump and the new node answering through its listener), and N-18 for two nodes sharing a listener.
• L2 (test_l2_tls.py, EE): N-6 validates each node against its own tls-name through the proxy, with no legacy-name fallback, and rejects a seed tls-name of the endpoint hostname; N-17 expects a CN-only certificate to be rejected. The C client still accepts a CN-only certificate through its deprecated legacy name check (FR-45) until the next major release, so N-17 is marked as a strict expected failure. It is reported as xfailed only when the handshake is accepted with the deprecation warning logged; any other failure fails it, and once the C client rejects the certificate it passes unexpectedly and fails, so the mark gets removed. conftest.py's fail-on-skip hook does not count expected failures as skips.
• L2c (test_l2_faults.py), ordered by ID. The client is the sync client, so these are the sync variants of the rows:
  • F-1 (idle-reset): every listener resets its connections with RST and is cycled while the client's connections sit idle in the pool. A raw probe connection per listener shows the RST, each node's closed connection count shows the client dropped the reset sockets, and no command fails to the caller.
  • F-2 (low-mtu): with the client network's MTU at 1200, a 64 KiB and a 1 MiB value write and read through every node's listener within a bound from socket_timeout and total_timeout, or fail with an error that is not a timeout.
  • F-3 (port-range): more concurrent commands than the proxy has source ports, with max_conns_per_node above that, fail with client errors within the timeout bound, never with a timeout. The client counts the errors, and commands succeed again once the load drops.
  • F-4 (default), two tests: a listener that holds data with no reply makes commands to its node time out within the timeout bound, without blocking a thread, and the node serves again once the toxic is removed. With max_error_rate and error_rate_window both 4, the timeouts trip the node's circuit breaker within the window, and later commands fail fast with MaxErrorRateExceeded. Java's 2 and 4 would not work here: the C client resets both to the defaults when their ratio is below 1. The test also sets timeout_delay to 0, since by default the C client hands a timed-out connection to recovery, and a timeout counts toward max_error_rate only when the connection is closed.
• L3 (test_l3_scale.py, test_l3_shared_endpoint.py and test_l3_tls_name.py), ordered by ID:
  • K-2: a client seeded once holds the pod the operator adds, at the endpoint hostname and the pod's own Service NodePort, within two tend intervals of the later of the peers-generation bump and the new node accepting a login through its Service NodePort, and keeps the nodes it held. A new node rejects the credentials with AEROSPIKE_INVALID_CREDENTIAL for about 3 seconds after it turns Ready and answers, so the budget starts at the first successful login.
  • K-3: with reads and writes running, the removed pod leaves the client's nodes, and once the remaining nodes are stable, errors, retries and failed commands stay flat over five tend intervals.
  • K-6 (L3 TLS job): the shared certificate's only dNSName is the tls-name, with no SAN for the endpoint. Every pod advertises the tls-name for its peers and its TLS Service NodePort as its TLS alternate address. The client holds every node there, with no legacy-name fallback, and a seed tls-name of the endpoint hostname fails the handshake.
  • K-9: a pod's Service keeps its UID and NodePort when the pod is rescheduled, and is recreated with a new NodePort when the pod is removed and added again. A NodePort Service pinned to pod 0 keeps its port throughout, and a client seeded only through it reaches every node.
  • K-10: every node is held at its advertised address and serves the keys it masters. K-10 reads the live NodePorts, since K-9 changes them.
  • The client exposes no partition map, so where Java checks that every partition is owned, these tests check from the nodes' own statistics that every node serves the keys it masters, once the cluster reports itself stable.

L5 driver

The Python client's side of the pluggable discovery L5 rows (W-1, W-2, W-3, W-5), which run on AWS through the l5-aws harness in aerospike-client-conformance.

• test/discovery/l5_driver.py: a driver to contract version 1 of l5-aws/DRIVER.md, modelled on the Java L5Driver. It builds a client from the harness's flags, runs the retrying load, writes ready, snapshot, log, served, watch, reset and error lines, and answers serve, watch, reset and quit. The harness's rows.py makes every assertion. --shape port-map builds the port-map translator (with use_services_alternate off), and --on-miss is honoured.
• test/discovery/l5-aws.manifest: L5_SETUP installs the toolchain on Amazon Linux 2023, L5_BUILD initializes the C client submodule (the harness clones without it) and builds and installs a wheel from the checked-out commit with python -m build, and L5_DRIVER runs the driver. .gitignore gets an exception, since *.manifest is ignored for PyInstaller.
• test/discovery/test_l2_l5_driver.py: the pass-through-ports L2 jobs, which seed one hostname with a port per node as L5 does, run the driver and check that it routes every node, serves all 4096 partitions, answers a watch and exits 0. L5 runs by hand, so this keeps the driver from breaking unnoticed between runs.

What the driver cannot report:

• Partition map: the client does not expose it, so snapshots leave out unowned, owners and longestUnownedMs, and watch replies ownershipChecked: false. The contract allows this; the harness then asserts on serve and command latency instead.
• active: the client does not expose a node's active flag. Every node it holds is reported active, since the C client deactivates a node in the same tend that removes it from the node list.

CLIENT-5853's done-when is "each row runs in the weekly job and passes". No weekly job exists yet: the strategy page (§3, §10, TG-6) lists L5 as run by hand with a weekly schedule planned and unowned, and run.sh runs from an operator's machine with SSO credentials and creates billable resources in the shared account. The AWS run itself is pending a go-ahead.

Testing

• python3 -m build from a clean C build: no warnings in the Python client sources.
• L0 passes locally with no server. actionlint (with shellcheck) and flake8 pass.
• Info, partition-ownership and served-by-owner helpers checked against a local server, including a wrong owner map failing.
• The L2 translation tests run locally against a three-node CE cluster on a Docker network whose private addresses the host cannot reach, each node published on its own port (the default profile's situation): N-2, N-8 (10 miss warnings, 192 keys on mapped masters, 108 on the victim's), M-5 (tend went from iteration 1 to 16 during a 3 s block; close returned in 1.0 s), then N-10 with one node advertising its published address, and N-16 with every node advertising localhost and its own port.
• In ci: CLIENT-5534 add pluggable discovery L0-L3 test workflow #1300's CI run, L2 and L3 passed: every node was held at its advertised hostname:listener port and served exactly the keys it masters, with no proxied commands; the N-16 scale-out node was active 84 to 205 ms after the peers-generation bump, under the 500 ms budget; in L3, both pods were held at as-endpoint.test with their own Service NodePorts.
• The L5 driver:
  • DRIVER.md's local check against a single server: ready, heartbeats 98-103 ms apart (also during serve), served 4096/4096, watch usable in 6 ms, error replies for malformed commands and a second watch, exit 0; every line passes rows.py's contract_problem. Exit codes: 2 for bad flags and for --user without AEROSPIKE_ADMIN_PASSWORD, 3 for an unreachable seed, 0 at end of input.
  • A server paused for 4 s under load: failures counted with lastError, the node reported unreachable, a WARN log line forwarded, then recovery and serve 4096/4096.
  • The manifest replayed as run.sh runs it, in amazonlinux:2023 as a sudo user: setup 10 s, build 47 s, then the installed driver served 4096/4096.
  • rows.py with the stub l5ctl from its test_rows.py, against a local three-node CE cluster that advertises one address with a port per node: W-1 pass-through passes routing, node-port, node-host and serve; W-2, with container restarts standing in for the Spot interruption and the rollout, passes (every node routed 200 ms after the cluster was ready, quiet 1.5 s later); W-3, with a fourth node advertised before a forwarder serves, passes (the new node was never held while unreachable, and was usable 59 ms after its listener served, bound 500 ms). W-5 needs per-zone endpoint addresses and was not simulated.
  • With --shape port-map against the three-node cluster: ready, every node at its listener, about 29k commands with no failure.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 17.77344% with 421 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.79%. Comparing base (a3b90b3) to head (b1e0a6c).

Files with missing lines	Patch %	Lines
src/main/discovery.c	11.13%	383 Missing ⚠️
aerospike_helpers/discovery/__init__.py	0.00%	19 Missing ⚠️
src/main/client/type.c	56.25%	14 Missing ⚠️
src/main/client/connect.c	25.00%	3 Missing ⚠️
src/main/exception.c	91.30%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            stage    #1303      +/-   ##
==========================================
- Coverage   86.03%   83.79%   -2.24%     
==========================================
  Files         104      106       +2     
  Lines       15577    16016     +439     
==========================================
+ Hits        13401    13420      +19     
- Misses       2176     2596     +420     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.