Skip to content

fix: findings from live testing 0.2.0 against Arc main (single node, edge sync, cluster) - #30

Merged
xe-nvdk merged 12 commits into
mainfrom
fix/live-test-findings
Nov 2, 2026
Merged

xe-nvdk merged 12 commits into
mainfrom
fix/live-test-findings

Conversation

@xe-nvdk

@xe-nvdk xe-nvdk commented Nov 2, 2026

Copy link
Copy Markdown
Member

Ran the 0.2.0 SDK against real Arc builds from main (1943fe87, the 2027.01.1 code), fixed what broke, and checked in the live suites so they can be re-run against any build.

Setups tested

Setup What it covered
Single node: native binary, auth on, enterprise dev license Delete, governance, query management, audit, reconciliation, retention, CQ, compaction, backup
Edge sync: hub and spoke as separate processes Register, rotate, enable/disable, sync, bundle export/import
Cluster: 3 writers and 1 reader Docker, built from source

SDK bugs found and fixed

Bug Fix
query_measurement retried Arc's governance 429 (5 requests, ~3 s) Retries 503 only, like the query POSTs
register_spoke(spoke_id) always failed: Arc refuses the default empty name name defaults to spoke_id
reconciliation.get_run and spoke status/ledger retried a 503 that won't change (no reconciliation / no edge-sync transport), stalling seconds before raising Raise at once
A non-dry-run delete could be reported as a failure after it ran. Arc may wait 30 s inside a delete (deleteLocalWait, peer flush), equal to the default client timeout. The caller got "Failed to connect" and lost unreachable_files. Row and database deletes get a 90 s floor; post/delete accept a per-request timeout
A read timeout said "Failed to connect" Now says the request was sent and may have been applied
cluster.remove_node returned Arc's 200 {"success": false} (no clustering) as if the node had been removed Raises ArcError; the id is escaped
The pandas/polars ImportError said pip install arc-client[...] Names arc-tsdb-client

Each fix has a unit test.

Docs corrected from observed behavior

  • Arrow dictionary-encoding thresholds.
  • The CQ execute end_time limit, and watermark_status on standalone.
  • Spoke ledger discovery.
  • What a permissions=[] token can't do: run SQL.
  • Data deletes must go to the primary writer.
  • assign_compactor's extra 409.

Live suites (opt-in, skipped without env)

  • tests/integration/test_live_parity.py: single node plus edge sync. 75 passed, 6 edge-sync skips when hub/spoke aren't running, and 2 xfail(strict=True) for the Arc bugs below.
  • tests/integration/test_live_cluster.py: 42 passed, plus 1 opt-in remove_node, which passed when enabled. One flaky fixture is fixed: it removed a file from only one node, and Arc's file puller healed it.
  • The run commands are in each module's docstring.

Arc issues filed

Not exercised

Checks

1087 unit + integration tests pass without env; ruff and mypy clean.

Exercises every new or changed SDK method against a real Arc built from
main: query/profile/query_id/Arrow options, governance 429, precision,
tag_columns, partial writes (LP, msgpack rows, buffered), health and
readiness, schema, deletes, auth, compaction, backup, CQ, retention,
reconciliation, diagnostics, cluster on standalone, RBAC, audit, query
management, edge sync hub/spoke/bundle, and the installation-id header.

Skipped unless ARC_TEST_URL is set; extra gates for licensed targets,
local storage access and edge-sync hub/spoke.
Found running the SDK against an Arc main build (1943fe87):
- query_measurement retried Arc's governance 429 as an ordinary GET (5
  requests, ~3s) before raising. It now retries 503 only, like the query
  POSTs.
- register_spoke(spoke_id) always failed: the default name "" is refused
  by Arc ("spoke name is required"). name now defaults to spoke_id.
- reconciliation.get_run and the spoke status/ledger reads retried a
  permanent 503 (no reconciliation / no edge-sync transport on this
  node) with backoff before raising. They now raise at once.
Arrow dictionary-encoding thresholds, the CQ execute end_time limit and
watermark_status on standalone, spoke ledger discovery, and what an
RBAC-only token (permissions=[]) can and cannot do.
…stly

Found on a live 4-node cluster (Arc main 1943fe87):
- A non-dry-run delete can wait 30 s inside Arc (deleteLocalWait, and the
  database delete's peer flush), equal to the default client timeout, so
  a delete that ran was reported as a connection failure and its
  unreachable_files were lost. Row and database deletes now use at least
  90 s; post/delete take a per-request timeout.
- A read timeout or a connection lost mid-response no longer says 'Failed
  to connect': the request was sent and may have been applied.
- cluster.remove_node raised nothing when Arc answered 200 with
  success=false (no clustering); it now raises ArcError and escapes the id.
- Docs: data deletes must go to the primary writer (others refuse with
  503, unlike writes); assign_compactor's 409 when no compactor lease is
  managed.
Remove the file from every node, not just the primary: Arc's delete waits
up to deleteLocalWait for listed files and the file puller heals a copy
missing on one node from any peer, so the old fixture raced that pull.
@xe-nvdk
xe-nvdk merged commit 1474b29 into main Nov 2, 2026
12 checks passed
@xe-nvdk
xe-nvdk deleted the fix/live-test-findings branch November 2, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant