feat(sdk): 0.2.0 parity with Arc main + opt-out installation telemetry

CHANGELOG.md

@@ -12,6 +12,70 @@ plus bug fixes and the first real HTTP-layer test coverage.
 
 ### Added
 
+- **Write path (Arc 2027.01.1 parity).**
+  - `ArcPartialWriteError` (`dropped`, `others_written`) for Arc's
+    `400 "partial write"`. It is never retried, and buffered writers discard
+    that batch and report it once, so a resend cannot duplicate stored rows.
+  - `ArcIngestRefusedError` (`reason`, `retry_safe`) for 503 refusals
+    (`X-Arc-Ingest-Fenced`, `X-Arc-Ingest-Refused`). Writes are resent
+    automatically except for `wal_not_durable` and line-protocol bodies
+    spanning several measurements, where Arc re-checks its fence per
+    measurement. A buffered writer keeps a refused batch and stays usable.
+  - `precision=` on `write_line_protocol`/`write_point`.
+  - `tag_columns=` on `write_columnar`/`write_dataframe` and buffered writes,
+    sent as msgpack `tag_keys`.
+- **Query path.**
+  - `profile=True` on `query`/`query_msgpack`, returned as `QueryResponse.profile`.
+  - `dictionary=`/`compression=` Arrow wire options.
+  - `QueryResponse.query_id`, and `arc.query_id` in the Arrow schema metadata,
+    for `client.queries.kill()`.
+  - `client.query.query_measurement()` for `GET /api/v1/query/:measurement`.
+- **Measurement schema**: `databases.get_measurement_schema()` and
+  `rebuild_measurement_schema()`.
+- **Readiness**: `client.readiness()` and `client.write_readiness()` return
+  `ReadyResponse` (`status`, `reason`, `tier`). A 503 is an answer, not an
+  error, and is not retried.
+- **Compaction**:
+  - `compaction.cycles()`, `compaction.cycle(id)`, `compaction.too_wide()`,
+    `compaction.clear_too_wide()` and `history(cycle_id=)` (Arc 2027.01.1+).
+  - `trigger(measurement=)` (Arc 2026.09.2+).
+- **Backup**: `create(databases=[...])`, `list(include_foreign=True)` and
+  `restore(mode="merge"|"replace")` (Arc 2027.01.1+). Restore still requires
+  `confirm=True`.
+- **Delete**: `allow_unreachable=True` on `delete.delete()` and
+  `databases.delete()` (Arc 2027.01.1+). A cluster delete then proceeds
+  despite unreachable files or unflushed peers, leaves those rows in place,
+  and names them in the response.
+- **Continuous queries**: `ExecuteCQResponse.watermark_status` (Arc 2027.01.1+).
+- **Cluster operator controls** (admin):
+  - `delete_file`, `demote_writer`, `assign_compactor`, `converge_voters`
+    (Arc 2026.09.2+).
+  - `stream_claims`, `release_stream_claim`, `abort_compaction_job`,
+    `policies`, `adopt_policies` (Arc 2027.01.1+).
+  - Destructive controls require `confirm=True`. `converge_voters` only
+    plans unless `dry_run=False, confirm=True`.
+- **Reconciliation** (`client.reconciliation`): `status()`, `trigger()` and
+  `get_run()`. `trigger()` is a dry run by default; acting requires
+  `dry_run=False, confirm=True`.
+- **Operational diagnostics** (`client.diagnostics`):
+  - Public JSON metrics, Prometheus text and timeseries.
+  - Admin `logs`, `memstats`, `duckdb_memory` and
+    `free_os_memory(confirm=True)`.
+  - `auth.cache_stats()` and `auth.invalidate_cache()`.
+- **Edge sync administration** (`client.edge_sync`):
+  - Hub: spoke registry and air-gap bundle import.
+  - Spoke: status, ledger, sync, requeue/dismiss, and bundle export/ack.
+  - `rotate_spoke_secret`, `delete_spoke` and `dismiss_failed` require
+    `confirm=True`.
+
+- **Installation ID for Arc's client telemetry.** Requests carry a random
+  per-installation `Arc-Client-Installation-Id` and a `User-Agent` that names
+  the Python version and platform, so Arc's opt-out telemetry can count SDK
+  installations the same way it counts arcli. The SDK itself still talks only
+  to the configured Arc server. Opt out with `DO_NOT_TRACK=1`,
+  `ARC_CLIENT_SEND_INSTALLATION_ID=0` or `send_installation_id=False`; pin the
+  ID in containers with `ARC_CLIENT_INSTALLATION_ID`. See README, "Privacy".
+
 - Arrow completion metadata in the existing `pyarrow.Table` schema for sync and
   async queries: policy cap and execution time are retained; failed, malformed,
   conflicting or unobservable announced completion metadata is rejected.
@@ -59,6 +123,15 @@ plus bug fixes and the first real HTTP-layer test coverage.
 
 ### Changed
 
+- MessagePack queries are stable. A stream that fails after the 200 raises
+  `ArcQueryError` (incomplete body) instead of returning partial rows.
+- `ready()` is a single probe: a 503 "not ready" is no longer retried with backoff.
+- `httpcore` is bounded `>=1.0.9,<1.1` instead of pinned `==1.0.9`.
+- Compaction docstrings state that every `/api/v1/compaction` route, reads
+  included, requires an admin token, as Arc enforces.
+- `User-Agent` now includes the Python version and platform
+  (`arc-client-python/0.2.0 (python/3.12.4; linux/amd64)`).
+
 - Default write compression is now **zstd** (was gzip). Passing
   `compression=True` still maps to gzip for backward compatibility.
 - Writes no longer send a `Content-Encoding` header; Arc auto-detects the codec
@@ -75,6 +148,21 @@ plus bug fixes and the first real HTTP-layer test coverage.
 
 ### Fixed
 
+- Query POSTs (`query`, `query_arrow`, `query_msgpack`, `estimate`) now retry
+  503 with `Retry-After`, which Arc sends during replication catch-up, for a
+  vanished partition, and when the cold tier is unavailable. A governance 429
+  (rate limit or quota) is raised at once. A transport failure mid-response
+  is still not replayed.
+- Query and write clients no longer hide typed errors. Auth, permission,
+  scoped-access, license, rate-limit and not-supported errors surface as
+  themselves.
+- `create_token(permissions=[])` and `update_token(permissions=[])` send the
+  empty list, creating an RBAC-only token, instead of silently producing a
+  read/write token.
+- `HealthResponse` models Arc's real `/health` (`time`, `uptime_sec`,
+  `storage`, `license`).
+- `dataframe_to_columnar(tag_columns=...)` is no longer a no-op.
+
 - MessagePack queries preserve cap metadata and exact timestamp nanoseconds.
   DELETE results retain partial-failure, cold-storage and peer diagnostics.
 - Retention and continuous-query clients accept Arc's list/null/flat response

README.md

@@ -154,6 +154,28 @@ tle = """ISS (ZARYA)
 client.write.write_tle(tle, measurement="satellite_tle")
 ```
 
+### Precision, Tags and Partial Writes
+
+```python
+client.write.write_line_protocol("cpu,host=a v=1 1700000000", precision="s")
+client.write.write_columnar("cpu", cols, tag_columns=["host", "region"])
+client.write.write_dataframe(df, "cpu", tag_columns=["host"])
+
+from arc_client import ArcPartialWriteError, ArcIngestRefusedError
+try:
+    client.write.write_line_protocol(lines)
+except ArcPartialWriteError as e:   # never resend: the rest was stored
+    print(e.dropped, e.others_written)
+except ArcIngestRefusedError as e:  # retries exhausted, or unsafe to resend
+    print(e.reason, e.retry_safe)
+```
+
+Arc refuses writes with a 503 during failover or under disk pressure. The SDK
+resends single-measurement writes that Arc reports as "nothing was written".
+It never resends a partial write, a `wal_not_durable` refusal, or a refusal
+of a line-protocol body spanning several measurements: Arc may already have
+stored part of those.
+
 ## Bulk Import
 
 Import files directly (admin token required). Each returns a typed result with
@@ -260,10 +282,12 @@ DataFrame conversions can drop schema metadata; inspect the Arrow table first
 when completeness matters.
 
 Trailer capture is tested with HTTPX 0.27/0.28, httpcore 1.0.9 and h11 0.16.
-Those dependency bounds are deliberate: the adapter observes HTTP/1.1 parser
-completion without replacing HTTPX's TLS, proxy, timeout or pooling behavior.
+httpcore is bounded `>=1.0.9,<1.1`, because the internals the adapter reads are
+unchanged across 1.0.x. If a release changes them, Arrow responses that
+announce trailers fail with `ArcQueryError` rather than return unverified
+metadata.
 
-### MessagePack Response (experimental)
+### MessagePack Response
 
 A compact columnar MessagePack response with native timestamp decoding. Returns
 the same `QueryResponse` shape as `query()` (row-major `data`), plus per-column
@@ -274,9 +298,9 @@ result = client.query.query_msgpack("SELECT * FROM default.cpu LIMIT 1000")
 print(result.columns, result.types)
 ```
 
-> **Experimental.** This uses `/api/v1/query/msgpack`, which requires the Arc
-> server to be built with the `duckdb_arrow` build tag. Against a server without
-> it, the call raises `ArcNotSupportedError` (HTTP 501).
+This uses `/api/v1/query/msgpack`, stable since Arc 2026.09.1. Older servers
+built without the `duckdb_arrow` tag answer 501, which raises
+`ArcNotSupportedError`.
 
 ### Query Estimation
 
@@ -296,6 +320,25 @@ for m in measurements:
     print(f"{m.measurement}: {m.file_count} files, {m.total_size_mb:.1f} MB")
 ```
 
+### Profiling, Query IDs and Measurement Reads
+
+```python
+r = client.query.query("SELECT * FROM cpu", profile=True)
+print(r.profile.total_ms, r.profile.rows_scanned, r.query_id)
+client.queries.kill(r.query_id)
+
+# Arrow wire encodings; the query ID is in the schema metadata
+t = client.query.query_arrow("SELECT * FROM cpu", dictionary=True, compression="zstd")
+t.schema.metadata[b"arc.query_id"]
+
+# Rows of one measurement without free-form SQL
+client.query.query_measurement("cpu", "metrics", limit=100, order_by="time ASC", where="host = 'a'")
+
+# Stable measurement schema (admin rebuild is queued, 202)
+s = client.databases.get_measurement_schema("metrics", "cpu")
+client.databases.rebuild_measurement_schema("metrics", "cpu")
+```
+
 ## Async Support
 
 All operations have async equivalents:
@@ -450,14 +493,58 @@ client.audit.logs(event_type="query", since="2024-01-01T00:00:00Z", limit=100)
 client.queries.active()
 client.queries.kill("query-id")
 
-# Cluster / compaction / tiering / backup / MQTT
+# Cluster / compaction / tiering / backup / MQTT (admin)
 client.cluster.status()
-client.compaction.trigger(tier="hourly,daily")
+client.cluster.converge_voters()                       # plan only
+client.cluster.converge_voters(dry_run=False, confirm=True)
+client.cluster.demote_writer("writer-1", confirm=True)
+client.cluster.policies(kind="retention")              # on the Raft leader
+
+client.compaction.trigger(database="metrics", measurement="cpu")
+client.compaction.cycles(limit=5)
+client.compaction.too_wide()
+
 client.tiering.set_policy("metrics", hot_only=True, hot_max_age_days=7)
-client.backup.create()
+
+client.backup.create(databases=["metrics"])            # scoped backup
+client.backup.list(include_foreign=True)
+client.backup.restore("backup-id", confirm=True, mode="merge")
+
 client.mqtt.list_subscriptions()
 ```
 
+## Operations and Diagnostics
+
+```python
+# Readiness: should this node get traffic / writes, and if not, why?
+r = client.readiness(); print(r.ready, r.reason, r.tier)
+client.write_readiness().status   # "ready" | "not_ready" | "not_write_target"
+
+# Metrics (public) and logs (admin)
+client.diagnostics.metrics()
+client.diagnostics.timeseries("api", duration_minutes=15)
+client.diagnostics.logs(level="warn", since_minutes=30, limit=200)
+client.auth.cache_stats()
+
+# Reconciliation (enterprise clustering): dry run first, then act
+run = client.reconciliation.trigger()                 # dry run
+client.reconciliation.trigger(dry_run=False, confirm=True)
+
+# Edge sync, on the hub
+secret = client.edge_sync.register_spoke("rocket-01", name="Rocket")
+print(secret.secret)  # shown only once
+client.edge_sync.import_bundle("/mnt/drive/bundle-1")
+
+# Edge sync, on the spoke
+client.edge_sync.spoke_status()
+client.edge_sync.run_sync()          # long-running: use a large client timeout
+client.edge_sync.requeue_failed(all_failed=True)
+```
+
+Destructive or disruptive calls (`reconciliation.trigger(dry_run=False)`,
+`diagnostics.free_os_memory`, `edge_sync.rotate_spoke_secret`,
+`edge_sync.delete_spoke`, `edge_sync.dismiss_failed`) require `confirm=True`.
+
 ## Configuration
 
 ```python
@@ -494,6 +581,8 @@ from arc_client.exceptions import (
     ArcLicenseError,       # Enterprise feature not licensed (403)
     ArcQueryError,         # Query execution errors
     ArcIngestionError,     # Write failures
+    ArcPartialWriteError,  # Some points dropped, others stored (400); never resend
+    ArcIngestRefusedError, # Write refused (503); .reason, .retry_safe
     ArcValidationError,    # Invalid input
     ArcNotFoundError,      # Resource not found (404)
     ArcRateLimitError,     # Rate limited (429)
@@ -509,6 +598,33 @@ except ArcConnectionError as e:
     print(f"Connection failed: {e}")
 ```
 
+## Privacy
+
+The SDK connects only to the Arc server you configure. It never contacts
+Basekick or any other service on its own.
+
+Requests identify the SDK version, the Python version and the OS/architecture
+in `User-Agent`. They also carry a random installation ID
+(`Arc-Client-Installation-Id`), created on first use and stored in
+`~/.config/arc-client-python/installation-id` (`%APPDATA%` on Windows). Arc may
+include this ID in its own opt-out telemetry so Basekick can count SDK
+installations across Arc servers. The ID is not derived from your machine, user
+or token, and nothing about your queries, databases or data is included.
+
+Disable the ID with any of:
+
+```bash
+DO_NOT_TRACK=1
+ARC_CLIENT_SEND_INSTALLATION_ID=0
+```
+
+```python
+ArcClient(host="localhost", token="...", send_installation_id=False)
+```
+
+In ephemeral containers, set `ARC_CLIENT_INSTALLATION_ID` to a fixed UUID so
+each start is not counted as a new installation.
+
 ## Documentation
 
 See the [full documentation](https://docs.basekick.net/arc/sdks/python]) for detailed guides and API reference.
@@ -535,7 +651,7 @@ Both sync and async clients provide the same methods. CQ replacements include
 `tag_columns`; retention replacements include `database`, `measurement`, and `buffer_days`.
 
 Query results retain `truncated`, `truncation_reason`, `rows_capped`, and `row_cap`
-when their wire format supplies them. Experimental `query_msgpack()` returns UTC
+when their wire format supplies them. `query_msgpack()` returns UTC
 `datetime` values when a timestamp is exactly representable at microsecond resolution.
 Submicrosecond or out-of-range timestamps remain `msgpack.Timestamp`, preserving
 `.seconds`, `.nanoseconds`, and `.to_unix_nano()`. Such cells require an explicit

pyproject.toml

@@ -28,7 +28,7 @@ classifiers = [
 
 dependencies = [
     "httpx>=0.27.0,<0.29",
-    "httpcore==1.0.9",
+    "httpcore>=1.0.9,<1.1",
     "h11>=0.16.0,<0.17",
     "msgpack>=1.0.7",
     "pyarrow>=15.0.0",

src/arc_client/__init__.py

@@ -15,9 +15,11 @@
     ArcConnectionError,
     ArcError,
     ArcIngestionError,
+    ArcIngestRefusedError,
     ArcLicenseError,
     ArcNotFoundError,
     ArcNotSupportedError,
+    ArcPartialWriteError,
     ArcPermissionDataUnavailableError,
     ArcPermissionError,
     ArcQueryError,
@@ -41,6 +43,8 @@
     "ArcAuthenticationError",
     "ArcQueryError",
     "ArcIngestionError",
+    "ArcPartialWriteError",
+    "ArcIngestRefusedError",
     "ArcValidationError",
     "ArcNotFoundError",
     "ArcRateLimitError",