Skip to content

feat(backup): report cluster restore modes and integrity #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
efegokdemir wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(backup): report cluster restore modes and integrity #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,8 @@ arcli logs --level warn --since 6h

`backup restore` is destructive and leaves metadata restoration to Arc's defaults: enabled on standalone nodes when present in the backup, skipped on cluster nodes. Use `--with-metadata` to request it explicitly (clusters refuse it), `--data-only` to explicitly skip metadata and config, or `--with-config` to also request config restoration. Stop writers and compaction before restoring. When the server reports staged metadata or a required restart, wait for the restore to complete before restarting.

Use `arcli backup restore --mode merge|replace` to choose how cluster data is restored. If omitted, Arc uses merge for full backups and may use replace for scoped backups on a cluster. `backup list` reports cluster-manifest gaps, while `backup show` includes samples and distinguishes missing data from files excluded or reconciled during the backup.

The command tree also covers API tokens, continuous queries, schedulers, predicate deletes, backups, cluster membership, and compaction. Run `arcli --help` or `arcli <command> --help` for the complete command reference.

`license status` reads public `/health` without sending a token or checking authentication. It works on standalone and clustered deployments and supports table and JSON output. It reports the server's tier, status, source, and optional expiry, days remaining, and site-license state. Expiry is normalized to UTC RFC3339 with fractional seconds when provided.
Expand Down
2 changes: 2 additions & 0 deletions docs/releases/unreleased.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,5 @@
- All successfully read license states exit 0. Configuration, request, decoding, and output failures exit nonzero. No token or database header is sent to `/health`, and no authentication verification is performed. Normal connection selection, timeout, TLS verification, and identity opt-out settings apply.

License installation and key/file configuration remain server responsibilities. Existing `ping -o json` license pass-through is unchanged. Closes [arcli#34](https://github.com/Basekick-Labs/arcli/issues/34).

- Backup commands support explicit `--mode merge|replace` restores and report the effective mode returned by Arc. Backup list/show and restore status now include cluster-manifest counts and samples; reconciled skips are excluded from incomplete counts, while unregistered and mid-run manifest changes remain informational. Part of [arcli#44](https://github.com/Basekick-Labs/arcli/issues/44).
114 changes: 70 additions & 44 deletions internal/client/backup.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,22 +38,26 @@ func (e *BackupBusyError) Error() string {

// BackupSummary is one entry of GET /api/v1/backup/.
//
// The three incompleteness counts arrive from Arc 26.09.3 on and are absent
// (zero) from older servers; each is its own population on the server:
// SkippedFiles are data files inside TotalFiles that were not stored,
// SkippedMetadataFiles are Iceberg metadata or compaction recovery state
// (not inside TotalFiles), UnaddressableFiles were never inventoried because
// no listing can return their key.
// Older Arc servers omit the incompleteness counts. SkippedFiles are data
// files inside TotalFiles that were not stored, SkippedMetadataFiles are
// Iceberg metadata or compaction recovery state (not inside TotalFiles), and
// UnaddressableFiles were never inventoried because no listing can return
// their key. Cluster manifest counts arrive on Arc 27.01.1+.
type BackupSummary struct {
BackupID string `json:"backup_id"`
CreatedAt time.Time `json:"created_at"`
BackupType string `json:"backup_type"`
TotalFiles int64 `json:"total_files"`
TotalBytes int64 `json:"total_size_bytes"`
DatabaseCount int `json:"database_count"`
SkippedFiles int64 `json:"skipped_files,omitempty"`
SkippedMetadataFiles int64 `json:"skipped_metadata_files,omitempty"`
UnaddressableFiles int64 `json:"unaddressable_files,omitempty"`
BackupID string `json:"backup_id"`
CreatedAt time.Time `json:"created_at"`
BackupType string `json:"backup_type"`
TotalFiles int64 `json:"total_files"`
TotalBytes int64 `json:"total_size_bytes"`
DatabaseCount int `json:"database_count"`
SkippedFiles int64 `json:"skipped_files,omitempty"`
SkippedMetadataFiles int64 `json:"skipped_metadata_files,omitempty"`
UnaddressableFiles int64 `json:"unaddressable_files,omitempty"`
SkippedReconciled int64 `json:"skipped_reconciled,omitempty"`
UnregisteredSkipped int64 `json:"unregistered_skipped,omitempty"`
ManifestOnlyFiles int64 `json:"manifest_only_files,omitempty"`
LeftManifestDuringRun int64 `json:"left_manifest_during_run,omitempty"`
ClusterManifestChecked *bool `json:"cluster_manifest_checked,omitempty"`
}

// BackupMeasurement / BackupDatabase / BackupManifest mirror
Expand Down Expand Up @@ -81,24 +85,32 @@ type BackupIcebergWarehouse struct {
}

type BackupManifest struct {
Version string `json:"version"`
BackupID string `json:"backup_id"`
CreatedAt time.Time `json:"created_at"`
BackupType string `json:"backup_type"`
Databases []BackupDatabase `json:"databases"`
TotalFiles int64 `json:"total_files"`
TotalSizeBytes int64 `json:"total_size_bytes"`
SkippedFiles int64 `json:"skipped_files,omitempty"`
SkippedMetadataFiles int64 `json:"skipped_metadata_files,omitempty"`
SkippedSample []string `json:"skipped_sample,omitempty"`
SkippedOverlongKeys int64 `json:"skipped_overlong_keys,omitempty"`
UnaddressableFiles int64 `json:"unaddressable_files,omitempty"`
UnaddressableSample []string `json:"unaddressable_sample,omitempty"`
IcebergWarehouse *BackupIcebergWarehouse `json:"iceberg_warehouse,omitempty"`
HasMetadata bool `json:"has_metadata"`
HasIcebergCatalog bool `json:"has_iceberg_catalog,omitempty"`
HasConfig bool `json:"has_config"`
Raw json.RawMessage `json:"-"`
Version string `json:"version"`
BackupID string `json:"backup_id"`
CreatedAt time.Time `json:"created_at"`
BackupType string `json:"backup_type"`
Databases []BackupDatabase `json:"databases"`
TotalFiles int64 `json:"total_files"`
TotalSizeBytes int64 `json:"total_size_bytes"`
SkippedFiles int64 `json:"skipped_files,omitempty"`
SkippedMetadataFiles int64 `json:"skipped_metadata_files,omitempty"`
SkippedSample []string `json:"skipped_sample,omitempty"`
SkippedOverlongKeys int64 `json:"skipped_overlong_keys,omitempty"`
UnaddressableFiles int64 `json:"unaddressable_files,omitempty"`
UnaddressableSample []string `json:"unaddressable_sample,omitempty"`
SkippedReconciled int64 `json:"skipped_reconciled,omitempty"`
UnregisteredSkipped int64 `json:"unregistered_skipped,omitempty"`
UnregisteredSample []string `json:"unregistered_sample,omitempty"`
ManifestOnlyFiles int64 `json:"manifest_only_files,omitempty"`
ManifestOnlySample []string `json:"manifest_only_sample,omitempty"`
LeftManifestDuringRun int64 `json:"left_manifest_during_run,omitempty"`
LeftManifestSample []string `json:"left_manifest_sample,omitempty"`
ClusterManifestChecked *bool `json:"cluster_manifest_checked,omitempty"`
IcebergWarehouse *BackupIcebergWarehouse `json:"iceberg_warehouse,omitempty"`
HasMetadata bool `json:"has_metadata"`
HasIcebergCatalog bool `json:"has_iceberg_catalog,omitempty"`
HasConfig bool `json:"has_config"`
Raw json.RawMessage `json:"-"`
}

// BackupProgress is GET /api/v1/backup/status when an operation has run
Expand All @@ -125,10 +137,20 @@ type BackupProgress struct {
UnaddressableSample []string `json:"unaddressable_sample,omitempty"`
// Restore only: what the restored backup already lacked when it was taken,
// and warehouse files left out because this node has no outside-root
// Iceberg warehouse.
BackupSkippedFiles int64 `json:"backup_skipped_files,omitempty"`
BackupUnaddressableFiles int64 `json:"backup_unaddressable_files,omitempty"`
IcebergWarehouseFilesSkipped int64 `json:"iceberg_warehouse_files_skipped,omitempty"`
// Iceberg warehouse. Newer Arc versions also report the selected mode,
// cluster registration outcomes, and files replaced by the restore.
BackupSkippedFiles int64 `json:"backup_skipped_files,omitempty"`
BackupUnaddressableFiles int64 `json:"backup_unaddressable_files,omitempty"`
IcebergWarehouseFilesSkipped int64 `json:"iceberg_warehouse_files_skipped,omitempty"`
Mode string `json:"mode,omitempty"`
FilesRegistered int64 `json:"files_registered,omitempty"`
RegistrationFailed int64 `json:"registration_failed,omitempty"`
RegistrationFailedSample []string `json:"registration_failed_sample,omitempty"`
SidecarMismatches int64 `json:"sidecar_mismatches,omitempty"`
SidecarMismatchSample []string `json:"sidecar_mismatch_sample,omitempty"`
ReplacedFiles int64 `json:"replaced_files,omitempty"`
BackupUnregisteredSkipped int64 `json:"backup_unregistered_skipped,omitempty"`
BackupManifestOnlyFiles int64 `json:"backup_manifest_only_files,omitempty"`
}

// BackupStatus is the decoded status endpoint: Idle when the server has
Expand All @@ -147,11 +169,13 @@ type BackupCreateOptions struct {
}

// RestoreOptions is the body of POST /api/v1/backup/restore. Confirm is
// always sent true by the client; the command layer owns the prompt.
// always sent true by the client; the command layer owns the prompt. A nil
// Mode leaves the effective mode to Arc, including its scoped-backup default.
type RestoreOptions struct {
RestoreData *bool
RestoreMetadata *bool
RestoreConfig *bool
Mode *string
}

// RestoreStarted is the 202 body of POST /api/v1/backup/restore.
Expand All @@ -160,6 +184,7 @@ type RestoreStarted struct {
Message string `json:"message"`
BackupID string `json:"backup_id"`
Status string `json:"status"`
Mode string `json:"mode,omitempty"`
RestartRequired bool `json:"restart_required"`
Staged bool `json:"staged"`
Raw json.RawMessage `json:"-"`
Expand Down Expand Up @@ -276,12 +301,13 @@ func (c *Client) RestoreBackup(ctx context.Context, id string, opts RestoreOptio
return nil, err
}
req := struct {
BackupID string `json:"backup_id"`
RestoreData *bool `json:"restore_data,omitempty"`
RestoreMetadata *bool `json:"restore_metadata,omitempty"`
RestoreConfig *bool `json:"restore_config,omitempty"`
Confirm bool `json:"confirm"`
}{id, opts.RestoreData, opts.RestoreMetadata, opts.RestoreConfig, true}
BackupID string `json:"backup_id"`
RestoreData *bool `json:"restore_data,omitempty"`
RestoreMetadata *bool `json:"restore_metadata,omitempty"`
RestoreConfig *bool `json:"restore_config,omitempty"`
Mode *string `json:"mode,omitempty"`
Confirm bool `json:"confirm"`
}{id, opts.RestoreData, opts.RestoreMetadata, opts.RestoreConfig, opts.Mode, true}
var out RestoreStarted
body, err := c.backupJSON(ctx, http.MethodPost, backupBase+"restore", req, 64<<10, &out)
if err != nil {
Expand Down
26 changes: 19 additions & 7 deletions internal/client/backup_incomplete_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,34 +15,43 @@ func TestBackup_IncompleteShapes(t *testing.T) {
cli, _ := newAuthTestClient(t, func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/v1/backup/":
_, _ = w.Write([]byte(`{"backups":[{"backup_id":"backup-20260917-000000-aaaaaaaa","created_at":"2026-09-17T00:00:00Z","backup_type":"full","total_files":21,"total_size_bytes":2086,"database_count":1,"skipped_files":3,"skipped_metadata_files":1,"unaddressable_files":2},{"backup_id":"backup-20260916-000000-bbbbbbbb","created_at":"2026-09-16T00:00:00Z","backup_type":"full","total_files":20,"total_size_bytes":2000,"database_count":1}],"count":2}`))
_, _ = w.Write([]byte(`{"backups":[{"backup_id":"backup-20260917-000000-aaaaaaaa","created_at":"2026-09-17T00:00:00Z","backup_type":"full","total_files":21,"total_size_bytes":2086,"database_count":1,"skipped_files":3,"skipped_metadata_files":1,"unaddressable_files":2,"skipped_reconciled":1,"unregistered_skipped":4,"manifest_only_files":2,"left_manifest_during_run":3,"cluster_manifest_checked":true},{"backup_id":"backup-20260916-000000-bbbbbbbb","created_at":"2026-09-16T00:00:00Z","backup_type":"full","total_files":20,"total_size_bytes":2000,"database_count":1}],"count":2}`))
case "/api/v1/backup/backup-20260917-000000-aaaaaaaa":
_, _ = w.Write([]byte(`{"version":"26.09.3","backup_id":"backup-20260917-000000-aaaaaaaa","created_at":"2026-09-17T00:00:00Z","backup_type":"full","databases":[{"name":"mydb","measurements":[{"name":"cpu","file_count":21,"size_bytes":2086}],"file_count":21,"size_bytes":2086}],"total_files":21,"total_size_bytes":2086,"skipped_files":3,"skipped_metadata_files":1,"skipped_sample":["mydb/cpu/2026/09/17/00/a.parquet","mydb/cpu/2026/09/17/00/b.parquet","` + long + `","mydb/cpu/metadata/00003-5f2c.metadata.json"],"skipped_overlong_keys":1,"unaddressable_files":2,"unaddressable_sample":["mydb/cpu/2026/09/17/00/.hidden.parquet","mydb/cpu/2026/09/17/00/bad key.parquet"],"iceberg_warehouse":{"path":"/srv/wh","file_count":4,"size_bytes":100,"skipped_files":1},"has_metadata":true,"has_config":false}`))
_, _ = w.Write([]byte(`{"version":"26.09.3","backup_id":"backup-20260917-000000-aaaaaaaa","created_at":"2026-09-17T00:00:00Z","backup_type":"full","databases":[{"name":"mydb","measurements":[{"name":"cpu","file_count":21,"size_bytes":2086}],"file_count":21,"size_bytes":2086}],"total_files":21,"total_size_bytes":2086,"skipped_files":3,"skipped_metadata_files":1,"skipped_sample":["mydb/cpu/2026/09/17/00/a.parquet","mydb/cpu/2026/09/17/00/b.parquet","` + long + `","mydb/cpu/metadata/00003-5f2c.metadata.json"],"skipped_overlong_keys":1,"unaddressable_files":2,"unaddressable_sample":["mydb/cpu/2026/09/17/00/.hidden.parquet","mydb/cpu/2026/09/17/00/bad key.parquet"],"skipped_reconciled":1,"unregistered_skipped":4,"unregistered_sample":["mydb/cpu/unregistered.parquet"],"manifest_only_files":2,"manifest_only_sample":["mydb/cpu/manifest-only.parquet"],"left_manifest_during_run":3,"left_manifest_sample":["mydb/cpu/left-manifest.parquet"],"cluster_manifest_checked":true,"iceberg_warehouse":{"path":"/srv/wh","file_count":4,"size_bytes":100,"skipped_files":1},"has_metadata":true,"has_config":false}`))
case "/api/v1/backup/status":
_, _ = w.Write([]byte(`{"operation":"restore","backup_id":"backup-20260917-000000-aaaaaaaa","status":"failed","total_files":17,"processed_files":16,"skipped_files":1,"unaddressable_files":1,"unaddressable_sample":["backup-20260917-000000-aaaaaaaa/data/mydb/cpu/2026/09/17/00/.hidden.parquet"],"skipped_sample":["backup-20260917-000000-aaaaaaaa/data/mydb/cpu/2026/09/17/00/c.parquet"],"missing_files":1,"backup_skipped_files":3,"backup_unaddressable_files":2,"iceberg_warehouse_files_skipped":4,"total_bytes":2086,"processed_bytes":1600,"started_at":"2026-09-17T01:00:00Z","completed_at":"2026-09-17T01:00:02Z","error":"restore incomplete: 1 objects could not be read from backup storage (skipped_sample)"}`))
_, _ = w.Write([]byte(`{"operation":"restore","backup_id":"backup-20260917-000000-aaaaaaaa","status":"failed","total_files":17,"processed_files":16,"skipped_files":1,"unaddressable_files":1,"unaddressable_sample":["backup-20260917-000000-aaaaaaaa/data/mydb/cpu/2026/09/17/00/.hidden.parquet"],"skipped_sample":["backup-20260917-000000-aaaaaaaa/data/mydb/cpu/2026/09/17/00/c.parquet"],"missing_files":1,"backup_skipped_files":2,"backup_unaddressable_files":2,"backup_unregistered_skipped":4,"backup_manifest_only_files":2,"mode":"replace","files_registered":15,"registration_failed":1,"registration_failed_sample":["mydb/cpu/unregistered.parquet"],"sidecar_mismatches":2,"sidecar_mismatch_sample":["mydb/cpu/sidecar-mismatch.parquet"],"replaced_files":8,"iceberg_warehouse_files_skipped":4,"total_bytes":2086,"processed_bytes":1600,"started_at":"2026-09-17T01:00:00Z","completed_at":"2026-09-17T01:00:02Z","error":"restore incomplete: 1 objects could not be read from backup storage (skipped_sample)"}`))
}
})
ctx := context.Background()
list, _, err := cli.ListBackups(ctx)
if err != nil || len(list) != 2 {
t.Fatalf("list=%+v err=%v", list, err)
}
if list[0].SkippedFiles != 3 || list[0].SkippedMetadataFiles != 1 || list[0].UnaddressableFiles != 2 {
if list[0].SkippedFiles != 3 || list[0].SkippedMetadataFiles != 1 || list[0].UnaddressableFiles != 2 ||
list[0].SkippedReconciled != 1 || list[0].UnregisteredSkipped != 4 || list[0].ManifestOnlyFiles != 2 ||
list[0].LeftManifestDuringRun != 3 || list[0].ClusterManifestChecked == nil || !*list[0].ClusterManifestChecked {
t.Errorf("incomplete entry = %+v", list[0])
}
if list[1].SkippedFiles != 0 || list[1].SkippedMetadataFiles != 0 || list[1].UnaddressableFiles != 0 {
if list[1].SkippedFiles != 0 || list[1].SkippedMetadataFiles != 0 || list[1].UnaddressableFiles != 0 ||
list[1].SkippedReconciled != 0 || list[1].UnregisteredSkipped != 0 || list[1].ManifestOnlyFiles != 0 ||
list[1].LeftManifestDuringRun != 0 || list[1].ClusterManifestChecked != nil {
t.Errorf("entry without the keys must read zero: %+v", list[1])
}
m, err := cli.GetBackup(ctx, "backup-20260917-000000-aaaaaaaa")
if err != nil {
t.Fatal(err)
}
if m.SkippedFiles != 3 || m.SkippedMetadataFiles != 1 || m.SkippedOverlongKeys != 1 || m.UnaddressableFiles != 2 {
if m.SkippedFiles != 3 || m.SkippedMetadataFiles != 1 || m.SkippedOverlongKeys != 1 || m.UnaddressableFiles != 2 ||
m.SkippedReconciled != 1 || m.UnregisteredSkipped != 4 || m.ManifestOnlyFiles != 2 || m.LeftManifestDuringRun != 3 ||
m.ClusterManifestChecked == nil || !*m.ClusterManifestChecked {
t.Errorf("manifest counts = %+v", m)
}
if len(m.SkippedSample) != 4 || m.SkippedSample[2] != long || len(m.UnaddressableSample) != 2 {
t.Errorf("manifest samples = %v / %v", m.SkippedSample, m.UnaddressableSample)
}
if len(m.UnregisteredSample) != 1 || len(m.ManifestOnlySample) != 1 || len(m.LeftManifestSample) != 1 {
t.Errorf("cluster manifest samples = %v / %v / %v", m.UnregisteredSample, m.ManifestOnlySample, m.LeftManifestSample)
}
if m.IcebergWarehouse == nil || m.IcebergWarehouse.SkippedFiles != 1 || m.IcebergWarehouse.Path != "/srv/wh" {
t.Errorf("iceberg warehouse = %+v", m.IcebergWarehouse)
}
Expand All @@ -52,7 +61,10 @@ func TestBackup_IncompleteShapes(t *testing.T) {
}
p := st.Progress
if len(p.SkippedSample) != 1 || len(p.UnaddressableSample) != 1 || p.UnaddressableFiles != 1 ||
p.BackupSkippedFiles != 3 || p.BackupUnaddressableFiles != 2 || p.IcebergWarehouseFilesSkipped != 4 {
p.BackupSkippedFiles != 2 || p.BackupUnaddressableFiles != 2 || p.IcebergWarehouseFilesSkipped != 4 ||
p.Mode != "replace" || p.FilesRegistered != 15 || p.RegistrationFailed != 1 || len(p.RegistrationFailedSample) != 1 ||
p.SidecarMismatches != 2 || len(p.SidecarMismatchSample) != 1 || p.ReplacedFiles != 8 ||
p.BackupUnregisteredSkipped != 4 || p.BackupManifestOnlyFiles != 2 {
t.Errorf("restore progress = %+v", p)
}
}
Loading
Loading