Skip to content

Preserve archives and delete only successfully exported node states - #590

Open
robertexternbrink wants to merge 6 commits into
ClusterCockpit:mainfrom
robertexternbrink:fix/critical-archive-retention
Open

robertexternbrink wants to merge 6 commits into
ClusterCockpit:mainfrom
robertexternbrink:fix/critical-archive-retention

Conversation

@robertexternbrink

Copy link
Copy Markdown

Publish complete archives without replacing existing output, preserve portable metadata, and stop retention cleanup after transfer failures. Delete only the exact node-state identities successfully exported so concurrent backfills and replacements remain intact, and prefer the latest inserted state when timestamps tie.

Validation: Archive, repository and retention package tests, including race checks, and build/vet passed locally. Checks used the published cc-lib v2.17.0 and line-protocol v2.4.0 dependencies without local replacements.

Audit findings: BUG-0017, BUG-0018, BUG-0019, BUG-0020
Replace cutoff-wide cleanup after a node-state export with exact archived IDs, timestamps and node identities. Recheck every node latest timestamp at deletion time and bound SQL parameter usage for large completed selections. Cover concurrent late inserts, reused identities, newly latest ties, unexported orphan rows, partial export failure, deletion failure and cancellation. Port only GB5 safety from the deployed 2026-10-26 source; retain the existing selection and archive format. Depends on BE14 archive-integrity.
Order the shared latest-state subquery by timestamp and then row ID. Two updates in the same second must expose the later state in detail, list and map reads, while an actually newer timestamp keeps priority over a higher ID backfill. Port the deployed GB6 correction without query batching or indexing features.
@spacehamster87

Copy link
Copy Markdown
Contributor

This review took a bit longer, as it found two larger discussion points which should be decided before merging, even though the PR itself builds cleanly and all tests pass (also with -race).

Blocking Job Retention

Description: With the copy or move retention policy, one job without archive files now blocks job retention completely, on every daily run: the transfer returns at the first such job, so neither the copy marker nor the cleanup ever advances. Such jobs are created regularly:

  • monitoring disabled (api/job.go:1062, api/nats.go:412 return before archiving)
  • failed archiving (archiver/archiveWorker.go:130/140)
  • walltime exceeded (repository/job.go:720)

FindJobsBetween does not filter on archive status, so these jobs are selected, and as the oldest jobs they are hit first. The same pattern applies to a missing or invalid cluster.json, which aborts the transfer for all clusters (retentionService.go:108/145; this trigger is rarer).

Proposal: Skip jobs with MonitoringStatusDisabled / MonitoringStatusArchivingFailed on purpose (as graph/util.go:47 already does) and abort only on real I/O or transfer errors. TestRetentionTransfersDoNotReportSuccessForSkippedJobs currently requires an abort for any LoadJobMeta error and would need to tell the two cases apart.

Deletion Timing Increase

Description: AI-assisted timing tests (freshly migrated DB, bundled SQLite 3.53.4, WAL, 1,000 nodes × 288 rows, 287k rows to delete, single runs with about ±1 s noise) show a notable increase for the new DeleteNodeStatesByIDs:

  • old single-statement DeleteNodeStatesBefore: 1.7 s (write lock held 1.7 s)
  • new: 32.0 s in 2,243 statements (about 19×; longest write-lock hold about 13 ms)

The existing node_state indexes are used by every lookup (checked with EXPLAIN QUERY PLAN).

Proposal: This keeps the introduced fix while reducing the measured time:

  • Return node_id from FindNodeStatesBefore instead of looking it up again for every row.
  • Write each batch as id IN (…) AND (id, time_stamp, node_id) IN (VALUES …) AND <latest guard>.
  • Commit every N batches. N is the knob between total time and how long node-state ingestion waits on the write lock: committing every 8 batches gives 9.5 s total with at most 76 ms per hold. A single transaction is fastest (3.7–4.9 s) but blocks ingestion for the whole run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants