Skip to content

Preserve routing and drain accepted NATS ingestion on shutdown - #589

Open
robertexternbrink wants to merge 6 commits into
ClusterCockpit:mainfrom
robertexternbrink:fix/critical-ingestion-nats
Open

robertexternbrink wants to merge 6 commits into
ClusterCockpit:mainfrom
robertexternbrink:fix/critical-ingestion-nats

Conversation

@robertexternbrink

Copy link
Copy Markdown

Resolve routing from each sample’s effective tags and carry subscription defaults with queued NATS messages. Propagate reader/subscription failures and drain accepted ingestion before the final checkpoint to prevent misrouting, callback races and shutdown data loss.

Validation: Full metricstore tests, including race checks, and build/vet passed locally. Checks used the published cc-lib v2.17.0 and line-protocol v2.4.0 dependencies without local replacements.

@spacehamster87 spacehamster87 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few flagged notes, otherwise in favor of the PR 👍

  • New fatal on startup: Subscription errors now end in cclog.Fatal (os.Exit(1)). In practice this fires for a malformed subscribe-to subject, where previously the subscription was silently missing (and the log falsely said "established"). While failing fast is reasonable, it is a behaviour change: one bad subject now stops the whole backend, including the web UI. Is that intended, or should it log and continue with the remaining subscriptions? @moebiusband73
  • Delayed NATS server is fine: Since cc-lib connects with RetryOnFailedConnect(true) and unlimited reconnects, an unreachable server at startup does not trigger the new fatal. @ipatix
  • Merge order with PR588: The DecodeLine hunk here still uses lvl/GetLevel, which PR588 removes. Whichever lands second needs a rebase; after PR588 the new check simply becomes prevCluster, prevHost = cluster, host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants