Skip to content

Keep metric writes reachable during concurrent retention - #588

Open
robertexternbrink wants to merge 2 commits into
ClusterCockpit:mainfrom
robertexternbrink:fix/BE02-retention-write-lifetime
Open

robertexternbrink wants to merge 2 commits into
ClusterCockpit:mainfrom
robertexternbrink:fix/BE02-retention-write-lifetime

Conversation

@robertexternbrink

Copy link
Copy Markdown

Keep ancestor levels locked until the leaf write completes and resolve ingestion writes from the root. Concurrent retention can otherwise detach a newly created level and lose successfully accepted samples.

Validation: Retention/write and ingestion regressions plus the metricstore package passed locally, including race checks.

BUG-cc-backend-T8103-F-e3fd58179aba-001

@spacehamster87 spacehamster87 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used AI-Assist to understand the change better. The change itself looks solid and worth merging. The review flagged three uncritical points:

  • Leftover lookup in DecodeLine. It still calls ms.GetLevel(st.selector) [lineprotocol.go:312]. That call can create a level, and its result is no longer used for writing; lvl only serves as a "cluster or host changed" flag now. The call and the variable could go, keeping prevCluster/prevHost.
  • Stale comment at [level.go:207-211]. It should say that the guarantee now comes from WriteToLevel holding the ancestor locks.
  • findLevelOrCreate still exists and keeps the early-release behaviour. It's still called from GetLevel, checkpoint loading [checkpoint.go:433] and WAL replay [walCheckpoint.go:627]. WAL replay even caches level pointers: Fine if both only run at startup before retention starts.

@robertexternbrink

Copy link
Copy Markdown
Author

Thanks for the review, @spacehamster87. I removed the redundant GetLevel lookup and lvl from DecodeLine, keeping prevCluster and prevHost for name reuse. I also updated the retention comment to explain that writes starting at the root hold ancestor locks until the leaf write completes. Follow-up commit

I checked the restore call sites as well. In production, checkpoint restore and WAL replay run during startup. Init waits for all restore workers, including WAL replay, to finish before starting retention. That makes the cached level pointers safe in the current restore path. Startup order · Restore workers

go test -race ./pkg/metricstore passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants