Repository navigation
DAOS-19591 container: resend EC boundary to reintegrated targets - #19356
Open
liuxuezhao wants to merge 1 commit into
Open
liuxuezhao wants to merge 1 commit into
liuxuezhao wants to merge 1 commit into
Conversation
|
Ticket title is 'Aurora daos_user: engine printing continuously in log "cont_agg_eph_sync() 66e9ae82/2416cc43: Sluggish EC boundary report from rank 117, 165933 Seconds." ' |
After a target is reintegrated its EC aggregation boundary restarts from zero and the container shard stays paused until the leader pushes the boundary again. That push never comes in two cases: 1. The rank's report of a zero epoch is ignored by the leader because ds_cont_leader_update_track_eph() only ever raises the per-rank value, so the global minimum does not change and cont_agg_eph_sync() skips the unchanged boundary forever. 2. If a target (SSD) of a live engine was evicted and reintegrated, ds_cont_tgt_ec_eph_query_ult() sees min_ec_agg_eph == 0 with a non-zero last reported epoch and skips the report entirely, so the leader does not even see the zero. Leader side: take a zero report as the rank's current value, so that a zero minimum tells that some target waits for the boundary. While that is the case, re-push the boundary every 30 seconds even if it is unchanged, once rebuild is done and reclaim is enabled. The pushed boundary cannot regress since the minimum is still raised to the rdb epoch. Rank side: always report a zero minimum epoch so the leader learns that a target of this rank waits for the boundary. Also fix the false "Sluggish EC boundary reporting" warning (#19015 on master): only warn when the epoch gap is not larger than the period observed by this leader (since it started tracking the container, or since rebuild ended / reclaim was enabled), a gap inherited from rdb at leader step-up is not a real stall. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
liuxuezhao
force-pushed
the
lxz/eph_ult_fix_28
branch
from
October 23, 2026 08:20
0cd0b48 to
46c809c
Compare
Collaborator
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19356/2/execution/node/1684/log |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a target is reintegrated its EC aggregation boundary restarts from zero and the container shard stays paused until the leader pushes the boundary again. That push never comes in two cases:
Leader side: remember that some rank waits for the boundary (cte_need_refresh) when a zero report is swallowed or when some rank has not reported yet, and periodically (every 30 seconds) re-push the boundary even if it is unchanged, once rebuild is done and reclaim is enabled.
Rank side: always report a zero minimum epoch so the leader learns that a target of this rank waits for the boundary.
Also back-port the false "Sluggish EC boundary reporting" warning fix from master (#19015): record when the leader starts tracking a container and only warn when it has been tracking it at least as long as the observed epoch gap, so a gap inherited from rdb at leader step-up is not reported as a stall.
Steps for the author:
After all prior steps are complete: