Skip to content

DAOS-19803 control: Fix rebuild timeout caused by self_heal lookup - #19395

Draft
mjmac wants to merge 1 commit into
masterfrom
mjmac/DAOS-19803
Draft

mjmac wants to merge 1 commit into
masterfrom
mjmac/DAOS-19803

Conversation

@mjmac

@mjmac mjmac commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Simplifies the implementation of dmg pool query by leveraging the
fact that the pool service's query RPC already returns a property list.

Instead of special-casing self_heal with a separate query mask and
PoolGetProp call, extend PoolQuery to allow for an optional set of
properties to be returned in a single call. This also future-proofs the
code for other properties that may be needed to construct a PoolQuery
response down the road. Removes unneeded DPI_SELF_HEAL_POLICY
bit from daos_pool.h, as the only consumer was the control plane.

Adds a new control plane error to indicate to RPC clients that the
targeted replica cannot service a request because none of its engines
are running. Control clients treat it as a redirect to the replica's
peers. The MS retry loop also now prefers a routing error over whichever
error arrived last, and retries against the replicas it has learned
rather than its original candidates. Together these fix rebuild test
timeouts caused by the prop query stalling for minutes on a replica
with no running engines.

Signed-off-by: Michael MacDonald mjmac@vectorsolutionsgroup.com

@github-actions

Copy link
Copy Markdown

Ticket title is 'rebuild/widely_striped.py:RbldWidelyStriped.test_rebuild_widely_striped - self-heal-property lookup caused a misleading rebuild timeout'
Status is 'To Do'
Labels: '2.8.1rc4,weekly_test'
https://daosio.atlassian.net/browse/DAOS-19803

@mjmac
mjmac force-pushed the mjmac/DAOS-19803 branch 3 times, most recently from 45371a3 to bd45104 Compare October 29, 2026 19:08
@mjmac mjmac changed the title DAOS-19803 control: return pool properties from PoolQuery DAOS-19803 control: Fix rebuild timeout caused by self_heal lookup Oct 29, 2026
Simplifies the implementation of dmg pool query by leveraging the
fact that the pool service's query RPC already returns a property list.

Instead of special-casing self_heal with a separate query mask and
PoolGetProp call, extend PoolQuery to allow for an optional set of
properties to be returned in a single call. This also future-proofs the
code for other properties that may be needed to construct a PoolQuery
response down the road. Removes unneeded DPI_SELF_HEAL_POLICY
bit from daos_pool.h, as the only consumer was the control plane.

Adds a new control plane error to indicate to RPC clients that the
targeted replica cannot service a request because none of its engines
are running. Control clients treat it as a redirect to the replica's
peers. The MS retry loop also now prefers a routing error over whichever
error arrived last, and retries against the replicas it has learned
rather than its original candidates. Together these fix rebuild test
timeouts caused by the prop query stalling for minutes on a replica
with no running engines.

Features: RbldWidelyStriped
Signed-off-by: Michael MacDonald <mjmac@vectorsolutionsgroup.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19395/10/execution/node/1700/log

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants