Skip to content

DAOS-19747 test: Extend test_dmg_system_reformat to verify multiple r… - #19191

Open
tanabarr wants to merge 7 commits into
masterfrom
tanabarr/ftest-system-erase-mdonssd-replicas
Open

DAOS-19747 test: Extend test_dmg_system_reformat to verify multiple r…#19191
tanabarr wants to merge 7 commits into
masterfrom
tanabarr/ftest-system-erase-mdonssd-replicas

Conversation

@tanabarr

@tanabarr tanabarr commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Extend DmgSystemReformatTest to cover dmg system erase/reformat across
storage modes (PMem/MD-on-SSD) and MS replica counts, instead of a
single fixed configuration.

Bump test_servers to 3 and add a setup: !mux single/multi_replica axis
varying mgmt_svc_replicas_qty.

Add a server_config: !mux standard/non_standard axis varying system
name and control port.

Add a pool: !mux default/md_on_ssd_p2 axis with a mem_ratio variant.

Add get_replicas() to assert the MS replica set is unchanged across
erase/reformat.

Wait for engine restart on every server_manager, not just the last.

Drop --force from system stop/storage format; retry format with a
longer timeout, tolerating a late "already running" response.

Fix system name forwarding on SystemErase's membership check and
replica fanout, which dropped non-default system names; authorise
SystemQuery for ComponentServer accordingly.

Add goroutine-stack diagnostics and a bounded Stop() wait to
execControlPlaneRestart to diagnose and avoid CI hangs during
multi-replica erase.

Features: control

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@tanabarr tanabarr self-assigned this Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Ticket title is 'Extend test_dmg_system_reformat to verify operation with multiple replicas'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-19747

@daosbuild3

daosbuild3 commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/1/execution/node/1140/log

@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from b16c487 to 47c67cd Compare October 4, 2026 21:34
@tanabarr
tanabarr changed the base branch from tanabarr/control-fmtreplace-rank to tanabarr/control-storage-format-status October 4, 2026 21:35
@tanabarr
tanabarr added this pull request to stack #19204 October 4, 2026 21:36
@daosbuild3

daosbuild3 commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/3/execution/node/1161/log

phender
phender requested changes Oct 5, 2026
View reviewed changes
Comment thread src/tests/ftest/control/dmg_system_reformat.yaml
Comment on lines +26 to +31
pool: !mux
default:
size: 90%
md_on_ssd_p2:
size: 90%
mem_ratio: 25

@phender phender Oct 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the same test can be run on a cluster configured with PMEM and a cluster configured to use MD on SSD, we cannot simply mux the two pool commands because the one with --mem-ratio will fail when the test is run on the cluster configured with PMEM.

To handle this, we use the filter-only feature in the test yaml (see example in ior/small.yaml), e.g.

launch:
  !filter-only : /run/pool/default    # yamllint disable-line rule:colons

pool: !mux
  default:
    size: 100%
    properties: rd_fac:0,space_rb:0
  md_on_ssd_p2:
    size: 100%
    properties: rd_fac:0,space_rb:0
    mem_ratio: 50

When the test is run launch.py optionally adds an extra yaml file that includes:

launch:
  !filter-only : /run/pool/md_on_ssd_p2

when it is being run with MD on SSD. This enables the second variant that creates a pool with --mem-ratio. On a cluster configured to use PMEM, this optional extra yaml is not created and only the default pool variant is run.

@tanabarr tanabarr Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread src/tests/ftest/control/dmg_system_reformat.yaml Outdated
engines:
0:
storage: auto
system_ram_reserved: 32

@phender phender Oct 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need reserved ram on HW clusters? I thought we have been removing this from tests that run on HW clusters.

@tanabarr tanabarr Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

@daosbuild3

daosbuild3 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/4/execution/node/1142/log

@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from 487619c to 0530576 Compare October 5, 2026 22:15
@daosbuild3

daosbuild3 commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/6/execution/node/1160/log

@tanabarr
tanabarr force-pushed the tanabarr/control-storage-format-status branch from 6c03e6d to ae11d67 Compare October 6, 2026 14:54
@tanabarr
tanabarr removed this pull request from stack #19204 October 8, 2026 13:18
@tanabarr
tanabarr changed the base branch from tanabarr/control-storage-format-status to tanabarr/control-fmtreplace-rank October 8, 2026 14:38
@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from 0530576 to 9512258 Compare October 8, 2026 14:51
@tanabarr
tanabarr requested a review from phender October 8, 2026 14:51
@daosbuild3

daosbuild3 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19191/8/display/redirect

@daosbuild3

daosbuild3 commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/9/execution/node/659/log

@tanabarr
tanabarr force-pushed the tanabarr/control-fmtreplace-rank branch from de5bbdf to 2a95e0c Compare October 10, 2026 14:32
@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from 9512258 to c3a278d Compare October 10, 2026 14:41
@daosbuild3

daosbuild3 commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/10/execution/node/1117/log

@daosbuild3

daosbuild3 commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/13/execution/node/1391/log

@daosbuild3

daosbuild3 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/14/execution/node/1134/log

@tanabarr
tanabarr force-pushed the tanabarr/control-fmtreplace-rank branch from 7477149 to 3bacfa7 Compare October 21, 2026 13:32
@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from da7f094 to a25042e Compare October 22, 2026 13:59
@daosbuild3

daosbuild3 commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/15/execution/node/1134/log

@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch 2 times, most recently from 39aed85 to 9c5c444 Compare October 23, 2026 13:10
@daosbuild3

daosbuild3 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/18/execution/node/1578/log

@daosbuild3

daosbuild3 commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Test stage Unit Test completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19191/19/display/redirect

Base automatically changed from tanabarr/control-fmtreplace-rank to master October 27, 2026 12:42
@daosbuild3

daosbuild3 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/20/execution/node/1594/log

@tanabarr
DAOS-19747 test: Extend test_dmg_system_reformat to verify multiple r…
fef784d 
…eplicas

Test-tag: hw,medium,control,dmg,system_reformat
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
@tanabarr
try to fix test yaml
027696c 
Test-tag: test_dmg_system_reformat
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
@tanabarr
attempt to resolve timing retry issue and ComponentServer access rela…
cedba6a 
…ted failures

Test-tag: test_dmg_system_reformat
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
@tanabarr
attempt to fix failures by correcting params and shorten custom sysname
3841df8 
Test-tag: test_dmg_system_reformat
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
@tanabarr
add hang diagnostics to System Erase control-plane restart
87e6f7a 
execControlPlaneRestart's GracefulStop()/Stop() sequence has been
observed to hang indefinitely in CI (dmg_system_reformat.py
multi-replica cases), with no visibility into what is blocking
progress. Add goroutine stack dumps on GracefulStop() timeout and
on a new Stop() timeout, plus explicit before/after logging around
Stop(), and stop waiting on Stop() past gracefulStopTimeout so the
exec() restart is never blocked indefinitely by a stuck Stop() call.

Test-tag: test_dmg_system_reformat
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from ca91f84 to 02c6808 Compare October 28, 2026 20:21
@daosbuild3

daosbuild3 commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Test stage Unit Test completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19191/21/display/redirect

@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from 2372ef4 to 70be471 Compare October 28, 2026 20:54
@tanabarr
tanabarr marked this pull request as ready for review October 28, 2026 21:01
@tanabarr
tanabarr requested review from a team as code owners October 28, 2026 21:01
@tanabarr

tanabarr commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@phender @daltonbohning probably ready for you guys to take a look please, it should be passing now

@tanabarr
fix system name forwarding
58119b0 
Features: control
Signed-off.-by: Tom Nabarro <thomas.nabarro@hpe.com>
@tanabarr
tanabarr force-pushed the tanabarr/ftest-system-erase-mdonssd-replicas branch from 70be471 to 58119b0 Compare October 28, 2026 21:07
@daosbuild3

daosbuild3 commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19191/24/execution/node/2013/log

@tanabarr
revert failsafe as ineffectual
b781f6d 
Test-tag: test_dmg_system_reformat
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@daltonbohning daltonbohning Awaiting requested review from daltonbohning
@phender phender Awaiting requested review from phender

Requested changes must be addressed to merge this pull request.

Assignees

@tanabarr tanabarr

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tanabarr @daosbuild3 @phender