Repository navigation
Run testsuite files in parallel - #726
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
xdelaruelle
force-pushed
the
testsuite-speed
branch
9 times, most recently
from
October 31, 2026 20:47
a7ee9b0 to
56bbc69
Compare
The lint sub-command runs a separate linter process for each file found in the enabled modulepaths. The whole-modulepath cases of 420-lint.exp were run against modulefiles.3, which holds more than 200 files, so each of these cases spawned nagelfar that many times and took about 3.5s. The five of them accounted for 18s of the 23s spent in this testfile. Add a modulefiles.lint fixture directory holding only what these cases need: modulefiles with a warning, an error or multiple problems, clean ones, a hidden one and a hidden dotfile name, forbidden modules with alias and virtual targets, and rc files with static errors. Point the whole-modulepath lint cases at it, which also lets them check the exact output rather than fragments. The single-file cases keep using the lint fixtures of modulefiles.3. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
…ries The modules.00-init series mixes the files bootstrapping the testsuite (global variables and procedures, initial environment, modulepath, cache pre-build and test environment checkpoint) with standalone checks of command-line switches and configuration behaviors that merely come first in the run order. Keeping these checks apart from the bootstrap makes clear which files every other series relies on, and lets the bootstrap series be run alone ahead of any series selection. Move every test file coming after 085-save_test_env.exp, plus the bad command name and bad argument number checks (070 and 071) that do not alter the testsuite running environment, into a new modules.01-general series. Only the version and shell type checks stay in 00-init, as they must run before any initialization is done. Drop the 00/080 entry from the setup file list of script/mt, as no such file exists anymore. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
The 090-uname-proc.exp file of the 50-cmds series holds no test: it only defines the variables describing the host that 095-uname.exp compares the uname modulefile command output against. Such a split means the test file cannot be run on its own, which gets in the way of running test files in separate groups. Move these definitions into 095-uname.exp and drop 090-uname-proc.exp. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
The channel name of the report file descriptor, shown in debug output when the pager is used, depends on the number of channels already open in the process. It gets a two-digit number on macOS, which the tests did not anticipate as they require a terminal that the macOS CI job did not provide so far. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
The expected listings of 070-spider-full.exp include modulefiles from directories whose names only differ by case, which are merged into a single directory on a case-insensitive filesystem. These tests require a terminal, which the macOS CI job did not provide so far, so they were never exercised on such a filesystem. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
The shell evaluating the script of the source-sh modulefile command or of the sh-to-mod sub-command inherits the standard input of the module command. As this script is run to collect the environment changes it makes, not to interact with the user, nothing is expected to be read from there. Yet the fish shell found on Cygwin starts an interactive session when a terminal is attached to its standard input, even if it is given a command to run, and then waits for an input that never comes. Run the shell with an empty standard input, so that the script evaluation cannot block on a terminal read, whatever the shell behavior. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
xdelaruelle
force-pushed
the
testsuite-speed
branch
from
November 1, 2026 09:24
56bbc69 to
5915ea2
Compare
xdelaruelle
force-pushed
the
testsuite-speed
branch
4 times, most recently
from
November 1, 2026 16:15
3e5bbdb to
8cea92b
Compare
A full testsuite run spawns one modulecmd.tcl process per test case, so its 12 minutes duration is bound by the single CPU DejaGnu uses, even on machines with many cores. Add the script/mpt driver that splits the selected test files into shards of at most 8 files from the same series, each run by its own runtest process in a private directory holding a copy of the testsuite directory (which is also the HOME of tests) and of the lib directory (the Tcl extension library file is replaced by some tests), plus links to the other entries of the object directory. Every shard runs the bootstrap files of the 00-init series first, plus the init and cleanup files of every series like script/mt does, as DejaGnu matches bare file names in every series directory. The 60-initx and 61-coll series, whose files depend on each other, are always run as a single shard. Each shard runs under a pseudo-terminal of its own through expect, as some tests change the column number of the terminal attached to the test harness. Shards are started biggest first and their results are merged into the usual modules.log and modules.sum files, with a summary section reflecting the whole run, so script/mtreview works unchanged on them. An interrupted run stops the running shards, merges the results obtained so far and cleans shard directories like a complete run does. A shard whose log file stops growing for 10 minutes (-t option) is considered stalled, for instance on a test waiting for an input: its processes are stopped and the run is reported in error, so a hanging test does not block the run until an external timeout. The driver is enabled with the JOBS variable of 'make test' and the -j option of script/mt. A full run takes 97s with 16 jobs and 177s with 4 jobs instead of 720s sequentially. Parallel run cannot be combined with coverage mode, as concurrent instrumented runs would corrupt the coverage logs. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Instrumented scripts of coverage mode write their coverage logs in tcl/ relatively to their current directory, so with several shards running concurrently in the same object directory these logs would be written and merged by every shard through the lock mechanism of Nagelfar, which gives up after a few retries. Give each shard a tcl/ directory of its own, made of links to the files of the object directory one, so every shard writes its own coverage logs. Once a shard is done, script/mpt merges its logs, made of incr and lappend commands over the counters of instrumented lines, with those already in the object directory like instrumented scripts do when they exit. Merged logs are written at the end of the run in the tcl/*.tcl_log files of the object directory, which are then marked up as usual. The COVERAGE=y and JOBS options of 'make test' can now be combined. A full coverage run takes 4m25s with 16 jobs instead of 26m33s sequentially, with the same lines reported as not covered. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Cygwin's fish package ships a /etc/fish/conf.d/01_fish_variables.fish configuration snippet that runs "pgrep fish | grep -v ^$fish_pid$ | xargs -r kill" when executed and leaves its own fish session without output. When test files run concurrently, a fish process started by one of them kills the fish processes started by the others, which makes the source-sh and sh-to-mod tests relying on fish fail or hang nondeterministically on Cygwin. The first fish-based test of a run also fails when it is the one executing this snippet. Mask this snippet with an empty file of the same name in a user configuration directory set in the testsuite directory, which fish is pointed at through the XDG_CONFIG_HOME environment variable for the whole modules and install testsuites. fish sources a single snippet per file name and looks at the user configuration directory first. This snippet is also what made the fish test of source-sh fail on Cygwin, so this test is not expected to fail there anymore. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
Use the -j option of script/mt on the test step of the Linux, macOS and Windows jobs, with as many jobs as processors available on the runner, to cut down the duration of these jobs. The MSYS2 job sets the MSYS environment variable to winsymlinks:native on this step, as symbolic links are created as copies of their target by default there, which does not suit the shard directories of a parallel run. Cygwin supports symbolic links by default, so its job needs no such setting. The log file of each shard is added to the artifact uploaded when a job fails, as the merged modules.log file does not hold the results of a shard that was stopped or that did not complete. The test step of the Cygwin and MSYS2 jobs is given a one hour timeout, so that a stalled run fails and provides these logs rather than waiting for the job timeout. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Xavier Delaruelle <xavier.delaruelle@cea.fr>
xdelaruelle
force-pushed
the
testsuite-speed
branch
from
November 1, 2026 18:50
8cea92b to
f4848d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A full run of the
modulestestsuite takes about 12 minutes, bound by the single CPU DejaGnu uses. This series cuts it down to 90 seconds on a 16-core machine and roughly halves the duration of the CI test jobs.make test JOBS=Nandscript/mt -j Nrun the test files concurrently: the newscript/mptdriver splits the selected files into shards of at most 8 files from the same series, each run by its ownruntestprocess under its own pseudo-terminal, in a private directory under.testshards/holding a copy oftestsuite/andlib/plus links to the other entries of the object directory. The results of all shards are merged into the usualmodules.logandmodules.sumfiles, soscript/mtreviewworks unchanged. An interrupted run stops the shards, merges what has been obtained so far and cleans up.tcl/*.tcl_logfiles, which the driver merges before the usual Nagelfar markup. A full coverage run takes 4m25s with 16 jobs instead of 26m33s, with the same lines reported as not covered.00-initseries is split: it now only holds the bootstrap files run by every shard, the other checks move to a new01-generalseries. The090-uname-proc.expsetup file is merged into095-uname.expso that test files remain self-contained.module linttests use a small dedicatedmodulefiles.lintfixture directory instead of linting the 460 files of the default modulepath, as Nagelfar is spawned once per linted file (420-lint.exp goes from 23s to 3s).script/mt -jwith the number of processors of the runner. Windows jobs keep a sequential run.