What happens
./mfc.sh test --no-build hands srun the path of a binary that was never built, instead of reporting that the case needs a build. The failure surfaces as a kernel-level execve error with no mention of the actual problem:
+ srun ... /.../build/install/gpu-mp-eca8fb2b47/bin/pre_process
error: execve(): /.../build/install/gpu-mp-eca8fb2b47/bin/pre_process: No such file or directory
srun: error: frontier10183: task 0: Exited with exit code 2
mfc: ERROR > :( /.../bin/pre_process failed with exit code 2.
Error: Submitting batch file for Interactive failed. It can be found here: .../tests/BEBF30E0/MFC.sh.
mfc: ERROR > main.py finished with a 143 exit code.
The harness even prints Build | syscheck, syscheck, pre_process, and simulation first and then proceeds to Run without compiling anything, so the log reads as though the build succeeded.
Why it matters
A case gets its own hashed install directory whenever its case.fpp differs from the default — an analytic initial condition (a patch_icpp expression) is the common way in. .github/workflows/common/test.sh always passes --no-build, and the Frontier AMD gpu-omp lane is the one lane whose Build step does not pre-build per-case binaries: because each amdflang device link is ~1 h, test.yml splits it into concurrent base (./mfc.sh build) and chem (./mfc.sh test --dry-run -a -o Chemistry) jobs, and neither covers a non-chemistry per-case build. Every other lane's single build step is ./mfc.sh test --dry-run -a, which does.
So the first case with an analytic IC to enter the suite fails on exactly one lane, after ~1.5 h of queue and build, with a message that points at srun and the filesystem rather than at the missing build. Diagnosing it meant reading the Build step's command line to notice what it does not build. It reproduced on both shards of that lane (jobs 105834021504 and 105834021537 of run 35415909570); AMD cpu, Frontier CCE and all NVHPC lanes were green on the same commit.
Suggested fix
Under --no-build, check each case's target binaries exist before launching and fail with the reason, naming the case and the build it would have needed, e.g.
mfc: ERROR > Test tests/BEBF30E0 (3D -> Example -> cyl_azimuthal_conduction_convergence)
needs its own build (gpu-mp-eca8fb2b47) because its case.fpp is
case-specific, but --no-build was passed and that build is absent.
That turns a ~90-minute opaque lane failure into an immediate, named one, and it would catch the same class of problem for a new chemistry mechanism, not just analytic ICs.
For reference, the instance that prompted this is fixed in #1906 by keeping analytic-IC cases out of the suite; this issue is only about the diagnostic.
What happens
./mfc.sh test --no-buildhandssrunthe path of a binary that was never built, instead of reporting that the case needs a build. The failure surfaces as a kernel-levelexecveerror with no mention of the actual problem:The harness even prints
Build | syscheck, syscheck, pre_process, and simulationfirst and then proceeds toRunwithout compiling anything, so the log reads as though the build succeeded.Why it matters
A case gets its own hashed install directory whenever its
case.fppdiffers from the default — an analytic initial condition (apatch_icppexpression) is the common way in..github/workflows/common/test.shalways passes--no-build, and the Frontier AMDgpu-omplane is the one lane whose Build step does not pre-build per-case binaries: because each amdflang device link is ~1 h,test.ymlsplits it into concurrentbase(./mfc.sh build) andchem(./mfc.sh test --dry-run -a -o Chemistry) jobs, and neither covers a non-chemistry per-case build. Every other lane's single build step is./mfc.sh test --dry-run -a, which does.So the first case with an analytic IC to enter the suite fails on exactly one lane, after ~1.5 h of queue and build, with a message that points at srun and the filesystem rather than at the missing build. Diagnosing it meant reading the Build step's command line to notice what it does not build. It reproduced on both shards of that lane (jobs 105834021504 and 105834021537 of run 35415909570); AMD
cpu, Frontier CCE and all NVHPC lanes were green on the same commit.Suggested fix
Under
--no-build, check each case's target binaries exist before launching and fail with the reason, naming the case and the build it would have needed, e.g.That turns a ~90-minute opaque lane failure into an immediate, named one, and it would catch the same class of problem for a new chemistry mechanism, not just analytic ICs.
For reference, the instance that prompted this is fixed in #1906 by keeping analytic-IC cases out of the suite; this issue is only about the diagnostic.