Level 3: ten production HPC applications, external source artifacts, trusted agent workspace (supersedes #4) by bowencui123 · Pull Request #5 · Deep-Learning-Profiling-Tools/HPC-Performance-AI

bowencui123 · 2026-09-11T01:50:12Z

Complete Level 3 delivery against main: the two application batches, the correctness/validator work, the scheme-3 source distribution, the agent workspace, GEOS' retirement and ExaCA's admission. This PR supersedes #4 — every commit of #4 is an ancestor of this branch (verified: 0dd99a1 is an ancestor of the head, no file of #4 reverted or lost), so #4 does not need to be merged separately and is being closed as superseded.

After this merges, main presents scheme 3 directly: clone, tools/prepare_benchmark.sh level3 <app>, build, validate. No user needs to check out a historical PR branch.

What is in this PR

1. Ten production applications (level3/<app>/), each pinned to an upstream commit with its real build system, an official input case and a validation criterion taken from the application's own mechanism where one exists: LAMMPS, SPARTA, WarpX, SPECFEM3D, nekRS (two variants), Nyx, CP2K, QMCPACK, DFT-FE, ExaCA. Every benchmark ships build.sh, run.sh, validate.sh, benchmark.yaml, optimization_scope.yaml, inputs/references and provenance/.

2. Correctness and reproducibility work (originally #4): per-application validators with justified tolerances, NaN/Inf rejection, real-exit-code capture, stale-log gates, the Nyx box-layout comparator (legal re-blocking → UNSUPPORTED_LAYOUT, defects → structural failure), the verdict classifier (level3/tools/l3_verdict.py) and the launcher's rank→GPU audit as required evidence.

3. Source distribution: project-controlled external artifacts + automatic materialization. Application source is not in git. Each benchmark carries provenance/source.lock*.yaml (schema hpcperf-source-lock-2): upstream repository/tag/commit, patch series with hashes, artifact filename, byte size, archive SHA-256, source_tree_sha256, licenses, redistribution status and — once published — the immutable asset URL. tools/prepare_benchmark.sh resolves --artifact → content-addressed cache → the lock's URL → mirrors, verifies size and SHA-256, extracts with a restricted extractor outside the benchmark directory, verifies the tree hash, scans for credentials/build output/escaping symlinks, and only then places src/ (+ deps/) atomically. It is idempotent, refuses a modified tree (DIRTY) unless --force-rematerialize, and is never called by a build. Design: level3/EXTERNAL_ARTIFACT_DESIGN.md.

4. The Git LFS design is gone. Scheme 2 (never pushed) was migrated archive by archive: copy into staging, verify size/SHA-256/full extraction/tree hash/scan, re-materialize through the cache, check, then delete. No active Git LFS dependency remains (level3/LFS_TO_ARTIFACT_MIGRATION.md). Historical commits mention LFS; the active implementation does not use it.

5. LLM optimization workspace with a trusted harness. tools/create_agent_workspace.sh makes a real per-run copy (never a symlink back to the canonical tree) with private build/ and .deps/, plus a trusted file-hash baseline kept in this repository. tools/check_workspace.py runs 17 checks; in --agent-mode the modifiable scope may change while inputs, references, validators, provenance, dependency source and the harness copies must match the trusted baseline. tools/validate_workspace.sh builds the workspace's current source for the same backend and variant it validates, records which runs belong to that iteration (from the manifest records appended during the run, not a tail -1 guess), and states the build provenance of the validated binary (built_this_iteration / verified_from_build_record / UNVERIFIED). Layered exit codes: 6 REFUSED (integrity — nothing built or run, never a scientific result), 7 BUILD_FAIL, 0/1/3/4 numerical as validate.sh.

6. GEOS retired. ParMETIS 4.0.3 may not be redistributed and the application was replaced: suite_status: retired, redistribution_status: blocked, no artifact staged or published, excluded from the suite counts and permanently from the release plan. Code, provenance and results stay.

7. ExaCA admitted as the tenth application — with an explicitly narrow basis. ExaCA 2.1.0 (MIT) with Kokkos 4.7.04 and nlohmann_json 3.12.0 as benchmark dependencies; build from the artifact verified (109 s); 1/2/4-GPU validation of the dirsolid smoke case under a project-defined statistical protocol (no upstream oracle exists for that case; the GrainID field is not bitwise reproducible). Protocol v2 was frozen before the acceptance runs and separates calibration (8 runs) from an independent holdout (9 runs, 9/9 PASS). strong/weak are completion-only, HIP untested, multi-node unverified.

Its upstream GoogleTest unit tests were built and run as well. Three separate results, deliberately not merged into one sentence:

First observation (2026-09-11): 30 of 52 CTest entries pass as upstream runs them. Release acceptance was put ON_HOLD on that basis and the original record is kept verbatim (level3/exaca/references/upstream_unit_tests.json).
Attribution of all 22 failures with before/after evidence, per test (references/upstream_unit_test_matrix.json): 7 TEST_FIXTURE (tstNucleation rebinds a local grain_id handle from create_mirror_view_and_copy instead of writing celldata's subview — instrumented, the subview held 0,0,0 while the handle held 1,2,3; tstOrientation reads an uncopied create_mirror_view; tstInterface calls KOKKOS_INLINE_FUNCTIONs from host loops and indexes a device view on the host), 13 UNSUPPORTED_CONFIG (host-space test variants inside a CUDA-enabled build; 23/23 pass in a separate Serial-only Kokkos build), 2 TEST_INFRA (CTest launches mpiexec without per-rank GPU binding; through the project launcher the same binary passes at 1/2/4 ranks and reproduces upstream's own references VolFractionNucleated 0.1882 ± 0.0100 and TimeStepOfOutput 4820 ± 1), 0 APPLICATION, 0 UNRESOLVED. 0 APPLICATION is scoped to this matrix, on this machine, for the tests upstream ships at d26e59cd: it says these 22 failures are not application defects, not that ExaCA has no defects here or anywhere else.
patched-upstream-tests: with the fixture-only patches applied, 23/23 of the applicable CUDA tests pass at 1/2/4 ranks. Those patches live in git (level3/exaca/patches/upstream-tests/, labelled patched-upstream-tests, recorded with the upstream commit and target-file hashes) and are not inside the frozen source artifact or the release payload. Adopting them would require a new source_version and a new artifact.

Therefore, stated plainly: unmodified upstream tests under CTest's own launcher still fail in this environment, no unit-test result is counted as a pass in this repository's totals, and release acceptance was restored on the basis of the project protocol, the completed attribution and the two official small full-application cases at 1/2/4 real GPUs.

8. Source artifacts published as a prerelease. level3/RELEASE_PLAN.md/.json: provider = this repository's GitHub Release assets, tag level3-source-hpcperf-l3-v1-rc1 (a source-artifact prerelease that asserts nothing about HIP, multi-node or full scientific acceptance), 11 assets, 1,197,001,594 bytes, each re-extracted and verified, with the application license and the bundled dependency/data licence review recorded separately, plus SHA256SUMS and one SOURCE_MANIFEST per artifact. The publish engine checks every HTTP status, JSON shape and object identity, never overwrites or deletes an asset, and refuses without per-run authorization; remote verification is split into an anonymous plan-URL check (expected hashes come from the lock, never from the URL) and the ordinary clone + prepare check, so there is no circular dependency. The plan was frozen at target commit 16dcf18a605c21dfcfb4aaddb2a35aef39bff8ee with sha256 52be4c5370ac6b67f4f324ad9556982ac4857e2e67fa2c5e76490a73ef3e1eb6, and that same value was passed as --expect-plan-sha256 to every publisher invocation from the preflight through the public release.

Published on 2026-09-12 as release 387492391, tag level3-source-hpcperf-l3-v1-rc1, prerelease, 24 assets. The tag was resolved through /git/ref/tags back to the plan target before the run was accepted. Every asset was checked for identity and state == uploaded at upload time, re-downloaded with credentials in the draft, and verified again by API digest before the release became public. All 11 archives were then downloaded anonymously, with the token removed from the process and an empty cache, and checked against size, archive sha256 and source_tree_sha256 taken only from the locks; the locks were updated with the real URLs only after that passed, in a separate commit, and the ordinary user path was then re-run from a clean clone of that commit. level3/PUBLICATION_RECORD.md records the whole sequence, including one defect found in the verification tooling during the run and the fix that closed it. GitHub immutable releases is not enabled on this repository (read-only probe; setting untouched), so the guarantee comes from the project rule (assets are never overwritten or deleted; a source change means a new source version and tag) and from the hash verification every consumer performs.

9. level3/README.md rewritten for first-time users: what Level 3 is, a prominent current-availability box, the catalog of ten applications with frozen versions and validated workloads, prerequisites split into download/build/run, a quick start that separates the future published flow from today's --artifact flow, GPU-selection semantics, the artifact/hash/cache/offline model, the workspace with its real directory tree and trusted-baseline rules, evidence levels with the criteria sources kept apart, known limitations and troubleshooting. The former policy sections moved verbatim into level3/BUILD_STRATEGY.md.

Status that this PR does not change

Nyx LyA heat/cool stays STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING (exit 3, not a pass, cause not established). QMCPACK's walker/cuSOLVER limit for its validated case stands. GEOS' flow/well unit-test failures remain recorded. HIP untested, multi-node unverified/blocked on this site, 8/40/80 GPUs are dry-run plans only. Eight applications keep historical build/validation evidence rather than a rebuild from the materialized workspace; DFT-FE's install fingerprint changed with the freeze. Per-claim evidence levels: level3/WORKSPACE_EVIDENCE.md.

Reading the diff

313 files, +771,974 / −14. 645,738 of those lines are the 12 provenance/SOURCE_MANIFEST*.json files (one line per source file: path, sha256, size, exec bit) and provenance dominates the rest. The reviewable change is ~6,400 lines of tools, ~10,500 lines of per-application scripts/contracts/READMEs and ~3,500 lines of Level 3 documentation. No application source is added by this PR.

Tests: level3/tools/tests/run_all.sh — seven CPU-only groups, all passing (infra 30, second-batch checkers 21, Nyx comparator 33, verdict classes 18, ExaCA validator 19, source-distribution tools 76, release/anonymous-fetch API mock 44), 241 checks in total. An asset whose state field is missing or null is now rejected exactly like a starter state, at upload time, before any publish request; two mock faults assert that.

Not in scope: no rebuild of the eight applications from src/, no large scaling experiments, no Level 1/2 change, no upload, no merge.

…EM3D/nekRS bring-up STEP 1-2: level3/APPLICATION_AUDIT.md covers all ten candidates (repository, release, commit, license, LOC, build system, GPU model, CUDA/HIP/MPI support, rank->GPU mapping, inputs, correctness mechanism, scaling inputs, build/disk estimates, B200+CUDA 13.2 and MI355X risk, containers, Spack, priority, blockers). level3/BUILD_STRATEGY.md compares Native / Spack / Apptainer / site-native per application and records BUILD_RECOMMENDATION (NATIVE for eight, NATIVE+SPACK_DEPS for CP2K and QMCPACK), the Spack and container policies, per-application dependency isolation and the modification classes used. STEP 3-6: per-application fetch/build/run/validate scripts and README under level3/<app>/, private trees under .deps/level3/<app>/ with schema l3-1 fingerprints (level3/tools/l3_common.sh), upstream sources as read-only shallow clones under _upstream/level3/. All five first-batch applications build natively for CUDA 13.2 / sm_100 and pass their upstream correctness mechanism at 1, 2 and 4 GPUs on dgx003 through the common launcher (one rank per GPU, per-rank GPU wrapper, mapping audited): LAMMPS stable_22Jul2025_update6 thermo vs shipped reference log (identical) SPARTA 27Aug2026 statistical stats vs shipped reference log WarpX 26.09 + AMReX 26.09 analytic Langmuir test + charge conservation SPECFEM3D v4.1.1 reference seismograms via upstream script nekRS v26.0 upstream --cimode CI on the analytic Ethier case Strong and weak inputs are defined and run at 1/4 GPUs; 8/40/80-GPU shapes are launcher dry-runs only (DRY-RUN / UNVALIDATED); multi-node is BLOCKED/ UNVERIFIED on this site; HIP build branches exist and are untested. Compatibility changes (no class E): SPECFEM3D two upstream-devel back-ports (CUDA 13 deviceOverlap guard, Blackwell device block) plus make-time GENCODE and SCOTCH-without-zlib; nekRS HYPRE sm_100 list and CUDA 13 / Thrust 3.2 compatibility (thrust::pair, explicit reverse_iterator/pair headers, not1 -> not_fn), Makefiles generator, OMPI_FC/-fno-lto/-fPIC/unset AR, osc ^ucx and ulimit -s at run time. Level 2 files are untouched. tools/runtime/README.md records the plan to move the shared launcher tools out of level2/tools without breaking Level 2 (Level 3 references them through HPCPERF_RUNTIME_DIR).

…se-solver decision Validators capture the real run exit code (timeout/nonzero/missing output -> FAIL), reject NaN/Inf (level3/tools/l3_check.py), require the complete step/field/trace/check sets, and write a per-run manifest. Dry-runs are routed to a .dryrun/ scratch directory and can no longer clobber real results. Fingerprint schema l3-2 records ordered patch-content hashes; the nekRS source-cache key is SHA + patch series hash. Level 3 builds strip the Level 2 .deps/install prefixes (LAMMPS/SPARTA rebuilt isolated, identical results). CPU-only negative tests: level3/tools/tests/run_all.sh (13/13). nekRS: the Ethier workload runs its HYPRE coarse solve on the CPU by default (cimode 2); the three HYPRE/Thrust patches are needed only for GPU HYPRE, which is now verified separately with cimode 3 (DEVICE coarse, 9/9 at 1/4 GPU, hypregpu variant). A patch-free cpucoarse variant (ENABLE_HYPRE_GPU=OFF) passes cimode 2 at 1/2/4 GPU and rejects a DEVICE request explicitly. Both variants are kept, isolated per variant. See level3/CORRECTNESS_FIXES.md and level3/nekrs/COMPATIBILITY.md.

…the second batch l3_paths_profile <app> <profile> gives every second-batch configuration its own .deps/level3/<app>/<profile>/{src,build,install,logs,cache} tree and build directory (profiles never share a mutable source tree or install); l3_version_mm derives the profile name components. l3_binary_backend_check now accepts a CUDA binary that links cudart statically (CMake's default for AMReX-based applications) when cuobjdump finds embedded device code, instead of reporting it as non-CUDA. Fingerprint schema unchanged (l3-2).

…/4 GPUs Native CMake build of Nyx 26.09 against a private AMReX 26.09 install (the AMReX commit Nyx pins cannot emit sm_100 through CMake: its convert_cuda_archs drops SM >= 10.0 and autodetects 8.6+PTX on this node; 26.09 is a strict descendant and resolves sm_100 correctly). Profiles cuda132-gcc133-adiabatic and a cpu-gcc133-adiabatic reference (with AMReX plotfile tools and particle_compare); double-precision particles as upstream's regression builds. Cases are upstream's decks: MiniSB (nightly GPU regression test, inputs.32 + ppm_type=0), LyA-adiabatic (inputs.rt.garuda), the 64^3 LyA science deck as a named adiabatic derivative for strong scaling, and the Scaling deck (RandomPerCell, labelled synthetic) for strong/weak. Fixed BoxArray across rank counts; ranks > boxes refused. validate.sh: completeness/finiteness, upstream's fcompare tolerance (2e-10) against a same-configuration rerun (1 GPU) or the 1-GPU run (2/4 GPUs), a CPU-backend reference at a pre-fixed 1e-8, baryon-mass conservation and exact DM counts. AMReX's particle_compare cannot compare across rank counts (header equality incl. next_id; exit code 0 even on "FAIL"), so nyx_particle_compare.py matches particles through their exact t=0 positions (checkpoints at step 0 and the final step) and applies the same norms. Results: VALIDATED_PASS at 1/2/4 GPUs for both official decks (max rel err 1.4e-10 vs 1-GPU, <= 1.8e-10 vs CPU, particles <= 2e-15, mass exact); strong/weak runs completed; 8/40/80 dry-runs planned or refused as designed.

…s; real grep; robust static-cudart check l3_clean_conda_build_env drops the conda env's CFLAGS/CXXFLAGS/LDFLAGS/AR/... (they pin -march=nocona and conda binutils, which broke OpenBLAS's AVX512 kernels and would apply to every system-GCC dependency build). l3_common.sh now unsets an exported `grep` shell function (a ugrep wrapper inherited from interactive sessions changed grep's exit codes inside the scripts) and l3_binary_backend_check captures cuobjdump's output before grepping so the callers' pipefail no longer turns cuobjdump's SIGPIPE on large binaries into a false 'not a CUDA binary'.

… GPUs; synthetic strong deck SUNDIALS probe first (its own CUDA example regression tests: 5/6 pass, the cuSolverSp batched-QR example differs only in integrator statistics and is not on Nyx's CVDiag path), then the staged heatcool profile (SUNDIALS with CVODE + ARKODE -- AMReX 26.09 requires the arkode component -- AMReX with SUNDIALS, Nyx HEATCOOL) plus a CPU heatcool reference. lya_heatcool runs Exec/LyA/inputs.rt as shipped. validate.sh applies per-case tolerances: the adiabatic decks keep 2e-10/1e-8; the heat/cool deck uses upstream's own nightly LyA tolerance 5e-5 and excludes the CVODE rate diagnostic I_R (O(1) noise even between identical runs) while every state variable agrees to <=1.6e-13 (rerun, 2/4 GPU, CPU). run.sh adds a strong mode for the synthetic RandomPerCell deck (fixed 256^3).

… validated at 1/2/4 GPUs Upstream toolchain (install_cp2k_toolchain.sh) with the B200 back-port of upstream commit 378b2fab (--gpu-ver=B200 -> ARCH_NUM 100; DBCSR H100 kernel parameters reused for B200), system GCC 14.2.1 for C/C++/Fortran, conda Open MPI 5.0.10, CUDA 13.2.78, CMAKE_CUDA_ARCHITECTURES=100. DBCSR 2.10.0 verified first (its own ctest suite, 4 ranks x 4 threads on the 4 B200s: 19/19). CP2K validation: adapted subset of upstream's regression tests (regtest-gpw-1 Ar/H2O-geoopt/pyridine, regtest-dm-ls-scf-1 H2-big-1/5) within upstream tolerances at 1, 2 and 4 GPUs; benchmarks/QS/H2O-64.inp GPW-DFT NVE MD (10 steps): all MD-step SCF cycles converged, GRID/DBCSR/PW GPU evidence from CP2K's own output, MD-step FORCE_EVAL energies within 8.7e-12 Ha of the 1-GPU run (pre-fixed tolerance 1e-8 Ha). The deck's own IGNORE_CONVERGENCE_FAILURE (initial ATOMIC-guess SCF, MAX_SCF=50) is reported explicitly and would fail without that declaration.

…on checkers test_l3_validators.sh exercises cp2k_md_summary.py, qmc_check.py and dftfe_check.py on genuine outputs (when present under build/level3/) and on mutated copies: truncated run, NaN, non-converged MD-step SCF, non-converged initial SCF without IGNORE_CONVERGENCE_FAILURE, zero accelerator devices, missing offload/CUDA banners, GRID tasks on CPU only, nonzero exit code, truncated/shifted scalar files, 'QMCPACK ERROR'. Each mutation must be rejected; the genuine output must be accepted.

…ith a private LLVM 23.1.0 toolchain Upstream's recommended NVIDIA configuration QMC_GPU="openmp;cuda" needs a Clang with NVPTX offload: LLVM 23.1.0 is built from source on local scratch (clang/lld + openmp/offload runtimes, GPU runtimes target nvptx64-nvidia-cuda -> libompdevice/libomptarget-nvptx.bc; the official binary release ships no device runtime), then probed before any application build (OMP_TARGET_OFFLOAD=MANDATORY, target region off the initial device, numerics vs host, MPI 2/4 ranks through the common launcher). HDF5 1.14.5 parallel, Boost 1.90 headers and OpenBLAS 0.3.30 are private per profile; the node's zlib-ng CMake package (missing libz.a) is bypassed for HDF5 via CMAKE_IGNORE_PATH. run.sh drives upstream's diamondC_2x1x1_pp batched VMC+DMC deck (verbatim for smoke; strong/weak change only the walker population); validate.sh runs upstream's unit and deterministic ctests on the built tree (64/64 and 526/526 passed) and applies upstream's own check_scalars.py criterion to the N-GPU run plus a cross-rank statistical consistency check; qmc_check.py gates on completion, offload/CUDA banners, reported device code paths and device memory allocated through the offload runtime.

… paths for system-toolchain builds C_INCLUDE_PATH/CPLUS_INCLUDE_PATH/CPATH/LIBRARY_PATH on this node point at other conda environments (empty include dirs, but linker search paths); they are now cleared together with the conda build variables in l3_clean_conda_build_env.

…II 9.6.2 / ELPA 2026.02.001 stack Recipe transcribed from upstream install_DFTFE (frontierDevelop) to this node: OpenBLAS -> ScaLAPACK 2.2.2 -> libxc 7.0.0 -> spglib -> ALGLIB -> p4est 2.8.7 (dftfe's Cray-oriented p4est-setup.sh given the Open MPI wrappers, LIBS=-lm and the 2.8.7 header location) -> Kokkos 4.6.00 Serial -> deal.II 9.6.2 -> ELPA with NVIDIA sm_100 kernels (-march=native for its AVX-512 probe; ScaLAPACK paths in LDFLAGS for its cublas check) -> dftfe_real (CMAKE_CUDA_ARCHITECTURES=100). deal.II 9.7.1 (the version the current recipe pairs with dftfe develop) was tried first and rejected: 9.7 removed Utilities::MPI::create_group, Triangulation::load(name, autopartition) and VtkFlags::ZlibCompressionLevel that release 1.2.0 still uses. One 2-line source patch (std::isnan for two unqualified isnan calls, GCC 14). run.sh drives upstream's GPU regression deck Input_MD_0.prm (32-atom Al BOMD) verbatim, a synthetic Al-supercell weak series and the 192-atom LLZO ground state; validate.sh compares with upstream's own GPU reference output through dftfe_check.py (pre-fixed tolerances) and requires the independent ELPA GPU probe (elpa_probe.sh: ELPA's residual/orthogonality tests on 1/2/4 GPUs). At 1/2/4 GPUs the Al BOMD energies, temperatures and forces reproduce upstream's reference at printed precision.

…eads one); BLAS-resolution guard The first build inherited conda's LDFLAGS (-Wl,--disable-new-dtags) and the MPI wrapper's rpath, so cp2k.psmp/libcp2k.so resolved libopenblas.so.0 to the conda pthreads OpenBLAS at run time ("OpenBLAS Warning : Detect OpenMP Loop"); the validations had passed with it, but the configuration was not the recorded one and H2O-128 ran 1.7x slower. build.sh now clears the conda build variables, passes the toolchain libopenblas.a as CP2K's CUSTOM BLAS/LAPACK and puts the toolchain library directories first in the RPATH; run.sh refuses to run unless ldd resolves BLAS under the toolchain (or it is linked statically) and records blas_resolved= in the manifest. Re-validated at 1/2/4 GPUs with the relinked binary; attempts 1-2 archived under install/ATTEMPT-*.

LyA heat/cool 64^3 on 1/2/4 GPUs: 2.85 / 2.62 / 2.65 s (too small to scale, fixed-cost dominated); dry-runs 8 planned, 40 IMBALANCED, 80 refused as designed. nyx_particle_compare.py made executable.

run.sh refuses more than HPCPERF_QMCPACK_MAX_WALKERS_PER_GPU (300) walkers per GPU unless HPCPERF_QMCPACK_FORCE_POPULATION is set: this build exhausts device memory at ~320 MB per walker (cuSOLVER INTERNAL_ERROR at 1024 walkers/GPU; open issue documented in the README, not worked around). Strong = verbatim 256-walker deck over N GPUs (534/332/211 s on 1/2/4), weak = 256 walkers per GPU (534/632/656 s); every run within upstream's check_scalars window. Manifest records walkers_per_gpu.

…ng validation 1/2/4 GPUs Build: upstream thirdPartyLibs superbuild (config-build.py -n) with three build-system patches (superlu_dist 65-char hash, RAJA_ENABLE_VECTORIZATION overridable -> OFF for nvcc 13.2 + GCC 14/x86-64-v3, hdf5 step generator and build commands), ENABLE_HYPREDRV=OFF (hypredrive step lacks Umpire's include; GEOS-documented option, unused by the beam workflow); GEOS with a private host-config (hypre on device, sm_100), TPL METIS/ParMETIS include dirs ahead of the conda MPI include dir (unrelated 32-bit metis.h), and the back-port of LLNL/blt 38b46203 for BLT's CUDA runtime smoke test (memoryClockRate removed in CUDA 13); tests kept at upstream's default ON. 1718 s at -j32. Validation re-implements geos-ats's checks for beamBending (curve_check.py metric ||u-u_ref||_2/N <= 2e-4 vs the analytic Euler-Bernoulli script, restart_check.py rules incl. LvArray permutation handling and the default exclusions, atol 1e-3 / rtol 1e-7) and was verified against upstream's published baseline first. Results: hypre GMRES+AMG on the device at 1/2/4 GPUs, curve metric 1.376e-4 (= upstream's baseline value), cross-rank rel L-inf <= 1.1e-8; shipped direct-solver deck vs the public restart baseline beamBending_smoke_01: 0 of 1191 arrays disagree (worst rel 3.6e-12); the three GPU-build LinearSolverParameters defaults are reported, not gating. Strong/weak/dry-run records: the official beam decks are too small to scale.

… 1/2/4 PASS with the probe elpa_probe.sh runs ELPA's own validate_*_gpu_analytic programs (the real binaries in build/elpa/.libs/; the top-level names are libtool relink wrappers) on 1/2/4 GPUs and applies ELPA's limits from test_analytic_template.F90 (max eigenvalue error 5e-14, eigenvector error 6e-10), exit 0, GPU timers present and a launcher audit with 0 mismatch. The two earlier versions looked for wrapper-script names and then for the %Error Residual/Orthogonality lines of the random-matrix programs (which the analytic ones never print; the empty grep aborted the script under pipefail) -- both recorded in the script header and README. Probe: PASS on all six runs (errors <= 7.3e-15 / 1.0e-11). Validation 1/2/4 re-run with the probe: PASS (al_md identical to upstream's GPU reference). README: results table, the std::isnan patch described as the one source change.

…ts fail on this GPU build) ctest on GPU 0 with upstream's default ENABLE_TESTS=ON: 254 of 261 passed. testMath (float asinhf vs host within 1 ulp) and testErrorHandling (abort on purpose under prterun) are test/harness artefacts; testCompMultiphaseFlow, testCompMultiphaseFlowHybrid, testThermalEstimator{Prod,Inj}Well and testReservoirThermalSinglePhaseMSWells_RateInj fail grossly (derivative and Jacobian checks off by O(1), ExternalError in the well solvers) -- cause not determined; the compositional-flow and well modules of this build are marked UNVERIFIED. The validated solid-mechanics beam workflow is unaffected.

SECOND_BATCH_STATUS.md (new): status table for Nyx, CP2K, QMCPACK, DFT-FE and GEOS (version/SHA, build strategy, compiler/Toolkit, dependency probe, CUDA build, 1/2/4-GPU validation, strong, weak/size sweep, 40/80 dry-run, multi-node, HIP, source changes, blocker), per-application sections with the measured numbers and the open items, cross-cutting notes, raw-material locations and the local commit list. APPLICATION_AUDIT.md / BUILD_STRATEGY.md: second-batch corrections (CP2K B200 back-port and BLAS relink, Nyx AMReX pin, QMCPACK toolchain, DFT-FE deal.II 9.6.2 + std::isnan patch, GEOS develop SHA, RAJA vectorization, BLT back-port, metis include order, unit-test findings), the 'what was actually built and how' table, Spack policy realised as NATIVE.

…epository) Environment facts of the validated node (CUDA 13.2.78, conda vs system GCC, Open MPI transport, no ROCm, Slurm slots), the git rules (no attribution trailers, local commits only, never push/PR/merge unprompted, never add the environment/dependency trees), the Level 2/3 per-application conventions (fetch/build/run/validate/checker/patches/README, per-profile isolation, fingerprints, manifests, common launcher interface and dry-run semantics), the validation principles and change policy, the node/toolchain pitfalls collected over Levels 1-3, the current state of the branches and the open decisions, and how reports are expected.

…BDIR for regression campaigns l3_clean_env.sh / l3_clean_env_exec: re-execute a command under env -i with an explicit allow-list (session/locale, Slurm, MPI/PMIx/UCX, CUDA, compilers and build flags, conda/python, OpenMP/BLAS knobs, HPCPERF_*). Used around every tool that records its process environment; CP2K's toolchain installer (which writes a declare -x dump into toolchain.env) now runs through it. Compatibility checked on dgx003: nvcc/mpirun/cmake/gcc/python and the launcher dry-run work under the wrapper with hpcperf_env.sh sourced, no secret-named variable is visible inside. HPCPERF_L3_RUN_SUBDIR (default run): every application's run.sh/validate.sh now builds its run directories under build/level3/<app>/<profile>/$L3_RUN_SUBDIR, so a regression campaign can write into a fresh sibling tree (run.regress-<sha>) without touching historical results. Mechanical, line-local edits in the 10 applications; level3/tools/tests/run_all.sh now also runs test_l3_validators.sh and test_nyx_validator.sh.

…ovenance corrected, negative tests compare() trusted an awk filter over fcompare's table: it dropped inf/nan and message rows before the finiteness check, treated any rc=1 as an over-tolerance case, and 'nv>=5' passed as a completeness check -- a reviewer's CPU injection (Ne abs=1 rel=inf on a zero reference, a missing variable with fcompare's warning, a NaN row) was accepted. nyx_fcompare_check.py now parses both plotfile Headers (variable set, dim, levels, time, domain, cell sizes, box arrays), runs fextrema on both files for raw finiteness and zero-reference detection, requires exactly one parsed row per variable and level (no message rows, no duplicates, no truncation), judges zero-reference fields by an absolute rule (exact by default), never drops a non-finite error, keeps diagnostic fields reported but ungated, and refuses parser/tool disagreement. validate.sh calls it, gains OFFLINE/report-dir/tools-dir hooks (offline re-checks never write into historical run directories), and downgrades a case with a diagnostic field to STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING (exit 3) instead of PASS. Provenance, corrected in the header and README: MiniSB 2e-10 is upstream's; LyA-adiabatic 2e-10 is a project choice 25x stricter than upstream's 5e-9; the heat/cool 5e-5 is upstream's plt00354 tolerance adopted after the first heat/cool validation at 2e-10 FAILED; the I_R exclusion is project-defined and was introduced at that time (upstream's report does compare I_R). README section 'I_R' records what the field is (SDC reaction integral, state-carried, a ~1e-3 residual of cancelling O(a^2 rho e/dt) terms in this deck), its O(1) run-to-run variation that leaves no imprint on any state field, why no state-derived absolute criterion accepts it, and what is still unidentified. Offline re-check of the saved 2026-09-06 plotfiles with the new comparator: MiniSB/LyA-adiabatic 1/2/4 GPUs PASS at 2e-10 / 1e-8 (worst Temp 1.4e-10 on MiniSB np4, 6.7e-14 on LyA); heat/cool 1/2/4 GPUs STATE_AND_PARTICLES_PASS, I_R_CHECK_PENDING. Tests: level3/tools/tests/test_nyx_validator.sh (26 checks: 16 comparator cases incl. the three reviewer injections, zero/zero, zero-vs- nonzero, duplicate, truncated, header mismatch, non-finite raw, diagnostic; 10 through the real validate.sh chain in OFFLINE mode).

…-listed environment wrapper

…tial deny-rule - nyx_fcompare_check.py: the BoxArray of every level is checked geometrically (physical extents -> cell boxes via prob_lo/dx): boxes inside the level domain, no overlap, level 0 covers the domain exactly; a missing box / overlap / coverage deficit / different covered cell sets is STRUCTURAL (exit 2), while a legal re-blocking (same cells, different partition) is UNSUPPORTED_LAYOUT (exit 4) -- reported explicitly, never a science FAIL, never skipped silently (fcompare would need --allow_diff_grids, not enabled). Same boxes in another order are compared normally. - nyx/validate.sh: comparator exit 4 -> case verdict UNSUPPORTED_LAYOUT, script exit 4; precedence FAIL(1) > UNSUPPORTED_LAYOUT(4) > I_R_CHECK_PENDING(3) > PASS(0); header states that I_R is consumed by the next step's predictor and that short-horizon state agreement does not prove it has no later effect. - l3_verdict.py (new): classes PASS / PENDING / UNSUPPORTED_LAYOUT / FAIL / MISSING from (exit code, log); exit 3/4 are never PASS; an exit code that contradicts the log is FAIL; campaign summary with launcher-audit columns. l3_common.sh: l3_run_recorded records a queue step's exit code and continues. - l3_clean_env.sh: credential deny-rule applied after the allow-list (a *_TOKEN/*_KEY/... name never survives through an allow-listed prefix); --show names denied variables (names only). Compatibility on dgx003 unchanged: nvcc, mpirun, gcc, cmake, python/numpy, nvidia-smi, SLURM_JOB_ID, CUDA_VISIBLE_DEVICES, OMPI_MCA_opal_cuda_support visible inside. - Tests: test_nyx_validator A17-A21 (re-blocking -> exit 4; missing box, overlap, out-of-domain -> STRUCTURAL; reordered boxes -> compared) and B11-B12 (real validate.sh chain: exit 4 UNSUPPORTED_LAYOUT, missing box -> FAIL); test_l3_infra 6a-6f (clean env) and 7a-7b (queue continues past exit 3); test_l3_verdict (13 checks). run_all.sh: 88 checks, 0 failures. - Offline re-judgement of the 2026-09-07 regression plotfiles with this comparator: identical verdicts (adiabatic 1/2/4 PASS, heat/cool 1/2/4 exit 3), 18/18 comparisons "identical box array".

…, second-batch table - level3/README.md: status table now carries the five second-batch applications with their real state (Nyx adiabatic decks PASS re-run 2026-09-07; heat/cool STATE_AND_PARTICLES_PASS; I_R_CHECK_PENDING -- not a pass; CP2K/QMCPACK/DFT-FE/ GEOS historical 2026-09-05/06 results, not re-run on GPU); the 2026-09-07 joint-HEAD regression (27 validate.sh calls: 24 PASS, 3 PENDING, 0 FAIL; four logs with unverified launcher ranks listed as a binding-evidence gap); verdict classes; strong/weak, 40/80-GPU, multi-node and HIP stated as they are; the Nyx heat/cool tolerance history recorded as the one exception to "no tolerance changed after a run". - SECOND_BATCH_STATUS.md: Nyx row/section split into adiabatic PASS and heat/cool PENDING ("Nothing remaining" removed; the I_R follow-up listed); joint-HEAD regression section with what was and was not re-run; CPU test groups and script versions; commit list updated (branch submitted as a Draft PR). - nyx/README.md: heat/cool case no longer "not built"; box-layout policy (UNSUPPORTED_LAYOUT vs STRUCTURAL); 2026-09-07 regression table; I_R section reworded -- I_R feeds the next predictor, 10-step state agreement is not proof of no later effect; the 1.9e-2 figure is a diagnostic estimate with its formula, inputs (tol 5e-5, max|a^2 rho e|/(dt a_half) = 3.7e2, dt = 2.99e5 code time units, a = 0.01094), units and assumptions, not a verified bound; heat/cool excluded from the correctness-pass count and performance summaries. - CLAUDE.md: validate.sh exit classes, queue/verdict helpers, the unwrapped run/profiler path as a follow-up, current state (2026-09-08) and open items.

…tract, LOC Level 3 application source is distributed as frozen source bundles (one archives/*.tar.zst per application, Git LFS) that materialize into level3/<app>/{src,deps}; build/run/validate read source only from there. - tools/hpcperf_source.py: source_tree_sha256 (hpcperf-tree-1: sorted paths, file content / symlink target, no mtime/uid/mode), file manifest, credential/artifact scan (name rules + token patterns, no value ever printed), escaping-symlink policy, deterministic GNU tar (fixed mtime, uid/gid 0, normalized modes) + zstd -19 single-thread, safe extraction, git HEAD-blob export (never the working tree), LFS pointer detection. - tools/freeze_benchmark_source.py: spec-driven freeze (pinned checkouts and declared submodules at recorded commits, sha256-pinned tarballs copied or downloaded, declared exclusions, approved patch series applied in order, scan -> hash -> equivalence check against the tree the recorded results were validated from -> archive); writes provenance/source.lock*.yaml, upstream*.lock, patch_series*.txt, original_vs_baseline*.diff, SOURCE_MANIFEST*.json, LICENSES*.md, equivalence*.{json,md} and the identity fields of benchmark.yaml. An UNEXPECTED equivalence difference stops the freeze; --verify-determinism rebuilds the archive and compares sha256. - tools/compare_source_trees.py: content comparison with classes identical / expected_patch_difference / expected_generated_difference / expected_normalization_excluded / expected_build_artifact / expected_added / missing_source / extra_source / unexpected_content_difference. - tools/prepare_benchmark.sh + hpcperf_materialize.py: LFS-pointer refusal with the git lfs pull hint, archive sha256, tree sha256 of the extracted bundle, layout/symlink/secret/artifact/provenance checks, atomic rename, idempotent on an identical tree, refuses a modified tree (exit 3) unless --force-rematerialize. - tools/check_workspace.py: 15 static checks (schema, scope, src/deps, canonical tree hash, entries, no external source references, symlinks, provenance, archive/tree hash consistency, inputs/references, scope inside the workspace, secrets, build artifacts, no _upstream/ dependency). - tools/create_agent_workspace.sh: per-run copy under workspaces/<run-id>/ (real src/deps copies, harness + environment symlinks at the workspace root, optional prebuilt dependency prefixes, readonly ranges chmod'ed, workspace.yaml with canonical/initial tree hashes, check run at the end). - tools/loc_report.py (cloc code LOC by ownership category incl. tarball dependencies), tools/source_archives_report.py (SOURCE_ARCHIVES.md), tools/readme_source_section.py (README/AUDIT sections). - level3/tools/l3_common.sh: l3_require_materialized, lock queries (l3_source_commit, l3_component_commit, l3_submodule_commit, l3_lock_patches, l3_source_tree_sha, l3_materialized_variant), PYTHONDONTWRITEBYTECODE=1. - Tests: tools/tests/test_source_tools.sh (29 checks on a synthetic benchmark: freeze/patch/equivalence/determinism, scan and symlink negatives, materialize idempotency/refusal/force/LFS pointer/corruption, check_workspace negatives, workspace creation), test_l3_infra 8a-8g (lock queries), all wired into level3/tools/tests/run_all.sh. - .gitattributes (LFS filter for level3/*/archives/*.tar.zst); .gitignore: materialized trees, markers, workspaces/, and a temporary guard for the archives because git-lfs is not available on the build node.

…e materialized bundle Per application: provenance/freeze_spec*.yaml (pinned upstream checkout at the recorded commit, declared submodules, exact dependency tarballs by sha256, declared exclusions, the approved patch series), benchmark.yaml (contract: backends, cases, entries, inputs/references, install root, prebuilt dependency prefixes, identity of the bundle), optimization_scope.yaml (what an agent may modify; bundled/benchmark-specific dependency source, inputs, references, validators and provenance are read-only; loc_categories), the generated provenance (source.lock, upstream.lock, patch_series, original_vs_baseline.diff, SOURCE_MANIFEST.json, LICENSES.md, equivalence, LOC, check_workspace). build.sh/run.sh/validate.sh: application source from $HERE/src, dependency sources from $HERE/deps, no fetch, no clone, no patch (the bundle is the patched baseline; patch names/hashes stay in the fingerprints); fetch.sh is freeze-time only. Builds that write into their source tree run from a build-side copy of the frozen tree (SPECFEM3D in-tree autotools, nekRS, DFT-FE include/git_info.h, GEOS LvArrayConfig.hpp, CP2K toolchain copy). GEOS pre-seeds the TPL superbuild downloads from deps/tpl-dist (CMake verifies URL_HASH and skips the download); CP2K pre-seeds the toolchain package tarballs from deps/cp2k-toolchain-dist (retrieve_package verifies the same sha256). nekRS keeps two frozen variants (hypregpu = v26.0 + 3 HYPRE CUDA-13 patches, cpucoarse = exact v26.0); build.sh/run.sh refuse a variant that is not the materialized one. Equivalence to the validated trees (provenance/equivalence*.md): every bundle is content-identical to the tree the recorded 1/2/4-GPU results were built from, up to declared build artifacts, the pre-applied patches, declared exclusions (SPECFEM3D doc/, Nyx's unused AMReX submodule pin, thirdPartyLibs' uberenv) and documentation files the validated private copies lacked (nekRS hypregpu, SPECFEM3D SCOTCH doc). No historical result was re-run. Tests updated to read references from the materialized bundle (freeze-time checkout as fallback); the Nyx validator takes the IC count from the run directory's own IC file.

…lication source sections, materialized LOC - level3/README.md: the frozen-bundle layout, tools, identity and the agent workspace contract; dependency isolation updated (level3/<app>/{src,deps} is the only application source input; _upstream/ is freeze-time only). - level3/SOURCE_ARCHIVES.md (generated by tools/source_archives_report.py): 11 archives, sizes, files, LOC, archive/tree sha256, LFS state, license status (ParMETIS blocker), equivalence, patches; Git LFS status and the git-lfs blocker on the build node. - level3/<app>/README.md: 'Source distribution (frozen bundle)' section per application (identity, patches, equivalence, LOC by ownership) generated by tools/readme_source_section.py; level3/APPLICATION_AUDIT.md: materialized source LOC table (cloc code lines) replacing whole-checkout figures. - CLAUDE.md: bundle layout, freeze/materialize/workspace rules, LFS guard.

…ent-addressed cache, trusted agent workspace Replace the abandoned Git LFS design (never pushed) by project-controlled external source artifacts + automatic materialization + self-contained agent workspaces: - tools/hpcperf_lock.py: source lock schema hpcperf-source-lock-2 (artifact filename/ size/sha256/source_tree_sha256, primary {url,status} with `unpublished` = null url, mirrors, cache layout, redistribution_status, source_scope); validate_lock rejects node-private locations, floating/non-https URLs, non-40-hex commits, LFS keys. - tools/hpcperf_source.py: LFS-pointer helpers removed; artifact naming, content- addressed cache (real read-only copies, never hard links), streaming download into .partial/ with size/magic/sha256 checks before the atomic rename. - tools/hpcperf_materialize.py / prepare_benchmark.sh: resolution --artifact > cache > primary url > mirrors, --offline, --status, DIRTY refusal (exit 3), artifact unavailable/unpublished (exit 4), hash/tree/unsafe-content mismatch (exit 5); extraction in level3/.materialize-staging/ outside the benchmark directory; status vocabulary NOT_PREPARED / LOCAL_ARTIFACT_VERIFIED / REMOTE_FETCH_VERIFIED / TREE_VERIFIED / MATERIALIZED / READY / DIRTY / INVALID. - tools/freeze_benchmark_source.py: artifacts go to $HPCPERF_ARTIFACT_STAGING/level3/ <app>/<source_version>/ (+ artifact.json, SHA256SUMS), never into the worktree; writes the schema-2 lock and the benchmark.yaml source_artifact identity. - tools/artifacts/: artifact_catalog.yaml (suite membership, naming, publish requirements), verify_artifact.py, migrate_from_lfs_bundle.py (copy, verify, rewrite lock, delete old only after the prepare test), generate_release_manifest.py (release_manifest.json + level3/SOURCE_ARTIFACTS.md, statuses only, no paths), publish_artifacts.sh --dry-run / publish_plan.py (PLAN/REFUSE; no provider adapter, nothing is uploaded). - tools/check_workspace.py: 17 checks; --agent-mode compares every file with the trusted baseline: modifiable-scope changes are recorded (files, diff, hashes), readonly/excluded/harness changes are READONLY TAMPERING. - tools/create_agent_workspace.sh: --dest (workspace outside the repository), trusted workspace_baseline.json (read-only, mirrored under <repo>/.hpcperf/); tools/ validate_workspace.sh: trusted harness (refuses tampering with exit 3, builds and validates inside the workspace, records iteration verdicts). - level3/tools/l3_common.sh: lock queries read schema 2 (l3_source_version added) with a PyYAML-capable interpreter (the project conda env has none). - tests: tools/tests/test_source_tools.sh rewritten for scheme 3 (62 checks: staging, verify, cache/offline/download/partial, extractor, lock validation, secret scan, check_workspace negatives, workspace isolation, readonly tampering, trusted validation, manifest, publish plan); infra tests updated (30). - .gitattributes (LFS rule) and tools/source_archives_report.py deleted; .gitignore: .artifacts/, .hpcperf/, level3/.materialize-staging/, *.tar.zst.

…-2 locks); retire GEOS from the default suite Every scheme-2 archive was copied byte-identically into the maintainer's local artifact staging as <app>[-<variant>]-hpcperf-l3-v1.tar.zst, re-verified (size, zstd magic, sha256, full extraction: layout, source_tree_sha256, entry count, symlinks, secret/ build-output scan), re-materialized from the staging copy through the cache (prepare test), checked with check_workspace (17/17), and only then deleted from the worktree. Locks rewritten to hpcperf-source-lock-2 (all provenance kept, migration block records the old path and the identical sha256, primary url null / unpublished); benchmark.yaml carries source_artifact / variants.<v> identity, upstream_version, redistribution_status, suite_status; freeze specs now name the source version hpcperf-l3-v1; fetch.sh comments no longer mention archives/. GEOS: RETIRED_FROM_DEFAULT_SUITE (ParMETIS 4.0.3 redistribution constraint + replacement decision): redistribution_status blocked, suite_status retired, no scheme-3 artifact staged (the old archive copy lives only in controlled local research storage), code/provenance/results kept. LAMMPS: provenance/agent_workspace_verification.yaml records the real agent-edit closed loop (workspace outside the repository; injected compile error fails the build; restore rebuilds in 90 s; 1-GPU trusted validation PASS; an agent edit is recompiled into a new binary and validated; tampered validator/reference REFUSED; canonical tree unchanged). Per-application README sections regenerated for the scheme-3 fields.

…(Kokkos 4.7.04, CUDA sm_100) ExaCA (LLNL/ExaAM, MIT; commit d26e59cd51e241a327c5267d43fd70537e5425f7) frozen as exaca-hpcperf-l3-v1.tar.zst: src/ = ExaCA, deps/kokkos = Kokkos 4.7.04 (82799e45; 4.6.02's nvcc_wrapper defaults to sm_70, rejected by CUDA 13.2), deps/json = the nlohmann_json 3.12.0 release tarball ExaCA would otherwise download. build.sh builds Kokkos, json and ExaCA out-of-source from the materialized artifact only (109 s at -j32, no patch). run.sh: directional-solidification deck (upstream Inp_DirSolidification physics, Inconel 625, G=5e5 K/m, R=3e5 K/s, nucleation density 10) with a decomposition-independent SurfaceSiteDensity substrate; one MPI rank per GPU, 1-D Y decomposition; modes smoke 128^3, strong 512x256x512 (a single rank cannot hold 512^3: ExaCA int-indexes the 26-neighbour arrays), weak 512x128x512 per rank; 40/80 GPUs dry-run only. validate.sh / exaca_check.py: the GrainID field is not bitwise reproducible (atomic captures), so the criteria are statistical invariants (all cells solidified, one global decomposition, log/field self-consistency, grain counts, nucleated fraction, top-layer grains, <001>-to-z misorientation) vs a frozen 1-GPU reference and vs the same build's 1-GPU run, tolerances = measured spread x >= 3. Results 2026-09-10: PASS at 1/2/4 GPUs; strong CA time 8.02/6.26/5.17 s (1/2/4), weak 4.88/6.16/6.80 s. benchmark.yaml, optimization_scope.yaml (modifiable: src/src, src/bin, src/analysis/src), provenance (source.lock schema 2, redistribution cleared, LOC: 6,512 application-owned, 6,368 agent-modifiable), check_workspace 17/17. Suite status: candidate -- all twenty admission criteria met on this node, maintainer confirmation pending.

…ord, SOURCE_ARTIFACTS, suite status - EXTERNAL_ARTIFACT_DESIGN.md: lifecycle (freeze -> staging -> publish -> git -> materialize -> workspace), artifact format, lock schema 2, staging vs publication statuses, cache and prepare rules, workspace/trusted-harness contract, versioning, license rules, tool map. - LFS_TO_ARTIFACT_MIGRATION.md: KEEP_AND_REUSE / MIGRATE_THEN_DELETE / DELETE_OBSOLETE / REVIEW_MANUALLY classification, per-archive migration table (11 archives, all verified, old copies deleted), residual scan (no active Git LFS dependency). - SOURCE_ARTIFACTS.md + tools/artifacts/release_manifest.json (generated): 9 retained applications / 10 artifacts, 1.11 GiB compressed; every artifact FROZEN, LOCAL_ARTIFACT_VERIFIED, LOCAL_MATERIALIZATION_VERIFIED, REMOTE_ARTIFACT_UNPUBLISHED; LAMMPS AGENT_WORKSPACE_VERIFIED; GEOS RETIRED; ExaCA CANDIDATE(pending). SOURCE_ARCHIVES.md (LFS report) removed. - README.md / BUILD_STRATEGY.md / APPLICATION_AUDIT.md / SECOND_BATCH_STATUS.md / CLAUDE.md: user flow = tools/prepare_benchmark.sh level3 <app> (no git lfs pull), source-input rules for build.sh, GEOS retired, ExaCA audit section and candidate row.

… 7 / numerical 0-1-3-4), trusted baseline only from the repository The workspace-integrity refusal and the Nyx scientific PENDING shared exit code 3, which a caller or summary could confuse. They are now separate layers: - tools/validate_workspace.sh: exit 6 REFUSED (integrity: readonly/harness tampering, untrusted or in-workspace baseline, broken layout -- nothing is built or run, no scientific verdict), 7 BUILD_FAIL (build layer), 0/1/3/4 propagated unchanged from validate.sh (numerical layer). reports/iter-N.verdict.yaml records `layer:` and `exit_code:`; a REFUSED or BUILD_FAIL result never enters a scientific or performance summary. - level3/tools/l3_verdict.py: classes REFUSED and BUILD_FAIL added; rc 3 stays PENDING only with the Nyx I_R_CHECK_PENDING line and is FAIL for a refusal; the summary counts the new classes separately and states that they are not scientific outcomes. - tools/check_workspace.py: the trusted baseline is the repository copy .hpcperf/workspace_baselines/<run-id>.json or an explicit --baseline outside the workspace root; the copy inside the workspace is not consulted (development only with --allow-workspace-baseline, which still cannot turn a forged file into a PASS). Agent mode now also verifies the harness copies of the workspace root (hpcperf_env.sh, level2/tools, level3/tools) -> HARNESS TAMPERING, and the docstring states that these are file-hash and permission checks, not an OS sandbox. - tools/create_agent_workspace.sh: baseline schema 2 records the harness files as well; the harness copies are made read-only; the workspace-local baseline is marked informational. - tests: 6 new source-tool checks (exit 6 for tampered validator/reference/harness/ replaced build.sh, untrusted in-workspace baseline, accepted external baseline) and 5 new verdict checks (rc 6/7 classes, rc 3 with a refusal line is FAIL).

…, negative tests, admitted as the tenth application The 2026-09-10 tolerances were derived from the same eight runs that were then declared PASS, so they were calibration, not acceptance. Protocol v2 separates both: - references/validation_protocol.{md,yaml}: frozen inputs, metrics and decision rule; spread = range (max - min) over the calibration runs, tolerance = max(3 x range, floor), applied to a single run vs the frozen reference (its own uncertainty is covered by the range-based rule), n = 8 -- explicitly an empirical range rule, no 3-sigma claim. - references/calibration.json: the 8 runs of 2026-09-10 (4 x 1 GPU, 1 x 2, 3 x 4) with per-metric range and standard deviation, re-labelled CALIBRATION; the single-grain probe (upstream Inp_SmallEquiaxedGrain, 64^3) showing that even one growing grain is not bitwise reproducible (3 cells) and its termination cycle depends on the rank count. Non-determinism evidence: 2.2 % of cells differ between identical 1-GPU runs with thousands of distinct id pairs and differing id sets (real spatial differences, not renumbering), consistent with the atomic_fetch_add steering vector and the atomic_compare_exchange cell capture in src/CAupdate.hpp (upstream states no determinism guarantee; GoogleTest unit tests are not available here). - v2 widens exactly the two tolerances that were below 3 x range (top_layer_grains 15 -> 25 %, mean_misorientation_z_top 0.5 -> 0.7 deg) BEFORE any holdout run. - references/holdout.json: 3 independent sets x fresh 1/2/4-GPU validate.sh runs (9 runs, new run ids, separate run directories), evaluated with the frozen rule: 9/9 PASS, launcher audit N verified / 0 mismatch each; observed deviations recorded (top_layer_grains 31 in one 1-GPU run = 6.9 %, well inside 25 %). No tolerance was changed after the holdout. - exaca_check.py: new `validate` subcommand (testable criteria evaluation) whose decomposition check verifies that the Y subdomains tile the box exactly once (offset chain with 1-cell halos, first 0, last end == Ny, sizes >= 2, N entries); the halo-sum formula alone would accept a missing plus a duplicated subdomain. - level3/tools/tests/test_exaca_validator.sh (19 checks, CPU only, synthetic fields): NaN/Inf statistics and reference, unsolidified cells, gap/duplicate/short/wrong-rank decompositions, log-vs-field inconsistency, out-of-tolerance statistics, missing 1-GPU run, ASCII/binary equivalence, and a stale field/log/stats in the run directory not standing in for a failed run. - ExaCA admitted (benchmark.yaml suite_status retained, catalog admission admitted, default_suite_order of ten): basis = project-defined statistical validation of the dirsolid smoke case only; strong/weak completion-only, HIP untested, multi-node unverified.

… regression, verdict layers, provider decision - level3/WORKSPACE_EVIDENCE.md: one column per claim (source equivalence + materialization, canonical build from the artifact, independent workspace build, agent-edit effectiveness, scientific validation with version/profile, GPU-binding audit, multi-GPU in a workspace, remote fetch) with levels VERIFIED / HISTORICAL / NOT_RUN / UNVERIFIED. The LAMMPS Ninja closed loop is not extrapolated: the Make/autotools and build-side-copy applications (SPECFEM3D, nekRS, DFT-FE, CP2K, GEOS) keep NOT_RUN for workspace build and agent edit; the eight applications not rebuilt from src/ this round keep HISTORICAL with the equivalence proof named. Readiness is defined as materialized variant + marker + check 17/17 (+ trusted baseline for iterations > 0), not "src/ exists". - level3/lammps/provenance/agent_workspace_verification.yaml: the 2-GPU regression in the existing out-of-repo workspace (iteration 7, existing binary 423fc91be978c0d9...): smoke case PASS against the upstream reference log and the 1-GPU run (all 8 quantities rel 0.0), agent-mode check PASS with the repository baseline, run id and input hash recorded; plus a 3000-step binding-evidence run in the same workspace with the launcher audit 2 verified / 0 mismatch (the ~1 s validation runs are too short for nvidia-smi sampling: unverified = observation gap, not a mismatch). 4-GPU workspace validation NOT_RUN, no dependency rebuild, no scaling experiment. - level3/README.md, EXTERNAL_ARTIFACT_DESIGN.md, CLAUDE.md: verdict layers of the trusted harness; trusted-baseline policy and the explicit statement that hash and permission checks are not an OS sandbox; provider decision (GitHub Release assets of this repository, first tag level3-source-hpcperf-l3-v1-rc1 as a prerelease that does not assert HIP/multi-node/full scientific acceptance); GitHub immutable releases probed read-only and NOT enabled, so the project rule (never overwrite or delete an asset, new source_version + tag on any source change, consumers verify size, sha256 and source_tree_sha256) carries the guarantee; application license and bundled dependency/data license review recorded separately. - SOURCE_ARTIFACTS.md + release_manifest.json regenerated: ten retained applications, eleven artifacts, all LOCAL_ARTIFACT_VERIFIED + LOCAL_MATERIALIZATION_VERIFIED + REMOTE_ARTIFACT_UNPUBLISHED; LAMMPS additionally AGENT_WORKSPACE_VERIFIED; GEOS RETIRED. Nyx PENDING, QMCPACK scale limit, HIP and multi-node statuses unchanged.

…pter, anonymous remote-fetch check - tools/artifacts/release_plan.py -> level3/RELEASE_PLAN.{md,json}: per artifact the asset filename, byte size, archive sha256, source_tree_sha256, planned asset URL (marked PLANNED; the locks stay url null / unpublished), application license and the separate bundled dependency/data license review, verification result, SHA256SUMS line, SOURCE_MANIFEST asset, patches, scientific status and build prerequisites; retired applications are excluded permanently and unadmitted candidates as CANDIDATE_EXCLUDED; the publication procedure is recorded as data. - tools/artifacts/github_release_upload.sh: adapter for the authorized upload step, UNTESTED against the live API. Refuses without HPCPERF_CONFIRM_UPLOAD=yes (granted per run by the maintainer only), without a token, when the target commit is not on origin, when a release with the tag exists, when an asset name is already present, or when a local file disagrees with the plan; uploads assets + SHA256SUMS + SOURCE_MANIFEST files to a DRAFT release and re-downloads each one (authenticated upload check, explicitly not REMOTE_FETCH_VERIFIED); never edits a lock. - tools/artifacts/remote_fetch_check.sh: the REMOTE_FETCH_VERIFIED test, to be run only after publication -- clean clone at the reviewed commit, empty cache, no credentials (refuses if GITHUB_TOKEN/GH_TOKEN is set), download through prepare_benchmark.sh, tree hash and check_workspace verified, then writes provenance/remote_fetch_verification*.yaml. - tools/artifacts/immutable_releases_probe.json: read-only probe result (not enabled, not enforced by the owner); the repository setting was not changed.

… 1.11 GiB, nothing published) Generated by tools/artifacts/release_plan.py with --full (every artifact re-extracted: layout, source_tree_sha256, entry count, symlinks, secret/build-output scan) against the maintainer's local staging. Eleven planned assets (ten retained applications, nekRS counted as its two variants) plus SHA256SUMS, one SOURCE_MANIFEST per artifact and the plan itself; GEOS excluded permanently (ParMETIS redistribution constraint). Provider: GitHub Release assets of this repository; the release is a prerelease of source artifacts and asserts nothing about HIP, multi-node or full scientific acceptance. Planned URLs are marked PLANNED and no lock was touched: every artifact stays url null / status unpublished until a published asset has been downloaded anonymously into a clean clone with an empty cache and verified. The plan must be regenerated at the final reviewed commit before an upload.

…ild and validation on one backend/variant, stop implying --skip-build compiled the edit tools/validate_workspace.sh previously took the binary and its hash from the last line of whatever run manifest it could glob, which could attribute an older run to the current iteration, and it built with build.sh's default backend while validating whatever backend the caller passed after `--`. - runs of the iteration: the manifest set, size and content are snapshotted before the validator runs; afterwards only the records APPENDED during the run are parsed (run.sh appends to run_manifest.txt, so the first record in a file is the oldest run). Every run of the iteration is listed with run id, ranks, exit code, binary, binary sha256 and the launcher's GPU-binding audit; when no manifest changed, the record says so instead of claiming a run. New helper tools/workspace_iteration_record.py. - one backend, one variant: --backend (default CUDA) is passed to build.sh and validate.sh; a backend given twice inconsistently is a usage error; the materialized variant from the marker is exported through benchmark.yaml's variant_env, and an environment variant that contradicts it is REFUSED before anything is built. - build provenance: each successful build appends a trusted record (source hash, backend, variant, sha256 of the produced executables) via tools/workspace_build_record.py. A verdict now carries build_provenance = built_this_iteration / verified_from_build_record / UNVERIFIED; --skip-build alone never asserts that the current source edit was compiled into the validated binary. - 8 new checks in tools/tests/test_source_tools.sh (76 total): provenance for built, skip-build-unchanged, skip-build-after-edit and rebuild; consecutive iterations report their own appended run; inconsistent backend refused; contradicting variant refused. Verified on the real LAMMPS workspace: iteration 9 reports its own run id and the current binary with build_provenance UNVERIFIED (--skip-build, no build record for that binary), numerical verdict PASS.

…a non-circular remote verification A curl that transfers successfully is not a successful API call, and the previous flow could only fill a lock URL after an ordinary prepare had worked, which itself needed the URL. - tools/artifacts/github_release_publish.py (new engine; the shell entry point keeps its guards and delegates): per-endpoint expected status codes, JSON-shape checks and object identity checks (release id, tag, target commit, draft/prerelease flags, upload_url addressing the same id, read-back of the created release, asset name/size/state). 401/403/404/422/5xx, timeouts, non-JSON bodies, identity mismatches and incomplete asset sets all fail with a non-zero exit and never print a published verdict. Assets are never overwritten or deleted: a pre-existing asset or an existing tag aborts the run. Modes: preflight (local: plan schema, staged archive size+sha256, working-tree SOURCE_MANIFEST hashes and the derived SHA256SUMS against the reviewed plan, the plan's own sha256), draft (create + upload + re-download every asset + verify the complete set), publish (flip the draft, then GET and re-verify), verify (optionally re-download). - tools/artifacts/verify_published_artifact.py + remote_fetch_check.sh --mode plan-url: the anonymous check that breaks the circular dependency. The URL comes from the reviewed plan, the expected size, archive sha256 and source_tree_sha256 come only from the trusted lock, so a URL can never redefine what the artifact must contain. It refuses to run with a token in the environment and uses a fresh empty cache. remote_fetch_check.sh --mode lock-entry then proves the ordinary user entry (clean clone of the lock-update commit, empty cache, plain tools/prepare_benchmark.sh); REMOTE_FETCH_VERIFIED means that second step passed. Both are recorded separately. - tools/artifacts/tests/{mock_github.py,test_release_publish_mock.sh}: CPU-only mock of the Releases API with 14 fault modes (unauthorized, forbidden, missing commit, existing tag, 422, 5xx on upload, invalid JSON, dangling release id, truncated asset, pre-existing asset, timeout, incomplete listing, publish not applied) plus the full success path and the anonymous-verification cases: 33 checks, wired into level3/tools/tests/run_all.sh. Nothing was uploaded: no release exists, and every lock keeps url null / unpublished.

…result (30/52 pass, evidence only) The earlier statement that GoogleTest is unavailable here was wrong: the artifact carries Kokkos' vendored fused GoogleTest 1.11.0 (deps/kokkos/tpls/gtest). Built as a build-side static library (no source change, no system install, artifact untouched), ExaCA's own test suite configures and builds in 62 s and runs 52 tests. Result: 30 pass, 22 fail, identical pass/fail pattern in two consecutive runs. Inspected causes: the host-space ("SERIAL") test variants abort in a CUDA-enabled build with Kokkos "attempt to access inaccessible memory space" (invalid configuration for those variants -- a Serial-only build would be needed); two CUDA suites read device views from the host without a mirror copy (tstOrientation's create_mirror_view without a copy, tstInterface's host access to "DOCenter"), i.e. test-side issues exposed by Kokkos 4.7's access checks; the Nucleation CUDA assertions and the Update CUDA aborts are not explained. Also recorded: the tests need `cmake --install` first, because they resolve material/orientation files by bare name through the installed share/ExaCA directory. Consequence: no unit-test result is counted as a PASS anywhere; ExaCA's acceptance continues to rest only on the project-defined statistical protocol for the dirsolid smoke case. Follow-ups noted in references/upstream_unit_tests.json.

… into BUILD_STRATEGY.md level3/README.md was a development timeline. It is now organised for a researcher who has just cloned the repository: what Level 3 is, a prominent current-availability box (source artifacts NOT published, code still on the PR branch, single-node 1/2/4 GPU CUDA only, workspace coverage partial), the catalog of the ten applications with frozen version, the workload that is actually validated and app-owned cloc LOC, prerequisites split into download / build / run, a quick start that separates the future published flow from today's --artifact flow, GPU-selection semantics, the artifact/hash/cache/offline model, the LLM workspace with its real directory tree and trusted-baseline rules, evidence status with the criteria sources kept apart, known limitations, and troubleshooting. GEOS appears only as retired; nekRS's two variants are named where they matter. The policy and convention sections that previously lived only in that README (Level 3 hard requirements, how the correctness criteria were chosen, per-application dependency isolation, the launcher contract, per-application layout) moved verbatim into level3/BUILD_STRATEGY.md and are linked from the README, so nothing is lost. WORKSPACE_EVIDENCE.md: LAMMPS 2-GPU workspace validation and its separate binding run, ExaCA's unit-test result as evidence only, the two remote-fetch steps, and the build-provenance levels per iteration. Root README: the Level 3 row now states the actual status (ten applications, artifacts unpublished, merge pending) instead of "planned"; Level 1/2 rows untouched.

…assets, 1.11 GiB, still unpublished) Target commit is the code/metadata commit that precedes this one (a plan cannot record its own commit id); the note now says so and asks for a regeneration plus --expect-plan-sha256 at the final reviewed head before any upload. Every artifact was re-extracted for this plan (layout, source_tree_sha256, entry count, symlinks, secret/build-output scan) and verified against the staged files: 11 assets, 1,197,001,594 bytes, plus SHA256SUMS, one SOURCE_MANIFEST per artifact and the plan itself. ExaCA is now included (admitted); GEOS stays excluded permanently. All locks remain url null / status unpublished.

…ation defects), add the ON_HOLD acceptance state, restore acceptance on the evidence Release acceptance was put ON_HOLD because 22 of 52 upstream unit tests failed with the CUDA Nucleation assertions and the 2/4-rank Update aborts unexplained. Both are now explained, per test, with before/after runs on this node (level3/exaca/references/upstream_unit_test_matrix.json: ctest name, backend, ranks, assertion or exit code, expected vs actual, command, class, evidence path): - TEST_FIXTURE (7): tstNucleation rebinds a local grain_id handle with create_mirror_view_and_copy instead of writing celldata's own subview, so on CUDA the fixture's grain IDs never reach the data the kernel and the assertions read. Instrumented proof before any fix: aliases_celldata=0, celldata_subview[2,6,8]=0,0,0 while the local handle held 1,2,3. tstOrientation reads a create_mirror_view that was never copied. tstInterface calls calcCritDiagonalLength and createNewOctahedron (KOKKOS_INLINE_FUNCTIONs) from host loops and indexes the device view octahedron_center_test ("DOCenter") on the host. Fixture-only patches under level3/exaca/patches/upstream-tests/ (label patched-upstream-tests; upstream commit, target hashes and patch hashes recorded) make all of them pass; every EXPECT_* and expected value is unchanged and no production source is touched. - UNSUPPORTED_CONFIG (13): the host-space ("SERIAL") test variants inside a CUDA-enabled build -- Kokkos' default execution space is Cuda, so application kernels touch HostSpace views. A separate Serial-only Kokkos build runs the same 23 tests: all pass. - TEST_INFRA (2): ExaCA_Update_test_CUDA at 2 and 4 ranks. CTest launches "mpiexec -n N <exe>" with no per-rank GPU assignment (all ranks on GPU 0) and the run segfaults inside runExaCA; through the validated launcher the same binary passes at np1/2/4. Six earlier failures were of the same class (data resolved via the install prefix, fixed by cmake --install). - APPLICATION: 0. UNRESOLVED: 0. Correction recorded: upstream DOES ship full-application numerical references -- inside tstUpdate.hpp, VolFractionNucleated 0.1882 +- 0.0100 (Inp_SmallDirSolidification) and TimeStepOfOutput 4820 +- 1 (Inp_SmallEquiaxedGrain, with an upstream FIXME about a race). Both pass here at 1, 2 and 4 real GPUs (0.1881/0.188/0.191 and 4820/4820/4820, launcher audit N verified / 0 mismatch). They apply to those small official cases only and do not relax the frozen 128^3 dirsolid protocol, which is unchanged (calibration + 9/9 holdout). Acceptance state machinery: the catalog and benchmark.yaml now carry release_acceptance with its history, and release_plan.py / publish_plan.py exclude an on_hold artifact from the plan and refuse it in the publish preflight (verified with a temporary catalog: 10 assets instead of 11, exaca REFUSE release_acceptance=on_hold). ExaCA's state is restored, with the condition recorded that unmodified upstream tests under CTest's own launcher still fail in this environment. The frozen source archive, its source_tree_sha256 and the nine other artifacts are untouched; the test patches are not part of the artifact or the release payload. Raw logs, ctest records and the per-suite matrices are kept in the gitignored .hpcperf/test-evidence/exaca-unit-tests/ (original failure logs preserved unmodified).

…et content before publication Two requirements that a release-only view cannot satisfy: - Git tag: /repos/.../git/ref/tags/<tag> is now consulted, not only /releases/tags/<tag>, because a tag can exist without a release. A lightweight tag resolves directly, an annotated tag through /git/tags/<sha>, to a full commit SHA. Before a draft the tag must not exist at all (tags are never reused or moved); at publication time the tag GitHub created must resolve to the plan's target commit. release.target_commitish is not used as evidence, since GitHub ignores it when the tag already exists. - Remote content: before the draft is flipped to public, every remote asset is verified by digest -- the API's own `digest` when present, otherwise by re-downloading and hashing -- against the reviewed plan (archives), the plan-derived SHA256SUMS, the recorded SOURCE_MANIFEST hashes and the local plan file. The asset list is paged to completion and every asset must be in state `uploaded`. Equal byte size is never accepted as evidence, and --no-redownload refuses when the API offers no digest instead of trusting the size. The same verification runs again after publication, together with the tag check. Mock coverage (tools/artifacts/tests/, CPU only, no real API): the mock now serves Git refs and annotated tag objects, asset digests and paged listings, and creates the tag when a draft is published. New faults: git-tag-exists-elsewhere, annotated-tag-wrong-commit, asset-content-mismatch (same size, different bytes), manifest-replaced, plan-replaced, asset-not-uploaded, no-digest. Each is asserted to block publication with the release still a draft, i.e. no public PATCH is ever issued on a failed check; the good path is asserted to verify content before publishing and the created tag afterwards. 42 checks.

…'s upstream suite; record ExaCA's acceptance state - level3/README.md: the 239/239 figure is named as THIS repository's harness, validator and publisher regression suite (7 groups, CPU only) and explicitly not ExaCA's upstream 30/52; the criteria list now distinguishes upstream reference output, analytic solutions, adapted subsets, cross-rank checks, CPU cross-checks, the project-defined statistical protocol AND upstream full-application references (ExaCA's two small official cases); ExaCA's three statements are kept apart (project smoke protocol passes / upstream applicable tests fail as upstream runs them / acceptance was on hold and is restored); a numerical PASS with build_provenance UNVERIFIED is kept as a scientific result for the binary that ran but must not be used as evidence of a verified source modification or an optimization gain; the availability box carries the acceptance state; known limitations updated with the attribution counts. - level3/WORKSPACE_EVIDENCE.md: ExaCA's row records the kernel-unit-test attribution, the full-application cases passing at 1/2/4 GPUs and the ON_HOLD -> restored transition. Unchanged: the unpublished/not-merged notices, Nyx PENDING, QMCPACK's limit, HIP and multi-node status, and the ten-application catalog.

…eptance decision (11 assets, still unpublished) Target commit c72d669 (the docs commit that precedes this metadata commit). ExaCA is planned again now that its release acceptance is restored; had it stayed on hold the plan would carry 10 assets and the publish preflight would refuse it with release_acceptance=on_hold (verified with a temporary catalog). Every artifact was re-extracted for this plan (layout, source_tree_sha256, entry count, symlinks, secret and build-output scan). Totals unchanged: 11 assets, 1,197,001,594 bytes, plus SHA256SUMS, one SOURCE_MANIFEST per artifact and the plan itself. GEOS remains permanently excluded. All locks keep url null / status unpublished; nothing has been uploaded.

…tus wording Two closing changes before the source-artifact prerelease is created. 1. `github_release_publish.py` no longer accepts an asset whose `state` is missing or null. Both places that judged an asset -- the upload response and the remote content verification that runs before the public PATCH -- required `state not in (None, "uploaded")`, so a response without the field, or with `state: null`, passed as if GitHub had reported a completed upload. The check is now `state != "uploaded"`, i.e. only an explicit `uploaded` is accepted and an absent field is a failure, not a default. `mock_github.py` gains the faults `asset-state-missing` (no `state` key at all) and `asset-state-null`, applied to both the upload response and the asset listing, and the mock suite asserts that each aborts at upload time: no draft id is carried forward and no publish request is ever sent. Release mock 42 -> 44 checks, `level3/tools/tests/run_all.sh` 239 -> 241. 2. ExaCA status wording, kept as three separate statements everywhere: the first observation (30 of 52 upstream unit tests passing as upstream runs them, and unmodified upstream CTest still failing in this environment today), the attribution of those 22 failures, and the patched-upstream-tests result (23/23 with the fixture-only patches plus per-rank GPU binding, 23/23 in a Serial-only build). `0 APPLICATION` is now explicitly scoped to this attribution matrix, on this machine, for the tests upstream ships at d26e59cd -- it is not a claim that ExaCA has no defects here or elsewhere. Both READMEs state that the fixture patches live in git under `level3/exaca/patches/upstream-tests/` and are deliberately not part of the frozen source artifact. The historical records keep their original text: the first-observation file `references/upstream_unit_tests.json` still carries its "unexplained" entry verbatim and now also carries that entry's resolution, and the catalog's `on_hold` history line is marked as the state at the moment the hold was placed. No failure record was overwritten. No production application source, dependency or archive content is touched by this commit; the 11 frozen artifacts and their hashes are unchanged.

RELEASE_PLAN.json/.md regenerated with --full (every staged artifact re-verified: lock validity, size + zstd magic + sha256, staging metadata agreement, restricted re-extraction, source-tree hash, manifest agreement, secret/build-output scan -- 9 checks per artifact, 11/11 PASS). plan.target_commit = 16dcf18, the commit that carries the code and the wording this release is cut from. 11 assets, 1,197,001,594 bytes, plus SHA256SUMS, one SOURCE_MANIFEST per artifact and this plan = 24 upload objects. GEOS stays excluded as RETIRED (never published). The locks still record primary {url: null, status: unpublished}; this commit plans the publication, it does not perform it. This file is the frozen plan for the upload: its sha256 is passed to --expect-plan-sha256 on every publisher invocation from the first upload to the public release, so a plan edited mid-flight aborts the run.

…d check The prerelease level3-source-hpcperf-l3-v1-rc1 exists (release 387492391, tag resolved through /git/ref/tags to 16dcf18, the plan target). All 11 source locks now carry `primary: {url: <release asset URL>, status: published}` instead of `{url: null, status: unpublished}`. Nothing else in any lock changed: the filename, size, archive sha256 and source_tree_sha256 are the ones that were frozen before the upload, and they are what the download was checked against. Order matters and was kept: the URL was written only after an anonymous fetch (no GITHUB_TOKEN, no GH_TOKEN, fresh empty cache) had downloaded each archive from the release and reproduced the size, the archive sha256 and the source_tree_sha256 that the lock already contained. A URL can never redefine what an artifact must be. Those records are added here as provenance/remote_artifact_verification*.yaml, 11 of them, all PASS. Also fixed, because it was found by that run: verify_published_artifact.py did not create a caller-supplied --cache/--scratch directory, so every anonymous check aborted with "No such file or directory" from zstd immediately AFTER a correct download and hash verification. The archives were never in question. remote_fetch_check.sh always passes paths inside a fresh mktemp -d, so this hit all 11. Test 4a2 covers it: both directories are now created, not assumed. Release mock 44 -> 45 checks, level3/tools/tests/run_all.sh 241 -> 242. REMOTE_FETCH_VERIFIED is not claimed by this commit. It requires the second, ordinary-user check: a clean clone of this commit, an empty cache and a plain tools/prepare_benchmark.sh, recorded separately.

…release The ordinary user entry now works and was measured, not assumed: for all 11 artifacts, a clean clone of the lock-update commit with an empty cache, no credentials and a plain tools/prepare_benchmark.sh downloads the archive from the URL in the lock, verifies size, archive sha256 and source_tree_sha256, and passes check_workspace. Both nekRS variants were fetched into separate directories. Records: provenance/remote_fetch_verification*.yaml, 11 PASS. level3/PUBLICATION_RECORD.md collects the whole sequence: the release identity, the seven checks that ran, the one tooling defect found during the run and its fix, and an explicit list of what this release does not claim. It also records that the token used for the upload is the one the 2026-09-07 security record lists as exposed with rotation not confirmed, authorized for this publication by the maintainer; no credential value was printed, written to a file or placed on a command line, and the anonymous checks ran with the token removed. Documentation now describes the published state, and only that: - level3/README.md: the availability box says the artifacts are published and the code is NOT merged into main, so a plain clone of the default branch still does not contain these tools. Quick start A is the published flow (clone, check out the branch or the tag, prepare downloads), B is the offline/local file flow. Remote fetch by an ordinary user moves from "not run" to VERIFIED. - SOURCE_ARTIFACTS.md: the 11 retained artifacts become REMOTE_FETCH_VERIFIED; GEOS stays unpublished and is stated as never to be published. - WORKSPACE_EVIDENCE.md: the remote-fetch column is VERIFIED for the 11, GEOS stays NOT_RUN by decision. - root README: Level 3 says published artifacts, unmerged code. Nothing about the scientific status changed here, and no such claim is added: Nyx LyA heat/cool stays PENDING, eight applications keep historical build evidence, ExaCA keeps the dirsolid protocol plus upstream's two small official cases, and unmodified upstream ExaCA tests still fail under CTest's own launcher in this environment.

SWE-bench and others added 30 commits September 4, 2026 21:28

Nyx: heat/cool strong-scaling timings and 8/40/80 dry-run record

c7b90f9

LyA heat/cool 64^3 on 1/2/4 GPUs: 2.85 / 2.62 / 2.65 s (too small to scale, fixed-cost dominated); dry-runs 8 planned, 40 IMBALANCED, 80 refused as designed. nyx_particle_compare.py made executable.

Nyx run.sh: run directory through $L3_RUN_SUBDIR (left out of 6711980)

fc4d2a1

CLAUDE.md: regression run trees (HPCPERF_L3_RUN_SUBDIR) and the allow…

74c1d4d

…-listed environment wrapper

bowencui123 added 7 commits September 10, 2026 18:11

bowencui123 changed the base branch from level3/second-batch-bringup to main September 11, 2026 06:10

bowencui123 added 5 commits September 11, 2026 02:57

bowencui123 changed the title ~~Level 3 source distribution: external source artifacts, trusted agent workspace, GEOS retired, ExaCA admitted~~ Level 3: ten production HPC applications, external source artifacts, trusted agent workspace (supersedes #4) Sep 11, 2026

bowencui123 mentioned this pull request Sep 11, 2026

Add Level 3 multi-GPU applications and validation safeguards #4

Closed

bowencui123 added 8 commits September 11, 2026 19:38

Navigation Menu

Uh oh!

Level 3: ten production HPC applications, external source artifacts, trusted agent workspace (supersedes #4) - #5

Level 3: ten production HPC applications, external source artifacts, trusted agent workspace (supersedes #4)#5
bowencui123 wants to merge 50 commits into
mainfrom
level3/source-freeze

bowencui123 commented Sep 11, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

bowencui123 commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is in this PR

Status that this PR does not change

Reading the diff

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bowencui123 commented Sep 11, 2026 •

edited

Loading