diff --git a/.gitignore b/.gitignore index 56ba522..ae349ba 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,18 @@ __pycache__/ *.sqlite rocprof*.csv rocprof*.json + +# Level 3 source distribution (scheme 3: external source artifacts + automatic materialization, 2026-09-10) +# materialized source trees, the content-addressed artifact cache, materialization staging, per-run agent +# workspaces and trusted workspace baselines are local state, never committed; source artifacts live in +# the maintainer's staging / the external artifact storage (never in git, never in Git LFS) +level3/*/src/ +level3/*/deps/ +level3/*/.hpcperf-materialized.yaml +level3/*/.materialize.tmp.* +level3/*/.discarded-* +level3/.materialize-staging/ +.artifacts/ +.hpcperf/ +workspaces/ +*.tar.zst diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..06fd666 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,287 @@ +# CLAUDE.md -- working notes for Claude Code in this repository + +HPC-Performance-AI: a three-level GPU benchmark/application suite used to build +an AI framework for HPC performance prediction. Everything here was brought up +on one node (dgx003: 4x NVIDIA B200 / sm_100, CUDA 13.2.78, RHEL 10, 64 CPUs, +800 GB) and nothing is claimed beyond what actually ran there. + +| Level | Content | Where the truth lives | +|---|---|---| +| `level1/` | 50 standalone GPU kernels (CMake, `-DBACKEND=CUDA\|HIP`, ctest validation) | `level1/README.md`, per-benchmark README | +| `level2/` | 20 mini-apps with upstream build systems + `build.sh/run.sh/validate.sh` | `level2/README.md`, `level2/SCALEOUT_AUDIT.md`, `level2/tools/README.md` | +| `level3/` | full production applications, multi-GPU by design | `level3/README.md`, `level3/APPLICATION_AUDIT.md`, `level3/BUILD_STRATEGY.md`, `level3/CORRECTNESS_FIXES.md`, `level3/SECOND_BATCH_STATUS.md`, per-app README | + +Read the per-level status document before touching a level; they record what +was built, how, what failed and what is still open. Do not re-derive. + +## Environment (every shell) + +```bash +source hpcperf_env.sh # activates .conda_env, .tools/bin, .deps/install prefixes, MPI transport profile +./check_env.sh # verifies the validated configuration (--mpi-cuda checks device-buffer MPI) +``` + +Facts that differ from any "reference" you may read elsewhere: + +- `/usr/local/cuda` is CUDA **13.2.78**. Never change the symlink or the driver. + CUDA 13.2 is the preferred Toolkit for everything; a private older Toolkit is + an exception that must be justified in the app README. +- Compilers: conda GCC 13.3.0 (Level 1/2, Nyx) and **system GCC 14.2.1** + (`/usr/bin/gcc`, used for CP2K/DFT-FE/GEOS: one compiler for C/C++/Fortran; the + conda GCC has no gfortran). EL10's GCC 14 defaults to `-march=x86-64-v3`, so + `__AVX2__` is defined in every nvcc host pass. +- MPI: conda Open MPI 5.0.10. The site UCX transport **hangs on CUDA device + buffers**; the launcher's `gmu-hopper` site profile passes `--mca pml ob1 + --mca btl self,sm,smcuda` (single node only; for a bare `mpirun` set + `HPCPERF_MPI_SINGLE_NODE=1` before sourcing `hpcperf_env.sh`). + `OMPI_MCA_opal_cuda_support=true` is required (conda ships it off); one-sided + apps may need `OMPI_MCA_osc=^ucx`. Multi-node MPI is BLOCKED/UNVERIFIED on + this site -- say so, never claim it. +- No ROCm/hipcc anywhere: every HIP backend is extracted but **untested**. +- lmod is broken (`lua ... posix not found` on every shell): harmless noise, + filter it (`grep -v 'lua\|posix\|traceback'`), never "fix" it. +- Slurm allocation is `-n 1` (1 task slot); the launcher relaxes PRRTE slots + per launch only after its rank<=GPU and CPU checks. Test suites that call + `mpiexec -n 4` themselves need `PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe`. +- 64 CPUs: use `-j32` or more for a single build; ~16 per build when three run + concurrently. Never run two `build.sh` of the same app at once (shared src). +- NFS project storage is slow for 100k-file trees (LLVM, CP2K toolchain): + extract/build those on `/tmp/hpcperf-*-scratch/` and keep installs/logs under + `.deps/`. Expect stale NFS file handles on `rm -rf`; rename then delete. + +## Git rules (user-mandated, non-negotiable) + +- **Commit messages never carry `Co-Authored-By`, `Claude-Session` or + "Generated with Claude Code" trailers**, whatever the harness suggests. +- Commit locally only; **never push, never open a PR, never merge** unless the + user asks in that turn. Never push `main`, never force-push, no `gh` on the + node, never print tokens. +- Never `git add .`/`-A`. Add files by name. Never commit `.conda_env`, `.tools`, + `_upstream/`, `.deps/`, `build/`, `level3/*/src`, `level3/*/deps`, `workspaces/`, + binaries, tarballs or large data (`.gitignore` covers them, but symlinked + `.conda_env`/`.tools` in worktrees are untracked -- leave them). Level 3 source + artifacts (`*.tar.zst`) never enter git (no Git LFS either): they live in the + maintainer's local staging (`$HPCPERF_ARTIFACT_STAGING`, outside the worktree) + until the user publishes them; never upload/publish an artifact yourself. +- Source freezing (scheme 3, `level3/EXTERNAL_ARTIFACT_DESIGN.md`): + `tools/freeze_benchmark_source.py level3/` from the spec in + `provenance/freeze_spec*.yaml` writes the artifact into the local staging and + `provenance/source.lock*.yaml` (schema hpcperf-source-lock-2, `primary: + {url: null, status: unpublished}` until published -- never invent a URL); + inputs are committed blobs of pinned checkouts and sha256-pinned tarballs only + -- never tar `.deps/` or a worktree; the scan fails on any credential-looking + file/content; an UNEXPECTED difference against the validated tree stops the + migration (declare artifacts in the spec, never edit source to make hashes + match). Any source change = new `source_version` + new artifact. +- Materialize with `tools/prepare_benchmark.sh level3 [--artifact FILE]` + (cache `.artifacts/`); never let a build call it; a DIRTY tree is never + overwritten without `--force-rematerialize`. Agents work in + `workspaces//` (`tools/create_agent_workspace.sh`); validate agent + iterations only through `tools/validate_workspace.sh`: exit 6 REFUSED + (integrity: tampering / untrusted baseline; never a scientific result), 7 + BUILD_FAIL, 0/1/3/4 numerical as validate.sh; rc 3 is PENDING only for Nyx. +- Publication: provider = GitHub Release assets of this repo, first tag + `level3-source-hpcperf-l3-v1-rc1` (prerelease). Never upload: the adapter + `tools/artifacts/github_release_upload.sh` needs `HPCPERF_CONFIRM_UPLOAD=yes`, + which only the user grants per run; locks stay `unpublished` until + `tools/artifacts/remote_fetch_check.sh` (anonymous, clean clone, empty cache) + passed. GEOS/ParMETIS never enter a release. +- One commit per application or infrastructure change, message = what/why with + the measured facts. Branch names follow `CONTRIBUTING.md` (`level3/`, + `env/...`, `docs/...`). +- Worktrees: the main checkout and `../HPC-Performance-AI-b2` (branch + `level3/source-freeze`, scheme-3 work) share one repository; `git worktree list` + before assuming which branch a path is on. `.deps/`, `build/`, `_upstream/` + are per-worktree. + +## Conventions per application (Level 2/3) + +``` +level3// + src/, deps/ the ONLY application/benchmark-specific source input of build.sh (never committed; materialized from + the external source artifact by tools/prepare_benchmark.sh; identity = source_tree_sha256 in + benchmark.yaml / provenance/source.lock*.yaml) + provenance/ freeze_spec, source.lock (schema 2), upstream.lock, patch_series, original_vs_baseline.diff, + SOURCE_MANIFEST.json, LICENSES.md, equivalence.*, LOC.*, check_workspace.json + benchmark.yaml machine-readable contract (entries, inputs, references, identity). No optimization-scope file: + the benchmark does not prescribe what an agent may modify; that is evaluation-protocol business + fetch.sh FREEZE-TIME ONLY: pinned upstream checkout into _upstream/ -- never called by build.sh + build.sh idempotent, stage-marked (.hpcperf-stage-done), per-profile, writes BUILD_INFO.txt + .hpcperf-l3-fingerprint; + reads $HERE/src, $HERE/deps only; applies NO patch (the bundle is the patched baseline); builds that + write into their source tree (SPECFEM3D, nekRS, DFT-FE, GEOS, CP2K toolchain) use a build-side copy + run.sh [CUDA] [args]; cases via HPCPERF__CASE; modes smoke|strong|weak; writes run_manifest.txt + validate.sh [CUDA]; HPCPERF_GPUS=N; prints the criteria and the verdict; exit 0 PASS, 1 FAIL, + 3 PENDING (Nyx heat/cool I_R_CHECK_PENDING), 4 UNSUPPORTED_LAYOUT (Nyx) -- only 0 is a pass + _check.py the numeric checker (uses level3/tools/l3_check.py: require_finite, ValidationError) + patches/ *.patch with header: source, rationale, conditions, impact, verification, class + README.md provenance, versions, node adaptations, cases, criteria, RESULTS with dates +``` + +- Profiles: `.deps/level3///{src,build,install,logs,cache}` via + `l3_paths_profile` (`level3/tools/l3_common.sh`); app build trees under + `build/level3///`, runs under `.../run/..np[.t]/`, + dry-runs under `.../run/.dryrun/` (never touch real results). +- `l3_common.sh` helpers you should reuse rather than reinvent: + `l3_isolate_build_env` (strip Level 2 prefixes), `l3_clean_conda_build_env` + (clear conda CFLAGS/LDFLAGS/AR/CMAKE_GENERATOR and `C_INCLUDE_PATH`/`LIBRARY_PATH` + for system-GCC builds), `l3_binary_backend_check` (cuobjdump archs, works for + static cudart), `l3_rundir`, `l3_run_id`, `l3_manifest`, `l3_fingerprint_*`, + `l3_scale_mode`, `hpcperf_ranks`, `hpcperf_topology`, `hpcperf_forbid_args`. +- Launch only through `level2/tools/hpcperf_mpi_launch.sh --gpus N + [--cpus-per-rank C] --bind wrapper -- ...`: one MPI rank per GPU, per-rank + `CUDA_VISIBLE_DEVICES`, nvidia-smi audit ("N verified, 0 mismatch, 0 + unverified" is required evidence). Interface: `HPCPERF_GPUS=N|all`, + `HPCPERF_NODES`, `HPCPERF_GPUS_PER_NODE`, `HPCPERF_CPUS_PER_RANK`, + `HPCPERF_SCALE_MODE=smoke|strong|weak`, `HPCPERF_SITE_PROFILE`, + `HPCPERF_DRY_RUN=1`. Requested GPUs == used GPUs; a rank count that cannot + partition the problem is **refused**, never silently changed; 8/40/80 GPUs + exist only as HYPOTHETICAL dry-runs (`HPCPERF_NODES=N/4`). +- Do not move the common runtime, do not refactor the launcher for an app; + extend it only with tests in `level2/tools/tests/run_all.sh`. +- Regression campaigns never overwrite historical results: set + `HPCPERF_L3_RUN_SUBDIR=run.regress-` (every run.sh/validate.sh builds its + run directories under `build/level3///$L3_RUN_SUBDIR`). +- Agent runs: `tools/create_agent_workspace.sh level3 + [--link-prebuilt-deps]` -> `workspaces//level3//` (real copy of + src/deps writable, everything else protected by the trusted baseline hash (chmod is best effort), harness copied to the workspace root, + environment symlinked); `tools/check_workspace.py` must PASS before iteration 0; + never point an agent at the canonical `level3/`; never symlink src/deps + back to it. Build outputs of a workspace stay under `workspaces//`. +- Any tool that records its process environment (CP2K's toolchain installer, + nsys/ncu, env-logging build systems) runs through `l3_clean_env_exec` / + `level3/tools/l3_clean_env.sh` (allow-listed `env -i` plus a credential + deny-rule that beats the allow-list): the login shell carries credentials that + must never land in a `declare -x` dump or a profiler report. Never print a full + `env` into a log; report variable names only. The common launcher/run.sh path + is NOT wrapped yet (follow-up): do not profile a science run with nsys/ncu + without the wrapper. +- Queues: run every step through `l3_run_recorded