Repository navigation
test: installed-binary tests with manifest-gated publish - #5
Merged
Merged
Conversation
Build -> upload -> independent install tests -> publish: - each build job writes a per-artifact manifest (filename, sha256, product version, product commit, engine sha, HiGHS version) - test-resource job ships the test suite and tools scripts as the r-install-tests artifact; install jobs never check out the product - three fresh-environment install jobs (linux x86_64 in an ubuntu:22.04 container keeping the clean-room checks, macos arm64, windows x86_64) install the exact binary of this run into an empty library in a spaced path, verify its identity against the build manifest, and run the full test suite, which now includes the portable ex1 .arb job (OPT suppression plus type-1 output and the native type-5 AR+ protection audit, validated against hand-derived expectations shared with the Python installed tests) - the publish job verifies every file against the build manifests, asserts platform coverage and lockstep, and publishes exactly the verified files plus an aggregated release-manifest.json - PRs get the build matrix plus all install tests; publishing stays tag-gated - replaces the inline clean-room smoke (subsumed by the full install test); the ldd/NEEDED and source-tarball checks stay in the floor job Verified locally: staged build, fresh spaced-path library, install-check green, full suite 409 passed 0 failed
The CRAN test dependencies are installed by install.packages, which needs a writable library; on the macos and windows runners the stripped default user library would not be one. The freshness guarantee is the empty per-job library (RLIB) first in the path, not a stripped environment.
There was a problem hiding this comment.
🟡 Changes recommended
Manifest generation and installation-test defects block the release pipeline, while unexpected audit statuses can bypass validation.
7 open issues
What changed in this PR
Adds installed-binary validation to the R release pipeline, with publishing gated on installation tests and artifact manifests.
Changes:
- Adds fresh-environment installation tests for Linux, macOS, and Windows.
- Adds an ex1 suppression and protection-audit test with portable fixtures.
- Adds artifact manifests and pre-publish verification.
| File | Description |
|---|---|
tools/install-check.R |
Checks installed package identity and native-library layout. |
tests/testthat/test_installed_arb.R |
Validates suppression, protection audit, and published output. |
tests/testthat/fixtures/ex1/sto.cdl |
Defines size-class codes. |
tests/testthat/fixtures/ex1/reg.cdl |
Defines region codes. |
tests/testthat/fixtures/ex1/ex1.tab |
Supplies input cells and manual suppression status. |
tests/testthat/fixtures/ex1/ex1.rda |
Defines table metadata. |
tests/testthat/fixtures/ex1/ex1.expected |
Records expected values and protection requirements. |
tests/testthat/fixtures/ex1/ex1.arb |
Supplies the example batch. |
.github/workflows/release.yml |
Adds manifests, installation jobs, and publishing gates. |
Review effort: Balanced
| echo " \"highs_version\": \"$HIGHS\"," | ||
| echo " \"platform\": \"${{ matrix.osname }}-${{ matrix.arch }}\"," | ||
| echo " \"files\": [" | ||
| echo " {\"name\": \"$BIN\", \"sha256\": \"$(sha_of "build/bin/$BIN")\", \"size\": $(wc -c < "build/bin/$BIN" | tr -d ' ')}]" |
| echo " \"highs_version\": \"$HIGHS\"," | ||
| echo " \"platform\": \"source\"," | ||
| echo " \"files\": [" | ||
| echo " {\"name\": \"$SRC\", \"sha256\": \"$(sha_of "build/dist/$SRC")\", \"size\": $(wc -c < "build/dist/$SRC" | tr -d ' ')}]" |
| } > "build/dist/manifest-source.json" | ||
| fi | ||
| cat build/bin/manifest-*.json | ||
| [ -f build/dist/manifest-source.json ] && cat build/dist/manifest-source.json |
| name: assemble install test resource | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v4 |
| expect_identical(as.numeric(r[14L]), 0) | ||
| expect_identical(as.numeric(r[15L]), 0) | ||
| } else { | ||
| skip(paste("unexpected status", st, "for", key)) |
|
|
||
| stopifnot(dir.exists(file.path(lib, "rtauargus.native"))) | ||
| pkg_path <- find.package("rtauargus.native", lib.loc = lib) | ||
| stopifnot(normalizePath(pkg_path) == file.path(lib, "rtauargus.native")) |
| if (length(main) != 1L) | ||
| stop("the rtauargus.native shared object is not loaded") | ||
| cat("loaded native library:", main, "\n") | ||
| if (!startsWith(main, normalizePath(pkg_path))) |
The manifest step ended with `[ -f ... ] && cat ...`; on macos and windows (no source tarball) the test is the last command and exits 1 under pipefail. Use an if/fi block.
On Windows the getLoadedDLLs path and normalizePath may disagree on the separator style, so startsWith could fail on a correct install. Normalize both sides; a no-op on POSIX.
…rence GNU tar on the Windows runner cannot write zips, so `tar -a -c -f x.zip` silently emitted a plain tar with a .zip name; R CMD INSTALL cannot unzip it (the 0.5.6 Windows binary shipped in exactly this state). Create the zip with PowerShell Compress-Archive and verify it with R's own zip reader (tools/check-zip.R). Also allow the main module's self-reference in the installed dylib check: it is linked against its own install name, as in the 0.5.6 binary.
…lve libR path R's unpackPkgZip derives the top-level directory it looks for from the zip file name (basename minus .zip and the trailing _[0-9.-]+), so a zip containing only rtauargus.native/ failed with 'cannot open ... DESCRIPTION'. Stage the installed package under the derived name before zipping, and teach tools/check-zip.R the expected directory. Also fix the macOS dylib reference check: the module references libR.dylib by its versioned framework path, but R RHOME returns the versionless R.framework/Resources symlink, so resolve through R.home() + normalizePath.
…d package Two root causes from run 39035454677: - windows install: R's unpackPkgZip derives both the top-level directory it looks for in the zip and the library directory it installs into from the zip file name (basename minus .zip and the trailing _[0-9.-]+), ignoring the DESCRIPTION Package field. The platform-suffixed name installed the package under rtauargus.native_1.3.7_r4.5.3_windows-x86/, so the install check could not find rtauargus.native/. Name the zip in the CRAN form rtauargus.native_<version>.zip (the platform stays in the manifest); the derived name is then exactly the package name. - macos install: the test-resource artifact carried the un-staged engine tests, which call tauargusengine:: explicitly, but the staged binary carries the binding functions in the rtauargus.native namespace (stage-r-package.sh rewrites the same tests in the build checkout). testthat 3.3.2 auto-skips missing :: packages only when on_cran(), so the bare linux container (NOT_CRAN unset) hid the mismatch with 19 skips while the macos and windows runners (NOT_CRAN=true) failed. The test-resource job now applies the same rewrite, with a guard that fails if stale references survive.
R CMD INSTALL of the correctly named zip now installs the package where expected, but the strict normalizePath equality failed on the windows install job. find.package is already restricted to the install library (a real mis-installation fails at the dir.exists check just above), so a case-only difference between normalizePath's disk-case answer and the constructed path is cosmetic on case-insensitive file systems: warn with both values instead of failing, and keep a hard failure with diagnostics for any real mismatch.
The round-7 diagnostic showed the mismatch was a separator style, not a case difference: found: C:\Users\runneradmin\rta install\Rlib\rtauargus.native expected: C:\Users\runneradmin\rta install\Rlib/rtauargus.native On Windows R defines FILESEP as '/' (src/include/Defn.h), so file.path() joins with forward slashes while normalizePath() outputs backslashes by default (winslash='\\'). The check compared the two raw outputs, which can never match on Windows. Normalize both sides; the directory identity is then compared in one canonical form on every platform.
The install URL in the release-notes template still pointed at the platform-suffixed zip name. R's zip installer derives the package directory from the file name, so the released asset is named rtauargus.native_<version>.zip. Note the broken 0.5.6 Windows asset (a plain tarball mislabeled as a .zip).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds realistic installed-binary (install + execute) release tests to the release pipeline.
What changes
tests/testthat/test_installed_arb.R): the complete portable ex1 .arb job through the publicrun_arb()driver: OPT suppression, type-1 matrix output and the native type-5 AR+ protection audit, validated against hand-derived expectations shared with the Python installed tests (exact cells and values, the preserved manual primary with adequate protection, every withheld cell strictly interior; the secondary pattern itself is not pinned).release-manifest.json.Verified locally: staged build of the package, fresh spaced-path library,
tools/install-check.Rgreen, full suite 409 passed / 0 failed / 20 skipped, plus a negative check (a corrupted expectation fails the protection assertions).