Repository navigation
fix: run pnpm, yarn 1 and yarn 4 the way their maintainers recommend #211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
15da8ab
27021a4
3f94699
fcf4f1d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -59,6 +59,40 @@ to finish, and every `--prepare` waits again right before the timed command | |
| `clean-helpers.sh`). The waits are untimed; in real use, the next install | ||
| rarely starts that soon. No other tool was seen leaving processes behind. | ||
|
|
||
| ## How each tool is installed and run | ||
|
|
||
| Each package manager is installed the way its maintainers recommend | ||
| (`suite/scripts/setup.sh`), and every timed command runs that tool directly, | ||
| the way a user would, with no extra launcher in front of it. | ||
|
|
||
| | Tool | Installed with | Timed command runs | | ||
| | ------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | ||
| | npm, vlt, bun, deno | `npm install -g <tool>@latest` | `<tool>` on `PATH` | | ||
| | pnpm | `npm install -g --allow-scripts=pnpm pnpm@latest` ([pnpm.io](https://pnpm.io/installation)) | `pnpm` on `PATH`, pnpm's native binary | | ||
| | yarn (1) | `npm install --global yarn@1` ([classic.yarnpkg.com](https://classic.yarnpkg.com/en/docs/install)) | that `yarn`, by full path | | ||
| | berry (Yarn 4) | `npm install -g corepack` ([yarnpkg.com](https://yarnpkg.com/getting-started/install)) | Corepack's `yarn` shim, by full path, with `packageManager: yarn@<version>` pinned | | ||
| | zpm (Yarn 6) | Yarn Switch (`curl -sS https://repo.yarnpkg.com/install \| bash`, [v6.yarnpkg.com](https://v6.yarnpkg.com/getting-started)) | Yarn Switch's `yarn`, by full path, with `packageManager: yarn@<version>` pinned | | ||
|
|
||
| Yarn 1, Yarn 4 and Yarn 6 all install a command called `yarn`, so yarn 1 and | ||
| Corepack get their own npm prefixes (`suite/scripts/pm-bins.sh`) and all three | ||
| are called by full path. npm 12 blocks install scripts by default; without | ||
| `--allow-scripts=pnpm`, pnpm's postinstall doesn't put its native binary in | ||
| place and `pnpm` runs through a JavaScript launcher instead. The berry version | ||
| is resolved once per job (`yarn@latest`) and pinned in `package.json` for each | ||
| berry run, as `yarn set version` would; `clean-helpers.sh` removes the pin | ||
| before the next tool runs. In `ci+setup`, the "setup" part of each timed | ||
| command is the same install (into a fresh prefix with an empty npm cache, so | ||
| it downloads every time); for berry that is installing Corepack plus | ||
|
Comment on lines
+84
to
+85
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Limit the fresh-prefix claim to the tools that use isolated caches. This guarantee applies to npm, Yarn Classic, pnpm and Berry's Corepack installation. Other installers, such as VLT's, still install globally with npm's default cache. Their prepare step populates that cache, and their uninstall step does not clear it. They therefore do not necessarily download the package manager again during the timed installation. State this distinction so the methodology describes the measured setup cost accurately. (docs.npmjs.com)
|
||
| Corepack's download of Yarn. | ||
|
|
||
| Until [#207](https://github.com/vltpkg/benchmarks/issues/207), pnpm, yarn and | ||
| berry ran through `corepack pnpm@latest`, `corepack yarn@1` and | ||
| `corepack yarn@latest`. That put a Node.js process (Corepack) in front of every | ||
| timed command, and for the `@latest` forms a registry lookup too: about half a | ||
| second per pnpm command, less for yarn. pnpm's (and to a lesser degree yarn's) | ||
| results drop at that point because that overhead is gone, not because pnpm | ||
| got faster. | ||
|
|
||
| ## Normalization: why per-package time matters | ||
|
|
||
| Package managers disagree about what a dependency graph contains. Comparing | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,9 @@ | ||
| # Exit on error | ||
| set -Eeuxo pipefail | ||
|
|
||
| # Where yarn classic, berry and zpm live (BENCH_YARN_BIN, BENCH_BERRY_BIN, ...) | ||
| source "$(dirname "${BASH_SOURCE[0]}")/pm-bins.sh" | ||
|
|
||
| # Function to safely remove files/directories | ||
| safe_remove() { | ||
| if [ -e "$1" ]; then | ||
|
|
@@ -17,9 +20,13 @@ clean_npm_cache() { | |
|
|
||
| # Function to safely clean yarn cache | ||
| clean_yarn_cache() { | ||
| if command -v corepack &> /dev/null; then | ||
| corepack yarn@1 cache clean --all --silent || true | ||
| if [ -x "$BENCH_YARN_BIN" ]; then | ||
| "$BENCH_YARN_BIN" cache clean --all --silent || true | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bypass the project pin check for Classic cache cleanup. If
|
||
| fi | ||
| # Yarn 1 refuses to run in a project whose packageManager names another | ||
| # yarn (a berry or zpm pin from the previous run), so also remove its cache | ||
| # directly: ~/.cache/yarn, or under $XDG_CACHE_HOME when that is set. | ||
| safe_remove "${XDG_CACHE_HOME:-$HOME/.cache}/yarn" | ||
| } | ||
|
|
||
| clean_berry_cache() { | ||
|
|
@@ -43,16 +50,16 @@ clean_zpm_cache() { | |
|
|
||
| # Function to safely clean pnpm cache | ||
| clean_pnpm_cache() { | ||
| if command -v corepack &> /dev/null; then | ||
| corepack pnpm cache delete '*' --silent || true | ||
| if command -v pnpm &> /dev/null; then | ||
| pnpm cache delete '*' --silent || true | ||
| # Remove the version-specific store directory reported by this pnpm | ||
| safe_remove "$(corepack pnpm store path 2>/dev/null | xargs)" | ||
| safe_remove "$(pnpm store path 2>/dev/null | xargs)" | ||
| fi | ||
| # Also wipe the entire metadata cache and store parent to catch any | ||
| # leftover version directories (v3, v9, v10, v11 …) that a single | ||
| # `pnpm store path` wouldn't return. | ||
| safe_remove "$HOME/.cache/pnpm" | ||
| safe_remove "$HOME/.local/share/pnpm/store" | ||
| safe_remove "${XDG_CACHE_HOME:-$HOME/.cache}/pnpm" | ||
| safe_remove "${XDG_DATA_HOME:-$HOME/.local/share}/pnpm/store" | ||
| } | ||
|
|
||
|
|
||
|
|
@@ -190,12 +197,15 @@ clean_lockfiles() { | |
| clean_package_manager_field() { | ||
| echo "Removing packageManager / devEngines.packageManager from package.json..." | ||
| if command -v vlt &> /dev/null; then | ||
| # The berry and zpm prepares pin packageManager to their own version; | ||
| # yarn classic and pnpm refuse to run in a project pinned to another | ||
| # package manager. | ||
| vlt pkg rm packageManager | ||
| # Recent corepack/yarn auto-pin a devEngines.packageManager entry (e.g. | ||
| # yarn) into package.json. npm and pnpm enforce devEngines and refuse to | ||
| # run any command when it doesn't match (EBADDEVENGINES), which breaks | ||
| # every benchmark that isn't that package manager. Strip it so each PM can | ||
| # run. vlt is used because it doesn't enforce devEngines, whereas npm/pnpm | ||
| # Some package managers can write a devEngines.packageManager entry into | ||
| # package.json. npm and pnpm enforce devEngines and refuse to run any | ||
| # command when it doesn't match (EBADDEVENGINES), which breaks every | ||
| # benchmark that isn't that package manager. Strip it so each PM can run. | ||
| # vlt is used because it doesn't enforce devEngines, whereas npm/pnpm | ||
| # would themselves fail on the field we're trying to remove. | ||
| vlt pkg rm devEngines.packageManager | ||
| fi | ||
|
|
@@ -288,10 +298,9 @@ clean_all() { | |
| clean_package_manager_files | ||
| clean_workspace_protocol | ||
| clean_all_cache | ||
| # Strip packageManager / devEngines.packageManager AFTER the cache cleans: | ||
| # the corepack-based cache cleans in clean_all_cache can re-pin a | ||
| # devEngines.packageManager entry into package.json, so this must run last | ||
| # to guarantee a neutral manifest before the next benchmark. | ||
| # Strip packageManager / devEngines.packageManager AFTER the cache cleans, | ||
| # so nothing they write survives: this must run last to guarantee a | ||
| # neutral manifest before the next benchmark. | ||
| clean_package_manager_field | ||
| clean_build_files | ||
| clean_build_output | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # Where setup.sh installs yarn classic and yarn berry, and the binaries the | ||
| # variations call. Sourced by setup.sh, variations/common.sh and | ||
| # clean-helpers.sh. | ||
| # | ||
| # Yarn 1 (`npm install --global yarn@1`), Yarn 4 (Corepack's `yarn` shim) and | ||
| # Yarn 6 (Yarn Switch) all install a `yarn` command. Each one lives in its own | ||
| # directory, none of them on PATH except Yarn Switch, and every benchmark calls | ||
| # them by full path so it's never ambiguous which one runs. pnpm has no clash | ||
| # and is called as plain `pnpm` (installed with `npm install -g` by setup.sh). | ||
| BENCH_PM_HOME="${BENCH_PM_HOME:-$HOME/.bench-pms}" | ||
|
|
||
| # Yarn 1, installed with npm into its own prefix. | ||
| BENCH_YARN_PREFIX="$BENCH_PM_HOME/yarn" | ||
| BENCH_YARN_BIN="$BENCH_YARN_PREFIX/bin/yarn" | ||
|
|
||
| # Yarn 4, through Corepack installed with npm into its own prefix: the corepack | ||
| # package ships `yarn` and `pnpm` shims of its own, which would collide with | ||
| # the npm-installed pnpm in the global prefix. Each berry benchmark pins the | ||
| # version setup.sh resolved in package.json#packageManager, like a project | ||
| # set up with `yarn set version`, and runs the `yarn` shim. | ||
| BENCH_COREPACK_PREFIX="$BENCH_PM_HOME/corepack" | ||
| BENCH_COREPACK_BIN="$BENCH_COREPACK_PREFIX/bin/corepack" | ||
| BENCH_BERRY_BIN="$BENCH_COREPACK_PREFIX/bin/yarn" | ||
| BENCH_BERRY_VERSION_FILE="$BENCH_PM_HOME/berry-version" | ||
| BENCH_BERRY_VERSION="${BENCH_BERRY_VERSION:-$(cat "$BENCH_BERRY_VERSION_FILE" 2>/dev/null || true)}" | ||
|
|
||
| # Yarn 6 (zpm), installed by the Yarn Switch installer. | ||
| BENCH_ZPM_BIN="$HOME/.yarn/switch/bin/yarn" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,18 +4,10 @@ | |
| set -e | ||
|
|
||
| # Environment Configuration | ||
| export COREPACK_ENABLE_STRICT=0 | ||
| export COREPACK_ENABLE_AUTO_PIN=0 | ||
| export YARN_ENABLE_IMMUTABLE_INSTALLS=false | ||
|
|
||
| # Persist corepack env vars for subsequent CI steps (e.g., benchmark runs). | ||
| # Without these, corepack 0.34+ auto-pins packageManager and refuses to run | ||
| # a different PM than what packageManager specifies — breaking benchmarks | ||
| # that cycle through multiple package managers on the same fixture. | ||
| if [ -n "${GITHUB_ENV:-}" ]; then | ||
| echo "COREPACK_ENABLE_STRICT=0" >> "$GITHUB_ENV" | ||
| echo "COREPACK_ENABLE_AUTO_PIN=0" >> "$GITHUB_ENV" | ||
| fi | ||
| # Install locations for yarn classic and yarn berry (BENCH_YARN_PREFIX, ...) | ||
| source "$(dirname "${BASH_SOURCE[0]}")/pm-bins.sh" | ||
|
|
||
| # Check Node version | ||
| REQUIRED_NODE_VERSION="26" | ||
|
|
@@ -45,7 +37,28 @@ echo "hyperfine: $HYPERFINE_VERSION" | |
|
|
||
| # Install Node.js package managers and tools | ||
| echo "Installing package managers and tools..." | ||
| npm install -g --allow-scripts=bun,deno,nx npm@latest corepack@latest vlt@latest bun@latest deno@latest nx@latest turbo@latest | ||
| npm install -g --allow-scripts=bun,deno,nx npm@latest vlt@latest bun@latest deno@latest nx@latest turbo@latest | ||
|
|
||
| # Install pnpm with npm, one of the methods in https://pnpm.io/installation. | ||
| # npm 12 blocks install scripts by default, but pnpm's postinstall is what puts | ||
| # its native binary in place of the JavaScript launcher, so allow it. | ||
| npm install -g --allow-scripts=pnpm pnpm@latest | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Remove legacy global Corepack before installing pnpm. If the previous setup ran on this machine, global
|
||
| PNPM_BIN="$(command -v pnpm)" | ||
| if [ "$PNPM_BIN" != "$(npm prefix -g)/bin/pnpm" ] || [ "$(head -c 4 "$(readlink -f "$PNPM_BIN")")" != $'\x7fELF' ]; then | ||
| echo "Error: expected pnpm on PATH to be the native binary installed by npm, got $PNPM_BIN ($(file -L "$PNPM_BIN" 2>/dev/null || true))" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Install Yarn 1 the way https://classic.yarnpkg.com/en/docs/install | ||
| # recommends (npm install --global yarn@1), into its own prefix: zpm's Yarn | ||
| # Switch and berry's Corepack shim are also called `yarn` (see pm-bins.sh). | ||
| # npm 12 skips yarn's preinstall script, which only removes a Corepack `yarn` | ||
| # symlink from the prefix being installed into; there is none in this one. | ||
| npm install -g --prefix "$BENCH_YARN_PREFIX" yarn@1 | ||
|
|
||
| # Install Corepack for Yarn 4, as https://yarnpkg.com/getting-started/install | ||
| # recommends (npm install -g corepack), into its own prefix (see pm-bins.sh). | ||
| npm install -g --prefix "$BENCH_COREPACK_PREFIX" corepack@latest | ||
|
|
||
| # Install Vite+ (vp) via npm (available as the `vite-plus` package) | ||
| npm install -g vite-plus@latest | ||
|
|
@@ -75,9 +88,12 @@ if ! command -v aube &>/dev/null; then | |
| fi | ||
| fi | ||
|
|
||
| # Configure Package Managers | ||
| echo "Configuring package managers..." | ||
| corepack enable yarn pnpm | ||
| # Resolve the berry version once, so every berry benchmark pins the same one | ||
| # in package.json#packageManager (and Corepack downloads it now, not in a | ||
| # timed run). | ||
| BERRY_VERSION="$("$BENCH_COREPACK_BIN" yarn@latest -v)" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use the selected Berry version for setup metadata. If
|
||
| mkdir -p "$BENCH_PM_HOME" | ||
| echo "$BERRY_VERSION" > "$BENCH_BERRY_VERSION_FILE" | ||
|
|
||
| # Install Yarn v6 Canary (zpm) | ||
| curl -sS https://repo.yarnpkg.com/install | bash | ||
|
|
@@ -92,10 +108,9 @@ mkdir -p ./results/ | |
| echo "Logging package manager versions..." | ||
| NPM_VERSION="$(npm -v)" | ||
| VLT_VERSION="$(vlt -v)" | ||
| YARN_VERSION="$(corepack yarn@1 -v)" | ||
| BERRY_VERSION="$(corepack yarn@latest -v)" | ||
| YARN_VERSION="$("$BENCH_YARN_BIN" -v)" | ||
| ZPM_VERSION="${BENCH_ZPM_VERSION:-$(curl -fsSL https://repo.yarnpkg.com/channels/default/canary)}" | ||
| PNPM_VERSION="$(corepack pnpm@latest -v)" | ||
| PNPM_VERSION="$(pnpm -v)" | ||
| BUN_VERSION="$(bun -v)" | ||
| DENO_VERSION="$(npm view deno@latest version)" | ||
| NX_VERSION="$(npm view nx@latest version)" | ||
|
|
@@ -120,6 +135,10 @@ echo "vp: $VP_VERSION" | |
| echo "aube: $AUBE_VERSION" | ||
| echo "upm: $UPM_VERSION" | ||
| echo "node: $NODE_VERSION" | ||
| echo "pnpm binary: $PNPM_BIN" | ||
| echo "yarn binary: $BENCH_YARN_BIN" | ||
| echo "berry binary: $BENCH_BERRY_BIN (packageManager: yarn@$BERRY_VERSION)" | ||
| echo "zpm binary: $BENCH_ZPM_BIN" | ||
|
|
||
| # Save versions to JSON file | ||
| echo "{ | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allow install scripts for bun, deno, and nx in the README install command.
The README command installs
bun,deno, andnxwithout the--allow-scriptsoption. npm 12 blocks lifecycle scripts by default. Those tools can be left without their setup step.suite/scripts/setup.shallows scripts for these packages, so the README should match it.Suggested fix
Committable suggestion