Skip to content

fix(ci): rewrite publish git/path dependencies for crates.io - #4

Merged
fffonion merged 2 commits into
masterfrom
subagent/pd-edge-ci-fix-20261117-1a3f5886
Nov 16, 2026
Merged

fffonion merged 2 commits into
masterfrom
subagent/pd-edge-ci-fix-20261117-1a3f5886

Conversation

@fffonion

@fffonion fffonion commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Fix Publish crates dependency rewriting so pd-host-function and aliased pd-vm git dependencies (as well as path dependencies) become crates.io versions, without retaining git/path/rev selectors; preserve package/features/default-features.
  • Add prepare-only Python regression coverage for tag 0.2.0 versions and alias/feature preservation.

Failure and version check

  • Publish crates run 43890104596 on tag 0.2.0 repeatedly failed selecting pd-host-function ^0.1.0. The existing rewrite only visited inline dependencies containing path; pd-edge-abi/Cargo.toml uses git + rev, so its old version remained. The vm alias had the same git selector.
  • The existing WORKSPACE_VERSION_OVERRIDES flow already derives pd-edge 0.2.0 from the tag when inputs.version is empty. pd-vm, pd-host-function and rustscript 0.22.2 registry versions were confirmed present, so no override change is needed.

Follow-up: root cause of the remaining 0.2.0 failure (run 43928737529)

The rewrite above was correct but structurally unable to work, because cargo publish -p <member> resolves the whole workspace before it packages anything. Rewriting pd-edge's path dependency into a registry requirement in place made the workspace unresolvable until pd-edge-abi 0.2.0 existed — and that only happens by publishing it from that same workspace:

publishing pd-edge-abi 0.2.0, attempt 1..18
error: failed to select a version for the requirement `pd-edge-abi = "^0.2.0"`
candidate versions found which didn't match: 0.1.1
location searched: crates.io index
required by package `pd-edge v0.2.0`

pd-edge-host-function 0.2.0 already published; skipping (it had gone out in the earlier run), so the job died on abi after ~6 minutes of pointless retries. --no-verify does not help: it skips the build, not resolution.

Fix (commit b1dd164):

  • .github/scripts/prepare_publish.py copies each released crate into its own standalone tree and rewrites that copy: path/git/rev/branch/tag are dropped, [workspace.package] inheritance (version, edition, …) is materialised, workspace crates get the release version, external crates get explicit registry versions (pd-vm / pd-host-function = 0.22.2), and dev-dependencies are removed (a crates.io consumer never builds them; reqwest must not gate a release). Any remaining path/git dependency without a known registry version fails the job loudly instead of shipping an unresolvable manifest.
  • Publish order is derived from the staged normal + build dependency graph (topological), so pd-edge-host-function → pd-edge-abi → pd-edge regardless of the declared order.
  • .github/scripts/publish_crates.py publishes with cargo publish --manifest-path <staged>/Cargo.toml, checks versions against the sparse index (index.crates.io) rather than only the crates.io API — the API answers before Cargo can see a version, which is the other way this exact error appears — and confirms a dependency is indexed before publishing its dependent. Already-published versions are skipped, and the deprecated --token flag is replaced by CARGO_REGISTRY_TOKEN.
  • publish-crates.yml now only derives the version and calls the two scripts; the rustscript checkout is gone because no published manifest references a path outside the crate.

Verification

  • python3 -m unittest discover -s .github/workflows -p test_publish_crates.py -v → 11 tests, OK. Fixtures under .github/scripts/fixtures/pd-edge-0.2.0/ are verbatim copies of the 0.2.0 manifests and reproduce the failure shape: they assert pd-edge-abi depends on no crate released after it, that pd-edge resolves edge_abi to { package = "pd-edge-abi", version = "0.2.0", default-features = false } with no path, that [dev-dependencies] is gone, and that a reversed order is re-sorted to host-function → abi → pd-edge.
  • python3 -m py_compile on both scripts and the test file: OK. yaml.safe_load on publish-crates.yml: OK. bash -n on both run: blocks: OK. git diff --check: OK.
  • Real cargo publish --dry-run --no-verify --manifest-path on the staged trees for 0.2.0:
    • pd-edge-abi: Packaged 22 files, 121.8KiB → aborting upload due to dry run (exit 0). This is the exact step that failed 18× in CI.
    • pd-edge-host-function: exit 0 (crate already on crates.io).
    • pd-edge: still reports failed to select a version for the requirement pd-edge-abi = "^0.2.0" — expected before abi is published, and precisely why the publisher now completes abi and waits for its index entry before touching pd-edge.
  • No publishing was executed and no tag/version was changed. crates.io still has no pd-edge* 0.2.0 for abi/pd-edge; the next tag run is the real end-to-end test.

fffonion and others added 2 commits November 17, 2026 00:59
`cargo publish -p <member>` resolves the whole workspace, so rewriting path
dependencies into registry requirements inside the checkout deadlocked the
0.2.0 tag: publishing pd-edge-abi failed 18 times with

    failed to select a version for the requirement `pd-edge-abi = "^0.2.0"`
    candidate versions found which didn't match: 0.1.1
    required by package `pd-edge v0.2.0`

because the workspace member pd-edge already demanded an abi version that only
exists after abi publishes.

- .github/scripts/prepare_publish.py: copy each released crate into its own
  standalone tree, drop path/git/rev/branch/tag selectors, apply the release
  version to workspace crates and the registry version to pd-vm /
  pd-host-function, remove dev-dependencies, materialise [workspace.package]
  inheritance, and derive the publish order from the normal/build graph.
- .github/scripts/publish_crates.py: publish by --manifest-path in that order,
  confirm each version in the sparse index (not only the crates.io API) before
  building on it, skip already-published versions, and stop retrying once a
  dependency is indexed.
- publish-crates.yml now calls the scripts, drops the rustscript checkout that
  staging made unnecessary and the deprecated `cargo publish --token`.
- tests run against verbatim 0.2.0 manifest fixtures plus the live manifests,
  covering the abi-before-pd-edge order, selector removal, dev-dep removal and
  the CLI dry-run plan.
@fffonion
fffonion merged commit 0d8db81 into master Nov 16, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant