Repository navigation
fix(upgrade): reclaim what a failed pull leaves behind - #3415
vpetersson-bot wants to merge 2 commits into
Conversation
A `docker compose pull` that dies partway abandons the bytes it already fetched in containerd's content store, where no image references them and `docker images` cannot see them. The cleanup that would collect them is deliberately the last step of the upgrade and only runs once the new stack is up, so it never runs on the path that needs it. A device upgrading because it is short of space therefore has less room on each retry than on the one before. - Reclaim on the pull's failure path, restricted to unreferenced untagged content via a new `--untagged-only` so the release the device is still running survives, which is why the sweep is otherwise deferred. - Report what was reclaimed: `docker image prune` frees this data without counting it and prints `0B`, so measure the content store directly. - Restart the services the low-RAM branch stopped before the pull, as the signature-check failure path already did. Measured on a Pi 5: an interrupted pull left 130 MB part-written and 39 MB of orphaned whole layers; prune freed all 169 MB and reported 0B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
All nine were on lines this branch adds, so they are fair. - Use `[[` for the conditionals introduced here (shelldre:S7688). Both scripts already mix `[` and `[[`; the pre-existing `[` tests are left alone rather than churned into this diff. - Give the two new functions an explicit `return` rather than inheriting the last command's status (shelldre:S7682). No behaviour change: the suite drives the real script, and all 15 tests still pass. shellcheck output is byte-identical to master's, so nothing here adds a finding of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
Issues Fixed
Fixes #3410.
Description
A
docker compose pullthat dies partway abandons the bytes it already fetched in containerd's content store. Nothing references them,docker imagescannot see them, and they do not expire. The one command that would collect them ispurge_stale_images.sh, which is deliberately the last step of an upgrade and only runs once the new stack is up — so it never runs on the path that needs it. A device upgrading because it is short of space has less room on each retry than on the one before: a ratchet, not a plateau.--untagged-onlyrestricts the sweep to unreferenced untagged content, so the superseded images — the release the device is still running — survive. That is the property the deferred ordering exists to protect, and keeping it is what makes reclaiming this early safe.docker image prunefrees this data without counting it and printsTotal reclaimed space: 0B, so the script measures the content store itself. On a device that is out of space, that number is the whole point.Not included: the pre-flight free-space check the issue floats as a companion. It is a behaviour change that can wrongly block an upgrade if the estimate is off, and is better judged on its own.
Verification
Reproduced and verified end to end on the Pi 5 and x86 testbeds, both on the containerd image store (Docker 29.6.1).
Interrupting a pull on the Pi 5, before the fix:
The leftovers split two ways: 130 MB of part-written layers under
ingest/, and 39 MB of whole layers underblobs/that finished downloading before the pull gave up. Measuring onlyingest/would have under-reported by 23%, which is why this counts the whole content store.After the fix, running the upgrade against a tag that cannot be pulled, with the leftovers of an earlier failed attempt already on disk:
Content store back to its 774,796 kB baseline; all four containers still
Up 40 hours;/api/v2/assets→ 200; all six Anthias images intact. Same result on x86 (prune reported0Bwhile freeing 37 MB; the failure path reclaimed and reported it, exit 1).Also checked that
--untagged-onlyspares a superseded, unreferenced Anthias image while the unflagged run still removes it, and that an unknown flag exits 2.Checklist
🤖 Generated with Claude Code