Repository navigation
fix(backup): swap restored assets in atomically, not in place - #3421
Open
vpetersson-bot wants to merge 1 commit into
Open
vpetersson-bot wants to merge 1 commit into
vpetersson-bot wants to merge 1 commit into
Conversation
`recover()` extracts every member straight to its final path, so an asset that already exists on the device is truncated and then refilled under its own name. Nothing stops the viewer fetching that same file over `/anthias_assets/<name>` while the write is in flight. Django's `FileResponse` measures `Content-Length` from the file as it stands when the response is built and then streams the same open fd to EOF. Rewriting the inode underneath that reader breaks the promise two ways: a restored copy that is larger makes the body overrun the header and uvicorn raises `Response content longer than Content-Length` (Sentry ANTHIAS-37), a smaller one makes it fall short and raises `Response content shorter than Content-Length`. The 500 is the visible half; the quiet half is that the viewer was handed a frame spliced from two different files. Every other writer of the media store already stages to a sibling and renames — `processing.py`, `celery_tasks.py`, `views.py`, `integrations/ingest.py`. Restore was the one path that did not, so it does now: each media-store file is extracted under `.restore-<hex>.tmp` beside its destination and `os.replace`d into place, which the kernel observes as a single inode swap. An in-flight response keeps reading the inode it opened and still matches the length it declared, and no request can observe a partial asset. The staging name is dot-prefixed and `.tmp`-suffixed on purpose: both of `views_files.anthias_assets`' transient-file guards already refuse to serve such a name, `_exclude_from_backup` keeps it out of the next archive, and the hourly `cleanup()` sweep collects it if the process dies between the extract and the rename. Config and database members keep the plain `tar.extract` path. On an upgraded device `.screenly/screenly.db` is a back-compat symlink into `.anthias/`, and extraction is meant to write *through* it; an `os.replace` there would replace the symlink with a regular file and leave the real database untouched. Also corrects the `before_send` comment that credited the `shorter` rule with ANTHIAS-37. ANTHIAS-37 is the `longer` variant, so that rule never matched it — and a genuine overrun is a bug in whatever produced the response, not noise to drop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What's wrong today
backup_helper.recover()extracts every tar member straight to its finalpath. For the media store that means an asset which already exists on the
device is truncated and then refilled under its own name, and nothing
stops the viewer fetching that same file over
/anthias_assets/<name>while the write is in flight.
Django's
FileResponsemeasuresContent-Lengthfrom the file as itstands when the response is built, then streams that same open fd to EOF.
Rewriting the inode underneath a live reader breaks the promise in both
directions:
RuntimeError: Response content longer than Content-LengthRuntimeError: Response content shorter than Content-LengthThe 500 is the visible half. The quiet half is that the viewer has already
been handed a frame spliced out of two different files.
The overrun is Sentry ANTHIAS-37 (
transaction: /anthias_assets/{filename},raised in
uvicorn/protocols/http/httptools_impl.py:sendoff Django'sasgi.py:send_response, no Anthias frame in the traceback — which is whyit reads as framework noise).
Every other writer of the media store already stages to a sibling and
renames:
processing.py(os.replace(staging, final_uri)),celery_tasks.py(.part→os.replace),views.py(chunked upload →os.replace),integrations/ingest.py(.part→os.replace). Restorewas the one path that did not.
What this changes
Media-store files are now extracted under
.restore-<hex>.tmpbesidetheir destination and
os.replaced into place, which the kernel observesas a single inode swap. An in-flight response keeps reading the inode it
opened and still matches the length it declared, and no request can
observe a partial asset.
The staging name is dot-prefixed and
.tmp-suffixed deliberately — allthree existing guards already cover that shape:
views_files.anthias_assets404s on a dot-prefixed path component andon a
.tmp/.partsuffix, so a staged file is never servable;_exclude_from_backupdrops it from the next archive;cleanup()sweep collects it if the process dies between theextract and the rename (the explicit
removeon the error path coversthe rest).
Config and database members keep the plain
tar.extractpath on purpose.On an upgraded device
.screenly/screenly.dbis a back-compat symlinkinto
.anthias/(bin/migrate_legacy_paths.sh) and extraction is meantto write through it — an
os.replacethere would replace the symlinkwith a regular file and leave the real database untouched. The existing
WAL-clearing contract in
recover()is unchanged.Also corrects the
before_sendcomment insettings.pythat credited theshorterrule with ANTHIAS-37. ANTHIAS-37 is thelongervariant, sothat rule never matched it; and a genuine overrun is a bug in whatever
produced the response, not noise to drop — so no filter is added here, the
cause is fixed instead.
How I know it's right
New regression test
test_recover_swaps_assets_atomicallyholds an openfd on an existing asset across a
recover()that lands a larger copy, andasserts the reader still sees the old bytes at the old length, that the
inode changed, that the new bytes are on disk afterwards, and that no
staging file is left behind. It fails on
master(the fd observes therewritten file) and passes here.
test_recover_clears_the_stale_wal_when_a_later_member_failspatchedTarFile.extracton an exact member name; it now matches the asset-dirprefix, since that member arrives under its staging name. Its assertions
are untouched.
uv run pytest -n auto -m "not integration"→ 2563 passed, 3 skipped.One pre-existing failure under
-n auto,test_viewer.py::TestWaitForWaylandSocket::test_probe_reports_not_ready_for_a_non_socket_path,reproduces identically on an unmodified tree and passes in isolation —
unrelated to this change.
uv run ruff check .,uv run ruff format --check .,uv run mypy .all clean.
🤖 Generated with Claude Code