Skip to content

fix(backup): swap restored assets in atomically, not in place - #3421

Open
vpetersson-bot wants to merge 1 commit into
masterfrom
agent/anthias-claude/def28ba47fbf
Open

vpetersson-bot wants to merge 1 commit into
masterfrom
agent/anthias-claude/def28ba47fbf

Conversation

@vpetersson-bot

Copy link
Copy Markdown
Contributor

What's wrong today

backup_helper.recover() extracts every tar member straight to its final
path. For the media store that means an asset which already exists on the
device is truncated and then refilled under its own name, and nothing
stops the viewer fetching that same file over /anthias_assets/<name>
while the write is in flight.

Django's FileResponse measures Content-Length from the file as it
stands when the response is built, then streams that same open fd to EOF.
Rewriting the inode underneath a live reader breaks the promise in both
directions:

restored copy what uvicorn raises
larger than the old one RuntimeError: Response content longer than Content-Length
smaller than the old one RuntimeError: Response content shorter than Content-Length

The 500 is the visible half. The quiet half is that the viewer has already
been handed a frame spliced out of two different files.

The overrun is Sentry ANTHIAS-37 (transaction: /anthias_assets/{filename},
raised in uvicorn/protocols/http/httptools_impl.py:send off Django's
asgi.py:send_response, no Anthias frame in the traceback — which is why
it reads as framework noise).

Every other writer of the media store already stages to a sibling and
renames: processing.py (os.replace(staging, final_uri)),
celery_tasks.py (.part → os.replace), views.py (chunked upload →
os.replace), integrations/ingest.py (.part → os.replace). Restore
was the one path that did not.

What this changes

Media-store files are now extracted under .restore-<hex>.tmp beside
their destination and os.replaced into place, which the kernel observes
as a single inode swap. An in-flight response keeps reading the inode it
opened and still matches the length it declared, and no request can
observe a partial asset.

The staging name is dot-prefixed and .tmp-suffixed deliberately — all
three existing guards already cover that shape:

  • views_files.anthias_assets 404s on a dot-prefixed path component and
    on a .tmp/.part suffix, so a staged file is never servable;
  • _exclude_from_backup drops it from the next archive;
  • the hourly cleanup() sweep collects it if the process dies between the
    extract and the rename (the explicit remove on the error path covers
    the rest).

Config and database members keep the plain tar.extract path on purpose.
On an upgraded device .screenly/screenly.db is a back-compat symlink
into .anthias/ (bin/migrate_legacy_paths.sh) and extraction is meant
to write through it — an os.replace there would replace the symlink
with a regular file and leave the real database untouched. The existing
WAL-clearing contract in recover() is unchanged.

Also corrects the before_send comment in settings.py that credited the
shorter rule with ANTHIAS-37. ANTHIAS-37 is the longer variant, so
that rule never matched it; and a genuine overrun is a bug in whatever
produced the response, not noise to drop — so no filter is added here, the
cause is fixed instead.

How I know it's right

New regression test test_recover_swaps_assets_atomically holds an open
fd on an existing asset across a recover() that lands a larger copy, and
asserts the reader still sees the old bytes at the old length, that the
inode changed, that the new bytes are on disk afterwards, and that no
staging file is left behind. It fails on master (the fd observes the
rewritten file) and passes here.

test_recover_clears_the_stale_wal_when_a_later_member_fails patched
TarFile.extract on an exact member name; it now matches the asset-dir
prefix, since that member arrives under its staging name. Its assertions
are untouched.

  • uv run pytest -n auto -m "not integration" → 2563 passed, 3 skipped.
    One pre-existing failure under -n auto,
    test_viewer.py::TestWaitForWaylandSocket::test_probe_reports_not_ready_for_a_non_socket_path,
    reproduces identically on an unmodified tree and passes in isolation —
    unrelated to this change.
  • uv run ruff check ., uv run ruff format --check ., uv run mypy .
    all clean.

🤖 Generated with Claude Code

`recover()` extracts every member straight to its final path, so an
asset that already exists on the device is truncated and then refilled
under its own name. Nothing stops the viewer fetching that same file
over `/anthias_assets/<name>` while the write is in flight.

Django's `FileResponse` measures `Content-Length` from the file as it
stands when the response is built and then streams the same open fd to
EOF. Rewriting the inode underneath that reader breaks the promise two
ways: a restored copy that is larger makes the body overrun the header
and uvicorn raises `Response content longer than Content-Length`
(Sentry ANTHIAS-37), a smaller one makes it fall short and raises
`Response content shorter than Content-Length`. The 500 is the visible
half; the quiet half is that the viewer was handed a frame spliced from
two different files.

Every other writer of the media store already stages to a sibling and
renames — `processing.py`, `celery_tasks.py`, `views.py`,
`integrations/ingest.py`. Restore was the one path that did not, so it
does now: each media-store file is extracted under `.restore-<hex>.tmp`
beside its destination and `os.replace`d into place, which the kernel
observes as a single inode swap. An in-flight response keeps reading
the inode it opened and still matches the length it declared, and no
request can observe a partial asset.

The staging name is dot-prefixed and `.tmp`-suffixed on purpose: both
of `views_files.anthias_assets`' transient-file guards already refuse
to serve such a name, `_exclude_from_backup` keeps it out of the next
archive, and the hourly `cleanup()` sweep collects it if the process
dies between the extract and the rename.

Config and database members keep the plain `tar.extract` path. On an
upgraded device `.screenly/screenly.db` is a back-compat symlink into
`.anthias/`, and extraction is meant to write *through* it; an
`os.replace` there would replace the symlink with a regular file and
leave the real database untouched.

Also corrects the `before_send` comment that credited the `shorter`
rule with ANTHIAS-37. ANTHIAS-37 is the `longer` variant, so that rule
never matched it — and a genuine overrun is a bug in whatever produced
the response, not noise to drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
@vpetersson-bot
vpetersson-bot requested a review from a team as a code owner October 25, 2026 09:17
Copilot AI lite review requested due to automatic review settings October 25, 2026 09:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants