Skip to content

fix(saude): never reuse or publish a partial CKAN download - #370

Open
devgtv wants to merge 1 commit into
AlertaDengue:mainfrom
devgtv:fix/saude-partial-download-reuse
Open

devgtv wants to merge 1 commit into
AlertaDengue:mainfrom
devgtv:fix/saude-partial-download-reuse

Conversation

@devgtv

@devgtv devgtv commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

fix(saude): never reuse or publish a partial CKAN download

download_resource() short-circuits on dest_path.exists() but streams
directly into that path, so a mid-stream drop leaves a truncated file that
the next call happily returns, and the engine then hashes, converts and
uploads it to S3 as the official artifact.

CKAN/Saude also ignore the requested filename and write their own derived
name inside output.parent, so the retry's cleanup of output deleted a path
that never existed. All Saude workers share management/tmp as dest_dir, so
two packages publishing a same-named resource also resolved to the same
dest_path and interleaved their writes.

download_resource() now streams into a sibling ...part file
and os.replace()s it on success, unlinking it on any BaseException, so the
rename is atomic. _download_raw_with_retry() gives each attempt its own
scratch directory and drops the whole directory on retry, which is the only
reliable cleanup when the origin chose the filename itself.

Rebased onto main: the FTP origin keeps its attempt directory, because
_retr_with_resume reads the existing bytes off it to resume with REST --
wiping it between attempts would restart every RETR from byte 0. The
per-attempt directory therefore applies to the origins that need it, and a
test pins both behaviours.


This is a re-submission of #353, which was closed without review. The
workflow runs on that PR sat in action_required for three days and then
expired unapproved, so CI never went green.

This workflow run required approval but was not approved before it expired.

Two notes on this one:

  • It conflicts semantically with Fix/ftp stale session #367 (Fix/ftp stale session), which
    added REST resume to the FTP path in the same function. Wiping the
    per-attempt directory between retries would restart every RETR from
    byte 0, so the attempt directory is now kept for FTP and the fresh
    directory applies to the origins that need it. A test pins both.
  • The suite passes (1787 passed, 6 skipped) and black/isort/flake8 are
    clean. The two remaining failures (test_export.py::TestToSQL::test_basic_duckdb,
    test_dbf_reader.py::test_numpy_object_array_types) also fail on a clean
    main and are unrelated to this change.

download_resource() short-circuits on dest_path.exists() but streams
directly into that path, so a mid-stream drop leaves a truncated file that
the next call happily returns, and the engine then hashes, converts and
uploads it to S3 as the official artifact.

CKAN/Saude also ignore the requested filename and write their own derived
name inside output.parent, so the retry's cleanup of output deleted a path
that never existed. All Saude workers share management/tmp as dest_dir, so
two packages publishing a same-named resource also resolved to the same
dest_path and interleaved their writes.

download_resource() now streams into a sibling .<name>.<uuid8>.part file
and os.replace()s it on success, unlinking it on any BaseException, so the
rename is atomic. _download_raw_with_retry() gives each attempt its own
scratch directory and drops the whole directory on retry, which is the only
reliable cleanup when the origin chose the filename itself.

Rebased onto main: the FTP origin keeps its attempt directory, because
_retr_with_resume reads the existing bytes off it to resume with REST --
wiping it between attempts would restart every RETR from byte 0. The
per-attempt directory therefore applies to the origins that need it, and a
test pins both behaviours.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.63314% with 4 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@9db5915). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pysus/api/saude/download.py 90.47% 2 Missing ⚠️
pysus/management/sync.py 88.88% 2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #370   +/-   ##
=======================================
  Coverage        ?   97.21%           
=======================================
  Files           ?      180           
  Lines           ?    23442           
  Branches        ?        0           
=======================================
  Hits            ?    22789           
  Misses          ?      653           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants