Conversation
download_resource() short-circuits on dest_path.exists() but streams directly into that path, so a mid-stream drop leaves a truncated file that the next call happily returns, and the engine then hashes, converts and uploads it to S3 as the official artifact. CKAN/Saude also ignore the requested filename and write their own derived name inside output.parent, so the retry's cleanup of output deleted a path that never existed. All Saude workers share management/tmp as dest_dir, so two packages publishing a same-named resource also resolved to the same dest_path and interleaved their writes. download_resource() now streams into a sibling .<name>.<uuid8>.part file and os.replace()s it on success, unlinking it on any BaseException, so the rename is atomic. _download_raw_with_retry() gives each attempt its own scratch directory and drops the whole directory on retry, which is the only reliable cleanup when the origin chose the filename itself. Rebased onto main: the FTP origin keeps its attempt directory, because _retr_with_resume reads the existing bytes off it to resume with REST -- wiping it between attempts would restart every RETR from byte 0. The per-attempt directory therefore applies to the origins that need it, and a test pins both behaviours.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #370 +/- ##
=======================================
Coverage ? 97.21%
=======================================
Files ? 180
Lines ? 23442
Branches ? 0
=======================================
Hits ? 22789
Misses ? 653
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(saude): never reuse or publish a partial CKAN download
download_resource() short-circuits on dest_path.exists() but streams
directly into that path, so a mid-stream drop leaves a truncated file that
the next call happily returns, and the engine then hashes, converts and
uploads it to S3 as the official artifact.
CKAN/Saude also ignore the requested filename and write their own derived
name inside output.parent, so the retry's cleanup of output deleted a path
that never existed. All Saude workers share management/tmp as dest_dir, so
two packages publishing a same-named resource also resolved to the same
dest_path and interleaved their writes.
download_resource() now streams into a sibling ...part file
and os.replace()s it on success, unlinking it on any BaseException, so the
rename is atomic. _download_raw_with_retry() gives each attempt its own
scratch directory and drops the whole directory on retry, which is the only
reliable cleanup when the origin chose the filename itself.
Rebased onto main: the FTP origin keeps its attempt directory, because
_retr_with_resume reads the existing bytes off it to resume with REST --
wiping it between attempts would restart every RETR from byte 0. The
per-attempt directory therefore applies to the origins that need it, and a
test pins both behaviours.
This is a re-submission of #353, which was closed without review. The
workflow runs on that PR sat in
action_requiredfor three days and thenexpired unapproved, so CI never went green.
Two notes on this one:
Fix/ftp stale session), whichadded REST resume to the FTP path in the same function. Wiping the
per-attempt directory between retries would restart every
RETRfrombyte 0, so the attempt directory is now kept for FTP and the fresh
directory applies to the origins that need it. A test pins both.
clean. The two remaining failures (
test_export.py::TestToSQL::test_basic_duckdb,test_dbf_reader.py::test_numpy_object_array_types) also fail on a cleanmainand are unrelated to this change.