Conversation
`BaseLocalFile.to_parquet` streams the rows out of `self.path` and
writes them with `pq.ParquetWriter(output_path, ...)`, which opens the
destination with `"wb"`. When `output_path` resolved to `self.path` the
source was truncated to zero length part-way through the very read it
was serving, and the Parquet bytes then replaced it:
>>> csv = await ExtensionFactory.instantiate("data.csv")
>>> await csv.to_parquet(output_path="data.csv") # succeeds!
>>> open("data.csv", "rb").read(4)
b'PAR1'
The source file was silently destroyed. `to_parquet()` on a Parquet
source hits this too, because `self.path.with_suffix(".parquet")` is
the source path when the source is already `.parquet`.
`DBF`, `DBC` and `Zip` already guard this by short-circuiting when the
output exists; this is the same guard, checked before any writing:
- a Parquet source is returned unchanged, since it needs no conversion
and `download_to_parquet` already tolerates `original_path ==
parquet_file.path`
- any other source raises `ConversionError` naming a usable
`output_path` instead of destroying the file
The comparison is against the resolved `output_path`, so a path that
only matches after normalisation (`sub/../data.csv`) is caught too.
devgtv
force-pushed
the
fix/parquet-partial-output
branch
from
October 2, 2026 19:18
20c19af to
b1ee260
Compare
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #356 +/- ##
=======================================
Coverage ? 97.19%
=======================================
Files ? 180
Lines ? 23351
Branches ? 0
=======================================
Hits ? 22697
Misses ? 654
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BaseLocalFile.to_parquetstreams rows out ofself.pathand writes them withpq.ParquetWriter(output_path, ...), which opens its destination with"wb". Whenoutput_pathresolved toself.path, the source was truncated to zero length part-way through the very read serving it, and the Parquet bytes then replaced it:The input file was silently destroyed and replaced by an unreadable one.
to_parquet()with no arguments on a Parquet source hits this as well, sinceself.path.with_suffix(".parquet")is the source path when the source is already.parquet.DBF,DBCandZipalready guard against this by short-circuiting when the output exists (pysus/api/extensions.py:538,:687). This adds the same guard to the base implementation, before anything is written.Fix
download_to_parquetalready anticipates this outcome with itsoriginal_path != parquet_file.pathcheck (pysus/api/client.py:596)ConversionErrornaming a usableoutput_pathsub/../data.csvis caught tooTests
Added 6 tests: CSV in place raises and leaves the file byte-identical, JSON in place likewise, a convoluted-but-equivalent path is caught, a Parquet source is a no-op returning the same object, the default
.csv→.parquetconversion still works, and the source survives it.4 of the 6 fail against the previous code.
1739 passed, 6 skippedblackandisortclean