Repository navigation
fix(transform): fall back to pdfa when primary PDF is empty - #651
TahaKhan998 wants to merge 1 commit into
Conversation
|
|
||
| @staticmethod | ||
| def _is_empty_or_corrupt_file(file_dump): | ||
| return file_dump["checksum"] == EMPTY_PDF_CHECKSUM or file_dump["size"] == 0 |
There was a problem hiding this comment.
I guess the size check should be enough and we wouldn't need to hardcode the checksum? WDYT?
There was a problem hiding this comment.
most of the corrupted files have that checksum and have a size of 321 bytes, not 0, so size alone would miss them. keeping both
| doi = self.record.body.get("pids", {}).get("doi", {}) | ||
| return doi.get("provider") == "external" | ||
|
|
||
| def _resolved_file_dumps(self): |
There was a problem hiding this comment.
I would rather go for a failure when a corrupted file is detected, then going for automatic solution - I think these files should be reviewed by the curators and fixed accordingly to individual use case
| FILE_SUBFORMATS_TO_DROP = ["pdfa", "unstamped"] | ||
|
|
||
| # Known empty/corrupt legacy PDF checksum (zero-byte file). | ||
| EMPTY_PDF_CHECKSUM = "6f58aa0bfac8a909ac578c1595d55a60" |
There was a problem hiding this comment.
i would check also the bytes because some checksums might not be correct
There was a problem hiding this comment.
is this still needed? seems to be used only in tests?
There was a problem hiding this comment.
yes still needed. transform_versions uses it in _is_empty_or_corrupt_file. And Zubeyde reported in the issue that this checksum is what signifies whether a pdf is corrupted and also i tried to look for records with corrupted files and every such record had this checksum as the md5 of the corrupted file.
0adec39 to
b8225f9
Compare
| ], | ||
| uow=uow, | ||
| ) | ||
| if file_data["mimetype"] == "application/pdf" and pdf_has_zero_pages( |
There was a problem hiding this comment.
could you add zero bytes check for other file types, not only PDF? we want to be able to identify all empty files and solve the problem not only for PDFs - it is safe to assume that we might other corrupted files.
also in here you are assuming that file_data mimetype is correct and defines a file type correctly. I would actually check it here and not rely on legacy metadata.
b8225f9 to
c7eda66
Compare
closes #622
some records get migrated with an empty/corrupt pdf even though legacy has a valid pdfa version that we currently skip.
if the primary file is empty (known bad checksum or size 0) and there's a pdfa for the same bibdoc, we use the pdfa content but keep the normal filename. if there's no pdfa we still migrate the empty file and log the recid. if the primary pdf is fine, nothing changes (pdfa still dropped).