Conversation
Pass the page number explicitly to _process_page() and _extract_images() instead of sharing it through self.current_page_number, which worker threads in process_batch() raced on. The race misapplied page-1-only formatting, corrupted PDFPage.page_number, and could collide extracted image filenames across pages. Add regression tests covering batch determinism, page-1-only formatting, sequential page numbering, and serial/batch parity. Signed-off-by: Kylinny <j.yao@wustl.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2319.
Hi, and thank you @rtorrero-lw for the excellent report — the diagnosis and local repro made the root cause easy to confirm.
Bug
With the default
batch_size=4,NaivePDFProcessorStrategy.process_batch()produced different markdown for the same PDF on different runs. In the issue's repro (12 pages, each with a Title Case line thatclean_pdf_text()only bolds on page 1), 30 runs gave 5 distinct outputs locally (bolding landed on page 1, on no page, or on other pages);batch_size=1was always stable.Root cause
Each worker thread wrote its page number to the shared
self.current_page_numberattribute and_process_page()read it back after text extraction — by which time another worker could have overwritten it. The same stale value also fedPDFPage.page_numberand the extracted image filenames (page_{n}_img_{m}, which could even collide across threads).Fix
Pass the page number down explicitly instead of storing it on
self, exactly as suggested in the report:_process_page(page, image_dir, page_num + 1)and_extract_images(page, image_dir, page_number). The now-unusedcurrent_page_numberattribute is removed. Invariants preserved:process()and batchedprocess_batch()paths (verified: both produce identical per-page markdown/html).clean_pdf_text()/clean_pdf_text_to_html()signatures are untouched; the page-1-only formatting now provably applies only to page 1.selfreads inside worker threads (extract_images,save_images_locally, …) are all read-only after__init__.Tests
New
tests/unit/test_pdf_page_number_race.py(hand-crafted multi-page PDFs, no new test dependencies):test_process_batch_is_deterministic— 10 runs withbatch_size=4give byte-identical outputtest_page_one_only_formatting_in_batch— author-line bolding appears on page 1 and only page 1test_page_numbers_sequential_in_batch—PDFPage.page_numbermatches page position under threadstest_process_page_uses_given_page_number(parametrized) —_process_pagehonors its argumenttest_process_and_process_batch_agree— serial/batch parityAll 7 new cases fail on the unfixed code and pass with the fix; the existing
test_pdf_html_escaping.pyPDF tests still pass.Before / after (issue's repro, 30 runs,
batch_size=4)[1],[],[1, 2],[1, 4],[1, 6])Happy to adjust anything on review. Thanks again for the great catch!