Skip to content

fix: page offset iterators past the first page's total - #1125

Merged
vdusek merged 3 commits into
masterfrom
fix/paginate-past-first-page-total
Oct 7, 2026
Merged

vdusek merged 3 commits into
masterfrom
fix/paginate-past-first-page-total

Conversation

@vdusek

@vdusek vdusek commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Offset iterators (list(), listItems()) stopped at the first page's total, so rows pushed during the loop were never read. They now page until a page scans no rows or limit is reached, like the Python client. Each page advances by the larger of x-apify-pagination-count and items.length, capped at the requested limit, because the header lags a fresh push.

Loosely related to #1085: RunClient.iterateDatasetItems() stays the tool for reading a dataset a run is still writing.

Closes: #1117

✍️ Drafted by Claude Code

Closes #1117

BREAKING CHANGE: iterating a `list()` method or `listItems()` no longer stops at the `total` reported by the first page. It pages until a page scans no rows or the caller's `limit` is reached, which takes one more request at the end.
@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 7, 2026
@vdusek vdusek self-assigned this Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ There are broken links in the documentation.

See more at https://github.com/apify/apify-client-js/actions/runs/37583957298#summary-112669732087

@vdusek
vdusek requested a review from barjin October 7, 2026 06:55
@vdusek
vdusek marked this pull request as ready for review October 7, 2026 06:55
@vdusek
vdusek requested a review from szaganek as a code owner October 7, 2026 06:55
@vdusek vdusek changed the title fix!: page offset iterators past the first page's total fix: page offset iterators past the first page's total Oct 7, 2026
@vdusek vdusek changed the title fix: page offset iterators past the first page's total fix!: page offset iterators past the first page's total Oct 7, 2026
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Oct 7, 2026
@vdusek

vdusek commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

I guess this is discutable after #1085, but we have it in Python already. Let me know your opinion @barjin.

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RE: your comment above - we should still do it. The total on the first page can be lagging because of API design. v2 can now miss some items if we do rapid writes / reads.

anyway, lgtm, thanks @vdusek!

@vdusek vdusek changed the title fix!: page offset iterators past the first page's total fix: page offset iterators past the first page's total Oct 7, 2026
@vdusek
vdusek merged commit 9cd54be into master Oct 7, 2026
11 of 13 checks passed
@vdusek
vdusek deleted the fix/paginate-past-first-page-total branch October 7, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Paginated iteration stops at the first page's total on a growing dataset

3 participants