Dataset iterators skip items when unwind returns more items than the rows scanned #1058

Summary

iterate_items advances the dataset offset by max(scanned_rows, len(items)). With unwind, a page returns more items than the rows it scanned, so the offset jumps past rows the next request never reads and those items are silently dropped.

Where

The max() appears in two layers, four sites:

The API applies offset and limit to the dataset's rows first and shapes the result afterwards. x-apify-pagination-count is set from min(total - offset, limit) before the transform stream runs, so it reports the rows scanned and never reflects unwind or the filters. Filters (clean, skip_empty, skip_hidden) leave len(items) below that number, unwind leaves it above. The max() handles the first case and breaks the second.

Reproduction

Driving the real iterator with a 4-row dataset where every row unwinds into three items, at chunk_size=2:

from dataclasses import dataclass

from apify_client._pagination import get_items_iterator

TOTAL_ROWS = 4
PARTS = ['a', 'b', 'c']


@dataclass
class Page:
    items: list[dict]
    count: int


calls = []


def callback(*, limit: int | None = None, offset: int | None = None) -> Page:
    offset = offset or 0
    calls.append(offset)
    rows = [{'id': i} for i in range(offset, min(offset + (limit or TOTAL_ROWS), TOTAL_ROWS))]
    # The API unwinds each row into three items after offset/limit picked the row window.
    items = [{**row, 'part': p} for row in rows for p in PARTS]
    # dataset.py builds the page as max(header, len(items)); the header is len(rows).
    return Page(items=items, count=max(len(rows), len(items)))


got = list(get_items_iterator(callback, chunk_size=2))
print('requested offsets:', calls)
print('yielded items:', len(got), 'of', TOTAL_ROWS * len(PARTS))

requested offsets: [0, 6]      # should be [0, 2]
yielded items: 6 of 12

Rows 2 and 3 are never requested. At the default chunk_size=1000, a 5000-row dataset with a 3x unwind yields about 2000 rows.

Nothing bounds the over-advance: get_items_iterator documents that "the total field is intentionally not consulted, because it can change between calls", so the offset runs off the end of the dataset and the loop stops on the resulting empty page.

Affected versions

Both max() sites arrived with #771 and have shipped since v3.0.0. #964 changed only the termination condition and left the bookkeeping alone.

Suggested fix

Follow the scanned count alone, which is right in both directions, and fall back to len(items) only for responses that expose no scanned count (the collection endpoints). apify-client-js took this route in #1044 after the same finding in review.

That forces a decision on the public DatasetItemsPage.count, which is currently max(scanned, returned) while its docstring says "Number of items in this page". apify-client-js keeps count == len(items) and carries the scanned number out of band, so the two clients currently disagree on this field.

Two related items

• There is no unwind coverage anywhere in tests/, which is why this went unnoticed.
• The get_cursor_iterator docstring says server-side filters "can drop every item on a page while a live cursor still points at more data". The API rules that out. For the request queue, getNextCursor() returns a cursor only when pushedItemsCount >= limit and limit is validated above zero. For the key-value store, nextExclusiveStartKey is last(keys)?.key, which is absent on an empty page. test_cursor_iterator_continues_past_fully_filtered_page covers a state the API cannot produce. The behavior is harmless, the justification is wrong.

✍️ Drafted by Claude Code