You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Live iteration over a run's dataset while the run is still running #1065
iterate_items only covers a dataset that is already complete. There is no way to consume a run's
items as they are pushed, so anyone who wants to process output while the Actor is still working
has to hand-roll the polling loop.
Proposal
A live iterator on RunClient / RunClientAsync, which already holds both the run status and the
dataset:
It polls while the run is READY or RUNNING, yields items as they appear, and returns once the
run reaches a terminal status and the dataset has been drained. Poll interval configurable, the
remaining arguments matching DatasetClient.iterate_items.
RunClient is the better home than DatasetClient: the termination condition needs the run status,
which the dataset client has no business knowing about.
Do not terminate on itemCount
Dataset.itemCount is bumped through a throttle (~5 s) while items are served fresh from S3 plus
the live buffer, so when a run finishes itemCount can still lag behind what is already readable.
Ending the final drain at read >= itemCount truncates the tail silently. Same root cause as #1058.
Drain until a page comes back short instead.
Useful for the overall shape, with two things to avoid: its final drain loop is bounded by a single dataset.itemCount read, which is the truncation above, and itemsThreshold withholds items until
enough have accumulated, which its own TODO already marks for removal.
Activity
feat: Add live iteration over a run's dataset items (#1079)