Skip to content

[Brook S7] Added Workflow Streams, which write through Local Activities. - #377

Open
moetemp wants to merge 1 commit into
moe/brook/review/s6-redis-memoryfrom
moe/brook/review/s7-workflow-streams
Open

moetemp wants to merge 1 commit into
moe/brook/review/s6-redis-memoryfrom
moe/brook/review/s7-workflow-streams

Conversation

@moetemp

@moetemp moetemp commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

This PR adds Workflow Streams (workflow.create_stream, create_or_get_stream, their _nowait forms, to_stream and WorkflowStream, which makes the Stream calls in Workflow code). They write through Local Activities.

  • A call takes the payload type as a type argument, as in workflow.create_stream[Turn]("session-7-turns"). A Workflow creates only standalone Streams, under an id of its choice, so the ref has no owner and the Stream doesn't end with the run.
  • create_stream fails with StreamAlreadyExistsError if the id exists, and create_or_get_stream gets it. The request id is <first execution run id>:<stream id>, so it's the same in each run of the chain. A Workflow Task retry, a retry run or a cron run that creates the Stream again gets the Stream that the first create made. A client sends a new request id with each create call. In a Workflow the request_id() source of [Brook S2] Generated the Stream service bindings with nexgen, and added the writer client. #372 now returns an empty id, so a create draws nothing from the user's workflow.random().
  • A run after continue-as-new uses the earlier run's Streams through the refs that it forwards, with workflow.to_stream. There's no workflow.get_stream.
  • Workflow code calls the generated Workflow API of [Brook S2] Generated the Stream service bindings with nexgen, and added the writer client. #372. A Workflow outbound interceptor gets those calls through start_system_nexus_operation and runs each one as a write Local Activity. So History keeps a marker per write and has no Nexus events.
  • Each publish is a Local Activity whose Producer epoch is the local_activity case with the Workflow Task that dispatched it ([Brook G3] Added activity.Info.workflow_task for Local Activities. #369). So a reader withdraws the writes of a failed task once its retry writes. A retry run's first publish to a Stream withdraws the run that it retries, and a create never does, since the Store refuses that on a Stream with no owner. The workflow_writer() and client_writer() sources of [Brook S2] Generated the Stream service bindings with nexgen, and added the writer client. #372 give the run's writer and the Activity attempt's writer.
  • A close carries no value. It's a regular Activity, which runs only after its task commits and closes as the run's Workflow Producer. So a task that fails first leaves the Stream open for its retry.
  • A write takes its place in the Stream's order at the call, since _support.start_now runs the generated call up to its first wait. So publish_nowait, close_nowait and the _nowait creates work in sync code. The error of a _nowait write fails the run once no other write is in flight, since no Local Activity marker may follow the fail command. A write queued behind a create that fails sends nothing and fails with the create's error, since the Stream can be another owner's.
  • publish returns a future and doesn't block the Workflow. A publish error that no caller retrieved fails the run at its end, and the Workflow's logger logs it when the write fails.
  • The end of the run waits for every write, also for a write that a Signal or Update handler starts while it waits. A cancel there doesn't stop the writes. The run waits for them, then ends as canceled.
  • async with closes at the block end, also when the run continues as new in the block. It defers the close to the run's end when a retry run may follow. A BaseException, such as an interrupt, follows the same rule as an exception. On a Stream bound to another run chain it raises StreamNotOwnerError at __aenter__.
  • Records convert with the Workflow's payload converter with no serialization context, as in Activities and clients.
  • A write in read-only code, such as a query handler or an update validator, raises temporalio.workflow.ReadOnlyContextError.

Why?

This is the main use of Streams. An agent Workflow writes its turns to a Stream, and a client or a UI reads them as the Workflow writes them.

This PR is over the size target, since the Workflow calls, the Local Activities and the interceptors only test together.

Open question. Workflow Streams use private parts of the Workflow runtime, such as the run's outbound interceptor chain, the context-free converters and the error that a _nowait call raises in the run. Should the SDK add a hook API in workflow.unsafe for them?

Part of the Streams review series.

Checklist

  1. How was this tested:
    • poe lint, poe gen-docs and pytest tests/contrib/streams on both backends.

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 25, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch from b298e44 to 90966d6 Compare October 26, 2026 00:14
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch 2 times, most recently from 326073b to d018d29 Compare October 26, 2026 02:08
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch 2 times, most recently from 1c5e48f to 84f1587 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch 2 times, most recently from b5bb616 to f5a8d77 Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch 2 times, most recently from c3f30a4 to b9572b1 Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from f5a8d77 to 6d49da8 Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch 2 times, most recently from 925cf34 to cc8e51c Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from 026e0a3 to 11cd676 Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch from cc8e51c to e7710c0 Compare October 28, 2026 08:05
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from 11cd676 to c68482a Compare October 28, 2026 08:05
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from effd2e0 to b445a38 Compare November 5, 2026 17:19
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch from eb66e79 to a912bfb Compare November 5, 2026 17:19
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch from a912bfb to 1c9ed91 Compare November 5, 2026 23:10
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from b445a38 to 27b6d1c Compare November 5, 2026 23:10
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch from 1c9ed91 to d57a2ae Compare November 6, 2026 09:06
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from 27b6d1c to 06f4615 Compare November 6, 2026 09:06
@moetemp
moetemp force-pushed the moe/brook/review/s7-workflow-streams branch from d57a2ae to aef29b9 Compare November 6, 2026 09:22
@moetemp
moetemp force-pushed the moe/brook/review/s6-redis-memory branch from 06f4615 to 84bc88f Compare November 6, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant