Skip to content

[DERCBOT-2176] Support delta streaming (streamMode) - #218

Draft
assouktim wants to merge 5 commits into
masterfrom
feature/stream-delta-and-sse-count
Draft

assouktim wants to merge 5 commits into
masterfrom
feature/stream-delta-and-sse-count

Conversation

@assouktim

Copy link
Copy Markdown

Title: Support Tock RAG streaming: streamMode: 'delta' option and fix SSE response counting

Context

Tock's web connector can now stream RAG answers progressively on the regular /sse?userid= channel when tock_web_sse=true (server-side
change: Tock PR #2129). It uses the standard streaming contract TRK already knows:

  • each partial frame is a message event with RESPONSE_ID and TOCK_STREAM_RESPONSE=true in its metadata;
  • the client replaces the message carrying the same RESPONSE_ID;
  • the final answer (footnotes, buttons…) keeps RESPONSE_ID and replaces the streamed message.

No new event type or endpoint is introduced. This PR makes TRK handle that flow correctly and adds an optional, bandwidth-friendly
delta mode.

Changes

  1. Fix: partial streamed frames no longer break the handledResponses counter

handleSseBotResponse counted every SSE frame, and handlePostBotResponse then sliced the POST response starting from that counter. With
N partial frames, a POST response arriving before the final SSE frame was sliced from index N and displayed nothing, and the final SSE
frame was then ignored.

Frames flagged TOCK_STREAM_RESPONSE=true are now handled but not counted. The POST response (or the final SSE frame) replaces the
message by RESPONSE_ID.

▎ Note for reviewers: with the Bot API streaming and tock_web_connector_merge_stream_response=false, frames carry fragments and the
▎ POST response re-dispatches them through UPDATE_MESSAGE. This is the one existing flow whose behavior I could not fully validate.
▎ Please check it on your side.

  1. New option streamMode: 'delta'

By default each frame carries the whole accumulated text (already rendered to HTML by the server). For long answers this transfers a
lot of redundant data. With streamMode: 'delta', TRK sends "streamMode": "delta" in the POST body and the server only sends the part of
the text that changed:

  • partial frames carry TOCK_STREAM_MODE=delta and TOCK_STREAM_KEEP=N;
  • the reducer (UPDATE_MESSAGE in TockState.tsx) keeps the first N characters of the current message and appends the frame text;
  • the final answer is always complete and replaces the message, so the display is resynchronized;
  • delta fragments are not stored in the local history, only the final answer is.

The server renders markdown exactly as in the default mode and sends the end of the rendered text, so the display is identical in both
modes (raw token fragments were tried first and abandoned: the display was condensed until the rendered final answer arrived).

Option wiring: TockOptions.streamMode, NetworkSettings.streamMode, copied by renderChat, and added to the POST body in useTock.ts only
when set.

  1. README

streamMode documented in the NetworkSettings and renderChat options tables, and the "Streamed responses" section completed.

Backward compatibility

  • streamMode is optional. Without it the request body, the frames and the display are unchanged.
  • Servers that do not stream (or have tock_web_sse=false) are unaffected.
  • The only behavioral change without opt-in is the counting fix in section 1.

Testing

  • tsc --noEmit, npm run build, eslint and prettier --check pass.
  • Manual tests in a browser against a Tock bot with RAG and tock_web_sse=true, with and without streamMode: 'delta': progressive
    display, same rendering in both modes, final answer replaces the bubble with footnotes.
  • There is no automated test suite in TRK (npm test is a stub).

@Fabilin Fabilin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. Just a small comment regarding the new configuration 😉

Comment thread src/TockOptions.ts
timeoutBetweenMessage?: number;
widgets?: { [id: string]: (props: unknown) => JSX.Element };
disableSse?: boolean;
streamMode?: 'delta';

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new field is redundant, TockOptions already contains the network settings through inheritance. Most properties in this specific interface are legacy support.

isFrameOfOtherRequest only filtered correlated frames while a local
request was in flight (inFlightRequestIds.size > 0). An idle tab thus
accepted a foreign tab's partial frames, then rejected the rest once it
sent its own question, leaving an orphan provisional bubble never
finalized nor recorded. For the same reason a late frame arriving after
the local request completed could still be handled.

Drop the size > 0 guard: a correlated frame is accepted only while its
request is in flight here. This also secures the error path - the
request id is removed from the in-flight set in sendMessage's finally
(on success or failure), so a late partial frame is now foreign and
dropped, and can no longer reset error to false via the reducer. Frames
without a request id (notifications) still always pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants