Skip to content

feat: Stream backpressure and drain (#43) - #25692

Merged
godsflaw merged 1 commit into
msgpack:masterfrom
enochgroot:feat/stream-drain
Sep 19, 2026
Merged

godsflaw merged 1 commit into
msgpack:masterfrom
enochgroot:feat/stream-drain

Conversation

@enochgroot

Copy link
Copy Markdown
Collaborator

Summary

  • Stream.send queues packed messages when the underlying writable returns false, then flushes FIFO on drain.
  • The Stream re-emits drain so callers can listen on msgpack.Stream, not only on the raw socket.
  • Pending sends are capped at 1024; a further send() throws. Version 3.3.0.

Motivation

Stream.send returned the boolean from write() but did not listen for drain and did not queue if the caller kept sending after backpressure. Slow sockets could drop messages.

Closes #43

Changes

  • Queue already-packed Buffers after write() returns false; flush FIFO on drain.
  • send() stays synchronous and returns the boolean from write(), or false if the message was queued.
  • Extra arguments (encoding, callback) are still forwarded on an immediate write. Queued flushes call write(buf) without inventing an encoding; a supplied callback runs after that buffer is written or if the queue is dropped.
  • Cap at 1024 pending messages. Further send() throws a catchable Error mentioning backpressure / queue full.
  • On underlying error / close / end, drop the queue and emit error if unsent messages remain. Handlers do not throw.
  • Read/data/msg path is unchanged (packed 0/null, bytes_remaining snapshot, error-on-malformed).
  • No src/msgpack.cc change.

Stacked PR

Parent #40 is still open: #25691 (feat/lazy-unpack @ 81bf44f). This branch is stacked on that. The diff vs master includes those commits; Stream backpressure is the commit on top.

Test Plan

  • Unit tests pass (npm test — 222/222)
  • JS coverage 100% statements/branches/functions/lines (npm run coverage:js)
  • Mock writable: first write returns true; send returns true; no queue
  • Mock writable: write returns false; second send is queued and flushed FIFO on drain
  • Queue cap 1024: 1025th pending send throws
  • Extra callback argument on send still runs (immediate and queued)
  • Packed 0 and null still emit as msg (no Stream: unpacking '0' doesn't trigger 'msg' emit  #44 regression)
  • No src/msgpack.cc change; worker_threads suite still green

Notes for Reviewers

- Queue packed messages when write() returns false
- Re-emit drain; cap pending sends at 1024
- Drop the queue on underlying error/close/end
- Version 3.3.0

Closes msgpack#43.
@godsflaw
godsflaw merged commit 4944d65 into msgpack:master Sep 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msgpack stream write 'drain'?

2 participants