Skip to content

fix(visualizer): cap live SSE streams, bound per-stream queues, never leak a subscription - #36

Open
lmajano wants to merge 14 commits into
developmentfrom
fix/sse-stream-hardening
Open

lmajano wants to merge 14 commits into
developmentfrom
fix/sse-stream-hardening

Conversation

@lmajano

@lmajano lmajano commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Description

Hardens the Live Tracker's Visualizer.stream() SSE action against three problems found by a reviewer with jcmd:

  1. No connection limit. Every open stream pins two threads (the Undertow worker blocked in SSE()'s latch, plus the callback thread). A loop of GET .../stream requests exhausts the worker pool and takes the whole app down when the routes are not secured.
  2. Unbounded queue. Each stream buffered events in a LinkedBlockingQueue() with no capacity, so a stalled or slow client grew memory without limit while rules kept firing.
  3. Subscription leak on failure. bus.subscribe() ran before SSE(), so if SSE() threw, the token was never unsubscribed.

Changes:

  • New models/metrics/LiveStreams.bx (LiveStreams@rulebox, singleton): an atomic stream-slot counter (tryAcquire / release), a bounded per-stream queue (newQueue, 1000 events) with offer() that drops the OLDEST event when full and never blocks the bus, and serve() which takes a slot, subscribes, runs the opener, and always unsubscribes and releases the slot in a finally.
  • handlers/Visualizer.bx stream() now goes through serve(). At the cap it returns HTTP 503 { "error": "Too many live tracker connections" }. The preHandler enabled-gate is untouched.
  • New setting visualizer.maxStreams (default 25), one line in ModuleConfig.bx. The handler reads it defensively (missing, non-numeric or < 1 falls back to 25), so it works whether or not the settings deep-merge fix has landed. Documented in docs/guides/visualizer.md.

Verified against the test-harness on BoxLang 1.18.0 (miniserver):

  • Before: 40 concurrent GET .../stream -> 32 streams open (all 32 Undertow worker threads taken), 8 requests hang, and GET .../index times out (8s). The app is down.
  • After: 40 concurrent -> 25 open, 15 get 503 {"error":"Too many live tracker connections"}; GET .../index still answers 200 in ~0.06s; after the clients leave, a new stream is accepted again (slots released by disconnect cleanup, ~2-28s depending on when the disconnect is noticed).
  • A published event still reaches an open stream (event: rule with the JSON payload).
  • Failure paths exercised through a scratch handler (removed): an opener that throws and a subscribe() that throws both leave activeCount() == 0 and no leaked bus subscribers; 50 publishes into a capacity-10 queue that is never drained keep exactly the newest 10.
  • The TestBox runner does not work under the miniserver (missing globber module), so the new specs are relying on CI.

Issues

Found during the 2.0.0 pre-release review of the Visualizer's live SSE stream (reviewer jcmd thread dumps). No separate GitHub issue was filed.

Type of change

  • Bug Fix
  • Improvement
  • New Feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Behavior change to note: more than maxStreams (default 25) simultaneous Live Tracker connections now get a 503 instead of being served.

Checklist

  • My code follows the style guidelines of this project cfformat
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works (RuleEventBusSpec.bx: cap rejects the N+1th stream, a failed open or failed subscribe releases the slot, the bounded queue drops the oldest; VisualizerHandlerSpec.bx: stream() answers 503 at the cap, with the default and with an unusable setting)
  • New and existing unit tests pass locally with my changes (TestBox cannot run under the miniserver here; boxlang check passes on every touched file and the behavior was verified live as above; relying on CI for the suite)

Noticed, not fixed

  • SSE( async: true ) still pins two threads per stream (worker blocked in the latch plus the callback thread). The cap bounds it; switching to a non-blocking approach would be a separate change.
  • On the very first concurrent burst against a cold server, 1 of 15 simultaneous rejected requests returned a 500 from ColdBox's DataMarshaller.renderContent (variables.requestService was null: a first-use initialization race inside ColdBox's renderData). It did not recur once warm. Not caused by this change.
  • The test-harness Application.bx only finds the module when the checkout directory is literally named rulebox (its moduleRootPath regex), so it fails to boot from a worktree under any other directory name.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX


Generated by Claude Code

claude added 4 commits October 3, 2026 11:53
ColdBox merges an app's moduleSettings over a module's defaults with a
shallow append, so the documented minimal config
`visualizer = { enabled = true }` replaced the whole default visualizer
struct and dropped metricsStore/datasourceName. RuleEventBus.onDIComplete
then threw on the missing key (its catch block re-read the same key, so
the error escaped), the bus was never built, and every RuleBook run failed
with a null publish() call.

- ModuleConfig: single visualizerDefaults() source; onLoad() deep-merges
  the defaults under whatever the app supplied.
- RuleEventBus, Visualizer handler, SQLiteMetricsStore: tolerate missing
  visualizer keys (defence in depth); resolveMetricsStore() no longer reads
  settings inside its own catch block.
- Add VisualizerSettingsSpec.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
… leak a subscription

stream() opened an SSE connection per tab with no limit (each pins two server
threads, so a loop of GET requests could exhaust the worker pool), buffered events
in an unbounded queue, and subscribed to the bus before SSE() so a failure opening
the stream leaked the subscription.

- New LiveStreams@rulebox model: atomic stream-slot counter, bounded per-stream
  queue (1000 events, drops the oldest when full, never blocks the bus), and
  serve() which subscribes, runs the opener, and always unsubscribes and releases
  the slot in a finally.
- New setting visualizer.maxStreams (default 25, read defensively in the handler);
  at the cap stream() answers HTTP 503 {"error":"Too many live tracker connections"}.
- Specs for the cap, slot release on failed open, and the bounded queue; docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
RuleEventBusSpec declared a second newBus() helper in the same run() scope as the
existing one, which BoxLang rejects at compile time ("Cannot define multiple
functions with the same name: newBus"); the whole bundle failed to load and the
runner produced no report. The new helpers are renamed (newStreamBus,
newLiveStreams).

The stream() cap specs in VisualizerHandlerSpec now call setup() first so execute()
gets a fresh request context instead of inheriting the previous spec's 404
renderData.

Verified with the real TestBox runner against the test-harness: the full suite is
144 passed, 0 failed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
ModuleConfig is now the single place that sets up and checks the visualizer
settings: onLoad() deep-merges the app's values over visualizerDefaults(), then
throws RuleBox.InvalidSettingException for a value of the wrong shape. The
handler, RuleEventBus and SQLiteMetricsStore go back to reading the settings
directly, with no scattered fallbacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
claude added 9 commits October 3, 2026 14:47
The handler reads visualizer.maxStreams directly; ModuleConfig fills the default
of 25 and rejects anything that is not a whole number of 1 or more at load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
The defaults do not depend on instance state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
The defaults do not depend on instance state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
A module is loaded once, so the defaults are a static struct instead of a
function. configure() copies it, and onLoad() fills the keys an app left out
with append( static.VISUALIZER_DEFAULTS, false ), so the static is never
written to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc5MDcm3tyR2yqPWs9RpgX
…ge' into fix/sse-stream-hardening

# Conflicts:
#	ModuleConfig.bx
…-hardening

# Conflicts:
#	ModuleConfig.bx
#	test-harness/tests/specs/VisualizerSettingsSpec.bx

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants