test(e2e): stagehand MQTT state and hot list

E2E_USER_STORIES.md

@@ -447,6 +447,49 @@ Config gaps caused several production incidents, and they are invisible to UI sp
 | CFG-04 | P1 | A missing `org`/parented-`level` zone is caught as `/misconfigured`, not as a blank page. | **partial** — WP-E2E-01 asserts the healthy path; the negative case is unasserted |
 | CFG-05 | P2 | Deployment-shaped run: the app served by nginx at `/workplace/` behind the `verified` cookie gate. | todo — a second project; `mintToken` already captures the cookie |
 
+## 5. Stagehand — MQTT state and the hot list
+
+Stagehand's hot list is built from MQTT: drivers write state to redis, `source` publishes
+each status retained to mosquitto, and stagehand subscribes through `/api/mqtt/`. The stack
+runs `source` and `mosquitto` for this, and the specs drive state with a purpose-built driver,
+`drivers/place/e2e/state_harness.cr` (a UDP device whose `connected` and other statuses are
+set through `exec`). See "MQTT" in `e2e/README.md`.
+
+**The broker is asserted directly, not only through the page.** Stagehand re-checks every
+MQTT alert over REST before showing it (`DashboardsService._updateDashboardAlertList`), which
+hides wrong data on the broker. Removing that re-check — it is what makes stagehand slow — is
+only safe once the MQTT rows below are all green.
+
+| ID | P | Story | Status |
+|----|---|-------|--------|
+| SH-MQTT-01 | P0 | A started module publishes `connected: true`, retained, at its full topic: org, `_` for an absent region and area, building, level, system, driver, module name, index. | **done** — `local/stagehand-mqtt.spec.ts` |
+| SH-MQTT-02 | P0 | `connected` follows the module through false and back to true. | **done** — `local/stagehand-mqtt.spec.ts` |
+| SH-MQTT-03 | P1 | Any status is published as the JSON value the driver set (boolean, number, string, object), not a string of it. | **done** — `local/stagehand-mqtt.spec.ts` |
+| SH-MQTT-04 | P0 | A client connecting later is handed the current value of every status, and nothing the module does not hold. | **done** — `local/stagehand-mqtt.spec.ts` |
+| SH-HOT-01 | P0 | The hot list shows a module while it reports `connected: false`, and not while it is connected; a reload agrees. Held against a disconnected control module. | **done** — `local/stagehand-hotlist.spec.ts` |
+| SH-MQTT-10 | P0 | Reordering a system's modules moves each module's state to its new index; nothing is left at the old one. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts`. The field report "alerts on the old index". Root cause: source numbered modules in database order (`Module.find_all`), not the system's, so a reorder changed nothing it published. |
+| SH-MQTT-11 | P0 | A module removed from a system (still existing) leaves no retained topics under that system. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts` |
+| SH-MQTT-12 | P0 | A deleted module leaves no retained topics. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts`. The field report "results for deleted modules". source worked out what to remove from the module's redis state and the current mappings, either of which can be gone first. |
+| SH-MQTT-13 | P1 | A deleted system leaves no retained topics. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts` |
+| SH-MQTT-14 | P1 | Renaming a module (`custom_name`) moves its state to the new name, and other modules' indexes with it. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts` |
+| SH-MQTT-15 | P1 | Moving a system between zones moves its topics. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts` |
+| SH-MQTT-16 | P1 | `source` that starts before the broker is reachable publishes once it is. | **done, not automated** — verified by hand (stop mosquitto, restart source, start mosquitto: all state published). Automating it needs a spec that can stop containers. source spec covers the retry path. |
+| SH-MQTT-17 | P1 | A status the driver clears is removed from the broker, not retained as `null`. | **done** — `local/stagehand-mqtt-lifecycle.spec.ts` |
+| SH-MQTT-18 | P0 | A module added to a system after it started has its state published. | **done** — every lifecycle spec's setup depends on it. Was a race: the state was written before source mapped the module, and nothing republished it. Found while building this coverage. |
+| SH-MQTT-19 | P0 | Two quick edits to one system leave source's mapping matching the latest. | **done** — the lifecycle setup makes them (two modules added back to back). pg-orm and placeos-resource handle each change on its own fiber, so source processed them out of order and kept the older module list. Found while building this coverage. |
+| SH-MQTT-20 | P1 | After a broker restart, every status is published again. | **done, not automated** — verified by hand (restart mosquitto: 32 of 32 topics back within 2 s; before the fix, 0 until each value changed). |
+| SH-MQTT-21 | P1 | Retained topics nothing maps to (left by a module deleted while source was down, or by an older source) are removed when source starts. | **done, not automated** — verified by hand with a planted topic. source spec `sweeps retained topics that nothing maps to`. |
+| SH-HOT-02 | P1 | A module added to a system after stagehand has loaded it can still raise an alert. | **done** — `local/stagehand-hotlist.spec.ts`. Stagehand cached each system's module list forever (`_system_modules_cache`); it now refetches, at most every 5 s, before dropping an alert for an unknown device. |
+
+**The lifecycle specs need source v1.22.1 or later** (PlaceOS/source#65). Against
+`placeos-2.2610.1` SH-MQTT-10 … 19 and SH-HOT-02 are red, which is what they are for. Until a
+PlaceOS release carries the fix, run them with `E2E_SOURCE_IMAGE=placeos/source:nightly` (a
+nightly built after 2026-10-14) or a local build (see `e2e/README.md`, "MQTT"). The advisory CI
+runs only the workplace suite, so it is not affected.
+
+**Not covered yet**: a status set with a TTL (`set_expire`) that expires in redis publishes
+nothing, so the broker keeps its last value. source would need redis keyspace notifications to see it.
+
 ---
 
 ## Notes & blockers

apps/stagehand/e2e/local/stagehand-hotlist.spec.ts

@@ -0,0 +1,196 @@
+/**
+ * Stagehand's hot list, end to end: a module that reports `connected: false`
+ * is listed, and one that is connected is not.
+ *
+ *   start a harness (UDP, so it comes up connected) -> not listed
+ *   harness reports connected: false                -> listed
+ *   harness reports connected: true                 -> gone again
+ *
+ * The broker is asserted alongside the page at every step. When stagehand
+ * stops re-checking alerts over REST, this spec is what says the page is still
+ * right; until then, the broker assertions are what say the data is.
+ *
+ * "Not listed" needs a control (see e2e/README.md, Conventions): the page shows
+ * a spinner, not an empty table, until its first alert. So the system also
+ * holds a second harness, `Projector_1`, that is disconnected first.
+ * Once the page lists the control, it has the system's state, and a missing
+ * `Display_1` row means what it says.
+ */
+import {
+    BrokerView,
+    systemFilter,
+} from '../../../../e2e/support/mqtt/broker-view';
+import {
+    expectHotList,
+    hotListRows,
+    openHotList,
+} from '../../../../e2e/support/stagehand/alerts.page';
+import { expect, test } from '../../../../e2e/support/stagehand/fixtures';
+import {
+    addHarness,
+    createSystem,
+    deleteSystem,
+    setConnected,
+    sweepSystems,
+    waitForConnected,
+} from '../../../../e2e/support/stagehand/stagehand.api';
+import { MQTT_PROPAGATION_MS } from '../../../../e2e/support/stagehand/stagehand.env';
+
+test.beforeAll(async ({ adminApi }) => {
+    await sweepSystems(adminApi);
+});
+
+test('a module is listed while it reports connected: false, and only then', async ({
+    page,
+    adminApi,
+    stagehandAdminToken,
+    harnessDriverId,
+    zones,
+}) => {
+    const system = await createSystem(adminApi, zones, 'hotlist');
+    const broker = await BrokerView.open(
+        stagehandAdminToken,
+        systemFilter(zones.org, system.id),
+    );
+    // Stagehand logs why it drops an alert; attached so a failure explains itself
+    const app_log: string[] = [];
+    page.on('console', (m) => {
+        if (/ALERTS/.test(m.text())) app_log.push(m.text());
+    });
+    try {
+        // Stagehand open first, so every change below arrives live
+        await openHotList(page);
+        await expectHotList(page, system.id, []);
+
+        // Both harnesses start connected, so neither is listed yet
+        const control = await addHarness(adminApi, system.id, harnessDriverId, {
+            custom_name: 'Projector',
+        });
+        const display = await addHarness(adminApi, system.id, harnessDriverId);
+        await waitForConnected(adminApi, control.id, true);
+        await waitForConnected(adminApi, display.id, true);
+        await broker.waitForValue(
+            'Projector',
+            1,
+            'connected',
+            true,
+            MQTT_PROPAGATION_MS,
+        );
+        await broker.waitForValue(
+            'Display',
+            1,
+            'connected',
+            true,
+            MQTT_PROPAGATION_MS,
+        );
+
+        // NOTE: both modules exist before the system's first alert. Stagehand
+        // caches a system's module list on its first alert and never refreshes
+        // it, so a module added after that cannot alert until a reload
+        // (SH-HOT-02). This spec is about the working path.
+        await setConnected(adminApi, control.id, false);
+        await broker.waitForValue(
+            'Projector',
+            1,
+            'connected',
+            false,
+            MQTT_PROPAGATION_MS,
+        );
+
+        await test.step('a connected module is not listed', async () => {
+            await expectHotList(page, system.id, ['Projector_1']);
+        });
+
+        await test.step('it is listed once it reports connected: false', async () => {
+            await setConnected(adminApi, display.id, false);
+            await broker.waitForValue(
+                'Display',
+                1,
+                'connected',
+                false,
+                MQTT_PROPAGATION_MS,
+            );
+            await expectHotList(page, system.id, ['Display_1', 'Projector_1']);
+
+            const [row] = (await hotListRows(page, system.id)).filter(
+                (r) => r.device === 'Display_1',
+            );
+            expect(row.cells.join(' ')).toContain('Disconnected');
+        });
+
+        await test.step('and leaves it when it reconnects', async () => {
+            await setConnected(adminApi, display.id, true);
+            await broker.waitForValue(
+                'Display',
+                1,
+                'connected',
+                true,
+                MQTT_PROPAGATION_MS,
+            );
+            await expectHotList(page, system.id, ['Projector_1']);
+        });
+
+        await test.step('a stagehand opened now agrees', async () => {
+            await page.reload();
+            await expectHotList(page, system.id, ['Projector_1']);
+        });
+    } finally {
+        await test
+            .info()
+            .attach('stagehand ALERTS log', { body: app_log.join('\n') });
+        await test
+            .info()
+            .attach('broker messages', { body: broker.describeLog() });
+        await broker.close();
+        await deleteSystem(adminApi, system.id);
+    }
+});
+
+test('SH-HOT-02: a module added after the page loaded the system can still be listed', async ({
+    page,
+    adminApi,
+    stagehandAdminToken,
+    harnessDriverId,
+    zones,
+}) => {
+    const system = await createSystem(adminApi, zones, 'late-module');
+    const broker = await BrokerView.open(
+        stagehandAdminToken,
+        systemFilter(zones.org, system.id),
+    );
+    const app_log: string[] = [];
+    page.on('console', (m) => {
+        if (/ALERTS/.test(m.text())) app_log.push(m.text());
+    });
+    try {
+        await openHotList(page);
+
+        // The system's first alert, which is when stagehand learns its modules
+        const control = await addHarness(adminApi, system.id, harnessDriverId, {
+            custom_name: 'Projector',
+        });
+        await waitForConnected(adminApi, control.id, true);
+        await setConnected(adminApi, control.id, false);
+        await expectHotList(page, system.id, ['Projector_1']);
+
+        // Added afterwards, then disconnected
+        const display = await addHarness(adminApi, system.id, harnessDriverId);
+        await waitForConnected(adminApi, display.id, true);
+        await setConnected(adminApi, display.id, false);
+        await broker.waitForValue(
+            'Display',
+            1,
+            'connected',
+            false,
+            MQTT_PROPAGATION_MS,
+        );
+
+        await expectHotList(page, system.id, ['Display_1', 'Projector_1']);
+    } finally {
+        await test
+            .info()
+            .attach('stagehand ALERTS log', { body: app_log.join('\n') });
+        await broker.close();
+        await deleteSystem(adminApi, system.id);
+    }
+});