From f1e3f1db854e63a765d51b1e4a901eb23d99de26 Mon Sep 17 00:00:00 2001 From: Abian Suarez Date: Mon, 19 Oct 2026 19:55:48 +0100 Subject: [PATCH 1/3] feat(extension): import .osrec recordings and apply their direction - New `.osrec` file format (lib/recording-file.ts): a raw recording (meta, markers, optional direction, video) that crosses browsers. The editor imports it from its empty screen, the popup's file icon, or a file packaged in the extension (`editor.html?import=`). - Automation API in the service worker (lib/automation.ts) for the recorder CLI over CDP: start, stop once saved, export in base64 chunks. Automated sessions open no editor tab on save. - Direction from the recorder's flow (lib/direction.ts): click tooltips, zooms and `hotspot: false` on the matching markers; waits cut from the clips as jump cuts; zoomed actions split their clip wide/zoomed/wide. Recordings made by hand are unchanged. --- apps/extension/CLAUDE.md | 10 +- .../__test__/unit/background.test.ts | 44 ++++ apps/extension/entrypoints/background.ts | 31 ++- apps/extension/entrypoints/editor/App.tsx | 8 +- .../editor/components/ImportScreen.tsx | 64 +++++ .../entrypoints/editor/components/Screens.tsx | 3 + .../core/__test__/unit/import-source.test.ts | 24 ++ .../core/__test__/unit/step-editor.test.ts | 8 + .../entrypoints/editor/core/import-source.ts | 11 + .../entrypoints/editor/core/step-editor.ts | 2 +- .../editor/hooks/use-recording-import.ts | 41 ++++ .../entrypoints/editor/hooks/use-recording.ts | 7 +- .../lib/__test__/unit/editable-step.test.ts | 40 ++++ .../entrypoints/editor/lib/editable-step.ts | 7 + apps/extension/entrypoints/popup/App.tsx | 60 +++-- .../lib/__test__/unit/automation.test.ts | 153 ++++++++++++ .../lib/__test__/unit/direction.test.ts | 79 +++++++ .../lib/__test__/unit/recording-file.test.ts | 221 ++++++++++++++++++ .../lib/__test__/unit/segments.test.ts | 92 ++++++++ apps/extension/lib/automation.ts | 83 +++++++ apps/extension/lib/db.ts | 36 +++ apps/extension/lib/direction.ts | 42 ++++ apps/extension/lib/messaging.ts | 2 + apps/extension/lib/recording-file.ts | 153 ++++++++++++ apps/extension/lib/segments.ts | 79 ++++++- 25 files changed, 1260 insertions(+), 40 deletions(-) create mode 100644 apps/extension/entrypoints/editor/components/ImportScreen.tsx create mode 100644 apps/extension/entrypoints/editor/core/__test__/unit/import-source.test.ts create mode 100644 apps/extension/entrypoints/editor/core/import-source.ts create mode 100644 apps/extension/entrypoints/editor/hooks/use-recording-import.ts create mode 100644 apps/extension/lib/__test__/unit/automation.test.ts create mode 100644 apps/extension/lib/__test__/unit/direction.test.ts create mode 100644 apps/extension/lib/__test__/unit/recording-file.test.ts create mode 100644 apps/extension/lib/automation.ts create mode 100644 apps/extension/lib/direction.ts create mode 100644 apps/extension/lib/recording-file.ts diff --git a/apps/extension/CLAUDE.md b/apps/extension/CLAUDE.md index c08c92a..f9a22ea 100644 --- a/apps/extension/CLAUDE.md +++ b/apps/extension/CLAUDE.md @@ -4,7 +4,7 @@ WXT + React Chrome extension (MV3) with two jobs: **capture** (continuous tab vi ## Commands -- `pnpm dev` / `pnpm build` (`:firefox` variants exist). Test manually by loading `.output/chrome-mv3` unpacked. There is no E2E test for recording. +- `pnpm dev` / `pnpm build` (`:firefox` variants exist). Test manually by loading `.output/chrome-mv3` unpacked. The recording E2E test lives in `packages/recorder` (`src/__test__/e2e`): it records a flow with this build in headless Chromium and imports the result in the editor. - `pnpm test`: Vitest in Node with WXT's `fakeBrowser` (in-memory `browser.*`/`chrome.*`). No real DOM, canvas or IndexedDB. - `postinstall` runs `wxt prepare` to generate `.wxt/` types. @@ -19,6 +19,14 @@ WXT + React Chrome extension (MV3) with two jobs: **capture** (continuous tab vi - Chrome burns the real cursor into tab captures. `lib/segments.ts` only keeps a video clip between clicks when there is activity (scroll/type/drag) or the gap is ≥3s. Otherwise the steps go photo → photo and the hotspot glides. Fast-forwarding idle stretches was tried and rejected. - The content script checks `ctx.isInvalid` (not `isInvalidated`) and detaches listeners in `ctx.onInvalidated`. An orphaned script after an extension reload would otherwise throw "Extension context invalidated" forever. +## Recordings from the recorder CLI + +- `lib/automation.ts` is the API `packages/recorder` calls over CDP (`globalThis.openshowcase` in the service worker): `start` → `stop` (waits for the save) → `exportFile` (with the flow's direction) → `readChunk` (base64). Its shape and `Direction` (`lib/direction.ts`) are mirrored in the recorder's `browser.ts`; change both. +- Direction: `cueClicks` puts the flow's `label`/`zoom`/`hotspot` on the matching click markers; `RecordingRecord.direction` keeps `cuts` and `zooms`. `buildStepPlan` uses them only when present (hand-made recordings are unchanged): a `hotspot: false` click is activity, not a photo; cuts split clips (jump cuts, and don't count toward the 3s long-gap rule); a zoom splits its clip into wide → zoomed (padded 0.5s) → wide contiguous pieces. +- A session started that way has `automated: true`, so saving it opens no editor tab. +- `.osrec` (`lib/recording-file.ts`): `OSREC` + version byte, uint32 LE header length, JSON header (recording meta, markers, optional direction), then the webm. The editor imports it from its empty screen (`ImportScreen`, also reached from the popup's file icon), or on load from a file packaged in the extension (`editor.html?import=imports/.osrec`, `core/import-source.ts`: what the recorder's review browser uses), into IndexedDB under a new id. +- `MarkerRecord.label` (the flow's `tooltip`) wins over `elementText` for the hotspot's tooltip. + ## Testing patterns - Code touching APIs `fakeBrowser` lacks (`tabCapture`, `offscreen`, `action`, IndexedDB) takes injectable deps. `startRecording(deps?: StartRecordingDeps)` in `background.ts` is the reference. diff --git a/apps/extension/entrypoints/__test__/unit/background.test.ts b/apps/extension/entrypoints/__test__/unit/background.test.ts index ad2344a..a476486 100644 --- a/apps/extension/entrypoints/__test__/unit/background.test.ts +++ b/apps/extension/entrypoints/__test__/unit/background.test.ts @@ -1,6 +1,7 @@ import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { fakeBrowser } from "wxt/testing/fake-browser"; import { + handleRecordingSaved, type StartRecordingDeps, startRecording, stopRecording, @@ -92,6 +93,49 @@ describe("startRecording", () => { expect(await recordingSession.getValue()).toMatchObject({ active: false }); expect(counterCalls).toContain(null); }); + + it("marks a session started by automation, so saving it opens no editor", async () => { + const unlisten = onMessage("offscreenStart", () => {}); + await startRecording(fakeDeps(), { automated: true }); + unlisten(); + expect(await recordingSession.getValue()).toMatchObject({ + active: true, + automated: true, + }); + }); +}); + +describe("handleRecordingSaved", () => { + const savedDeps = { + setCounterIcon: async () => {}, + closeOffscreenDocumentIfOpen: async () => {}, + }; + + it("ends the session and opens the editor on the recording", async () => { + await recordingSession.setValue({ active: true, recordingId: "rec-1" }); + await handleRecordingSaved("rec-1", savedDeps); + + expect(await recordingSession.getValue()).toMatchObject({ active: false }); + const tabs = await fakeBrowser.tabs.query({}); + expect(tabs.map((t) => t.url)).toContainEqual( + expect.stringContaining("/editor.html?recordingId=rec-1"), + ); + }); + + it("opens no editor for an automated recording (the recorder CLI takes it)", async () => { + await recordingSession.setValue({ + active: true, + recordingId: "rec-1", + automated: true, + }); + await handleRecordingSaved("rec-1", savedDeps); + + expect(await recordingSession.getValue()).toMatchObject({ active: false }); + const tabs = await fakeBrowser.tabs.query({}); + expect(tabs.map((t) => t.url)).not.toContainEqual( + expect.stringContaining("editor.html"), + ); + }); }); describe("stopRecording", () => { diff --git a/apps/extension/entrypoints/background.ts b/apps/extension/entrypoints/background.ts index f6e23e0..81ca565 100644 --- a/apps/extension/entrypoints/background.ts +++ b/apps/extension/entrypoints/background.ts @@ -1,5 +1,8 @@ +import { createAutomation } from "@/lib/automation"; import { addMarker, + getMarkersByRecording, + getRecording, markRecordingInterrupted, markRecordingStarted, createRecording as saveRecording, @@ -24,6 +27,17 @@ export default defineBackground(() => { onMessage("recordingSaved", ({ data }) => handleRecordingSaved(data.recordingId), ); + + // The recorder CLI (packages/recorder) drives recordings over CDP, through this global: + // there's no toolbar click in its automated Chromium. Only reachable from DevTools/CDP. + Object.assign(globalThis, { + openshowcase: createAutomation({ + startRecording: () => startRecording(undefined, { automated: true }), + stopRecording, + getRecording, + getMarkers: getMarkersByRecording, + }), + }); }); async function ensureOffscreenDocument(): Promise { @@ -95,6 +109,7 @@ const defaultStartRecordingDeps: StartRecordingDeps = { export async function startRecording( deps: StartRecordingDeps = defaultStartRecordingDeps, + { automated = false }: { automated?: boolean } = {}, ): Promise { const tab = await deps.getActiveTab(); if (!tab?.id) return; @@ -111,6 +126,7 @@ export async function startRecording( recordingId, tabId: tab.id, clickCount: 0, + automated, }); await deps.setCounterIcon(0); @@ -212,13 +228,22 @@ async function handleMarker( }); } -async function handleRecordingSaved(recordingId: string): Promise { +export interface RecordingSavedDeps { + setCounterIcon: (count: number | null) => Promise; + closeOffscreenDocumentIfOpen: () => Promise; +} + +export async function handleRecordingSaved( + recordingId: string, + deps: RecordingSavedDeps = { setCounterIcon, closeOffscreenDocumentIfOpen }, +): Promise { const state = await recordingSession.getValue(); if (state.recordingId === recordingId) { await patchRecordingSessionState({ active: false }); } - await setCounterIcon(null); - await closeOffscreenDocumentIfOpen(); + await deps.setCounterIcon(null); + await deps.closeOffscreenDocumentIfOpen(); + if (state.automated && state.recordingId === recordingId) return; // When the recording ends, open the editor directly. await browser.tabs.create({ url: browser.runtime.getURL(`/editor.html?recordingId=${recordingId}`), diff --git a/apps/extension/entrypoints/editor/App.tsx b/apps/extension/entrypoints/editor/App.tsx index e6b6198..bbae5dd 100644 --- a/apps/extension/entrypoints/editor/App.tsx +++ b/apps/extension/entrypoints/editor/App.tsx @@ -1,6 +1,7 @@ import { Toaster } from "@/components/ui/sonner"; import { TooltipProvider } from "@/components/ui/tooltip"; import { useColorScheme } from "@/hooks/use-color-scheme"; +import { ImportScreen } from "./components/ImportScreen"; import { LoadingScreen, MessageScreen } from "./components/Screens"; import { Editor } from "./Editor"; import { useRecording } from "./hooks/use-recording"; @@ -15,12 +16,7 @@ function App() { return ( - {state.status === "missing-id" && ( - - )} + {state.status === "missing-id" && } {state.status === "loading" && } {state.status === "empty" && ( (null); + + return ( +
+ + Record a tab from the OpenShowcase toolbar button, or import a{" "} + {RECORDING_FILE_EXTENSION}{" "} + recording made by the recorder CLI. + + } + action={ + <> + + { + const file = e.target.files?.[0]; + if (file) importFile(file); + e.target.value = ""; + }} + /> + + } + /> + {dragging && ( +
+ Drop to import this recording +
+ )} +
+ ); +} diff --git a/apps/extension/entrypoints/editor/components/Screens.tsx b/apps/extension/entrypoints/editor/components/Screens.tsx index ba6db68..ae45664 100644 --- a/apps/extension/entrypoints/editor/components/Screens.tsx +++ b/apps/extension/entrypoints/editor/components/Screens.tsx @@ -37,9 +37,11 @@ function Shimmer({ className }: { className: string }) { export function MessageScreen({ title, text, + action, }: { title: string; text: React.ReactNode; + action?: React.ReactNode; }) { return (
@@ -53,6 +55,7 @@ export function MessageScreen({ {text}

+ {action}
{ + it("reads a recording the recorder CLI put in the extension's imports folder", () => { + expect(importPathFromQuery("?import=imports/factura-venta.osrec")).toBe( + "imports/factura-venta.osrec", + ); + }); + + it("ignores pages without one", () => { + expect(importPathFromQuery("?recordingId=abc")).toBeUndefined(); + expect(importPathFromQuery("")).toBeUndefined(); + }); + + it("refuses anything outside the imports folder or not a recording", () => { + expect(importPathFromQuery("?import=../secret.osrec")).toBeUndefined(); + expect(importPathFromQuery("?import=imports/../x.osrec")).toBeUndefined(); + expect( + importPathFromQuery("?import=https://evil.example/x.osrec"), + ).toBeUndefined(); + expect(importPathFromQuery("?import=imports/x.js")).toBeUndefined(); + }); +}); diff --git a/apps/extension/entrypoints/editor/core/__test__/unit/step-editor.test.ts b/apps/extension/entrypoints/editor/core/__test__/unit/step-editor.test.ts index 7e089cf..9e7be3b 100644 --- a/apps/extension/entrypoints/editor/core/__test__/unit/step-editor.test.ts +++ b/apps/extension/entrypoints/editor/core/__test__/unit/step-editor.test.ts @@ -287,6 +287,14 @@ describe("defaultPanZoom", () => { expect(pz.y).toBeCloseTo(1 - half); }); + it("takes another scale, still kept inside the image", () => { + expect(StepEditor.defaultPanZoom({ x: 0.5, y: 0.98 }, 2.5)).toEqual({ + x: 0.5, + y: 0.8, + scale: 2.5, + }); + }); + it("centers on the image when there is no focus point", () => { expect(StepEditor.defaultPanZoom(undefined)).toMatchObject({ x: 0.5, diff --git a/apps/extension/entrypoints/editor/core/import-source.ts b/apps/extension/entrypoints/editor/core/import-source.ts new file mode 100644 index 0000000..21ca90b --- /dev/null +++ b/apps/extension/entrypoints/editor/core/import-source.ts @@ -0,0 +1,11 @@ +/** + * A recording the recorder CLI's `open` command copied into this extension's `imports/` + * folder, named in the editor's URL (`?import=imports/.osrec`). Only plain file names + * in that folder: the editor fetches it from its own extension origin. + */ +export function importPathFromQuery(search: string): string | undefined { + const path = new URLSearchParams(search).get("import"); + return path && /^imports\/[\w.-]+\.osrec$/.test(path) && !path.includes("..") + ? path + : undefined; +} diff --git a/apps/extension/entrypoints/editor/core/step-editor.ts b/apps/extension/entrypoints/editor/core/step-editor.ts index 4ad9ebd..af12743 100644 --- a/apps/extension/entrypoints/editor/core/step-editor.ts +++ b/apps/extension/entrypoints/editor/core/step-editor.ts @@ -189,8 +189,8 @@ const DEFAULT_ZOOM_SCALE = 1.8; */ export function defaultPanZoom( focus: { x: number; y: number } | undefined, + scale = DEFAULT_ZOOM_SCALE, ): PanZoom { - const scale = DEFAULT_ZOOM_SCALE; const half = 0.5 / scale; const clampToImage = (v: number) => Math.min(1 - half, Math.max(half, v)); return { diff --git a/apps/extension/entrypoints/editor/hooks/use-recording-import.ts b/apps/extension/entrypoints/editor/hooks/use-recording-import.ts new file mode 100644 index 0000000..be2f36a --- /dev/null +++ b/apps/extension/entrypoints/editor/hooks/use-recording-import.ts @@ -0,0 +1,41 @@ +import { useCallback, useEffect, useState } from "react"; +import { toast } from "sonner"; +import { importRecording } from "@/lib/db"; +import { unpackRecording } from "@/lib/recording-file"; + +/** + * Imports an `.osrec` file into the database, then reopens the editor on it. `packaged` is a + * recording shipped inside the extension (see core/import-source.ts), imported on mount. + */ +export function useRecordingImport(packaged?: string) { + const [importing, setImporting] = useState(packaged != null); + + const importBlob = useCallback(async (load: () => Promise) => { + setImporting(true); + try { + const recordingId = await importRecording( + await unpackRecording(await load()), + ); + window.location.replace(`?recordingId=${recordingId}`); + } catch (error) { + toast.error(error instanceof Error ? error.message : String(error)); + setImporting(false); + } + }, []); + + useEffect(() => { + if (!packaged) return; + void importBlob(async () => { + const response = await fetch(`/${packaged}`); + if (!response.ok) throw new Error(`Couldn't read ${packaged}`); + return response.blob(); + }); + }, [packaged, importBlob]); + + const importFile = useCallback( + (file: File) => importBlob(async () => file), + [importBlob], + ); + + return { importing, importFile }; +} diff --git a/apps/extension/entrypoints/editor/hooks/use-recording.ts b/apps/extension/entrypoints/editor/hooks/use-recording.ts index 0ff09b8..eeec84b 100644 --- a/apps/extension/entrypoints/editor/hooks/use-recording.ts +++ b/apps/extension/entrypoints/editor/hooks/use-recording.ts @@ -46,7 +46,12 @@ export function useRecording(recordingId: string | null): RecordingState { return; } const initialSteps = stepsFromPlan( - buildStepPlan(markers, recording.durationSec, recording.startedAt), + buildStepPlan( + markers, + recording.durationSec, + recording.startedAt, + recording.direction, + ), ); if (initialSteps.length === 0) { setState({ status: "empty" }); diff --git a/apps/extension/entrypoints/editor/lib/__test__/unit/editable-step.test.ts b/apps/extension/entrypoints/editor/lib/__test__/unit/editable-step.test.ts index e909bca..0b67f77 100644 --- a/apps/extension/entrypoints/editor/lib/__test__/unit/editable-step.test.ts +++ b/apps/extension/entrypoints/editor/lib/__test__/unit/editable-step.test.ts @@ -27,6 +27,14 @@ describe("autoTooltipLabel", () => { ); }); + it("uses the label the recording chose over the element text", () => { + expect( + autoTooltipLabel( + marker({ elementText: "New", label: "Create your first invoice" }), + ), + ).toBe("Create your first invoice"); + }); + it("falls back to a generic label when there is no element text", () => { expect(autoTooltipLabel(marker({ elementText: undefined }))).toBe( "Click here", @@ -61,6 +69,38 @@ describe("stepsFromPlan", () => { }); }); + it("zooms a photo step on its click when the recording asked for it", () => { + const [step] = stepsFromPlan([ + { + kind: "photo", + time: 1, + marker: marker({ xFrac: 0.5, yFrac: 0.5, zoom: 2 }), + }, + ]); + expect(step?.data.kind === "photo" && step.data.panZoom).toEqual({ + x: 0.5, + y: 0.5, + scale: 2, + }); + }); + + it("frames a clip where the plan asked for it, kept inside the image", () => { + const [step] = stepsFromPlan([ + { + kind: "video", + startTime: 0, + endTime: 2, + playbackRate: 1, + zoom: { x: 0.99, y: 0.5, scale: 2 }, + }, + ]); + expect(step?.data.kind === "video" && step.data.panZoom).toEqual({ + x: 0.75, + y: 0.5, + scale: 2, + }); + }); + it("gives a photo step no hotspot when the marker has no click position", () => { const plan: StepPlan[] = [ { diff --git a/apps/extension/entrypoints/editor/lib/editable-step.ts b/apps/extension/entrypoints/editor/lib/editable-step.ts index ac3636a..e7326c0 100644 --- a/apps/extension/entrypoints/editor/lib/editable-step.ts +++ b/apps/extension/entrypoints/editor/lib/editable-step.ts @@ -6,6 +6,7 @@ import { } from "@rustrak/openshowcase-schema"; import type { MarkerRecord } from "@/lib/db"; import type { StepPlan } from "@/lib/segments"; +import { defaultPanZoom } from "../core/step-editor"; export interface PhotoData { kind: "photo"; @@ -35,6 +36,7 @@ export interface EditableStep { export const DEFAULT_HOTSPOT_LABEL = "Click here"; export function autoTooltipLabel(marker: MarkerRecord): string { + if (marker.label) return marker.label; if (marker.elementText) return `Click “${marker.elementText}”`; return DEFAULT_HOTSPOT_LABEL; } @@ -57,12 +59,14 @@ export function stepsFromPlan(plan: StepPlan[]): EditableStep[] { ...defaultHotspotStyle, } : undefined; + const { zoom } = item.marker; return { id: newStepId(), data: { kind: "photo", sourceTime: item.time, hotspot, + ...(zoom && { panZoom: defaultPanZoom(hotspot, zoom) }), } satisfies PhotoData, }; } @@ -73,6 +77,9 @@ export function stepsFromPlan(plan: StepPlan[]): EditableStep[] { trimStart: item.startTime, trimEnd: item.endTime, playbackRate: item.playbackRate, + ...(item.zoom && { + panZoom: defaultPanZoom(item.zoom, item.zoom.scale), + }), } satisfies VideoData, }; }); diff --git a/apps/extension/entrypoints/popup/App.tsx b/apps/extension/entrypoints/popup/App.tsx index 88b4d5a..0194b40 100644 --- a/apps/extension/entrypoints/popup/App.tsx +++ b/apps/extension/entrypoints/popup/App.tsx @@ -1,5 +1,6 @@ import { ArrowUpRight, + FileUp, MonitorPlay, MousePointerClick, Square, @@ -58,6 +59,11 @@ function App() { window.close(); } + async function openImport() { + await browser.tabs.create({ url: browser.runtime.getURL("/editor.html") }); + window.close(); + } + async function openEditor(recordingId: string) { await browser.tabs.create({ url: browser.runtime.getURL(`/editor.html?recordingId=${recordingId}`), @@ -73,24 +79,17 @@ function App() { {session.active ? ( ) : ( - - - } +
+ + + + - - - Test an exported demo - - + +
)} @@ -124,6 +123,35 @@ function App() { ); } +function HeaderAction({ + label, + onClick, + children, +}: { + label: string; + onClick: () => void; + children: React.ReactNode; +}) { + return ( + + + } + > + {children} + + {label} + + ); +} + function IdleView({ tab, onStart, diff --git a/apps/extension/lib/__test__/unit/automation.test.ts b/apps/extension/lib/__test__/unit/automation.test.ts new file mode 100644 index 0000000..cfc8f24 --- /dev/null +++ b/apps/extension/lib/__test__/unit/automation.test.ts @@ -0,0 +1,153 @@ +import { beforeEach, describe, expect, it } from "vitest"; +import { fakeBrowser } from "wxt/testing/fake-browser"; +import { type AutomationDeps, createAutomation } from "@/lib/automation"; +import type { MarkerRecord, RecordingRecord } from "@/lib/db"; +import { unpackRecording } from "@/lib/recording-file"; +import { recordingSession } from "@/lib/recording-storage"; + +beforeEach(() => { + fakeBrowser.reset(); +}); + +const finished: RecordingRecord = { + id: "rec-1", + createdAt: 1_000, + startedAt: 2_000, + status: "complete", + videoBlob: new Blob([new Uint8Array([9, 8, 7])], { type: "video/webm" }), + mimeType: "video/webm", + durationSec: 3, + width: 2560, + height: 1600, + tabId: 1, +}; + +const click: MarkerRecord = { + id: "m-1", + recordingId: "rec-1", + kind: "click", + capturedAt: 2_500, + pageUrl: "https://example.com", + xFrac: 0.5, + yFrac: 0.5, +}; + +function fakeDeps(overrides: Partial = {}): AutomationDeps { + return { + startRecording: async () => { + await recordingSession.setValue({ active: true, recordingId: "rec-1" }); + }, + stopRecording: async () => {}, + getRecording: async () => finished, + getMarkers: async () => [click], + sleep: async () => {}, + ...overrides, + }; +} + +function fromBase64(base64: string): number[] { + return [...atob(base64)].map((c) => c.charCodeAt(0)); +} + +describe("start", () => { + it("returns the id of the recording it started", async () => { + expect(await createAutomation(fakeDeps()).start()).toBe("rec-1"); + }); + + it("throws when the recording didn't start", async () => { + const automation = createAutomation( + fakeDeps({ startRecording: async () => {} }), + ); + await expect(automation.start()).rejects.toThrow(/didn't start/); + }); +}); + +describe("stop", () => { + it("waits until the recording is saved", async () => { + const states: RecordingRecord["status"][] = [ + "recording", + "recording", + "complete", + ]; + let reads = 0; + let stopped = false; + const automation = createAutomation( + fakeDeps({ + stopRecording: async () => { + stopped = true; + }, + getRecording: async () => ({ + ...finished, + status: states[Math.min(reads++, states.length - 1)] ?? "complete", + }), + }), + ); + await automation.stop("rec-1"); + expect(stopped).toBe(true); + expect(reads).toBe(3); + }); + + it("throws when the recording was interrupted", async () => { + const automation = createAutomation( + fakeDeps({ + getRecording: async () => ({ ...finished, status: "interrupted" }), + }), + ); + await expect(automation.stop("rec-1")).rejects.toThrow(/interrupted/); + }); + + it("gives up when the recording is never saved", async () => { + const automation = createAutomation( + fakeDeps({ + getRecording: async () => ({ ...finished, status: "recording" }), + }), + ); + await expect(automation.stop("rec-1")).rejects.toThrow(/never saved/); + }); +}); + +describe("exportFile / readChunk", () => { + it("exports the recording as a file with the flow's click labels", async () => { + const automation = createAutomation(fakeDeps()); + const size = await automation.exportFile("rec-1", { + clicks: [{ start: 2_450, end: 2_600, label: "Open the menu", zoom: 2 }], + cuts: [{ start: 2_700, end: 2_800 }], + zooms: [], + }); + + let base64 = ""; + const chunk = 7; + const bytes: number[] = []; + for (let offset = 0; offset < size; offset += chunk) { + base64 = await automation.readChunk(offset, chunk); + bytes.push(...fromBase64(base64)); + } + expect(bytes).toHaveLength(size); + + const file = await unpackRecording(new Blob([new Uint8Array(bytes)])); + expect(file.markers).toEqual([ + { + kind: "click", + capturedAt: 2_500, + pageUrl: "https://example.com", + xFrac: 0.5, + yFrac: 0.5, + label: "Open the menu", + zoom: 2, + }, + ]); + expect(file.direction).toEqual({ + cuts: [{ start: 2_700, end: 2_800 }], + zooms: [], + }); + expect([...new Uint8Array(await file.video.arrayBuffer())]).toEqual([ + 9, 8, 7, + ]); + }); + + it("throws when reading before exporting", async () => { + await expect(createAutomation(fakeDeps()).readChunk(0, 10)).rejects.toThrow( + /exportFile/, + ); + }); +}); diff --git a/apps/extension/lib/__test__/unit/direction.test.ts b/apps/extension/lib/__test__/unit/direction.test.ts new file mode 100644 index 0000000..e1db902 --- /dev/null +++ b/apps/extension/lib/__test__/unit/direction.test.ts @@ -0,0 +1,79 @@ +import { describe, expect, it } from "vitest"; +import type { MarkerRecord } from "@/lib/db"; +import { type ClickCue, cueClicks } from "@/lib/direction"; + +function marker( + capturedAt: number, + kind: MarkerRecord["kind"] = "click", +): MarkerRecord { + return { + id: `m-${capturedAt}`, + recordingId: "rec-1", + kind, + capturedAt, + pageUrl: "https://example.com", + }; +} + +const label = (start: number, end: number, text: string): ClickCue => ({ + start, + end, + label: text, +}); + +describe("cueClicks", () => { + it("labels the click pressed while the flow's click was running", () => { + const labelled = cueClicks( + [marker(1_050), marker(2_030)], + [label(1_000, 1_100, "Open invoices"), label(2_000, 2_100, "New one")], + ); + expect(labelled.map((m) => m.label)).toEqual(["Open invoices", "New one"]); + }); + + it("tolerates a press stamped just outside the click's window", () => { + const [labelled] = cueClicks( + [marker(990)], + [label(1_000, 1_100, "Open invoices")], + ); + expect(labelled?.label).toBe("Open invoices"); + }); + + it("leaves clicks no flow step caused unlabelled", () => { + const labelled = cueClicks( + [marker(500), marker(1_050)], + [label(1_000, 1_100, "Open invoices")], + ); + expect(labelled.map((m) => m.label)).toEqual([undefined, "Open invoices"]); + }); + + it("gives one label to only one click (a double-fired press keeps the first)", () => { + const labelled = cueClicks( + [marker(1_040), marker(1_060)], + [label(1_000, 1_100, "Open invoices")], + ); + expect(labelled.map((m) => m.label)).toEqual(["Open invoices", undefined]); + }); + + it("never labels activity markers", () => { + const labelled = cueClicks( + [marker(1_050, "type")], + [label(1_000, 1_100, "Open invoices")], + ); + expect(labelled[0]?.label).toBeUndefined(); + }); + + it("carries the click's zoom and hotspot choice to its marker", () => { + const [labelled] = cueClicks( + [marker(1_050)], + [{ start: 1_000, end: 1_100, zoom: 2, hotspot: false }], + ); + expect(labelled).toMatchObject({ zoom: 2, hotspot: false }); + expect(labelled).not.toHaveProperty("label"); + }); + + it("doesn't mutate the markers it's given", () => { + const markers = [marker(1_050)]; + cueClicks(markers, [label(1_000, 1_100, "Open invoices")]); + expect(markers[0]?.label).toBeUndefined(); + }); +}); diff --git a/apps/extension/lib/__test__/unit/recording-file.test.ts b/apps/extension/lib/__test__/unit/recording-file.test.ts new file mode 100644 index 0000000..a6fd1fe --- /dev/null +++ b/apps/extension/lib/__test__/unit/recording-file.test.ts @@ -0,0 +1,221 @@ +import { describe, expect, it } from "vitest"; +import type { MarkerRecord, RecordingRecord } from "@/lib/db"; +import { + fileFromRecords, + packRecording, + type RecordingFile, + recordsFromFile, + unpackRecording, +} from "@/lib/recording-file"; + +const video = new Blob([new Uint8Array([1, 2, 3, 4, 5])], { + type: "video/webm", +}); + +function recordingFile(overrides: Partial = {}): RecordingFile { + return { + recording: { + createdAt: 1_000, + startedAt: 2_000, + mimeType: "video/webm", + durationSec: 4.5, + width: 2560, + height: 1600, + }, + markers: [ + { + kind: "click", + capturedAt: 2_500, + pageUrl: "https://example.com", + xFrac: 0.25, + yFrac: 0.5, + selector: "#new", + elementText: "New invoice", + label: "Create your first invoice", + }, + { kind: "scroll", capturedAt: 3_000, pageUrl: "https://example.com" }, + ], + video, + ...overrides, + }; +} + +async function bytes(blob: Blob): Promise { + return [...new Uint8Array(await blob.arrayBuffer())]; +} + +describe("packRecording / unpackRecording", () => { + it("round-trips the recording, its markers and the video bytes", async () => { + const file = recordingFile(); + const unpacked = await unpackRecording(await packRecording(file)); + + expect(unpacked.recording).toEqual(file.recording); + expect(unpacked.markers).toEqual(file.markers); + expect(await bytes(unpacked.video)).toEqual([1, 2, 3, 4, 5]); + expect(unpacked.video.type).toBe("video/webm"); + }); + + it("round-trips the recorder's direction", async () => { + const direction = { + cuts: [{ start: 2_600, end: 2_900 }], + zooms: [{ start: 2_500, end: 2_800, x: 0.4, y: 0.6, scale: 2 }], + }; + const unpacked = await unpackRecording( + packRecording(recordingFile({ direction })), + ); + expect(unpacked.direction).toEqual(direction); + }); + + it("rejects a malformed direction", async () => { + const packed = packRecording( + recordingFile({ + direction: { cuts: "soon" } as unknown as RecordingFile["direction"], + }), + ); + await expect(unpackRecording(packed)).rejects.toThrow(/damaged/); + }); + + it("keeps non-ASCII text intact", async () => { + const file = recordingFile({ + markers: [ + { + kind: "click", + capturedAt: 2_500, + pageUrl: "https://example.com", + label: "Créa tu factura “ya” 🚀", + }, + ], + }); + const unpacked = await unpackRecording(await packRecording(file)); + expect(unpacked.markers[0]?.label).toBe("Créa tu factura “ya” 🚀"); + }); + + it("rejects a file that isn't a recording", async () => { + await expect( + unpackRecording(new Blob(["PK\u0003\u0004 a zip, not a recording"])), + ).rejects.toThrow(/not an OpenShowcase recording/); + }); + + it("rejects a file too short to hold a header", async () => { + await expect(unpackRecording(new Blob(["OS"]))).rejects.toThrow( + /not an OpenShowcase recording/, + ); + }); + + it("rejects a newer format version", async () => { + const packed = new Uint8Array( + await (await packRecording(recordingFile())).arrayBuffer(), + ); + packed[5] = 99; + await expect(unpackRecording(new Blob([packed]))).rejects.toThrow( + /newer version/, + ); + }); + + it("rejects a truncated header", async () => { + const packed = await packRecording(recordingFile()); + await expect(unpackRecording(packed.slice(0, 20))).rejects.toThrow( + /damaged/, + ); + }); + + it("rejects a header without the recording's timing", async () => { + const packed = await packRecording( + recordingFile({ + recording: { + ...recordingFile().recording, + startedAt: null as unknown as number, + }, + }), + ); + await expect(unpackRecording(packed)).rejects.toThrow(/damaged/); + }); + + it("rejects a recording without video", async () => { + const packed = await packRecording(recordingFile({ video: new Blob([]) })); + await expect(unpackRecording(packed)).rejects.toThrow(/no video/); + }); +}); + +describe("fileFromRecords / recordsFromFile", () => { + const record: RecordingRecord = { + id: "rec-1", + createdAt: 1_000, + startedAt: 2_000, + status: "complete", + videoBlob: video, + mimeType: "video/webm", + durationSec: 4.5, + width: 2560, + height: 1600, + tabId: 7, + }; + const markers: MarkerRecord[] = [ + { + id: "m-1", + recordingId: "rec-1", + kind: "click", + capturedAt: 2_500, + pageUrl: "https://example.com", + xFrac: 0.25, + yFrac: 0.5, + }, + ]; + + it("drops the database ids and the tab from the file", () => { + const file = fileFromRecords(record, markers); + expect(file.recording).toEqual({ + createdAt: 1_000, + startedAt: 2_000, + mimeType: "video/webm", + durationSec: 4.5, + width: 2560, + height: 1600, + }); + expect(file.markers).toEqual([ + { + kind: "click", + capturedAt: 2_500, + pageUrl: "https://example.com", + xFrac: 0.25, + yFrac: 0.5, + }, + ]); + expect(file.video).toBe(video); + }); + + it("refuses a recording that hasn't finished", () => { + expect(() => + fileFromRecords({ ...record, status: "recording" }, markers), + ).toThrow(/not finished/); + expect(() => fileFromRecords({ ...record, videoBlob: null }, [])).toThrow( + /not finished/, + ); + }); + + it("keeps the direction both ways", () => { + const direction = { cuts: [{ start: 1, end: 2 }], zooms: [] }; + const file = fileFromRecords({ ...record, direction }, markers); + expect(file.direction).toEqual(direction); + expect(recordsFromFile(file, "rec-new").recording.direction).toEqual( + direction, + ); + }); + + it("gives an imported file fresh ids under the new recording", () => { + let n = 0; + const { recording, markers: imported } = recordsFromFile( + fileFromRecords(record, markers), + "rec-new", + () => `id-${++n}`, + ); + expect(recording).toEqual({ + ...record, + id: "rec-new", + tabId: -1, + }); + expect(imported).toEqual([ + { ...markers[0], id: "id-1", recordingId: "rec-new" }, + ]); + }); +}); diff --git a/apps/extension/lib/__test__/unit/segments.test.ts b/apps/extension/lib/__test__/unit/segments.test.ts index a0fc62c..9ec4904 100644 --- a/apps/extension/lib/__test__/unit/segments.test.ts +++ b/apps/extension/lib/__test__/unit/segments.test.ts @@ -126,3 +126,95 @@ describe("buildStepPlan", () => { expect(plan[0]?.kind).toBe("photo"); }); }); + +describe("buildStepPlan with a recording's direction (recorder CLI)", () => { + const noDirection = { cuts: [], zooms: [] }; + + it("plays a click marked hotspot: false inside a clip instead of stopping on it", () => { + const plan = buildStepPlan( + [ + clickMarker(1000), + clickMarker(1800, { hotspot: false }), + clickMarker(2500), + ], + 5, + 0, + noDirection, + ); + // the quiet click is activity: the stretch around it stays a clip + expect(plan.map((s) => s.kind)).toEqual([ + "video", + "photo", + "video", + "photo", + "video", + ]); + }); + + it("cuts the waits out of the clips, as jump cuts", () => { + const plan = buildStepPlan([clickMarker(1000)], 8, 0, { + cuts: [{ start: 3000, end: 6000 }], + zooms: [], + }); + expect(spans(plan)).toEqual([ + [0, 0.967], + [0.967, 0.967], + [0.967, 3], + [6, 8], + ]); + }); + + it("drops what's left of a clip when a cut leaves too little of it", () => { + const plan = buildStepPlan([clickMarker(1000)], 4, 0, { + cuts: [{ start: 1300, end: 4000 }], + zooms: [], + }); + expect(plan.map((s) => s.kind)).toEqual(["video", "photo"]); + }); + + it("doesn't keep a clip for a long gap that is mostly a wait", () => { + const plan = buildStepPlan([clickMarker(1000), clickMarker(5000)], 8, 0, { + cuts: [{ start: 1500, end: 4500 }], + zooms: [], + }); + expect(plan.map((s) => s.kind)).toEqual([ + "video", + "photo", + "photo", + "video", + ]); + }); + + it("frames a clip on the action the flow zoomed into", () => { + const plan = buildStepPlan( + [clickMarker(1000), activityMarker("type", 2000)], + 5, + 0, + { + cuts: [], + zooms: [{ start: 1500, end: 2500, x: 0.3, y: 0.6, scale: 2 }], + }, + ); + // The camera moves in a little before the action and out a little after it: the clip + // splits there, into contiguous pieces that play straight through. + expect( + plan.map((s) => + s.kind === "video" + ? [s.startTime, s.endTime, s.zoom?.scale ?? null] + : "photo", + ), + ).toEqual([[0, 0.967, null], "photo", [0.967, 3, 2], [3, 5, null]]); + }); + + it("keeps a zoomed clip whole when what's left around the action is too short", () => { + const plan = buildStepPlan([clickMarker(1000)], 3.2, 0, { + cuts: [], + zooms: [{ start: 1500, end: 2500, x: 0.3, y: 0.6, scale: 2 }], + }); + expect(plan.at(-1)).toMatchObject({ + startTime: 0.967, + endTime: 3.2, + zoom: { scale: 2 }, + }); + }); +}); diff --git a/apps/extension/lib/automation.ts b/apps/extension/lib/automation.ts new file mode 100644 index 0000000..ad44d88 --- /dev/null +++ b/apps/extension/lib/automation.ts @@ -0,0 +1,83 @@ +import type { MarkerRecord, RecordingRecord } from "@/lib/db"; +import { cueClicks, type Direction } from "@/lib/direction"; +import { fileFromRecords, packRecording } from "@/lib/recording-file"; +import { recordingSession } from "@/lib/recording-storage"; + +export interface AutomationDeps { + startRecording: () => Promise; + stopRecording: () => Promise; + getRecording: (id: string) => Promise; + getMarkers: (id: string) => Promise; + sleep?: (ms: number) => Promise; +} + +const POLL_MS = 100; +/** Stopping remuxes the whole video (offscreen/lib/remux.ts): minutes of 60fps take a while. */ +const SAVE_TIMEOUT_MS = 120_000; + +/** + * What the recorder CLI calls in the service worker (over CDP) to record a tab it drives: + * start, stop once saved, then pull the recording out as an `.osrec` file in base64 chunks + * (a single CDP message can't carry a long recording). + */ +export function createAutomation(deps: AutomationDeps) { + const sleep = + deps.sleep ?? + ((ms: number) => new Promise((resolve) => setTimeout(resolve, ms))); + let exported: Blob | undefined; + + return { + /** Records the active tab; resolves with the recording's id once capture runs. */ + async start(): Promise { + await deps.startRecording(); + const state = await recordingSession.getValue(); + if (!state.active || !state.recordingId) + throw new Error( + "The recording didn't start (see the extension's service worker console)", + ); + return state.recordingId; + }, + + async stop(recordingId: string): Promise { + await deps.stopRecording(); + for (let waited = 0; waited <= SAVE_TIMEOUT_MS; waited += POLL_MS) { + const status = (await deps.getRecording(recordingId))?.status; + if (status === "complete") return; + if (status === "interrupted") + throw new Error("The recording was interrupted"); + await sleep(POLL_MS); + } + throw new Error("The recording was never saved"); + }, + + /** Packs the recording with the flow's direction; resolves with its size. */ + async exportFile(recordingId: string, direction?: Direction) { + const [record, markers] = await Promise.all([ + deps.getRecording(recordingId), + deps.getMarkers(recordingId), + ]); + if (!record) throw new Error(`No recording ${recordingId}`); + const { clicks = [], cuts = [], zooms = [] } = direction ?? {}; + exported = packRecording( + fileFromRecords( + { ...record, ...(direction && { direction: { cuts, zooms } }) }, + cueClicks(markers, clicks), + ), + ); + return exported.size; + }, + + async readChunk(offset: number, length: number): Promise { + if (!exported) throw new Error("Call exportFile before readChunk"); + const bytes = new Uint8Array( + await exported.slice(offset, offset + length).arrayBuffer(), + ); + let binary = ""; + for (let i = 0; i < bytes.length; i += 0x8000) + binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000)); + return btoa(binary); + }, + }; +} + +export type Automation = ReturnType; diff --git a/apps/extension/lib/db.ts b/apps/extension/lib/db.ts index 707f66e..4aefe1d 100644 --- a/apps/extension/lib/db.ts +++ b/apps/extension/lib/db.ts @@ -1,4 +1,5 @@ import { type DBSchema, type IDBPDatabase, openDB } from "idb"; +import { type RecordingFile, recordsFromFile } from "@/lib/recording-file"; export type RecordingStatus = "recording" | "complete" | "interrupted"; @@ -13,6 +14,21 @@ export interface RecordingRecord { width: number | null; height: number | null; tabId: number; + /** Editing the recorder CLI's flow asked for (absent on recordings made by hand). */ + direction?: RecordingDirection; +} + +export interface TimeRange { + start: number; + end: number; +} + +/** Wall-clock times, like the markers' `capturedAt`. */ +export interface RecordingDirection { + /** Waits for the app: cut out of the clips. */ + cuts: TimeRange[]; + /** Clips that play an action the flow zoomed into frame on it. */ + zooms: (TimeRange & { x: number; y: number; scale: number })[]; } export type MarkerKind = "click" | "scroll" | "type" | "drag"; @@ -27,6 +43,12 @@ export interface MarkerRecord { yFrac?: number; selector?: string; elementText?: string; + /** Tooltip text chosen by whoever recorded (the recorder CLI's flow), over the element text. */ + label?: string; + /** The flow's camera zoom on this click's photo step. */ + zoom?: number; + /** false: the click plays inside a clip instead of becoming a photo step. */ + hotspot?: boolean; pageUrl: string; } @@ -153,3 +175,17 @@ export async function listRecentRecordings( .sort((a, b) => b.createdAt - a.createdAt) .slice(0, limit); } + +/** Stores a recording file (made elsewhere, e.g. by the recorder CLI); resolves with its new id. */ +export async function importRecording(file: RecordingFile): Promise { + const id = crypto.randomUUID(); + const { recording, markers } = recordsFromFile(file, id); + const db = await getDb(); + const tx = db.transaction(["recordings", "markers"], "readwrite"); + await Promise.all([ + tx.objectStore("recordings").put(recording), + ...markers.map((marker) => tx.objectStore("markers").put(marker)), + tx.done, + ]); + return id; +} diff --git a/apps/extension/lib/direction.ts b/apps/extension/lib/direction.ts new file mode 100644 index 0000000..d19e3dc --- /dev/null +++ b/apps/extension/lib/direction.ts @@ -0,0 +1,42 @@ +import type { MarkerRecord, RecordingDirection } from "@/lib/db"; + +/** A click the recorder CLI performed: wall-clock window around it + what the flow asked of it. */ +export interface ClickCue { + start: number; + end: number; + label?: string; + zoom?: number; + hotspot?: boolean; +} + +/** Everything the recorder CLI's flow asks of a recording, matched onto it at export. */ +export interface Direction extends RecordingDirection { + clicks: ClickCue[]; +} + +/** The press can be stamped a little before the CLI's own `start` (two clocks, rounding). */ +const TOLERANCE_MS = 50; + +/** Gives each click marker what the flow click it fell within asked for, once per cue. */ +export function cueClicks( + markers: MarkerRecord[], + cues: ClickCue[], +): MarkerRecord[] { + const unused = [...cues]; + return markers.map((marker) => { + if ((marker.kind ?? "click") !== "click") return marker; + const index = unused.findIndex( + (c) => + marker.capturedAt >= c.start - TOLERANCE_MS && + marker.capturedAt <= c.end + TOLERANCE_MS, + ); + if (index === -1) return marker; + const [{ label, zoom, hotspot } = {}] = unused.splice(index, 1); + return { + ...marker, + ...(label != null && { label }), + ...(zoom != null && { zoom }), + ...(hotspot != null && { hotspot }), + }; + }); +} diff --git a/apps/extension/lib/messaging.ts b/apps/extension/lib/messaging.ts index edea79f..589138e 100644 --- a/apps/extension/lib/messaging.ts +++ b/apps/extension/lib/messaging.ts @@ -46,6 +46,8 @@ export interface RecordingSessionState { recordingId?: string; tabId?: number; clickCount?: number; + /** Started by the recorder CLI: it takes the recording, so no editor opens on save. */ + automated?: boolean; } export const defaultRecordingSessionState: RecordingSessionState = { diff --git a/apps/extension/lib/recording-file.ts b/apps/extension/lib/recording-file.ts new file mode 100644 index 0000000..3342737 --- /dev/null +++ b/apps/extension/lib/recording-file.ts @@ -0,0 +1,153 @@ +import type { + MarkerRecord, + RecordingDirection, + RecordingRecord, +} from "@/lib/db"; + +/** + * A raw recording as a single file (`.osrec`): what a recording made somewhere else (the + * recorder CLI's automated Chromium) hands to this extension's editor. Layout: + * `OSREC` + version byte, header length (uint32 LE), header JSON (UTF-8), video bytes. + */ +export const RECORDING_FILE_EXTENSION = ".osrec"; + +const MAGIC = "OSREC"; +const VERSION = 1; +const PREFIX_BYTES = MAGIC.length + 1 + 4; + +export interface RecordingFile { + recording: { + createdAt: number; + startedAt: number; + mimeType: string; + durationSec: number; + width: number; + height: number; + }; + markers: Omit[]; + /** Editing the recorder CLI's flow asked for. */ + direction?: RecordingDirection; + video: Blob; +} + +export function packRecording(file: RecordingFile): Blob { + const header = new TextEncoder().encode( + JSON.stringify({ + recording: file.recording, + markers: file.markers, + direction: file.direction, + }), + ); + const prefix = new Uint8Array(PREFIX_BYTES); + prefix.set(new TextEncoder().encode(MAGIC)); + prefix[MAGIC.length] = VERSION; + new DataView(prefix.buffer).setUint32(MAGIC.length + 1, header.length, true); + return new Blob([prefix, header, file.video]); +} + +export async function unpackRecording(blob: Blob): Promise { + const prefix = new Uint8Array( + await blob.slice(0, PREFIX_BYTES).arrayBuffer(), + ); + if ( + prefix.length < PREFIX_BYTES || + new TextDecoder().decode(prefix.slice(0, MAGIC.length)) !== MAGIC + ) + throw new Error("This file is not an OpenShowcase recording"); + if (prefix[MAGIC.length] !== VERSION) + throw new Error( + "This recording was made by a newer version of OpenShowcase. Update the extension.", + ); + + const headerLength = new DataView(prefix.buffer).getUint32( + MAGIC.length + 1, + true, + ); + const headerEnd = PREFIX_BYTES + headerLength; + const damaged = () => new Error("This recording file is damaged"); + if (blob.size < headerEnd) throw damaged(); + + let header: Pick; + try { + header = JSON.parse(await blob.slice(PREFIX_BYTES, headerEnd).text()); + } catch { + throw damaged(); + } + const { recording, markers, direction } = header ?? {}; + const numbers = [ + recording?.createdAt, + recording?.startedAt, + recording?.durationSec, + recording?.width, + recording?.height, + ]; + if ( + !numbers.every((n) => typeof n === "number" && Number.isFinite(n)) || + typeof recording.mimeType !== "string" || + !Array.isArray(markers) || + !markers.every( + (m) => typeof m?.capturedAt === "number" && typeof m.pageUrl === "string", + ) || + (direction != null && + !(Array.isArray(direction.cuts) && Array.isArray(direction.zooms))) + ) + throw damaged(); + + const video = blob.slice(headerEnd, blob.size, recording.mimeType); + if (video.size === 0) throw new Error("This recording has no video"); + return { recording, markers, ...(direction && { direction }), video }; +} + +/** A finished recording from the database, as a file. */ +export function fileFromRecords( + record: RecordingRecord, + markers: MarkerRecord[], +): RecordingFile { + const { videoBlob, startedAt, mimeType, durationSec, width, height } = record; + if ( + record.status !== "complete" || + !videoBlob || + startedAt == null || + mimeType == null || + durationSec == null || + width == null || + height == null + ) + throw new Error("The recording is not finished"); + return { + recording: { + createdAt: record.createdAt, + startedAt, + mimeType, + durationSec, + width, + height, + }, + markers: markers.map(({ id: _id, recordingId: _rid, ...marker }) => marker), + ...(record.direction && { direction: record.direction }), + video: videoBlob, + }; +} + +/** The database records of an imported file. It wasn't recorded from any tab here. */ +export function recordsFromFile( + file: RecordingFile, + recordingId: string, + newId: () => string = () => crypto.randomUUID(), +): { recording: RecordingRecord; markers: MarkerRecord[] } { + return { + recording: { + id: recordingId, + ...file.recording, + status: "complete", + videoBlob: file.video, + tabId: -1, + ...(file.direction && { direction: file.direction }), + }, + markers: file.markers.map((marker) => ({ + ...marker, + id: newId(), + recordingId, + })), + }; +} diff --git a/apps/extension/lib/segments.ts b/apps/extension/lib/segments.ts index 537cc1c..9e549bb 100644 --- a/apps/extension/lib/segments.ts +++ b/apps/extension/lib/segments.ts @@ -1,4 +1,4 @@ -import type { MarkerRecord } from "@/lib/db"; +import type { MarkerRecord, RecordingDirection } from "@/lib/db"; /** Margin before the press for the photo step: 1/30s (two frames of the 60fps recording). */ export const FRAME_SEC = 1 / 30; @@ -7,10 +7,21 @@ const MIN_VIDEO_SEC = 0.6; /** Between two clicks with no activity, a stretch at least this long is still kept as a clip * (something probably happened: loading, an animation); shorter ones go photo → photo. */ const LONG_GAP_SEC = 3; +/** The camera starts moving in this long before a zoomed action and leaves this long after. */ +const ZOOM_PAD_SEC = 0.5; +/** Shorter than this, the unzoomed rest of a clip joins the zoomed part instead. */ +const MIN_SPLIT_SEC = 0.6; export type StepPlan = | { kind: "photo"; time: number; marker: MarkerRecord } - | { kind: "video"; startTime: number; endTime: number; playbackRate: number }; + | { + kind: "video"; + startTime: number; + endTime: number; + playbackRate: number; + /** Camera framing the recording asked for (center + scale). */ + zoom?: { x: number; y: number; scale: number }; + }; const ms = (t: number) => Math.round(t * 1000) / 1000; @@ -29,17 +40,34 @@ export function buildStepPlan( markers: MarkerRecord[], durationSec: number, recordingStartedAt: number, + direction?: RecordingDirection, ): StepPlan[] { const toSec = (capturedAt: number) => (capturedAt - recordingStartedAt) / 1000; const clamp = (t: number) => Math.max(0, Math.min(durationSec, t)); + const isQuietClick = (m: MarkerRecord) => m.hotspot === false; + const cuts = (direction?.cuts ?? []).map((c) => ({ + start: toSec(c.start), + end: toSec(c.end), + })); + const zooms = (direction?.zooms ?? []).map((z) => ({ + ...z, + start: toSec(z.start), + end: toSec(z.end), + })); + const cutWithin = (start: number, end: number) => + cuts.reduce( + (sum, c) => + sum + Math.max(0, Math.min(end, c.end) - Math.max(start, c.start)), + 0, + ); // Photo instants: the frame just before the press (`capturedAt` is the pointerdown time). const photos: { time: number; marker: MarkerRecord }[] = []; for (const marker of [...markers].sort( (a, b) => a.capturedAt - b.capturedAt, )) { - if ((marker.kind ?? "click") !== "click") continue; + if ((marker.kind ?? "click") !== "click" || isQuietClick(marker)) continue; const pressed = toSec(marker.capturedAt); if (pressed < -1 || pressed > durationSec) continue; const time = ms(clamp(pressed - FRAME_SEC)); @@ -49,21 +77,48 @@ export function buildStepPlan( photos.push({ time, marker }); } + // A click marked hotspot: false plays inside the clip, like scrolling or typing. const activityTimes = markers - .filter((m) => m.kind && m.kind !== "click") + .filter((m) => (m.kind && m.kind !== "click") || isQuietClick(m)) .map((m) => toSec(m.capturedAt)); const hasActivityBetween = (start: number, end: number) => activityTimes.some((t) => t > start && t < end); const plan: StepPlan[] = []; + const push = (start: number, end: number, zoom?: (typeof zooms)[number]) => + plan.push({ + kind: "video", + startTime: ms(start), + endTime: ms(end), + playbackRate: 1, + ...(zoom && { zoom: { x: zoom.x, y: zoom.y, scale: zoom.scale } }), + }); + // A zoomed action splits its clip: wide, zoomed in on the action, wide again. The pieces are + // contiguous, so the player rolls straight through while the camera moves. + // ponytail: one zoom per clip piece; a second one in the same piece is ignored. + const piece = (start: number, end: number) => { + if (end - start < MIN_VIDEO_SEC) return; + const zoom = zooms.find( + (z) => z.start - ZOOM_PAD_SEC < end && z.end + ZOOM_PAD_SEC > start, + ); + if (!zoom) return push(start, end); + let from = Math.max(start, zoom.start - ZOOM_PAD_SEC); + let to = Math.min(end, zoom.end + ZOOM_PAD_SEC); + if (from - start < MIN_SPLIT_SEC) from = start; + if (end - to < MIN_SPLIT_SEC) to = end; + if (from > start) push(start, from); + push(from, to, zoom); + if (to < end) push(to, end); + }; + // The cuts (waits for the app) come out of a clip as jump cuts. const clip = (start: number, end: number) => { - if (end - start >= MIN_VIDEO_SEC) - plan.push({ - kind: "video", - startTime: ms(start), - endTime: ms(end), - playbackRate: 1, - }); + let from = start; + for (const c of [...cuts].sort((a, b) => a.start - b.start)) { + if (c.end <= from || c.start >= end) continue; + piece(from, c.start); + from = c.end; + } + piece(from, end); }; let cursor = 0; @@ -72,7 +127,7 @@ export function buildStepPlan( const keep = !betweenClicks || hasActivityBetween(cursor, photo.time) || - photo.time - cursor >= LONG_GAP_SEC; + photo.time - cursor - cutWithin(cursor, photo.time) >= LONG_GAP_SEC; if (keep) clip(cursor, photo.time); plan.push({ kind: "photo", time: photo.time, marker: photo.marker }); cursor = photo.time; From 2af8c0354d70370af0f97633e398bd36f0978a33 Mon Sep 17 00:00:00 2001 From: Abian Suarez Date: Mon, 19 Oct 2026 19:55:49 +0100 Subject: [PATCH 2/3] feat(recorder): add a CLI and Claude Code skill to record demos with an agent `@rustrak/openshowcase-recorder` plays a JSON flow in headless Chromium while the extension records it through its own capture pipeline: - `inspect`: runs the flow fast and prints where it ends as an accessibility tree (screenshot only on request, `--within` to narrow). - `record`: paced for the camera, with an injected cursor that moves like a hand, targets scrolled into comfortable view and waits cut out. The flow directs the camera (`zoom`, `hotspot: false`, `tooltip`). - `open`: reviews a recording in the editor of a separate browser. - `login`: saves a session once; every run works on a throwaway copy of the profile. - `install-skill`: adds the skill that teaches an agent to direct short, cinematic user guides. valibot moves to the `core` catalog now that two workspaces use it. --- CLAUDE.md | 1 + README.md | 15 + packages/recorder/CLAUDE.md | 26 ++ packages/recorder/LICENSE | 21 + packages/recorder/README.md | 55 +++ packages/recorder/package.json | 53 +++ packages/recorder/rslib.config.ts | 17 + packages/recorder/skill/SKILL.md | 92 +++++ .../recorder/src/__test__/e2e/record.test.ts | 382 ++++++++++++++++++ .../recorder/src/__test__/unit/cli.test.ts | 110 +++++ .../src/__test__/unit/extension-id.test.ts | 17 + .../recorder/src/__test__/unit/flow.test.ts | 150 +++++++ .../recorder/src/__test__/unit/human.test.ts | 82 ++++ .../src/__test__/unit/inspect-format.test.ts | 79 ++++ .../src/__test__/unit/install-skill.test.ts | 35 ++ .../recorder/src/__test__/unit/open.test.ts | 89 ++++ .../src/__test__/unit/run-flow.test.ts | 283 +++++++++++++ packages/recorder/src/bin.ts | 4 + packages/recorder/src/browser.ts | 230 +++++++++++ packages/recorder/src/cli.ts | 206 ++++++++++ packages/recorder/src/cursor.ts | 102 +++++ packages/recorder/src/flow.ts | 167 ++++++++ packages/recorder/src/human.ts | 73 ++++ packages/recorder/src/inspect.ts | 96 +++++ packages/recorder/src/install-skill.ts | 19 + packages/recorder/src/login.ts | 24 ++ packages/recorder/src/open.ts | 65 +++ packages/recorder/src/record.ts | 94 +++++ packages/recorder/src/run-flow.ts | 307 ++++++++++++++ packages/recorder/tsconfig.build.json | 4 + packages/recorder/tsconfig.json | 19 + packages/recorder/vitest.config.ts | 28 ++ packages/schema/package.json | 2 +- pnpm-lock.yaml | 270 ++++++++++++- pnpm-workspace.yaml | 1 + turbo.json | 8 + 36 files changed, 3223 insertions(+), 3 deletions(-) create mode 100644 packages/recorder/CLAUDE.md create mode 100644 packages/recorder/LICENSE create mode 100644 packages/recorder/README.md create mode 100644 packages/recorder/package.json create mode 100644 packages/recorder/rslib.config.ts create mode 100644 packages/recorder/skill/SKILL.md create mode 100644 packages/recorder/src/__test__/e2e/record.test.ts create mode 100644 packages/recorder/src/__test__/unit/cli.test.ts create mode 100644 packages/recorder/src/__test__/unit/extension-id.test.ts create mode 100644 packages/recorder/src/__test__/unit/flow.test.ts create mode 100644 packages/recorder/src/__test__/unit/human.test.ts create mode 100644 packages/recorder/src/__test__/unit/inspect-format.test.ts create mode 100644 packages/recorder/src/__test__/unit/install-skill.test.ts create mode 100644 packages/recorder/src/__test__/unit/open.test.ts create mode 100644 packages/recorder/src/__test__/unit/run-flow.test.ts create mode 100644 packages/recorder/src/bin.ts create mode 100644 packages/recorder/src/browser.ts create mode 100644 packages/recorder/src/cli.ts create mode 100644 packages/recorder/src/cursor.ts create mode 100644 packages/recorder/src/flow.ts create mode 100644 packages/recorder/src/human.ts create mode 100644 packages/recorder/src/inspect.ts create mode 100644 packages/recorder/src/install-skill.ts create mode 100644 packages/recorder/src/login.ts create mode 100644 packages/recorder/src/open.ts create mode 100644 packages/recorder/src/record.ts create mode 100644 packages/recorder/src/run-flow.ts create mode 100644 packages/recorder/tsconfig.build.json create mode 100644 packages/recorder/tsconfig.json create mode 100644 packages/recorder/vitest.config.ts diff --git a/CLAUDE.md b/CLAUDE.md index d9c57e9..37b2321 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,6 +22,7 @@ Each workspace has its own `CLAUDE.md` with its build, test and gotcha notes. Re - **The exported bundle is pure data:** `steps.json` + `assets/*.avif` (photo frames, encoded at export) + `assets/recording.mp4` (only if there are video steps: just the ranges they play, re-encoded at export; `.webm` if H.264 encoding is unavailable). It never ships a player runtime. The player is an npm component in the host app (`player-react` / `player-vue`). - **`packages/schema` is the single source of truth** for the demo format. Read `packages/schema/src/index.ts` for the current shape rather than trusting any copy of it. - Pipeline: `content.ts` markers + continuous tab video → `apps/extension/lib/segments.ts` derives steps (click → photo step with hotspot, activity between clicks → video clip) → editor → export (`editor/lib/export-video.ts` re-encodes the played ranges) → exporter → player. +- **Agent recording** (`packages/recorder`): a CLI plays a JSON flow in headless Chromium with the real extension loaded, which records it through the same pipeline. The recording travels as an `.osrec` file (`apps/extension/lib/recording-file.ts`) that the editor imports. The human always reviews in the editor; the recorder never exports a demo itself. ## Conventions diff --git a/README.md b/README.md index 336c662..a71f254 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,20 @@ powers the interactive demos in the [Rustrak docs](https://rustrak.github.io/rus export +## Let an AI agent record it + +`@rustrak/openshowcase-recorder` lets an agent like Claude Code record the demo +for you. It scripts the flow, checks it against your app, and records it in a +headless Chromium with the extension, without touching your browser. You import +the result in the editor and polish what doesn't convince you. + +```sh +npx @rustrak/openshowcase-recorder install-skill # adds the Claude Code skill +``` + +Then ask: *"record a demo of creating an invoice"*. See the +[recorder's README](./packages/recorder/README.md) to run it yourself. + ## Embed a demo Put the `.openshowcase` file in your site's static files and point the player @@ -77,6 +91,7 @@ apps/extension/ Chrome extension: recorder + editor (WXT + React) packages/schema/ Demo format: Valibot schema + TypeScript types packages/player-core/ Framework-agnostic playback engine packages/exporter/ Builds the exported bundle in the browser +packages/recorder/ CLI + agent skill: records a scripted flow headlessly adapters/player-react/ for React adapters/player-vue/ for Vue ``` diff --git a/packages/recorder/CLAUDE.md b/packages/recorder/CLAUDE.md new file mode 100644 index 0000000..404e1a4 --- /dev/null +++ b/packages/recorder/CLAUDE.md @@ -0,0 +1,26 @@ +# @rustrak/openshowcase-recorder + +Node CLI that lets an AI agent record a demo: it drives a headless Chromium through a JSON flow (`src/flow.ts`) while the **real extension** records it, and writes the result as an `.osrec` file the extension's editor imports. Ships the agent skill (`skill/SKILL.md`, installed with `install-skill`). + +## Commands + +- `pnpm build`: rslib bundles `src/bin.ts` to `dist/cli.js` and copies `skill/` and the built extension (`apps/extension/.output/chrome-mv3`) into `dist/`. Turbo builds the extension first (`turbo.json`). +- `pnpm test`: two Vitest projects in Node. `unit` (some tests launch a plain headless Chromium for `run-flow`), `e2e` (records with the built extension, then imports the file in its editor). e2e needs the extension built; Turbo's `test` task does it. +- Try it: `node dist/cli.js inspect `, `node dist/cli.js record flow.json --profile /tmp/p`. + +## How recording works + +- `browser.ts` `launch()`: `chromium` channel (Chrome-branded builds drop `--load-extension`), `--allowlisted-extension-id=` + `--auto-accept-this-tab-capture` (without them tabCapture fails with "Extension has not been invoked"). The id is computed from the extension's real path (`extensionIdForPath`, tested against a real id). +- No Playwright viewport emulation (`viewport: null`): tabCapture records the real tab, and an emulated viewport made click `yFrac` disagree with the video. `fitViewport` resizes the window over CDP so the page is exactly the flow's size: even headless keeps ~87px for invisible browser UI. `--force-device-scale-factor=2` gives the extension's 2x capture real pixels. +- The CLI calls the extension's automation API (`apps/extension/lib/automation.ts`, `globalThis.openshowcase` in its service worker) through `worker.evaluate`. The `Automation` interface in `browser.ts` mirrors it: keep them in sync (e2e catches a mismatch). The file comes out in 8 MB base64 chunks: one CDP message can't carry a long recording. +- `runFlow` returns a **direction** (wall-clock ms): `clicks` (tooltip/zoom/`hotspot: false`, matched to the extension's click markers by `lib/direction.ts`), `cuts` (every `wait`, `goto`, and app waits ≥200ms: cut from the clips as jump cuts) and `zooms` (fill/select/hover zoom: the clip splits wide → zoomed → wide in the extension's `lib/segments.ts`). The `.osrec` format lives only in the extension (`lib/recording-file.ts`); the e2e test reads its header inline. +- Camera feel: `cursor.ts` injects a cursor (headless draws none; human recordings have Chrome's burned in) built with DOM APIs only (CSP/Trusted Types safe), following the real CDP mouse. `human.ts` gives minimum-jerk Bézier paths; `approach()` in run-flow waits for the target (off camera), smooth-scrolls it to the middle if it's outside 12–85% of the height, then moves there. `fill` clicks into an unfocused field first, as a `hotspot: false` click. +- `open.ts`: the review browser ("studio", `~/.openshowcase/studio`) is a plain detached Chromium, not automated, so downloads and the playground work and it outlives the CLI. The file is copied into the studio's own extension copy (`imports/`) and the editor imports it via `editor.html?import=` (extension's `core/import-source.ts`). Chromium forwards the URL to an already-open studio as a new tab. +- `inspect` is the agent's eyes: text first. No screenshot unless `--screenshot`, link URLs stripped from the tree (`compactSnapshot`), `--within ` narrows it. +- Only `login` writes to the profile (`keepChanges`). `inspect` and `record` run on a temp copy (`scratchCopy`, caches skipped), so re-recording starts from the same browser state. Server-side effects of a flow are real. + +## Gotchas + +- Don't use `new URL("./x", import.meta.url)`: Rspack treats it as an asset import and fails the build. Use `join(dirname(fileURLToPath(import.meta.url)), "x")`. +- `RECORDING_PACE` drives how the demo feels (pauses before clicks show the hover state in the photo step; `fill` types key by key so it becomes a clip). `inspect` uses `INSTANT_PACE`. +- Flow targets map to Playwright locators in `locate()`; prefer role+name in docs and the skill. diff --git a/packages/recorder/LICENSE b/packages/recorder/LICENSE new file mode 100644 index 0000000..a662c43 --- /dev/null +++ b/packages/recorder/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 AbianS + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/packages/recorder/README.md b/packages/recorder/README.md new file mode 100644 index 0000000..c17b250 --- /dev/null +++ b/packages/recorder/README.md @@ -0,0 +1,55 @@ +# @rustrak/openshowcase-recorder + +Lets an AI agent (or a script) record an [OpenShowcase](https://github.com/rustrak/openshowcase) interactive demo. You describe the flow as a JSON file. The recorder plays it in a headless Chromium, paced for the camera, while the OpenShowcase extension records it exactly as it records you. You get an `.osrec` file to review and polish in the editor. + +## Install + +```sh +npm install -D @rustrak/openshowcase-recorder +npx playwright install chromium # once, if you don't have Playwright's Chromium +``` + +## Use it with Claude Code + +```sh +npx openshowcase-recorder install-skill +``` + +This adds the `openshowcase-demo` skill to `.claude/skills/` in your project. Then ask Claude something like *"record a demo of creating an invoice"*. It explores the app, writes the flow, checks it, records it and hands you the `.osrec` file. + +## Use it yourself + +```sh +npx openshowcase-recorder login https://app.example.com # sign in once, close the window +npx openshowcase-recorder inspect https://app.example.com # accessibility tree (--screenshot for an image) +npx openshowcase-recorder inspect demos/invoice.flow.json # run the flow fast, see where it ends +npx openshowcase-recorder record demos/invoice.flow.json --open # → invoice.osrec, opened for review +``` + +`--open` (or `open `) opens the recording in the OpenShowcase editor, in a separate review browser that stays open and keeps your recordings. You can also import the file in your own Chrome: OpenShowcase toolbar button → **Import a recording**. + +Recordings look like a person did them: a visible cursor that moves like a hand, targets scrolled into comfortable view, waits cut out of the video. The flow directs the camera: `"zoom": 1.8` on a click, fill, select or hover, and `"hotspot": false` on clicks that should just play instead of stopping the demo. + +A flow: + +```json +{ + "url": "https://app.example.com/invoices", + "viewport": { "width": 1280, "height": 800 }, + "steps": [ + { "action": "click", "target": { "role": "button", "name": "New invoice" }, "tooltip": "Create your first invoice" }, + { "action": "fill", "target": { "label": "Customer" }, "value": "Acme Corp", "zoom": 1.8 }, + { "action": "click", "target": { "role": "button", "name": "Save" }, "tooltip": "Save it" } + ] +} +``` + +Actions: `click`, `fill`, `press`, `select`, `hover`, `scroll`, `goto`, `wait` (always cut from the video). Targets: `role` (+ `name`), `label`, `placeholder`, `text`, `testId` or `css`, with optional `exact` and `nth`. The [skill](./skill/SKILL.md) documents them all. + +Steps really run against your app. Browser state (localStorage, cookies) is thrown away after every run, so each run starts from what `login` saved, but server-side changes stay. + +Options: `--profile ` (default `~/.openshowcase/profile`), `--extension ` (record with your own extension build), `--headed` (show the window). + +## License + +[MIT](./LICENSE) diff --git a/packages/recorder/package.json b/packages/recorder/package.json new file mode 100644 index 0000000..5d56b95 --- /dev/null +++ b/packages/recorder/package.json @@ -0,0 +1,53 @@ +{ + "name": "@rustrak/openshowcase-recorder", + "version": "0.2.0", + "description": "CLI that records an OpenShowcase demo by driving a headless Chromium through a scripted flow, for AI agents", + "keywords": [ + "openshowcase", + "interactive-demo", + "product-demo", + "recorder", + "playwright", + "ai-agent" + ], + "license": "MIT", + "author": "AbianS", + "homepage": "https://github.com/rustrak/openshowcase/tree/main/packages/recorder#readme", + "repository": { + "type": "git", + "url": "git+https://github.com/rustrak/openshowcase.git", + "directory": "packages/recorder" + }, + "bugs": { + "url": "https://github.com/rustrak/openshowcase/issues" + }, + "type": "module", + "bin": { + "openshowcase-recorder": "./dist/cli.js" + }, + "files": [ + "dist" + ], + "publishConfig": { + "access": "public" + }, + "engines": { + "node": "^22.19.0 || ^24.5.0 || >=26.0.0" + }, + "scripts": { + "build": "rslib build && node -e \"require('node:fs').cpSync('../../apps/extension/.output/chrome-mv3', 'dist/extension', { recursive: true })\"", + "check-types": "tsc --noEmit", + "test": "vitest run", + "test:watch": "vitest" + }, + "dependencies": { + "playwright": "catalog:tooling", + "valibot": "catalog:core" + }, + "devDependencies": { + "@rslib/core": "catalog:tooling", + "@types/node": "24.19.2", + "typescript": "catalog:core", + "vitest": "catalog:tooling" + } +} diff --git a/packages/recorder/rslib.config.ts b/packages/recorder/rslib.config.ts new file mode 100644 index 0000000..5258137 --- /dev/null +++ b/packages/recorder/rslib.config.ts @@ -0,0 +1,17 @@ +import { defineConfig } from "@rslib/core"; + +// A Node CLI: `playwright` and `valibot` stay real imports (autoExternal). The published +// package also carries the built extension (the automated Chromium loads it to record) +// and the agent skill that `install-skill` copies into a project. +export default defineConfig({ + source: { + tsconfigPath: "./tsconfig.build.json", + entry: { cli: "./src/bin.ts" }, + }, + lib: [{ format: "esm", syntax: "es2022" }], + output: { + target: "node", + // The extension is copied by the build script instead: rslib would reformat its JS. + copy: [{ from: "skill", to: "skill" }], + }, +}); diff --git a/packages/recorder/skill/SKILL.md b/packages/recorder/skill/SKILL.md new file mode 100644 index 0000000..ba797ce --- /dev/null +++ b/packages/recorder/skill/SKILL.md @@ -0,0 +1,92 @@ +--- +name: openshowcase-demo +description: Record an interactive product demo of a web app flow with OpenShowcase. Use when the user asks to record, film, capture or make a demo, walkthrough, product tour, user guide or tutorial of a flow in their app (e.g. "make a demo of creating an invoice"). +--- + +# Record an OpenShowcase demo + +You direct a short interactive user guide: you script the flow as a JSON file, check it against the real app, and record it in a headless Chromium with a human-looking cursor. The user reviews the result in the OpenShowcase editor and polishes what doesn't convince them. Recording runs in the background and never touches the user's own browser. + +The tool is a CLI. If `command -v openshowcase-recorder` finds it, run it as `openshowcase-recorder `; otherwise `npx @rustrak/openshowcase-recorder `. Below, `osr` stands for whichever works. If it says Chromium is missing, run `npx playwright install chromium` once. + +## What makes a good demo + +A user guide nobody finishes is worthless. Think like a video editor, not a test script. + +- **Short.** 20–40 seconds of video, 4–8 hotspot steps. If the flow is longer, show the essential path and leave the rest out (or split it into several demos and say so). +- **One idea per hotspot.** A hotspot stops the demo until the viewer clicks it, so it must teach something: a decision, a key field, the button that matters. Its `tooltip` says what the viewer achieves, in their language ("Choose who you're invoicing"), never the mechanics ("Click the combobox"). +- **Routine clicks don't stop the demo.** Opening a dropdown before picking from it, focusing a field, confirming a dialog, moving through a menu, the second and third repetition of something already shown: give those `"hotspot": false`. They play in the video, cursor and all, without interrupting. +- **Repetition is shown once.** Adding three invoice lines? Show the first one properly (hotspots, maybe a zoom), let the others run quickly with `"hotspot": false`, or leave them out entirely. +- **Zoom like a camera operator.** Start wide so the viewer knows where they are. Zoom (`"zoom": 1.5`–`2.2`) into a small area when the detail matters: a field being typed, a cell, a small button, a total that updates. Don't zoom on page changes or large layouts, and don't zoom every step: alternate wide shots and close-ups, and end on the result. +- **No dead time.** Every `wait` is cut out of the video automatically, and so is time spent waiting for a target to appear or a page to load. So wait for the app freely (`{ "action": "wait", "for": }` is best), but never use a wait to "show" something: the step's own pause and the final shot do that. +- **Framing is automatic.** Before using a target near the top or bottom edge, the recorder scrolls it to the middle of the screen with a smooth scroll. Add a `scroll` step only to show something on purpose (e.g. the totals at the end). +- **Realistic, made-up data**, short values (typing takes screen time). + +## Workflow + +1. **Agree on the demo.** Ask what the viewer should learn, the start URL and the account to use, unless the user already said. Prefer a demo or seeded account, and never script destructive actions (delete, pay, send, invite) unless the user confirms it's safe. + +2. **Sign in, if the app needs it.** Run `osr login ` and ask the user to sign in in the window that opens and close it when done. The session is saved in the recording profile and reused by every later command: a one-time step. + +3. **Look at the start page.** `osr inspect --screenshot /tmp/osr-start.png`, and look at that screenshot once, to understand the layout and plan the shots: what's small enough to deserve a zoom, what's far down the page. Everything else you read from the accessibility tree. + +4. **Write the flow step by step.** Create `demos/.flow.json` (format below). After adding a few steps, run `osr inspect demos/.flow.json`. It runs the flow fast, without recording, and prints the page where it ends (or which step failed, with the page at that moment) as an accessibility tree: roles and names, exactly what targets use. + - **Save tokens:** read the tree, not screenshots. Take another screenshot (`--screenshot `) only when the tree can't answer (a canvas, an icon-only UI, a layout question for a zoom). On big apps, narrow the tree to the part you're working in with `--within ` (e.g. `--within main`, `--within "[role=dialog]"`) instead of reading the whole page each time. + - Inspect **really performs** the steps. Browser state is thrown away after every run (each run starts from what `login` saved), but anything the app saves on its server stays. Make the flow work when run again (e.g. unique names), or ask the user to reset the data before recording. + +5. **Edit the flow like a cut.** Before recording, reread it against "What makes a good demo": which clicks deserve a hotspot, where a zoom helps, what's repetitive, whether it's too long. + +6. **Check it end to end.** `osr inspect demos/.flow.json` must report `Steps: all N ran`. + +7. **Record and open it.** `osr record demos/.flow.json --out demos/.osrec --open`. It prints the demo's size: `Demo: N hotspot steps, about Xs of video after cutting Ys of waiting`. If it's over ~40s or has more than ~8 hotspots, tighten the flow (step 5) and record again. `--open` opens the recording in the OpenShowcase editor, in a separate review browser that stays open: tell the user it's there to review, tweak (text, zoom, trims) and export. `osr open ` reopens any recording. + +When the app's UI changes, re-run steps 6 and 7 with the same flow: the flow is the source of truth for the demo. + +## Flow format + +```json +{ + "url": "https://app.example.com/invoices", + "viewport": { "width": 1440, "height": 900 }, + "steps": [ + { "action": "click", "target": { "role": "button", "name": "New invoice" }, "tooltip": "Create an invoice" }, + { "action": "wait", "for": { "role": "combobox", "name": "Customer" } }, + { "action": "click", "target": { "role": "combobox", "name": "Customer" }, "hotspot": false }, + { "action": "fill", "target": { "role": "combobox", "name": "Customer" }, "value": "Acme", "zoom": 1.8 }, + { "action": "click", "target": { "role": "option", "name": "Acme Corp" }, "tooltip": "Pick who you're invoicing" }, + { "action": "click", "target": { "role": "button", "name": "Save" }, "tooltip": "Save it" }, + { "action": "wait", "for": { "text": "Invoice saved" } } + ] +} +``` + +`viewport` is optional (default 1280x800): the size of the demo. + +| action | fields | does | +|---|---|---| +| `click` | `target`, `tooltip`?, `hotspot`?, `zoom`? | Moves the cursor there like a hand and clicks. A photo step with a hotspot showing `tooltip`, unless `"hotspot": false` (then it just plays in the video). `zoom` frames the photo step on it | +| `fill` | `target`, `value`, `zoom`? | Clicks into the field if it isn't focused, clears it and types `value` key by key. `zoom` frames the typing | +| `press` | `key`, `target`? | Presses a key (`Enter`, `Tab`, `Meta+A`…), on the target or the page | +| `select` | `target`, `value`, `zoom`? | Picks a ` + + +`; + +let server: Server; +let url: string; +const work = mkdtempSync(join(tmpdir(), "recorder-e2e-")); + +beforeAll(async () => { + if (!existsSync(join(extensionDir, "manifest.json"))) + throw new Error( + "Build the extension first: pnpm --filter @rustrak/openshowcase-extension build", + ); + server = createServer((_, res) => { + res.setHeader("content-type", "text/html"); + res.end(APP); + }); + await new Promise((resolve) => server.listen(0, resolve)); + const address = server.address(); + url = `http://localhost:${typeof address === "object" && address?.port}/`; +}); + +afterAll(() => { + server?.close(); +}); + +/** The `.osrec` header (lib/recording-file.ts in the extension): magic, version, JSON. */ +function readHeader(path: string) { + const bytes = readFileSync(path); + expect(bytes.subarray(0, 5).toString()).toBe("OSREC"); + const length = bytes.readUInt32LE(6); + return { + header: JSON.parse(bytes.subarray(10, 10 + length).toString("utf8")) as { + recording: { durationSec: number; width: number; height: number }; + markers: { + kind: string; + label?: string; + elementText?: string; + xFrac?: number; + yFrac?: number; + zoom?: number; + hotspot?: boolean; + }[]; + direction?: { cuts: unknown[]; zooms: unknown[] }; + }, + videoBytes: bytes.length - 10 - length, + }; +} + +const flow = () => + parseFlow({ + url, + steps: [ + { + action: "click", + target: { role: "button", name: "New invoice" }, + tooltip: "Create your first invoice", + }, + { action: "fill", target: { label: "Customer" }, value: "Acme" }, + { + action: "click", + target: { role: "button", name: "Save" }, + tooltip: "Save it", + }, + ], + }); + +describe("record", () => { + const profileDir = join(work, "profile"); + const out = join(work, "invoice.osrec"); + + it("records the flow through the extension into an .osrec file", async () => { + const result = await record(flow(), { + profileDir, + extensionDir, + headed: false, + out, + }); + expect(result).toMatchObject({ out, hotspots: 2 }); + + const { header, videoBytes } = readHeader(out); + expect(videoBytes).toBeGreaterThan(10_000); + // Retina: twice the 1280x800 viewport. + expect(header.recording).toMatchObject({ width: 2560, height: 1600 }); + expect(header.recording.durationSec).toBeGreaterThan(3); + + // Hotspots land where the button is drawn in the video: #new's center sits ~90px in + // and ~100px down (under the heading) on the 1280x800 page. A viewport that differs + // from the captured tab would skew these. + const [first] = header.markers; + expect(first?.xFrac).toBeGreaterThan(0.03); + expect(first?.xFrac).toBeLessThan(0.12); + expect(first?.yFrac).toBeGreaterThan(0.08); + expect(first?.yFrac).toBeLessThan(0.16); + + const clicks = header.markers.filter((m) => m.kind === "click"); + // The fill clicks into the field first, like a person: a click with no hotspot. + expect(clicks.map((m) => m.label)).toEqual([ + "Create your first invoice", + undefined, + "Save it", + ]); + expect(clicks.map((m) => m.hotspot)).toEqual([undefined, false, undefined]); + expect(clicks.map((m) => m.elementText)).toEqual([ + "New invoice", + undefined, + "Save", + ]); + expect(header.markers.some((m) => m.kind === "type")).toBe(true); + expect(header.direction).toMatchObject({ cuts: [], zooms: [] }); + }); + + it("opens in the extension's editor as steps with the flow's tooltips", async () => { + const { context, close } = await launch({ + profileDir: join(work, "editor-profile"), + extensionDir, + headless: true, + }); + try { + const page = context.pages()[0] ?? (await context.newPage()); + const id = extensionIdForPath(extensionDir); + await page.goto(`chrome-extension://${id}/editor.html`); + await page.locator('input[type="file"]').setInputFiles(out); + await page.waitForURL(/recordingId=/); + + const steps = page.getByLabel(/^Step \d+\. Drag to reorder$/); + // lead-in clip, click photo, typing clip, click photo, tail clip + await expect.poll(() => steps.count(), { timeout: 15_000 }).toBe(5); + + await steps.nth(1).click(); + await expect + .poll(() => page.getByText("Create your first invoice").count()) + .toBeGreaterThan(0); + } finally { + await close(); + } + }); +}); + +describe("a profile used before", () => { + it("records from a profile a login run already used", async () => { + // A login run leaves the extension's service worker registered in the profile, with + // its script cached in "Service Worker/ScriptCache". The run's copy must keep it. + const profileDir = join(work, "used-profile"); + const { context, close } = await launch({ + profileDir, + extensionDir, + headless: true, + keepChanges: true, + }); + await (context.pages()[0] ?? (await context.newPage())).goto(url); + await close(); + + const result = await record(parseFlow({ url, steps: [] }), { + profileDir, + extensionDir, + headed: false, + out: join(work, "used-profile.osrec"), + }); + expect(result.bytes).toBeGreaterThan(1_000); + }); +}); + +describe("a profile used before, then the extension changes", () => { + it("runs the extension's current code, not what the profile cached", async () => { + const ownExtension = join(work, "changing-extension"); + cpSync(extensionDir, ownExtension, { recursive: true }); + const profileDir = join(work, "stale-profile"); + const login = await launch({ + profileDir, + extensionDir: ownExtension, + headless: true, + keepChanges: true, + }); + await login.close(); + + // A newer build of the extension, same path (so same id), same version. + appendFileSync( + join(ownExtension, "background.js"), + "\nglobalThis.__build = 2;\n", + ); + const run = await launch({ + profileDir, + extensionDir: ownExtension, + headless: true, + }); + try { + expect( + await run.worker.evaluate( + () => (globalThis as { __build?: number }).__build, + ), + ).toBe(2); + } finally { + await run.close(); + } + }); +}); + +describe("direction", () => { + const out = join(work, "directed.osrec"); + + it("carries zooms and quiet clicks into the editor's steps", async () => { + await record( + parseFlow({ + url, + steps: [ + { + action: "click", + target: { role: "button", name: "New invoice" }, + hotspot: false, + }, + { + action: "fill", + target: { label: "Customer" }, + value: "Acme", + zoom: 2, + }, + { + action: "click", + target: { role: "button", name: "Save" }, + tooltip: "Save it", + zoom: 1.6, + }, + { action: "wait", ms: 1500 }, + ], + }), + { + profileDir: join(work, "directed-profile"), + extensionDir, + headed: false, + out, + }, + ); + const { header } = readHeader(out); + expect(header.markers.filter((m) => m.kind === "click")).toEqual([ + expect.objectContaining({ hotspot: false }), + expect.objectContaining({ hotspot: false }), + expect.objectContaining({ label: "Save it", zoom: 1.6 }), + ]); + expect(header.direction?.zooms).toHaveLength(1); + expect(header.direction?.cuts).toHaveLength(1); + }); + + it("opens in the review browser's editor straight from the file", async () => { + let studioUrl = ""; + await openInStudio(out, { + extensionDir, + studioDir: join(work, "studio"), + spawn: () => ({ unref: () => {} }), + }); + const { context, close } = await launch({ + profileDir: join(work, "studio", "profile"), + extensionDir: join(work, "studio", "extension"), + headless: true, + keepChanges: true, + }); + try { + const page = context.pages()[0] ?? (await context.newPage()); + const id = extensionIdForPath( + realpathSync(join(work, "studio", "extension")), + ); + studioUrl = `chrome-extension://${id}/editor.html?import=imports/directed.osrec`; + await page.goto(studioUrl); + await page.waitForURL(/recordingId=/); + const steps = page.getByLabel(/^Step \d+\. Drag to reorder$/); + // One photo (Save). The quiet click and the typing play as clips, split where the + // camera zooms in on the typing and back out (wide, zoomed, wide); the wait is cut. + await expect.poll(() => steps.count(), { timeout: 15_000 }).toBe(5); + await expect + .poll(() => page.getByText("Save it").count()) + .toBeGreaterThan(0); + } finally { + await close(); + } + }); +}); + +describe("inspect", () => { + it("reports the page after the flow, with its accessibility tree", async () => { + const inspection = await inspect(flow(), { + profileDir: join(work, "inspect-profile"), + extensionDir, + headed: false, + screenshot: join(work, "shot.png"), + }); + expect(inspection).toMatchObject({ ran: 3, total: 3, title: "Invoices" }); + expect(inspection.error).toBeUndefined(); + expect(inspection.snapshot).toContain('heading "Saved"'); + expect(existsSync(join(work, "shot.png"))).toBe(true); + }); + + it("leaves the profile as it was: every run starts from the same state", async () => { + const options = { + profileDir: join(work, "state-profile"), + extensionDir, + headed: false, + }; + // The flow saves to localStorage; the next run must not see it. + await inspect(flow(), options); + const next = await inspect(parseFlow({ url, steps: [] }), options); + expect(next.snapshot).toContain('heading "Invoices"'); + }); + + it("starts from what a login run saved to the profile", async () => { + const profileDir = join(work, "login-profile"); + const { context, close } = await launch({ + profileDir, + extensionDir, + headless: true, + keepChanges: true, + }); + const page = context.pages()[0] ?? (await context.newPage()); + await page.goto(url); + await page.evaluate(() => localStorage.setItem("saved", "1")); + await close(); + + const next = await inspect(parseFlow({ url, steps: [] }), { + profileDir, + extensionDir, + headed: false, + }); + expect(next.snapshot).toContain('heading "Saved before"'); + }); + + it("stops at a failing step and shows the page as it was", async () => { + const inspection = await inspect( + parseFlow({ + url, + steps: [ + { action: "click", target: { role: "button", name: "Delete" } }, + ], + }), + { + profileDir: join(work, "inspect-profile"), + extensionDir, + headed: false, + }, + ); + expect(inspection.ran).toBe(0); + expect(inspection.error).toMatch( + /^Step 1 \(click role=button name="Delete"\) failed/, + ); + expect(inspection.snapshot).toContain('button "New invoice"'); + }); +}); diff --git a/packages/recorder/src/__test__/unit/cli.test.ts b/packages/recorder/src/__test__/unit/cli.test.ts new file mode 100644 index 0000000..3659f7e --- /dev/null +++ b/packages/recorder/src/__test__/unit/cli.test.ts @@ -0,0 +1,110 @@ +import { homedir } from "node:os"; +import { join, resolve } from "node:path"; +import { describe, expect, it } from "vitest"; +import { BUNDLED_EXTENSION_DIR, DEFAULT_PROFILE_DIR } from "../../browser"; +import { parseCommand, UsageError } from "../../cli"; + +describe("parseCommand", () => { + it("records a flow to a file named after it by default", () => { + expect(parseCommand(["record", "flows/invoice.json"])).toEqual({ + command: "record", + flow: "flows/invoice.json", + out: resolve("invoice.osrec"), + profileDir: DEFAULT_PROFILE_DIR, + extensionDir: BUNDLED_EXTENSION_DIR, + headed: false, + open: false, + }); + }); + + it("names the file after the flow, without a .flow suffix", () => { + expect(parseCommand(["record", "demos/invoice.flow.json"])).toMatchObject({ + out: resolve("invoice.osrec"), + }); + }); + + it("opens the recording in the review browser right after, if asked", () => { + expect(parseCommand(["record", "f.json", "--open"])).toMatchObject({ + open: true, + }); + }); + + it("opens a recording in the review browser", () => { + expect(parseCommand(["open", "demos/x.osrec"])).toEqual({ + command: "open", + file: resolve("demos/x.osrec"), + extensionDir: BUNDLED_EXTENSION_DIR, + }); + expect(() => parseCommand(["open"])).toThrow(/recording/); + }); + + it("takes an output file, a profile, an extension build and a visible window", () => { + expect( + parseCommand([ + "record", + "flow.json", + "--out", + "demos/x.osrec", + "--profile", + "~/p", + "--extension", + "ext", + "--headed", + ]), + ).toMatchObject({ + out: resolve("demos/x.osrec"), + profileDir: join(homedir(), "p"), + extensionDir: resolve("ext"), + headed: true, + }); + }); + + it("inspects a flow or a URL, with a screenshot path", () => { + expect( + parseCommand(["inspect", "https://example.com", "--screenshot", "a.png"]), + ).toMatchObject({ + command: "inspect", + flow: "https://example.com", + screenshot: resolve("a.png"), + }); + }); + + it("takes no screenshot unless asked, and can narrow the tree to a part of the page", () => { + const command = parseCommand(["inspect", "flow.json", "--within", "main"]); + expect(command).toMatchObject({ within: "main" }); + expect(command).not.toHaveProperty("screenshot"); + }); + + it("logs in at a URL", () => { + expect(parseCommand(["login", "https://example.com"])).toMatchObject({ + command: "login", + url: "https://example.com", + }); + }); + + it("installs the skill in the current project or a given one", () => { + expect(parseCommand(["install-skill"])).toEqual({ + command: "install-skill", + projectDir: process.cwd(), + }); + expect(parseCommand(["install-skill", "--dir", "../app"])).toEqual({ + command: "install-skill", + projectDir: resolve("../app"), + }); + }); + + it("shows help with no command or --help", () => { + expect(parseCommand([])).toEqual({ command: "help" }); + expect(parseCommand(["--help"])).toEqual({ command: "help" }); + expect(parseCommand(["record", "--help"])).toEqual({ command: "help" }); + }); + + it("rejects an unknown command, a missing argument and an unknown option", () => { + expect(() => parseCommand(["film"])).toThrow(UsageError); + expect(() => parseCommand(["record"])).toThrow(/flow file/); + expect(() => parseCommand(["login"])).toThrow(/URL/); + expect(() => parseCommand(["record", "f.json", "--nope"])).toThrow( + UsageError, + ); + }); +}); diff --git a/packages/recorder/src/__test__/unit/extension-id.test.ts b/packages/recorder/src/__test__/unit/extension-id.test.ts new file mode 100644 index 0000000..b92c75b --- /dev/null +++ b/packages/recorder/src/__test__/unit/extension-id.test.ts @@ -0,0 +1,17 @@ +import { describe, expect, it } from "vitest"; +import { extensionIdForPath } from "../../browser"; + +describe("extensionIdForPath", () => { + it("matches the id Chromium gives an unpacked extension at that path", () => { + // Read from a real Chromium launch with --load-extension at this path. + expect( + extensionIdForPath( + "/Users/abiansuarezbrito/Documents/openshowcase/apps/extension/.output/chrome-mv3", + ), + ).toBe("hgcmjhpgnammaphhnaomjpapgjbmeopf"); + }); + + it("is 32 letters from a to p", () => { + expect(extensionIdForPath("/any/where")).toMatch(/^[a-p]{32}$/); + }); +}); diff --git a/packages/recorder/src/__test__/unit/flow.test.ts b/packages/recorder/src/__test__/unit/flow.test.ts new file mode 100644 index 0000000..8515872 --- /dev/null +++ b/packages/recorder/src/__test__/unit/flow.test.ts @@ -0,0 +1,150 @@ +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { describeStep, FlowError, loadFlow, parseFlow } from "../../flow"; + +describe("parseFlow", () => { + it("accepts a flow with every kind of step and target", () => { + const flow = parseFlow({ + url: "https://app.example.com", + viewport: { width: 1440, height: 900 }, + steps: [ + { + action: "click", + target: { role: "button", name: "New invoice" }, + tooltip: "Create your first invoice", + }, + { action: "fill", target: { label: "Customer" }, value: "Acme" }, + { action: "press", key: "Enter" }, + { action: "press", key: "Tab", target: { placeholder: "Search" } }, + { action: "select", target: { testId: "currency" }, value: "EUR" }, + { action: "hover", target: { text: "Pricing", exact: true } }, + { action: "scroll", by: 400 }, + { action: "scroll", target: { css: "#footer" } }, + { action: "goto", url: "https://app.example.com/settings" }, + { action: "wait", ms: 500 }, + { action: "wait", for: { role: "dialog", nth: 0 } }, + ], + }); + expect(flow.steps).toHaveLength(11); + expect(flow.viewport).toEqual({ width: 1440, height: 900 }); + }); + + it("takes camera direction: zoom on actions, clicks without a hotspot", () => { + const flow = parseFlow({ + url: "https://example.com", + steps: [ + { action: "click", target: { text: "Menu" }, hotspot: false }, + { action: "click", target: { text: "Total" }, zoom: 2, tooltip: "T" }, + { action: "fill", target: { label: "Qty" }, value: "2", zoom: 1.6 }, + { action: "select", target: { label: "VAT" }, value: "21", zoom: 1.5 }, + { action: "hover", target: { text: "Help" }, zoom: 1.4 }, + ], + }); + expect(flow.steps[0]).toMatchObject({ hotspot: false }); + expect(flow.steps[1]).toMatchObject({ zoom: 2 }); + }); + + it("keeps zoom between 1.2 and 3", () => { + for (const zoom of [1, 3.5]) { + expect(() => + parseFlow({ + url: "https://example.com", + steps: [{ action: "click", target: { text: "A" }, zoom }], + }), + ).toThrow(/steps\.0\.zoom/); + } + }); + + it("defaults the viewport to 1280x800", () => { + const flow = parseFlow({ url: "https://example.com", steps: [] }); + expect(flow.viewport).toEqual({ width: 1280, height: 800 }); + }); + + it("reports every problem with its path", () => { + expect(() => + parseFlow({ + url: "not a url", + steps: [{ action: "click" }, { action: "teleport" }], + }), + ).toThrow(FlowError); + try { + parseFlow({ url: "not a url", steps: [{ action: "click" }] }); + } catch (error) { + const message = (error as FlowError).message; + expect(message).toMatch(/url/); + expect(message).toMatch(/steps\.0\.target/); + } + }); + + it("explains what a target needs", () => { + expect(() => + parseFlow({ + url: "https://example.com", + steps: [{ action: "click", target: { name: "Save" } }], + }), + ).toThrow(/role, text, label, placeholder, testId or css/); + }); + + it("rejects a wait with neither ms nor for", () => { + expect(() => + parseFlow({ + url: "https://example.com", + steps: [{ action: "wait" }], + }), + ).toThrow(/ms or for/); + }); + + it("rejects an unknown action", () => { + expect(() => + parseFlow({ + url: "https://example.com", + steps: [{ action: "teleport" }], + }), + ).toThrow(/steps\.0\.action/); + }); +}); + +describe("loadFlow", () => { + it("reads a flow file", async () => { + const dir = mkdtempSync(join(tmpdir(), "flow-")); + const path = join(dir, "demo.json"); + writeFileSync( + path, + JSON.stringify({ url: "https://example.com", steps: [] }), + ); + expect((await loadFlow(path)).url).toBe("https://example.com"); + }); + + it("turns a bare URL into a flow with no steps", async () => { + expect(await loadFlow("https://example.com/app")).toEqual({ + url: "https://example.com/app", + viewport: { width: 1280, height: 800 }, + steps: [], + }); + }); + + it("says which file isn't valid JSON", async () => { + const dir = mkdtempSync(join(tmpdir(), "flow-")); + const path = join(dir, "broken.json"); + writeFileSync(path, "{ nope"); + await expect(loadFlow(path)).rejects.toThrow(/broken\.json.*JSON/); + }); +}); + +describe("describeStep", () => { + it("names the action and its target briefly", () => { + expect( + describeStep({ + action: "click", + target: { role: "button", name: "Save" }, + }), + ).toBe('click role=button name="Save"'); + expect(describeStep({ action: "press", key: "Enter" })).toBe("press Enter"); + expect(describeStep({ action: "wait", ms: 300 })).toBe("wait 300ms"); + expect(describeStep({ action: "goto", url: "https://example.com/x" })).toBe( + "goto https://example.com/x", + ); + }); +}); diff --git a/packages/recorder/src/__test__/unit/human.test.ts b/packages/recorder/src/__test__/unit/human.test.ts new file mode 100644 index 0000000..bf84fe9 --- /dev/null +++ b/packages/recorder/src/__test__/unit/human.test.ts @@ -0,0 +1,82 @@ +import { describe, expect, it } from "vitest"; +import { aimPoint, humanPath, MOVE_TICK_MS } from "../../human"; + +const seq = + (...values: number[]) => + () => + values.shift() ?? 0.5; + +/** Distance of p from the straight line a→b. */ +function offLine( + p: { x: number; y: number }, + a: { x: number; y: number }, + b: { x: number; y: number }, +) { + const dx = b.x - a.x; + const dy = b.y - a.y; + return ( + Math.abs(dy * p.x - dx * p.y + b.x * a.y - b.y * a.x) / Math.hypot(dx, dy) + ); +} + +describe("humanPath", () => { + const from = { x: 100, y: 100 }; + const to = { x: 700, y: 400 }; + + it("ends exactly on the target", () => { + const path = humanPath(from, to, seq(0.9)); + expect(path.at(-1)).toEqual(to); + }); + + it("takes longer for longer moves, within a human range", () => { + const short = humanPath(from, { x: 160, y: 100 }, seq(0.5)); + const long = humanPath(from, to, seq(0.5)); + expect(long.length).toBeGreaterThan(short.length); + for (const path of [short, long]) { + const ms = path.length * MOVE_TICK_MS; + expect(ms).toBeGreaterThanOrEqual(250); + expect(ms).toBeLessThanOrEqual(950); + } + }); + + it("arcs a little instead of moving in a straight line", () => { + const path = humanPath(from, to, seq(0.95)); + const maxOff = Math.max(...path.map((p) => offLine(p, from, to))); + const dist = Math.hypot(to.x - from.x, to.y - from.y); + expect(maxOff).toBeGreaterThan(5); + expect(maxOff).toBeLessThan(dist * 0.2); + }); + + it("accelerates, then slows down onto the target", () => { + const path = [from, ...humanPath(from, to, seq(0.5))]; + const steps = path + .slice(1) + .map((p, i) => + Math.hypot(p.x - (path[i]?.x ?? 0), p.y - (path[i]?.y ?? 0)), + ); + const middle = steps[Math.floor(steps.length / 2)] ?? 0; + expect(steps[0]).toBeLessThan(middle / 2); + expect(steps.at(-1)).toBeLessThan(middle / 2); + }); + + it("doesn't move when already there", () => { + expect(humanPath(to, { x: 701, y: 400 }, seq(0.5))).toEqual([ + { x: 701, y: 400 }, + ]); + }); +}); + +describe("aimPoint", () => { + it("aims near the middle of the element, not always dead center", () => { + const box = { x: 100, y: 200, width: 200, height: 40 }; + const a = aimPoint(box, seq(0, 0)); + const b = aimPoint(box, seq(1, 1)); + for (const p of [a, b]) { + expect(p.x).toBeGreaterThanOrEqual(100 + 200 * 0.3); + expect(p.x).toBeLessThanOrEqual(100 + 200 * 0.7); + expect(p.y).toBeGreaterThanOrEqual(200 + 40 * 0.3); + expect(p.y).toBeLessThanOrEqual(200 + 40 * 0.7); + } + expect(a).not.toEqual(b); + }); +}); diff --git a/packages/recorder/src/__test__/unit/inspect-format.test.ts b/packages/recorder/src/__test__/unit/inspect-format.test.ts new file mode 100644 index 0000000..0f4cb01 --- /dev/null +++ b/packages/recorder/src/__test__/unit/inspect-format.test.ts @@ -0,0 +1,79 @@ +import { describe, expect, it } from "vitest"; +import { compactSnapshot, formatInspection } from "../../inspect"; + +describe("formatInspection", () => { + it("lists where the flow ended up, then the page's accessibility tree", () => { + expect( + formatInspection({ + url: "https://example.com/invoices", + title: "Invoices", + screenshot: "/tmp/shot.png", + ran: 2, + total: 2, + snapshot: '- heading "Invoices" [level=1]', + }), + ).toBe( + [ + "URL: https://example.com/invoices", + "Title: Invoices", + "Screenshot: /tmp/shot.png", + "Steps: all 2 ran", + "", + "Accessibility tree:", + '- heading "Invoices" [level=1]', + ].join("\n"), + ); + }); + + it("says a bare URL ran no steps", () => { + const text = formatInspection({ + url: "https://example.com", + title: "", + screenshot: "/tmp/shot.png", + ran: 0, + total: 0, + snapshot: "", + }); + expect(text).toContain("Steps: none (just the start page)"); + }); + + it("leaves the screenshot line out when none was taken", () => { + const text = formatInspection({ + url: "https://example.com", + title: "", + ran: 0, + total: 0, + snapshot: "", + }); + expect(text).not.toContain("Screenshot"); + }); + + it("puts a failed step up front", () => { + const text = formatInspection({ + url: "https://example.com", + title: "", + screenshot: "/tmp/shot.png", + ran: 1, + total: 3, + error: 'Step 2 (click role=button name="Save") failed: Timeout', + snapshot: '- button "Cancel"', + }); + expect(text).toContain( + 'Steps: 1 of 3 ran. Step 2 (click role=button name="Save") failed: Timeout', + ); + }); +}); + +describe("compactSnapshot", () => { + it("drops link URLs: targets use names, and they cost tokens", () => { + expect( + compactSnapshot( + [ + '- link "Invoices":', + " - /url: /sale/invoices", + '- button "New"', + ].join("\n"), + ), + ).toBe(['- link "Invoices"', '- button "New"'].join("\n")); + }); +}); diff --git a/packages/recorder/src/__test__/unit/install-skill.test.ts b/packages/recorder/src/__test__/unit/install-skill.test.ts new file mode 100644 index 0000000..3c02b87 --- /dev/null +++ b/packages/recorder/src/__test__/unit/install-skill.test.ts @@ -0,0 +1,35 @@ +import { existsSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { installSkill } from "../../install-skill"; + +const skillDir = join(import.meta.dirname, "../../../skill"); + +describe("installSkill", () => { + it("copies the skill into the project's .claude/skills", async () => { + const project = mkdtempSync(join(tmpdir(), "project-")); + const installed = await installSkill(project, skillDir); + + expect(installed).toBe( + join(project, ".claude", "skills", "openshowcase-demo", "SKILL.md"), + ); + expect(readFileSync(installed, "utf8")).toBe( + readFileSync(join(skillDir, "SKILL.md"), "utf8"), + ); + }); + + it("updates a skill installed before", async () => { + const project = mkdtempSync(join(tmpdir(), "project-")); + const installed = await installSkill(project, skillDir); + writeFileSync(installed, "old"); + await installSkill(project, skillDir); + expect(readFileSync(installed, "utf8")).not.toBe("old"); + }); + + it("ships a skill Claude Code can load (name + description frontmatter)", () => { + const skill = readFileSync(join(skillDir, "SKILL.md"), "utf8"); + expect(existsSync(join(skillDir, "SKILL.md"))).toBe(true); + expect(skill).toMatch(/^---\nname: openshowcase-demo\ndescription: .+\n/); + }); +}); diff --git a/packages/recorder/src/__test__/unit/open.test.ts b/packages/recorder/src/__test__/unit/open.test.ts new file mode 100644 index 0000000..d46699f --- /dev/null +++ b/packages/recorder/src/__test__/unit/open.test.ts @@ -0,0 +1,89 @@ +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { extensionIdForPath } from "../../browser"; +import { openInStudio } from "../../open"; + +function fixture() { + const dir = mkdtempSync(join(tmpdir(), "studio-test-")); + const extensionDir = join(dir, "built-extension"); + mkdirSync(extensionDir); + writeFileSync(join(extensionDir, "manifest.json"), "{}"); + const file = join(dir, "Factura de venta.osrec"); + writeFileSync(file, "OSREC…"); + return { dir, extensionDir, file, studioDir: join(dir, "studio") }; +} + +describe("openInStudio", () => { + it("opens the recording in the editor of a separate, lasting browser", async () => { + const { extensionDir, file, studioDir } = fixture(); + const calls: { command: string; args: string[]; detached?: boolean }[] = []; + const url = await openInStudio(file, { + extensionDir, + studioDir, + executablePath: "/bin/chromium", + spawn: (command, args, options) => { + calls.push({ command, args, detached: options.detached }); + return { unref: () => {} }; + }, + }); + + const studioExtension = join(studioDir, "extension"); + // The extension gets its own copy, with the recording inside it. + expect(existsSync(join(studioExtension, "manifest.json"))).toBe(true); + expect( + readFileSync( + join(studioExtension, "imports", "Factura-de-venta.osrec"), + "utf8", + ), + ).toBe("OSREC…"); + + const id = extensionIdForPath(realpathSync(studioExtension)); + expect(url).toBe( + `chrome-extension://${id}/editor.html?import=imports/Factura-de-venta.osrec`, + ); + expect(calls).toHaveLength(1); + expect(calls[0]?.command).toBe("/bin/chromium"); + expect(calls[0]?.detached).toBe(true); + expect(calls[0]?.args).toContain( + `--user-data-dir=${join(studioDir, "profile")}`, + ); + expect(calls[0]?.args).toContain( + `--load-extension=${realpathSync(studioExtension)}`, + ); + expect(calls[0]?.args.at(-1)).toBe(url); + }); + + it("only keeps the latest import", async () => { + const { extensionDir, file, studioDir, dir } = fixture(); + const spawn = () => ({ unref: () => {} }); + const other = join(dir, "other.osrec"); + writeFileSync(other, "x"); + await openInStudio(other, { extensionDir, studioDir, spawn }); + await openInStudio(file, { extensionDir, studioDir, spawn }); + expect( + existsSync(join(studioDir, "extension", "imports", "other.osrec")), + ).toBe(false); + }); + + it("refuses a file that isn't a recording", async () => { + const { extensionDir, studioDir, dir } = fixture(); + const notes = join(dir, "notes.txt"); + writeFileSync(notes, "x"); + await expect( + openInStudio(notes, { + extensionDir, + studioDir, + spawn: () => ({ unref: () => {} }), + }), + ).rejects.toThrow(/\.osrec/); + }); +}); diff --git a/packages/recorder/src/__test__/unit/run-flow.test.ts b/packages/recorder/src/__test__/unit/run-flow.test.ts new file mode 100644 index 0000000..e51d9d9 --- /dev/null +++ b/packages/recorder/src/__test__/unit/run-flow.test.ts @@ -0,0 +1,283 @@ +import { type Browser, chromium, type Page } from "playwright"; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { installCursor } from "../../cursor"; +import type { Step } from "../../flow"; +import { + INSTANT_PACE, + type Pace, + RECORDING_PACE, + runFlow, + StepError, +} from "../../run-flow"; + +const PAGE = ` + + +

Invoices

+ + + + + +Second page +
+
+
+ +`; + +let browser: Browser; +let page: Page; + +beforeAll(async () => { + browser = await chromium.launch({ channel: "chromium", headless: true }); +}); + +afterAll(async () => { + await browser.close(); +}); + +beforeEach(async () => { + page = await browser.newPage({ viewport: { width: 800, height: 600 } }); + page.setDefaultTimeout(2_000); + await page.setContent(PAGE); +}); + +const run = (steps: Step[]) => runFlow(page, steps, INSTANT_PACE); + +describe("runFlow", () => { + it("clicks a target found by role and name", async () => { + await run([ + { action: "click", target: { role: "button", name: "New invoice" } }, + ]); + await expect( + page.getByRole("button", { name: "Clicked" }).count(), + ).resolves.toBe(1); + }); + + it("returns the window and direction of each click that asks for some", async () => { + const before = Date.now(); + const { clicks } = await run([ + { + action: "click", + target: { role: "button", name: "New invoice" }, + tooltip: "Start a new invoice", + zoom: 2, + }, + { action: "click", target: { text: "Hover me" } }, + { action: "click", target: { text: "Hover me" }, hotspot: false }, + ]); + expect(clicks).toHaveLength(2); + const [first, quiet] = clicks; + expect(first).toMatchObject({ label: "Start a new invoice", zoom: 2 }); + expect(first?.start).toBeGreaterThanOrEqual(before); + expect(first?.end).toBeGreaterThanOrEqual(first?.start ?? Infinity); + expect(quiet).toMatchObject({ hotspot: false }); + expect(quiet?.end).toBeLessThanOrEqual(Date.now()); + }); + + it("clicks into a field before typing, as a click with no hotspot", async () => { + const { clicks } = await run([ + { action: "fill", target: { label: "Customer" }, value: "Acme" }, + ]); + expect(clicks).toEqual([expect.objectContaining({ hotspot: false })]); + }); + + it("doesn't click a field again when it already has focus", async () => { + await page.focus("#customer"); + const { clicks } = await run([ + { action: "fill", target: { label: "Customer" }, value: "Acme" }, + ]); + expect(clicks).toEqual([]); + }); + + it("frames the clip on a field it zooms into while typing", async () => { + const { zooms } = await run([ + { + action: "fill", + target: { label: "Customer" }, + value: "Acme", + zoom: 1.8, + }, + ]); + const box = await page.locator("#customer").boundingBox(); + expect(zooms).toHaveLength(1); + const [zoom] = zooms; + expect(zoom?.scale).toBe(1.8); + expect(zoom?.x).toBeCloseTo( + ((box?.x ?? 0) + (box?.width ?? 0) / 2) / 800, + 2, + ); + expect(zoom?.y).toBeCloseTo( + ((box?.y ?? 0) + (box?.height ?? 0) / 2) / 600, + 2, + ); + expect(zoom?.end).toBeGreaterThan(zoom?.start ?? Infinity); + }); + + it("cuts waits out of the video", async () => { + const { cuts } = await run([{ action: "wait", ms: 120 }]); + expect(cuts).toHaveLength(1); + const [cut] = cuts; + expect((cut?.end ?? 0) - (cut?.start ?? 0)).toBeGreaterThanOrEqual(110); + }); + + it("cuts the time spent waiting for a target to show up", async () => { + const { cuts } = await run([{ action: "click", target: { text: "Late" } }]); + expect(cuts).toHaveLength(1); + expect((cuts[0]?.end ?? 0) - (cuts[0]?.start ?? 0)).toBeGreaterThan(300); + }); + + it("brings a target near the edge to the middle of the screen before using it", async () => { + await run([{ action: "click", target: { text: "Far away" } }]); + const box = await page.getByRole("button", { name: "Done" }).boundingBox(); + expect(box?.y).toBeGreaterThan(600 * 0.12); + expect((box?.y ?? 0) + (box?.height ?? 0)).toBeLessThan(600 * 0.85); + }); + + it("moves the mouse like a hand on camera, and the cursor follows it", async () => { + const context = await browser.newContext({ + viewport: { width: 800, height: 600 }, + }); + await installCursor(context); + // A real navigation (setContent's document.write drops the init script's listeners) + await context.route("http://demo.test/", (route) => + route.fulfill({ contentType: "text/html", body: PAGE }), + ); + const camera = await context.newPage(); + camera.setDefaultTimeout(2_000); + await camera.goto("http://demo.test/"); + await camera.evaluate(() => { + (window as unknown as { moves: number }).moves = 0; + document.addEventListener("mousemove", () => { + (window as unknown as { moves: number }).moves++; + }); + }); + const quick: Pace = { ...RECORDING_PACE, leadIn: 0, afterStep: 0, tail: 0 }; + await runFlow( + camera, + [{ action: "click", target: { role: "button", name: "New invoice" } }], + quick, + ); + + expect( + await camera.evaluate( + () => (window as unknown as { moves: number }).moves, + ), + ).toBeGreaterThan(10); + const box = await camera + .getByRole("button", { name: "Clicked" }) + .boundingBox(); + const [x, y] = await camera.evaluate(() => { + const host = document.querySelector("openshowcase-cursor") as HTMLElement; + return host.style.transform.match(/-?[\d.]+/g)?.map(Number) ?? []; + }); + expect(x).toBeGreaterThan(box?.x ?? Infinity); + expect(x).toBeLessThan((box?.x ?? 0) + (box?.width ?? 0)); + expect(y).toBeGreaterThan(box?.y ?? Infinity); + expect(y).toBeLessThan((box?.y ?? 0) + (box?.height ?? 0)); + await context.close(); + }); + + it("types a value key by key into a field found by its label", async () => { + await run([ + { action: "fill", target: { label: "Customer" }, value: "Acme" }, + ]); + expect(await page.inputValue("#customer")).toBe("Acme"); + const keys = await page.evaluate( + () => (window as unknown as { keys: string[] }).keys, + ); + expect(keys.filter((k) => k.length === 1).join("")).toBe("Acme"); + }); + + it("replaces what a field already had", async () => { + await page.fill("#customer", "old"); + await run([ + { action: "fill", target: { label: "Customer" }, value: "new" }, + ]); + expect(await page.inputValue("#customer")).toBe("new"); + }); + + it("presses keys, on a target or on the page", async () => { + await run([ + { action: "fill", target: { placeholder: "Search" }, value: "ab" }, + { action: "press", key: "Backspace", target: { placeholder: "Search" } }, + { action: "press", key: "Backspace" }, + ]); + expect(await page.inputValue("#search")).toBe(""); + }); + + it("selects an option, hovers, and waits for an element to appear", async () => { + await run([ + { action: "select", target: { testId: "currency" }, value: "EUR" }, + { action: "hover", target: { role: "button", name: "Hover me" } }, + { action: "wait", for: { role: "dialog" } }, + { action: "wait", ms: 10 }, + ]); + expect(await page.inputValue("select")).toBe("EUR"); + expect( + await page.getAttribute("button:nth-of-type(2)", "data-hovered"), + ).toBe("yes"); + }); + + it("scrolls by pixels and to a target", async () => { + await run([{ action: "scroll", by: 500 }]); + await expect.poll(() => page.evaluate(() => window.scrollY)).toBe(500); + await run([{ action: "scroll", target: { css: "footer" } }]); + await expect + .poll(() => page.evaluate(() => window.scrollY)) + .toBeGreaterThan(2000); + }); + + it("navigates", async () => { + await run([{ action: "goto", url: "data:text/html,

Other

" }]); + await expect(page.getByRole("heading").textContent()).resolves.toBe( + "Other", + ); + }); + + it("picks the nth match of an ambiguous target", async () => { + await run([{ action: "click", target: { role: "button", nth: 0 } }]); + await expect( + page.getByRole("button", { name: "Clicked" }).count(), + ).resolves.toBe(1); + }); + + it("says which step failed and why", async () => { + const error = await run([ + { action: "click", target: { role: "button", name: "New invoice" } }, + { action: "click", target: { role: "button", name: "Delete" } }, + ]).catch((e: unknown) => e); + expect(error).toBeInstanceOf(StepError); + expect((error as StepError).index).toBe(1); + expect((error as StepError).message).toMatch( + /^Step 2 \(click role=button name="Delete"\) failed: /, + ); + }); + + it("reports progress before each step", async () => { + const seen: number[] = []; + await runFlow( + page, + [ + { action: "wait", ms: 1 }, + { action: "wait", ms: 1 }, + ], + INSTANT_PACE, + (index) => seen.push(index), + ); + expect(seen).toEqual([0, 1]); + }); +}); diff --git a/packages/recorder/src/bin.ts b/packages/recorder/src/bin.ts new file mode 100644 index 0000000..eed3596 --- /dev/null +++ b/packages/recorder/src/bin.ts @@ -0,0 +1,4 @@ +#!/usr/bin/env node +import { run } from "./cli"; + +process.exitCode = await run(process.argv.slice(2)); diff --git a/packages/recorder/src/browser.ts b/packages/recorder/src/browser.ts new file mode 100644 index 0000000..71008c9 --- /dev/null +++ b/packages/recorder/src/browser.ts @@ -0,0 +1,230 @@ +import { createHash } from "node:crypto"; +import { existsSync, realpathSync } from "node:fs"; +import { cp, mkdtemp, rm } from "node:fs/promises"; +import { homedir, tmpdir } from "node:os"; +import { basename, dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { type BrowserContext, chromium, type Worker } from "playwright"; + +interface TimeRange { + start: number; + end: number; +} + +/** + * What the flow asks of the recording, in wall-clock ms (the clock of the extension's + * markers). Mirrors `Direction` in apps/extension/lib/direction.ts: keep in sync. + */ +export interface Direction { + /** Clicks with a tooltip, zoom or hotspot choice, matched to click markers by time. */ + clicks: (TimeRange & { label?: string; zoom?: number; hotspot?: boolean })[]; + /** Waits for the app, cut out of the video. */ + cuts: TimeRange[]; + /** Actions to frame the clip on (centers as fractions of the page). */ + zooms: (TimeRange & { x: number; y: number; scale: number })[]; +} + +/** The extension's automation API (apps/extension/lib/automation.ts). Keep in sync. */ +interface Automation { + start(): Promise; + stop(recordingId: string): Promise; + exportFile(recordingId: string, direction: Direction): Promise; + readChunk(offset: number, length: number): Promise; +} + +export const DEFAULT_PROFILE_DIR = join(homedir(), ".openshowcase", "profile"); + +/** The extension the published package ships next to this file (see rslib.config.ts). */ +export const BUNDLED_EXTENSION_DIR = join( + dirname(fileURLToPath(import.meta.url)), + "extension", +); + +/** Chromium's id for an unpacked extension: its path's SHA-256, first 32 hex digits as a–p. */ +export function extensionIdForPath(path: string): string { + const hex = createHash("sha256").update(path).digest("hex").slice(0, 32); + return [...hex] + .map((digit) => String.fromCharCode(97 + Number.parseInt(digit, 16))) + .join(""); +} + +export interface LaunchOptions { + profileDir: string; + extensionDir: string; + headless: boolean; + viewport?: { width: number; height: number }; + /** Write to the profile itself (login). Otherwise the run works on a throwaway copy. */ + keepChanges?: boolean; +} + +export interface Launched { + context: BrowserContext; + /** The extension's service worker, where `globalThis.openshowcase` lives. */ + worker: Worker; + /** Closes the browser, then deletes the profile copy it ran on. */ + close: () => Promise; +} + +/** + * A persistent Chromium (the profile keeps the user's logins) with the extension loaded. + * The two allowlist flags let the extension capture a tab without the toolbar click + * tabCapture otherwise requires. `chromium` channel: Chrome-branded builds drop + * --load-extension, and new headless runs extensions. + */ +export async function launch(options: LaunchOptions): Promise { + const extensionDir = realpathSync(options.extensionDir); + const id = extensionIdForPath(extensionDir); + const { width, height } = options.viewport ?? { width: 1280, height: 800 }; + const userDataDir = options.keepChanges + ? options.profileDir + : await scratchCopy(options.profileDir); + const context = await chromium.launchPersistentContext(userDataDir, { + channel: "chromium", + headless: options.headless, + // The real window size, not Playwright's viewport emulation: tabCapture records the + // real tab, so an emulated viewport would make click positions disagree with the video. + viewport: null, + args: [ + `--window-size=${width},${height}`, + // Retina-sharp capture: the extension records at 2x the tab's CSS size. + "--force-device-scale-factor=2", + `--disable-extensions-except=${extensionDir}`, + `--load-extension=${extensionDir}`, + `--allowlisted-extension-id=${id}`, + "--auto-accept-this-tab-capture", + // A headed window behind others must keep painting, or the capture stalls. + "--disable-backgrounding-occluded-windows", + "--disable-renderer-backgrounding", + "--disable-background-timer-throttling", + ], + }); + const close = async () => { + await context.close(); + if (userDataDir !== options.profileDir) + // Chromium may still be flushing files as it exits: retry a busy/non-empty dir. + await rm(userDataDir, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 100, + }); + }; + + const isOurs = (w: Worker) => w.url().startsWith(`chrome-extension://${id}/`); + let worker: Worker; + try { + worker = + context.serviceWorkers().find(isOurs) ?? + (await context.waitForEvent("serviceworker", { + predicate: isOurs, + timeout: 10_000, + })); + } catch (error) { + await close(); + throw new Error( + `The OpenShowcase extension didn't start from ${extensionDir}. Is it a built Chrome extension (manifest.json inside)?`, + { cause: error }, + ); + } + try { + await fitViewport(context, width, height); + } catch (error) { + await close(); + throw error; + } + return { context, worker, close }; +} + +/** + * Files of a profile a run doesn't need: rebuildable caches (HTTP, compiled code, GPU), the + * lock of a Chromium using it, and the service worker registrations. The extension's worker + * is registered there with its script cached: a copy would run the extension as it was at + * login (stale code), and a copy without the cached script makes the worker die on start. + * Without the folder the extension registers fresh, from its files. Logins live in cookies + * and storage, not there; a site's own worker just registers again. + */ +const SKIPPED_PROFILE_FILES = new Set([ + "Service Worker", + "Cache", + "Code Cache", + "GPUCache", + "GPUPersistentCache", + "GraphiteDawnCache", + "DawnGraphiteCache", + "DawnWebGPUCache", + "ShaderCache", + "SingletonLock", + "SingletonSocket", + "SingletonCookie", +]); + +/** + * A temporary copy of the profile: the run keeps its logins, but whatever the flow does in + * the browser (localStorage, cookies, drafts) is gone afterwards, so every inspect and + * record starts from the same state. + */ +async function scratchCopy(profileDir: string): Promise { + const dir = await mkdtemp(join(tmpdir(), "openshowcase-run-")); + if (existsSync(profileDir)) + await cp(profileDir, dir, { + recursive: true, + filter: (src) => !SKIPPED_PROFILE_FILES.has(basename(src)), + }); + return dir; +} + +/** + * Resizes the window so the page itself is `width`x`height`: even headless Chromium keeps + * room for (invisible) browser UI, so --window-size alone leaves the page shorter. + */ +async function fitViewport( + context: BrowserContext, + width: number, + height: number, +): Promise { + const page = context.pages()[0] ?? (await context.newPage()); + const inner = () => + page.evaluate(() => [window.innerWidth, window.innerHeight] as const); + const [innerWidth, innerHeight] = await inner(); + if (innerWidth === width && innerHeight === height) return; + const [outerWidth, outerHeight] = await page.evaluate( + () => [window.outerWidth, window.outerHeight] as const, + ); + const cdp = await context.newCDPSession(page); + const { windowId } = await cdp.send("Browser.getWindowForTarget"); + await cdp.send("Browser.setWindowBounds", { + windowId, + bounds: { + width: width + outerWidth - innerWidth, + height: height + outerHeight - innerHeight, + }, + }); + await cdp.detach(); + for (let i = 0; i < 50; i++) { + const [w, h] = await inner(); + if (w === width && h === height) return; + await page.waitForTimeout(20); + } + throw new Error(`Couldn't size the page to ${width}x${height}`); +} + +/** Calls the extension's automation API (lib/automation.ts) inside its service worker. */ +export function automation( + worker: Worker, + method: K, + ...args: Parameters +): Promise>> { + return worker.evaluate( + ([method, args]) => { + const api = (globalThis as unknown as { openshowcase?: Automation }) + .openshowcase; + if (!api) + throw new Error( + "This build of the extension has no automation API. Rebuild it.", + ); + // biome-ignore lint/suspicious/noExplicitAny: forwarding a typed call across CDP + return (api[method] as (...a: any[]) => unknown)(...args); + }, + [method, args] as const, + ) as Promise>>; +} diff --git a/packages/recorder/src/cli.ts b/packages/recorder/src/cli.ts new file mode 100644 index 0000000..90eb90e --- /dev/null +++ b/packages/recorder/src/cli.ts @@ -0,0 +1,206 @@ +import { homedir } from "node:os"; +import { basename, extname, join, resolve } from "node:path"; +import { parseArgs } from "node:util"; +import { BUNDLED_EXTENSION_DIR, DEFAULT_PROFILE_DIR } from "./browser"; +import { loadFlow } from "./flow"; +import { formatInspection, inspect } from "./inspect"; +import { BUNDLED_SKILL_DIR, installSkill } from "./install-skill"; +import { login } from "./login"; +import { openInStudio } from "./open"; +import { record } from "./record"; + +export const USAGE = `openshowcase-recorder: record an OpenShowcase demo from a scripted flow + +Usage: + openshowcase-recorder inspect [--within ] [--screenshot ] + Run the flow fast, without recording, and print where it ends up: URL and + the accessibility tree (roles and names for targets), only the part + matching --within if given. A failing step is reported, with the page as + it was when it failed. --screenshot also saves an image of it. + openshowcase-recorder record [--out ] [--open] + Record the flow in a headless Chromium, paced for the camera, with a + visible cursor. --open then opens it for review (see open). + openshowcase-recorder open + Open a recording in the OpenShowcase editor, in a separate review browser + that stays open (your recordings are kept there). Returns right away. + openshowcase-recorder login + Open a window on the recording profile to sign in once. Close it when done. + +Steps really run against the app, so whatever they change on its server stays +changed. Browser state (localStorage, cookies) doesn't: inspect and record run on +a throwaway copy of the profile, so every run starts from what login saved. + openshowcase-recorder install-skill [--dir ] + Install the Claude Code skill into /.claude/skills. + +Options: + --profile Browser profile (logins) to use. Default: ${DEFAULT_PROFILE_DIR} + --extension A built OpenShowcase extension to record with (default: bundled) + --headed Show the browser window + -h, --help Show this help`; + +export class UsageError extends Error { + override readonly name = "UsageError"; +} + +interface BrowserOptions { + profileDir: string; + extensionDir: string; + headed: boolean; +} + +export type Command = + | { command: "help" } + | ({ + command: "record"; + flow: string; + out: string; + open: boolean; + } & BrowserOptions) + | { command: "open"; file: string; extensionDir: string } + | ({ + command: "inspect"; + flow: string; + screenshot?: string; + within?: string; + } & BrowserOptions) + | ({ command: "login"; url: string } & BrowserOptions) + | { command: "install-skill"; projectDir: string }; + +const expandHome = (path: string) => + path === "~" || path.startsWith("~/") ? join(homedir(), path.slice(1)) : path; + +export function parseCommand(argv: string[]): Command { + let parsed: ReturnType; + try { + parsed = parse(argv); + } catch (error) { + throw new UsageError((error as Error).message); + } + const { values, positionals } = parsed; + const [command, arg] = positionals; + if (values.help || !command) return { command: "help" }; + + const browser: BrowserOptions = { + profileDir: resolve(expandHome(values.profile ?? DEFAULT_PROFILE_DIR)), + extensionDir: values.extension + ? resolve(expandHome(values.extension)) + : BUNDLED_EXTENSION_DIR, + headed: values.headed ?? false, + }; + + switch (command) { + case "record": { + if (!arg) throw new UsageError("record needs a flow file"); + const out = + values.out ?? + `${basename(arg, extname(arg)).replace(/\.flow$/, "") || "recording"}.osrec`; + return { + command, + flow: arg, + out: resolve(out), + ...browser, + open: values.open ?? false, + }; + } + case "open": + if (!arg) throw new UsageError("open needs a recording (.osrec)"); + return { + command, + file: resolve(arg), + extensionDir: browser.extensionDir, + }; + case "inspect": + if (!arg) throw new UsageError("inspect needs a flow file or a URL"); + return { + command, + flow: arg, + ...(values.screenshot && { screenshot: resolve(values.screenshot) }), + ...(values.within && { within: values.within }), + ...browser, + }; + case "login": + if (!arg) throw new UsageError("login needs the URL to sign in at"); + return { command, url: arg, ...browser }; + case "install-skill": + return { + command, + projectDir: resolve(values.dir ?? process.cwd()), + }; + default: + throw new UsageError(`Unknown command "${command}"`); + } +} + +function parse(argv: string[]) { + return parseArgs({ + args: argv, + allowPositionals: true, + options: { + out: { type: "string" }, + screenshot: { type: "string" }, + within: { type: "string" }, + profile: { type: "string" }, + extension: { type: "string" }, + dir: { type: "string" }, + headed: { type: "boolean" }, + open: { type: "boolean" }, + help: { type: "boolean", short: "h" }, + }, + }); +} + +/** Runs a command; resolves with the process exit code. */ +export async function run(argv: string[]): Promise { + const log = (line: string) => process.stdout.write(`${line}\n`); + let command: Command; + try { + command = parseCommand(argv); + } catch (error) { + process.stderr.write(`${(error as Error).message}\n\n${USAGE}\n`); + return 2; + } + + try { + switch (command.command) { + case "help": + log(USAGE); + return 0; + case "record": { + const flow = await loadFlow(command.flow); + const result = await record(flow, { ...command, log }); + log(`Recorded → ${result.out} (${(result.bytes / 1e6).toFixed(1)} MB)`); + log( + `Demo: ${result.hotspots} hotspot steps, about ${Math.round(result.videoSec)}s of video after cutting ${Math.round(result.cutSec)}s of waiting.`, + ); + if (command.open) { + await openInStudio(result.out, command); + log("Opened in the review browser."); + } else log(`Review it: openshowcase-recorder open ${result.out}`); + return 0; + } + case "open": + await openInStudio(command.file, command); + log(`Opened ${command.file} in the review browser.`); + return 0; + case "inspect": { + const flow = await loadFlow(command.flow); + const inspection = await inspect(flow, command); + log(formatInspection(inspection)); + return inspection.error ? 1 : 0; + } + case "login": + log(`Sign in in the window that opened, then close it.`); + await login(command.url, command); + log(`Saved to the profile at ${command.profileDir}`); + return 0; + case "install-skill": + log( + `Installed ${await installSkill(command.projectDir, BUNDLED_SKILL_DIR)}`, + ); + return 0; + } + } catch (error) { + process.stderr.write(`${(error as Error).message}\n`); + return 1; + } +} diff --git a/packages/recorder/src/cursor.ts b/packages/recorder/src/cursor.ts new file mode 100644 index 0000000..eb143a0 --- /dev/null +++ b/packages/recorder/src/cursor.ts @@ -0,0 +1,102 @@ +import type { BrowserContext } from "playwright"; + +/** + * Headless Chromium draws no cursor, while a recording made by hand has Chrome's own cursor + * burned in. This puts one in the page, following the (real, CDP-driven) mouse, with a ripple + * on press. Built with DOM APIs only (no HTML strings, no style attributes) so a page's CSP + * or Trusted Types don't block it, and in a shadow root so the page's CSS doesn't touch it. + */ +export async function installCursor(context: BrowserContext): Promise { + await context.addInitScript(cursorScript); +} + +function cursorScript() { + // ponytail: top frame only; over an iframe the cursor stops until the mouse leaves it. + if (window.top !== window) return; + const KEY = "openshowcase-cursor"; + const SVG = "http://www.w3.org/2000/svg"; + + const host = document.createElement("openshowcase-cursor"); + host.style.cssText = + "position:fixed;left:0;top:0;width:0;height:0;z-index:2147483647;pointer-events:none;opacity:0;"; + const root = host.attachShadow({ mode: "open" }); + + const ripple = document.createElement("div"); + ripple.style.cssText = + "position:absolute;left:-16px;top:-16px;width:32px;height:32px;border-radius:50%;background:rgba(20,20,20,.18);border:2px solid rgba(255,255,255,.9);box-sizing:border-box;opacity:0;"; + + // A macOS-style arrow, tip at (0, 0). + const svg = document.createElementNS(SVG, "svg"); + svg.setAttribute("width", "22"); + svg.setAttribute("height", "30"); + svg.setAttribute("viewBox", "-1 -1 15 21"); + svg.style.cssText = + "position:absolute;left:-1.5px;top:-1.5px;overflow:visible;filter:drop-shadow(0 1px 1.5px rgba(0,0,0,.35));transform-origin:2px 2px;"; + const arrow = document.createElementNS(SVG, "path"); + arrow.setAttribute( + "d", + "M0 0 L0 16 L4 12.3 L6.8 18.6 L9.3 17.5 L6.6 11.4 L11.8 11.4 Z", + ); + arrow.setAttribute("fill", "#111"); + arrow.setAttribute("stroke", "#fff"); + arrow.setAttribute("stroke-width", "1.3"); + arrow.setAttribute("stroke-linejoin", "round"); + svg.append(arrow); + root.append(ripple, svg); + + const place = (x: number, y: number) => { + host.style.transform = `translate(${x}px, ${y}px)`; + host.style.opacity = "1"; + try { + sessionStorage.setItem(KEY, JSON.stringify({ x, y })); + } catch { + /* storage off: the cursor reappears on the next move */ + } + }; + // Where the cursor was before this page load (same tab, same origin). + try { + const last = JSON.parse(sessionStorage.getItem(KEY) ?? "null"); + if (last) place(last.x, last.y); + } catch { + /* no previous position */ + } + + addEventListener("mousemove", (e) => place(e.clientX, e.clientY), { + capture: true, + passive: true, + }); + addEventListener( + "mousedown", + (e) => { + place(e.clientX, e.clientY); + svg.animate( + [ + { transform: "scale(1)" }, + { transform: "scale(.85)" }, + { transform: "scale(1)" }, + ], + { duration: 220, easing: "ease-out" }, + ); + ripple.animate( + [ + { opacity: 1, transform: "scale(.3)" }, + { opacity: 0, transform: "scale(1.3)" }, + ], + { duration: 450, easing: "cubic-bezier(.2,.7,.3,1)" }, + ); + }, + { capture: true, passive: true }, + ); + + // Under , not : frameworks re-render the body. Re-attach if something removes it. + const mount = () => { + if (document.documentElement && !host.isConnected) + document.documentElement.append(host); + }; + mount(); + document.addEventListener("readystatechange", mount); + new MutationObserver(mount).observe(document, { + childList: true, + subtree: false, + }); +} diff --git a/packages/recorder/src/flow.ts b/packages/recorder/src/flow.ts new file mode 100644 index 0000000..fb1b627 --- /dev/null +++ b/packages/recorder/src/flow.ts @@ -0,0 +1,167 @@ +import { readFile } from "node:fs/promises"; +import * as v from "valibot"; + +/** + * The script of a demo (a JSON file the agent writes): where to start, at what size, and + * the steps to perform. Targets are Playwright locators by accessibility role/name first, + * so a flow keeps working when the markup changes. + */ +const TARGET_KEYS = [ + "role", + "text", + "label", + "placeholder", + "testId", + "css", +] as const; + +const Target = v.pipe( + v.object({ + role: v.optional(v.string()), + /** Accessible name, with `role`. */ + name: v.optional(v.string()), + text: v.optional(v.string()), + label: v.optional(v.string()), + placeholder: v.optional(v.string()), + testId: v.optional(v.string()), + css: v.optional(v.string()), + /** Exact (case-sensitive, whole-string) match for name/text/label/placeholder. */ + exact: v.optional(v.boolean()), + /** Which match, 0-based, when the target matches several elements. */ + nth: v.optional(v.pipe(v.number(), v.integer(), v.minValue(0))), + }), + v.check( + (target) => TARGET_KEYS.filter((key) => target[key] != null).length === 1, + "a target needs exactly one of role, text, label, placeholder, testId or css", + ), +); + +/** Camera zoom on the action (photo step or clip), centered on its target. */ +const Zoom = v.optional(v.pipe(v.number(), v.minValue(1.2), v.maxValue(3))); + +const Step = v.variant("action", [ + v.object({ + action: v.literal("click"), + target: Target, + /** The hotspot's tooltip for this click in the demo. */ + tooltip: v.optional(v.string()), + /** false: no photo step; the click just plays in the video, cursor and all. */ + hotspot: v.optional(v.boolean()), + zoom: Zoom, + }), + /** Types the value visibly, character by character. */ + v.object({ + action: v.literal("fill"), + target: Target, + value: v.string(), + zoom: Zoom, + }), + v.object({ + action: v.literal("press"), + key: v.string(), + target: v.optional(Target), + }), + v.object({ + action: v.literal("select"), + target: Target, + value: v.string(), + zoom: Zoom, + }), + v.object({ action: v.literal("hover"), target: Target, zoom: Zoom }), + /** Smooth scroll: to a target, or by a number of pixels (default 600). */ + v.object({ + action: v.literal("scroll"), + target: v.optional(Target), + by: v.optional(v.number()), + }), + v.object({ action: v.literal("goto"), url: v.pipe(v.string(), v.url()) }), + v.pipe( + v.object({ + action: v.literal("wait"), + ms: v.optional(v.pipe(v.number(), v.minValue(0))), + for: v.optional(Target), + }), + v.check( + (step) => step.ms != null || step.for != null, + "a wait needs ms or for", + ), + ), +]); + +const Flow = v.object({ + url: v.pipe(v.string(), v.url()), + viewport: v.optional( + v.object({ + width: v.pipe(v.number(), v.integer(), v.minValue(320)), + height: v.pipe(v.number(), v.integer(), v.minValue(240)), + }), + { width: 1280, height: 800 }, + ), + steps: v.array(Step), +}); + +export type Target = v.InferOutput; +export type Step = v.InferOutput; +export type Flow = v.InferOutput; + +export class FlowError extends Error { + override readonly name = "FlowError"; +} + +export function parseFlow(input: unknown): Flow { + const result = v.safeParse(Flow, input); + if (result.success) return result.output; + const lines = result.issues.map( + (issue) => `- ${v.getDotPath(issue) ?? "(root)"}: ${issue.message}`, + ); + throw new FlowError(`Invalid flow:\n${lines.join("\n")}`); +} + +/** A flow file, or a bare URL (a flow with no steps: handy to look at a page first). */ +export async function loadFlow(pathOrUrl: string): Promise { + if (/^https?:\/\//.test(pathOrUrl)) + return parseFlow({ url: pathOrUrl, steps: [] }); + const text = await readFile(pathOrUrl, "utf8"); + let json: unknown; + try { + json = JSON.parse(text); + } catch (error) { + throw new FlowError( + `${pathOrUrl} is not valid JSON: ${(error as Error).message}`, + ); + } + return parseFlow(json); +} + +export function describeTarget(target: Target): string { + return Object.entries(target) + .filter(([, value]) => value != null) + .map(([key, value]) => + key === "role" || typeof value !== "string" + ? `${key}=${value}` + : `${key}=${JSON.stringify(value)}`, + ) + .join(" "); +} + +/** One line for logs and errors: `click role=button name="Save"`. */ +export function describeStep(step: Step): string { + switch (step.action) { + case "press": + return step.target + ? `press ${step.key} on ${describeTarget(step.target)}` + : `press ${step.key}`; + case "goto": + return `goto ${step.url}`; + case "wait": + return step.for + ? `wait for ${describeTarget(step.for)}` + : `wait ${step.ms}ms`; + case "scroll": + return step.target + ? `scroll to ${describeTarget(step.target)}` + : `scroll by ${step.by ?? 600}px`; + default: + return `${step.action} ${describeTarget(step.target)}`; + } +} diff --git a/packages/recorder/src/human.ts b/packages/recorder/src/human.ts new file mode 100644 index 0000000..4377449 --- /dev/null +++ b/packages/recorder/src/human.ts @@ -0,0 +1,73 @@ +/** How the recorder moves the mouse so the cursor in the demo reads as a person's hand. */ + +export interface Point { + x: number; + y: number; +} + +/** One mouse move per frame. */ +export const MOVE_TICK_MS = 16; + +/** Minimum-jerk easing: the speed profile of a real reaching movement (slow, fast, slow). */ +const ease = (t: number) => t * t * t * (10 - 15 * t + 6 * t * t); + +/** + * Points (one per frame) from `from` to `to`: a gentle arc to one side, eased, and longer for + * longer distances (Fitts's law, roughly 0.3–0.9s). + */ +export function humanPath( + from: Point, + to: Point, + random: () => number = Math.random, +): Point[] { + const dx = to.x - from.x; + const dy = to.y - from.y; + const dist = Math.hypot(dx, dy); + if (dist < 3) return [to]; + + const ms = Math.min(900, Math.max(280, 250 + 120 * Math.log2(1 + dist / 30))); + const ticks = Math.round(ms / MOVE_TICK_MS); + // Perpendicular bend: up to ±12% of the distance, capped so long moves stay subtle. + const bend = Math.min(90, dist * 0.12) * (random() * 2 - 1); + const nx = -dy / dist; + const ny = dx / dist; + const c1 = { + x: from.x + dx * 0.3 + nx * bend, + y: from.y + dy * 0.3 + ny * bend, + }; + const c2 = { + x: from.x + dx * 0.75 + nx * bend * 0.5, + y: from.y + dy * 0.75 + ny * bend * 0.5, + }; + + const points: Point[] = []; + for (let i = 1; i < ticks; i++) { + const t = ease(i / ticks); + const u = 1 - t; + points.push({ + x: + u * u * u * from.x + + 3 * u * u * t * c1.x + + 3 * u * t * t * c2.x + + t * t * t * to.x, + y: + u * u * u * from.y + + 3 * u * u * t * c1.y + + 3 * u * t * t * c2.y + + t * t * t * to.y, + }); + } + points.push(to); + return points; +} + +/** Where a person would click on a box: near its middle, a little off center. */ +export function aimPoint( + box: { x: number; y: number; width: number; height: number }, + random: () => number = Math.random, +): Point { + return { + x: box.x + box.width * (0.4 + random() * 0.2), + y: box.y + box.height * (0.4 + random() * 0.2), + }; +} diff --git a/packages/recorder/src/inspect.ts b/packages/recorder/src/inspect.ts new file mode 100644 index 0000000..e95bd79 --- /dev/null +++ b/packages/recorder/src/inspect.ts @@ -0,0 +1,96 @@ +import { launch } from "./browser"; +import type { Flow } from "./flow"; +import { INSTANT_PACE, runFlow, StepError } from "./run-flow"; + +export interface Inspection { + url: string; + title: string; + /** Only when asked for: an image costs an agent far more than the tree. */ + screenshot?: string; + /** Steps that ran (all of them, unless one failed). */ + ran: number; + total: number; + error?: string; + /** Playwright's ARIA snapshot of the page: the roles and names targets use. */ + snapshot: string; +} + +export interface InspectOptions { + profileDir: string; + extensionDir: string; + headed: boolean; + screenshot?: string; + /** CSS selector of the part of the page to describe (a dialog, `main`…). */ + within?: string; +} + +/** Runs the flow without pacing or recording and reports the page where it ends (or fails). */ +export async function inspect( + flow: Flow, + options: InspectOptions, +): Promise { + const { context, close } = await launch({ + ...options, + headless: !options.headed, + viewport: flow.viewport, + }); + try { + const page = context.pages()[0] ?? (await context.newPage()); + page.setDefaultTimeout(5_000); + await page.goto(flow.url); + + let ran = flow.steps.length; + let error: string | undefined; + try { + await runFlow(page, flow.steps, INSTANT_PACE); + } catch (e) { + if (!(e instanceof StepError)) throw e; + ran = e.index; + error = e.message; + } + + if (options.screenshot) await page.screenshot({ path: options.screenshot }); + return { + url: page.url(), + title: await page.title(), + ...(options.screenshot && { screenshot: options.screenshot }), + ran, + total: flow.steps.length, + ...(error && { error }), + snapshot: compactSnapshot( + await page + .locator(options.within ?? "body") + .first() + .ariaSnapshot(), + ), + }; + } finally { + await close(); + } +} + +export function formatInspection(inspection: Inspection): string { + const steps = inspection.error + ? `${inspection.ran} of ${inspection.total} ran. ${inspection.error}` + : inspection.total === 0 + ? "none (just the start page)" + : `all ${inspection.total} ran`; + return [ + `URL: ${inspection.url}`, + `Title: ${inspection.title}`, + ...(inspection.screenshot ? [`Screenshot: ${inspection.screenshot}`] : []), + `Steps: ${steps}`, + "", + "Accessibility tree:", + inspection.snapshot, + ].join("\n"); +} + +/** The ARIA snapshot without link URLs: flows target names, and every line costs tokens. */ +export function compactSnapshot(snapshot: string): string { + return snapshot + .split("\n") + .filter((line) => !/^\s*- \/url: /.test(line)) + .map((line) => line.replace(/^(\s*- link .*):$/, "$1")) + .join("\n"); +} diff --git a/packages/recorder/src/install-skill.ts b/packages/recorder/src/install-skill.ts new file mode 100644 index 0000000..1b49fff --- /dev/null +++ b/packages/recorder/src/install-skill.ts @@ -0,0 +1,19 @@ +import { cp } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +/** The skill the published package ships next to this file (see rslib.config.ts). */ +export const BUNDLED_SKILL_DIR = join( + dirname(fileURLToPath(import.meta.url)), + "skill", +); + +/** Copies the skill into `/.claude/skills/openshowcase-demo`; resolves with its SKILL.md. */ +export async function installSkill( + projectDir: string, + skillDir: string, +): Promise { + const dest = join(projectDir, ".claude", "skills", "openshowcase-demo"); + await cp(skillDir, dest, { recursive: true, force: true }); + return join(dest, "SKILL.md"); +} diff --git a/packages/recorder/src/login.ts b/packages/recorder/src/login.ts new file mode 100644 index 0000000..0b802a8 --- /dev/null +++ b/packages/recorder/src/login.ts @@ -0,0 +1,24 @@ +import { launch } from "./browser"; + +/** Opens a visible window on the profile at `url` and waits for the user to close it. */ +export async function login( + url: string, + options: { profileDir: string; extensionDir: string }, +): Promise { + const { context, close } = await launch({ + ...options, + headless: false, + keepChanges: true, + }); + const page = context.pages()[0] ?? (await context.newPage()); + await page.goto(url); + await new Promise((resolve) => { + const closedAll = () => { + if (context.pages().length === 0) resolve(); + }; + for (const p of context.pages()) p.on("close", closedAll); + context.on("page", (p) => p.on("close", closedAll)); + context.on("close", () => resolve()); + }); + await close(); +} diff --git a/packages/recorder/src/open.ts b/packages/recorder/src/open.ts new file mode 100644 index 0000000..5e51cf7 --- /dev/null +++ b/packages/recorder/src/open.ts @@ -0,0 +1,65 @@ +import { spawn as nodeSpawn } from "node:child_process"; +import { realpathSync } from "node:fs"; +import { copyFile, cp, mkdir, rm } from "node:fs/promises"; +import { homedir } from "node:os"; +import { basename, join } from "node:path"; +import { chromium } from "playwright"; +import { extensionIdForPath } from "./browser"; + +/** A browser of its own for reviewing recordings: profile (editor's IndexedDB) + extension copy. */ +export const STUDIO_DIR = join(homedir(), ".openshowcase", "studio"); + +export interface OpenOptions { + extensionDir: string; + studioDir?: string; + executablePath?: string; + spawn?: ( + command: string, + args: string[], + options: { detached: boolean; stdio: "ignore" }, + ) => { unref: () => void }; +} + +/** + * Opens the recording in the extension's editor, in a separate Chromium that outlives this + * command: a plain browser (not automated), so downloads and the playground work as usual and + * recordings stay in its profile. The file goes into the studio's own copy of the extension, + * which the editor imports from (`?import=`, see the extension's core/import-source.ts). If the + * studio is already open, Chromium hands the URL to it as a new tab. + */ +export async function openInStudio( + file: string, + options: OpenOptions, +): Promise { + if (!file.endsWith(".osrec")) + throw new Error(`${file} is not a recording (.osrec)`); + const studioDir = options.studioDir ?? STUDIO_DIR; + const extensionDir = join(studioDir, "extension"); + const imports = join(extensionDir, "imports"); + + await rm(imports, { recursive: true, force: true }); + await cp(options.extensionDir, extensionDir, { + recursive: true, + force: true, + }); + await mkdir(imports, { recursive: true }); + const name = basename(file).replace(/[^\w.-]/g, "-"); + await copyFile(file, join(imports, name)); + + const realExtension = realpathSync(extensionDir); + const url = `chrome-extension://${extensionIdForPath(realExtension)}/editor.html?import=imports/${name}`; + const spawn = options.spawn ?? nodeSpawn; + spawn( + options.executablePath ?? chromium.executablePath(), + [ + `--user-data-dir=${join(studioDir, "profile")}`, + `--disable-extensions-except=${realExtension}`, + `--load-extension=${realExtension}`, + "--no-first-run", + "--no-default-browser-check", + url, + ], + { detached: true, stdio: "ignore" }, + ).unref(); + return url; +} diff --git a/packages/recorder/src/record.ts b/packages/recorder/src/record.ts new file mode 100644 index 0000000..8813684 --- /dev/null +++ b/packages/recorder/src/record.ts @@ -0,0 +1,94 @@ +import { open } from "node:fs/promises"; +import { automation, launch } from "./browser"; +import { installCursor } from "./cursor"; +import { describeStep, type Flow } from "./flow"; +import { RECORDING_PACE, runFlow } from "./run-flow"; + +/** Base64 chunks pulled from the service worker: well under CDP's message size limit. */ +const CHUNK_BYTES = 8 * 1024 * 1024; + +export interface RecordOptions { + profileDir: string; + extensionDir: string; + headed: boolean; + out: string; + log?: (line: string) => void; +} + +/** + * Records the flow through the extension's own capture pipeline (tab video + click markers, + * exactly as a human recording) and writes it as an `.osrec` file. + */ +export async function record( + flow: Flow, + options: RecordOptions, +): Promise<{ + out: string; + bytes: number; + /** Clicks that stop the demo on a hotspot. */ + hotspots: number; + /** Recording length once the waits are cut (the video steps can only play less). */ + videoSec: number; + cutSec: number; +}> { + const log = options.log ?? (() => {}); + const { context, worker, close } = await launch({ + ...options, + headless: !options.headed, + viewport: flow.viewport, + }); + try { + await installCursor(context); + const page = context.pages()[0] ?? (await context.newPage()); + page.setDefaultTimeout(10_000); + await page.goto(flow.url); + // The extension records the active tab. + await page.bringToFront(); + + const recordingId = await automation(worker, "start"); + const startedAt = Date.now(); + log(`Recording ${flow.url}`); + const direction = await runFlow( + page, + flow.steps, + RECORDING_PACE, + (i, step) => log(` ${i + 1}/${flow.steps.length} ${describeStep(step)}`), + ); + await automation(worker, "stop", recordingId); + const recordedSec = (Date.now() - startedAt) / 1000; + const cutSec = + direction.cuts.reduce((sum, c) => sum + c.end - c.start, 0) / 1000; + + const bytes = await automation( + worker, + "exportFile", + recordingId, + direction, + ); + const file = await open(options.out, "w"); + try { + for (let offset = 0; offset < bytes; offset += CHUNK_BYTES) { + const chunk = await automation( + worker, + "readChunk", + offset, + CHUNK_BYTES, + ); + await file.write(Buffer.from(chunk, "base64")); + } + } finally { + await file.close(); + } + return { + out: options.out, + bytes, + hotspots: flow.steps.filter( + (s) => s.action === "click" && s.hotspot !== false, + ).length, + videoSec: recordedSec - cutSec, + cutSec, + }; + } finally { + await close(); + } +} diff --git a/packages/recorder/src/run-flow.ts b/packages/recorder/src/run-flow.ts new file mode 100644 index 0000000..a95a472 --- /dev/null +++ b/packages/recorder/src/run-flow.ts @@ -0,0 +1,307 @@ +import type { Locator, Page } from "playwright"; +import type { Direction } from "./browser"; +import { describeStep, type Step, type Target } from "./flow"; +import { aimPoint, humanPath, MOVE_TICK_MS, type Point } from "./human"; + +/** How things happen on camera. A recording should feel like a person; inspecting doesn't. */ +export interface Pace { + /** Before the first step: the viewer sees where the demo starts. */ + leadIn: number; + /** Between the cursor reaching a target and pressing it. */ + beforeClick: number; + /** After every step, so its result is on screen. */ + afterStep: number; + /** Per typed character. */ + typingDelay: number; + /** Between 100px wheel ticks of a scroll; 0 scrolls in one go (and instantly). */ + scrollTick: number; + /** After the last step: the demo ends on the result. */ + tail: number; + /** Move the mouse along a human path (otherwise it jumps). */ + humanMoves: boolean; +} + +export const RECORDING_PACE: Pace = { + leadIn: 800, + beforeClick: 180, + afterStep: 500, + typingDelay: 55, + scrollTick: 16, + tail: 1200, + humanMoves: true, +}; + +export const INSTANT_PACE: Pace = { + leadIn: 0, + beforeClick: 0, + afterStep: 0, + typingDelay: 0, + scrollTick: 0, + tail: 0, + humanMoves: false, +}; + +const DEFAULT_SCROLL_PX = 600; +const WHEEL_TICK_PX = 100; +/** A wait for the app shorter than this isn't worth a cut. */ +const MIN_CUT_MS = 200; +/** A target is used where the viewer can see it well: between these fractions of the height. */ +const COMFORT_TOP = 0.12; +const COMFORT_BOTTOM = 0.85; + +export class StepError extends Error { + override readonly name = "StepError"; + constructor( + readonly index: number, + readonly step: Step, + cause: unknown, + ) { + const reason = ( + cause instanceof Error ? cause.message : String(cause) + ).split("\n")[0]; + super(`Step ${index + 1} (${describeStep(step)}) failed: ${reason}`, { + cause, + }); + } +} + +export function locate(page: Page, target: Target): Locator { + const { exact } = target; + const base = + target.role != null + ? page.getByRole(target.role as Parameters[0], { + name: target.name, + exact, + }) + : target.text != null + ? page.getByText(target.text, { exact }) + : target.label != null + ? page.getByLabel(target.label, { exact }) + : target.placeholder != null + ? page.getByPlaceholder(target.placeholder, { exact }) + : target.testId != null + ? page.getByTestId(target.testId) + : page.locator(target.css ?? ""); + return target.nth != null ? base.nth(target.nth) : base; +} + +/** + * Performs the steps on the page at the given pace, and returns the direction for the editor: + * what each click asked for (with its wall-clock window, matched to the extension's click + * markers), the waits to cut out of the video, and the actions to zoom into. + */ +export async function runFlow( + page: Page, + steps: Step[], + pace: Pace, + onStep?: (index: number, step: Step) => void, + random: () => number = Math.random, +): Promise { + const pause = (ms: number) => (ms > 0 ? page.waitForTimeout(ms) : undefined); + const direction: Direction = { clicks: [], cuts: [], zooms: [] }; + const [width, height] = await page.evaluate( + () => [window.innerWidth, window.innerHeight] as const, + ); + let mouse: Point = { x: width / 2, y: height / 2 }; + if (pace.humanMoves) await page.mouse.move(mouse.x, mouse.y); + + /** Time the app made us wait: cut from the video when it's long enough to notice. */ + const offCamera = async (wait: () => Promise) => { + const start = Date.now(); + await wait(); + if (Date.now() - start >= MIN_CUT_MS) + direction.cuts.push({ start, end: Date.now() }); + }; + + await pause(pace.leadIn); + for (const [index, step] of steps.entries()) { + onStep?.(index, step); + try { + await perform(step); + await offCamera(() => page.waitForLoadState()); + } catch (error) { + throw new StepError(index, step, error); + } + await pause(pace.afterStep); + } + await pause(pace.tail); + return direction; + + async function moveTo(point: Point): Promise { + if (!pace.humanMoves) { + mouse = point; + return; + } + for (const p of humanPath(mouse, point, random)) { + await page.mouse.move(p.x, p.y); + await page.waitForTimeout(MOVE_TICK_MS); + } + mouse = point; + } + + /** Waits for the target (off camera), centers it if it's near an edge, then aims at it. */ + async function approach(locator: Locator) { + await offCamera(() => locator.waitFor({ state: "visible" })); + let box = await locator.boundingBox(); + if ( + box && + (box.y < height * COMFORT_TOP || + box.y + box.height > height * COMFORT_BOTTOM) + ) { + await locator.evaluate( + (el, smooth) => + el.scrollIntoView({ + behavior: smooth ? "smooth" : "instant", + block: "center", + }), + pace.scrollTick > 0, + ); + box = await settled(locator); + } + if (!box) throw new Error("The target has no size on the page"); + const aim = aimPoint(box, random); + await moveTo(aim); + return { + box, + /** Where to press, relative to the element (what Playwright's `position` takes). */ + position: { x: aim.x - box.x, y: aim.y - box.y }, + focus: { + x: (box.x + box.width / 2) / width, + y: (box.y + box.height / 2) / height, + }, + }; + } + + /** The target's box once a smooth scroll has stopped moving it. */ + async function settled(locator: Locator) { + let box = await locator.boundingBox(); + for (let i = 0; i < 30; i++) { + await page.waitForTimeout(50); + const next = await locator.boundingBox(); + if (next?.y === box?.y) return next; + box = next; + } + return box; + } + + async function zoomed( + zoom: number | undefined, + focus: Point, + action: () => Promise, + ) { + const start = Date.now(); + await action(); + if (zoom) + direction.zooms.push({ start, end: Date.now(), ...focus, scale: zoom }); + } + + async function perform(step: Step): Promise { + switch (step.action) { + case "click": { + const target = locate(page, step.target); + const { position } = await approach(target); + await pause(pace.beforeClick); + const start = Date.now(); + await target.click({ position }); + const { tooltip, zoom, hotspot } = step; + if (tooltip != null || zoom != null || hotspot != null) + direction.clicks.push({ + start, + end: Date.now(), + ...(tooltip != null && { label: tooltip }), + ...(zoom != null && { zoom }), + ...(hotspot != null && { hotspot }), + }); + return; + } + case "fill": { + const target = locate(page, step.target); + const { position, focus } = await approach(target); + const focused = await target.evaluate( + (el) => + el === document.activeElement || + el.contains(document.activeElement), + ); + await zoomed(step.zoom, focus, async () => { + if (!focused) { + await pause(pace.beforeClick); + const start = Date.now(); + await target.click({ position }); + // A person clicks into the field first; it's not a step of its own. + direction.clicks.push({ start, end: Date.now(), hotspot: false }); + } + await target.fill(""); + await target.pressSequentially(step.value, { + delay: pace.typingDelay, + }); + }); + return; + } + case "press": + if (step.target) await locate(page, step.target).press(step.key); + else await page.keyboard.press(step.key); + return; + case "select": { + const target = locate(page, step.target); + const { focus } = await approach(target); + await zoomed(step.zoom, focus, async () => { + await pause(pace.beforeClick); + await target.selectOption(step.value); + }); + return; + } + case "hover": { + const target = locate(page, step.target); + const { position, focus } = await approach(target); + await zoomed(step.zoom, focus, async () => { + await target.hover({ position }); + await pause(pace.afterStep); + }); + return; + } + case "scroll": + if (step.target) { + const target = locate(page, step.target); + await offCamera(() => target.waitFor({ state: "visible" })); + await target.evaluate( + (el, smooth) => + el.scrollIntoView({ + behavior: smooth ? "smooth" : "instant", + block: "center", + }), + pace.scrollTick > 0, + ); + await settled(target); + return; + } + await wheel(step.by ?? DEFAULT_SCROLL_PX); + return; + case "goto": + await offCamera(() => page.goto(step.url)); + return; + case "wait": { + // Waiting is for the app, never for the viewer: always cut, whatever its length. + const start = Date.now(); + if (step.for) + await locate(page, step.for).waitFor({ state: "visible" }); + if (step.ms) await page.waitForTimeout(step.ms); + direction.cuts.push({ start, end: Date.now() }); + return; + } + } + } + + /** Wheel ticks rather than a jump: the clip shows a real scroll, and the page sees wheel events. */ + async function wheel(by: number): Promise { + if (pace.scrollTick === 0) { + await page.mouse.wheel(0, by); + return; + } + const ticks = Math.ceil(Math.abs(by) / WHEEL_TICK_PX); + for (let i = 0; i < ticks; i++) { + const left = Math.abs(by) - i * WHEEL_TICK_PX; + await page.mouse.wheel(0, Math.sign(by) * Math.min(WHEEL_TICK_PX, left)); + await page.waitForTimeout(pace.scrollTick); + } + } +} diff --git a/packages/recorder/tsconfig.build.json b/packages/recorder/tsconfig.build.json new file mode 100644 index 0000000..6079c7f --- /dev/null +++ b/packages/recorder/tsconfig.build.json @@ -0,0 +1,4 @@ +{ + "extends": "./tsconfig.json", + "exclude": ["src/**/__test__"] +} diff --git a/packages/recorder/tsconfig.json b/packages/recorder/tsconfig.json new file mode 100644 index 0000000..1108495 --- /dev/null +++ b/packages/recorder/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "Preserve", + "rootDir": "src", + "lib": ["ES2022", "DOM", "DOM.Iterable"], + "types": ["node"], + "strict": true, + "noUncheckedIndexedAccess": true, + "verbatimModuleSyntax": true, + "isolatedModules": true, + "moduleDetection": "force", + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "noEmit": true + }, + "include": ["src"] +} diff --git a/packages/recorder/vitest.config.ts b/packages/recorder/vitest.config.ts new file mode 100644 index 0000000..e9f8d13 --- /dev/null +++ b/packages/recorder/vitest.config.ts @@ -0,0 +1,28 @@ +import { defineConfig } from "vitest/config"; + +// Node, not browser mode: the CLI drives Chromium itself through Playwright. Unit tests that +// need a page launch a plain headless Chromium; e2e records for real with the built extension +// (apps/extension/.output/chrome-mv3, built first by Turbo). +export default defineConfig({ + test: { + projects: [ + { + test: { + name: "unit", + include: ["src/__test__/unit/**/*.test.ts"], + environment: "node", + testTimeout: 20_000, + }, + }, + { + test: { + name: "e2e", + include: ["src/__test__/e2e/**/*.test.ts"], + environment: "node", + testTimeout: 120_000, + hookTimeout: 60_000, + }, + }, + ], + }, +}); diff --git a/packages/schema/package.json b/packages/schema/package.json index ad3be6c..a79bf0b 100644 --- a/packages/schema/package.json +++ b/packages/schema/package.json @@ -60,6 +60,6 @@ } ], "dependencies": { - "valibot": "1.5.0" + "valibot": "catalog:core" } } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b238650..c99cfc4 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -167,6 +167,9 @@ catalogs: typescript: specifier: 7.0.2 version: 7.0.2 + valibot: + specifier: 1.5.0 + version: 1.5.0 react: '@types/react': specifier: 19.3.0 @@ -530,10 +533,32 @@ importers: specifier: catalog:tooling version: 5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)) + packages/recorder: + dependencies: + playwright: + specifier: catalog:tooling + version: 1.63.0 + valibot: + specifier: catalog:core + version: 1.5.0(typescript@7.0.2) + devDependencies: + '@rslib/core': + specifier: catalog:tooling + version: 1.0.1(@microsoft/api-extractor@7.59.2(@types/node@24.19.2))(typescript@7.0.2) + '@types/node': + specifier: 24.19.2 + version: 24.19.2 + typescript: + specifier: catalog:core + version: 7.0.2 + vitest: + specifier: catalog:tooling + version: 5.0.1(@types/node@24.19.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + packages/schema: dependencies: valibot: - specifier: 1.5.0 + specifier: catalog:core version: 1.5.0(typescript@7.0.2) devDependencies: '@rslib/core': @@ -1548,6 +1573,9 @@ packages: '@types/har-format@1.2.16': resolution: {integrity: sha512-fluxdy7ryD3MV6h8pTfTYpy/xQzCFC7m89nOH9y94cNqJ1mDIDPut7MnRHI3F6qRmh/cT2fUjG1MLdCNb4hE9A==} + '@types/node@24.19.2': + resolution: {integrity: sha512-93d49wCWJkCyN87MxLpNN6KXPCwFw7bVUg79mGUV1OS40JGWwVCr8n61DXCSqz7CV6/VfPkb7xec7uP2M+QHWQ==} + '@types/node@26.6.2': resolution: {integrity: sha512-X1P21scMv4zGKLYqjdGjaKa7COa0RKVYYZZN/NfvLQ1JegxFhdhpZG/Lyn8AXx6CDUavKAd11v6BvfpkDByK8g==} @@ -2803,6 +2831,9 @@ packages: uhyphen@0.2.0: resolution: {integrity: sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA==} + undici-types@7.24.6: + resolution: {integrity: sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==} + undici-types@8.9.0: resolution: {integrity: sha512-KTDyRTYX8sWmKXAikPHHSyc63CRPETMctyjKFupcC6OBLXT3xsN0e9aF7m+mIXutFWpUXuedtowG7iLOzp0kQg==} @@ -3480,6 +3511,15 @@ snapshots: tinyglobby: 0.2.17 yaml: 2.9.1 + '@microsoft/api-extractor-model@7.33.13(@types/node@24.19.2)': + dependencies: + '@microsoft/tsdoc': 0.17.0 + '@microsoft/tsdoc-config': 0.18.2 + '@rushstack/node-core-library': 5.24.2(@types/node@24.19.2) + transitivePeerDependencies: + - '@types/node' + optional: true + '@microsoft/api-extractor-model@7.33.13(@types/node@26.6.2)': dependencies: '@microsoft/tsdoc': 0.17.0 @@ -3489,6 +3529,25 @@ snapshots: - '@types/node' optional: true + '@microsoft/api-extractor@7.59.2(@types/node@24.19.2)': + dependencies: + '@microsoft/api-extractor-model': 7.33.13(@types/node@24.19.2) + '@microsoft/tsdoc': 0.17.0 + '@microsoft/tsdoc-config': 0.18.2 + '@rushstack/node-core-library': 5.24.2(@types/node@24.19.2) + '@rushstack/rig-package': 0.7.3 + '@rushstack/terminal': 0.24.5(@types/node@24.19.2) + '@rushstack/ts-command-line': 5.3.15(@types/node@24.19.2) + diff: 8.0.4 + minimatch: 10.2.3 + resolve: 1.22.12 + semver: 7.7.4 + source-map: 0.6.1 + typescript: 5.9.3 + transitivePeerDependencies: + - '@types/node' + optional: true + '@microsoft/api-extractor@7.59.2(@types/node@26.6.2)': dependencies: '@microsoft/api-extractor-model': 7.33.13(@types/node@26.6.2) @@ -3605,6 +3664,17 @@ snapshots: - '@vue/compiler-sfc' - vue + '@rslib/core@1.0.1(@microsoft/api-extractor@7.59.2(@types/node@24.19.2))(typescript@7.0.2)': + dependencies: + '@rsbuild/core': 2.2.8 + rsbuild-plugin-dts: 1.0.1(@microsoft/api-extractor@7.59.2(@types/node@24.19.2))(@rsbuild/core@2.2.8)(typescript@7.0.2) + optionalDependencies: + '@microsoft/api-extractor': 7.59.2(@types/node@24.19.2) + typescript: 7.0.2 + transitivePeerDependencies: + - '@module-federation/runtime-tools' + - core-js + '@rslib/core@1.0.1(@microsoft/api-extractor@7.59.2(@types/node@26.6.2))(typescript@6.0.3)': dependencies: '@rsbuild/core': 2.2.8 @@ -3704,6 +3774,20 @@ snapshots: optionalDependencies: '@rspack/core': 2.2.6(@swc/helpers@0.5.23) + '@rushstack/node-core-library@5.24.2(@types/node@24.19.2)': + dependencies: + ajv: 8.20.0 + ajv-draft-04: 1.0.0(ajv@8.20.0) + ajv-formats: 3.0.1(ajv@8.20.0) + fs-extra: 11.3.6 + import-lazy: 4.0.0 + jju: 1.4.0 + resolve: 1.22.12 + semver: 7.7.4 + optionalDependencies: + '@types/node': 24.19.2 + optional: true + '@rushstack/node-core-library@5.24.2(@types/node@26.6.2)': dependencies: ajv: 8.20.0 @@ -3718,6 +3802,11 @@ snapshots: '@types/node': 26.6.2 optional: true + '@rushstack/problem-matcher@0.2.1(@types/node@24.19.2)': + optionalDependencies: + '@types/node': 24.19.2 + optional: true + '@rushstack/problem-matcher@0.2.1(@types/node@26.6.2)': optionalDependencies: '@types/node': 26.6.2 @@ -3729,6 +3818,15 @@ snapshots: resolve: 1.22.12 optional: true + '@rushstack/terminal@0.24.5(@types/node@24.19.2)': + dependencies: + '@rushstack/node-core-library': 5.24.2(@types/node@24.19.2) + '@rushstack/problem-matcher': 0.2.1(@types/node@24.19.2) + supports-color: 8.1.1 + optionalDependencies: + '@types/node': 24.19.2 + optional: true + '@rushstack/terminal@0.24.5(@types/node@26.6.2)': dependencies: '@rushstack/node-core-library': 5.24.2(@types/node@26.6.2) @@ -3738,6 +3836,16 @@ snapshots: '@types/node': 26.6.2 optional: true + '@rushstack/ts-command-line@5.3.15(@types/node@24.19.2)': + dependencies: + '@rushstack/terminal': 0.24.5(@types/node@24.19.2) + '@types/argparse': 1.0.38 + argparse: 1.0.10 + string-argv: 0.3.2 + transitivePeerDependencies: + - '@types/node' + optional: true + '@rushstack/ts-command-line@5.3.15(@types/node@26.6.2)': dependencies: '@rushstack/terminal': 0.24.5(@types/node@26.6.2) @@ -3893,6 +4001,10 @@ snapshots: '@types/har-format@1.2.16': {} + '@types/node@24.19.2': + dependencies: + undici-types: 7.24.6 + '@types/node@26.6.2': dependencies: undici-types: 8.9.0 @@ -3977,6 +4089,34 @@ snapshots: vite: 8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1) vue: 3.5.43(typescript@6.0.3) + '@vitest/browser-playwright@5.0.1(playwright@1.63.0)(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1)': + dependencies: + '@vitest/browser': 5.0.1(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1) + '@vitest/mocker': 5.0.1(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + playwright: 1.63.0 + tinyrainbow: 3.1.1 + vitest: 5.0.1(@types/node@24.19.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + transitivePeerDependencies: + - bufferutil + - msw + - utf-8-validate + - vite + optional: true + + '@vitest/browser-playwright@5.0.1(playwright@1.63.0)(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1)': + dependencies: + '@vitest/browser': 5.0.1(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1) + '@vitest/mocker': 5.0.1(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + playwright: 1.63.0 + tinyrainbow: 3.1.1 + vitest: 5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + transitivePeerDependencies: + - bufferutil + - msw + - utf-8-validate + - vite + optional: true + '@vitest/browser-playwright@5.0.1(playwright@1.63.0)(vite@8.3.0(@types/node@26.6.2))(vitest@5.0.1)': dependencies: '@vitest/browser': 5.0.1(vite@8.3.0(@types/node@26.6.2))(vitest@5.0.1) @@ -3990,6 +4130,44 @@ snapshots: - utf-8-validate - vite + '@vitest/browser@5.0.1(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1)': + dependencies: + '@blazediff/core': 1.10.0 + '@vitest/mocker': 5.0.1(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + '@vitest/ui': 5.0.1(vitest@5.0.1) + '@vitest/utils': 5.0.1 + magic-string: 1.4.1 + pngjs: 7.0.0 + sirv: 3.0.2 + tinyrainbow: 3.1.1 + vitest: 5.0.1(@types/node@24.19.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + ws: 8.21.3 + transitivePeerDependencies: + - bufferutil + - msw + - utf-8-validate + - vite + optional: true + + '@vitest/browser@5.0.1(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1)': + dependencies: + '@blazediff/core': 1.10.0 + '@vitest/mocker': 5.0.1(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + '@vitest/ui': 5.0.1(vitest@5.0.1) + '@vitest/utils': 5.0.1 + magic-string: 1.4.1 + pngjs: 7.0.0 + sirv: 3.0.2 + tinyrainbow: 3.1.1 + vitest: 5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + ws: 8.21.3 + transitivePeerDependencies: + - bufferutil + - msw + - utf-8-validate + - vite + optional: true + '@vitest/browser@5.0.1(vite@8.3.0(@types/node@26.6.2))(vitest@5.0.1)': dependencies: '@blazediff/core': 1.10.0 @@ -4008,6 +4186,24 @@ snapshots: - utf-8-validate - vite + '@vitest/mocker@5.0.1(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))': + dependencies: + '@jridgewell/trace-mapping': 0.3.31 + '@vitest/spy': 5.0.1 + estree-walker: 3.0.3 + magic-string: 1.4.1 + optionalDependencies: + vite: 8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1) + + '@vitest/mocker@5.0.1(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))': + dependencies: + '@jridgewell/trace-mapping': 0.3.31 + '@vitest/spy': 5.0.1 + estree-walker: 3.0.3 + magic-string: 1.4.1 + optionalDependencies: + vite: 8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1) + '@vitest/mocker@5.0.1(vite@8.3.0(@types/node@26.6.2))': dependencies: '@jridgewell/trace-mapping': 0.3.31 @@ -4031,7 +4227,7 @@ snapshots: pathe: 2.0.3 sirv: 3.0.2 tinyrainbow: 3.1.1 - vitest: 5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)) + vitest: 5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) '@vitest/utils@5.0.1': dependencies: @@ -4871,6 +5067,14 @@ snapshots: '@rolldown/binding-win32-arm64-msvc': 1.2.9 '@rolldown/binding-win32-x64-msvc': 1.2.9 + rsbuild-plugin-dts@1.0.1(@microsoft/api-extractor@7.59.2(@types/node@24.19.2))(@rsbuild/core@2.2.8)(typescript@7.0.2): + dependencies: + '@ast-grep/napi': 0.45.3 + '@rsbuild/core': 2.2.8 + optionalDependencies: + '@microsoft/api-extractor': 7.59.2(@types/node@24.19.2) + typescript: 7.0.2 + rsbuild-plugin-dts@1.0.1(@microsoft/api-extractor@7.59.2(@types/node@26.6.2))(@rsbuild/core@2.2.8)(typescript@6.0.3): dependencies: '@ast-grep/napi': 0.45.3 @@ -5052,6 +5256,8 @@ snapshots: uhyphen@0.2.0: {} + undici-types@7.24.6: {} + undici-types@8.9.0: optional: true @@ -5114,6 +5320,20 @@ snapshots: optionalDependencies: typescript: 7.0.2 + vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1): + dependencies: + lightningcss: 1.33.0 + picomatch: 4.0.7 + postcss: 8.5.28 + rolldown: 1.2.9 + tinyglobby: 0.2.17 + optionalDependencies: + '@types/node': 24.19.2 + esbuild: 0.28.2 + fsevents: 2.3.3 + jiti: 2.7.0 + yaml: 2.9.1 + vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1): dependencies: lightningcss: 1.33.0 @@ -5128,6 +5348,52 @@ snapshots: jiti: 2.7.0 yaml: 2.9.1 + vitest@5.0.1(@types/node@24.19.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)): + dependencies: + '@types/chai': 5.2.3 + '@vitest/mocker': 5.0.1(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + chai: 6.2.2 + es-module-lexer: 2.3.2 + expect-type: 1.4.0 + magic-string: 1.4.1 + obug: 2.2.1 + picomatch: 4.0.7 + std-env: 4.2.0 + tinybench: 6.1.4 + tinyexec: 1.3.0 + tinyglobby: 0.2.17 + vite: 8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1) + why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 24.19.2 + '@vitest/browser-playwright': 5.0.1(playwright@1.63.0)(vite@8.3.0(@types/node@24.19.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1) + '@vitest/ui': 5.0.1(vitest@5.0.1) + transitivePeerDependencies: + - msw + + vitest@5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)): + dependencies: + '@types/chai': 5.2.3 + '@vitest/mocker': 5.0.1(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1)) + chai: 6.2.2 + es-module-lexer: 2.3.2 + expect-type: 1.4.0 + magic-string: 1.4.1 + obug: 2.2.1 + picomatch: 4.0.7 + std-env: 4.2.0 + tinybench: 6.1.4 + tinyexec: 1.3.0 + tinyglobby: 0.2.17 + vite: 8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1) + why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 26.6.2 + '@vitest/browser-playwright': 5.0.1(playwright@1.63.0)(vite@8.3.0(@types/node@26.6.2)(esbuild@0.28.2)(jiti@2.7.0)(yaml@2.9.1))(vitest@5.0.1) + '@vitest/ui': 5.0.1(vitest@5.0.1) + transitivePeerDependencies: + - msw + vitest@5.0.1(@types/node@26.6.2)(@vitest/browser-playwright@5.0.1)(@vitest/ui@5.0.1)(vite@8.3.0(@types/node@26.6.2)): dependencies: '@types/chai': 5.2.3 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index b50600d..c7b0281 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -24,6 +24,7 @@ catalogs: # What every workspace builds with. core: typescript: 7.0.2 + valibot: 1.5.0 # TypeScript 7 ships only the native `tsc`: its package no longer exports the JS compiler # API. player-vue's toolchain still loads that API (vue-tsc, and @vue/compiler-sfc through diff --git a/turbo.json b/turbo.json index 7c160fa..eb43861 100644 --- a/turbo.json +++ b/turbo.json @@ -16,6 +16,14 @@ "test": { "dependsOn": ["^build"] }, + "@rustrak/openshowcase-recorder#build": { + "dependsOn": ["^build", "@rustrak/openshowcase-extension#build"], + "inputs": ["$TURBO_DEFAULT$", ".env*"], + "outputs": ["dist/**"] + }, + "@rustrak/openshowcase-recorder#test": { + "dependsOn": ["^build", "@rustrak/openshowcase-extension#build"] + }, "size": { "dependsOn": ["build"] }, From c0119a58ddd770768d536127e92f8e293773f7a5 Mon Sep 17 00:00:00 2001 From: Abian Suarez Date: Mon, 19 Oct 2026 19:55:50 +0100 Subject: [PATCH 3/3] chore: add changeset for the agent recorder --- .changeset/agent-recorder.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 .changeset/agent-recorder.md diff --git a/.changeset/agent-recorder.md b/.changeset/agent-recorder.md new file mode 100644 index 0000000..cf7ed93 --- /dev/null +++ b/.changeset/agent-recorder.md @@ -0,0 +1,9 @@ +--- +"@rustrak/openshowcase-recorder": minor +"@rustrak/openshowcase-extension": minor +--- + +AI agents can record demos: + +- New `@rustrak/openshowcase-recorder` CLI: plays a JSON flow in headless Chromium while the extension records it, with a cursor that moves like a hand, targets scrolled into view and waits cut out, and writes an `.osrec` file. The flow directs the camera (`zoom`, `hotspot: false`). `inspect` shows where a flow ends up as an accessibility tree, `login` saves a session once, `open` reviews a recording in a separate browser with the editor, `install-skill` adds the Claude Code skill to a project. +- The extension imports `.osrec` recordings (popup file icon, or the editor's empty screen) and applies their direction: tooltips, zooms, quiet clicks and cuts.