feat: let an AI agent record cinematic demos (recorder CLI + skill)

.changeset/agent-recorder.md

@@ -0,0 +1,9 @@
+---
+"@rustrak/openshowcase-recorder": minor
+"@rustrak/openshowcase-extension": minor
+---
+
+AI agents can record demos:
+
+- New `@rustrak/openshowcase-recorder` CLI: plays a JSON flow in headless Chromium while the extension records it, with a cursor that moves like a hand, targets scrolled into view and waits cut out, and writes an `.osrec` file. The flow directs the camera (`zoom`, `hotspot: false`). `inspect` shows where a flow ends up as an accessibility tree, `login` saves a session once, `open` reviews a recording in a separate browser with the editor, `install-skill` adds the Claude Code skill to a project.
+- The extension imports `.osrec` recordings (popup file icon, or the editor's empty screen) and applies their direction: tooltips, zooms, quiet clicks and cuts.

CLAUDE.md

@@ -22,6 +22,7 @@ Each workspace has its own `CLAUDE.md` with its build, test and gotcha notes. Re
 - **The exported bundle is pure data:** `steps.json` + `assets/*.avif` (photo frames, encoded at export) + `assets/recording.mp4` (only if there are video steps: just the ranges they play, re-encoded at export; `.webm` if H.264 encoding is unavailable). It never ships a player runtime. The player is an npm component in the host app (`player-react` / `player-vue`).
 - **`packages/schema` is the single source of truth** for the demo format. Read `packages/schema/src/index.ts` for the current shape rather than trusting any copy of it.
 - Pipeline: `content.ts` markers + continuous tab video → `apps/extension/lib/segments.ts` derives steps (click → photo step with hotspot, activity between clicks → video clip) → editor → export (`editor/lib/export-video.ts` re-encodes the played ranges) → exporter → player.
+- **Agent recording** (`packages/recorder`): a CLI plays a JSON flow in headless Chromium with the real extension loaded, which records it through the same pipeline. The recording travels as an `.osrec` file (`apps/extension/lib/recording-file.ts`) that the editor imports. The human always reviews in the editor; the recorder never exports a demo itself.
 
 ## Conventions
 

README.md

@@ -41,6 +41,20 @@ powers the interactive demos in the [Rustrak docs](https://rustrak.github.io/rus
 
 <img width="2000" height="1000" alt="export" src="https://github.com/user-attachments/assets/e93fa6a5-1197-462e-ad87-e28adbb459de" />
 
+## Let an AI agent record it
+
+`@rustrak/openshowcase-recorder` lets an agent like Claude Code record the demo
+for you. It scripts the flow, checks it against your app, and records it in a
+headless Chromium with the extension, without touching your browser. You import
+the result in the editor and polish what doesn't convince you.
+
+```sh
+npx @rustrak/openshowcase-recorder install-skill   # adds the Claude Code skill
+```
+
+Then ask: *"record a demo of creating an invoice"*. See the
+[recorder's README](./packages/recorder/README.md) to run it yourself.
+
 ## Embed a demo
 
 Put the `.openshowcase` file in your site's static files and point the player
@@ -77,6 +91,7 @@ apps/extension/        Chrome extension: recorder + editor (WXT + React)
 packages/schema/       Demo format: Valibot schema + TypeScript types
 packages/player-core/  Framework-agnostic playback engine
 packages/exporter/     Builds the exported bundle in the browser
+packages/recorder/     CLI + agent skill: records a scripted flow headlessly
 adapters/player-react/ <InteractiveDemo /> for React
 adapters/player-vue/   <InteractiveDemo /> for Vue
 ```

apps/extension/CLAUDE.md

@@ -4,7 +4,7 @@ WXT + React Chrome extension (MV3) with two jobs: **capture** (continuous tab vi
 
 ## Commands
 
-- `pnpm dev` / `pnpm build` (`:firefox` variants exist). Test manually by loading `.output/chrome-mv3` unpacked. There is no E2E test for recording.
+- `pnpm dev` / `pnpm build` (`:firefox` variants exist). Test manually by loading `.output/chrome-mv3` unpacked. The recording E2E test lives in `packages/recorder` (`src/__test__/e2e`): it records a flow with this build in headless Chromium and imports the result in the editor.
 - `pnpm test`: Vitest in Node with WXT's `fakeBrowser` (in-memory `browser.*`/`chrome.*`). No real DOM, canvas or IndexedDB.
 - `postinstall` runs `wxt prepare` to generate `.wxt/` types.
 
@@ -19,6 +19,14 @@ WXT + React Chrome extension (MV3) with two jobs: **capture** (continuous tab vi
 - Chrome burns the real cursor into tab captures. `lib/segments.ts` only keeps a video clip between clicks when there is activity (scroll/type/drag) or the gap is ≥3s. Otherwise the steps go photo → photo and the hotspot glides. Fast-forwarding idle stretches was tried and rejected.
 - The content script checks `ctx.isInvalid` (not `isInvalidated`) and detaches listeners in `ctx.onInvalidated`. An orphaned script after an extension reload would otherwise throw "Extension context invalidated" forever.
 
+## Recordings from the recorder CLI
+
+- `lib/automation.ts` is the API `packages/recorder` calls over CDP (`globalThis.openshowcase` in the service worker): `start` → `stop` (waits for the save) → `exportFile` (with the flow's direction) → `readChunk` (base64). Its shape and `Direction` (`lib/direction.ts`) are mirrored in the recorder's `browser.ts`; change both.
+- Direction: `cueClicks` puts the flow's `label`/`zoom`/`hotspot` on the matching click markers; `RecordingRecord.direction` keeps `cuts` and `zooms`. `buildStepPlan` uses them only when present (hand-made recordings are unchanged): a `hotspot: false` click is activity, not a photo; cuts split clips (jump cuts, and don't count toward the 3s long-gap rule); a zoom splits its clip into wide → zoomed (padded 0.5s) → wide contiguous pieces.
+- A session started that way has `automated: true`, so saving it opens no editor tab.
+- `.osrec` (`lib/recording-file.ts`): `OSREC` + version byte, uint32 LE header length, JSON header (recording meta, markers, optional direction), then the webm. The editor imports it from its empty screen (`ImportScreen`, also reached from the popup's file icon), or on load from a file packaged in the extension (`editor.html?import=imports/<name>.osrec`, `core/import-source.ts`: what the recorder's review browser uses), into IndexedDB under a new id.
+- `MarkerRecord.label` (the flow's `tooltip`) wins over `elementText` for the hotspot's tooltip.
+
 ## Testing patterns
 
 - Code touching APIs `fakeBrowser` lacks (`tabCapture`, `offscreen`, `action`, IndexedDB) takes injectable deps. `startRecording(deps?: StartRecordingDeps)` in `background.ts` is the reference.

apps/extension/entrypoints/__test__/unit/background.test.ts

@@ -1,6 +1,7 @@
 import { afterEach, beforeEach, describe, expect, it } from "vitest";
 import { fakeBrowser } from "wxt/testing/fake-browser";
 import {
+  handleRecordingSaved,
   type StartRecordingDeps,
   startRecording,
   stopRecording,
@@ -92,6 +93,49 @@ describe("startRecording", () => {
     expect(await recordingSession.getValue()).toMatchObject({ active: false });
     expect(counterCalls).toContain(null);
   });
+
+  it("marks a session started by automation, so saving it opens no editor", async () => {
+    const unlisten = onMessage("offscreenStart", () => {});
+    await startRecording(fakeDeps(), { automated: true });
+    unlisten();
+    expect(await recordingSession.getValue()).toMatchObject({
+      active: true,
+      automated: true,
+    });
+  });
+});
+
+describe("handleRecordingSaved", () => {
+  const savedDeps = {
+    setCounterIcon: async () => {},
+    closeOffscreenDocumentIfOpen: async () => {},
+  };
+
+  it("ends the session and opens the editor on the recording", async () => {
+    await recordingSession.setValue({ active: true, recordingId: "rec-1" });
+    await handleRecordingSaved("rec-1", savedDeps);
+
+    expect(await recordingSession.getValue()).toMatchObject({ active: false });
+    const tabs = await fakeBrowser.tabs.query({});
+    expect(tabs.map((t) => t.url)).toContainEqual(
+      expect.stringContaining("/editor.html?recordingId=rec-1"),
+    );
+  });
+
+  it("opens no editor for an automated recording (the recorder CLI takes it)", async () => {
+    await recordingSession.setValue({
+      active: true,
+      recordingId: "rec-1",
+      automated: true,
+    });
+    await handleRecordingSaved("rec-1", savedDeps);
+
+    expect(await recordingSession.getValue()).toMatchObject({ active: false });
+    const tabs = await fakeBrowser.tabs.query({});
+    expect(tabs.map((t) => t.url)).not.toContainEqual(
+      expect.stringContaining("editor.html"),
+    );
+  });
 });
 
 describe("stopRecording", () => {

apps/extension/entrypoints/background.ts

@@ -1,5 +1,8 @@
+import { createAutomation } from "@/lib/automation";
 import {
   addMarker,
+  getMarkersByRecording,
+  getRecording,
   markRecordingInterrupted,
   markRecordingStarted,
   createRecording as saveRecording,
@@ -24,6 +27,17 @@
   onMessage("recordingSaved", ({ data }) =>
     handleRecordingSaved(data.recordingId),
   );
+
+  // The recorder CLI (packages/recorder) drives recordings over CDP, through this global:
+  // there's no toolbar click in its automated Chromium. Only reachable from DevTools/CDP.
+  Object.assign(globalThis, {
+    openshowcase: createAutomation({
+      startRecording: () => startRecording(undefined, { automated: true }),
+      stopRecording,
+      getRecording,
+      getMarkers: getMarkersByRecording,
+    }),
+  });
 });
 
 async function ensureOffscreenDocument(): Promise<void> {
@@ -95,6 +109,7 @@
 
 export async function startRecording(
   deps: StartRecordingDeps = defaultStartRecordingDeps,
+  { automated = false }: { automated?: boolean } = {},
 ): Promise<void> {
   const tab = await deps.getActiveTab();
   if (!tab?.id) return;
@@ -111,6 +126,7 @@
       recordingId,
       tabId: tab.id,
       clickCount: 0,
+      automated,
     });
     await deps.setCounterIcon(0);
 
@@ -143,7 +159,7 @@
  */
 function drawCounter(size: number, count: number): ImageData {
   const canvas = new OffscreenCanvas(size, size);
   const ctx = canvas.getContext("2d")!;
   ctx.clearRect(0, 0, size, size);
   ctx.fillStyle = "#DC2626";
   ctx.beginPath();
@@ -212,13 +228,22 @@
   });
 }
 
-async function handleRecordingSaved(recordingId: string): Promise<void> {
+export interface RecordingSavedDeps {
+  setCounterIcon: (count: number | null) => Promise<void>;
+  closeOffscreenDocumentIfOpen: () => Promise<void>;
+}
+
+export async function handleRecordingSaved(
+  recordingId: string,
+  deps: RecordingSavedDeps = { setCounterIcon, closeOffscreenDocumentIfOpen },
+): Promise<void> {
   const state = await recordingSession.getValue();
   if (state.recordingId === recordingId) {
     await patchRecordingSessionState({ active: false });
   }
-  await setCounterIcon(null);
-  await closeOffscreenDocumentIfOpen();
+  await deps.setCounterIcon(null);
+  await deps.closeOffscreenDocumentIfOpen();
+  if (state.automated && state.recordingId === recordingId) return;
   // When the recording ends, open the editor directly.
   await browser.tabs.create({
     url: browser.runtime.getURL(`/editor.html?recordingId=${recordingId}`),

apps/extension/entrypoints/editor/App.tsx

@@ -1,6 +1,7 @@
 import { Toaster } from "@/components/ui/sonner";
 import { TooltipProvider } from "@/components/ui/tooltip";
 import { useColorScheme } from "@/hooks/use-color-scheme";
+import { ImportScreen } from "./components/ImportScreen";
 import { LoadingScreen, MessageScreen } from "./components/Screens";
 import { Editor } from "./Editor";
 import { useRecording } from "./hooks/use-recording";
@@ -15,12 +16,7 @@ function App() {
 
   return (
     <TooltipProvider delay={500}>
-      {state.status === "missing-id" && (
-        <MessageScreen
-          title="No recording selected"
-          text="Record a tab from the OpenShowcase toolbar button — the editor opens by itself when you stop."
-        />
-      )}
+      {state.status === "missing-id" && <ImportScreen />}
       {state.status === "loading" && <LoadingScreen />}
       {state.status === "empty" && (
         <MessageScreen

apps/extension/entrypoints/editor/components/ImportScreen.tsx

@@ -0,0 +1,64 @@
+import { FileUp, LoaderCircle } from "lucide-react";
+import { useRef } from "react";
+import { Button } from "@/components/ui/button";
+import { RECORDING_FILE_EXTENSION } from "@/lib/recording-file";
+import { useFileDrop } from "../../playground/hooks/use-file-drop";
+import { importPathFromQuery } from "../core/import-source";
+import { useRecordingImport } from "../hooks/use-recording-import";
+import { MessageScreen } from "./Screens";
+
+/** The editor with no recording: record one, or import a file made by the recorder CLI. */
+export function ImportScreen() {
+  const { importing, importFile } = useRecordingImport(
+    importPathFromQuery(window.location.search),
+  );
+  const dragging = useFileDrop(importFile);
+  const inputRef = useRef<HTMLInputElement>(null);
+
+  return (
+    <div className="relative h-full">
+      <MessageScreen
+        title="No recording selected"
+        text={
+          <>
+            Record a tab from the OpenShowcase toolbar button, or import a{" "}
+            <code className="font-mono">{RECORDING_FILE_EXTENSION}</code>{" "}
+            recording made by the recorder CLI.
+          </>
+        }
+        action={
+          <>
+            <Button
+              className="gap-1.5"
+              disabled={importing}
+              onClick={() => inputRef.current?.click()}
+            >
+              {importing ? (
+                <LoaderCircle className="size-3.5 animate-spin" />
+              ) : (
+                <FileUp className="size-3.5" />
+              )}
+              Import a recording
+            </Button>
+            <input
+              ref={inputRef}
+              type="file"
+              accept={RECORDING_FILE_EXTENSION}
+              hidden
+              onChange={(e) => {
+                const file = e.target.files?.[0];
+                if (file) importFile(file);
+                e.target.value = "";
+              }}
+            />
+          </>
+        }
+      />
+      {dragging && (
+        <div className="pointer-events-none absolute inset-2 flex items-center justify-center rounded-xl border-2 border-dashed border-primary bg-window/70 text-ui font-medium animate-in fade-in-0 duration-150">
+          Drop to import this recording
+        </div>
+      )}
+    </div>
+  );
+}

apps/extension/entrypoints/editor/components/Screens.tsx

@@ -37,9 +37,11 @@ function Shimmer({ className }: { className: string }) {
 export function MessageScreen({
   title,
   text,
+  action,
 }: {
   title: string;
   text: React.ReactNode;
+  action?: React.ReactNode;
 }) {
   return (
     <div className="flex h-full items-center justify-center bg-window p-8">
@@ -53,6 +55,7 @@ export function MessageScreen({
             {text}
           </p>
         </div>
+        {action}
         <div className="mt-6 flex flex-col items-center gap-2">
           <OpenShowcaseWordmark
             still

apps/extension/entrypoints/editor/core/__test__/unit/import-source.test.ts

@@ -0,0 +1,24 @@
+import { describe, expect, it } from "vitest";
+import { importPathFromQuery } from "../../import-source";
+
+describe("importPathFromQuery", () => {
+  it("reads a recording the recorder CLI put in the extension's imports folder", () => {
+    expect(importPathFromQuery("?import=imports/factura-venta.osrec")).toBe(
+      "imports/factura-venta.osrec",
+    );
+  });
+
+  it("ignores pages without one", () => {
+    expect(importPathFromQuery("?recordingId=abc")).toBeUndefined();
+    expect(importPathFromQuery("")).toBeUndefined();
+  });
+
+  it("refuses anything outside the imports folder or not a recording", () => {
+    expect(importPathFromQuery("?import=../secret.osrec")).toBeUndefined();
+    expect(importPathFromQuery("?import=imports/../x.osrec")).toBeUndefined();
+    expect(
+      importPathFromQuery("?import=https://evil.example/x.osrec"),
+    ).toBeUndefined();
+    expect(importPathFromQuery("?import=imports/x.js")).toBeUndefined();
+  });
+});

apps/extension/entrypoints/editor/core/__test__/unit/step-editor.test.ts

@@ -287,6 +287,14 @@ describe("defaultPanZoom", () => {
     expect(pz.y).toBeCloseTo(1 - half);
   });
 
+  it("takes another scale, still kept inside the image", () => {
+    expect(StepEditor.defaultPanZoom({ x: 0.5, y: 0.98 }, 2.5)).toEqual({
+      x: 0.5,
+      y: 0.8,
+      scale: 2.5,
+    });
+  });
+
   it("centers on the image when there is no focus point", () => {
     expect(StepEditor.defaultPanZoom(undefined)).toMatchObject({
       x: 0.5,

apps/extension/entrypoints/editor/core/import-source.ts

@@ -0,0 +1,11 @@
+/**
+ * A recording the recorder CLI's `open` command copied into this extension's `imports/`
+ * folder, named in the editor's URL (`?import=imports/<name>.osrec`). Only plain file names
+ * in that folder: the editor fetches it from its own extension origin.
+ */
+export function importPathFromQuery(search: string): string | undefined {
+  const path = new URLSearchParams(search).get("import");
+  return path && /^imports\/[\w.-]+\.osrec$/.test(path) && !path.includes("..")
+    ? path
+    : undefined;
+}

apps/extension/entrypoints/editor/core/step-editor.ts

@@ -84,7 +84,7 @@
   duration: number,
 ): EditableStep[] {
   const step = steps.find((s) => s.id === id);
   if (!step || step.data.kind !== "photo") return steps;
   const source = step.data.sourceTime;
   return patchStep(steps, id, (s) => ({
     ...s,
@@ -110,7 +110,7 @@
 ): SplitVideoResult | undefined {
   const index = steps.findIndex((s) => s.id === id);
   const step = steps[index];
   if (!step || step.data.kind !== "video") return undefined;
   const first: EditableStep = {
     id: createId(),
     data: { ...step.data, trimEnd: time },
@@ -189,8 +189,8 @@
  */
 export function defaultPanZoom(
   focus: { x: number; y: number } | undefined,
+  scale = DEFAULT_ZOOM_SCALE,
 ): PanZoom {
-  const scale = DEFAULT_ZOOM_SCALE;
   const half = 0.5 / scale;
   const clampToImage = (v: number) => Math.min(1 - half, Math.max(half, v));
   return {