SCAL-295232 Add Liveboard schedule webhook example

[prathum-pandey-ts]

Adds rest-api/liveboard-schedule-webhook: a receiver for ThoughtSpot LIVEBOARD_SCHEDULE webhooks that uploads the exported files to Google Drive. Follows the KPI monitor webhook example in starters/kpi-monitor.

• src/main.ts: Express receiver for direct (multipart) and S3 storage deliveries; bearer token check, ack within 5 s, dedupe on msgUniqueId, upload to Drive (or ./out)
• src/setup.ts: storage-config, create-webhook, route-schedules and validate via @thoughtspot/rest-api-sdk
• src/demo.ts: npm run dev (StackBlitz demo) and npm test (smoke run) using sample deliveries from the payload docs
• package-lock.json resolves to the public npm registry

[snyk-io]

⛔ Snyk checks have failed. 3 issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (3)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
⛔	Code Security	0	3	0	0	3 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[ashoka1981]

Code Review: SCAL-295232 — Liveboard Schedule Webhook Example

Overall: This is a well-structured, well-documented developer example that clearly demonstrates the end-to-end flow of receiving ThoughtSpot LIVEBOARD_SCHEDULE webhook events and uploading files to Google Drive. The README is excellent — the Mermaid sequence diagram, delivery-mode comparison table, step-by-step setup guide, and "Not covered" / "Before you start" caveats are exactly what the Jira ticket asks for. The demo harness (npm run dev) that works without any cloud credentials is a strong addition.

There are a few issues to address before merging, primarily around a path validation pattern that will be copied by adopters, and some robustness gaps in the queue/dedup design that should at least be documented.

---

🔴 Blocking Issues

#	File	Issue
1	src/main.ts	tempFile() path traversal guard is fragile — uses dirname(dirname()) depth heuristic instead of the idiomatic startsWith(TMP_ROOT + path.sep) check. Breaks on macOS (symlink /tmp → /private/tmp) and is non-obvious to readers who will copy this pattern.
2	src/main.ts	Unbounded seen Set — grows without limit for the process lifetime. Any developer who adapts this to a long-running deployment will silently leak memory. Needs at minimum a TTL eviction or a prominent code comment + README caveat.
3	src/main.ts	Queue error propagation — if processDelivery throws an unhandled rejection, the promise chain can break silently, stalling all future deliveries. Each link needs a .catch() wrapper.

🟡 Suggestions

#	File	Issue
4	src/main.ts	seen.delete(key) in the catch block creates a window where a retry can be admitted while the failed delivery's queue slot is still settling — potential duplicate processing. Worth a code comment documenting this limitation.
5	src/main.ts	fetchStored S3 download has no size cap — unlike the 25 MB limit on multipart bodies. A comment noting this or adding a size check would help adopters.
6	src/main.ts	takeManifest — JSON.parse without try/catch on attachment content. If a non-JSON file happens to have application/json content type, this throws a 400 → ThoughtSpot retries → permanent failure loop.
7	src/main.ts	S3Client and Google Drive client are re-instantiated per file. For a sample this is fine, but a comment suggesting singleton/lazy patterns for production would help adopters.
8	src/setup.ts	as any casts suppress type checking silently. Use @ts-expect-error with a comment explaining the SDK type gap, so it fails loudly when types are added.
9	README.md	Explicitly state that the in-memory dedup set and queue do not survive process restarts and are single-instance only. The "Not covered" section is the right place.
10	README.md	Warn that leaving RECEIVER_TOKEN unset makes the endpoint publicly writable.

ℹ️ Nits

#	File	Issue
11	src/main.ts	takeManifest mutates the caller's attachments array via splice — a surprising side effect. Consider returning the filtered list instead.
12	src/main.ts	idle() captures a snapshot of queue at call time; a delivery appended after the snapshot but before the await resolves makes await idle() unreliable in tests. Worth a JSDoc comment.
13	src/main.ts	Consider adding section-boundary comments (e.g., // --- STORAGE ADAPTER ---) to make extension points visually obvious for adopters who want to swap S3 for GCS or Drive for another destination.

---

Jira Scope Coverage

The PR covers LIVEBOARD_SCHEDULE events, AWS S3 storage, the receiver-server pattern, Google Drive as downstream, and beta/enablement caveats — all core requirements from SCAL-295232. GCP/GCS storage fetch is declared out of scope in the README's "Not covered" section, which is acceptable given the Jira scope is large. The setup script and README walk the complete path end-to-end, which directly addresses the field feedback that the confusion is about the flow, not the API.

[ashoka1981]

🔴 Blocking — Unbounded memory growth

This Set grows without limit for the process lifetime. Any developer adapting this for a long-running deployment will silently leak memory.

Suggested fix: add a TTL-based eviction (e.g., a Map<string, number> with timestamps, purged periodically), or at minimum add a prominent comment:

// ⚠️ In-memory only — grows without bound, lost on restart.
// For production, replace with Redis SET NX EX or a persistent store.
const seen = new Set<string>();

Also add this caveat to the README's "Not covered" section.

[ashoka1981]

🔴 Blocking — Unhandled rejection can poison the queue

If processDelivery throws an error that escapes the try/catch (e.g., a synchronous throw before the first await), the promise chain breaks and all future deliveries silently stall forever.

Wrap each link defensively:

queue = queue.then(() =>
  processDelivery(event, key, files, stored, dir)
    .catch(err => console.error(`[${event.eventId}] unhandled:`, err))
);

[ashoka1981]

🟡 Suggestion — seen.delete(key) creates a duplicate-processing window

Timeline: (1) Request A admitted, seen.add(key). (2) processDelivery(A) starts, yields at an await. (3) Request B (retry) arrives — correctly rejected as duplicate. (4) processDelivery(A) fails, seen.delete(key). (5) Request C (another retry) arrives — admitted and queued, potentially overlapping with A's cleanup.

This is acceptable for a sample app, but add a comment documenting the limitation:

// NOTE: A retry arriving after this delete but before the queue settles
// can cause duplicate processing. For production, use an atomic
// check-and-set in a persistent store instead of in-memory Set.

[ashoka1981]

🟡 Suggestion — No size limit on S3 downloads

The multipart path enforces MAX_BYTES via busboy, but the S3 fetch path pipes the object body to disk with no size constraint. A misconfigured or malicious S3 object could fill the temp filesystem.

Consider adding a comment noting the gap, or wrapping the stream with a size-limiting transform.

[ashoka1981]

🟡 Suggestion — JSON.parse without try/catch risks permanent retry loop

If a non-JSON attachment happens to have application/json as its content type, this JSON.parse throws → the outer catch returns 400 → ThoughtSpot retries → same 400 forever.

Wrap in try/catch and continue on parse failure:

let json;
try { json = JSON.parse(await readFile(file.path, 'utf8')); }
catch { continue; }

[ashoka1981]

🟡 Suggestion — as any sets a poor precedent for adopters

Developer examples get copied verbatim. as any silently disables all type checking. If the SDK types are incomplete, prefer @ts-expect-error with a comment explaining why:

// @ts-expect-error — SDK types don't yet include storage_destination
client.createWebhookConfiguration({ ... });

This will fail loudly (and usefully) once the SDK adds the types.

[ashoka1981]

🟡 Suggestion — Re-instantiating SDK clients per file

Both S3Client and the Drive client are created fresh on every file upload. Node.js caches the dynamic import() after the first call, but the client constructor (credential loading, HTTP agent pool) is re-executed each time.

For a sample this is acceptable, but consider adding a comment:

// TIP: In production, hoist the S3Client to module scope (lazy singleton)
// to reuse the HTTP connection pool and avoid repeated credential loading.

[ashoka1981]

ℹ️ Nit — idle() has snapshot semantics

idle() returns the current value of queue, but a new delivery appended after the call (but before the await resolves) won't be waited on. This makes await idle() unreliable if the server is still accepting requests.

Consider adding a JSDoc comment:

/** Resolves when all currently-queued deliveries complete.
 *  Not safe if new requests can still arrive — close the server first. */
export const idle = () => queue;

[ashoka1981]

ℹ️ Nit — Add a section-boundary comment for extensibility

The README's "Not covered" section tells adopters to add GCS support on top of this function. Make the extension point visually obvious:

// --- STORAGE ADAPTER --------------------------------------------------------
// Replace this function to fetch from a different source (GCS, Azure Blob).
// Input:  StoredFile with bucket name and object key from the delivery manifest
// Output: file content written to `dest` under TMP_ROOT
// ---------------------------------------------------------------------------