diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b9fc8f8..5ac9cfd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,7 +32,7 @@ jobs: plugins: uses: rocketflare-dev/rocketflare/.github/workflows/plugin-ci.yml@main with: - plugin_subdirs: '["plugins/analytics", "plugins/web-knowledge", "plugins/connectors", "plugins/m365", "plugins/issue-reports"]' + plugin_subdirs: '["plugins/analytics", "plugins/web-knowledge", "plugins/connectors", "plugins/m365", "plugins/issue-reports", "plugins/inbox", "plugins/freeagent"]' # On a kit-released dispatch, prove against the ref that was just cut — used verbatim, with # the range NOT consulted. That is what makes the notification worth sending: resolving the # range again would pick the same versions as the last run and prove nothing new. Empty on diff --git a/README.md b/README.md index 282d670..2b72fec 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,8 @@ sees what and its known gaps. They are also published at | [**connectors**](plugins/connectors/README.md) | Organisation connections: admin consent once, then directory and calendar sync every 15 minutes; Settings → Connections, My calendar, CLI and the `connectors` setup skill | kit 0.13.0 and a provider | | [**m365**](plugins/m365/README.md) | Microsoft 365 as a `connectors` provider: Entra admin consent, app-only Graph delta sync of users, groups and calendars | kit 0.12.0 and `connectors` | | [**issue-reports**](plugins/issue-reports/README.md) | Ideas & issues: anyone talks a bug or an idea through with an agent that asks until it is clear, checks for a duplicate, and logs it as a GitHub issue (or a comment) once they approve the draft | kit 0.21.0 and a GitHub token | +| [**inbox**](plugins/inbox/README.md) | Inbound email through Cloudflare Email Routing: per-organisation address and sender allow-list, SPF/DKIM/DMARC checks, parsed bodies and attachments, Gmail/Proton forwarding-confirmation codes, an Inbox page, CLI, read-only agent tools, and an `inbox:consumers` extension other plugins build on | kit 0.22.0 and Email Routing on the domain | +| [**freeagent**](plugins/freeagent/README.md) | FreeAgent (UK accounting) per organisation: OAuth connection, a 30-minute incremental mirror of bank transactions, explanations, categories, contacts, bills and invoices, Settings → FreeAgent and a FreeAgent page, read tools plus approval-gated write tools for agents, analytics cubes, CLI, a service API and a `freeagent:sync-listeners` extension for plugins built on top | kit 0.21.0 and a FreeAgent app | ## Installing one diff --git a/plugins/freeagent/README.md b/plugins/freeagent/README.md new file mode 100644 index 0000000..db2f65e --- /dev/null +++ b/plugins/freeagent/README.md @@ -0,0 +1,103 @@ +# freeagent + +> Connect each organisation's FreeAgent company (UK accounting) and get its books mirrored into +> your app — for pages, agents, analytics and the plugins you build on top. + +An admin approves your FreeAgent app once, over OAuth. Every 30 minutes the plugin syncs the +company's bank accounts, bank transactions, explanations, categories, contacts, bills and invoices +into tenant-scoped `freeagent_*` tables, incrementally. Agents read that mirror and can propose an +explanation or a bill for a person to approve. Other plugins — a receipts matcher, say — explain +transactions and attach receipts through an exported service API, and hear about every sync through +the `freeagent:sync-listeners` extension. + +## What it adds + +| Area | What you get | +|---|---| +| Connection | One FreeAgent company per organisation. OAuth approval through `POST /api/freeagent/connect` and the public callback `/api/hooks/freeagent/callback` (signed state). Refresh and access tokens stored sealed. A refused refresh token marks the connection **needs reconnecting** | +| Sync | `freeagent.sync`, queued every 30 minutes per connected organisation, after connecting, and by **Sync now**. Incremental by `updated_since`, paged, across every bank account. The first sync reaches back about 400 days for transactions | +| Client | One typed wrapper: always `X-Api-Version: 2026-09-01`, token refresh 2 minutes before expiry (single-flight), bounded back-off on 429 `Retry-After`, `Link` pagination, typed `FreeAgentError`, and the token never sent off the FreeAgent API host | +| UI | **Settings → FreeAgent**: connect, reconnect, disconnect, environment, company, last sync, errors, Sync now. **FreeAgent** page: unexplained transactions and invoices (overdue filter) | +| Agent tools | Read: `freeagent_company`, `freeagent_search_transactions`, `freeagent_list_categories`, `freeagent_list_invoices`, `freeagent_list_bills`. Write, approval required, admin requesters only: `freeagent_explain_transaction`, `freeagent_create_bill`. None offered until the organisation is connected | +| Service API | `freeagentClientFor`, `explainTransaction`, `attachToExplanation`, `deleteExplanation`, `createBill`, `listUnexplainedTransactions` — each write updates the mirror row it touched | +| Extension | `freeagent:sync-listeners`: a plugin contributes `{ id, onSync(summary, ctx) }`, called after every successful sync | +| Analytics | When the `analytics` plugin is installed: cubes `FreeAgentTransactions`, `FreeAgentExplanations` (+ `FreeAgentCategories`), `FreeAgentInvoices`, `FreeAgentBills` — spend and income by month and category, unexplained count and amount, outstanding and overdue. Inert without analytics | +| Data | `freeagent_connections`, `freeagent_sync_state` and seven mirror tables, all with row-level security; money as `numeric` | +| CLI | `rocketflare freeagent status` and `rocketflare freeagent sync` | +| Skill | `/freeagent`: registering the app (sandbox and production), secrets, connecting, verifying, troubleshooting | + +## Requirements + +- Rocketflare kit **0.21.0** or later. +- A FreeAgent app registered at [dev.freeagent.com](https://dev.freeagent.com) with the redirect + URI `{APP_URL}/api/hooks/freeagent/callback`. +- `OAUTH_ENCRYPTION_KEY` (the kit's bootstrap sets it) and the kit's `JOBS_QUEUE`. + +## Install + +```bash +pnpm plugin add https://github.com/rocketflare-dev/rocketflare-plugins.git@ --subdir plugins/freeagent +pnpm plugin add https://github.com/rocketflare-dev/rocketflare-plugins.git@ --subdir plugins/freeagent --apply +pnpm db:generate --name plugin-freeagent- && pnpm db:migrate +``` + +Then set the three keys — `FREEAGENT_CLIENT_ID`, `FREEAGENT_CLIENT_SECRET`, `FREEAGENT_ENVIRONMENT` +(`sandbox` | `production`, unset means `sandbox`) — in `apps/web/.dev.vars` locally and with +`wrangler secret put` deployed. All three are declared as secrets so no toml edit is needed; the +environment is not sensitive and may go under `[vars]` in both tomls instead. `pnpm provision +cloudflare ` writes the `*/30 * * * *` cron. Run `/freeagent` for the walk-through. + +## Building on it + +```ts +import { freeagentExtensions } from '@rocketflare/shared/plugins/freeagent/index' +import { + attachToExplanation, + explainTransaction, + type FreeAgentSyncListener, + listUnexplainedTransactions, +} from '@/plugins/freeagent' + +const rematch: FreeAgentSyncListener = { + id: 'receipts', + async onSync(summary, ctx) { + await ctx.enqueue({ type: 'receipts.match', payload: { tenantId: summary.tenantId } }) + }, +} + +// in a job: +const open = await listUnexplainedTransactions(ctx, tenantId, { from: '2026-04-06' }) +const explanation = await explainTransaction(ctx, tenantId, { + bankTransactionUrl: open[0].url, + categoryUrl, // from freeagent_categories + grossValue: open[0].unexplainedAmount, // signed like the transaction + datedOn: open[0].datedOn, + description: 'Train ticket', +}) +await attachToExplanation(ctx, tenantId, explanation.url, { + bytes, filename: 'receipt.pdf', contentType: 'application/pdf', +}) + +export const receiptsServer = { + // … + extensions: freeagentExtensions({ syncListeners: [rematch] }), +} +``` + +Declare `requires: { plugins: ["freeagent"] }` in your `plugin.json`. Writes throw `FreeAgentError`; +branch on `err.code` (`needs_reconnect`, `rate_limited`, `validation`, …). + +## Who sees what + +Every member can see the FreeAgent page and the mirrored books, and their agent runs get the read +tools. Owners and admins connect, disconnect and sync, and only their runs are offered the two write +tools — each call still waits for a person to approve it. + +## Known gaps + +- One FreeAgent company per organisation; no accountancy-practice (multi-client) API. +- Polling only (FreeAgent has no webhooks): changes appear within 30 minutes, or on Sync now. +- The first sync mirrors about 400 days of transactions and explanations; older history is not fetched. +- A deleted contact, bill or invoice stays in the mirror (FreeAgent's `updated_since` does not report deletions). Explanations deleted in FreeAgent are reconciled through their transaction. +- No per-user rate-limit budgeting beyond the bounded 429 back-off; a very large first sync may span several queue retries. +- Disconnecting deletes the mirror as well as the tokens. diff --git a/plugins/freeagent/apps/cli/src/plugins/freeagent/index.ts b/plugins/freeagent/apps/cli/src/plugins/freeagent/index.ts new file mode 100644 index 0000000..6f7e185 --- /dev/null +++ b/plugins/freeagent/apps/cli/src/plugins/freeagent/index.ts @@ -0,0 +1,59 @@ +/** + * `rocketflare freeagent …` — the plugin's CLI half. + * + * rocketflare freeagent status [--json] GET /api/freeagent/status + * rocketflare freeagent sync [--json] POST /api/freeagent/sync (queues a sync; admin) + */ +import { + FREEAGENT_ID, + freeagentShared, + freeagentStatusSchema, + freeagentSyncResponseSchema, +} from '@rocketflare/shared/plugins/freeagent/index' +import type { CliPlugin, CommandContext } from '../api' +import { formatDate, requireClient } from '../api' + +export async function runFreeagentStatus(ctx: CommandContext): Promise { + const { data, raw } = await requireClient(ctx).request('GET', '/api/freeagent/status', { + schema: freeagentStatusSchema, + }) + const connection = data.connected + ? 'connected' + : data.status === 'needs_reconnect' + ? 'needs reconnecting (Settings → FreeAgent)' + : 'not connected' + ctx.out.data(raw, () => + [ + `FreeAgent: ${data.configured ? `configured (${data.environment})` : 'not configured on this deployment'}`, + `Connection: ${connection}`, + `Company: ${data.companyName ?? '-'}${data.companyCurrency ? ` (${data.companyCurrency})` : ''}`, + `Last sync: ${formatDate(data.lastSyncAt)}`, + `Mirrored: ${data.counts.transactions} transactions (${data.counts.unexplained} unexplained), ${data.counts.invoices} invoices, ${data.counts.bills} bills`, + ...(data.lastError + ? [`Last error: ${data.lastError} (${formatDate(data.lastErrorAt)})`] + : []), + ].join('\n') + ) +} + +export async function runFreeagentSync(ctx: CommandContext): Promise { + const { data, raw } = await requireClient(ctx).request('POST', '/api/freeagent/sync', { + schema: freeagentSyncResponseSchema, + }) + ctx.out.data(raw, () => `Sync queued (job ${data.jobId}).`) +} + +export const freeagentCli: CliPlugin = { + shared: freeagentShared, + register(program, action) { + const root = program.command(FREEAGENT_ID).description('FreeAgent accounting (a plugin)') + root + .command('status', { isDefault: true }) + .description('show the connection, company and last sync') + .action(action(ctx => runFreeagentStatus(ctx))) + root + .command('sync') + .description('queue a sync of the FreeAgent books now') + .action(action(ctx => runFreeagentSync(ctx))) + }, +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/CLAUDE.md b/plugins/freeagent/apps/web/src/plugins/freeagent/CLAUDE.md new file mode 100644 index 0000000..e73b441 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/CLAUDE.md @@ -0,0 +1,25 @@ +# freeagent plugin — rules for this directory + +An organisation's FreeAgent books: OAuth (`api/routes.ts` connect → FreeAgent → `api/hooks.ts` +callback) → sealed tokens on `freeagent_connections` → `freeagent.sync` (cron `*/30`, "Sync now", +the callback) → `services/sync.ts` → the `freeagent_*` mirror → agent tools, the page, cubes, and +every `freeagent:sync-listeners` extension. + +- **Every FreeAgent call goes through `FreeAgentClient`** (`services/client.ts`): it sends + `X-Api-Version: 2026-09-01`, refreshes tokens, waits out 429s (bounded), follows `Link` + pagination and refuses any URL off the connection's API host. Never `fetch` FreeAgent directly. +- **`fetch` is injected.** Services take `deps` (default `freeagentHttp`); tests pass a fake, and + route tests swap `freeagentHttp.fetch` for one test. No test reaches FreeAgent. +- **Tokens are sealed** (`sealSecret`) in `*_enc` columns, never returned, never logged, nor are + approval codes. Log a `FreeAgentError.code`, not a body. +- **`invalid_grant` → `needs_reconnect`**, recorded on the row; the cron skips it, tools vanish. +- **Money is a decimal string** end to end (`numeric` columns, zod `decimalStringSchema`). Never + `Number()` an amount. +- **Mirror rows are keyed `(tenant_id, url)`** and written only through `services/mirror.ts` + (`upsertMirror`), by the sync and by every write — a write updates the row it touched. +- **The one cross-tenant read** is the cron fan-out in `api/scheduled.ts` (tenant ids only). It + needs no `unscopedAllowlist` entry: the kit scan sees the `tenantId` it selects. +- **Cubes are plain data** contributed under `analytics:*` keys (`cubes/index.ts`): no import of + analytics or drizzle-cube, inert without it. A new cube needs an isolation case beside it. +- **Listeners** (`FreeAgentSyncListener`) are called after a successful sync; a throw is logged and + does not fail the sync. They must be quick and idempotent. diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/api/hooks.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/api/hooks.ts new file mode 100644 index 0000000..ecca973 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/api/hooks.ts @@ -0,0 +1,108 @@ +/** + * `/api/hooks/freeagent` — this plugin's PUBLIC mount (no `authMiddleware`). + * + * GET /callback FreeAgent's approval redirect lands here with `code` and `state` + * + * What makes an unauthenticated route safe is that it trusts nothing the request says except what + * it can VERIFY: `state`, signed by `POST /api/freeagent/connect` for one tenant and one admin, for + * this purpose only, expiring in fifteen minutes (FreeAgent's own code lifetime). A missing, + * tampered or expired state is a 401 and nothing else — no hint which part failed. Only after that + * does the handler know which organisation it is acting for. + * + * Then: exchange the code (Basic auth with the operator's app), read `GET /v2/company` with the + * new token, store both tokens SEALED, enqueue the first sync and send the admin back to + * Settings → FreeAgent. The code and the tokens are never logged; a failure is logged by code. + */ +import { + FREEAGENT_CONNECTION_ENTITY, + FREEAGENT_SUBJECT, + FREEAGENT_SYNC_JOB, +} from '@rocketflare/shared/plugins/freeagent/index' +import { z } from 'zod' +import type { PublicCtx } from '@/plugins/api' +import { createRouter, publicCtx, recordActivity, verifyState } from '@/plugins/api' +import { FreeAgentClient } from '../services/client' +import { saveConnection } from '../services/connection' +import { + appCredentials, + FREEAGENT_HOSTS, + freeagentConfig, + freeagentHttp, + redirectUri, + settingsReturnUrl, +} from '../services/env' +import { FreeAgentError } from '../services/errors' +import { mapCompany } from '../services/mapping' +import { exchangeCode } from '../services/tokens' + +export const CONNECT_STATE_PURPOSE = 'freeagent.connect' +/** FreeAgent's approval code lives 15 minutes; the state need not outlive it. */ +export const CONNECT_STATE_TTL_SECONDS = 15 * 60 + +export const connectStateSchema = z.object({ + tenantId: z.string().uuid(), + userId: z.string().uuid(), +}) + +export const freeagentHooksRouter = createRouter() + +freeagentHooksRouter.get('/callback', async c => { + const ctx: PublicCtx = publicCtx(c) + const query = new URL(c.req.url).searchParams + // Parsed even though it is signed: a signature proves who wrote it, not what shape it has. + const state = connectStateSchema.safeParse( + await verifyState(ctx.config, CONNECT_STATE_PURPOSE, query.get('state') ?? '') + ) + if (!state.success) ctx.unauthorized('Invalid or expired FreeAgent approval') + const { tenantId, userId } = state.data + const back = (outcome: { connected: true } | { error: string }) => + c.redirect(settingsReturnUrl(ctx.appUrl, outcome), 302) + + // The admin clicked "Deny" (or FreeAgent refused): FreeAgent sends `error`, no code. + const denied = query.get('error') + if (denied) + return back({ error: denied === 'access_denied' ? 'access_denied' : 'approval_failed' }) + const code = query.get('code') + if (!code) return back({ error: 'missing_code' }) + const credentials = appCredentials(ctx.config) + if (!credentials) return back({ error: 'not_configured' }) + + const environment = freeagentConfig(ctx.config).FREEAGENT_ENVIRONMENT + const deps = freeagentHttp + let connectionId: string + let companyName: string | null + try { + const tokens = await exchangeCode( + environment, + credentials, + { code, redirectUri: redirectUri(ctx.appUrl) }, + deps + ) + const client = new FreeAgentClient({ + baseUrl: FREEAGENT_HOSTS[environment], + accessToken: async () => tokens.accessToken, + deps, + }) + const company = mapCompany(await client.get('/v2/company')) + companyName = company.companyName + const row = await saveConnection(ctx, { tenantId, userId, environment, tokens, company }) + connectionId = row.id + } catch (err) { + const codeName = err instanceof FreeAgentError ? err.code : 'internal' + ctx.logger.warn({ tenantId, code: codeName }, 'freeagent: completing the connection failed') + return back({ error: codeName === 'invalid_grant' ? 'code_expired' : 'exchange_failed' }) + } + + await ctx.enqueue({ type: FREEAGENT_SYNC_JOB, payload: { tenantId, requestedBy: userId } }) + // Awaited rather than deferred: both are one cheap write, and the admin is waiting on a redirect. + await recordActivity(ctx.db, { + tenantId, + userId, + type: 'freeagent.connected', + subjectType: FREEAGENT_SUBJECT, + subjectId: connectionId, + metadata: { companyName, environment }, + }).catch(() => {}) + await ctx.nudge(tenantId, FREEAGENT_CONNECTION_ENTITY).catch(() => {}) + return back({ connected: true }) +}) diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/api/routes.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/api/routes.ts new file mode 100644 index 0000000..23dad4e --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/api/routes.ts @@ -0,0 +1,198 @@ +/** + * `/api/freeagent` — the authenticated mount: status, connect, disconnect, sync now, and the two + * mirror lists the FreeAgent page shows (unexplained transactions, recent invoices). + * + * Connecting is a browser round trip through FreeAgent: `POST /connect` answers the approve URL + * with a SIGNED state naming this tenant and this admin, and the public callback + * (`api/hooks.ts`) trusts nothing but that state. Every read here filters by the session's tenant; + * nothing reaches FreeAgent itself except through the queued sync. + */ +import { + FREEAGENT_BOOKS_ENTITY, + FREEAGENT_CONNECTION_ENTITY, + FREEAGENT_SUBJECT, + FREEAGENT_SYNC_JOB, + type FreeAgentInvoice, + type FreeAgentInvoiceListQuery, + type FreeAgentTransaction, + type FreeAgentTransactionListQuery, + freeagentInvoiceListQuerySchema, + freeagentTransactionListQuerySchema, +} from '@rocketflare/shared/plugins/freeagent/index' +import { and, count, desc, eq, type SQL } from 'drizzle-orm' +import type { RequestCtx } from '@/plugins/api' +import { createRouter, recordActivity, requestCtx, signState, validate } from '@/plugins/api' +import { + freeagentBankAccounts, + freeagentBankTransactions, + freeagentContacts, + freeagentInvoices, +} from '../db/schema' +import { deleteConnection, loadConnection } from '../services/connection' +import { appCredentials, approveUrl, freeagentConfig, redirectUri } from '../services/env' +import { isOverdue } from '../services/mapping' +import { overdueSql } from '../services/overdue' +import { CONNECT_STATE_PURPOSE, CONNECT_STATE_TTL_SECONDS } from './hooks' +import { loadStatus } from './status' + +export const freeagentRouter = createRouter() + +freeagentRouter.get('/status', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', FREEAGENT_SUBJECT) + return c.json(await loadStatus(ctx.db, ctx.config, ctx.tenantId)) +}) + +freeagentRouter.post('/connect', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('manage', FREEAGENT_SUBJECT) + const credentials = appCredentials(ctx.config) + if (!credentials) { + ctx.unavailable( + 'FreeAgent is not set up on this deployment (FREEAGENT_CLIENT_ID / FREEAGENT_CLIENT_SECRET)', + 'freeagent_not_configured' + ) + } + const state = await signState( + ctx.config, + CONNECT_STATE_PURPOSE, + { tenantId: ctx.tenantId, userId: ctx.userId }, + { ttlSeconds: CONNECT_STATE_TTL_SECONDS } + ) + return c.json({ + approveUrl: approveUrl(freeagentConfig(ctx.config).FREEAGENT_ENVIRONMENT, { + clientId: credentials.clientId, + redirectUri: redirectUri(ctx.config.APP_URL), + state, + }), + }) +}) + +freeagentRouter.delete('/connection', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('manage', FREEAGENT_SUBJECT) + const existing = await loadConnection(ctx.db, ctx.tenantId) + if (!existing) ctx.notFound('FreeAgent is not connected', 'freeagent_not_connected') + await deleteConnection(ctx.db, ctx.tenantId) + ctx.defer(() => + recordActivity(ctx.db, { + tenantId: ctx.tenantId, + userId: ctx.userId, + type: 'freeagent.disconnected', + subjectType: FREEAGENT_SUBJECT, + subjectId: existing.id, + metadata: { companyName: existing.companyName, environment: existing.environment }, + }) + ) + ctx.nudge(FREEAGENT_CONNECTION_ENTITY) + ctx.nudge(FREEAGENT_BOOKS_ENTITY) + return c.body(null, 204) +}) + +freeagentRouter.post('/sync', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('manage', FREEAGENT_SUBJECT) + const connection = await loadConnection(ctx.db, ctx.tenantId) + if (!connection) ctx.notFound('FreeAgent is not connected', 'freeagent_not_connected') + if (connection.status !== 'connected') { + ctx.conflict('Reconnect to FreeAgent before syncing', 'freeagent_needs_reconnect') + } + const job = await ctx.enqueue({ + type: FREEAGENT_SYNC_JOB, + payload: { tenantId: ctx.tenantId, requestedBy: ctx.userId }, + }) + return c.json({ jobId: job.id, type: FREEAGENT_SYNC_JOB }, 202) +}) + +freeagentRouter.get( + '/transactions', + validate('query', freeagentTransactionListQuerySchema), + async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', FREEAGENT_SUBJECT) + const query = ctx.valid('query') + const filters: SQL[] = [eq(freeagentBankTransactions.tenantId, ctx.tenantId)] + if (query.unexplained !== undefined) { + filters.push(eq(freeagentBankTransactions.isUnexplained, query.unexplained)) + } + if (query.bankAccountUrl) { + filters.push(eq(freeagentBankTransactions.bankAccountUrl, query.bankAccountUrl)) + } + const where = and(...filters) + const [rows, totals] = await Promise.all([ + ctx.db + .select({ tx: freeagentBankTransactions, accountName: freeagentBankAccounts.name }) + .from(freeagentBankTransactions) + .leftJoin( + freeagentBankAccounts, + and( + eq(freeagentBankAccounts.tenantId, freeagentBankTransactions.tenantId), + eq(freeagentBankAccounts.url, freeagentBankTransactions.bankAccountUrl) + ) + ) + .where(where) + .orderBy(desc(freeagentBankTransactions.datedOn), desc(freeagentBankTransactions.id)) + .limit(query.pageSize) + .offset((query.page - 1) * query.pageSize), + ctx.db.select({ total: count() }).from(freeagentBankTransactions).where(where), + ]) + const items = rows.map( + ({ tx, accountName }): FreeAgentTransaction => ({ + id: tx.id, + url: tx.url, + bankAccountUrl: tx.bankAccountUrl, + bankAccountName: accountName, + datedOn: tx.datedOn, + amount: tx.amount, + unexplainedAmount: tx.unexplainedAmount, + isUnexplained: tx.isUnexplained, + description: tx.description, + }) + ) + return c.json(ctx.page(items, totals[0]?.total ?? 0, query)) + } +) + +freeagentRouter.get('/invoices', validate('query', freeagentInvoiceListQuerySchema), async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', FREEAGENT_SUBJECT) + const query = ctx.valid('query') + const filters: SQL[] = [eq(freeagentInvoices.tenantId, ctx.tenantId)] + if (query.overdue) filters.push(overdueSql(freeagentInvoices)) + const where = and(...filters) + const [rows, totals] = await Promise.all([ + ctx.db + .select({ inv: freeagentInvoices, contactName: freeagentContacts.displayName }) + .from(freeagentInvoices) + .leftJoin( + freeagentContacts, + and( + eq(freeagentContacts.tenantId, freeagentInvoices.tenantId), + eq(freeagentContacts.url, freeagentInvoices.contactUrl) + ) + ) + .where(where) + .orderBy(desc(freeagentInvoices.datedOn), desc(freeagentInvoices.id)) + .limit(query.pageSize) + .offset((query.page - 1) * query.pageSize), + ctx.db.select({ total: count() }).from(freeagentInvoices).where(where), + ]) + const today = new Date().toISOString().slice(0, 10) + const items = rows.map( + ({ inv, contactName }): FreeAgentInvoice => ({ + id: inv.id, + url: inv.url, + reference: inv.reference, + contactName, + status: inv.status, + datedOn: inv.datedOn, + dueOn: inv.dueOn, + currency: inv.currency, + totalValue: inv.totalValue, + paidValue: inv.paidValue, + dueValue: inv.dueValue, + overdue: isOverdue(inv, today), + }) + ) + return c.json(ctx.page(items, totals[0]?.total ?? 0, query)) +}) diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/api/scheduled.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/api/scheduled.ts new file mode 100644 index 0000000..4d6889e --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/api/scheduled.ts @@ -0,0 +1,43 @@ +/** + * The half-hourly tick: one `freeagent.sync` per connected organisation. Contributed through + * `ServerPlugin.scheduledTasks`; the same expression is `plugin.json`'s `crons`, which + * `pnpm provision cloudflare ` writes into both tomls. + * + * This is the plugin's one cross-tenant read: it runs in the Worker's `scheduled` handler with no + * request behind it, reads only tenant ids, and every write it causes happens in a job that + * re-reads its connection under the tenant in its payload. It needs no `unscopedAllowlist` entry: + * the kit's scan sees the `tenantId` it selects, and the test refuses a stale one. + * + * A tenant that needs reconnecting is skipped — no sync can succeed for it. Overlap with a sync + * still running is harmless: the job's claim turns the second one into a no-op. + */ +import { FREEAGENT_SYNC_JOB } from '@rocketflare/shared/plugins/freeagent/index' +import { eq } from 'drizzle-orm' +import type { CronCtx, JobInput, ScheduledTask } from '@/plugins/api' +import { cronCtx } from '@/plugins/api' +import { freeagentConnections } from '../db/schema' + +export const FREEAGENT_CRON = '*/30 * * * *' + +export async function enqueueSyncs(ctx: CronCtx): Promise { + const connected = await ctx.db + .select({ tenantId: freeagentConnections.tenantId }) + .from(freeagentConnections) + .where(eq(freeagentConnections.status, 'connected')) + const jobs: JobInput[] = connected.map(row => ({ + type: FREEAGENT_SYNC_JOB, + payload: { tenantId: row.tenantId }, + })) + // Queue batches are capped at 100 messages; `enqueueMany` chunks for us. + if (jobs.length > 0) await ctx.enqueueMany(jobs) + return jobs.length +} + +export const enqueueSyncsTask: ScheduledTask = { + name: 'freeagent.enqueueSyncs', + run: async raw => { + const ctx = cronCtx(raw) + const queued = await enqueueSyncs(ctx) + ctx.logger.info({ queued }, 'freeagent.enqueueSyncs') + }, +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/api/status.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/api/status.ts new file mode 100644 index 0000000..3ff891c --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/api/status.ts @@ -0,0 +1,74 @@ +/** + * The status Settings, the CLI and the `freeagent_company` tool all show: is the deployment set up, + * is this organisation connected, to which company, when did it last sync and what went wrong. + * Never a token — `hasCredential`-style, the browser learns only that one exists. + */ +import type { + FreeAgentEnvironment, + FreeAgentStatus, +} from '@rocketflare/shared/plugins/freeagent/index' +import { and, count, eq } from 'drizzle-orm' +import type { Database, PluginConfig } from '@/plugins/api' +import { + freeagentBankAccounts, + freeagentBankTransactions, + freeagentBills, + freeagentInvoices, +} from '../db/schema' +import { loadConnection } from '../services/connection' +import { appCredentials, freeagentConfig } from '../services/env' + +export async function loadStatus( + db: Database, + config: PluginConfig, + tenantId: string +): Promise { + const connection = await loadConnection(db, tenantId) + const total = async ( + table: typeof freeagentBills | typeof freeagentInvoices | typeof freeagentBankAccounts + ) => { + const t = table as typeof freeagentBills + const [row] = await db.select({ n: count() }).from(t).where(eq(t.tenantId, tenantId)) + return row?.n ?? 0 + } + const [transactions, unexplained] = connection + ? await Promise.all([ + db + .select({ n: count() }) + .from(freeagentBankTransactions) + .where(eq(freeagentBankTransactions.tenantId, tenantId)), + db + .select({ n: count() }) + .from(freeagentBankTransactions) + .where( + and( + eq(freeagentBankTransactions.tenantId, tenantId), + eq(freeagentBankTransactions.isUnexplained, true) + ) + ), + ]) + : [[], []] + return { + configured: appCredentials(config) !== null, + environment: freeagentConfig(config).FREEAGENT_ENVIRONMENT, + connected: connection?.status === 'connected', + status: (connection?.status as FreeAgentStatus['status']) ?? null, + connectionEnvironment: (connection?.environment as FreeAgentEnvironment | undefined) ?? null, + companyName: connection?.companyName ?? null, + companyCurrency: connection?.companyCurrency ?? null, + connectedAt: connection?.connectedAt ?? null, + lastSyncAt: connection?.lastSyncAt ?? null, + lastSyncStartedAt: connection?.lastSyncStartedAt ?? null, + lastError: connection?.lastError ?? null, + lastErrorAt: connection?.lastErrorAt ?? null, + counts: connection + ? { + bankAccounts: await total(freeagentBankAccounts), + transactions: transactions[0]?.n ?? 0, + unexplained: unexplained[0]?.n ?? 0, + invoices: await total(freeagentInvoices), + bills: await total(freeagentBills), + } + : { bankAccounts: 0, transactions: 0, unexplained: 0, invoices: 0, bills: 0 }, + } +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/cubes/index.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/cubes/index.ts new file mode 100644 index 0000000..58de469 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/cubes/index.ts @@ -0,0 +1,357 @@ +/** + * drizzle-cube cubes over the mirror, contributed to the `analytics` plugin through `extensions`. + * + * **Inert without analytics, and deliberately so.** The contribution goes under analytics' own + * extension keys (`analytics:cubes`, `analytics:cubeIsolationCases`) as plain data: nothing here + * imports the analytics plugin or `drizzle-cube`, so this plugin needs neither installed and + * declares no `requires.plugins`. When analytics IS installed it reads these, narrows them, serves + * them at `/cubejs-api`, and its mandatory cube-isolation test runs the cases below against them. + * `defineCube(name, def)` in drizzle-cube is `{ name, ...def }`, which is the shape written here. + * + * Every cube's `sql()` scopes its base query by the security context's tenant (`tenantOf`), which + * throws rather than compiling `tenant_id = NULL`. Joins match on the URL AND the tenant: FreeAgent + * category URLs (`/v2/categories/285`) are the same in every company, so a URL-only join would + * cross organisations. `tests/api/cubes.test.ts` proves the scoping without analytics installed. + * + * Member names are frozen once released: dashboards reference them. + */ +import { eq, gt, lt, type SQL, sql } from 'drizzle-orm' +import type { Database } from '@/plugins/api' +import { + freeagentBankTransactions, + freeagentBills, + freeagentCategories, + freeagentExplanations, + freeagentInvoices, +} from '../db/schema' +import { outstandingSql, overdueSql } from '../services/overdue' + +/** The slice of drizzle-cube's `QueryContext` a cube reads. */ +export interface CubeQueryContext { + securityContext: Record +} + +export function tenantOf(ctx: CubeQueryContext): string { + const tenantId = ctx.securityContext.tenantId + if (typeof tenantId !== 'string' || tenantId.length === 0) { + throw new Error('freeagent cube queried without a tenant in the security context') + } + return tenantId +} + +/** One cube, as drizzle-cube's `defineCube` would return it. */ +export interface FreeAgentCube { + name: string + title: string + description: string + sql: (ctx: CubeQueryContext) => { from: unknown; where: SQL | undefined } + joins?: Record + dimensions: Record> + measures: Record> +} + +const tx = freeagentBankTransactions +const ex = freeagentExplanations +const cat = freeagentCategories +const inv = freeagentInvoices +const bill = freeagentBills + +export const freeagentCategoriesCube: FreeAgentCube = { + name: 'FreeAgentCategories', + title: 'FreeAgent Categories', + description: 'Accounting categories (nominal codes) in FreeAgent', + sql: ctx => ({ from: cat, where: eq(cat.tenantId, tenantOf(ctx)) }), + dimensions: { + id: { name: 'id', title: 'ID', type: 'string', sql: cat.id, primaryKey: true }, + url: { name: 'url', title: 'URL', type: 'string', sql: cat.url }, + description: { name: 'description', title: 'Category', type: 'string', sql: cat.description }, + nominalCode: { + name: 'nominalCode', + title: 'Nominal Code', + type: 'string', + sql: cat.nominalCode, + }, + kind: { name: 'kind', title: 'Group', type: 'string', sql: cat.kind }, + }, + measures: { + count: { name: 'count', title: 'Categories', type: 'count', sql: cat.id }, + }, +} + +export const freeagentTransactionsCube: FreeAgentCube = { + name: 'FreeAgentTransactions', + title: 'FreeAgent Bank Transactions', + description: + 'Bank transactions mirrored from FreeAgent: money in and out, and what is unexplained', + sql: ctx => ({ from: tx, where: eq(tx.tenantId, tenantOf(ctx)) }), + dimensions: { + id: { name: 'id', title: 'ID', type: 'string', sql: tx.id, primaryKey: true }, + url: { name: 'url', title: 'URL', type: 'string', sql: tx.url }, + datedOn: { name: 'datedOn', title: 'Date', type: 'time', sql: tx.datedOn }, + bankAccountUrl: { + name: 'bankAccountUrl', + title: 'Bank Account', + type: 'string', + sql: tx.bankAccountUrl, + }, + description: { name: 'description', title: 'Description', type: 'string', sql: tx.description }, + isUnexplained: { + name: 'isUnexplained', + title: 'Unexplained', + type: 'boolean', + sql: tx.isUnexplained, + }, + direction: { + name: 'direction', + title: 'Direction', + type: 'string', + sql: sql`case when ${tx.amount} < 0 then 'out' else 'in' end`, + }, + }, + measures: { + count: { name: 'count', title: 'Transactions', type: 'count', sql: tx.id }, + net: { name: 'net', title: 'Net Movement', type: 'sum', sql: tx.amount }, + moneyIn: { + name: 'moneyIn', + title: 'Money In', + type: 'sum', + sql: tx.amount, + filters: [() => gt(tx.amount, '0')], + }, + moneyOut: { + name: 'moneyOut', + title: 'Money Out', + type: 'sum', + sql: tx.amount, + filters: [() => lt(tx.amount, '0')], + }, + unexplainedCount: { + name: 'unexplainedCount', + title: 'Unexplained Transactions', + type: 'count', + sql: tx.id, + filters: [() => eq(tx.isUnexplained, true)], + }, + unexplainedAmount: { + name: 'unexplainedAmount', + title: 'Unexplained Amount', + type: 'sum', + sql: tx.unexplainedAmount, + }, + }, +} + +export const freeagentExplanationsCube: FreeAgentCube = { + name: 'FreeAgentExplanations', + title: 'FreeAgent Explanations', + description: 'How bank transactions were explained: spend and income by category', + sql: ctx => ({ from: ex, where: eq(ex.tenantId, tenantOf(ctx)) }), + joins: { + FreeAgentCategories: { + targetCube: () => freeagentCategoriesCube, + relationship: 'belongsTo', + on: [ + { source: ex.categoryUrl, target: cat.url }, + { source: ex.tenantId, target: cat.tenantId }, + ], + }, + }, + dimensions: { + id: { name: 'id', title: 'ID', type: 'string', sql: ex.id, primaryKey: true }, + datedOn: { name: 'datedOn', title: 'Date', type: 'time', sql: ex.datedOn }, + entryType: { name: 'entryType', title: 'Type', type: 'string', sql: ex.entryType }, + categoryUrl: { + name: 'categoryUrl', + title: 'Category URL', + type: 'string', + sql: ex.categoryUrl, + }, + }, + measures: { + count: { name: 'count', title: 'Explanations', type: 'count', sql: ex.id }, + gross: { name: 'gross', title: 'Gross Value', type: 'sum', sql: ex.grossValue }, + spend: { + name: 'spend', + title: 'Spend', + type: 'sum', + sql: sql`-${ex.grossValue}`, + filters: [() => lt(ex.grossValue, '0')], + }, + income: { + name: 'income', + title: 'Income', + type: 'sum', + sql: ex.grossValue, + filters: [() => gt(ex.grossValue, '0')], + }, + salesTax: { name: 'salesTax', title: 'Sales Tax', type: 'sum', sql: ex.salesTaxValue }, + withAttachments: { + name: 'withAttachments', + title: 'With Attachments', + type: 'count', + sql: ex.id, + filters: [() => gt(ex.attachmentCount, 0)], + }, + }, +} + +function documentCube( + name: 'FreeAgentInvoices' | 'FreeAgentBills', + table: typeof freeagentInvoices | typeof freeagentBills +): FreeAgentCube { + const t = table as typeof freeagentInvoices + const noun = name === 'FreeAgentInvoices' ? 'Invoices' : 'Bills' + return { + name, + title: `FreeAgent ${noun}`, + description: + name === 'FreeAgentInvoices' + ? 'Sales invoices: billed, outstanding and overdue' + : 'Supplier bills: owed, outstanding and overdue', + sql: ctx => ({ from: t, where: eq(t.tenantId, tenantOf(ctx)) }), + dimensions: { + id: { name: 'id', title: 'ID', type: 'string', sql: t.id, primaryKey: true }, + url: { name: 'url', title: 'URL', type: 'string', sql: t.url }, + reference: { name: 'reference', title: 'Reference', type: 'string', sql: t.reference }, + status: { name: 'status', title: 'Status', type: 'string', sql: t.status }, + currency: { name: 'currency', title: 'Currency', type: 'string', sql: t.currency }, + datedOn: { name: 'datedOn', title: 'Date', type: 'time', sql: t.datedOn }, + dueOn: { name: 'dueOn', title: 'Due', type: 'time', sql: t.dueOn }, + }, + measures: { + count: { name: 'count', title: noun, type: 'count', sql: t.id }, + total: { name: 'total', title: 'Total', type: 'sum', sql: t.totalValue }, + paid: { name: 'paid', title: 'Paid', type: 'sum', sql: t.paidValue }, + outstanding: { + name: 'outstanding', + title: 'Outstanding', + type: 'sum', + sql: t.dueValue, + filters: [() => outstandingSql(table)], + }, + overdueCount: { + name: 'overdueCount', + title: `Overdue ${noun}`, + type: 'count', + sql: t.id, + filters: [() => overdueSql(table)], + }, + overdueAmount: { + name: 'overdueAmount', + title: 'Overdue Amount', + type: 'sum', + sql: t.dueValue, + filters: [() => overdueSql(table)], + }, + }, + } +} + +export const freeagentInvoicesCube = documentCube('FreeAgentInvoices', freeagentInvoices) +export const freeagentBillsCube = documentCube('FreeAgentBills', freeagentBills) + +export const FREEAGENT_CUBES: readonly FreeAgentCube[] = [ + freeagentBillsCube, + freeagentCategoriesCube, + freeagentExplanationsCube, + freeagentInvoicesCube, + freeagentTransactionsCube, +] + +// ---- Isolation cases (the analytics plugin's `CubeIsolationCase` shape) ------------------------- + +type Rows = Array> + +/** + * One case per cube. Analytics seeds tenant A then tenant B, then queries as each; the seed puts + * the tenant's id into every row's URL, and `expect` checks each side sees only its own — and that + * it sees them at all. The seeded ids are remembered in call order, which is how `expect` (given + * only `'a' | 'b'`) knows which tenant it is looking at. + */ +function isolationCase( + cube: string, + member: string, + seed: (db: Database, tenantId: string) => Promise +) { + const seeded: string[] = [] + return { + cube, + seed: async (db: Database, ctx: { tenantId: string; userId: string }) => { + seeded.push(ctx.tenantId) + await seed(db, ctx.tenantId) + }, + query: { measures: [`${cube}.count`], dimensions: [`${cube}.${member}`] }, + expect: (rows: Rows, side: 'a' | 'b') => { + const mine = seeded[seeded.length - (side === 'a' ? 2 : 1)] ?? '' + const other = seeded[seeded.length - (side === 'a' ? 1 : 2)] ?? '' + // Plain throws, not vitest's `expect`: this file ships in the Worker bundle. + const values = rows.map(r => String(r[`${cube}.${member}`])) + if (values.length === 0) throw new Error(`${cube}: no rows for side ${side}`) + for (const v of values) { + if (!mine || !v.includes(mine) || (other && v.includes(other))) { + throw new Error(`${cube}: side ${side} saw a row that is not its own (${v})`) + } + } + }, + } +} + +const fa = (tenantId: string, path: string) => + `https://api.sandbox.freeagent.com/v2/${path}/${tenantId}` + +export const FREEAGENT_CUBE_ISOLATION_CASES = [ + isolationCase('FreeAgentTransactions', 'url', async (db, tenantId) => { + await db.insert(tx).values({ + tenantId, + url: fa(tenantId, 'bank_transactions'), + bankAccountUrl: fa(tenantId, 'bank_accounts'), + datedOn: '2026-10-01', + amount: '-12.00', + unexplainedAmount: '-12.00', + isUnexplained: true, + }) + }), + isolationCase('FreeAgentCategories', 'url', async (db, tenantId) => { + await db.insert(cat).values({ + tenantId, + url: fa(tenantId, 'categories'), + description: 'Travel', + kind: 'admin_expenses', + }) + }), + isolationCase('FreeAgentExplanations', 'categoryUrl', async (db, tenantId) => { + await db.insert(ex).values({ + tenantId, + url: fa(tenantId, 'bank_transaction_explanations'), + categoryUrl: fa(tenantId, 'categories'), + grossValue: '-12.00', + datedOn: '2026-10-01', + }) + }), + isolationCase('FreeAgentInvoices', 'url', async (db, tenantId) => { + await db.insert(inv).values({ + tenantId, + url: fa(tenantId, 'invoices'), + status: 'Open', + totalValue: '100.00', + dueValue: '100.00', + dueOn: '2026-01-01', + }) + }), + isolationCase('FreeAgentBills', 'url', async (db, tenantId) => { + await db.insert(bill).values({ + tenantId, + url: fa(tenantId, 'bills'), + status: 'Open', + totalValue: '50.00', + dueValue: '50.00', + dueOn: '2026-01-01', + }) + }), +] + +/** `analyticsExtensions({ cubes, cubeIsolationCases })`, spelled without importing analytics. */ +export const FREEAGENT_ANALYTICS_EXTENSIONS: Record = { + 'analytics:cubes': FREEAGENT_CUBES, + 'analytics:cubeIsolationCases': FREEAGENT_CUBE_ISOLATION_CASES, +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/db/schema/freeagent.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/db/schema/freeagent.ts new file mode 100644 index 0000000..7d502ef --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/db/schema/freeagent.ts @@ -0,0 +1,285 @@ +/** + * The freeagent plugin's tables (D31). Kit conventions throughout: `tenantRef()` first, + * `timestamps()`, every index led by `tenant_id`, `tenantIsolation()` on each table. + * + * - `freeagent_connections` — one row per organisation (unique `tenant_id`): the environment it + * was made in, the refresh token and the cached access token (both SEALED with `sealSecret`, + * never plaintext), the company it approved, the sync claim and the last outcome. + * - `freeagent_sync_state` — one `updated_since` cursor per (organisation, resource). Per-account + * resources carry the account in the key (`bank_transactions:`). + * - The mirror: `freeagent_bank_accounts`, `freeagent_bank_transactions`, `freeagent_explanations`, + * `freeagent_categories`, `freeagent_contacts`, `freeagent_bills`, `freeagent_invoices`. Each row + * is keyed by its FreeAgent URL — FreeAgent identifies everything by URL, and so does every + * foreign reference here (`bank_account_url`, `category_url`, `contact_url`), which keeps an + * upsert one statement and lets a row arrive before the row it points at. Unique per tenant on + * `(tenant_id, url)`; the sync upserts on exactly that. + * + * Money is `numeric`, which drizzle returns as a STRING — never a float. FreeAgent's own + * timestamps are kept as `fa_updated_at` beside the kit's `updated_at`, which is when WE wrote it. + */ +import { + boolean, + date, + index, + integer, + numeric, + pgTable, + text, + timestamp, + uniqueIndex, + uuid, +} from 'drizzle-orm/pg-core' +import { tenantIsolation, tenantRef, tenants, timestamps, users } from '../../../../db/schema/kit' + +/** Money: 2 dp is FreeAgent's, the extra headroom is for exchange-rate arithmetic upstream. */ +const money = (name: string) => numeric(name, { precision: 18, scale: 2 }) +/** A percentage such as `20` or `5.5`. */ +const rate = (name: string) => numeric(name, { precision: 7, scale: 4 }) + +export const freeagentConnections = pgTable( + 'freeagent_connections', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + /** `sandbox` | `production` — which API host the tokens belong to. */ + environment: text('environment').notNull(), + /** `connected` | `needs_reconnect`. */ + status: text('status').notNull().default('connected'), + refreshTokenEnc: text('refresh_token_enc').notNull(), + accessTokenEnc: text('access_token_enc'), + accessTokenExpiresAt: timestamp('access_token_expires_at', { withTimezone: true }), + companyUrl: text('company_url'), + companyName: text('company_name'), + companyCurrency: text('company_currency'), + connectedByUserId: uuid('connected_by_user_id').references(() => users.id, { + onDelete: 'set null', + }), + connectedAt: timestamp('connected_at', { withTimezone: true }).notNull().defaultNow(), + /** The DB claim one sync holds — a second job (cron overlapping "Sync now") acks without work. */ + syncClaimedUntil: timestamp('sync_claimed_until', { withTimezone: true }), + lastSyncStartedAt: timestamp('last_sync_started_at', { withTimezone: true }), + lastSyncAt: timestamp('last_sync_at', { withTimezone: true }), + lastError: text('last_error'), + lastErrorAt: timestamp('last_error_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_connections_tenant_uidx').on(table.tenantId), + tenantIsolation('freeagent_connections'), + ] +) + +export const freeagentSyncState = pgTable( + 'freeagent_sync_state', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + /** `contacts`, `bills`, `invoices`, `bank_transactions:`, … */ + resource: text('resource').notNull(), + /** The next pass asks for `updated_since` this; null = never synced (a full pass). */ + updatedSince: timestamp('updated_since', { withTimezone: true }), + lastSyncedAt: timestamp('last_synced_at', { withTimezone: true }), + itemsSynced: integer('items_synced').notNull().default(0), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_sync_state_resource_uidx').on(table.tenantId, table.resource), + tenantIsolation('freeagent_sync_state'), + ] +) + +export const freeagentBankAccounts = pgTable( + 'freeagent_bank_accounts', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + name: text('name').notNull(), + /** `StandardBankAccount` | `CreditCardAccount` | `PaypalAccount`. */ + type: text('type'), + currency: text('currency'), + currentBalance: money('current_balance'), + bankName: text('bank_name'), + isPrimary: boolean('is_primary').notNull().default(false), + status: text('status'), + faUpdatedAt: timestamp('fa_updated_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_bank_accounts_url_uidx').on(table.tenantId, table.url), + tenantIsolation('freeagent_bank_accounts'), + ] +) + +export const freeagentBankTransactions = pgTable( + 'freeagent_bank_transactions', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + bankAccountUrl: text('bank_account_url').notNull(), + datedOn: date('dated_on', { mode: 'string' }).notNull(), + amount: money('amount').notNull(), + description: text('description'), + fullDescription: text('full_description'), + unexplainedAmount: money('unexplained_amount').notNull().default('0'), + /** `unexplained_amount <> 0` — stored so the matcher's main query is one index range. */ + isUnexplained: boolean('is_unexplained').notNull().default(false), + isManual: boolean('is_manual').notNull().default(false), + faCreatedAt: timestamp('fa_created_at', { withTimezone: true }), + faUpdatedAt: timestamp('fa_updated_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_bank_transactions_url_uidx').on(table.tenantId, table.url), + index('freeagent_bank_transactions_tenant_dated_idx').on(table.tenantId, table.datedOn), + index('freeagent_bank_transactions_tenant_unexplained_idx').on( + table.tenantId, + table.isUnexplained, + table.datedOn + ), + tenantIsolation('freeagent_bank_transactions'), + ] +) + +export const freeagentExplanations = pgTable( + 'freeagent_explanations', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + bankTransactionUrl: text('bank_transaction_url'), + bankAccountUrl: text('bank_account_url'), + categoryUrl: text('category_url'), + /** FreeAgent's `type` / `entry_type`, e.g. `Payment`, `Business Entertaining`. */ + entryType: text('entry_type'), + datedOn: date('dated_on', { mode: 'string' }), + description: text('description'), + grossValue: money('gross_value'), + salesTaxRate: rate('sales_tax_rate'), + salesTaxValue: money('sales_tax_value'), + attachmentCount: integer('attachment_count').notNull().default(0), + isDeletable: boolean('is_deletable'), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_explanations_url_uidx').on(table.tenantId, table.url), + index('freeagent_explanations_tenant_transaction_idx').on( + table.tenantId, + table.bankTransactionUrl + ), + tenantIsolation('freeagent_explanations'), + ] +) + +export const freeagentCategories = pgTable( + 'freeagent_categories', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + description: text('description').notNull(), + nominalCode: text('nominal_code'), + /** `admin_expenses` | `cost_of_sales` | `income` | `general` — the key FreeAgent listed it under. */ + kind: text('kind').notNull(), + groupDescription: text('group_description'), + allowableForTax: boolean('allowable_for_tax'), + taxReportingName: text('tax_reporting_name'), + autoSalesTaxRate: text('auto_sales_tax_rate'), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_categories_url_uidx').on(table.tenantId, table.url), + tenantIsolation('freeagent_categories'), + ] +) + +export const freeagentContacts = pgTable( + 'freeagent_contacts', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + /** Organisation name, else "first last" — what every list shows. */ + displayName: text('display_name').notNull(), + organisationName: text('organisation_name'), + firstName: text('first_name'), + lastName: text('last_name'), + email: text('email'), + status: text('status'), + faUpdatedAt: timestamp('fa_updated_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_contacts_url_uidx').on(table.tenantId, table.url), + tenantIsolation('freeagent_contacts'), + ] +) + +export const freeagentBills = pgTable( + 'freeagent_bills', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + contactUrl: text('contact_url'), + reference: text('reference'), + datedOn: date('dated_on', { mode: 'string' }), + dueOn: date('due_on', { mode: 'string' }), + paidOn: date('paid_on', { mode: 'string' }), + status: text('status'), + currency: text('currency'), + totalValue: money('total_value'), + netValue: money('net_value'), + salesTaxValue: money('sales_tax_value'), + paidValue: money('paid_value'), + dueValue: money('due_value'), + faUpdatedAt: timestamp('fa_updated_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_bills_url_uidx').on(table.tenantId, table.url), + index('freeagent_bills_tenant_dated_idx').on(table.tenantId, table.datedOn), + tenantIsolation('freeagent_bills'), + ] +) + +export const freeagentInvoices = pgTable( + 'freeagent_invoices', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + url: text('url').notNull(), + contactUrl: text('contact_url'), + reference: text('reference'), + datedOn: date('dated_on', { mode: 'string' }), + dueOn: date('due_on', { mode: 'string' }), + paidOn: date('paid_on', { mode: 'string' }), + /** `Draft`, `Open`, `Overdue`, `Paid`, … as FreeAgent spells it. */ + status: text('status'), + currency: text('currency'), + netValue: money('net_value'), + totalValue: money('total_value'), + paidValue: money('paid_value'), + dueValue: money('due_value'), + faUpdatedAt: timestamp('fa_updated_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + uniqueIndex('freeagent_invoices_url_uidx').on(table.tenantId, table.url), + index('freeagent_invoices_tenant_dated_idx').on(table.tenantId, table.datedOn), + tenantIsolation('freeagent_invoices'), + ] +) + +export type FreeAgentConnectionRow = typeof freeagentConnections.$inferSelect +export type FreeAgentSyncStateRow = typeof freeagentSyncState.$inferSelect +export type FreeAgentBankAccountRow = typeof freeagentBankAccounts.$inferSelect +export type FreeAgentBankTransactionRow = typeof freeagentBankTransactions.$inferSelect +export type NewFreeAgentBankTransactionRow = typeof freeagentBankTransactions.$inferInsert +export type FreeAgentExplanationRow = typeof freeagentExplanations.$inferSelect +export type NewFreeAgentExplanationRow = typeof freeagentExplanations.$inferInsert +export type FreeAgentCategoryRow = typeof freeagentCategories.$inferSelect +export type FreeAgentContactRow = typeof freeagentContacts.$inferSelect +export type FreeAgentBillRow = typeof freeagentBills.$inferSelect +export type FreeAgentInvoiceRow = typeof freeagentInvoices.$inferSelect diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/db/schema/index.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/db/schema/index.ts new file mode 100644 index 0000000..4289210 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/db/schema/index.ts @@ -0,0 +1,2 @@ +/** What `apps/web/src/plugins/schema.ts` re-exports for this plugin. */ +export * from './freeagent' diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/index.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/index.ts new file mode 100644 index 0000000..29ac189 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/index.ts @@ -0,0 +1,66 @@ +/** + * `freeagent` — the server entry (D31). Read top to bottom: the authed mount, the public OAuth + * callback, the one job, the half-hourly cron, the agent tools, the grants and the analytics + * contribution. + * + * What it exports beyond the `ServerPlugin` is the service API another plugin builds on — a + * receipts matcher explains transactions and attaches receipts through these, and rematches after + * each sync through `FreeAgentSyncListener` (`freeagent:sync-listeners`). They are part of this + * plugin's four-entry surface, and its semver covers them. + */ +import { + FREEAGENT_SUBJECT, + FREEAGENT_SYNC_JOB, + freeagentShared, +} from '@rocketflare/shared/plugins/freeagent/index' +import type { ServerPlugin } from '@/plugins/api' +import { toolCtx } from '@/plugins/api' +import { freeagentHooksRouter } from './api/hooks' +import { freeagentRouter } from './api/routes' +import { enqueueSyncsTask, FREEAGENT_CRON } from './api/scheduled' +import { FREEAGENT_ANALYTICS_EXTENSIONS } from './cubes' +import { handleFreeagentSync } from './jobs/sync' +import { freeagentAgentTools } from './tools/freeagent-tools' + +export type { FreeAgentClient, Query as FreeAgentQuery } from './services/client' +export type { FreeAgentCtx } from './services/connection' +export { freeagentClientFor } from './services/connection' +export type { FreeAgentErrorCode } from './services/errors' +export { FreeAgentError, isFreeAgentError } from './services/errors' +export type { FreeAgentSyncListener, FreeAgentSyncSummary } from './services/sync' +export type { + AttachmentInput, + BillView, + ExplanationView, + UnexplainedTransaction, +} from './services/writes' +export { + attachToExplanation, + createBill, + deleteExplanation, + explainTransaction, + listUnexplainedTransactions, +} from './services/writes' + +export const freeagentServer = { + shared: freeagentShared, + mounts: [['/api/freeagent', freeagentRouter]], + /** `/api/hooks/` is the only prefix a public mount may take — the kit's config test says so. */ + publicMounts: [['/api/hooks/freeagent', freeagentHooksRouter]], + jobHandlers: { [FREEAGENT_SYNC_JOB]: handleFreeagentSync }, + scheduledTasks: { [FREEAGENT_CRON]: [enqueueSyncsTask] }, + agentTools: raw => freeagentAgentTools(toolCtx(raw)), + /** + * Additive, over this plugin's own subject. Owners and admins connect, disconnect and sync; + * everyone may read the mirrored books (the read tools are offered to every member's runs, so + * the page should not pretend otherwise). Writes through tools are admin-only and approved. + */ + grants: { + owner: can => can('manage', FREEAGENT_SUBJECT), + admin: can => can('manage', FREEAGENT_SUBJECT), + support: can => can('read', FREEAGENT_SUBJECT), + member: can => can('read', FREEAGENT_SUBJECT), + }, + /** Inert unless the analytics plugin is installed — see `cubes/index.ts`. */ + extensions: FREEAGENT_ANALYTICS_EXTENSIONS, +} satisfies ServerPlugin diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/jobs/sync.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/jobs/sync.ts new file mode 100644 index 0000000..08361bb --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/jobs/sync.ts @@ -0,0 +1,18 @@ +/** + * `freeagent.sync` — registered in `ServerPlugin.jobHandlers`. The work is `services/sync.ts`; + * this is the adapter. A throw (rate limited, FreeAgent down, Postgres unavailable) retries with + * the kit's backoff, which is safe because the sync is incremental and idempotent; a connection + * that needs reconnecting returns normally, because no retry can fix it. + */ +import type { JobOf } from '@rocketflare/shared/jobs' +import type { FREEAGENT_SYNC_JOB } from '@rocketflare/shared/plugins/freeagent/index' +import type { JobHandler } from '@/plugins/api' +import { jobCtx } from '@/plugins/api' +import { syncTenant } from '../services/sync' + +export const handleFreeagentSync: JobHandler = async ( + job: JobOf, + ctx +) => { + await syncTenant(jobCtx(ctx), job.payload.tenantId) +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/plugin.json b/plugins/freeagent/apps/web/src/plugins/freeagent/plugin.json new file mode 100644 index 0000000..efe2995 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/plugin.json @@ -0,0 +1,72 @@ +{ + "id": "freeagent", + "label": "FreeAgent", + "version": "3.5.1", + "repo": "https://github.com/rocketflare-dev/rocketflare-plugins.git", + "minKit": "0.21.0", + "anchor": "apps/web/src/plugins/freeagent/plugin.json", + "paths": [ + "apps/web/src/plugins/freeagent/**", + "packages/shared/src/plugins/freeagent/**", + "apps/cli/src/plugins/freeagent/**", + "docs/plugins/freeagent/**" + ], + "registries": [ + "apps/web/src/plugins/server.ts", + "apps/web/src/plugins/ui.ts", + "apps/web/src/plugins/schema.ts", + "packages/shared/src/plugins/index.ts", + "apps/cli/src/plugins/index.ts" + ], + "requires": { + "surfaces": [], + "plugins": [] + }, + "uses": {}, + "dependencies": { + "apps/web": {}, + "packages/shared": {}, + "apps/cli": {} + }, + "bindings": [], + "crons": ["*/30 * * * *"], + "apiPrefixes": [], + "coreEdits": [], + "vars": [ + { + "key": "FREEAGENT_CLIENT_ID", + "secret": true + }, + { + "key": "FREEAGENT_CLIENT_SECRET", + "secret": true + }, + { + "key": "FREEAGENT_ENVIRONMENT", + "example": "sandbox", + "secret": true + } + ], + "workerExports": [], + "skills": ["freeagent"], + "schema": { + "tables": [ + "freeagent_connections", + "freeagent_sync_state", + "freeagent_bank_accounts", + "freeagent_bank_transactions", + "freeagent_explanations", + "freeagent_categories", + "freeagent_contacts", + "freeagent_bills", + "freeagent_invoices" + ], + "rlsExcluded": [] + }, + "migrations": [ + "freeagent_connections — one row per tenant (unique tenant_id): environment, status, sealed refresh/access tokens, company, sync claim and last outcome; RLS policy", + "freeagent_sync_state — one updated_since cursor per (tenant, resource), unique; RLS policy", + "freeagent_bank_accounts / freeagent_bank_transactions / freeagent_explanations / freeagent_categories / freeagent_contacts / freeagent_bills / freeagent_invoices — the mirror, each unique on (tenant_id, url), money as numeric; RLS policies" + ], + "subdir": "plugins/freeagent" +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/client.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/client.ts new file mode 100644 index 0000000..29dd3d7 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/client.ts @@ -0,0 +1,229 @@ +/** + * `FreeAgentClient` — the one typed fetch wrapper every call to FreeAgent goes through. + * + * What it does, so no caller has to: + * + * - **Headers.** `Authorization: Bearer`, JSON both ways, a `User-Agent` (FreeAgent requires + * one) and ALWAYS `X-Api-Version: 2026-09-01`, which opts into multiple attachments per + * explanation (`attachments[]`; the single `attachment` field goes on 1 Dec 2026). + * - **Tokens.** It asks its `accessToken` source for a token per request; the source refreshes + * within two minutes of expiry, single-flight (`services/tokens.ts`). A 401 forces ONE refresh + * and one retry — a token revoked early looks exactly like that. + * - **429.** FreeAgent allows 120 requests a minute and 3600 an hour per user. A 429 is waited out + * for its `Retry-After` (capped at a minute) up to three times, then becomes a + * `FreeAgentError('rate_limited')` carrying `retryAfterSeconds` — a sync job throws it and the + * queue's own backoff takes over, which is cheaper than holding a Worker open for an hour. + * - **Pagination.** `per_page=100` (the maximum) and the `Link` header's `rel="next"`. + * - **Scope of the token.** Every URL it fetches — a path, a resource URL a caller passed in, a + * `Link` it was handed — must be on the connection's API host. Anything else is refused BEFORE + * the request, because the bearer token would go with it. + * - **Errors.** Anything not OK is a `FreeAgentError` with a code and FreeAgent's own message. + */ +import { FREEAGENT_API_VERSION } from '@rocketflare/shared/plugins/freeagent/index' +import type { FreeAgentDeps } from './env' +import { codeForStatus, FreeAgentError, freeagentErrorMessage } from './errors' + +export const FREEAGENT_USER_AGENT = 'Rocketflare FreeAgent plugin (+https://rocketflare.dev)' +export const FREEAGENT_PER_PAGE = 100 +const MAX_RATE_LIMIT_RETRIES = 3 +const MAX_RETRY_AFTER_MS = 60_000 +/** A runaway `Link` chain is a bug upstream, not a reason to page forever. */ +const MAX_PAGES = 500 + +export type QueryValue = string | number | boolean | null | undefined +export type Query = Record + +/** Supplies a bearer token; `force` means the last one was refused — refresh regardless of expiry. */ +export type AccessTokenSource = (opts: { force: boolean }) => Promise + +export interface FreeAgentClientOptions { + baseUrl: string + accessToken: AccessTokenSource + deps: FreeAgentDeps +} + +/** `; rel="next", <…>; rel="last"` → `{ next, last }`. Pure. */ +export function parseLinkHeader(header: string | null): Record { + const links: Record = {} + if (!header) return links + for (const part of header.split(',')) { + const match = /<([^>]+)>\s*;\s*rel="?([^";]+)"?/.exec(part.trim()) + if (match?.[1] && match[2]) links[match[2]] = match[1] + } + return links +} + +/** `Retry-After` as milliseconds: delta-seconds or an HTTP date. Null when absent or unreadable. */ +export function retryAfterMs(header: string | null, now = Date.now()): number | null { + if (!header) return null + const seconds = Number(header) + if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000) + const at = Date.parse(header) + return Number.isNaN(at) ? null : Math.max(0, at - now) +} + +export class FreeAgentClient { + readonly baseUrl: string + readonly #origin: string + readonly #accessToken: AccessTokenSource + readonly #deps: FreeAgentDeps + + constructor(opts: FreeAgentClientOptions) { + this.baseUrl = opts.baseUrl.replace(/\/+$/, '') + this.#origin = new URL(this.baseUrl).origin + this.#accessToken = opts.accessToken + this.#deps = opts.deps + } + + /** + * A path (`/v2/company`) or a full URL on this connection's host → a URL. Throws `bad_response` + * for anything off-host: a resource URL is caller-supplied, and the token must never leave. + */ + resolve(pathOrUrl: string, query: Query = {}): URL { + let url: URL + try { + url = new URL(pathOrUrl, `${this.baseUrl}/`) + } catch { + throw new FreeAgentError('bad_response', `Not a FreeAgent URL: ${pathOrUrl}`) + } + if (url.origin !== this.#origin || !url.pathname.startsWith('/v2/')) { + throw new FreeAgentError( + 'bad_response', + `Refusing a URL outside this connection's FreeAgent API (${this.#origin}/v2/)` + ) + } + for (const [k, v] of Object.entries(query)) { + if (v !== undefined && v !== null && v !== '') url.searchParams.set(k, String(v)) + } + return url + } + + /** True when `url` is a resource on this connection's API — for validating caller input. */ + owns(url: string): boolean { + try { + this.resolve(url) + return true + } catch { + return false + } + } + + async request( + method: 'GET' | 'POST' | 'PUT' | 'DELETE', + pathOrUrl: string, + opts: { query?: Query; body?: unknown } = {} + ): Promise<{ data: T; response: Response }> { + const url = this.resolve(pathOrUrl, opts.query) + let forced = false + let rateLimited = 0 + for (;;) { + const token = await this.#accessToken({ force: forced }) + const headers = new Headers({ + Authorization: `Bearer ${token}`, + Accept: 'application/json', + 'User-Agent': FREEAGENT_USER_AGENT, + 'X-Api-Version': FREEAGENT_API_VERSION, + }) + if (opts.body !== undefined) headers.set('Content-Type', 'application/json') + let response: Response + try { + response = await this.#deps.fetch(url.toString(), { + method, + headers, + ...(opts.body !== undefined && { body: JSON.stringify(opts.body) }), + }) + } catch (err) { + throw new FreeAgentError( + 'network', + `FreeAgent did not answer: ${err instanceof Error ? err.message : String(err)}` + ) + } + + if (response.status === 401 && !forced) { + forced = true + continue + } + if (response.status === 429) { + const wait = retryAfterMs(response.headers.get('Retry-After')) ?? 1000 + if (rateLimited < MAX_RATE_LIMIT_RETRIES && wait <= MAX_RETRY_AFTER_MS) { + rateLimited++ + await this.#deps.sleep(wait) + continue + } + throw new FreeAgentError('rate_limited', 'FreeAgent rate limit reached; try again later', { + status: 429, + retryAfterSeconds: Math.ceil(wait / 1000), + }) + } + + const text = await response.text() + let body: unknown = null + if (text) { + try { + body = JSON.parse(text) + } catch { + if (response.ok) { + throw new FreeAgentError('bad_response', 'FreeAgent answered with something not JSON', { + status: response.status, + }) + } + } + } + if (!response.ok) { + const message = + freeagentErrorMessage(body) ?? `FreeAgent answered ${response.status} for ${method}` + throw new FreeAgentError(codeForStatus(response.status), message, { + status: response.status, + }) + } + return { data: body as T, response } + } + } + + async get(pathOrUrl: string, query?: Query): Promise { + return (await this.request('GET', pathOrUrl, { query })).data + } + + async post(pathOrUrl: string, body: unknown): Promise { + return (await this.request('POST', pathOrUrl, { body })).data + } + + async put(pathOrUrl: string, body: unknown): Promise { + return (await this.request('PUT', pathOrUrl, { body })).data + } + + async delete(pathOrUrl: string): Promise { + await this.request('DELETE', pathOrUrl) + } + + /** + * Every page of a list endpoint, one array per page. `key` is the list's root element + * (`bank_transactions`). Follows `Link: rel="next"` — through `resolve`, so a next link pointing + * anywhere else stops the walk with an error rather than leaking the token. + */ + async *pages(path: string, key: string, query: Query = {}): AsyncGenerator { + let next: string | null = this.resolve(path, { + per_page: FREEAGENT_PER_PAGE, + ...query, + }).toString() + for (let page = 0; next && page < MAX_PAGES; page++) { + const { data, response }: { data: unknown; response: Response } = await this.request( + 'GET', + next + ) + const items = (data as Record | null)?.[key] + if (!Array.isArray(items)) { + throw new FreeAgentError('bad_response', `FreeAgent's ${path} answer has no "${key}" list`) + } + yield items as T[] + next = parseLinkHeader(response.headers.get('Link')).next ?? null + } + } + + /** Every item of a list endpoint, all pages. For small lists; the sync walks `pages`. */ + async all(path: string, key: string, query: Query = {}): Promise { + const out: T[] = [] + for await (const page of this.pages(path, key, query)) out.push(...page) + return out + } +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/connection.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/connection.ts new file mode 100644 index 0000000..8987671 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/connection.ts @@ -0,0 +1,176 @@ +/** + * The organisation's connection: read it, build a client for it, store a new one, remove it. + * + * `freeagentClientFor` is the front door every write and the sync use, and the one exported to + * other plugins: it answers a client bound to THIS tenant's tokens and API host, or throws a + * `FreeAgentError` saying exactly why it cannot (`not_configured`, `not_connected`, + * `needs_reconnect`). Every query here names the tenant; the cron's fan-out is the plugin's only + * cross-tenant read and lives in `api/scheduled.ts`. + */ +import type { FreeAgentEnvironment } from '@rocketflare/shared/plugins/freeagent/index' +import { eq } from 'drizzle-orm' +import type { Database, PluginConfig } from '@/plugins/api' +import { sealSecret } from '@/plugins/api' +import { + type FreeAgentConnectionRow, + freeagentBankAccounts, + freeagentBankTransactions, + freeagentBills, + freeagentCategories, + freeagentConnections, + freeagentContacts, + freeagentExplanations, + freeagentInvoices, + freeagentSyncState, +} from '../db/schema' +import { FreeAgentClient } from './client' +import { appCredentials, FREEAGENT_HOSTS, type FreeAgentDeps, freeagentHttp } from './env' +import { FreeAgentError } from './errors' +import type { TokenSet } from './tokens' +import { connectionTokenSource } from './tokens' + +/** What every service here needs from whichever context called it (request, job, tool). */ +export interface FreeAgentCtx { + db: Database + config: PluginConfig + logger?: { warn: (obj: object, msg: string) => void; error: (obj: object, msg: string) => void } +} + +export async function loadConnection( + db: Database, + tenantId: string +): Promise { + const [row] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + .limit(1) + return row +} + +/** True when tools and writes can work right now: a row, `connected`, and an app configured. */ +export async function isUsable(ctx: FreeAgentCtx, tenantId: string): Promise { + if (!appCredentials(ctx.config)) return false + const row = await loadConnection(ctx.db, tenantId) + return row?.status === 'connected' +} + +/** A client for an already-loaded connection row (the sync holds one from its claim). */ +export function clientForConnection( + ctx: FreeAgentCtx, + connection: FreeAgentConnectionRow, + deps: FreeAgentDeps = freeagentHttp +): FreeAgentClient { + const credentials = appCredentials(ctx.config) + if (!credentials) { + throw new FreeAgentError( + 'not_configured', + 'FreeAgent is not set up on this deployment (FREEAGENT_CLIENT_ID / FREEAGENT_CLIENT_SECRET)' + ) + } + if (connection.status !== 'connected') { + throw new FreeAgentError('needs_reconnect', 'Reconnect to FreeAgent in Settings → FreeAgent') + } + const environment = connection.environment as FreeAgentEnvironment + return new FreeAgentClient({ + baseUrl: FREEAGENT_HOSTS[environment] ?? FREEAGENT_HOSTS.sandbox, + accessToken: connectionTokenSource(ctx.db, ctx.config, connection, credentials, deps), + deps, + }) +} + +/** + * A FreeAgent client for one organisation — the exported entry point for other plugins. + * Throws `FreeAgentError` (`not_configured` | `not_connected` | `needs_reconnect`). + */ +export async function freeagentClientFor( + ctx: FreeAgentCtx, + tenantId: string, + deps: FreeAgentDeps = freeagentHttp +): Promise { + const connection = await loadConnection(ctx.db, tenantId) + if (!connection) { + throw new FreeAgentError('not_connected', 'This organisation has not connected FreeAgent') + } + return clientForConnection(ctx, connection, deps) +} + +/** Every mirrored row and cursor for one tenant — the books we can no longer keep current. */ +export async function clearMirror(db: Database, tenantId: string): Promise { + for (const table of [ + freeagentExplanations, + freeagentBankTransactions, + freeagentBankAccounts, + freeagentCategories, + freeagentContacts, + freeagentBills, + freeagentInvoices, + freeagentSyncState, + ]) { + await db.delete(table).where(eq(table.tenantId, tenantId)) + } +} + +/** + * Store a fresh approval: tokens sealed, the company recorded, status `connected`. A reconnect to + * a DIFFERENT company (another FreeAgent login) clears the old mirror first, so two companies' + * books are never mixed under one organisation; a reconnect to the same one keeps its cursors. + */ +export async function saveConnection( + ctx: FreeAgentCtx, + input: { + tenantId: string + userId: string + environment: FreeAgentEnvironment + tokens: TokenSet + company: { + companyUrl: string | null + companyName: string | null + companyCurrency: string | null + } + } +): Promise { + if (!input.tokens.refreshToken) { + throw new FreeAgentError('bad_response', 'FreeAgent did not issue a refresh token') + } + const existing = await loadConnection(ctx.db, input.tenantId) + if ( + existing && + (existing.environment !== input.environment || + (existing.companyUrl && existing.companyUrl !== input.company.companyUrl)) + ) { + await clearMirror(ctx.db, input.tenantId) + } + const values = { + environment: input.environment, + status: 'connected', + refreshTokenEnc: await sealSecret(ctx.config, input.tokens.refreshToken), + accessTokenEnc: await sealSecret(ctx.config, input.tokens.accessToken), + accessTokenExpiresAt: input.tokens.expiresAt, + companyUrl: input.company.companyUrl, + companyName: input.company.companyName, + companyCurrency: input.company.companyCurrency, + connectedByUserId: input.userId, + connectedAt: new Date(), + syncClaimedUntil: null, + lastError: null, + lastErrorAt: null, + } + const [row] = await ctx.db + .insert(freeagentConnections) + .values({ tenantId: input.tenantId, ...values }) + .onConflictDoUpdate({ target: freeagentConnections.tenantId, set: values }) + .returning() + if (!row) throw new Error('freeagent_connections: upsert returned no row') + return row +} + +/** Disconnect: the tokens go, and with them the mirror and its cursors. */ +export async function deleteConnection(db: Database, tenantId: string): Promise { + const deleted = await db + .delete(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + .returning({ id: freeagentConnections.id }) + await clearMirror(db, tenantId) + return deleted.length > 0 +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/env.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/env.ts new file mode 100644 index 0000000..0cd8670 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/env.ts @@ -0,0 +1,86 @@ +/** + * Where FreeAgent lives for this deployment, and the outbound `fetch` everything uses. + * + * Sandbox and production are two hosts with the same paths. A connection remembers the + * environment it was made in, so changing `FREEAGENT_ENVIRONMENT` never sends a production token + * to the sandbox: existing connections keep their host and Settings shows the mismatch. + * + * `freeagentHttp` is the one place a real `fetch` (and a real timer) is named. Every service takes + * `deps` defaulting to it, so a unit test passes its own; the API tests that drive the real routes + * swap `freeagentHttp.fetch` for the length of a test and put it back. No global is stubbed. + */ +import { + type FreeAgentConfig, + type FreeAgentEnvironment, + freeagentConfigSchema, +} from '@rocketflare/shared/plugins/freeagent/index' +import type { PluginConfig } from '@/plugins/api' + +export const FREEAGENT_HOSTS: Record = { + sandbox: 'https://api.sandbox.freeagent.com', + production: 'https://api.freeagent.com', +} + +/** The public mount's callback — what the operator registers as the app's redirect URI. */ +export const FREEAGENT_CALLBACK_PATH = '/api/hooks/freeagent/callback' + +/** `fetch` as the client calls it. Injected so tests never reach FreeAgent. */ +export type FetchLike = (input: string | URL | Request, init?: RequestInit) => Promise + +export interface FreeAgentDeps { + fetch: FetchLike + /** Waits out a 429's `Retry-After`. Injected so a test does not actually wait. */ + sleep: (ms: number) => Promise +} + +/** Unbound `fetch` throws "Illegal invocation" on Workers once it is called as a method. */ +export const freeagentHttp: FreeAgentDeps = { + fetch: (input, init) => fetch(input, init), + sleep: ms => new Promise(resolve => setTimeout(resolve, ms)), +} + +/** This plugin's slice of the Worker config, parsed (defaults applied). */ +export function freeagentConfig(config: PluginConfig): FreeAgentConfig { + return freeagentConfigSchema.parse(config) +} + +export interface FreeAgentAppCredentials { + clientId: string + clientSecret: string +} + +/** The operator's app, or null when the deployment has not set both secrets. */ +export function appCredentials(config: PluginConfig): FreeAgentAppCredentials | null { + const c = freeagentConfig(config) + if (!c.FREEAGENT_CLIENT_ID || !c.FREEAGENT_CLIENT_SECRET) return null + return { clientId: c.FREEAGENT_CLIENT_ID, clientSecret: c.FREEAGENT_CLIENT_SECRET } +} + +export function redirectUri(appUrl: string): string { + return new URL(FREEAGENT_CALLBACK_PATH, appUrl).toString() +} + +/** `GET /v2/approve_app` — FreeAgent's consent page, which redirects back with `code` + `state`. */ +export function approveUrl( + environment: FreeAgentEnvironment, + params: { clientId: string; redirectUri: string; state: string } +): string { + const url = new URL('/v2/approve_app', FREEAGENT_HOSTS[environment]) + url.searchParams.set('client_id', params.clientId) + url.searchParams.set('response_type', 'code') + url.searchParams.set('redirect_uri', params.redirectUri) + url.searchParams.set('state', params.state) + return url.toString() +} + +/** Where the admin lands afterwards: Settings → FreeAgent, with the outcome in the query. */ +export function settingsReturnUrl( + appUrl: string, + outcome: { connected: true } | { error: string } +): string { + const url = new URL('/settings', appUrl) + url.searchParams.set('tab', 'freeagent') + if ('connected' in outcome) url.searchParams.set('connected', 'freeagent') + else url.searchParams.set('connectError', outcome.error) + return url.toString() +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/errors.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/errors.ts new file mode 100644 index 0000000..ed13c84 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/errors.ts @@ -0,0 +1,93 @@ +/** + * `FreeAgentError` — every way talking to FreeAgent can fail, as ONE typed error with a code. + * + * Callers branch on `code`, never on a message: the sync decides retry-or-stop from it, the routes + * map it to a status, and a tool turns it into a hint the model can act on. The message is safe to + * show and to log — it never carries a token, a code or a client secret, because nothing that + * builds one puts them in. + */ + +export const FREEAGENT_ERROR_CODES = [ + /** The deployment has no FreeAgent app (`FREEAGENT_CLIENT_ID` / `_SECRET`). */ + 'not_configured', + /** This organisation has no connection. */ + 'not_connected', + /** FreeAgent refused the refresh token (`invalid_grant`): an admin must reconnect. */ + 'needs_reconnect', + /** The token endpoint said `invalid_grant` — internal; surfaced to callers as `needs_reconnect`. */ + 'invalid_grant', + /** 429 after the bounded back-off; `retryAfterSeconds` says when. */ + 'rate_limited', + 'unauthorized', + 'forbidden', + 'not_found', + /** 422 / 400: FreeAgent rejected what we sent. The message is FreeAgent's. */ + 'validation', + /** 5xx. */ + 'upstream', + /** The request never got an answer. */ + 'network', + /** An answer we could not read, or a URL outside the connection's API host. */ + 'bad_response', +] as const +export type FreeAgentErrorCode = (typeof FREEAGENT_ERROR_CODES)[number] + +export class FreeAgentError extends Error { + readonly code: FreeAgentErrorCode + readonly status: number | null + readonly retryAfterSeconds: number | null + + constructor( + code: FreeAgentErrorCode, + message: string, + opts: { status?: number | null; retryAfterSeconds?: number | null } = {} + ) { + super(message) + this.name = 'FreeAgentError' + this.code = code + this.status = opts.status ?? null + this.retryAfterSeconds = opts.retryAfterSeconds ?? null + } +} + +export function isFreeAgentError(err: unknown, code?: FreeAgentErrorCode): err is FreeAgentError { + return err instanceof FreeAgentError && (code === undefined || err.code === code) +} + +/** The code an HTTP status maps to, for a response that was not OK and not a 429. */ +export function codeForStatus(status: number): FreeAgentErrorCode { + if (status === 401) return 'unauthorized' + if (status === 403) return 'forbidden' + if (status === 404) return 'not_found' + if (status === 400 || status === 422) return 'validation' + if (status >= 500) return 'upstream' + return 'bad_response' +} + +/** + * FreeAgent's error body, read loosely: `{ errors: { error: { message } } }`, + * `{ errors: [{ message }] }`, `{ errors: { error: [...] } }`, or OAuth's + * `{ error, error_description }`. Answers the first human message found, or null. + */ +export function freeagentErrorMessage(body: unknown): string | null { + if (!body || typeof body !== 'object') return null + const record = body as Record + if (typeof record.error_description === 'string') return record.error_description + const messages: string[] = [] + const visit = (value: unknown, depth: number) => { + if (depth > 4 || messages.length >= 3) return + if (typeof value === 'string') return + if (Array.isArray(value)) { + for (const v of value) visit(v, depth + 1) + return + } + if (value && typeof value === 'object') { + const v = value as Record + if (typeof v.message === 'string') messages.push(v.message) + else for (const child of Object.values(v)) visit(child, depth + 1) + } + } + visit(record.errors, 0) + if (messages.length > 0) return messages.join('; ') + return typeof record.error === 'string' ? record.error : null +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/mapping.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/mapping.ts new file mode 100644 index 0000000..cd90fe3 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/mapping.ts @@ -0,0 +1,376 @@ +/** + * FreeAgent JSON → mirror rows. Pure, so the whole translation is table-tested without a database + * (`tests/config/mapping.test.ts`). + * + * Each mapper parses with a PERMISSIVE zod schema (`passthrough`, everything but the URL optional) + * and fails loudly on the one thing it cannot do without: a resource with no `url` is unusable, + * because the URL is the key. Field names are FreeAgent's, checked against dev.freeagent.com. + * + * Money stays a decimal STRING from end to end. FreeAgent sends `"-730.0"`; Postgres `numeric` + * takes it as is; nothing here ever goes through `Number`. + */ +import { z } from 'zod' +import { FreeAgentError } from './errors' + +/** FreeAgent sends decimals as strings, but be lenient about a bare number. */ +const decimal = z + .union([z.string(), z.number()]) + .transform(v => String(v).trim()) + .refine(v => /^-?\d+(\.\d+)?$/.test(v), 'not a decimal') +const optionalDecimal = decimal.nullish().transform(v => v ?? null) +const optionalString = z + .string() + .nullish() + .transform(v => (v === undefined || v === null || v === '' ? null : v)) +const optionalDate = z + .string() + .nullish() + .transform(v => (v && /^\d{4}-\d{2}-\d{2}/.test(v) ? v.slice(0, 10) : null)) +const optionalTimestamp = z + .string() + .nullish() + .transform(v => { + if (!v) return null + const d = new Date(v) + return Number.isNaN(d.getTime()) ? null : d + }) +const optionalBoolean = z + .union([z.boolean(), z.string()]) + .nullish() + .transform(v => (v === undefined || v === null ? null : v === true || v === 'true')) + +function parse(schema: z.ZodType, raw: unknown, what: string): T { + const result = schema.safeParse(raw) + if (!result.success) { + const issue = result.error.issues[0] + throw new FreeAgentError( + 'bad_response', + `Unreadable FreeAgent ${what}: ${issue?.path.join('.') || ''} ${issue?.message ?? ''}`.trim() + ) + } + return result.data +} + +/** `"0.0"`, `"-0"`, `"0"` → false; anything else that is a decimal → true. */ +export function isNonZeroDecimal(value: string | null | undefined): boolean { + if (!value) return false + return !/^-?0*(\.0*)?$/.test(value.trim()) +} + +// ---- Bank accounts ------------------------------------------------------------------------------ + +const bankAccountSchema = z + .object({ + url: z.string().url(), + name: z.string().nullish(), + type: optionalString, + currency: optionalString, + current_balance: optionalDecimal, + bank_name: optionalString, + is_primary: optionalBoolean, + status: optionalString, + updated_at: optionalTimestamp, + }) + .passthrough() + +export function mapBankAccount(raw: unknown) { + const a = parse(bankAccountSchema, raw, 'bank account') + return { + url: a.url, + name: a.name || a.bank_name || 'Bank account', + type: a.type, + currency: a.currency, + currentBalance: a.current_balance, + bankName: a.bank_name, + isPrimary: a.is_primary ?? false, + status: a.status, + faUpdatedAt: a.updated_at, + } +} + +// ---- Explanations ------------------------------------------------------------------------------- + +const explanationSchema = z + .object({ + url: z.string().url(), + bank_transaction: optionalString, + bank_account: optionalString, + category: optionalString, + type: optionalString, + entry_type: optionalString, + dated_on: optionalDate, + description: optionalString, + gross_value: optionalDecimal, + sales_tax_rate: optionalDecimal, + sales_tax_value: optionalDecimal, + is_deletable: optionalBoolean, + /** `X-Api-Version: 2026-09-01` — an array of `{ url }`. */ + attachments: z.array(z.unknown()).nullish(), + /** The pre-2026-09-01 single attachment, for a response that still carries it. */ + attachment: z.unknown().nullish(), + }) + .passthrough() + +export function mapExplanation(raw: unknown) { + const e = parse(explanationSchema, raw, 'bank transaction explanation') + return { + url: e.url, + bankTransactionUrl: e.bank_transaction, + bankAccountUrl: e.bank_account, + categoryUrl: e.category, + entryType: e.type ?? e.entry_type, + datedOn: e.dated_on, + description: e.description, + grossValue: e.gross_value, + salesTaxRate: e.sales_tax_rate, + salesTaxValue: e.sales_tax_value, + attachmentCount: Array.isArray(e.attachments) ? e.attachments.length : e.attachment ? 1 : 0, + isDeletable: e.is_deletable, + } +} + +/** + * The SUBSET a bank transaction embeds in `bank_transaction_explanations` (no category, no tax): + * written on conflict without touching the columns only the explanations endpoint knows. + */ +export function mapEmbeddedExplanation(raw: unknown, transactionUrl: string) { + const e = mapExplanation(raw) + return { + url: e.url, + bankTransactionUrl: e.bankTransactionUrl ?? transactionUrl, + bankAccountUrl: e.bankAccountUrl, + entryType: e.entryType, + datedOn: e.datedOn, + description: e.description, + grossValue: e.grossValue, + isDeletable: e.isDeletable, + } +} + +// ---- Bank transactions -------------------------------------------------------------------------- + +const bankTransactionSchema = z + .object({ + url: z.string().url(), + bank_account: z.string(), + dated_on: z.string().regex(/^\d{4}-\d{2}-\d{2}/), + amount: decimal, + description: optionalString, + full_description: optionalString, + unexplained_amount: optionalDecimal, + is_manual: optionalBoolean, + created_at: optionalTimestamp, + updated_at: optionalTimestamp, + bank_transaction_explanations: z.array(z.unknown()).nullish(), + }) + .passthrough() + +export function mapBankTransaction(raw: unknown) { + const t = parse(bankTransactionSchema, raw, 'bank transaction') + const unexplained = t.unexplained_amount ?? '0' + return { + row: { + url: t.url, + bankAccountUrl: t.bank_account, + datedOn: t.dated_on.slice(0, 10), + amount: t.amount, + description: t.description, + fullDescription: t.full_description, + unexplainedAmount: unexplained, + isUnexplained: isNonZeroDecimal(unexplained), + isManual: t.is_manual ?? false, + faCreatedAt: t.created_at, + faUpdatedAt: t.updated_at, + }, + /** Null when the answer did not include the list at all — then nothing may be reconciled. */ + explanations: + t.bank_transaction_explanations === undefined || t.bank_transaction_explanations === null + ? null + : t.bank_transaction_explanations.map(e => mapEmbeddedExplanation(e, t.url)), + } +} + +// ---- Categories --------------------------------------------------------------------------------- + +export const CATEGORY_KINDS = { + admin_expenses_categories: 'admin_expenses', + cost_of_sales_categories: 'cost_of_sales', + income_categories: 'income', + general_categories: 'general', +} as const + +const categorySchema = z + .object({ + url: z.string().url(), + description: z.string().nullish(), + nominal_code: optionalString, + group_description: optionalString, + allowable_for_tax: optionalBoolean, + tax_reporting_name: optionalString, + auto_sales_tax_rate: z + .union([z.string(), z.number()]) + .nullish() + .transform(v => (v === undefined || v === null ? null : String(v))), + }) + .passthrough() + +/** `GET /v2/categories` answers four lists under four keys; flatten them, keeping the key. */ +export function mapCategories(raw: unknown) { + if (!raw || typeof raw !== 'object') { + throw new FreeAgentError('bad_response', 'Unreadable FreeAgent categories answer') + } + const out = [] + for (const [key, kind] of Object.entries(CATEGORY_KINDS)) { + const list = (raw as Record)[key] + if (!Array.isArray(list)) continue + for (const item of list) { + const c = parse(categorySchema, item, 'category') + out.push({ + url: c.url, + description: c.description || c.nominal_code || c.url, + nominalCode: c.nominal_code, + kind, + groupDescription: c.group_description, + allowableForTax: c.allowable_for_tax, + taxReportingName: c.tax_reporting_name, + autoSalesTaxRate: c.auto_sales_tax_rate, + }) + } + } + return out +} + +// ---- Contacts ----------------------------------------------------------------------------------- + +const contactSchema = z + .object({ + url: z.string().url(), + organisation_name: optionalString, + first_name: optionalString, + last_name: optionalString, + email: optionalString, + status: optionalString, + updated_at: optionalTimestamp, + }) + .passthrough() + +export function mapContact(raw: unknown) { + const c = parse(contactSchema, raw, 'contact') + const person = [c.first_name, c.last_name].filter(Boolean).join(' ') + return { + url: c.url, + displayName: c.organisation_name || person || c.email || c.url, + organisationName: c.organisation_name, + firstName: c.first_name, + lastName: c.last_name, + email: c.email, + status: c.status, + faUpdatedAt: c.updated_at, + } +} + +// ---- Bills and invoices ------------------------------------------------------------------------- + +const documentSchema = z + .object({ + url: z.string().url(), + contact: optionalString, + reference: optionalString, + dated_on: optionalDate, + due_on: optionalDate, + paid_on: optionalDate, + status: optionalString, + currency: optionalString, + total_value: optionalDecimal, + net_value: optionalDecimal, + sales_tax_value: optionalDecimal, + paid_value: optionalDecimal, + due_value: optionalDecimal, + updated_at: optionalTimestamp, + }) + .passthrough() + +export function mapBill(raw: unknown) { + const b = parse(documentSchema, raw, 'bill') + return { + url: b.url, + contactUrl: b.contact, + reference: b.reference, + datedOn: b.dated_on, + dueOn: b.due_on, + paidOn: b.paid_on, + status: b.status, + currency: b.currency, + totalValue: b.total_value, + netValue: b.net_value, + salesTaxValue: b.sales_tax_value, + paidValue: b.paid_value, + dueValue: b.due_value, + faUpdatedAt: b.updated_at, + } +} + +export function mapInvoice(raw: unknown) { + const i = parse(documentSchema, raw, 'invoice') + return { + url: i.url, + contactUrl: i.contact, + reference: i.reference, + datedOn: i.dated_on, + dueOn: i.due_on, + paidOn: i.paid_on, + status: i.status, + currency: i.currency, + netValue: i.net_value, + totalValue: i.total_value, + paidValue: i.paid_value, + dueValue: i.due_value, + faUpdatedAt: i.updated_at, + } +} + +// ---- Company ------------------------------------------------------------------------------------ + +const companySchema = z.object({ + company: z + .object({ + url: optionalString, + name: z.string().nullish(), + currency: optionalString, + }) + .passthrough(), +}) + +export function mapCompany(raw: unknown) { + const { company } = parse(companySchema, raw, 'company') + return { + companyUrl: company.url, + companyName: company.name ?? null, + companyCurrency: company.currency, + } +} + +// ---- Overdue ------------------------------------------------------------------------------------ + +/** Statuses after which nothing is owed, or before which nothing is due yet. */ +export const SETTLED_STATUSES = [ + 'Draft', + 'Scheduled To Email', + 'Paid', + 'Overpaid', + 'Refunded', + 'Written-off', + 'Zero Value', + 'Cancelled', +] as const + +/** An invoice or bill is overdue when something is still owed and its due date has passed. */ +export function isOverdue( + doc: { status: string | null; dueOn: string | null; dueValue: string | null }, + today: string +): boolean { + if (doc.status === 'Overdue') return true + if (!doc.dueOn || !isNonZeroDecimal(doc.dueValue)) return false + if (doc.status && (SETTLED_STATUSES as readonly string[]).includes(doc.status)) return false + return doc.dueOn < today +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/mirror.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/mirror.ts new file mode 100644 index 0000000..863fe8d --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/mirror.ts @@ -0,0 +1,116 @@ +/** + * Writing the mirror: one batched UPSERT per page, keyed on `(tenant_id, url)`. + * + * The sync and the writes both come through here, so a row a write touched and the row the next + * sync brings back are the same row. A batch upsert sets each column from `excluded` — the incoming + * value — and only the columns the mapper produced, which is what lets the PARTIAL explanation a + * bank transaction embeds update its own fields without wiping the category the explanations + * endpoint wrote. + */ +import { and, eq, getTableColumns, inArray, notInArray, type SQL, sql } from 'drizzle-orm' +import type { PgTable } from 'drizzle-orm/pg-core' +import type { Database } from '@/plugins/api' +import { + freeagentBankAccounts, + freeagentBankTransactions, + freeagentBills, + freeagentCategories, + freeagentContacts, + freeagentExplanations, + freeagentInvoices, +} from '../db/schema' + +/** The mirror tables, all shaped `{ tenantId, url, … }` with a unique `(tenant_id, url)`. */ +export const MIRROR_TABLES = { + bankAccounts: freeagentBankAccounts, + bankTransactions: freeagentBankTransactions, + explanations: freeagentExplanations, + categories: freeagentCategories, + contacts: freeagentContacts, + bills: freeagentBills, + invoices: freeagentInvoices, +} as const +export type MirrorTable = (typeof MIRROR_TABLES)[keyof typeof MIRROR_TABLES] + +/** Postgres caps a statement at 65 535 parameters; 200 rows × ~15 columns is far below it. */ +const BATCH = 200 + +/** + * Upsert `rows` (each already mapped, without `tenantId`) into one mirror table for one tenant. + * Every row in a call must carry the same keys — true of every mapper's output. + */ +export async function upsertMirror( + db: Database, + table: MirrorTable, + tenantId: string, + rows: readonly Record[] +): Promise { + if (rows.length === 0) return 0 + const columns = getTableColumns(table as PgTable) as Record + const keys = Object.keys(rows[0] ?? {}).filter(k => k !== 'url' && k in columns) + const set: Record = { updatedAt: new Date() } + for (const key of keys) { + const column = columns[key] + if (column) set[key] = sql.raw(`excluded."${column.name}"`) + } + const target = table as typeof freeagentBills + for (let i = 0; i < rows.length; i += BATCH) { + const values = rows.slice(i, i + BATCH).map(r => ({ ...r, tenantId })) + await db + .insert(target) + .values(values as (typeof freeagentBills.$inferInsert)[]) + .onConflictDoUpdate({ target: [target.tenantId, target.url], set }) + } + return rows.length +} + +/** + * A transaction's embedded explanation list is the truth about which explanations it HAS: one + * deleted in FreeAgent never comes back from `updated_since`, but its transaction does (its + * unexplained amount moved). So remove this transaction's rows whose URL is no longer listed. + */ +export async function pruneExplanations( + db: Database, + tenantId: string, + bankTransactionUrl: string, + keepUrls: readonly string[] +): Promise { + await db + .delete(freeagentExplanations) + .where( + and( + eq(freeagentExplanations.tenantId, tenantId), + eq(freeagentExplanations.bankTransactionUrl, bankTransactionUrl), + ...(keepUrls.length > 0 ? [notInArray(freeagentExplanations.url, [...keepUrls])] : []) + ) + ) +} + +/** Delete mirror rows by URL — categories and bank accounts are re-listed whole each sync. */ +export async function deleteMissing( + db: Database, + table: typeof freeagentCategories | typeof freeagentBankAccounts, + tenantId: string, + seenUrls: readonly string[] +): Promise { + const target = table as typeof freeagentCategories + await db + .delete(target) + .where( + and( + eq(target.tenantId, tenantId), + ...(seenUrls.length > 0 ? [notInArray(target.url, [...seenUrls])] : []) + ) + ) +} + +export async function deleteByUrl( + db: Database, + table: MirrorTable, + tenantId: string, + urls: readonly string[] +): Promise { + if (urls.length === 0) return + const target = table as typeof freeagentBills + await db.delete(target).where(and(eq(target.tenantId, tenantId), inArray(target.url, [...urls]))) +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/overdue.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/overdue.ts new file mode 100644 index 0000000..865b336 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/overdue.ts @@ -0,0 +1,29 @@ +/** + * "Overdue" and "outstanding" as SQL, over a bill or invoice table — the same rule `isOverdue` + * (`./mapping`) applies in TypeScript, so the page, the tools and the cubes agree on what the word + * means: something is still owed, and (for overdue) its due date has passed. FreeAgent's own + * `Overdue` status counts whatever the dates say, because it is FreeAgent's call. + */ +import { and, eq, gt, inArray, lt, not, or, type SQL, sql } from 'drizzle-orm' +import type { freeagentBills, freeagentInvoices } from '../db/schema' +import { SETTLED_STATUSES } from './mapping' + +type Doc = typeof freeagentBills | typeof freeagentInvoices + +/** Owed, and not a draft or settled document. */ +export function outstandingSql(table: Doc): SQL { + const t = table as typeof freeagentInvoices + return and( + gt(t.dueValue, '0'), + or(sql`${t.status} is null`, not(inArray(t.status, [...SETTLED_STATUSES]))) + ) as SQL +} + +/** Outstanding and past its due date (today, in the database's clock), or marked Overdue. */ +export function overdueSql(table: Doc): SQL { + const t = table as typeof freeagentInvoices + return or( + eq(t.status, 'Overdue'), + and(outstandingSql(table), lt(t.dueOn, sql`current_date`)) + ) as SQL +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/sync.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/sync.ts new file mode 100644 index 0000000..1c07b5d --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/sync.ts @@ -0,0 +1,413 @@ +/** + * `freeagent.sync`: one organisation's books → the mirror, incrementally. + * + * FreeAgent has no webhooks, so this polls. Order, and why: + * + * 1. **Claim** — one conditional UPDATE on the connection (`sync_claimed_until`). A second job for + * the same tenant (the cron overlapping "Sync now", a redelivery) sees the claim and acks + * without work. The claim expires on its own, so a job that died never wedges the tenant. + * 2. **Company** — name and currency, so Settings stays right after a rename. + * 3. **Bank accounts, categories** — small, and FreeAgent offers no `updated_since` for either, + * so both are listed whole and anything no longer listed is removed. + * 4. **Contacts, then per bank account: transactions and explanations, then bills, invoices** — + * incremental by `updated_since`, paged, one cursor each in `freeagent_sync_state`. + * Transactions run across EVERY bank account. Each transaction carries its explanation list, + * which reconciles deletions (`pruneExplanations`); the explanations endpoint then fills in the + * category and tax the embedded copy lacks. + * 5. **Listeners** — every `freeagent:sync-listeners` contribution, so a receipts matcher can + * rematch against what just changed. A listener's throw is logged; it fails nothing. + * + * **Cursors.** A cursor is the moment its pass STARTED, less a five-minute overlap for clock skew + * between us and FreeAgent, written only when that resource's pass completed. Upserts are + * idempotent, so the overlap costs a few re-read rows and never a missed one; a pass that failed + * half-way leaves its cursor where it was and the next run repeats it. + * + * **First sync.** Transactions and explanations start `FIRST_SYNC_LOOKBACK_DAYS` back + * (`from_date`) rather than at the company's beginning: a matcher wants this year's unexplained + * lines, and a decade of history would cost the rate limit an afternoon. + * + * **Failure.** A `rate_limited` or `upstream` error is rethrown so the queue retries with backoff; + * `needs_reconnect` (the token was refused) is recorded and NOT retried — no retry can fix it. + */ +import { + FREEAGENT_BOOKS_ENTITY, + FREEAGENT_CONNECTION_ENTITY, + FREEAGENT_EXTENSION_KEYS, +} from '@rocketflare/shared/plugins/freeagent/index' +import { and, eq, isNull, lt, or } from 'drizzle-orm' +import { z } from 'zod' +import type { JobCtx } from '@/plugins/api' +import { extensions } from '@/plugins/api/peers' +import { + type FreeAgentConnectionRow, + freeagentBankAccounts, + freeagentCategories, + freeagentConnections, + freeagentSyncState, +} from '../db/schema' +import type { FreeAgentClient, Query } from './client' +import { clientForConnection } from './connection' +import { type FreeAgentDeps, freeagentHttp } from './env' +import { FreeAgentError, isFreeAgentError } from './errors' +import { + mapBankAccount, + mapBankTransaction, + mapBill, + mapCategories, + mapCompany, + mapContact, + mapExplanation, + mapInvoice, +} from './mapping' +import { deleteMissing, MIRROR_TABLES, pruneExplanations, upsertMirror } from './mirror' + +/** How long one sync may hold the tenant before another may take over. */ +export const SYNC_CLAIM_MS = 15 * 60_000 +/** Overlap subtracted from a pass's start when it becomes the next `updated_since`. */ +export const CURSOR_OVERLAP_MS = 5 * 60_000 +export const FIRST_SYNC_LOOKBACK_DAYS = 400 + +/** What a listener is handed after a successful sync. */ +export interface FreeAgentSyncSummary { + tenantId: string + startedAt: Date + finishedAt: Date + /** Rows upserted per resource family in this pass. */ + counts: Record +} + +/** + * A plugin that wants to react to fresh books contributes one of these under + * `freeagent:sync-listeners` (`extensions: freeagentExtensions({ syncListeners: [l] })`). Called + * after every successful sync. Keep it quick — enqueue your own job — and idempotent: a sync with + * nothing new still calls it. A throw is logged; it does not stop the other listeners. + */ +export interface FreeAgentSyncListener { + id: string + onSync(summary: FreeAgentSyncSummary, ctx: JobCtx): Promise +} + +const listenerSchema = z.object({ + id: z.string().min(1), + onSync: z.custom(v => typeof v === 'function'), +}) + +/** Every installed listener, narrowed — a malformed contribution fails loudly, by plugin. */ +export function freeagentSyncListeners(): FreeAgentSyncListener[] { + return extensions(FREEAGENT_EXTENSION_KEYS.syncListeners).map(raw => { + const parsed = listenerSchema.safeParse(raw) + if (!parsed.success) { + throw new Error( + `freeagent: a contributed sync listener is malformed: ${parsed.error.message}` + ) + } + return parsed.data as FreeAgentSyncListener + }) +} + +export type SyncOutcome = + | { status: 'synced'; summary: FreeAgentSyncSummary } + | { status: 'skipped'; reason: 'not_connected' | 'claimed' } + | { status: 'needs_reconnect'; message: string } + +interface SyncOptions { + deps?: FreeAgentDeps + listeners?: readonly FreeAgentSyncListener[] + now?: () => Date +} + +/** Take the tenant's sync claim, or answer why not. */ +async function claim( + ctx: JobCtx, + tenantId: string, + now: Date +): Promise { + const [row] = await ctx.db + .update(freeagentConnections) + .set({ + syncClaimedUntil: new Date(now.getTime() + SYNC_CLAIM_MS), + lastSyncStartedAt: now, + }) + .where( + and( + eq(freeagentConnections.tenantId, tenantId), + eq(freeagentConnections.status, 'connected'), + or( + isNull(freeagentConnections.syncClaimedUntil), + lt(freeagentConnections.syncClaimedUntil, now) + ) + ) + ) + .returning() + if (row) return row + const [existing] = await ctx.db + .select({ status: freeagentConnections.status }) + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + .limit(1) + return existing?.status === 'connected' ? 'claimed' : 'not_connected' +} + +async function cursorFor(ctx: JobCtx, tenantId: string, resource: string): Promise { + const [row] = await ctx.db + .select({ updatedSince: freeagentSyncState.updatedSince }) + .from(freeagentSyncState) + .where( + and(eq(freeagentSyncState.tenantId, tenantId), eq(freeagentSyncState.resource, resource)) + ) + .limit(1) + return row?.updatedSince ?? null +} + +async function saveCursor( + ctx: JobCtx, + tenantId: string, + resource: string, + updatedSince: Date, + items: number, + now: Date +): Promise { + const values = { updatedSince, lastSyncedAt: now, itemsSynced: items } + await ctx.db + .insert(freeagentSyncState) + .values({ tenantId, resource, ...values }) + .onConflictDoUpdate({ + target: [freeagentSyncState.tenantId, freeagentSyncState.resource], + set: { ...values, updatedAt: now }, + }) +} + +function isoDate(d: Date): string { + return d.toISOString().slice(0, 10) +} + +/** + * One incremental resource: page through `path` from its cursor, hand each page to `onPage`, and + * move the cursor only once every page has landed. + */ +async function syncIncremental( + ctx: JobCtx, + client: FreeAgentClient, + tenantId: string, + spec: { + resource: string + path: string + key: string + query?: Query + /** On the first pass only — bounds history (`from_date`). */ + firstPassQuery?: Query + onPage: (items: unknown[]) => Promise + }, + passStartedAt: Date +): Promise { + const since = await cursorFor(ctx, tenantId, spec.resource) + const query: Query = { + ...spec.query, + ...(since ? { updated_since: since.toISOString() } : spec.firstPassQuery), + } + let count = 0 + for await (const page of client.pages(spec.path, spec.key, query)) { + count += await spec.onPage(page) + } + const next = new Date(passStartedAt.getTime() - CURSOR_OVERLAP_MS) + await saveCursor(ctx, tenantId, spec.resource, next, count, new Date()) + return count +} + +/** Every step after the claim. Throws; `syncTenant` records the outcome. */ +async function runSync( + ctx: JobCtx, + client: FreeAgentClient, + connection: FreeAgentConnectionRow, + startedAt: Date +): Promise> { + const { tenantId } = connection + const counts: Record = {} + + // Company: cheap, and keeps the name Settings shows current. + const company = mapCompany(await client.get('/v2/company')) + await ctx.db + .update(freeagentConnections) + .set({ companyName: company.companyName, companyCurrency: company.companyCurrency }) + .where( + and(eq(freeagentConnections.tenantId, tenantId), eq(freeagentConnections.id, connection.id)) + ) + + // Listed whole: no updated_since on either endpoint. + const accounts = (await client.all('/v2/bank_accounts', 'bank_accounts')).map(mapBankAccount) + counts.bankAccounts = await upsertMirror(ctx.db, MIRROR_TABLES.bankAccounts, tenantId, accounts) + await deleteMissing( + ctx.db, + freeagentBankAccounts, + tenantId, + accounts.map(a => a.url) + ) + + const categories = mapCategories(await client.get('/v2/categories')) + counts.categories = await upsertMirror(ctx.db, MIRROR_TABLES.categories, tenantId, categories) + await deleteMissing( + ctx.db, + freeagentCategories, + tenantId, + categories.map(c => c.url) + ) + + counts.contacts = await syncIncremental( + ctx, + client, + tenantId, + { + resource: 'contacts', + path: '/v2/contacts', + key: 'contacts', + query: { view: 'all' }, + onPage: items => + upsertMirror(ctx.db, MIRROR_TABLES.contacts, tenantId, items.map(mapContact)), + }, + startedAt + ) + + const firstPass: Query = { + from_date: isoDate(new Date(startedAt.getTime() - FIRST_SYNC_LOOKBACK_DAYS * 86_400_000)), + } + counts.bankTransactions = 0 + counts.explanations = 0 + for (const account of accounts) { + counts.bankTransactions += await syncIncremental( + ctx, + client, + tenantId, + { + resource: `bank_transactions:${account.url}`, + path: '/v2/bank_transactions', + key: 'bank_transactions', + query: { bank_account: account.url, view: 'all' }, + firstPassQuery: firstPass, + onPage: async items => { + const mapped = items.map(mapBankTransaction) + await upsertMirror( + ctx.db, + MIRROR_TABLES.bankTransactions, + tenantId, + mapped.map(m => m.row) + ) + for (const m of mapped) { + if (m.explanations === null) continue + await upsertMirror(ctx.db, MIRROR_TABLES.explanations, tenantId, m.explanations) + await pruneExplanations( + ctx.db, + tenantId, + m.row.url, + m.explanations.map(e => e.url) + ) + } + return mapped.length + }, + }, + startedAt + ) + counts.explanations += await syncIncremental( + ctx, + client, + tenantId, + { + resource: `bank_transaction_explanations:${account.url}`, + path: '/v2/bank_transaction_explanations', + key: 'bank_transaction_explanations', + query: { bank_account: account.url }, + firstPassQuery: firstPass, + onPage: items => + upsertMirror(ctx.db, MIRROR_TABLES.explanations, tenantId, items.map(mapExplanation)), + }, + startedAt + ) + } + + counts.bills = await syncIncremental( + ctx, + client, + tenantId, + { + resource: 'bills', + path: '/v2/bills', + key: 'bills', + query: { view: 'all' }, + onPage: items => upsertMirror(ctx.db, MIRROR_TABLES.bills, tenantId, items.map(mapBill)), + }, + startedAt + ) + counts.invoices = await syncIncremental( + ctx, + client, + tenantId, + { + resource: 'invoices', + path: '/v2/invoices', + key: 'invoices', + query: { view: 'all' }, + onPage: items => + upsertMirror(ctx.db, MIRROR_TABLES.invoices, tenantId, items.map(mapInvoice)), + }, + startedAt + ) + return counts +} + +export async function syncTenant( + ctx: JobCtx, + tenantId: string, + opts: SyncOptions = {} +): Promise { + const now = opts.now ?? (() => new Date()) + const startedAt = now() + const claimed = await claim(ctx, tenantId, startedAt) + if (claimed === 'not_connected' || claimed === 'claimed') { + return { status: 'skipped', reason: claimed } + } + const release = (patch: Partial) => + ctx.db + .update(freeagentConnections) + .set({ syncClaimedUntil: null, ...patch }) + .where( + and(eq(freeagentConnections.tenantId, tenantId), eq(freeagentConnections.id, claimed.id)) + ) + + let counts: Record + try { + const client = clientForConnection(ctx, claimed, opts.deps ?? freeagentHttp) + counts = await runSync(ctx, client, claimed, startedAt) + } catch (err) { + const message = err instanceof Error ? err.message : String(err) + if (isFreeAgentError(err, 'needs_reconnect')) { + // The token source already marked the row; just let go of the claim. + await release({}) + await ctx.nudge(tenantId, FREEAGENT_CONNECTION_ENTITY) + ctx.logger.warn({ tenantId }, 'freeagent.sync: the connection needs an admin to reconnect') + return { status: 'needs_reconnect', message } + } + await release({ lastError: message.slice(0, 1000), lastErrorAt: now() }) + await ctx.nudge(tenantId, FREEAGENT_CONNECTION_ENTITY) + ctx.logger.error( + { tenantId, code: err instanceof FreeAgentError ? err.code : 'internal', err: message }, + 'freeagent.sync failed' + ) + throw err + } + + const finishedAt = now() + await release({ lastSyncAt: finishedAt, lastError: null, lastErrorAt: null }) + const summary: FreeAgentSyncSummary = { tenantId, startedAt, finishedAt, counts } + for (const listener of opts.listeners ?? freeagentSyncListeners()) { + try { + await listener.onSync(summary, ctx) + } catch (err) { + ctx.logger.error( + { tenantId, listener: listener.id, err: err instanceof Error ? err.message : String(err) }, + 'freeagent: sync listener failed' + ) + } + } + await ctx.nudge(tenantId, FREEAGENT_CONNECTION_ENTITY) + await ctx.nudge(tenantId, FREEAGENT_BOOKS_ENTITY) + return { status: 'synced', summary } +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/tokens.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/tokens.ts new file mode 100644 index 0000000..5d9ca0a --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/tokens.ts @@ -0,0 +1,242 @@ +/** + * The OAuth half: exchanging the approval code, refreshing, and handing the client a token. + * + * FreeAgent's token endpoint takes the app's credentials as HTTP Basic and a form body + * (`grant_type=authorization_code|refresh_token`). Access tokens last an hour; refresh tokens about + * twenty years, and a refresh answers a NEW refresh token, which replaces the stored one. + * + * Both tokens live on `freeagent_connections`, sealed (`sealSecret`), never plaintext and never in a + * log line. The access token is cached there rather than in memory because every queue message may + * land on a fresh isolate; it is refreshed when it has less than two minutes left, and the refresh + * is SINGLE-FLIGHT per token source — a sync that fires twenty requests on a cold cache refreshes + * once, which matters against FreeAgent's 15-refreshes-a-minute limit. + * + * `invalid_grant` on refresh means the approval was revoked (or the token is from another + * environment): the connection is marked `needs_reconnect` with the reason, the tools disappear, + * the cron stops enqueueing it, and Settings asks an admin to reconnect. + */ +import type { FreeAgentEnvironment } from '@rocketflare/shared/plugins/freeagent/index' +import { and, eq } from 'drizzle-orm' +import type { Database, PluginConfig } from '@/plugins/api' +import { openSecret, sealSecret } from '@/plugins/api' +import { type FreeAgentConnectionRow, freeagentConnections } from '../db/schema' +import type { AccessTokenSource } from './client' +import { FREEAGENT_HOSTS, type FreeAgentAppCredentials, type FreeAgentDeps } from './env' +import { FreeAgentError, freeagentErrorMessage } from './errors' + +/** Refresh when a cached token has less than this left. */ +export const REFRESH_MARGIN_MS = 2 * 60_000 + +export interface TokenSet { + accessToken: string + /** Absent when FreeAgent did not rotate it; keep the one we have. */ + refreshToken: string | null + expiresAt: Date +} + +/** Base64 of `id:secret` without relying on Node's `Buffer`. */ +function basicAuth(credentials: FreeAgentAppCredentials): string { + const raw = `${credentials.clientId}:${credentials.clientSecret}` + const bytes = new TextEncoder().encode(raw) + let binary = '' + for (const b of bytes) binary += String.fromCharCode(b) + return `Basic ${btoa(binary)}` +} + +/** One call to `/v2/token_endpoint`. Never logs or echoes what it sent. */ +async function tokenRequest( + environment: FreeAgentEnvironment, + credentials: FreeAgentAppCredentials, + form: Record, + deps: FreeAgentDeps, + now: Date +): Promise { + let response: Response + try { + response = await deps.fetch(`${FREEAGENT_HOSTS[environment]}/v2/token_endpoint`, { + method: 'POST', + headers: { + Authorization: basicAuth(credentials), + Accept: 'application/json', + 'Content-Type': 'application/x-www-form-urlencoded', + }, + body: new URLSearchParams(form).toString(), + }) + } catch (err) { + throw new FreeAgentError( + 'network', + `FreeAgent's token endpoint did not answer: ${err instanceof Error ? err.message : String(err)}` + ) + } + const body = (await response.json().catch(() => null)) as Record | null + if (!response.ok) { + if (response.status === 429) { + throw new FreeAgentError('rate_limited', 'Too many token refreshes; try again shortly', { + status: 429, + }) + } + if (body?.error === 'invalid_grant') { + throw new FreeAgentError( + 'invalid_grant', + 'FreeAgent no longer accepts this approval — reconnect to FreeAgent', + { status: response.status } + ) + } + if (body?.error === 'invalid_client') { + throw new FreeAgentError( + 'not_configured', + 'FreeAgent refused the app credentials — check FREEAGENT_CLIENT_ID and FREEAGENT_CLIENT_SECRET', + { status: response.status } + ) + } + throw new FreeAgentError( + response.status >= 500 ? 'upstream' : 'bad_response', + freeagentErrorMessage(body) ?? `FreeAgent's token endpoint answered ${response.status}`, + { status: response.status } + ) + } + const accessToken = body?.access_token + if (typeof accessToken !== 'string' || accessToken.length === 0) { + throw new FreeAgentError('bad_response', "FreeAgent's token answer has no access token") + } + const expiresIn = Number(body?.expires_in) + return { + accessToken, + refreshToken: typeof body?.refresh_token === 'string' ? body.refresh_token : null, + // An absent or odd lifetime is treated as FreeAgent's documented hour. + expiresAt: new Date(now.getTime() + (Number.isFinite(expiresIn) ? expiresIn : 3600) * 1000), + } +} + +export function exchangeCode( + environment: FreeAgentEnvironment, + credentials: FreeAgentAppCredentials, + input: { code: string; redirectUri: string }, + deps: FreeAgentDeps, + now = new Date() +): Promise { + return tokenRequest( + environment, + credentials, + { grant_type: 'authorization_code', code: input.code, redirect_uri: input.redirectUri }, + deps, + now + ) +} + +export function refreshTokens( + environment: FreeAgentEnvironment, + credentials: FreeAgentAppCredentials, + refreshToken: string, + deps: FreeAgentDeps, + now = new Date() +): Promise { + return tokenRequest( + environment, + credentials, + { grant_type: 'refresh_token', refresh_token: refreshToken }, + deps, + now + ) +} + +/** Mark the connection as needing an admin, with the reason Settings shows. */ +export async function markNeedsReconnect( + db: Database, + connection: Pick, + reason: string +): Promise { + await db + .update(freeagentConnections) + .set({ + status: 'needs_reconnect', + accessTokenEnc: null, + accessTokenExpiresAt: null, + lastError: reason, + lastErrorAt: new Date(), + }) + .where( + and( + eq(freeagentConnections.id, connection.id), + eq(freeagentConnections.tenantId, connection.tenantId) + ) + ) +} + +/** + * A token source for one connection row: the cached access token while it has more than two + * minutes left, otherwise ONE refresh shared by every caller waiting on it. The row object is + * updated in place so later calls on the same source see the new token without a re-read. + */ +export function connectionTokenSource( + db: Database, + config: PluginConfig, + connection: FreeAgentConnectionRow, + credentials: FreeAgentAppCredentials, + deps: FreeAgentDeps, + now: () => Date = () => new Date() +): AccessTokenSource { + let inFlight: Promise | null = null + + const refresh = async (): Promise => { + const environment = connection.environment as FreeAgentEnvironment + let tokens: TokenSet + try { + tokens = await refreshTokens( + environment, + credentials, + await openSecret(config, connection.refreshTokenEnc), + deps, + now() + ) + } catch (err) { + if (err instanceof FreeAgentError && err.code === 'invalid_grant') { + await markNeedsReconnect(db, connection, err.message) + connection.status = 'needs_reconnect' + throw new FreeAgentError('needs_reconnect', err.message, { status: err.status }) + } + throw err + } + const accessTokenEnc = await sealSecret(config, tokens.accessToken) + const refreshTokenEnc = tokens.refreshToken + ? await sealSecret(config, tokens.refreshToken) + : connection.refreshTokenEnc + await db + .update(freeagentConnections) + .set({ accessTokenEnc, accessTokenExpiresAt: tokens.expiresAt, refreshTokenEnc }) + .where( + and( + eq(freeagentConnections.id, connection.id), + eq(freeagentConnections.tenantId, connection.tenantId) + ) + ) + connection.accessTokenEnc = accessTokenEnc + connection.accessTokenExpiresAt = tokens.expiresAt + connection.refreshTokenEnc = refreshTokenEnc + return tokens.accessToken + } + + return async ({ force }) => { + if (connection.status !== 'connected') { + throw new FreeAgentError('needs_reconnect', 'Reconnect to FreeAgent in Settings → FreeAgent') + } + if ( + !force && + connection.accessTokenEnc && + connection.accessTokenExpiresAt && + connection.accessTokenExpiresAt.getTime() - now().getTime() > REFRESH_MARGIN_MS + ) { + try { + return await openSecret(config, connection.accessTokenEnc) + } catch { + // Sealed under a rotated key — refresh below rather than failing the call. + } + } + if (!inFlight) { + inFlight = refresh().finally(() => { + inFlight = null + }) + } + return inFlight + } +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/services/writes.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/services/writes.ts new file mode 100644 index 0000000..ab820d6 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/services/writes.ts @@ -0,0 +1,299 @@ +/** + * Writes to FreeAgent, and the mirror read other plugins build on — the exported service API. + * + * Every write updates the mirror row it touched in the same call, so a matcher that explains a + * transaction sees it explained at once rather than after the next half-hourly sync: the new + * explanation is upserted, and the transaction is re-read from FreeAgent for its new + * `unexplained_amount` (FreeAgent computes it; we never guess it). + * + * Inputs are parsed with the shared zod schemas before anything is sent, and every resource URL a + * caller passes is checked to be on the connection's own API host (`FreeAgentClient.resolve`) — a + * URL is how FreeAgent names things, and the bearer token must never follow one anywhere else. + */ +import { + FREEAGENT_ATTACHMENT_MAX_BYTES, + FREEAGENT_ATTACHMENT_TYPES, + type FreeAgentCreateBillInput, + type FreeAgentExplainInput, + freeagentCreateBillInputSchema, + freeagentExplainInputSchema, +} from '@rocketflare/shared/plugins/freeagent/index' +import { and, asc, desc, eq, gte, lte, type SQL } from 'drizzle-orm' +import type { z } from 'zod' +import { + type FreeAgentBankTransactionRow, + freeagentBankAccounts, + freeagentBankTransactions, + freeagentExplanations, +} from '../db/schema' +import type { FreeAgentClient } from './client' +import { type FreeAgentCtx, freeagentClientFor } from './connection' +import { type FreeAgentDeps, freeagentHttp } from './env' +import { FreeAgentError } from './errors' +import { mapBankTransaction, mapBill, mapExplanation } from './mapping' +import { deleteByUrl, MIRROR_TABLES, pruneExplanations, upsertMirror } from './mirror' + +/** Parse with a shared schema, turning a zod failure into `FreeAgentError('validation')`. */ +function input(schema: z.ZodType, value: unknown): T { + const parsed = schema.safeParse(value) + if (!parsed.success) { + const issue = parsed.error.issues[0] + throw new FreeAgentError( + 'validation', + `${issue?.path.join('.') || 'input'}: ${issue?.message ?? 'invalid'}` + ) + } + return parsed.data +} + +function requireOwned(client: FreeAgentClient, url: string, what: string): void { + if (!client.owns(url)) { + throw new FreeAgentError('validation', `${what} is not a URL on this FreeAgent connection`) + } +} + +/** Re-read one transaction and write it (and its explanation list) to the mirror. */ +async function refreshTransaction( + ctx: FreeAgentCtx, + client: FreeAgentClient, + tenantId: string, + bankTransactionUrl: string +): Promise { + const body = await client.get<{ bank_transaction?: unknown }>(bankTransactionUrl) + const mapped = mapBankTransaction(body.bank_transaction) + await upsertMirror(ctx.db, MIRROR_TABLES.bankTransactions, tenantId, [mapped.row]) + if (mapped.explanations) { + await upsertMirror(ctx.db, MIRROR_TABLES.explanations, tenantId, mapped.explanations) + await pruneExplanations( + ctx.db, + tenantId, + mapped.row.url, + mapped.explanations.map(e => e.url) + ) + } +} + +export type ExplanationView = ReturnType + +/** + * Explain (part of) a bank transaction: `POST /v2/bank_transaction_explanations`. `grossValue` is + * signed like the transaction (money out is negative). Answers the explanation as mirrored. + */ +export async function explainTransaction( + ctx: FreeAgentCtx, + tenantId: string, + raw: FreeAgentExplainInput, + deps: FreeAgentDeps = freeagentHttp +): Promise { + const body = input(freeagentExplainInputSchema, raw) + const client = await freeagentClientFor(ctx, tenantId, deps) + requireOwned(client, body.bankTransactionUrl, 'bankTransactionUrl') + requireOwned(client, body.categoryUrl, 'categoryUrl') + const created = await client.post<{ bank_transaction_explanation?: unknown }>( + '/v2/bank_transaction_explanations', + { + bank_transaction_explanation: { + bank_transaction: body.bankTransactionUrl, + category: body.categoryUrl, + gross_value: body.grossValue, + dated_on: body.datedOn, + description: body.description, + ...(body.salesTaxRate !== undefined && { sales_tax_rate: body.salesTaxRate }), + ...(body.receiptReference !== undefined && { receipt_reference: body.receiptReference }), + }, + } + ) + const explanation = mapExplanation(created.bank_transaction_explanation) + await upsertMirror(ctx.db, MIRROR_TABLES.explanations, tenantId, [ + { + ...explanation, + bankTransactionUrl: explanation.bankTransactionUrl ?? body.bankTransactionUrl, + }, + ]) + await refreshTransaction(ctx, client, tenantId, body.bankTransactionUrl) + return explanation +} + +export interface AttachmentInput { + bytes: ArrayBuffer | Uint8Array + filename: string + contentType: string + description?: string +} + +/** FreeAgent's spelling of a PDF is `application/x-pdf`; accept the standard one and translate. */ +export function freeagentContentType(contentType: string): string | null { + const type = contentType.toLowerCase().split(';')[0]?.trim() ?? '' + const normalised = type === 'application/pdf' ? 'application/x-pdf' : type + return (FREEAGENT_ATTACHMENT_TYPES as readonly string[]).includes(normalised) ? normalised : null +} + +function toBase64(bytes: Uint8Array): string { + let binary = '' + const CHUNK = 0x8000 + for (let i = 0; i < bytes.length; i += CHUNK) { + binary += String.fromCharCode(...bytes.subarray(i, i + CHUNK)) + } + return btoa(binary) +} + +/** + * Attach a file to an explanation: `POST {explanation}/attachments` (API version 2026-09-01, + * up to 50 per explanation). PNG, JPEG, GIF or PDF, at most 5 MB — refused here, before any bytes + * leave. Answers the explanation's attachment count after the upload, which the mirror now holds. + */ +export async function attachToExplanation( + ctx: FreeAgentCtx, + tenantId: string, + explanationUrl: string, + file: AttachmentInput, + deps: FreeAgentDeps = freeagentHttp +): Promise<{ attachmentCount: number; attachmentUrls: string[] }> { + const bytes = file.bytes instanceof Uint8Array ? file.bytes : new Uint8Array(file.bytes) + const contentType = freeagentContentType(file.contentType) + if (!contentType) { + throw new FreeAgentError( + 'validation', + `FreeAgent accepts PNG, JPEG, GIF or PDF attachments, not ${file.contentType}` + ) + } + if (bytes.byteLength === 0 || bytes.byteLength > FREEAGENT_ATTACHMENT_MAX_BYTES) { + throw new FreeAgentError('validation', 'An attachment must be between 1 byte and 5 MB') + } + const client = await freeagentClientFor(ctx, tenantId, deps) + requireOwned(client, explanationUrl, 'explanationUrl') + const filename = file.filename.replace(/[\\/\r\n"]/g, '_').slice(0, 200) || 'attachment' + const answer = await client.post<{ attachments?: Array<{ url?: string }> }>( + `${explanationUrl.replace(/\/+$/, '')}/attachments`, + { + attachments: [ + { + data: toBase64(bytes), + file_name: filename, + content_type: contentType, + description: file.description ?? filename, + }, + ], + } + ) + const attachments = Array.isArray(answer.attachments) ? answer.attachments : [] + await ctx.db + .update(freeagentExplanations) + .set({ attachmentCount: attachments.length }) + .where( + and( + eq(freeagentExplanations.tenantId, tenantId), + eq(freeagentExplanations.url, explanationUrl) + ) + ) + return { + attachmentCount: attachments.length, + attachmentUrls: attachments.map(a => a.url).filter((u): u is string => typeof u === 'string'), + } +} + +/** Delete an explanation, and re-read its transaction (which is now unexplained again). */ +export async function deleteExplanation( + ctx: FreeAgentCtx, + tenantId: string, + explanationUrl: string, + deps: FreeAgentDeps = freeagentHttp +): Promise { + const client = await freeagentClientFor(ctx, tenantId, deps) + requireOwned(client, explanationUrl, 'explanationUrl') + const [mirrored] = await ctx.db + .select({ bankTransactionUrl: freeagentExplanations.bankTransactionUrl }) + .from(freeagentExplanations) + .where( + and( + eq(freeagentExplanations.tenantId, tenantId), + eq(freeagentExplanations.url, explanationUrl) + ) + ) + .limit(1) + await client.delete(explanationUrl) + await deleteByUrl(ctx.db, MIRROR_TABLES.explanations, tenantId, [explanationUrl]) + if (mirrored?.bankTransactionUrl) { + await refreshTransaction(ctx, client, tenantId, mirrored.bankTransactionUrl) + } +} + +export type BillView = ReturnType + +/** Create a bill: `POST /v2/bills`. Answers the bill as mirrored. */ +export async function createBill( + ctx: FreeAgentCtx, + tenantId: string, + raw: FreeAgentCreateBillInput, + deps: FreeAgentDeps = freeagentHttp +): Promise { + const body = input(freeagentCreateBillInputSchema, raw) + const client = await freeagentClientFor(ctx, tenantId, deps) + requireOwned(client, body.contactUrl, 'contactUrl') + for (const item of body.items) requireOwned(client, item.categoryUrl, 'categoryUrl') + const created = await client.post<{ bill?: unknown }>('/v2/bills', { + bill: { + contact: body.contactUrl, + reference: body.reference, + dated_on: body.datedOn, + due_on: body.dueOn, + bill_items: body.items.map(item => ({ + category: item.categoryUrl, + description: item.description, + total_value: item.totalValue, + ...(item.salesTaxRate !== undefined && { sales_tax_rate: item.salesTaxRate }), + })), + }, + }) + const bill = mapBill(created.bill) + await upsertMirror(ctx.db, MIRROR_TABLES.bills, tenantId, [bill]) + return bill +} + +export interface UnexplainedTransaction extends FreeAgentBankTransactionRow { + bankAccountName: string | null +} + +/** + * Unexplained bank transactions from the MIRROR (no FreeAgent call) — what a matcher iterates. + * Newest first; narrow by account and date range. + */ +export async function listUnexplainedTransactions( + ctx: Pick, + tenantId: string, + opts: { + bankAccountUrl?: string + from?: string + to?: string + limit?: number + oldestFirst?: boolean + } = {} +): Promise { + const filters: SQL[] = [ + eq(freeagentBankTransactions.tenantId, tenantId), + eq(freeagentBankTransactions.isUnexplained, true), + ] + if (opts.bankAccountUrl) + filters.push(eq(freeagentBankTransactions.bankAccountUrl, opts.bankAccountUrl)) + if (opts.from) filters.push(gte(freeagentBankTransactions.datedOn, opts.from)) + if (opts.to) filters.push(lte(freeagentBankTransactions.datedOn, opts.to)) + const rows = await ctx.db + .select({ tx: freeagentBankTransactions, bankAccountName: freeagentBankAccounts.name }) + .from(freeagentBankTransactions) + .leftJoin( + freeagentBankAccounts, + and( + eq(freeagentBankAccounts.tenantId, freeagentBankTransactions.tenantId), + eq(freeagentBankAccounts.url, freeagentBankTransactions.bankAccountUrl) + ) + ) + .where(and(...filters)) + .orderBy( + opts.oldestFirst + ? asc(freeagentBankTransactions.datedOn) + : desc(freeagentBankTransactions.datedOn), + asc(freeagentBankTransactions.url) + ) + .limit(Math.min(Math.max(opts.limit ?? 500, 1), 5000)) + return rows.map(r => ({ ...r.tx, bankAccountName: r.bankAccountName })) +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/tests/api/cubes.test.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/api/cubes.test.ts new file mode 100644 index 0000000..7a2df80 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/api/cubes.test.ts @@ -0,0 +1,61 @@ +/** + * The contributed cubes scope by tenant — proven here WITHOUT the analytics plugin, by running each + * cube's own base query (`sql(ctx)` → `{ from, where }`) against Postgres as two tenants, seeded by + * the same isolation cases analytics would run. When analytics is installed its cube-isolation test + * runs those cases again through `/cubejs-api`. + */ +import { createTestTenant, setupTestDatabase } from '@testkit/integration' +import { describe, expect, it } from 'vitest' +import { + FREEAGENT_ANALYTICS_EXTENSIONS, + FREEAGENT_CUBE_ISOLATION_CASES, + FREEAGENT_CUBES, + tenantOf, +} from '../../cubes' + +const db = setupTestDatabase() + +describe('freeagent cubes', () => { + it('has exactly one isolation case per cube, under the analytics keys', () => { + expect(FREEAGENT_CUBE_ISOLATION_CASES.map(c => c.cube).sort()).toEqual( + FREEAGENT_CUBES.map(c => c.name).sort() + ) + expect(Object.keys(FREEAGENT_ANALYTICS_EXTENSIONS).sort()).toEqual([ + 'analytics:cubeIsolationCases', + 'analytics:cubes', + ]) + }) + + it('refuses a query with no tenant', () => { + expect(() => tenantOf({ securityContext: {} })).toThrow() + for (const cube of FREEAGENT_CUBES) expect(() => cube.sql({ securityContext: {} })).toThrow() + }) + + it('every cube’s base query answers only the querying tenant’s rows', async () => { + const a = (await createTestTenant(db)).id + const b = (await createTestTenant(db)).id + for (const c of FREEAGENT_CUBE_ISOLATION_CASES) { + await c.seed(db, { tenantId: a, userId: a }) + await c.seed(db, { tenantId: b, userId: b }) + } + for (const cube of FREEAGENT_CUBES) { + for (const [mine, other] of [ + [a, b], + [b, a], + ] as const) { + const { from, where } = cube.sql({ securityContext: { tenantId: mine } }) + // biome-ignore lint/suspicious/noExplicitAny: the cube's `from` is one of this plugin's tables + const rows = (await db + .select() + .from(from as any) + .where(where)) as Array<{ tenantId: string }> + expect(rows.length, cube.name).toBeGreaterThan(0) + expect( + rows.every(r => r.tenantId === mine), + cube.name + ).toBe(true) + expect(JSON.stringify(rows)).not.toContain(other) + } + } + }) +}) diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/tests/api/freeagent.test.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/api/freeagent.test.ts new file mode 100644 index 0000000..9a7f7ff --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/api/freeagent.test.ts @@ -0,0 +1,998 @@ +/** + * The plugin end to end on real Postgres, with a FAKE FreeAgent behind an injected fetch: the + * OAuth callback (bad state → 401; success stores sealed tokens), token refresh and `invalid_grant` + * → needs reconnect, an incremental sync (upserts, cursors, pagination, deletions reconciled), + * explain + attach updating the mirror, tools hidden until connected, the cron fan-out, and tenant + * isolation — another organisation can neither read nor change anything through the mounts. + * + * Nothing here reaches FreeAgent: services take `deps`, and the one test that drives the real + * callback route swaps `freeagentHttp.fetch` for its duration. + */ +import { readFileSync } from 'node:fs' +import { + createExecutionContext, + createTestEnv, + createTestSession, + createTestTenant, + createTestUser, + dispatchScheduled, + json, + linkUserToTenant, + request, + sessionCookieHeader, + setupTestDatabase, + stubs, + type TestEnv, + waitOnExecutionContext, +} from '@testkit/integration' +import { makeCronCtx, makeJobCtx, makeToolCtx } from '@testkit/unit' +import { and, eq } from 'drizzle-orm' +import { afterEach, beforeAll, describe, expect, it } from 'vitest' +import { openSecret, signState } from '@/plugins/api' +import { CONNECT_STATE_PURPOSE } from '../../api/hooks' +import { enqueueSyncs, FREEAGENT_CRON } from '../../api/scheduled' +import { + freeagentBankTransactions, + freeagentBills, + freeagentCategories, + freeagentConnections, + freeagentContacts, + freeagentExplanations, + freeagentInvoices, + freeagentSyncState, +} from '../../db/schema' +import { freeagentClientFor, saveConnection } from '../../services/connection' +import { type FetchLike, type FreeAgentDeps, freeagentHttp } from '../../services/env' +import { syncTenant } from '../../services/sync' +import { + attachToExplanation, + explainTransaction, + listUnexplainedTransactions, +} from '../../services/writes' +import { FREEAGENT_TOOL_NAMES, freeagentAgentTools } from '../../tools/freeagent-tools' + +const db = setupTestDatabase() +const API = 'https://api.sandbox.freeagent.com' +const ORIGIN = { Origin: 'http://localhost:3000' } + +function freeagentEnv(): TestEnv { + return createTestEnv({ + FREEAGENT_CLIENT_ID: 'test-client', + FREEAGENT_CLIENT_SECRET: 'test-secret', + FREEAGENT_ENVIRONMENT: 'sandbox', + } as Partial) +} + +// ---- A fake FreeAgent ----------------------------------------------------------------------------- + +interface Call { + method: string + url: URL + headers: Headers + body: string | null +} +type Handler = (call: Call) => Response | Promise + +function reply(body: unknown, init: ResponseInit = {}): Response { + return new Response(JSON.stringify(body), { + ...init, + headers: { 'Content-Type': 'application/json', ...(init.headers as Record) }, + }) +} + +/** Routes keyed `METHOD /path`; records every call. An unknown route is a 404 the test will see. */ +function fakeFreeAgent(routes: Record) { + const calls: Call[] = [] + const fetch: FetchLike = async (input, init) => { + const url = new URL(String(input)) + const call: Call = { + method: init?.method ?? 'GET', + url, + headers: new Headers(init?.headers), + body: typeof init?.body === 'string' ? init.body : null, + } + calls.push(call) + const handler = routes[`${call.method} ${url.pathname}`] + return handler + ? handler(call) + : reply({ errors: { error: { message: 'no route' } } }, { status: 404 }) + } + const deps: FreeAgentDeps = { fetch, sleep: async () => {} } + return { deps, calls, routes } +} + +function tokenReply(n: number) { + return reply({ + access_token: `access-${n}`, + token_type: 'bearer', + expires_in: 3600, + refresh_token: `refresh-${n}`, + refresh_token_expires_in: 631151957, + }) +} + +const company = { company: { url: `${API}/v2/company`, name: 'GuideMode Ltd', currency: 'GBP' } } + +/** Connect a tenant directly (no browser): a fresh access token, so no refresh on first use. */ +async function connect(tenantId: string, userId: string, env: TestEnv, expiresInMs = 3_600_000) { + const ctx = makeJobCtx({ db, env }) + await saveConnection(ctx, { + tenantId, + userId, + environment: 'sandbox', + tokens: { + accessToken: 'access-0', + refreshToken: 'refresh-0', + expiresAt: new Date(Date.now() + expiresInMs), + }, + company: { + companyUrl: `${API}/v2/company`, + companyName: 'GuideMode Ltd', + companyCurrency: 'GBP', + }, + }) +} + +async function tenantWithOwner() { + const tenantId = (await createTestTenant(db)).id + const owner = await createTestUser(db) + await linkUserToTenant(db, owner.id, tenantId, 'owner') + const cookie = { + ...sessionCookieHeader(await createTestSession(db, owner.id, tenantId)), + ...ORIGIN, + } + return { tenantId, userId: owner.id, cookie } +} + +// ---- The books the fake serves -------------------------------------------------------------------- + +const ACCOUNT = `${API}/v2/bank_accounts/1` +const TX1 = `${API}/v2/bank_transactions/101` +const TX2 = `${API}/v2/bank_transactions/102` +const EXP1 = `${API}/v2/bank_transaction_explanations/501` +const CATEGORY = `${API}/v2/categories/285` + +function booksRoutes(state: { pass: number }): Record { + return { + 'GET /v2/company': () => reply(company), + 'GET /v2/bank_accounts': () => + reply({ + bank_accounts: [ + { + url: ACCOUNT, + name: 'Business Current', + type: 'StandardBankAccount', + currency: 'GBP', + current_balance: '1500.25', + is_primary: true, + }, + ], + }), + 'GET /v2/categories': () => + reply({ + admin_expenses_categories: [{ url: CATEGORY, description: 'Travel', nominal_code: '285' }], + income_categories: [ + { url: `${API}/v2/categories/001`, description: 'Sales', nominal_code: '001' }, + ], + }), + // Two pages, to prove the sync follows Link. + 'GET /v2/contacts': ({ url }) => + url.searchParams.get('page') === '2' + ? reply({ contacts: [{ url: `${API}/v2/contacts/2`, organisation_name: 'Rail Co' }] }) + : reply( + { contacts: [{ url: `${API}/v2/contacts/1`, organisation_name: 'Acme Ltd' }] }, + { headers: { Link: `<${API}/v2/contacts?page=2&per_page=100>; rel="next"` } } + ), + 'GET /v2/bank_transactions': () => + state.pass === 1 + ? reply({ + bank_transactions: [ + { + url: TX1, + bank_account: ACCOUNT, + dated_on: '2026-10-01', + amount: '-42.50', + description: 'TRAINLINE', + unexplained_amount: '-42.50', + bank_transaction_explanations: [], + }, + { + url: TX2, + bank_account: ACCOUNT, + dated_on: '2026-10-02', + amount: '1200.00', + description: 'ACME PAYMENT', + unexplained_amount: '0.0', + bank_transaction_explanations: [ + { + url: EXP1, + bank_account: ACCOUNT, + dated_on: '2026-10-02', + entry_type: 'Invoice Receipt', + gross_value: '1200.0', + }, + ], + }, + ], + }) + : // Second pass: the explanation on TX2 was deleted in FreeAgent. + reply({ + bank_transactions: [ + { + url: TX2, + bank_account: ACCOUNT, + dated_on: '2026-10-02', + amount: '1200.00', + description: 'ACME PAYMENT', + unexplained_amount: '1200.00', + bank_transaction_explanations: [], + }, + ], + }), + 'GET /v2/bank_transaction_explanations': () => + reply({ + bank_transaction_explanations: + state.pass === 1 + ? [ + { + url: EXP1, + bank_transaction: TX2, + bank_account: ACCOUNT, + category: `${API}/v2/categories/001`, + gross_value: '1200.0', + sales_tax_rate: '20.0', + sales_tax_value: '200.0', + dated_on: '2026-10-02', + attachments: [], + }, + ] + : [], + }), + 'GET /v2/bills': () => + reply({ + bills: [ + { + url: `${API}/v2/bills/1`, + contact: `${API}/v2/contacts/2`, + reference: 'B-1', + dated_on: '2026-09-01', + due_on: '2026-09-30', + status: 'Open', + total_value: '60.0', + due_value: '60.0', + }, + ], + }), + 'GET /v2/invoices': () => + reply({ + invoices: [ + { + url: `${API}/v2/invoices/1`, + contact: `${API}/v2/contacts/1`, + reference: 'INV-001', + dated_on: '2026-09-01', + due_on: '2026-09-15', + status: 'Open', + total_value: '1200.0', + paid_value: '0.0', + due_value: '1200.0', + }, + ], + }), + } +} + +afterEach(() => { + freeagentHttp.fetch = (input, init) => fetch(input, init) +}) + +// ---- OAuth ---------------------------------------------------------------------------------------- + +describe('connecting', () => { + let tenantId: string + let userId: string + let cookie: Record + let memberCookie: Record + + beforeAll(async () => { + ;({ tenantId, userId, cookie } = await tenantWithOwner()) + const member = await createTestUser(db) + await linkUserToTenant(db, member.id, tenantId, 'member') + memberCookie = { + ...sessionCookieHeader(await createTestSession(db, member.id, tenantId)), + ...ORIGIN, + } + }) + + it('answers FreeAgent’s approve URL to an admin, never to a member, and 503s unconfigured', async () => { + const env = freeagentEnv() + const res = await request( + '/api/freeagent/connect', + { method: 'POST', headers: cookie }, + { env } + ) + expect(res.status).toBe(200) + const { approveUrl } = await json<{ approveUrl: string }>(res) + const url = new URL(approveUrl) + expect(`${url.origin}${url.pathname}`).toBe(`${API}/v2/approve_app`) + expect(url.searchParams.get('client_id')).toBe('test-client') + expect(url.searchParams.get('response_type')).toBe('code') + expect(url.searchParams.get('redirect_uri')).toMatch(/\/api\/hooks\/freeagent\/callback$/) + expect(url.searchParams.get('state')).toBeTruthy() + + const denied = await request( + '/api/freeagent/connect', + { method: 'POST', headers: memberCookie }, + { env } + ) + expect(denied.status).toBe(403) + const bare = await request('/api/freeagent/connect', { method: 'POST', headers: cookie }) + expect(bare.status).toBe(503) + }) + + it('401s a callback whose state is missing, forged or for another purpose', async () => { + const env = freeagentEnv() + const config = makeJobCtx({ db, env }).config + const wrongPurpose = await signState(config, 'something.else', { tenantId, userId }) + for (const state of ['', 'garbage', 'a.b', wrongPurpose]) { + const res = await request( + `/api/hooks/freeagent/callback?code=abc&state=${encodeURIComponent(state)}`, + {}, + { env } + ) + expect(res.status).toBe(401) + } + const [row] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + expect(row).toBeUndefined() + }) + + it('exchanges the code, stores the tokens SEALED, records the company and queues a sync', async () => { + const env = freeagentEnv() + const config = makeJobCtx({ db, env }).config + const fake = fakeFreeAgent({ + 'POST /v2/token_endpoint': () => tokenReply(1), + 'GET /v2/company': () => reply(company), + }) + freeagentHttp.fetch = fake.deps.fetch + const state = await signState(config, CONNECT_STATE_PURPOSE, { tenantId, userId }) + const res = await request( + `/api/hooks/freeagent/callback?code=the-code&state=${encodeURIComponent(state)}`, + {}, + { env } + ) + expect(res.status).toBe(302) + const location = new URL(res.headers.get('location') ?? '') + expect(location.pathname).toBe('/settings') + expect(location.searchParams.get('tab')).toBe('freeagent') + expect(location.searchParams.get('connected')).toBe('freeagent') + + // The token request: Basic auth with the app, the code, the same redirect URI. + const tokenCall = fake.calls.find(c => c.url.pathname === '/v2/token_endpoint') + expect(tokenCall?.headers.get('Authorization')).toBe(`Basic ${btoa('test-client:test-secret')}`) + const form = new URLSearchParams(tokenCall?.body ?? '') + expect(form.get('grant_type')).toBe('authorization_code') + expect(form.get('code')).toBe('the-code') + expect(form.get('redirect_uri')).toMatch(/\/api\/hooks\/freeagent\/callback$/) + + const [row] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + expect(row).toMatchObject({ + status: 'connected', + environment: 'sandbox', + companyName: 'GuideMode Ltd', + companyCurrency: 'GBP', + connectedByUserId: userId, + }) + expect(row?.refreshTokenEnc).not.toContain('refresh-1') + expect(row?.accessTokenEnc).not.toContain('access-1') + expect(await openSecret(config, row?.refreshTokenEnc ?? '')).toBe('refresh-1') + expect(await openSecret(config, row?.accessTokenEnc ?? '')).toBe('access-1') + expect(stubs(env).queue.messages.map(m => (m.body as { type: string }).type)).toContain( + 'freeagent.sync' + ) + + // Status never carries a token. + const status = await request('/api/freeagent/status', { headers: memberCookie }, { env }) + const body = await status.text() + expect(JSON.parse(body)).toMatchObject({ + configured: true, + connected: true, + companyName: 'GuideMode Ltd', + }) + expect(body).not.toContain('refresh-1') + expect(body).not.toContain('access-1') + }) + + it('sends a denied approval back to Settings with the reason', async () => { + const env = freeagentEnv() + const config = makeJobCtx({ db, env }).config + const state = await signState(config, CONNECT_STATE_PURPOSE, { tenantId, userId }) + const res = await request( + `/api/hooks/freeagent/callback?error=access_denied&state=${encodeURIComponent(state)}`, + {}, + { env } + ) + expect(res.status).toBe(302) + expect(new URL(res.headers.get('location') ?? '').searchParams.get('connectError')).toBe( + 'access_denied' + ) + }) +}) + +// ---- Tokens --------------------------------------------------------------------------------------- + +describe('token refresh', () => { + it('refreshes once near expiry (single-flight), rotating the stored refresh token', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + await connect(tenantId, userId, env, 60_000) // one minute left: inside the two-minute margin + let refreshes = 0 + const fake = fakeFreeAgent({ + 'POST /v2/token_endpoint': () => tokenReply(++refreshes), + 'GET /v2/company': ({ headers }) => + reply({ ...company, seenToken: headers.get('Authorization') }), + }) + const ctx = makeJobCtx({ db, env }) + const client = await freeagentClientFor(ctx, tenantId, fake.deps) + const answers = await Promise.all([ + client.get<{ seenToken: string }>('/v2/company'), + client.get<{ seenToken: string }>('/v2/company'), + client.get<{ seenToken: string }>('/v2/company'), + ]) + expect(refreshes).toBe(1) + expect(answers.map(a => a.seenToken)).toEqual([ + 'Bearer access-1', + 'Bearer access-1', + 'Bearer access-1', + ]) + const form = new URLSearchParams( + fake.calls.find(c => c.url.pathname === '/v2/token_endpoint')?.body ?? '' + ) + expect(form.get('grant_type')).toBe('refresh_token') + expect(form.get('refresh_token')).toBe('refresh-0') + const [row] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + expect(await openSecret(ctx.config, row?.refreshTokenEnc ?? '')).toBe('refresh-1') + }) + + it('marks the connection needs_reconnect on invalid_grant, and the sync stops there', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + await connect(tenantId, userId, env, 0) // expired + const fake = fakeFreeAgent({ + 'POST /v2/token_endpoint': () => + reply({ error: 'invalid_grant', error_description: 'revoked' }, { status: 400 }), + }) + const ctx = makeJobCtx({ db, env }) + const outcome = await syncTenant(ctx, tenantId, { deps: fake.deps, listeners: [] }) + expect(outcome.status).toBe('needs_reconnect') + const [row] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + expect(row).toMatchObject({ + status: 'needs_reconnect', + accessTokenEnc: null, + syncClaimedUntil: null, + }) + expect(row?.lastError).toMatch(/reconnect/i) + await expect(freeagentClientFor(ctx, tenantId, fake.deps)).rejects.toMatchObject({ + code: 'needs_reconnect', + }) + // The cron no longer queues it, and no tools are offered. + expect(await freeagentAgentTools(makeToolCtx({ db, env, tenantId }), fake.deps)).toEqual([]) + }) +}) + +// ---- Sync ----------------------------------------------------------------------------------------- + +describe('sync', () => { + it('mirrors the books, pages, writes cursors, then syncs incrementally and reconciles deletions', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + await connect(tenantId, userId, env) + const state = { pass: 1 } + const fake = fakeFreeAgent(booksRoutes(state)) + const ctx = makeJobCtx({ db, env }) + const heard: string[] = [] + const listener = { + id: 'test', + onSync: async (s: { tenantId: string }) => void heard.push(s.tenantId), + } + const broken = { + id: 'broken', + onSync: async () => { + throw new Error('boom') + }, + } + + const first = await syncTenant(ctx, tenantId, { + deps: fake.deps, + listeners: [broken, listener], + }) + expect(first.status).toBe('synced') + expect(heard).toEqual([tenantId]) // a throwing listener did not stop the next + + const txs = await db + .select() + .from(freeagentBankTransactions) + .where(eq(freeagentBankTransactions.tenantId, tenantId)) + expect(txs.map(t => [t.url, t.amount, t.isUnexplained]).sort()).toEqual([ + [TX1, '-42.50', true], + [TX2, '1200.00', false], + ]) + const [exp] = await db + .select() + .from(freeagentExplanations) + .where(eq(freeagentExplanations.tenantId, tenantId)) + expect(exp).toMatchObject({ + url: EXP1, + bankTransactionUrl: TX2, + categoryUrl: `${API}/v2/categories/001`, + grossValue: '1200.00', + salesTaxRate: '20.0000', + }) + const contacts = await db + .select() + .from(freeagentContacts) + .where(eq(freeagentContacts.tenantId, tenantId)) + expect(contacts.map(c => c.displayName).sort()).toEqual(['Acme Ltd', 'Rail Co']) // both pages + expect( + await db.select().from(freeagentCategories).where(eq(freeagentCategories.tenantId, tenantId)) + ).toHaveLength(2) + expect( + await db.select().from(freeagentBills).where(eq(freeagentBills.tenantId, tenantId)) + ).toHaveLength(1) + expect( + await db.select().from(freeagentInvoices).where(eq(freeagentInvoices.tenantId, tenantId)) + ).toHaveLength(1) + + // The first pass of transactions is bounded (from_date) and not incremental. + const firstTxCall = fake.calls.find(c => c.url.pathname === '/v2/bank_transactions') + expect(firstTxCall?.url.searchParams.get('bank_account')).toBe(ACCOUNT) + expect(firstTxCall?.url.searchParams.get('from_date')).toMatch(/^\d{4}-\d{2}-\d{2}$/) + expect(firstTxCall?.url.searchParams.get('updated_since')).toBeNull() + for (const call of fake.calls) expect(call.headers.get('X-Api-Version')).toBe('2026-09-01') + + const cursors = await db + .select() + .from(freeagentSyncState) + .where(eq(freeagentSyncState.tenantId, tenantId)) + expect(cursors.map(c => c.resource).sort()).toEqual( + [ + 'bills', + 'contacts', + 'invoices', + `bank_transaction_explanations:${ACCOUNT}`, + `bank_transactions:${ACCOUNT}`, + ].sort() + ) + const [conn] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + expect(conn?.lastSyncAt).toBeInstanceOf(Date) + expect(conn?.syncClaimedUntil).toBeNull() + + // Second pass: incremental, and the explanation deleted upstream is removed here. + state.pass = 2 + fake.calls.length = 0 + expect((await syncTenant(ctx, tenantId, { deps: fake.deps, listeners: [] })).status).toBe( + 'synced' + ) + const secondTxCall = fake.calls.find(c => c.url.pathname === '/v2/bank_transactions') + expect(secondTxCall?.url.searchParams.get('updated_since')).toBeTruthy() + expect(secondTxCall?.url.searchParams.get('from_date')).toBeNull() + const [tx2] = await db + .select() + .from(freeagentBankTransactions) + .where( + and( + eq(freeagentBankTransactions.tenantId, tenantId), + eq(freeagentBankTransactions.url, TX2) + ) + ) + expect(tx2).toMatchObject({ isUnexplained: true, unexplainedAmount: '1200.00' }) + expect( + await db + .select() + .from(freeagentExplanations) + .where(eq(freeagentExplanations.tenantId, tenantId)) + ).toEqual([]) + expect(txs).toHaveLength(2) + expect( + await db + .select() + .from(freeagentBankTransactions) + .where(eq(freeagentBankTransactions.tenantId, tenantId)) + ).toHaveLength(2) + }) + + it('skips a tenant another sync has claimed, and records a failure for the retry', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + await connect(tenantId, userId, env) + await db + .update(freeagentConnections) + .set({ syncClaimedUntil: new Date(Date.now() + 60_000) }) + .where(eq(freeagentConnections.tenantId, tenantId)) + const fake = fakeFreeAgent(booksRoutes({ pass: 1 })) + const ctx = makeJobCtx({ db, env }) + expect(await syncTenant(ctx, tenantId, { deps: fake.deps, listeners: [] })).toEqual({ + status: 'skipped', + reason: 'claimed', + }) + expect(fake.calls).toHaveLength(0) + + await db + .update(freeagentConnections) + .set({ syncClaimedUntil: null }) + .where(eq(freeagentConnections.tenantId, tenantId)) + const limited = fakeFreeAgent({ + 'GET /v2/company': () => reply({}, { status: 429, headers: { 'Retry-After': '3600' } }), + }) + await expect( + syncTenant(ctx, tenantId, { deps: limited.deps, listeners: [] }) + ).rejects.toMatchObject({ code: 'rate_limited' }) + const [row] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, tenantId)) + expect(row).toMatchObject({ + syncClaimedUntil: null, + lastError: 'FreeAgent rate limit reached; try again later', + }) + }) + + it('the cron queues one sync per connected organisation, through the host dispatcher', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + await connect(tenantId, userId, env) + const queued = await enqueueSyncs(makeCronCtx({ db, env })) + expect(queued).toBeGreaterThanOrEqual(1) + const payloads = stubs(env).queue.messages.map( + m => m.body as { type: string; payload: { tenantId: string } } + ) + expect( + payloads.filter(p => p.type === 'freeagent.sync').map(p => p.payload.tenantId) + ).toContain(tenantId) + + // The manifest's cron (what reaches both tomls) is the expression the task is registered + // under, and the host's dispatcher really runs it. + const manifest = JSON.parse( + readFileSync(new URL('../../plugin.json', import.meta.url), 'utf8') + ) as { crons: string[] } + expect(manifest.crons).toEqual([FREEAGENT_CRON]) + const execution = createExecutionContext() + const report = await dispatchScheduled(FREEAGENT_CRON, freeagentEnv(), execution) + await waitOnExecutionContext(execution) + expect(report).toEqual( + expect.arrayContaining([ + expect.objectContaining({ task: 'freeagent.enqueueSyncs', status: 'ok' }), + ]) + ) + }) +}) + +// ---- Writes --------------------------------------------------------------------------------------- + +describe('explain and attach', () => { + it('explains a transaction, attaches a receipt, and the mirror follows both', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + await connect(tenantId, userId, env) + const state = { pass: 1 } + const books = fakeFreeAgent(booksRoutes(state)) + const ctx = makeJobCtx({ db, env }) + await syncTenant(ctx, tenantId, { deps: books.deps, listeners: [] }) + expect((await listUnexplainedTransactions(ctx, tenantId)).map(t => t.url)).toEqual([TX1]) + + const NEW_EXP = `${API}/v2/bank_transaction_explanations/777` + let attachments: Array<{ url: string }> = [] + const writes = fakeFreeAgent({ + 'POST /v2/bank_transaction_explanations': ({ body }) => { + const sent = JSON.parse(body ?? '{}').bank_transaction_explanation + expect(sent).toEqual({ + bank_transaction: TX1, + category: CATEGORY, + gross_value: '-42.50', + dated_on: '2026-10-01', + description: 'Train to London', + sales_tax_rate: '0', + }) + return reply( + { + bank_transaction_explanation: { + url: NEW_EXP, + bank_transaction: TX1, + bank_account: ACCOUNT, + category: CATEGORY, + gross_value: '-42.5', + sales_tax_rate: '0.0', + dated_on: '2026-10-01', + description: 'Train to London', + attachments: [], + }, + }, + { status: 201 } + ) + }, + 'GET /v2/bank_transactions/101': () => + reply({ + bank_transaction: { + url: TX1, + bank_account: ACCOUNT, + dated_on: '2026-10-01', + amount: '-42.50', + unexplained_amount: '0.0', + bank_transaction_explanations: [{ url: NEW_EXP, gross_value: '-42.5' }], + }, + }), + 'POST /v2/bank_transaction_explanations/777/attachments': ({ body }) => { + const sent = JSON.parse(body ?? '{}') + expect(sent.attachments).toHaveLength(1) + expect(sent.attachments[0]).toMatchObject({ + file_name: 'receipt.pdf', + content_type: 'application/x-pdf', + description: 'Trainline receipt', + }) + expect(atob(sent.attachments[0].data)).toBe('%PDF-1.4 receipt') + attachments = [...attachments, { url: `${API}/v2/attachments/${attachments.length + 1}` }] + return reply({ attachments }, { status: 201 }) + }, + }) + + const explanation = await explainTransaction( + ctx, + tenantId, + { + bankTransactionUrl: TX1, + categoryUrl: CATEGORY, + grossValue: '-42.50', + datedOn: '2026-10-01', + description: 'Train to London', + salesTaxRate: '0', + }, + writes.deps + ) + expect(explanation.url).toBe(NEW_EXP) + const [tx1] = await db + .select() + .from(freeagentBankTransactions) + .where( + and( + eq(freeagentBankTransactions.tenantId, tenantId), + eq(freeagentBankTransactions.url, TX1) + ) + ) + expect(tx1).toMatchObject({ isUnexplained: false, unexplainedAmount: '0.00' }) + expect(await listUnexplainedTransactions(ctx, tenantId)).toEqual([]) + + const bytes = new TextEncoder().encode('%PDF-1.4 receipt') + const result = await attachToExplanation( + ctx, + tenantId, + NEW_EXP, + { + bytes, + filename: 'receipt.pdf', + contentType: 'application/pdf', + description: 'Trainline receipt', + }, + writes.deps + ) + expect(result.attachmentCount).toBe(1) + const [mirrored] = await db + .select() + .from(freeagentExplanations) + .where( + and(eq(freeagentExplanations.tenantId, tenantId), eq(freeagentExplanations.url, NEW_EXP)) + ) + expect(mirrored).toMatchObject({ + categoryUrl: CATEGORY, + grossValue: '-42.50', + attachmentCount: 1, + }) + + // Refused before any bytes leave: wrong type, too big, or a URL off the connection's host. + await expect( + attachToExplanation( + ctx, + tenantId, + NEW_EXP, + { bytes, filename: 'x.docx', contentType: 'application/msword' }, + writes.deps + ) + ).rejects.toMatchObject({ code: 'validation' }) + await expect( + attachToExplanation( + ctx, + tenantId, + NEW_EXP, + { + bytes: new Uint8Array(5 * 1024 * 1024 + 1), + filename: 'big.png', + contentType: 'image/png', + }, + writes.deps + ) + ).rejects.toMatchObject({ code: 'validation' }) + await expect( + explainTransaction( + ctx, + tenantId, + { + bankTransactionUrl: 'https://evil.example/v2/bank_transactions/1', + categoryUrl: CATEGORY, + grossValue: '-1', + datedOn: '2026-10-01', + description: 'x', + }, + writes.deps + ) + ).rejects.toMatchObject({ code: 'validation' }) + await expect( + explainTransaction( + ctx, + tenantId, + { + bankTransactionUrl: TX1, + categoryUrl: CATEGORY, + grossValue: 'twelve', + datedOn: '2026-10-01', + description: 'x', + }, + writes.deps + ) + ).rejects.toMatchObject({ code: 'validation' }) + }) +}) + +// ---- Tools ---------------------------------------------------------------------------------------- + +describe('agent tools', () => { + it('offers nothing until connected; read tools to everyone, write tools to admins only', async () => { + const { tenantId, userId } = await tenantWithOwner() + const env = freeagentEnv() + expect(await freeagentAgentTools(makeToolCtx({ db, env, tenantId }))).toEqual([]) + await connect(tenantId, userId, env) + // Configured AND connected: connected alone, on a deployment without the app, is not enough. + expect(await freeagentAgentTools(makeToolCtx({ db, tenantId }))).toEqual([]) + + const admin = await freeagentAgentTools(makeToolCtx({ db, env, tenantId, bypass: true })) + expect(admin.map(t => t.name).sort()).toEqual(Object.values(FREEAGENT_TOOL_NAMES).sort()) + for (const t of admin) { + const writes = + t.name === FREEAGENT_TOOL_NAMES.explainTransaction || + t.name === FREEAGENT_TOOL_NAMES.createBill + expect(Boolean(t.requiresApproval), t.name).toBe(writes) + } + const member = await freeagentAgentTools( + makeToolCtx({ db, env, tenantId, bypass: false, userId }) + ) + expect(member.map(t => t.name)).not.toContain(FREEAGENT_TOOL_NAMES.explainTransaction) + expect(member.map(t => t.name)).toContain(FREEAGENT_TOOL_NAMES.searchTransactions) + + await syncTenant(makeJobCtx({ db, env }), tenantId, { + deps: fakeFreeAgent(booksRoutes({ pass: 1 })).deps, + listeners: [], + }) + const search = member.find(t => t.name === FREEAGENT_TOOL_NAMES.searchTransactions) + const out = JSON.parse((await search?.handler?.({ unexplainedOnly: true })) ?? '{}') + expect(out.transactions.map((t: { url: string }) => t.url)).toEqual([TX1]) + const invoices = member.find(t => t.name === FREEAGENT_TOOL_NAMES.listInvoices) + const overdue = JSON.parse((await invoices?.handler?.({ overdueOnly: true })) ?? '{}') + expect(overdue.invoices).toEqual([ + expect.objectContaining({ reference: 'INV-001', overdue: true, contact: 'Acme Ltd' }), + ]) + }) +}) + +// ---- Tenant isolation ----------------------------------------------------------------------------- + +describe('tenant isolation', () => { + it('another organisation can neither read nor change anything through the mounts', async () => { + const a = await tenantWithOwner() + // The second organisation, built inline so the shape of the proof is visible here. + const otherTenantId = (await createTestTenant(db)).id + const outsider = await createTestUser(db) + await linkUserToTenant(db, outsider.id, otherTenantId, 'owner') + const b = { + tenantId: otherTenantId, + userId: outsider.id, + cookie: { + ...sessionCookieHeader(await createTestSession(db, outsider.id, otherTenantId)), + ...ORIGIN, + }, + } + const env = freeagentEnv() + await connect(a.tenantId, a.userId, env) + await syncTenant(makeJobCtx({ db, env }), a.tenantId, { + deps: fakeFreeAgent(booksRoutes({ pass: 1 })).deps, + listeners: [], + }) + + // A sees its books. + const mine = await json<{ items: { url: string }[] }>( + await request('/api/freeagent/transactions', { headers: a.cookie }, { env }) + ) + expect(mine.items.map(t => t.url).sort()).toEqual([TX1, TX2]) + + // B sees nothing of A's, and is not connected. + const status = await json<{ + connected: boolean + companyName: string | null + counts: { transactions: number } + }>(await request('/api/freeagent/status', { headers: b.cookie }, { env })) + expect(status).toMatchObject({ + connected: false, + companyName: null, + counts: { transactions: 0 }, + }) + for (const path of [ + '/api/freeagent/transactions', + '/api/freeagent/transactions?unexplained=true', + '/api/freeagent/invoices', + ]) { + const list = await json<{ items: unknown[] }>( + await request(path, { headers: b.cookie }, { env }) + ) + expect(list.items, path).toEqual([]) + } + + // B can neither disconnect nor sync A. + expect( + (await request('/api/freeagent/connection', { method: 'DELETE', headers: b.cookie }, { env })) + .status + ).toBe(404) + expect( + (await request('/api/freeagent/sync', { method: 'POST', headers: b.cookie }, { env })).status + ).toBe(404) + const [stillA] = await db + .select() + .from(freeagentConnections) + .where(eq(freeagentConnections.tenantId, a.tenantId)) + expect(stillA?.status).toBe('connected') + expect( + await db + .select() + .from(freeagentBankTransactions) + .where(eq(freeagentBankTransactions.tenantId, a.tenantId)) + ).toHaveLength(2) + + // A's state cannot be replayed into B's callback either: a state names its tenant. + const config = makeJobCtx({ db, env }).config + const forged = await signState(config, CONNECT_STATE_PURPOSE, { + tenantId: 'not-a-uuid', + userId: b.userId, + }) + expect( + ( + await request( + `/api/hooks/freeagent/callback?code=x&state=${encodeURIComponent(forged)}`, + {}, + { env } + ) + ).status + ).toBe(401) + + // A's own disconnect removes A's tokens and mirror — and only A's. + expect( + (await request('/api/freeagent/connection', { method: 'DELETE', headers: a.cookie }, { env })) + .status + ).toBe(204) + expect( + await db + .select() + .from(freeagentBankTransactions) + .where(eq(freeagentBankTransactions.tenantId, a.tenantId)) + ).toEqual([]) + }) +}) diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/tests/config/client.test.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/config/client.test.ts new file mode 100644 index 0000000..4966842 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/config/client.test.ts @@ -0,0 +1,161 @@ +/** + * `FreeAgentClient` with an injected fetch: headers, 429 back-off (bounded), 401 → one forced + * refresh, `Link` pagination, the off-host refusal and the error mapping. No database, no network. + */ +import { describe, expect, it, vi } from 'vitest' +import { FreeAgentClient, parseLinkHeader, retryAfterMs } from '../../services/client' +import type { FetchLike } from '../../services/env' +import { FreeAgentError } from '../../services/errors' + +const BASE = 'https://api.sandbox.freeagent.com' + +function json(body: unknown, init: ResponseInit = {}): Response { + return new Response(JSON.stringify(body), { + ...init, + headers: { 'Content-Type': 'application/json', ...(init.headers as Record) }, + }) +} + +function client(fetch: FetchLike, token = vi.fn(async (_opts: { force: boolean }) => 'tok')) { + const sleep = vi.fn(async (_ms: number) => {}) + return { + c: new FreeAgentClient({ baseUrl: BASE, accessToken: token, deps: { fetch, sleep } }), + sleep, + token, + } +} + +describe('pure helpers', () => { + it('parses Link headers and Retry-After', () => { + expect( + parseLinkHeader( + `<${BASE}/v2/contacts?page=2&per_page=100>; rel="next", <${BASE}/v2/contacts?page=5>; rel="last"` + ) + ).toEqual({ + next: `${BASE}/v2/contacts?page=2&per_page=100`, + last: `${BASE}/v2/contacts?page=5`, + }) + expect(parseLinkHeader(null)).toEqual({}) + expect(retryAfterMs('60')).toBe(60_000) + expect(retryAfterMs(new Date(1_000_000 + 5_000).toUTCString(), 1_000_000)).toBeLessThanOrEqual( + 5_000 + ) + expect(retryAfterMs('soon')).toBeNull() + }) +}) + +describe('requests', () => { + it('always sends the API version, a bearer token and a user agent', async () => { + const fetch = vi.fn(async () => json({ company: { name: 'x' } })) + const { c } = client(fetch) + await c.get('/v2/company') + const [url, init] = fetch.mock.calls[0] ?? [] + const headers = new Headers(init?.headers) + expect(url).toBe(`${BASE}/v2/company`) + expect(headers.get('X-Api-Version')).toBe('2026-09-01') + expect(headers.get('Authorization')).toBe('Bearer tok') + expect(headers.get('User-Agent')).toBeTruthy() + }) + + it('waits out a 429 for its Retry-After, then succeeds', async () => { + let calls = 0 + const fetch: FetchLike = async () => + ++calls < 3 ? json({}, { status: 429, headers: { 'Retry-After': '2' } }) : json({ ok: true }) + const { c, sleep } = client(fetch) + expect(await c.get('/v2/company')).toEqual({ ok: true }) + expect(sleep.mock.calls.map(([ms]) => ms)).toEqual([2000, 2000]) + }) + + it('gives up after three waits, or at once when Retry-After is too long', async () => { + const always: FetchLike = async () => json({}, { status: 429, headers: { 'Retry-After': '1' } }) + const { c, sleep } = client(always) + await expect(c.get('/v2/company')).rejects.toMatchObject({ + code: 'rate_limited', + retryAfterSeconds: 1, + }) + expect(sleep).toHaveBeenCalledTimes(3) + + const hour: FetchLike = async () => + json({}, { status: 429, headers: { 'Retry-After': '3600' } }) + const second = client(hour) + await expect(second.c.get('/v2/company')).rejects.toMatchObject({ + code: 'rate_limited', + retryAfterSeconds: 3600, + }) + expect(second.sleep).not.toHaveBeenCalled() + }) + + it('forces one token refresh on a 401, and no more', async () => { + const statuses = [401, 200] + const fetch: FetchLike = async () => json({ ok: true }, { status: statuses.shift() ?? 401 }) + const { c, token } = client(fetch) + await c.get('/v2/company') + expect(token.mock.calls.map(([o]) => o.force)).toEqual([false, true]) + + const never: FetchLike = async () => + json({ errors: { error: { message: 'Access denied' } } }, { status: 401 }) + await expect(client(never).c.get('/v2/company')).rejects.toMatchObject({ + code: 'unauthorized', + message: 'Access denied', + }) + }) + + it('maps FreeAgent errors to codes with FreeAgent’s message', async () => { + const fetch: FetchLike = async () => + json({ errors: [{ message: 'Gross value is not a number' }] }, { status: 422 }) + await expect(client(fetch).c.post('/v2/bills', {})).rejects.toMatchObject({ + code: 'validation', + status: 422, + message: 'Gross value is not a number', + }) + const down: FetchLike = async () => new Response('oops', { status: 503 }) + await expect(client(down).c.get('/v2/company')).rejects.toMatchObject({ code: 'upstream' }) + const offline: FetchLike = async () => { + throw new TypeError('fetch failed') + } + await expect(client(offline).c.get('/v2/company')).rejects.toMatchObject({ code: 'network' }) + }) + + it('never sends the token off the connection’s API host', async () => { + const fetch = vi.fn(async () => json({})) + const { c } = client(fetch) + await expect(c.get('https://evil.example/v2/company')).rejects.toBeInstanceOf(FreeAgentError) + await expect(c.get('https://api.freeagent.com/v2/company')).rejects.toMatchObject({ + code: 'bad_response', + }) + expect(c.owns(`${BASE}/v2/bank_transactions/1`)).toBe(true) + expect(c.owns(`${BASE}/elsewhere`)).toBe(false) + expect(fetch).not.toHaveBeenCalled() + }) +}) + +describe('pagination', () => { + it('asks for 100 a page and follows rel="next" until it runs out', async () => { + const fetch = vi.fn(async input => { + const url = new URL(String(input)) + const page = Number(url.searchParams.get('page') ?? '1') + const next = page < 3 ? `<${BASE}/v2/contacts?page=${page + 1}&per_page=100>; rel="next"` : '' + return json( + { contacts: [{ url: `${BASE}/v2/contacts/${page}` }] }, + { headers: next ? { Link: next } : {} } + ) + }) + const { c } = client(fetch) + const all = await c.all<{ url: string }>('/v2/contacts', 'contacts', { view: 'all' }) + expect(all.map(x => x.url.split('/').pop())).toEqual(['1', '2', '3']) + const first = new URL(String(fetch.mock.calls[0]?.[0])) + expect(first.searchParams.get('per_page')).toBe('100') + expect(first.searchParams.get('view')).toBe('all') + }) + + it('stops at a next link pointing off-host', async () => { + const fetch: FetchLike = async () => + json( + { contacts: [] }, + { headers: { Link: '; rel="next"' } } + ) + await expect(client(fetch).c.all('/v2/contacts', 'contacts')).rejects.toMatchObject({ + code: 'bad_response', + }) + }) +}) diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/tests/config/mapping.test.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/config/mapping.test.ts new file mode 100644 index 0000000..3c5c8de --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/tests/config/mapping.test.ts @@ -0,0 +1,214 @@ +/** FreeAgent JSON → mirror rows, as tables (`services/mapping.ts`). Pure: no database, no network. */ +import { describe, expect, it } from 'vitest' +import { FreeAgentError } from '../../services/errors' +import { + isNonZeroDecimal, + isOverdue, + mapBankAccount, + mapBankTransaction, + mapBill, + mapCategories, + mapCompany, + mapContact, + mapExplanation, + mapInvoice, +} from '../../services/mapping' + +const API = 'https://api.sandbox.freeagent.com/v2' + +describe('decimals', () => { + it('treats every spelling of zero as zero, and keeps strings as strings', () => { + for (const zero of ['0', '0.0', '-0.00', '0.', '00.000']) + expect(isNonZeroDecimal(zero)).toBe(false) + for (const v of ['-0.01', '12', '-730.0']) expect(isNonZeroDecimal(v)).toBe(true) + expect(isNonZeroDecimal(null)).toBe(false) + }) +}) + +describe('bank transactions', () => { + it('maps amounts as strings and derives is_unexplained', () => { + const { row, explanations } = mapBankTransaction({ + url: `${API}/bank_transactions/1`, + amount: '-730.0', + bank_account: `${API}/bank_accounts/1`, + dated_on: '2026-10-01', + description: 'TRAINLINE', + full_description: 'TRAINLINE.COM/LONDON', + unexplained_amount: '-730.0', + is_manual: false, + updated_at: '2026-10-02T09:00:00.000Z', + bank_transaction_explanations: [], + }) + expect(row).toMatchObject({ + amount: '-730.0', + unexplainedAmount: '-730.0', + isUnexplained: true, + datedOn: '2026-10-01', + description: 'TRAINLINE', + }) + expect(row.faUpdatedAt).toEqual(new Date('2026-10-02T09:00:00.000Z')) + expect(explanations).toEqual([]) + }) + + it('reads an explained transaction, its embedded explanations, and a missing list as null', () => { + const explained = mapBankTransaction({ + url: `${API}/bank_transactions/2`, + amount: 100, + bank_account: `${API}/bank_accounts/1`, + dated_on: '2026-10-01', + unexplained_amount: '0.0', + bank_transaction_explanations: [ + { + url: `${API}/bank_transaction_explanations/9`, + bank_account: `${API}/bank_accounts/1`, + dated_on: '2026-10-01', + entry_type: 'Business Entertaining', + gross_value: '100.0', + is_deletable: true, + }, + ], + }) + expect(explained.row).toMatchObject({ amount: '100', isUnexplained: false }) + expect(explained.explanations).toEqual([ + expect.objectContaining({ + url: `${API}/bank_transaction_explanations/9`, + bankTransactionUrl: `${API}/bank_transactions/2`, + entryType: 'Business Entertaining', + grossValue: '100.0', + }), + ]) + expect(explained.explanations?.[0]).not.toHaveProperty('categoryUrl') + const bare = mapBankTransaction({ + url: `${API}/bank_transactions/3`, + amount: '1', + bank_account: `${API}/bank_accounts/1`, + dated_on: '2026-10-01', + }) + expect(bare.explanations).toBeNull() + }) + + it('refuses a transaction it cannot key, with a typed error', () => { + expect(() => mapBankTransaction({ amount: '1' })).toThrow(FreeAgentError) + try { + mapBankTransaction({ url: 'x' }) + } catch (err) { + expect((err as FreeAgentError).code).toBe('bad_response') + } + }) +}) + +describe('explanations', () => { + it('counts the 2026-09-01 attachments array, and falls back to the single attachment', () => { + const base = { + url: `${API}/bank_transaction_explanations/1`, + bank_transaction: `${API}/bank_transactions/1`, + category: `${API}/categories/285`, + gross_value: '-12.5', + sales_tax_rate: '20.0', + sales_tax_value: '-2.08', + type: 'Payment', + dated_on: '2026-10-01', + } + expect( + mapExplanation({ + ...base, + attachments: [{ url: `${API}/attachments/1` }, { url: `${API}/attachments/2` }], + }) + ).toMatchObject({ + categoryUrl: `${API}/categories/285`, + grossValue: '-12.5', + salesTaxRate: '20.0', + salesTaxValue: '-2.08', + entryType: 'Payment', + attachmentCount: 2, + }) + expect(mapExplanation({ ...base, attachment: { url: 'x' } }).attachmentCount).toBe(1) + expect(mapExplanation(base).attachmentCount).toBe(0) + }) +}) + +describe('everything else', () => { + it('flattens the four category lists, keeping which one each came from', () => { + const rows = mapCategories({ + admin_expenses_categories: [ + { + url: `${API}/categories/285`, + description: 'Accommodation and Meals', + nominal_code: '285', + allowable_for_tax: true, + auto_sales_tax_rate: 'Standard rate', + }, + ], + income_categories: [ + { url: `${API}/categories/001`, description: 'Sales', nominal_code: '001' }, + ], + general_categories: [ + { url: `${API}/categories/750`, description: 'Loan', nominal_code: '750' }, + ], + }) + expect(rows.map(r => [r.nominalCode, r.kind])).toEqual([ + ['285', 'admin_expenses'], + ['001', 'income'], + ['750', 'general'], + ]) + expect(rows[0]).toMatchObject({ allowableForTax: true, autoSalesTaxRate: 'Standard rate' }) + }) + + it('names a contact by organisation, else by person', () => { + expect( + mapContact({ url: `${API}/contacts/1`, organisation_name: 'Acme Ltd', first_name: 'A' }) + .displayName + ).toBe('Acme Ltd') + expect( + mapContact({ url: `${API}/contacts/2`, first_name: 'Ada', last_name: 'Lovelace' }).displayName + ).toBe('Ada Lovelace') + }) + + it('maps bills, invoices, accounts and the company', () => { + expect( + mapInvoice({ + url: `${API}/invoices/1`, + contact: `${API}/contacts/1`, + reference: 'INV-001', + dated_on: '2026-09-01', + due_on: '2026-10-01', + status: 'Open', + total_value: '1200.0', + paid_value: '0.0', + due_value: '1200.0', + }) + ).toMatchObject({ + reference: 'INV-001', + totalValue: '1200.0', + dueValue: '1200.0', + dueOn: '2026-10-01', + }) + expect( + mapBill({ url: `${API}/bills/1`, total_value: '50', sales_tax_value: '8.33' }) + ).toMatchObject({ totalValue: '50', salesTaxValue: '8.33' }) + expect( + mapBankAccount({ + url: `${API}/bank_accounts/1`, + name: 'Current', + current_balance: '10.5', + is_primary: true, + }) + ).toMatchObject({ name: 'Current', currentBalance: '10.5', isPrimary: true }) + expect( + mapCompany({ company: { url: `${API}/company`, name: 'GuideMode Ltd', currency: 'GBP' } }) + ).toEqual({ + companyUrl: `${API}/company`, + companyName: 'GuideMode Ltd', + companyCurrency: 'GBP', + }) + }) + + it('calls something overdue only when it is owed and past due', () => { + const today = '2026-10-24' + expect(isOverdue({ status: 'Open', dueOn: '2026-10-01', dueValue: '10.0' }, today)).toBe(true) + expect(isOverdue({ status: 'Open', dueOn: '2026-11-01', dueValue: '10.0' }, today)).toBe(false) + expect(isOverdue({ status: 'Open', dueOn: '2026-10-01', dueValue: '0.0' }, today)).toBe(false) + expect(isOverdue({ status: 'Draft', dueOn: '2026-10-01', dueValue: '10.0' }, today)).toBe(false) + expect(isOverdue({ status: 'Overdue', dueOn: null, dueValue: null }, today)).toBe(true) + }) +}) diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/tools/freeagent-tools.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/tools/freeagent-tools.ts new file mode 100644 index 0000000..e11b41f --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/tools/freeagent-tools.ts @@ -0,0 +1,369 @@ +/** + * Agent tools over the organisation's FreeAgent books. + * + * Five read tools answer from the MIRROR (no FreeAgent call, no rate limit spent): company and + * sync status, transaction search, categories, invoices and bills. Two write tools call FreeAgent + * and are `requiresApproval: true` — the loop stops and a person approves the exact arguments + * before the handler runs (`Tool.requiresApproval`, enforced by the kit's tool loop, never here). + * + * The builder offers nothing to a tenant that is not connected (`[]`), so a model is never shown a + * tool that could only answer "not connected". The write tools are offered only to an admin-level + * requester (`scope.bypass`): a member's run may read the books but not propose changes to them. + * + * Tools are bound to the RUN's tenant (`ctx.tenantId`); no input names a tenant. + */ +import { + freeagentCreateBillInputSchema, + freeagentExplainInputSchema, +} from '@rocketflare/shared/plugins/freeagent/index' +import { and, asc, desc, eq, gte, ilike, lte, or, type SQL, sql } from 'drizzle-orm' +import { z } from 'zod' +import type { Tool, ToolCtx } from '@/plugins/api' +import { defineTool } from '@/plugins/api' +import { loadStatus } from '../api/status' +import { + freeagentBankAccounts, + freeagentBankTransactions, + freeagentBills, + freeagentCategories, + freeagentContacts, + freeagentInvoices, +} from '../db/schema' +import { isUsable } from '../services/connection' +import type { FreeAgentDeps } from '../services/env' +import { freeagentHttp } from '../services/env' +import { isFreeAgentError } from '../services/errors' +import { isOverdue } from '../services/mapping' +import { overdueSql } from '../services/overdue' +import { createBill, explainTransaction } from '../services/writes' + +export const FREEAGENT_TOOL_NAMES = { + company: 'freeagent_company', + searchTransactions: 'freeagent_search_transactions', + listCategories: 'freeagent_list_categories', + listInvoices: 'freeagent_list_invoices', + listBills: 'freeagent_list_bills', + explainTransaction: 'freeagent_explain_transaction', + createBill: 'freeagent_create_bill', +} as const + +const DEFAULT_LIMIT = 25 +const MAX_LIMIT = 200 + +const limitSchema = z.coerce + .number() + .int() + .min(1) + .max(MAX_LIMIT) + .optional() + .describe(`How many to return (default ${DEFAULT_LIMIT}, max ${MAX_LIMIT})`) +const dateSchema = z.string().regex(/^\d{4}-\d{2}-\d{2}$/, 'YYYY-MM-DD') +const amountSchema = z.string().regex(/^-?\d+(\.\d+)?$/, 'A decimal such as 12.50') + +function escapeLike(text: string): string { + return `%${text.replace(/[%_\\]/g, m => `\\${m}`)}%` +} + +function today(): string { + return new Date().toISOString().slice(0, 10) +} + +/** A FreeAgent failure as something the model can act on, never a stack. */ +function toolError(err: unknown): string { + if (isFreeAgentError(err)) { + return JSON.stringify({ + error: err.code, + message: err.message, + ...(err.code === 'needs_reconnect' && { + hint: 'Tell the user an admin must reconnect FreeAgent in Settings → FreeAgent.', + }), + ...(err.code === 'rate_limited' && { + hint: `FreeAgent is rate limiting; retry in about ${err.retryAfterSeconds ?? 60} seconds.`, + }), + ...(err.code === 'validation' && { + hint: 'FreeAgent rejected the input. Check the URLs come from the freeagent_* tools and the amount is signed like the transaction.', + }), + }) + } + throw err +} + +export function companyTool(ctx: ToolCtx): Tool> { + return defineTool({ + name: FREEAGENT_TOOL_NAMES.company, + description: + "The organisation's FreeAgent connection: company name, currency, environment, when the books " + + 'were last synced, and how many accounts, transactions (and unexplained ones), invoices and bills are mirrored.', + schema: z.object({}).strict() as unknown as z.ZodType>, + async handler() { + const s = await loadStatus(ctx.db, ctx.config, ctx.tenantId) + return JSON.stringify({ + company: s.companyName, + currency: s.companyCurrency, + environment: s.connectionEnvironment, + status: s.status, + lastSyncAt: s.lastSyncAt?.toISOString() ?? null, + lastError: s.lastError, + counts: s.counts, + note: 'Figures come from a mirror synced every 30 minutes; recent changes in FreeAgent may not show yet.', + }) + }, + }) +} + +export const searchTransactionsInputSchema = z.object({ + from: dateSchema.optional().describe('Earliest date, inclusive'), + to: dateSchema.optional().describe('Latest date, inclusive'), + amount: amountSchema + .optional() + .describe('Exact amount, signed (money out is negative, e.g. -12.50)'), + minAmount: amountSchema.optional().describe('Smallest signed amount, inclusive'), + maxAmount: amountSchema.optional().describe('Largest signed amount, inclusive'), + text: z.string().trim().max(200).optional().describe('Match against the bank description'), + unexplainedOnly: z.boolean().optional().describe('Only transactions not yet fully explained'), + limit: limitSchema, +}) +export type SearchTransactionsInput = z.infer + +export function searchTransactionsTool(ctx: ToolCtx): Tool { + return defineTool({ + name: FREEAGENT_TOOL_NAMES.searchTransactions, + description: + 'Search bank transactions in FreeAgent by date range, signed amount (money out is negative), ' + + 'description text, or unexplained only. Answers each transaction URL (use it with ' + + 'freeagent_explain_transaction), date, amount, unexplained amount and bank account.', + schema: searchTransactionsInputSchema, + async handler(input) { + const t = freeagentBankTransactions + const filters: SQL[] = [eq(t.tenantId, ctx.tenantId)] + if (input.from) filters.push(gte(t.datedOn, input.from)) + if (input.to) filters.push(lte(t.datedOn, input.to)) + if (input.amount) filters.push(eq(t.amount, input.amount)) + if (input.minAmount) filters.push(gte(t.amount, input.minAmount)) + if (input.maxAmount) filters.push(lte(t.amount, input.maxAmount)) + if (input.unexplainedOnly) filters.push(eq(t.isUnexplained, true)) + if (input.text) { + const match = or( + ilike(t.description, escapeLike(input.text)), + ilike(t.fullDescription, escapeLike(input.text)) + ) + if (match) filters.push(match) + } + const rows = await ctx.db + .select({ tx: t, account: freeagentBankAccounts.name }) + .from(t) + .leftJoin( + freeagentBankAccounts, + and( + eq(freeagentBankAccounts.tenantId, t.tenantId), + eq(freeagentBankAccounts.url, t.bankAccountUrl) + ) + ) + .where(and(...filters)) + .orderBy(desc(t.datedOn), asc(t.url)) + .limit(input.limit ?? DEFAULT_LIMIT) + if (rows.length === 0) { + return JSON.stringify({ + transactions: [], + hint: 'No mirrored transaction matches. Widen the date range or drop a filter; the mirror holds about the last 400 days.', + }) + } + return JSON.stringify({ + transactions: rows.map(({ tx, account }) => ({ + url: tx.url, + datedOn: tx.datedOn, + amount: tx.amount, + unexplainedAmount: tx.unexplainedAmount, + unexplained: tx.isUnexplained, + description: tx.description, + bankAccount: account, + bankAccountUrl: tx.bankAccountUrl, + })), + }) + }, + }) +} + +export const listCategoriesInputSchema = z.object({ + search: z.string().trim().max(100).optional().describe('Match description or nominal code'), + kind: z + .enum(['admin_expenses', 'cost_of_sales', 'income', 'general']) + .optional() + .describe('Which group of categories'), +}) +export type ListCategoriesInput = z.infer + +export function listCategoriesTool(ctx: ToolCtx): Tool { + return defineTool({ + name: FREEAGENT_TOOL_NAMES.listCategories, + description: + 'List FreeAgent accounting categories (nominal codes): URL, description, code, group and ' + + 'automatic VAT rate. Use a category URL with freeagent_explain_transaction or freeagent_create_bill.', + schema: listCategoriesInputSchema, + async handler(input) { + const c = freeagentCategories + const filters: SQL[] = [eq(c.tenantId, ctx.tenantId)] + if (input.kind) filters.push(eq(c.kind, input.kind)) + if (input.search) { + const match = or( + ilike(c.description, escapeLike(input.search)), + ilike(c.nominalCode, escapeLike(input.search)) + ) + if (match) filters.push(match) + } + const rows = await ctx.db + .select() + .from(c) + .where(and(...filters)) + .orderBy(asc(c.nominalCode)) + .limit(300) + return JSON.stringify({ + categories: rows.map(r => ({ + url: r.url, + description: r.description, + nominalCode: r.nominalCode, + kind: r.kind, + group: r.groupDescription, + autoSalesTaxRate: r.autoSalesTaxRate, + })), + ...(rows.length === 0 && { hint: 'No category matches; try a shorter search or none.' }), + }) + }, + }) +} + +export const listDocumentsInputSchema = z.object({ + overdueOnly: z.boolean().optional().describe('Only those still owed and past their due date'), + status: z + .string() + .trim() + .max(40) + .optional() + .describe('FreeAgent status, e.g. Open, Overdue, Paid, Draft'), + limit: limitSchema, +}) +export type ListDocumentsInput = z.infer + +function documentsTool(ctx: ToolCtx, kind: 'invoices' | 'bills'): Tool { + const table = kind === 'invoices' ? freeagentInvoices : freeagentBills + const t = table as typeof freeagentInvoices + return defineTool({ + name: kind === 'invoices' ? FREEAGENT_TOOL_NAMES.listInvoices : FREEAGENT_TOOL_NAMES.listBills, + description: + kind === 'invoices' + ? 'List sales invoices from FreeAgent, newest first: reference, client, dates, total, paid, still due, and whether overdue.' + : 'List supplier bills from FreeAgent, newest first: reference, supplier, dates, total, paid, still due, and whether overdue.', + schema: listDocumentsInputSchema, + async handler(input) { + const filters: SQL[] = [eq(t.tenantId, ctx.tenantId)] + if (input.overdueOnly) filters.push(overdueSql(table)) + if (input.status) filters.push(sql`lower(${t.status}) = lower(${input.status})`) + const rows = await ctx.db + .select({ doc: t, contact: freeagentContacts.displayName }) + .from(t) + .leftJoin( + freeagentContacts, + and(eq(freeagentContacts.tenantId, t.tenantId), eq(freeagentContacts.url, t.contactUrl)) + ) + .where(and(...filters)) + .orderBy(desc(t.datedOn), asc(t.url)) + .limit(input.limit ?? DEFAULT_LIMIT) + const day = today() + return JSON.stringify({ + [kind]: rows.map(({ doc, contact }) => ({ + url: doc.url, + reference: doc.reference, + contact, + status: doc.status, + datedOn: doc.datedOn, + dueOn: doc.dueOn, + currency: doc.currency, + total: doc.totalValue, + paid: doc.paidValue, + due: doc.dueValue, + overdue: isOverdue(doc, day), + })), + ...(rows.length === 0 && { hint: `No ${kind} match. Say so rather than inventing any.` }), + }) + }, + }) +} + +export const listInvoicesTool = (ctx: ToolCtx) => documentsTool(ctx, 'invoices') +export const listBillsTool = (ctx: ToolCtx) => documentsTool(ctx, 'bills') + +export type ExplainTransactionToolInput = z.infer + +export function explainTransactionTool( + ctx: ToolCtx, + deps: FreeAgentDeps = freeagentHttp +): Tool { + return defineTool({ + name: FREEAGENT_TOOL_NAMES.explainTransaction, + description: + 'Explain a bank transaction in FreeAgent (WRITES to the books; a person approves first): the ' + + 'transaction URL from freeagent_search_transactions, a category URL from freeagent_list_categories, ' + + 'the gross value signed like the transaction (money out negative), the date, a description, and optionally the VAT rate.', + schema: freeagentExplainInputSchema, + requiresApproval: true, + approvalMessage: 'Explain this bank transaction in FreeAgent?', + allowEdits: true, + async handler(input) { + try { + const explanation = await explainTransaction(ctx, ctx.tenantId, input, deps) + return JSON.stringify({ explained: true, explanation }) + } catch (err) { + return toolError(err) + } + }, + }) +} + +export type CreateBillToolInput = z.infer + +export function createBillTool( + ctx: ToolCtx, + deps: FreeAgentDeps = freeagentHttp +): Tool { + return defineTool({ + name: FREEAGENT_TOOL_NAMES.createBill, + description: + 'Create a supplier bill in FreeAgent (WRITES to the books; a person approves first): the ' + + 'supplier contact URL, a reference, dated and due dates, and one or more items each with a ' + + 'category URL, description and total including VAT.', + schema: freeagentCreateBillInputSchema, + requiresApproval: true, + approvalMessage: 'Create this bill in FreeAgent?', + allowEdits: true, + async handler(input) { + try { + const bill = await createBill(ctx, ctx.tenantId, input, deps) + return JSON.stringify({ created: true, bill }) + } catch (err) { + return toolError(err) + } + }, + }) +} + +/** + * What `agentTools` offers this run: nothing unless the organisation is connected (and the + * deployment configured); the two write tools only for an admin-level requester. + */ +export async function freeagentAgentTools( + ctx: ToolCtx, + deps: FreeAgentDeps = freeagentHttp +): Promise { + if (!(await isUsable(ctx, ctx.tenantId))) return [] + const tools: Tool[] = [ + companyTool(ctx) as Tool, + searchTransactionsTool(ctx) as Tool, + listCategoriesTool(ctx) as Tool, + listInvoicesTool(ctx) as Tool, + listBillsTool(ctx) as Tool, + ] + if (ctx.scope.bypass) { + tools.push(explainTransactionTool(ctx, deps) as Tool, createBillTool(ctx, deps) as Tool) + } + return tools +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/ui/hooks/useFreeAgent.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/hooks/useFreeAgent.ts new file mode 100644 index 0000000..e05c5bf --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/hooks/useFreeAgent.ts @@ -0,0 +1,74 @@ +import { + type FreeAgentInvoiceListQuery, + type FreeAgentTransactionListQuery, + freeagentConnectResponseSchema, + freeagentInvoiceListResponseSchema, + freeagentStatusSchema, + freeagentSyncResponseSchema, + freeagentTransactionListResponseSchema, +} from '@rocketflare/shared/plugins/freeagent/index' +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query' +import { api } from '@/plugins/api/ui' +import { freeagentBooksKeys, freeagentConnectionKeys } from '../query-keys' + +const BASE = '/api/freeagent' + +function params(query: Record): string { + const p = new URLSearchParams() + for (const [k, v] of Object.entries(query)) if (v !== undefined) p.set(k, String(v)) + return p.toString() +} + +export function useFreeAgentStatus() { + return useQuery({ + queryKey: freeagentConnectionKeys.status(), + queryFn: () => api.get(`${BASE}/status`, { schema: freeagentStatusSchema }), + }) +} + +export function useConnectFreeAgent() { + return useMutation({ + mutationFn: () => api.post(`${BASE}/connect`, {}, { schema: freeagentConnectResponseSchema }), + }) +} + +export function useDisconnectFreeAgent() { + const queryClient = useQueryClient() + return useMutation({ + mutationFn: () => api.delete(`${BASE}/connection`), + onSuccess: async () => { + await queryClient.invalidateQueries({ queryKey: freeagentConnectionKeys.all }) + await queryClient.invalidateQueries({ queryKey: freeagentBooksKeys.all }) + }, + }) +} + +export function useSyncFreeAgent() { + const queryClient = useQueryClient() + return useMutation({ + mutationFn: () => api.post(`${BASE}/sync`, {}, { schema: freeagentSyncResponseSchema }), + onSuccess: () => queryClient.invalidateQueries({ queryKey: freeagentConnectionKeys.all }), + }) +} + +export function useFreeAgentTransactions( + query: Partial> +) { + return useQuery({ + queryKey: freeagentBooksKeys.transactions(query), + queryFn: () => + api.get(`${BASE}/transactions?${params(query)}`, { + schema: freeagentTransactionListResponseSchema, + }), + }) +} + +export function useFreeAgentInvoices( + query: Partial> +) { + return useQuery({ + queryKey: freeagentBooksKeys.invoices(query), + queryFn: () => + api.get(`${BASE}/invoices?${params(query)}`, { schema: freeagentInvoiceListResponseSchema }), + }) +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/ui/index.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/index.ts new file mode 100644 index 0000000..f37c625 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/index.ts @@ -0,0 +1,41 @@ +/** + * `freeagent` — the UI entry (D31): one page (`/freeagent`), one nav item, one settings tab. Pages + * are `lazy()` so none of this rides in the main bundle; the route, the nav item and the tab share + * one guard, so a link never points at a page its reader cannot open. + */ +import { BanknotesIcon } from '@heroicons/react/24/outline' +import { FREEAGENT_SUBJECT, freeagentShared } from '@rocketflare/shared/plugins/freeagent/index' +import { createElement, lazy, Suspense } from 'react' +import type { NavGuard, UiPlugin } from '@/plugins/api/ui-wiring' +import { freeagentQueryKeys } from './query-keys' + +const FreeAgentPage = lazy(() => import('./pages/FreeAgentPage')) +const FreeAgentSettingsPage = lazy(() => import('./pages/FreeAgentSettings')) + +const readFreeAgent: NavGuard = { action: 'read', subject: FREEAGENT_SUBJECT } + +export const freeagentUi = { + shared: freeagentShared, + routes: [{ path: '/freeagent', Component: FreeAgentPage, guard: readFreeAgent }], + nav: [ + { + items: [{ to: '/freeagent', label: 'FreeAgent', icon: BanknotesIcon, guard: readFreeAgent }], + }, + ], + settingsTabs: ({ can }) => + can('read', FREEAGENT_SUBJECT) + ? [ + { + id: 'freeagent', + label: 'FreeAgent', + icon: createElement(BanknotesIcon, { className: 'w-4 h-4' }), + content: createElement( + Suspense, + { fallback: null }, + createElement(FreeAgentSettingsPage) + ), + }, + ] + : [], + queryKeys: freeagentQueryKeys, +} satisfies UiPlugin diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/ui/pages/FreeAgentPage.tsx b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/pages/FreeAgentPage.tsx new file mode 100644 index 0000000..06a3c55 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/pages/FreeAgentPage.tsx @@ -0,0 +1,204 @@ +/** + * FreeAgent: the organisation's unexplained bank transactions and its recent invoices, from the + * mirror. Two plain tables — the place to see what a receipts matcher (or a person) still has to + * explain, and who still owes money. Amounts are shown exactly as FreeAgent sent them. + */ +import { BanknotesIcon } from '@heroicons/react/24/outline' +import { useState } from 'react' +import { + EmptyState, + PageHeader, + PaginationControls, + SectionPanel, + SkeletonRows, + timeAgo, +} from '@/plugins/api/ui' +import { + useFreeAgentInvoices, + useFreeAgentStatus, + useFreeAgentTransactions, +} from '../hooks/useFreeAgent' + +function Money({ value, currency }: { value: string | null; currency?: string | null }) { + if (value === null) return — + const negative = value.trim().startsWith('-') + return ( + + {value} + {currency ? ` ${currency}` : ''} + + ) +} + +export default function FreeAgentPage() { + const status = useFreeAgentStatus() + const connected = status.data?.status != null + return ( +
+ + {status.isPending ? ( + + + + ) : !connected ? ( + + + + ) : ( + <> + + + + )} +
+ ) +} + +function UnexplainedTransactions() { + const [page, setPage] = useState(1) + const list = useFreeAgentTransactions({ unexplained: 'true', page, pageSize: 25 }) + return ( + + {list.isPending ? ( +
+ +
+ ) : list.isError || !list.data ? ( +

The transactions could not be loaded.

+ ) : list.data.items.length === 0 ? ( +

Everything is explained.

+ ) : ( + <> +
+ + + + + + + + + + + + {list.data.items.map(t => ( + + + + + + + + ))} + +
DateDescriptionAccountAmountUnexplained
{t.datedOn}{t.description ?? '—'}{t.bankAccountName ?? '—'} + + + +
+
+ + + )} + + ) +} + +function RecentInvoices() { + const [page, setPage] = useState(1) + const [overdue, setOverdue] = useState(false) + const list = useFreeAgentInvoices({ page, pageSize: 25, ...(overdue && { overdue: 'true' }) }) + return ( + + { + setOverdue(e.target.checked) + setPage(1) + }} + /> + Overdue only + + } + > + {list.isPending ? ( +
+ +
+ ) : list.isError || !list.data ? ( +

The invoices could not be loaded.

+ ) : list.data.items.length === 0 ? ( +

+ {overdue ? 'Nothing is overdue.' : 'No invoices yet.'} +

+ ) : ( + <> +
+ + + + + + + + + + + + + + {list.data.items.map(i => ( + + + + + + + + + + ))} + +
ReferenceClientDatedDueStatusTotalStill due
{i.reference ?? '—'}{i.contactName ?? '—'}{i.datedOn ?? '—'}{i.dueOn ?? '—'} + + {i.overdue ? 'Overdue' : (i.status ?? '—')} + + + + + +
+
+ + + )} + + ) +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/ui/pages/FreeAgentSettings.tsx b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/pages/FreeAgentSettings.tsx new file mode 100644 index 0000000..aef4cb6 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/pages/FreeAgentSettings.tsx @@ -0,0 +1,203 @@ +/** + * Settings → FreeAgent: connect, reconnect or disconnect this organisation's FreeAgent company, + * see which environment the deployment uses, when the books last synced and what went wrong, and + * sync now. Read-only without `manage FreeAgent`. + * + * Connecting leaves the app: the API answers FreeAgent's approve URL and the browser goes there; + * FreeAgent sends it back to `/api/hooks/freeagent/callback`, which lands here with `?connected=` + * or `?connectError=` — read once, turned into a toast, and dropped from the address bar. + */ +import type { FreeAgentStatus } from '@rocketflare/shared/plugins/freeagent/index' +import { useEffect, useState } from 'react' +import { + ApiError, + ConfirmModal, + formatDateTime, + SectionPanel, + SectionPanelSkeleton, + SettingRow, + showToast, + timeAgo, + usePermissions, +} from '@/plugins/api/ui' +import { + useConnectFreeAgent, + useDisconnectFreeAgent, + useFreeAgentStatus, + useSyncFreeAgent, +} from '../hooks/useFreeAgent' + +const CONNECT_ERRORS: Record = { + access_denied: 'FreeAgent access was not granted.', + approval_failed: 'FreeAgent did not approve the connection.', + missing_code: 'FreeAgent sent no approval code. Try again.', + not_configured: 'This deployment has no FreeAgent app configured.', + code_expired: 'The approval expired before it was used. Try again.', + exchange_failed: 'Completing the connection with FreeAgent failed. Try again.', +} + +function useCallbackToast() { + useEffect(() => { + const url = new URL(window.location.href) + const connected = url.searchParams.get('connected') + const error = url.searchParams.get('connectError') + if (connected !== 'freeagent' && !error) return + if (connected === 'freeagent') + showToast('FreeAgent connected. The first sync has started.', 'success') + if (error) showToast(CONNECT_ERRORS[error] ?? `Connecting failed (${error}).`, 'error') + url.searchParams.delete('connected') + url.searchParams.delete('connectError') + window.history.replaceState(null, '', url.toString()) + }, []) +} + +export default function FreeAgentSettingsPage() { + useCallbackToast() + const status = useFreeAgentStatus() + if (status.isPending) return + if (status.isError || !status.data) { + return ( + +

The FreeAgent status could not be loaded.

+ + ) + } + return +} + +function errorMessage(err: unknown, fallback: string): string { + return err instanceof ApiError ? err.message : fallback +} + +function Connection({ status }: { status: FreeAgentStatus }) { + const { can } = usePermissions() + const canManage = can('manage', 'FreeAgent') + const connect = useConnectFreeAgent() + const disconnect = useDisconnectFreeAgent() + const sync = useSyncFreeAgent() + const [confirmDisconnect, setConfirmDisconnect] = useState(false) + const hasConnection = status.status !== null + const mismatch = + status.connectionEnvironment !== null && status.connectionEnvironment !== status.environment + + const startConnect = () => + connect.mutate(undefined, { + onSuccess: ({ approveUrl }) => window.location.assign(approveUrl), + onError: err => showToast(errorMessage(err, 'Could not start connecting'), 'error'), + }) + + return ( + + + + {status.environment} + + + {!status.configured && ( +
+ This deployment has no FreeAgent app. An operator sets FREEAGENT_CLIENT_ID and + FREEAGENT_CLIENT_SECRET (see the freeagent skill). +
+ )} + + {status.connected ? ( + connected + ) : status.status === 'needs_reconnect' ? ( + needs reconnecting + ) : ( + not connected + )} + + {hasConnection && ( + <> + + + {status.companyName ?? '—'} + {status.companyCurrency ? ` (${status.companyCurrency})` : ''} + + + + + {timeAgo(status.lastSyncAt)} + {status.lastSyncAt && + ` · ${status.counts.transactions} transactions, ${status.counts.unexplained} unexplained`} + + + + )} + {mismatch && ( +
+ This connection was made in {status.connectionEnvironment}; the deployment now uses{' '} + {status.environment}. Reconnect to switch. +
+ )} + {status.lastError && ( +
+ + {status.lastError} + {status.lastErrorAt && ` (${timeAgo(status.lastErrorAt)})`} + +
+ )} + {canManage && ( +
+ {status.connected && ( + + )} + {hasConnection && ( + + )} + +
+ )} + setConfirmDisconnect(false)} + onConfirm={() => + disconnect.mutate(undefined, { + onSuccess: () => { + setConfirmDisconnect(false) + showToast('FreeAgent disconnected', 'success') + }, + onError: err => showToast(errorMessage(err, 'Could not disconnect'), 'error'), + }) + } + /> + + ) +} diff --git a/plugins/freeagent/apps/web/src/plugins/freeagent/ui/query-keys.ts b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/query-keys.ts new file mode 100644 index 0000000..afa4bf8 --- /dev/null +++ b/plugins/freeagent/apps/web/src/plugins/freeagent/ui/query-keys.ts @@ -0,0 +1,23 @@ +/** Query-key families. Each root is an `entity.changed` entity the server nudges. */ +import { + FREEAGENT_BOOKS_ENTITY, + FREEAGENT_CONNECTION_ENTITY, +} from '@rocketflare/shared/plugins/freeagent/index' + +export const freeagentConnectionKeys = { + all: [FREEAGENT_CONNECTION_ENTITY] as const, + status: () => [FREEAGENT_CONNECTION_ENTITY, 'status'] as const, +} + +export const freeagentBooksKeys = { + all: [FREEAGENT_BOOKS_ENTITY] as const, + transactions: (filters: Record) => + [FREEAGENT_BOOKS_ENTITY, 'transactions', filters] as const, + invoices: (filters: Record) => + [FREEAGENT_BOOKS_ENTITY, 'invoices', filters] as const, +} + +export const freeagentQueryKeys = { + [FREEAGENT_CONNECTION_ENTITY]: freeagentConnectionKeys, + [FREEAGENT_BOOKS_ENTITY]: freeagentBooksKeys, +} as const diff --git a/plugins/freeagent/docs/plugins/freeagent/upgrades/README.md b/plugins/freeagent/docs/plugins/freeagent/upgrades/README.md new file mode 100644 index 0000000..e6c9535 --- /dev/null +++ b/plugins/freeagent/docs/plugins/freeagent/upgrades/README.md @@ -0,0 +1,28 @@ +# Release notes, written for an agent + +One file per release of this plugin, `X.Y.Z.md`, plus `unreleased.md`, which collects entries +between releases. `CHANGELOG.md` at the repository root is the human index. These files are the +porting instructions: `pnpm plugin upgrade freeagent` drives the kit's `scripts/upgrade.mjs`, +which reads their frontmatter. + +## The shape + +Frontmatter comes first, then four fixed headings in this order: **What changed**, **How to +apply**, **Conflicts to expect**, **Verify**. + +| Field | Means | +|---|---| +| `version` · `previous` | This release and the one before it. `previous: null` appears only on the first release. `plugin upgrade` walks this chain, so a gap is permanent | +| `date` | `YYYY-MM-DD` | +| `breaking` | Does an adopter have to change their own code? | +| `migrations` | **Descriptions of the schema change, never file names.** The host generates the DDL with `pnpm db:generate`; this plugin never ships a migration | +| `areas` · `touches_surfaces` · `requires_surfaces` | Where the change lands | +| `manual` | Is there a step no tool can do? | + +## The rules + +- **Released history is never rewritten.** Every app pins a commit of this repository. +- **A behaviour change adds an entry to `unreleased.md` in the same commit.** + `node scripts/release.mjs X.Y.Z` folds it into `X.Y.Z.md`. +- **Never rename anything across releases**, whether a table, a column, a provider id or a tool + name. Expand and contract instead. diff --git a/plugins/freeagent/docs/plugins/freeagent/upgrades/unreleased.md b/plugins/freeagent/docs/plugins/freeagent/upgrades/unreleased.md new file mode 100644 index 0000000..9feadf0 --- /dev/null +++ b/plugins/freeagent/docs/plugins/freeagent/upgrades/unreleased.md @@ -0,0 +1,41 @@ +--- +version: unreleased +previous: null +date: null +breaking: false +migrations: + - freeagent_connections — one row per tenant (unique tenant_id): environment, status, sealed refresh and access tokens, company, sync claim and last outcome; RLS policy + - freeagent_sync_state — one updated_since cursor per (tenant, resource), unique; RLS policy + - freeagent_bank_accounts, freeagent_bank_transactions, freeagent_explanations, freeagent_categories, freeagent_contacts, freeagent_bills, freeagent_invoices — the mirror, each unique on (tenant_id, url), money as numeric; RLS policies +areas: [api, ui, cli] +touches_surfaces: [] +requires_surfaces: [] +manual: true +--- + +## What changed + +First release of `freeagent`: an organisation connects its FreeAgent company over OAuth, and its books are mirrored every 30 minutes for pages, agents, analytics and other plugins. + +- OAuth through `POST /api/freeagent/connect` and the public callback `/api/hooks/freeagent/callback`; tokens sealed; `invalid_grant` marks the connection `needs_reconnect`. +- `freeagent.sync` (cron `*/30 * * * *`, "Sync now", and after connecting) mirrors bank accounts, transactions, explanations, categories, contacts, bills and invoices incrementally by `updated_since`, then calls every `freeagent:sync-listeners` extension. +- Every request sends `X-Api-Version: 2026-09-01` (multiple attachments per explanation). +- Service API for other plugins: `freeagentClientFor`, `explainTransaction`, `attachToExplanation`, `deleteExplanation`, `createBill`, `listUnexplainedTransactions`. +- Agent tools: five read tools over the mirror; `freeagent_explain_transaction` and `freeagent_create_bill` need approval and are offered to admin-level requesters only. +- UI: Settings → FreeAgent and a FreeAgent page; CLI `rocketflare freeagent status|sync`; drizzle-cube cubes contributed to `analytics` when it is installed. + +## How to apply + +1. Install with `pnpm plugin add https://github.com/rocketflare-dev/rocketflare-plugins.git@ --subdir plugins/freeagent --apply` on a kit at 0.21.0 or later. +2. Run `pnpm db:generate --name plugin-freeagent-`, read the SQL (nine `CREATE TABLE`s and their policies), then `pnpm db:migrate`. +3. Register a FreeAgent app and set `FREEAGENT_CLIENT_ID`, `FREEAGENT_CLIENT_SECRET` and `FREEAGENT_ENVIRONMENT` — the `freeagent` skill (`/freeagent register`, `/freeagent secrets`) walks both. `pnpm provision cloudflare ` writes the `*/30 * * * *` cron into both tomls. + +## Conflicts to expect + +None — a first release adds files only. + +## Verify + +1. `pnpm plugin check` passes. +2. `pnpm gate` passes, including `src/plugins/freeagent/tests/**`. +3. After connecting, `pnpm cli freeagent status` shows the company and a last-sync time. diff --git a/plugins/freeagent/docs/upgrades/README.md b/plugins/freeagent/docs/upgrades/README.md new file mode 100644 index 0000000..e6c9535 --- /dev/null +++ b/plugins/freeagent/docs/upgrades/README.md @@ -0,0 +1,28 @@ +# Release notes, written for an agent + +One file per release of this plugin, `X.Y.Z.md`, plus `unreleased.md`, which collects entries +between releases. `CHANGELOG.md` at the repository root is the human index. These files are the +porting instructions: `pnpm plugin upgrade freeagent` drives the kit's `scripts/upgrade.mjs`, +which reads their frontmatter. + +## The shape + +Frontmatter comes first, then four fixed headings in this order: **What changed**, **How to +apply**, **Conflicts to expect**, **Verify**. + +| Field | Means | +|---|---| +| `version` · `previous` | This release and the one before it. `previous: null` appears only on the first release. `plugin upgrade` walks this chain, so a gap is permanent | +| `date` | `YYYY-MM-DD` | +| `breaking` | Does an adopter have to change their own code? | +| `migrations` | **Descriptions of the schema change, never file names.** The host generates the DDL with `pnpm db:generate`; this plugin never ships a migration | +| `areas` · `touches_surfaces` · `requires_surfaces` | Where the change lands | +| `manual` | Is there a step no tool can do? | + +## The rules + +- **Released history is never rewritten.** Every app pins a commit of this repository. +- **A behaviour change adds an entry to `unreleased.md` in the same commit.** + `node scripts/release.mjs X.Y.Z` folds it into `X.Y.Z.md`. +- **Never rename anything across releases**, whether a table, a column, a provider id or a tool + name. Expand and contract instead. diff --git a/plugins/freeagent/docs/upgrades/unreleased.md b/plugins/freeagent/docs/upgrades/unreleased.md new file mode 100644 index 0000000..9feadf0 --- /dev/null +++ b/plugins/freeagent/docs/upgrades/unreleased.md @@ -0,0 +1,41 @@ +--- +version: unreleased +previous: null +date: null +breaking: false +migrations: + - freeagent_connections — one row per tenant (unique tenant_id): environment, status, sealed refresh and access tokens, company, sync claim and last outcome; RLS policy + - freeagent_sync_state — one updated_since cursor per (tenant, resource), unique; RLS policy + - freeagent_bank_accounts, freeagent_bank_transactions, freeagent_explanations, freeagent_categories, freeagent_contacts, freeagent_bills, freeagent_invoices — the mirror, each unique on (tenant_id, url), money as numeric; RLS policies +areas: [api, ui, cli] +touches_surfaces: [] +requires_surfaces: [] +manual: true +--- + +## What changed + +First release of `freeagent`: an organisation connects its FreeAgent company over OAuth, and its books are mirrored every 30 minutes for pages, agents, analytics and other plugins. + +- OAuth through `POST /api/freeagent/connect` and the public callback `/api/hooks/freeagent/callback`; tokens sealed; `invalid_grant` marks the connection `needs_reconnect`. +- `freeagent.sync` (cron `*/30 * * * *`, "Sync now", and after connecting) mirrors bank accounts, transactions, explanations, categories, contacts, bills and invoices incrementally by `updated_since`, then calls every `freeagent:sync-listeners` extension. +- Every request sends `X-Api-Version: 2026-09-01` (multiple attachments per explanation). +- Service API for other plugins: `freeagentClientFor`, `explainTransaction`, `attachToExplanation`, `deleteExplanation`, `createBill`, `listUnexplainedTransactions`. +- Agent tools: five read tools over the mirror; `freeagent_explain_transaction` and `freeagent_create_bill` need approval and are offered to admin-level requesters only. +- UI: Settings → FreeAgent and a FreeAgent page; CLI `rocketflare freeagent status|sync`; drizzle-cube cubes contributed to `analytics` when it is installed. + +## How to apply + +1. Install with `pnpm plugin add https://github.com/rocketflare-dev/rocketflare-plugins.git@ --subdir plugins/freeagent --apply` on a kit at 0.21.0 or later. +2. Run `pnpm db:generate --name plugin-freeagent-`, read the SQL (nine `CREATE TABLE`s and their policies), then `pnpm db:migrate`. +3. Register a FreeAgent app and set `FREEAGENT_CLIENT_ID`, `FREEAGENT_CLIENT_SECRET` and `FREEAGENT_ENVIRONMENT` — the `freeagent` skill (`/freeagent register`, `/freeagent secrets`) walks both. `pnpm provision cloudflare ` writes the `*/30 * * * *` cron into both tomls. + +## Conflicts to expect + +None — a first release adds files only. + +## Verify + +1. `pnpm plugin check` passes. +2. `pnpm gate` passes, including `src/plugins/freeagent/tests/**`. +3. After connecting, `pnpm cli freeagent status` shows the company and a last-sync time. diff --git a/plugins/freeagent/packages/shared/src/plugins/freeagent/index.ts b/plugins/freeagent/packages/shared/src/plugins/freeagent/index.ts new file mode 100644 index 0000000..bb104c1 --- /dev/null +++ b/plugins/freeagent/packages/shared/src/plugins/freeagent/index.ts @@ -0,0 +1,250 @@ +/** + * `freeagent` — an organisation's FreeAgent books (UK accounting), shared half (D31). + * + * One OAuth connection per organisation. The deployment registers ONE FreeAgent app + * (`FREEAGENT_CLIENT_ID` / `FREEAGENT_CLIENT_SECRET`, sandbox or production by + * `FREEAGENT_ENVIRONMENT`); an admin approves it for their company and the callback stores the + * refresh token sealed. FreeAgent has no webhooks, so a half-hourly cron enqueues `freeagent.sync` + * per connected organisation, which mirrors bank accounts, transactions, explanations, categories, + * contacts, bills and invoices into `freeagent_*` tables (incremental by `updated_since`). Agents + * read the mirror; the two write tools (explain a transaction, create a bill) need approval. After + * each sync every `freeagent:sync-listeners` extension is called, which is how a receipts matcher + * built on top rematches. + * + * Everything it keys carries the id: tables `freeagent_*`, jobs `freeagent.*`, `/api/freeagent`, + * `/api/hooks/freeagent`, query-key roots `freeagent:…`, the CLI command `rocketflare freeagent`. + * + * Money is a decimal STRING everywhere (`"-12.50"`), as FreeAgent sends it and as Postgres + * `numeric` returns it. Never a float. + */ +import { z } from 'zod' +import { paginatedResponse, paginationQuerySchema } from '../../pagination' +import type { SharedPlugin } from '../types' + +export const FREEAGENT_ID = 'freeagent' + +/** The one job: mirror one organisation's books, incrementally. */ +export const FREEAGENT_SYNC_JOB = 'freeagent.sync' + +/** CASL subject: the connection and everything mirrored through it. */ +export const FREEAGENT_SUBJECT = 'FreeAgent' + +/** Query-key roots, and the `entity.changed` entities the server nudges. */ +export const FREEAGENT_CONNECTION_ENTITY = 'freeagent:connection' +export const FREEAGENT_BOOKS_ENTITY = 'freeagent:books' + +/** + * The extension key a listener plugin contributes under (D31 decision 6). In the shared half + * because the shared half is a leaf — a listener reaching for the server entry at module scope + * could find it half-evaluated through the server barrel. + */ +export const FREEAGENT_EXTENSION_KEYS = { syncListeners: `${FREEAGENT_ID}:sync-listeners` } as const + +export function freeagentExtensions(contribution: { + syncListeners: readonly unknown[] +}): Record { + return { [FREEAGENT_EXTENSION_KEYS.syncListeners]: contribution.syncListeners } +} + +/** The API version every request sends: multiple attachments per explanation (`attachments[]`). */ +export const FREEAGENT_API_VERSION = '2026-09-01' + +/** What FreeAgent accepts as an explanation attachment, and how big. */ +export const FREEAGENT_ATTACHMENT_TYPES = [ + 'image/png', + 'image/x-png', + 'image/jpeg', + 'image/jpg', + 'image/gif', + 'application/x-pdf', +] as const +export const FREEAGENT_ATTACHMENT_MAX_BYTES = 5 * 1024 * 1024 + +/** A decimal amount as a string: `-12.5`, `1000.00`. What FreeAgent sends and accepts. */ +export const FREEAGENT_DECIMAL_RE = /^-?\d{1,13}(\.\d{1,4})?$/ +export const decimalStringSchema = z + .string() + .trim() + .regex(FREEAGENT_DECIMAL_RE, 'A decimal amount such as 12.50 or -12.50') +const dateSchema = z.string().regex(/^\d{4}-\d{2}-\d{2}$/, 'A date as YYYY-MM-DD') + +// ---- Config ------------------------------------------------------------------------------------- + +export const FREEAGENT_ENVIRONMENTS = ['sandbox', 'production'] as const +export const freeagentEnvironmentSchema = z.enum(FREEAGENT_ENVIRONMENTS) +export type FreeAgentEnvironment = z.infer + +const optional = z + .string() + .trim() + .optional() + .transform(value => (value ? value : undefined)) + +/** + * The operator's FreeAgent app. Both credentials unset → connecting is off and Settings says so. + * All three are declared as secrets in the manifest so the plugin needs no toml edit; + * `FREEAGENT_ENVIRONMENT` is not sensitive and a host may put it under `[vars]` in BOTH tomls + * instead. Unset, it is `sandbox` — a forgotten var touches nobody's real books. + */ +export const freeagentConfigShape = { + FREEAGENT_CLIENT_ID: optional, + FREEAGENT_CLIENT_SECRET: optional, + FREEAGENT_ENVIRONMENT: z + .string() + .trim() + .optional() + .transform(value => (value ? value : 'sandbox')) + .pipe(freeagentEnvironmentSchema), +} satisfies z.ZodRawShape + +export const freeagentConfigSchema = z.object(freeagentConfigShape) +export type FreeAgentConfig = z.infer + +// ---- Connection --------------------------------------------------------------------------------- + +/** `connected` works; `needs_reconnect` means FreeAgent refused the refresh token (`invalid_grant`). */ +export const FREEAGENT_CONNECTION_STATUSES = ['connected', 'needs_reconnect'] as const +export const freeagentConnectionStatusSchema = z.enum(FREEAGENT_CONNECTION_STATUSES) +export type FreeAgentConnectionStatus = z.infer + +export const freeagentStatusSchema = z.object({ + /** The deployment has an app (client id and secret). */ + configured: z.boolean(), + /** What new connections use. */ + environment: freeagentEnvironmentSchema, + connected: z.boolean(), + status: freeagentConnectionStatusSchema.nullable(), + /** The environment the stored connection was made in — may differ after a config change. */ + connectionEnvironment: freeagentEnvironmentSchema.nullable(), + companyName: z.string().nullable(), + companyCurrency: z.string().nullable(), + connectedAt: z.coerce.date().nullable(), + lastSyncAt: z.coerce.date().nullable(), + lastSyncStartedAt: z.coerce.date().nullable(), + lastError: z.string().nullable(), + lastErrorAt: z.coerce.date().nullable(), + counts: z.object({ + bankAccounts: z.number().int(), + transactions: z.number().int(), + unexplained: z.number().int(), + invoices: z.number().int(), + bills: z.number().int(), + }), +}) +export type FreeAgentStatus = z.infer + +/** `POST /api/freeagent/connect` — where to send the admin's browser. */ +export const freeagentConnectResponseSchema = z.object({ approveUrl: z.string().url() }) +export type FreeAgentConnectResponse = z.infer + +/** `POST /api/freeagent/sync` — the envelope of the queued sync. */ +export const freeagentSyncResponseSchema = z.object({ + jobId: z.string().uuid(), + type: z.literal(FREEAGENT_SYNC_JOB), +}) +export type FreeAgentSyncResponse = z.infer + +// ---- The mirror, as the UI and CLI read it ------------------------------------------------------- + +export const freeagentTransactionSchema = z.object({ + id: z.string().uuid(), + url: z.string(), + bankAccountUrl: z.string(), + bankAccountName: z.string().nullable(), + datedOn: z.string(), + amount: z.string(), + unexplainedAmount: z.string(), + isUnexplained: z.boolean(), + description: z.string().nullable(), +}) +export type FreeAgentTransaction = z.infer + +export const freeagentTransactionListQuerySchema = paginationQuerySchema.extend({ + unexplained: z + .enum(['true', 'false']) + .optional() + .transform(v => (v === undefined ? undefined : v === 'true')), + bankAccountUrl: z.string().url().optional(), +}) +export type FreeAgentTransactionListQuery = z.infer +export const freeagentTransactionListResponseSchema = paginatedResponse(freeagentTransactionSchema) +export type FreeAgentTransactionListResponse = z.infer< + typeof freeagentTransactionListResponseSchema +> + +export const freeagentInvoiceSchema = z.object({ + id: z.string().uuid(), + url: z.string(), + reference: z.string().nullable(), + contactName: z.string().nullable(), + status: z.string().nullable(), + datedOn: z.string().nullable(), + dueOn: z.string().nullable(), + currency: z.string().nullable(), + totalValue: z.string().nullable(), + paidValue: z.string().nullable(), + dueValue: z.string().nullable(), + overdue: z.boolean(), +}) +export type FreeAgentInvoice = z.infer + +export const freeagentInvoiceListQuerySchema = paginationQuerySchema.extend({ + overdue: z + .enum(['true', 'false']) + .optional() + .transform(v => v === 'true'), +}) +export type FreeAgentInvoiceListQuery = z.infer +export const freeagentInvoiceListResponseSchema = paginatedResponse(freeagentInvoiceSchema) +export type FreeAgentInvoiceListResponse = z.infer + +// ---- Writes (the exported service API and the two approval-gated tools) -------------------------- + +export const freeagentExplainInputSchema = z.object({ + bankTransactionUrl: z.string().url(), + categoryUrl: z.string().url(), + /** Signed like the transaction: money out is negative. */ + grossValue: decimalStringSchema, + datedOn: dateSchema, + description: z.string().trim().min(1).max(500), + /** A percentage (`20`), or omitted to let FreeAgent apply the category's automatic rate. */ + salesTaxRate: decimalStringSchema.optional(), + receiptReference: z.string().trim().max(100).optional(), +}) +export type FreeAgentExplainInput = z.infer + +export const freeagentBillItemSchema = z.object({ + categoryUrl: z.string().url(), + description: z.string().trim().min(1).max(500), + /** Including tax. */ + totalValue: decimalStringSchema, + salesTaxRate: decimalStringSchema.optional(), +}) +export const freeagentCreateBillInputSchema = z.object({ + contactUrl: z.string().url(), + reference: z.string().trim().min(1).max(100), + datedOn: dateSchema, + dueOn: dateSchema, + items: z.array(freeagentBillItemSchema).min(1).max(40), +}) +export type FreeAgentCreateBillInput = z.infer + +// ---- Jobs --------------------------------------------------------------------------------------- + +export const freeagentSyncPayloadSchema = z.object({ + tenantId: z.string().uuid(), + /** Who asked, for the audit trail; absent for the cron. */ + requestedBy: z.string().uuid().optional(), +}) +export type FreeAgentSyncPayload = z.infer + +// ---- The plugin --------------------------------------------------------------------------------- + +export const freeagentShared = { + id: FREEAGENT_ID, + label: 'FreeAgent', + version: '3.5.1', + subjects: [FREEAGENT_SUBJECT], + jobs: [z.object({ type: z.literal(FREEAGENT_SYNC_JOB), payload: freeagentSyncPayloadSchema })], + config: freeagentConfigShape, +} as const satisfies SharedPlugin diff --git a/plugins/freeagent/rocketflare-plugin.json b/plugins/freeagent/rocketflare-plugin.json new file mode 100644 index 0000000..a9dbae2 --- /dev/null +++ b/plugins/freeagent/rocketflare-plugin.json @@ -0,0 +1,171 @@ +{ + "id": "freeagent", + "label": "FreeAgent", + "version": "3.5.1", + "repo": "https://github.com/rocketflare-dev/rocketflare-plugins.git", + "minKit": "0.21.0", + "anchor": "apps/web/src/plugins/freeagent/plugin.json", + "paths": [ + "apps/web/src/plugins/freeagent/**", + "packages/shared/src/plugins/freeagent/**", + "apps/cli/src/plugins/freeagent/**", + "docs/plugins/freeagent/**" + ], + "registries": [ + "apps/web/src/plugins/server.ts", + "apps/web/src/plugins/ui.ts", + "apps/web/src/plugins/schema.ts", + "packages/shared/src/plugins/index.ts", + "apps/cli/src/plugins/index.ts" + ], + "requires": { + "surfaces": [], + "plugins": [] + }, + "uses": { + "@/plugins/api": [ + "createRouter", + "cronCtx", + "CronCtx", + "Database", + "defineTool", + "jobCtx", + "JobCtx", + "JobHandler", + "JobInput", + "openSecret", + "PluginConfig", + "publicCtx", + "PublicCtx", + "recordActivity", + "requestCtx", + "RequestCtx", + "ScheduledTask", + "sealSecret", + "ServerPlugin", + "signState", + "Tool", + "toolCtx", + "ToolCtx", + "validate", + "verifyState" + ], + "@/plugins/api/peers": [ + "extensions" + ], + "@/plugins/api/ui-wiring": [ + "NavGuard", + "UiPlugin" + ], + "@/plugins/api/ui": [ + "api", + "ApiError", + "ConfirmModal", + "EmptyState", + "formatDateTime", + "PageHeader", + "PaginationControls", + "SectionPanel", + "SectionPanelSkeleton", + "SettingRow", + "showToast", + "SkeletonRows", + "timeAgo", + "usePermissions" + ], + "@/db/schema/kit": [ + "tenantIsolation", + "tenantRef", + "tenants", + "timestamps", + "users" + ], + "'../api' (apps/cli/src/plugins/api.ts)": [ + "CliPlugin", + "CommandContext", + "formatDate", + "requireClient" + ], + "@testkit/integration": [ + "createExecutionContext", + "createTestEnv", + "createTestSession", + "createTestTenant", + "createTestUser", + "dispatchScheduled", + "json", + "linkUserToTenant", + "request", + "sessionCookieHeader", + "setupTestDatabase", + "stubs", + "TestEnv", + "waitOnExecutionContext" + ], + "@testkit/unit": [ + "makeCronCtx", + "makeJobCtx", + "makeToolCtx" + ], + "@rocketflare/shared/jobs": [ + "JobOf" + ], + "@rocketflare/shared/pagination": [ + "paginatedResponse", + "paginationQuerySchema" + ], + "@rocketflare/shared/plugins/types": [ + "SharedPlugin" + ] + }, + "dependencies": { + "apps/web": {}, + "packages/shared": {}, + "apps/cli": {} + }, + "bindings": [], + "crons": [ + "*/30 * * * *" + ], + "apiPrefixes": [], + "coreEdits": [], + "vars": [ + { + "key": "FREEAGENT_CLIENT_ID", + "secret": true + }, + { + "key": "FREEAGENT_CLIENT_SECRET", + "secret": true + }, + { + "key": "FREEAGENT_ENVIRONMENT", + "example": "sandbox", + "secret": true + } + ], + "workerExports": [], + "skills": [ + "freeagent" + ], + "schema": { + "tables": [ + "freeagent_connections", + "freeagent_sync_state", + "freeagent_bank_accounts", + "freeagent_bank_transactions", + "freeagent_explanations", + "freeagent_categories", + "freeagent_contacts", + "freeagent_bills", + "freeagent_invoices" + ], + "rlsExcluded": [] + }, + "migrations": [ + "freeagent_connections — one row per tenant (unique tenant_id): environment, status, sealed refresh/access tokens, company, sync claim and last outcome; RLS policy", + "freeagent_sync_state — one updated_since cursor per (tenant, resource), unique; RLS policy", + "freeagent_bank_accounts / freeagent_bank_transactions / freeagent_explanations / freeagent_categories / freeagent_contacts / freeagent_bills / freeagent_invoices — the mirror, each unique on (tenant_id, url), money as numeric; RLS policies" + ], + "subdir": "plugins/freeagent" +} diff --git a/plugins/freeagent/skills/freeagent/SKILL.md b/plugins/freeagent/skills/freeagent/SKILL.md new file mode 100644 index 0000000..ca5b708 --- /dev/null +++ b/plugins/freeagent/skills/freeagent/SKILL.md @@ -0,0 +1,128 @@ +--- +name: freeagent +description: Set up, connect, verify and troubleshoot the `freeagent` plugin — registering a FreeAgent app at dev.freeagent.com (sandbox and production), the redirect URI, setting FREEAGENT_CLIENT_ID / FREEAGENT_CLIENT_SECRET / FREEAGENT_ENVIRONMENT locally and deployed, connecting an organisation's FreeAgent company, checking the sync, and decoding errors such as needs reconnecting, rate limited, invalid redirect URI or an empty mirror. Use when someone asks to connect FreeAgent, set up UK accounting/bookkeeping integration, sync bank transactions or invoices from FreeAgent, or asks why FreeAgent stopped syncing. +argument-hint: "[explain | register | secrets | connect | verify | troubleshoot]" +--- + +# /freeagent — FreeAgent, set up end to end + +The `freeagent` plugin connects ONE FreeAgent company per organisation over OAuth, mirrors its +bank accounts, transactions, explanations, categories, contacts, bills and invoices into +`freeagent_*` tables every 30 minutes, gives agents read tools (and two approval-gated write +tools), and exports a service API other plugins (a receipts matcher) build on. Code rules: +`apps/web/src/plugins/freeagent/CLAUDE.md`. + +`$ARGUMENTS` picks the mode. With none: run **Status**, then ask which mode they need. + +## Rules + +- Anything at dev.freeagent.com or in FreeAgent itself is the user's to click. Give the exact path; + never claim you did it. +- **Never** echo, log or paste a client secret, an access token, a refresh token or an approval + code. Secrets go in through stdin (`wrangler secret put`) or `apps/web/.dev.vars`, never a toml, + never a commit, never a chat transcript. +- Sandbox first. A sandbox app and a production app are separate registrations with separate + credentials; tokens from one never work against the other. + +## Status + +1. `pnpm cli freeagent status` — configured?, environment, connection, company, last sync, last + error, mirrored counts. +2. Deployed: `pnpm --filter @rocketflare/web exec wrangler tail` during a sync. Lines that matter: + `freeagent.sync failed` (with a `code`), `freeagent.sync: the connection needs an admin to + reconnect`, `freeagent: completing the connection failed` (with a `code`), + `freeagent.enqueueSyncs` (how many tenants the cron queued). + +## Mode: explain + +In this order: the deployment registers one FreeAgent app; an admin of each organisation clicks +**Connect FreeAgent** in Settings → FreeAgent and approves it in FreeAgent; FreeAgent redirects to +`{APP_URL}/api/hooks/freeagent/callback`, which stores the tokens sealed and queues the first sync; +every 30 minutes the cron queues `freeagent.sync` per connected organisation; the sync is +incremental (`updated_since`), and the first one reaches back about 400 days for transactions; +agents read the mirror and may propose an explanation or a bill, which a person approves. + +## Mode: register + +1. Sign in at **https://dev.freeagent.com** → **My Apps** → **Create New App**. +2. Name it after the app (e.g. "GuideMode"). **OAuth redirect URIs**: add exactly + `{APP_URL}/api/hooks/freeagent/callback` for every origin that will connect — e.g. + `http://localhost:3001/api/hooks/freeagent/callback` for local dev (the Worker's port, not + Vite's), `https://staging.example.com/api/hooks/freeagent/callback`, + `https://app.example.com/api/hooks/freeagent/callback`. The match is exact: scheme, host, port + and path. +3. Copy the **OAuth identifier** (client id) and **OAuth secret** (client secret). They are the + same for sandbox and production API hosts only if FreeAgent says so — treat the sandbox + registration as its own app. +4. **Sandbox**: create a sandbox company at **https://signup.sandbox.freeagent.com** (it is a + separate FreeAgent login), and add some bank transactions there to have something to sync. + Production: the real company's FreeAgent login approves the app at connect time. + +## Mode: secrets + +The plugin reads three keys (all declared as secrets so no toml edit is needed): + +| Key | Value | +|---|---| +| `FREEAGENT_CLIENT_ID` | the OAuth identifier | +| `FREEAGENT_CLIENT_SECRET` | the OAuth secret | +| `FREEAGENT_ENVIRONMENT` | `sandbox` or `production` (unset = `sandbox`). Not sensitive: a host may instead put it under `[vars]` in BOTH tomls | + +**Locally**: add to `apps/web/.dev.vars` (git-ignored) — the user types the values, or you write +the keys and they fill in the values: + +``` +FREEAGENT_CLIENT_ID= +FREEAGENT_CLIENT_SECRET= +FREEAGENT_ENVIRONMENT=sandbox +``` + +Also add the three keys with empty values to `apps/web/.dev.vars.example`. Restart `pnpm dev`. +`OAUTH_ENCRYPTION_KEY` must be set too (the kit's bootstrap writes it) — tokens are sealed with it. + +**Deployed** (per environment, values from stdin so they never reach shell history): + +```bash +pnpm --filter @rocketflare/web exec wrangler secret put FREEAGENT_CLIENT_ID # paste, Enter +pnpm --filter @rocketflare/web exec wrangler secret put FREEAGENT_CLIENT_SECRET +printf production | pnpm --filter @rocketflare/web exec wrangler secret put FREEAGENT_ENVIRONMENT +# staging: add --config wrangler.staging.toml to each +``` + +Or `pnpm provision secrets `, which reads them from the shell or `apps/web/.provision.env`. +The `*/30 * * * *` cron is written into both tomls by `pnpm provision cloudflare `. + +## Mode: connect + +1. Signed in as an owner or admin: **Settings → FreeAgent**. The Environment badge must say what + you expect; if the panel says the deployment has no FreeAgent app, do **secrets** first. +2. **Connect FreeAgent** → FreeAgent's sign-in → approve. FreeAgent sends the browser back to + Settings with "FreeAgent connected. The first sync has started." +3. A **Reconnect** to a different FreeAgent company clears the old mirror first, so two companies' + books are never mixed. + +## Mode: verify + +1. `pnpm cli freeagent status` — `Connection: connected`, the company name, and within a minute or + two a `Last sync` time and non-zero counts. +2. The **FreeAgent** page lists unexplained transactions and recent invoices. +3. `pnpm cli freeagent sync` queues another sync; the last-sync time moves. +4. In chat, ask "what's unexplained in FreeAgent this month?" — the agent should call + `freeagent_search_transactions`. + +## Mode: troubleshoot + +| Symptom | Cause | Fix | +|---|---|---| +| FreeAgent shows "redirect_uri mismatch" / invalid redirect | the callback URL is not registered exactly | add `{APP_URL}/api/hooks/freeagent/callback` (right scheme, host, port) in the app at dev.freeagent.com | +| Callback answers 401 "Invalid or expired FreeAgent approval" | the approval took over 15 minutes, or the link was reused | click Connect again | +| Toast "Completing the connection … failed" | wrong secret, sandbox credentials against production (or vice versa) | check the three keys match ONE registration and environment | +| Status "needs reconnecting" | FreeAgent refused the refresh token (`invalid_grant`): the approval was revoked in FreeAgent, or the environment changed | an admin clicks Reconnect | +| `Last error: FreeAgent rate limit reached` | 120 requests/min or 3600/hour per user | nothing; the queue retries with backoff. A big first sync may take a few runs | +| Connected but no transactions | the first sync looks back ~400 days; older history is not mirrored, or the sandbox company is empty | add transactions in FreeAgent and Sync now | +| Settings warns the connection was made in another environment | `FREEAGENT_ENVIRONMENT` changed after connecting | Reconnect | +| Write tools never offered | the requester is not an admin, or the organisation is not connected | by design; members get read tools only | + +Testing back-off by hand (sandbox only): FreeAgent lowers the limit to 5 requests a minute when a +request carries `X-RateLimit-Test: true`. The plugin never sends it; use curl against the sandbox +with a token if you need to see a 429. diff --git a/plugins/inbox/README.md b/plugins/inbox/README.md new file mode 100644 index 0000000..c23ba11 --- /dev/null +++ b/plugins/inbox/README.md @@ -0,0 +1,75 @@ +# inbox + +> Inbound email for your app: forward receipts, invoices and notices to an address on your domain +> and they arrive parsed, with their attachments, ready for other plugins to act on. + +Mail reaches the Worker through **Cloudflare Email Routing** and the kit's `email` entry. Each +organisation picks a local part (`receipts`). Mail to `receipts@`, or +`receipts+anything@`, is checked against the organisation's sender allow-list and the SPF, DKIM +and DMARC verdicts Cloudflare recorded. Accepted mail is stored raw in R2 and parsed in a queued +job. Other plugins subscribe to parsed messages through the `inbox:consumers` extension. + +## What it adds + +| Area | What you get | +|---|---| +| Email | A catch-all `emailHandlers` entry. Unknown addresses, senders that aren't allowed and failed authentication are bounced with a reason the forwarding mailbox shows | +| UI | **Inbox**: messages, sender and forwarder, authentication verdicts, text body, attachments, raw/HTML downloads, reprocess and delete. **Settings → Inbox**: address, allow-list, require authentication, on/off | +| Forwarding | Gmail and Proton forwarding confirmations are let through before anyone is allow-listed, and their **code and link** are shown at the top of the message | +| Knowledge | Each parsed message (headers + text) and its PDF/Office/HTML/text attachments are indexed into the kit's knowledge base (`inbox:message`, `inbox:attachment`), so chat can search your mail. Upserted on reprocess, removed on delete, switchable in Settings | +| Agent tools | `inbox_list_messages` and `inbox_get_message`, both read-only | +| Extension | `inbox:consumers`: a plugin contributes `{ id, onMessage(message, ctx) }` and is called once per parsed message, with attachment refs it reads through `readInboxAttachment` | +| API | `GET/PUT /api/inbox/settings`, `GET /api/inbox/messages`, `GET /api/inbox/messages/:id`, `…/:id/raw`, `…/:id/html`, `GET /api/inbox/attachments/:id`, `POST …/:id/reprocess`, `DELETE …/:id` | +| Data | `inbox_settings`, `inbox_messages` and `inbox_attachments`, all with row-level security. Bytes live in R2 under `tenants//inbox/` and are purged with the tenant | +| CLI | `rocketflare inbox list [--status]` and `rocketflare inbox status` | +| Skill | `/inbox`: Email Routing setup, Gmail and Proton forwarding, testing, troubleshooting | + +## Requirements + +- Rocketflare kit **0.22.0** or later (the `emailHandlers` slot and `storage()` on background contexts). +- The receiving domain on Cloudflare, with Email Routing enabled and a rule that sends the address to the Worker. +- The kit's `FILES` (R2) and `JOBS_QUEUE` bindings, which every kit app already has, and its knowledge surface (`feature-knowledge`) for indexing. + +## Install + +```bash +pnpm plugin add https://github.com/rocketflare-dev/rocketflare-plugins.git@ --subdir plugins/inbox +pnpm plugin add https://github.com/rocketflare-dev/rocketflare-plugins.git@ --subdir plugins/inbox --apply +pnpm db:generate --name plugin-inbox- && pnpm db:migrate +``` + +Then run `/inbox setup`. + +## Building on it + +```ts +import { inboxExtensions } from '@rocketflare/shared/plugins/inbox/index' +import { type InboxConsumer, readInboxAttachment } from '@/plugins/inbox' + +const receipts: InboxConsumer = { + id: 'receipts', + async onMessage(message, ctx) { + // Quick and idempotent: enqueue your own job, keyed on message.messageId. + await ctx.enqueue({ type: 'receipts.extract', payload: { tenantId: message.tenantId, messageId: message.messageId } }) + }, +} + +export const receiptsServer = { + // … + extensions: inboxExtensions({ consumers: [receipts] }), +} +``` + +Declare `requires: { plugins: ["inbox"] }` in your `plugin.json`. + +## Who sees what + +Every member can read the inbox and its attachments. Owners and admins change the settings, reprocess messages and delete them. + +## Known gaps + +- Bounced messages aren't stored or listed. The bounce reason goes only to the forwarding mailbox and the Worker log. +- One local part per organisation. +- No outbound replies. +- The routing rule is a manual dashboard step; provisioning doesn't create it. +- The HTML body is a download only and is never rendered in the app. diff --git a/plugins/inbox/apps/cli/src/plugins/inbox/index.ts b/plugins/inbox/apps/cli/src/plugins/inbox/index.ts new file mode 100644 index 0000000..3de398b --- /dev/null +++ b/plugins/inbox/apps/cli/src/plugins/inbox/index.ts @@ -0,0 +1,65 @@ +/** + * `rocketflare inbox …` — the plugin's CLI half. + * + * rocketflare inbox list [--status parsed|received|failed] [--json] GET /api/inbox/messages + * rocketflare inbox status [--json] GET /api/inbox/settings + */ +import { + INBOX_ID, + inboxMessageListResponseSchema, + inboxSettingsSchema, + inboxShared, +} from '@rocketflare/shared/plugins/inbox/index' +import type { CliPlugin, CommandContext } from '../api' +import { formatDate, renderTable, requireClient } from '../api' + +export async function runInboxList( + ctx: CommandContext, + options: { status?: string } +): Promise { + const query = options.status ? `?status=${encodeURIComponent(options.status)}` : '' + const { data, raw } = await requireClient(ctx).request('GET', `/api/inbox/messages${query}`, { + schema: inboxMessageListResponseSchema, + }) + ctx.out.data(raw, () => + data.items.length === 0 + ? 'No messages.' + : renderTable(data.items, [ + { header: 'Received', value: m => formatDate(m.receivedAt) }, + { header: 'From', value: m => m.fromAddress ?? m.envelopeFrom }, + { header: 'Subject', value: m => m.subject }, + { header: 'Files', value: m => m.attachmentCount }, + { header: 'Status', value: m => m.status }, + ]) + ) +} + +export async function runInboxStatus(ctx: CommandContext): Promise { + const { data, raw } = await requireClient(ctx).request('GET', '/api/inbox/settings', { + schema: inboxSettingsSchema, + }) + ctx.out.data(raw, () => + [ + `Inbox: ${data.enabled ? 'on' : 'off'}`, + `Address: ${data.localPart ?? 'not set'}`, + `Allowed senders: ${data.allowedSenders.length ? data.allowedSenders.join(', ') : 'none'}`, + `Authentication: ${data.requireAuthentication ? 'required' : 'not required'}`, + ].join('\n') + ) +} + +export const inboxCli: CliPlugin = { + shared: inboxShared, + register(program, action) { + const root = program.command(INBOX_ID).description('inbound email (a plugin)') + root + .command('list', { isDefault: true }) + .description('list received messages, newest first') + .option('--status ', 'received | parsed | failed') + .action(action((ctx, cmd) => runInboxList(ctx, cmd.opts()))) + root + .command('status') + .description('show the inbox address and allow-list') + .action(action(ctx => runInboxStatus(ctx))) + }, +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/CLAUDE.md b/plugins/inbox/apps/web/src/plugins/inbox/CLAUDE.md new file mode 100644 index 0000000..31950c7 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/CLAUDE.md @@ -0,0 +1,31 @@ +# inbox plugin — rules for this directory + +Inbound email for an organisation: Cloudflare Email Routing → the kit's `email` entry → +`emailHandlers` (this plugin claims `*`) → `services/receive.ts` → R2 + `inbox_messages` → +`inbox.parse` → `services/parse.ts` → every `inbox:consumers` extension. + +- **`email()` stays cheap.** `receiveMessage` decides (address → organisation, allow-list, + authentication), stores the raw bytes, writes ONE row and enqueues. No MIME parsing there — + that is `inbox.parse`, which has the queue's retries and its own CPU budget. +- **Refuse before you write.** Every rejection is `message.setReject(reason)` (a permanent bounce + the forwarding mailbox shows its owner) and happens before any R2 or Postgres write. A throw + AFTER the decision is rethrown by the kit as a temporary failure, so the sender retries. +- **The one unscoped query** is `settingsForLocalPart` — a message names an address, not a + tenant. It is in `unscopedAllowlist` with its reason; every other query carries `tenant_id`. +- **R2 keys live under `tenants//inbox//`** (`messagePrefix`), so the kit's + tenant purge removes them and deleting a message is one prefix. +- **Never render a body.** The HTML body and raw `.eml` are served as `attachment` with + `nosniff`; the UI shows the text body only. These are bytes a stranger chose. +- **Parse is idempotent.** Attachments are unique per (message, sha256) and upserted, so a + reprocess keeps attachment ids a consumer may have stored. Consumers must be idempotent too. +- **Consumers** (`InboxConsumer`, exported from `index.ts`) are called for `kind = 'message'` only, + never for forwarding confirmations. A consumer's throw is logged and written to the message's + `error`; it does not stop the others and does not retry the job. +- **Knowledge** (`services/knowledge.ts`): after consumers, a parsed message (headers + text) and + each non-image attachment are ingested with `source: 'inbox:message' | 'inbox:attachment'` and the + row's id as `externalId` — upserts, so a reprocess is safe. Best-effort: failures land on the + message's `error`, never fail the job. Files need an owner: the settings' last editor, else the + oldest owner. Deleting a message calls `forgetMessage` BEFORE the rows go (the ids are the keys). +- **Forwarding confirmations** (Gmail's `forwarding-noreply@google.com`, Proton domains) are + accepted before the allow-list names anyone — that is the only way to read the code — but must + still authenticate. Logic and tables: `services/rules.ts`, `tests/config/rules.test.ts`. diff --git a/plugins/inbox/apps/web/src/plugins/inbox/api/routes.ts b/plugins/inbox/apps/web/src/plugins/inbox/api/routes.ts new file mode 100644 index 0000000..0ccca19 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/api/routes.ts @@ -0,0 +1,258 @@ +/** + * `/api/inbox` — settings, the message list, one message, its raw `.eml`, its HTML body, an + * attachment's bytes, reprocess and delete. + * + * Downloads are always `Content-Disposition: attachment` with `X-Content-Type-Options: nosniff`: + * these are bytes a stranger chose, and rendering them on this origin would hand that stranger a + * page with the user's cookies. The HTML body in particular is served as an attachment, never + * inline. + */ +import { + INBOX_MESSAGE_SUBJECT, + INBOX_MESSAGES_ENTITY, + INBOX_PARSE_JOB, + INBOX_SETTINGS_ENTITY, + INBOX_SETTINGS_SUBJECT, + type InboxAttachment, + type InboxAuth, + type InboxConfirmation, + type InboxMessage, + type InboxMessageListQuery, + type InboxMessageSummary, + inboxMessageListQuerySchema, + type UpdateInboxSettings, + updateInboxSettingsSchema, +} from '@rocketflare/shared/plugins/inbox/index' +import { and, count, desc, eq, type SQL } from 'drizzle-orm' +import type { RequestCtx } from '@/plugins/api' +import { + createRouter, + isUniqueViolation, + recordActivity, + requestCtx, + validate, +} from '@/plugins/api' +import { type InboxMessageRow, inboxAttachments, inboxMessages } from '../db/schema' +import { forgetMessage } from '../services/knowledge' +import { messagePrefix } from '../services/receive' +import { loadSettings, saveSettings, toSettings } from './settings' + +export const inboxRouter = createRouter() + +function toSummary(row: InboxMessageRow): InboxMessageSummary { + return { + id: row.id, + status: row.status as InboxMessageSummary['status'], + kind: row.kind as InboxMessageSummary['kind'], + envelopeFrom: row.envelopeFrom, + fromAddress: row.fromAddress, + fromName: row.fromName, + toAddress: row.toAddress, + subaddress: row.subaddress, + subject: row.subject, + sentAt: row.sentAt, + receivedAt: row.receivedAt, + attachmentCount: row.attachmentCount, + rawSize: row.rawSize, + } +} + +function download(body: ReadableStream, contentType: string, filename: string, size?: number) { + const headers = new Headers({ + 'Content-Type': contentType, + 'Content-Disposition': `attachment; filename="${filename.replace(/"/g, '')}"`, + 'X-Content-Type-Options': 'nosniff', + 'Cache-Control': 'private, no-store', + }) + if (size !== undefined) headers.set('Content-Length', String(size)) + return new Response(body, { headers }) +} + +async function loadMessage(ctx: RequestCtx, id: string): Promise { + const [row] = await ctx.db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, ctx.tenantId), eq(inboxMessages.id, id))) + .limit(1) + return row ?? ctx.notFound('Message not found', 'inbox_message_not_found') +} + +// ---- settings ----------------------------------------------------------------------------------- + +inboxRouter.get('/settings', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', INBOX_SETTINGS_SUBJECT) + return c.json(toSettings(await loadSettings(ctx.db, ctx.tenantId))) +}) + +inboxRouter.put('/settings', validate('json', updateInboxSettingsSchema), async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('manage', INBOX_SETTINGS_SUBJECT) + const body = ctx.valid('json') + const existing = await loadSettings(ctx.db, ctx.tenantId) + const localPart = body.localPart ?? existing?.localPart ?? null + if ((body.enabled ?? existing?.enabled) && !localPart) { + ctx.badRequest('Choose an address before turning the inbox on', 'inbox_local_part_required') + } + let row: Awaited> + try { + row = await saveSettings(ctx.db, ctx.tenantId, ctx.userId, body) + } catch (err) { + if (isUniqueViolation(err)) { + ctx.conflict('That address is already taken', 'inbox_local_part_taken') + } + throw err + } + ctx.defer(() => + recordActivity(ctx.db, { + tenantId: ctx.tenantId, + userId: ctx.userId, + type: 'inbox.settings.updated', + subjectType: INBOX_SETTINGS_SUBJECT, + subjectId: row.id, + metadata: { + enabled: row.enabled, + localPart: row.localPart, + allowedSenders: row.allowedSenders.length, + }, + }) + ) + ctx.nudge(INBOX_SETTINGS_ENTITY) + return c.json(toSettings(row)) +}) + +// ---- messages ----------------------------------------------------------------------------------- + +inboxRouter.get('/messages', validate('query', inboxMessageListQuerySchema), async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', INBOX_MESSAGE_SUBJECT) + const query = ctx.valid('query') + const filters: SQL[] = [eq(inboxMessages.tenantId, ctx.tenantId)] + if (query.status) filters.push(eq(inboxMessages.status, query.status)) + if (query.kind) filters.push(eq(inboxMessages.kind, query.kind)) + const where = and(...filters) + const [rows, totals] = await Promise.all([ + ctx.db + .select() + .from(inboxMessages) + .where(where) + .orderBy(desc(inboxMessages.receivedAt)) + .limit(query.pageSize) + .offset((query.page - 1) * query.pageSize), + ctx.db.select({ total: count() }).from(inboxMessages).where(where), + ]) + return c.json(ctx.page(rows.map(toSummary), totals[0]?.total ?? 0, query)) +}) + +inboxRouter.get('/messages/:id', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', INBOX_MESSAGE_SUBJECT) + const row = await loadMessage(ctx, ctx.uuid('id')) + const attachments = await ctx.db + .select() + .from(inboxAttachments) + .where(and(eq(inboxAttachments.tenantId, ctx.tenantId), eq(inboxAttachments.messageId, row.id))) + .orderBy(inboxAttachments.createdAt) + const body: InboxMessage = { + ...toSummary(row), + messageIdHeader: row.messageIdHeader, + auth: { spf: null, dkim: null, dmarc: null, ...(row.auth as Partial) }, + textBody: row.textBody, + hasHtmlBody: row.htmlKey !== null, + confirmation: (row.confirmation as InboxConfirmation | null) ?? null, + error: row.error, + attachments: attachments.map( + (a): InboxAttachment => ({ + id: a.id, + filename: a.filename, + contentType: a.contentType, + size: a.size, + sha256: a.sha256, + inline: a.inline, + }) + ), + } + return c.json(body) +}) + +inboxRouter.get('/messages/:id/raw', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', INBOX_MESSAGE_SUBJECT) + const row = await loadMessage(ctx, ctx.uuid('id')) + const object = await ctx.storage().get(row.rawKey) + if (!object) ctx.notFound('The raw message is missing', 'inbox_raw_missing') + return download(object.body, 'message/rfc822', `${row.id}.eml`, object.size) +}) + +inboxRouter.get('/messages/:id/html', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', INBOX_MESSAGE_SUBJECT) + const row = await loadMessage(ctx, ctx.uuid('id')) + if (!row.htmlKey) ctx.notFound('This message has no HTML body', 'inbox_html_missing') + const object = await ctx.storage().get(row.htmlKey) + if (!object) ctx.notFound('The HTML body is missing', 'inbox_html_missing') + return download(object.body, 'text/html; charset=utf-8', `${row.id}.html`, object.size) +}) + +inboxRouter.get('/attachments/:id', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('read', INBOX_MESSAGE_SUBJECT) + const id = ctx.uuid('id') + const [row] = await ctx.db + .select() + .from(inboxAttachments) + .where(and(eq(inboxAttachments.tenantId, ctx.tenantId), eq(inboxAttachments.id, id))) + .limit(1) + if (!row) ctx.notFound('Attachment not found', 'inbox_attachment_not_found') + const object = await ctx.storage().get(row.storageKey) + if (!object) ctx.notFound('The attachment is missing', 'inbox_attachment_missing') + return download(object.body, row.contentType, row.filename, object.size) +}) + +inboxRouter.post('/messages/:id/reprocess', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('update', INBOX_MESSAGE_SUBJECT) + const row = await loadMessage(ctx, ctx.uuid('id')) + const job = await ctx.enqueue({ + type: INBOX_PARSE_JOB, + payload: { tenantId: ctx.tenantId, messageId: row.id }, + }) + return c.json({ jobId: job.id, type: INBOX_PARSE_JOB }, 202) +}) + +inboxRouter.delete('/messages/:id', async c => { + const ctx: RequestCtx = requestCtx(c) + ctx.guard('delete', INBOX_MESSAGE_SUBJECT) + const row = await loadMessage(ctx, ctx.uuid('id')) + const attachmentIds = await ctx.db + .select({ id: inboxAttachments.id }) + .from(inboxAttachments) + .where(and(eq(inboxAttachments.tenantId, ctx.tenantId), eq(inboxAttachments.messageId, row.id))) + // Knowledge first: its documents are found by these ids, which the cascade is about to remove. + await forgetMessage( + ctx, + ctx.tenantId, + row.id, + attachmentIds.map(a => a.id) + ) + await ctx.db + .delete(inboxMessages) + .where(and(eq(inboxMessages.tenantId, ctx.tenantId), eq(inboxMessages.id, row.id))) + // The row is the truth; the objects follow. A failure here leaves orphans under the tenant's + // prefix, which the tenant purge still removes. + const storage = ctx.storage() + const objects = await storage.list(`${messagePrefix(ctx.tenantId, row.id)}/`) + if (objects.length > 0) await storage.deleteMany(objects.map(o => o.key)) + ctx.defer(() => + recordActivity(ctx.db, { + tenantId: ctx.tenantId, + userId: ctx.userId, + type: 'inbox.message.deleted', + subjectType: INBOX_MESSAGE_SUBJECT, + subjectId: row.id, + metadata: { fromAddress: row.fromAddress }, + }) + ) + ctx.nudge(INBOX_MESSAGES_ENTITY, row.id) + return c.body(null, 204) +}) diff --git a/plugins/inbox/apps/web/src/plugins/inbox/api/settings.ts b/plugins/inbox/apps/web/src/plugins/inbox/api/settings.ts new file mode 100644 index 0000000..c76a245 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/api/settings.ts @@ -0,0 +1,59 @@ +/** + * `inbox_settings`: read, shape and upsert one organisation's row. The local part is unique across + * tenants (it is how a message finds its organisation), so a taken one is a 409 at the route rather + * than a raw unique violation. + */ +import type { InboxSettings, UpdateInboxSettings } from '@rocketflare/shared/plugins/inbox/index' +import { eq } from 'drizzle-orm' +import type { Database } from '@/plugins/api' +import { type InboxSettingsRow, inboxSettings } from '../db/schema' + +export async function loadSettings( + db: Database, + tenantId: string +): Promise { + const [row] = await db + .select() + .from(inboxSettings) + .where(eq(inboxSettings.tenantId, tenantId)) + .limit(1) + return row +} + +export function toSettings(row: InboxSettingsRow | undefined): InboxSettings { + return { + enabled: row?.enabled ?? false, + localPart: row?.localPart ?? null, + allowedSenders: row?.allowedSenders ?? [], + requireAuthentication: row?.requireAuthentication ?? true, + ingestToKnowledge: row?.ingestToKnowledge ?? true, + updatedAt: row?.updatedAt ?? null, + } +} + +export async function saveSettings( + db: Database, + tenantId: string, + userId: string | null, + body: UpdateInboxSettings +): Promise { + const values = { + ...(body.enabled !== undefined && { enabled: body.enabled }), + ...(body.localPart !== undefined && { localPart: body.localPart }), + ...(body.allowedSenders !== undefined && { + allowedSenders: [...new Set(body.allowedSenders)], + }), + ...(body.requireAuthentication !== undefined && { + requireAuthentication: body.requireAuthentication, + }), + ...(body.ingestToKnowledge !== undefined && { ingestToKnowledge: body.ingestToKnowledge }), + updatedByUserId: userId, + } + const [row] = await db + .insert(inboxSettings) + .values({ tenantId, ...values }) + .onConflictDoUpdate({ target: inboxSettings.tenantId, set: values }) + .returning() + if (!row) throw new Error('inbox_settings: upsert returned no row') + return row +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/db/schema/inbox.ts b/plugins/inbox/apps/web/src/plugins/inbox/db/schema/inbox.ts new file mode 100644 index 0000000..6c84b20 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/db/schema/inbox.ts @@ -0,0 +1,120 @@ +/** + * The inbox plugin's tables (D31). Kit conventions throughout: `tenantRef()` first, `timestamps()`, + * every index led by `tenant_id`, `tenantIsolation()` on each table. + * + * - `inbox_settings` — one row per organisation: the local part it receives on (UNIQUE across + * tenants — it is how an inbound message finds its organisation, so that one lookup is the + * plugin's only unscoped query), the sender allow-list and whether authentication is required. + * - `inbox_messages` — one row per accepted message. The raw RFC 822 bytes live in R2 under + * `tenants//inbox//raw.eml` (purged with the tenant by the kit); the row carries + * what the list and the consumers need. `message_id_header` is unique per tenant when present, + * so a message delivered twice (two forwarding rules, a retried SMTP hand-off) is stored once. + * - `inbox_attachments` — one row per extracted part, cascading with its message, bytes in R2. + */ +import { sql } from 'drizzle-orm' +import { + boolean, + index, + integer, + jsonb, + pgTable, + text, + timestamp, + uniqueIndex, + uuid, +} from 'drizzle-orm/pg-core' +import { tenantIsolation, tenantRef, tenants, timestamps, users } from '../../../../db/schema/kit' + +export const inboxSettings = pgTable( + 'inbox_settings', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + enabled: boolean('enabled').notNull().default(false), + localPart: text('local_part'), + allowedSenders: text('allowed_senders').array().notNull().default(sql`'{}'::text[]`), + requireAuthentication: boolean('require_authentication').notNull().default(true), + /** Index each parsed message and its documents into the knowledge base (chat can search it). */ + ingestToKnowledge: boolean('ingest_to_knowledge').notNull().default(true), + updatedByUserId: uuid('updated_by_user_id').references(() => users.id, { + onDelete: 'set null', + }), + ...timestamps(), + }, + table => [ + uniqueIndex('inbox_settings_tenant_uidx').on(table.tenantId), + uniqueIndex('inbox_settings_local_part_uidx').on(table.localPart), + tenantIsolation('inbox_settings'), + ] +) + +export const inboxMessages = pgTable( + 'inbox_messages', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + status: text('status').notNull().default('received'), + kind: text('kind').notNull().default('message'), + envelopeFrom: text('envelope_from').notNull(), + fromAddress: text('from_address'), + fromName: text('from_name'), + toAddress: text('to_address').notNull(), + subaddress: text('subaddress'), + subject: text('subject'), + messageIdHeader: text('message_id_header'), + sentAt: timestamp('sent_at', { withTimezone: true }), + receivedAt: timestamp('received_at', { withTimezone: true }).notNull().defaultNow(), + rawKey: text('raw_key').notNull(), + rawSize: integer('raw_size').notNull(), + htmlKey: text('html_key'), + textBody: text('text_body'), + /** `{ spf, dkim, dmarc }` from `Authentication-Results`. */ + auth: jsonb('auth').notNull().default({}), + /** `{ provider, code, url }` for a forwarding confirmation, else null. */ + confirmation: jsonb('confirmation'), + attachmentCount: integer('attachment_count').notNull().default(0), + error: text('error'), + parsedAt: timestamp('parsed_at', { withTimezone: true }), + ...timestamps(), + }, + table => [ + index('inbox_messages_tenant_received_idx').on(table.tenantId, table.receivedAt), + index('inbox_messages_tenant_status_idx').on(table.tenantId, table.status), + uniqueIndex('inbox_messages_tenant_message_id_uidx') + .on(table.tenantId, table.messageIdHeader) + .where(sql`${table.messageIdHeader} IS NOT NULL`), + tenantIsolation('inbox_messages'), + ] +) + +export const inboxAttachments = pgTable( + 'inbox_attachments', + { + id: uuid('id').primaryKey().defaultRandom(), + tenantId: tenantRef(tenants), + messageId: uuid('message_id') + .notNull() + .references(() => inboxMessages.id, { onDelete: 'cascade' }), + filename: text('filename').notNull(), + contentType: text('content_type').notNull(), + size: integer('size').notNull(), + sha256: text('sha256').notNull(), + storageKey: text('storage_key').notNull(), + inline: boolean('inline').notNull().default(false), + ...timestamps(), + }, + table => [ + index('inbox_attachments_tenant_message_idx').on(table.tenantId, table.messageId), + uniqueIndex('inbox_attachments_message_sha_uidx').on( + table.tenantId, + table.messageId, + table.sha256 + ), + tenantIsolation('inbox_attachments'), + ] +) + +export type InboxSettingsRow = typeof inboxSettings.$inferSelect +export type InboxMessageRow = typeof inboxMessages.$inferSelect +export type NewInboxMessageRow = typeof inboxMessages.$inferInsert +export type InboxAttachmentRow = typeof inboxAttachments.$inferSelect diff --git a/plugins/inbox/apps/web/src/plugins/inbox/db/schema/index.ts b/plugins/inbox/apps/web/src/plugins/inbox/db/schema/index.ts new file mode 100644 index 0000000..5abec76 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/db/schema/index.ts @@ -0,0 +1,2 @@ +/** What `apps/web/src/plugins/schema.ts` re-exports for this plugin. */ +export * from './inbox' diff --git a/plugins/inbox/apps/web/src/plugins/inbox/index.ts b/plugins/inbox/apps/web/src/plugins/inbox/index.ts new file mode 100644 index 0000000..b6e5128 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/index.ts @@ -0,0 +1,71 @@ +/** + * `inbox` — the server entry (D31). Read top to bottom: the catch-all email handler, the one job, + * the authed mount, two read-only agent tools, and the grants. What it exports beyond the + * `ServerPlugin` is the consumer contract — `InboxConsumer`, `InboxParsedMessage` and the two + * readers — which is how another plugin (a receipts matcher, a ticketing bridge) builds on mail. + * + * The handler claims `*`: Email Routing decides which addresses reach the Worker at all, and this + * plugin decides which ORGANISATION an address belongs to (`inbox_settings.local_part`). A plugin + * that wants a dedicated address of its own claims that local part exactly, which wins over `*`. + */ +import { + INBOX_MESSAGE_SUBJECT, + INBOX_PARSE_JOB, + INBOX_SETTINGS_SUBJECT, + inboxShared, +} from '@rocketflare/shared/plugins/inbox/index' +import type { ServerPlugin, Tool } from '@/plugins/api' +import { EMAIL_CATCH_ALL, emailCtx, toolCtx } from '@/plugins/api' +import { inboxRouter } from './api/routes' +import { handleInboxParse } from './jobs/parse' +import { receiveMessage } from './services/receive' +import { getInboxMessageTool, listInboxMessagesTool } from './tools/inbox-tools' + +export type { InboxAttachmentRef, InboxConsumer, InboxParsedMessage } from './services/parse' +export { readInboxAttachment, readInboxHtml } from './services/parse' + +export const inboxServer = { + shared: inboxShared, + mounts: [['/api/inbox', inboxRouter]], + emailHandlers: [ + { + localPart: EMAIL_CATCH_ALL, + name: 'inbox', + handle: async (message, raw) => { + const ctx = emailCtx(raw) + await receiveMessage(message, ctx, { + authenticationOptional: ctx.config.APP_ENV === 'development', + }) + }, + }, + ], + jobHandlers: { [INBOX_PARSE_JOB]: handleInboxParse }, + agentTools: raw => { + const ctx = toolCtx(raw) + return [listInboxMessagesTool(ctx) as Tool, getInboxMessageTool(ctx) as Tool] + }, + /** Additive, over this plugin's own subjects: admins configure and tidy, members read. */ + grants: { + owner: can => { + can('manage', INBOX_MESSAGE_SUBJECT) + can('manage', INBOX_SETTINGS_SUBJECT) + }, + admin: can => { + can('manage', INBOX_MESSAGE_SUBJECT) + can('manage', INBOX_SETTINGS_SUBJECT) + }, + support: can => { + can('read', INBOX_MESSAGE_SUBJECT) + can('read', INBOX_SETTINGS_SUBJECT) + }, + member: can => { + can('read', INBOX_MESSAGE_SUBJECT) + can('read', INBOX_SETTINGS_SUBJECT) + }, + }, + unscopedAllowlist: { + 'src/plugins/inbox/services/receive.ts': + 'settingsForLocalPart resolves the organisation FROM the recipient address — an inbound ' + + 'message names no tenant, and the unique index on local_part makes the answer one row.', + }, +} satisfies ServerPlugin diff --git a/plugins/inbox/apps/web/src/plugins/inbox/jobs/parse.ts b/plugins/inbox/apps/web/src/plugins/inbox/jobs/parse.ts new file mode 100644 index 0000000..157876e --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/jobs/parse.ts @@ -0,0 +1,15 @@ +/** + * `inbox.parse` — registered in `ServerPlugin.jobHandlers`. The work is `services/parse.ts`; this + * is the adapter. A throw (R2 or Postgres unavailable, a malformed consumer contribution) retries + * with the kit's backoff, which is safe because the parse is idempotent. + */ +import type { JobOf } from '@rocketflare/shared/jobs' +import type { INBOX_PARSE_JOB } from '@rocketflare/shared/plugins/inbox/index' +import type { JobHandler } from '@/plugins/api' +import { jobCtx } from '@/plugins/api' +import { parseMessage } from '../services/parse' + +export const handleInboxParse: JobHandler = ( + job: JobOf, + ctx +) => parseMessage(jobCtx(ctx), job.payload.tenantId, job.payload.messageId) diff --git a/plugins/inbox/apps/web/src/plugins/inbox/plugin.json b/plugins/inbox/apps/web/src/plugins/inbox/plugin.json new file mode 100644 index 0000000..9537dee --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/plugin.json @@ -0,0 +1,118 @@ +{ + "id": "inbox", + "label": "Inbox", + "version": "3.5.1", + "repo": "https://github.com/rocketflare-dev/rocketflare-plugins.git", + "minKit": "0.22.0", + "anchor": "apps/web/src/plugins/inbox/plugin.json", + "paths": [ + "apps/web/src/plugins/inbox/**", + "packages/shared/src/plugins/inbox/**", + "apps/cli/src/plugins/inbox/**", + "docs/plugins/inbox/**" + ], + "registries": [ + "apps/web/src/plugins/server.ts", + "apps/web/src/plugins/ui.ts", + "apps/web/src/plugins/schema.ts", + "packages/shared/src/plugins/index.ts", + "apps/cli/src/plugins/index.ts" + ], + "requires": { + "surfaces": ["feature-knowledge"], + "plugins": [] + }, + "uses": { + "@/plugins/api": [ + "createRouter", + "Database", + "defineTool", + "EMAIL_CATCH_ALL", + "emailCtx", + "EmailCtx", + "emailLocalPart", + "emailSubaddress", + "isUniqueViolation", + "jobCtx", + "JobCtx", + "JobHandler", + "recordActivity", + "requestCtx", + "RequestCtx", + "ServerPlugin", + "Tool", + "toolCtx", + "ToolCtx", + "validate" + ], + "@/plugins/api/peers": ["extensions"], + "@/plugins/api/ui-wiring": ["NavGuard", "UiPlugin"], + "@/plugins/api/ui": [ + "api", + "ApiError", + "ConfirmModal", + "EmptyState", + "formatBytes", + "formatDateTime", + "PageHeader", + "PaginationControls", + "SectionPanel", + "SectionPanelSkeleton", + "SettingInput", + "SettingRow", + "SettingToggle", + "showToast", + "SkeletonRows", + "timeAgo", + "usePermissions" + ], + "@/db/schema/kit": ["tenantIsolation", "tenantRef", "tenants", "timestamps", "users"], + "'../api' (apps/cli/src/plugins/api.ts)": [ + "CliPlugin", + "CommandContext", + "formatDate", + "renderTable", + "requireClient" + ], + "@testkit/integration": [ + "createTestEnv", + "createTestSession", + "createTestTenant", + "createTestUser", + "json", + "linkUserToTenant", + "request", + "sessionCookieHeader", + "setupTestDatabase", + "stubs" + ], + "@testkit/unit": ["makeEmailCtx", "makeJobCtx"], + "@rocketflare/shared/jobs": ["JobOf"], + "@rocketflare/shared/pagination": ["paginatedResponse", "paginationQuerySchema"], + "@rocketflare/shared/plugins/types": ["SharedPlugin"] + }, + "dependencies": { + "apps/web": { + "postal-mime": "^4.1.4" + }, + "packages/shared": {}, + "apps/cli": {} + }, + "bindings": [], + "crons": [], + "apiPrefixes": [], + "coreEdits": [], + "vars": [], + "workerExports": [], + "skills": ["inbox"], + "schema": { + "tables": ["inbox_settings", "inbox_messages", "inbox_attachments"], + "rlsExcluded": [] + }, + "migrations": [ + "inbox_settings \u2014 one row per tenant (unique tenant_id) plus a GLOBALLY unique local_part, the sender allow-list and require_authentication; RLS policy", + "inbox_messages \u2014 one row per accepted message, raw bytes in R2; partial unique (tenant_id, message_id_header); RLS policy", + "inbox_attachments \u2014 one row per kept part, cascading with its message, bytes in R2; unique (tenant_id, message_id, sha256); RLS policy" + ], + "subdir": "plugins/inbox" +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/services/knowledge.ts b/plugins/inbox/apps/web/src/plugins/inbox/services/knowledge.ts new file mode 100644 index 0000000..3078069 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/services/knowledge.ts @@ -0,0 +1,138 @@ +/** + * The inbox's half of the knowledge base (D34): after a parse, the message (headers + text body) + * becomes one document and each PDF/Office/HTML/text attachment another, so chat can answer "what + * did we pay AWS in March?" from mail nobody filed. Upserts by `externalId` (the message id, the + * attachment id), so a reprocess replaces rather than duplicates; deleting a message deletes them. + * + * Best-effort, never fatal to the parse: knowledge is a convenience on top of the inbox, and an + * embeddings outage must not leave a receipt unparsed or a consumer uncalled. Failures come back + * as strings the caller records on the message. Images are not ingested — the kit converts + * documents, not pictures — and forwarding confirmations never are. + * + * Visibility is the organisation's (`tenant`), matching the inbox itself: every member may read + * the inbox, so every member may find its mail in search. + */ +import { INBOX_KNOWLEDGE_SOURCES } from '@rocketflare/shared/plugins/inbox/index' +import { and, asc, eq } from 'drizzle-orm' +import { tenantUsers } from '@/db/schema/kit' +import type { JobCtx, PluginContext } from '@/plugins/api' +import { deleteIngestedDocument, ingestDocument, ingestDocumentFile } from '@/plugins/api' +import type { InboxMessageRow, InboxSettingsRow } from '../db/schema' +import type { InboxAttachmentRef } from './parse' +import { readInboxAttachment } from './parse' + +/** Error codes that mean "this attachment is not a document", not "something broke". */ +const SKIP_CODES = new Set(['unsupported_media_type']) + +function codeOf(err: unknown): string | undefined { + return typeof err === 'object' && err !== null && 'code' in err + ? String((err as { code: unknown }).code) + : undefined +} + +/** + * Who owns an ingested FILE (the kit requires one — a stored original cascades from its user): + * whoever last configured the inbox, else the organisation's longest-standing owner. + */ +export async function documentOwner( + ctx: Pick, + tenantId: string, + settings: Pick | undefined +): Promise { + if (settings?.updatedByUserId) return settings.updatedByUserId + const [owner] = await ctx.db + .select({ userId: tenantUsers.userId }) + .from(tenantUsers) + .where(and(eq(tenantUsers.tenantId, tenantId), eq(tenantUsers.role, 'owner'))) + .orderBy(asc(tenantUsers.joinedAt)) + .limit(1) + return owner?.userId ?? null +} + +/** The searchable text of one message: who, when, what, then the body. */ +export function messageDocumentText(row: InboxMessageRow): string { + const from = row.fromName ? `${row.fromName} <${row.fromAddress}>` : (row.fromAddress ?? '') + return [ + `From: ${from}`, + `Forwarded by: ${row.envelopeFrom}`, + row.sentAt ? `Date: ${row.sentAt.toISOString().slice(0, 10)}` : null, + `Subject: ${row.subject ?? '(no subject)'}`, + '', + row.textBody ?? '', + ] + .filter((line): line is string => line !== null) + .join('\n') +} + +/** Ingest one parsed message and its document attachments. Returns failures, never throws. */ +export async function ingestMessage( + ctx: JobCtx, + row: InboxMessageRow, + attachments: readonly InboxAttachmentRef[], + settings: InboxSettingsRow | undefined +): Promise { + const failures: string[] = [] + const title = row.subject || `Email from ${row.fromAddress ?? row.envelopeFrom}` + try { + await ingestDocument(ctx, { + tenantId: row.tenantId, + userId: null, + source: INBOX_KNOWLEDGE_SOURCES.message, + externalId: row.id, + title, + text: messageDocumentText(row), + }) + } catch (err) { + ctx.logger.warn({ err, messageId: row.id }, 'inbox: knowledge ingest of the message failed') + failures.push(`knowledge: ${err instanceof Error ? err.message : String(err)}`) + } + + const files = attachments.filter(a => !a.contentType.startsWith('image/')) + if (files.length === 0) return failures + const owner = await documentOwner(ctx, row.tenantId, settings) + if (!owner) return [...failures, 'knowledge: no owner to attach documents to'] + for (const attachment of files) { + try { + const file = await readInboxAttachment(ctx, row.tenantId, attachment.id) + if (!file) continue + await ingestDocumentFile(ctx, { + tenantId: row.tenantId, + userId: owner, + source: INBOX_KNOWLEDGE_SOURCES.attachment, + externalId: attachment.id, + title: `${attachment.filename} — ${title}`, + filename: attachment.filename, + file: new Blob([file.bytes], { type: file.contentType }), + }) + } catch (err) { + if (SKIP_CODES.has(codeOf(err) ?? '')) continue + ctx.logger.warn( + { err, attachmentId: attachment.id }, + 'inbox: knowledge ingest of an attachment failed' + ) + failures.push(`knowledge (${attachment.filename}): ${codeOf(err) ?? 'failed'}`) + } + } + return failures +} + +/** Remove a message's documents. Idempotent; a missing document is not an error. */ +export async function forgetMessage( + ctx: PluginContext, + tenantId: string, + messageId: string, + attachmentIds: readonly string[] +): Promise { + await deleteIngestedDocument(ctx, { + tenantId, + source: INBOX_KNOWLEDGE_SOURCES.message, + externalId: messageId, + }) + for (const id of attachmentIds) { + await deleteIngestedDocument(ctx, { + tenantId, + source: INBOX_KNOWLEDGE_SOURCES.attachment, + externalId: id, + }) + } +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/services/parse.ts b/plugins/inbox/apps/web/src/plugins/inbox/services/parse.ts new file mode 100644 index 0000000..ccb40da --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/services/parse.ts @@ -0,0 +1,299 @@ +/** + * `inbox.parse`: the raw message in R2 → bodies, attachments, a confirmation code, and a call to + * every consumer plugin. Safe to re-run (the reprocess route does exactly that): attachments are + * unique per (message, sha256), the row is overwritten, and consumers are required to be + * idempotent too. + */ +import { + INBOX_EXTENSION_KEYS, + INBOX_MESSAGES_ENTITY, + INBOX_TEXT_BODY_MAX, + type InboxConfirmation, +} from '@rocketflare/shared/plugins/inbox/index' +import { and, eq } from 'drizzle-orm' +import PostalMime from 'postal-mime' +import { z } from 'zod' +import type { JobCtx } from '@/plugins/api' +import { extensions } from '@/plugins/api/peers' +import { type InboxMessageRow, inboxAttachments, inboxMessages, inboxSettings } from '../db/schema' +import { ingestMessage } from './knowledge' +import { messagePrefix } from './receive' +import { + extractConfirmation, + extractForwardedOriginal, + isForwardSubject, + keepPart, + safeFilename, +} from './rules' + +/** One stored attachment, as a consumer sees it. Read the bytes with `readInboxAttachment`. */ +export interface InboxAttachmentRef { + id: string + filename: string + contentType: string + size: number + sha256: string + inline: boolean +} + +/** What a consumer is handed for each parsed message. */ +export interface InboxParsedMessage { + tenantId: string + messageId: string + envelopeFrom: string + fromAddress: string | null + fromName: string | null + toAddress: string + subaddress: string | null + subject: string | null + sentAt: Date | null + receivedAt: Date + textBody: string | null + hasHtmlBody: boolean + attachments: readonly InboxAttachmentRef[] +} + +/** + * A plugin that wants inbound mail contributes one of these under `inbox:consumers` + * (`extensions: inboxExtensions({ consumers: [myConsumer] })`). Called for every parsed message of + * kind `message` (never for forwarding confirmations), after its attachments are stored. Keep it + * quick — enqueue your own job — and idempotent: a reprocess calls it again. A throw is logged and + * recorded on the message; it does not stop the other consumers. + */ +export interface InboxConsumer { + id: string + onMessage(message: InboxParsedMessage, ctx: JobCtx): Promise +} + +const consumerSchema = z.object({ + id: z.string().min(1), + onMessage: z.custom(v => typeof v === 'function'), +}) + +/** Every installed consumer, narrowed — a malformed contribution fails loudly, by plugin. */ +export function inboxConsumers(): InboxConsumer[] { + return extensions(INBOX_EXTENSION_KEYS.consumers).map(raw => { + const parsed = consumerSchema.safeParse(raw) + if (!parsed.success) { + throw new Error(`inbox: a contributed consumer is malformed: ${parsed.error.message}`) + } + return parsed.data as InboxConsumer + }) +} + +async function sha256Hex(bytes: ArrayBuffer): Promise { + const digest = await crypto.subtle.digest('SHA-256', bytes) + return [...new Uint8Array(digest)].map(b => b.toString(16).padStart(2, '0')).join('') +} + +function toArrayBuffer(content: ArrayBuffer | Uint8Array | string): ArrayBuffer { + if (typeof content === 'string') return new TextEncoder().encode(content).buffer as ArrayBuffer + if (content instanceof Uint8Array) { + return content.buffer.slice( + content.byteOffset, + content.byteOffset + content.byteLength + ) as ArrayBuffer + } + return content +} + +export function toParsedMessage( + row: InboxMessageRow, + attachments: readonly InboxAttachmentRef[] +): InboxParsedMessage { + return { + tenantId: row.tenantId, + messageId: row.id, + envelopeFrom: row.envelopeFrom, + fromAddress: row.fromAddress, + fromName: row.fromName, + toAddress: row.toAddress, + subaddress: row.subaddress, + subject: row.subject, + sentAt: row.sentAt, + receivedAt: row.receivedAt, + textBody: row.textBody, + hasHtmlBody: row.htmlKey !== null, + attachments, + } +} + +export async function parseMessage( + ctx: JobCtx, + tenantId: string, + messageId: string, + consumers: readonly InboxConsumer[] = inboxConsumers() +): Promise { + const [row] = await ctx.db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + .limit(1) + if (!row) { + // Deleted between receive and parse: nothing to do, and retrying cannot bring it back. + ctx.logger.warn({ tenantId, messageId }, 'inbox.parse: message no longer exists') + return + } + + const storage = ctx.storage() + const object = await storage.get(row.rawKey) + if (!object) { + await ctx.db + .update(inboxMessages) + .set({ status: 'failed', error: 'The raw message is missing from storage' }) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + return + } + const raw = await new Response(object.body).arrayBuffer() + const email = await PostalMime.parse(raw) + + const prefix = messagePrefix(tenantId, messageId) + let htmlKey: string | null = null + if (email.html) { + htmlKey = `${prefix}/body.html` + await storage.put(htmlKey, email.html, { contentType: 'text/html; charset=utf-8' }) + } + + const kept: InboxAttachmentRef[] = [] + for (const [index, part] of (email.attachments ?? []).entries()) { + const bytes = toArrayBuffer(part.content) + const contentType = (part.mimeType || 'application/octet-stream').toLowerCase() + const inline = part.disposition === 'inline' || Boolean(part.related) + if (!keepPart({ inline, contentType, size: bytes.byteLength })) continue + const sha256 = await sha256Hex(bytes) + const filename = safeFilename(part.filename, `attachment-${index + 1}`) + const attachmentId = crypto.randomUUID() + const storageKey = `${prefix}/attachments/${sha256.slice(0, 16)}-${filename}` + await storage.put(storageKey, bytes, { contentType }) + const [saved] = await ctx.db + .insert(inboxAttachments) + .values({ + id: attachmentId, + tenantId, + messageId, + filename, + contentType, + size: bytes.byteLength, + sha256, + storageKey, + inline, + }) + // A reprocess finds the same bytes: keep the existing row (and its id, which a consumer + // may already have recorded) rather than minting a second. + .onConflictDoUpdate({ + target: [inboxAttachments.tenantId, inboxAttachments.messageId, inboxAttachments.sha256], + set: { filename, contentType, storageKey, inline }, + }) + .returning() + if (saved) { + kept.push({ + id: saved.id, + filename: saved.filename, + contentType: saved.contentType, + size: saved.size, + sha256: saved.sha256, + inline: saved.inline, + }) + } + } + + const text = email.text ?? null + let confirmation = row.confirmation as InboxConfirmation | null + if (row.kind === 'confirmation' && confirmation) { + confirmation = extractConfirmation(confirmation.provider, row.subject, text) + } + + // A hand-forwarded message names its real sender in the quoted header block. + let fromAddress = email.from?.address?.toLowerCase() ?? row.fromAddress + let fromName = email.from?.name || row.fromName + if (row.kind === 'message' && isForwardSubject(email.subject ?? row.subject)) { + const original = extractForwardedOriginal(text) + if (original) { + fromAddress = original.fromAddress + fromName = original.fromName ?? fromName + } + } + + const [updated] = await ctx.db + .update(inboxMessages) + .set({ + status: 'parsed', + fromAddress, + fromName, + subject: email.subject ?? row.subject, + sentAt: email.date ? new Date(email.date) : row.sentAt, + textBody: text ? text.slice(0, INBOX_TEXT_BODY_MAX) : null, + htmlKey, + confirmation, + attachmentCount: kept.length, + error: null, + parsedAt: new Date(), + }) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + .returning() + + if (updated && updated.kind === 'message') { + const failures: string[] = [] + const parsed = toParsedMessage(updated, kept) + for (const consumer of consumers) { + try { + await consumer.onMessage(parsed, ctx) + } catch (err) { + ctx.logger.error({ err, consumer: consumer.id, messageId }, 'inbox: consumer failed') + failures.push(`${consumer.id}: ${err instanceof Error ? err.message : String(err)}`) + } + } + const [settings] = await ctx.db + .select() + .from(inboxSettings) + .where(eq(inboxSettings.tenantId, tenantId)) + .limit(1) + if (settings?.ingestToKnowledge ?? true) { + failures.push(...(await ingestMessage(ctx, updated, kept, settings))) + } + if (failures.length > 0) { + await ctx.db + .update(inboxMessages) + .set({ error: failures.join('; ').slice(0, 2000) }) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + } + } + await ctx.nudge(tenantId, INBOX_MESSAGES_ENTITY, messageId) +} + +/** The bytes of one attachment, for a consumer (or the download route). Null when gone. */ +export async function readInboxAttachment( + ctx: Pick, + tenantId: string, + attachmentId: string +): Promise<{ bytes: ArrayBuffer; filename: string; contentType: string } | null> { + const [row] = await ctx.db + .select() + .from(inboxAttachments) + .where(and(eq(inboxAttachments.tenantId, tenantId), eq(inboxAttachments.id, attachmentId))) + .limit(1) + if (!row) return null + const object = await ctx.storage().get(row.storageKey) + if (!object) return null + return { + bytes: await new Response(object.body).arrayBuffer(), + filename: row.filename, + contentType: row.contentType, + } +} + +/** The HTML body of one message, for a consumer that renders it (a receipt with no PDF). */ +export async function readInboxHtml( + ctx: Pick, + tenantId: string, + messageId: string +): Promise { + const [row] = await ctx.db + .select({ htmlKey: inboxMessages.htmlKey }) + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + .limit(1) + if (!row?.htmlKey) return null + const object = await ctx.storage().get(row.htmlKey) + return object ? new Response(object.body).text() : null +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/services/receive.ts b/plugins/inbox/apps/web/src/plugins/inbox/services/receive.ts new file mode 100644 index 0000000..b56abc1 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/services/receive.ts @@ -0,0 +1,168 @@ +/** + * The `email()` half: decide, store, queue — and nothing slower. Parsing a 20 MB PDF-laden message + * inside the Email Worker would spend its CPU budget on a sender who is waiting for an SMTP answer, + * so the raw bytes go to R2 and `inbox.parse` does the rest. + * + * Order matters for what a failure means. Everything that can REFUSE (unknown address, sender, + * authentication) runs before any write and answers `setReject` — a permanent bounce the forwarding + * mailbox will show its owner. Everything after that is storage, and a throw there is rethrown by + * the kit as a temporary failure, so the sender retries rather than the message being lost. + */ +import { + INBOX_MESSAGES_ENTITY, + INBOX_PARSE_JOB, + type InboxConfirmation, +} from '@rocketflare/shared/plugins/inbox/index' +import { and, eq } from 'drizzle-orm' +import type { Database, EmailCtx } from '@/plugins/api' +import { emailLocalPart, emailSubaddress } from '@/plugins/api' +import { type InboxSettingsRow, inboxMessages, inboxSettings } from '../db/schema' +import { acceptMessage, parseAuthenticationResults } from './rules' + +/** The R2 prefix for one message: raw bytes, HTML body and attachments all live beneath it. */ +export function messagePrefix(tenantId: string, messageId: string): string { + return `tenants/${tenantId}/inbox/${messageId}` +} + +/** + * The organisation that receives on this local part. The plugin's one cross-tenant read: a message + * names an address, not a tenant, and the unique index on `local_part` makes the answer one row. + */ +export async function settingsForLocalPart( + db: Database, + localPart: string +): Promise { + const [row] = await db + .select() + .from(inboxSettings) + .where(eq(inboxSettings.localPart, localPart)) + .limit(1) + return row +} + +/** `"Name" ` → `{ address, name }`; null when no address can be read. */ +export function parseAddressHeader( + value: string | null +): { address: string; name: string | null } | null { + if (!value) return null + const angle = /^(.*?)<([^>]+@[^>]+)>/.exec(value) + if (angle?.[2]) { + const name = angle[1]?.trim().replace(/^"|"$/g, '') || null + return { address: angle[2].trim().toLowerCase(), name } + } + const bare = /([^\s<>",]+@[^\s<>",]+)/.exec(value) + return bare?.[1] ? { address: bare[1].toLowerCase(), name: null } : null +} + +/** `Date:` → a Date, or null when absent or unparseable. */ +function headerDate(value: string | null): Date | null { + if (!value) return null + const d = new Date(value) + return Number.isNaN(d.getTime()) ? null : d +} + +/** Why a message was refused, in the words the forwarding mailbox will show its owner. */ +const REJECT_REASONS = { + address: 'Unknown recipient', + disabled: 'This inbox is not accepting mail', + sender: 'Sender not allowed', + authentication: 'Sender could not be authenticated', +} as const + +export type ReceiveOutcome = + | { status: 'rejected'; reason: keyof typeof REJECT_REASONS } + | { status: 'duplicate'; messageId: string } + | { status: 'stored'; messageId: string; tenantId: string } + +/** + * Take one message. Exported so tests drive it with a fake `ForwardableEmailMessage`; the + * registered handler in `index.ts` is a one-line adapter. + */ +export async function receiveMessage( + message: ForwardableEmailMessage, + ctx: EmailCtx, + options: { authenticationOptional?: boolean } = {} +): Promise { + const reject = (reason: keyof typeof REJECT_REASONS): ReceiveOutcome => { + // Addresses and the reason only — never a subject or body in a log line. + ctx.logger.warn({ to: message.to, from: message.from, reason }, 'inbox: message refused') + message.setReject(REJECT_REASONS[reason]) + return { status: 'rejected', reason } + } + + const settings = await settingsForLocalPart(ctx.db, emailLocalPart(message.to)) + if (!settings) return reject('address') + + const headerFrom = parseAddressHeader(message.headers.get('from')) + const auth = parseAuthenticationResults( + [message.headers.get('authentication-results')].filter((h): h is string => Boolean(h)) + ) + const verdict = acceptMessage({ + enabled: settings.enabled, + allowedSenders: settings.allowedSenders, + // Local `wrangler dev` deliveries carry no Authentication-Results at all. + requireAuthentication: settings.requireAuthentication && !options.authenticationOptional, + envelopeFrom: message.from, + headerFrom: headerFrom?.address ?? null, + auth, + }) + if (!verdict.accept) return reject(verdict.reason) + + const tenantId = settings.tenantId + const messageIdHeader = message.headers.get('message-id')?.trim() || null + if (messageIdHeader) { + const [existing] = await ctx.db + .select({ id: inboxMessages.id }) + .from(inboxMessages) + .where( + and( + eq(inboxMessages.tenantId, tenantId), + eq(inboxMessages.messageIdHeader, messageIdHeader) + ) + ) + .limit(1) + if (existing) { + ctx.logger.info({ messageId: existing.id }, 'inbox: duplicate delivery ignored') + return { status: 'duplicate', messageId: existing.id } + } + } + + const id = crypto.randomUUID() + const rawKey = `${messagePrefix(tenantId, id)}/raw.eml` + const raw = await new Response(message.raw).arrayBuffer() + await ctx.storage().put(rawKey, raw, { contentType: 'message/rfc822' }) + + const confirmation: InboxConfirmation | null = verdict.confirmationProvider + ? { provider: verdict.confirmationProvider, code: null, url: null } + : null + const inserted = await ctx.db + .insert(inboxMessages) + .values({ + id, + tenantId, + kind: confirmation ? 'confirmation' : 'message', + envelopeFrom: message.from.toLowerCase(), + fromAddress: headerFrom?.address ?? null, + fromName: headerFrom?.name ?? null, + toAddress: message.to.toLowerCase(), + subaddress: emailSubaddress(message.to), + subject: message.headers.get('subject'), + messageIdHeader, + sentAt: headerDate(message.headers.get('date')), + rawKey, + rawSize: raw.byteLength, + auth, + confirmation, + }) + // Two deliveries racing past the lookup above: the unique index decides, the loser is a no-op. + .onConflictDoNothing() + .returning({ id: inboxMessages.id }) + if (inserted.length === 0) { + await ctx.storage().delete(rawKey) + return { status: 'duplicate', messageId: id } + } + + await ctx.enqueue({ type: INBOX_PARSE_JOB, payload: { tenantId, messageId: id } }) + await ctx.nudge(tenantId, INBOX_MESSAGES_ENTITY, id) + return { status: 'stored', messageId: id, tenantId } +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/services/rules.ts b/plugins/inbox/apps/web/src/plugins/inbox/services/rules.ts new file mode 100644 index 0000000..9105399 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/services/rules.ts @@ -0,0 +1,185 @@ +/** + * The inbox's decisions as pure functions — who may send, what Cloudflare said about them, whether + * a message is a forwarding confirmation, and who REALLY sent a forwarded message. No I/O, so every + * rule is a table test (`tests/config/rules.test.ts`). + */ +import type { InboxAuth, InboxConfirmation } from '@rocketflare/shared/plugins/inbox/index' + +/** `Me+Tag@Example.com` → `me@example.com`: lower-cased, subaddress dropped. */ +export function normaliseAddress(address: string): string { + const trimmed = address.trim().toLowerCase() + const at = trimmed.lastIndexOf('@') + if (at === -1) return trimmed + const local = trimmed.slice(0, at) + const plus = local.indexOf('+') + return `${plus === -1 ? local : local.slice(0, plus)}${trimmed.slice(at)}` +} + +/** `a@b.com` matches `a@b.com` or `@b.com`; `+tags` are ignored on both sides. */ +export function senderMatches(address: string, allowList: readonly string[]): boolean { + const normal = normaliseAddress(address) + const domain = normal.slice(normal.lastIndexOf('@')) + return allowList.some(entry => { + const e = entry.trim().toLowerCase() + return e.startsWith('@') ? domain === e : normaliseAddress(e) === normal + }) +} + +const AUTH_METHODS = ['spf', 'dkim', 'dmarc'] as const + +/** + * The verdicts in `Authentication-Results` (RFC 8601) — the FIRST result per method across every + * such header, which is the one the receiving edge (Cloudflare) prepended. `null` where absent. + */ +export function parseAuthenticationResults(headers: readonly string[]): InboxAuth { + const auth: InboxAuth = { spf: null, dkim: null, dmarc: null } + for (const header of headers) { + for (const method of AUTH_METHODS) { + if (auth[method] !== null) continue + const match = new RegExp(`\\b${method}=([a-z]+)`, 'i').exec(header) + if (match?.[1]) auth[method] = match[1].toLowerCase() + } + } + return auth +} + +/** Any one of SPF, DKIM or DMARC passing. Forwarded mail usually passes SPF for the forwarder. */ +export function isAuthenticated(auth: InboxAuth): boolean { + return auth.spf === 'pass' || auth.dkim === 'pass' || auth.dmarc === 'pass' +} + +/** Senders of forwarding confirmations — accepted before an allow-list exists, so the code shows. */ +const CONFIRMATION_SENDERS: Record = { + 'forwarding-noreply@google.com': 'gmail', + '@proton.me': 'proton', + '@protonmail.com': 'proton', + '@protonmail.ch': 'proton', +} + +/** Which provider a confirmation sender belongs to, or null for everyone else. */ +export function confirmationProvider(envelopeFrom: string): InboxConfirmation['provider'] | null { + for (const [entry, provider] of Object.entries(CONFIRMATION_SENDERS)) { + if (senderMatches(envelopeFrom, [entry])) return provider + } + return null +} + +export type AcceptVerdict = + | { accept: true; confirmationProvider: InboxConfirmation['provider'] | null } + | { accept: false; reason: 'disabled' | 'sender' | 'authentication' } + +/** + * Whether to take a message at all. A forwarding confirmation is accepted from its provider even + * before the allow-list names anybody (that is the only way to see the code), but it still has to + * authenticate — anyone can claim to be Google. Everything else needs an allow-listed envelope + * sender OR `From:` header, and authentication when the organisation requires it. + */ +export function acceptMessage(input: { + enabled: boolean + allowedSenders: readonly string[] + requireAuthentication: boolean + envelopeFrom: string + headerFrom: string | null + auth: InboxAuth +}): AcceptVerdict { + const authenticated = isAuthenticated(input.auth) + const allowed = + senderMatches(input.envelopeFrom, input.allowedSenders) || + (input.headerFrom !== null && senderMatches(input.headerFrom, input.allowedSenders)) + const provider = allowed ? null : confirmationProvider(input.envelopeFrom) + if (provider) { + return authenticated + ? { accept: true, confirmationProvider: provider } + : { accept: false, reason: 'authentication' } + } + if (!input.enabled) return { accept: false, reason: 'disabled' } + if (!allowed) return { accept: false, reason: 'sender' } + if (input.requireAuthentication && !authenticated) { + return { accept: false, reason: 'authentication' } + } + return { accept: true, confirmationProvider: null } +} + +/** + * The code and link in a forwarding-confirmation message. Gmail puts the code in the subject + * (`(#123456789) Gmail Forwarding Confirmation`) and the body (`Confirmation code: 123456789`), + * and the link on `mail-settings.google.com` / `mail.google.com/mail/vf-`; Proton sends a link. + */ +export function extractConfirmation( + provider: InboxConfirmation['provider'], + subject: string | null, + text: string | null +): InboxConfirmation { + const body = text ?? '' + const code = + /confirmation code:\s*(\d{6,})/i.exec(body)?.[1] ?? + /\(#(\d{6,})\)/.exec(subject ?? '')?.[1] ?? + null + const urls = body.match(/https:\/\/[^\s<>"')]+/g) ?? [] + const pattern = + provider === 'gmail' + ? /mail-settings\.google\.com|mail\.google\.com\/mail\/vf-/ + : provider === 'proton' + ? /proton\.me|protonmail\./ + : /./ + const url = urls.find(u => pattern.test(u)) ?? null + return { provider, code, url } +} + +/** Is this the subject line of a hand-forwarded message? */ +export function isForwardSubject(subject: string | null): boolean { + return /^\s*(fwd?|fw)\s*:/i.test(subject ?? '') +} + +export interface ForwardedOriginal { + fromAddress: string + fromName: string | null + subject: string | null +} + +/** + * The original sender of a HAND-forwarded message, read from the quoted header block mail clients + * insert (`---------- Forwarded message ---------` / `Begin forwarded message:` / Outlook's + * `From: … Sent: …`). Automatic forwarding keeps the original `From:` header, so this is only + * consulted when the subject says `Fwd:`. Null when no block can be found. + */ +export function extractForwardedOriginal(text: string | null): ForwardedOriginal | null { + if (!text) return null + const marker = + /-{2,}\s*Forwarded message\s*-{2,}|Begin forwarded message:|-{2,}\s*Original Message\s*-{2,}/i.exec( + text + ) + const block = marker ? text.slice(marker.index, marker.index + 2000) : text.slice(0, 2000) + const from = /^\s*>?\s*From:\s*(.+)$/im.exec(block)?.[1]?.trim() + if (!from) return null + const angle = /^(.*?)<([^>]+@[^>]+)>/.exec(from) + const bare = /([^\s<>"]+@[^\s<>"]+)/.exec(from) + const fromAddress = (angle?.[2] ?? bare?.[1])?.trim().toLowerCase() + if (!fromAddress) return null + const name = angle?.[1]?.trim().replace(/^"|"$/g, '') || null + const subject = /^\s*>?\s*Subject:\s*(.+)$/im.exec(block)?.[1]?.trim() ?? null + return { fromAddress, fromName: name, subject } +} + +/** A filename safe for an R2 key and a `Content-Disposition` header. */ +export function safeFilename(name: string | null | undefined, fallback: string): string { + const cleaned = (name ?? '') + .normalize('NFKD') + .replace(/[^\w.\- ]+/g, '') + .replace(/\s+/g, ' ') + .trim() + .slice(0, 120) + return cleaned || fallback +} + +/** + * Inline parts worth keeping: anything that is not an image, and images large enough not to be a + * logo or a tracking pixel. A receipt sent as an inline image is still kept; a 2 KB footer is not. + */ +export const INLINE_IMAGE_MIN_BYTES = 20_000 + +export function keepPart(part: { inline: boolean; contentType: string; size: number }): boolean { + if (!part.inline) return true + if (!part.contentType.startsWith('image/')) return true + return part.size >= INLINE_IMAGE_MIN_BYTES +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/tests/api/inbox.test.ts b/plugins/inbox/apps/web/src/plugins/inbox/tests/api/inbox.test.ts new file mode 100644 index 0000000..3c8e07e --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/tests/api/inbox.test.ts @@ -0,0 +1,485 @@ +/** + * The inbox end to end on real Postgres and the in-memory R2/queue: settings, the receive + * decisions (address, sender, authentication, confirmations, duplicates), parsing a real MIME + * message into a text body and attachments, the consumer call, the download routes, and tenant + * isolation — another organisation can neither list, read, download nor delete. + */ +import { + createTestEnv, + createTestSession, + createTestTenant, + createTestUser, + json, + linkUserToTenant, + request, + sessionCookieHeader, + setupTestDatabase, + stubs, +} from '@testkit/integration' +import { makeEmailCtx, makeJobCtx } from '@testkit/unit' +import { and, eq } from 'drizzle-orm' +import { beforeAll, describe, expect, it, vi } from 'vitest' +import { allTables } from '@/plugins/api/peers' +import { inboxAttachments, inboxMessages } from '../../db/schema' +import type { InboxConsumer, InboxParsedMessage } from '../../index' +import { parseMessage, readInboxAttachment } from '../../services/parse' +import { receiveMessage } from '../../services/receive' + +const db = setupTestDatabase() +const BASE = '/api/inbox' +const ORIGIN = { Origin: 'http://localhost:3000' } +const AUTH_PASS = 'mx.cloudflare.net; spf=pass smtp.mailfrom=me@gmail.com; dkim=pass; dmarc=pass' + +let tenantId: string +let localPart: string +let ownerCookie: Record +let memberCookie: Record +let otherCookie: Record + +/** A small multipart message: a text body and one PDF attachment. */ +function mime(opts: { + from?: string + subject?: string + messageId?: string + body?: string + pdf?: string +}): string { + const boundary = 'b1_inbox_test' + const pdf = Buffer.from(opts.pdf ?? '%PDF-1.4 receipt').toString('base64') + return [ + `From: ${opts.from ?? 'Stripe '}`, + 'To: me@gmail.com', + `Subject: ${opts.subject ?? 'Your receipt #1234'}`, + `Message-ID: ${opts.messageId ?? `<${crypto.randomUUID()}@stripe.com>`}`, + 'Date: Thu, 23 Oct 2026 10:00:00 +0000', + 'MIME-Version: 1.0', + `Content-Type: multipart/mixed; boundary="${boundary}"`, + '', + `--${boundary}`, + 'Content-Type: text/plain; charset=utf-8', + '', + opts.body ?? 'Thanks for your payment of £12.00.', + `--${boundary}`, + 'Content-Type: application/pdf; name="receipt.pdf"', + 'Content-Disposition: attachment; filename="receipt.pdf"', + 'Content-Transfer-Encoding: base64', + '', + pdf, + `--${boundary}--`, + '', + ].join('\r\n') +} + +/** The slice of `ForwardableEmailMessage` the plugin reads, built from a raw MIME string. */ +function message(raw: string, envelope: { from: string; to: string; auth?: string | null }) { + const headerBlock = raw.split('\r\n\r\n')[0] ?? '' + const headers = new Headers() + for (const line of headerBlock.split('\r\n')) { + const idx = line.indexOf(':') + if (idx > 0) headers.append(line.slice(0, idx).trim(), line.slice(idx + 1).trim()) + } + if (envelope.auth !== null) headers.set('Authentication-Results', envelope.auth ?? AUTH_PASS) + return { + from: envelope.from, + to: envelope.to, + headers, + raw: new Response(raw).body as ReadableStream, + rawSize: raw.length, + canBeForwarded: true, + setReject: vi.fn(), + forward: vi.fn(), + reply: vi.fn(), + } as unknown as ForwardableEmailMessage & { setReject: ReturnType } +} + +beforeAll(async () => { + tenantId = (await createTestTenant(db)).id + const otherTenantId = (await createTestTenant(db)).id + localPart = `receipts-${crypto.randomUUID().slice(0, 8)}` + + const owner = await createTestUser(db) + await linkUserToTenant(db, owner.id, tenantId, 'owner') + ownerCookie = { + ...sessionCookieHeader(await createTestSession(db, owner.id, tenantId)), + ...ORIGIN, + } + + const member = await createTestUser(db) + await linkUserToTenant(db, member.id, tenantId, 'member') + memberCookie = { + ...sessionCookieHeader(await createTestSession(db, member.id, tenantId)), + ...ORIGIN, + } + + const outsider = await createTestUser(db) + await linkUserToTenant(db, outsider.id, otherTenantId, 'owner') + otherCookie = { + ...sessionCookieHeader(await createTestSession(db, outsider.id, otherTenantId)), + ...ORIGIN, + } + + const res = await request( + `${BASE}/settings`, + { method: 'PUT', headers: ownerCookie }, + { json: { enabled: true, localPart, allowedSenders: ['me@gmail.com'] } } + ) + expect(res.status).toBe(200) +}) + +describe('settings', () => { + it('lets members read and only admins write', async () => { + const read = await request(`${BASE}/settings`, { headers: memberCookie }) + expect(await json(read)).toMatchObject({ + enabled: true, + localPart, + requireAuthentication: true, + }) + const write = await request( + `${BASE}/settings`, + { method: 'PUT', headers: memberCookie }, + { json: { enabled: false } } + ) + expect(write.status).toBe(403) + }) + + it('refuses a local part another organisation owns', async () => { + const res = await request( + `${BASE}/settings`, + { method: 'PUT', headers: otherCookie }, + { json: { localPart } } + ) + expect(res.status).toBe(409) + expect(await json(res)).toMatchObject({ code: 'inbox_local_part_taken' }) + }) + + it('refuses turning on without an address', async () => { + const res = await request( + `${BASE}/settings`, + { method: 'PUT', headers: otherCookie }, + { json: { enabled: true } } + ) + expect(res.status).toBe(400) + }) +}) + +describe('receive', () => { + it('rejects an address no organisation owns, before touching storage', async () => { + const env = createTestEnv() + const msg = message(mime({}), { from: 'me@gmail.com', to: 'nobody-here@example.test' }) + const outcome = await receiveMessage(msg, makeEmailCtx({ db, env })) + expect(outcome).toEqual({ status: 'rejected', reason: 'address' }) + expect(msg.setReject).toHaveBeenCalledWith('Unknown recipient') + expect(stubs(env).queue.messages).toHaveLength(0) + }) + + it('rejects a sender not on the allow-list, and one that failed authentication', async () => { + const env = createTestEnv() + const stranger = message(mime({ from: 'x@spam.test' }), { + from: 'x@spam.test', + to: `${localPart}@example.test`, + }) + expect(await receiveMessage(stranger, makeEmailCtx({ db, env }))).toEqual({ + status: 'rejected', + reason: 'sender', + }) + const forged = message(mime({}), { + from: 'me@gmail.com', + to: `${localPart}@example.test`, + auth: 'mx.cloudflare.net; spf=fail; dkim=none; dmarc=fail', + }) + expect(await receiveMessage(forged, makeEmailCtx({ db, env }))).toEqual({ + status: 'rejected', + reason: 'authentication', + }) + }) + + it('stores, queues and parses an allowed message, then calls consumers', async () => { + const env = createTestEnv() + const msg = message(mime({}), { + from: 'me+caf_=x@gmail.com', + to: `${localPart}+gmail@example.test`, + }) + const outcome = await receiveMessage(msg, makeEmailCtx({ db, env })) + expect(outcome.status).toBe('stored') + if (outcome.status !== 'stored') return + expect(msg.setReject).not.toHaveBeenCalled() + expect(stubs(env).queue.messages.map(m => (m.body as { type: string }).type)).toEqual([ + 'inbox.parse', + ]) + + const seen: InboxParsedMessage[] = [] + const consumer: InboxConsumer = { id: 'test', onMessage: async m => void seen.push(m) } + const jobCtx = makeJobCtx({ db, env }) + await parseMessage(jobCtx, tenantId, outcome.messageId, [consumer]) + + const [row] = await db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, outcome.messageId))) + expect(row).toMatchObject({ + status: 'parsed', + kind: 'message', + fromAddress: 'receipts@stripe.com', + subaddress: 'gmail', + subject: 'Your receipt #1234', + attachmentCount: 1, + }) + expect(row?.textBody).toContain('£12.00') + + expect(seen).toHaveLength(1) + expect(seen[0]?.attachments).toEqual([ + expect.objectContaining({ filename: 'receipt.pdf', contentType: 'application/pdf' }), + ]) + const attachmentId = seen[0]?.attachments[0]?.id ?? '' + const file = await readInboxAttachment(jobCtx, tenantId, attachmentId) + expect(new TextDecoder().decode(file?.bytes)).toBe('%PDF-1.4 receipt') + + // A reprocess finds the same bytes and keeps the same attachment row. + await parseMessage(jobCtx, tenantId, outcome.messageId, [consumer]) + const attachments = await db + .select() + .from(inboxAttachments) + .where(eq(inboxAttachments.messageId, outcome.messageId)) + expect(attachments.map(a => a.id)).toEqual([attachmentId]) + expect(seen).toHaveLength(2) + }) + + it('stores a message delivered twice only once', async () => { + const env = createTestEnv() + const raw = mime({ messageId: `` }) + const to = `${localPart}@example.test` + const first = await receiveMessage( + message(raw, { from: 'me@gmail.com', to }), + makeEmailCtx({ db, env }) + ) + const second = await receiveMessage( + message(raw, { from: 'me@gmail.com', to }), + makeEmailCtx({ db, env }) + ) + expect(first.status).toBe('stored') + expect(second).toEqual({ + status: 'duplicate', + messageId: first.status === 'stored' ? first.messageId : '', + }) + }) + + it('records a hand-forwarded message under its original sender', async () => { + const env = createTestEnv() + const raw = mime({ + from: 'Me ', + subject: 'Fwd: Invoice INV-9', + body: 'fyi\n\n---------- Forwarded message ---------\nFrom: AWS \nSubject: Invoice INV-9\n', + }) + const outcome = await receiveMessage( + message(raw, { from: 'me@gmail.com', to: `${localPart}@example.test` }), + makeEmailCtx({ db, env }) + ) + if (outcome.status !== 'stored') throw new Error('not stored') + await parseMessage(makeJobCtx({ db, env }), tenantId, outcome.messageId, []) + const [row] = await db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, outcome.messageId))) + expect(row).toMatchObject({ fromAddress: 'billing@aws.example', fromName: 'AWS' }) + }) + + it('accepts a Gmail forwarding confirmation and surfaces its code', async () => { + const env = createTestEnv() + const raw = [ + 'From: Gmail Team ', + 'Subject: (#812345678) Gmail Forwarding Confirmation - Receive Mail from me@gmail.com', + `Message-ID: <${crypto.randomUUID()}@google.com>`, + 'Content-Type: text/plain; charset=utf-8', + '', + 'Confirmation code: 812345678', + 'https://mail-settings.google.com/mail/vf-abc', + '', + ].join('\r\n') + const outcome = await receiveMessage( + message(raw, { from: 'forwarding-noreply@google.com', to: `${localPart}@example.test` }), + makeEmailCtx({ db, env }) + ) + if (outcome.status !== 'stored') throw new Error('not stored') + const seen: InboxParsedMessage[] = [] + await parseMessage(makeJobCtx({ db, env }), tenantId, outcome.messageId, [ + { id: 'test', onMessage: async m => void seen.push(m) }, + ]) + const res = await request(`${BASE}/messages/${outcome.messageId}`, { headers: memberCookie }) + expect(await json(res)).toMatchObject({ + kind: 'confirmation', + confirmation: { + provider: 'gmail', + code: '812345678', + url: 'https://mail-settings.google.com/mail/vf-abc', + }, + }) + // A confirmation is not mail anybody subscribed to. + expect(seen).toEqual([]) + }) +}) + +describe('knowledge base', () => { + const { documents } = allTables() + + async function docsFor(externalIds: string[]) { + const rows = await db + .select({ source: documents.source, externalId: documents.externalId }) + .from(documents) + .where(eq(documents.tenantId, tenantId)) + return rows.filter(r => r.externalId && externalIds.includes(r.externalId)) + } + + it('ingests the message and its PDF, and forgets both when the message is deleted', async () => { + const env = createTestEnv() + const outcome = await receiveMessage( + message(mime({ subject: 'Knowledge receipt' }), { + from: 'me@gmail.com', + to: `${localPart}@example.test`, + }), + makeEmailCtx({ db, env }) + ) + if (outcome.status !== 'stored') throw new Error('not stored') + await parseMessage(makeJobCtx({ db, env }), tenantId, outcome.messageId, []) + const [att] = await db + .select() + .from(inboxAttachments) + .where( + and( + eq(inboxAttachments.tenantId, tenantId), + eq(inboxAttachments.messageId, outcome.messageId) + ) + ) + const ids = [outcome.messageId, att?.id ?? ''] + expect((await docsFor(ids)).map(d => d.source).sort()).toEqual([ + 'inbox:attachment', + 'inbox:message', + ]) + + // A reprocess upserts: still one document each. + await parseMessage(makeJobCtx({ db, env }), tenantId, outcome.messageId, []) + expect(await docsFor(ids)).toHaveLength(2) + + const res = await request( + `${BASE}/messages/${outcome.messageId}`, + { method: 'DELETE', headers: ownerCookie }, + { env } + ) + expect(res.status).toBe(204) + expect(await docsFor(ids)).toEqual([]) + }) + + it('ingests nothing when the organisation turned it off', async () => { + const off = await request( + `${BASE}/settings`, + { method: 'PUT', headers: ownerCookie }, + { json: { ingestToKnowledge: false } } + ) + expect(await json(off)).toMatchObject({ ingestToKnowledge: false }) + const env = createTestEnv() + const outcome = await receiveMessage( + message(mime({}), { from: 'me@gmail.com', to: `${localPart}@example.test` }), + makeEmailCtx({ db, env }) + ) + if (outcome.status !== 'stored') throw new Error('not stored') + await parseMessage(makeJobCtx({ db, env }), tenantId, outcome.messageId, []) + expect(await docsFor([outcome.messageId])).toEqual([]) + await request( + `${BASE}/settings`, + { method: 'PUT', headers: ownerCookie }, + { json: { ingestToKnowledge: true } } + ) + }) +}) + +describe('routes and tenant isolation', () => { + let messageId: string + let attachmentId: string + const env = createTestEnv() + + beforeAll(async () => { + const outcome = await receiveMessage( + message(mime({ subject: 'Isolation receipt' }), { + from: 'me@gmail.com', + to: `${localPart}@example.test`, + }), + makeEmailCtx({ db, env }) + ) + if (outcome.status !== 'stored') throw new Error('not stored') + messageId = outcome.messageId + await parseMessage(makeJobCtx({ db, env }), tenantId, messageId, []) + const [att] = await db + .select() + .from(inboxAttachments) + .where(eq(inboxAttachments.messageId, messageId)) + attachmentId = att?.id ?? '' + }) + + it('lists and downloads for a member, always as an attachment', async () => { + const list = await request(`${BASE}/messages`, { headers: memberCookie }, { env }) + const body = await json<{ items: { id: string }[] }>(list) + expect(body.items.map(m => m.id)).toContain(messageId) + + const file = await request( + `${BASE}/attachments/${attachmentId}`, + { headers: memberCookie }, + { env } + ) + expect(file.status).toBe(200) + expect(file.headers.get('content-disposition')).toBe('attachment; filename="receipt.pdf"') + expect(file.headers.get('x-content-type-options')).toBe('nosniff') + expect(await file.text()).toBe('%PDF-1.4 receipt') + + const raw = await request( + `${BASE}/messages/${messageId}/raw`, + { headers: memberCookie }, + { env } + ) + expect(raw.headers.get('content-type')).toBe('message/rfc822') + }) + + it('never lets another organisation list, read, download or delete', async () => { + const list = await request(`${BASE}/messages`, { headers: otherCookie }, { env }) + const body = await json<{ items: { id: string }[] }>(list) + expect(body.items.map(m => m.id)).not.toContain(messageId) + for (const path of [ + `${BASE}/messages/${messageId}`, + `${BASE}/messages/${messageId}/raw`, + `${BASE}/attachments/${attachmentId}`, + ]) { + expect((await request(path, { headers: otherCookie }, { env })).status).toBe(404) + } + const deleted = await request( + `${BASE}/messages/${messageId}`, + { method: 'DELETE', headers: otherCookie }, + { env } + ) + expect(deleted.status).toBe(404) + const [row] = await db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + expect(row).toBeDefined() + }) + + it('lets a member read but not delete; an owner deletes the row and its objects', async () => { + const denied = await request( + `${BASE}/messages/${messageId}`, + { method: 'DELETE', headers: memberCookie }, + { env } + ) + expect(denied.status).toBe(403) + const res = await request( + `${BASE}/messages/${messageId}`, + { method: 'DELETE', headers: ownerCookie }, + { env } + ) + expect(res.status).toBe(204) + const [row] = await db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, tenantId), eq(inboxMessages.id, messageId))) + expect(row).toBeUndefined() + const left = await stubs(env).files.list({ prefix: `tenants/${tenantId}/inbox/${messageId}/` }) + expect(left.objects).toHaveLength(0) + }) +}) diff --git a/plugins/inbox/apps/web/src/plugins/inbox/tests/config/rules.test.ts b/plugins/inbox/apps/web/src/plugins/inbox/tests/config/rules.test.ts new file mode 100644 index 0000000..c3ca417 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/tests/config/rules.test.ts @@ -0,0 +1,180 @@ +/** The inbox's pure decisions, as tables (`services/rules.ts`). */ +import { describe, expect, it } from 'vitest' +import { + acceptMessage, + confirmationProvider, + extractConfirmation, + extractForwardedOriginal, + isForwardSubject, + keepPart, + normaliseAddress, + parseAuthenticationResults, + safeFilename, + senderMatches, +} from '../../services/rules' + +const PASS = { spf: 'pass', dkim: null, dmarc: null } +const FAIL = { spf: 'fail', dkim: 'fail', dmarc: 'fail' } + +describe('addresses', () => { + it('normalises case and drops the subaddress', () => { + expect(normaliseAddress(' Me+Caf_=x@Gmail.COM ')).toBe('me@gmail.com') + expect(normaliseAddress('no-at-sign')).toBe('no-at-sign') + }) + + it('matches whole addresses and @domains, ignoring +tags', () => { + expect(senderMatches('me+caf_=receipts=x.dev@gmail.com', ['me@gmail.com'])).toBe(true) + expect(senderMatches('billing@stripe.com', ['@stripe.com'])).toBe(true) + expect(senderMatches('billing@evilstripe.com', ['@stripe.com'])).toBe(false) + expect(senderMatches('you@gmail.com', ['me@gmail.com'])).toBe(false) + }) +}) + +describe('authentication', () => { + it('reads the first verdict per method across headers', () => { + expect( + parseAuthenticationResults([ + 'mx.cloudflare.net; dkim=pass header.d=gmail.com; spf=pass smtp.mailfrom=me@gmail.com; dmarc=pass', + 'other; spf=fail', + ]) + ).toEqual({ spf: 'pass', dkim: 'pass', dmarc: 'pass' }) + expect(parseAuthenticationResults([])).toEqual({ spf: null, dkim: null, dmarc: null }) + }) +}) + +describe('acceptMessage', () => { + const base = { + enabled: true, + allowedSenders: ['me@gmail.com'], + requireAuthentication: true, + envelopeFrom: 'me@gmail.com', + headerFrom: 'receipts@vendor.com', + auth: PASS, + } + + it('accepts an allow-listed, authenticated forwarder', () => { + expect(acceptMessage(base)).toEqual({ accept: true, confirmationProvider: null }) + }) + + it('accepts on the From header when the envelope is not listed', () => { + expect( + acceptMessage({ ...base, envelopeFrom: 'bounce@srs.vendor.com', headerFrom: 'me@gmail.com' }) + ).toEqual({ accept: true, confirmationProvider: null }) + }) + + it('refuses an unlisted sender, a failed authentication and a disabled inbox', () => { + expect(acceptMessage({ ...base, envelopeFrom: 'x@y.com', headerFrom: null })).toEqual({ + accept: false, + reason: 'sender', + }) + expect(acceptMessage({ ...base, auth: FAIL })).toEqual({ + accept: false, + reason: 'authentication', + }) + expect(acceptMessage({ ...base, enabled: false })).toEqual({ + accept: false, + reason: 'disabled', + }) + }) + + it('skips authentication only when the organisation does not require it', () => { + expect(acceptMessage({ ...base, auth: FAIL, requireAuthentication: false }).accept).toBe(true) + }) + + it('lets an authenticated Gmail confirmation through before anyone is allow-listed', () => { + const confirmation = { + ...base, + enabled: false, + allowedSenders: [], + envelopeFrom: 'forwarding-noreply@google.com', + headerFrom: 'forwarding-noreply@google.com', + } + expect(acceptMessage(confirmation)).toEqual({ accept: true, confirmationProvider: 'gmail' }) + expect(acceptMessage({ ...confirmation, auth: FAIL })).toEqual({ + accept: false, + reason: 'authentication', + }) + }) + + it('treats an allow-listed Proton mailbox as a sender, not a confirmation', () => { + expect( + acceptMessage({ ...base, allowedSenders: ['me@proton.me'], envelopeFrom: 'me@proton.me' }) + ).toEqual({ accept: true, confirmationProvider: null }) + expect(confirmationProvider('noreply@proton.me')).toBe('proton') + expect(confirmationProvider('me@gmail.com')).toBeNull() + }) +}) + +describe('extractConfirmation', () => { + it('finds Gmail’s code in the body or the subject, and its link', () => { + const text = + 'me@gmail.com has requested to automatically forward mail to your email address.\n' + + 'Confirmation code: 812345678\n\nTo allow, click: https://mail-settings.google.com/mail/vf-%5BANGjdJ%5D-abc\n' + + 'Other: https://support.google.com/mail' + expect(extractConfirmation('gmail', null, text)).toEqual({ + provider: 'gmail', + code: '812345678', + url: 'https://mail-settings.google.com/mail/vf-%5BANGjdJ%5D-abc', + }) + expect( + extractConfirmation('gmail', '(#998877665) Gmail Forwarding Confirmation', 'no code here') + .code + ).toBe('998877665') + }) + + it('finds Proton’s link', () => { + expect( + extractConfirmation( + 'proton', + 'Confirm forwarding', + 'Open https://account.proton.me/confirm?x=1' + ).url + ).toBe('https://account.proton.me/confirm?x=1') + }) +}) + +describe('forwarded originals', () => { + it('recognises forward subjects', () => { + expect(isForwardSubject('Fwd: Your receipt')).toBe(true) + expect(isForwardSubject('FW: Invoice')).toBe(true) + expect(isForwardSubject('Your receipt')).toBe(false) + }) + + it('reads the Gmail, Apple and Outlook header blocks', () => { + expect( + extractForwardedOriginal( + 'hi\n\n---------- Forwarded message ---------\nFrom: Stripe \nDate: Mon\nSubject: Your receipt #1234\nTo: me@gmail.com\n' + ) + ).toEqual({ + fromAddress: 'receipts@stripe.com', + fromName: 'Stripe', + subject: 'Your receipt #1234', + }) + expect( + extractForwardedOriginal( + 'Begin forwarded message:\n\n> From: "AWS" \n> Subject: Invoice\n' + ) + ).toMatchObject({ fromAddress: 'no-reply@aws.amazon.com', fromName: 'AWS' }) + expect( + extractForwardedOriginal( + '-----Original Message-----\nFrom: billing@vendor.co.uk\nSent: Tuesday\nSubject: Bill\n' + ) + ).toMatchObject({ fromAddress: 'billing@vendor.co.uk', fromName: null }) + expect(extractForwardedOriginal('just a note')).toBeNull() + }) +}) + +describe('parts and filenames', () => { + it('keeps attachments and big inline images, drops logos', () => { + expect(keepPart({ inline: false, contentType: 'image/png', size: 10 })).toBe(true) + expect(keepPart({ inline: true, contentType: 'image/png', size: 2_000 })).toBe(false) + expect(keepPart({ inline: true, contentType: 'image/jpeg', size: 200_000 })).toBe(true) + expect(keepPart({ inline: true, contentType: 'application/pdf', size: 10 })).toBe(true) + }) + + it('makes filenames safe for keys and headers', () => { + expect(safeFilename('Invoice "March"/2026.pdf', 'x')).toBe('Invoice March2026.pdf') + expect(safeFilename('', 'attachment-1')).toBe('attachment-1') + expect(safeFilename(null, 'attachment-2')).toBe('attachment-2') + }) +}) diff --git a/plugins/inbox/apps/web/src/plugins/inbox/tools/inbox-tools.ts b/plugins/inbox/apps/web/src/plugins/inbox/tools/inbox-tools.ts new file mode 100644 index 0000000..ac4defe --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/tools/inbox-tools.ts @@ -0,0 +1,152 @@ +/** + * Two read-only tools for chat and agents: list recent inbound messages, and read one (text body, + * windowed, plus its attachment list). Nothing here writes, so nothing needs approval. Every + * member may read the inbox (`read InboxMessage`), so the tenant predicate is the whole scope. + */ +import { and, desc, eq, ilike, or } from 'drizzle-orm' +import { z } from 'zod' +import type { Tool, ToolCtx } from '@/plugins/api' +import { defineTool } from '@/plugins/api' +import { inboxAttachments, inboxMessages } from '../db/schema' + +export const LIST_INBOX_MESSAGES_TOOL = 'inbox_list_messages' +export const GET_INBOX_MESSAGE_TOOL = 'inbox_get_message' + +const DEFAULT_LIMIT = 20 +const MAX_LIMIT = 100 +const DEFAULT_WINDOW = 8_000 + +export const listInboxMessagesInputSchema = z.object({ + search: z + .string() + .trim() + .max(200) + .optional() + .describe('Match against the sender address, sender name or subject (case-insensitive)'), + limit: z.coerce + .number() + .int() + .min(1) + .max(MAX_LIMIT) + .optional() + .describe(`How many to list (default ${DEFAULT_LIMIT}, max ${MAX_LIMIT}), newest first`), +}) +export type ListInboxMessagesInput = z.infer + +export function listInboxMessagesTool(ctx: ToolCtx): Tool { + return defineTool({ + name: LIST_INBOX_MESSAGES_TOOL, + description: + 'List email received by this workspace’s inbox (forwarded receipts, invoices, notices), ' + + 'newest first: sender, subject, dates and attachment count. Use inbox_get_message to read one.', + schema: listInboxMessagesInputSchema, + async handler(input) { + const filters = [eq(inboxMessages.tenantId, ctx.tenantId)] + if (input.search) { + const like = `%${input.search.replace(/[%_\\]/g, m => `\\${m}`)}%` + const match = or( + ilike(inboxMessages.fromAddress, like), + ilike(inboxMessages.fromName, like), + ilike(inboxMessages.subject, like) + ) + if (match) filters.push(match) + } + const rows = await ctx.db + .select() + .from(inboxMessages) + .where(and(...filters)) + .orderBy(desc(inboxMessages.receivedAt)) + .limit(input.limit ?? DEFAULT_LIMIT) + if (rows.length === 0) { + return JSON.stringify({ + messages: [], + hint: input.search + ? 'No inbound message matches that search. Try a shorter term, or none.' + : 'The inbox has received nothing yet. Say so rather than inventing a message.', + }) + } + return JSON.stringify({ + messages: rows.map(r => ({ + messageId: r.id, + from: r.fromAddress, + fromName: r.fromName, + subject: r.subject, + kind: r.kind, + status: r.status, + sentAt: r.sentAt?.toISOString() ?? null, + receivedAt: r.receivedAt.toISOString(), + attachments: r.attachmentCount, + source: r.subaddress, + })), + }) + }, + }) +} + +export const getInboxMessageInputSchema = z.object({ + messageId: z.string().uuid().describe('The messageId from inbox_list_messages'), + offset: z.coerce.number().int().min(0).optional().describe('Character offset into the body'), + maxChars: z.coerce + .number() + .int() + .min(500) + .max(20_000) + .optional() + .describe(`How much of the body to return (default ${DEFAULT_WINDOW})`), +}) +export type GetInboxMessageInput = z.infer + +export function getInboxMessageTool(ctx: ToolCtx): Tool { + return defineTool({ + name: GET_INBOX_MESSAGE_TOOL, + description: + 'Read one inbound email: headers, its plain-text body (windowed by offset/maxChars) and the ' + + 'names and types of its attachments.', + schema: getInboxMessageInputSchema, + async handler(input) { + const [row] = await ctx.db + .select() + .from(inboxMessages) + .where(and(eq(inboxMessages.tenantId, ctx.tenantId), eq(inboxMessages.id, input.messageId))) + .limit(1) + if (!row) { + return JSON.stringify({ + error: 'not_found', + hint: 'No message with that id in this workspace. List messages with inbox_list_messages.', + }) + } + const attachments = await ctx.db + .select() + .from(inboxAttachments) + .where( + and(eq(inboxAttachments.tenantId, ctx.tenantId), eq(inboxAttachments.messageId, row.id)) + ) + const body = row.textBody ?? '' + const offset = input.offset ?? 0 + const maxChars = input.maxChars ?? DEFAULT_WINDOW + return JSON.stringify({ + messageId: row.id, + from: row.fromAddress, + fromName: row.fromName, + forwardedBy: row.envelopeFrom, + subject: row.subject, + sentAt: row.sentAt?.toISOString() ?? null, + receivedAt: row.receivedAt.toISOString(), + body: body.slice(offset, offset + maxChars), + bodyLength: body.length, + ...(offset + maxChars < body.length && { + hint: `Body continues — call again with offset ${offset + maxChars}.`, + }), + ...(row.status !== 'parsed' && { + note: `This message is ${row.status}; the body may be empty.`, + }), + attachments: attachments.map(a => ({ + attachmentId: a.id, + filename: a.filename, + contentType: a.contentType, + size: a.size, + })), + }) + }, + }) +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/ui/hooks/useInbox.ts b/plugins/inbox/apps/web/src/plugins/inbox/ui/hooks/useInbox.ts new file mode 100644 index 0000000..7cf53f7 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/ui/hooks/useInbox.ts @@ -0,0 +1,64 @@ +import { + type InboxMessageListQuery, + inboxMessageListResponseSchema, + inboxMessageSchema, + inboxReprocessResponseSchema, + inboxSettingsSchema, + type UpdateInboxSettings, +} from '@rocketflare/shared/plugins/inbox/index' +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query' +import { api } from '@/plugins/api/ui' +import { inboxMessageKeys, inboxSettingsKeys } from '../query-keys' + +const BASE = '/api/inbox' + +export function useInboxMessages(query: Partial) { + const params = new URLSearchParams() + for (const [k, v] of Object.entries(query)) if (v !== undefined) params.set(k, String(v)) + return useQuery({ + queryKey: inboxMessageKeys.list(query), + queryFn: () => + api.get(`${BASE}/messages?${params.toString()}`, { schema: inboxMessageListResponseSchema }), + }) +} + +export function useInboxMessage(id: string | null) { + return useQuery({ + queryKey: inboxMessageKeys.detail(id ?? ''), + queryFn: () => api.get(`${BASE}/messages/${id}`, { schema: inboxMessageSchema }), + enabled: Boolean(id), + }) +} + +export function useReprocessInboxMessage() { + const queryClient = useQueryClient() + return useMutation({ + mutationFn: (id: string) => + api.post(`${BASE}/messages/${id}/reprocess`, {}, { schema: inboxReprocessResponseSchema }), + onSuccess: () => queryClient.invalidateQueries({ queryKey: inboxMessageKeys.all }), + }) +} + +export function useDeleteInboxMessage() { + const queryClient = useQueryClient() + return useMutation({ + mutationFn: (id: string) => api.delete(`${BASE}/messages/${id}`), + onSuccess: () => queryClient.invalidateQueries({ queryKey: inboxMessageKeys.all }), + }) +} + +export function useInboxSettings() { + return useQuery({ + queryKey: inboxSettingsKeys.all, + queryFn: () => api.get(`${BASE}/settings`, { schema: inboxSettingsSchema }), + }) +} + +export function useUpdateInboxSettings() { + const queryClient = useQueryClient() + return useMutation({ + mutationFn: (body: UpdateInboxSettings) => + api.put(`${BASE}/settings`, body, { schema: inboxSettingsSchema }), + onSuccess: () => queryClient.invalidateQueries({ queryKey: inboxSettingsKeys.all }), + }) +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/ui/index.ts b/plugins/inbox/apps/web/src/plugins/inbox/ui/index.ts new file mode 100644 index 0000000..7490b69 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/ui/index.ts @@ -0,0 +1,36 @@ +/** + * `inbox` — the UI entry (D31): one page (`/inbox`), one nav item, one settings tab. Pages are + * `lazy()` so none of this rides in the main bundle. + */ +import { EnvelopeIcon } from '@heroicons/react/24/outline' +import { + INBOX_MESSAGE_SUBJECT, + INBOX_SETTINGS_SUBJECT, + inboxShared, +} from '@rocketflare/shared/plugins/inbox/index' +import { createElement, lazy, Suspense } from 'react' +import type { NavGuard, UiPlugin } from '@/plugins/api/ui-wiring' +import { inboxQueryKeys } from './query-keys' + +const InboxPage = lazy(() => import('./pages/InboxPage')) +const InboxSettingsPage = lazy(() => import('./pages/InboxSettings')) + +const readInbox: NavGuard = { action: 'read', subject: INBOX_MESSAGE_SUBJECT } + +export const inboxUi = { + shared: inboxShared, + routes: [{ path: '/inbox', Component: InboxPage, guard: readInbox }], + nav: [{ items: [{ to: '/inbox', label: 'Inbox', icon: EnvelopeIcon, guard: readInbox }] }], + settingsTabs: ({ can }) => + can('read', INBOX_SETTINGS_SUBJECT) + ? [ + { + id: 'inbox', + label: 'Inbox', + icon: createElement(EnvelopeIcon, { className: 'w-4 h-4' }), + content: createElement(Suspense, { fallback: null }, createElement(InboxSettingsPage)), + }, + ] + : [], + queryKeys: inboxQueryKeys, +} satisfies UiPlugin diff --git a/plugins/inbox/apps/web/src/plugins/inbox/ui/pages/InboxPage.tsx b/plugins/inbox/apps/web/src/plugins/inbox/ui/pages/InboxPage.tsx new file mode 100644 index 0000000..6bb2a9b --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/ui/pages/InboxPage.tsx @@ -0,0 +1,271 @@ +/** + * Inbox: what arrived, newest first, and one message beside it — sender, forwarding mailbox, + * authentication verdicts, the text body and the attachments. A forwarding confirmation shows its + * code and link at the top, because that is the only reason anybody opens one. + * + * Bodies are shown as TEXT only. The HTML body and the raw message are downloads (served as + * attachments by the API), never rendered here: they are markup a stranger wrote. + */ +import { EnvelopeIcon, PaperClipIcon } from '@heroicons/react/24/outline' +import type { InboxMessage, InboxMessageSummary } from '@rocketflare/shared/plugins/inbox/index' +import { useState } from 'react' +import { + ConfirmModal, + EmptyState, + formatBytes, + formatDateTime, + PageHeader, + PaginationControls, + SectionPanel, + SkeletonRows, + showToast, + timeAgo, + usePermissions, +} from '@/plugins/api/ui' +import { + useDeleteInboxMessage, + useInboxMessage, + useInboxMessages, + useReprocessInboxMessage, +} from '../hooks/useInbox' + +const STATUS_BADGE: Record = { + received: 'badge-ghost', + parsed: 'badge-success', + failed: 'badge-error', +} + +export default function InboxPage() { + const [page, setPage] = useState(1) + const [selected, setSelected] = useState(null) + const list = useInboxMessages({ page, pageSize: 25 }) + + return ( +
+ +
+ + {list.isPending ? ( +
+ +
+ ) : list.isError || !list.data ? ( +

The inbox could not be loaded.

+ ) : list.data.items.length === 0 ? ( + + ) : ( + <> +
    + {list.data.items.map(m => ( +
  • + +
    • + ))} +
+ + + )} + + {selected ? ( + setSelected(null)} /> + ) : ( + +

Choose a message to read it.

+ + )} +
+
+ ) +} + +function MessageDetail({ id, onDeleted }: { id: string; onDeleted: () => void }) { + const message = useInboxMessage(id) + if (message.isPending) { + return ( + + + + ) + } + if (message.isError || !message.data) { + return ( + +

This message could not be loaded.

+ + ) + } + return +} + +function Verdict({ label, value }: { label: string; value: string | null }) { + const tone = value === 'pass' ? 'badge-success' : value ? 'badge-warning' : 'badge-ghost' + return {`${label} ${value ?? '—'}`} +} + +function MessageBody({ message, onDeleted }: { message: InboxMessage; onDeleted: () => void }) { + const { can } = usePermissions() + const reprocess = useReprocessInboxMessage() + const remove = useDeleteInboxMessage() + const [confirmDelete, setConfirmDelete] = useState(false) + + return ( + + + Download .eml + + {message.hasHtmlBody && ( + + Download HTML + + )} + {can('update', 'InboxMessage') && ( + + )} + {can('delete', 'InboxMessage') && ( + + )} + + } + > + {message.confirmation && ( +
+
+

+ Forwarding confirmation + {message.confirmation.provider !== 'other' && + ` from ${message.confirmation.provider}`} +

+ {message.confirmation.code && ( +

+ Code: {message.confirmation.code} +

+ )} + {message.confirmation.url && ( + + Open the confirmation link + + )} + {!message.confirmation.code && !message.confirmation.url && ( +

No code or link was found — read the body below.

+ )} +
+
+ )} +
+
From
+
+ {message.fromName ? `${message.fromName} <${message.fromAddress}>` : message.fromAddress} +
+
Forwarded by
+
{message.envelopeFrom}
+
To
+
{message.toAddress}
+
Sent
+
{message.sentAt ? formatDateTime(message.sentAt) : '—'}
+
Received
+
{formatDateTime(message.receivedAt)}
+
Checks
+
+ + + +
+
+ {message.error &&

{message.error}

} + {message.attachments.length > 0 && ( +
    + {message.attachments.map(a => ( +
  • + + + {a.filename} + + + {a.contentType} · {formatBytes(a.size)} + +
    • + ))} +
+ )} + 
+        {message.textBody ?? (message.status === 'parsed' ? '(no text body)' : 'Not parsed yet.')}
+
+ setConfirmDelete(false)} + onConfirm={() => + remove.mutate(message.id, { + onSuccess: () => { + setConfirmDelete(false) + onDeleted() + }, + }) + } + /> + + ) +} diff --git a/plugins/inbox/apps/web/src/plugins/inbox/ui/pages/InboxSettings.tsx b/plugins/inbox/apps/web/src/plugins/inbox/ui/pages/InboxSettings.tsx new file mode 100644 index 0000000..4fb0ec8 --- /dev/null +++ b/plugins/inbox/apps/web/src/plugins/inbox/ui/pages/InboxSettings.tsx @@ -0,0 +1,156 @@ +/** + * Settings → Inbox: the address this organisation receives on, who may send to it, and whether + * mail must authenticate. Read-only without `manage InboxSettings`. + * + * The domain half of the address is whatever Cloudflare Email Routing routes to this Worker, which + * the app cannot know — the page shows this site's host as the likely one and says so. + */ +import { + INBOX_LOCAL_PART_RE, + INBOX_SENDER_RE, + type InboxSettings as Settings, +} from '@rocketflare/shared/plugins/inbox/index' +import { useState } from 'react' +import { + ApiError, + SectionPanel, + SectionPanelSkeleton, + SettingInput, + SettingRow, + SettingToggle, + showToast, + usePermissions, +} from '@/plugins/api/ui' +import { useInboxSettings, useUpdateInboxSettings } from '../hooks/useInbox' + +export default function InboxSettingsPage() { + const settings = useInboxSettings() + if (settings.isPending) return + if (settings.isError || !settings.data) { + return ( + +

The inbox settings could not be loaded.

+ + ) + } + return +} + +function parseSenders(text: string): string[] { + return text + .split(/[\s,;]+/) + .map(s => s.trim().toLowerCase()) + .filter(Boolean) +} + +function SettingsForm({ settings }: { settings: Settings }) { + const { can } = usePermissions() + const canManage = can('manage', 'InboxSettings') + const update = useUpdateInboxSettings() + const [enabled, setEnabled] = useState(settings.enabled) + const [localPart, setLocalPart] = useState(settings.localPart ?? '') + const [senders, setSenders] = useState(settings.allowedSenders.join('\n')) + const [requireAuth, setRequireAuth] = useState(settings.requireAuthentication) + const [ingest, setIngest] = useState(settings.ingestToKnowledge) + + const parsedSenders = parseSenders(senders) + const badSender = parsedSenders.find(s => !INBOX_SENDER_RE.test(s)) + const localPartError = + localPart && !INBOX_LOCAL_PART_RE.test(localPart) + ? 'Lower-case letters, digits, dots, dashes or underscores' + : undefined + const host = typeof window === 'undefined' ? 'your-domain' : window.location.hostname + + const save = () => + update.mutate( + { + enabled, + ...(localPart && { localPart }), + allowedSenders: parsedSenders, + requireAuthentication: requireAuth, + ingestToKnowledge: ingest, + }, + { + onSuccess: () => showToast('Inbox settings saved', 'success'), + onError: err => + showToast(err instanceof ApiError ? err.message : 'Could not save the settings', 'error'), + } + ) + + return ( + + setLocalPart(v.trim().toLowerCase())} + placeholder="receipts" + disabled={!canManage} + error={localPartError} + /> + +
+