From 144cac52fb4a81f468e80c6297547af182d0eac5 Mon Sep 17 00:00:00 2001 From: apecode Date: Tue, 15 Sep 2026 10:46:02 +0800 Subject: [PATCH] =?UTF-8?q?=E8=BF=81=E7=A7=BB=E5=88=B0=20SQLite=20?= =?UTF-8?q?=E4=BA=8B=E5=AE=9E=E6=BA=90=E5=B9=B6=E5=8A=A0=E5=85=A5=E5=8E=9F?= =?UTF-8?q?=E7=94=9F=E5=A4=87=E4=BB=BD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 本次变更： - 将 SQLite 明确为正常运行时的本地事实源 - 新增 backup create / verify 与完整性检查 - doctor 改为只读检查数据库，init 防止 demo 覆盖已有组合 - 将 ledger 降级为显式、有损的旧版互操作导出/导入 - 同步更新 CLI、README、中文文档、Skill 和合成 demo 验证： - pnpm -r test（63 core + 4 connector-yzyx + 11 CLI tests passed） - pnpm typecheck - pnpm -r build - CI skill-drift check - git diff --check - 隔离临时数据库上的 init / doctor / backup create / backup verify smoke test 范围： - 单用户、本地 MVP；不引入多用户、分布式事务或高强度安全模型 - 未包含个人账户、九坤资料、内部 endpoint 或真实数据 --- CHANGELOG.md | 17 ++ README.md | 74 ++--- README.zh-CN.md | 61 ++-- SECURITY.md | 17 +- examples/README.md | 37 ++- package.json | 5 +- packages/cli/README.md | 25 +- packages/cli/package.json | 5 +- packages/cli/src/__tests__/backup-cli.test.ts | 75 +++++ packages/cli/src/__tests__/doctor.test.ts | 120 ++++++++ packages/cli/src/__tests__/init.test.ts | 103 +++++++ packages/cli/src/__tests__/ledger.test.ts | 112 ++++++++ packages/cli/src/commands/backup.ts | 78 +++++ packages/cli/src/commands/doctor.ts | 272 +++++++++--------- packages/cli/src/commands/init.ts | 173 ++++++----- packages/cli/src/commands/ledger.ts | 145 ++++++---- packages/cli/src/index.ts | 4 +- packages/cli/src/utils/config.ts | 4 +- packages/core/package.json | 2 +- packages/core/src/__tests__/backup.test.ts | 129 +++++++++ packages/core/src/db/backup.ts | 212 ++++++++++++++ packages/core/src/db/connection.ts | 53 +++- packages/core/src/db/schema.ts | 2 +- packages/core/src/index.ts | 1 + packages/core/src/ledger/read.ts | 4 +- packages/core/src/ledger/sync.ts | 7 +- packages/core/src/ledger/types.ts | 9 +- packages/core/src/ledger/write.ts | 39 +-- scripts/demo.mjs | 13 +- skills/finsight/SKILL.md | 67 +++-- 30 files changed, 1425 insertions(+), 440 deletions(-) create mode 100644 packages/cli/src/__tests__/backup-cli.test.ts create mode 100644 packages/cli/src/__tests__/doctor.test.ts create mode 100644 packages/cli/src/__tests__/init.test.ts create mode 100644 packages/cli/src/__tests__/ledger.test.ts create mode 100644 packages/cli/src/commands/backup.ts create mode 100644 packages/core/src/__tests__/backup.test.ts create mode 100644 packages/core/src/db/backup.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 214a978..d8c9734 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). We use [SemVer](https://semver.org/) for versioning — `0.x` releases batch related changes, breaking changes can land in `0.x → 0.y`. +## [Unreleased] + +### Changed + +- SQLite is now documented and treated as FinSight's authoritative local data + store during normal CLI and web operation. +- The legacy plain-text ledger is explicit interoperability only; it is not + automatically synchronized and is not the canonical recovery format. +- `doctor` checks SQLite integrity and reads the database without migrations or + writes. `init` refuses to load demo data into a populated portfolio database. + +### Added + +- `finsight backup create` creates an integrity-checked native SQLite backup + with SHA-256 metadata and restrictive local file permissions. +- `finsight backup verify ` verifies an existing native SQLite backup. + ## [0.1.0] — 2026-06-02 — Initial public release The first public version of FinSight. A local-first, AI-friendly personal diff --git a/README.md b/README.md index b2847d4..ff6d851 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ **See your money. Know where it sits. Decide what's next.** A personal portfolio tracker that runs on your laptop, plays well with AI tools, -and keeps your data in plain text files you can read and edit yourself. +and keeps your data in a local SQLite database you can back up and inspect. [![CI](https://github.com/ApeCodeAI/finsight/actions/workflows/ci.yml/badge.svg)](https://github.com/ApeCodeAI/finsight/actions/workflows/ci.yml) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](./LICENSE) @@ -38,9 +38,10 @@ rebalance?", you're stuck taking screenshots. FinSight takes the other path: -- 🗂 **Your files are the truth.** Holdings live in plain text on your disk - (YAML — readable by humans). Back them up with `git`, edit them in any editor, - open them on any machine. The app's database is just a working copy. +- 🗂 **SQLite is the truth.** Holdings, transactions, snapshots, and decisions + are stored locally in `~/.finsight/data/finsight.db`. Create a native, + integrity-checked backup with `finsight backup create` whenever you want a + recovery point. - 🌍 **You pick the base currency.** USD, CNY, JPY, EUR — pick one, everything else gets converted automatically. No region-locked product here. - 🤖 **AI is a first-class user.** Every command can output structured JSON, and @@ -49,14 +50,14 @@ FinSight takes the other path: - 🔒 **Stays on your machine.** No cloud account, no signup, no telemetry. The dashboard runs at `localhost`. -> A good portfolio tracker should be a file format first, a command-line tool -> second, and a dashboard third — in that order. +> A good portfolio tracker should be local-first, command-line friendly, and +> easy for an AI agent to inspect and operate. ## ⚡ Install ```bash npm install -g finsight -finsight init # asks for base currency, locale, where to store your files +finsight init # asks for base currency and locale finsight overview # see your portfolio ``` @@ -122,12 +123,12 @@ $ finsight overview --json | jq | Decision | Most trackers | FinSight | |---|---|---| -| Where data lives | Their cloud | Your laptop, in plain text files | -| Source of truth | Their database | Your text files; the app's DB is a copy | +| Where data lives | Their cloud | Your laptop, in a local SQLite database | +| Source of truth | Their database | Your local SQLite database | | AI access | Maybe a chat box | A documented JSON output any AI can read | | Base currency | Hard-coded | `finsight config set base-currency JPY` | -| Schema changes | Their migration | You edit the file | -| Backup | Vendor lock-in | `git commit` your folder | +| Schema changes | Their migration | Local SQLite migrations | +| Backup | Vendor lock-in | `finsight backup create` | | Account model | Sign up first | None — runs as you, on your machine | ## 🎯 What it does and doesn't do @@ -154,7 +155,7 @@ For budgeting, you can pair FinSight with: - [**Actual**](https://actualbudget.org/) — local-first envelope budgeting (open source) - [**YNAB**](https://www.ynab.com/) / [**Lunch Money**](https://lunchmoney.app/) — commercial, polished -Since FinSight uses plain text files, running both side by side is easy. +Since FinSight runs locally and exposes JSON, running both side by side is easy. ## 🤖 Let an AI agent drive it @@ -195,18 +196,14 @@ wire it into your specific assistant. └─────────┬───────────┴─────────┬───────────┘ │ │ ┌────▼─────┐ ┌─────▼─────┐ - │ Hono API │ ◄─────► │ SQLite │ ← working copy - └────┬─────┘ └─────▲─────┘ + │ Hono API │ ◄─────► │ SQLite │ ← authoritative local store + └────┬─────┘ └─────┬─────┘ │ │ - │ finsight ledger sync (once a day) + │ backup create / verify │ ▼ │ ┌───────────────┐ - └────────────►│ your folder │ ← source of truth - │ accounts.yaml│ (git-tracked) - │ fx-rates.yaml│ - │ transactions │ - │ snapshots/ │ - │ decisions/ │ + └────────────►│ native backup │ ← recovery copy + │ *.sqlite3 │ └───────────────┘ ``` @@ -224,7 +221,9 @@ finsight config set ledger-dir ~/notes/finance/ledger ``` Your settings live in `~/.finsight/config.json`. The `FINSIGHT_DB_PATH` -environment variable can override where the SQLite working copy lives. +environment variable can override where the authoritative SQLite database lives. +The optional `ledger-dir` setting is only for explicit legacy export/import +interoperability; it is not required for normal use. ### 🔒 Note on security @@ -278,11 +277,15 @@ finsight quote update --dry-run # preview, don't write ```bash finsight reconcile # compare to what your broker app shows finsight reconcile log # past cross-check results -finsight ledger sync # save today's database snapshot into your folder -finsight ledger verify # check that the DB and your files match -finsight ledger restore --yes # rebuild the DB from your files (disaster recovery) +finsight backup create --json # create a verified native SQLite backup +finsight backup verify --json # verify an existing native backup +finsight doctor --json # inspect DB integrity and portfolio health ``` +`ledger sync`, `ledger verify`, and `ledger restore` are retained only as +explicit legacy interoperability commands. Ledger export is lossy and is not +the canonical recovery path. + **Hand the portfolio to an LLM** ```bash finsight context | pbcopy # Markdown briefing → clipboard @@ -338,18 +341,19 @@ for the full guide. ## 🗃 What lives in your folder ``` -ledger/ -├── README.md (auto-generated) -├── accounts.yaml — your accounts + holdings (you can edit this directly) -├── fx-rates.yaml — exchange rates between currencies -├── transactions.jsonl — every event, appended one per line -├── snapshots/ — daily JSON snapshots -└── decisions/ — markdown notes per investment decision +legacy-ledger/ # only if you explicitly configure ledger-dir +├── README.md (auto-generated) +├── accounts.yaml — exported account + holding state +├── fx-rates.jsonl — exported dated exchange rates +├── transactions.jsonl — exported transactions +├── snapshots.jsonl — exported snapshots +├── reconciliations.jsonl — exported reconciliation rows +└── decisions/ — exported decision notes ``` -This layout is deliberately simple. Paste `accounts.yaml` into any AI and -ask for advice; commit the diff after a buy; restore the database from any -point in your git history. +This directory is an optional compatibility format. It is generated only by +an explicit ledger command and can be useful for inspection or interoperability, +but it is not a native SQLite backup and is not imported automatically. ## 🚧 Status diff --git a/README.zh-CN.md b/README.zh-CN.md index fdd7fa8..e5b78ae 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -36,18 +36,20 @@ Built by **[ApeCode.ai](https://apecode.ai)** · Sponsored by **[BytePass.ai](ht FinSight 走另一条路： -- 🗂 **你的文件就是真相。** 持仓存在你硬盘上的纯文本里（YAML，人能读懂），用 `git` 备份，任何编辑器都能改，任何机器都能打开。App 里那个数据库只是工作副本。 +- 🗂 **SQLite 是事实源。** 持仓、交易、快照和决策都保存在本机 + `~/.finsight/data/finsight.db`。需要恢复点时，运行 + `finsight backup create` 创建经过完整性检查的原生备份。 - 🌍 **基准币你说了算。** USD / CNY / JPY / EUR，挑一个，其他币种自动换算到这个。没有地区锁定。 - 🤖 **AI 是一等用户。** 每个命令都能输出结构化 JSON；仓库里有一个 markdown 文件（叫 [skill](./skills/finsight/SKILL.md)），任何 AI 助手（Claude / Cursor / Codex / ChatGPT）读完就知道怎么帮你操作。 - 🔒 **数据不出本机。** 无云端、无注册、无埋点。Dashboard 默认只跑在 `localhost`。 -> 一个好的投资追踪工具，应该先是一个文件格式，然后是命令行，最后才是 Dashboard。 +> 一个好的投资追踪工具，应该本地优先、命令行友好，而且方便 AI agent 检查和操作。 ## ⚡ 安装 ```bash npm install -g finsight -finsight init # 问你基准币、地区、文件存哪里 +finsight init # 问你基准币和地区 finsight overview # 看你的组合 ``` @@ -109,12 +111,12 @@ $ finsight overview --json | jq | 维度 | 大多数工具 | FinSight | |---|---|---| -| 数据存哪儿 | 它的云 | 你的笔记本上，纯文本文件 | -| 真相在谁手里 | 它的数据库 | 你的文件；App 的 DB 只是副本 | +| 数据存哪儿 | 它的云 | 你的笔记本上，本地 SQLite 数据库 | +| 真相在谁手里 | 它的数据库 | 你的本地 SQLite 数据库 | | AI 怎么用 | 也许给你一个聊天框 | 有文档的 JSON 输出，任何 AI 都能读 | | 基准币 | 写死 | `finsight config set base-currency JPY` | -| Schema 变化 | 它的迁移脚本 | 你自己改文件 | -| 备份 | 绑在它身上 | `git commit` 你的文件夹 | +| Schema 变化 | 它的迁移脚本 | 本地 SQLite 迁移 | +| 备份 | 绑在它身上 | `finsight backup create` | | 账户模型 | 必须注册 | 没有 —— 以你的身份在你机器上跑 | ## 🎯 在做什么 / 不在做什么 @@ -174,18 +176,14 @@ finsight context --json # 结构化数据，给自主 agent 用 └─────────┬───────────┴─────────┬───────────┘ │ │ ┌────▼─────┐ ┌─────▼─────┐ - │ Hono API │ ◄─────► │ SQLite │ ← 工作副本 - └────┬─────┘ └─────▲─────┘ + │ Hono API │ ◄─────► │ SQLite │ ← 本地事实源 + └────┬─────┘ └─────┬─────┘ │ │ - │ finsight ledger sync (每日) + │ backup create / verify │ ▼ │ ┌───────────────┐ - └────────────►│ 你的文件夹 │ ← 真相（git 追踪） - │ accounts.yaml│ - │ fx-rates.yaml│ - │ transactions │ - │ snapshots/ │ - │ decisions/ │ + └────────────►│ 原生 SQLite 备份 │ ← 恢复副本 + │ *.sqlite3 │ └───────────────┘ ``` @@ -203,7 +201,8 @@ finsight config set ledger-dir ~/notes/finance/ledger ``` 配置存在 `~/.finsight/config.json`，`FINSIGHT_DB_PATH` 环境变量可以 -改 SQLite 工作副本的位置。 +改事实源 SQLite 数据库的位置。`ledger-dir` 是可选的旧版导出/导入 +互操作配置，日常使用不需要配置。 ### 🔒 关于安全 @@ -255,11 +254,14 @@ finsight quote update --dry-run # 预览不写入 ```bash finsight reconcile # 和券商 App 显示的对比 finsight reconcile log # 历史对账记录 -finsight ledger sync # 把今天的 DB 镜像存到你文件夹里 -finsight ledger verify # 检查 DB 和文件是否一致 -finsight ledger restore --yes # 灾难恢复：从文件重建 DB +finsight backup create --json # 创建已校验的原生 SQLite 备份 +finsight backup verify --json # 验证已有原生备份 +finsight doctor --json # 检查数据库完整性和组合状态 ``` +`ledger sync`、`ledger verify`、`ledger restore` 只保留作显式的旧版 +互操作命令。ledger 导出是有损格式，不是规范的灾难恢复路径。 + **把组合喂给 LLM** ```bash finsight context | pbcopy # Markdown 简报 → 剪贴板 @@ -313,17 +315,18 @@ provider。新增语言改 `packages/core/src/i18n/index.ts`。完整说明见 ## 🗃 你文件夹里有什么 ``` -ledger/ -├── README.md (自动生成) -├── accounts.yaml — 你的账户 + 持仓（可以直接手编辑） -├── fx-rates.yaml — 币种间的汇率 -├── transactions.jsonl — 每个事件一行，只追加不修改 -├── snapshots/ — 每日 JSON 快照 -└── decisions/ — 每个投资决策一份 markdown 笔记 +legacy-ledger/ # 只有显式配置 ledger-dir 时才会有 +├── README.md (自动生成) +├── accounts.yaml — 导出的账户和持仓状态 +├── fx-rates.jsonl — 导出的带日期汇率 +├── transactions.jsonl — 导出的交易 +├── snapshots.jsonl — 导出的快照 +├── reconciliations.jsonl — 导出的对账记录 +└── decisions/ — 导出的决策笔记 ``` -这个结构故意保持简单。把 `accounts.yaml` 喂给任何 AI 问意见； -买完之后 commit 一下 diff；任意 git 历史点都能 restore 出当时的 DB。 +这是可选的兼容格式，只会在显式执行 ledger 命令时生成，适合检查或 +与旧工具互操作；它不是原生 SQLite 备份，也不会自动导入。 ## 🚧 项目状态 diff --git a/SECURITY.md b/SECURITY.md index 894af6a..5bf73d9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,8 +1,13 @@ # Security policy FinSight is a **local-first, single-user** tool. Your data lives on your -machine (`~/.finsight/`) and your vault directory. There is no server-side -component, no telemetry, no account system. +machine (`~/.finsight/`) in a local SQLite database and optional native backup +files. There is no cloud account, telemetry, or multi-user account system. + +The optional ledger directory is only an explicitly generated legacy +export/import format. It is not synchronized automatically and is not the +canonical recovery source; use `finsight backup create` for native recovery +copies. That said, a few security-relevant things still apply. @@ -29,8 +34,8 @@ In scope: is bound to a non-loopback interface - Connector code that mishandles untrusted upstream responses (e.g., quote feed returning malicious payloads) -- Vault/ledger code that could be tricked into writing outside the - configured `ledger_dir` +- Ledger export/import code that could write outside the explicitly configured + `ledger_dir` Out of scope: @@ -44,8 +49,8 @@ Out of scope: ## Hardening tips for users -- Keep your `ledger_dir` in a private git repo or an encrypted filesystem. - It contains your full position list. +- Keep native backups and any configured `ledger_dir` in a private or encrypted + location; they contain portfolio data. - Don't pipe `finsight context` into a third-party LLM if you consider your portfolio sensitive — `finsight context --json` and the Markdown form both include account names and dollar amounts. diff --git a/examples/README.md b/examples/README.md index 2a497f7..44a1423 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,22 +1,45 @@ # Example portfolios -Drop one of these YAML files into your `ledger_dir` as `accounts.yaml` to bootstrap a demo. +These YAML files are synthetic demo data. The normal FinSight source of truth is +local SQLite; examples are loaded by `finsight init --demo` or by the demo +script. | File | Profile | Base currency | |---|---|---| | `seed-portfolio.en.yaml` | US-leaning: VOO / VTI / AAPL / NVDA / cash / BTC | USD | | `seed-portfolio.zh.yaml` | China-leaning: A-share funds + HK + US stocks | CNY | -| `tickers.cn.yaml` | Chinese-name → ticker mapping (drop in ledger as `tickers.yaml`) | — | +| `tickers.cn.yaml` | Chinese-name → ticker mapping for optional legacy exports | — | ## Use as demo ```bash -finsight ledger init ~/finsight-demo -cp examples/seed-portfolio.en.yaml ~/finsight-demo/accounts.yaml -finsight ledger restore --yes +finsight init --non-interactive \ + --base-currency USD \ + --display-locale en-US \ + --labels-language en \ + --demo en finsight overview ``` -## Use as template +Or run the isolated dashboard demo: -Treat these as a starting point — edit account names, paste in your real positions, then `finsight ledger restore --yes`. +```bash +pnpm demo +# pnpm demo zh +``` + +The demo uses a temporary database and does not touch the user's real +`~/.finsight/` data. + +## Optional legacy export + +If an older tool needs the plain-text representation, configure a directory +explicitly and export from SQLite: + +```bash +finsight ledger init ~/finsight-legacy-export +finsight ledger export +``` + +The export is for interoperability and inspection. It is not a native backup +and should not be treated as the canonical recovery source. diff --git a/package.json b/package.json index 2ede763..ade4d66 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,7 @@ "version": "0.1.0", "private": true, "type": "module", - "description": "Local-first, AI-friendly personal portfolio tracker. Plain-text files are the source of truth; every command outputs JSON for AI tools.", + "description": "Local-first, AI-friendly personal portfolio tracker. SQLite is the authoritative local store; every command outputs JSON for AI tools.", "keywords": [ "portfolio", "personal-finance", @@ -12,8 +12,7 @@ "yfinance", "ai-friendly", "local-first", - "vault", - "obsidian", + "sqlite", "cli" ], "homepage": "https://github.com/ApeCodeAI/finsight#readme", diff --git a/packages/cli/README.md b/packages/cli/README.md index acfc3f3..e9f719b 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -4,14 +4,11 @@ See your money. Know where it sits. Decide what's next. -`finsight` is a CLI for tracking a personal investment portfolio across -multiple accounts and currencies. Your data lives in plain text files -on your disk — no cloud, no signup, no telemetry. Every command can -output structured JSON, so AI assistants (Claude, Cursor, Codex, -ChatGPT) can drive the tool on your behalf. +`finsight` tracks a personal investment portfolio across multiple accounts and +currencies. Data stays in a local SQLite database, every command can output +structured JSON, and AI assistants can drive the CLI on your behalf. -[Full documentation, screenshots, and roadmap on -GitHub →](https://github.com/ApeCodeAI/finsight) +[Full documentation and roadmap on GitHub →](https://github.com/ApeCodeAI/finsight) ## Install @@ -24,7 +21,7 @@ Requires Node.js >= 22. ## Quickstart ```bash -finsight init # interactive setup — base currency, locale, vault path +finsight init # interactive setup — base currency and locale finsight overview # net worth + breakdown by asset class finsight overview --json | jq # same data, machine-readable ``` @@ -42,6 +39,8 @@ finsight overview --json | jq # same data, machine-readable discrepancy is. - **AI briefings** via `finsight context` — Markdown or JSON payload for handing the portfolio to an LLM. +- **Native SQLite backups** via `finsight backup create` and + `finsight backup verify`. ## What it doesn't do @@ -57,10 +56,16 @@ finsight symbol show PDD # PDD across all your accounts finsight trade buy PDD 100 # auto-fetches today's price finsight quote update # refresh prices + FX finsight reconcile # compare to broker app +finsight backup create --json # create a verified native backup +finsight backup verify --json +finsight doctor --json # inspect local DB health finsight context | pbcopy # hand briefing to ChatGPT/Claude -finsight ledger sync # save daily snapshot to your folder ``` +The `ledger` commands remain available only for explicit legacy +export/import interoperability. Ledger export is lossy and is not the native +recovery path. + All read commands support `--json`. Exit codes are semantic: `0` ok · `1` bad input · `2` rule violated · `3` not found · `4` internal. @@ -87,7 +92,7 @@ once, then drives the tool. Tell your AI: > Read `skills/finsight/SKILL.md` and start tracking my portfolio. -> My salary lands in 招商 on the 15th — record it as a deposit. +> Record operations with the CLI and use `--json` for machine-readable output. ## Project diff --git a/packages/cli/package.json b/packages/cli/package.json index afac194..c61765f 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -36,7 +36,7 @@ "vitest": "^3.2.0", "typescript": "^5.8.0" }, - "description": "Local-first, AI-friendly personal portfolio tracker. Your files are truth, every CLI command outputs JSON for AI agents.", + "description": "Local-first, AI-friendly personal portfolio tracker. SQLite is authoritative, every CLI command outputs JSON for AI agents.", "keywords": [ "portfolio", "personal-finance", @@ -46,8 +46,7 @@ "local-first", "cli", "yfinance", - "obsidian", - "vault" + "sqlite" ], "repository": { "type": "git", diff --git a/packages/cli/src/__tests__/backup-cli.test.ts b/packages/cli/src/__tests__/backup-cli.test.ts new file mode 100644 index 0000000..2a917e5 --- /dev/null +++ b/packages/cli/src/__tests__/backup-cli.test.ts @@ -0,0 +1,75 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { getDb } from "@finsight/core"; +import { createBackupCommand } from "../commands/backup.js"; + +const tempDirs: string[] = []; +const originalDbPath = process.env.FINSIGHT_DB_PATH; + +function makeTempDir(): string { + const dir = mkdtempSync(path.join(tmpdir(), "finsight-backup-cli-test-")); + tempDirs.push(dir); + return dir; +} + +afterEach(() => { + vi.restoreAllMocks(); + if (originalDbPath === undefined) delete process.env.FINSIGHT_DB_PATH; + else process.env.FINSIGHT_DB_PATH = originalDbPath; + for (const dir of tempDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("backup CLI", () => { + it("creates and verifies SQLite backups with structured JSON", async () => { + const root = makeTempDir(); + const sourcePath = path.join(root, "source.db"); + const destinationDir = path.join(root, "backups"); + const db = getDb(sourcePath); + db.$client.close(); + process.env.FINSIGHT_DB_PATH = sourcePath; + + const lines: string[] = []; + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + lines.push(String(chunk)); + return true; + }); + + await createBackupCommand().parseAsync( + ["create", "--dir", destinationDir, "--json"], + { from: "user" }, + ); + const created = JSON.parse(lines.pop() ?? "null") as Record; + + expect(created).toMatchObject({ + ok: true, + operation: "backup.create", + source_path: sourcePath, + source_integrity: "ok", + backup_integrity: "ok", + mode: "0600", + }); + expect(created.backup_path).toEqual(expect.any(String)); + expect(created.sha256).toMatch(/^[a-f0-9]{64}$/); + expect(created.bytes).toEqual(expect.any(Number)); + + await createBackupCommand().parseAsync( + ["verify", String(created.backup_path), "--json"], + { from: "user" }, + ); + const verified = JSON.parse(lines.pop() ?? "null") as Record; + + expect(verified).toEqual({ + ok: true, + operation: "backup.verify", + backup_path: created.backup_path, + integrity: "ok", + sha256: created.sha256, + bytes: created.bytes, + mode: "0600", + }); + }); +}); diff --git a/packages/cli/src/__tests__/doctor.test.ts b/packages/cli/src/__tests__/doctor.test.ts new file mode 100644 index 0000000..82e49b1 --- /dev/null +++ b/packages/cli/src/__tests__/doctor.test.ts @@ -0,0 +1,120 @@ +import { spawnSync } from "node:child_process"; +import { + existsSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { getDb } from "@finsight/core"; +import { afterEach, describe, expect, it } from "vitest"; +import { collectStorageHealthChecks } from "../commands/doctor.js"; + +const tempDirs: string[] = []; +const cliEntry = fileURLToPath(new URL("../index.ts", import.meta.url)); + +function makeTempDir(): string { + const dir = mkdtempSync(path.join(tmpdir(), "finsight-doctor-test-")); + tempDirs.push(dir); + return dir; +} + +function runDoctor(home: string, databasePath: string) { + return spawnSync(process.execPath, ["--import", "tsx", cliEntry, "doctor", "--json"], { + cwd: path.dirname(cliEntry), + encoding: "utf8", + env: { ...process.env, HOME: home, FINSIGHT_DB_PATH: databasePath }, + }); +} + +afterEach(() => { + for (const dir of tempDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("doctor storage checks", () => { + it("treats an unconfigured legacy ledger as normal and checks SQLite integrity", () => { + const root = makeTempDir(); + const databasePath = path.join(root, "finsight.db"); + const db = getDb(databasePath); + db.$client.close(); + + const checks = collectStorageHealthChecks({ + databasePath, + ledgerDir: undefined, + }); + + expect(checks).toContainEqual({ + name: "db.integrity", + level: "ok", + message: `SQLite integrity check passed: ${databasePath}`, + }); + expect(checks).toContainEqual({ + name: "ledger.legacy", + level: "ok", + message: "Legacy ledger export/import is not configured (normal).", + }); + expect(checks.some((check) => check.name === "ledger.sync")).toBe(false); + }); + + it("reports a missing database without creating it", () => { + const home = makeTempDir(); + const databasePath = path.join(home, "missing", "finsight.db"); + + const result = runDoctor(home, databasePath); + + expect(result.status).toBe(2); + expect(result.stdout).toBe(""); + expect(JSON.parse(result.stderr.trim())).toMatchObject({ + code: "DATA_CONFLICT", + }); + expect(existsSync(databasePath)).toBe(false); + expect(existsSync(path.dirname(databasePath))).toBe(false); + }); + + it("reads healthy content without mutating the database", () => { + const home = makeTempDir(); + const databasePath = path.join(home, "healthy.db"); + const db = getDb(databasePath); + db.$client + .prepare( + `INSERT INTO accounts + (id, name, type, currency, balance, is_active, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + "doctor-account", + "Doctor account", + "cash", + "USD", + 50, + 1, + "2026-09-15T00:00:00.000Z", + "2026-09-15T00:00:00.000Z", + ); + db.$client.pragma("journal_mode = DELETE"); + db.$client.exec("CREATE TABLE incomes (id TEXT PRIMARY KEY)"); + db.$client.close(); + const before = readFileSync(databasePath); + + const result = runDoctor(home, databasePath); + + const after = readFileSync(databasePath); + expect(result.status).toBe(0); + expect(result.stderr).toBe(""); + const payload = JSON.parse(result.stdout.trim()) as { + checks: Array<{ name: string; message: string }>; + }; + expect(payload.checks).toContainEqual( + expect.objectContaining({ + name: "db.content", + message: expect.stringContaining("accounts=1"), + }), + ); + expect(after).toEqual(before); + }); +}); diff --git a/packages/cli/src/__tests__/init.test.ts b/packages/cli/src/__tests__/init.test.ts new file mode 100644 index 0000000..465784e --- /dev/null +++ b/packages/cli/src/__tests__/init.test.ts @@ -0,0 +1,103 @@ +import { spawnSync } from "node:child_process"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { getDb } from "@finsight/core"; +import { afterEach, describe, expect, it } from "vitest"; +import { collectInitAnswersFromFlags } from "../commands/init.js"; + +const tempDirs: string[] = []; +const cliEntry = fileURLToPath(new URL("../index.ts", import.meta.url)); +const repoRoot = path.resolve(path.dirname(cliEntry), "../../.."); + +function makeTempDir(): string { + const dir = mkdtempSync(path.join(tmpdir(), "finsight-init-test-")); + tempDirs.push(dir); + return dir; +} + +afterEach(() => { + for (const dir of tempDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("init defaults", () => { + it("does not configure or default a legacy ledger directory", () => { + expect(collectInitAnswersFromFlags({})).toEqual({ + base_currency: "USD", + display_locale: "en-US", + labels_language: "en", + demo: "none", + }); + }); + + it("rejects demo loading into an existing authoritative database", () => { + const home = makeTempDir(); + const databasePath = path.join(home, "authoritative.db"); + const db = getDb(databasePath); + db.$client + .prepare( + `INSERT INTO accounts + (id, name, type, currency, balance, is_active, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + "existing-account", + "Authoritative portfolio", + "brokerage", + "USD", + 1234, + 1, + "2026-09-15T00:00:00.000Z", + "2026-09-15T00:00:00.000Z", + ); + db.$client.close(); + + const result = spawnSync( + process.execPath, + [ + "--import", + "tsx", + cliEntry, + "init", + "--non-interactive", + "--demo", + "en", + "--force", + "--json", + ], + { + cwd: path.dirname(cliEntry), + encoding: "utf8", + env: { + ...process.env, + HOME: home, + FINSIGHT_DB_PATH: databasePath, + FINSIGHT_HOME: repoRoot, + }, + }, + ); + + expect(result.status).toBe(2); + expect(result.stdout).toBe(""); + expect(JSON.parse(result.stderr.trim())).toMatchObject({ + code: "DATA_CONFLICT", + }); + + const preserved = getDb(databasePath); + expect( + preserved.$client + .prepare("SELECT id, name, balance FROM accounts") + .all(), + ).toEqual([ + { + id: "existing-account", + name: "Authoritative portfolio", + balance: 1234, + }, + ]); + preserved.$client.close(); + }); +}); diff --git a/packages/cli/src/__tests__/ledger.test.ts b/packages/cli/src/__tests__/ledger.test.ts new file mode 100644 index 0000000..cb0b309 --- /dev/null +++ b/packages/cli/src/__tests__/ledger.test.ts @@ -0,0 +1,112 @@ +import { spawnSync } from "node:child_process"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { getDb } from "@finsight/core"; +import { afterEach, describe, expect, it } from "vitest"; +import { + requireLegacyImportConfirmation, + resolveLegacyLedgerDir, +} from "../commands/ledger.js"; + +const tempDirs: string[] = []; +const cliEntry = fileURLToPath(new URL("../index.ts", import.meta.url)); + +function makeTempDir(): string { + const dir = mkdtempSync(path.join(tmpdir(), "finsight-ledger-test-")); + tempDirs.push(dir); + return dir; +} + +afterEach(() => { + for (const dir of tempDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("legacy ledger export destination", () => { + it.each(["sync", "export"])( + "%s fails closed when neither --dir nor ledger-dir is configured", + () => { + expect(() => resolveLegacyLedgerDir(undefined, undefined)).toThrow( + /No legacy ledger directory specified/, + ); + expect(() => resolveLegacyLedgerDir(" ", "")).toThrow( + /No legacy ledger directory specified/, + ); + }, + ); + + it("resolves an explicit directory without falling back to cwd", () => { + expect(resolveLegacyLedgerDir("./legacy-export", undefined)).toBe( + path.resolve("legacy-export"), + ); + }); + + it("requires --yes for a lossy legacy import even in JSON mode", () => { + expect(() => requireLegacyImportConfirmation(false)).toThrow( + /requires --yes/, + ); + expect(() => requireLegacyImportConfirmation(true)).not.toThrow(); + }); +}); + +describe("ledger purge confirmation", () => { + it("requires --yes in JSON mode and leaves the database unchanged", () => { + const home = makeTempDir(); + const databasePath = path.join(home, "authoritative.db"); + const db = getDb(databasePath); + db.$client + .prepare( + `INSERT INTO accounts + (id, name, type, currency, balance, is_active, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run("account-1", "Cash", "cash", "USD", 10, 1, "2025-01-01", "2025-01-01"); + db.$client + .prepare( + `INSERT INTO transactions + (id, account_id, type, amount, fee, currency, traded_at, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run("tx-1", "account-1", "deposit", 10, 0, "USD", "2025-01-01", "2025-01-01"); + db.$client + .prepare( + `INSERT INTO snapshots + (id, snapshot_date, total_net_worth, created_at) + VALUES (?, ?, ?, ?)`, + ) + .run("snapshot-1", "2025-01-01", 10, "2025-01-01"); + db.$client.pragma("journal_mode = DELETE"); + db.$client.close(); + const before = readFileSync(databasePath); + + const result = spawnSync( + process.execPath, + [ + "--import", + "tsx", + cliEntry, + "ledger", + "purge", + "--before", + "2026-01-01", + "--json", + ], + { + cwd: path.dirname(cliEntry), + encoding: "utf8", + env: { ...process.env, HOME: home, FINSIGHT_DB_PATH: databasePath }, + }, + ); + + expect(result.status).toBe(1); + expect(result.stdout).toBe(""); + expect(JSON.parse(result.stderr.trim())).toMatchObject({ + code: "USER_ERROR", + error: expect.stringContaining("--yes"), + }); + expect(readFileSync(databasePath)).toEqual(before); + }); +}); diff --git a/packages/cli/src/commands/backup.ts b/packages/cli/src/commands/backup.ts new file mode 100644 index 0000000..77ce95c --- /dev/null +++ b/packages/cli/src/commands/backup.ts @@ -0,0 +1,78 @@ +import { Command } from "commander"; +import chalk from "chalk"; +import path from "node:path"; +import { + createSqliteBackup, + verifySqliteBackup, +} from "@finsight/core"; +import { emitJson, fail } from "../utils/exit.js"; +import { printInfo, printSuccess } from "../utils/display.js"; + +export function createBackupCommand(): Command { + const command = new Command("backup").description( + "Create and verify native backups of the authoritative SQLite database", + ); + + command + .command("create") + .description( + "Create an integrity-checked SQLite backup (default: ~/.finsight/backups)", + ) + .option("--dir ", "Backup directory (default: ~/.finsight/backups)") + .option("--json", "Emit structured JSON") + .action(async (opts) => { + try { + const result = await createSqliteBackup({ + destinationDir: opts.dir ? expandHome(opts.dir) : undefined, + }); + const payload = { ok: true, operation: "backup.create", ...result }; + if (opts.json) { + emitJson(payload); + return; + } + printSuccess(`Created verified SQLite backup: ${result.backup_path}`); + printInfo(`SHA-256: ${result.sha256}`); + printInfo(`Bytes: ${result.bytes} · mode: ${result.mode}`); + } catch (error) { + fail("DATA_CONFLICT", errorMessage(error), { json: opts.json }); + } + }); + + command + .command("verify") + .description("Verify SQLite integrity and report backup metadata") + .argument("", "Existing SQLite backup file") + .option("--json", "Emit structured JSON") + .action(async (file, opts) => { + try { + const result = await verifySqliteBackup(expandHome(file)); + const payload = { ok: true, operation: "backup.verify", ...result }; + if (opts.json) { + emitJson(payload); + return; + } + printSuccess(`Verified SQLite backup: ${result.backup_path}`); + printInfo(`Integrity: ${chalk.green(result.integrity)}`); + printInfo(`SHA-256: ${result.sha256}`); + printInfo(`Bytes: ${result.bytes} · mode: ${result.mode}`); + } catch (error) { + fail("DATA_CONFLICT", errorMessage(error), { json: opts.json }); + } + }); + + return command; +} + +export const backupCmd = createBackupCommand(); + +function expandHome(value: string): string { + if (value === "~") return process.env.HOME ?? value; + if (value.startsWith("~/")) { + return path.join(process.env.HOME ?? "", value.slice(2)); + } + return value; +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} diff --git a/packages/cli/src/commands/doctor.ts b/packages/cli/src/commands/doctor.ts index 8f37517..e55d996 100644 --- a/packages/cli/src/commands/doctor.ts +++ b/packages/cli/src/commands/doctor.ts @@ -10,30 +10,73 @@ import { listTransactionsNeedingRationale, getActiveTarget, getEffectiveConfig, - getLedgerDir, - verifyLedgerVsDb, + getDbPath, + getReadOnlyDb, + checkSqliteIntegrity, } from "@finsight/core"; -import { initDb } from "../utils/config.js"; -import { emitJson, ExitCode } from "../utils/exit.js"; +import { emitJson, fail, ExitCode } from "../utils/exit.js"; type CheckLevel = "ok" | "warn" | "error"; -interface Check { +export interface Check { name: string; level: CheckLevel; message: string; hint?: string; } +export function collectStorageHealthChecks(input: { + databasePath: string; + ledgerDir?: string; +}): Check[] { + const checks: Check[] = []; + try { + const result = checkSqliteIntegrity(input.databasePath); + checks.push({ + name: "db.integrity", + level: "ok", + message: `SQLite integrity check passed: ${result.database_path}`, + }); + } catch (error) { + checks.push({ + name: "db.integrity", + level: "error", + message: error instanceof Error ? error.message : String(error), + hint: "Recover only from a verified SQLite backup.", + }); + } + + if (!input.ledgerDir?.trim()) { + checks.push({ + name: "ledger.legacy", + level: "ok", + message: "Legacy ledger export/import is not configured (normal).", + }); + } else if (!existsSync(input.ledgerDir)) { + checks.push({ + name: "ledger.legacy", + level: "warn", + message: `Configured legacy ledger directory is missing: ${input.ledgerDir}`, + hint: "Update or clear ledger-dir if legacy interoperability is no longer used.", + }); + } else { + checks.push({ + name: "ledger.legacy", + level: "ok", + message: `Legacy ledger interoperability configured: ${input.ledgerDir}`, + }); + } + return checks; +} + export const doctorCmd = new Command("doctor") .description( - "Health check: verify config, vault, DB, prices, sync state. Run when something feels off.", + "Health check: verify config, SQLite integrity/content, prices, and portfolio state.", ) .option("--json", "Emit JSON") .action((opts) => { const checks: Check[] = []; const cfg = getEffectiveConfig(); - // 1. Config sanity if (!cfg.base_currency) { checks.push({ name: "config.base_currency", @@ -49,142 +92,122 @@ export const doctorCmd = new Command("doctor") }); } - // 2. Ledger configured + exists - const ledgerDir = getLedgerDir(); - if (!ledgerDir) { - checks.push({ - name: "ledger.configured", - level: "warn", - message: "No ledger directory configured (DB is unbacked).", - hint: "finsight ledger init ", - }); - } else if (!existsSync(ledgerDir)) { - checks.push({ - name: "ledger.exists", - level: "error", - message: `Ledger directory missing: ${ledgerDir}`, - hint: "finsight ledger init to recreate", - }); - } else { - checks.push({ - name: "ledger.exists", - level: "ok", - message: `Ledger at ${ledgerDir}`, + // Inspect the authoritative SQLite database before any writable open. + const databasePath = getDbPath(); + const storageChecks = collectStorageHealthChecks({ + databasePath, + ledgerDir: cfg.ledger_dir || undefined, + }); + checks.push(...storageChecks); + const integrityError = storageChecks.find( + (check) => check.name === "db.integrity" && check.level === "error", + ); + if (integrityError) { + fail("DATA_CONFLICT", integrityError.message, { + json: opts.json, + hint: integrityError.hint, }); } - // 3. DB writable + has content - const db = initDb(); - const accs = listAccounts(db, { includeInactive: true }); - const posCount = listPositions(db).length; - const txCount = listTransactions(db).length; - const snapCount = listSnapshots(db).length; - const decCount = listDecisions(db).length; - checks.push({ - name: "db.content", - level: accs.length === 0 ? "warn" : "ok", - message: `accounts=${accs.length} positions=${posCount} tx=${txCount} snapshots=${snapCount} decisions=${decCount}`, - hint: - accs.length === 0 - ? "Empty DB. Try `finsight init` to seed demo data, or `finsight account add`." - : undefined, - }); - - // 4. Sync state (DB vs vault) - if (ledgerDir && existsSync(ledgerDir)) { + let databaseReadError: unknown; + try { + const db = getReadOnlyDb(databasePath); try { - const diff = verifyLedgerVsDb(db, ledgerDir); - if (diff.in_sync) { + const accs = listAccounts(db, { includeInactive: true }); + const allPositions = listPositions(db); + const allTransactions = listTransactions(db); + const posCount = allPositions.length; + const txCount = allTransactions.length; + const snapCount = listSnapshots(db).length; + const decCount = listDecisions(db).length; + checks.push({ + name: "db.content", + level: accs.length === 0 ? "warn" : "ok", + message: `accounts=${accs.length} positions=${posCount} tx=${txCount} snapshots=${snapCount} decisions=${decCount}`, + hint: + accs.length === 0 + ? "Empty DB. Try `finsight init` to seed demo data, or `finsight account add`." + : undefined, + }); + + const stale = allPositions.filter((p) => p.current_price === 0); + if (stale.length > 0) { checks.push({ - name: "ledger.sync", + name: "prices.stale", + level: "warn", + message: `${stale.length} position(s) have current_price = 0`, + hint: "finsight quote update (refreshes from Yahoo Finance + 天天基金)", + }); + } else if (posCount > 0) { + checks.push({ + name: "prices.stale", level: "ok", - message: "DB and vault are in sync.", + message: "All positions have a current price.", }); - } else { + } + + const needsReview = allTransactions.filter((t) => t.needs_review === 1); + if (needsReview.length > 0) { checks.push({ - name: "ledger.sync", + name: "transactions.needs_review", level: "warn", - message: `Out of sync: ${diff.db_accounts}/${diff.ledger_accounts} accs, ${diff.db_transactions}/${diff.ledger_transactions} tx, ${diff.db_snapshots}/${diff.ledger_snapshots} snap`, - hint: "finsight ledger sync (or `restore --yes` if vault is newer)", + message: `${needsReview.length} transaction(s) flagged needs_review = 1`, + hint: "finsight trade list --needs-review (then `transaction confirm ` to clear)", }); } - } catch (e) { - checks.push({ - name: "ledger.sync", - level: "warn", - message: `Couldn't verify: ${e instanceof Error ? e.message : String(e)}`, - }); - } - } - - // 5. Stale prices: any position with current_price still at 0 or untouched - const stale = listPositions(db).filter((p) => p.current_price === 0); - if (stale.length > 0) { - checks.push({ - name: "prices.stale", - level: "warn", - message: `${stale.length} position(s) have current_price = 0`, - hint: "finsight quote update (refreshes from Yahoo Finance + 天天基金)", - }); - } else if (posCount > 0) { - checks.push({ - name: "prices.stale", - level: "ok", - message: "All positions have a current price.", - }); - } - // 6. needs_review transactions (price fallbacks) - const needsReview = listTransactions(db).filter((t) => t.needs_review === 1); - if (needsReview.length > 0) { - checks.push({ - name: "transactions.needs_review", - level: "warn", - message: `${needsReview.length} transaction(s) flagged needs_review = 1`, - hint: "finsight trade list --needs-review (then `transaction confirm ` to clear)", - }); - } + const pendingRat = listTransactionsNeedingRationale(db); + if (pendingRat.length > 0) { + checks.push({ + name: "decisions.pending_rationale", + level: "warn", + message: `${pendingRat.length} buy/sell without a rationale decision`, + hint: "finsight decision review (then `decision add --type rationale --tx ...`)", + }); + } - // 7. Pending rationale (buy/sell without a rationale decision) - const pendingRat = listTransactionsNeedingRationale(db); - if (pendingRat.length > 0) { - checks.push({ - name: "decisions.pending_rationale", - level: "warn", - message: `${pendingRat.length} buy/sell without a rationale decision`, - hint: "finsight decision review (then `decision add --type rationale --tx ...`)", - }); + const target = getActiveTarget(db); + if (!target) { + checks.push({ + name: "target.active", + level: "warn", + message: "No active target allocation set.", + hint: "finsight target add --active --alloc ... (or skip if you don't care)", + }); + } else { + checks.push({ + name: "target.active", + level: "ok", + message: `Active target: ${target.name}`, + }); + } + } finally { + db.$client.close(); + } + } catch (error) { + databaseReadError = error; } - - // 8. Active target - const target = getActiveTarget(db); - if (!target) { - checks.push({ - name: "target.active", - level: "warn", - message: "No active target allocation set.", - hint: "finsight target add --active --alloc ... (or skip if you don't care)", - }); - } else { - checks.push({ - name: "target.active", - level: "ok", - message: `Active target: ${target.name}`, - }); + if (databaseReadError) { + fail( + "DATA_CONFLICT", + `Cannot read the authoritative SQLite database: ${databaseReadError instanceof Error ? databaseReadError.message : String(databaseReadError)}`, + { + json: opts.json, + hint: "Run a verified recovery or migration command; doctor never modifies the database.", + }, + ); } - // 9. DB file size sanity - if (cfg.db_path && existsSync(cfg.db_path)) { - const size = statSync(cfg.db_path).size; + if (existsSync(databasePath)) { + const size = statSync(databasePath).size; const mb = (size / 1024 / 1024).toFixed(2); checks.push({ name: "db.size", level: "ok", - message: `${mb} MB at ${cfg.db_path}`, + message: `${mb} MB at ${databasePath}`, }); } - // Aggregate const errorCount = checks.filter((c) => c.level === "error").length; const warnCount = checks.filter((c) => c.level === "warn").length; @@ -193,16 +216,9 @@ export const doctorCmd = new Command("doctor") checks, summary: { ok: checks.length - errorCount - warnCount, warn: warnCount, error: errorCount }, }); - process.exit( - errorCount > 0 - ? ExitCode.DATA_CONFLICT - : warnCount > 0 - ? ExitCode.OK // warnings shouldn't fail scripts - : ExitCode.OK, - ); + process.exit(errorCount > 0 ? ExitCode.DATA_CONFLICT : ExitCode.OK); } - // Pretty print console.log(); for (const c of checks) { const icon = diff --git a/packages/cli/src/commands/init.ts b/packages/cli/src/commands/init.ts index 8a93b2c..a270110 100644 --- a/packages/cli/src/commands/init.ts +++ b/packages/cli/src/commands/init.ts @@ -1,10 +1,10 @@ import { Command } from "commander"; import path from "node:path"; -import { existsSync, copyFileSync, readdirSync } from "node:fs"; +import { existsSync, copyFileSync, mkdtempSync, rmSync } from "node:fs"; import { fileURLToPath } from "node:url"; -import { homedir } from "node:os"; +import { homedir, tmpdir } from "node:os"; import chalk from "chalk"; -import { input, select, confirm } from "@inquirer/prompts"; +import { input, select } from "@inquirer/prompts"; import { readConfig, writeConfig, @@ -12,29 +12,29 @@ import { ensureLedgerSkeleton, ensureReadme, rebuildDbFromLedger, + findPopulatedPortfolioTables, + getDbPath, DEFAULTS, } from "@finsight/core"; import { initDb } from "../utils/config.js"; import { emitJson, fail, ExitCode } from "../utils/exit.js"; -import { printSuccess, printInfo } from "../utils/display.js"; +import { printSuccess } from "../utils/display.js"; -interface InitAnswers { +export interface InitAnswers { base_currency: string; display_locale: string; labels_language: string; - ledger_dir: string; + ledger_dir?: string; demo: "en" | "zh" | "none"; } function findExamplesDir(): string | null { - // 1. From the source tree (development) const fromCli = path.resolve( fileURLToPath(import.meta.url), "../../../..", "examples", ); if (existsSync(fromCli)) return fromCli; - // 2. From the repo root if FINSIGHT_HOME points there const home = process.env.FINSIGHT_HOME; if (home) { const fromHome = path.join(home, "examples"); @@ -43,28 +43,9 @@ function findExamplesDir(): string | null { return null; } -/** - * Suggest a sensible default ledger directory. If the user appears to have an - * Obsidian vault at one of the common locations, prefer placing the ledger - * under it so portfolio data and notes live side by side. Otherwise drop a - * plain `~/finsight-vault`. - */ -function defaultLedgerSuggestion(): string { - const obsidianCandidates = [ - path.join(homedir(), "Documents", "Obsidian Vault", "finsight"), - path.join(homedir(), "Obsidian", "finsight"), - path.join(homedir(), "notes", "finsight"), - path.join(homedir(), "vault", "finsight"), - ]; - for (const candidate of obsidianCandidates) { - if (existsSync(path.dirname(candidate))) return candidate; - } - return path.join(homedir(), "finsight-vault"); -} - export const initCmd = new Command("init") .description("First-time interactive setup wizard") - .option("--force", "Overwrite existing config") + .option("--force", "Overwrite existing config only; never replace portfolio data") .option( "--non-interactive", "Use flags only; suitable for CI / scripts. Combine with --base-currency etc.", @@ -72,7 +53,10 @@ export const initCmd = new Command("init") .option("--base-currency ", "Override base currency (e.g. USD)") .option("--display-locale ", "Override Intl locale (e.g. en-US)") .option("--labels-language ", "Override labels language (en / zh)") - .option("--ledger-dir ", "Override ledger directory") + .option( + "--ledger-dir ", + "Configure an optional legacy ledger export/import directory", + ) .option( "--demo ", "Load demo data: en / zh / none", @@ -91,22 +75,47 @@ export const initCmd = new Command("init") } const answers = opts.nonInteractive - ? collectFromFlags(opts) - : await collectFromPrompts(opts); + ? collectInitAnswersFromFlags(opts) + : await collectFromPrompts(); + + if (answers.demo !== "none") { + let populatedTables: string[]; + try { + populatedTables = findPopulatedPortfolioTables(getDbPath()); + } catch (error) { + fail( + "DATA_CONFLICT", + `Cannot safely inspect the authoritative SQLite database before loading demo data: ${error instanceof Error ? error.message : String(error)}`, + { + json: opts.json, + hint: "Use a new empty database path for demo data.", + }, + ); + } + if (populatedTables.length > 0) { + fail( + "DATA_CONFLICT", + `Refusing to load demo data because the authoritative SQLite database contains portfolio data (${populatedTables.join(", ")}).`, + { + json: opts.json, + hint: "Use a new empty database path; --force only overwrites config.", + }, + ); + } + } - // Persist config const cfg = readConfig(); cfg.base_currency = answers.base_currency; cfg.display_locale = answers.display_locale; cfg.labels_language = answers.labels_language; - cfg.ledger_dir = answers.ledger_dir; + if (answers.ledger_dir !== undefined) cfg.ledger_dir = answers.ledger_dir; writeConfig(cfg); - // Skeleton + README - ensureLedgerSkeleton(answers.ledger_dir); - ensureReadme(answers.ledger_dir); + if (answers.ledger_dir) { + ensureLedgerSkeleton(answers.ledger_dir); + ensureReadme(answers.ledger_dir); + } - // Demo data let demoLoaded = false; let demoCounts: | { @@ -122,29 +131,32 @@ export const initCmd = new Command("init") if (!examplesDir) { if (!opts.json) { process.stderr.write( - chalk.yellow( - "⚠ Could not locate examples/ directory — skipping demo data.\n", - ), + chalk.yellow("⚠ Could not locate examples/ directory — skipping demo data.\n"), ); } } else { const src = path.join(examplesDir, `seed-portfolio.${answers.demo}.yaml`); - const tickerSrc = path.join(examplesDir, `tickers.cn.yaml`); - const dst = path.join(answers.ledger_dir, "accounts.yaml"); + const tickerSrc = path.join(examplesDir, "tickers.cn.yaml"); if (!existsSync(src)) { if (!opts.json) { - process.stderr.write( - chalk.yellow(`⚠ Demo source not found: ${src}\n`), - ); + process.stderr.write(chalk.yellow(`⚠ Demo source not found: ${src}\n`)); } } else { - copyFileSync(src, dst); - if (answers.demo === "zh" && existsSync(tickerSrc)) { - copyFileSync(tickerSrc, path.join(answers.ledger_dir, "tickers.yaml")); + const importDir = + answers.ledger_dir ?? mkdtempSync(path.join(tmpdir(), "finsight-demo-import-")); + try { + ensureLedgerSkeleton(importDir); + copyFileSync(src, path.join(importDir, "accounts.yaml")); + if (answers.ledger_dir && answers.demo === "zh" && existsSync(tickerSrc)) { + copyFileSync(tickerSrc, path.join(importDir, "tickers.yaml")); + } + const db = initDb(); + demoCounts = rebuildDbFromLedger(db, importDir); + db.$client.close(); + demoLoaded = true; + } finally { + if (!answers.ledger_dir) rmSync(importDir, { recursive: true, force: true }); } - const db = initDb(); - demoCounts = rebuildDbFromLedger(db, answers.ledger_dir); - demoLoaded = true; } } } @@ -153,7 +165,7 @@ export const initCmd = new Command("init") emitJson({ ok: true, config: cfg, - ledger_dir: answers.ledger_dir, + ledger_dir: cfg.ledger_dir ?? null, demo_loaded: demoLoaded, demo_variant: answers.demo, ...(demoCounts ?? {}), @@ -163,31 +175,27 @@ export const initCmd = new Command("init") console.log(); printSuccess(`Wrote ${configPath()}`); - printSuccess(`Created vault at ${answers.ledger_dir}`); + if (answers.ledger_dir) { + printSuccess(`Configured legacy ledger interoperability at ${answers.ledger_dir}`); + } if (demoLoaded && demoCounts) { - printSuccess( - `Loaded ${demoCounts.accounts} demo accounts (${demoCounts.positions} positions)`, - ); + printSuccess(`Loaded ${demoCounts.accounts} demo accounts (${demoCounts.positions} positions)`); } console.log(); console.log(chalk.bold("Next steps:")); console.log(` ${chalk.cyan("finsight overview")} # See your portfolio`); console.log(` ${chalk.cyan("finsight web")} # Open the dashboard`); console.log(` ${chalk.cyan("finsight account add")} # Add your real accounts`); - if (answers.demo !== "none") { - console.log( - chalk.dim( - ` ${chalk.cyan("finsight ledger restore --yes")} # rebuild from vault any time`, - ), - ); - } + console.log( + chalk.dim(` ${chalk.cyan("finsight backup create")} # create a verified DB backup`), + ); console.log(); }); -async function collectFromPrompts(opts: { force?: boolean }): Promise { +async function collectFromPrompts(): Promise { console.log(); console.log(chalk.bold(" Welcome to FinSight!")); - console.log(chalk.dim(" Local-first portfolio tracker · AI-friendly · vault-backed")); + console.log(chalk.dim(" Local-first portfolio tracker · AI-friendly · SQLite-backed")); console.log(); const base_currency = await select({ @@ -207,12 +215,10 @@ async function collectFromPrompts(opts: { force?: boolean }): Promise", "Vault ledger directory (e.g. ~/finsight-vault or ~/Documents/Obsidian Vault/finsight)") + .description("Configure an optional legacy ledger export/import directory") + .argument("", "Legacy ledger directory") .option("--json", "Emit JSON") .action((dir, opts) => { const resolved = path.resolve(expandHome(dir)); @@ -40,21 +61,21 @@ ledgerCmd process.exit(ExitCode.OK); } printSuccess(`ledger_dir configured: ${resolved}`); - printInfo("Daily workflow:"); - printInfo(" · Run finsight commands normally (writes only to local DB)."); - printInfo(" · Once a day: `finsight ledger sync` to mirror DB → vault."); - printInfo(" · DB lost / corrupted: `finsight ledger restore` to rebuild from vault."); + printInfo("SQLite remains the sole source of truth."); + printInfo("Use ledger commands only for explicit legacy export/import interoperability."); }); ledgerCmd .command("sync") - .description("Mirror current DB → vault ledger (run daily as a backup)") - .option("--dir ", "Override ledger directory (default: from config)") + .description("Legacy export: write current SQLite data to a vault ledger") + .option("--dir ", "Override legacy ledger directory (default: from config)") .option("--json", "Emit JSON") .action((opts) => { - const root = path.resolve(expandHome(opts.dir ?? getLedgerDir() ?? "")); - if (!root) { - fail("USER_ERROR", "No ledger directory configured. Run `finsight ledger init ` first.", { + let root: string; + try { + root = resolveLegacyLedgerDir(opts.dir, getLedgerDir()); + } catch (error) { + fail("USER_ERROR", error instanceof Error ? error.message : String(error), { json: opts.json, }); } @@ -64,25 +85,29 @@ ledgerCmd emitJson({ ok: true, dir: root, ...result }); process.exit(ExitCode.OK); } - printSuccess(`Synced DB → ${root}`); + printSuccess(`Exported SQLite data → legacy ledger at ${root}`); printInfo(`accounts: ${result.accounts} · positions: ${result.positions}`); printInfo( `transactions: ${result.transactions} · snapshots: ${result.snapshots} · fx: ${result.fx_rates} · decisions: ${result.decisions ?? 0} · reconciliations: ${result.reconciliations}`, ); - printInfo("Commit the vault changes to git when convenient."); + printInfo("This lossy export is not a native backup or source of truth."); }); // alias: export is the legacy name for sync (kept for backwards compat with any // scripts already calling it) ledgerCmd .command("export") - .description("Alias for `sync` — mirror DB → vault") - .option("--dir ", "Override ledger directory") + .description("Legacy export: alias for `sync`") + .option("--dir ", "Override legacy ledger directory") .option("--json", "Emit JSON") .action((opts) => { - const root = path.resolve(expandHome(opts.dir ?? getLedgerDir() ?? "")); - if (!root) { - fail("USER_ERROR", "No ledger directory configured.", { json: opts.json }); + let root: string; + try { + root = resolveLegacyLedgerDir(opts.dir, getLedgerDir()); + } catch (error) { + fail("USER_ERROR", error instanceof Error ? error.message : String(error), { + json: opts.json, + }); } const db = initDb(); const result = dumpDbToLedger(db, root); @@ -90,12 +115,12 @@ ledgerCmd emitJson({ ok: true, dir: root, ...result }); process.exit(ExitCode.OK); } - printSuccess(`Synced DB → ${root}`); + printSuccess(`Exported SQLite data → legacy ledger at ${root}`); }); ledgerCmd .command("restore") - .description("Rebuild DB from vault ledger (disaster recovery; WIPES local DB)") + .description("Legacy import: replace SQLite data from a ledger (LOSSY; WIPES DB)") .option("--yes", "Skip confirmation") .option("--json", "Emit JSON") .action((opts) => { @@ -106,15 +131,23 @@ ledgerCmd if (!existsSync(root)) { fail("NOT_FOUND", `Ledger directory not found: ${root}`, { json: opts.json }); } - if (!opts.yes && !opts.json) { - process.stderr.write( - chalk.yellow( - "⚠ This will WIPE the local DB and rebuild from vault.\n" + - " Any CLI writes made since the last `ledger sync` will be lost.\n" + - " Run with --yes to proceed.\n", - ), - ); - process.exit(ExitCode.USER_ERROR); + if (!opts.yes) { + if (!opts.json) { + process.stderr.write( + chalk.yellow( + "⚠ LEGACY LOSSY IMPORT: this will WIPE the authoritative SQLite DB.\n" + + " Ledger files do not preserve every SQLite field or row.\n" + + " First run `finsight backup create` and keep the verified backup.\n", + ), + ); + } + try { + requireLegacyImportConfirmation(false); + } catch (error) { + fail("USER_ERROR", error instanceof Error ? error.message : String(error), { + json: opts.json, + }); + } } const db = initDb(); const result = rebuildDbFromLedger(db, root); @@ -122,7 +155,7 @@ ledgerCmd emitJson({ ok: true, dir: root, ...result }); process.exit(ExitCode.OK); } - printSuccess(`Restored DB from ${root}`); + printSuccess(`Imported legacy ledger into SQLite from ${root}`); printInfo(`accounts: ${result.accounts} · positions: ${result.positions}`); printInfo( `transactions: ${result.transactions} · snapshots: ${result.snapshots} · fx: ${result.fx_rates} · decisions: ${result.decisions ?? 0} · reconciliations: ${result.reconciliations}`, @@ -131,7 +164,7 @@ ledgerCmd ledgerCmd .command("verify") - .description("Compare DB vs vault ledger; non-zero exit if they diverge") + .description("Legacy check: compare exported ledger counts with SQLite") .option("--json", "Emit JSON") .action((opts) => { const root = getLedgerDir(); @@ -158,12 +191,13 @@ ledgerCmd } console.log(table.toString()); if (diff.in_sync) { - printSuccess("DB and vault ledger are in sync."); + printSuccess("Legacy export counts match SQLite."); } else { process.stderr.write( chalk.yellow( - "⚠ DB and vault diverge. If DB is newer, run `finsight ledger sync`.\n" + - " If you manually edited vault YAML, run `finsight ledger restore --yes`.\n", + "⚠ Legacy ledger and authoritative SQLite counts differ.\n" + + " Export again only if you explicitly need interoperability.\n" + + " Never import it as canonical recovery without a verified DB backup.\n", ), ); process.exit(ExitCode.DATA_CONFLICT); @@ -186,6 +220,20 @@ ledgerCmd .option("--dry-run", "Show what would be deleted without changing the DB") .option("--json", "Emit JSON") .action(async (opts) => { + if (!opts.dryRun && !opts.yes) { + if (!opts.json) { + process.stderr.write( + chalk.yellow( + "⚠ Destructive purge requires --yes. Run with --dry-run to preview.\n", + ), + ); + } + fail("USER_ERROR", "Ledger purge requires --yes unless --dry-run is used.", { + json: opts.json, + hint: "Create a verified native backup before purging historical data.", + }); + } + const db = initDb(); // Preview the impact first (always, before any write). @@ -214,19 +262,6 @@ ledgerCmd process.exit(ExitCode.OK); } - if (!opts.yes && !opts.json) { - process.stderr.write( - chalk.yellow( - `⚠ Will delete ${preview.transactions_would_delete} transactions ` + - `and ${preview.snapshots_would_delete} snapshots ` + - `(traded_at / snapshot_date < ${opts.before}).\n` + - ` Destructive, no automatic backup.\n` + - ` Run with --yes to proceed, or --dry-run for JSON preview.\n`, - ), - ); - process.exit(ExitCode.USER_ERROR); - } - try { const result = purgeHistoricalBefore(db, opts.before); if (opts.json) { @@ -237,7 +272,7 @@ ledgerCmd `Purged: ${result.transactions_deleted} transactions, ` + `${result.snapshots_deleted} snapshots (cutoff < ${result.cutoff})`, ); - printInfo("Run `finsight ledger sync` to mirror to vault."); + printInfo("Run `finsight backup create` to capture a verified native backup."); } catch (e) { fail("USER_ERROR", e instanceof Error ? e.message : String(e), { json: opts.json, @@ -247,7 +282,7 @@ ledgerCmd ledgerCmd .command("status") - .description("Show current ledger configuration") + .description("Show optional legacy ledger interoperability configuration") .option("--json", "Emit JSON") .action((opts) => { const cfg = readConfig(); @@ -262,12 +297,12 @@ ledgerCmd (existsSync(cfg.ledger_dir) ? chalk.green("exists") : chalk.red("missing")), ); console.log(); - console.log(chalk.dim(" Workflow:")); - console.log(chalk.dim(" finsight ledger sync — daily DB → vault")); - console.log(chalk.dim(" finsight ledger verify — check sync state")); - console.log(chalk.dim(" finsight ledger restore — vault → DB (disaster recovery)")); + console.log(chalk.dim(" Legacy interoperability only; SQLite is authoritative.")); + console.log(chalk.dim(" finsight ledger sync — explicit SQLite → ledger export")); + console.log(chalk.dim(" finsight ledger verify — compare exported counts")); + console.log(chalk.dim(" finsight ledger restore — lossy ledger → SQLite import")); } else { - console.log(" No ledger configured. Run `finsight ledger init `."); + console.log(" No legacy ledger configured (normal). SQLite is authoritative."); } }); diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 5346b30..74ae069 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -18,13 +18,14 @@ import { importCmd } from "./commands/import-cmd.js"; import { overviewCmd } from "./commands/overview.js"; import { performanceCmd } from "./commands/performance.js"; import { doctorCmd } from "./commands/doctor.js"; +import { backupCmd } from "./commands/backup.js"; import { webCmd } from "./commands/web.js"; const program = new Command(); program .name("finsight") .description( - "Local-first portfolio tracker. Every command supports --json.\n\n" + + "Local-first portfolio tracker. SQLite is the authoritative store. Every command supports --json.\n\n" + " AI agents: read `skills/finsight/SKILL.md`, then start with\n" + " `finsight context` (LLM-ready briefing) or `finsight doctor` (health check).", ) @@ -49,6 +50,7 @@ program.addCommand(importCmd); program.addCommand(overviewCmd); program.addCommand(performanceCmd); program.addCommand(doctorCmd); +program.addCommand(backupCmd); program.addCommand(webCmd); program.parse(process.argv); diff --git a/packages/cli/src/utils/config.ts b/packages/cli/src/utils/config.ts index 59423f1..f7fbc7d 100644 --- a/packages/cli/src/utils/config.ts +++ b/packages/cli/src/utils/config.ts @@ -8,8 +8,8 @@ let _db: AppDatabase | null = null; * 2. config.db_path in ~/.finsight/config.json * 3. default: ~/.finsight/data/finsight.db * - * The vault ledger (when configured) is a daily backup, not the runtime source - * of truth — see `finsight ledger sync` / `finsight ledger restore`. + * SQLite is the sole source of truth. The optional ledger commands are explicit, + * legacy export/import interoperability and are never run automatically. */ export function initDb(_opts: { skipRebuild?: boolean } = {}): AppDatabase { if (!_db) { diff --git a/packages/core/package.json b/packages/core/package.json index a318147..374266f 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -22,7 +22,7 @@ "typescript": "^5.8.0", "drizzle-kit": "^0.31.0" }, - "description": "FinSight core — schema, services, ledger, config, i18n. Vault YAML is truth, SQLite is cache.", + "description": "FinSight core — authoritative SQLite schema, services, native backups, config, and legacy ledger interoperability.", "repository": { "type": "git", "url": "git+https://github.com/ApeCodeAI/finsight.git", diff --git a/packages/core/src/__tests__/backup.test.ts b/packages/core/src/__tests__/backup.test.ts new file mode 100644 index 0000000..d0ea5db --- /dev/null +++ b/packages/core/src/__tests__/backup.test.ts @@ -0,0 +1,129 @@ +import Database from "better-sqlite3"; +import { + existsSync, + mkdtempSync, + readFileSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + checkSqliteIntegrity, + createSqliteBackup, + verifySqliteBackup, +} from "../db/backup.js"; + +const tempDirs: string[] = []; + +function makeTempDir(): string { + const dir = mkdtempSync(path.join(tmpdir(), "finsight-backup-test-")); + tempDirs.push(dir); + return dir; +} + +afterEach(() => { + for (const dir of tempDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("createSqliteBackup", () => { + it("creates a verified online backup that includes committed WAL data", async () => { + const root = makeTempDir(); + const sourcePath = path.join(root, "source.db"); + const destinationDir = path.join(root, "backups"); + const source = new Database(sourcePath); + source.pragma("journal_mode = WAL"); + source.exec("CREATE TABLE entries (value TEXT NOT NULL)"); + source.prepare("INSERT INTO entries (value) VALUES (?)").run("from-wal"); + + try { + const result = await createSqliteBackup({ + sourcePath, + destinationDir, + now: new Date("2026-09-15T02:00:00.123Z"), + }); + + expect(path.dirname(result.backup_path)).toBe(destinationDir); + expect(path.basename(result.backup_path)).toMatch( + /^finsight-20260915T020000123Z-[a-f0-9]{32}\.sqlite3$/, + ); + expect(result.source_integrity).toBe("ok"); + expect(result.backup_integrity).toBe("ok"); + expect(result.sha256).toMatch(/^[a-f0-9]{64}$/); + expect(result.bytes).toBe(statSync(result.backup_path).size); + expect(result.mode).toBe("0600"); + expect(statSync(result.backup_path).mode & 0o777).toBe(0o600); + + const backup = new Database(result.backup_path, { + readonly: true, + fileMustExist: true, + }); + try { + expect( + backup.prepare("SELECT value FROM entries").get(), + ).toEqual({ value: "from-wal" }); + } finally { + backup.close(); + } + } finally { + source.close(); + } + }); +}); + +describe("checkSqliteIntegrity", () => { + it("reports integrity for an existing authoritative database", () => { + const root = makeTempDir(); + const sourcePath = path.join(root, "source.db"); + const source = new Database(sourcePath); + source.exec("CREATE TABLE entries (value TEXT NOT NULL)"); + source.close(); + + expect(checkSqliteIntegrity(sourcePath)).toEqual({ + database_path: sourcePath, + integrity: "ok", + }); + }); +}); + +describe("verifySqliteBackup", () => { + it("checks integrity and reports stable metadata", async () => { + const root = makeTempDir(); + const sourcePath = path.join(root, "source.db"); + const source = new Database(sourcePath); + source.exec( + "CREATE TABLE entries (value TEXT NOT NULL); INSERT INTO entries VALUES ('saved')", + ); + source.close(); + + const created = await createSqliteBackup({ + sourcePath, + destinationDir: path.join(root, "backups"), + }); + const verified = await verifySqliteBackup(created.backup_path); + + expect(verified).toEqual({ + backup_path: created.backup_path, + integrity: "ok", + sha256: created.sha256, + bytes: created.bytes, + mode: "0600", + }); + }); + + it("rejects a corrupt backup with an integrity-check error", async () => { + const root = makeTempDir(); + const backupPath = path.join(root, "corrupt.sqlite3"); + writeFileSync(backupPath, "not a sqlite database", { mode: 0o600 }); + + await expect(verifySqliteBackup(backupPath)).rejects.toThrow( + /SQLite integrity check failed/, + ); + expect(existsSync(backupPath)).toBe(true); + expect(readFileSync(backupPath, "utf8")).toBe("not a sqlite database"); + }); +}); diff --git a/packages/core/src/db/backup.ts b/packages/core/src/db/backup.ts new file mode 100644 index 0000000..71b483e --- /dev/null +++ b/packages/core/src/db/backup.ts @@ -0,0 +1,212 @@ +import Database from "better-sqlite3"; +import { createHash, randomBytes } from "node:crypto"; +import { + chmodSync, + createReadStream, + existsSync, + mkdtempSync, + mkdirSync, + renameSync, + rmSync, + statSync, +} from "node:fs"; +import { homedir } from "node:os"; +import path from "node:path"; +import { getDbPath } from "../config/index.js"; + +export interface CreateSqliteBackupOptions { + sourcePath?: string; + destinationDir?: string; + now?: Date; +} + +export interface SqliteBackupResult { + source_path: string; + backup_path: string; + created_at: string; + source_integrity: "ok"; + backup_integrity: "ok"; + sha256: string; + bytes: number; + mode: string; +} + +export interface SqliteBackupVerification { + backup_path: string; + integrity: "ok"; + sha256: string; + bytes: number; + mode: string; +} + +export interface SqliteIntegrityResult { + database_path: string; + integrity: "ok"; +} + +export function getDefaultBackupDir(): string { + return path.join(homedir(), ".finsight", "backups"); +} + +export function checkSqliteIntegrity( + databasePath = getDbPath(), +): SqliteIntegrityResult { + const resolvedPath = path.resolve(databasePath); + const database = new Database(resolvedPath, { + readonly: true, + fileMustExist: true, + }); + try { + assertIntegrity(database, resolvedPath); + return { database_path: resolvedPath, integrity: "ok" }; + } finally { + database.close(); + } +} + +/** + * Create a native SQLite online backup in a private temporary directory, + * validate it, then move it into the backup directory. + * + * FinSight is a single-user local MVP. This deliberately uses a small, + * retryable file workflow rather than a hardened multi-process protocol. + */ +export async function createSqliteBackup( + options: CreateSqliteBackupOptions = {}, +): Promise { + const sourcePath = path.resolve(options.sourcePath ?? getDbPath()); + const destinationDir = path.resolve( + options.destinationDir ?? getDefaultBackupDir(), + ); + const now = options.now ?? new Date(); + + mkdirSync(destinationDir, { recursive: true, mode: 0o700 }); + const timestamp = formatFilenameTimestamp(now); + const suffix = randomBytes(16).toString("hex"); + const backupPath = path.join( + destinationDir, + `finsight-${timestamp}-${suffix}.sqlite3`, + ); + const stagingDirectory = mkdtempSync( + path.join(path.dirname(destinationDir), ".finsight-backup-stage-"), + ); + chmodSync(stagingDirectory, 0o700); + const temporaryPath = path.join(stagingDirectory, path.basename(backupPath)); + + try { + const source = new Database(sourcePath, { + readonly: true, + fileMustExist: true, + }); + try { + assertIntegrity(source, sourcePath); + await source.backup(temporaryPath); + } finally { + source.close(); + } + + chmodSync(temporaryPath, 0o600); + const backup = new Database(temporaryPath, { + readonly: true, + fileMustExist: true, + }); + try { + assertIntegrity(backup, temporaryPath); + } finally { + backup.close(); + } + + const sha256 = await sha256File(temporaryPath); + const stagedStats = statSync(temporaryPath); + if (formatMode(stagedStats.mode) !== "0600") { + throw new Error(`SQLite backup permissions are not 0600: ${temporaryPath}`); + } + if (existsSync(backupPath)) { + throw new Error(`Backup destination already exists: ${backupPath}`); + } + + renameSync(temporaryPath, backupPath); + const publishedStats = statSync(backupPath); + return { + source_path: sourcePath, + backup_path: backupPath, + created_at: now.toISOString(), + source_integrity: "ok", + backup_integrity: "ok", + sha256, + bytes: publishedStats.size, + mode: formatMode(publishedStats.mode), + }; + } finally { + rmSync(stagingDirectory, { recursive: true, force: true }); + } +} + +export async function verifySqliteBackup( + backupPath: string, +): Promise { + const resolvedPath = path.resolve(backupPath); + const backup = new Database(resolvedPath, { + readonly: true, + fileMustExist: true, + }); + try { + assertIntegrity(backup, resolvedPath); + } finally { + backup.close(); + } + + const sha256 = await sha256File(resolvedPath); + const stats = statSync(resolvedPath); + return { + backup_path: resolvedPath, + integrity: "ok", + sha256, + bytes: stats.size, + mode: formatMode(stats.mode), + }; +} + +function assertIntegrity( + database: InstanceType, + filePath: string, +): void { + let rows: Array<{ integrity_check: string }>; + try { + rows = database.pragma("integrity_check") as Array<{ + integrity_check: string; + }>; + } catch (error) { + const detail = error instanceof Error ? error.message : String(error); + throw new Error( + `SQLite integrity check failed for ${filePath}: ${detail}`, + { cause: error }, + ); + } + const messages = rows.map((row) => row.integrity_check); + if (messages.length !== 1 || messages[0] !== "ok") { + throw new Error( + `SQLite integrity check failed for ${filePath}: ${messages.join("; ") || "no result"}`, + ); + } +} + +function formatFilenameTimestamp(date: Date): string { + if (Number.isNaN(date.getTime())) throw new Error("Invalid backup timestamp"); + return date.toISOString().replace(/[-:.]/g, ""); +} + +function formatMode(mode: number): string { + return (mode & 0o777).toString(8).padStart(4, "0"); +} + +async function sha256File(filePath: string): Promise { + const hash = createHash("sha256"); + await new Promise((resolve, reject) => { + const stream = createReadStream(filePath); + stream.on("data", (chunk) => hash.update(chunk)); + stream.on("error", reject); + stream.on("end", resolve); + }); + return hash.digest("hex"); +} diff --git a/packages/core/src/db/connection.ts b/packages/core/src/db/connection.ts index 2ecf680..aca07eb 100644 --- a/packages/core/src/db/connection.ts +++ b/packages/core/src/db/connection.ts @@ -1,6 +1,6 @@ import Database from "better-sqlite3"; import { drizzle } from "drizzle-orm/better-sqlite3"; -import { mkdirSync } from "node:fs"; +import { existsSync, mkdirSync } from "node:fs"; import { dirname } from "node:path"; import * as schema from "./schema.js"; import { getDbPath as configDbPath } from "../config/index.js"; @@ -168,3 +168,54 @@ export function getDb(dbPath?: string) { pushSchema(sqlite); return createDb(sqlite); } + +/** Open an existing database without creating, migrating, or writing to it. */ +export function getReadOnlyDb(dbPath?: string) { + const resolvedPath = dbPath ?? configDbPath(); + const sqlite = new Database(resolvedPath, { + readonly: true, + fileMustExist: true, + }); + sqlite.pragma("query_only = ON"); + return createDb(sqlite); +} + +const PORTFOLIO_DATA_TABLES = [ + "accounts", + "positions", + "transactions", + "snapshots", + "exchange_rates", + "decisions", + "targets", + "reconciliations", +] as const; + +/** Inspect an existing database without creating it or applying migrations. */ +export function findPopulatedPortfolioTables(dbPath?: string): string[] { + const resolvedPath = dbPath ?? configDbPath(); + if (!existsSync(resolvedPath)) return []; + + const sqlite = new Database(resolvedPath, { + readonly: true, + fileMustExist: true, + }); + try { + const existingTables = new Set( + ( + sqlite + .prepare("SELECT name FROM sqlite_master WHERE type = 'table'") + .all() as Array<{ name: string }> + ).map((row) => row.name), + ); + return PORTFOLIO_DATA_TABLES.filter((table) => { + if (!existingTables.has(table)) return false; + const row = sqlite + .prepare(`SELECT 1 AS present FROM "${table}" LIMIT 1`) + .get() as { present: number } | undefined; + return row?.present === 1; + }); + } finally { + sqlite.close(); + } +} diff --git a/packages/core/src/db/schema.ts b/packages/core/src/db/schema.ts index 88b68de..a3b1690 100644 --- a/packages/core/src/db/schema.ts +++ b/packages/core/src/db/schema.ts @@ -72,7 +72,7 @@ export const transactions = sqliteTable("transactions", { // ── decisions ───────────────────────────────────────────── // Investment decisions, theses, retrospectives, and daily notes — anything // the user wants to reason about that's *not* a price-changing event. -// Stored in DB; `finsight ledger sync` mirrors each row to a markdown file +// Stored in DB; an explicit legacy `finsight ledger sync` can export each row // under `ledger/decisions/-.md`. export const decisions = sqliteTable("decisions", { id: text("id").primaryKey(), diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 34323df..31eac67 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -1,5 +1,6 @@ export * from "./db/schema.js"; export * from "./db/connection.js"; +export * from "./db/backup.js"; export * from "./types.js"; export * from "./services/account.js"; export * from "./services/position.js"; diff --git a/packages/core/src/ledger/read.ts b/packages/core/src/ledger/read.ts index 61ad754..2c2a340 100644 --- a/packages/core/src/ledger/read.ts +++ b/packages/core/src/ledger/read.ts @@ -48,8 +48,8 @@ function readJsonl(file: string): T[] { /** * Read snapshots, preferring `snapshots.jsonl`. Falls back to the legacy - * `snapshots/*.json` folder so older vaults still load (then `ledger sync` - * will rewrite as JSONL and remove the folder). + * `snapshots/*.json` folder so older exports still load (then an explicit + * legacy `ledger sync` rewrites as JSONL and removes the folder). */ function readSnapshots(p: ReturnType): LedgerSnapshotFile[] { if (existsSync(p.snapshots)) { diff --git a/packages/core/src/ledger/sync.ts b/packages/core/src/ledger/sync.ts index 969719c..a1f83e5 100644 --- a/packages/core/src/ledger/sync.ts +++ b/packages/core/src/ledger/sync.ts @@ -40,7 +40,7 @@ import { } from "./types.js"; /* ───────────────────────────────────────────────────────────────────────── - DB -> Ledger (one-shot export, used for initial migration) + DB -> Ledger (explicit legacy interoperability export) ─────────────────────────────────────────────────────────────────────── */ export function dumpDbToLedger(db: AppDatabase, root: string): { @@ -197,7 +197,7 @@ export function dumpDbToLedger(db: AppDatabase, root: string): { } /* ───────────────────────────────────────────────────────────────────────── - Ledger -> DB (rebuild cache on startup or via `finsight ledger rebuild`) + Ledger -> DB (explicit legacy, potentially lossy import) ─────────────────────────────────────────────────────────────────────── */ export function rebuildDbFromLedger(db: AppDatabase, root: string): { @@ -211,7 +211,8 @@ export function rebuildDbFromLedger(db: AppDatabase, root: string): { } { const ledger = readLedger(root); - // Wipe all tables first to ensure ledger is the SOLE truth. + // Legacy import replaces supported SQLite tables. Callers must require + // explicit confirmation and warn that the ledger is not lossless. db.delete(positions).run(); db.delete(transactions).run(); db.delete(snapshots).run(); diff --git a/packages/core/src/ledger/types.ts b/packages/core/src/ledger/types.ts index 9d7b805..1e0fdb0 100644 --- a/packages/core/src/ledger/types.ts +++ b/packages/core/src/ledger/types.ts @@ -1,7 +1,6 @@ /** - * File-first ledger schema. The vault directory is the single source of truth; - * the SQLite database is a derived cache. These types describe what lives - * inside `/projects/finsight/ledger/`. + * Legacy plain-text interoperability schema. SQLite is FinSight's sole source + * of truth; these types describe an explicitly exported/imported ledger. * * Format rules: * - YAML = stateful document or lookup dict (one current truth) @@ -18,8 +17,8 @@ * ├── reconciliations.jsonl ← broker-vs-computed reconciliation events * └── decisions/YYYY-MM/.md ← prose + frontmatter * - * Decision logs (markdown) live in `ledger/decisions/` and are NOT loaded - * into the DB — they're navigated via Obsidian / git. + * This representation is intentionally retained for backwards compatibility + * and is not a lossless native backup format. */ export const LEDGER_SCHEMA_VERSION = 1; diff --git a/packages/core/src/ledger/write.ts b/packages/core/src/ledger/write.ts index a4f47cb..9f79c47 100644 --- a/packages/core/src/ledger/write.ts +++ b/packages/core/src/ledger/write.ts @@ -20,33 +20,34 @@ import { type LedgerTransaction, } from "./types.js"; -const README_TEMPLATE = `# finsight ledger +const README_TEMPLATE = `# FinSight legacy ledger export -这是 finsight 的真相源（file-first）。SQLite 数据库 (~/.finsight/data/finsight.db) 只是从这里重建出来的派生缓存。 +SQLite (~/.finsight/data/finsight.db, unless configured otherwise) is FinSight's sole source of truth. +This directory is an explicitly generated, lossy interoperability format. It is not a native backup and is never synchronized or imported automatically. -## 文件格式约定 +## File formats -- **YAML** = stateful document / lookup dict（当前状态、人/AI 都能改） -- **JSONL** = append-only 时间序列（一行一个不可变事件；AI \`jq\` / grep 友好） -- **Markdown** = prose body + YAML frontmatter +- **YAML** = exported current state / lookup data +- **JSONL** = exported time-series rows +- **Markdown** = exported prose with YAML frontmatter -## 文件说明 +## Files -- \`accounts.yaml\` — 账户元信息 + 嵌套持仓（YAML — 当前状态） -- \`transactions.jsonl\` — 交易事件流，一行一笔 -- \`snapshots.jsonl\` — 每日净资产快照，一行一日（含 per-account/position 明细） -- \`fx-rates.jsonl\` — 汇率历史，一行一个 (date, from, to, rate) -- \`reconciliations.jsonl\` — broker-vs-computed 对账记录 -- \`decisions/YYYY-MM/.md\` — 决策日志，markdown body + YAML frontmatter +- \`accounts.yaml\` — exported account metadata + nested open positions +- \`transactions.jsonl\` — exported transactions +- \`snapshots.jsonl\` — exported net-worth snapshots +- \`fx-rates.jsonl\` — exported (date, from, to, rate) rows +- \`reconciliations.jsonl\` — exported broker-vs-computed reconciliation rows +- \`decisions/YYYY-MM/.md\` — exported decision entries -## 工作流 +## Commands -- \`finsight ledger init\` — 一次性配置 ledger 目录到 ~/.finsight/config.json -- \`finsight ledger export\` — 把当前 DB 一次性 dump 到 ledger（迁移用） -- \`finsight ledger rebuild\` — 强制从 ledger 重建 DB cache -- \`finsight ledger verify\` — 检查 ledger 和 DB cache 是否一致 +- \`finsight ledger init \` — configure this optional legacy directory +- \`finsight ledger sync\` / \`export\` — explicitly export SQLite data here +- \`finsight ledger verify\` — compare exported row counts with SQLite +- \`finsight ledger restore --yes\` — explicit, lossy import that wipes supported SQLite tables -每次 finsight 启动时会自动读 ledger 并重建 DB cache。所以你可以放心手编辑 \`accounts.yaml\` —— 下次任何 finsight 命令都会用最新内容。 +For recovery, use \`finsight backup create\` and verify the resulting native SQLite file with \`finsight backup verify \`. Do not treat this ledger as canonical recovery. `; export function writeAccountsFile( diff --git a/scripts/demo.mjs b/scripts/demo.mjs index f1eadf0..c7d71d8 100755 --- a/scripts/demo.mjs +++ b/scripts/demo.mjs @@ -2,8 +2,8 @@ /** * One-shot demo bootstrap: * 1. Build all packages (if dist is missing) - * 2. Spin up a fresh demo vault + DB in /tmp/finsight-demo - * 3. Load examples/seed-portfolio.{en,zh}.yaml + * 2. Spin up a fresh demo DB in /tmp/finsight-demo + * 3. Load synthetic examples/seed-portfolio.{en,zh}.yaml into SQLite * 4. Start the web dashboard * * Usage: @@ -20,11 +20,8 @@ import { fileURLToPath } from "node:url"; const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); const demoDir = "/tmp/finsight-demo"; const demoDb = path.join(demoDir, "finsight.db"); -const demoVault = path.join(demoDir, "vault"); const variant = (process.argv[2] ?? "en") === "zh" ? "zh" : "en"; -const example = path.join(repoRoot, "examples", `seed-portfolio.${variant}.yaml`); -const tickersExample = path.join(repoRoot, "examples", "tickers.cn.yaml"); function log(msg) { process.stdout.write(`\x1b[36m›\x1b[0m ${msg}\n`); @@ -68,10 +65,10 @@ if (existsSync(demoDir)) { log(`Wiping previous demo at ${demoDir}`); rmSync(demoDir, { recursive: true, force: true }); } -mkdirSync(demoVault, { recursive: true }); +mkdirSync(demoDir, { recursive: true }); // ── 3. Configure + load demo ────────────────────────────────────────────── -log("Bootstrapping demo (vault + config + seed data)…"); +log("Bootstrapping demo (SQLite + config + seed data)…"); const env = { ...process.env, FINSIGHT_DB_PATH: demoDb, @@ -87,8 +84,6 @@ exec("node", [ variant === "zh" ? "zh-CN" : "en-US", "--labels-language", variant, - "--ledger-dir", - demoVault, "--demo", variant, "--force", diff --git a/skills/finsight/SKILL.md b/skills/finsight/SKILL.md index e724896..680fc56 100644 --- a/skills/finsight/SKILL.md +++ b/skills/finsight/SKILL.md @@ -5,7 +5,7 @@ description: > multiple accounts/currencies, record salary deposits and consumption withdrawals, log investment decisions (rationale / target / stop-loss / retro), reconcile computed balances against broker apps, compute money-weighted annualized return - (XIRR), or sync everything to a plain-text vault. FinSight is a local-first + (XIRR), or create/verify local SQLite backups. FinSight is a local-first portfolio tracker — NOT a budgeting / expense-tracking tool. For budgeting, point the user at Beancount / Actual / YNAB instead. version: 0.1.0 @@ -16,17 +16,17 @@ version: 0.1.0 A driver's manual for operating FinSight on behalf of a user. FinSight is a local-first portfolio tracker; the CLI is the canonical interface (the web dashboard is a read-mostly view for the human). Every command supports -`--json`, exit codes are semantic, and the vault is plain-text — so an AI -agent can drive it end-to-end without ever needing a UI. +`--json`, exit codes are semantic, and SQLite is the authoritative local store +— so an AI agent can drive it end-to-end without ever needing a UI. ## Mental Model — Read This First -- **Source of truth = vault** (`/accounts.yaml` + `transactions.jsonl` + - `snapshots.jsonl` + `decisions/`). SQLite at `~/.finsight/data/finsight.db` - is a derived cache rebuilt from the vault. -- **DB ↔ vault sync** is one-way during normal use (DB → vault via - `finsight ledger sync`). Restore the other direction (`ledger restore`) - only for disaster recovery. +- **Source of truth = local SQLite** at `~/.finsight/data/finsight.db` (or the + configured `FINSIGHT_DB_PATH`). Normal CLI and web operations read and write + this database directly. +- **Native backups** use `finsight backup create` and + `finsight backup verify `. The optional plain-text ledger is legacy + interoperability only; it is never synchronized or imported automatically. - **In scope**: portfolio tracking, allocation, decision journal, broker reconciliation, performance/XIRR. - **Out of scope**: detailed expense categorization, monthly budgets, @@ -52,13 +52,14 @@ For first-run setup, run the interactive wizard: finsight init ``` -It picks base currency, locale, and the vault directory. If the user is -non-interactive (you're driving), set config explicitly: +It picks base currency and locale. A legacy ledger directory is optional. If +the user is non-interactive (you're driving), set config explicitly: ```bash finsight config set base-currency CNY # or USD, HKD, etc. finsight config set labels-language zh # or en -finsight config set ledger-dir /path/to/vault +# Optional legacy interoperability only: +finsight config set ledger-dir /path/to/legacy-export ``` ## Daily Workflow @@ -73,7 +74,8 @@ finsight trade deposit --date 2026-06-15 \ - `` is fuzzy-matched (partial substring works). - `` is positive in the account's native currency. - Always use `--date`; do NOT rely on "today" unless the user explicitly says so. -- `--note` is what shows up in the vault; keep it human-readable. +- `--note` is stored on the transaction; keep it human-readable for the CLI, + AI briefing, and any optional legacy export. ### When the user spends or transfers money OUT of the FinSight universe @@ -140,13 +142,20 @@ finsight reconcile --broker-total 162589.52 \ Logs the delta between FinSight's computed total and what the broker shows. Exit code 2 (DATA_CONFLICT) when the delta is large (>5%). -### Daily sync to vault (end of session) +### Create a native backup (when a recovery point is useful) ```bash -finsight ledger sync --json +finsight backup create --json ``` -Mirrors DB → vault. Commit the vault changes to git when convenient. +The result includes the backup path, integrity status, SHA-256, byte count, and +file mode. Verify an existing backup with: + +```bash +finsight backup verify ~/.finsight/backups/.sqlite3 --json +``` + +Do not use the optional legacy ledger as canonical recovery. ## Performance / Annualized Return @@ -176,13 +185,14 @@ Any account with 0 cashflows is dragging XIRR away from reality. ### Starting Fresh (when historical data is unreliable) ```bash +finsight backup create --json finsight ledger purge --before 2026-05-28 --yes --json -finsight ledger sync --json ``` Destructive: deletes transactions and snapshots before the cutoff. Leaves -accounts, positions, decisions, targets, reconciliations intact. Use when -imported historical data is incomplete enough to mislead XIRR. +accounts, positions, decisions, targets, reconciliations intact. Create a +native backup first. Use when imported historical data is incomplete enough to +mislead XIRR. ## Target Allocation @@ -213,6 +223,7 @@ finsight snapshot list --json finsight decision list --json finsight trade list --json finsight reconcile log --json +finsight doctor --json ``` `finsight context` is the single best command for "give the AI the user's @@ -234,7 +245,8 @@ whole portfolio state in one shot" — pipe it to your prompt. | `reconcile` | Log broker-vs-computed delta + `log` | | `snapshot` | `take / list / show / diff` | | `performance` | XIRR + total return per account + overall | -| `ledger` | `sync / restore / verify / purge / status / init` | +| `backup` | Create / verify native SQLite backups | +| `ledger` | `explicit legacy export / restore / verify / purge / status / init` | | `import` | `youzhiyouhang / md / yzyx-batch` | | `overview` | Single-shot summary card | | `context` | LLM-ready briefing (markdown or json) | @@ -261,8 +273,8 @@ In `--json` mode, errors are emitted to stderr as inside the portfolio; deposits cross the boundary into it. - **Don't record an internal transfer as deposit+withdraw** — use `transfer` so XIRR sees them as net-zero. -- **Don't edit the SQLite DB directly** — use the CLI. The vault is the - source of truth and CLI writes go through schema validation. +- **Don't edit the SQLite DB directly** — use the CLI. SQLite is the source of + truth and CLI writes go through schema validation. - **Don't omit `--date` for backdated events** — defaulting to today corrupts the time series. - **Don't import 转入转出 rows manually** — `finsight import yzyx-batch ` @@ -272,9 +284,10 @@ In `--json` mode, errors are emitted to stderr as coverage overstates returns. Either complete coverage or `ledger purge --before ` and start fresh. -## Vault Format +## Optional Legacy Ledger Format -After any `ledger sync`, the vault has this shape: +After an explicit `ledger sync` or `ledger export`, the optional legacy export +directory has this shape: ``` ledger/ @@ -287,9 +300,9 @@ ledger/ └── decisions/YYYY-MM/.md # markdown body + YAML frontmatter ``` -Format rule: **JSONL = append-only time series; YAML = stateful document; -Markdown = prose body with frontmatter**. The vault is meant to be -git-tracked. +Format rule: **JSONL = exported time-series rows; YAML = exported current +state; Markdown = exported prose with frontmatter**. This is an interoperability +format, not a native backup or automatic source of truth. ## Future Patterns (for context — not implemented yet)