From 6ec4b0da245c1c5c09c6aaf758d4817bf9a567d6 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 23 Sep 2026 12:17:00 +0000 Subject: [PATCH] docs: sync documentation with recent changes - README.md: the tier-2 benchmark CLI table and the eval-tiers paragraph still described --baseline as a flat accuracy-drop gate; add --max-regressions and --max-cost-increase to the flag list and describe the paired McNemar regression gate, noting --max-accuracy-drop is now deprecated. (#156) - docs/adapters/architecture.md, docs/extensions/add-adapter.md, docs/adapters/sdk.md: drop references to supports_schema_introspection, supports_dry_run and supports_cost_estimate, which no longer exist on DatasourceCapability (only supports_sql and supports_rest remain); fix the now-incorrect claim that fetch_schema_snapshot() is gated by a capability flag. (#163) - packages/nl2sql/README.md: the demo now seeds three SQLite databases (Chinook, support, webanalytics), not just Chinook (#157); Settings moved from a panel to its own routed page (#154); and nl2sql demo --hosted, a public-facing mode with no server-side key, was entirely undocumented here (#165). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01Bft4Niqsd2GWnPopt1VAhd --- README.md | 8 ++++++-- docs/adapters/architecture.md | 2 +- docs/adapters/sdk.md | 2 +- docs/extensions/add-adapter.md | 3 +-- packages/nl2sql/README.md | 16 ++++++++++++---- 5 files changed, 21 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index a6baaea2..f9d0bbdb 100644 --- a/README.md +++ b/README.md @@ -214,7 +214,7 @@ folder (`cd nl2sql-demo`). Every command has `--help`. | `nl2sql feedback export --good` | Write thumbs-up runs as draft gold entries to a separate YAML for review | `--out` (default `feedback_gold_drafts.yaml`) | | `nl2sql feedback clear` | Delete every rating | `--yes` | | `nl2sql benchmark --tier 1` | Tier 1: the hand-written gold plans through the validator, generator and executor, with a local fake LLM (no key) | `--role`, `--include-ids`, `--export-path` | -| `nl2sql benchmark --tier 2` | Tier 2: the real model end to end on the gold questions, per config, under a cost cap | `--max-cost USD` (required), `--model` (repeatable), `--llm PRESET\|PATH` (repeatable), `--passes`, `--questions`, `--note`, `--baseline`, `--results-dir` | +| `nl2sql benchmark --tier 2` | Tier 2: the real model end to end on the gold questions, per config, under a cost cap | `--max-cost USD` (required), `--model` (repeatable), `--llm PRESET\|PATH` (repeatable), `--passes`, `--questions`, `--note`, `--baseline`, `--max-regressions`, `--max-accuracy-drop` (deprecated), `--max-cost-increase`, `--results-dir` | | `nl2sql benchmark` | Without `--tier`: the gold questions through the full pipeline with the configured LLM | `--iterations` (default 3), `--bench-config-path`, `--role`, `--include-ids` | | `nl2sql benchmark retrieval` | Table and column recall of schema retrieval on the gold questions; no key, no LLM | `--record`, `--note`, `--baseline`, `--questions`, `--results-dir` | | `nl2sql benchmark presets` | List the built-in tier 2 LLM configs (`gpt-5.4`, `gpt-5.4-mini-helpers`, `claude-planner`) | | @@ -398,7 +398,11 @@ Details: [Security Model](docs/security/model.md), - **Tier 2** (real model, costs money): end to end per config, with accuracy, answerability precision and recall, cost, latency, determinism and answer faithfulness (whether the answer's numbers and names come from the rows). - Stops before spend could pass `--max-cost`; `--baseline` fails on a regression. + Stops before spend could pass `--max-cost`. `--baseline` pairs each run's + questions with the baseline's and fails when more than `--max-regressions` + (default 2) flip from pass to fail, or when a smaller drop is significant by + McNemar's test; the old flat-threshold `--max-accuracy-drop` gate still + works if passed explicitly, but is deprecated in favor of `--max-regressions`. - **Retrieval recall** (no key): the share of each question's needed tables and columns that schema retrieval sends the planner. - **Feedback:** playground ratings and guardrail rates via `nl2sql feedback stats`; diff --git a/docs/adapters/architecture.md b/docs/adapters/architecture.md index 56f8027b..dbb2dd95 100644 --- a/docs/adapters/architecture.md +++ b/docs/adapters/architecture.md @@ -42,7 +42,7 @@ See `sdk.md` for the authoritative adapter interface reference and required fiel ## Capability-driven routing -Adapters expose capabilities (e.g., `supports_sql`, `supports_schema_introspection`). The registry reads them once, at registration, and `DatasourceRegistry.supports(datasource_id, *capabilities)` is the one check every consumer asks. It fails closed: an unknown datasource, or an adapter whose `capabilities()` is missing or raises, supports nothing. These capabilities drive: +Adapters expose capabilities (`supports_sql`, `supports_rest`). The registry reads them once, at registration, and `DatasourceRegistry.supports(datasource_id, *capabilities)` is the one check every consumer asks. It fails closed: an unknown datasource, or an adapter whose `capabilities()` is missing or raises, supports nothing. These capabilities drive: - **Subgraph selection** (`resolve_subgraph()` in routing). - **Executor selection** (`ExecutorNode._supports_sql()` and `SqlExecutorService.validate_request()`, gating the SQL executor). diff --git a/docs/adapters/sdk.md b/docs/adapters/sdk.md index 4c78b759..fd0efb9a 100644 --- a/docs/adapters/sdk.md +++ b/docs/adapters/sdk.md @@ -40,7 +40,7 @@ Executes a plan-specific request and returns a normalized `ResultFrame`. #### `fetch_schema_snapshot()` -Required only if `supports_schema_introspection` is advertised. +Required by every adapter; there is no capability flag gating it. ### Optional Methods (SQL adapters) diff --git a/docs/extensions/add-adapter.md b/docs/extensions/add-adapter.md index 644a4a3f..de176d29 100644 --- a/docs/extensions/add-adapter.md +++ b/docs/extensions/add-adapter.md @@ -39,8 +39,7 @@ Secrets are resolved by `SecretManager` before the registry initializes. Ensure `capabilities()` advertises the correct set for routing and execution: - `supports_sql` if SQL execution is supported. -- `supports_schema_introspection` if schema can be fetched. -- `supports_dry_run` or `supports_cost_estimate` if available. +- `supports_rest` if REST execution is supported. ## Source references diff --git a/packages/nl2sql/README.md b/packages/nl2sql/README.md index ea98cfc4..8e71b284 100644 --- a/packages/nl2sql/README.md +++ b/packages/nl2sql/README.md @@ -31,18 +31,26 @@ pip install "nl2sql-engine[demo]" nl2sql demo ``` -That writes a demo project into `./nl2sql-demo` with the Chinook sample -database, indexes its schema locally (no key; the first run downloads a ~79 MB -ONNX embedding model) and opens the playground on . +That writes a demo project into `./nl2sql-demo`, copies in three SQLite +databases -- [Chinook](https://github.com/nadeem4/nl2sql/blob/main/THIRD_PARTY_NOTICES.md), +`support` and `webanalytics` -- indexes their schemas locally (no key; the +first run downloads a ~79 MB ONNX embedding model) and opens the playground on +. **Answering a question needs a model:** `--api-key`, or `OPENAI_API_KEY`, `OPENROUTER_API_KEY` or `ANTHROPIC_API_KEY` in the environment (Claude needs `pip install "nl2sql-engine[demo,anthropic]"`), a key pasted into the -playground's Settings panel, or a reachable Ollama. Without one the demo runs +playground's Settings page, or a reachable Ollama. Without one the demo runs in replay mode, which has no recorded answers out of the box, so it can show the schema and the index but answers nothing. `nl2sql demo --record` (with an OpenAI or OpenRouter key) records the guided questions for later key-free runs. +`nl2sql demo --hosted` runs a public-facing variant instead: the server holds +no API key, each visitor pastes their own into the page for that question +only, and Settings, Rebuild, ratings and `--record` are refused. See +[Hosted demo](https://github.com/nadeem4/nl2sql/blob/main/docs/deployment/hosted-demo.md) +in the full docs. + The playground shows each answer's plan, validation checks, SQL, rows and cost, a per-node Debug view, a Retrieval inspector over the live index, and a right/wrong rating per answer.