Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ folder (`cd nl2sql-demo`). Every command has `--help`.
| `nl2sql feedback export --good` | Write thumbs-up runs as draft gold entries to a separate YAML for review | `--out` (default `feedback_gold_drafts.yaml`) |
| `nl2sql feedback clear` | Delete every rating | `--yes` |
| `nl2sql benchmark --tier 1` | Tier 1: the hand-written gold plans through the validator, generator and executor, with a local fake LLM (no key) | `--role`, `--include-ids`, `--export-path` |
| `nl2sql benchmark --tier 2` | Tier 2: the real model end to end on the gold questions, per config, under a cost cap | `--max-cost USD` (required), `--model` (repeatable), `--llm PRESET\|PATH` (repeatable), `--passes`, `--questions`, `--note`, `--baseline`, `--results-dir` |
| `nl2sql benchmark --tier 2` | Tier 2: the real model end to end on the gold questions, per config, under a cost cap | `--max-cost USD` (required), `--model` (repeatable), `--llm PRESET\|PATH` (repeatable), `--passes`, `--questions`, `--note`, `--baseline`, `--max-regressions`, `--max-accuracy-drop` (deprecated), `--max-cost-increase`, `--results-dir` |
| `nl2sql benchmark` | Without `--tier`: the gold questions through the full pipeline with the configured LLM | `--iterations` (default 3), `--bench-config-path`, `--role`, `--include-ids` |
| `nl2sql benchmark retrieval` | Table and column recall of schema retrieval on the gold questions; no key, no LLM | `--record`, `--note`, `--baseline`, `--questions`, `--results-dir` |
| `nl2sql benchmark presets` | List the built-in tier 2 LLM configs (`gpt-5.4`, `gpt-5.4-mini-helpers`, `claude-planner`) | |
Expand Down Expand Up @@ -398,7 +398,11 @@ Details: [Security Model](docs/security/model.md),
- **Tier 2** (real model, costs money): end to end per config, with accuracy,
answerability precision and recall, cost, latency, determinism and answer
faithfulness (whether the answer's numbers and names come from the rows).
Stops before spend could pass `--max-cost`; `--baseline` fails on a regression.
Stops before spend could pass `--max-cost`. `--baseline` pairs each run's
questions with the baseline's and fails when more than `--max-regressions`
(default 2) flip from pass to fail, or when a smaller drop is significant by
McNemar's test; the old flat-threshold `--max-accuracy-drop` gate still
works if passed explicitly, but is deprecated in favor of `--max-regressions`.
- **Retrieval recall** (no key): the share of each question's needed tables and
columns that schema retrieval sends the planner.
- **Feedback:** playground ratings and guardrail rates via `nl2sql feedback stats`;
Expand Down
2 changes: 1 addition & 1 deletion docs/adapters/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ See `sdk.md` for the authoritative adapter interface reference and required fiel

## Capability-driven routing

Adapters expose capabilities (e.g., `supports_sql`, `supports_schema_introspection`). The registry reads them once, at registration, and `DatasourceRegistry.supports(datasource_id, *capabilities)` is the one check every consumer asks. It fails closed: an unknown datasource, or an adapter whose `capabilities()` is missing or raises, supports nothing. These capabilities drive:
Adapters expose capabilities (`supports_sql`, `supports_rest`). The registry reads them once, at registration, and `DatasourceRegistry.supports(datasource_id, *capabilities)` is the one check every consumer asks. It fails closed: an unknown datasource, or an adapter whose `capabilities()` is missing or raises, supports nothing. These capabilities drive:

- **Subgraph selection** (`resolve_subgraph()` in routing).
- **Executor selection** (`ExecutorNode._supports_sql()` and `SqlExecutorService.validate_request()`, gating the SQL executor).
Expand Down
2 changes: 1 addition & 1 deletion docs/adapters/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Executes a plan-specific request and returns a normalized `ResultFrame`.

#### `fetch_schema_snapshot()`

Required only if `supports_schema_introspection` is advertised.
Required by every adapter; there is no capability flag gating it.

### Optional Methods (SQL adapters)

Expand Down
3 changes: 1 addition & 2 deletions docs/extensions/add-adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,7 @@ Secrets are resolved by `SecretManager` before the registry initializes.
Ensure `capabilities()` advertises the correct set for routing and execution:

- `supports_sql` if SQL execution is supported.
- `supports_schema_introspection` if schema can be fetched.
- `supports_dry_run` or `supports_cost_estimate` if available.
- `supports_rest` if REST execution is supported.

## Source references

Expand Down
16 changes: 12 additions & 4 deletions packages/nl2sql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,26 @@ pip install "nl2sql-engine[demo]"
nl2sql demo
```

That writes a demo project into `./nl2sql-demo` with the Chinook sample
database, indexes its schema locally (no key; the first run downloads a ~79 MB
ONNX embedding model) and opens the playground on <http://127.0.0.1:8765/>.
That writes a demo project into `./nl2sql-demo`, copies in three SQLite
databases -- [Chinook](https://github.com/nadeem4/nl2sql/blob/main/THIRD_PARTY_NOTICES.md),
`support` and `webanalytics` -- indexes their schemas locally (no key; the
first run downloads a ~79 MB ONNX embedding model) and opens the playground on
<http://127.0.0.1:8765/>.

**Answering a question needs a model:** `--api-key`, or `OPENAI_API_KEY`,
`OPENROUTER_API_KEY` or `ANTHROPIC_API_KEY` in the environment (Claude needs
`pip install "nl2sql-engine[demo,anthropic]"`), a key pasted into the
playground's Settings panel, or a reachable Ollama. Without one the demo runs
playground's Settings page, or a reachable Ollama. Without one the demo runs
in replay mode, which has no recorded answers out of the box, so it can show the
schema and the index but answers nothing. `nl2sql demo --record` (with an
OpenAI or OpenRouter key) records the guided questions for later key-free runs.

`nl2sql demo --hosted` runs a public-facing variant instead: the server holds
no API key, each visitor pastes their own into the page for that question
only, and Settings, Rebuild, ratings and `--record` are refused. See
[Hosted demo](https://github.com/nadeem4/nl2sql/blob/main/docs/deployment/hosted-demo.md)
in the full docs.

The playground shows each answer's plan, validation checks, SQL, rows and cost,
a per-node Debug view, a Retrieval inspector over the live index, and a
right/wrong rating per answer.
Expand Down
Loading