Added generic and database specific skills and debugged validator tool by akshayravi13 · Pull Request #20 · SVijayB/sequel2sql

akshayravi13 · 2026-03-03T05:23:05Z

Description

This PR is a cumulative integration of work across several feature branches (generic-skills → clean-benchmark → validator-final-fix → testing-confirmed-fix). It introduces a complete tooling overhaul for the Sequel2SQL agent — adding validated SQL fix retrieval, a reworked AST validator, per-database semantic model skills, an error taxonomy knowledge base, and a restructured benchmark pipeline.

🔧 Validator Overhaul (`src/ast_parsers/`)

Rewrote the SQL validator to use EXPLAIN on the live database instead of fragile AST-only parsing.
Introduced a new tag-based error system (tags.py) replacing the old error_codes.py / errors.py / error_context.py modules (all deleted).
New result.py model with ValidationResult and ValidationErrorOut — cleaner Pydantic output.
Edge-case handling: statement timeouts in _explain no longer surface as SQL errors; schema cache invalidation after preprocess_sql ensures dynamically created tables are recognized.
Comprehensive validator benchmarks added (tests/run_validator_benchmark.py, tests/run_validate_with_db_benchmark.py) with full results in tests/output/.

🧠 Semantic Model Skills (`src/skills/`)

Added per-database semantic model skills for all 10 benchmark databases (financial, card_games, codebase_community, etc.).
Each skill includes a SKILL.md entrypoint plus domain-specific references: gotchas.md, metrics.md, and query_patterns.md.
Skills are loaded via pydantic-ai-skills using load_skill('<db>-semantic-model').
New skills_config.py filters instructions to only the relevant DB skill, saving ~350–400 tokens per run.

📚 Error Taxonomy (`src/error_taxonomy/`)

Created 10 taxonomy skill files covering all error categories: aggregation, syntax, join_related, semantic, logical, structural, filter_conditions, set_operations, subquery_formulation, value_representation.
generic_skills.py provides get_error_taxonomy_skill(category) — looks up best-practice fix strategies by category.
Registered as a tool on both benchmark and webui agents.

🗃️ Confirmed Fixes Knowledge Base (`src/db_confirmed_fixes/`)

db_confirmed_fixes.json — 50 curated (intent → error SQL → corrected SQL → explanation) entries mapped to their benchmark db_id.
info.py — one-time seeder script that embeds intents into per-database ChromaDB instances under chroma/<db_id>/.
retriever.py — find_similar_confirmed_fixes() returns top-4 semantically similar fixes (no similarity threshold — raw ranking). save_confirmed_fix() with deduplication and pruning.
Retrieval test (tests/test_confirmed_fix_retrieval.py): 86% top-1 accuracy, 100% top-4 recall across all 50 entries.

🤖 Agent & Prompt Updates (`src/agent/`)

Benchmark agent now has 4 tools: execute_sql_query, validate_query, get_error_taxonomy_skill, find_similar_confirmed_fixes_tool.
WebUI agent has 7 tools (above 4 + analyze_and_fix_sql, describe_database_schema, save_confirmed_fix_tool).
analyze_and_fix_sql fully restored — performs validation, schema-aware table filtering, similar example retrieval, taxonomy lookups, and confirmed fix retrieval.
Benchmark prompt rewritten: describes all 4 tools, adds a Skills section for semantic models, frames validate_query and confirmed fixes as advisory guidance (not literal truth).
WebUI and base prompts updated with confirmed fixes awareness.

📊 Benchmark Pipeline (`benchmark/`)

Added sequel2sql_client.py — a dedicated client for running the Sequel2SQL agent pipeline through the benchmark harness.
Updated config.py, api_client.py, ui.py, and main.py for the new pipeline integration.
Pydantic model output (BenchmarkOutput) for structured agent responses.

Related Issue

This PR closes #

Motivation and Context

The Sequel2SQL agent previously lacked structured knowledge about common SQL error patterns and had no way to leverage previously confirmed fixes. The validator was brittle (AST-only, no live database checking), and there was no per-database semantic model to guide query repair.

This PR establishes a complete knowledge-augmented pipeline:

Semantic models give the agent domain-specific awareness per database.
Error taxonomy provides structured fix strategies by error category.
Confirmed fixes act as a curated RAG layer — real, validated corrections retrievable by intent similarity.
A robust validator catches syntax and schema errors against the live DB, with proper edge-case handling.

Together, these give the agent significantly richer context for producing correct SQL fixes in both benchmark and production (webui) modes.

Screenshots (if any)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.

…to display more benchmark info

…enchmark

…final-fix

…l mistmatches

Copilot

Pull request overview

This PR expands Sequel2SQL with database-specific semantic-model skills, an error-taxonomy knowledge base, and a confirmed-fixes (ChromaDB) retrieval layer, while also refactoring validator/result modeling and updating the benchmark harness to use the revised agent pipeline.

Changes:

Added per-database semantic-model skill packs (SKILL.md + reference resources) and wiring for skills loading.
Introduced error taxonomy markdown “skills” plus a retrieval helper (get_error_taxonomy_skill).
Added confirmed-fix persistence/retrieval via ChromaDB, updated prompts/benchmark pipeline, and refactored validator result models.

Reviewed changes

Copilot reviewed 94 out of 107 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
uv.lock	Adds `pydantic-ai-skills` lock entry
pyproject.toml	Adds `pydantic-ai-skills` dependency + dev group
.gitignore	Ignores confirmed-fixes chroma dir
tests/test_validator_fixes.py	Adds validator regression tests
tests/test_skills_loading.py	Adds skills toolset tests
tests/test_confirmed_fix_retrieval.py	Adds confirmed-fix retrieval test script
tests/inspect_chroma_db.py	Updates chroma inspection script
src/skills/toxicology-semantic-model/SKILL.md	Adds toxicology semantic skill
src/skills/toxicology-semantic-model/references/gotchas.md	Toxicology gotchas
src/skills/toxicology-semantic-model/references/metrics.md	Toxicology metrics
src/skills/toxicology-semantic-model/references/query_patterns.md	Toxicology query patterns
src/skills/superhero-semantic-model/SKILL.md	Adds superhero semantic skill
src/skills/superhero-semantic-model/references/gotchas.md	Superhero gotchas
src/skills/superhero-semantic-model/references/metrics.md	Superhero metrics
src/skills/superhero-semantic-model/references/query_patterns.md	Superhero query patterns
src/skills/student-club-semantic-model/SKILL.md	Adds student_club semantic skill
src/skills/student-club-semantic-model/references/gotchas.md	Student club gotchas
src/skills/student-club-semantic-model/references/metrics.md	Student club metrics
src/skills/student-club-semantic-model/references/query_patterns.md	Student club query patterns
src/skills/formula-1-semantic-model/SKILL.md	Adds formula_1 semantic skill
src/skills/formula-1-semantic-model/references/gotchas.md	Formula 1 gotchas
src/skills/formula-1-semantic-model/references/metrics.md	Formula 1 metrics
src/skills/formula-1-semantic-model/references/query_patterns.md	Formula 1 query patterns
src/skills/financial-semantic-model/SKILL.md	Adds financial semantic skill
src/skills/financial-semantic-model/references/gotchas.md	Financial gotchas
src/skills/financial-semantic-model/references/metrics.md	Financial metrics
src/skills/financial-semantic-model/references/query_patterns.md	Financial query patterns
src/skills/european-football-2-semantic-model/SKILL.md	Adds european_football_2 semantic skill
src/skills/european-football-2-semantic-model/references/gotchas.md	European football gotchas
src/skills/european-football-2-semantic-model/references/metrics.md	European football metrics
src/skills/european-football-2-semantic-model/references/query_patterns.md	European football query patterns
src/skills/debit-card-specializing-semantic-model/SKILL.md	Adds debit_card_specializing semantic skill
src/skills/debit-card-specializing-semantic-model/references/gotchas.md	Debit card gotchas
src/skills/debit-card-specializing-semantic-model/references/metrics.md	Debit card metrics
src/skills/debit-card-specializing-semantic-model/references/query_patterns.md	Debit card query patterns
src/skills/codebase-community-semantic-model/SKILL.md	Adds codebase_community semantic skill
src/skills/codebase-community-semantic-model/references/gotchas.md	Codebase community gotchas
src/skills/codebase-community-semantic-model/references/metrics.md	Codebase community metrics
src/skills/codebase-community-semantic-model/references/query_patterns.md	Codebase community query patterns
src/skills/card-games-semantic-model/SKILL.md	Adds card_games semantic skill
src/skills/card-games-semantic-model/references/gotchas.md	Card games gotchas
src/skills/card-games-semantic-model/references/metrics.md	Card games metrics
src/skills/card-games-semantic-model/references/query_patterns.md	Card games query patterns
src/skills/california-schools-semantic-model/SKILL.md	Adds california_schools semantic skill
src/skills/california-schools-semantic-model/references/gotchas.md	California schools gotchas
src/skills/california-schools-semantic-model/references/metrics.md	California schools metrics
src/skills/california-schools-semantic-model/references/query_patterns.md	California schools query patterns
src/query_intent_vectordb/search_similar_query.py	Adjusts distance→similarity conversion
src/query_intent_vectordb/embed_query_intent.py	Sets Chroma cosine space metadata
src/error_taxonomy/init.py	Adds taxonomy package init
src/error_taxonomy/generic_skills.py	Adds taxonomy skill loader
src/error_taxonomy/aggregation.md	Adds aggregation taxonomy guidance
src/error_taxonomy/filter_conditions.md	Adds filter taxonomy guidance
src/error_taxonomy/join_related.md	Adds join taxonomy guidance
src/error_taxonomy/logical.md	Adds logical taxonomy guidance
src/error_taxonomy/semantic.md	Adds schema-link taxonomy guidance
src/error_taxonomy/set_operations.md	Adds set-ops taxonomy guidance
src/error_taxonomy/structural.md	Adds select/structural taxonomy guidance
src/error_taxonomy/subquery_formulation.md	Adds subquery taxonomy guidance
src/error_taxonomy/syntax.md	Adds syntax taxonomy guidance
src/error_taxonomy/value_representation.md	Adds literal/value taxonomy guidance
src/db_confirmed_fixes/init.py	Adds confirmed-fixes package init
src/db_confirmed_fixes/README.md	Documents confirmed-fixes store
src/db_confirmed_fixes/db_confirmed_fixes.json	Adds seeded confirmed-fix corpus
src/db_confirmed_fixes/info.py	Adds seeding script
src/db_confirmed_fixes/retriever.py	Adds save/find/prune logic
src/database/database.py	Changes SQL execution guard behavior
src/ast_parsers/result.py	Introduces unified Pydantic result models
src/ast_parsers/init.py	Updates ast_parsers public API exports
src/agent/skills_config.py	Adds skills toolset + DB-skill instructions helper
src/agent/prompts/base_prompt.py	Updates tool/policy guidance text
src/agent/prompts/webui_prompt.py	Adds confirmed-fix workflow instructions
src/agent/prompts/benchmark_prompt.py	Rewrites benchmark prompt as standalone
benchmark/src/ui.py	Adds single-query mode + counts update
benchmark/src/sequel2sql_client.py	Runs full pipeline; adds usage limits & preprocess/cleanup execution
benchmark/src/prompt_generator.py	Adds single-record prompt extraction
benchmark/src/inference_engine.py	Adds inter-task sleep
benchmark/src/config.py	Updates provider configs (Gemini/Codestral)
benchmark/src/checkpoint_manager.py	Persists run metadata in checkpoint
benchmark/src/api_client.py	Updates provider description text
benchmark/main.py	Adds interactive single-query mode + run_config persistence
benchmark/README.md	Updates benchmark README model references
benchmark.sh	Updates model list output
src/chroma_db/f60bef43-d232-42a0-b54c-c354655c0ae1/length.bin	Adds Chroma persistence artifact
src/chroma_db/f60bef43-d232-42a0-b54c-c354655c0ae1/header.bin	Adds Chroma persistence artifact
src/chroma_db/f60bef43-d232-42a0-b54c-c354655c0ae1/link_lists.bin	Adds Chroma persistence artifact
src/chroma_db/be58dce8-6c8c-4bad-8bca-670bb1820eb2/length.bin	Adds Chroma persistence artifact

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-03T05:38:11Z

+        candidates = []
+        ids_to_increment = []
+        metadatas_to_update = []
+
+        for doc, meta, dist, doc_id in zip(docs, metas, dists, ids):
+            similarity = 1.0 - dist
+            candidates.append(
+                {
+                    "intent": doc,
+                    "error_sql": meta.get("error_sql", ""),
+                    "corrected_sql": meta.get("corrected_sql", ""),
+                    "explanation": meta.get("explanation", ""),
+                    "similarity": round(similarity, 4),
+                }
+            )
+        # Best-effort increment usage_count
+        if ids_to_increment:
+            try:
+                collection.update(
+                    ids=ids_to_increment,
+                    metadatas=metadatas_to_update
+                )
+            except Exception:
+                pass


find_similar_confirmed_fixes declares ids_to_increment / metadatas_to_update and has a "Best-effort increment usage_count" block, but those lists are never populated. As a result, usage_count is never updated even though the function appears to support it. Either implement the increment logic (e.g., read current usage_count from meta, increment, and queue updates) or remove the dead code to avoid misleading behavior.

Copilot · 2026-03-03T05:38:12Z

+Each database gets its own ChromaDB collection under `chroma/<database_name>/`. Fixes are embedded by their **intent** (natural language description) using ChromaDB's default sentence-transformer model.
+
+### Core Functions (`store.py`)
+
+| Function | Description |
+|---|---|
+| `save_confirmed_fix()` | Stores a confirmed fix with deduplication (intent similarity ≥ 0.8) |
+| `find_similar_confirmed_fixes()` | Retrieves up to N fixes with cosine similarity ≥ 0.75 |
+| `prune_confirmed_fixes()` | Tiered cleanup when collection exceeds 500 items |
+


This README describes store.py under a db_skills/ directory, but the implementation in this PR lives under src/db_confirmed_fixes/ (not db_skills) and the main module is retriever.py (plus info.py). Please update the module names and directory structure in the documentation so it matches the actual code layout.

Copilot · 2026-03-03T05:38:13Z

+def test_db_skill_hint_present():
+	"""get_db_skill_hint returns a load_skill hint when a skill exists for the DB."""
+	from src.agent.skills_config import get_db_skill_hint
+	hint = get_db_skill_hint("california_schools_template")
+	assert hint != ""
+	assert "load_skill" in hint
+	assert "california-schools-template-semantic-model" in hint
+
+
+def test_db_skill_hint_absent():
+	"""get_db_skill_hint returns empty string when no skill exists for the DB."""
+	from src.agent.skills_config import get_db_skill_hint
+	hint = get_db_skill_hint("nonexistent_db")
+	assert hint == ""


This test imports get_db_skill_hint from src.agent.skills_config, but skills_config.py defines get_db_skill_instructions (no get_db_skill_hint symbol). As written, the test will fail at import time. Either export a get_db_skill_hint alias (and keep its previous semantics), or update the tests (and any call sites) to use the new function name consistently.

Copilot · 2026-03-03T05:38:13Z

+def test_skill_discovery():
+	"""SkillsToolset discovers all expected skills in the skills directory."""
+	toolset = SkillsToolset(directories=[str(SKILLS_DIR)])
+	skill_names = list(toolset.skills.keys())
+	assert "db-query" in skill_names, f"db-query skill not found. Found: {skill_names}"
+	assert "california-schools-template-semantic-model" in skill_names, (
+		f"california-schools-template-semantic-model not found. Found: {skill_names}"
+	)


test_skill_discovery asserts that a skill named california-schools-template-semantic-model exists, but the skills directory in this PR contains california-schools-semantic-model (and no *-template-* variant). This will make the discovery test fail (and also suggests the DB→skill naming convention is inconsistent). Align the skill name: in SKILL.md (or the test expectations) so the generated DB skill name matches what SkillsToolset actually discovers.

aravindh28 · 2026-03-03T05:48:08Z

@copilot can you make the changes for all the md files that you suggested? (the example names under skills)

Copilot · 2026-03-03T05:48:15Z

@aravindh28 I've opened a new pull request, #22, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: aravindh28 <54079536+aravindh28@users.noreply.github.com>

Align SKILL.md `read_skill_resource` examples with actual skill IDs

…ncrement - save_confirmed_fix: initialize results = None before the conditional query block so the dedup check behaves deterministically instead of relying on the broad except to mask NameError - find_similar_confirmed_fixes: populate ids_to_increment and metadatas_to_update so usage_count is actually incremented on each retrieval (was previously dead code)

- Rename references from db_skills/store.py to db_confirmed_fixes/retriever.py - Add info.py and db_confirmed_fixes.json to documentation - Update directory tree to reflect current structure

- Point CHROMA_BASE at src/db_confirmed_fixes/chroma/ instead of the old src/db_skills/chroma/ path - Support inspecting all DBs (no args) or a specific one (CLI arg)

- Replace weak DDL-only blocklist with allowlist (SELECT/WITH/EXPLAIN) - Block DML (INSERT/UPDATE/DELETE) and all DDL/utility commands - Use word-boundary regex to avoid false positives on column names - Strip string literals before scanning to handle WHERE x = 'DELETE' - Strip SQL comments before checking first token

SVijayB · 2026-03-03T15:15:36Z

Working on this branch atm, will test all changes with new benchmarking scores and then merge to main

SVijayB and others added 30 commits February 17, 2026 23:00

Added code for error taxonomy and two new skills

278d6ae

added Mistral model

0ba0f37

Merge branch 'main' into generic-skills

9ba7ba9

Fixed ast - moved to using explain on live env

73c0732

Fixed retry attempts and removed forced SELECT queries, updated logs …

a94a53e

…to display more benchmark info

Updated sql agent to get full schema if all tables are wrong when called

92e1b5d

Implemented single execute mode

39e6f9e

Updated claude imd

664cef0

Switched from gemma3 to gemini

d0ca994

updated query intent metadata with new query analyzer

5211929

added db specific skills

333deb6

replaced generic skill with database skill for storing corrected queries

ef2cc78

adjusted similarity search metric

51a6933

added db skills benchmark

7df80c0

removed validate query and added timeout per query

f95e905

db specific skills addition

87fba4b

Added semantic model skills for top 50 query DB

9b3965f

uv lock

da27d0e

Rename California semantic model folder name.

6f52ca9

Merge remote-tracking branch 'origin/db-specific-skills' into clean-b…

c31865c

…enchmark

modified benchmark prompt and added usage limit handling

c56cf52

removed similar example tools, increased sleep between tool calls

a0b061a

v1 of skills revision

298017e

removed 'template' name

cc14ffe

added codestral model option

8857533

Merge remote-tracking branch 'origin/clean-benchmark' into validator-…

86ef8a8

…final-fix

handled edge cases for validator

c6bec4d

addded skill path issue fix for windows

e33d914

benchmark prompt change

8f9d58e

minor bm prompt change

743dc6d

aravindh28 and others added 7 commits February 28, 2026 16:48

removed tool context bloat, moved error-taxonomy, fixed semantic mode…

68c8c38

…l mistmatches

revert temp and tool use prompt

1cbd776

pydantic model for bm agent output

465a114

added explain timeout handling and SRE handling

3b2a4c8

reworked db confirmed fixes testing suite

dc26aad

added benchmark test for db confirmed fix retrieval

f59c62c

added all tools to benchmark

f859355

akshayravi13 requested review from SVijayB and aravindh28 March 3, 2026 05:23

Copilot AI mentioned this pull request Mar 3, 2026

[WIP] Add generic and database specific skills and debug validator tool #21

Closed

6 tasks

Repository owner deleted a comment from Copilot AI Mar 3, 2026

aravindh28 requested a review from Copilot March 3, 2026 05:29

Copilot started reviewing on behalf of aravindh28 March 3, 2026 05:30 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

Initial plan

169b794

Copilot AI mentioned this pull request Mar 3, 2026

Align SKILL.md read_skill_resource examples with actual skill IDs #22

Merged

Copilot AI and others added 6 commits March 3, 2026 05:50

Fix skill markdown example names to match skill IDs

074c429

Co-authored-by: aravindh28 <54079536+aravindh28@users.noreply.github.com>

Merge pull request #22 from SVijayB/copilot/sub-pr-20-again

ed27686

Align SKILL.md `read_skill_resource` examples with actual skill IDs

docs: update README to match actual module layout

417394f

- Rename references from db_skills/store.py to db_confirmed_fixes/retriever.py - Add info.py and db_confirmed_fixes.json to documentation - Update directory tree to reflect current structure

fix: update inspect_chroma_db to use correct db_confirmed_fixes path

6afef6f

- Point CHROMA_BASE at src/db_confirmed_fixes/chroma/ instead of the old src/db_skills/chroma/ path - Support inspecting all DBs (no args) or a specific one (CLI arg)

SVijayB added 4 commits March 3, 2026 09:46

added nvidea nim integration

207b990

Added default deepseek for testing

74da170

complete prompt rework

62e4442

Added select statement only type queries and improved pre-processing

90bbfbd

SVijayB merged commit 78f895f into main Mar 10, 2026

Navigation Menu

Added generic and database specific skills and debugged validator tool - #20

Added generic and database specific skills and debugged validator tool#20
SVijayB merged 49 commits into
mainfrom
testing-confirmed-fix

akshayravi13 commented Mar 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Uh oh!

Uh oh!

aravindh28 commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

SVijayB commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

akshayravi13 commented Mar 3, 2026

Description

🔧 Validator Overhaul (src/ast_parsers/)

🧠 Semantic Model Skills (src/skills/)

📚 Error Taxonomy (src/error_taxonomy/)

🗃️ Confirmed Fixes Knowledge Base (src/db_confirmed_fixes/)

🤖 Agent & Prompt Updates (src/agent/)

📊 Benchmark Pipeline (benchmark/)

Related Issue

Motivation and Context

Screenshots (if any)

Types of changes

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aravindh28 commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

SVijayB commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

🔧 Validator Overhaul (`src/ast_parsers/`)

🧠 Semantic Model Skills (`src/skills/`)

📚 Error Taxonomy (`src/error_taxonomy/`)

🗃️ Confirmed Fixes Knowledge Base (`src/db_confirmed_fixes/`)

🤖 Agent & Prompt Updates (`src/agent/`)

📊 Benchmark Pipeline (`benchmark/`)