ci: benchmark and plan-guard lint changes against base

.claude/skills/new-lint/SKILL.md

@@ -283,6 +283,7 @@ Then check:
 - [ ] `docs/XXXX_<name>.md` exists
 - [ ] `test/sql/XXXX_<name>.sql` exists
 - [ ] `test/expected/XXXX_<name>.out` exists
+- [ ] `bench/fixture/` creates objects the new lint reports, so the benchmark measures it on a non-empty result
 
 ---
 

.github/workflows/bench.yml

@@ -0,0 +1,144 @@
+name: benchmark
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+    paths:
+      - "lints/**"
+      - "bin/compile.py"
+      - "bench/**"
+      - ".github/workflows/bench.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: bench-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  measure:
+    name: Measure (${{ matrix.target }})
+    # Label events other than the two waivers would only repeat the last run.
+    if: >-
+      github.event_name != 'pull_request'
+      || !contains(fromJSON('["labeled","unlabeled"]'), github.event.action)
+      || contains(fromJSON('["perf-accepted","lint-results-changed"]'), github.event.label.name)
+    strategy:
+      fail-fast: false
+      matrix:
+        # Small schemas only check rows: their timings sit inside runner noise.
+        include:
+          - { target: synthetic, rounds: 5 }
+          - { target: carbon, rounds: 5 }
+          - { target: deskcomm, rounds: 5 }
+          - { target: basejump, rounds: 0 }
+          - { target: atomic-crm, rounds: 0 }
+          - { target: wacrm, rounds: 0 }
+          - { target: grida, rounds: 0 }
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    services:
+      postgres:
+        image: supabase/postgres:17.6.1.169
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd "pg_isready -U postgres -h localhost"
+          --health-interval 5s --health-timeout 5s --health-retries 20
+    env:
+      PGHOST: localhost
+      PGPORT: "5432"
+      PGPASSWORD: postgres
+    steps:
+      - name: Checkout head
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          path: head
+          persist-credentials: false
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+
+      - name: Checkout base
+        if: github.event_name == 'pull_request'
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          path: base
+          persist-credentials: false
+          ref: ${{ github.event.pull_request.base.sha }}
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Measure
+        env:
+          TARGET: ${{ matrix.target }}
+          ROUNDS: ${{ matrix.rounds }}
+          BASE: ${{ github.event_name == 'pull_request' && 'base' || '' }}
+        run: |
+          args=(--head head --rounds "$ROUNDS" --out "results-$TARGET.json")
+          if [ -n "$BASE" ]; then args+=(--base "$BASE"); fi
+          if [ "$TARGET" != synthetic ]; then
+            migrations=$(python head/bench/bench.py fetch "$TARGET" --dest "project-$TARGET")
+            args+=(--project "$TARGET=$migrations")
+          fi
+          python head/bench/bench.py measure "${args[@]}"
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: bench-${{ matrix.target }}
+          path: results-${{ matrix.target }}.json
+
+  verdict:
+    name: Verdict
+    needs: measure
+    if: always() && needs.measure.result != 'skipped'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+    steps:
+      - name: Checkout head
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+          sparse-checkout: bench
+          persist-credentials: false
+
+      - uses: actions/download-artifact@v4
+        with:
+          pattern: bench-*
+          merge-multiple: true
+
+      - name: Compare
+        env:
+          ACCEPT_PERF: ${{ contains(github.event.pull_request.labels.*.name, 'perf-accepted') && '--accept-perf' || '' }}
+          ACCEPT_RESULTS: ${{ contains(github.event.pull_request.labels.*.name, 'lint-results-changed') && '--accept-results' || '' }}
+          MEASURED: ${{ needs.measure.result }}
+        run: |
+          set +e
+          # shellcheck disable=SC2086 # the waiver flags are empty or one word
+          python3 bench/bench.py compare results-*.json --markdown report.md $ACCEPT_PERF $ACCEPT_RESULTS
+          status=$?
+          if [ "$MEASURED" != success ]; then
+            printf '\n**A measure job failed**, so some targets are missing above.\n' >> report.md
+            status=1
+          fi
+          cat report.md >> "$GITHUB_STEP_SUMMARY"
+          exit $status
+
+      - name: Comment on PR
+        # Fork PRs get a read-only token; they still have the job summary.
+        if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && hashFiles('report.md') != ''
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR: ${{ github.event.pull_request.number }}
+        run: |
+          id=$(gh api "repos/$GITHUB_REPOSITORY/issues/$PR/comments" --paginate \
+            --jq '.[] | select(.body | startswith("<!-- splinter-bench -->")) | .id' | head -n1)
+          if [ -n "$id" ]; then
+            gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$id" -F body=@report.md >/dev/null
+          else
+            gh api "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@report.md >/dev/null
+          fi

README.md

@@ -95,3 +95,7 @@ To run the test suite:
 ```sh
 docker rmi -f dockerfiles-test && SUPABASE_VERSION=15.1.1.13 docker-compose -f dockerfiles/docker-compose.yml run --rm test
 ```
+
+## Benchmark
+
+PRs that change a lint are benchmarked against their base on a synthetic schema at three sizes and on real open-source Supabase projects. The check fails when findings change, a lint errors, or its plan, buffer reads or run time regress. See [bench/README.md](bench/README.md).

bench/README.md

@@ -0,0 +1,59 @@
+# Lint benchmark
+
+Times every lint, and the full `splinter.sql` union, on Supabase schemas, comparing a base checkout with a head checkout. CI runs it on every PR that touches `lints/`, `bin/compile.py` or `bench/`. Each target runs as its own job on `supabase/postgres`, and one PR comment collects the results and is updated on every run.
+
+## Targets
+
+- **synthetic**: `fixture/` built at 2,000 tables, a scaling stress test that surfaces lints whose cost grows faster than the catalog. Every lint has findings. `--sizes small=20,large=2000` builds several sizes cumulatively on one database.
+- **Real projects**: the open-source apps from the dogfooding corpus, fetched from GitHub at the commits pinned in `projects/projects.json`. Their `supabase/migrations` are applied in order on the bare image, after `projects/services-stub.sql` stands in for the tables storage-api, GoTrue and realtime would create. Each migration runs in one transaction; one that still fails is rolled back, skipped and listed in the report.
+
+| target | tables | policies | `pg_depend` rows | migrations applied |
+| -- | --: | --: | --: | --: |
+| synthetic / large | 2,197 | 2,736 | 64k | – |
+| carbon | 520 | 1,501 | 41k | 1,230 / 1,230 |
+| deskcomm | 283 | 669 | 17k | 520 / 530 (baseline dump + migrations) |
+| grida | 174 | 89 | 7k | 76 / 87 (timescaledb is not in the image) |
+| wacrm | 113 | 114 | 5k | 54 / 54 |
+| atomic-crm | 88 | 36 | 3k | 28 / 28 |
+| basejump | 82 | 13 | 2k | 4 / 4 |
+
+Table counts include the image's own schemas.
+
+Only the large targets are timed. The small projects run with `--rounds 0`: their full run takes 50–100 ms, mostly fixed overhead, and per-lint differences stay inside runner noise, so they only check that every lint returns the same rows as base. Their real-world SQL still catches rewrites that break on patterns the fixture lacks.
+
+## What fails the check
+
+| Guard | Rule | Waiver label |
+| -- | -- | -- |
+| Lint errors on head | Any error | none |
+| Results | Row count and md5 of the sorted rows (as `jsonb`, so column names count) differ from base | `lint-results-changed` |
+| Plan | Head's plan gains a catalog scan that runs once per outer row and reads more than 50 blocks per loop, i.e. a whole catalog or index instead of an index probe | `perf-accepted` |
+| Buffers | `EXPLAIN (ANALYZE, BUFFERS)` reads at least 10% and 100 blocks more than base | `perf-accepted` |
+| Wall time | The median is at least 15% and 3 ms slower, and every head round is slower than every base round | `perf-accepted` |
+| Base errors | The lint errors on base, so there is nothing to compare | `perf-accepted` |
+
+Buffer counts are deterministic, so they catch plan regressions that are too small to see in wall time on a noisy runner. The plan pass runs twice and keeps the second, so catalog-cache warm-up does not count. Wall time runs base and head in interleaved rounds on the same database, alternating which goes first, and is measured server-side around the query.
+
+## When you change the benchmark
+
+- When you add a lint, extend `fixture/` so it has findings.
+- A project whose initial schema is a dump outside its migrations sets `baseline` (applied first) and, for the extensions the dump assumes, `setup` in `projects/projects.json`.
+- When you add a project, pin a commit in `projects/projects.json` and add it to the matrix in `.github/workflows/bench.yml`, timed (`rounds: 5`) only if it is large enough for lint timings to move beyond noise.
+
+## Run locally
+
+```bash
+docker run -d --name splinter-bench -e POSTGRES_PASSWORD=postgres -p 54329:5432 supabase/postgres:17.6.1.169
+git worktree add /tmp/splinter-base origin/main
+export PGHOST=localhost PGPORT=54329 PGPASSWORD=postgres
+python3 bench/bench.py measure --base /tmp/splinter-base --out synthetic.json
+python3 bench/bench.py compare synthetic.json
+```
+
+For a project, use a fresh container and:
+
+```bash
+python3 bench/bench.py measure --base /tmp/splinter-base --project "carbon=$(python3 bench/bench.py fetch carbon --dest /tmp/carbon)" --out carbon.json
+```
+
+Without `--base` you get absolute numbers: only lint errors fail, and repeated catalog scans are listed. `compare` takes several result files. The schema load is not idempotent, so every `measure` needs a fresh container.