Skip to content

ci: phase the Claude PR review — full review once, then delta-only follow-ups - #90

Merged
AlexKantor87 merged 2 commits into
mainfrom
ci/phased-claude-review
Oct 22, 2026
Merged

AlexKantor87 merged 2 commits into
mainfrom
ci/phased-claude-review

Conversation

@dangrondahl

Copy link
Copy Markdown
Contributor

Closes #89. Supersedes #83 (folded in as its own commit, b3d6841).

Why

Every push re-ran a full Opus review. Each round found fresh nits instead of checking whether the last round's findings were fixed: 6 full runs on #87, 3 on #45 for an unchanged dependency diff.

What changes

  • Triggers: adds ready_for_review and skips drafts. The fork guard stays on every job.
  • Rebase detection: on synchronize, a shell step compares the PR's own diff with the previous push's using git patch-id. A pure rebase onto newer main runs no review in any job. If the old SHA can't be fetched after a force-push, the step assumes the diff changed.
  • Dependabot / catalog jobs: run their full review again only when their diff changed.
  • General job: a full review on open. Later pushes get a Sonnet follow-up scoped to the new commits: git diff BEFORE AFTER, or git range-diff after a rebase so main's changes aren't included. It reports resolved / partially addressed / not addressed for earlier findings, resolves fixed threads, and flags only bugs, security issues and regressions. If the first full review was cancelled, it reviews the whole PR instead.
  • Full review prompt: adds a "what not to report" section, cuts Phase 3 to one sentence, and lets the reviewer read CI results with gh pr checks instead of running npm.
  • Outdated rules in the prompt: the general prompt now points at CLAUDE.md instead of restating it. The restated list said org falls back to config.org, the opposite of the rule, and left the version out of User-Agent. The dependency prompt now says tools are registered in src/server.ts.
  • CLAUDE.md documents the phasing.

Testing

  • Rebase detection was tested in a throwaway repo: a pure rebase compared equal, a real change differed, and range-diff output was checked.
  • actionlint (with shellcheck) passes.

Needs a manual review

The action refuses to run when its workflow differs from the default branch (see CLAUDE.md, #33), so this PR can't review itself. It takes effect after merge; check the first follow-up run on the next PR.

Points to look at:

  • Model: the follow-up uses claude-sonnet-5-5; the full reviews stay on claude-opus-5.
  • Thread resolving: not confirmed that the job's pull-requests: write is enough for the GraphQL resolveReviewThread mutation. If it isn't, threads stay open and the rest still works.
  • gh api access: the follow-up allows Bash(gh api:*), as in the CLI repo's workflow. The token is limited to contents: read and pull-requests: write.
  • Limits: the follow-up is capped at 30 turns and $2; the full reviews keep 60.

Folds in #83, the only other open PR that edits claude-review.yaml.
…llow-ups

Every push re-ran a full Opus review, which found fresh nits each round instead
of checking whether the last round's findings were fixed, and Dependabot rebases
re-reviewed unchanged dependency diffs.

- Full review on opened, reopened and ready_for_review; drafts are skipped.
- On synchronize, a git patch-id step compares the PR's own diff with the
  previous push's. A pure rebase runs no review in any job.
- Dependabot and catalog PRs get their full review again only when their diff
  changed.
- General PRs get a Sonnet follow-up scoped to the new commits (git diff, or
  git range-diff after a rebase), which reports on earlier findings, resolves
  fixed threads, and flags only bugs, security issues and regressions.
- The full general review gets an explicit bar for what not to report, and
  can read CI results with gh pr checks instead of reviewing statically.
- The general prompt points at CLAUDE.md for invariants instead of restating
  them. The restated list had drifted: it said org falls back to config.org,
  the opposite of the rule, and dropped the version from User-Agent. The
  dependency prompt now places tool registration in src/server.ts.

Closes #89
@dangrondahl
dangrondahl enabled auto-merge (squash) October 22, 2026 11:20
@AlexKantor87
AlexKantor87 disabled auto-merge October 22, 2026 11:28
@AlexKantor87
AlexKantor87 merged commit f4623d3 into main Oct 22, 2026
5 checks passed
@AlexKantor87
AlexKantor87 deleted the ci/phased-claude-review branch October 22, 2026 11:29
dangrondahl added a commit that referenced this pull request Oct 23, 2026
Release prep for **0.6.0**. A draft for now: it gets rebased once #97
(CodeQL) merges, then marked ready.

`npm version minor --no-git-tag-version` moves `package.json`,
`package-lock.json` and the generated `src/version.ts` together.
`manifest.json` gets the version at pack time. Nothing else changes.

## What's in 0.6.0 (since 0.5.0)

### ⚠️ Changes that affect existing setups
- **No configured org (#65).** `KOSLI_ORG` and the Desktop extension's
*Organization* setting are removed. Every org-scoped call names its org
through the execute tools' `org` input, so no request lands in an org
the caller didn't name. An existing config that still sets `KOSLI_ORG`
starts fine and logs that the value is ignored, but calls that relied on
the default are now refused until an org is given.
- **Node ≥ 22 (#38).** It was ≥ 20.

### Fixes
- **Evidence downloads work again (#88, fixes #86).** Since 16 Oct,
every `get_attestation_evidence` call failed: our token was forwarded to
the pre-signed S3 URL. Text files come back as text, and binary files
return a clear error.
- **The `.mcpb` bundle starts on Node 22.0 – 22.6 (#87).** It didn't
before.
- **No known vulnerabilities in the bundled dependencies (#96).** `npm
audit --omit=dev` is clean.

### Features
- **`org` input on the execute tools (#65).**
- **`User-Agent` carries the version (#41):**
`kosli-mcp-server/<version>`.

### Dependencies
- **Runtime:** `@modelcontextprotocol/sdk` 1.29.0 → 1.32.1 (#29, #71,
#79).
- **Dev:** TypeScript 7 (#45), with emitted output byte-identical to
5.9. vitest 5 (#93).
- **Catalog:** regenerated from the OpenAPI spec (#50, #53, #60, #69,
#78).

### Release confidence (CI only)
Typecheck of tests and scripts, a stdio smoke test against `dist/` and
the unzipped bundle (including Node 22.0.0), `mcpb validate`, and
`release.yml` publishing only after every check (#87). There's also a
phased Claude review (#90), Dependabot grouping with a 7-day cooldown,
and CodeQL (#97).

## Checked locally on this branch

- Build, typecheck, 122 tests.
- The smoke test reports `server advertises version 0.6.0`, both from
`dist/` and from the unzipped bundle.
- `kosli-mcp-server-0.6.0.mcpb`: the manifest says 0.6.0 and `mcpb
validate` passes.

## Before tagging (after merge)
- [ ] Rebase onto `main` once #97 merges, then mark ready.
- [ ] Install the 0.6.0 `.mcpb` in Claude Desktop, including an upgrade
over 0.5.0 with an org configured.
- [ ] Run `KOSLI_API_TOKEN=… KOSLI_SMOKE_ORG=<org> npm run smoke` on
`main`.
- [ ] Download one piece of evidence via `get_attestation_evidence`.
- [ ] `git tag -a v0.6.0 -m "v0.6.0" <merge commit> && git push origin
v0.6.0`. `release.yml` does the rest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: phase the Claude PR review — full review once, then delta-only follow-ups

2 participants