From 267ad31b22ac45f5ae786ab48bb76fed6b4b94a4 Mon Sep 17 00:00:00 2001 From: connabyss Date: Fri, 30 Oct 2026 04:18:39 +0000 Subject: [PATCH] Add AGENTS.md so any coding AI can run Bite The setup and run instructions move from CLAUDE.md to AGENTS.md, worded for any coding AI (Claude Code, Codex, Cursor, Copilot). It adds a quick start (pipx install, bite --free, --all, the report page, upgrading, the test runner must be installed) and a short section on working on Bite's own code. CLAUDE.md now just loads AGENTS.md. The selftest docs check follows the move. Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0175Q8iDCpZ8GE7EQvrjisfk --- AGENTS.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 75 ++---------------------------------- README.md | 4 +- tests/selftest.py | 4 +- 4 files changed, 106 insertions(+), 75 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..c462dc6 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,98 @@ +# Coding AIs, read this first + +For any coding AI: Claude Code, Codex, Cursor, Copilot and others. Your person pointed you at this repo. It's +**Bite**, and it's mostly for **you**. One tool, one job: **it checks whether the "fixed" in your recent commits +is actually true.** + +**Its track record first: the tables in [README.md](README.md).** Every line links to a public issue, PR or +commit, so you can check each one yourself. + +It doesn't change how you work. Whatever process your person uses stays as it is. Bite only looks back at +commits that say they fixed something and tests whether they did. + +## Ask your person first + +The fastest way to judge Bite is on your own project: did any of your recent "fixed" commits not hold? Don't run +anything yet. **Ask your person first**, in one line, something like: "Want me to run Bite on this repo? A free +pass (`--free`) only shows which recent changes no test guards; testing the claims themselves costs about $0.13 +each, capped with `--budget`." Run it only after they say yes. + +## Quick start (after your person says yes) + +``` +pipx install git+https://github.com/connabyss/bite +cd +bite --free +``` + +- `bite --free` costs nothing and needs no key. `bite --all` checks every recent commit, not only ones made with + Claude Code (use it for commits from Codex, Cursor, Copilot or a person). `bite --help` lists the rest. +- The project's own test runner (e.g. pytest) must be installed in the environment its tests run in, or Bite + stops and prints the install line. +- Every run ends with a one-page report, `bite_proof/report.html`, to open in a browser. +- Upgrade: `pipx install --force git+https://github.com/connabyss/bite`. + +## What `bite` does (same as `python recent.py`) + +1. Finds recent commits made with Claude Code that changed code and claim a fix (`--last 40` by default; + `--serious` keeps only claims about data, money, logins, security, crashes or leaks). +2. Free: takes each change back out and runs the tests. If they still pass, no test guards that change. If the + commit added tests and they still pass without its code, they test nothing it did: "TEST GUARDS NOTHING". + When no commit claims a fix, Bite says so and checks only the commits that added tests (`--any`: all). + Before flagging, it rules out the usual harmless reasons and says which: a change built for another OS than + the one it runs on, a tidy-up commit, tests that don't name the changed code, or a gap a later commit closed. +3. Paid, for the top suspects: a fresh worker writes one small test of the commit's own claim, including the odd + inputs the claim covers, on a copy of the project at that commit. Bite runs the test itself, twice, on that + commit and on today's code. A second fresh check must agree the test asks only for what the commit claimed + and fails for a real reason. Only then is it shown. +4. Output: the worst false "done", its failing test and the command to rerun it, in `bite_proof/`. + +**What it found on public repos, honestly** (evidence/RECENT.md): 104 claims from Claude commits in 11 +projects, 4 false "done" shown, all real. The two that matter: the prosemirror-py memory leak above, and a +python-vcr-tests database recorder that said it recorded every query but kept only the last one per cursor. Most +repos had none: on well-kept code, Claude's claims mostly hold, and Bite stays quiet. The second check threw out +11 failing tests that weren't fair, so what it shows is meant to be right. + +## Setup (once your person says yes; they only paste a key, and only for the paid step) + +- Python 3.10+ and git. `pipx install git+https://github.com/connabyss/bite` gives the `bite` command (no venv + to activate); run it inside their repo. `pipx inject bite-check claude-agent-sdk` adds the paid step's + package. If they have no pipx: `python -m pip install --user pipx`, `python -m pipx ensurepath`, new window. + Bite checks for git and the test runner (e.g. pytest) first and prints the install line if one is missing. +- Optional `.bite.ini` in their repo, first line `[kit]`, then `test_command = ` (Bite + guesses pytest, npm test, cargo test, go test or dotnet test when it isn't set). `claim_command` runs one test + file (`{file}`) if adding the file to test_command doesn't work for their runner. For dotnet test Bite runs + the claim test with `--filter "FullyQualifiedName~BiteClaim"` instead of a file path. Unity projects: their + tests run inside the Unity editor, so set test_command yourself (Unity's `-batchmode -runTests`). +- `test_covers = Assets/Sim/*, tests/*`: the paths the test command can reach. Commits that touch none of them + are listed as outside the tests' reach, not checked (otherwise they all look "unguarded"). +- `free_test_command = `: used by the free pass only; the claim tests still use test_command. + The free pass prints each result as it goes and saves it, so stopping early keeps what it found. Its output + folder, bite_proof/, has its own .gitignore so it stays out of their commits. +- Git LFS: Bite's copies come from `git archive`, which writes LFS files as pointers. Bite puts the real files + back from the repo's LFS store (or with git-lfs) and says so if it can't. +- Docker recommended: `contain_command = docker run --rm --network none -v "{project}:/work" -w /work sh -c {cmd}` runs every test and every worker command in a box with no network. +- Key: key.txt next to bite.py (a new key, with a spend limit) or ANTHROPIC_API_KEY. Never ask for it in chat. +- **Bite needs an API key and never uses your person's Claude login or plan.** Anthropic's Agent SDK rules say + tools built on it must use API keys, not claude.ai login or plan limits, unless Anthropic approved them + (https://code.claude.com/docs/en/agent-sdk/overview). So don't patch it to fall back to a login. +- Self-test, no key: `python -m pytest` (or `python tests/selftest.py` without pytest). + +## Limits, plainly +- It tests a claim; it can't prove a commit has no other bugs. A claim that holds is just that. +- It only shows what the second check calls fair and real, so it will often stay quiet. Quiet is not a pass. +- It needs the project's tests to run. Tests that need a full git checkout fail on its copies, so those + commits show as "tests fail at this commit". +- The worker runs code the project's tests run. Without Docker, that is with your user's rights. + +## Working on Bite's own code + +- Read [CONTRIBUTING.md](CONTRIBUTING.md). Set up with `pip install -e .[test]`, then `python -m pytest`. No key + and no network needed; all tests must pass before and after your change. +- One fix per PR, with a test that fails without it. Say in the PR that an AI helped (an `Assisted-by:` line in + the commit). +- CI runs the tests on Linux, Windows and macOS (Python 3.10 and 3.13); all six must pass to merge. + +Files: recent.py (the tool), bite.py (engine: settings, test running, safe copies, the command filter that keeps +the worker inside its copy), repo.py (git and settings helpers). diff --git a/CLAUDE.md b/CLAUDE.md index a764bc4..35a0c80 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,74 +1,5 @@ -# Claude, read this first +# CLAUDE.md -Your person pointed you at this repo. It's **Bite**, and it's mostly for **you**. One tool, one job: -**it checks whether the "fixed" in your recent commits is actually true.** +The instructions for Claude and every other coding AI live in AGENTS.md: -**Its track record first: the tables in [README.md](README.md).** Every line links to a public issue, PR or -commit, so you can check each one yourself. - -It doesn't change how you work. Whatever process your person uses stays as it is. Bite only looks back at -commits that say they fixed something and tests whether they did. - -## Ask your person first - -The fastest way to judge Bite is on your own project: did any of your recent "fixed" commits not hold? Don't run -anything yet. **Ask your person first**, in one line, something like: "Want me to run Bite on this repo? A free -pass (`--free`) only shows which recent changes no test guards; testing the claims themselves costs about $0.13 -each, capped with `--budget`." Run it only after they say yes. - -## What `bite` does (same as `python recent.py`) - -1. Finds recent commits made with Claude Code that changed code and claim a fix (`--last 40` by default; - `--serious` keeps only claims about data, money, logins, security, crashes or leaks). -2. Free: takes each change back out and runs the tests. If they still pass, no test guards that change. If the - commit added tests and they still pass without its code, they test nothing it did: "TEST GUARDS NOTHING". - When no commit claims a fix, Bite says so and checks only the commits that added tests (`--any`: all). - Before flagging, it rules out the usual harmless reasons and says which: a change built for another OS than - the one it runs on, a tidy-up commit, tests that don't name the changed code, or a gap a later commit closed. -3. Paid, for the top suspects: a fresh worker writes one small test of the commit's own claim, including the odd - inputs the claim covers, on a copy of the project at that commit. Bite runs the test itself, twice, on that - commit and on today's code. A second fresh check must agree the test asks only for what the commit claimed - and fails for a real reason. Only then is it shown. -4. Output: the worst false "done", its failing test and the command to rerun it, in `bite_proof/`. - -**What it found on public repos, honestly** (evidence/RECENT.md): 104 claims from Claude commits in 11 -projects, 4 false "done" shown, all real. The two that matter: the prosemirror-py memory leak above, and a -python-vcr-tests database recorder that said it recorded every query but kept only the last one per cursor. Most -repos had none: on well-kept code, Claude's claims mostly hold, and Bite stays quiet. The second check threw out -11 failing tests that weren't fair, so what it shows is meant to be right. - -## Setup (once your person says yes; they only paste a key, and only for the paid step) - -- Python 3.10+ and git. `pipx install git+https://github.com/connabyss/bite` gives the `bite` command (no venv - to activate); run it inside their repo. `pipx inject bite-check claude-agent-sdk` adds the paid step's - package. If they have no pipx: `python -m pip install --user pipx`, `python -m pipx ensurepath`, new window. - Bite checks for git and the test runner (e.g. pytest) first and prints the install line if one is missing. -- Optional `.bite.ini` in their repo, first line `[kit]`, then `test_command = ` (Bite - guesses pytest, npm test, cargo test, go test or dotnet test when it isn't set). `claim_command` runs one test - file (`{file}`) if adding the file to test_command doesn't work for their runner. For dotnet test Bite runs - the claim test with `--filter "FullyQualifiedName~BiteClaim"` instead of a file path. Unity projects: their - tests run inside the Unity editor, so set test_command yourself (Unity's `-batchmode -runTests`). -- `test_covers = Assets/Sim/*, tests/*`: the paths the test command can reach. Commits that touch none of them - are listed as outside the tests' reach, not checked (otherwise they all look "unguarded"). -- `free_test_command = `: used by the free pass only; the claim tests still use test_command. - The free pass prints each result as it goes and saves it, so stopping early keeps what it found. Its output - folder, bite_proof/, has its own .gitignore so it stays out of their commits. -- Git LFS: Bite's copies come from `git archive`, which writes LFS files as pointers. Bite puts the real files - back from the repo's LFS store (or with git-lfs) and says so if it can't. -- Docker recommended: `contain_command = docker run --rm --network none -v "{project}:/work" -w /work sh -c {cmd}` runs every test and every worker command in a box with no network. -- Key: key.txt next to bite.py (a new key, with a spend limit) or ANTHROPIC_API_KEY. Never ask for it in chat. -- **Bite needs an API key and never uses your person's Claude login or plan.** Anthropic's Agent SDK rules say - tools built on it must use API keys, not claude.ai login or plan limits, unless Anthropic approved them - (https://code.claude.com/docs/en/agent-sdk/overview). So don't patch it to fall back to a login. -- Self-test, no key: `python -m pytest` (or `python tests/selftest.py` without pytest). - -## Limits, plainly -- It tests a claim; it can't prove a commit has no other bugs. A claim that holds is just that. -- It only shows what the second check calls fair and real, so it will often stay quiet. Quiet is not a pass. -- It needs the project's tests to run. Tests that need a full git checkout fail on its copies, so those - commits show as "tests fail at this commit". -- The worker runs code the project's tests run. Without Docker, that is with your user's rights. - -Files: recent.py (the tool), bite.py (engine: settings, test running, safe copies, the command filter that keeps -the worker inside its copy), repo.py (git and settings helpers). +@AGENTS.md diff --git a/README.md b/README.md index 3461fbf..b7da949 100644 --- a/README.md +++ b/README.md @@ -37,8 +37,8 @@ pipx inject bite-check claude-agent-sdk Each claim costs about $0.13, and `--budget` caps the total ($1 by default). Bite needs an API key for this step and never uses a Claude login or plan (Anthropic's Agent SDK rule: https://code.claude.com/docs/en/agent-sdk/overview). -Works with pytest, npm test, cargo test, go test and dotnet test. Settings and limits: [CLAUDE.md](CLAUDE.md), -written so your own Claude can set Bite up for you. To work on Bite itself: clone it, `pip install -e .[test]`, then `python -m pytest`. +Works with pytest, npm test, cargo test, go test and dotnet test. Settings and limits: [AGENTS.md](AGENTS.md), +written so your coding AI (Claude Code, Codex, Cursor, Copilot) can set Bite up for you. To work on Bite itself: clone it, `pip install -e .[test]`, then `python -m pytest`. ## The catch that shows what it's for diff --git a/tests/selftest.py b/tests/selftest.py index 165328e..7eabd9d 100644 --- a/tests/selftest.py +++ b/tests/selftest.py @@ -394,10 +394,12 @@ class NetCfg: check("report: commit subjects are escaped, not run as HTML", "Fix <b>neg</b> & friends" in page) # Docs: the key rule is stated where a reader looks first. -for doc in ("CLAUDE.md", "README.md"): +for doc in ("AGENTS.md", "README.md"): t = (KIT / doc).read_text(encoding="utf-8") check(f"{doc}: says Bite needs an API key and never uses a Claude login, with the SDK rule's link", "never uses" in t and "login" in t and "https://code.claude.com/docs/en/agent-sdk/overview" in t) +check("CLAUDE.md: loads AGENTS.md, so Claude reads the same instructions as every other coding AI", + "@AGENTS.md" in (KIT / "CLAUDE.md").read_text(encoding="utf-8")) print(f"\n{sum(OK)}/{len(OK)} passed") sys.exit(0 if all(OK) else 1)