Bite report: {e(root.name)}
+Claims tested
Free check: does a test notice if each change is taken out?
+{f'No commits were checked.
'} + +diff --git a/README.md b/README.md index 9b2f7e9..b0efff6 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ bite --free `bite` now works in any terminal window. `bite --free` needs no key and costs nothing. It takes each recent change back out, runs your tests, and lists the changes no test guards, plus any new test that still passes without its commit's code (it tests nothing). If something it needs is missing, like git or pytest, it says -so and gives you the line that installs it. To update Bite later: `pipx upgrade bite-check`. +so and gives you the line that installs it. Each run ends with a one-page report, `bite_proof/report.html`, to open in your browser. To update Bite later: `pipx upgrade bite-check`. To test the claims themselves, add the paid step's one extra package and set `ANTHROPIC_API_KEY`, then run `bite`: diff --git a/recent.py b/recent.py index d031da5..e24b610 100644 --- a/recent.py +++ b/recent.py @@ -1,13 +1,13 @@ """ -Bite recent: did your coding agent's recent "done" commits really do what they said? +Bite: did your coding agent's recent "done" commits really do what they said? - python recent.py check the last 40 commits made with Claude Code (needs a key for step 2) - python recent.py --free step 1 only: no key, no cost - python recent.py --all every commit, not only Claude-made ones - python recent.py --last 80 --suspects 3 --budget 1.00 - python recent.py --serious only fix claims about data, money, logins, security, crashes or leaks - python recent.py --any check commits even when none of them claims a fix - (installed with pip install -e ., the same runs as: bite, bite --free, ...) + bite check the last 40 commits made with Claude Code (needs a key for step 2) + bite --free step 1 only: no key, no cost + bite --all every commit, not only Claude-made ones + bite --last 80 --suspects 3 --budget 1.00 + bite --serious only fix claims about data, money, logins, security, crashes or leaks + bite --any check commits even when none of them claims a fix + (from a clone without installing: python recent.py, python recent.py --free, ...) Step 1 (free): for each commit that changed code, the change is taken back out and the tests are run. If they still pass, no test shows the change does anything ("unguarded"). If the commit added or changed tests and @@ -18,13 +18,13 @@ test itself, twice, at that commit and on today's code, and a second fresh check must agree that the test asks only for what the commit claimed and fails for a real reason. Only then is it shown as a false "done". -Proof is written to bite_proof/ in the repo: each failing test plus the command that runs it. +Results are written to bite_proof/ in the repo: report.html (one page to open in a browser), and each +failing test plus the command that runs it. Settings: .bite.ini in the repo (optional; first line [kit], then test_command = ...). claim_command runs one test file ({file}); without it Bite adds the file to test_command (for dotnet test it adds a --filter instead). test_covers = the paths the test command can reach (globs, comma-separated, e.g. Assets/Sim/*, tests/*): commits that touch none of them are listed as outside the tests' reach, not checked. free_test_command = a faster subset of the tests for step 1 (step 2 still uses test_command). -Needs bite.py and repo.py next to it. Bite needs an API key for step 2 and never uses a Claude login or plan: Anthropic's rules for tools built on the Agent SDK (https://code.claude.com/docs/en/agent-sdk/overview) say they must use API keys. @@ -514,6 +514,79 @@ def save_log(root: Path, tried: list, cs: list) -> None: {k: c.get(k) for k in ("sha", "subject", "undo", "score")} for c in cs]}, indent=1), encoding="utf-8") +def plain(undo: str) -> tuple[str, str]: + """The free check's result as (kind, words) for the report: kind is held, gap, odd or skip.""" + if undo.startswith(("guarded", "closed since")): + return "held", "Held: a test fails when this change is taken out" + if undo == "TEST GUARDS NOTHING": + return "odd", "Its own tests still pass without its code: they check nothing it changed" + if undo.startswith("unguarded"): + return "gap", "No test notices if this change is taken out" + return "skip", "Couldn't check: " + undo.replace("not flagged: ", "") + + +def commit_link(root: Path) -> str: + """https://github.com/owner/name/commit/ for a GitHub origin, else "" (the report then shows plain shas).""" + url = repo.git(root, "remote", "get-url", "origin") or "" + m = re.search(r"github\.com[:/]([^/]+/[^/]+?)(?:\.git)?/?$", url) + return f"https://github.com/{m.group(1)}/commit/" if m else "" + + +def write_html(root: Path, cs: list, tried: list, seconds: float) -> Path: + """One page that says what was checked and how each commit did, to open in a browser or screenshot.""" + import html + from datetime import datetime + e = html.escape + link = commit_link(root) + checked = [c for c in cs if c.get("undo")] + kinds = [plain(c["undo"])[0] for c in checked] + false_done = [r for r in tried if r.get("status") == "FALSE DONE"] + sha = lambda x: (f'{x[:8]}' if link else x[:8]) # noqa: E731 + rows = "".join(f'
{sha(c["sha"])}{sha(r["sha"])}No commits were checked.
'} + +