diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 22049ea..7cf8211 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,9 @@ jobs: run: uv sync - name: Lint run: uv run ruff check python evals tests scripts + - name: Browser inspection tests (no dependencies) + working-directory: . + run: node --test tools/browser-inspection/*.test.mjs tools/inspection-experiments/*.test.mjs tools/inspection-transport/*.test.mjs - name: Rust checks run: | cargo fmt --all --check diff --git a/.gitignore b/.gitignore index bda6b3d..e0b68bc 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,9 @@ *.pyc .DS_Store /external/ + +# Local capability trial scratch; qualification summaries live in results/. +/jevdual/output/ + +# Owned restart fixture/profile scratch; sanitized qualification lives beside it. +/jevdual/results/restart-reconciliation-20261010/fixture/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2a966a0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,17 @@ +# Working in Footwork (for agents and people) + +Read `CONTRIBUTING.md` first; its rules apply to every change. Two habits matter most for an agent +working here across sessions: + +- **Keep the engineering notebook.** Before a substantive test or harness change, write a + `prediction` in `jevdual/docs/notebook/-.md`: hypothesis, test, what it predicts, + and an outcome that would count against it. Append an `update` when evidence changes the diagnosis + (resolving a prediction as held, failed or inconclusive), and a `summary` when a package closes or is + handed off. Refer to traces, receipts, tests and commits; never copy them. Skip routine steps. + Format and rules: `jevdual/docs/notebook/README.md`; check with + `uv run python -m evals.notebook docs/notebook --check` from `jevdual/`. +- **Commit the prediction before the test that resolves it.** A prediction committed with its own + result cannot be told apart from one written afterwards. + +The notebook is also where a session picks up after its context is lost: read the open predictions +and the last update of the current package before doing anything else. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 64b9ef3..d533cd3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,6 +18,9 @@ Footwork is research code with an evaluation discipline. The discipline is the p - **Secrets never appear in the repo.** Task secrets are fixture demo logins scoped to their origin; `scripts/redact_results.py` runs on every run directory before it is committed; traces and `run.log` are git-ignored. +- **Engineering keeps a notebook.** Debugging and harness changes between experiments record a + prediction before the test and an update when the evidence changes it, in + `jevdual/docs/notebook/` (format in its README; `tests/test_notebook.py` checks it). - **A bug found by driving becomes a scenario first.** `jevdual/tests/test_session_scenarios.py` runs the real `footwork` commands against a simulated Mac (`tests/fixtures/native_world.py`); reproduce the bug there, see it fail, then fix the harness, not the scenario. diff --git a/README.md b/README.md index 53b1f93..cc588bf 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,11 @@ The Python package is `jevdual` (in [`jevdual/`](jevdual/)); Footwork is the pro from a run directory under [`jevdual/results/`](jevdual/results/), and every experiment was pre-registered in [`jevdual/docs/experiments/`](jevdual/docs/experiments/) before it ran. +Optional [browser inspection utilities](tools/browser-inspection/README.md) collect redacted, +timestamped evidence through an existing tab/CDP adapter. They are dependency-free ESM helpers, +separate from the Python driver and verification gates. Their contract is tested offline; live +capture fidelity must be calibrated on the selected browser. + ## Drive it ```sh @@ -28,13 +33,35 @@ footwork done "624" # the verifier judges the claim against the win ``` Also `do type | append | key | hotkey | menu | scroll | enter`, `status`, and `footwork run` for the whole -loop with Jev as System 1 and an LLM as System 2 on escalation. Nothing dispatches on an id from an older +loop: by default the measured **guarded** arm (an LLM decides every step behind the guard), or `--arm dual` +for Jev as System 1 with the LLM on escalation. Nothing dispatches on an id from an older observation; text comes from the task or from named secrets (`{NAME}` read from `$FOOTWORK_SECRET_NAME`, never printed); `--authorize "