From 8e71e79ac0f2c1dcb4a130f7b9195ba834abe0ad Mon Sep 17 00:00:00 2001 From: Protechnocrats <213904364+protechnocrats@users.noreply.github.com> Date: Mon, 2 Nov 2026 14:22:55 -0500 Subject: [PATCH] Clean up the documents after removing the evaluation harness Skill files: - WORKFLOW.md, REPORTING.md, DIFFERENTIAL-TESTING.md, BENCHMARKING.md, and ALGORITHM-ARSENAL.md rewritten for clarity: shorter sentences, each instruction stated once, benchmarking bullets grouped by stage. No rule, step, threshold, example, or named section removed; every cross-reference in the skill still resolves. - BEHAVIOR-TRAPS.md: numeric identity in hash keys, new size limits, and parameterized batched queries, from changes Opus made in rounds 8 and 9. - WRITING-NEW-CODE.md: cache keys include user, tenant, and permissions; choices that take outside input must hold on hostile input. Repository documents, now that evals/ and examples/ are gone: - README: evidence rewritten from the Opus results, with links to the results and examples in the repository history; contributing section points to CONTRIBUTING.md. - CONTRIBUTING, SECURITY, RELEASING, docs/ARCHITECTURE, PR template: references to evals/ removed; contributors write for coding-tier models. - CHANGELOG [Unreleased]: what the next release actually contains, including a Removed section. - docs/EVAL-PLAN.md: notes that the harness is to be rebuilt. - CI: the token-budget, cross-reference, rule-evidence, and eval-script steps removed; docs/RULE-EVIDENCE.md removed. Co-Authored-By: Claude Opus 5.5 --- .github/pull_request_template.md | 7 ++- .github/workflows/checks.yml | 11 ----- CHANGELOG.md | 41 ++++++++---------- CONTRIBUTING.md | 46 ++++++++------------ README.md | 35 ++++++--------- RELEASING.md | 6 +-- SECURITY.md | 7 +-- docs/ARCHITECTURE.md | 33 ++------------ docs/EVAL-PLAN.md | 2 +- docs/RULE-EVIDENCE.md | 40 ----------------- skills/ALGORITHM-ARSENAL.md | 10 +---- skills/BEHAVIOR-TRAPS.md | 4 +- skills/BENCHMARKING.md | 33 ++++++++------ skills/DIFFERENTIAL-TESTING.md | 4 +- skills/REPORTING.md | 11 +++-- skills/WORKFLOW.md | 74 +++++++++++++++----------------- skills/WRITING-NEW-CODE.md | 4 +- 17 files changed, 127 insertions(+), 241 deletions(-) delete mode 100644 docs/RULE-EVIDENCE.md diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index a954726..33f12c1 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,11 +1,10 @@ ## What changes for someone using the skill - + ## Checklist - [ ] `CHANGELOG.md` `[Unreleased]` has a line for any change to what the skill does -- [ ] `evals/token-budget.sh` passes; any raised budget has its reason in the script - [ ] Generic files stay language-neutral; mandatory rules are in `skills/SKILL.md` and the file that carries them out -- [ ] Behavior changes were measured before and after on a tuning fixture, or this says why not -- [ ] For eval changes: `evals/selftest.sh` passes (and `--probes` if a probe changed) +- [ ] Links, file names, quoted section names, and rule numbers this change touches still point at something +- [ ] Behavior changes were measured before and after, or this says why not diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index a8a76d0..0aa4c48 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -29,16 +29,5 @@ jobs: set -euo pipefail test -f skills/SKILL.md grep -q '^name: algorithmic-intelligence$' skills/SKILL.md - - name: Token budgets (evals/token-budget.sh) - run: bash evals/token-budget.sh - - name: Cross-references in the skill resolve (evals/check-refs.py) - run: python3 evals/check-refs.py - - name: Rule evidence table is current (evals/rule-cases.py --check) - run: python3 evals/rule-cases.py --check - - name: Eval scripts parse - run: | - set -euo pipefail - for f in evals/*.sh evals/claims/*.sh; do bash -n "$f"; done - python3 -m py_compile evals/*.py evals/probes/*.py - name: CHANGELOG.md has an Unreleased section run: grep -q '^## \[Unreleased\]' CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 881a9d9..baa3707 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,40 +8,33 @@ Versions follow [semantic versioning](https://semver.org): while the major versi ### Changed -- **`SKILL.md` rewritten for clarity:** shorter sentences, one statement of each instruction, rules 6 and 8 split into paragraphs. No rule, threshold, example, or named section was removed (7,898 tokens, from 8,181). -- **A fourth way to keep behavior: take the fast path, and fall back to the original where it cannot match** (for example, send in batches and resend a failed batch one item at a time). In round 9, both Opus sessions with the skill said no behavior-keeping version of the SMS batching existed, although this one does. +- **Clearer instructions throughout:** `SKILL.md`, `WORKFLOW.md`, `REPORTING.md`, `DIFFERENTIAL-TESTING.md`, `BENCHMARKING.md`, and `ALGORITHM-ARSENAL.md` rewritten in shorter sentences, with each instruction stated once and the rest pointing to it. No rule, step, threshold, example, or named section was removed. +- **Before filing a speedup as a behavior change, the agent looks for a version that keeps the behavior:** redo costly work only when its input changed, hand out copies of cached values, take the fast path and fall back to the original where it cannot match (for example, resend a failed batch one item at a time), or read data that something outside the operation may change once per operation. A behavior change is listed only when no such version exists, with each way tried. When the developer declines a behavior change, the agent applies the version that keeps the behavior. +- **Changes to how code calls another system are now verifiable.** Batching, deduplicating, or caching calls to a database or service is checked against a fake that implements the collaborator's documented contract (limits, missing items, partial failure, error order), comparing everything callers observe and the requested items as a set. Only behavior the documentation does not settle stays unverified. +- **The agent states its mode first, then reads the file for that mode before looking at code** (`WORKFLOW.md` or `WRITING-NEW-CODE.md`), and `DIFFERENTIAL-TESTING.md` before writing differential tests. - **The agent says when it runs on a small model.** If it knows its model and the model is a small, fast, or low-cost tier, it adds one line after the mode: the skill is tested with coding-tier models such as Claude Sonnet and Opus. -- **One "Reproduce" section can cover every item** in `APPLY-RESULTS.md` when one harness tests them all, instead of repeating it per item. -- **Changes to how code calls another system are now verifiable.** Batching, deduplicating, or caching calls to a database or service is checked against a fake that implements the collaborator's documented contract (limits, missing items, partial failure, error order), comparing everything callers observe and the requested items as a set. Only behavior the documentation does not settle stays unverified. In round 7 the skill held such changes back as unverifiable even when they were safe. -- **Before filing a speedup as a behavior change, the agent looks for a version that keeps the behavior:** redo costly work only when its input changed, hand out copies of cached values, or read data that something outside the operation may change once per operation (only results the original could also produce). A behavior change is listed only when no such version exists, with each way tried. When the developer declines a behavior change, the agent applies the version that keeps the behavior. In round 8, every Haiku session with the skill left two items slow for lack of this. -- **The agent reads the file for its mode before looking at code** (`WORKFLOW.md` or `WRITING-NEW-CODE.md`), and `DIFFERENTIAL-TESTING.md` before writing differential tests. In round 8, 6 of 8 sessions read neither. -- **The agent states its mode first.** The instruction now opens `SKILL.md` and the first step of each workflow, since sessions often skipped it. +- **New behavior traps,** from changes coding-tier models made in testing: numeric identity in hash keys (`1` against `1.0`), new size limits an optimization introduces, and batched queries that must keep values as bound parameters. In write mode, a cache key must include the user, tenant, and permissions when they change the result. - **The A/A run of a differential test is required only when the behavior can vary between runs** (time, randomness, hash or iteration order, concurrency, external systems); deterministic code records "not needed". -- **The end-to-end share is its own field** in the apply results and part of each item's prediction, since sessions often left it out. +- **The end-to-end share is its own field** in the apply results and part of each item's prediction. +- **One "Reproduce" section can cover every item** in `APPLY-RESULTS.md` when one harness tests them all. ### Added -- **Six clinic cases** (evals only), so java-clinic and its held-out twin rb-clinic cover what only kotlin-shop covered: a secret comparison that must keep uniform timing (J11, RB10), an NP-hard grouped knapsack with a pseudo-polynomial solution (J12, RB11), and 3SUM counting with a conjectured O(n²) bound (J13, RB12), each with probe checks confirmed against a correct and a tempting wrong rewrite. -- **Full grading** (evals only): `evals/GRADING.md` defines every category a round is graded on (validity, behavior safety, case outcomes, protocol, analysis, verification, measurement, report completeness, skill use, cost), and `evals/grade.py` grades every session on all of them and writes a scorecard. Judged categories use a model with a fixed rubric, validated on a planted defect. `run-pilot.sh` allows more read-only commands and gains `--skip-permissions` for disposable machines, since permission refusals of loops made 4 of round 8's 16 sessions invalid. +- **For contributors:** `docs/ARCHITECTURE.md` (what loads when, where each idea lives), `CONTRIBUTING.md`, issue and pull-request templates, `CODEOWNERS`, and a dev container with the JDK, Ruby, Node, Rust, Python, and Swift toolchains. +- **Installing a specific release** is documented in the README (the tag's `skills/` folder URL). +- `docs/EVAL-PLAN.md`: the plan for the next evaluations (hostile inputs, security side effects of optimizations, unusual problems, and write mode, on Sonnet and Opus). -- **Harness guards** (none are installed with the skill or add to an agent's context): `evals/check-refs.py` fails CI on broken links, file names, section names, and rule numbers in the skill; `evals/rule-cases.py` generates `docs/RULE-EVIDENCE.md`, the eval cases that test each rule, from the answer keys, and CI fails when it is out of date; `evals/check-report.py` grades a session's report files against the formats in `REPORTING.md`. `CONTRIBUTING.md` gains "Pruning rules" (a rule with no case needs a case, not deletion) and "Guards". 32 answer-key cases now name the rules they test by number. +### Removed -- **For contributors:** `docs/ARCHITECTURE.md` (what loads when, where each idea lives), `CONTRIBUTING.md`, issue and pull-request templates, `CODEOWNERS`, and a dev container with every toolchain the fixtures need (not yet built and tested). -- **Free self-tests:** `evals/selftest.sh` tests the eval runner with a stub agent (no API, no cost); `--probes` also checks the Java and Ruby grading probes against committed correct and wrong rewrites. -- **More runnable fixtures:** `run-pilot.sh` now runs the report flow on kotlin-shop, swift-shop, c-shop, and rust-shop; its stall watchdog no longer relies on Linux-only `stat` options. -- **Results overview:** `evals/results/README.md` summarizes every round and each clinic case's history. -- **Installing a specific release** is documented in the README (the tag's `skills/` folder URL). -- `examples/`: a real report-and-apply run on the Java fixture, unedited, and one item compared with and without the skill (hostile input on a hash map), linked from the top of the README. +- **The evaluation harness (`evals/`) and the examples (`examples/`),** to be rebuilt per `docs/EVAL-PLAN.md`. Both stay readable in the [repository history](https://github.com/protechnocrats/algorithmic-intelligence/tree/da47e4b), with every round's results. +- **The CI checks that depended on it** (token budgets, cross-references, rule evidence) and `docs/RULE-EVIDENCE.md`. ### Evaluation -- Evaluations now report coding-tier models only (Claude Sonnet and Opus). Round 8's results are corrected: the judge found behavior changes applied on "apply all" in 2 of 3 Opus baselines, which the probe does not test. -- Round 9 ([results](evals/results/2026-11-02-round9.md)), Java project with three new cases, 5 Haiku 5.5 pairs and 3 Opus 5.5 pairs: with Haiku, behavior differences in 0 of 4 valid sessions with the skill and 3 of 3 without (each changed a public method's signature to make sums exact); with Opus, 0 of 2 and 0 of 1. 5 of 15 sessions were invalid because of harness permission refusals. -- The evaluation cost cap is now hard: each turn may spend only what is left of it. Round 9 went $5.97 over its cap because the cap only stopped new sessions from starting. -- `grade.py` counts a probe that cannot compile against a session's code as a behavior difference, not as "no difference", and marks a session invalid when more than half of its shell commands were refused. -- `grade.py` scores write-mode cases only on write sessions, and the judge's cost cap no longer has a $0.50 floor per call. -- Round 8 ([results](evals/results/2026-11-02-round8.md)), held-out Ruby project, 5 Haiku 5.5 pairs and 3 Opus 5.5 pairs: with Haiku, behavior differences in 0 of 3 valid sessions with the skill and 4 of 4 without (corrupted labels); with Opus, 0 of 2 and 0 of 3 (4 sessions invalid: harness permission refusals), and the baseline was faster and cheaper. With the skill, Haiku held back two speedups as behavior changes where equivalent versions exist. -- Round 7 ([results](evals/results/2026-11-01-round7.md)), Haiku 5.5: on the held-out Ruby project, neither skill session changed behavior while the baseline corrupted user-visible labels; on the Java project, the baseline's "optimization" became 3.5 times slower on attacker-chosen keys, which none of 11 skill sessions did. Write mode loaded the skill with the README's recommended project-instruction line. +Results on Claude Opus 5.5 from the rounds since 0.1.0 ([results in the history](https://github.com/protechnocrats/algorithmic-intelligence/tree/da47e4b/evals/results)), graded on every category including a judge reading each session's code and reports: +- Without the skill, Opus applied behavior changes on a general "apply all" in all 3 sessions where this could be scored. With it, Opus held every one for review, in 4 of 4 sessions. +- Bugs were reported to the developer in 4 of 4 sessions with the skill and 0 of 3 without. +- Sessions with the skill cost 1.7 to 3 times as much. ## [0.1.0] - 2026-11-01 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 29f01af..8dc2c2a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,55 +1,45 @@ # Contributing -Thank you for helping. This project is a skill (Markdown instructions that coding agents follow) plus the evaluations that measure it. Read [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) first: it shows which file holds what and what an agent loads when. +Thank you for helping. This project is a skill: Markdown instructions that coding agents follow. Read [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) first. It shows which file holds what, and what an agent loads when. ## What helps most - **New discovery methods and techniques,** judged by whether they change how the agent thinks, verifies, or measures, not by coverage alone. - **Language companion files** (see below). -- **Evaluation cases,** especially ones where an agent without the skill is likely to fail, and real-world performance problems. -- **Evidence:** runs of the evals on other agents and models. +- **Real-world performance problems** where an agent without the skill gets it wrong: slow code, hostile inputs, or an optimization that broke behavior or security. +- **Evidence** from running the skill with other agents and models. ## How changes reach `main` 1. Branch from `main` and open a pull request. Direct pushes to `main` are blocked. -2. The `skill-checks` workflow must pass: skill name and layout, token budgets, eval scripts parse, `CHANGELOG.md` has an `[Unreleased]` section. +2. The `skill-checks` workflow must pass: it checks the skill's name and layout, and that `CHANGELOG.md` has an `[Unreleased]` section. 3. If the change affects what the skill does, add a line to `[Unreleased]` in `CHANGELOG.md`, written for someone who installs the skill: what the agent now does differently. -4. Resolve review threads; merge with squash or rebase. +4. Resolve review threads, then merge with squash or rebase. ## Changing the skill -- **Put each instruction where it belongs.** A mandatory rule goes in `skills/SKILL.md` and in the file that carries it out; anything else lives in exactly one file, and others point to it. -- **Write for the weakest agent.** Every instruction must be followable by a small model without guessing: explicit, no shorthand, examples where a rule is easy to misread. -- **Keep generic files language-neutral.** Name concepts ("an operation that copies a collection"), not one language's functions; language-specific names belong in `skills/languages/`. -- **Run `evals/token-budget.sh`.** If a needed instruction exceeds a budget, raise the budget in the script with the reason. Never shorten or move instructions to fit. -- **Measure behavior changes.** A change meant to alter what agents do should be tested before and after on a tuning fixture (see `evals/README.md`), never tuned on a held-out fixture. +- **Put each instruction where it belongs.** A mandatory rule goes in `skills/SKILL.md` and in the file that carries it out. Anything else lives in exactly one file, and others point to it. +- **Write for coding-tier models.** The skill is tested with Claude Sonnet and Opus. Every instruction must be followable without guessing: explicit, no shorthand, and an example where a rule is easy to misread. +- **Keep generic files language-neutral.** Name concepts ("an operation that copies a collection"), not one language's functions. Language-specific names belong in `skills/languages/`. +- **Keep references whole.** When you rename a file or a section, or renumber a rule, update every link, file name, quoted section name, and rule number that points at it. An agent that follows a reference to nothing skips that step. +- **Do not trim instructions to save tokens.** Every file is loaded only when a step names it. Remove repetition, never content. +- **Measure behavior changes.** A change meant to alter what agents do should be tested before and after on the same cases, with and without the change. -## Pruning rules +## Removing rules -Every line of `skills/SKILL.md` is loaded on every run, so a rule there must earn its place. [docs/RULE-EVIDENCE.md](docs/RULE-EVIDENCE.md) lists the eval cases that test each rule. It is generated from the "Rules tested" column of the answer keys by `evals/rule-cases.py`, and CI fails when it is out of date, so it cannot drift from the cases. - -- **A rule with no case needs a case, not deletion.** An empty row means the rule has not been tested yet, not that it does nothing. -- **Remove or move a rule only after a before-and-after run** on a tuning fixture shows the same results without it, and only with the maintainer's approval. -- **This applies to `SKILL.md`.** Technique files, `DISCOVERY.md`, and language files load only when a step routes to them; they cover code no eval reaches (kernels, databases, compilers) and are not pruned for lack of a case. -- When you add a case, name the rules it tests, by number, in its answer key. - -## Guards - -A written rule is a request; a script is a check. Before adding a guard to the skill (a script the agent must run), measure the failure it would catch: `evals/check-report.py` grades a session's reports against the formats in `REPORTING.md`. Add a guard to the skill only for a failure that keeps happening across sessions, since its instruction is loaded on every run. CI keeps the skill itself consistent: `evals/check-refs.py` fails on links, file names, section names, and rule numbers that no longer exist. +Every line of `skills/SKILL.md` is loaded on every run, so a rule there must earn its place. But a rule with no test case needs a case, not deletion: it has not been tested yet, which does not mean it does nothing. Remove or move a rule only after a before-and-after run shows the same results without it, and only with the maintainer's approval. Technique files, `DISCOVERY.md`, and language files load only when a step routes to them; they cover code few tests reach (kernels, databases, compilers) and are not pruned for lack of a case. ## Language companion files -Use [`skills/languages/GENERIC.md`](skills/languages/GENERIC.md) as the checklist, follow the structure of the existing files, and add a row to `skills/languages/INDEX.md`. A profile the skill saved in your project (`.algorithmic-intelligence/languages/.md`) is a good starting point. Cover the language and its standard library only, not frameworks. Cite official documentation for every complexity and ordering claim. Where a claim is implementation behavior rather than documented behavior, say so, and add a check under `evals/claims/` (see `evals/claims/kotlin/`). +Use [`skills/languages/GENERIC.md`](skills/languages/GENERIC.md) as the checklist, follow the structure of the existing files, and add a row to `skills/languages/INDEX.md`. A profile the skill saved in your project (`.algorithmic-intelligence/languages/.md`) is a good starting point. Cover the language and its standard library only, not frameworks. Cite official documentation for every complexity and ordering claim, and where a claim is implementation behavior rather than documented behavior, say so. -## Evaluation cases +## Evaluation -Follow "Adding a case" in [evals/README.md](evals/README.md). Every case states, before any run, why an agent without the skill is likely to fail it; invented problems need their intended answer checked against a brute-force reference; probe checks are confirmed against a correct rewrite and the tempting wrong one (add both to `evals/probes/selftest/`). +The evaluation harness is being rebuilt. [docs/EVAL-PLAN.md](docs/EVAL-PLAN.md) describes what it will test (hostile inputs, security side effects of optimizations, unusual problems, and write mode, on Sonnet and Opus). Earlier rounds and their tooling are in the git history before the `evals/` folder was removed. -## Running things +## Toolchains -- **Free:** `evals/check-refs.py`, `evals/rule-cases.py`, `evals/check-report.py `, `evals/token-budget.sh`, `evals/selftest.sh` (the harness, with a stub agent), `evals/selftest.sh --probes` (the graders), the fixtures' own test suites, `evals/claims/kotlin-claims.sh`. -- **Costs money:** `evals/run-pilot.sh` and `evals/trigger-test.sh` start real agent sessions. Always pass `--max-total-usd`, and read [SECURITY.md](SECURITY.md) on running them safely. -- **Toolchains:** the dev container in `.devcontainer/` has the JDK, Ruby, Node, Rust, Python, and Swift the fixtures need. +The dev container in `.devcontainer/` has the JDK, Ruby, Node, Rust, Python, and Swift toolchains. ## Reporting problems diff --git a/README.md b/README.md index bfcc3ca..a0de061 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Algorithmic Intelligence (`algorithmic-intelligence`) is a skill for AI coding a Sometimes the best engineering decision is to leave the code alone, and the skill says so when it is. -**See it first:** [`examples/`](examples/) has a real, unedited report from one session, and one item compared with and without the skill. +**See it first:** [a real, unedited report](https://github.com/protechnocrats/algorithmic-intelligence/tree/da47e4b/examples) from one session on a Java project, and one item compared with and without the skill. ## Quick start @@ -117,7 +117,7 @@ Agents load a skill by matching your request against the skill's description. Re Use the algorithmic-intelligence skill when writing new code that loops over collections or data, and for any question about the speed or complexity of existing code. ``` -This matters most for smaller models. In testing, Haiku 5.5 loaded the skill for 2 of 24 new-code requests from the description alone, and for 22 of 24 with this line, while still skipping trivial requests (0 of 24). (Measured before the skill was renamed from `complexity-optimizer` on 2026-11-01; the description is unchanged.) A broader line such as "use it when writing new code" also loads it for trivial helpers. +Without it, agents often skip the skill for new code: in early testing, even Opus 5.5 did not load it for 1 of 3 write-mode requests. A broader line such as "use it when writing new code" also loads it for trivial helpers. ### Verify and update @@ -244,16 +244,16 @@ Companion files cover the language and its standard library only. Frameworks and ## Evidence -The [`evals/`](evals/README.md) folder tests whether an agent with the skill makes the right call on code where the right answer is known, against the same agent without it. It has: +The skill was tested by running the same coding agent on small projects with planted problems, with and without the skill, and checking each session's final code against the original with trap inputs. The harness and every round's results are in the [repository history](https://github.com/protechnocrats/algorithmic-intelligence/tree/da47e4b/evals); a new harness is being built ([docs/EVAL-PLAN.md](docs/EVAL-PLAN.md)). -- **Test projects** in Kotlin, TypeScript, Swift, C, Rust, Java, Ruby, and Python, with planted cases: straightforward fixes, exponential and cubic starting points, costs that only appear along the call chain, code reached only through a plugin registration, techniques whose assumptions do not hold, ordering and floating-point traps, planted bugs, security traps, invented problems that need the discovery method (including one that only looks like a textbook problem and is NP-hard), verification over a finite domain, external limits such as rate limits and open-file limits, and code that is not worth changing. -- **Two held-out projects** (Python and Ruby) that are never used to tune the skill, only to measure it. The Ruby project also tests the path for a language without a companion file. -- **Answer keys, scorecards, and grading probes** that run trap inputs against each session's final code. Each check in the Java and Ruby probes was confirmed against a correct rewrite (no difference reported) and the tempting wrong one (caught). -- **Tooling:** a headless runner, per-session cost and token statistics, a check of which prompts load the skill, a token-budget check, and tests of language-file claims that official documentation does not settle. +Results on Claude Opus 5.5 so far: -**Results so far** (round 3, in [`evals/results/`](evals/results/)): on the held-out Python project, report-flow cases passed at 1.00 with the skill against 0.50 without it on Opus 5.5, and 0.83 against 0.39 on Haiku 5.5. On both models the code was correct even without the skill; the difference was judgment. Without the skill, every baseline run applied an unapproved concurrency change and shipped a hash-based fix that is quadratic on attacker-chosen keys. With the skill, no Opus run did either; Haiku held the concurrency change for review but still shipped the quadratic fix in two of three runs. +- **Hostile input** (held-out Python project): every run without the skill shipped a hash-based fix that becomes quadratic when an attacker chooses colliding IDs, and applied a concurrency change nobody approved. No run with the skill did either. Report-flow cases passed at 1.00 with the skill against 0.50 without. +- **Behavior changes** (Ruby and Java projects, graded by a separate model reading each session's code and reports): without the skill, Opus applied behavior changes on a general "apply all" in all 3 sessions where this could be scored (edge cases handled differently, new size limits, changed error messages, shared mutable results). With the skill, it held every one for the developer's review, in 4 of 4 sessions. +- **Bugs** found along the way were reported to the developer in 4 of 4 sessions with the skill and 0 of 3 without. +- **Cost:** sessions with the skill cost 1.7 to 3 times as much, mostly for writing and running the differential tests and benchmarks that verify each change. -**Limits:** these runs used Claude Code only; the skill is written to be agent-neutral, but other agents have not been evaluated yet. The Java and Ruby projects were added after round 3 and have not been run yet, and the Swift project has not yet been run on a Swift toolchain. See [`evals/README.md`](evals/README.md) to run the evaluations with and without the skill. +**Limits:** small samples, so treat these as early signals. All runs used Claude Code; other agents have not been evaluated. Claude Sonnet has not been tested yet. ## Files @@ -279,8 +279,8 @@ The skill's knowledge base is modular and organized around its workflow. The age - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): what each part of the repository does, what the agent loads in each mode, and where each idea lives. - [CONTRIBUTING.md](CONTRIBUTING.md): how changes reach `main`, and the rules for changing the skill. -- `.devcontainer/`: every toolchain the evaluation fixtures need (JDK, Ruby, Node, Rust, Python, Swift), and a sandbox for running agent sessions. -- [evals/results/README.md](evals/results/README.md): every evaluation round so far, summarized. +- `.devcontainer/`: the JDK, Ruby, Node, Rust, Python, and Swift toolchains, and a sandbox for running agent sessions. +- [docs/EVAL-PLAN.md](docs/EVAL-PLAN.md): the plan for the next evaluations. Earlier rounds are in the [repository history](https://github.com/protechnocrats/algorithmic-intelligence/tree/da47e4b/evals/results). ## Releases and security @@ -289,18 +289,7 @@ The skill's knowledge base is modular and organized around its workflow. The age ## Contributing -Changes reach `main` only through pull requests that pass the repository checks, and every pull request that changes the skill adds a line to the `[Unreleased]` section of [CHANGELOG.md](CHANGELOG.md). Contributions are welcome, particularly: - -- New discovery methods and algorithmic techniques. -- Language companion files. -- Improvements to verification and benchmarking. -- Evaluation fixtures and test cases, especially real-world performance problems. - -When contributing algorithmic or language-specific knowledge, document assumptions, complexity, behavioral considerations, and authoritative references, and judge each addition by whether it changes how the agent thinks, verifies, or measures, not by coverage alone. - -**Language companion files:** use [`languages/GENERIC.md`](skills/languages/GENERIC.md) as the checklist for what to cover, follow the structure of the existing files, and add a row to the table in `languages/INDEX.md`. A profile that the skill saved in your project (`.algorithmic-intelligence/languages/.md`) is a good starting point. Cover the language and its standard library only. Cite official documentation for every complexity and ordering claim, and list it in the file's "References" section. Where a claim is implementation behavior rather than documented behavior, say so, and add a check for it under `evals/claims/` (see `evals/claims/kotlin/` for an example). - -**Changes to the skill itself:** put each instruction in the file where it belongs, written so that any agent, from the smallest model to the largest, can follow it without guessing. Then run `evals/token-budget.sh`: budgets make growth visible and are raised when a needed instruction exceeds them, never met by trimming instructions. New eval cases follow "Adding a case" in [`evals/README.md`](evals/README.md). +Contributions are welcome, particularly new discovery methods and algorithmic techniques, language companion files, improvements to verification and benchmarking, and real-world performance problems. Judge each addition by whether it changes how the agent thinks, verifies, or measures, not by coverage alone, and document its assumptions, complexity, behavioral considerations, and authoritative references. [CONTRIBUTING.md](CONTRIBUTING.md) has the details: how changes reach `main`, the rules for changing the skill, and the checklist for language companion files. **Understand deeply. Reason algorithmically. Engineer exceptionally.** diff --git a/RELEASING.md b/RELEASING.md index 3c2d259..0b6d0d2 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -6,17 +6,17 @@ Every release has a version tag and published release notes. The notes come from Versions follow [semantic versioning](https://semver.org), and tags are written `v..` (for example `v0.1.0`). -- **Patch** (`v0.1.1`): wording fixes, corrected facts or citations, eval and tooling fixes. No intended change in how agents behave. +- **Patch** (`v0.1.1`): wording fixes, corrected facts or citations, and tooling fixes. No intended change in how agents behave. - **Minor** (`v0.2.0`): new or changed instructions, rules, modes, techniques, or language files, anything that can change what an agent does. While the major version is 0, a minor release may also change behavior incompatibly. - **Major** (`v1.0.0`): the first stable release, and later any incompatible change after it. ## Between releases -Every pull request that changes the skill adds a line to the `[Unreleased]` section of `CHANGELOG.md`, describing the change for someone who installs the skill: what the agent now does differently, not which file changed. The pull request checks fail if the `[Unreleased]` heading is missing. If a change is measured (an eval round), link its results file. +Every pull request that changes the skill adds a line to the `[Unreleased]` section of `CHANGELOG.md`, describing the change for someone who installs the skill: what the agent now does differently, not which file changed. The pull request checks fail if the `[Unreleased]` heading is missing. If a change was measured in an evaluation round, link its results. ## Making a release -1. **Prepare it in a pull request.** In `CHANGELOG.md`, rename `[Unreleased]` to the new version with today's date (`## [0.2.0] - YYYY-MM-DD`), add a short summary at the top of the section, and start a new empty `## [Unreleased]` section above it. Run `evals/token-budget.sh`. Merge the pull request once its checks pass. +1. **Prepare it in a pull request.** In `CHANGELOG.md`, rename `[Unreleased]` to the new version with today's date (`## [0.2.0] - YYYY-MM-DD`), add a short summary at the top of the section, and start a new empty `## [Unreleased]` section above it. Merge the pull request once its checks pass. 2. **Tag the merge commit on `main`:** ```bash diff --git a/SECURITY.md b/SECURITY.md index 5df7129..d23908c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,15 +24,10 @@ You can expect an acknowledgment within 7 days, and an assessment with a planned ## What counts as a security issue here -This project is a set of instructions that coding agents follow, plus evaluation scripts. A vulnerability is anything in it that can make an agent, or a person running the scripts, act against the developer's interests. For example: +This project is a set of instructions that coding agents follow. A vulnerability is anything in it that can make an agent act against the developer's interests. For example: - **Instructions that lead an agent to run, change, delete, or send something the developer did not ask for,** or to skip an approval the skill promises (the mandatory rules say behavior changes, dependencies, and commits need the developer's explicit approval). - **Prompt injection through files the skill tells agents to read.** The skill loads language profiles saved in the project (`.algorithmic-intelligence/languages/.md`) and tells agents to treat them as data. A way to make an agent follow instructions placed in such a file, in a fetched web page, or in project code is a vulnerability. - **Guidance that produces insecure code,** for example advice that removes a limit, uses a non-constant-time comparison on secrets, or picks a structure that is quadratic on attacker-chosen input where the skill claims to prevent that (`skills/techniques/hostile-inputs.md`). -- **Evaluation scripts** (`evals/`) that run code or write files outside the run directories they create, or that expose credentials. Not security issues (open a normal issue instead): an agent producing slow code, a wrong complexity claim, or a missed optimization, unless it creates one of the risks above. - -## Running the evaluation scripts safely - -The scripts in `evals/` start real coding-agent sessions on your machine, and those sessions can run commands (tests, benchmarks, Python). They work in copies of the fixtures outside this repository, but they are not sandboxed. Run them only on a machine where that is acceptable, with a spending limit (`--max-total-usd`), and never with credentials in the environment that the sessions should not have. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1cd7afd..b422a6e 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,14 +2,9 @@ A map of the repository for contributors: what each part does, what loads when, and where each idea lives. For what the skill does, start with the [README](../README.md). -## Two halves +## The skill -| Part | What it is | Who uses it | -|---|---|---| -| `skills/` | The skill: Markdown instructions that a coding agent loads. Everything an installed user gets. | Coding agents | -| `evals/` | The evidence: test projects with planted problems, answer keys, grading probes, and the scripts that run agent sessions and measure them. Never installed. | Maintainers and contributors | - -The skill has no code. Its "runtime" is the agent reading the files, so the main engineering constraints are **what an agent loads, when, and how much it costs**, and **whether a small model can follow every instruction without guessing**. +`skills/` is everything an installed user gets: Markdown instructions that a coding agent loads. The skill has no code. Its "runtime" is the agent reading the files, so the main engineering constraints are **what an agent loads, when, and how much it costs**, and **whether a coding-tier model can follow every instruction without guessing**. The evaluation harness is being rebuilt ([EVAL-PLAN.md](EVAL-PLAN.md)). ## What loads when @@ -33,9 +28,7 @@ The agent always loads `skills/SKILL.md` first (the description in its frontmatt └── apply mode adds: DIFFERENTIAL-TESTING.md, BEHAVIOR-TRAPS.md, BENCHMARKING.md ``` -`evals/check-refs.py` checks that every link, file name, quoted section name, and rule number in the skill still points at something, so a rename cannot leave an agent following a reference to nothing. - -`evals/token-budget.sh` measures exactly this loading per mode and fails when it grows past a budget. Budgets make growth visible; when a needed instruction exceeds one, the budget is raised (with the reason recorded in the script), never met by trimming instructions. +Every link, file name, quoted section name, and rule number in the skill must point at something: an agent that follows a reference to nothing skips that step. Every file loads only when a step names it, so a file's size is paid only in the modes that load it. ## Where each idea lives @@ -56,26 +49,6 @@ Each concept has one home; other files point to it. A mandatory instruction is stated in `SKILL.md` and again in the file that executes it, because an agent working through a phase may not reread `SKILL.md`. Anything else appears once. -## The evaluation pipeline - -``` -fixtures/ ──new-run.sh──▶ a fresh git repo outside this one (+ the skill, for with-skill runs) - │ - run-pilot.sh: claude -p, three turns (report, "apply all", decline) - │ logs/.*.jsonl, progress.log - ┌────────────────────────┼─────────────────────────┐ - probes/ answers/.md cost-breakdown.py, session-stats.py - (trap inputs, original (expected outcome per case, (cost by activity, refusals, - vs the session's code) why a baseline fails) skill files read) - └────────────────────────┴──────────────▶ results/-.md -``` - -- **Fixtures** are small projects with planted cases. **Tuning** fixtures may shape the skill's wording; **held-out** ones (`py-logistics`, `rb-clinic`) only measure it. -- **Probes** compare a session's final code with the original fixture on trap inputs. `evals/probes/selftest/` holds a correct and a deliberately wrong rewrite for each probed case, and `evals/selftest.sh --probes` checks that the probe accepts the first and catches the second. -- **Report grader:** `evals/check-report.py` checks a session's report files for every field the templates in `REPORTING.md` require. It reads the fields from the templates, so it follows format changes. -- **Rule evidence:** `evals/rule-cases.py` builds [RULE-EVIDENCE.md](RULE-EVIDENCE.md) from the answer keys: which cases test each rule. -- **The runner** caps spending (`--max-total-usd`), stops stalled turns, and never lets a stopped session end the batch. `evals/selftest.sh` tests it with a stub agent at no cost. - ## Releases and rules `main` changes only through pull requests that pass the `skill-checks` workflow; `v*` tags publish a GitHub release with that version's section of `CHANGELOG.md`. See [RELEASING.md](../RELEASING.md). diff --git a/docs/EVAL-PLAN.md b/docs/EVAL-PLAN.md index 16bbe94..2c73de0 100644 --- a/docs/EVAL-PLAN.md +++ b/docs/EVAL-PLAN.md @@ -1,6 +1,6 @@ # Evaluation plan: adversarial inputs, security, unusual problems, and write mode on Sonnet and Opus -Status: proposed, 2026-11-02. Nothing in this plan spends money until its phase's cap is agreed. +Status: proposed, 2026-11-02. Nothing in this plan spends money until its phase's cap is agreed. The `evals/` folder was removed on 2026-11-02. The tools this plan names (`run-pilot.sh`, `grade.py`, the probes, `selftest.sh`) are in the repository history at commit `da47e4b`; phase 0 restores or rewrites the ones it needs, and the existing fixtures it mentions come back only if phase 0 brings them back. ## Why diff --git a/docs/RULE-EVIDENCE.md b/docs/RULE-EVIDENCE.md deleted file mode 100644 index 2120b46..0000000 --- a/docs/RULE-EVIDENCE.md +++ /dev/null @@ -1,40 +0,0 @@ -# Rule evidence - - - -Which evaluation cases test each mandatory rule in `skills/SKILL.md`, and each skill file named in an answer key. A case "tests" a rule when its answer key says so; whether a run passed is in `evals/results/`. - -**Reading this table:** a rule with no case has not been tested yet. That calls for writing a case, not for deleting the rule. A rule is removed only after a before-and-after run shows the same results without it, and with the maintainer's approval (`CONTRIBUTING.md`, "Pruning rules"). Every case also tests rule 3 implicitly, since the probes compare each session's code with the original; only cases that name it are listed. - -| Rule | Cases | Count | -|---|---|---| -| 1. Time is invaluable. Space is cheap, but not infinite | java-clinic J2 | 1 | -| 2. Time first, then space | **none yet** | 0 | -| 3. Understand before changing, and verify behavior is unchanged | c-shop C4, c-shop C5, java-clinic J13, kotlin-shop K3, py-logistics PW1, rb-clinic RB12, rust-shop R4, rust-shop R6, rust-shop R7, swift-shop S3, swift-shop S7, ts-shop T7 (2 held out) | 12 | -| 4. Bugs are always documented, and never fixed on purpose | c-shop C7, c-shop C8, java-clinic J7, kotlin-shop K4, kotlin-shop K5, kotlin-shop K6, py-logistics P3, rb-clinic RB7, rust-shop R8, rust-shop R9, swift-shop S5, swift-shop S9, ts-shop T5 (2 held out) | 13 | -| 5. Behavior changes are never silent | java-clinic J3, java-clinic J8, java-clinic J11, kotlin-shop K10, py-logistics P2, rb-clinic RB2, rb-clinic RB10, ts-shop T4, ts-shop T6 (3 held out) | 9 | -| 6. Strive for the best possible complexity | c-shop C1, c-shop C2, c-shop C6, c-shop C10, java-clinic J5, java-clinic J6, java-clinic J10, java-clinic J12, java-clinic J13, kotlin-shop K1, kotlin-shop K2, kotlin-shop K12, kotlin-shop K13, kotlin-shop K14, py-logistics P1, py-logistics P6, rb-clinic RB5, rb-clinic RB6, rb-clinic RB9, rb-clinic RB11, rb-clinic RB12, rust-shop R1, rust-shop R2, rust-shop R3, rust-shop R5, rust-shop R11, rust-shop R12, swift-shop S2, swift-shop S4, swift-shop S6, ts-shop T1, ts-shop T2, ts-shop T10, ts-shop T11, ts-shop T12 (7 held out) | 35 | -| 7. Data structures must earn their place | swift-shop S1, ts-shop T2 | 2 | -| 8. Measure, do not only reason | c-shop C9, java-clinic J4, kotlin-shop K9, kotlin-shop K15, py-logistics P4, rb-clinic RB4, rb-clinic RB5, rust-shop R10, swift-shop S10, ts-shop T9 (3 held out) | 10 | -| 9. State which case you mean | java-clinic J9, java-clinic J11, java-clinic J12, kotlin-shop K8, py-logistics P5, rb-clinic RB10, rb-clinic RB11 (3 held out) | 7 | -| 10. Look beyond CPU loops | java-clinic J1, kotlin-shop K10, rb-clinic RB1 (1 held out) | 3 | -| 11. Keep the public contract | ts-shop T3 | 1 | -| 12. Start green | **none yet** | 0 | -| 13. Respect the project's idioms | **none yet** | 0 | -| 14. Prefer the project language's own data structures over the host platform's | kotlin-shop K1, kotlin-shop K3, kotlin-shop K6, kotlin-shop K11, kotlin-shop K12 | 5 | - -## Skill files named by cases - -| File | Cases | Count | -|---|---|---| -| `BEHAVIOR-TRAPS.md` | java-clinic J8, rb-clinic RB2 (1 held out) | 2 | -| `DIFFERENTIAL-TESTING.md` | java-clinic J7 | 1 | -| `DISCOVERY.md` | java-clinic J5, java-clinic J12, py-logistics P1, rb-clinic RB5, rb-clinic RB11 (3 held out) | 5 | -| `languages/GENERIC.md` | rb-clinic RB8, rb-clinic RB12, rb-clinic RBW1 (3 held out) | 3 | -| `techniques/hostile-inputs.md` | java-clinic J9, py-logistics P5 (1 held out) | 2 | - -## Cases that name no rule - -These test concepts (hidden inputs, cost of existing code from new code, resource bounds) that no single rule owns. Tag one with a rule in its answer key when a rule clearly owns it. - -c-shop C3, java-clinic JW1, kotlin-shop K7, rb-clinic RB3, rb-clinic RB8, rb-clinic RBW1, swift-shop S8, ts-shop T8, ts-shop TW1 diff --git a/skills/ALGORITHM-ARSENAL.md b/skills/ALGORITHM-ARSENAL.md index d45e04c..2cb9f82 100644 --- a/skills/ALGORITHM-ARSENAL.md +++ b/skills/ALGORITHM-ARSENAL.md @@ -106,15 +106,7 @@ Many large wins come from changing the problem rather than speeding up the loop. ## Behavior-changing techniques -Some techniques in `CORE-TECHNIQUES.md` and the domain files necessarily change observable behavior: element order, laziness, side effects skipped, concurrency, error and crash behavior, approximate instead of exact results, different floating-point results, different model outputs, different images or geometry, or the shape of what callers receive. Rule 5 applies to every one of them: - -- Never apply a behavior-changing optimization silently. -- Document it in the report under "Behavior changes for review," with the exact behavior before and after. -- Apply it only after the developer explicitly approves that item (a general "apply all" is not approval), then add the `BEHAVIOR-CHANGE` marker from `REPORTING.md`. - -This includes changes that look like improvements: a function that fails faster, returns results in a "more sensible" order, or no longer throws an exception callers may rely on. Those are behavior changes, and the developer decides. - -The one exception is a bug removed as a side effect of the best optimization, such as an iterative rewrite that no longer overflows the stack. Rule 4 in `SKILL.md` governs it. +Some techniques in `CORE-TECHNIQUES.md` and the domain files necessarily change observable behavior: element order, laziness, side effects skipped, concurrency, error and crash behavior, approximate instead of exact results, different floating-point results, different model outputs, different images or geometry, or the shape of what callers receive. Rule 5 applies to every one of them: first look for a version that keeps the behavior; otherwise list the item under "Behavior changes for review" with the exact behavior before and after, apply it only after the developer explicitly approves that item (a general "apply all" is not approval), and then add the `BEHAVIOR-CHANGE` marker from `REPORTING.md`. That includes changes that look like improvements: failing faster, a "more sensible" order, no longer throwing an exception callers may rely on. The one exception is a bug removed as a side effect of the best optimization, such as an iterative rewrite that no longer overflows the stack (rule 4). ## When the best technique is not available diff --git a/skills/BEHAVIOR-TRAPS.md b/skills/BEHAVIOR-TRAPS.md index 4be49f7..e9802e1 100644 --- a/skills/BEHAVIOR-TRAPS.md +++ b/skills/BEHAVIOR-TRAPS.md @@ -19,6 +19,7 @@ Rewrites that look equivalent but are not. Walk this list for every candidate in - **Mutable keys.** An object used as a hash key and mutated afterward can no longer be found. - **Case, culture, and normalization.** String comparisons that ignore case or depend on locale must use the same comparer in the new structure. - **Floating-point keys.** `NaN`, `-0.0` versus `0.0`, and rounding make float keys unsafe in hash structures. +- **Numeric identity.** `1` and `1.0`, or `2.50` and `2.5` as decimals, may be equal under `==` but not as hash keys (Ruby `eql?`, Java `Integer` versus `Long`, `BigDecimal` scale), or the reverse. A list scan with `==` matched them; a hash structure may not. ## Duplicates and nulls @@ -43,7 +44,6 @@ Rewrites that look equivalent but are not. Walk this list for every candidate in For numerical code, matching outputs on ordinary inputs is not enough evidence. Test the edge values listed in `DIFFERENTIAL-TESTING.md` (NaN, infinities, signed zero, subnormals, extreme magnitudes). - - **Overflow.** A closed-form formula or a different summation order can overflow where the loop did not, or the reverse. Some languages wrap silently, others trap or crash. - **Floating-point association.** `(a + b) + c` is not `a + (b + c)`. Prefix sums, reordering, and parallel reduction change results. - **Integer division and rounding.** Rewritten arithmetic can round differently. @@ -146,6 +146,7 @@ An optimization can be faster while making the system less secure. Treat every i - **Constant-time code.** Do not replace constant-time comparisons or computations (secrets, tokens, MACs, password hashes) with early-exit or data-dependent versions. - **Authorization and validation.** Early exits, caching, batching, and reordering must not skip, reorder, or weaken authentication, authorization, or input validation checks. +- **Query building.** Batching per-item queries into one (`WHERE id IN (...)`) must keep the values as bound parameters, never concatenated into the query text, and chunk to the driver's parameter limit. - **Cache keys.** A cache key must include every input that affects the result, including user, tenant, permission, locale, and configuration context. Omitting one can leak data across users. - **Adversarial inputs.** Hash-based structures can degrade to their worst case under hash flooding when attackers control keys; prefer structures with worst-case guarantees (or randomized hashing) for untrusted input. - **Resource exhaustion.** A faster average case with a much worse worst case (unbounded caches, quadratic behavior on pathological input, unbounded recursion) can create a denial-of-service risk. @@ -155,6 +156,7 @@ If an existing security weakness is found, it is a bug: document it and report i ## Errors and limits - **Different exceptions.** A new structure may throw a different exception type, at a different point, or not at all. +- **New size limits.** An optimization that works only up to a size the original never limited (a bitmask over at most 64 items, a table sized for the test data, a fixed-size buffer) adds an error or a wrong result above that size. Check the largest size callers can pass, not the sizes in the tests. - **Recursion depth.** Turning iteration into recursion (or the reverse) changes stack overflow behavior on large inputs. Removing an overflow on valid input is a bug fixed as a side effect (rule 4); introducing one is a regression and is never acceptable. - **Resource limits.** A faster algorithm that uses more memory may hit OOM where the original did not (rule 1: memory must stay bounded). diff --git a/skills/BENCHMARKING.md b/skills/BENCHMARKING.md index b585bb2..0df1ded 100644 --- a/skills/BENCHMARKING.md +++ b/skills/BENCHMARKING.md @@ -4,24 +4,29 @@ How to measure speed and memory so the numbers can be trusted. Benchmarks are ev ## Benchmarks -- **Predict first.** Before running anything, write down the expected speedup at realistic `n` and the expected growth exponent of each version (the report's "Prediction" field). Compare the measurement with it afterward. A prediction that misses badly means the analysis is wrong somewhere: find out where before keeping the change. +**Before running** +- **Predict first.** Write down the expected speedup at realistic `n` and the expected growth exponent of each version (the report's "Prediction" field), and compare the measurement with it afterward. A prediction that misses badly means the analysis is wrong somewhere: find out where before keeping the change. - Use the project's existing benchmark tooling when present (see the language files). Otherwise write a simple timing harness in test code: warm up, run many iterations, report the median, and keep the result in a variable the compiler cannot discard. -- Measure the reference and the candidate at realistic `n` and at a larger `n` (for example, 10× realistic) to show the growth difference. -- **Doubling experiments.** To check a complexity claim, time both versions at N, 2N, 4N, and 8N after warm-up, and compute the exponent b ≈ log₂(T(2N) / T(N)): near 1 is linear, slightly above 1 fits n log n, near 2 is quadratic. Predict the next size's time before measuring it. Start N large enough that fixed overhead does not dominate, and expect the constant to shift where the working set crosses a cache or memory boundary. Report the measured exponent next to the claimed complexity; a mismatch means the analysis missed a cost. -- Use the same inputs for both versions. -- Include worst-case and adversarial inputs where relevant: already-sorted or reverse-sorted data for sorting and selection, colliding hash keys for hash-based structures, deep or degenerate shapes for trees and recursion. When untrusted parties control the input, search for the worst inputs as in "Finding worst-case inputs" in `techniques/hostile-inputs.md`. -- Warm up where the runtime has a JIT or caches. Run multiple times and report the median and the variance (or min and max), not a single run. +- Give both versions the same inputs, the same machine, the same build type, and the same load. Prefer release builds: JIT, garbage collection, and debug builds distort results. +- **Use representative inputs.** Uniform random data can hide or exaggerate gains: real inputs are often partly sorted, skewed, or full of duplicates. Benchmark on production-like data (or a recorded sample) as well as generated data. +- **Include worst-case and adversarial inputs** where relevant: already-sorted or reverse-sorted data for sorting and selection, colliding hash keys for hash-based structures, deep or degenerate shapes for trees and recursion. When untrusted parties control the input, search for the worst inputs as in "Finding worst-case inputs" in `techniques/hostile-inputs.md`. + +**Running** +- Measure both versions at realistic `n` and at a larger `n` (for example, 10× realistic) to show the growth difference. +- **Doubling experiments.** To check a complexity claim, time both versions at N, 2N, 4N, and 8N after warm-up, and compute the exponent b ≈ log₂(T(2N) / T(N)): near 1 is linear, slightly above 1 fits n log n, near 2 is quadratic. Predict the next size's time before measuring it. Start N large enough that fixed overhead does not dominate, and expect the constant to shift where the working set crosses a cache or memory boundary. A measured exponent that does not match the claimed complexity means the analysis missed a cost. +- Warm up where the runtime has a JIT or caches, and run many times: report the median and the spread (or min and max), never a single run. - **Alternate the versions.** Run reference and candidate in alternating or randomized order (A, B, B, A, …), not all of one and then all of the other, so drift in temperature, CPU frequency, background load, or caches affects both equally. -- **Control noise.** On the benchmark machine, avoid other heavy load, and note CPU frequency scaling and turbo settings; pin to cores where the platform allows. In cloud or shared environments, expect more noise and take more samples. -- **Measure the noise first (an A/A run).** Before comparing reference and candidate, benchmark the reference against an identical copy of itself, in the same harness and order. The difference you see is the noise floor: a candidate's gain smaller than it is "no measurable difference." An A/A run that shows a large difference means the setup itself is biased; small changes to the environment (environment-variable size, link order, memory layout) can shift results by more than the effect being measured (Mytkowicz et al., 2009), so vary or randomize the setup and repeat in fresh processes before trusting any number. -- **Separate the effect from the noise.** Report a confidence interval for the difference (for example, a bootstrap interval over the per-run times), or at least show that the gap between medians exceeds the run-to-run spread. A difference inside the noise is "no measurable difference," not a speedup. Use the benchmark tool's own statistics when it has them (JMH, BenchmarkDotNet, Criterion, Google Benchmark). -- **Use representative inputs.** Uniform random data can hide or exaggerate gains: real inputs are often partly sorted, skewed, or full of duplicates. Benchmark on production-like data (or a recorded sample) as well as generated data, and on the adversarial inputs listed above. -- Report absolute times together with percentage change. Percentages alone can make a tiny gain look large. -- Run the reference and the candidate in equivalent environments: same machine, same build type, same load. -- Record machine and runtime details with the numbers. JIT, garbage collection, and debug builds distort results; prefer release builds and repeated runs. +- **Control noise.** Avoid other heavy load on the benchmark machine, note CPU frequency scaling and turbo settings, and pin to cores where the platform allows. In cloud or shared environments, expect more noise and take more samples. + +**Separating the effect from noise** +- **Measure the noise first (an A/A run).** Before comparing, benchmark the reference against an identical copy of itself, in the same harness and order. The difference is the noise floor: a gain smaller than it is "no measurable difference." An A/A run that shows a large difference means the setup itself is biased: small changes to the environment (environment-variable size, link order, memory layout) can shift results by more than the effect being measured (Mytkowicz et al., 2009), so vary or randomize the setup and repeat in fresh processes before trusting any number. +- **Show the difference exceeds the noise.** Report a confidence interval for the difference (for example, a bootstrap interval over the per-run times), or at least show that the gap between medians exceeds the run-to-run spread. A difference inside the noise is "no measurable difference," not a speedup. Use the benchmark tool's own statistics when it has them (JMH, BenchmarkDotNet, Criterion, Google Benchmark). + +**Reporting** +- Report absolute times together with the percentage change; percentages alone can make a tiny gain look large. Record machine and runtime details with the numbers. +- Label every number measured. Big-O improvements and workload estimates are labeled estimated and never presented as measured. - Do not commit benchmark harnesses into production code. Keep them in test or benchmark sources if the developer wants them. - **Offer a regression guard.** Propose keeping the benchmark (in the project's benchmark tooling, or as a separate job in continuous integration) with a threshold relative to a recorded baseline, so a later change that brings the cost back is caught. Never put timing assertions in unit tests: they are flaky. A test that checks a size-dependent count instead (comparisons, queries, allocations) is a stable alternative where the code exposes one. The developer decides whether to keep it. -- Label every number as measured. Big-O improvements and workload estimates are labeled estimated and never presented as measured results. ## Component and end-to-end diff --git a/skills/DIFFERENTIAL-TESTING.md b/skills/DIFFERENTIAL-TESTING.md index 5a85ee5..3a35de2 100644 --- a/skills/DIFFERENTIAL-TESTING.md +++ b/skills/DIFFERENTIAL-TESTING.md @@ -43,7 +43,7 @@ Compare all of these, not only the return value: - **Exceptions.** Same exception type, and the same message when callers or logs depend on it. Same behavior on null, empty, and invalid input. - **Side effects.** Calls to collaborators: which ones, with what arguments, how many times, and in what order. Writes, logs, events, and analytics. - **State.** Any object or global state the function changes. -- **Aliasing and identity.** Whether the function behaves the same when arguments are the same object or share parts (the same list passed twice, a list and a view of it, an object and a collection that contains it), and whether it returns the same object identity where callers can observe it (it returns an argument itself rather than a copy, or reuses a cached or internal object). A deep-copy snapshot of the inputs cannot detect these differences: test them with inputs built to alias, and compare identity (`===`, `is`, `ReferenceEquals`, pointer equality) where the original's identity behavior is observable. +- **Aliasing and identity.** Whether the function behaves the same when arguments are the same object or share parts (the same list passed twice, a list and a view of it, an object and a collection that contains it). Whether it returns the same object identity where callers can observe it: an argument itself rather than a copy, or a cached or internal object. A deep-copy snapshot of the inputs cannot detect these differences: test them with inputs built to alias, and compare identity (`===`, `is`, `ReferenceEquals`, pointer equality) where the original's identity behavior is observable. - **Laziness and short-circuiting.** If the original stops early (for example, `any`, `first`, a `break`), the candidate must not evaluate more elements when evaluation has side effects or can throw. Floating-point results must match exactly unless the developer approves a tolerance. Reordering additions changes results. @@ -107,7 +107,7 @@ These checks hold the new code to the contract, not just to the old code's outpu A differential test that passes is evidence only if it could have failed. Before you count it, show that it can: - **Plant a difference (mutation testing).** In a temporary copy of the code under test (the original, before the new version exists; the new version, once it does), make one small change at a time that the test must catch: an off-by-one at a boundary, a reversed tie order, a dropped edge case (empty input, a duplicate, `NaN`), a changed exception type, a skipped side effect. Also try the tempting wrong rewrite you rejected during analysis (for example the unstable sort, or the version without the wraparound). Run the differential test against each mutant. A mutant it does not catch marks a property the test cannot verify: add inputs or comparisons until it is caught, or report that property as unverified. Mutants are never committed; delete them when done. -- **Compare the original with itself (an A/A run) when the behavior can vary between runs:** the code or the test reads the time, uses randomness, depends on hash or iteration order, runs concurrently, or calls an external system. Run the differential test with the original on both sides. It must pass. For code that is deterministic on its inputs, write "A/A not needed: deterministic" instead of running it. If it fails, something nondeterministic (time, randomness, hash order, concurrency) is being compared: control it (fix the seed, inject the clock, compare as multisets where order is not part of the contract) rather than loosening the comparison until it passes. +- **Compare the original with itself (an A/A run) when the behavior can vary between runs:** the code or the test reads the time, uses randomness, depends on hash or iteration order, runs concurrently, or calls an external system. Run the differential test with the original on both sides; it must pass. If it fails, something nondeterministic is being compared: control it (fix the seed, inject the clock, compare as multisets where order is not part of the contract) rather than loosening the comparison until it passes. For code that is deterministic on its inputs, write "A/A not needed: deterministic" instead of running it. Record which mutants were caught in "Reporting verification." diff --git a/skills/REPORTING.md b/skills/REPORTING.md index e2ac409..18c4118 100644 --- a/skills/REPORTING.md +++ b/skills/REPORTING.md @@ -8,9 +8,12 @@ For a small result (one or two items), present the report in the conversation. O - `BUGS-FOUND.md` — written in Phase 4: every bug noticed during the analysis. None are fixed on purpose. - `APPLY-RESULTS.md` — written at the end of apply mode (Phase 6): what was applied and how it was verified and measured, changes marked as needing testing, behavior changes decided, changes tried and rejected while applying, and bugs found while applying. It refers to items by the IDs in `OPTIMIZATION-REPORT.md`. -**Write each file once, and do not edit it afterward.** Text an agent writes stays in the conversation, and every later step sends the whole conversation to the model again, so a report's size is paid for again on every step after it is written, and every edit adds another copy of the changed text. Keep findings in your working notes as you analyze; write `OPTIMIZATION-REPORT.md` and `BUGS-FOUND.md` once, after Phase 3 is complete for every candidate. In apply mode, never edit or rewrite `OPTIMIZATION-REPORT.md`: the developer is deciding from it, and its analysis must stay as written. Record each item's results in your notes and write `APPLY-RESULTS.md` once, at close-out. If you find a mistake in a file you already wrote, fix it in one combined edit and say what changed. When the developer answers after `APPLY-RESULTS.md` is written (approvals, declines, dependency decisions), record all of the answers in one combined edit, not one edit per answer. If the developer asks for progress before then, answer in the conversation. +**Write each file once, and do not edit it afterward.** Text you write stays in the conversation, and every later step sends the whole conversation to the model again: a report's size is paid again on every step after it, and every edit adds another copy of the changed text. +- Keep findings in your working notes while you analyze. Write `OPTIMIZATION-REPORT.md` and `BUGS-FOUND.md` once, after Phase 3 is complete for every candidate. +- In apply mode, never edit or rewrite `OPTIMIZATION-REPORT.md`: the developer is deciding from it, and its analysis must stay as written. Keep each item's results in your notes and write `APPLY-RESULTS.md` once, at close-out. +- If you find a mistake in a file you already wrote, fix it in one combined edit and say what changed. When the developer answers after `APPLY-RESULTS.md` is written (approvals, declines, dependency decisions), record all the answers in one combined edit, not one per answer. If the developer asks for progress before then, answer in the conversation. -**Keep each file as short as the decisions allow.** Every field below appears, but in one line unless the item needs more: no narrative, and each fact stated once (refer to the item or section that already says it). An item may expand a field into short sub-bullets only when the developer has to decide something about it: an item under "Behavior changes for review," unverified equivalence, an exposure (untrusted input, secrets, a hard deadline), or a new dependency. +**Keep each file as short as the decisions allow.** Every field below appears, in one line unless the item needs more: no narrative, and each fact stated once (refer to the item or section that already says it). Expand a field into short sub-bullets only when the developer has to decide something about it: an item under "Behavior changes for review," unverified equivalence, an exposure (untrusted input, secrets, a hard deadline), or a new dependency. ## Report header @@ -22,7 +25,7 @@ Start `OPTIMIZATION-REPORT.md` (or the in-conversation report) with these sectio ## Optimization item format -Rank items by expected real-world impact. Use the full format for nontrivial items (rule 6) and for any item with behavior risks or an exposure; straightforward items and items not worth changing use the compact format. The format marks the difficulty: full format means nontrivial, compact means straightforward or not worth changing. +Rank items by expected real-world impact. Use the full format for nontrivial items (rule 6) and for any item with behavior risks or an exposure, and the compact format for straightforward items and items not worth changing. The format itself tells the reader the difficulty. Compact format: @@ -134,7 +137,7 @@ Also list every marked change in a "Needs testing" section of `APPLY-RESULTS.md` ## Final summary -End every run with a short summary in the conversation. When the report is in files, give each point below in a line or two and point to the files for the detail; do not restate them in the conversation, since the developer already has them. When the report itself is in the conversation, it carries these points and no separate summary is needed: +End every run with a short summary in the conversation. When the report is in files, give each point below in a line or two and point to the files for the detail, without restating them. When the report itself is in the conversation, it already carries these points and needs no separate summary: - Items applied and verified, with the verification level and a one-line summary of the evidence for each. - Items applied with developer approval but marked `NEEDS-TESTING`. diff --git a/skills/WORKFLOW.md b/skills/WORKFLOW.md index ec85408..494c0cb 100644 --- a/skills/WORKFLOW.md +++ b/skills/WORKFLOW.md @@ -1,45 +1,41 @@ # Workflow -Follow these phases in order in report and apply mode. Report mode stops after Phase 4. Write mode does not use this file; it follows `WRITING-NEW-CODE.md`. +Follow these phases in order in report and apply mode. Report mode stops after Phase 4. Write mode follows `WRITING-NEW-CODE.md` instead. ## Phase 1: Setup -1. If you have not yet, state the mode in one line ("First, state the mode" in `SKILL.md`). Detect languages and versions as described in `SKILL.md`, then load the matching `languages/` files. For a language without a companion file, look for a saved profile at `.algorithmic-intelligence/languages/.md` in the project's root directory and load it if it exists; otherwise build one, as `languages/GENERIC.md` describes. Either way, if the profile needs to be created or updated, ask whether to save it there (the question and its purpose are in `GENERIC.md`, "Saving the profile"), together with the commit question in step 4. +1. If you have not yet, state the mode in one line ("First, state the mode" in `SKILL.md`). Detect languages and versions as `SKILL.md` describes, and load the matching `languages/` files. For a language without one, load the saved profile at `.algorithmic-intelligence/languages/.md` in the project's root if it exists, or build one as `languages/GENERIC.md` describes. If the profile is new or changed, ask whether to save it there ("Saving the profile" in `GENERIC.md`), together with the commit question in step 4. 2. Identify the test framework, how tests are run, and whether a property-based testing library or a verification tool (bounded model checker, concurrency checker, proof tool; see the language file) is already present. Do not add one without asking. -3. Identify available profilers and benchmark tooling. If none exists, plan a simple in-test timing harness (see `BENCHMARKING.md`) rather than adding a dependency. -4. Check version control state. In apply mode, recommend a dedicated branch and confirm the working tree is clean, or that the developer accepts mixing changes. Then settle commits: commits are opt-in ("Commits are opt-in" in `SKILL.md`). If neither the conversation nor the project's instructions authorize or forbid them, ask once now whether to commit each optimization separately, and remember the answer for the rest of the task. -5. Run the existing test suite. Record pass and fail counts and the names of the failing tests. If anything fails before you change code (rule 12): - - **Report mode:** continue. Put the failing tests at the top of the report, do not use failing tests as evidence (do not profile through them, and do not treat their expected values as the code's contract), and note which candidates the failures could affect. - - **Apply mode:** report the failures and stop before changing any code, until the developer decides how to proceed. - - If there is no suite, or it cannot run here, follow rule 12. +3. Identify available profilers and benchmark tooling. If there are none, plan a simple in-test timing harness (`BENCHMARKING.md`) rather than adding a dependency. +4. Check version control. In apply mode, recommend a dedicated branch and confirm the working tree is clean, or that the developer accepts mixing changes. Then settle commits ("Commits are opt-in" in `SKILL.md`): if neither the conversation nor the project's instructions authorize or forbid them, ask once now whether to commit each optimization separately, and keep the answer for the rest of the task. +5. Run the existing test suite. Record pass and fail counts and the names of failing tests. If any fail before you change code, follow rule 12: report mode continues, with the failing tests at the top of the report and noted against the candidates they could affect; apply mode stops before changing code until the developer decides. Never use a failing test as evidence: do not profile through it, and do not treat its expected values as the code's contract. If there is no suite, or it cannot run here, follow rule 12. ## Phase 2: Survey -**Confirm the scope first.** Write down what the developer asked you to cover: the whole repository, a module, a directory, a file, a function, or a slow operation they named. Survey that scope. Outside it, read code only as far as needed to establish a candidate's cost and callers (step 4, "Cost along the call chain"): follow calls out of the scope, but do not start surveying unrelated modules or the source of third-party dependencies. If the scope is too large to inspect completely in this session, start with the evidence below (profiles, the developer's complaints, hot paths), and keep a list of the parts you did not inspect. The report must include that list ("Scope" in `REPORTING.md`), so the developer never mistakes an unreviewed area for one with no problems. +**Confirm the scope first.** Write down what the developer asked you to cover: the whole repository, a module, a directory, a file, a function, or a slow operation they named. Survey that scope. Outside it, read code only as far as needed to establish a candidate's cost and callers (step 4, "Cost along the call chain"): follow calls out of the scope, but do not survey unrelated modules or the source of third-party dependencies. If the scope is too large to inspect completely in this session, start with the evidence below (profiles, the developer's complaints, hot paths) and keep a list of the parts you did not inspect. The report must include that list ("Scope" in `REPORTING.md`), so the developer never mistakes an unreviewed area for one with no problems. -Find candidates. Prefer evidence over guessing: +Find candidates, preferring evidence to guessing: -- Profiler output, slow-trace logs, or a specific complaint from the developer come first. When the code can run with realistic input (the test suite, a benchmark, a local environment) and no profile exists, run a profiler before choosing candidates and record where the time goes. In report mode, profile only through what already exists (the test suite, existing benchmarks, an entry point the project provides) or with scratch files outside the project, and leave the working tree exactly as it was. If it cannot run, say why, and treat every candidate's bottleneck evidence as estimated. -- Then scan for structural signals: nested loops over the same or related collections (including triple-nested loops), recursion that branches more than once per call, enumeration of subsets or permutations, linear searches inside loops (`contains`, `indexOf`, `find`, `includes`), repeated sorting, removal from the front of arrays, string building in loops, recursion without memoization over overlapping subproblems, database or network calls inside loops, work repeated on every UI frame, game frame, or recomposition, filters that visit every neighbor of every pixel, element-by-element loops over arrays or tensors, polling that rescans a whole dataset to find changes, aggregates recomputed from base data on every read, and requests that spend longer waiting in queues, pools, or locks than being processed. -- Use the "hidden costs" section of each loaded language file. +- **Profiles and complaints come first:** profiler output, slow-trace logs, or a specific complaint from the developer. When the code can run with realistic input (the test suite, a benchmark, a local environment) and no profile exists, profile it before choosing candidates and record where the time goes. In report mode, profile only through what already exists (the test suite, existing benchmarks, an entry point the project provides) or with scratch files outside the project, and leave the working tree exactly as it was. If the code cannot run, say why, and label every candidate's bottleneck evidence estimated. +- **Then scan for structural signals:** nested loops over the same or related collections (including triple-nested loops), recursion that branches more than once per call, enumeration of subsets or permutations, linear searches inside loops (`contains`, `indexOf`, `find`, `includes`), repeated sorting, removal from the front of arrays, string building in loops, recursion without memoization over overlapping subproblems, database or network calls inside loops, work repeated on every UI frame, game frame, or recomposition, filters that visit every neighbor of every pixel, element-by-element loops over arrays or tensors, polling that rescans a whole dataset to find changes, aggregates recomputed from base data on every read, and requests that wait longer in queues, pools, or locks than they take to process. +- **Use the "Hidden costs" section** of each loaded language file. For each candidate, establish before analyzing it: -1. **Its contract.** Read the function and the functions it calls. Write what it requires, what it returns (including order, ties, and duplicates), and what it changes, throws, or calls. This is the specification every later check is held to. -2. **Who uses it, and how it is reached.** Search all callers, including reflection, dependency injection, serialization, and UI bindings where relevant. Note public API exposure. Then follow the callers upward, caller by caller, until you reach an **entry point**: code that the program, a framework, or an outside party starts. Entry points include `main` functions and command-line commands; request handlers and routes; event, message, and queue listeners; scheduled and background jobs; UI event handlers and lifecycle callbacks; tests; and public or exported APIs of a library (whose callers are outside the repository). +1. **Its contract.** Read the function and the functions it calls. Write what it requires, what it returns (including order, ties, and duplicates), and what it changes, throws, or calls. Every later check is held to this specification. +2. **Who uses it, and how it is reached.** Search all callers, including reflection, dependency injection, serialization, and UI bindings where relevant, and note public API exposure. Then follow the callers upward until you reach an **entry point**: code that the program, a framework, or an outside party starts. Entry points include `main` functions and command-line commands; request handlers and routes; event, message, and queue listeners; scheduled and background jobs; UI event handlers and lifecycle callbacks; tests; and the public or exported API of a library (whose callers are outside the repository). **Reachability.** Write down which of these you found: - - **Reached from an entry point:** name it. Its frequency (per request, per frame, nightly) is what multiplies the cost in step 4. + - **Reached from an entry point:** name it. Its frequency (per request, per frame, nightly) multiplies the cost in step 4. - **Reached only from tests:** say that no production entry point was found. - - **No entry point found:** the chain ends at a function, class, or module that nothing appears to call, often one registered with a dependency-injection container. Do not conclude it is unused. Code reached indirectly does not show up in a search for direct calls. Search for: dependency-injection registrations and bindings (annotations such as `@Inject`, `@Component`, `@Service`, `@Bean`, `@Provides`, module and container configuration in code, XML, YAML, or JSON); classpath or package scanning; service loaders and plugin registries (`META-INF/services`, entry points in package metadata); routing tables and handler maps; event subscriptions; scheduler and job configuration; reflection and lookup by name (class names or method names as strings, `getattr`, dynamic imports); and build or deployment configuration that selects implementations. If one of these reaches the code, treat it as reached from that entry point. If none does, record the reachability as unknown, with the list of places you searched. + - **No entry point found:** the chain ends at a function, class, or module that nothing appears to call, often one registered with a dependency-injection container. Do not conclude it is unused: code reached indirectly does not show up in a search for direct calls. Search for dependency-injection registrations and bindings (annotations such as `@Inject`, `@Component`, `@Service`, `@Bean`, `@Provides`, and module and container configuration in code, XML, YAML, or JSON); classpath or package scanning; service loaders and plugin registries (`META-INF/services`, entry points in package metadata); routing tables and handler maps; event subscriptions; scheduler and job configuration; reflection and lookup by name (class or method names as strings, `getattr`, dynamic imports); and build or deployment configuration that selects implementations. If one of these reaches the code, treat it as reached from that entry point. If none does, record the reachability as unknown, with the list of places you searched. Code with unknown reachability keeps its item. Its cost along the call chain is unknown, not zero: rank it below items on proven hot paths (Phase 4), never list it as "not worth changing" for that reason, never delete it, and ask the developer whether the code is used. 3. **How often and how large.** Estimate realistic `n` (and every other size variable) and call frequency from callers, data sources, UI limits, pagination, and tests. Write the assumption down. -4. **Cost along the call chain.** Multiply the function's cost by how many times its callers invoke it per operation: loops in callers, per-item callbacks, per-frame or per-request calls. A function that is optimal for a single call can be the inner loop of a cubic computation. When it is, analyze the caller as the candidate; the fix often belongs there (for example, build an index once, then look up) rather than in the function, which may stay unchanged. -5. **Hot or cold.** Main thread, per-frame, per-request, per-item, startup, or rare background work. -6. **Which resource limits it.** CPU, memory, I/O, network, a lock, or a queue. For services and systems, check each resource's utilization, saturation (work waiting for it), and errors: the USE method. Time spent waiting is not fixed by a faster algorithm in the waiting code; see `techniques/queues-and-latency.md` and `techniques/hardware-and-concurrency.md`. -7. **Exposure.** Does an untrusted party control any input (requests, files, network data, user-supplied patterns or keys), does the code handle secrets (keys, tokens, passwords, private data), or must it meet a hard deadline? If any applies, judge it by its worst case rather than its expected case, and load `techniques/hostile-inputs.md`. When the code uses a randomized structure (a sketch, a filter, a randomized cache, a keyed hash table) and the untrusted party can see outputs that depend on its random choices, the adversary is adaptive: follow "Adaptive adversaries" in that file. +4. **Cost along the call chain.** Multiply the function's cost by how many times its callers invoke it per operation: loops in callers, per-item callbacks, per-frame or per-request calls. A function that is optimal for one call can be the inner loop of a cubic computation. Then analyze the caller as the candidate: the fix often belongs there (for example, build an index once, then look up), and the function may stay unchanged. +5. **Hot or cold.** Main thread, per frame, per request, per item, startup, or rare background work. +6. **Which resource limits it.** CPU, memory, I/O, network, a lock, or a queue. For services and systems, check each resource's utilization, saturation (work waiting for it), and errors: the USE method. A faster algorithm in the waiting code does not fix time spent waiting; see `techniques/queues-and-latency.md` and `techniques/hardware-and-concurrency.md`. +7. **Exposure.** Does an untrusted party control any input (requests, files, network data, user-supplied patterns or keys), does the code handle secrets (keys, tokens, passwords, private data), or must it meet a hard deadline? If any applies, judge it by its worst case, not its expected case, and load `techniques/hostile-inputs.md`. If the code uses a randomized structure (a sketch, a filter, a randomized cache, a keyed hash table) and the untrusted party can see outputs that depend on its random choices, the adversary is adaptive: follow "Adaptive adversaries" in that file. Drop candidates where `n` is provably tiny and the path is cold (rule 8 defines both), but list them in the report as "not worth changing" so the developer sees they were considered. A path with unknown reachability (step 2) is not cold: keep those candidates. @@ -48,20 +44,20 @@ Drop candidates where `n` is provably tiny and the path is cold (rule 8 defines For each remaining candidate, record: 1. **Problem statement.** Restate the function as an abstract problem ("count pairs with sum k," "find the latest event per user," "check membership against a fixed list"). Abstract statements match known techniques. -2. **Current complexity.** Time and space, in every size variable, labeled worst, average, or amortized. Include the cost of every call the code makes, built-in operations and library calls as well as project functions (for example, a built-in operation that walks a whole collection is linear in its size, so inside a loop over n items it makes the loop O(n²)), each multiplied by how often it runs, and the work done implicitly (copies, conversions, hashing and comparing keys, a sort's key function), as "Count every call at its real cost" under rule 6 in `SKILL.md` requires; then the total along the call chain from Phase 2. In the item's "Cost now" field, name the calls that make up the dominant term, with each one's cost and its source. -3. **Lower bound.** The best complexity any correct algorithm could achieve, with the reason and its basis (proven, conditional, conjectured, workload-dependent, or unknown). See "Lower bounds" in `ALGORITHM-ARSENAL.md`. If no bound can be established, write "unknown," say why, and give the strongest bound you can support. Never invent one (rule 6). -4. **Candidates.** Routed by `ALGORITHM-ARSENAL.md`, from every matching signal in the tables of `CORE-TECHNIQUES.md` (and `ADVANCED-TECHNIQUES.md` when the core techniques do not reach the lower bound or the problem domain calls for it), in proportion to difficulty (rule 6). Straightforward: the strongest candidate plus a brief reason alternatives are unnecessary. Nontrivial: at least three plausible approaches, including at least one combination when the problem has more than one dimension (ordering plus lookup, range plus count, and so on). For each: time, space (preprocessing separately from per-query cost), the assumptions it needs and the evidence from the code that they hold, standard library or dependency, behavior risks. If no table row matches the abstract problem, or the best match stays above the lower bound for no stated reason, work through `DISCOVERY.md` before choosing, and add what it produces to the candidates. -5. **Choice and why not better.** The best approach and why. If it does not reach the lower bound, explain why the gap remains, using the reasons at the end of "Lower bounds" in `ALGORITHM-ARSENAL.md` (an unapproved dependency, preprocessing the call pattern cannot amortize, an assumption the code does not satisfy, or constant factors that lose at realistic sizes), and which `DISCOVERY.md` moves were tried when no table row reached it. -6. **Prediction.** The expected speedup at realistic sizes and the expected growth exponent of both versions, written down now, before anything is measured. Also predict the end-to-end effect: the share of the whole operation's time this code takes, and therefore the most the change can save (rule 8). -7. **Space check.** Only after time is optimal: can the chosen approach use less memory without changing its time complexity? Confirm every new cache or index is bounded. Then multiply: how many copies of this structure can exist at the same time (one per process, worker, thread, connection, request, or tenant)? Report the total across all of them, not one copy (rule 1). +2. **Current complexity.** Time and space in every size variable, labeled worst, average, or amortized, counting every call and all implicit work as "Count every call at its real cost" under rule 6 in `SKILL.md` requires; then the total along the call chain from Phase 2. In the item's "Cost now" field, name the calls that make up the dominant term, each with its cost and the source of that cost. +3. **Lower bound.** The best complexity any correct algorithm could achieve, with the reason and its basis (proven, conditional, conjectured, workload-dependent, or unknown); see "Lower bounds" in `ALGORITHM-ARSENAL.md`. If none can be established, write "unknown," say why, and give the strongest bound you can support. Never invent one (rule 6). +4. **Candidates.** Routed by `ALGORITHM-ARSENAL.md`, from every matching signal in the tables of `CORE-TECHNIQUES.md` (and `ADVANCED-TECHNIQUES.md` when the core techniques do not reach the lower bound or the domain calls for it), in proportion to difficulty (rule 6). Straightforward: the strongest candidate and a brief reason alternatives are unnecessary. Nontrivial: at least three plausible approaches, including at least one combination when the problem has more than one dimension (ordering plus lookup, range plus count, and so on). For each: time, space (preprocessing separately from per-query cost), the assumptions it needs and the evidence from the code that they hold, standard library or dependency, and behavior risks. If no table row matches the abstract problem, or the best match stays above the lower bound for no stated reason, work through `DISCOVERY.md` before choosing and add what it produces to the candidates. +5. **Choice and why not better.** The best approach and why. If it does not reach the lower bound, explain the gap with the reasons at the end of "Lower bounds" in `ALGORITHM-ARSENAL.md` (an unapproved dependency, preprocessing the call pattern cannot amortize, an assumption the code does not satisfy, or constant factors that lose at realistic sizes), and name the `DISCOVERY.md` moves tried when no table row reached it. +6. **Prediction.** Written now, before anything is measured: the expected speedup at realistic sizes, the expected growth exponent of both versions, and the end-to-end effect (the share of the whole operation's time this code takes, and so the most the change can save; rule 8). +7. **Space check.** Only once time is optimal: can the chosen approach use less memory without changing its time complexity? Confirm every new cache or index is bounded. Then multiply by the number of copies that can exist at once (one per process, worker, thread, connection, request, or tenant), and report the total, not one copy (rule 1). 8. **Behavior risks.** Walk `BEHAVIOR-TRAPS.md` and list every trap the change could hit. 9. **Verification plan.** The highest verification level the change can reach cheaply ("Verification levels" in `DIFFERENTIAL-TESTING.md`), the tool if one is needed, and which observable behavior no level can cover here (real database ordering, timing, platform rendering). -10. **Known behavior changes.** Any observable behavior the approach changes, even for the better ("Behavior-changing techniques" in `ALGORITHM-ARSENAL.md` lists the usual ones). Before listing one, look for a version that keeps the behavior ("Look for a version that keeps the behavior" under rule 5 in `SKILL.md`); if one exists, it is the item and the faster version is the option for review. Behavior changes go in the report's "Behavior changes for review" section and are never applied without explicit approval (rule 5). -11. **Bugs noticed.** Anything that looks wrong in the original. Record it for the bug log; do not fix it. Note whether the code will be optimized (rule 4): if not, it stays exactly as is; if so, note whether the chosen approach keeps the bug or removes it as a side effect. +10. **Known behavior changes.** Any observable behavior the approach changes, even for the better ("Behavior-changing techniques" in `ALGORITHM-ARSENAL.md` lists the usual ones). Before listing one, look for a version that keeps the behavior ("Look for a version that keeps the behavior" under rule 5 in `SKILL.md`); if one exists, it is the item, and the faster version is the option for review. Behavior changes go under the report's "Behavior changes for review" and are never applied without explicit approval (rule 5). +11. **Bugs noticed.** Anything that looks wrong in the original. Record it for the bug log; do not fix it. Note whether the code will be optimized (rule 4): if not, it stays exactly as is; if so, whether the chosen approach keeps the bug or removes it as a side effect. ## Phase 4: Report -Write the report defined in `REPORTING.md` in one pass, once every candidate has been analyzed ("Where reports go" in `REPORTING.md`). Rank items by expected real-world impact: hot path and large `n` first, then complexity gap, then risk. Items whose reachability is unknown (Phase 2, step 2) rank below items reached from an entry point with known frequency, and above items not worth changing; ask the developer whether their code is used, since the answer can move them to the top or remove the need to change them. +Once every candidate has been analyzed, write the report defined in `REPORTING.md` in one pass ("Where reports go" in `REPORTING.md`). Rank items by expected real-world impact: hot path and large `n` first, then complexity gap, then risk. Items with unknown reachability (Phase 2, step 2) rank below items reached from an entry point with known frequency and above items not worth changing; ask the developer whether their code is used, since the answer can move them to the top or remove the need to change them. In report mode, stop here and present the report. Ask which items to apply and raise any dependency requests. Ask separately about each item under "Behavior changes for review": a general answer such as "apply all" covers only items whose equivalence is verified (rule 5). @@ -69,16 +65,16 @@ In report mode, stop here and present the report. Ask which items to apply and r For each approved item, one at a time: -1. **Characterize the original.** Write differential tests per `DIFFERENTIAL-TESTING.md`, with the original implementation copied verbatim as the reference, and turn the contract into executable checks ("The contract as executable properties"). Run them against the unchanged code first. They must pass, which shows the harness is wired correctly. Then check the check: the tests must catch planted differences, and pass with the original on both sides when the behavior can vary between runs ("Check the check" in `DIFFERENTIAL-TESTING.md`). Keep one differential-test file and one benchmark program for the whole apply phase and add each item to them ("Work in few, large steps" in `SKILL.md`). +1. **Characterize the original.** Write differential tests per `DIFFERENTIAL-TESTING.md`, with the original implementation copied verbatim as the reference, and turn the contract into executable checks ("The contract as executable properties"). Run them against the unchanged code first: they must pass, which shows the harness is wired correctly. Then check the check: the tests must catch planted differences, and pass with the original on both sides when behavior can vary between runs ("Check the check" in `DIFFERENTIAL-TESTING.md`). Keep one differential-test file and one benchmark program for the whole apply phase, and add each item to them ("Work in few, large steps" in `SKILL.md`). 2. **Change the code.** Make the smallest change that implements the chosen approach. Keep the signature and contract. Add a short comment where the technique is not obvious. -3. **Verify.** Run the differential tests and contract checks, then the full existing suite, then any bounded or exhaustive check from the verification plan. Re-walk the behavior traps for this change and confirm each is covered by a test or ruled out by analysis. Record the level reached. -4. **Benchmark.** Measure time and memory for the original and the new version at realistic `n`, at a larger `n`, and on worst-case inputs where relevant, following `BENCHMARKING.md`. Also state the end-to-end effect: what share of the whole operation's time the changed code took before the change, and the resulting end-to-end saving, measured where possible ("Component and end-to-end" in `BENCHMARKING.md`). Record the numbers as measured, with the measured growth exponent, and compare them with the prediction from Phase 3. If they disagree, find out why before keeping the change: a wrong prediction means a cost the analysis missed. If the new version is not faster at realistic sizes, do not keep it unless the developer agrees for scaling reasons; a change that is not kept goes under "Tried and rejected" in the report. If benchmarking cannot be run, say so and report the expected gain as estimated only. -5. **Handle unverified parts and behavior changes.** If any behavior cannot be verified, or the change alters behavior in any way, stop work on this item: restore its original code, do not commit it, add it to "Behavior changes for review," and continue with the next item. Ask about every such item at the end (rule 5). An earlier "apply all" is not approval for it. If the developer approves that item, apply it and add the `NEEDS-TESTING` or `BEHAVIOR-CHANGE` marker from `REPORTING.md`. If the developer declines it, apply the version that keeps the behavior, when one exists, through steps 1 to 4 (rule 5, "Declined is not the end of the item"). -6. **Re-run after the final edit.** If anything changed after the last test run (review fixes, comments, formatting), run the differential tests and the full suite again. Results from an earlier version of the code do not count. -7. **Commit, only if authorized.** If the developer authorized commits (Phase 1, step 4), make one commit per optimization, with a message that states the before and after complexity, the benchmark result, the verification level, and the tests added. Do not mix bug fixes, formatting, or unrelated refactors into the commit. If commits are not authorized, do not commit: record in the report the exact files and functions this item changed, so the developer can review, commit, or revert it on its own. +3. **Verify.** Run the differential tests and contract checks, then the full existing suite, then any bounded or exhaustive check from the verification plan. Walk the behavior traps again for this change, and confirm each is covered by a test or ruled out by analysis. Record the level reached. +4. **Benchmark.** Measure time and memory for the original and the new version at realistic `n`, at a larger `n`, and on worst-case inputs where relevant (`BENCHMARKING.md`). State the end-to-end effect: the share of the whole operation's time the changed code took before the change, and the resulting saving, measured where possible ("Component and end-to-end" in `BENCHMARKING.md`). Record the numbers as measured, with the measured growth exponent, and compare them with the Phase 3 prediction. If they disagree, find out why before keeping the change: a wrong prediction means the analysis missed a cost. If the new version is not faster at realistic sizes, do not keep it unless the developer agrees for scaling reasons; a change not kept goes under "Tried and rejected." If benchmarking cannot be run, say so and report the gain as estimated only. +5. **Handle unverified parts and behavior changes.** If any behavior cannot be verified, or the change alters behavior in any way, stop work on this item: restore its original code, do not commit it, add it to "Behavior changes for review," and go on to the next item. Ask about every such item at the end; an earlier "apply all" does not approve it (rule 5). If the developer approves it, apply it with the `NEEDS-TESTING` or `BEHAVIOR-CHANGE` marker from `REPORTING.md`. If the developer declines it, apply the version that keeps the behavior, when one exists, through steps 1 to 4 (rule 5, "Declined is not the end of the item"). +6. **Run again after the final edit.** If anything changed after the last test run (review fixes, comments, formatting), run the differential tests and the full suite again. Results from an earlier version of the code do not count. +7. **Commit, only if authorized.** If the developer authorized commits (Phase 1, step 4), make one commit per optimization, with a message that states the complexity before and after, the benchmark result, the verification level, and the tests added. Do not mix bug fixes, formatting, or unrelated refactors into it. Without authorization, do not commit: record in the report the exact files and functions this item changed, so the developer can review, commit, or revert it on its own. -If a differential test fails, the optimization is wrong. Fix the optimization, never the test. If the failure reveals that the original behavior is itself a bug, record it in the bug log and follow rule 4: if the best optimization removes the bug as a side effect, confirm that outputs differ only on the bug-triggering inputs and document it; otherwise keep matching the original. +If a differential test fails, the optimization is wrong: fix the optimization, never the test. If the failure shows that the original behavior is itself a bug, record it in the bug log and follow rule 4: if the best optimization removes the bug as a side effect, confirm that outputs differ only on inputs that trigger the bug and document it; otherwise keep matching the original. ## Phase 6: Close out -Write `APPLY-RESULTS.md` once, in one pass ("Apply results format" in `REPORTING.md`), and leave `OPTIMIZATION-REPORT.md` exactly as the analysis wrote it: what was applied, skipped, or rejected; final benchmark numbers against the predictions; changes tried and rejected while applying; bugs found while applying; and the full lists of `NEEDS-TESTING` and `BEHAVIOR-CHANGE` items. State plainly which items were verified and at which level (with the evidence), which were accepted by the developer with unverified behavior, and which knowingly change behavior. Call an item proven only at verification level 4. Offer to keep each item's benchmark as a regression guard ("Offer a regression guard" in `BENCHMARKING.md`), and list the ones the developer kept. +Write `APPLY-RESULTS.md` once, in one pass ("Apply results format" in `REPORTING.md`), and leave `OPTIMIZATION-REPORT.md` exactly as the analysis wrote it. Cover what was applied, skipped, or rejected; final benchmark numbers against the predictions; changes tried and rejected while applying; bugs found while applying; and the full lists of `NEEDS-TESTING` and `BEHAVIOR-CHANGE` items. State plainly which items were verified and at which level (with the evidence), which the developer accepted with unverified behavior, and which knowingly change behavior. Call an item proven only at verification level 4. Offer to keep each item's benchmark as a regression guard ("Offer a regression guard" in `BENCHMARKING.md`), and list the ones the developer kept. diff --git a/skills/WRITING-NEW-CODE.md b/skills/WRITING-NEW-CODE.md index d160a81..be2dab0 100644 --- a/skills/WRITING-NEW-CODE.md +++ b/skills/WRITING-NEW-CODE.md @@ -37,8 +37,8 @@ The mandatory rules in `SKILL.md` apply as follows. "As written" means exactly a 1. **Setup.** Start your reply by stating the mode in one line ("Write mode: …"; see "First, state the mode" in `SKILL.md`). Detect the language and version, and load the language companion file (`SKILL.md`, "Language detection"). Its "hidden costs" section lists the traps to avoid while writing. For a language without a companion file, load the profile saved in the project at `.algorithmic-intelligence/languages/.md` if it exists; otherwise build one and ask whether to save it there, as `languages/GENERIC.md` describes. The saved profile is the one file outside the request that write mode may create or update, and only with that permission. 2. **Size the workload.** For every part of the new code that loops over data, queries a database or service, or runs per item, per request, or per frame, write down the expected input sizes, how they grow, and how often the code runs. Take them from the request, the existing code (limits, pagination, volume constants, schemas), or reasonable estimates, and state which. If a size decides the design and cannot be estimated at all, ask one short question; otherwise state your assumption and continue. Also note the exposure (`WORKFLOW.md`, Phase 2, step 7): untrusted input, secrets, or a hard deadline means choosing by worst case with `techniques/hostile-inputs.md`. 3. **Separate the parts that matter.** Code whose inputs are provably small and that does not run hot (rule 8 defines both) gets the simplest clear implementation. Everything else gets steps 4 to 6. -4. **Choose algorithms and structures.** When you estimate a design's cost, count every call it makes at its real cost, including built-in operations, library calls, and calls into existing code ("Count every call at its real cost" under rule 6 in `SKILL.md`): a short built-in call inside a loop is not free. Restate the work as an abstract problem, then use the six questions and the "Start from the current complexity" table in `ALGORITHM-ARSENAL.md`, then the sections of `CORE-TECHNIQUES.md` it points to (and `ADVANCED-TECHNIQUES.md` where its topics apply). Check each technique's "Valid when" column against the request and the data. When the right choice is obvious (a hash map instead of a repeated linear search), use it. When it is not, compare the plausible approaches briefly and pick one. When nothing in those files fits the problem (a custom rule or aggregate that no table names), use `DISCOVERY.md` to derive an approach instead of falling back to the obvious nested loop. -5. **Account for existing code you call.** Calling an existing function is part of your code's cost. If you call it inside a loop, multiply its cost by the loop count (WORKFLOW.md, Phase 2, "Cost along the call chain"). Structure the new code so it calls slow existing code as few times as possible: once per distinct input instead of once per item, or once before the loop instead of inside it. Reusing results within one call of your code is safe when nothing can change them during that call (no writes, no hidden inputs: see `BEHAVIOR-TRAPS.md`). A cache that outlives one call is allowed only with an explicit bound (rule 1) and invalidation for every input it depends on (`BEHAVIOR-TRAPS.md`, "Caching traps"); prefer the per-call scope whenever it is enough. +4. **Choose algorithms and structures.** When you estimate a design's cost, count every call it makes at its real cost, including built-in operations, library calls, and calls into existing code ("Count every call at its real cost" under rule 6 in `SKILL.md`): a short built-in call inside a loop is not free. Restate the work as an abstract problem, then use the six questions and the "Start from the current complexity" table in `ALGORITHM-ARSENAL.md`, then the sections of `CORE-TECHNIQUES.md` it points to (and `ADVANCED-TECHNIQUES.md` where its topics apply). Check each technique's "Valid when" column against the request and the data. When the code takes input from users or other systems, the choice must also hold on hostile input and keep queries parameterized ("Security-sensitive behavior" in `BEHAVIOR-TRAPS.md`). When the right choice is obvious (a hash map instead of a repeated linear search), use it. When it is not, compare the plausible approaches briefly and pick one. When nothing in those files fits the problem (a custom rule or aggregate that no table names), use `DISCOVERY.md` to derive an approach instead of falling back to the obvious nested loop. +5. **Account for existing code you call.** Calling an existing function is part of your code's cost. If you call it inside a loop, multiply its cost by the loop count (WORKFLOW.md, Phase 2, "Cost along the call chain"). Structure the new code so it calls slow existing code as few times as possible: once per distinct input instead of once per item, or once before the loop instead of inside it. Reusing results within one call of your code is safe when nothing can change them during that call (no writes, no hidden inputs: see `BEHAVIOR-TRAPS.md`). A cache that outlives one call is allowed only with an explicit bound (rule 1), invalidation for every input it depends on (`BEHAVIOR-TRAPS.md`, "Caching traps"), and a key that includes every input that changes the result, including the user, tenant, and permissions ("Security-sensitive behavior" in `BEHAVIOR-TRAPS.md`); prefer the per-call scope whenever it is enough. 6. **Write it.** Follow the project's idioms and the language file. Add a short comment wherever the algorithm or structure is not obvious, naming it and its complexity (for example, `// Index by SKU once: O(n + m) instead of O(n·m)`). 7. **Test it.** See "Tests". 8. **Summarize.** See "Summary".