feat(agent): cap the agent's context window with agent.context_window

.claude/notes/README.md

@@ -13,6 +13,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`.
 ## Contents
 
 - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity
+- [context-window.md](context-window.md) — the `agent.context_window` cap, per harness
 - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging
 - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading
 - [lint-rules.md](lint-rules.md) — why each CE lint rule exists

.claude/notes/context-window.md

@@ -0,0 +1,170 @@
+# Capping the agent's context window
+
+Analysis and design for `agent.context_window`: one harness-neutral field that caps the
+context window an agent works within, set per agent config or per experiment variant.
+
+## Problem and evidence
+
+A benchmark of Claude Code on Sonnet 5 (Bedrock, 1M-token window) let runs grow to
+250k-480k tokens of context. Above about 150k tokens each extended-thinking step got
+about 10x slower, so most runs hit the 2-hour task limit. The operator wants to measure
+pass rate and cost at several context caps, for example one variant at 200k and one at
+400k, on the same tasks.
+
+Before this change no harness-neutral way existed:
+
+- **claude-code**: `claude_settings` (`--settings`) could carry `autoCompactWindow`. That
+  works for one harness only, and when `claude_settings` is a file path a variant cannot
+  merge one key into it.
+- **codex**: `CodexAgentConfig` has no pass-through. `CodexAgent._build_thread_options`
+  builds the thread `config` dict itself, so no value reached Codex.
+- **antigravity, opencode, pi, delegate**: no knob is wired.
+
+## How each harness controls its window
+
+### claude-code
+
+Verified in the CLI the SDK spawns. `claude-agent-sdk` runs its **bundled** CLI before it
+looks for `claude` on `PATH` (`SubprocessCLITransport._find_cli`), so the CLI that runs is
+the one the SDK wheel bundles, not the one the agent image installs with npm:
+
+| coder_eval | claude-agent-sdk (uv.lock) | bundled CLI | image npm CLI |
+|---|---|---|---|
+| 0.12.1 | 0.2.124 | 2.1.216 | 2.1.177 |
+| 0.12.12 | 0.2.159 | 2.1.281 | 2.1.281 |
+
+`environment_info.claude_code_cli` records `claude -v` from `PATH`, so a 0.12.1 run
+records 2.1.177 while 2.1.216 ran. That is a separate defect and is not fixed here.
+
+Three inputs set the auto-compact window. Both bundled CLIs (2.1.216 and 2.1.281) contain
+the same resolution function. The highest precedence comes first:
+
+1. `CLAUDE_CODE_AUTO_COMPACT_WINDOW` environment variable: a plain token count. It
+   outranks the flag and every settings scope.
+2. `--autocompact <auto|tokens>`: added in 2.1.221 (CLI reference). It is absent from
+   2.1.216 as a parsed option.
+3. `autoCompactWindow` settings key. The settings schema is
+   `int().min(100000).max(1000000).optional().catch(undefined)`, so an out-of-range
+   value is **silently dropped**.
+
+The effective window is `min(model context window, configured value)`. Auto-compaction
+starts when the context approaches that window.
+
+### codex
+
+Verified in the `openai-codex-cli-bin` 0.156.1 binary (the pin) and its source at tag
+`rust-v0.156.1`:
+
+- `model_context_window` ("Size of the context window for the model, in tokens"):
+  `with_config_overrides` sets the model's `context_window` to
+  `min(value, max_context_window)`. Three things follow from it. The default
+  auto-compact limit becomes 90% of it. The hard cap, which forces compaction, becomes
+  `context_window * effective_context_window_percent / 100`. The remaining-window figure
+  the model is shown also changes.
+- `model_auto_compact_token_limit` ("Token usage threshold triggering auto-compaction"):
+  the trigger only, clamped to 90% of the resolved window. It does not change the window.
+- `model_auto_compact_token_limit_scope` (`total` by default, or `body_after_prefix`) and
+  `model_post_turn_compact_threshold_percent` (turn-end compaction, off by default) tune
+  when compaction starts. They are not caps.
+
+The thread `config` dict `thread_start` takes is the same override surface that
+coder_eval already uses for `enabled_tools` and `model_providers`.
+
+### antigravity, opencode, pi, delegate
+
+None of them has a context-window knob wired in coder_eval. They are out of scope, and
+they reject the field.
+
+## Options considered
+
+**Per-harness pass-through** (a Codex `config` pass-through next to `claude_settings`).
+It is flexible, but a variant would have to name a different key per harness, and the
+recorded value would mean something different on each. It also opens a free-form
+pass-through on Codex with no denylist. Rejected.
+
+**A field under `run_limits`.** Run limits are caps that the orchestrator or the agent
+enforces on a run (turns, time, tokens, USD), and the parity table lists them. A context
+window is not a spend cap. It is a harness setting that changes what the agent does, and
+it depends on the agent type, which `run_limits` does not know. The guide also says that
+`agent:` holds no run-time caps. Rejected.
+
+**A field on the agent config.** One name, accepted on a type-less config so that
+experiment defaults and variants can set it, and validated against the concrete type
+when the config resolves. **Chosen.**
+
+Naming: `context_window`, in tokens. It names the quantity the operator caps, matches
+Codex's `model_context_window`, and avoids "auto-compact", which describes a mechanism of
+one harness. `max_context_tokens` was considered, but it reads as a per-request input
+limit, which neither harness enforces.
+
+## Recommended design
+
+`BaseAgentConfig.context_window: int | None` (default `None`, `gt=0`).
+
+- Each config class declares `_context_window_range: ClassVar[tuple[int, int | None] |
+  None]`. `None` (the base default) means that the harness cannot apply the field.
+  `ClaudeCodeAgentConfig` declares `(100_000, 1_000_000)` and `CodexAgentConfig` declares
+  `(32_768, None)`.
+- `check_context_window_supported` (model validator) rejects an unsupported type or an
+  out-of-range value as soon as the type is known. A plugin agent inherits `None`, so it
+  also rejects the field until it opts in.
+- The field uses the default `replace` merge strategy. It is set at any of the five
+  layers, and with `-D agent.context_window=N`.
+
+Per-harness mapping:
+
+- **claude-code**: `CLAUDE_CODE_AUTO_COMPACT_WINDOW=<N>` in the SDK `env`. The
+  environment variable was chosen over the settings key and the flag for four reasons.
+  It has the highest precedence, so host, project and `claude_settings` values cannot
+  override the cap. It works whether `claude_settings` is a dict or a file path. Both
+  bundled CLIs read it, which the flag does not. `extra_args` stays framework-owned.
+  `claude_settings.autoCompactWindow` together with `context_window` is a load error,
+  because the environment variable would silently override it.
+- **codex**: `model_context_window: <N>` in the thread `config`. No trigger key is set,
+  so Codex compacts at its default 90% of the window.
+
+## Validation
+
+- `gt=0` on every config, so a type-less config rejects nonsense as well.
+- claude-code: 100000-1000000, the CLI's documented range. Outside that range the CLI
+  drops a settings value silently, so coder_eval enforces the range itself.
+- codex: 32768 and up. Codex clamps the value to the model's catalog maximum, and the
+  floor rejects a window too small to work in, such as a YAML `yes` coerced to 1.
+- every other type: rejected at load, naming the supported types. A variant that
+  switches `type` to an unsupported harness fails when the experiment resolves.
+
+## Recording
+
+The resolved agent config is persisted as `agent_config` on every task row (`run.json`
+and `task.json`). `agent.context_window` therefore appears per task, and config lineage
+records which layer set it (for example, the variant). The experiment report groups by
+variant, so a cap per variant groups without new report code. On claude-code the
+`sdk_options` dump also shows the environment variable in `env`.
+
+## Open questions
+
+- **A silent clamp to the model's window.** Neither harness reports that the model's
+  window lowered the cap. A cap above the model's window runs at the model's window.
+  Recording the effective window would need a per-model catalog in coder_eval.
+- **Trigger points differ.** Codex compacts at 90% of the window. Claude Code compacts
+  when the context approaches the window, minus an internal buffer. Equal caps therefore
+  do not compact at exactly the same token count. A separate harness-neutral trigger
+  field is possible, but it is not added until a benchmark needs it.
+- **A host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` and uncapped runs.** Decided: the SDK starts
+  the CLI with `os.environ` underneath `options.env`, so when the field is unset the
+  claude-code agent blanks a host value. A run recorded as uncapped is never capped by
+  the host. This changes every claude-code run on a host that sets the variable,
+  including runs that never set `context_window`. Codex has no equivalent: an unset
+  field leaves a host `model_context_window` in `$CODEX_HOME/config.toml` in effect.
+- **The recorded CLI version is wrong.** See the version table above:
+  `environment_info.claude_code_cli` should record the bundled CLI version.
+- **Other harnesses.** Pi can lower a model's window with
+  `providers.<provider>.modelOverrides.<model>.contextWindow` in `models.json`. A mapping
+  was written and then reverted (d1757a60), because no live run backed it and no task
+  uses `type: pi`; it comes back with a live test. OpenCode (`limit.context` and
+  `limit.input` in provider config) needs a `limit.output` coder_eval does not know, and
+  an unpinned CLI. Antigravity's `CompactionConfig(token_threshold=N)` only moves the
+  trigger. Delegate keeps its history in the backend. Each needs its own verification
+  before it opts in.
+- **How often a run compacted is not recorded.** A cap A/B cannot yet tell "cap never
+  reached" from "compacted N times" from the task row alone.

docs/AB_EXPERIMENTS.md

@@ -23,6 +23,7 @@ to the orchestrator are involved.
 - [What a Variant Can Override](#what-a-variant-can-override)
 - [Recipe: A/B a Skill](#recipe-ab-a-skill)
 - [Recipe: A/B a Model](#recipe-ab-a-model)
+- [Recipe: A/B a Context Window](#recipe-ab-a-context-window)
 - [Recipe: A/B a Prompt](#recipe-ab-a-prompt)
 - [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop)
 - [Replicates (Statistical Power)](#replicates-statistical-power)
@@ -222,6 +223,28 @@ variants:
     agent: { model: claude-opus-5 }
 ```
 
+## Recipe: A/B a Context Window
+
+`agent.context_window` caps the context the agent works within, in tokens: the
+harness compacts the conversation before it outgrows the cap. Use it to measure
+pass rate and cost against the cap on a model with a large window. Each task row
+records the cap in `agent_config.context_window`.
+
+```yaml
+experiment_id: context-window
+description: "The same model with a 200k and a 400k context cap"
+
+variants:
+  - variant_id: cap-200k
+    agent: { context_window: 200000 }
+  - variant_id: cap-400k
+    agent: { context_window: 400000 }
+```
+
+An agent type that cannot apply the cap fails the experiment at load.
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness) lists which
+agent types accept it and what the cap does on each.
+
 ## Recipe: A/B a Prompt
 
 Use `prompt_mutations` (transform the task prompt) or `initial_prompt` /

docs/EXTENDING.md

@@ -80,6 +80,22 @@ class MyAgentConfig(BaseAgentConfig):
 The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config
 isn't an instance of the registered `config_class`, so keep them paired.
 
+To accept `agent.context_window`, declare the inclusive range of tokens your harness
+can cap its context window to, and apply `self.config.context_window` when the agent
+starts. A config class that declares nothing rejects the field at load, which is the
+right default: a capped variant must never run uncapped.
+
+```python
+from typing import ClassVar
+
+class MyAgentConfig(BaseAgentConfig):
+    type: Literal["my-agent"] = "my-agent"
+    _context_window_range: ClassVar[tuple[int, int | None] | None] = (32_768, None)
+```
+
+Add your harness to the table in
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness).
+
 ### The `Agent` ABC — implementation checklist
 
 Implement these three abstract methods:

docs/TASK_DEFINITION_GUIDE.md

@@ -170,6 +170,7 @@ agent:
     - "Write"
     - "Bash"
   model: "claude-sonnet-5"            # Optional: specific model
+  context_window: 200000              # Optional: cap the context window, in tokens (claude-code, codex)
   sdk_options:                        # Optional: agent SDK pass-through (keys depend on `type`)
     effort: high                      # claude-code: any non-framework-managed ClaudeAgentOptions field
 ```
@@ -188,6 +189,13 @@ validates its own keys at YAML load:
 Deep-merged across the 5-layer config chain. Override via CLI with the
 repeatable `-D agent.sdk_options.KEY=VALUE`.
 
+**`context_window`** caps the context window the agent works within, in tokens: the
+harness compacts the conversation before it outgrows the cap, and never runs past the
+model's own window. Override via CLI with `-D agent.context_window=400000`. Which agent
+types accept it, in what range, what each does with it and what an unset value means:
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness). An agent
+type that cannot apply it rejects it at load.
+
 **Permission Modes:**
 - `default` — Default permission handling
 - `acceptEdits` — Auto-accept file edits (recommended for evaluations)

docs/agents/CLAUDE_CODE.md

@@ -104,6 +104,7 @@ agent:
 | `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. |
 | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
 | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
+| `context_window` | `int \| null` (100000-1000000) | Auto-compact window, in tokens, sent as `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. It outranks `--autocompact` and every settings scope, and is capped to the model's window. Setting `claude_settings.autoCompactWindow` too is a load error. See [Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness). |
 | `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. |
 | `ignore_patterns` | `list[str] \| null` | Gitignore-style overrides for the workspace copy used by judge sub-agents (supports `!` negation). |
 

docs/agents/CODEX.md

@@ -192,6 +192,13 @@ The agent maps `permission_mode` to the Codex SDK's `Sandbox`. The approval mode
 
 `allowed_tools` / `disallowed_tools` are normalized (`Bash` → `shell`, `Write`/`Edit` → `apply_patch`, etc.) and passed as `enabled_tools` / `disabled_tools` in the thread `config`. **Note:** the Codex SDK does not currently enforce `disabled_tools`; do not rely on it as a security boundary (the agent logs a warning when it is set).
 
+### Context Window
+
+`agent.context_window` is passed as `model_context_window` in the thread `config`. Codex
+uses it as the model's context window, capped to the model's catalog maximum, and
+auto-compacts at 90% of it. See
+[Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness).
+
 ### Skills Discovery
 
 The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills/` directory:

docs/agents/HARNESS_PARITY.md

@@ -637,6 +637,38 @@ A timeout is a *failure* (partial turn captured, error status); the turn cap is
 *clean stop*. Conflating them is the mistake this page exists to prevent: a task
 whose cap fires should not look like a task whose harness hung.
 
+## The context window cap per harness
+
+`agent.context_window` is not a run limit, but the same promise applies: one value
+must mean the same cap on every harness that accepts it. It is the context window, in
+tokens, that the harness compacts the conversation within. A harness that cannot apply
+it rejects it when the config loads. It is never ignored, because a variant that runs
+uncapped under a "capped" label is a wrong measurement.
+
+| | claude-code | codex | antigravity | opencode | pi | delegate |
+|---|---|---|---|---|---|---|
+| mechanism | `CLAUDE_CODE_AUTO_COMPACT_WINDOW` in the CLI environment | thread config `model_context_window` | rejected | rejected | rejected | rejected |
+| accepted range | 100000-1000000 | 32768 and up | | | | |
+| when compaction starts | when the context approaches the window | at 90% of the window (Codex's default auto-compact limit) | | | | |
+| above the model's own window | capped to the model's window by the CLI | capped to the model's catalog maximum by Codex | | | | |
+| unset | the model's full window: a host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` is blanked | the host's Codex config: a `model_context_window` in `$CODEX_HOME/config.toml` still applies | | | | |
+
+- **claude-code**: the environment variable outranks `--autocompact` and every settings
+  scope, so a host or project `autoCompactWindow` cannot change the cap. Setting
+  `claude_settings.autoCompactWindow` together with `context_window` fails at load when
+  `claude_settings` is an inline dict; a settings file's `autoCompactWindow` is not read,
+  and is overridden the same way. When `context_window` is unset, a host-set
+  `CLAUDE_CODE_AUTO_COMPACT_WINDOW` is blanked, so a run recorded as uncapped is never
+  capped by the host. This applies to every claude-code run, capped or not.
+- **codex**: the value replaces the model's context window, so the hard-cap
+  compaction at the usable window (a per-model share of it, 95% for Codex's fallback
+  model metadata) also moves. The floor of 32768 rejects a window too small to work in,
+  such as a dropped `000` or a YAML `yes`, which would load as 1. When `context_window`
+  is unset, coder_eval sets nothing, so a host `model_context_window` in
+  `$CODEX_HOME/config.toml` still applies.
+- Neither harness reports when the model's own window has lowered the cap. A cap
+  above the model's window runs at the model's window.
+
 ## `agent.plugins[].path` accepts different depths per harness
 
 Not a run limit, but the same promise: one task file, three harnesses, same meaning.

src/coder_eval/agents/claude_code_agent.py

@@ -253,6 +253,8 @@ def _turn_usage_slice(
 
 _JSON_START_SEARCH_LIMIT = 200
 
+AUTO_COMPACT_WINDOW_ENV = "CLAUDE_CODE_AUTO_COMPACT_WINDOW"
+
 
 class _ClaudeTurnState:
     """Per-turn mutable scratch state for one ``ClaudeCodeAgent.communicate`` call.
@@ -1268,6 +1270,14 @@ def _build_claude_query(
             cost_log_tags=cost_log_tags,
         )
         effective_model = self._resolve_effective_model(self.config.model, env, route_model)
+        if self.config.context_window is not None:
+            # The env var outranks --autocompact and every settings scope, and is
+            # read by every CLI the pinned SDKs bundle.
+            env[AUTO_COMPACT_WINDOW_ENV] = str(self.config.context_window)
+        elif os.environ.get(AUTO_COMPACT_WINDOW_ENV):
+            # The SDK spawns the CLI with {**os.environ, **options.env}, so a host-set
+            # window would cap a variant recorded as uncapped. Empty is falsy there.
+            env[AUTO_COMPACT_WINDOW_ENV] = ""
 
         disallowed_tools = list(self.config.disallowed_tools or [])
         # Do not allow ToolSearch. This is required to keep Bedrock backend in sync with the other backends.

src/coder_eval/agents/codex_agent.py

@@ -1500,6 +1500,10 @@ def _build_thread_options(self) -> dict[str, Any]:
                 + f"api_version={'set' if api_version else 'unset'})"
             )
 
+        if self.config.context_window is not None:
+            # Codex then auto-compacts at 90% of this window.
+            tool_config["model_context_window"] = self.config.context_window
+
         if tool_config:
             options["config"] = tool_config