diff --git a/.claude/notes/README.md b/.claude/notes/README.md index 641e4f1d..e74a8b89 100644 --- a/.claude/notes/README.md +++ b/.claude/notes/README.md @@ -13,6 +13,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`. ## Contents - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity +- [context-window.md](context-window.md) — the `agent.context_window` cap, per harness - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading - [lint-rules.md](lint-rules.md) — why each CE lint rule exists diff --git a/.claude/notes/context-window.md b/.claude/notes/context-window.md new file mode 100644 index 00000000..b1611532 --- /dev/null +++ b/.claude/notes/context-window.md @@ -0,0 +1,170 @@ +# Capping the agent's context window + +Analysis and design for `agent.context_window`: one harness-neutral field that caps the +context window an agent works within, set per agent config or per experiment variant. + +## Problem and evidence + +A benchmark of Claude Code on Sonnet 5 (Bedrock, 1M-token window) let runs grow to +250k-480k tokens of context. Above about 150k tokens each extended-thinking step got +about 10x slower, so most runs hit the 2-hour task limit. The operator wants to measure +pass rate and cost at several context caps, for example one variant at 200k and one at +400k, on the same tasks. + +Before this change no harness-neutral way existed: + +- **claude-code**: `claude_settings` (`--settings`) could carry `autoCompactWindow`. That + works for one harness only, and when `claude_settings` is a file path a variant cannot + merge one key into it. +- **codex**: `CodexAgentConfig` has no pass-through. `CodexAgent._build_thread_options` + builds the thread `config` dict itself, so no value reached Codex. +- **antigravity, opencode, pi, delegate**: no knob is wired. + +## How each harness controls its window + +### claude-code + +Verified in the CLI the SDK spawns. `claude-agent-sdk` runs its **bundled** CLI before it +looks for `claude` on `PATH` (`SubprocessCLITransport._find_cli`), so the CLI that runs is +the one the SDK wheel bundles, not the one the agent image installs with npm: + +| coder_eval | claude-agent-sdk (uv.lock) | bundled CLI | image npm CLI | +|---|---|---|---| +| 0.12.1 | 0.2.124 | 2.1.216 | 2.1.177 | +| 0.12.12 | 0.2.159 | 2.1.281 | 2.1.281 | + +`environment_info.claude_code_cli` records `claude -v` from `PATH`, so a 0.12.1 run +records 2.1.177 while 2.1.216 ran. That is a separate defect and is not fixed here. + +Three inputs set the auto-compact window. Both bundled CLIs (2.1.216 and 2.1.281) contain +the same resolution function. The highest precedence comes first: + +1. `CLAUDE_CODE_AUTO_COMPACT_WINDOW` environment variable: a plain token count. It + outranks the flag and every settings scope. +2. `--autocompact `: added in 2.1.221 (CLI reference). It is absent from + 2.1.216 as a parsed option. +3. `autoCompactWindow` settings key. The settings schema is + `int().min(100000).max(1000000).optional().catch(undefined)`, so an out-of-range + value is **silently dropped**. + +The effective window is `min(model context window, configured value)`. Auto-compaction +starts when the context approaches that window. + +### codex + +Verified in the `openai-codex-cli-bin` 0.156.1 binary (the pin) and its source at tag +`rust-v0.156.1`: + +- `model_context_window` ("Size of the context window for the model, in tokens"): + `with_config_overrides` sets the model's `context_window` to + `min(value, max_context_window)`. Three things follow from it. The default + auto-compact limit becomes 90% of it. The hard cap, which forces compaction, becomes + `context_window * effective_context_window_percent / 100`. The remaining-window figure + the model is shown also changes. +- `model_auto_compact_token_limit` ("Token usage threshold triggering auto-compaction"): + the trigger only, clamped to 90% of the resolved window. It does not change the window. +- `model_auto_compact_token_limit_scope` (`total` by default, or `body_after_prefix`) and + `model_post_turn_compact_threshold_percent` (turn-end compaction, off by default) tune + when compaction starts. They are not caps. + +The thread `config` dict `thread_start` takes is the same override surface that +coder_eval already uses for `enabled_tools` and `model_providers`. + +### antigravity, opencode, pi, delegate + +None of them has a context-window knob wired in coder_eval. They are out of scope, and +they reject the field. + +## Options considered + +**Per-harness pass-through** (a Codex `config` pass-through next to `claude_settings`). +It is flexible, but a variant would have to name a different key per harness, and the +recorded value would mean something different on each. It also opens a free-form +pass-through on Codex with no denylist. Rejected. + +**A field under `run_limits`.** Run limits are caps that the orchestrator or the agent +enforces on a run (turns, time, tokens, USD), and the parity table lists them. A context +window is not a spend cap. It is a harness setting that changes what the agent does, and +it depends on the agent type, which `run_limits` does not know. The guide also says that +`agent:` holds no run-time caps. Rejected. + +**A field on the agent config.** One name, accepted on a type-less config so that +experiment defaults and variants can set it, and validated against the concrete type +when the config resolves. **Chosen.** + +Naming: `context_window`, in tokens. It names the quantity the operator caps, matches +Codex's `model_context_window`, and avoids "auto-compact", which describes a mechanism of +one harness. `max_context_tokens` was considered, but it reads as a per-request input +limit, which neither harness enforces. + +## Recommended design + +`BaseAgentConfig.context_window: int | None` (default `None`, `gt=0`). + +- Each config class declares `_context_window_range: ClassVar[tuple[int, int | None] | + None]`. `None` (the base default) means that the harness cannot apply the field. + `ClaudeCodeAgentConfig` declares `(100_000, 1_000_000)` and `CodexAgentConfig` declares + `(32_768, None)`. +- `check_context_window_supported` (model validator) rejects an unsupported type or an + out-of-range value as soon as the type is known. A plugin agent inherits `None`, so it + also rejects the field until it opts in. +- The field uses the default `replace` merge strategy. It is set at any of the five + layers, and with `-D agent.context_window=N`. + +Per-harness mapping: + +- **claude-code**: `CLAUDE_CODE_AUTO_COMPACT_WINDOW=` in the SDK `env`. The + environment variable was chosen over the settings key and the flag for four reasons. + It has the highest precedence, so host, project and `claude_settings` values cannot + override the cap. It works whether `claude_settings` is a dict or a file path. Both + bundled CLIs read it, which the flag does not. `extra_args` stays framework-owned. + `claude_settings.autoCompactWindow` together with `context_window` is a load error, + because the environment variable would silently override it. +- **codex**: `model_context_window: ` in the thread `config`. No trigger key is set, + so Codex compacts at its default 90% of the window. + +## Validation + +- `gt=0` on every config, so a type-less config rejects nonsense as well. +- claude-code: 100000-1000000, the CLI's documented range. Outside that range the CLI + drops a settings value silently, so coder_eval enforces the range itself. +- codex: 32768 and up. Codex clamps the value to the model's catalog maximum, and the + floor rejects a window too small to work in, such as a YAML `yes` coerced to 1. +- every other type: rejected at load, naming the supported types. A variant that + switches `type` to an unsupported harness fails when the experiment resolves. + +## Recording + +The resolved agent config is persisted as `agent_config` on every task row (`run.json` +and `task.json`). `agent.context_window` therefore appears per task, and config lineage +records which layer set it (for example, the variant). The experiment report groups by +variant, so a cap per variant groups without new report code. On claude-code the +`sdk_options` dump also shows the environment variable in `env`. + +## Open questions + +- **A silent clamp to the model's window.** Neither harness reports that the model's + window lowered the cap. A cap above the model's window runs at the model's window. + Recording the effective window would need a per-model catalog in coder_eval. +- **Trigger points differ.** Codex compacts at 90% of the window. Claude Code compacts + when the context approaches the window, minus an internal buffer. Equal caps therefore + do not compact at exactly the same token count. A separate harness-neutral trigger + field is possible, but it is not added until a benchmark needs it. +- **A host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` and uncapped runs.** Decided: the SDK starts + the CLI with `os.environ` underneath `options.env`, so when the field is unset the + claude-code agent blanks a host value. A run recorded as uncapped is never capped by + the host. This changes every claude-code run on a host that sets the variable, + including runs that never set `context_window`. Codex has no equivalent: an unset + field leaves a host `model_context_window` in `$CODEX_HOME/config.toml` in effect. +- **The recorded CLI version is wrong.** See the version table above: + `environment_info.claude_code_cli` should record the bundled CLI version. +- **Other harnesses.** Pi can lower a model's window with + `providers..modelOverrides..contextWindow` in `models.json`. A mapping + was written and then reverted (d1757a60), because no live run backed it and no task + uses `type: pi`; it comes back with a live test. OpenCode (`limit.context` and + `limit.input` in provider config) needs a `limit.output` coder_eval does not know, and + an unpinned CLI. Antigravity's `CompactionConfig(token_threshold=N)` only moves the + trigger. Delegate keeps its history in the backend. Each needs its own verification + before it opts in. +- **How often a run compacted is not recorded.** A cap A/B cannot yet tell "cap never + reached" from "compacted N times" from the task row alone. diff --git a/docs/AB_EXPERIMENTS.md b/docs/AB_EXPERIMENTS.md index afe35d66..76b1551a 100644 --- a/docs/AB_EXPERIMENTS.md +++ b/docs/AB_EXPERIMENTS.md @@ -23,6 +23,7 @@ to the orchestrator are involved. - [What a Variant Can Override](#what-a-variant-can-override) - [Recipe: A/B a Skill](#recipe-ab-a-skill) - [Recipe: A/B a Model](#recipe-ab-a-model) +- [Recipe: A/B a Context Window](#recipe-ab-a-context-window) - [Recipe: A/B a Prompt](#recipe-ab-a-prompt) - [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop) - [Replicates (Statistical Power)](#replicates-statistical-power) @@ -222,6 +223,28 @@ variants: agent: { model: claude-opus-5 } ``` +## Recipe: A/B a Context Window + +`agent.context_window` caps the context the agent works within, in tokens: the +harness compacts the conversation before it outgrows the cap. Use it to measure +pass rate and cost against the cap on a model with a large window. Each task row +records the cap in `agent_config.context_window`. + +```yaml +experiment_id: context-window +description: "The same model with a 200k and a 400k context cap" + +variants: + - variant_id: cap-200k + agent: { context_window: 200000 } + - variant_id: cap-400k + agent: { context_window: 400000 } +``` + +An agent type that cannot apply the cap fails the experiment at load. +[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness) lists which +agent types accept it and what the cap does on each. + ## Recipe: A/B a Prompt Use `prompt_mutations` (transform the task prompt) or `initial_prompt` / diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index 84214674..32fdccad 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -80,6 +80,22 @@ class MyAgentConfig(BaseAgentConfig): The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config isn't an instance of the registered `config_class`, so keep them paired. +To accept `agent.context_window`, declare the inclusive range of tokens your harness +can cap its context window to, and apply `self.config.context_window` when the agent +starts. A config class that declares nothing rejects the field at load, which is the +right default: a capped variant must never run uncapped. + +```python +from typing import ClassVar + +class MyAgentConfig(BaseAgentConfig): + type: Literal["my-agent"] = "my-agent" + _context_window_range: ClassVar[tuple[int, int | None] | None] = (32_768, None) +``` + +Add your harness to the table in +[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness). + ### The `Agent` ABC — implementation checklist Implement these three abstract methods: diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index 373210d5..d3caa64d 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -170,6 +170,7 @@ agent: - "Write" - "Bash" model: "claude-sonnet-5" # Optional: specific model + context_window: 200000 # Optional: cap the context window, in tokens (claude-code, codex) sdk_options: # Optional: agent SDK pass-through (keys depend on `type`) effort: high # claude-code: any non-framework-managed ClaudeAgentOptions field ``` @@ -188,6 +189,13 @@ validates its own keys at YAML load: Deep-merged across the 5-layer config chain. Override via CLI with the repeatable `-D agent.sdk_options.KEY=VALUE`. +**`context_window`** caps the context window the agent works within, in tokens: the +harness compacts the conversation before it outgrows the cap, and never runs past the +model's own window. Override via CLI with `-D agent.context_window=400000`. Which agent +types accept it, in what range, what each does with it and what an unset value means: +[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness). An agent +type that cannot apply it rejects it at load. + **Permission Modes:** - `default` — Default permission handling - `acceptEdits` — Auto-accept file edits (recommended for evaluations) diff --git a/docs/agents/CLAUDE_CODE.md b/docs/agents/CLAUDE_CODE.md index e3473089..4145e99b 100644 --- a/docs/agents/CLAUDE_CODE.md +++ b/docs/agents/CLAUDE_CODE.md @@ -104,6 +104,7 @@ agent: | `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. | | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). | | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. | +| `context_window` | `int \| null` (100000-1000000) | Auto-compact window, in tokens, sent as `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. It outranks `--autocompact` and every settings scope, and is capped to the model's window. Setting `claude_settings.autoCompactWindow` too is a load error. See [Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness). | | `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. | | `ignore_patterns` | `list[str] \| null` | Gitignore-style overrides for the workspace copy used by judge sub-agents (supports `!` negation). | diff --git a/docs/agents/CODEX.md b/docs/agents/CODEX.md index 29a98517..12b8e33e 100644 --- a/docs/agents/CODEX.md +++ b/docs/agents/CODEX.md @@ -192,6 +192,13 @@ The agent maps `permission_mode` to the Codex SDK's `Sandbox`. The approval mode `allowed_tools` / `disallowed_tools` are normalized (`Bash` → `shell`, `Write`/`Edit` → `apply_patch`, etc.) and passed as `enabled_tools` / `disabled_tools` in the thread `config`. **Note:** the Codex SDK does not currently enforce `disabled_tools`; do not rely on it as a security boundary (the agent logs a warning when it is set). +### Context Window + +`agent.context_window` is passed as `model_context_window` in the thread `config`. Codex +uses it as the model's context window, capped to the model's catalog maximum, and +auto-compacts at 90% of it. See +[Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness). + ### Skills Discovery The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills/` directory: diff --git a/docs/agents/HARNESS_PARITY.md b/docs/agents/HARNESS_PARITY.md index 463655e9..b7d3790e 100644 --- a/docs/agents/HARNESS_PARITY.md +++ b/docs/agents/HARNESS_PARITY.md @@ -637,6 +637,38 @@ A timeout is a *failure* (partial turn captured, error status); the turn cap is *clean stop*. Conflating them is the mistake this page exists to prevent: a task whose cap fires should not look like a task whose harness hung. +## The context window cap per harness + +`agent.context_window` is not a run limit, but the same promise applies: one value +must mean the same cap on every harness that accepts it. It is the context window, in +tokens, that the harness compacts the conversation within. A harness that cannot apply +it rejects it when the config loads. It is never ignored, because a variant that runs +uncapped under a "capped" label is a wrong measurement. + +| | claude-code | codex | antigravity | opencode | pi | delegate | +|---|---|---|---|---|---|---| +| mechanism | `CLAUDE_CODE_AUTO_COMPACT_WINDOW` in the CLI environment | thread config `model_context_window` | rejected | rejected | rejected | rejected | +| accepted range | 100000-1000000 | 32768 and up | | | | | +| when compaction starts | when the context approaches the window | at 90% of the window (Codex's default auto-compact limit) | | | | | +| above the model's own window | capped to the model's window by the CLI | capped to the model's catalog maximum by Codex | | | | | +| unset | the model's full window: a host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` is blanked | the host's Codex config: a `model_context_window` in `$CODEX_HOME/config.toml` still applies | | | | | + +- **claude-code**: the environment variable outranks `--autocompact` and every settings + scope, so a host or project `autoCompactWindow` cannot change the cap. Setting + `claude_settings.autoCompactWindow` together with `context_window` fails at load when + `claude_settings` is an inline dict; a settings file's `autoCompactWindow` is not read, + and is overridden the same way. When `context_window` is unset, a host-set + `CLAUDE_CODE_AUTO_COMPACT_WINDOW` is blanked, so a run recorded as uncapped is never + capped by the host. This applies to every claude-code run, capped or not. +- **codex**: the value replaces the model's context window, so the hard-cap + compaction at the usable window (a per-model share of it, 95% for Codex's fallback + model metadata) also moves. The floor of 32768 rejects a window too small to work in, + such as a dropped `000` or a YAML `yes`, which would load as 1. When `context_window` + is unset, coder_eval sets nothing, so a host `model_context_window` in + `$CODEX_HOME/config.toml` still applies. +- Neither harness reports when the model's own window has lowered the cap. A cap + above the model's window runs at the model's window. + ## `agent.plugins[].path` accepts different depths per harness Not a run limit, but the same promise: one task file, three harnesses, same meaning. diff --git a/src/coder_eval/agents/claude_code_agent.py b/src/coder_eval/agents/claude_code_agent.py index 3a626769..aa34da93 100644 --- a/src/coder_eval/agents/claude_code_agent.py +++ b/src/coder_eval/agents/claude_code_agent.py @@ -253,6 +253,8 @@ def _turn_usage_slice( _JSON_START_SEARCH_LIMIT = 200 +AUTO_COMPACT_WINDOW_ENV = "CLAUDE_CODE_AUTO_COMPACT_WINDOW" + class _ClaudeTurnState: """Per-turn mutable scratch state for one ``ClaudeCodeAgent.communicate`` call. @@ -1268,6 +1270,14 @@ def _build_claude_query( cost_log_tags=cost_log_tags, ) effective_model = self._resolve_effective_model(self.config.model, env, route_model) + if self.config.context_window is not None: + # The env var outranks --autocompact and every settings scope, and is + # read by every CLI the pinned SDKs bundle. + env[AUTO_COMPACT_WINDOW_ENV] = str(self.config.context_window) + elif os.environ.get(AUTO_COMPACT_WINDOW_ENV): + # The SDK spawns the CLI with {**os.environ, **options.env}, so a host-set + # window would cap a variant recorded as uncapped. Empty is falsy there. + env[AUTO_COMPACT_WINDOW_ENV] = "" disallowed_tools = list(self.config.disallowed_tools or []) # Do not allow ToolSearch. This is required to keep Bedrock backend in sync with the other backends. diff --git a/src/coder_eval/agents/codex_agent.py b/src/coder_eval/agents/codex_agent.py index bb3746a1..2430eda3 100644 --- a/src/coder_eval/agents/codex_agent.py +++ b/src/coder_eval/agents/codex_agent.py @@ -1500,6 +1500,10 @@ def _build_thread_options(self) -> dict[str, Any]: + f"api_version={'set' if api_version else 'unset'})" ) + if self.config.context_window is not None: + # Codex then auto-compacts at 90% of this window. + tool_config["model_context_window"] = self.config.context_window + if tool_config: options["config"] = tool_config diff --git a/src/coder_eval/models/agent_config.py b/src/coder_eval/models/agent_config.py index 27ce5974..e65f627a 100644 --- a/src/coder_eval/models/agent_config.py +++ b/src/coder_eval/models/agent_config.py @@ -122,6 +122,9 @@ class BaseAgentConfig(BaseModel): # Cross-field merge exclusion: setting either prompt field at any layer clears # the sibling. ClassVar -> not a model field. _merge_exclusive_groups: ClassVar[tuple[tuple[str, ...], ...]] = (("system_prompt", "system_prompt_file"),) + # Inclusive (min, max) tokens the harness can cap its context window to; None + # means the harness has no such knob and ``context_window`` is rejected. + _context_window_range: ClassVar[tuple[int, int | None] | None] = None type: str | None = Field( default=None, @@ -162,6 +165,17 @@ class BaseAgentConfig(BaseModel): "Mutually exclusive with system_prompt." ), ) + context_window: int | None = Field( + default=None, + gt=0, + description=( + "Cap, in tokens, on the context window the agent works within: the harness compacts " + "the conversation before it outgrows this size, and never runs past the model's own " + "window. None leaves the harness's default. Each agent type declares whether it applies " + "the cap and in what range; one that cannot rejects it at load. See " + "docs/agents/HARNESS_PARITY.md#the-context-window-cap-per-harness." + ), + ) # Customizable ignore patterns for file tracking ignore_patterns: list[str] = MergeField( @@ -203,12 +217,43 @@ def check_prompt_exclusivity(self) -> Self: raise ValueError("Only one of 'system_prompt' or 'system_prompt_file' can be provided, not both") return self + @model_validator(mode="after") + def check_context_window_supported(self) -> Self: + """Reject a ``context_window`` the resolved agent type cannot apply. + + A type-less config defers the check to the concrete config it resolves to. + An unsupported or out-of-range value fails here, at load, because a harness + that drops it would run the variant uncapped and still label it capped. + + Rationale: .claude/notes/context-window.md § Recommended design + """ + if self.context_window is None or self.type is None: + return self + bounds = type(self)._context_window_range + if bounds is None: + raise ValueError( + f"context_window is not supported by agent type {self.type!s}; " + + f"supported agent types: {', '.join(_kinds_supporting_context_window())}" + ) + low, high = bounds + if self.context_window < low or (high is not None and self.context_window > high): + allowed = f"{low}-{high}" if high is not None else f">= {low}" + raise ValueError( + f"context_window {self.context_window} is out of range for agent type {self.type!s} " + + f"(allowed: {allowed} tokens)" + ) + return self + class ClaudeCodeAgentConfig(BaseAgentConfig): """Claude Code agent configuration.""" type: Literal[AgentKind.CLAUDE_CODE] # type: ignore[assignment] + # The CLI's own accepted auto-compact window range; it silently drops a + # settings value outside it, so the bound is enforced here instead. + _context_window_range: ClassVar[tuple[int, int | None] | None] = (100_000, 1_000_000) + system_prompt_mode: SystemPromptMode = Field( default="append", description=( @@ -285,12 +330,38 @@ def check_replace_mode_has_prompt(self) -> Self: ) return self + @model_validator(mode="after") + def check_single_auto_compact_source(self) -> Self: + """Reject ``context_window`` alongside ``claude_settings.autoCompactWindow``. + + ``context_window`` reaches the CLI as ``CLAUDE_CODE_AUTO_COMPACT_WINDOW``, which + outranks the settings key, so the settings value would be silently ignored. Only an + inline dict can be checked here; when ``claude_settings`` is a file path, the file's + ``autoCompactWindow`` is overridden the same way, and the run is still capped at + ``context_window``, which is what the task row records. + """ + if ( + self.context_window is not None + and isinstance(self.claude_settings, dict) + and "autoCompactWindow" in self.claude_settings + ): + raise ValueError( + "context_window and claude_settings.autoCompactWindow both set the auto-compact window; " + + "set only context_window" + ) + return self + class CodexAgentConfig(BaseAgentConfig): """Codex agent configuration.""" type: Literal[AgentKind.CODEX] # type: ignore[assignment] + # Codex clamps the value to the model's catalog maximum itself and compacts at 90% of it. + # The floor rejects a window too small to work in, such as a dropped "000" or a YAML + # `yes` that loads as 1. + _context_window_range: ClassVar[tuple[int, int | None] | None] = (32_768, None) + # Mirrors google.antigravity.types.ThinkingLevel as a plain Literal so this module # imports without the optional SDK -- base installs must load every config class. @@ -546,6 +617,17 @@ def parse_agent_config(**kwargs: Any) -> BaseAgentConfig: return registration.config_class.model_validate(kwargs) +def _kinds_supporting_context_window() -> list[str]: + """Registered agent kinds whose config class can apply ``context_window``.""" + from coder_eval.agents.registry import AgentRegistry + + return [ + kind + for kind in sorted(AgentRegistry.list_kinds()) + if (reg := AgentRegistry.get(kind)) is not None and reg.config_class._context_window_range is not None + ] + + def _coerce_agent_config(value: Any) -> Any: """Coerce a raw agent-config dict to its registered subclass via the registry. diff --git a/src/coder_eval/reports/experiment.py b/src/coder_eval/reports/experiment.py index 2edb3232..4501c92a 100644 --- a/src/coder_eval/reports/experiment.py +++ b/src/coder_eval/reports/experiment.py @@ -25,7 +25,7 @@ paired_comparison, ) from .html import write_experiment_html, write_variant_html -from .markdown import ReportGenerator, resolve_agent_settings +from .markdown import ReportGenerator, resolve_agent_settings, resolve_context_window logger = logging.getLogger(__name__) @@ -536,7 +536,11 @@ def generate_variant_report(variant_id: str, result: ExperimentResult, run_dir: settings_source, is_sdk = resolve_agent_settings(task_dicts) if settings_source: lines.append("") - lines.extend(ReportGenerator._generate_agent_settings_section(settings_source, is_sdk)) + lines.extend( + ReportGenerator._generate_agent_settings_section( + settings_source, is_sdk, context_window=resolve_context_window(task_dicts) + ) + ) # Installed Tools installed_tools_lines = ReportGenerator._generate_installed_tools_section(task_dicts) diff --git a/src/coder_eval/reports/html.py b/src/coder_eval/reports/html.py index 16f04880..eb202f6e 100644 --- a/src/coder_eval/reports/html.py +++ b/src/coder_eval/reports/html.py @@ -1045,7 +1045,10 @@ def _render_agent_settings(result: EvaluationResult) -> str: return "" rows = collect_agent_settings_rows( - settings, is_sdk, system_prompt_semantics=(result.environment_info or {}).get("system_prompt_semantics") + settings, + is_sdk, + system_prompt_semantics=(result.environment_info or {}).get("system_prompt_semantics"), + context_window=result.agent_config.context_window if result.agent_config is not None else None, ) body = "".join(f"{_esc(label)}{_esc(value)}" for label, value in rows) return f""" diff --git a/src/coder_eval/reports/markdown.py b/src/coder_eval/reports/markdown.py index 70ffc52c..6146729c 100644 --- a/src/coder_eval/reports/markdown.py +++ b/src/coder_eval/reports/markdown.py @@ -63,6 +63,19 @@ def resolve_agent_settings(task_dicts: list[dict[str, Any]]) -> tuple[dict[str, return None, False +def resolve_context_window(task_dicts: list[dict[str, Any]]) -> int | None: + """The ``agent.context_window`` the tasks ran with, or None when they ran uncapped. + + Read from ``agent_config`` because the value never reaches ``sdk_options``: Claude Code + receives it in the CLI environment and Codex in the thread config. + """ + for task in task_dicts: + config = task.get("agent_config") + if isinstance(config, dict) and config.get("context_window") is not None: + return int(config["context_window"]) + return None + + def _unwrap_system_prompt(value: Any) -> tuple[str | None, Literal["replace"] | None]: """Reduce a persisted ``sdk_options.system_prompt`` value to (text, regime). @@ -90,6 +103,7 @@ def collect_agent_settings_rows( is_sdk: bool, *, system_prompt_semantics: str | None = None, + context_window: int | None = None, ) -> list[tuple[str, str]]: """Extract ordered label/value pairs from an agent settings dict. @@ -110,6 +124,9 @@ def collect_agent_settings_rows( model = settings_source.get("model") if model: rows.append(("Model", str(model))) + window = context_window if context_window is not None else settings_source.get("context_window") + if window is not None: + rows.append(("Context Window", f"{int(window):,} tokens")) if is_sdk: for key, label in ( @@ -651,7 +668,10 @@ def generate_markdown(summary: RunSummary, run_dir: Path | None = None) -> str: semantics = (summary.environment_info or {}).get("system_prompt_semantics") lines.extend( ReportGenerator._generate_agent_settings_section( - settings_source, is_sdk, system_prompt_semantics=semantics + settings_source, + is_sdk, + system_prompt_semantics=semantics, + context_window=resolve_context_window(summary.task_results), ) ) @@ -672,11 +692,12 @@ def _generate_agent_settings_section( is_sdk: bool, *, system_prompt_semantics: str | None = None, + context_window: int | None = None, ) -> list[str]: """Generate Agent Settings markdown lines from a settings dict.""" lines = ["## Agent Settings", ""] for label, value in collect_agent_settings_rows( - settings_source, is_sdk, system_prompt_semantics=system_prompt_semantics + settings_source, is_sdk, system_prompt_semantics=system_prompt_semantics, context_window=context_window ): lines.append(f"- **{label}**: {value}") return lines diff --git a/tests/test_agent_context_window.py b/tests/test_agent_context_window.py new file mode 100644 index 00000000..2eb5f49e --- /dev/null +++ b/tests/test_agent_context_window.py @@ -0,0 +1,152 @@ +"""``agent.context_window``: one harness-neutral cap, applied per harness or rejected at load.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from coder_eval.agents.claude_code_agent import AUTO_COMPACT_WINDOW_ENV, ClaudeCodeAgent +from coder_eval.models import AgentKind, BaseAgentConfig, TaskDefinition, parse_agent_config +from coder_eval.orchestration.config_merge import Layer, resolve_root, validate_paths +from coder_eval.orchestration.overrides import OverrideError, apply_overrides + + +def _claude_env(**config_kwargs) -> dict[str, str]: + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, **config_kwargs)) + agent.working_directory = Path(".") + options, _transport, _model = agent._build_claude_query("hi", None, None, lambda _line: None) + return options.env + + +def _task(agent: dict) -> TaskDefinition: + return TaskDefinition.model_validate( + {"task_id": "cw", "description": "d", "initial_prompt": "p", "agent": agent, "success_criteria": []} + ) + + +class TestClaudeCode: + def test_a_capped_variant_sends_the_window_to_the_cli(self): + assert _claude_env(context_window=200_000)[AUTO_COMPACT_WINDOW_ENV] == "200000" + + def test_an_uncapped_variant_leaves_the_cli_default(self): + assert AUTO_COMPACT_WINDOW_ENV not in _claude_env() + + def test_an_uncapped_variant_ignores_a_window_set_on_the_host(self, monkeypatch): + monkeypatch.setenv(AUTO_COMPACT_WINDOW_ENV, "150000") + assert _claude_env()[AUTO_COMPACT_WINDOW_ENV] == "" + assert _claude_env(context_window=300_000)[AUTO_COMPACT_WINDOW_ENV] == "300000" + + @pytest.mark.parametrize("window", [99_999, 1_000_001]) + def test_a_window_outside_the_cli_range_fails_at_load(self, window): + with pytest.raises(ValidationError, match="out of range for agent type claude-code"): + parse_agent_config(type=AgentKind.CLAUDE_CODE, context_window=window) + + @pytest.mark.parametrize("window", [100_000, 1_000_000]) + def test_the_ends_of_the_cli_range_are_accepted(self, window): + assert parse_agent_config(type=AgentKind.CLAUDE_CODE, context_window=window).context_window == window + + def test_a_second_auto_compact_source_fails_at_load(self): + with pytest.raises(ValidationError, match=r"claude_settings.autoCompactWindow"): + parse_agent_config( + type=AgentKind.CLAUDE_CODE, + context_window=200_000, + claude_settings={"autoCompactWindow": 400_000}, + ) + + +class TestCodex: + def test_a_capped_variant_sets_the_model_context_window(self): + pytest.importorskip("openai_codex") + from coder_eval.agents.codex_agent import CodexAgent + + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, context_window=400_000)) + assert agent._build_thread_options()["config"]["model_context_window"] == 400_000 + + def test_an_uncapped_variant_leaves_the_codex_default(self): + pytest.importorskip("openai_codex") + from coder_eval.agents.codex_agent import CodexAgent + + options = CodexAgent(parse_agent_config(type=AgentKind.CODEX))._build_thread_options() + assert "model_context_window" not in options.get("config", {}) + + @pytest.mark.parametrize("window", [32_767, 1_000, True]) + def test_a_window_too_small_to_work_in_fails_at_load(self, window): + with pytest.raises(ValidationError, match="out of range for agent type codex"): + parse_agent_config(type=AgentKind.CODEX, context_window=window) + + def test_the_floor_itself_is_accepted(self): + assert parse_agent_config(type=AgentKind.CODEX, context_window=32_768).context_window == 32_768 + + +class TestUnsupportedHarnesses: + @pytest.mark.parametrize( + "kind", [AgentKind.ANTIGRAVITY, AgentKind.OPENCODE, AgentKind.PI, AgentKind.DELEGATE, AgentKind.NONE] + ) + def test_a_harness_without_the_knob_fails_at_load(self, kind): + with pytest.raises(ValidationError, match="supported agent types: claude-code, codex"): + parse_agent_config(type=kind, context_window=200_000) + + def test_a_non_positive_window_fails_even_before_the_type_is_known(self): + with pytest.raises(ValidationError): + parse_agent_config(context_window=0) + + +class TestReports: + def test_the_agent_settings_name_the_window_a_capped_run_used(self): + from coder_eval.reports.markdown import collect_agent_settings_rows, resolve_context_window + + tasks = [{"agent_config": {"type": "codex"}}, {"agent_config": {"type": "codex", "context_window": 200_000}}] + window = resolve_context_window(tasks) + assert window == 200_000 + sdk_rows = dict(collect_agent_settings_rows({"permission_mode": "acceptEdits"}, True, context_window=window)) + assert sdk_rows["Context Window"] == "200,000 tokens" + config_rows = dict(collect_agent_settings_rows({"context_window": 400_000}, False)) + assert config_rows["Context Window"] == "400,000 tokens" + + def test_an_uncapped_run_shows_no_window(self): + from coder_eval.reports.markdown import collect_agent_settings_rows, resolve_context_window + + assert resolve_context_window([{"agent_config": {"type": "claude-code"}}, {}]) is None + assert "Context Window" not in dict(collect_agent_settings_rows({"permission_mode": "acceptEdits"}, True)) + + +class TestConfigLayers: + def test_a_type_less_task_defers_the_check_to_the_resolved_type(self): + assert isinstance(parse_agent_config(context_window=50_000), BaseAgentConfig) + + def test_a_variant_caps_the_task_it_runs_and_records_it(self): + lineage: dict = {} + resolved = resolve_root( + "agent", + [ + Layer(source="task", patch={"type": "claude-code"}), + Layer(source="variant", patch={"context_window": 200_000}, detail="variant cap-200k"), + ], + lineage=lineage, + ) + assert resolved is not None + assert resolved.model_dump()["context_window"] == 200_000 + assert lineage["agent.context_window"].source == "variant" + + def test_a_variant_switching_to_an_unsupported_type_fails_at_load(self): + with pytest.raises(ValidationError, match="not supported by agent type opencode"): + resolve_root( + "agent", + [ + Layer(source="experiment-defaults", patch={"context_window": 200_000}), + Layer(source="variant", patch={"type": "opencode"}), + ], + ) + + def test_a_cli_override_sets_the_cap(self): + validate_paths(["agent.context_window"]) + task = _task({"type": "codex"}) + apply_overrides(task, {"agent.context_window": 400_000}) + assert task.agent is not None and task.agent.context_window == 400_000 + + def test_a_cli_override_out_of_range_reads_as_a_clean_error(self): + task = _task({"type": "claude-code"}) + with pytest.raises(OverrideError, match=r"-D agent: .*out of range"): + apply_overrides(task, {"agent.context_window": 50_000})