feat: eval baseline arm — measure LCD vs bare Claude Code

.claude-plugin/marketplace.json

@@ -12,7 +12,7 @@
       "name": "lcd",
       "source": "./",
       "description": "Lean Context Development — adaptive-weight agentic dev methodology (Quick / Standard / Deep lanes).",
-      "version": "0.16.0"
+      "version": "0.17.0"
     }
   ]
 }

.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "lcd",
   "displayName": "Lean Context Development",
-  "version": "0.16.0",
+  "version": "0.17.0",
   "description": "Context-economy agentic dev methodology: process weight adapts to the work (Quick / Standard / Deep lanes) and durable artifacts make a context reset cheap.",
   "author": {
     "name": "stid"

CHANGELOG.md

@@ -6,6 +6,30 @@ All notable changes to this project are documented here. Format loosely follows
 Development is tracked by decision IDs (`D-NNN`) in the project's internal decision log; entries
 cite them for traceability.
 
+## [0.17.0] — 2026-08-22
+
+The benchmark gains its missing comparison: a bare-agent baseline arm, so "context economy
+pays" is a measured claim, not an argument (dev-harness decision D-025).
+
+### Added
+- **`evals/run-eval.sh --baseline`** — bare-agent arm: no plugin loads, the workspace copy is
+  stripped of the fixture's methodology onboarding (strip verified fail-closed), the run is
+  driven by a new frozen prompt (`evals/feature-prompt-baseline.md`) whose feature paragraph
+  is byte-identical to the LCD prompt's (test-enforced), and the finish line is an `EVAL-DONE`
+  marker. `--baseline` + `--plugin-dir` is refused as a contradiction.
+- **`evals/grade.sh --baseline`** — grades bare workspaces with the identical row shape
+  (`audit: n/a`); fail-closed both ways (residue under `--baseline` refused; bare workspace
+  without it still refused). Golden-locked grading criteria unchanged.
+- **`tests/test-eval-baseline.sh`** — fixture tests for all of the above.
+
+### Changed
+- **Both eval arms now run config-isolated, symmetrically:** `CLAUDE_CONFIG_DIR` points at a
+  fresh empty dir and `--settings '{"enabledPlugins":[]}'` is passed, so user-level installed
+  plugins and the personal `~/.claude/CLAUDE.md` no longer leak into benchmark runs (they did
+  by default; `--plugin-dir` adds to, not replaces, the installed set). The arms differ by
+  exactly one flag: `--plugin-dir`. Rows recorded before this isolation are not comparable
+  with rows after it.
+
 ## [0.16.0] — 2026-08-22
 
 From a full review of the plugin's prompts against Anthropic's current published guidance

evals/feature-prompt-baseline.md

@@ -0,0 +1,22 @@
+# Scripted feature — baseline arm (FROZEN — the benchmark's constant)
+
+> This prompt is handed verbatim to `claude -p` inside a fresh bare-project workspace
+> derived from `evals/fixture/` with the methodology onboarding stripped. Its feature
+> paragraph is byte-identical to `feature-prompt.md`'s (enforced by
+> `tests/test-eval-baseline.sh`); only the process wrapper differs — that difference IS
+> the experiment. Identical wording across runs of one experiment — do not edit between
+> runs.
+
+You are working in a small Node.js project. Build the following feature end-to-end,
+writing tests for everything you add and committing as you go:
+
+Add a `stats` operation exposed via both the CLI and HTTP: it returns the record
+count, a last-updated indicator, and a checksum over the records; validation and
+error semantics must be identical on both surfaces, with the shared logic living in
+the core module both surfaces already use.
+
+Finish with the full test suite green. Work autonomously — do not ask questions, and do
+not stop between steps: keep going in this same turn until the feature is complete and
+the suite is green. When, and only when, that holds, create a file named `EVAL-DONE` at
+the repository root as your final action. If hard-blocked, record why in `BLOCKED.md` at
+the repository root and stop.

evals/grade.sh

@@ -13,12 +13,19 @@
 # verdict, not the grader's exit code); 2 on unusable input (fail closed: this is an
 # instrument, unlike the plugin's fail-open hooks).
 #
-# Usage: grade.sh <workspace-dir> <slug> [arm]
+# Usage: grade.sh [--baseline] <workspace-dir> <slug> [arm]
+#   --baseline: grade a bare-agent workspace (D-025 in the dev harness). Fail-closed in
+#   BOTH directions: --baseline refuses a workspace that still contains docs/lcd (residue
+#   = silently-wrong baseline), and its absence keeps refusing non-LCD workspaces.
+#   The audit column short-circuits to n/a; every other column grades identically.
 
 set -uo pipefail
 
+baseline=0
+if [[ "${1:-}" == "--baseline" ]]; then baseline=1; shift; fi
+
 if [[ $# -lt 2 ]]; then
-  echo "usage: $0 <workspace-dir> <slug> [arm]" >&2
+  echo "usage: $0 [--baseline] <workspace-dir> <slug> [arm]" >&2
   exit 2
 fi
 
@@ -27,13 +34,20 @@ script_dir="$(cd "$(dirname "$0")" && pwd)"
 plugin_bin="$script_dir/../bin"
 
 [[ -d "$ws" ]] || { echo "error: workspace not found: $ws" >&2; exit 2; }
-[[ -d "$ws/docs/lcd" ]] || { echo "error: $ws is not an LCD workspace" >&2; exit 2; }
+if [[ $baseline -eq 1 ]]; then
+  [[ -d "$ws/docs/lcd" ]] && { echo "error: --baseline but workspace contains docs/lcd (methodology residue): $ws" >&2; exit 2; }
+else
+  [[ -d "$ws/docs/lcd" ]] || { echo "error: $ws is not an LCD workspace" >&2; exit 2; }
+fi
 work="$ws/docs/lcd/work/$slug"
 
 # --- audit (the same cross-path gate the methodology ships) ------------------------
 # A run that stopped before producing spec+plan is a valid (failed) result, not a
-# grader error: record it as INCOMPLETE so the experiment row still lands.
-if [[ -f "$work/spec.md" && -f "$work/plan.md" ]]; then
+# grader error: record it as INCOMPLETE so the experiment row still lands. A baseline
+# workspace has no audit by definition: n/a, not a failure.
+if [[ $baseline -eq 1 ]]; then
+  audit_res="n/a"; nonok="n/a"
+elif [[ -f "$work/spec.md" && -f "$work/plan.md" ]]; then
   table="$(LCD_ROOT="$ws" LCD_SPECS_DIR="docs/lcd/work" bash "$plugin_bin/audit-crosspath.sh" "$slug" 2>/dev/null)"
   if [[ $? -eq 0 ]]; then audit_res="PASS"; else audit_res="BLOCKED"; fi
   nonok="$(printf '%s\n' "$table" \

evals/run-eval.sh

@@ -14,9 +14,19 @@
 # CONTRIBUTING for the evidence-in-PR policy (contributors run locally with their own key).
 #
 # Usage:
-#   evals/run-eval.sh --arm <label> [--plugin-dir <plugin-root>] [--fixture evals/fixture]
-#                     [--results evals/results.md] [--workspace <dir>]
-#                     [--slug stats-surfaces] [--dry-run]
+#   evals/run-eval.sh --arm <label> [--baseline] [--plugin-dir <plugin-root>]
+#                     [--fixture evals/fixture] [--results evals/results.md]
+#                     [--workspace <dir>] [--slug stats-surfaces] [--dry-run]
+#
+# --baseline: bare-agent arm (D-025 in the dev harness). No plugin loads; the workspace
+# copy is stripped of the fixture's methodology onboarding (.claude/, docs/) and the strip
+# is verified fail-closed; the frozen baseline prompt drives the run and the finish line
+# is an EVAL-DONE marker instead of the audit artifact. Contradicts --plugin-dir.
+#
+# Isolation (BOTH arms, symmetric): every model call runs with CLAUDE_CONFIG_DIR pointed
+# at a fresh empty dir (bypasses user-level plugins, settings and CLAUDE.md; auth survives
+# via the OS keychain) and --settings '{"enabledPlugins":[]}'. The arms differ by exactly
+# one flag: the plugin arm adds --plugin-dir.
 # Env:
 #   EVAL_CLAUDE_BIN    model CLI (default: claude)
 #   EVAL_CLAUDE_MODEL  model for the run (default: claude-fable-5 — the experiment's premise
@@ -33,18 +43,21 @@ plugin_root="$(cd "$(dirname "$0")/.." && pwd)"
 
 arm=""
 plugin_dir="$plugin_root"
+plugin_dir_explicit=0
+baseline=0
 fixture="$plugin_root/evals/fixture"
 results="$plugin_root/evals/results.md"
 workspace=""
 slug="stats-surfaces"
 dry_run=0
 
-usage() { echo "usage: $0 --arm <label> [--plugin-dir DIR] [--fixture DIR] [--results FILE] [--workspace DIR] [--slug SLUG] [--dry-run]" >&2; }
+usage() { echo "usage: $0 --arm <label> [--baseline] [--plugin-dir DIR] [--fixture DIR] [--results FILE] [--workspace DIR] [--slug SLUG] [--dry-run]" >&2; }
 
 while [[ $# -gt 0 ]]; do
   case "$1" in
     --arm)        arm="${2:-}"; shift 2 ;;
-    --plugin-dir) plugin_dir="${2:-}"; shift 2 ;;
+    --baseline)   baseline=1; shift ;;
+    --plugin-dir) plugin_dir="${2:-}"; plugin_dir_explicit=1; shift 2 ;;
     --fixture)    fixture="${2:-}"; shift 2 ;;
     --results)    results="${2:-}"; shift 2 ;;
     --workspace)  workspace="${2:-}"; shift 2 ;;
@@ -60,18 +73,26 @@ claude_bin="${EVAL_CLAUDE_BIN:-claude}"
 claude_model="${EVAL_CLAUDE_MODEL:-claude-fable-5}"
 claude_flags="${EVAL_CLAUDE_FLAGS:---output-format json --dangerously-skip-permissions}"
 max_legs="${EVAL_MAX_LEGS:-8}"
-prompt_file="$plugin_root/evals/feature-prompt.md"
+if [[ $baseline -eq 1 ]]; then
+  prompt_file="$plugin_root/evals/feature-prompt-baseline.md"
+else
+  prompt_file="$plugin_root/evals/feature-prompt.md"
+fi
 [[ -n "$workspace" ]] || workspace="$(mktemp -d "${TMPDIR:-/tmp}/lcd-eval-XXXXXX")/$arm"
+config_dir="$(dirname "$workspace")/claude-config-$arm"
+isolation_flags=(--settings '{"enabledPlugins":[]}')
 
 # --- preflight: fail closed, naming the precondition -------------------------------
 refuse() { echo "refused: $1" >&2; exit 2; }
 
+[[ $baseline -eq 1 && $plugin_dir_explicit -eq 1 ]] \
+  && refuse "--baseline contradicts --plugin-dir (the baseline arm runs with no plugin)"
 [[ -d "$fixture" && -f "$fixture/package.json" ]] \
   || refuse "fixture not found or not a project: $fixture"
 [[ -f "$fixture/.claude/rules/lcd-conventions.md" ]] \
   || refuse "fixture is not LCD-onboarded: $fixture"
-[[ -f "$plugin_dir/.claude-plugin/plugin.json" ]] \
-  || refuse "plugin variant not found: $plugin_dir (no .claude-plugin/plugin.json)"
+[[ $baseline -eq 0 && ! -f "$plugin_dir/.claude-plugin/plugin.json" ]] \
+  && refuse "plugin variant not found: $plugin_dir (no .claude-plugin/plugin.json)"
 [[ -f "$prompt_file" ]] || refuse "feature prompt missing: $prompt_file"
 command -v "$claude_bin" >/dev/null 2>&1 \
   || refuse "model CLI not found: $claude_bin (set EVAL_CLAUDE_BIN)"
@@ -81,18 +102,30 @@ if [[ $dry_run -eq 1 ]]; then
   echo "DRY RUN — resolved plan (no model call, nothing materialized):"
   echo "  arm: $arm"
   echo "  fixture: $fixture"
-  echo "  plugin: $plugin_dir"
+  if [[ $baseline -eq 1 ]]; then
+    echo "  plugin: none (baseline)"
+  else
+    echo "  plugin: $plugin_dir"
+  fi
   echo "  slug: $slug"
   echo "  prompt: $prompt_file"
   echo "  workspace: $workspace"
   echo "  results: $results"
+  echo "  isolation: CLAUDE_CONFIG_DIR=$config_dir · --settings '{\"enabledPlugins\":[]}'"
   echo "  model: $claude_bin --model $claude_model $claude_flags (max $max_legs legs)"
   exit 0
 fi
 
 # --- materialize the throwaway workspace -------------------------------------------
 mkdir -p "$(dirname "$workspace")"
 cp -R "$fixture" "$workspace"
+if [[ $baseline -eq 1 ]]; then
+  # Strip the methodology onboarding, then VERIFY the strip (AC-2: a run that cannot
+  # guarantee absence must not start — a "baseline" with residue is silently wrong).
+  rm -rf "$workspace/.claude" "$workspace/docs" 2>/dev/null
+  [[ -e "$workspace/.claude" || -e "$workspace/docs" ]] \
+    && refuse "baseline strip left methodology residue in $workspace"
+fi
 git -C "$workspace" init -q -b main
 git -C "$workspace" add -A
 git -C "$workspace" -c user.email=eval@local -c user.name=eval \
@@ -106,27 +139,40 @@ git -C "$workspace" tag eval-baseline
 # IS_SANDBOX=1: permit --dangerously-skip-permissions under root in containerized
 # environments — acceptable here because the run is confined to the throwaway workspace.
 echo "running arm '$arm' in $workspace …" >&2
-audit_artifact="$workspace/docs/lcd/work/$slug/audit.md"
-nudge="Continue the LCD Deep-lane pipeline for '$slug' from wherever it stands (check docs/lcd/work/$slug/). Do not stop until the audit for $slug has run with all rows OK and the full test suite is green, or you are hard-blocked (record why in the JOURNAL)."
+if [[ $baseline -eq 1 ]]; then
+  finish_artifact="$workspace/EVAL-DONE"
+  finish_label="EVAL-DONE marker"
+  nudge="Continue building the 'stats' feature per the original instructions. Do not stop until the feature is complete with the full test suite green — then create a file named EVAL-DONE at the repository root as your final action. If hard-blocked, record why in BLOCKED.md and stop."
+  plugin_args=()
+else
+  finish_artifact="$workspace/docs/lcd/work/$slug/audit.md"
+  finish_label="audit artifact"
+  nudge="Continue the LCD Deep-lane pipeline for '$slug' from wherever it stands (check docs/lcd/work/$slug/). Do not stop until the audit for $slug has run with all rows OK and the full test suite is green, or you are hard-blocked (record why in the JOURNAL)."
+  plugin_args=(--plugin-dir "$plugin_dir")
+fi
+mkdir -p "$config_dir"
 leg=0
-while [[ ! -f "$audit_artifact" && $leg -lt $max_legs ]]; do
+while [[ ! -f "$finish_artifact" && $leg -lt $max_legs ]]; do
   leg=$((leg + 1))
   if [[ $leg -eq 1 ]]; then
     leg_args=(-p "$(cat "$prompt_file")")
   else
     leg_args=(-p --continue "$nudge")
   fi
   # shellcheck disable=SC2086
-  (cd "$workspace" && IS_SANDBOX=1 "$claude_bin" "${leg_args[@]}" \
-      --model "$claude_model" --plugin-dir "$plugin_dir" $claude_flags \
+  (cd "$workspace" && IS_SANDBOX=1 CLAUDE_CONFIG_DIR="$config_dir" \
+      "$claude_bin" "${leg_args[@]}" \
+      --model "$claude_model" "${plugin_args[@]}" "${isolation_flags[@]}" $claude_flags \
       > "run-$leg.json" 2>> run.stderr)
   run_code=$?
   [[ $run_code -eq 0 ]] || echo "warning: leg $leg exited $run_code (continuing — the row is the verdict)" >&2
 done
-echo "run finished after $leg leg(s); audit artifact $( [[ -f "$audit_artifact" ]] && echo present || echo ABSENT )" >&2
+echo "run finished after $leg leg(s); $finish_label $( [[ -f "$finish_artifact" ]] && echo present || echo ABSENT )" >&2
 
 # --- grade + append -----------------------------------------------------------------
-row="$(bash "$plugin_root/evals/grade.sh" "$workspace" "$slug" "$arm")" \
+grade_args=()
+[[ $baseline -eq 1 ]] && grade_args=(--baseline)
+row="$(bash "$plugin_root/evals/grade.sh" "${grade_args[@]}" "$workspace" "$slug" "$arm")" \
   || refuse "grading failed for $workspace"
 # Stamp the model id on the row (here, not in the golden-locked grader): rows are
 # comparable only within one model id — a silent model change must be visible per row.