Complete benchmark framework core and camera pilot

[yuecideng]

Description

Complete the public benchmark framework proposed in #679 and keep the camera pilot as its first end-to-end consumer. The framework now provides a shared experiment definition, deterministic matrix expansion, budgeted execution, stage/continuous/attempt measurements, standard artifacts, provenance, offline aggregation, paired comparison checks, legacy conversion, and dependency-free report tables/figures.

Refs #679. The public framework is intentionally local and file based for v1; domain execution and validation remain in their existing owners.

Public framework

• scripts/benchmark/core/contracts.py defines ExperimentDefinition, Budget, MetricDefinition, RunRecord, RawObservation, and AggregateMetric. Execution, quality, task, and data states remain separate.
• scripts/benchmark/core/planning.py expands stable Cartesian parameter matrices and enforces run, attempt, and wall-time budgets. Retries consume the same attempt ledger.
• scripts/benchmark/core/measurement.py provides separate APIs for staged operations, continuous operation windows, bounded attempts/episodes, and resource samples.
• scripts/benchmark/core/execution.py retains deterministic isolated subprocess runs, timeout/interruption cleanup, not-run rows, and common raw/quality/artifact-index records.
• scripts/benchmark/core/artifacts.py creates the standard definition.json, effective_config.yaml, assets_manifest.json, raw.jsonl, metrics.json, quality.json, and artifact_index.json files. JSONL records and evidence registration are idempotent and reject conflicting duplicates.
• scripts/benchmark/reporting/ preserves units, populations, denominators, missing reasons, uncertainty, and source run IDs; supports paired eligibility/ratios, legacy row conversion, Markdown tables, CSV plotting data, deterministic SVG charts, and offline report rebuilding.

Camera pilot integration

The camera pilot now freezes an ExperimentDefinition and budget, uses the common execution/artifact protocol, and writes raw and quality evidence for both EmbodiChain/DexSim and Isaac Lab. Its simulator-specific adapters remain isolated under scripts/benchmark/rendering/.

Dependencies: no new package dependencies. Simulator workers still use the existing DexSim-capable environment and installed Isaac Lab revision documented in scripts/benchmark/rendering/README.md.

Type of change

• New feature (non-breaking change which adds functionality)

Validation

• Black 26.3.1: black . and final black --check . passed.

• 104 focused tests passed:

  python -m pytest -q -c /dev/null --noconftest -p no:cacheprovider \
    tests/benchmark/core tests/benchmark/reporting \
    tests/benchmark/rendering/test_camera_pilot.py tests/test_main.py \
    tests/test_agent_context_map.py tests/test_agent_context_tools.py

• Public API documentation gate: 2,257/2,257 exports documented.

• Context-map validation, affected-context review, benchmark skill validation, git diff --check, and standard-library-only import smoke tests passed.

• Live validation on RTX 5090: one fresh EmbodiChain process and one fresh Isaac Lab process, each completing 30 warm-up and 300 measured captures. Freshness and nonempty-image checks passed.

• The live run produced the full standard artifact set, including two raw.jsonl records, two quality records, four indexed worker artifacts, aggregate metrics.json, and an offline report rebuilt with python -S.

• Image quality has not been independently calibrated, so the pilot remains not_qualified and does not emit an equal-quality speedup claim.

Screenshots

The live run stores sample images and camera-movement probe images alongside raw results. Generated outputs remain outside version control.

Checklist

• I have run the black . command to format the code base.
• I reviewed affected documentation and agent context and updated the framework overview, routing, and benchmark guidance.
• Public API documentation coverage passes (python docs/scripts/check_api_docs.py); no packaged public API documentation changes are needed.
• I have added tests that prove the feature works.
• Dependencies have been reviewed; no dependency-file changes are needed.

G-03 minimal vertical slice

Add a simulator-free expert-generation fixture under scripts/benchmark/expert_generation/:

• GenerationCase, GenerationAttempt, and PersistenceReceipt keep source lineage, execution, measured validation, task outcome, and persistence state independent.
• The bounded runner reuses the public budget ledger, JSONL artifact store, common attempt aggregation, and offline reporting helpers.
• Accepted yield requires completed execution, passed validation, passed task outcome, and a confirmed receipt. Repeated commit_id values are counted as duplicates; budget-exhausted attempts remain not_run.
• python -m scripts.benchmark expert-generation --fixture produces standard artifacts and an attempt/stage/confirmed-yield report without a simulator.

The executor callback is the integration point for #670's Candidate Coordinator, GenerationSession, Physical Executor, Measured Validator, and EpisodeSink. This fixture does not introduce a competing production lifecycle.

#670 GenerationSession bridge

Add scripts/benchmark/expert_generation/session_adapter.py, which translates a production host's proposal, measured ExpertEpisode, and CommitReceipt through the existing GenerationSession lifecycle:

propose → mark_rollout_started → accept_episode → EpisodeSink receipt → apply_receipt

The adapter is covered with CPU tests for confirmed receipts and measured-validation rejection. It keeps simulator and physical-host construction outside the benchmark runner; the remaining integration work is to provide the real Candidate Coordinator / Physical Executor / Measured Validator host implementation.