[dtice25]

Summary

Lands rollout step 2 of the reliability benchmark design (docs/2026-09-29-benchmark-types.md in benchmarks-platform): the runner side of type: 'reliability'.

reliability config block — defineBenchmarkConfig now accepts type: 'reliability' with a strict reliability block mirroring the platform's config.reliability zod schema:

reliability: {
  interval: '1m',                       // required, /^(\d+)([smh])$/, >= 30s
  graceMs?: number,
  degradedWhen?: { data: Record<string, string | number | boolean> },
  target?: { uptime: number },          // 0 < x < 1
  layout?: 'series',
  sections?: (uptime | slot-grid | metric | breakdown | incidents)[],
  feedCard?: { strip?: '24h', stat?: 'uptime' },
}

Pairing is enforced both ways (block requires the type, type requires the block). Unlike the platform — which skips unknown section types at render time — the runner is the strict check: unknown section types, windows, stats, and outcomes throw at definition time. A metric section must name a key in display.metrics or the built-in 'latency' (RELIABILITY_LATENCY_METRIC).

Upsert — the materialize path sends config.reliability alongside index/display/scoring, so a reliability-typed bench file passes the platform's type-vs-config check on first run.

--scheduled-for <ISO-8601> — new flag stamped into config.trigger.scheduledFor on the run (read by reliability-slots.ts's assignedAt). A scheduled run that reports late fills its real slot instead of whichever slot it landed in by creation time; without the flag the platform falls back to nearest-slot. Exposed on CliArgs/ResolvedRunConfig/runConfigToJson.

Testing

27 new bench-config.test.ts cases cover the full block: type/block pairing, interval parsing + 30s minimum, target.uptime range, unknown keys, each section type's options, unknown section types, and the metric-vs-display.metrics rule including the latency exemption. runner.test.ts adds --scheduled-for parse/reject cases, trigger JSON stamping, and a config.reliability upsert assertion. 221 tests pass; root pnpm typecheck clean.

Link to Devin session: https://app.devin.ai/sessions/fdf951a9b9184f11a6d65e3c5056e907
Open in Devin Desktop: https://app.devin.ai/desktop/session/fdf951a9b9184f11a6d65e3c5056e907?variant=devin
Requested by: @dtice25

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[devin-ai-integration]

✅ Devin resolved the finding on 988b973

Fixed by Devin (1)

• Ambiguous timestamps misassign reliability slots

View all findings in Devin Review