Repository navigation
[feature] Public mock fixture suite (or keeperhub-mock Docker image) for adapter testing #53
Description
Friction
Building sbo3l-keeperhub-adapter, every CI run of the live submission test must either:
- Burn a real workflow execution by hitting a real
wfb_<token>-protected webhook. Costs money (KH platform billing per execution), pollutes the workflow's run history with synthetic CI traffic, and requires the operator to provision long-lived test credentials with cross-team permissions. - Mock the entire HTTP surface in-process (
crates/sbo3l-keeperhub-adapter/src/lib.rs#L97-L99—KeeperHubExecutor::local_mock()). Cheap and deterministic, but the mock's response shape is OUR guess at what KH returns. We've reverse-engineered it fromcurl -vagainst one workflow; it diverges from production in ways we'll only discover when adapter authors using our crate hit prod for real.
Both options are unsafe at scale. Adapter authors after us will write their own bespoke mocks (cost: hours per author × N authors), each with subtle drift from production.
Reproduction
# Our adapter's local_mock returns:
cargo run --example submit_signed_receipt -p sbo3l-keeperhub-adapter
# == mock execute result ==
# sponsor=keeperhub mock=true execution_ref=kh-01HTAWX5K3R8YV9NQB7C6P2DGZ
# But we don't know if production returns:
# { "executionId": "..." } ← what our mock assumes
# { "id": "..." } ← what our parser falls back to
# { "execution_id": "..." } ← snake_case variant?
# { "data": { "id": "..." } } ← nested envelope?
# The submit_live_to parser falls back through `executionId` → `id`
# (lib.rs#L317-L319), but a public canonical answer is missing.Proposed fix (pick one or both)
Option A — keeperhub-mock Docker image
Publish ghcr.io/keeperhub/mock:latest — a tiny HTTP server that responds to /api/workflows/<any-id>/webhook with deterministic fixtures matching the production webhook contract. Adapter authors run it as a sidecar in CI:
services:
keeperhub-mock:
image: ghcr.io/keeperhub/mock:latest
ports: ['18080:8080']Then the adapter test points at http://localhost:18080/api/workflows/<id>/webhook instead of prod. Drift between mock and prod is detectable by version-pinning the image.
Option B — public JSON fixture suite
Publish a versioned keeperhub/fixtures repo (or KeeperHub/cli/testdata/) with one JSON file per webhook response shape:
fixtures/
├── webhook_submit_201_minimal.json # canonical 201 success
├── webhook_submit_200_with_metadata.json # 200 with optional fields
├── webhook_submit_400_envelope_malformed.json
├── webhook_submit_422_schema_mismatch.json
├── webhook_submit_429_rate_limited.json
└── webhook_submit_503_transient.json
Adapter authors use these as serde_json::from_str(include_str!(...)) to validate their parser without any network. Pairs naturally with issue #52 (error code catalog) — every fixture corresponds to one row in the catalog.
Why both options together?
- Docker image = integration testing (real HTTP, real timeouts, real connection failures)
- JSON fixtures = unit testing (parser correctness, no network)
Most adapters need both. Other webhook-platform vendors (Stripe's stripe-mock, Auth0's fixtures) ship both for the same reason.
Reference PR draft (consumer side)
B2JK-Industry/SBO3L-ethglobal-openagents-2026 PR — branch agent/dev2/kh-bf-additional-pr-draft-2: shows our adapter's test suite migrated from local_mock() (in-process stub) to KeeperHubMockServer::from_image() (Option A) plus golden-fixture comparisons (Option B). Removes ~80 lines of bespoke mock code.
Context
- Filed as part of SBO3L's KeeperHub Builder Feedback submission for ETHGlobal Open Agents 2026
- Related: [docs] Publish JSON schema for action submission/result envelope #48 (envelope schema), [docs+feature] HTTP error code catalog + adapter retry semantics for workflow webhook #52 (error code catalog) — both improve fixture quality
Activity
docs(kh-cs): consumer-side adapter shape for KH mock fixture suite (K…
test(fixtures): add execution recovery contract pack v1
test: KeeperHub#53 execution recovery pack with loader, idempotency, …
suisuss commentedon Aug 13, 2026
Thanks for the detailed writeup, and for including a reproduction and a reference PR on the consumer side - this is one of the better-scoped issues we get.
Confirming the friction is real: there is no public fixture or mock server anywhere in this repo for the
/api/workflows/<id>/webhooksubmission response.testdata,mock, andfixtureall return nothing onmain, and more to the point, this repo has no code path that calls that endpoint at all -khnever submits to a workflow webhook, so nothing here currently exercises or documents that contract.That gap is also why I'm not accepting this as filed yet.
KeeperHub/clidoesn't implement or call/api/workflows/*/webhook- that surface belongs to whichever service owns workflow triggers, not this CLI. If we publish fixtures for it here, this repo commits to tracking a contract that nothing in its own test suite exercises, so drift between the fixtures and production would go undetected until an adapter author hits it in the wild - which is the exact failure mode this issue exists to prevent.Option A: host it here anyway (
testdata/or aghcr.io/keeperhub/mockimage built and versioned from this repo), and accept that as an open-ended maintenance commitment with no local CI signal if it goes stale.Option B: the team that owns the webhook endpoint publishes and versions the canonical fixtures - in that service's own repo, or the standalone
keeperhub/fixturesrepo you sketched as an alternative - and this issue gets redirected there once that exists.That's a resourcing and ownership call, not something I should settle unilaterally in triage, so I'm taking it to the core team.
One thing worth flagging so it doesn't cause confusion while this is open: PR #97 references this issue and does solid, tested work - idempotency keys, receipt-vs-status classification, cold-start handling - but it fixtures
GET /api/execute/{id}/status, the CLI's own direct-execution polling endpoint. That's a different surface from the webhook submission response ambiguity (executionIdvsidvsexecution_idvs nesteddata.id) that this issue reports. It's worth merging on its own terms, but it does not resolve this issue, and this issue will not move toacceptedon the strength of it.Marking
needs-discussionfor now - I'll follow up here once we've settled ownership.feat: execution recovery safeguards (#97)