[slayerjain]

What has changed?

The T2 flow (KEPLOY_PROVEN=1) had no documentation. The docs had no hits for KEPLOY_PROVEN, verdicts.yaml or "quarantine", and keploy record --help and keploy test --help never mention the switch. A customer could not find the flow, and could not follow it.

This PR adds two pages under Running Keploy, links them from the CLI reference, and points the GitHub CI page at the T2 job:

• running-keploy/proven-flow: Record, judge and replay tests in CI.

  • What the flow is: record; auto-replay's full and confirming rounds; judging; one local run of CI's own command; then CI. It also says when to use the flow. The page names no Keploy version: it tells a reader who already has Keploy to run keploy update (see Gated on).

  • Requirements:

    • Linux amd64 with kernel 5.10+, or arm64 with 6.6+ (arm64 is unproven: see Gated on);
    • Go 1.20+;
    • sudo;
    • a Keploy account, with an API key from Settings → API Keys.

    Unix sockets are unsupported. When you record, a Unix-socket call leaves the whole test set NOT JUDGED: nothing is written for it, and keploy record exits 1.

  • Install and sign-in, and the switch's value. The commands Keploy prints for you to run carry the switch: the NEXT block's commands (except keploy mock), the QUARANTINE? paste line, and each reason's remedy.

  • Record commands with the switch:

    • native (start the app's dependencies first);
    • docker run, with how its -p, --network and --add-host reach Keploy's agent container, and how to reach a database on the Docker host;
    • docker compose.

    Also: the proxy and --low-latency capture modes, with their caveats; a warning that --delay is not a keploy record flag, and what auto-replay waits for instead; and mock.real with each kind of entry.

  • How to read what auto-replay prints:

    • the JUDGED line and every line under it, the new SET FAILED lines included;
    • the NOT JUDGED line, and where its cause is;
    • the CI would be red and CI runs none of the tests this record judged lines;
    • record's exit codes.

  • The NEXT block, as the in-flight NEXT fix prints it (see Gated on). A judging round is not the suite's verdict:

    • keploy record's block is recorded. It wires no CI, and gives CI's own command once, on its run: line.
    • A new section, Run CI's command once, has the reader run KEPLOY_PROVEN=1 keploy test where they recorded. That run's green block carries the caveats line for the quarantined tests, the commit: line and the CI recipe.
    • The field table gains ask and run.

  • Exactly what to commit: the commit: line of that green block. That is keploy.yml, keploy/.gitignore, tests/, mappings.yaml (or mappings.json), verdicts.yaml and config.yaml (or config.json), and never mocks.yaml or secret.yaml. The line names every test set on disk, judged or not, so the page says to delete a set Keploy didn't judge before running CI's command.

  • The CI recipe: a GitHub Actions job with the enterprise installer, KEPLOY_PROVEN=1 and keploy test, run as the runner's own user. It lists every line that run can print.

  • RECORDING LOST capture-loss evidence, including what Keploy really does today with a request whose recording overlaps a loss.

  • Time freezing: Keploy patches the Go binary, nothing to rebuild (links the existing Time Freezing page), and the second UNFROZEN form.

  • The flags and settings the switch refuses.

• running-keploy/verdicts-yaml: a reference for the file.
  • When an entry skips a test.
  • Every field. rounds gains set_failed.
  • What a URL keeps of its path and query.
  • Each class and reason, with what to do. New:
    • keploy/set_failed and keploy/set_capture_incomplete: a set that failed for what no test owns is quarantined whole;
    • app-infra/real_not_reached.
  • Tests with no verdict, and the delete-then-record recovery. CI's command replays an unjudged set left on disk UNLISTED.
  • Merge conflicts.
• running-keploy/cli-commands: a pointer to the T2 flow. It also takes -d, --delay out of keploy record's flags and example: keploy record -c "node src/app.js" -d 10 exits 8 with unknown shorthand flag: 'd' in -d.
• ci-cd/github: a tip that points a KEPLOY_PROVEN=1 user at the T2 CI job.
• server/{linux,macos,windows}/installation: the docker keploy record line drops --delay 10. keploy record defines no --delay, so the command exited 8 with unknown flag: --delay.

Review round 2 (on 53cc6bc), and what this round changed

Round 2 found that 53cc6bc documented t2-next-ci r4's round-4 fix, 13a1d52e9, which was in no pushed branch. The gate still named r4 at 235114959, and on 235114959 the page was wrong in three places:

• (a) a gated run of only some sets (-t) was verified_green, with a commit: line naming a set that never ran;
• (b) a verdicts.yaml, mappings.yaml or config.yaml written after the gated run left status verified_green, even when it quarantined the last test CI runs;
• (c) a record that quarantined every test it judged, beside another set CI runs, gave verified_empty while record exited 0.

This round (docs/t2-proven-flow @ ea77395):

The BLOCKER: the page follows r4's round-4 fix, and that fix is now pushed and proven. The page documents what 13a1d52e9 does, because 235114959's behaviour in (a) and (b) is a false green. r4's owner has 13a1d52e9 and its e2e commit d75fffdde only in a local worktree; origin's train/fix/t2-next-ci-r4 is still 235114959. So the proof PR, keploy/enterprise#3030 (train/test/t2-docs-proof-r6-next-next at 14ab25274), is built on d75fffdde, which puts both commits on origin. Its pipeline, 12241, runs:

• (b) as phase 9's keploy quarantine after the gated run;
• (c) and (a) as phase 12's new step 4.

#3030 supersedes #3026, which is closed. The gate below names r4 by the behaviour the page needs, not by a SHA taken mid-flight.

SHOULD-FIX:

• The commit: line. It names every test set on disk: gatherProvenCI walks Signals.TestSets. Step 6 and Commit the test sets now say so, and say to delete a set Keploy didn't judge before running CI's command. The reason not to git add keploy/ is that it takes whatever keploy/.gitignore doesn't keep out, which can include a set's secret.yaml. OSS replay ignores /*/secret.yaml only once it replays a set whose config names secrets.
• --delay on record. It is gone from server/{linux,macos,windows}/installation.md too.
• Gate SHAs. The gates are now behaviours on branches, with each head and what main already has, as of this round.
• Capture-loss proof. The page's caution box already describes the agent's drop. The product fix and a green phase 12b are now a gate (5), and product issue 3 is marked as blocking release 1.

NITs:

• keploy update moves into the install section, for a machine that already has Keploy.
• The verdicts.yaml example says keploy: <version>, with a line that says so.
• URL shaping says letters joined by - _ . ~ : are kept (hello-world), and which segments of letters are {}, as models.PathShape does.
• After keploy mock upload, the page now says what the printed remedy says: run KEPLOY_PROVEN=1 keploy status again. On r4 that status is recorded (listsChanged: the upload writes config.yaml after the run) and gives CI's command.
• The onboarding quarantine rate is product issue 1, now marked as blocking a release-1 "onboardable" claim.

Also from 13a1d52e9:

• The field table's phase row lists recorded for a green -t run and for a run older than a list.
• It says a run with the switch that stopped short is red.
• The NEXT block section says that a record that quarantined everything it judged, beside a set CI runs, is recorded and keeps its exit code.

How every claim was checked

• Code. I checked every message, field, reason and remedy against:

  • enterprise main at a4b49aaf8 (latest tag v3.8.79);
  • keploy v3.6.154, which E-C7a, E-C12 and r4 pin;
  • the gating branches below, at the heads named. This round, I re-read r4's d75fffdde for each claim the review named:
    • ComputePhase: partialRun and listsChanged are recorded, and stoppedShort is red;
    • listsWrittenAfter: verdicts.yaml, mappings.yaml/.json and config.yaml/.json, by mtime against the run's directory;
    • judgingPhase: verified_empty only on NoTestRunsInCI;
    • gatherProvenCI: every set in Signals.TestSets;
    • uploadFirst's printed remedy;
    • models.PathShape;
    • OSS routes/record.go massSuppressed (WARN at 25%);
    • the OSS flag table (--delay under test only, containerName an alias of container-name).

• A CI run of the combination: keploy/enterprise#3030, test-only, at 14ab25274, pipeline 12241. It is d75fffdde (r4 with 13a1d52e9), plus:

  • E-C7a's two commits;
  • the freeze check at 0478b97b7;
  • the T2 lane;
  • round 6's test-only commits.

  Results:

  • Both T2 lanes passed all seven cycles: t2-flow-e2e (6.12.74, woodpecker-main) and t2-flow-e2e-debian11 (5.10.0-46). The cycles are native, docker run and compose, each in proxy and --low-latency mode, plus native-runner-user.
  • Each cycle walks the page's flow:
    1. record;
    2. judging;
    3. record's NEXT is recorded, with CI's command on run:;
    4. the gate, which passes exactly the K kept tests and is verified_green, with the caveat, commit: and the CI recipe;
    5. a commit of exactly the commit: line, and a fresh clone;
    6. CI's keploy test with the database down: K of K.
  • Kept per cycle (35 tests each):
    • 5.10: 16, 12, 18, 16, 16, 17 and 25;
    • 6.12: 13, 13, 16, 14, 13, 13 and 10.
  • The negative held: CI's command on the checkout with the app broken failed 13 of 16 tests (5.10) and 10 of 13 (6.12).
  • RECORDING LOST on the documented path. Both lanes' native-low-latency cycle lost reads ("capture could not read N piece(s)") on several tests, and every agent: recording complete line had tcs_suppressed_total: 0. On 5.10, 4 such tests replayed and were kept, and 5 were quarantined with the capture evidence on their entry. On 6.12 it was 4 and 6. The gate and CI then passed exactly the kept tests. So the page's per-test mechanism has a green run. The agent's drop described in the caution box is a separate path, phase 12b's (gate 5).
  • In auto-replay-e2e:
    • Phase 9 prints the page's examples byte for byte, except the app command: the JUDGED and QUARANTINED lines, record's recorded block, and the gated run's QUARANTINED, not run, SET and green blocks. Then d75fffdde's new check holds. After the gated run, keploy quarantine puts the last test CI runs on the list, and status is recorded with CI's command and no commit: line, and names keploy/test-set-0/verdicts.yaml changed after the latest run. This is review item (b).
    • Phase 12, step 4:
      • A set that fails on its own, recorded beside a set CI runs, exits 0, and record's NEXT is recorded with CI's command. CI's command then passes. This is item (c).
      • A gated -t test-set-0 run is recorded and wires nothing. This is item (a).
    • Phase 12a: a Unix-socket record is NOT JUDGED and exits 1. CI's command with the unjudged set on disk exits 1. After rm -rf, CI's command is green, with only the judged set on its commit: line.
    • Phase 12b fails at the record wrote no get-unread-1, as on every proof pipeline since 12143: the agent sent 3 tests and suppressed 2 (product issue 3, gate 5). It is the only failing step of 12241.

• Unit tests of #3030's tree on .189 (proven, cliflow, composeapp, the pkg/service/test freeze tests, tests/ciscript T2/WallClock/KeployTestGate, wait-utils-test.sh, auto-replay-assert-test.sh, and cmd/enterprise/cli's record, judging, gate, quarantine and proven tests) pass. The one exception is TestProvenReplayWritesItsArtifacts, which compares a temp path the box resolves through a symlink. It passes on the laptop.

Gated on

Some of what these pages document is enterprise code that hasn't merged or been released yet. Do not merge this PR before a Keploy Enterprise release that contains all of the following is published at keploy.io/ent/install.sh. The page names no version. It tells a reader who already has Keploy to run keploy update, so it describes the latest release.

Each gate is a behaviour on a branch. The SHAs are each branch's head as of this round (enterprise main at a4b49aaf8, 2026-10-22), so check each gate against the head that merges, not against these SHAs.

1. The NEXT/CI-recipe fix, train/fix/t2-next-ci-r4 (or train/fix/t2-next-ci, if it lands under that name), with its round-4 fix. The page needs:

   • record's NEXT is recorded and gives CI's command once, on run:; only the gated keploy test is green and wires CI (a8c9b3120);
   • the gate's and judging's texts carry the switch (4802e41d1);
   • 13a1d52e9:
     • a gated run that replayed only some sets is recorded, never green (partialRun);
     • a set's verdicts.yaml, mappings or config written after the gated run makes it recorded (listsChanged);
     • a run under the switch that stopped short is red;
     • a record that quarantined everything it judged, beside a set CI runs, is recorded.

   Origin's r4 is still 235114959, which has none of 13a1d52e9. The owner's local head, d75fffdde, has it, and #3030 put d75fffdde on origin and proved it. If the owner pushes a different head, re-check against it:

   • the record and gated-run blocks (phase 9);
   • the phase row;
   • Keep the suite healthy;
   • the keploy mock upload paragraph.

2. E-C12, keploy/enterprise#3023 (train/batch-ec12 at 5a9088968).

   • main already has its two record-exit commits, as d9de81c77 and 9de0edb0f (git cherry shows E-C12's copies as applied), so #3023 needs a rebase.
   • The page still needs the rest of #3023, which main lacks:
     • the record-loss judging (77cd1234d, 177758edf);
     • the set-failure judging (8f9a44dde through f3158bae3): keploy/set_failed, keploy/set_capture_incomplete and app-infra/real_not_reached, the SET FAILED lines, the set_failed round outcome, and exit 1 when CI would be red or run no test.

3. E-C7a, keploy/enterprise#2972 (train/batch-ec7 at b99296a87), for the two commits E-C12 lacks:

   • 6ad9e03f8: diff keys shaped as {};
   • 78a43c4a9: a --low-latency capture's losses are judged under the switch.

   Whichever of E-C7a and E-C12 lands second must drop or reconcile its copies of the record-loss commits.

4. train/fix/freeze-check-compose-service (head 0478b97b7; 46a0ef2f6 was gated in round 6). The page tells a compose app to pass its service's name (--container-name api). Without this fix, CI's keploy test of such an app logs ERROR Time freezing could not be verified ... on a green run.

5. The capture-loss path the page describes needs a green e2e. auto-replay-e2e phase 12b is E-C7a's own lane for 78a43c4a9. It has failed on every proof pipeline since 12143, 12234 included: the agent drops the request whose capture lost data instead of recording it as evidence (product issue 3). The page's caution box documents today's behaviour. If the agent is fixed before this PR merges, rewrite the box and the bullet list above it to match the fix.

6. arm64 is unproven. The page says the flow runs on arm64 with kernel 6.6 or later, as release 1's scope does. But this Woodpecker has no arm64 runner, and every proof here is amd64 (5.10 and 6.12). Before merging, get the T2 lane green on an arm64 6.6 host. Otherwise, take arm64 out of the page.

The T2 lane itself, train/fix/t2-default-path-lane (CI-AR-01, head 7c58f2ae1), still reads record's NEXT for the commit: line. On r4 it needs the change #3030 makes in a test-only commit, a gate step that runs CI's command first. That is for its owner to restack.

Product issues found while checking (not fixed here)

1. Blocks calling release 1 onboardable: Keploy quarantines most of the sample's suite on an unchanged app.
   • In pipeline 12241, samples-go's Postgres app had 35 tests per cycle. Keploy kept 10 to 25 of them, and quarantined the rest as keploy/unmocked_call (often no_schema_candidates) or keploy/unconsumed_mock: 25 of 35 in the 6.12 docker-low-latency cycle.
   • In the --low-latency cycles, many of those come with "capture could not read N piece(s)" on Postgres INSERTs.
   • This is not a false green: CI runs only the kept tests. But it is Keploy failing to mock or match its own recording, and recording again only draws another subset. A first-run customer who follows this page loses most of their suite. The page tells the reader to report such tests.
2. One test with no verdict sinks its whole test set at record time. The set's other tests replayed, but they are neither kept nor quarantined. The NOT JUDGED line names the cause, but not the remedy: that is only in a WARN and an ERROR log line's next_step.
3. Blocks release 1, under the no-silent-drop rule: the agent drops a recorded request whose capture had a hole, and charges the loss to another test.
   • It happens when a parser stops mid-stream and opens an orphan window. In phase 12b (12234 and 12241), the agent sent 3 tests and suppressed 2, get-unread-1 among them. The loss was charged to get-health-1, which replayed and was kept: RECORDING LOST test-set-0/get-health-1 ... it replays, so it is kept.
   • tcs_suppressed_total is logged at INFO below 25% of the session's requests (OSS routes/record.go, massSuppressed), and nothing in keploy record's output names the dropped requests.
   • The fix belongs in the OSS agent's orphan suppression, under the switch: record the request with its loss as evidence, and let auto-replay judge it.
   • Phase 12b is E-C7a's own lane for this, red since 12143. It needs an owner. The page documents today's behaviour in a caution box (gate 5).
4. Some printed commands still lack the switch:
   • the next_step of record's incomplete-verdict log lines (record_verdict.go);
   • the --must-pass refusal's keploy quarantine <set>/<test> --reason.
5. Under the switch, keploy test's delay messages are wrong. It still logs Delay is set to 5 seconds ... use --delay, and the --health-url help says it preserves the fixed --delay behavior. In fact it waits for readiness, then waits --delay.
6. The keploy/capture_incomplete remedy is stale. It still names "a unix socket", though a Unix-socket call now leaves tests NOT JUDGED.
7. Docs, outside this PR: cli-commands.md's keploy rerecord section documents a command that no longer exists. Neither enterprise nor OSS v3.6.154 registers rerecord.

Type of change

• Documentation update (if none of the other choices apply).

How Has This Been Tested?

• npx prettier@2.8.8 --check passes on every changed file: both pages, the CLI reference, the GitHub CI page, the three server installation pages and the sidebar.
• All seven changed pages compile as MDX (@mdx-js/mdx 3 with GFM and directives, after Docusaurus's heading-ID and comment handling). A control file with a bare <tests> fails that check, so it does catch errors.
• Every internal link and anchor the PR adds resolves (a script over both pages' headings), including the new #gate.
• npm run build runs in this PR's Deploy PR Build and Check job. Every check (deploy-preview, prettier, run-lint, asset-budget, DCO) is green on ea77395.

Checklist:

• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.