From 46078a3b8c1d0d1a073ca006fc1e5ccd280cac97 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 13:01:05 +0530 Subject: [PATCH 01/14] docs: document the KEPLOY_PROVEN record, judge and CI flow The docs had no page for the T2 flow: nothing named KEPLOY_PROVEN, verdicts.yaml or quarantine, and keploy record/test --help never mention the switch, so a customer could neither find the flow nor follow it. Add two pages under Running Keploy: - proven-flow: what the flow is and when to use it; requirements (Linux amd64 5.10+, arm64 6.6+, Go 1.20+, TCP dependencies only: Unix sockets are unsupported); install and sign-in; the record commands with the switch for native, docker run and docker compose apps, in proxy and --low-latency modes; why --delay is not a keploy record flag; mock.real; how to read the auto-replay's log lines, the JUDGED line and the lines under it, record's exit codes and the NEXT block; exactly what to commit (never mocks.yaml); the GitHub Actions job with the enterprise installer, KEPLOY_PROVEN=1 and keploy test, and what that run prints; RECORDING LOST capture-loss evidence; time freezing, which patches a Go binary with nothing to rebuild; and the flags and settings the switch refuses. - verdicts-yaml: the file's format, when an entry skips a test, every field, each class and reason with what to do, and merge conflicts. Every message, field, reason and remedy is taken from enterprise main and keploy main. The NEXT block's commit: line and CI recipe, the registry upload of a set with quarantined tests, and keploy test's warning without the switch follow enterprise train/fix/t2-next-ci; the RECORDING LOST lines, record's incomplete-verdict exit and the shaping of diff keys follow the E-C7a vehicle (enterprise #2972). Signed-off-by: slayerjain --- .../running-keploy/proven-flow.md | 390 ++++++++++++++++++ .../running-keploy/verdicts-yaml.md | 118 ++++++ .../version-4.0.0-sidebars.json | 2 + 3 files changed, 510 insertions(+) create mode 100644 versioned_docs/version-4.0.0/running-keploy/proven-flow.md create mode 100644 versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md new file mode 100644 index 000000000..7135b38c3 --- /dev/null +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -0,0 +1,390 @@ +--- +id: proven-flow +title: Record, judge and replay tests in CI +sidebar_label: Judged record and replay (KEPLOY_PROVEN) +description: Record API tests with KEPLOY_PROVEN=1, let auto-replay judge which of them replay, commit verdicts.yaml, and run only the kept tests in CI with keploy test. +tags: + - record + - replay + - auto-replay + - ci + - verdicts +keywords: + - KEPLOY_PROVEN + - verdicts.yaml + - quarantined tests + - keploy auto-replay + - keploy record judging + - keploy test in CI + - mappings.yaml + - RECORDING LOST +--- + +With `KEPLOY_PROVEN=1`, `keploy record` does more than save tests. When you stop recording, its auto-replay replays every test it just recorded against the recorded mocks, learns what legitimately changes between runs, and judges each test. A test that replays is kept. A test that doesn't is quarantined in `keploy//verdicts.yaml`, with the reason. Your CI then runs `keploy test` with the same switch: it skips the quarantined tests and replays the rest, so a green run means every test it ran replayed against its own recording. + +:::note What this release covers +The flow is for Go applications on Linux: amd64 with kernel 5.10 or later, and arm64 with kernel 6.6 or later. Calls over Unix sockets aren't supported. See [Requirements](#requirements). +::: + +## When to use it + +Use this flow when you want recorded API tests to run in CI with no dependencies running, and you want to trust the result: + +- A kept test passed two replays in which every call it made was answered from the recording, and every call recorded for it was made. +- A test that can't replay yet, because of a dependency Keploy can't mock, a call it couldn't capture, or an app that isn't deterministic, doesn't block you. Auto-replay filters it out and says why, and the rest of the suite runs. + +Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described in the [CLI reference](/docs/running-keploy/cli-commands/), and `keploy test` ignores `verdicts.yaml`. + +## How the flow works + +1. **Record.** `keploy record` captures each request to your app as a test case, and each call your app makes to its dependencies as a mock. +2. **Replay every test (the full round).** When you stop recording, auto-replay replays each new test with its mocks. Your app's clock is frozen 61.5 seconds after the moment the test was recorded, so values that come from the clock change. Keploy learns what changed: a response field that differs becomes noise on the test case, and a value your app generated and sent to a dependency becomes a variable that the mock follows. +3. **Replay as CI will (the confirming round).** Keploy runs `keploy test` the way your CI will run it, skipping the tests it's about to quarantine. +4. **Judge.** A test that passed both rounds is kept. A test that didn't is written to `keploy//verdicts.yaml` with a class and a reason. Keploy prints a `JUDGED` line for each test set, then a `NEXT` block that says what to commit and how to wire CI. +5. **Run in CI.** You commit the `keploy` folder without its `mocks.yaml` files. Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. + +## Requirements {#requirements} + +- **Linux.** amd64 with kernel 5.10 or later, or arm64 with kernel 6.6 or later. Keploy behaves the same on both. macOS and Windows aren't part of this release. +- **A Go application,** built with Go 1.20 or later. Other languages aren't part of this release. +- **sudo rights.** Keploy loads eBPF programs, so it runs its agent with `sudo` and may ask for your password. On a CI runner, the runner's passwordless `sudo` is enough. +- **Docker**, if your app runs with `docker run` or `docker compose`. +- **A Keploy account.** `keploy record` puts a judged test set's mocks in the Keploy registry, and `keploy test` downloads them in CI. Sign in once with `keploy login`. In CI, Keploy signs in with an API key: in [app.keploy.io](https://app.keploy.io), open your user profile, go to **Account Settings → API Keys**, generate a token, and set it as the `KEPLOY_API_KEY` environment variable. +- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. A test that calls a dependency over a Unix socket gets no verdict: Keploy reports it as `NOT JUDGED`, with the remedy to reach that dependency over a TCP address (for example `localhost:5432` instead of a socket path), and a CI run that reaches it fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. + +## Install Keploy and sign in + +```bash +curl --silent -O -L https://keploy.io/ent/install.sh && source install.sh +keploy login +``` + +`https://keploy.io/install.sh` hands over to the same installer, so either one installs the same Keploy. + +Then turn the flow on for your shell: + +```bash +export KEPLOY_PROVEN=1 +``` + +Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. + +## Record your tests + +Pick the command that matches how you run your app. + +**Native.** Build your Go binary and start it with `-c`: + +```bash +go build -o server . +KEPLOY_PROVEN=1 keploy record -c "./server" +``` + +Start the Go binary itself in `-c`, not a shell, `make` or a script that starts it. Keploy freezes the clock of the Go binary that `-c` starts, as described in [Freeze time](#freeze-time). + +**Docker.** Use the `docker run` command you already use, and pass the container's name: + +```bash +docker build -t my-app:latest . +KEPLOY_PROVEN=1 keploy record -c "docker run -p 8080:8080 --name my-app my-app:latest" --container-name my-app +``` + +**Docker Compose.** Pass the app's service name, or its `container_name`, from your compose file. A compose app needs `--container-name`: + +```bash +KEPLOY_PROVEN=1 keploy record -c "docker compose up" --container-name api +``` + +When Keploy replays a compose app, in auto-replay and in CI, only the app and Keploy's agent run, with the services they can't start without. The app's dependency services stay down, and Keploy answers the app's calls to them from the mocks. A service you name in the command (`docker compose up api redis`) runs too, which is how a [`mock.real`](#mock-real) dependency that is a compose service stays up. + +Then send traffic to your app: `curl`, your API test suite, or a script that walks through your endpoints. **Send one request at a time.** When a request starts while another one is still open, Keploy warns `some tests were recorded while another test's request was still open`: it maps each dependency call made while both were open to the test whose request came first, which may not be the test that made it. Such tests can fail auto-replay and be quarantined. + +To stop recording, press **Ctrl+C once**, then wait. Keploy says: + +```text +auto-replay: recording stopped — now replaying what was just recorded to check for noise. Press Ctrl+C again to skip and exit. +``` + +Auto-replay runs both rounds and prints its verdict. Don't press Ctrl+C a second time: that skips auto-replay, so nothing is judged, no `verdicts.yaml` is written, and `keploy record` exits non-zero. To record without a verdict on purpose, pass `--disableAutoReplay`, or set `record.disableAutoReplay: true` in `keploy.yml`. + +:::caution Don't pass `--delay` to keploy record +`--delay` is a `keploy test` flag. `keploy record` rejects it as an unknown flag and records nothing. Under the switch, auto-replay and `keploy test` wait for your app to be ready before they send the first test: it listens on the port the test connects to, or the health URL you gave answers (`--auto-replay-health-url` on `keploy record`, `--health-url` on `keploy test`). +::: + +### Choose a capture mode {#capture-mode} + +Keploy records in one of two modes. Auto-replay and `keploy test` replay the same way in both: Keploy's proxy serves the mocks. + +| | Proxy (default) | Low latency (`--low-latency`) | +| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| How Keploy records | Your app's calls to its dependencies go through Keploy's proxy. | Keploy reads your app's traffic in the kernel with eBPF. Nothing sits between your app and its dependencies. | +| HTTPS dependencies while recording | Your app must trust Keploy's CA. Keploy sets this up for common runtimes; see [TLS Docker Support](/docs/running-keploy/docker-tls/). | No CA is needed while recording. | +| HTTPS dependencies while replaying | Your app must trust Keploy's CA. | Your app must trust Keploy's CA: replay always goes through the proxy. | +| Kernels | amd64 5.10 or later, arm64 6.6 or later. | The same: low-latency capture loads on arm64 from 6.4, but this flow is supported from 6.6. | + +To record in low-latency mode, add the flag to the record command: + +```bash +KEPLOY_PROVEN=1 keploy record -c "./server" --low-latency +``` + +`--low-latency` is a `keploy record` flag only. With it, Keploy runs itself with `sudo` from the start, even for a native app. + +A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in is `NOT JUDGED`, with the remedy to make that client trust Keploy's CA. + +### Reach a dependency for real {#mock-real} + +Under the switch, Keploy records and mocks every call your app makes. To keep a dependency live in recording, auto-replay and CI, list it under `mock.real` in `keploy.yml`, as a host and port, an address, or a container: + +```yaml +mock: + real: + - auth.example.com:443 + - localhost:6379 + - container:redis +``` + +A test that uses a `mock.real` dependency runs on the real clock, and CI must start that dependency: a run in which a listed destination is never reached fails. For a compose app, name a `mock.real` service in the compose command you record and test with, such as `docker compose up api redis`, or Keploy's replay leaves it down. The `--pass-through-ports`, `--global-passthrough` and `--opportunistic-tls-intercept` flags, and the `passThroughPorts` and `bypassRules` settings, don't apply under the switch; `mock.real` replaces them. + +## Read what auto-replay prints {#read-the-output} + +While it replays, auto-replay logs what it learns. A response field that changed is marked as noise on the test case, and a value your app generated and echoed to a dependency is learned as an `echo`, which the mock answers with this run's value: + +```text +INFO auto-noise: marked diff fields as noise on testcase {"test_set_id": "test-set-0", "test_case": "get-date-1", "fields": ["body.served_at"]} +INFO learned a field that changed between record and replay {"Type": "http", "mock": "mock-0", "field": "body.key", "as": "echo"} +``` + +Keploy learns noise only for a test whose dependency calls were all answered. A test that made a call with no recorded answer keeps failing, and is quarantined with that call as its evidence: + +```text +WARN auto-noise: keploy could not mock a dependency call this testcase made; its differences were not learned as noise and it stays failed +``` + +After the confirming round, Keploy prints its verdict for each test set, then the `NEXT` block: + +```text +JUDGED test-set-0: 5 replayable, 2 quarantined (1 flaky/request_changed, 1 flaky/unstable_order); wrote keploy/test-set-0/verdicts.yaml: commit it + QUARANTINED test-set-0/post-learn-batch-1 flaky/unstable_order: HTTP POST /batch to localhost:8096 (no_match) sent an array in another order + QUARANTINED test-set-0/post-learn-order-1 flaky/request_changed: HTTP POST /orders to localhost:8096 (no_match) changed +``` + +### The JUDGED line + +`JUDGED : replayable, quarantined ( /, ...); ` + +- **Replayable** tests passed both rounds. CI runs them. +- **Quarantined** tests are listed in `verdicts.yaml`, so CI skips them. The counts in brackets group them by [class and reason](/docs/running-keploy/verdicts-yaml/#reasons). +- A test that Keploy couldn't judge is in neither count. The confirming round's output names it on a `NOT JUDGED` line, with the reason. + +The line ends with one of: + +| Ending | Meaning | +| -------------------------------------------------- | ------------------------------------------------- | +| `wrote keploy//verdicts.yaml: commit it` | Keploy added or changed entries. Commit the file. | +| `keploy//verdicts.yaml unchanged` | The file already says what Keploy decided. | +| `it needs no verdicts.yaml` | Every test replays, and the set has no file. | + +Under the `JUDGED` line, Keploy can print: + +| Line | Meaning | +| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `QUARANTINED / /: ` | The test is now in `verdicts.yaml`. The evidence names the calls, mocks or response fields without their bodies; the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/#fields) says what a URL keeps. See [what to do for each reason](/docs/running-keploy/verdicts-yaml/#reasons). | +| `RUNS AGAIN /: ...` | The test replays now, so its old entry no longer applies, and CI runs it again. | +| `PRUNED /: its recording is no longer on disk` | Keploy dropped an entry whose test's recording is gone. | +| `RECORDING LOST /: ...` | The recording of this test lost data. See [Capture-loss evidence](#recording-lost). | +| `NOT CHARGED TO A TEST (in CI's shape): ...` | A dependency call made outside every test that Keploy couldn't answer or capture, such as a background poller. CI fails on it. | +| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI would run no test of this set, and a run in which no test ran fails. Fix the reasons and record again. | + +Other lines a test set can get: + +- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, for example because a round stopped early or the app never answered a test. Fix what the reason names and record again. A set with no test case gets `NOT JUDGED : it has no test case to judge`. +- `NOT WRITTEN : keploy//verdicts.yaml changed while keploy record judged, so it stays as it was`: something else changed the file meanwhile, such as a `git pull` or another Keploy run, so nothing Keploy decided for the set was written. The same line ends `could not be written, so it stays as it was` when the write failed, for example on file permissions. Run `KEPLOY_PROVEN=1 keploy test --judge`: it replays the set and prints a `keploy quarantine` command for each test that fails, which you run if you agree (it writes a `manual` entry). Or delete the set and record it again. +- `QUARANTINE? / ...`, followed by a `keploy quarantine` command: a test failed the way CI runs it, and no entry skips it. Keploy never adds that entry by itself. Run the printed command, with the switch set, if you agree, or fix the test. +- `keploy record's judging: CI would be red as these lists stand`: what's printed above it would fail CI. Fix it before you wire CI. + +### Exit codes + +`keploy record` exits 0 when judging finished and CI would be green as the lists stand, even if it quarantined tests. When it prints `keploy record's judging: CI would be red as these lists stand`, go by that line, not by the exit code: fix what it names before you wire CI. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code. When Keploy recorded tests but couldn't finish judging them, because auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, or a second Ctrl+C stopped it, `keploy record` prints `keploy record's verdict is incomplete: ` and exits non-zero, so a script can't take an unjudged recording for a judged one. + +### The NEXT block + +Every `keploy record` and `keploy test` ends with a `NEXT` block, and `KEPLOY_PROVEN=1 keploy status` prints it any time (add `--json` for a machine-readable copy). After a green record it looks like this: + +```text +=== KEPLOY NEXT === +phase: verified_green +summary: All 5 test(s) passed across 1 test-set(s). Suite is green and dependency-free. +caveats: + - 2 quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only. `KEPLOY_PROVEN=1 keploy test --all` runs them; `KEPLOY_PROVEN=1 keploy test --judge` drops the entries of those that now pass. +ask: Want me to (a) cover another resource, (b) wire CI so this runs on every PR, or (c) stop here? +do: If wiring CI: the Keploy registry holds each test set's mocks; CI downloads them by the hash in the set's config.yaml (KEPLOY_API_KEY required in CI). Commit exactly the paths on the commit: line (relative to the directory that holds keploy/: run `git add` there, and CI's keploy test there too): ... +commit: keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml +options: + - cover another resource → KEPLOY_PROVEN=1 keploy record -c './server' --metadata 'name=' + - wire CI: add a job that installs keploy and runs `KEPLOY_PROVEN=1 keploy test` on every PR (see do:) + - stop +protocol: ... +=================== +``` + +| Field | What it tells you | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `phase` | Where the suite is: `recorded` (not replayed since it changed), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | +| `summary` | The last run in one sentence. | +| `failures` | Each failing test, with what differed and a remedy. | +| `caveats` | What a green doesn't cover, such as the quarantined tests it didn't run. Read them before you call the suite green. | +| `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | +| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | +| `options` | Commands for the next step, each with the switch where the flow needs it. | + +For a Docker or Compose app, add the `--container-name` you recorded with to the `keploy record` and `keploy test` commands the block prints. + +When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. Upload them with the `keploy mock upload -t ` command it prints, then run `KEPLOY_PROVEN=1 keploy status` again. + +## Commit the test sets {#commit} + +From the directory that holds `keploy/`, `git add` exactly the paths on the `NEXT` block's `commit:` line: + +```bash +git add keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml +``` + +| Path | What it is | Commit it? | +| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| `keploy.yml` | Keploy's settings for your app, such as its command and `mock.real`. | Yes | +| `keploy/.gitignore` | Keeps `mocks.yaml`, `reports/` and Keploy's other local files out of the repository. | Yes | +| `keploy//tests/`, and `assets/` when present | The test cases, with the noise auto-replay learned. | Yes | +| `keploy//mappings.yaml` | Which recorded calls each test owns. Under the switch, only `keploy record` writes it. | Yes | +| `keploy//verdicts.yaml` | The tests CI skips, and why. See the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/). | Yes | +| `keploy//config.yaml` | Names the upload of the set's mocks in the Keploy registry. | Yes | +| `keploy//seed.sh`, when present | Your seed script for the set. | Yes | +| `keploy//mocks.yaml` | The recorded dependency calls. | **Never.** It's in the Keploy registry, and `keploy/.gitignore` keeps it out. Don't remove that line. | +| `keploy//mocks.json` or `mocks.gob` | The recorded dependency calls, in the JSON or gob storage format. The registry takes only `mocks.yaml`. | Yes: CI reads it from the checkout, and the `commit:` line lists it. | +| `keploy//secret.yaml` | Secrets that `keploy sanitize` took out of the recording. | Never | +| `keploy/reports/` | Run reports. | No | + +## Run the kept tests in CI {#ci} + +The CI job checks out the repository, builds your app, installs Keploy, and runs `keploy test` with the switch on, in the directory that holds `keploy/`. Store your Keploy API key as a CI secret named `KEPLOY_API_KEY`. In GitHub Actions, add `.github/workflows/keploy.yml`: + +```yaml +name: keploy +on: [pull_request] + +jobs: + keploy-test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + - name: Build the app + run: go build -o server . + - name: Install Keploy + run: curl --silent -O -L https://keploy.io/ent/install.sh && bash install.sh -isCI + - name: Replay the kept tests + env: + KEPLOY_PROVEN: "1" + KEPLOY_API_KEY: ${{ secrets.KEPLOY_API_KEY }} + KEPLOY_INVOKER: ci + run: keploy test -c "./server" +``` + +- **`KEPLOY_PROVEN: "1"` is required.** Without it, `keploy test` ignores `verdicts.yaml`, runs every test, the quarantined ones too, and judges them the old way, so it can pass tests the flow quarantined. +- **Use the command you recorded with.** For a Docker app, build the image first, then run the same `docker run` command with the same `--container-name`. For a Compose app, run the same `docker compose up` command with the same `--container-name`. +- **Run it where `keploy/` is.** If `keploy/` isn't at the repository root, for example in a monorepo, set `defaults.run.working-directory` on the job to the directory that holds it, and point `go-version-file` at that directory's `go.mod`: `defaults.run` applies to `run:` steps only. +- **Your app's dependencies don't need to run.** Keploy answers your app's calls from the mocks it downloads from the Keploy registry. Give your app the same configuration it had when you recorded, such as the environment variables it reads at startup. Only a `mock.real` dependency must run in CI. +- **`KEPLOY_INVOKER: ci`** makes a missing sign-in fail at once, instead of waiting for a browser login. +- Don't add `--mappings` or `--freezeTime`. Under the switch, `keploy test` uses the committed `mappings.yaml` and always freezes your app's clock. + +The CI run first lists what it skips, and why, with what to do for each reason: + +```text +QUARANTINED, not run (2): `keploy test --all` runs them + test-set-0/post-learn-batch-1 flaky/unstable_order HTTP POST /batch to localhost:8096 (no_match) sent an array in another order since 2026-10-20 + test-set-0/post-learn-order-1 flaky/request_changed HTTP POST /orders to localhost:8096 (no_match) changed since 2026-10-20 + by class: keploy 0 · app-infra 0 · flaky 2 · manual 0 + flaky/request_changed: a request field changed between runs in a way keploy does not learn: make it deterministic, then re-record + flaky/unstable_order: the app sent an array in another order than it recorded: give it a stable order in the app, then re-record +``` + +After the replay it prints a `SET` line for each test set, and a `VERDICT failed: ` line when the run fails: + +```text +SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run +``` + +`keploy test` exits 0 when every test it ran passed and it printed no `VERDICT failed` line. It exits non-zero when a test failed, when no test ran (for example, every test is quarantined), when a test has no verdict (`NOT JUDGED`), and when the run fails for a reason no single test shows: a `HELD` test, a call charged to no test, or a `mock.real` destination never reached. The test report counts a quarantined test as ignored, and JUnit output marks it skipped, with the message `quarantined by keploy ()`. Other lines you can see: + +- `UNLISTED : no verdicts.yaml, so none of its tests is quarantined`: the set has no `verdicts.yaml`, so every test of it runs. That's expected for a set in which every test replays. +- `STALE /`: the test's entry names another recording, so the test runs. This happens after a hand edit of the test case, or a merge that brought another version of it. +- `UNREADABLE : ...`: Keploy couldn't read the set's `verdicts.yaml`, for example because it holds merge-conflict markers, so it skips none of the set's tests. +- `HELD /`: the test ran, and Keploy holds evidence against it, such as a call no mock answered, so the run fails even if the test passed. + +## Keep the suite healthy + +- **When the app changes on purpose,** record the affected flows again. `keploy record` writes them to a new test set and judges it. Then delete the old test cases they replace, so CI doesn't run them; `KEPLOY_PROVEN=1 keploy test --judge` drops their entries (`PRUNED`). +- **After you fix what a quarantine reason names,** record that flow again, as above, or run `KEPLOY_PROVEN=1 keploy test --judge`. It replays the test sets twice, read-only, drops the entries of tests that now pass, and prints a `keploy quarantine` command for each test that fails. It never adds a skip by itself. +- **To quarantine a test by hand,** for example while a shared sandbox is down: + + ```bash + KEPLOY_PROVEN=1 keploy quarantine test-set-0/get-orders-3 --reason "payments sandbox is down until Friday" + ``` + + The reason is one line of at most 200 bytes. The entry's class is `manual`, and it applies until the test's recording changes. + +- **To run the quarantined tests too,** for example to check a fix, run `KEPLOY_PROVEN=1 keploy test --all`. +- **When CI goes red,** run `KEPLOY_PROVEN=1 keploy status` and follow its `NEXT` block. A call the app now makes differently, or no longer makes, is what a regression looks like, so check the code change before you record again. + +## Capture-loss evidence: RECORDING LOST {#recording-lost} + +Sometimes Keploy can't capture all of a call while it records, for example when its capture buffer runs out of room. Under the switch, such a loss never fails the recording. It's evidence for auto-replay, which judges each affected test the same way as every other: + +- A test whose recording lost nothing it needs replays, and is kept. +- A test whose recording lost what it needs fails to replay, for example on a call with no recorded answer or on a different response. It's quarantined with the usual reason, and its entry also names what the recording lost. +- A test that lost data and that no round could judge is quarantined as `keploy/capture_incomplete`, never kept. + +Under the `JUDGED` line, Keploy prints a `RECORDING LOST` line for each test whose recording lost data, with what was lost and what Keploy decided: + +```text + RECORDING LOST test-set-0/get-orders-1: capture incomplete: keploy dropped captured bytes 1 time(s) for want of room, so the calls they belong to were not recorded whole; it replays, so it is kept + RECORDING LOST test-set-0/post-orders-2: capture incomplete: keploy dropped captured bytes 1 time(s) for want of room, so the calls they belong to were not recorded whole; it does not replay, so it is quarantined (keploy/unmocked_call), its entry naming what the recording lost +``` + +A loss that no single test holds, such as one after the last test, gets one line for the test set, ending with whether auto-replay judged the set's tests without it. A kept test's line is a warning, not a failure: the test replayed against its recording. If many tests lose data, the [capture-loss guide](/docs/running-keploy/capture-loss/) says what causes each kind of loss and what to change. + +On a kernel that can count them, Keploy also reports capture runs the kernel skipped (`during its recording the kernel skipped N capture run(s) ...`). That count is evidence too, never a verdict on its own: on a kernel that can't count them, Keploy judges each test the same way, without that line. + +When auto-replay doesn't run (`--disableAutoReplay`), nothing judges the recording, so a capture loss fails `keploy record` as it does without the switch. When auto-replay runs but can't judge a set whose recording lost data, `keploy record` says its verdict is incomplete and exits non-zero. + +## Freeze time {#freeze-time} + +Under the switch, Keploy always freezes your app's clock when it replays: you don't pass `--freezeTime`. In auto-replay's full round, the clock reads 61.5 seconds after the moment each test was recorded, so time-dependent fields differ and are learned. In the confirming round and in CI, it reads the recorded moment. + +For a Go app there's nothing to rebuild. Keploy patches a copy of your compiled Go binary at run time and runs that copy, natively and in Docker; your own binary and image aren't modified. When Keploy can't patch a binary, it logs one `time-freeze[go]` line that names the binary and the reason, the app runs on the real clock, and `keploy test` lists it on an `UNFROZEN` line, so a pass that relied on the real clock shows. When staging the frozen clock fails for an operational reason, such as a Docker daemon error or an image Keploy couldn't pull, Keploy refuses to start the app unfrozen and says why. See [Time Freezing](/docs/keploy-cloud/time-freezing/) for what Keploy patches and the few cases it can't. + +## Flags and settings the switch replaces {#refused-flags} + +Under `KEPLOY_PROVEN=1`, these flags fail the command at once, with an error that says what replaces them, so a command copied from an older guide doesn't run the old way: + +| Command | Flag | Use instead | +| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `keploy record` | `--auto-replay-health-poll-timeout` | Nothing: auto-replay waits for your app's readiness. | +| `keploy record` | `--auto-replay-mock-noise-detection`, `--auto-replay-schema-noise-detection`, `--auto-replay-mock-noise-strict`, `--auto-replay-schema-noise-strict` | Nothing: auto-replay's learn step learns what changes. | +| `keploy record`, `keploy test` | `--pass-through-ports`, `--global-passthrough`, `--opportunistic-tls-intercept` | [`mock.real`](#mock-real) in `keploy.yml`. | +| `keploy test` | `--mocking` | [`mock.real`](#mock-real) in `keploy.yml`. | +| `keploy test` | `--health-poll-timeout` | Nothing: the first test waits for your app's readiness. | +| `keploy test` | `--must-pass`, `--max-fail-attempts` | `keploy quarantine`, with a reason. | +| `keploy test` | `--flaky-check-retry` | `keploy test --judge`. | +| `keploy test` | `--mock-noise-detection`, `--schema-noise-detection` | Nothing: `keploy record`'s auto-replay learns what changes, and a replay writes no recording. | + +Keploy also ignores the matching `keploy.yml` settings, such as `test.ignoredTests`, `test.selectedTests` (select test sets with `-t` instead), `test.mustPass`, `test.healthPollTimeout`, `record.passThroughPorts` and `bypassRules`, and says so in a warning when they're set. + +## Related + +- [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/): every field, class and reason, and what to do for each. +- [Capture loss warnings](/docs/running-keploy/capture-loss/): why Keploy couldn't record some bytes, and what to change. +- [Time Freezing](/docs/keploy-cloud/time-freezing/): how Keploy freezes a Go app's clock with nothing to rebuild. +- [Mock Registry](/docs/keploy-cloud/mock-registry/): where your mocks live instead of the repository. diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md new file mode 100644 index 000000000..73abad80d --- /dev/null +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -0,0 +1,118 @@ +--- +id: verdicts-yaml +title: verdicts.yaml reference +sidebar_label: verdicts.yaml reference +description: What keploy//verdicts.yaml holds, when an entry skips a test, and what to do for each quarantine class and reason. +tags: + - replay + - ci + - verdicts + - reference +keywords: + - verdicts.yaml + - KEPLOY_PROVEN + - quarantined tests + - keploy quarantine + - keploy test --judge + - unmocked_call + - capture_incomplete + - unstable_order +--- + +Under `KEPLOY_PROVEN=1`, each test set can have a `keploy//verdicts.yaml`: the tests of that set that `keploy test` doesn't replay, and why. It's a deny list. Every test the file doesn't name runs as usual. You commit it with the test set, and your CI's `keploy test` reads it. To learn how the file gets written, see [Record, judge and replay tests in CI](/docs/running-keploy/proven-flow/). + +Three commands write it: + +- `keploy record` adds an entry for each test it recorded that its auto-replay couldn't replay. +- `keploy quarantine` adds an entry you write by hand, with your reason. +- `keploy test --judge` only removes: it drops the entries of tests that now pass, and prints a `keploy quarantine` command for each test that fails. + +`keploy test` without `KEPLOY_PROVEN=1` ignores the file, says so in a warning, and runs every test. + +## Example + + +```yaml +# keploy verdicts: the tests of this set that keploy does not replay, and why. +# Every test not listed here runs as usual. An entry applies only while the test's +# recording is one that `rec` names; after any other change the test runs again. +version: 1 +quarantined: + get-orders-3: + class: manual + reason: payments sandbox is down until Friday + rec: [sha256:7c01d2e911111111111111111111111111111111111111111111111111111111] + since: 2026-10-20 + by: quarantine + keploy: 3.7.0 + post-learn-batch-1: + class: flaky + reason: unstable_order + rec: [sha256:41aa0c3e00000000000000000000000000000000000000000000000000000000] + since: 2026-10-20 + by: record + keploy: 3.7.0 + rounds: {full: failed, confirm: skipped} + calls: + - {protocol: HTTP, call: POST /batch, destination: localhost:8096, phase: no_match} +``` + +## When an entry skips a test {#rules} + +- **An entry applies only to the recording it names.** `rec` lists the hashes of the test's recordings that the entry quarantines: the hash of the test case file and the request bodies stored beside it in `assets/`, not of its mocks. When Keploy itself rewrites the test case, for example to add noise it learned, it moves the entry onto the new hash. Any other change to the test case, such as a hand edit or a merge that brought another version of it, makes the entry stale: `keploy test` prints `STALE` for it and runs the test. +- **A file Keploy can't read skips nothing.** A file with merge-conflict markers, more than one YAML document, a version other than `1`, or YAML that doesn't parse is unreadable. `keploy test` prints `UNREADABLE` for the set and runs all of its tests. +- **So a missing, stale or broken file can only run too much.** It can turn CI red, never falsely green. +- **An entry outlives a deleted test until you judge its set.** When the test case is gone, `KEPLOY_PROVEN=1 keploy test --judge` drops its entry and prints `PRUNED`. Until then the entry skips nothing, because no test carries its name. + +Don't add entries by editing the file: an entry whose `rec` doesn't name the test's current recording never applies. Use `keploy quarantine`, which names it for you. + +## Fields {#fields} + +| Field | Meaning | +| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `version` | The file's format. This release reads and writes `1` only. | +| `quarantined` | One entry per quarantined test, keyed by the test case's name in the set, such as `get-orders-3`. | +| `class` | Who has to act: `keploy`, `app-infra`, `flaky` or `manual`. See [Classes and reasons](#reasons). | +| `reason` | Why the test doesn't replay. For `manual`, your own text from `keploy quarantine --reason`, one line of at most 200 bytes. | +| `rec` | The recordings the entry applies to: `sha256:<64 hex digits>`, optionally followed by `@` for a recording published to a Keploy cloud branch, or `none` for a test with no recording. | +| `since` | The UTC date the test entered quarantine. | +| `by` | What wrote the entry: `record` for `keploy record`, `quarantine` for `keploy quarantine`. | +| `keploy` | The Keploy version that wrote the reason. | +| `rounds` | Each judging round's outcome for the test, each `passed`, `failed` or `skipped`: `full` (auto-replay's full round, with nothing skipped), `confirm` (the confirming round, run as CI runs `keploy test`), and `record` (the run against real infrastructure, which only `keploy mock record` has). A test that's about to be quarantined is `skipped` in `confirm`, as CI will skip it. | +| `calls` | Up to five dependency calls behind the reason: `protocol`, `call` (the method and the URL's shape for HTTP and gRPC, the command alone for other protocols), `destination`, and `phase`, the step of mock matching the call failed at (such as `no_match` or `body_mismatch`) or `record` for a call the recording lost. | +| `mocks` | Up to five recorded mocks behind the reason, by `name` and `kind`. | +| `diff` | The paths of the response fields that differed, such as `body.total`. A key that may be a value, such as an email address that a map is keyed by, is written `{}`. | + +The evidence holds no request or response bodies, query values or headers. A URL keeps only the path segments that read as names, and writes every other one as `{}`: numbers, IDs, UUIDs, hashes, email addresses and tokens with a digit in them. So `/users/alice@example.com/orders/12345` is `/users/{}/orders/{}`. A segment Keploy knows holds a value, such as the place of a variable it learned, is `{}` too. Any other segment made only of letters, such as a slug or a lowercase user name, can't be told from a name, so it's kept: `/users/alice/orders/12345` is `/users/alice/orders/{}`. Review the file before you commit it if your URLs carry such values. + +Keploy writes each entry's fields on fixed lines, one entry per test, sorted by test name, so changes to tests that are far apart in the file merge in Git without a conflict. Two branches that change the same test's entry, or add entries next to each other, can conflict: see [Resolve a merge conflict](#merge-conflict). + +## Classes and reasons {#reasons} + +`keploy test` prints what to do for each class and reason under its `QUARANTINED, not run` list. To re-record a test, record its flow again with `KEPLOY_PROVEN=1 keploy record` and the command you recorded it with: Keploy writes the new recording to a new test set and judges it there. Then delete the old test case from its set, and `KEPLOY_PROVEN=1 keploy test --judge` drops its entry (`PRUNED`). + +| Class and reason | What happened | What to do | +| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `keploy/unmocked_call` | The app made a call its recording has no answer for, or one call more than it recorded. | If the app changed on purpose, re-record the test. If it didn't, check the diff for a regression first. | +| `keploy/unconsumed_mock` | A recorded call was never made. | If the app changed on purpose, re-record the test. If it didn't, check the diff for a regression first. | +| `keploy/mock_missing` | The replay of the build that was just recorded was served other mocks than its recording maps. | Re-record the test. | +| `keploy/capture_incomplete` | Keploy couldn't capture one of the test's calls, such as a TLS handshake it couldn't join or a UDP reply, or the test's recording lost data that no round could judge. | Give the dependency a TCP address Keploy can mock, or list it in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), then re-record. For a recording that lost data, see [Capture-loss evidence](/docs/running-keploy/proven-flow/#recording-lost). | +| `flaky/replay_only` | The test failed on replay with every call answered: the app is nondeterministic (the clock, randomness), or a real dependency's state changed since it was recorded. | Fix that, then re-record. | +| `flaky/flipped` | The test passed in one round and failed in the next: it depends on test order, on another test, or on how the command runs it. | Remove the dependency, then re-record. | +| `flaky/request_changed` | A request field changed between runs in a way Keploy doesn't learn. | Make the field deterministic, then re-record. | +| `flaky/unstable_order` | The app sent an array in another order than it recorded, or an array came back in another order. | Give the array a stable order in the app, then re-record. | +| `app-infra/failed_in_record` | The test failed against real infrastructure when it was recorded, so it has no recording. Only `keploy mock record` writes this reason. | Fix the app or its infrastructure, then record it again. | +| `manual/` | You quarantined the test with `keploy quarantine`. | Fix what your reason names, then remove the entry with `KEPLOY_PROVEN=1 keploy test --judge` once the test passes, or record the test again. | + +A class or reason this Keploy doesn't know, written by a newer one, is shown as it is, and the entry still applies. + +A test can also end with no verdict at all, which is not a quarantine: Keploy prints `NOT JUDGED` for it and writes nothing. That happens when the test calls a dependency over a Unix socket, which Keploy doesn't support, or when Keploy couldn't read or check everything the test did. Such a test fails CI until you fix what the `NOT JUDGED` line names. + +## Resolve a merge conflict {#merge-conflict} + +When two branches change the same test's entry, or add entries next to each other, Git can leave conflict markers in the file, and Keploy then skips none of the set's tests. Run `git checkout --conflict=diff3 keploy//verdicts.yaml` to see both sides' entries whole. Keep both sides' entries, and for a test both sides list, keep every `rec` item of both. Then commit the file. + +## Related + +- [Record, judge and replay tests in CI](/docs/running-keploy/proven-flow/): the flow that writes and reads this file. +- [Capture loss warnings](/docs/running-keploy/capture-loss/): why Keploy couldn't record some bytes, and what to change. diff --git a/versioned_sidebars/version-4.0.0-sidebars.json b/versioned_sidebars/version-4.0.0-sidebars.json index e04e855ef..6b7754454 100644 --- a/versioned_sidebars/version-4.0.0-sidebars.json +++ b/versioned_sidebars/version-4.0.0-sidebars.json @@ -43,6 +43,8 @@ ] }, "running-keploy/cli-commands", + "running-keploy/proven-flow", + "running-keploy/verdicts-yaml", "running-keploy/rename-testcases", "running-keploy/docker-tls", "running-keploy/configuration-file", From e8d11adc659c585cb563a9fea0061d30865b936e Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 14:41:12 +0530 Subject: [PATCH 02/14] docs: say what keploy record's NEXT block and tests with no verdict really are The T2 flow page showed a caveats line in the NEXT block after a green record. keploy record never prints one: its summary counts the confirming round's tests only, and only the JUDGED line names the quarantined ones. The page now shows the record's block as it is, and the caveat where it comes, in a later keploy test's block. Tests with no verdict get their own section in the verdicts.yaml reference, with every cause keploy has: a Unix socket call leaves its test and every later one NOT JUDGED, a call outside every test leaves the whole set unjudged, and server-first destinations, refused connections, unread containers and unclassified processes are causes too. Also: - record: start the app's dependencies before you record a native app; - the switch: record and test refuse a value they cannot read, the other commands take it as off; - readiness: auto-replay and keploy test wait for the app, then their delay (--auto-replay-delay on record, --delay on test); - the record's second Ctrl+C exits 130, and its incomplete-verdict line names the exit code; - --opportunistic-tls-intercept is a record flag, --mocking a test flag; some refusals come after sign-in; - the NOTE line: judging counts the set red and does not upload it, and CI fails only when no test of the whole run runs; - keploy mock record writes verdicts.yaml too; - a URL's query keeps its names and writes each value as {}; - the manual quarantine example is a test that fails, which keploy test --judge keeps; - the CLI reference links the page from record, test and Related. Signed-off-by: slayerjain --- .../running-keploy/cli-commands.md | 5 ++ .../running-keploy/proven-flow.md | 49 ++++++++++++------- .../running-keploy/verdicts-yaml.md | 26 ++++++++-- 3 files changed, 56 insertions(+), 24 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/cli-commands.md b/versioned_docs/version-4.0.0/running-keploy/cli-commands.md index edada8f29..d0513a436 100755 --- a/versioned_docs/version-4.0.0/running-keploy/cli-commands.md +++ b/versioned_docs/version-4.0.0/running-keploy/cli-commands.md @@ -42,6 +42,8 @@ Here are some examples of how to use some common flags: The `record` mode in Keploy allows the user to record Keploy testcases from the API calls. The recorded testcases and generated mocks are then saved in the `keploy` directory in the current working directory. +To have Keploy replay each test it records, judge which ones replay, and run only those in CI, record with `KEPLOY_PROVEN=1`: see [Record, judge and replay tests in CI](/docs/running-keploy/proven-flow/). + Usage: ```bash @@ -129,6 +131,8 @@ keploy record -c "node src/app.js" --rerecord "test-set-0" The `test` mode in Keploy allows the user to run the recoded testcases from the API calls and execute assertion. A detailed report is produced after the tests are executed and it's then saved in the yaml format in `keploy/reports` directory in the current working directory. +With `KEPLOY_PROVEN=1`, `keploy test` skips the tests that `keploy//verdicts.yaml` quarantines: see [Record, judge and replay tests in CI](/docs/running-keploy/proven-flow/) and the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/). + Usage: ```bash @@ -469,6 +473,7 @@ keploy example [flags] ## Related - [Configuration File](/docs/running-keploy/configuration-file/) — set these flags once in `keploy.yaml`. +- [Record, judge and replay tests in CI](/docs/running-keploy/proven-flow/) — `KEPLOY_PROVEN=1`: auto-replay judges each recorded test, and CI runs only the kept ones. - [Recording Filters](/docs/running-keploy/recording-filters/) — control which requests get recorded. - [Keploy Templatize](/docs/running-keploy/keploy-templatize/) — details on the `templatize` command. - [Keploy Passthrough](/docs/running-keploy/keploy-passthrough/) — using the `--passThroughPorts` flag. diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 7135b38c3..5b5b388b8 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -50,7 +50,7 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i - **sudo rights.** Keploy loads eBPF programs, so it runs its agent with `sudo` and may ask for your password. On a CI runner, the runner's passwordless `sudo` is enough. - **Docker**, if your app runs with `docker run` or `docker compose`. - **A Keploy account.** `keploy record` puts a judged test set's mocks in the Keploy registry, and `keploy test` downloads them in CI. Sign in once with `keploy login`. In CI, Keploy signs in with an API key: in [app.keploy.io](https://app.keploy.io), open your user profile, go to **Account Settings → API Keys**, generate a token, and set it as the `KEPLOY_API_KEY` environment variable. -- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. A test that calls a dependency over a Unix socket gets no verdict: Keploy reports it as `NOT JUDGED`, with the remedy to reach that dependency over a TCP address (for example `localhost:5432` instead of a socket path), and a CI run that reaches it fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. +- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it in the run get no verdict: Keploy reports them as `NOT JUDGED`, with the remedy to reach that dependency over a TCP address (for example `localhost:5432` instead of a socket path), mocked or listed in [`mock.real`](#mock-real). A call made outside every test, for example while your app starts, leaves the whole test set without a verdict. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes. ## Install Keploy and sign in @@ -67,7 +67,7 @@ Then turn the flow on for your shell: export KEPLOY_PROVEN=1 ``` -Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. +Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. ## Record your tests @@ -82,6 +82,8 @@ KEPLOY_PROVEN=1 keploy record -c "./server" Start the Go binary itself in `-c`, not a shell, `make` or a script that starts it. Keploy freezes the clock of the Go binary that `-c` starts, as described in [Freeze time](#freeze-time). +**Start your app's dependencies first.** Keploy records your app's real calls to its database and the other services it calls, with their answers, so they must be up while you record, as they are when you run the app yourself. For example, start a database your compose file defines with `docker compose up -d db`, and give your app the same environment variables it always reads, such as its database address. Only replay, in auto-replay and in CI, runs without them. A compose app is different: `docker compose up` starts its dependency services with it. + **Docker.** Use the `docker run` command you already use, and pass the container's name: ```bash @@ -102,13 +104,13 @@ Then send traffic to your app: `curl`, your API test suite, or a script that wal To stop recording, press **Ctrl+C once**, then wait. Keploy says: ```text -auto-replay: recording stopped — now replaying what was just recorded to check for noise. Press Ctrl+C again to skip and exit. +auto-replay: recording stopped — now replaying what was just recorded to check for noise. Press Ctrl+C again to skip and exit. Under KEPLOY_PROVEN=1 a skip leaves keploy record with no verdict on what it recorded, so it does not exit 0: a recording that did not fail on its own exits 128+N for the signal (130 for Ctrl+C). ``` -Auto-replay runs both rounds and prints its verdict. Don't press Ctrl+C a second time: that skips auto-replay, so nothing is judged, no `verdicts.yaml` is written, and `keploy record` exits non-zero. To record without a verdict on purpose, pass `--disableAutoReplay`, or set `record.disableAutoReplay: true` in `keploy.yml`. +Auto-replay runs both rounds and prints its verdict. Don't press Ctrl+C a second time: that skips auto-replay, so nothing is judged, no `verdicts.yaml` is written, and `keploy record` exits 130. To record without a verdict on purpose, pass `--disableAutoReplay`, or set `record.disableAutoReplay: true` in `keploy.yml`. :::caution Don't pass `--delay` to keploy record -`--delay` is a `keploy test` flag. `keploy record` rejects it as an unknown flag and records nothing. Under the switch, auto-replay and `keploy test` wait for your app to be ready before they send the first test: it listens on the port the test connects to, or the health URL you gave answers (`--auto-replay-health-url` on `keploy record`, `--health-url` on `keploy test`). +`--delay` is a `keploy test` flag. `keploy record` rejects it as an unknown flag and records nothing: its auto-replay's delay is `--auto-replay-delay`. Under the switch, auto-replay and `keploy test` wait for your app to be ready before they send the first test: it listens on the port the test connects to, or the health URL you gave answers (`--auto-replay-health-url` on `keploy record`, `--health-url` on `keploy test`). Then they wait their delay, 15 seconds for auto-replay (`--auto-replay-delay`) and 5 for `keploy test` (`--delay`), and send it. ::: ### Choose a capture mode {#capture-mode} @@ -130,7 +132,7 @@ KEPLOY_PROVEN=1 keploy record -c "./server" --low-latency `--low-latency` is a `keploy record` flag only. With it, Keploy runs itself with `sudo` from the start, even for a native app. -A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in is `NOT JUDGED`, with the remedy to make that client trust Keploy's CA. +A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in is `NOT JUDGED`, with the remedy to make that client trust Keploy's CA. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). ### Reach a dependency for real {#mock-real} @@ -144,7 +146,7 @@ mock: - container:redis ``` -A test that uses a `mock.real` dependency runs on the real clock, and CI must start that dependency: a run in which a listed destination is never reached fails. For a compose app, name a `mock.real` service in the compose command you record and test with, such as `docker compose up api redis`, or Keploy's replay leaves it down. The `--pass-through-ports`, `--global-passthrough` and `--opportunistic-tls-intercept` flags, and the `passThroughPorts` and `bypassRules` settings, don't apply under the switch; `mock.real` replaces them. +A test that uses a `mock.real` dependency runs on the real clock, and CI must start that dependency: a run in which a listed destination is never reached fails. For a compose app, name a `mock.real` service in the compose command you record and test with, such as `docker compose up api redis`, or Keploy's replay leaves it down. The `--pass-through-ports` and `--global-passthrough` flags, `keploy record`'s `--opportunistic-tls-intercept` and `keploy test`'s `--mocking`, and the `keploy.yml` settings that match them, such as `passThroughPorts` and `bypassRules`, don't apply under the switch; `mock.real` replaces them. ## Read what auto-replay prints {#read-the-output} @@ -175,7 +177,7 @@ JUDGED test-set-0: 5 replayable, 2 quarantined (1 flaky/request_changed, 1 flak - **Replayable** tests passed both rounds. CI runs them. - **Quarantined** tests are listed in `verdicts.yaml`, so CI skips them. The counts in brackets group them by [class and reason](/docs/running-keploy/verdicts-yaml/#reasons). -- A test that Keploy couldn't judge is in neither count. The confirming round's output names it on a `NOT JUDGED` line, with the reason. +- A test that Keploy couldn't judge is in neither count. The confirming round's output names it on a `NOT JUDGED` line, with the reason: see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). The line ends with one of: @@ -194,7 +196,7 @@ Under the `JUDGED` line, Keploy can print: | `PRUNED /: its recording is no longer on disk` | Keploy dropped an entry whose test's recording is gone. | | `RECORDING LOST /: ...` | The recording of this test lost data. See [Capture-loss evidence](#recording-lost). | | `NOT CHARGED TO A TEST (in CI's shape): ...` | A dependency call made outside every test that Keploy couldn't answer or capture, such as a background poller. CI fails on it. | -| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI would run no test of this set, and a run in which no test ran fails. Fix the reasons and record again. | +| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI runs no test of this set. Judging counts the set as red: it prints `CI would be red`, and doesn't put the set's mocks in the Keploy registry. CI's `keploy test` fails when no test of the whole run runs. Fix the reasons and record again. | Other lines a test set can get: @@ -205,18 +207,16 @@ Other lines a test set can get: ### Exit codes -`keploy record` exits 0 when judging finished and CI would be green as the lists stand, even if it quarantined tests. When it prints `keploy record's judging: CI would be red as these lists stand`, go by that line, not by the exit code: fix what it names before you wire CI. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code. When Keploy recorded tests but couldn't finish judging them, because auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, or a second Ctrl+C stopped it, `keploy record` prints `keploy record's verdict is incomplete: ` and exits non-zero, so a script can't take an unjudged recording for a judged one. +`keploy record` exits 0 when judging finished and CI would be green as the lists stand, even if it quarantined tests. When it prints `keploy record's judging: CI would be red as these lists stand`, go by that line, not by the exit code: fix what it names before you wire CI. When Keploy recorded tests but couldn't finish judging them, because auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, or a second Ctrl+C stopped it, `keploy record` prints `keploy record's verdict is incomplete: ; it exits `, so a script can't take an unjudged recording for a judged one. It exits 1, or 128 plus the signal's number when a signal stopped auto-replay or its judging: 130 for Ctrl+C. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. ### The NEXT block -Every `keploy record` and `keploy test` ends with a `NEXT` block, and `KEPLOY_PROVEN=1 keploy status` prints it any time (add `--json` for a machine-readable copy). After a green record it looks like this: +Every `keploy record` and `keploy test` ends with a `NEXT` block, and `KEPLOY_PROVEN=1 keploy status` prints it any time (add `--json` for a machine-readable copy). After the record above, which kept 5 tests and quarantined 2, it looks like this: ```text === KEPLOY NEXT === phase: verified_green summary: All 5 test(s) passed across 1 test-set(s). Suite is green and dependency-free. -caveats: - - 2 quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only. `KEPLOY_PROVEN=1 keploy test --all` runs them; `KEPLOY_PROVEN=1 keploy test --judge` drops the entries of those that now pass. ask: Want me to (a) cover another resource, (b) wire CI so this runs on every PR, or (c) stop here? do: If wiring CI: the Keploy registry holds each test set's mocks; CI downloads them by the hash in the set's config.yaml (KEPLOY_API_KEY required in CI). Commit exactly the paths on the commit: line (relative to the directory that holds keploy/: run `git add` there, and CI's keploy test there too): ... commit: keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml @@ -228,12 +228,22 @@ protocol: ... =================== ``` +:::caution A green record's NEXT block doesn't mention the quarantined tests +The record's `summary` is its confirming round's: it counts the tests that round ran, and leaves out the ones it skipped because the record quarantines them. The block has no `caveats` line for them either. Only the `JUDGED` line and the `QUARANTINED` lines under it name the quarantined tests, so read those before you call the suite green. The `NEXT` block of a later `keploy test`, such as your CI run, lists them under `caveats`: + +```text +caveats: + - 2 quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only. `KEPLOY_PROVEN=1 keploy test --all` runs them; `KEPLOY_PROVEN=1 keploy test --judge` drops the entries of those that now pass. +``` + +::: + | Field | What it tells you | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `phase` | Where the suite is: `recorded` (not replayed since it changed), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | | `summary` | The last run in one sentence. | | `failures` | Each failing test, with what differed and a remedy. | -| `caveats` | What a green doesn't cover, such as the quarantined tests it didn't run. Read them before you call the suite green. | +| `caveats` | What a green doesn't cover, such as the quarantined tests a `keploy test` run didn't run. Read them before you call the suite green. | | `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | | `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | | `options` | Commands for the next step, each with the switch where the flow needs it. | @@ -327,13 +337,13 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run - **When the app changes on purpose,** record the affected flows again. `keploy record` writes them to a new test set and judges it. Then delete the old test cases they replace, so CI doesn't run them; `KEPLOY_PROVEN=1 keploy test --judge` drops their entries (`PRUNED`). - **After you fix what a quarantine reason names,** record that flow again, as above, or run `KEPLOY_PROVEN=1 keploy test --judge`. It replays the test sets twice, read-only, drops the entries of tests that now pass, and prints a `keploy quarantine` command for each test that fails. It never adds a skip by itself. -- **To quarantine a test by hand,** for example while a shared sandbox is down: +- **To quarantine a test by hand,** for example a test that fails since a change you made on purpose, until you record it again: ```bash - KEPLOY_PROVEN=1 keploy quarantine test-set-0/get-orders-3 --reason "payments sandbox is down until Friday" + KEPLOY_PROVEN=1 keploy quarantine test-set-0/get-orders-3 --reason "fails since the orders API change in PR 812 until it is re-recorded" ``` - The reason is one line of at most 200 bytes. The entry's class is `manual`, and it applies until the test's recording changes. + The reason is one line of at most 200 bytes. The entry's class is `manual`, and it applies until the test's recording changes. Quarantine only a test that fails: `KEPLOY_PROVEN=1 keploy test --judge` drops the entry of a test that passes, a manual one too. - **To run the quarantined tests too,** for example to check a fix, run `KEPLOY_PROVEN=1 keploy test --all`. - **When CI goes red,** run `KEPLOY_PROVEN=1 keploy status` and follow its `NEXT` block. A call the app now makes differently, or no longer makes, is what a regression looks like, so check the code change before you record again. @@ -367,13 +377,14 @@ For a Go app there's nothing to rebuild. Keploy patches a copy of your compiled ## Flags and settings the switch replaces {#refused-flags} -Under `KEPLOY_PROVEN=1`, these flags fail the command at once, with an error that says what replaces them, so a command copied from an older guide doesn't run the old way: +Under `KEPLOY_PROVEN=1`, Keploy refuses these flags before it starts your app, with an error that says what replaces them, so a command copied from an older guide doesn't run the old way. Keploy checks some of them after it signs in, so without a session you see the sign-in error first: | Command | Flag | Use instead | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `keploy record` | `--auto-replay-health-poll-timeout` | Nothing: auto-replay waits for your app's readiness. | | `keploy record` | `--auto-replay-mock-noise-detection`, `--auto-replay-schema-noise-detection`, `--auto-replay-mock-noise-strict`, `--auto-replay-schema-noise-strict` | Nothing: auto-replay's learn step learns what changes. | -| `keploy record`, `keploy test` | `--pass-through-ports`, `--global-passthrough`, `--opportunistic-tls-intercept` | [`mock.real`](#mock-real) in `keploy.yml`. | +| `keploy record`, `keploy test` | `--pass-through-ports`, `--global-passthrough` | [`mock.real`](#mock-real) in `keploy.yml`. | +| `keploy record` | `--opportunistic-tls-intercept` | [`mock.real`](#mock-real) in `keploy.yml`. | | `keploy test` | `--mocking` | [`mock.real`](#mock-real) in `keploy.yml`. | | `keploy test` | `--health-poll-timeout` | Nothing: the first test waits for your app's readiness. | | `keploy test` | `--must-pass`, `--max-fail-attempts` | `keploy quarantine`, with a reason. | diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index 73abad80d..727ec032a 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -21,9 +21,10 @@ keywords: Under `KEPLOY_PROVEN=1`, each test set can have a `keploy//verdicts.yaml`: the tests of that set that `keploy test` doesn't replay, and why. It's a deny list. Every test the file doesn't name runs as usual. You commit it with the test set, and your CI's `keploy test` reads it. To learn how the file gets written, see [Record, judge and replay tests in CI](/docs/running-keploy/proven-flow/). -Three commands write it: +These commands write it: - `keploy record` adds an entry for each test it recorded that its auto-replay couldn't replay. +- `keploy mock record`, which records against your real infrastructure, does the same for the tests it records, and is the only one that writes `app-infra/failed_in_record`. - `keploy quarantine` adds an entry you write by hand, with your reason. - `keploy test --judge` only removes: it drops the entries of tests that now pass, and prints a `keploy quarantine` command for each test that fails. @@ -40,7 +41,7 @@ version: 1 quarantined: get-orders-3: class: manual - reason: payments sandbox is down until Friday + reason: fails since the orders API change in PR 812 until it is re-recorded rec: [sha256:7c01d2e911111111111111111111111111111111111111111111111111111111] since: 2026-10-20 by: quarantine @@ -76,14 +77,14 @@ Don't add entries by editing the file: an entry whose `rec` doesn't name the tes | `reason` | Why the test doesn't replay. For `manual`, your own text from `keploy quarantine --reason`, one line of at most 200 bytes. | | `rec` | The recordings the entry applies to: `sha256:<64 hex digits>`, optionally followed by `@` for a recording published to a Keploy cloud branch, or `none` for a test with no recording. | | `since` | The UTC date the test entered quarantine. | -| `by` | What wrote the entry: `record` for `keploy record`, `quarantine` for `keploy quarantine`. | +| `by` | What wrote the entry: `record` for `keploy record` and `keploy mock record`, `quarantine` for `keploy quarantine`. | | `keploy` | The Keploy version that wrote the reason. | | `rounds` | Each judging round's outcome for the test, each `passed`, `failed` or `skipped`: `full` (auto-replay's full round, with nothing skipped), `confirm` (the confirming round, run as CI runs `keploy test`), and `record` (the run against real infrastructure, which only `keploy mock record` has). A test that's about to be quarantined is `skipped` in `confirm`, as CI will skip it. | | `calls` | Up to five dependency calls behind the reason: `protocol`, `call` (the method and the URL's shape for HTTP and gRPC, the command alone for other protocols), `destination`, and `phase`, the step of mock matching the call failed at (such as `no_match` or `body_mismatch`) or `record` for a call the recording lost. | | `mocks` | Up to five recorded mocks behind the reason, by `name` and `kind`. | | `diff` | The paths of the response fields that differed, such as `body.total`. A key that may be a value, such as an email address that a map is keyed by, is written `{}`. | -The evidence holds no request or response bodies, query values or headers. A URL keeps only the path segments that read as names, and writes every other one as `{}`: numbers, IDs, UUIDs, hashes, email addresses and tokens with a digit in them. So `/users/alice@example.com/orders/12345` is `/users/{}/orders/{}`. A segment Keploy knows holds a value, such as the place of a variable it learned, is `{}` too. Any other segment made only of letters, such as a slug or a lowercase user name, can't be told from a name, so it's kept: `/users/alice/orders/12345` is `/users/alice/orders/{}`. Review the file before you commit it if your URLs carry such values. +The evidence holds no request or response bodies, query values or headers. A URL keeps only the path segments that read as names, and writes every other one as `{}`: numbers, IDs, UUIDs, hashes, email addresses and tokens with a digit in them. So `/users/alice@example.com/orders/12345` is `/users/{}/orders/{}`. A segment Keploy knows holds a value, such as the place of a variable it learned, is `{}` too. A query keeps its parameters' names, by the same rule, and writes every value as `{}`: `?expand=items&page=2` is `?expand={}&page={}`. Any other segment made only of letters, such as a slug or a lowercase user name, can't be told from a name, so it's kept: `/users/alice/orders/12345` is `/users/alice/orders/{}`. Review the file before you commit it if your URLs carry such values. Keploy writes each entry's fields on fixed lines, one entry per test, sorted by test name, so changes to tests that are far apart in the file merge in Git without a conflict. Two branches that change the same test's entry, or add entries next to each other, can conflict: see [Resolve a merge conflict](#merge-conflict). @@ -106,7 +107,22 @@ Keploy writes each entry's fields on fixed lines, one entry per test, sorted by A class or reason this Keploy doesn't know, written by a newer one, is shown as it is, and the entry still applies. -A test can also end with no verdict at all, which is not a quarantine: Keploy prints `NOT JUDGED` for it and writes nothing. That happens when the test calls a dependency over a Unix socket, which Keploy doesn't support, or when Keploy couldn't read or check everything the test did. Such a test fails CI until you fix what the `NOT JUDGED` line names. +## Tests with no verdict {#not-judged} + +A test can also end with no verdict at all, which is not a quarantine: Keploy prints `NOT JUDGED` for it, with the reason, and writes no entry for it. In CI, a test with no verdict fails the run, and the run's `VERDICT failed` line says what to do. It happens when the test called something Keploy can't capture, or can't tell it captured, or when Keploy couldn't read its own evidence about the test: + +| Cause | Which tests have no verdict | What to do | +| ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | +| Called a dependency over a Unix socket, which Keploy doesn't support. | That test and every later one. | Reach the dependency over a TCP address, mocked or listed in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real). | +| Ran after Keploy's report of Unix-socket calls stopped, so whether a test made one can't be told. | That test and every later one. | Report it to Keploy, with the agent's log. Nothing you change in your app ends it. | +| Ran while Keploy couldn't read a process's PID namespace, so whether its calls were your app's can't be told. | That test and every later one. | Report it to Keploy, with the agent's log, which names the process. | +| Hung up on its TLS handshake with Keploy without a word, which a client that doesn't trust Keploy's CA can do. | That test. | Make the client trust Keploy's CA, for example with `NODE_EXTRA_CA_CERTS` for Node.js, or `--cacert` for the curl command line. | +| Sent data to a server-first protocol Keploy has no parser for, such as SMTP, FTP, POP3, IMAP or NATS, which it relays. | That test. | List the destination in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), to reach it for real. | +| Connected to a destination that refused the connection at once, such as a Redis error or a MySQL error packet. | That test. | Find why the destination refused your app's connections, such as its connection limit or protected mode. | +| Called a container whose configuration the Docker daemon didn't give Keploy, so the call's container can't be told. | That test. | Make `docker inspect` of the container answer: another operation on it, such as a stop's grace period or a hung `exec`, holds it. | +| Keploy couldn't read or check everything the test did: the calls no mock answered, the mocks it used, or its learned values against its calls. | That test. | It's Keploy's failure, not your app's: report it to Keploy, with the run's log. | + +A call to something Keploy can't capture that your app makes outside every test, for example while it starts, or after the last test, leaves the whole test set without a verdict. ## Resolve a merge conflict {#merge-conflict} From 43ae47eb628a49b4b365315c4284678e1a2ec676 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 17:24:35 +0530 Subject: [PATCH 03/14] docs: a test with no verdict sinks its set at record time; what a capture loss costs Review round 2 of the T2 flow pages, checked against a CI run of the gating combination (keploy/enterprise#3001, pipeline 12172): - A test with no verdict (a Unix socket, an abandoned TLS handshake, ...) leaves its whole test set NOT JUDGED when you record: nothing is written for the set and keploy record exits 1. The pages say so, quote the real NOT JUDGED, RECORDING LOST and incomplete-verdict lines, and say where the remedy is (the next_step of the WARN and the ERROR above them). - RECORDING LOST: the agent can leave a request that overlaps a capture loss out of the recording altogether (tcs_suppressed_total), and charge the loss to another test, which may be kept. A caution box says so, and how to notice it. - mock.real container entries are container:, with no tag, and match every container of that image. - docker run: how its -p, --network and --add-host reach Keploy's agent container, and how to reach a database on the Docker host; keploy test reads -c and --container-name from the keploy.yml record created. - UNFROZEN's pid-namespace form, which every native run prints when Keploy runs inside a container, and why it doesn't fail the run. - The CI run's other lines (RAN ANYWAY, UNATTRIBUTED, NOT JUDGED, ENDED, NOT RUN, REAL, LIVE, MOCKED, UNFROZEN), mappings.json and config.json in the commit table, the API key path (Settings, API Keys), a version note (Keploy Enterprise v3.8.80 and later), and a pointer from the GitHub CI page to the T2 job. Signed-off-by: slayerjain --- versioned_docs/version-4.0.0/ci-cd/github.md | 4 + .../running-keploy/proven-flow.md | 80 +++++++++++++------ .../running-keploy/verdicts-yaml.md | 21 ++++- 3 files changed, 77 insertions(+), 28 deletions(-) diff --git a/versioned_docs/version-4.0.0/ci-cd/github.md b/versioned_docs/version-4.0.0/ci-cd/github.md index eb9352882..ac51a5888 100644 --- a/versioned_docs/version-4.0.0/ci-cd/github.md +++ b/versioned_docs/version-4.0.0/ci-cd/github.md @@ -34,6 +34,10 @@ import ProductTier from '@site/src/components/ProductTier'; +:::tip Run only the tests Keploy judged +If you record with `KEPLOY_PROVEN=1`, CI must run `keploy test` with the same switch, so that it skips the tests `verdicts.yaml` quarantines. See [Run the kept tests in CI](/docs/running-keploy/proven-flow/#ci) for that job. +::: + Keploy can be integrated with GitHub by two methods:- 1. [Using Shell Scripts](#shell-scripts) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 5b5b388b8..5f9aa51fb 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -24,6 +24,8 @@ With `KEPLOY_PROVEN=1`, `keploy record` does more than save tests. When you stop :::note What this release covers The flow is for Go applications on Linux: amd64 with kernel 5.10 or later, and arm64 with kernel 6.6 or later. Calls over Unix sockets aren't supported. See [Requirements](#requirements). + +This page describes Keploy Enterprise v3.8.80 and later. Run `keploy --version` to see yours, and `keploy update` to update it. ::: ## When to use it @@ -31,7 +33,8 @@ The flow is for Go applications on Linux: amd64 with kernel 5.10 or later, and a Use this flow when you want recorded API tests to run in CI with no dependencies running, and you want to trust the result: - A kept test passed two replays in which every call it made was answered from the recording, and every call recorded for it was made. -- A test that can't replay yet, because of a dependency Keploy can't mock, a call it couldn't capture, or an app that isn't deterministic, doesn't block you. Auto-replay filters it out and says why, and the rest of the suite runs. +- A test that doesn't replay, because it made a call its recording has no answer for, a call changed, or the app isn't deterministic, doesn't block you. Auto-replay quarantines it with the reason, and CI runs the rest of the suite. +- A test that Keploy can't judge at all is different. One that called a dependency over a Unix socket is an example. When you record, such a test leaves its whole test set without a verdict: Keploy writes nothing for the set, and `keploy record` exits 1. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described in the [CLI reference](/docs/running-keploy/cli-commands/), and `keploy test` ignores `verdicts.yaml`. @@ -49,8 +52,8 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i - **A Go application,** built with Go 1.20 or later. Other languages aren't part of this release. - **sudo rights.** Keploy loads eBPF programs, so it runs its agent with `sudo` and may ask for your password. On a CI runner, the runner's passwordless `sudo` is enough. - **Docker**, if your app runs with `docker run` or `docker compose`. -- **A Keploy account.** `keploy record` puts a judged test set's mocks in the Keploy registry, and `keploy test` downloads them in CI. Sign in once with `keploy login`. In CI, Keploy signs in with an API key: in [app.keploy.io](https://app.keploy.io), open your user profile, go to **Account Settings → API Keys**, generate a token, and set it as the `KEPLOY_API_KEY` environment variable. -- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it in the run get no verdict: Keploy reports them as `NOT JUDGED`, with the remedy to reach that dependency over a TCP address (for example `localhost:5432` instead of a socket path), mocked or listed in [`mock.real`](#mock-real). A call made outside every test, for example while your app starts, leaves the whole test set without a verdict. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes. +- **A Keploy account.** `keploy record` puts a judged test set's mocks in the Keploy registry, and `keploy test` downloads them in CI. Sign in once with `keploy login`. In CI, Keploy signs in with an API key: in [app.keploy.io](https://app.keploy.io), open **Settings → API Keys**, create a key, and set it as the `KEPLOY_API_KEY` environment variable. +- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it have no verdict, so the test set has none either. When you record, Keploy writes nothing for that test set, not even for its other tests, prints `NOT JUDGED` for it, and `keploy record` exits 1. Reach that dependency over a TCP address instead, for example `localhost:5432` instead of a socket path, mocked or listed in [`mock.real`](#mock-real), and record again. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes, and where Keploy names each one. ## Install Keploy and sign in @@ -91,6 +94,8 @@ docker build -t my-app:latest . KEPLOY_PROVEN=1 keploy record -c "docker run -p 8080:8080 --name my-app my-app:latest" --container-name my-app ``` +Keploy runs your container in the network of its own agent container, and moves the command's `-p`, `--network` and `--add-host` options to that container, so your app reaches what it reaches without Keploy. To reach a database on the Docker host, such as one you started with `docker compose up -d db`, use `host.docker.internal` and the port the host publishes, as you would without Keploy: for example `--add-host host.docker.internal:host-gateway -e DB_HOST=host.docker.internal -e DB_PORT=5433`. + **Docker Compose.** Pass the app's service name, or its `container_name`, from your compose file. A compose app needs `--container-name`: ```bash @@ -132,11 +137,11 @@ KEPLOY_PROVEN=1 keploy record -c "./server" --low-latency `--low-latency` is a `keploy record` flag only. With it, Keploy runs itself with `sudo` from the start, even for a native app. -A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in is `NOT JUDGED`, with the remedy to make that client trust Keploy's CA. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). +A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in has no verdict, so when you record, its whole test set is `NOT JUDGED`, as for a Unix socket. The remedy is to make that client trust Keploy's CA. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). ### Reach a dependency for real {#mock-real} -Under the switch, Keploy records and mocks every call your app makes. To keep a dependency live in recording, auto-replay and CI, list it under `mock.real` in `keploy.yml`, as a host and port, an address, or a container: +Under the switch, Keploy records and mocks every call your app makes. To keep a dependency live in recording, auto-replay and CI, list it under `mock.real` in `keploy.yml`: ```yaml mock: @@ -146,6 +151,12 @@ mock: - container:redis ``` +Each entry is one of: + +- **A name and port,** such as `auth.example.com:443`. +- **An address and port,** such as `localhost:6379`, `10.0.4.7:6379` or `[::1]:5432`. `localhost` means both `127.0.0.1` and `::1`. +- **`container:` and an image repository, without a tag,** such as `container:redis` for a container that runs `redis:7-alpine`. It matches every container of that image, on every port. It isn't a container's name or a compose service's name: for a service that runs `image: redis:7-alpine`, write `container:redis`, whatever the service is called. An entry with a tag or a digest, such as `container:redis:7`, is refused. + A test that uses a `mock.real` dependency runs on the real clock, and CI must start that dependency: a run in which a listed destination is never reached fails. For a compose app, name a `mock.real` service in the compose command you record and test with, such as `docker compose up api redis`, or Keploy's replay leaves it down. The `--pass-through-ports` and `--global-passthrough` flags, `keploy record`'s `--opportunistic-tls-intercept` and `keploy test`'s `--mocking`, and the `keploy.yml` settings that match them, such as `passThroughPorts` and `bypassRules`, don't apply under the switch; `mock.real` replaces them. ## Read what auto-replay prints {#read-the-output} @@ -177,7 +188,7 @@ JUDGED test-set-0: 5 replayable, 2 quarantined (1 flaky/request_changed, 1 flak - **Replayable** tests passed both rounds. CI runs them. - **Quarantined** tests are listed in `verdicts.yaml`, so CI skips them. The counts in brackets group them by [class and reason](/docs/running-keploy/verdicts-yaml/#reasons). -- A test that Keploy couldn't judge is in neither count. The confirming round's output names it on a `NOT JUDGED` line, with the reason: see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). +- A test set with a test that Keploy couldn't judge gets no `JUDGED` line. It gets a `NOT JUDGED` line instead, nothing is written for it, and `keploy record` exits 1: see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). The line ends with one of: @@ -200,7 +211,7 @@ Under the `JUDGED` line, Keploy can print: Other lines a test set can get: -- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, for example because a round stopped early or the app never answered a test. Fix what the reason names and record again. A set with no test case gets `NOT JUDGED : it has no test case to judge`. +- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name and record the set again. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example. A set with no test case gets `NOT JUDGED : it has no test case to judge`. - `NOT WRITTEN : keploy//verdicts.yaml changed while keploy record judged, so it stays as it was`: something else changed the file meanwhile, such as a `git pull` or another Keploy run, so nothing Keploy decided for the set was written. The same line ends `could not be written, so it stays as it was` when the write failed, for example on file permissions. Run `KEPLOY_PROVEN=1 keploy test --judge`: it replays the set and prints a `keploy quarantine` command for each test that fails, which you run if you agree (it writes a `manual` entry). Or delete the set and record it again. - `QUARANTINE? / ...`, followed by a `keploy quarantine` command: a test failed the way CI runs it, and no entry skips it. Keploy never adds that entry by itself. Run the printed command, with the switch set, if you agree, or fix the test. - `keploy record's judging: CI would be red as these lists stand`: what's printed above it would fail CI. Fix it before you wire CI. @@ -248,7 +259,7 @@ caveats: | `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | | `options` | Commands for the next step, each with the switch where the flow needs it. | -For a Docker or Compose app, add the `--container-name` you recorded with to the `keploy record` and `keploy test` commands the block prints. +The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. Upload them with the `keploy mock upload -t ` command it prints, then run `KEPLOY_PROVEN=1 keploy status` again. @@ -260,19 +271,19 @@ From the directory that holds `keploy/`, `git add` exactly the paths on the `NEX git add keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml ``` -| Path | What it is | Commit it? | -| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| `keploy.yml` | Keploy's settings for your app, such as its command and `mock.real`. | Yes | -| `keploy/.gitignore` | Keeps `mocks.yaml`, `reports/` and Keploy's other local files out of the repository. | Yes | -| `keploy//tests/`, and `assets/` when present | The test cases, with the noise auto-replay learned. | Yes | -| `keploy//mappings.yaml` | Which recorded calls each test owns. Under the switch, only `keploy record` writes it. | Yes | -| `keploy//verdicts.yaml` | The tests CI skips, and why. See the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/). | Yes | -| `keploy//config.yaml` | Names the upload of the set's mocks in the Keploy registry. | Yes | -| `keploy//seed.sh`, when present | Your seed script for the set. | Yes | -| `keploy//mocks.yaml` | The recorded dependency calls. | **Never.** It's in the Keploy registry, and `keploy/.gitignore` keeps it out. Don't remove that line. | -| `keploy//mocks.json` or `mocks.gob` | The recorded dependency calls, in the JSON or gob storage format. The registry takes only `mocks.yaml`. | Yes: CI reads it from the checkout, and the `commit:` line lists it. | -| `keploy//secret.yaml` | Secrets that `keploy sanitize` took out of the recording. | Never | -| `keploy/reports/` | Run reports. | No | +| Path | What it is | Commit it? | +| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `keploy.yml` | Keploy's settings for your app, such as its command and `mock.real`. | Yes | +| `keploy/.gitignore` | Keeps Keploy's local files, such as `reports/`, out of the repository. Once Keploy has put a set's mocks in the registry, it keeps `mocks.yaml` out too. | Yes | +| `keploy//tests/`, and `assets/` when present | The test cases, with the noise auto-replay learned. | Yes | +| `keploy//mappings.yaml` | Which recorded calls each test owns. Under the switch, only `keploy record` writes it. `mappings.json` in the JSON storage format. | Yes | +| `keploy//verdicts.yaml` | The tests CI skips, and why. See the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/). | Yes | +| `keploy//config.yaml` | Names the upload of the set's mocks in the Keploy registry. `config.json` in the JSON storage format. | Yes | +| `keploy//seed.sh`, when present | Your seed script for the set. | Yes | +| `keploy//mocks.yaml` | The recorded dependency calls. | **Never.** It's in the Keploy registry. When Keploy puts it there, it adds `/*/mocks.yaml` to `keploy/.gitignore` to keep it out of the repository. Don't remove that line. | +| `keploy//mocks.json` or `mocks.gob` | The recorded dependency calls, in the JSON or gob storage format. The registry takes only `mocks.yaml`. | Yes: CI reads it from the checkout, and the `commit:` line lists it. | +| `keploy//secret.yaml` | Secrets that `keploy sanitize` took out of the recording. | Never | +| `keploy/reports/` | Run reports. | No | ## Run the kept tests in CI {#ci} @@ -303,7 +314,7 @@ jobs: ``` - **`KEPLOY_PROVEN: "1"` is required.** Without it, `keploy test` ignores `verdicts.yaml`, runs every test, the quarantined ones too, and judges them the old way, so it can pass tests the flow quarantined. -- **Use the command you recorded with.** For a Docker app, build the image first, then run the same `docker run` command with the same `--container-name`. For a Compose app, run the same `docker compose up` command with the same `--container-name`. +- **Use the command you recorded with.** For a Docker app, build the image first, for example with `docker build -t my-app:latest .`, then run `keploy test` with the same `docker run` command. For a Compose app, run it with the same `docker compose up` command. `keploy test` takes `--container-name` from the committed `keploy.yml`, as [the NEXT block](#the-next-block) says. If your `docker run` command names a network with `--network`, create that network in the job first, with `docker network create`. - **Run it where `keploy/` is.** If `keploy/` isn't at the repository root, for example in a monorepo, set `defaults.run.working-directory` on the job to the directory that holds it, and point `go-version-file` at that directory's `go.mod`: `defaults.run` applies to `run:` steps only. - **Your app's dependencies don't need to run.** Keploy answers your app's calls from the mocks it downloads from the Keploy registry. Give your app the same configuration it had when you recorded, such as the environment variables it reads at startup. Only a `mock.real` dependency must run in CI. - **`KEPLOY_INVOKER: ci`** makes a missing sign-in fail at once, instead of waiting for a browser login. @@ -330,8 +341,17 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run - `UNLISTED : no verdicts.yaml, so none of its tests is quarantined`: the set has no `verdicts.yaml`, so every test of it runs. That's expected for a set in which every test replays. - `STALE /`: the test's entry names another recording, so the test runs. This happens after a hand edit of the test case, or a merge that brought another version of it. +- `RAN ANYWAY / /`: a quarantined test that ran because you passed `--all`. - `UNREADABLE : ...`: Keploy couldn't read the set's `verdicts.yaml`, for example because it holds merge-conflict markers, so it skips none of the set's tests. - `HELD /`: the test ran, and Keploy holds evidence against it, such as a call no mock answered, so the run fails even if the test passed. +- `UNATTRIBUTED : , charged to no test`: calls that Keploy couldn't answer or capture, made outside every test, such as by a background poller. The run fails. +- `NOT JUDGED /: , so it has no verdict`: the test has no verdict, for one of the causes in [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). The run fails, and its `VERDICT failed` line gives the remedy. A set Keploy couldn't judge as a whole gets `NOT JUDGED : ...` instead. +- `ENDED : `: `keploy test` ended the set other than `PASSED`, for example `FAILED` with no failing test. The run fails. +- `NOT RUN : the run stopped before it`: none of the set's tests ran. +- `REAL /: ( connections, failed)`: the test reached a [`mock.real`](#mock-real) dependency for real. +- `LIVE /: :`: the test called a listener of the run's own processes, such as a database your app starts as its child process. Keploy lets that call through, and never records it. +- `MOCKED /: (mocked: ...)`: the test called a local destination outside the run, such as another container or a process you started yourself, and Keploy answered it from the mocks. If that destination is your app, the test tests nothing: start the app with the `-c` command you give Keploy, then record again. The line ends with that remedy. +- `UNFROZEN : `: a test or a process that ran on the real clock, or whose clock Keploy couldn't check. See [Freeze time](#freeze-time). ## Keep the suite healthy @@ -350,12 +370,16 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run ## Capture-loss evidence: RECORDING LOST {#recording-lost} -Sometimes Keploy can't capture all of a call while it records, for example when its capture buffer runs out of room. Under the switch, such a loss never fails the recording. It's evidence for auto-replay, which judges each affected test the same way as every other: +Sometimes Keploy can't capture all of what your app's connections carry while it records. For example, its capture buffer runs out of room, or, in low-latency mode, the app sends bytes that were never in its memory, such as a file it sends with `sendfile`. Under the switch, such a loss doesn't fail `keploy record` by itself. Keploy charges each loss to the test that was open when it counted the loss, and auto-replay judges that test the same way as every other: - A test whose recording lost nothing it needs replays, and is kept. - A test whose recording lost what it needs fails to replay, for example on a call with no recorded answer or on a different response. It's quarantined with the usual reason, and its entry also names what the recording lost. - A test that lost data and that no round could judge is quarantined as `keploy/capture_incomplete`, never kept. +:::caution A loss can cost you tests, and land on another test +When a capture loss, or memory pressure, overlaps a request, Keploy's agent can leave that request out of the recording altogether, so that it never saves a test whose mocks are incomplete. That request then has no test case, so nothing judges it, and its loss is charged to another test of the set. That test may replay and be kept: its `RECORDING LOST` line then names a test that didn't lose what it needs, and says it's kept. The agent counts the requests it left out as `tcs_suppressed_total` on its `agent: recording complete` line, which it logs at `WARN` level when they're a quarter or more of the session's requests. So when a `RECORDING LOST` line appears, compare the test cases on disk with the requests you sent, and record the missing ones again. A loss of bytes that were never in your app's memory happens in low-latency mode only, so record those flows in proxy mode. +::: + Under the `JUDGED` line, Keploy prints a `RECORDING LOST` line for each test whose recording lost data, with what was lost and what Keploy decided: ```text @@ -363,7 +387,7 @@ Under the `JUDGED` line, Keploy prints a `RECORDING LOST` line for each test who RECORDING LOST test-set-0/post-orders-2: capture incomplete: keploy dropped captured bytes 1 time(s) for want of room, so the calls they belong to were not recorded whole; it does not replay, so it is quarantined (keploy/unmocked_call), its entry naming what the recording lost ``` -A loss that no single test holds, such as one after the last test, gets one line for the test set, ending with whether auto-replay judged the set's tests without it. A kept test's line is a warning, not a failure: the test replayed against its recording. If many tests lose data, the [capture-loss guide](/docs/running-keploy/capture-loss/) says what causes each kind of loss and what to change. +A test with no verdict, such as one that called a Unix socket, gets a `RECORDING LOST` line too, ending `its set was not judged`, under its set's `NOT JUDGED` line. A loss that no single test holds, such as one after the last test, gets one line for the test set, ending with whether auto-replay judged the set's tests without it. A kept test's line is a warning, not a failure: the test replayed against its recording. If many tests lose data, the [capture-loss guide](/docs/running-keploy/capture-loss/) says what causes each kind of loss and what to change. On a kernel that can count them, Keploy also reports capture runs the kernel skipped (`during its recording the kernel skipped N capture run(s) ...`). That count is evidence too, never a verdict on its own: on a kernel that can't count them, Keploy judges each test the same way, without that line. @@ -375,6 +399,14 @@ Under the switch, Keploy always freezes your app's clock when it replays: you do For a Go app there's nothing to rebuild. Keploy patches a copy of your compiled Go binary at run time and runs that copy, natively and in Docker; your own binary and image aren't modified. When Keploy can't patch a binary, it logs one `time-freeze[go]` line that names the binary and the reason, the app runs on the real clock, and `keploy test` lists it on an `UNFROZEN` line, so a pass that relied on the real clock shows. When staging the frozen clock fails for an operational reason, such as a Docker daemon error or an image Keploy couldn't pull, Keploy refuses to start the app unfrozen and says why. See [Time Freezing](/docs/keploy-cloud/time-freezing/) for what Keploy patches and the few cases it can't. +`UNFROZEN` has a second form, which doesn't mean your app ran on the real clock: + +```text +UNFROZEN the run's processes: keploy's agent runs in a pid namespace of its own (a container), where the process numbers its proxy reports are not its own, so it cannot check that each process reads the run's clock +``` + +Keploy prints it on every run of a native app, one you start with `-c "./server"`, when Keploy itself runs inside a container: a GitHub Actions job with `container:`, GitLab's Docker executor, or any CI step that runs in a container. Its agent then runs in a PID namespace of its own. Keploy patched your app's binary as usual, and its log says so (`time-freeze[go]: patched the Go app`). What it couldn't do is check, process by process, that each one reads the frozen clock. The line doesn't fail the run. A native app on a runner that's a virtual machine, such as GitHub Actions' `ubuntu-latest`, doesn't get it, and neither does an app you run with `docker run` or `docker compose`. + ## Flags and settings the switch replaces {#refused-flags} Under `KEPLOY_PROVEN=1`, Keploy refuses these flags before it starts your app, with an error that says what replaces them, so a command copied from an older guide doesn't run the old way. Keploy checks some of them after it signs in, so without a session you see the sign-in error first: diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index 727ec032a..4d2ab8674 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -45,14 +45,14 @@ quarantined: rec: [sha256:7c01d2e911111111111111111111111111111111111111111111111111111111] since: 2026-10-20 by: quarantine - keploy: 3.7.0 + keploy: 3.8.80 post-learn-batch-1: class: flaky reason: unstable_order rec: [sha256:41aa0c3e00000000000000000000000000000000000000000000000000000000] since: 2026-10-20 by: record - keploy: 3.7.0 + keploy: 3.8.80 rounds: {full: failed, confirm: skipped} calls: - {protocol: HTTP, call: POST /batch, destination: localhost:8096, phase: no_match} @@ -109,7 +109,20 @@ A class or reason this Keploy doesn't know, written by a newer one, is shown as ## Tests with no verdict {#not-judged} -A test can also end with no verdict at all, which is not a quarantine: Keploy prints `NOT JUDGED` for it, with the reason, and writes no entry for it. In CI, a test with no verdict fails the run, and the run's `VERDICT failed` line says what to do. It happens when the test called something Keploy can't capture, or can't tell it captured, or when Keploy couldn't read its own evidence about the test: +A test can also end with no verdict at all, which is not a quarantine. It happens when the test called something Keploy can't capture, or can't tell it captured, or when Keploy couldn't read its own evidence about the test. Keploy writes no entry for such a test, and what happens next depends on where it runs: + +- **When you record.** A test with no verdict leaves its whole test set without one. Keploy judges none of the set's tests: it prints a `NOT JUDGED` line for the set instead of a `JUDGED` line, writes no `verdicts.yaml` for it, and doesn't put its mocks in the Keploy registry. `keploy record` says its verdict is incomplete, and exits 1. For an app that called the Docker daemon's socket in one test, it looks like this: + + ```text + NOT JUDGED test-set-0: round F stopped early (status INTERNAL_ERR: not judged: 2 test(s) ran while keploy could not capture, or could not tell it captured, every call the application made (get-docker-ping-1, get-whoami-2): the application called a Unix socket keploy cannot capture (1 time(s)) while this test or one before it ran (/var/run/docker.sock), so keploy cannot prove this test's calls were all captured); nothing was written for it + RECORDING LOST test-set-0/get-docker-ping-1: not judged: the application called a Unix socket keploy cannot capture (1 time(s)) while this test or one before it ran (/var/run/docker.sock), so keploy cannot prove this test's calls were all captured; its set was not judged + RECORDING LOST test-set-0/get-whoami-2: not judged: the application called a Unix socket keploy cannot capture (1 time(s)) while this test or one before it ran (/var/run/docker.sock), so keploy cannot prove this test's calls were all captured; its set was not judged + keploy record's verdict is incomplete: 1 of the 1 test set it recorded is not judged (test-set-0); it exits 1 + ``` + + The `NOT JUDGED` line names the cause and the tests that have no verdict, and a `RECORDING LOST` line under it names each of them. What to do is in the `next_step` of the log lines above them: the `WARN` Keploy logged when the call happened (`the application called a Unix socket keploy cannot capture: ...`), and the replay's `ERROR` line that starts `test-set is not judged:`. Here it's `reach that dependency over a TCP address (mocked, or listed in mock.real)`. Fix the cause, then record the set again. The `NEXT` block that follows says `verified_red`, although the set isn't judged rather than failed. + +- **In CI.** The tests the second column below names have no verdict. `keploy test` prints a `NOT JUDGED` line for them, and the run fails. Its `VERDICT failed` line says what to do. | Cause | Which tests have no verdict | What to do | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | @@ -122,7 +135,7 @@ A test can also end with no verdict at all, which is not a quarantine: Keploy pr | Called a container whose configuration the Docker daemon didn't give Keploy, so the call's container can't be told. | That test. | Make `docker inspect` of the container answer: another operation on it, such as a stop's grace period or a hung `exec`, holds it. | | Keploy couldn't read or check everything the test did: the calls no mock answered, the mocks it used, or its learned values against its calls. | That test. | It's Keploy's failure, not your app's: report it to Keploy, with the run's log. | -A call to something Keploy can't capture that your app makes outside every test, for example while it starts, or after the last test, leaves the whole test set without a verdict. +A call to something Keploy can't capture that your app makes outside every test, for example while it starts, or after the last test, leaves the whole test set without a verdict, in CI too. ## Resolve a merge conflict {#merge-conflict} From 5b031b571c23caad62b896b6b0eea213ccc009a0 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 19:16:36 +0530 Subject: [PATCH 04/14] docs: recover a NOT JUDGED set by deleting it; keploy-class quarantines are Keploy's to fix Review round 2 of the T2 flow pages found steps a customer could not follow to a green CI, and claims the code or the proof does not back: - Recording again never replaces a set: keploy record writes a new test-set-N and leaves the unjudged one on disk, where the NEXT block's commit: line (gatherProvenCI, every set on disk) still lists it and do: can ask to upload its mocks. The NOT JUDGED, Unix-socket and NOTE steps now say to delete keploy/ first, and the upload advice is scoped to sets CI replays green. - Every verdicts.yaml entry keploy record or keploy mock record writes is decided on the build that recorded the test (proven ModeKeployRecord and ModeMockRecord are same-build; ModeJudge never adds), so a keploy/* quarantine is Keploy failing to answer or match a call: report it. The printed "check the diff" remedy does not apply to it. - The CLI reference no longer lists -d/--delay for keploy record (record has no such flag in keploy v3.6.154 or on main; it exits 8). - Record's exit code: it still exits 0 when it prints "CI would be red". - The page names no unreleased version, and no longer claims arm64 behaves the same without an arm64 run. - The uninitialized phase, the NOT CHARGED (replaying every test) line, the BAKED and NOT BAKED lines and the "bake ... was put back" suffix. Signed-off-by: slayerjain --- .../running-keploy/cli-commands.md | 8 +-- .../running-keploy/proven-flow.md | 51 ++++++++++--------- .../running-keploy/verdicts-yaml.md | 23 +++++++-- 3 files changed, 46 insertions(+), 36 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/cli-commands.md b/versioned_docs/version-4.0.0/running-keploy/cli-commands.md index d0513a436..397e315ef 100755 --- a/versioned_docs/version-4.0.0/running-keploy/cli-commands.md +++ b/versioned_docs/version-4.0.0/running-keploy/cli-commands.md @@ -29,7 +29,7 @@ Here are some examples of how to use some common flags: | Command | Flags Available | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `record` | `-c, --command`, `--config-path`, `--containerName`, `-d, --delay`, `--metadata`, `-n, --networkName`, `--passThroughPorts`, `-p, --path`, `--proxyport`, `--debug` | +| `record` | `-c, --command`, `--config-path`, `--containerName`, `--metadata`, `-n, --networkName`, `--passThroughPorts`, `-p, --path`, `--proxyport`, `--debug` | | `test` | `--apiTimeout`, `-c, --command`, `--config-path`, `--containerName`, `-d, --delay`, `--mongoPassword`, `-n, --net, --networkName`, `--passThroughPorts`, `-p, --path`, `--proxyport`, `-t, --testsets`, `--debug`, `-g, --generateTestReport`, `--removeUnusedMocks`, `--coverage`, `--goCoverage`, `--ignoreOrdering`, `--skip-preview`, `--proto-dir`, `--proto-file`, `--proto-include` | | `gen` (Deprecated) | `--sourceFilePath`, `--testFilePath`,`--coverageReportPath`,`--testCommand`,`--coverageFormat`,`--expectedCoverage`,`--maxIterations`,`--testDir`,`--llmBaseUrl`,`--model`,`--llmApiVersion` | | `normalize` | `-p, --path`, `--test-run`, `--tests` | @@ -76,12 +76,6 @@ keploy record -c "docker compose up" --container-name "my-app-container" > **Podman:** on Linux, with the Keploy that `install.sh` installs (v3.8.60 or later), an application started in rootful Podman with `podman run` or `podman compose` works the same way: `keploy record -c "podman compose up" --container-name "my-app-container"`. See [Platform requirements](/docs/concepts/platform-requirements/#podman) for what `podman compose` needs. -- `-d, --delay uint` - Delay in seconds to run user application. The default is 5 seconds. - -```bash -keploy record -c "node src/app.js" -d 10 -``` - - `--metadata string` - Key-value pairs to be added as metadata in the config.yaml file. If a `name` key is provided, it will be used as the test set name. ```bash diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 5f9aa51fb..c04830db3 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -25,7 +25,7 @@ With `KEPLOY_PROVEN=1`, `keploy record` does more than save tests. When you stop :::note What this release covers The flow is for Go applications on Linux: amd64 with kernel 5.10 or later, and arm64 with kernel 6.6 or later. Calls over Unix sockets aren't supported. See [Requirements](#requirements). -This page describes Keploy Enterprise v3.8.80 and later. Run `keploy --version` to see yours, and `keploy update` to update it. +Run `keploy update` before you start, so that you have the latest Keploy Enterprise release. ::: ## When to use it @@ -48,12 +48,12 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i ## Requirements {#requirements} -- **Linux.** amd64 with kernel 5.10 or later, or arm64 with kernel 6.6 or later. Keploy behaves the same on both. macOS and Windows aren't part of this release. +- **Linux.** amd64 with kernel 5.10 or later, or arm64 with kernel 6.6 or later. macOS and Windows aren't part of this release. - **A Go application,** built with Go 1.20 or later. Other languages aren't part of this release. - **sudo rights.** Keploy loads eBPF programs, so it runs its agent with `sudo` and may ask for your password. On a CI runner, the runner's passwordless `sudo` is enough. - **Docker**, if your app runs with `docker run` or `docker compose`. - **A Keploy account.** `keploy record` puts a judged test set's mocks in the Keploy registry, and `keploy test` downloads them in CI. Sign in once with `keploy login`. In CI, Keploy signs in with an API key: in [app.keploy.io](https://app.keploy.io), open **Settings → API Keys**, create a key, and set it as the `KEPLOY_API_KEY` environment variable. -- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it have no verdict, so the test set has none either. When you record, Keploy writes nothing for that test set, not even for its other tests, prints `NOT JUDGED` for it, and `keploy record` exits 1. Reach that dependency over a TCP address instead, for example `localhost:5432` instead of a socket path, mocked or listed in [`mock.real`](#mock-real), and record again. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes, and where Keploy names each one. +- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it have no verdict, so the test set has none either. When you record, Keploy writes nothing for that test set, not even for its other tests, prints `NOT JUDGED` for it, and `keploy record` exits 1. Reach that dependency over a TCP address instead, for example `localhost:5432` instead of a socket path, mocked or listed in [`mock.real`](#mock-real). Then delete the test set Keploy didn't judge, `keploy/`, and record again: a new recording goes to a new test set, and the old one would stay on the `NEXT` block's `commit:` line. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes, and where Keploy names each one. ## Install Keploy and sign in @@ -200,25 +200,28 @@ The line ends with one of: Under the `JUDGED` line, Keploy can print: -| Line | Meaning | -| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `QUARANTINED / /: ` | The test is now in `verdicts.yaml`. The evidence names the calls, mocks or response fields without their bodies; the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/#fields) says what a URL keeps. See [what to do for each reason](/docs/running-keploy/verdicts-yaml/#reasons). | -| `RUNS AGAIN /: ...` | The test replays now, so its old entry no longer applies, and CI runs it again. | -| `PRUNED /: its recording is no longer on disk` | Keploy dropped an entry whose test's recording is gone. | -| `RECORDING LOST /: ...` | The recording of this test lost data. See [Capture-loss evidence](#recording-lost). | -| `NOT CHARGED TO A TEST (in CI's shape): ...` | A dependency call made outside every test that Keploy couldn't answer or capture, such as a background poller. CI fails on it. | -| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI runs no test of this set. Judging counts the set as red: it prints `CI would be red`, and doesn't put the set's mocks in the Keploy registry. CI's `keploy test` fails when no test of the whole run runs. Fix the reasons and record again. | +| Line | Meaning | +| ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `QUARANTINED / /: ` | The test is now in `verdicts.yaml`. The evidence names the calls, mocks or response fields without their bodies; the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/#fields) says what a URL keeps. See [what to do for each reason](/docs/running-keploy/verdicts-yaml/#reasons). | +| `RUNS AGAIN /: ...` | The test replays now, so its old entry no longer applies, and CI runs it again. | +| `PRUNED /: its recording is no longer on disk` | Keploy dropped an entry whose test's recording is gone. | +| `RECORDING LOST /: ...` | The recording of this test lost data. See [Capture-loss evidence](#recording-lost). | +| `NOT CHARGED TO A TEST (in CI's shape): ...` | A dependency call made outside every test that Keploy couldn't answer or capture, such as a background poller, in the confirming round, which runs as CI does. CI fails on it, and judging counts the set as red. | +| `NOT CHARGED TO A TEST (replaying every test): ...` | The same, seen in the full round, which replays every test. It's evidence only: the confirming round decides whether CI fails, and prints the `(in CI's shape)` line when it does. | +| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI runs no test of this set. Judging counts the set as red: it prints `CI would be red`, and doesn't put the set's mocks in the Keploy registry. CI's `keploy test` fails when no test of the whole run runs. Don't commit the set: delete it, act on [its reasons](/docs/running-keploy/verdicts-yaml/#reasons), and record again. | Other lines a test set can get: -- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name and record the set again. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example. A set with no test case gets `NOT JUDGED : it has no test case to judge`. -- `NOT WRITTEN : keploy//verdicts.yaml changed while keploy record judged, so it stays as it was`: something else changed the file meanwhile, such as a `git pull` or another Keploy run, so nothing Keploy decided for the set was written. The same line ends `could not be written, so it stays as it was` when the write failed, for example on file permissions. Run `KEPLOY_PROVEN=1 keploy test --judge`: it replays the set and prints a `keploy quarantine` command for each test that fails, which you run if you agree (it writes a `manual` entry). Or delete the set and record it again. +- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name, delete the set's directory, `keploy/`, and record again. Don't commit the set or upload its mocks: `keploy record` writes a new set when you record again, and leaves this one on the `NEXT` block's `commit:` line until you delete it. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example. A set with no test case gets `NOT JUDGED : it has no test case to judge`. +- `NOT WRITTEN : keploy//verdicts.yaml changed while keploy record judged, so it stays as it was`: something else changed the file meanwhile, such as a `git pull` or another Keploy run, so nothing Keploy decided for the set was written. The same line ends `could not be written, so it stays as it was` when the write failed, for example on file permissions. Run `KEPLOY_PROVEN=1 keploy test --judge`: it replays the set and prints a `keploy quarantine` command for each test that fails, which you run if you agree (it writes a `manual` entry). Or delete the set and record it again. Until its `verdicts.yaml` says what you decided, don't commit the set or upload its mocks. +- `NOT JUDGED` and `NOT WRITTEN` lines can end `, and the bake of its mappings.yaml's worker scope was put back`. Before the confirming round, Keploy had moved into the set's `mappings.yaml` the recorded calls your app made outside the test that recorded them (see `BAKED` below). Nothing was decided for the set, so Keploy put the file back as `keploy record` wrote it. +- `BAKED : answers from on, in the set's worker scope (mappings.yaml worker:): ...`, printed before the `JUDGED` line: the full round saw your app make a recorded call outside the test that recorded it, for example from a background job, so Keploy moved that mock to the set's worker scope in `mappings.yaml`. `` is `the set's start`, or `after began`: the earliest point at which the full round saw the call. It's part of what you commit with `mappings.yaml`. `NOT BAKED : ...` says Keploy couldn't read or write the file, so those calls stay with their tests. - `QUARANTINE? / ...`, followed by a `keploy quarantine` command: a test failed the way CI runs it, and no entry skips it. Keploy never adds that entry by itself. Run the printed command, with the switch set, if you agree, or fix the test. - `keploy record's judging: CI would be red as these lists stand`: what's printed above it would fail CI. Fix it before you wire CI. ### Exit codes -`keploy record` exits 0 when judging finished and CI would be green as the lists stand, even if it quarantined tests. When it prints `keploy record's judging: CI would be red as these lists stand`, go by that line, not by the exit code: fix what it names before you wire CI. When Keploy recorded tests but couldn't finish judging them, because auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, or a second Ctrl+C stopped it, `keploy record` prints `keploy record's verdict is incomplete: ; it exits `, so a script can't take an unjudged recording for a judged one. It exits 1, or 128 plus the signal's number when a signal stopped auto-replay or its judging: 130 for Ctrl+C. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. +`keploy record` exits 0 when judging finished and CI would be green as the lists stand, even if it quarantined tests. When judging finished and CI would be red, it prints `keploy record's judging: CI would be red as these lists stand, for what is shown above. keploy record keeps its own exit code`, and still exits 0, unless its verdict is also incomplete, as described next. So don't script on the exit code alone: check for that line too, and fix what it names before you wire CI. When Keploy recorded tests but couldn't finish judging them, because auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, or a second Ctrl+C stopped it, `keploy record` prints `keploy record's verdict is incomplete: ; it exits `, so a script can't take an unjudged recording for a judged one. It exits 1, or 128 plus the signal's number when a signal stopped auto-replay or its judging: 130 for Ctrl+C. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. ### The NEXT block @@ -249,19 +252,19 @@ caveats: ::: -| Field | What it tells you | -| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `phase` | Where the suite is: `recorded` (not replayed since it changed), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | -| `summary` | The last run in one sentence. | -| `failures` | Each failing test, with what differed and a remedy. | -| `caveats` | What a green doesn't cover, such as the quarantined tests a `keploy test` run didn't run. Read them before you call the suite green. | -| `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | -| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | -| `options` | Commands for the next step, each with the switch where the flow needs it. | +| Field | What it tells you | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (not replayed since it changed), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | +| `summary` | The last run in one sentence. | +| `failures` | Each failing test, with what differed and a remedy. | +| `caveats` | What a green doesn't cover, such as the quarantined tests a `keploy test` run didn't run. Read them before you call the suite green. | +| `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | +| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | +| `options` | Commands for the next step, each with the switch where the flow needs it. | The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. -When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. Upload them with the `keploy mock upload -t ` command it prints, then run `KEPLOY_PROVEN=1 keploy status` again. +When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. For a set that judging kept green, upload them with the `keploy mock upload -t ` command it prints, then run `KEPLOY_PROVEN=1 keploy status` again. The block says this of any set on disk whose mocks the registry doesn't hold, including the ones `keploy record` didn't upload on purpose: a set it printed `NOT JUDGED` for, or one judging counted red (`CI would be red`, or a `NOTE` that CI runs none of it). Don't upload or commit those: delete the set's directory, `keploy/`, and record again. For a `NOT WRITTEN` set, settle its `verdicts.yaml` first, as its line says. ## Commit the test sets {#commit} diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index 4d2ab8674..dd84a7858 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -90,13 +90,17 @@ Keploy writes each entry's fields on fixed lines, one entry per test, sorted by ## Classes and reasons {#reasons} -`keploy test` prints what to do for each class and reason under its `QUARANTINED, not run` list. To re-record a test, record its flow again with `KEPLOY_PROVEN=1 keploy record` and the command you recorded it with: Keploy writes the new recording to a new test set and judges it there. Then delete the old test case from its set, and `KEPLOY_PROVEN=1 keploy test --judge` drops its entry (`PRUNED`). +`keploy test` prints what to do for each class and reason under its `QUARANTINED, not run` list. + +Every entry that `keploy record` or `keploy mock record` writes was decided on the build that recorded the test: auto-replay replays each recording against the same app, right after you stop recording. So the `keploy` class means what it says: Keploy has to act. On an app that didn't change, Keploy couldn't answer one of the test's calls from its recording, or couldn't match the calls it recorded for it. Checking your diff finds nothing, because there's no diff, and recording the same traffic again can quarantine a different set of tests rather than none. Report such a test to Keploy, with the entry's `calls:` or `mocks:` evidence and the log of the `keploy record` that wrote it. Keploy's printed remedy for `keploy/unmocked_call` and `keploy/unconsumed_mock` says to re-record the test if the app changed on purpose, and to check the diff otherwise: neither applies to an entry from a record's own judging. + +To re-record a test, record its flow again with `KEPLOY_PROVEN=1 keploy record` and the command you recorded it with: Keploy writes the new recording to a new test set and judges it there. Then delete the old test case from its set, and `KEPLOY_PROVEN=1 keploy test --judge` drops its entry (`PRUNED`). | Class and reason | What happened | What to do | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `keploy/unmocked_call` | The app made a call its recording has no answer for, or one call more than it recorded. | If the app changed on purpose, re-record the test. If it didn't, check the diff for a regression first. | -| `keploy/unconsumed_mock` | A recorded call was never made. | If the app changed on purpose, re-record the test. If it didn't, check the diff for a regression first. | -| `keploy/mock_missing` | The replay of the build that was just recorded was served other mocks than its recording maps. | Re-record the test. | +| `keploy/unmocked_call` | On the build that recorded it, the test made a call Keploy had no recorded answer for, or one call more than it recorded. | Report it to Keploy, as [above](#reasons). One cause is yours: requests that overlapped while you recorded. Then record those flows again, [one request at a time](/docs/running-keploy/proven-flow/#record-your-tests). | +| `keploy/unconsumed_mock` | On the build that recorded it, the test never made a call Keploy recorded for it. | The same as for `keploy/unmocked_call`. | +| `keploy/mock_missing` | The replay of the build that was just recorded was served other mocks than its recording maps. | Report it to Keploy, as [above](#reasons). Keploy's printed remedy is to re-record the test: a new recording is judged afresh, and it isn't sure to be kept. | | `keploy/capture_incomplete` | Keploy couldn't capture one of the test's calls, such as a TLS handshake it couldn't join or a UDP reply, or the test's recording lost data that no round could judge. | Give the dependency a TCP address Keploy can mock, or list it in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), then re-record. For a recording that lost data, see [Capture-loss evidence](/docs/running-keploy/proven-flow/#recording-lost). | | `flaky/replay_only` | The test failed on replay with every call answered: the app is nondeterministic (the clock, randomness), or a real dependency's state changed since it was recorded. | Fix that, then re-record. | | `flaky/flipped` | The test passed in one round and failed in the next: it depends on test order, on another test, or on how the command runs it. | Remove the dependency, then re-record. | @@ -120,7 +124,16 @@ A test can also end with no verdict at all, which is not a quarantine. It happen keploy record's verdict is incomplete: 1 of the 1 test set it recorded is not judged (test-set-0); it exits 1 ``` - The `NOT JUDGED` line names the cause and the tests that have no verdict, and a `RECORDING LOST` line under it names each of them. What to do is in the `next_step` of the log lines above them: the `WARN` Keploy logged when the call happened (`the application called a Unix socket keploy cannot capture: ...`), and the replay's `ERROR` line that starts `test-set is not judged:`. Here it's `reach that dependency over a TCP address (mocked, or listed in mock.real)`. Fix the cause, then record the set again. The `NEXT` block that follows says `verified_red`, although the set isn't judged rather than failed. + The `NOT JUDGED` line names the cause and the tests that have no verdict, and a `RECORDING LOST` line under it names each of them. What to do is in the `next_step` of the log lines above them: the `WARN` Keploy logged when the call happened (`the application called a Unix socket keploy cannot capture: ...`), and the replay's `ERROR` line that starts `test-set is not judged:`. Here it's `reach that dependency over a TCP address (mocked, or listed in mock.real)`. The `NEXT` block that follows says `verified_red`, although the set isn't judged rather than failed. + + To recover, fix the cause, delete the set's directory, then record again: + + ```bash + rm -rf keploy/test-set-0 + KEPLOY_PROVEN=1 keploy record -c "./server" + ``` + + Delete the set first. Recording again doesn't replace it: `keploy record` writes a new test set, such as `test-set-1`, and leaves the set it didn't judge on disk. The `NEXT` block's `commit:` line lists every set on disk, that one too. Committed, it runs in CI with no `verdicts.yaml` (`UNLISTED`): every one of its tests runs, though Keploy judged none of them, so CI can fail on tests that a judged set would have quarantined. For the same reason, don't upload its mocks with `keploy mock upload`, even when the `NEXT` block's `do:` asks for it: that advice is for a set CI replays green. If `rm` is refused, Keploy wrote the files as root: use `sudo rm -rf`. - **In CI.** The tests the second column below names have no verdict. `keploy test` prints a `NOT JUDGED` line for them, and the run fails. Its `VERDICT failed` line says what to do. From 1e033711a354f127a81ed5d4a83198e7165fdba3 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 20:15:27 +0530 Subject: [PATCH 05/14] docs: a T2 run without the switch is no verdict, and the NEXT block's commands carry it The NEXT/CI-recipe fix (enterprise train/fix/t2-next-ci) moved to 09b8328ab, rebased onto main with four commits added. Two of them change what a customer sees: - 282e3c547: a keploy test run without KEPLOY_PROVEN on a judged suite (one whose sets hold a verdicts.yaml) ignores the file and runs the quarantined tests the legacy way, so status, and the NEXT block that run ends with, no longer take it for the suite's verdict: the phase is recorded, and the block asks for a run with the switch. The phase table and the CI recipe's KEPLOY_PROVEN bullet say so. - 67ba7145f: every keploy command a T2 NEXT block gives carries KEPLOY_PROVEN=1, in every phase, and the caveat a judged suite gets without the switch says the suite leaves the flow only when each set's verdicts.yaml is deleted. The options row, the switch's paragraph and a new "stays in this flow" bullet say so. 8db80f77b changes do: only past the part the page quotes, and 09b8328ab only a WARN for a mocks.yaml that is not a regular file. Signed-off-by: slayerjain --- .../running-keploy/proven-flow.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index c04830db3..1b28c663d 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -70,7 +70,7 @@ Then turn the flow on for your shell: export KEPLOY_PROVEN=1 ``` -Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. +Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set, and every `keploy` command in the block carries `KEPLOY_PROVEN=1`. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. ## Record your tests @@ -252,15 +252,15 @@ caveats: ::: -| Field | What it tells you | -| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (not replayed since it changed), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | -| `summary` | The last run in one sentence. | -| `failures` | Each failing test, with what differed and a remedy. | -| `caveats` | What a green doesn't cover, such as the quarantined tests a `keploy test` run didn't run. Read them before you call the suite green. | -| `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | -| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | -| `options` | Commands for the next step, each with the switch where the flow needs it. | +| Field | What it tells you | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (not replayed since it changed, or last replayed without the switch: such a run ignored `verdicts.yaml`, so it's no verdict on the suite), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | +| `summary` | The last run in one sentence. | +| `failures` | Each failing test, with what differed and a remedy. | +| `caveats` | What a green doesn't cover, such as the quarantined tests a `keploy test` run didn't run. Read them before you call the suite green. | +| `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | +| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | +| `options` | Commands for the next step. Every `keploy` command in the block carries the switch. | The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. @@ -316,7 +316,7 @@ jobs: run: keploy test -c "./server" ``` -- **`KEPLOY_PROVEN: "1"` is required.** Without it, `keploy test` ignores `verdicts.yaml`, runs every test, the quarantined ones too, and judges them the old way, so it can pass tests the flow quarantined. +- **`KEPLOY_PROVEN: "1"` is required.** Without it, `keploy test` ignores `verdicts.yaml`, runs every test, the quarantined ones too, and judges them the old way, so it can pass tests the flow quarantined. The `NEXT` block such a run ends with, and `keploy status` after it, don't take it for the suite's verdict: they report the suite as `recorded`, and ask for a run with the switch. - **Use the command you recorded with.** For a Docker app, build the image first, for example with `docker build -t my-app:latest .`, then run `keploy test` with the same `docker run` command. For a Compose app, run it with the same `docker compose up` command. `keploy test` takes `--container-name` from the committed `keploy.yml`, as [the NEXT block](#the-next-block) says. If your `docker run` command names a network with `--network`, create that network in the job first, with `docker network create`. - **Run it where `keploy/` is.** If `keploy/` isn't at the repository root, for example in a monorepo, set `defaults.run.working-directory` on the job to the directory that holds it, and point `go-version-file` at that directory's `go.mod`: `defaults.run` applies to `run:` steps only. - **Your app's dependencies don't need to run.** Keploy answers your app's calls from the mocks it downloads from the Keploy registry. Give your app the same configuration it had when you recorded, such as the environment variables it reads at startup. Only a `mock.real` dependency must run in CI. @@ -369,6 +369,7 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run The reason is one line of at most 200 bytes. The entry's class is `manual`, and it applies until the test's recording changes. Quarantine only a test that fails: `KEPLOY_PROVEN=1 keploy test --judge` drops the entry of a test that passes, a manual one too. - **To run the quarantined tests too,** for example to check a fix, run `KEPLOY_PROVEN=1 keploy test --all`. +- **The suite stays in this flow** while any of its test sets holds a `verdicts.yaml`. Deleting every set's `verdicts.yaml` takes it out, and drops each quarantine and its reason with it, so CI would run those tests again. Do that only to leave the flow on purpose, never to make a run pass. - **When CI goes red,** run `KEPLOY_PROVEN=1 keploy status` and follow its `NEXT` block. A call the app now makes differently, or no longer makes, is what a regression looks like, so check the code change before you record again. ## Capture-loss evidence: RECORDING LOST {#recording-lost} From 036d6ee9fe1f8317317210dfa3e742472105c76e Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 21:08:21 +0530 Subject: [PATCH 06/14] docs: a green record's NEXT block says its quarantined tests did not run Enterprise train/fix/record-next-quarantine (32d465d83) fixes the record NEXT block that called a partly quarantined suite green with no caveat: the confirming round's report counts its quarantine skips, so record's green block now carries the same caveat a later keploy test's does ("N quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only ..."). The T2 lane's judging step holds it on every cycle (CI-AR-01 at 5ec0c3f6e). The page documents the fixed block, and drops the caution box that said the block had no caveat. The PR is gated on that branch. Signed-off-by: slayerjain --- .../version-4.0.0/running-keploy/proven-flow.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 1b28c663d..2996f13e0 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -231,6 +231,8 @@ Every `keploy record` and `keploy test` ends with a `NEXT` block, and `KEPLOY_PR === KEPLOY NEXT === phase: verified_green summary: All 5 test(s) passed across 1 test-set(s). Suite is green and dependency-free. +caveats: + - 2 quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only. `KEPLOY_PROVEN=1 keploy test --all` runs them; `KEPLOY_PROVEN=1 keploy test --judge` drops the entries of those that now pass. ask: Want me to (a) cover another resource, (b) wire CI so this runs on every PR, or (c) stop here? do: If wiring CI: the Keploy registry holds each test set's mocks; CI downloads them by the hash in the set's config.yaml (KEPLOY_API_KEY required in CI). Commit exactly the paths on the commit: line (relative to the directory that holds keploy/: run `git add` there, and CI's keploy test there too): ... commit: keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml @@ -242,22 +244,14 @@ protocol: ... =================== ``` -:::caution A green record's NEXT block doesn't mention the quarantined tests -The record's `summary` is its confirming round's: it counts the tests that round ran, and leaves out the ones it skipped because the record quarantines them. The block has no `caveats` line for them either. Only the `JUDGED` line and the `QUARANTINED` lines under it name the quarantined tests, so read those before you call the suite green. The `NEXT` block of a later `keploy test`, such as your CI run, lists them under `caveats`: - -```text -caveats: - - 2 quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only. `KEPLOY_PROVEN=1 keploy test --all` runs them; `KEPLOY_PROVEN=1 keploy test --judge` drops the entries of those that now pass. -``` - -::: +The `summary` counts the tests the confirming round ran, so it leaves out the ones that round skipped because the record quarantines them. The `caveats` line says how many those are: a green covers the other tests only. The `JUDGED` line and the `QUARANTINED` lines under it name each one. The `NEXT` block of a later `keploy test`, such as your CI run, has the same caveat. | Field | What it tells you | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (not replayed since it changed, or last replayed without the switch: such a run ignored `verdicts.yaml`, so it's no verdict on the suite), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | | `summary` | The last run in one sentence. | | `failures` | Each failing test, with what differed and a remedy. | -| `caveats` | What a green doesn't cover, such as the quarantined tests a `keploy test` run didn't run. Read them before you call the suite green. | +| `caveats` | What a green doesn't cover, such as the quarantined tests the run skipped, a record's confirming round included. Read them before you call the suite green. | | `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | | `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | | `options` | Commands for the next step. Every `keploy` command in the block carries the switch. | From 2954edb049c771f014b5f3047c2feb907b649fe9 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Wed, 21 Oct 2026 21:52:44 +0530 Subject: [PATCH 07/14] docs: the keploy mock commands a T2 NEXT block gives carry no switch The NEXT block's do: tells a set whose mocks the registry lacks to run `keploy mock upload -t `, and the uninitialized block offers `keploy mock status`: cliflow builds both without Config.keploy, so neither carries KEPLOY_PROVEN=1 (67ba7145f leaves the mock flow's commands as they are). The two places that say every command in the block carries the switch now say so of every command but those. Signed-off-by: slayerjain --- versioned_docs/version-4.0.0/running-keploy/proven-flow.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 2996f13e0..8254b1d18 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -70,7 +70,7 @@ Then turn the flow on for your shell: export KEPLOY_PROVEN=1 ``` -Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set, and every `keploy` command in the block carries `KEPLOY_PROVEN=1`. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. +Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set, and every `keploy` command in the block carries `KEPLOY_PROVEN=1`, except the `keploy mock` commands, which don't need it. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. ## Record your tests @@ -254,7 +254,7 @@ The `summary` counts the tests the confirming round ran, so it leaves out the on | `caveats` | What a green doesn't cover, such as the quarantined tests the run skipped, a record's confirming round included. Read them before you call the suite green. | | `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | | `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | -| `options` | Commands for the next step. Every `keploy` command in the block carries the switch. | +| `options` | Commands for the next step. Every `keploy` command in the block carries the switch, except the `keploy mock` commands, which don't need it. | The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. From 81342660a8644361d2acd1b596b19edb6f2105b2 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 08:36:12 +0530 Subject: [PATCH 08/14] docs: record's NEXT gives CI's command once, and that run wires CI; set-level reasons Under the in-flight NEXT fix (enterprise train/fix/t2-next-ci-r4, 235114959) a judging round is never the suite's verdict: keploy record's NEXT block is recorded, wires no CI and gives CI's command once on its run: line, and only the gated keploy test's green block carries the quarantined caveat, the commit: line and the CI recipe. The page showed a caveat on record's block that record never prints. - How the flow works and the NEXT section: record's real recorded block, a new "Run CI's command once" step (#gate) with that run's green block, and the field table's ask and run. - E-C12 (enterprise#3023), which r4 is built on: record exits 1 when CI would be red or would run no test; the SET FAILED lines; keploy/set_failed, keploy/set_capture_incomplete and app-infra/real_not_reached; the set_failed round outcome. - The NOTE row (a NOTE set prints no "CI would be red"), the switch on the QUARANTINE? paste line and CI's QUARANTINED header, the SET line's "N not judged", and where an unjudged set left on disk goes: CI's command replays it UNLISTED (pipeline 12234 phase 12a: its Unix-socket test left every later test of the run, the new set's too, unjudged). Signed-off-by: slayerjain --- .../running-keploy/proven-flow.md | 103 ++++++++++++------ .../running-keploy/verdicts-yaml.md | 59 +++++----- 2 files changed, 99 insertions(+), 63 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 8254b1d18..6b159441e 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -43,8 +43,9 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i 1. **Record.** `keploy record` captures each request to your app as a test case, and each call your app makes to its dependencies as a mock. 2. **Replay every test (the full round).** When you stop recording, auto-replay replays each new test with its mocks. Your app's clock is frozen 61.5 seconds after the moment the test was recorded, so values that come from the clock change. Keploy learns what changed: a response field that differs becomes noise on the test case, and a value your app generated and sent to a dependency becomes a variable that the mock follows. 3. **Replay as CI will (the confirming round).** Keploy runs `keploy test` the way your CI will run it, skipping the tests it's about to quarantine. -4. **Judge.** A test that passed both rounds is kept. A test that didn't is written to `keploy//verdicts.yaml` with a class and a reason. Keploy prints a `JUDGED` line for each test set, then a `NEXT` block that says what to commit and how to wire CI. -5. **Run in CI.** You commit the `keploy` folder without its `mocks.yaml` files. Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. +4. **Judge.** A test that passed both rounds is kept. A test that didn't is written to `keploy//verdicts.yaml` with a class and a reason. Keploy prints a `JUDGED` line for each test set, then a `NEXT` block that gives CI's command, `KEPLOY_PROVEN=1 keploy test`. +5. **Run CI's command once.** You run that command on your machine. It skips the quarantined tests and replays the rest, exactly as CI will. Its result is the suite's verdict, and when it's green, its `NEXT` block says what to commit and how to wire CI. +6. **Run in CI.** You commit the `keploy` folder without its `mocks.yaml` files. Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. ## Requirements {#requirements} @@ -53,7 +54,7 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i - **sudo rights.** Keploy loads eBPF programs, so it runs its agent with `sudo` and may ask for your password. On a CI runner, the runner's passwordless `sudo` is enough. - **Docker**, if your app runs with `docker run` or `docker compose`. - **A Keploy account.** `keploy record` puts a judged test set's mocks in the Keploy registry, and `keploy test` downloads them in CI. Sign in once with `keploy login`. In CI, Keploy signs in with an API key: in [app.keploy.io](https://app.keploy.io), open **Settings → API Keys**, create a key, and set it as the `KEPLOY_API_KEY` environment variable. -- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it have no verdict, so the test set has none either. When you record, Keploy writes nothing for that test set, not even for its other tests, prints `NOT JUDGED` for it, and `keploy record` exits 1. Reach that dependency over a TCP address instead, for example `localhost:5432` instead of a socket path, mocked or listed in [`mock.real`](#mock-real). Then delete the test set Keploy didn't judge, `keploy/`, and record again: a new recording goes to a new test set, and the old one would stay on the `NEXT` block's `commit:` line. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes, and where Keploy names each one. +- **TCP dependencies.** Keploy doesn't capture calls over Unix sockets. When your app calls a dependency over a Unix socket, the test it happened in and every test after it have no verdict, so the test set has none either. When you record, Keploy writes nothing for that test set, not even for its other tests, prints `NOT JUDGED` for it, and `keploy record` exits 1. Reach that dependency over a TCP address instead, for example `localhost:5432` instead of a socket path, mocked or listed in [`mock.real`](#mock-real). Then delete the test set Keploy didn't judge, `keploy/`, and record again: a new recording goes to a new test set, and the old one would stay on disk, where `keploy test` replays every one of its tests, judged by nothing. A CI run with a test that has no verdict fails. The sockets of system services that carry no dependency traffic, such as nscd, systemd-resolved, journald, syslog (`/dev/log`) and D-Bus, don't count. The Docker daemon's socket does: an app that drives Docker, for example with Testcontainers, has a real dependency there. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) for the other causes, and where Keploy names each one. ## Install Keploy and sign in @@ -70,7 +71,7 @@ Then turn the flow on for your shell: export KEPLOY_PROVEN=1 ``` -Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set, and every `keploy` command in the block carries `KEPLOY_PROVEN=1`, except the `keploy mock` commands, which don't need it. The commands Keploy prints, such as the `keploy quarantine` line under `QUARANTINE?`, assume the switch is set in your shell. The examples on this page set it on each command, so you can copy any one of them on its own. +Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set. The commands Keploy prints for you to run carry `KEPLOY_PROVEN=1`: every `keploy` command in the `NEXT` block, except the `keploy mock` commands, which don't need it; the `keploy quarantine` line under `QUARANTINE?`; and the commands in each reason's remedy. A few log lines name a command in their `next_step` without it, so set the switch on every Keploy command of this flow all the same. The examples on this page set it on each command, so you can copy any one of them on its own. ## Record your tests @@ -174,7 +175,7 @@ Keploy learns noise only for a test whose dependency calls were all answered. A WARN auto-noise: keploy could not mock a dependency call this testcase made; its differences were not learned as noise and it stays failed ``` -After the confirming round, Keploy prints its verdict for each test set, then the `NEXT` block: +After the confirming round, Keploy prints its verdict for each test set: ```text JUDGED test-set-0: 5 replayable, 2 quarantined (1 flaky/request_changed, 1 flaky/unstable_order); wrote keploy/test-set-0/verdicts.yaml: commit it @@ -200,41 +201,71 @@ The line ends with one of: Under the `JUDGED` line, Keploy can print: -| Line | Meaning | -| ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `QUARANTINED / /: ` | The test is now in `verdicts.yaml`. The evidence names the calls, mocks or response fields without their bodies; the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/#fields) says what a URL keeps. See [what to do for each reason](/docs/running-keploy/verdicts-yaml/#reasons). | -| `RUNS AGAIN /: ...` | The test replays now, so its old entry no longer applies, and CI runs it again. | -| `PRUNED /: its recording is no longer on disk` | Keploy dropped an entry whose test's recording is gone. | -| `RECORDING LOST /: ...` | The recording of this test lost data. See [Capture-loss evidence](#recording-lost). | -| `NOT CHARGED TO A TEST (in CI's shape): ...` | A dependency call made outside every test that Keploy couldn't answer or capture, such as a background poller, in the confirming round, which runs as CI does. CI fails on it, and judging counts the set as red. | -| `NOT CHARGED TO A TEST (replaying every test): ...` | The same, seen in the full round, which replays every test. It's evidence only: the confirming round decides whether CI fails, and prints the `(in CI's shape)` line when it does. | -| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI runs no test of this set. Judging counts the set as red: it prints `CI would be red`, and doesn't put the set's mocks in the Keploy registry. CI's `keploy test` fails when no test of the whole run runs. Don't commit the set: delete it, act on [its reasons](/docs/running-keploy/verdicts-yaml/#reasons), and record again. | +| Line | Meaning | +| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `QUARANTINED / /: ` | The test is now in `verdicts.yaml`. The evidence names the calls, mocks or response fields without their bodies; the [verdicts.yaml reference](/docs/running-keploy/verdicts-yaml/#fields) says what a URL keeps. See [what to do for each reason](/docs/running-keploy/verdicts-yaml/#reasons). | +| `RUNS AGAIN /: ...` | The test replays now, so its old entry no longer applies, and CI runs it again. | +| `PRUNED /: its recording is no longer on disk` | Keploy dropped an entry whose test's recording is gone. | +| `RECORDING LOST /: ...` | The recording of this test lost data. See [Capture-loss evidence](#recording-lost). | +| `SET FAILED (in CI's shape): ` or `SET FAILED (replaying every test): `, then the same label with `, its evidence: ` | A round's `keploy test` failed the whole set for what none of its tests owns: a call your app made before the set's first test, or after its last, that no recording answers, or a call the set's worker scope recorded that your app no longer makes. CI's `keploy test` fails the set for it whichever of its tests run, so Keploy keeps no test of the set, whichever round saw it: it quarantines them all as `keploy/set_failed`, or `keploy/set_capture_incomplete` when Keploy also couldn't capture all that the set did. When a [`mock.real`](#mock-real) destination that was never reached is all the set failed for, the label adds `, for what only the recordings of list`, and only those tests are quarantined, as `app-infra/real_not_reached`. | +| `SET FAILED only replaying every test, after , which CI skips on their own account: ...` | Only the full round failed the set, after it replayed tests that CI skips anyway. The failure may be what those tests did, or it may come again whichever tests run, so Keploy still keeps no test of the set. Fix those tests first. | +| `NOT CHARGED TO A TEST (in CI's shape): ...` | A dependency call made outside every test that Keploy couldn't answer or capture, such as a background poller's, in the confirming round, which runs as CI does. CI's `keploy test` fails the set on it, so Keploy keeps no test of the set, as for `SET FAILED`. | +| `NOT CHARGED TO A TEST (replaying every test): ...` | The same, seen in the full round, which replays every test. It counts as the confirming round's would. | +| `NOTE : every test of it CI reaches is quarantined, so CI runs none of it` | CI runs no test of this set, so none of it is verified, and Keploy doesn't put the set's mocks in the Keploy registry. Don't commit the set: delete it, act on [its reasons](/docs/running-keploy/verdicts-yaml/#reasons), and record again. When every set the record judged gets this line, Keploy says so after the last one: see the two `CI runs none of the tests this record judged` lines below. | Other lines a test set can get: -- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name, delete the set's directory, `keploy/`, and record again. Don't commit the set or upload its mocks: `keploy record` writes a new set when you record again, and leaves this one on the `NEXT` block's `commit:` line until you delete it. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example. A set with no test case gets `NOT JUDGED : it has no test case to judge`. +- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name, delete the set's directory, `keploy/`, and record again. Don't commit the set or upload its mocks: `keploy record` writes a new set when you record again, and leaves this one on disk until you delete it, where `keploy test` replays every one of its tests. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example. A set with no test case gets `NOT JUDGED : it has no test case to judge`. - `NOT WRITTEN : keploy//verdicts.yaml changed while keploy record judged, so it stays as it was`: something else changed the file meanwhile, such as a `git pull` or another Keploy run, so nothing Keploy decided for the set was written. The same line ends `could not be written, so it stays as it was` when the write failed, for example on file permissions. Run `KEPLOY_PROVEN=1 keploy test --judge`: it replays the set and prints a `keploy quarantine` command for each test that fails, which you run if you agree (it writes a `manual` entry). Or delete the set and record it again. Until its `verdicts.yaml` says what you decided, don't commit the set or upload its mocks. - `NOT JUDGED` and `NOT WRITTEN` lines can end `, and the bake of its mappings.yaml's worker scope was put back`. Before the confirming round, Keploy had moved into the set's `mappings.yaml` the recorded calls your app made outside the test that recorded them (see `BAKED` below). Nothing was decided for the set, so Keploy put the file back as `keploy record` wrote it. - `BAKED : answers from on, in the set's worker scope (mappings.yaml worker:): ...`, printed before the `JUDGED` line: the full round saw your app make a recorded call outside the test that recorded it, for example from a background job, so Keploy moved that mock to the set's worker scope in `mappings.yaml`. `` is `the set's start`, or `after began`: the earliest point at which the full round saw the call. It's part of what you commit with `mappings.yaml`. `NOT BAKED : ...` says Keploy couldn't read or write the file, so those calls stay with their tests. -- `QUARANTINE? / ...`, followed by a `keploy quarantine` command: a test failed the way CI runs it, and no entry skips it. Keploy never adds that entry by itself. Run the printed command, with the switch set, if you agree, or fix the test. -- `keploy record's judging: CI would be red as these lists stand`: what's printed above it would fail CI. Fix it before you wire CI. +- `QUARANTINE? / ...`, followed by a `KEPLOY_PROVEN=1 keploy quarantine ...` command: a test failed the way CI runs it, and no entry skips it. Keploy never adds that entry by itself. Run the printed command if you agree, or fix the test. +- `keploy record's judging: CI would be red as these lists stand, for what is shown above, so keploy record does not exit 0`: a test CI runs is red, or CI fails a set for what no test owns, and judging decided no entry for it. Fix it before you wire CI. +- `keploy record's judging: CI runs none of the tests this record judged, every one of them quarantined for what is shown above, and no other test set of the project has a test CI runs: CI's keploy test would run no test and fail, so keploy record does not exit 0`: every test the record judged is quarantined, and CI's `keploy test` fails a run in which no test runs. +- `keploy record's judging: CI runs none of the tests this record judged: every one of them is quarantined, for what is shown above. Another test set of the project has a test CI runs, so keploy record keeps its own exit code ...`: the same, but CI still runs the tests of another set, so CI isn't failed for it. None of what this record judged is verified. ### Exit codes -`keploy record` exits 0 when judging finished and CI would be green as the lists stand, even if it quarantined tests. When judging finished and CI would be red, it prints `keploy record's judging: CI would be red as these lists stand, for what is shown above. keploy record keeps its own exit code`, and still exits 0, unless its verdict is also incomplete, as described next. So don't script on the exit code alone: check for that line too, and fix what it names before you wire CI. When Keploy recorded tests but couldn't finish judging them, because auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, or a second Ctrl+C stopped it, `keploy record` prints `keploy record's verdict is incomplete: ; it exits `, so a script can't take an unjudged recording for a judged one. It exits 1, or 128 plus the signal's number when a signal stopped auto-replay or its judging: 130 for Ctrl+C. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. +`keploy record` exits 0 when it judged every test set it recorded, wrote what it decided, and CI would be green as the lists stand, even if it quarantined tests. Otherwise it prints `keploy record's verdict is incomplete: ; it exits `, so that a script can't take an unjudged recording, or one that CI would fail, for a judged one. That happens when auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, a second Ctrl+C stopped it, CI would be red as the lists stand, or CI would run no test at all. It exits 1, or 128 plus the signal's number when a signal stopped the recording's auto-replay or its judging: 130 for Ctrl+C, and 129 for a hang-up, such as a closed terminal, that stopped the recording. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. ### The NEXT block -Every `keploy record` and `keploy test` ends with a `NEXT` block, and `KEPLOY_PROVEN=1 keploy status` prints it any time (add `--json` for a machine-readable copy). After the record above, which kept 5 tests and quarantined 2, it looks like this: +Every `keploy record` and `keploy test` ends with a `NEXT` block, and `KEPLOY_PROVEN=1 keploy status` prints it any time (add `--json` for a machine-readable copy). A judging round is never the suite's verdict, so `keploy record`'s block is never green, and it wires no CI: when its verdict is complete, it gives you CI's own command to run once. After the record above, which kept 5 tests and quarantined 2, it looks like this: + +```text +=== KEPLOY NEXT === +phase: recorded +summary: keploy record judged the suite: the tests it kept passed both judging rounds. A judging round is not the suite's verdict: CI's own command is, `KEPLOY_PROVEN=1 keploy test -c './server'`, and it has not run on what keploy record wrote yet. keploy/test-set-0/verdicts.yaml quarantines 2 test(s) of test-set-0 (1 flaky/request_changed, 1 flaky/unstable_order), and CI's keploy test skips them. +ask: Run CI's command once, on what keploy record wrote? Its result is the suite's verdict, and its NEXT block wires CI. +run: KEPLOY_PROVEN=1 keploy test -c './server' +do: Run `KEPLOY_PROVEN=1 keploy test -c './server'` once and follow the NEXT block it ends with. It is the command CI runs: it skips what keploy//verdicts.yaml quarantines, as CI does, and it writes the run's quarantine.yaml, so its result is the suite's verdict, and its NEXT block wires CI once it is green. ... +options: + - run CI's command once: its result is the suite's verdict → KEPLOY_PROVEN=1 keploy test -c './server' + - re-record, once each reason is fixed → KEPLOY_PROVEN=1 keploy record -c './server' --metadata 'name=' +protocol: ... +=================== +``` + +When the confirming round failed tests that judging then quarantined, the `summary` starts `keploy record's confirming round failed what it then quarantined`, and when a set failed on its own, it says so, with why. When judging's verdict failed, for example for a `NOT JUDGED` set or a CI that would be red, the block is `verified_red`, and when CI would run no test, `verified_empty`: fix what they name, then record again. + +#### Run CI's command once {#gate} + +Run the command on the `run:` line where you recorded, in the directory that holds `keploy/`: + +```bash +KEPLOY_PROVEN=1 keploy test -c "./server" +``` + +It runs exactly as your CI will: it skips the quarantined tests, replays the rest against their mocks, and applies every rule CI applies. Your app's dependencies don't need to run, except a [`mock.real`](#mock-real) one. Its result is the suite's verdict. When it's green, its `NEXT` block wires CI: ```text === KEPLOY NEXT === phase: verified_green -summary: All 5 test(s) passed across 1 test-set(s). Suite is green and dependency-free. +summary: 5 test(s) passed across 1 test-set(s), and CI's keploy test is green: it skips the 2 test(s) keploy//verdicts.yaml quarantines (1 flaky/request_changed, 1 flaky/unstable_order), and none of those is verified. caveats: - 2 quarantined test(s) did not run (keploy//verdicts.yaml): this green covers the other tests only. `KEPLOY_PROVEN=1 keploy test --all` runs them; `KEPLOY_PROVEN=1 keploy test --judge` drops the entries of those that now pass. ask: Want me to (a) cover another resource, (b) wire CI so this runs on every PR, or (c) stop here? -do: If wiring CI: the Keploy registry holds each test set's mocks; CI downloads them by the hash in the set's config.yaml (KEPLOY_API_KEY required in CI). Commit exactly the paths on the commit: line (relative to the directory that holds keploy/: run `git add` there, and CI's keploy test there too): ... +do: The quarantined tests are not verified: ... If wiring CI: the Keploy registry holds each test set's mocks; CI downloads them by the hash in the set's config.yaml (KEPLOY_API_KEY required in CI). Commit exactly the paths on the commit: line (relative to the directory that holds keploy/: run `git add` there, and CI's keploy test there too): ... The CI job: check out; install Keploy (`curl --silent -O -L https://keploy.io/ent/install.sh && bash install.sh -isCI`); export KEPLOY_API_KEY from a CI secret; in the directory that holds keploy/, run `KEPLOY_PROVEN=1 keploy test -c './server'` with KEPLOY_INVOKER=ci so a missing session fails fast instead of waiting on a browser login. ... commit: keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml options: - cover another resource → KEPLOY_PROVEN=1 keploy record -c './server' --metadata 'name=' @@ -244,17 +275,19 @@ protocol: ... =================== ``` -The `summary` counts the tests the confirming round ran, so it leaves out the ones that round skipped because the record quarantines them. The `caveats` line says how many those are: a green covers the other tests only. The `JUDGED` line and the `QUARANTINED` lines under it name each one. The `NEXT` block of a later `keploy test`, such as your CI run, has the same caveat. +The `summary` counts the tests the run ran. The `caveats` line says how many quarantined tests it skipped: a green covers the other tests only. The `JUDGED` line and the `QUARANTINED` lines under it name each one, and so does this run's `QUARANTINED, not run` list, which [CI prints too](#ci). When the run fails, its block is `verified_red`, with each failing test: fix what it names before you wire CI. -| Field | What it tells you | -| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (not replayed since it changed, or last replayed without the switch: such a run ignored `verdicts.yaml`, so it's no verdict on the suite), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). | -| `summary` | The last run in one sentence. | -| `failures` | Each failing test, with what differed and a remedy. | -| `caveats` | What a green doesn't cover, such as the quarantined tests the run skipped, a record's confirming round included. Read them before you call the suite green. | -| `do` | The next step in full. After a green record it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | -| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. | -| `options` | Commands for the next step. Every `keploy` command in the block carries the switch, except the `keploy mock` commands, which don't need it. | +| Field | What it tells you | +| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (judged but not yet run as CI runs it, not replayed since it changed, or last replayed without the switch: such a run ignored `verdicts.yaml`, so it's no verdict on the suite), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). Only a `keploy test` with the switch that ran to its end is green or red: `keploy record`'s judging rounds aren't. | +| `summary` | The last run in one sentence, or what `keploy record`'s judging decided. | +| `failures` | Each failing test, with what differed and a remedy. | +| `caveats` | What a green doesn't cover, such as the quarantined tests the run skipped. Read them before you call the suite green. | +| `ask` | The question to answer next. | +| `run` | The command to run next, when there is one: after a record, CI's command. | +| `do` | The next step in full. After a green `keploy test` it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | +| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. Only a green block has it. | +| `options` | Commands for the next step. Every `keploy` command in the block carries the switch, except the `keploy mock` commands, which don't need it. | The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. @@ -262,7 +295,7 @@ When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checko ## Commit the test sets {#commit} -From the directory that holds `keploy/`, `git add` exactly the paths on the `NEXT` block's `commit:` line: +Once [CI's command](#gate) is green, `git add` exactly the paths on the `commit:` line of its `NEXT` block, from the directory that holds `keploy/`: ```bash git add keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml @@ -320,7 +353,7 @@ jobs: The CI run first lists what it skips, and why, with what to do for each reason: ```text -QUARANTINED, not run (2): `keploy test --all` runs them +QUARANTINED, not run (2): `KEPLOY_PROVEN=1 keploy test --all` runs them test-set-0/post-learn-batch-1 flaky/unstable_order HTTP POST /batch to localhost:8096 (no_match) sent an array in another order since 2026-10-20 test-set-0/post-learn-order-1 flaky/request_changed HTTP POST /orders to localhost:8096 (no_match) changed since 2026-10-20 by class: keploy 0 · app-infra 0 · flaky 2 · manual 0 @@ -342,7 +375,7 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run - `UNREADABLE : ...`: Keploy couldn't read the set's `verdicts.yaml`, for example because it holds merge-conflict markers, so it skips none of the set's tests. - `HELD /`: the test ran, and Keploy holds evidence against it, such as a call no mock answered, so the run fails even if the test passed. - `UNATTRIBUTED : , charged to no test`: calls that Keploy couldn't answer or capture, made outside every test, such as by a background poller. The run fails. -- `NOT JUDGED /: , so it has no verdict`: the test has no verdict, for one of the causes in [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). The run fails, and its `VERDICT failed` line gives the remedy. A set Keploy couldn't judge as a whole gets `NOT JUDGED : ...` instead. +- `NOT JUDGED /: , so it has no verdict`: the test has no verdict, for one of the causes in [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). The run fails, the set's `SET` line ends `, not judged`, and the `VERDICT failed` line gives the remedy. A set Keploy couldn't judge as a whole gets `NOT JUDGED : ...` instead. - `ENDED : `: `keploy test` ended the set other than `PASSED`, for example `FAILED` with no failing test. The run fails. - `NOT RUN : the run stopped before it`: none of the set's tests ran. - `REAL /: ( connections, failed)`: the test reached a [`mock.real`](#mock-real) dependency for real. @@ -352,7 +385,7 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run ## Keep the suite healthy -- **When the app changes on purpose,** record the affected flows again. `keploy record` writes them to a new test set and judges it. Then delete the old test cases they replace, so CI doesn't run them; `KEPLOY_PROVEN=1 keploy test --judge` drops their entries (`PRUNED`). +- **When the app changes on purpose,** record the affected flows again. `keploy record` writes them to a new test set and judges it. Then delete the old test cases they replace, so CI doesn't run them; `KEPLOY_PROVEN=1 keploy test --judge` drops their entries (`PRUNED`). Run [CI's command](#gate) once more before you commit, as after your first record. - **After you fix what a quarantine reason names,** record that flow again, as above, or run `KEPLOY_PROVEN=1 keploy test --judge`. It replays the test sets twice, read-only, drops the entries of tests that now pass, and prints a `keploy quarantine` command for each test that fails. It never adds a skip by itself. - **To quarantine a test by hand,** for example a test that fails since a change you made on purpose, until you record it again: diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index dd84a7858..7a26484af 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -69,20 +69,20 @@ Don't add entries by editing the file: an entry whose `rec` doesn't name the tes ## Fields {#fields} -| Field | Meaning | -| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `version` | The file's format. This release reads and writes `1` only. | -| `quarantined` | One entry per quarantined test, keyed by the test case's name in the set, such as `get-orders-3`. | -| `class` | Who has to act: `keploy`, `app-infra`, `flaky` or `manual`. See [Classes and reasons](#reasons). | -| `reason` | Why the test doesn't replay. For `manual`, your own text from `keploy quarantine --reason`, one line of at most 200 bytes. | -| `rec` | The recordings the entry applies to: `sha256:<64 hex digits>`, optionally followed by `@` for a recording published to a Keploy cloud branch, or `none` for a test with no recording. | -| `since` | The UTC date the test entered quarantine. | -| `by` | What wrote the entry: `record` for `keploy record` and `keploy mock record`, `quarantine` for `keploy quarantine`. | -| `keploy` | The Keploy version that wrote the reason. | -| `rounds` | Each judging round's outcome for the test, each `passed`, `failed` or `skipped`: `full` (auto-replay's full round, with nothing skipped), `confirm` (the confirming round, run as CI runs `keploy test`), and `record` (the run against real infrastructure, which only `keploy mock record` has). A test that's about to be quarantined is `skipped` in `confirm`, as CI will skip it. | -| `calls` | Up to five dependency calls behind the reason: `protocol`, `call` (the method and the URL's shape for HTTP and gRPC, the command alone for other protocols), `destination`, and `phase`, the step of mock matching the call failed at (such as `no_match` or `body_mismatch`) or `record` for a call the recording lost. | -| `mocks` | Up to five recorded mocks behind the reason, by `name` and `kind`. | -| `diff` | The paths of the response fields that differed, such as `body.total`. A key that may be a value, such as an email address that a map is keyed by, is written `{}`. | +| Field | Meaning | +| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `version` | The file's format. This release reads and writes `1` only. | +| `quarantined` | One entry per quarantined test, keyed by the test case's name in the set, such as `get-orders-3`. | +| `class` | Who has to act: `keploy`, `app-infra`, `flaky` or `manual`. See [Classes and reasons](#reasons). | +| `reason` | Why the test doesn't replay. For `manual`, your own text from `keploy quarantine --reason`, one line of at most 200 bytes. | +| `rec` | The recordings the entry applies to: `sha256:<64 hex digits>`, optionally followed by `@` for a recording published to a Keploy cloud branch, or `none` for a test with no recording. | +| `since` | The UTC date the test entered quarantine. | +| `by` | What wrote the entry: `record` for `keploy record` and `keploy mock record`, `quarantine` for `keploy quarantine`. | +| `keploy` | The Keploy version that wrote the reason. | +| `rounds` | Each judging round's outcome for the test, each `passed`, `failed`, `skipped`, or `set_failed` (the round's `keploy test` failed the test's set for what no test owns, while the test itself passed): `full` (auto-replay's full round, with nothing skipped), `confirm` (the confirming round, run as CI runs `keploy test`), and `record` (the run against real infrastructure, which only `keploy mock record` has). A test that's about to be quarantined is `skipped` in `confirm`, as CI will skip it. | +| `calls` | Up to five dependency calls behind the reason: `protocol`, `call` (the method and the URL's shape for HTTP and gRPC, the command alone for other protocols), `destination`, and `phase`, the step of mock matching the call failed at (such as `no_match` or `body_mismatch`) or `record` for a call the recording lost. | +| `mocks` | Up to five recorded mocks behind the reason, by `name` and `kind`. | +| `diff` | The paths of the response fields that differed, such as `body.total`. A key that may be a value, such as an email address that a map is keyed by, is written `{}`. | The evidence holds no request or response bodies, query values or headers. A URL keeps only the path segments that read as names, and writes every other one as `{}`: numbers, IDs, UUIDs, hashes, email addresses and tokens with a digit in them. So `/users/alice@example.com/orders/12345` is `/users/{}/orders/{}`. A segment Keploy knows holds a value, such as the place of a variable it learned, is `{}` too. A query keeps its parameters' names, by the same rule, and writes every value as `{}`: `?expand=items&page=2` is `?expand={}&page={}`. Any other segment made only of letters, such as a slug or a lowercase user name, can't be told from a name, so it's kept: `/users/alice/orders/12345` is `/users/alice/orders/{}`. Review the file before you commit it if your URLs carry such values. @@ -92,22 +92,25 @@ Keploy writes each entry's fields on fixed lines, one entry per test, sorted by `keploy test` prints what to do for each class and reason under its `QUARANTINED, not run` list. -Every entry that `keploy record` or `keploy mock record` writes was decided on the build that recorded the test: auto-replay replays each recording against the same app, right after you stop recording. So the `keploy` class means what it says: Keploy has to act. On an app that didn't change, Keploy couldn't answer one of the test's calls from its recording, or couldn't match the calls it recorded for it. Checking your diff finds nothing, because there's no diff, and recording the same traffic again can quarantine a different set of tests rather than none. Report such a test to Keploy, with the entry's `calls:` or `mocks:` evidence and the log of the `keploy record` that wrote it. Keploy's printed remedy for `keploy/unmocked_call` and `keploy/unconsumed_mock` says to re-record the test if the app changed on purpose, and to check the diff otherwise: neither applies to an entry from a record's own judging. +Every entry that `keploy record` or `keploy mock record` writes was decided on the build that recorded the test: auto-replay replays each recording against the same app, right after you stop recording. So for the reasons of one test's calls, `keploy/unmocked_call`, `keploy/unconsumed_mock` and `keploy/mock_missing`, the `keploy` class means what it says: Keploy has to act. On an app that didn't change, Keploy couldn't answer one of the test's calls from its recording, or couldn't match the calls it recorded for it. Checking your diff finds nothing, because there's no diff, and recording the same traffic again can quarantine a different set of tests rather than none. Report such a test to Keploy, with the entry's `calls:` or `mocks:` evidence and the log of the `keploy record` that wrote it. Keploy's printed remedy for `keploy/unmocked_call` and `keploy/unconsumed_mock` says to re-record the test if the app changed on purpose, and to check the diff otherwise: neither applies to an entry from a record's own judging. The two reasons of a whole set, `keploy/set_failed` and `keploy/set_capture_incomplete`, are different: see their rows. To re-record a test, record its flow again with `KEPLOY_PROVEN=1 keploy record` and the command you recorded it with: Keploy writes the new recording to a new test set and judges it there. Then delete the old test case from its set, and `KEPLOY_PROVEN=1 keploy test --judge` drops its entry (`PRUNED`). -| Class and reason | What happened | What to do | -| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `keploy/unmocked_call` | On the build that recorded it, the test made a call Keploy had no recorded answer for, or one call more than it recorded. | Report it to Keploy, as [above](#reasons). One cause is yours: requests that overlapped while you recorded. Then record those flows again, [one request at a time](/docs/running-keploy/proven-flow/#record-your-tests). | -| `keploy/unconsumed_mock` | On the build that recorded it, the test never made a call Keploy recorded for it. | The same as for `keploy/unmocked_call`. | -| `keploy/mock_missing` | The replay of the build that was just recorded was served other mocks than its recording maps. | Report it to Keploy, as [above](#reasons). Keploy's printed remedy is to re-record the test: a new recording is judged afresh, and it isn't sure to be kept. | -| `keploy/capture_incomplete` | Keploy couldn't capture one of the test's calls, such as a TLS handshake it couldn't join or a UDP reply, or the test's recording lost data that no round could judge. | Give the dependency a TCP address Keploy can mock, or list it in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), then re-record. For a recording that lost data, see [Capture-loss evidence](/docs/running-keploy/proven-flow/#recording-lost). | -| `flaky/replay_only` | The test failed on replay with every call answered: the app is nondeterministic (the clock, randomness), or a real dependency's state changed since it was recorded. | Fix that, then re-record. | -| `flaky/flipped` | The test passed in one round and failed in the next: it depends on test order, on another test, or on how the command runs it. | Remove the dependency, then re-record. | -| `flaky/request_changed` | A request field changed between runs in a way Keploy doesn't learn. | Make the field deterministic, then re-record. | -| `flaky/unstable_order` | The app sent an array in another order than it recorded, or an array came back in another order. | Give the array a stable order in the app, then re-record. | -| `app-infra/failed_in_record` | The test failed against real infrastructure when it was recorded, so it has no recording. Only `keploy mock record` writes this reason. | Fix the app or its infrastructure, then record it again. | -| `manual/` | You quarantined the test with `keploy quarantine`. | Fix what your reason names, then remove the entry with `KEPLOY_PROVEN=1 keploy test --judge` once the test passes, or record the test again. | +| Class and reason | What happened | What to do | +| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `keploy/unmocked_call` | On the build that recorded it, the test made a call Keploy had no recorded answer for, or one call more than it recorded. | Report it to Keploy, as [above](#reasons). One cause is yours: requests that overlapped while you recorded. Then record those flows again, [one request at a time](/docs/running-keploy/proven-flow/#record-your-tests). | +| `keploy/unconsumed_mock` | On the build that recorded it, the test never made a call Keploy recorded for it. | The same as for `keploy/unmocked_call`. | +| `keploy/mock_missing` | The replay of the build that was just recorded was served other mocks than its recording maps. | Report it to Keploy, as [above](#reasons). Keploy's printed remedy is to re-record the test: a new recording is judged afresh, and it isn't sure to be kept. | +| `keploy/capture_incomplete` | Keploy couldn't capture one of the test's calls, such as a TLS handshake it couldn't join or a UDP reply, or the test's recording lost data that no round could judge. | Give the dependency a TCP address Keploy can mock, or list it in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), then re-record. For a recording that lost data, see [Capture-loss evidence](/docs/running-keploy/proven-flow/#recording-lost). | +| `keploy/set_failed` | A judging round's `keploy test` failed the test's set for what none of its tests owns: a call your app made before the set's first test, or after its last, that no recording answers, or a call the set's worker scope recorded that your app no longer makes. CI's `keploy test` fails the set for it whichever of its tests run, so every test of the set is quarantined, and CI doesn't start your app for the set. | The entry's `calls:` name what failed the set. When `keploy record` wrote the entry, your app didn't change in between, so something it does outside every test differs from one run to the next, such as a write at start-up that carries the time or a random ID, or a call it makes on some runs only: make that deterministic, then record the set again. That's Keploy's printed remedy. If the calls carry no such value, report it to Keploy, as [above](#reasons). Recording an app that didn't change again quarantines the same tests again. | +| `keploy/set_capture_incomplete` | The same failure of the whole set, while Keploy couldn't capture all that the set did, in a judging round or in its recording. The failure may be what the capture lost, not your app. | Fix what the evidence names first: give a dependency Keploy can't capture a TCP address it can mock, or list it in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real). Then record the set again. If the set still fails on its own with nothing lost, check your diff for a regression. | +| `flaky/replay_only` | The test failed on replay with every call answered: the app is nondeterministic (the clock, randomness), or a real dependency's state changed since it was recorded. | Fix that, then re-record. | +| `flaky/flipped` | The test passed in one round and failed in the next: it depends on test order, on another test, or on how the command runs it. | Remove the dependency, then re-record. | +| `flaky/request_changed` | A request field changed between runs in a way Keploy doesn't learn. | Make the field deterministic, then re-record. | +| `flaky/unstable_order` | The app sent an array in another order than it recorded, or an array came back in another order. | Give the array a stable order in the app, then re-record. | +| `app-infra/real_not_reached` | The test's recording lists a [`mock.real`](/docs/running-keploy/proven-flow/#mock-real) destination that a judging round never reached, and that's all its set failed for. CI's `keploy test` fails the set while a test that lists it runs. | Start the destination where the tests run, CI included, then run `KEPLOY_PROVEN=1 keploy test --judge` to bring the test back. If the test no longer uses it, re-record the test. | +| `app-infra/failed_in_record` | The test failed against real infrastructure when it was recorded, so it has no recording. Only `keploy mock record` writes this reason. | Fix the app or its infrastructure, then record it again. | +| `manual/` | You quarantined the test with `keploy quarantine`. | Fix what your reason names, then remove the entry with `KEPLOY_PROVEN=1 keploy test --judge` once the test passes, or record the test again. | A class or reason this Keploy doesn't know, written by a newer one, is shown as it is, and the entry still applies. @@ -133,7 +136,7 @@ A test can also end with no verdict at all, which is not a quarantine. It happen KEPLOY_PROVEN=1 keploy record -c "./server" ``` - Delete the set first. Recording again doesn't replace it: `keploy record` writes a new test set, such as `test-set-1`, and leaves the set it didn't judge on disk. The `NEXT` block's `commit:` line lists every set on disk, that one too. Committed, it runs in CI with no `verdicts.yaml` (`UNLISTED`): every one of its tests runs, though Keploy judged none of them, so CI can fail on tests that a judged set would have quarantined. For the same reason, don't upload its mocks with `keploy mock upload`, even when the `NEXT` block's `do:` asks for it: that advice is for a set CI replays green. If `rm` is refused, Keploy wrote the files as root: use `sudo rm -rf`. + Delete the set first. Recording again doesn't replace it: `keploy record` writes a new test set, such as `test-set-1`, and leaves the set it didn't judge on disk. [CI's command](/docs/running-keploy/proven-flow/#gate) then replays that set too, with no `verdicts.yaml` (`UNLISTED`): every one of its tests runs, though Keploy judged none of them. The run can fail on them: replayed again, the Unix-socket call above leaves its test and every later test of the run without a verdict, the new set's tests too. When the run passes, its `commit:` line lists that set too. Committed, it runs the same way in CI, so CI can fail on tests that a judged set would have quarantined. For the same reason, don't upload its mocks with `keploy mock upload`, even when the `NEXT` block's `do:` asks for it: that advice is for a set CI replays green. If `rm` is refused, Keploy wrote the files as root: use `sudo rm -rf`. - **In CI.** The tests the second column below names have no verdict. `keploy test` prints a `NOT JUDGED` line for them, and the run fails. Its `VERDICT failed` line says what to do. From 53cc6bc25a62e78cdd8ecb6687b75653837bd3b4 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 09:56:17 +0530 Subject: [PATCH 09/14] docs: commit only the green run's commit: line, and record's exit 0 is not the suite's verdict Review round 2 of keploy/docs#937. - How the flow works, step 6: commit exactly the paths on the commit: line of CI's command's green NEXT block, never a mocks.yaml or a secret.yaml, and not `git add keploy/`. keploy/.gitignore keeps out only /reports/, .verdicts-* and /*/mocks.yaml, so `git add keploy/` also takes secret.yaml and a set Keploy did not judge. - Exit codes: exit 0 means judging found nothing that makes CI red as the lists stand. It is not the suite's verdict: CI's keploy test decides by its own rules and replays every set on disk, so a script takes only a green `KEPLOY_PROVEN=1 keploy test` without -t. - Keep the suite healthy: run CI's command again, without -t, after a record, keploy quarantine, keploy test --judge, keploy mock upload or a git pull that changed a set's verdicts.yaml, mappings.yaml or config.yaml. Until then status reports the suite as recorded (r4's 13a1d52e9: listsChanged, partialRun). After keploy mock upload, run CI's command again rather than only keploy status. - --disableAutoReplay: its NEXT block is the legacy recorded one that asks to mark noise by hand (next.go's Recorded branch); in this flow, delete the set and record it again without the flag. - The commands printed without the switch now include the --must-pass refusal's keploy quarantine and record's incomplete-verdict next_step. - A TLS handshake hung up on, for a Go app: Go's crypto/tls sends bad_certificate when it rejects a certificate, which Keploy counts as tls_handshake_failed, not tls_handshake_abandoned; a Go client hangs up without a word only when it gives up (a cancelled context closes the connection with no alert). Keploy's Node.js and curl remedies are for other programs the app runs. Signed-off-by: slayerjain --- .../version-4.0.0/running-keploy/proven-flow.md | 17 ++++++++++------- .../running-keploy/verdicts-yaml.md | 9 ++++++++- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 6b159441e..02ed7004f 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -45,7 +45,7 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i 3. **Replay as CI will (the confirming round).** Keploy runs `keploy test` the way your CI will run it, skipping the tests it's about to quarantine. 4. **Judge.** A test that passed both rounds is kept. A test that didn't is written to `keploy//verdicts.yaml` with a class and a reason. Keploy prints a `JUDGED` line for each test set, then a `NEXT` block that gives CI's command, `KEPLOY_PROVEN=1 keploy test`. 5. **Run CI's command once.** You run that command on your machine. It skips the quarantined tests and replays the rest, exactly as CI will. Its result is the suite's verdict, and when it's green, its `NEXT` block says what to commit and how to wire CI. -6. **Run in CI.** You commit the `keploy` folder without its `mocks.yaml` files. Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. +6. **Run in CI.** You commit exactly the paths on the `commit:` line of that green `NEXT` block, never a `mocks.yaml` or a `secret.yaml`: not `git add keploy/`, which can also take a test set that Keploy didn't judge. See [Commit the test sets](#commit). Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. ## Requirements {#requirements} @@ -71,7 +71,7 @@ Then turn the flow on for your shell: export KEPLOY_PROVEN=1 ``` -Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set. The commands Keploy prints for you to run carry `KEPLOY_PROVEN=1`: every `keploy` command in the `NEXT` block, except the `keploy mock` commands, which don't need it; the `keploy quarantine` line under `QUARANTINE?`; and the commands in each reason's remedy. A few log lines name a command in their `next_step` without it, so set the switch on every Keploy command of this flow all the same. The examples on this page set it on each command, so you can copy any one of them on its own. +Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set. The commands Keploy prints for you to run carry `KEPLOY_PROVEN=1`: every `keploy` command in the `NEXT` block, except the `keploy mock` commands, which don't need it; the `keploy quarantine` line under `QUARANTINE?`; and the commands in each reason's remedy. A few log lines and errors name a command without it, such as the `next_step` of `keploy record`'s incomplete-verdict log line and the `keploy quarantine` command in the `--must-pass` refusal, so set the switch on every Keploy command of this flow all the same. The examples on this page set it on each command, so you can copy any one of them on its own. ## Record your tests @@ -113,7 +113,7 @@ To stop recording, press **Ctrl+C once**, then wait. Keploy says: auto-replay: recording stopped — now replaying what was just recorded to check for noise. Press Ctrl+C again to skip and exit. Under KEPLOY_PROVEN=1 a skip leaves keploy record with no verdict on what it recorded, so it does not exit 0: a recording that did not fail on its own exits 128+N for the signal (130 for Ctrl+C). ``` -Auto-replay runs both rounds and prints its verdict. Don't press Ctrl+C a second time: that skips auto-replay, so nothing is judged, no `verdicts.yaml` is written, and `keploy record` exits 130. To record without a verdict on purpose, pass `--disableAutoReplay`, or set `record.disableAutoReplay: true` in `keploy.yml`. +Auto-replay runs both rounds and prints its verdict. Don't press Ctrl+C a second time: that skips auto-replay, so nothing is judged, no `verdicts.yaml` is written, and `keploy record` exits 130. To record without a verdict on purpose, pass `--disableAutoReplay`, or set `record.disableAutoReplay: true` in `keploy.yml`. Such a recording isn't judged, and no noise is learned for it. Its `NEXT` block is then the one for a recording that nothing replayed: it asks you to run `keploy test` twice and add each field that differs to that test case's noise list. Don't do that in this flow, where auto-replay learns the noise. To have the set judged, delete its directory, `keploy/`, and record it again without the flag. :::caution Don't pass `--delay` to keploy record `--delay` is a `keploy test` flag. `keploy record` rejects it as an unknown flag and records nothing: its auto-replay's delay is `--auto-replay-delay`. Under the switch, auto-replay and `keploy test` wait for your app to be ready before they send the first test: it listens on the port the test connects to, or the health URL you gave answers (`--auto-replay-health-url` on `keploy record`, `--health-url` on `keploy test`). Then they wait their delay, 15 seconds for auto-replay (`--auto-replay-delay`) and 5 for `keploy test` (`--delay`), and send it. @@ -138,7 +138,7 @@ KEPLOY_PROVEN=1 keploy record -c "./server" --low-latency `--low-latency` is a `keploy record` flag only. With it, Keploy runs itself with `sudo` from the start, even for a native app. -A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in has no verdict, so when you record, its whole test set is `NOT JUDGED`, as for a Unix socket. The remedy is to make that client trust Keploy's CA. See [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). +A client that rejects Keploy's certificate without a word, or hangs up just after its TLS handshake, leaves Keploy unable to tell whether a call was lost. The test it happened in has no verdict, so when you record, its whole test set is `NOT JUDGED`, as for a Unix socket. A Go client doesn't reject a certificate that way: it sends an alert. It does hang up without a word when it gives up, for example on a cancelled context. See [A TLS handshake hung up on](/docs/running-keploy/verdicts-yaml/#tls-hung-up) for what to do. ### Reach a dependency for real {#mock-real} @@ -226,7 +226,9 @@ Other lines a test set can get: ### Exit codes -`keploy record` exits 0 when it judged every test set it recorded, wrote what it decided, and CI would be green as the lists stand, even if it quarantined tests. Otherwise it prints `keploy record's verdict is incomplete: ; it exits `, so that a script can't take an unjudged recording, or one that CI would fail, for a judged one. That happens when auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, a second Ctrl+C stopped it, CI would be red as the lists stand, or CI would run no test at all. It exits 1, or 128 plus the signal's number when a signal stopped the recording's auto-replay or its judging: 130 for Ctrl+C, and 129 for a hang-up, such as a closed terminal, that stopped the recording. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. +`keploy record` exits 0 when it judged every test set it recorded, wrote what it decided, and its judging found nothing that makes CI red as the lists stand, even if it quarantined tests. Exit 0 isn't the suite's verdict, and it doesn't mean CI will be green. CI's `keploy test` decides by its own rules, not by judging's, and it replays every test set on disk, including one this record didn't judge, such as a set left from an earlier recording. So a script should take only a green [`KEPLOY_PROVEN=1 keploy test`](#gate), run over every test set (without `-t`), as the suite's verdict. + +In every other case, `keploy record` prints `keploy record's verdict is incomplete: ; it exits `, so that a script can't take an unjudged recording, or one that its judging found CI would fail, for a judged one. That happens when auto-replay judged nothing, a test set was `NOT JUDGED` or `NOT WRITTEN`, a second Ctrl+C stopped it, judging found CI would be red as the lists stand, or CI would run no test at all. It exits 1, or 128 plus the signal's number when a signal stopped the recording's auto-replay or its judging: 130 for Ctrl+C, and 129 for a hang-up, such as a closed terminal, that stopped the recording. When the recording fails on its own, for example because your app exits with an error, `keploy record` exits with that failure's code, and says so on the same line. With `--disableAutoReplay`, nothing is judged, and the exit code is the recording's own. ### The NEXT block @@ -291,7 +293,7 @@ The `summary` counts the tests the run ran. The `caveats` line says how many qua The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. -When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. For a set that judging kept green, upload them with the `keploy mock upload -t ` command it prints, then run `KEPLOY_PROVEN=1 keploy status` again. The block says this of any set on disk whose mocks the registry doesn't hold, including the ones `keploy record` didn't upload on purpose: a set it printed `NOT JUDGED` for, or one judging counted red (`CI would be red`, or a `NOTE` that CI runs none of it). Don't upload or commit those: delete the set's directory, `keploy/`, and record again. For a `NOT WRITTEN` set, settle its `verdicts.yaml` first, as its line says. +When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. For a set that judging kept green, upload them with the `keploy mock upload -t ` command it prints, then run [CI's command](#gate) once more: the upload writes the set's `config.yaml`, which CI reads. The block says this of any set on disk whose mocks the registry doesn't hold, including the ones `keploy record` didn't upload on purpose: a set it printed `NOT JUDGED` for, or one judging counted red (`CI would be red`, or a `NOTE` that CI runs none of it). Don't upload or commit those: delete the set's directory, `keploy/`, and record again. For a `NOT WRITTEN` set, settle its `verdicts.yaml` first, as its line says. ## Commit the test sets {#commit} @@ -395,7 +397,8 @@ SET test-set-0: 5 ran, 5 passed, 0 failed, 2 quarantined not run The reason is one line of at most 200 bytes. The entry's class is `manual`, and it applies until the test's recording changes. Quarantine only a test that fails: `KEPLOY_PROVEN=1 keploy test --judge` drops the entry of a test that passes, a manual one too. -- **To run the quarantined tests too,** for example to check a fix, run `KEPLOY_PROVEN=1 keploy test --all`. +- **After any change to what CI reads, run [CI's command](#gate) once more before you commit,** with no `-t`, so that it replays every test set as CI does. That means after a record, a `keploy quarantine`, a `keploy test --judge`, a `keploy mock upload`, or a `git pull` that changed a set's `verdicts.yaml`, `mappings.yaml` or `config.yaml`. Until you do, `keploy status` doesn't call the suite green. It reports a run made before those files last changed as `recorded`, whether that run was green or red, and so it does a green run of only some test sets (`-t`). Each time, its `NEXT` block gives CI's command. Commit what the green run's `commit:` line names. +- **To run the quarantined tests too,** for example to check a fix, run `KEPLOY_PROVEN=1 keploy test --all`. It isn't CI's command, so run CI's command once more before you commit. - **The suite stays in this flow** while any of its test sets holds a `verdicts.yaml`. Deleting every set's `verdicts.yaml` takes it out, and drops each quarantine and its reason with it, so CI would run those tests again. Do that only to leave the flow on purpose, never to make a run pass. - **When CI goes red,** run `KEPLOY_PROVEN=1 keploy status` and follow its `NEXT` block. A call the app now makes differently, or no longer makes, is what a regression looks like, so check the code change before you record again. diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index 7a26484af..5c2d735f8 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -145,7 +145,7 @@ A test can also end with no verdict at all, which is not a quarantine. It happen | Called a dependency over a Unix socket, which Keploy doesn't support. | That test and every later one. | Reach the dependency over a TCP address, mocked or listed in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real). | | Ran after Keploy's report of Unix-socket calls stopped, so whether a test made one can't be told. | That test and every later one. | Report it to Keploy, with the agent's log. Nothing you change in your app ends it. | | Ran while Keploy couldn't read a process's PID namespace, so whether its calls were your app's can't be told. | That test and every later one. | Report it to Keploy, with the agent's log, which names the process. | -| Hung up on its TLS handshake with Keploy without a word, which a client that doesn't trust Keploy's CA can do. | That test. | Make the client trust Keploy's CA, for example with `NODE_EXTRA_CA_CERTS` for Node.js, or `--cacert` for the curl command line. | +| Hung up on its TLS handshake with Keploy without a word, which a client that doesn't trust Keploy's CA can do, and so can one that gave up. | That test. | See [A TLS handshake hung up on](#tls-hung-up). | | Sent data to a server-first protocol Keploy has no parser for, such as SMTP, FTP, POP3, IMAP or NATS, which it relays. | That test. | List the destination in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), to reach it for real. | | Connected to a destination that refused the connection at once, such as a Redis error or a MySQL error packet. | That test. | Find why the destination refused your app's connections, such as its connection limit or protected mode. | | Called a container whose configuration the Docker daemon didn't give Keploy, so the call's container can't be told. | That test. | Make `docker inspect` of the container answer: another operation on it, such as a stop's grace period or a hung `exec`, holds it. | @@ -153,6 +153,13 @@ A test can also end with no verdict at all, which is not a quarantine. It happen A call to something Keploy can't capture that your app makes outside every test, for example while it starts, or after the last test, leaves the whole test set without a verdict, in CI too. +### A TLS handshake hung up on {#tls-hung-up} + +A client that closes its TLS handshake with Keploy without a word may have rejected Keploy's certificate, or may have given up. Nothing on the wire tells the two apart, so Keploy can't tell whether a call was lost. + +- **A Go client.** Go's `crypto/tls` doesn't reject a certificate without a word: it sends an alert, which Keploy counts as a failed handshake, a call lost, and not as this cause: auto-replay judges that test as it judges any test whose recording lost a call (see [Capture-loss evidence](/docs/running-keploy/proven-flow/#recording-lost)). A Go client hangs up without a word when it gives up, for example when its context is cancelled or its deadline passes during the handshake. Find why it gave up, then delete the set and record it again, [as above](#not-judged). +- **Another program your app runs,** such as `curl`. Make it trust Keploy's CA. Keploy's message names the usual ways: `NODE_EXTRA_CA_CERTS` for Node.js, and `--cacert`, `CURL_CA_BUNDLE` or `SSL_CERT_FILE` for the curl command line. + ## Resolve a merge conflict {#merge-conflict} When two branches change the same test's entry, or add entries next to each other, Git can leave conflict markers in the file, and Keploy then skips none of the set's tests. Run `git checkout --conflict=diff3 keploy//verdicts.yaml` to see both sides' entries whole. Keep both sides' entries, and for a test both sides list, keep every `rec` item of both. Then commit the file. From 6f73a5edd79c925a096ef7a53a6455f45a263e94 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 10:42:54 +0530 Subject: [PATCH 10/14] docs: the commit: line names every set on disk, and record takes no --delay anywhere Review round 2 of keploy/docs#937. - How the flow works, step 6, and Commit the test sets: the commit: line of the green NEXT block names the files of every test set on disk (gatherProvenCI walks Signals.TestSets), judged or not, because CI's keploy test replays every set the checkout holds. So delete a set Keploy did not judge before running CI's command. The reason not to `git add keploy/` is that it takes every file keploy/.gitignore does not keep out, which can include a set's secret.yaml (keploy test ignores /*/secret.yaml only once it replays a set whose config names secrets). - After keploy mock upload: say what the printed remedy says (run `KEPLOY_PROVEN=1 keploy status` again, provenci.go uploadFirst), and what status then does: the upload writes config.yaml after the run, so status is recorded and gives CI's command (t2-next-ci r4's listsChanged). - The capture-loss caution: the agent logs `agent: recording complete`, with tcs_suppressed_total, at INFO unless a quarter or more of the session's requests were left out (routes/record.go massSuppressed), so read the count even when no warning shows. - keploy update moves from the note above the install section to the install section, for a machine that already has Keploy. - verdicts.yaml example: `keploy: ` instead of a release that does not exist, with a line that says so. - URL shaping (models.PathShape): letters joined by - _ . ~ : are kept too (hello-world); a segment with a digit other than an API version is {}; the few letter segments that are {} read as hex, as a random token in both cases, or are very long. - server/{linux,macos,windows}/installation.md: the docker record line drops `--delay 10`. keploy record defines no --delay (OSS registers it under test only, cli/provider/cmd.go), so the command exited 8 with `unknown flag: --delay`. Signed-off-by: slayerjain --- .../version-4.0.0/running-keploy/proven-flow.md | 12 ++++++------ .../version-4.0.0/running-keploy/verdicts-yaml.md | 8 +++++--- .../version-4.0.0/server/linux/installation.md | 2 +- .../version-4.0.0/server/macos/installation.md | 2 +- .../version-4.0.0/server/windows/installation.md | 2 +- 5 files changed, 14 insertions(+), 12 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 02ed7004f..8e03fabca 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -24,8 +24,6 @@ With `KEPLOY_PROVEN=1`, `keploy record` does more than save tests. When you stop :::note What this release covers The flow is for Go applications on Linux: amd64 with kernel 5.10 or later, and arm64 with kernel 6.6 or later. Calls over Unix sockets aren't supported. See [Requirements](#requirements). - -Run `keploy update` before you start, so that you have the latest Keploy Enterprise release. ::: ## When to use it @@ -45,7 +43,7 @@ Without `KEPLOY_PROVEN=1`, `keploy record` and `keploy test` work as described i 3. **Replay as CI will (the confirming round).** Keploy runs `keploy test` the way your CI will run it, skipping the tests it's about to quarantine. 4. **Judge.** A test that passed both rounds is kept. A test that didn't is written to `keploy//verdicts.yaml` with a class and a reason. Keploy prints a `JUDGED` line for each test set, then a `NEXT` block that gives CI's command, `KEPLOY_PROVEN=1 keploy test`. 5. **Run CI's command once.** You run that command on your machine. It skips the quarantined tests and replays the rest, exactly as CI will. Its result is the suite's verdict, and when it's green, its `NEXT` block says what to commit and how to wire CI. -6. **Run in CI.** You commit exactly the paths on the `commit:` line of that green `NEXT` block, never a `mocks.yaml` or a `secret.yaml`: not `git add keploy/`, which can also take a test set that Keploy didn't judge. See [Commit the test sets](#commit). Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. +6. **Run in CI.** You commit exactly the paths on the `commit:` line of that green `NEXT` block. That line never names a `mocks.yaml` or a `secret.yaml`, but it does name every test set on disk, judged or not, so delete a set you don't want in CI before you run CI's command. Don't run `git add keploy/` instead: it takes every file under `keploy/` that `keploy/.gitignore` doesn't keep out, which can include a set's `secret.yaml`. See [Commit the test sets](#commit). Keploy puts the mocks of each test set that CI would replay green in the Keploy registry, and `KEPLOY_PROVEN=1 keploy test` in CI downloads them, skips the quarantined tests, and replays the rest. ## Requirements {#requirements} @@ -63,7 +61,7 @@ curl --silent -O -L https://keploy.io/ent/install.sh && source install.sh keploy login ``` -`https://keploy.io/install.sh` hands over to the same installer, so either one installs the same Keploy. +`https://keploy.io/install.sh` hands over to the same installer, so either one installs the same Keploy. If Keploy is already installed on your machine, run `keploy update` instead, so that you have the latest Keploy Enterprise release. Then turn the flow on for your shell: @@ -293,7 +291,7 @@ The `summary` counts the tests the run ran. The `caveats` line says how many qua The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. -When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. For a set that judging kept green, upload them with the `keploy mock upload -t ` command it prints, then run [CI's command](#gate) once more: the upload writes the set's `config.yaml`, which CI reads. The block says this of any set on disk whose mocks the registry doesn't hold, including the ones `keploy record` didn't upload on purpose: a set it printed `NOT JUDGED` for, or one judging counted red (`CI would be red`, or a `NOTE` that CI runs none of it). Don't upload or commit those: delete the set's directory, `keploy/`, and record again. For a `NOT WRITTEN` set, settle its `verdicts.yaml` first, as its line says. +When `do:` says the Keploy registry doesn't hold a test set's mocks, a CI checkout would have no mocks for that set and its tests would fail. For a set that judging kept green, upload them with the `keploy mock upload -t ` command it prints, then run `KEPLOY_PROVEN=1 keploy status` again, as `do:` says. The upload writes the set's `config.yaml`, which CI reads, after the run, so `keploy status` reports the suite as `recorded` and gives [CI's command](#gate) to run once more: commit what that run's green block names. The block says this of any set on disk whose mocks the registry doesn't hold, including the ones `keploy record` didn't upload on purpose: a set it printed `NOT JUDGED` for, or one judging counted red (`CI would be red`, or a `NOTE` that CI runs none of it). Don't upload or commit those: delete the set's directory, `keploy/`, and record again. For a `NOT WRITTEN` set, settle its `verdicts.yaml` first, as its line says. ## Commit the test sets {#commit} @@ -303,6 +301,8 @@ Once [CI's command](#gate) is green, `git add` exactly the paths on the `commit: git add keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml ``` +The `commit:` line names the files of every test set on disk, not only the sets Keploy judged, because CI's `keploy test` replays every set the checkout holds. A green run replayed each of them, but a set that Keploy didn't judge, such as one left on disk by a `NOT JUDGED` record or recorded with `--disableAutoReplay`, has no `verdicts.yaml` to skip what doesn't replay. Delete such a set, `keploy/`, and run CI's command once more before you commit: see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). + | Path | What it is | Commit it? | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `keploy.yml` | Keploy's settings for your app, such as its command and `mock.real`. | Yes | @@ -411,7 +411,7 @@ Sometimes Keploy can't capture all of what your app's connections carry while it - A test that lost data and that no round could judge is quarantined as `keploy/capture_incomplete`, never kept. :::caution A loss can cost you tests, and land on another test -When a capture loss, or memory pressure, overlaps a request, Keploy's agent can leave that request out of the recording altogether, so that it never saves a test whose mocks are incomplete. That request then has no test case, so nothing judges it, and its loss is charged to another test of the set. That test may replay and be kept: its `RECORDING LOST` line then names a test that didn't lose what it needs, and says it's kept. The agent counts the requests it left out as `tcs_suppressed_total` on its `agent: recording complete` line, which it logs at `WARN` level when they're a quarter or more of the session's requests. So when a `RECORDING LOST` line appears, compare the test cases on disk with the requests you sent, and record the missing ones again. A loss of bytes that were never in your app's memory happens in low-latency mode only, so record those flows in proxy mode. +When a capture loss, or memory pressure, overlaps a request, Keploy's agent can leave that request out of the recording altogether, so that it never saves a test whose mocks are incomplete. That request then has no test case, so nothing judges it, and its loss is charged to another test of the set. That test may replay and be kept: its `RECORDING LOST` line then names a test that didn't lose what it needs, and says it's kept. The agent counts the requests it left out as `tcs_suppressed_total` on its `agent: recording complete` line. It logs that line at `INFO` level, and at `WARN` only when they're a quarter or more of the session's requests, so a few left-out requests raise no warning. So when a `RECORDING LOST` line appears, read that count, compare the test cases on disk with the requests you sent, and record the missing ones again. A loss of bytes that were never in your app's memory happens in low-latency mode only, so record those flows in proxy mode. ::: Under the `JUDGED` line, Keploy prints a `RECORDING LOST` line for each test whose recording lost data, with what was lost and what Keploy decided: diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index 5c2d735f8..f10acf72e 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -45,19 +45,21 @@ quarantined: rec: [sha256:7c01d2e911111111111111111111111111111111111111111111111111111111] since: 2026-10-20 by: quarantine - keploy: 3.8.80 + keploy: post-learn-batch-1: class: flaky reason: unstable_order rec: [sha256:41aa0c3e00000000000000000000000000000000000000000000000000000000] since: 2026-10-20 by: record - keploy: 3.8.80 + keploy: rounds: {full: failed, confirm: skipped} calls: - {protocol: HTTP, call: POST /batch, destination: localhost:8096, phase: no_match} ``` +In a real file, `keploy` holds the version of the Keploy release that wrote the entry where this example says ``, and each `rec` hash is your recording's own. + ## When an entry skips a test {#rules} - **An entry applies only to the recording it names.** `rec` lists the hashes of the test's recordings that the entry quarantines: the hash of the test case file and the request bodies stored beside it in `assets/`, not of its mocks. When Keploy itself rewrites the test case, for example to add noise it learned, it moves the entry onto the new hash. Any other change to the test case, such as a hand edit or a merge that brought another version of it, makes the entry stale: `keploy test` prints `STALE` for it and runs the test. @@ -84,7 +86,7 @@ Don't add entries by editing the file: an entry whose `rec` doesn't name the tes | `mocks` | Up to five recorded mocks behind the reason, by `name` and `kind`. | | `diff` | The paths of the response fields that differed, such as `body.total`. A key that may be a value, such as an email address that a map is keyed by, is written `{}`. | -The evidence holds no request or response bodies, query values or headers. A URL keeps only the path segments that read as names, and writes every other one as `{}`: numbers, IDs, UUIDs, hashes, email addresses and tokens with a digit in them. So `/users/alice@example.com/orders/12345` is `/users/{}/orders/{}`. A segment Keploy knows holds a value, such as the place of a variable it learned, is `{}` too. A query keeps its parameters' names, by the same rule, and writes every value as `{}`: `?expand=items&page=2` is `?expand={}&page={}`. Any other segment made only of letters, such as a slug or a lowercase user name, can't be told from a name, so it's kept: `/users/alice/orders/12345` is `/users/alice/orders/{}`. Review the file before you commit it if your URLs carry such values. +The evidence holds no request or response bodies, query values or headers. A URL keeps only the path segments that read as names, and writes every other one as `{}`: numbers, IDs, UUIDs, hashes, email addresses, and any segment with a digit in it other than an API version such as `v1`, so `oauth2` and `acct_42` are `{}` too. So `/users/alice@example.com/orders/12345` is `/users/{}/orders/{}`. A segment Keploy knows holds a value, such as the place of a variable it learned, is `{}` too. A query keeps its parameters' names, by the same rule, and writes every value as `{}`: `?expand=items&page=2` is `?expand={}&page={}`. Any other segment of letters can't be told from a name, so it's kept. That includes letters joined by `-`, `_`, `.`, `~` or `:`, such as a slug like `hello-world`, and a lowercase user name: `/users/alice/orders/12345` is `/users/alice/orders/{}`. The few segments of letters that are `{}` hold a part that reads as a hash (eight or more of the letters `a` to `f` alone, such as `deadbeef`), a part in both cases that doesn't read as camelCase words (a random token such as `NffrFeUfNVaHib`), or are very long. Review the file before you commit it if your URLs carry such values. Keploy writes each entry's fields on fixed lines, one entry per test, sorted by test name, so changes to tests that are far apart in the file merge in Git without a conflict. Two branches that change the same test's entry, or add entries next to each other, can conflict: see [Resolve a merge conflict](#merge-conflict). diff --git a/versioned_docs/version-4.0.0/server/linux/installation.md b/versioned_docs/version-4.0.0/server/linux/installation.md index d3825f48b..83c2eb90f 100644 --- a/versioned_docs/version-4.0.0/server/linux/installation.md +++ b/versioned_docs/version-4.0.0/server/linux/installation.md @@ -93,7 +93,7 @@ docker network create keploy-network Now, we will record the testcases. ```shell -keploy record -c "docker run -p : --name --network keploy-network --rm " --containerName "" --delay 10 +keploy record -c "docker run -p : --name --network keploy-network --rm " --containerName "" ``` #### Run the Testcases diff --git a/versioned_docs/version-4.0.0/server/macos/installation.md b/versioned_docs/version-4.0.0/server/macos/installation.md index 93554f878..921f4100c 100644 --- a/versioned_docs/version-4.0.0/server/macos/installation.md +++ b/versioned_docs/version-4.0.0/server/macos/installation.md @@ -153,7 +153,7 @@ Here are few points to consider before recording! Utilize keploy to capture testcases. **Execute** the following command within your application's **root directory**. ```shell -keploy record -c "docker run -p : --name --network keploy-network --rm " --containerName "" --delay 10 +keploy record -c "docker run -p : --name --network keploy-network --rm " --containerName "" ``` Make API Calls using Postman, or cURL commands. diff --git a/versioned_docs/version-4.0.0/server/windows/installation.md b/versioned_docs/version-4.0.0/server/windows/installation.md index 196a9373e..e705e7f3b 100644 --- a/versioned_docs/version-4.0.0/server/windows/installation.md +++ b/versioned_docs/version-4.0.0/server/windows/installation.md @@ -145,7 +145,7 @@ docker network create keploy-network Now, we record the testcases. ```shell -keploy record -c "docker run -p : --name --network keploy-network --rm " --containerName "" --delay 10 +keploy record -c "docker run -p : --name --network keploy-network --rm " --containerName "" ``` #### Run the Test Mode From 80f8530878135471ee430e67970e7653d46d0b3b Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 10:46:56 +0530 Subject: [PATCH 11/14] docs: a NEXT block is green only over every set as the lists stand The phase row of the NEXT block's field table said only a run "to its end" is green or red. Under t2-next-ci r4's 13a1d52e9, status takes a gated run that replayed only some sets (-t, Signals.partialRun), or that a set's verdicts.yaml, mappings or config postdates (Signals. listsChanged), for recorded, and a run under the switch that stopped before its end (stoppedShort) for red. The row now lists those recorded cases and says which run is green. Commit the test sets: a set Keploy did not judge was passed once by the green run, but never replayed in both judging rounds, so nothing found which of its tests do not replay: say that, not that the run did not replay it. Signed-off-by: slayerjain --- .../running-keploy/proven-flow.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 8e03fabca..eb67654d9 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -277,17 +277,17 @@ protocol: ... The `summary` counts the tests the run ran. The `caveats` line says how many quarantined tests it skipped: a green covers the other tests only. The `JUDGED` line and the `QUARANTINED` lines under it name each one, and so does this run's `QUARANTINED, not run` list, which [CI prints too](#ci). When the run fails, its block is `verified_red`, with each failing test: fix what it names before you wire CI. -| Field | What it tells you | -| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (judged but not yet run as CI runs it, not replayed since it changed, or last replayed without the switch: such a run ignored `verdicts.yaml`, so it's no verdict on the suite), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). Only a `keploy test` with the switch that ran to its end is green or red: `keploy record`'s judging rounds aren't. | -| `summary` | The last run in one sentence, or what `keploy record`'s judging decided. | -| `failures` | Each failing test, with what differed and a remedy. | -| `caveats` | What a green doesn't cover, such as the quarantined tests the run skipped. Read them before you call the suite green. | -| `ask` | The question to answer next. | -| `run` | The command to run next, when there is one: after a record, CI's command. | -| `do` | The next step in full. After a green `keploy test` it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | -| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. Only a green block has it. | -| `options` | Commands for the next step. Every `keploy` command in the block carries the switch, except the `keploy mock` commands, which don't need it. | +| Field | What it tells you | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `phase` | Where the suite is: `uninitialized` (no test set in `keploy/` yet), `recorded` (judged but not yet run as CI runs it; not replayed since it changed; last replayed without the switch, a run that ignored `verdicts.yaml`; last replayed green over only some test sets, with `-t`; or last replayed before a set's `verdicts.yaml`, mappings or config changed), `verified_green`, `verified_red`, `verified_empty` (the last run scored nothing, so it isn't green), or `ci_wired` (green, with a `.github/workflows/keploy.yml` beside `keploy/`). Only a `keploy test` with the switch, run to its end over every test set as the lists stand, is green: `keploy record`'s judging rounds aren't. A run with the switch that stopped before its end is red. | +| `summary` | The last run in one sentence, or what `keploy record`'s judging decided. | +| `failures` | Each failing test, with what differed and a remedy. | +| `caveats` | What a green doesn't cover, such as the quarantined tests the run skipped. Read them before you call the suite green. | +| `ask` | The question to answer next. | +| `run` | The command to run next, when there is one: after a record, CI's command. | +| `do` | The next step in full. After a green `keploy test` it's the CI recipe: what to commit, how to install Keploy, and the `keploy test` command. | +| `commit` | Exactly the paths to `git add`, relative to the directory that holds `keploy/`. Only a green block has it. | +| `options` | Commands for the next step. Every `keploy` command in the block carries the switch, except the `keploy mock` commands, which don't need it. | The commands the block prints leave out `--container-name`. When `keploy record` creates `keploy.yml`, it saves your `-c` command and `--container-name` there, and `keploy test` reads them from the committed file. If your `keploy.yml` existed before you recorded, pass the same `-c` and `--container-name` to `keploy test` yourself. @@ -301,7 +301,7 @@ Once [CI's command](#gate) is green, `git add` exactly the paths on the `commit: git add keploy.yml keploy/.gitignore keploy/test-set-0/tests keploy/test-set-0/mappings.yaml keploy/test-set-0/verdicts.yaml keploy/test-set-0/config.yaml ``` -The `commit:` line names the files of every test set on disk, not only the sets Keploy judged, because CI's `keploy test` replays every set the checkout holds. A green run replayed each of them, but a set that Keploy didn't judge, such as one left on disk by a `NOT JUDGED` record or recorded with `--disableAutoReplay`, has no `verdicts.yaml` to skip what doesn't replay. Delete such a set, `keploy/`, and run CI's command once more before you commit: see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). +The `commit:` line names the files of every test set on disk, not only the sets Keploy judged, because CI's `keploy test` replays every set the checkout holds. A green run passed each of them once. But a set that Keploy didn't judge, such as one left on disk by a `NOT JUDGED` record or recorded with `--disableAutoReplay`, was never replayed in both judging rounds, so nothing found which of its tests don't replay, and it has no `verdicts.yaml` to skip them: CI can fail on them later. Delete such a set, `keploy/`, and run CI's command once more before you commit: see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged). | Path | What it is | Commit it? | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | From ea773950b082d77f7c162543ba9df5ce7ad04285 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 10:47:18 +0530 Subject: [PATCH 12/14] docs: record's NEXT is recorded when it quarantined all it judged beside a set CI runs On t2-next-ci r4's 13a1d52e9, judgingRunsNothing beside another test set that CI runs gives record's NEXT block the recorded phase, with CI's command and a summary that none of the tests it judged is verified, while keploy record keeps its own exit code. Only a project in which CI would run no test at all gets verified_empty (JudgingReport. NoTestRunsInCI). Say both. Signed-off-by: slayerjain --- versioned_docs/version-4.0.0/running-keploy/proven-flow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index eb67654d9..94f034d37 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -246,7 +246,7 @@ protocol: ... =================== ``` -When the confirming round failed tests that judging then quarantined, the `summary` starts `keploy record's confirming round failed what it then quarantined`, and when a set failed on its own, it says so, with why. When judging's verdict failed, for example for a `NOT JUDGED` set or a CI that would be red, the block is `verified_red`, and when CI would run no test, `verified_empty`: fix what they name, then record again. +When the confirming round failed tests that judging then quarantined, the `summary` starts `keploy record's confirming round failed what it then quarantined`, and when a set failed on its own, it says so, with why. When judging's verdict failed, for example for a `NOT JUDGED` set or a CI that would be red, the block is `verified_red`, and when CI would run no test at all, `verified_empty`: fix what they name, then record again. When judging quarantined every test it judged but another test set of the project has a test CI runs, `keploy record` keeps its own exit code and its block is `recorded`: its `summary` says that none of the tests it judged is verified, and it gives CI's command, which runs the other sets' tests. #### Run CI's command once {#gate} From e74048a8b4198ed3283d20298074a049486623a7 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 13:12:44 +0530 Subject: [PATCH 13/14] docs: a keploy-class quarantine can be a clock-only call; delete a NOT JUDGED set first Review round 2 of keploy/docs#937. - verdicts.yaml, Classes and reasons: a keploy/unmocked_call or keploy/unconsumed_mock entry from a record's own judging is not only Keploy's to act on. Record's full round replays each test 61.5 s past its recorded instant (record_judge.go), and Classify charges any miss in that round to keploy/unmocked_call, whatever the clock. So an app that compares its clock with a time a dependency gave it (a token's expiry) makes a call its recording lacks, and the test is quarantined, though CI reads the recorded instant. keploy test --judge replays both rounds at the recorded instant (no clock lead), drops the entry of a test that passes both, and prints RUNS AGAIN. keploy/enterprise#3037's phase 12c proves it end to end. A call made only on some runs and overlapping requests are the other causes on the customer's side; the rest is Keploy's to fix, and the reader reports it. - The unmocked_call row says the same, and Freeze time links to it. - A NOT JUDGED set: the verified_red NEXT block's run:, do: and options (keploy test, keploy test --judge, keploy record) get the set judged none of them while it is on disk (pipeline 12241, phase 12a), so the pages say to delete it first, whatever the block says. - KEPLOY_PROVEN=yes: keploy record and keploy test check the sign-in in PreRunE, before ValidateFlags refuses the value, so without a session they exit 5 at the sign-in error first (checked on a build of the proof tree). Signed-off-by: slayerjain --- .../version-4.0.0/running-keploy/proven-flow.md | 8 ++++---- .../version-4.0.0/running-keploy/verdicts-yaml.md | 12 +++++++++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 94f034d37..74a0cc28f 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -69,7 +69,7 @@ Then turn the flow on for your shell: export KEPLOY_PROVEN=1 ``` -Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`, and the other commands take it as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set. The commands Keploy prints for you to run carry `KEPLOY_PROVEN=1`: every `keploy` command in the `NEXT` block, except the `keploy mock` commands, which don't need it; the `keploy quarantine` line under `QUARANTINE?`; and the commands in each reason's remedy. A few log lines and errors name a command without it, such as the `next_step` of `keploy record`'s incomplete-verdict log line and the `keploy quarantine` command in the `--must-pass` refusal, so set the switch on every Keploy command of this flow all the same. The examples on this page set it on each command, so you can copy any one of them on its own. +Set it to `1`. `keploy record` and `keploy test` refuse a value they can't read as on or off, such as `yes`. They check your sign-in first, so without a session they stop at the sign-in error (exit 5), and refuse the value only once you're signed in. The other commands take such a value as off. Set the switch for every Keploy command in this flow: `record`, `test`, `status` and `quarantine`. Without it, `keploy record` doesn't judge, `keploy test` ignores `verdicts.yaml` (it warns that it does, and runs every test, the quarantined ones too), and `keploy quarantine`, `keploy test --all` and `keploy test --judge` don't exist. Once a test set holds a `verdicts.yaml`, a Keploy command run without the switch still ends with this flow's `NEXT` block, with a caveat that the switch isn't set. The commands Keploy prints for you to run carry `KEPLOY_PROVEN=1`: every `keploy` command in the `NEXT` block, except the `keploy mock` commands, which don't need it; the `keploy quarantine` line under `QUARANTINE?`; and the commands in each reason's remedy. A few log lines and errors name a command without it, such as the `next_step` of `keploy record`'s incomplete-verdict log line and the `keploy quarantine` command in the `--must-pass` refusal, so set the switch on every Keploy command of this flow all the same. The examples on this page set it on each command, so you can copy any one of them on its own. ## Record your tests @@ -213,7 +213,7 @@ Under the `JUDGED` line, Keploy can print: Other lines a test set can get: -- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name, delete the set's directory, `keploy/`, and record again. Don't commit the set or upload its mocks: `keploy record` writes a new set when you record again, and leaves this one on disk until you delete it, where `keploy test` replays every one of its tests. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example. A set with no test case gets `NOT JUDGED : it has no test case to judge`. +- `NOT JUDGED : ; nothing was written for it`: Keploy couldn't judge the set, so it decided nothing for any of its tests: it writes no `verdicts.yaml` for the set, and doesn't put the set's mocks in the Keploy registry. It happens when a test of the set has no verdict, such as one that called a Unix socket, or when a round stopped early or the app never answered a test. `` says which round stopped and why: for a test with no verdict, it names the cause and the tests, and a `RECORDING LOST` line under it names each of those tests. What to do is in the `next_step` of the replay's `ERROR` line above it, which starts `test-set is not judged:`. Fix what they name, delete the set's directory, `keploy/`, and record again, in that order, whatever the `NEXT` block says: none of the commands that block gives gets the set judged while the set is on disk. Don't commit the set or upload its mocks: `keploy record` writes a new set when you record again, and leaves this one on disk until you delete it, where `keploy test` replays every one of its tests. [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged) shows an example, and why. A set with no test case gets `NOT JUDGED : it has no test case to judge`. - `NOT WRITTEN : keploy//verdicts.yaml changed while keploy record judged, so it stays as it was`: something else changed the file meanwhile, such as a `git pull` or another Keploy run, so nothing Keploy decided for the set was written. The same line ends `could not be written, so it stays as it was` when the write failed, for example on file permissions. Run `KEPLOY_PROVEN=1 keploy test --judge`: it replays the set and prints a `keploy quarantine` command for each test that fails, which you run if you agree (it writes a `manual` entry). Or delete the set and record it again. Until its `verdicts.yaml` says what you decided, don't commit the set or upload its mocks. - `NOT JUDGED` and `NOT WRITTEN` lines can end `, and the bake of its mappings.yaml's worker scope was put back`. Before the confirming round, Keploy had moved into the set's `mappings.yaml` the recorded calls your app made outside the test that recorded them (see `BAKED` below). Nothing was decided for the set, so Keploy put the file back as `keploy record` wrote it. - `BAKED : answers from on, in the set's worker scope (mappings.yaml worker:): ...`, printed before the `JUDGED` line: the full round saw your app make a recorded call outside the test that recorded it, for example from a background job, so Keploy moved that mock to the set's worker scope in `mappings.yaml`. `` is `the set's start`, or `after began`: the earliest point at which the full round saw the call. It's part of what you commit with `mappings.yaml`. `NOT BAKED : ...` says Keploy couldn't read or write the file, so those calls stay with their tests. @@ -246,7 +246,7 @@ protocol: ... =================== ``` -When the confirming round failed tests that judging then quarantined, the `summary` starts `keploy record's confirming round failed what it then quarantined`, and when a set failed on its own, it says so, with why. When judging's verdict failed, for example for a `NOT JUDGED` set or a CI that would be red, the block is `verified_red`, and when CI would run no test at all, `verified_empty`: fix what they name, then record again. When judging quarantined every test it judged but another test set of the project has a test CI runs, `keploy record` keeps its own exit code and its block is `recorded`: its `summary` says that none of the tests it judged is verified, and it gives CI's command, which runs the other sets' tests. +When the confirming round failed tests that judging then quarantined, the `summary` starts `keploy record's confirming round failed what it then quarantined`, and when a set failed on its own, it says so, with why. When judging's verdict failed, for example for a `NOT JUDGED` set or a CI that would be red, the block is `verified_red`, and when CI would run no test at all, `verified_empty`: fix what they name, then record again. For a `NOT JUDGED` set, delete the set's directory before anything else: the `verified_red` block's `run:` and `options` give `keploy test`, `keploy test --judge` and `keploy record`, and none of them recovers while the set is on disk (see [Tests with no verdict](/docs/running-keploy/verdicts-yaml/#not-judged)). When judging quarantined every test it judged but another test set of the project has a test CI runs, `keploy record` keeps its own exit code and its block is `recorded`: its `summary` says that none of the tests it judged is verified, and it gives CI's command, which runs the other sets' tests. #### Run CI's command once {#gate} @@ -429,7 +429,7 @@ When auto-replay doesn't run (`--disableAutoReplay`), nothing judges the recordi ## Freeze time {#freeze-time} -Under the switch, Keploy always freezes your app's clock when it replays: you don't pass `--freezeTime`. In auto-replay's full round, the clock reads 61.5 seconds after the moment each test was recorded, so time-dependent fields differ and are learned. In the confirming round and in CI, it reads the recorded moment. +Under the switch, Keploy always freezes your app's clock when it replays: you don't pass `--freezeTime`. In auto-replay's full round, the clock reads 61.5 seconds after the moment each test was recorded, so time-dependent fields differ and are learned. In the confirming round and in CI, it reads the recorded moment. So a call your app makes only once time has passed, such as the refresh of a token that a dependency said expires within that minute, can happen in the full round only, and the test is then quarantined for it: see [verdicts.yaml's classes and reasons](/docs/running-keploy/verdicts-yaml/#reasons) for how to bring it back. For a Go app there's nothing to rebuild. Keploy patches a copy of your compiled Go binary at run time and runs that copy, natively and in Docker; your own binary and image aren't modified. When Keploy can't patch a binary, it logs one `time-freeze[go]` line that names the binary and the reason, the app runs on the real clock, and `keploy test` lists it on an `UNFROZEN` line, so a pass that relied on the real clock shows. When staging the frozen clock fails for an operational reason, such as a Docker daemon error or an image Keploy couldn't pull, Keploy refuses to start the app unfrozen and says why. See [Time Freezing](/docs/keploy-cloud/time-freezing/) for what Keploy patches and the few cases it can't. diff --git a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md index f10acf72e..9379f450e 100644 --- a/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md +++ b/versioned_docs/version-4.0.0/running-keploy/verdicts-yaml.md @@ -94,13 +94,19 @@ Keploy writes each entry's fields on fixed lines, one entry per test, sorted by `keploy test` prints what to do for each class and reason under its `QUARANTINED, not run` list. -Every entry that `keploy record` or `keploy mock record` writes was decided on the build that recorded the test: auto-replay replays each recording against the same app, right after you stop recording. So for the reasons of one test's calls, `keploy/unmocked_call`, `keploy/unconsumed_mock` and `keploy/mock_missing`, the `keploy` class means what it says: Keploy has to act. On an app that didn't change, Keploy couldn't answer one of the test's calls from its recording, or couldn't match the calls it recorded for it. Checking your diff finds nothing, because there's no diff, and recording the same traffic again can quarantine a different set of tests rather than none. Report such a test to Keploy, with the entry's `calls:` or `mocks:` evidence and the log of the `keploy record` that wrote it. Keploy's printed remedy for `keploy/unmocked_call` and `keploy/unconsumed_mock` says to re-record the test if the app changed on purpose, and to check the diff otherwise: neither applies to an entry from a record's own judging. The two reasons of a whole set, `keploy/set_failed` and `keploy/set_capture_incomplete`, are different: see their rows. +Every entry that `keploy record` or `keploy mock record` writes was decided on the build that recorded the test: auto-replay replays each recording against the same app, right after you stop recording. So for the reasons of one test's calls, `keploy/unmocked_call`, `keploy/unconsumed_mock` and `keploy/mock_missing`, your code didn't change, and checking your diff finds nothing. Keploy's printed remedy for `keploy/unmocked_call` and `keploy/unconsumed_mock` says to re-record the test if the app changed on purpose, and to check the diff otherwise: neither applies to an entry from a record's own judging. The `keploy` class doesn't mean that only Keploy can act, though. Before you report a `keploy/unmocked_call` or `keploy/unconsumed_mock` entry, check these causes on your side: + +- **A call your app makes only once time has passed.** The full round replays each test with your app's clock 61.5 seconds after the moment the test was recorded (see [Freeze time](/docs/running-keploy/proven-flow/#freeze-time)). An app that compares its clock with a time a dependency gave it, such as the expiry of a token an auth server issued, can then make a call its recording doesn't have, here a token refresh, or skip one it has. Keploy quarantines such a test as `keploy/unmocked_call` for the extra call, or `keploy/unconsumed_mock` for the skipped one, the same way as a call it couldn't match: the class doesn't tell the two apart. CI reads the recorded moment, so the test can pass there all the same. Run `KEPLOY_PROVEN=1 keploy test --judge`: both of its rounds replay at the recorded moment, as CI does. When the test passes both, it drops the test's entry and prints `RUNS AGAIN`, so CI runs the test again. Then run [CI's command](/docs/running-keploy/proven-flow/#gate) once more before you commit. When the test still fails, the entry stays, and the clock wasn't the cause. +- **A call your app makes only on some runs,** for example one that depends on random sampling, or on which of two goroutines gets there first. Make it deterministic, then re-record. +- **Requests that overlapped while you recorded.** Keploy maps each dependency call made while two requests were open to the test whose request came first. Record those flows again, [one request at a time](/docs/running-keploy/proven-flow/#record-your-tests). + +If none of these is the cause, Keploy couldn't answer one of the test's calls from its recording, or couldn't match the calls it recorded for it, and recording the same traffic again can quarantine a different set of tests rather than none. Report such a test to Keploy, with the entry's `calls:` or `mocks:` evidence and the log of the `keploy record` that wrote it. The two reasons of a whole set, `keploy/set_failed` and `keploy/set_capture_incomplete`, are different: see their rows. To re-record a test, record its flow again with `KEPLOY_PROVEN=1 keploy record` and the command you recorded it with: Keploy writes the new recording to a new test set and judges it there. Then delete the old test case from its set, and `KEPLOY_PROVEN=1 keploy test --judge` drops its entry (`PRUNED`). | Class and reason | What happened | What to do | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `keploy/unmocked_call` | On the build that recorded it, the test made a call Keploy had no recorded answer for, or one call more than it recorded. | Report it to Keploy, as [above](#reasons). One cause is yours: requests that overlapped while you recorded. Then record those flows again, [one request at a time](/docs/running-keploy/proven-flow/#record-your-tests). | +| `keploy/unmocked_call` | On the build that recorded it, the test made a call Keploy had no recorded answer for, or one call more than it recorded. | Check the causes on your side [above](#reasons) first: a call your app makes only once time has passed, for which `KEPLOY_PROVEN=1 keploy test --judge` brings the test back; a call it makes only on some runs; or requests that overlapped while you recorded. If none is the cause, report it to Keploy. | | `keploy/unconsumed_mock` | On the build that recorded it, the test never made a call Keploy recorded for it. | The same as for `keploy/unmocked_call`. | | `keploy/mock_missing` | The replay of the build that was just recorded was served other mocks than its recording maps. | Report it to Keploy, as [above](#reasons). Keploy's printed remedy is to re-record the test: a new recording is judged afresh, and it isn't sure to be kept. | | `keploy/capture_incomplete` | Keploy couldn't capture one of the test's calls, such as a TLS handshake it couldn't join or a UDP reply, or the test's recording lost data that no round could judge. | Give the dependency a TCP address Keploy can mock, or list it in [`mock.real`](/docs/running-keploy/proven-flow/#mock-real), then re-record. For a recording that lost data, see [Capture-loss evidence](/docs/running-keploy/proven-flow/#recording-lost). | @@ -129,7 +135,7 @@ A test can also end with no verdict at all, which is not a quarantine. It happen keploy record's verdict is incomplete: 1 of the 1 test set it recorded is not judged (test-set-0); it exits 1 ``` - The `NOT JUDGED` line names the cause and the tests that have no verdict, and a `RECORDING LOST` line under it names each of them. What to do is in the `next_step` of the log lines above them: the `WARN` Keploy logged when the call happened (`the application called a Unix socket keploy cannot capture: ...`), and the replay's `ERROR` line that starts `test-set is not judged:`. Here it's `reach that dependency over a TCP address (mocked, or listed in mock.real)`. The `NEXT` block that follows says `verified_red`, although the set isn't judged rather than failed. + The `NOT JUDGED` line names the cause and the tests that have no verdict, and a `RECORDING LOST` line under it names each of them. What to do is in the `next_step` of the log lines above them: the `WARN` Keploy logged when the call happened (`the application called a Unix socket keploy cannot capture: ...`), and the replay's `ERROR` line that starts `test-set is not judged:`. Here it's `reach that dependency over a TCP address (mocked, or listed in mock.real)`. The `NEXT` block that follows says `verified_red`, although the set isn't judged rather than failed. Don't follow its commands for this set. Its `run:`, `do:` and `options` give `KEPLOY_PROVEN=1 keploy test`, `KEPLOY_PROVEN=1 keploy test --judge` and `KEPLOY_PROVEN=1 keploy record`, and none of them gets the set judged while it's on disk: `keploy test` replays it with no `verdicts.yaml` and fails, and its own `NEXT` block is `verified_red` again; `--judge` replays the recording Keploy couldn't judge; and a new record writes another set beside it. An agent that follows the block without deleting the set goes round that loop. Recover as below instead. To recover, fix the cause, delete the set's directory, then record again: From c75a876c6de6c37099ea2c42e76069505d81bdb6 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 22 Oct 2026 13:12:44 +0530 Subject: [PATCH 14/14] docs: test.selectedTests's warning suggests -t, but a set CI is not to run is one to delete Review round 2 of keploy/docs#937 named the refused-settings paragraph's "select test sets with -t instead" among the steps that lead a reader into a run that is not the suite's verdict. Under t2-next-ci r4 a gated run over only some sets is recorded (partialRun), and its NEXT block says a set CI is not to run is one to delete, not to leave out with -t (cliflow/next.go). The paragraph now says what the warning suggests, and that CI's keploy test replays every set on disk. Signed-off-by: slayerjain --- versioned_docs/version-4.0.0/running-keploy/proven-flow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md index 74a0cc28f..fd04104b3 100644 --- a/versioned_docs/version-4.0.0/running-keploy/proven-flow.md +++ b/versioned_docs/version-4.0.0/running-keploy/proven-flow.md @@ -457,7 +457,7 @@ Under `KEPLOY_PROVEN=1`, Keploy refuses these flags before it starts your app, w | `keploy test` | `--flaky-check-retry` | `keploy test --judge`. | | `keploy test` | `--mock-noise-detection`, `--schema-noise-detection` | Nothing: `keploy record`'s auto-replay learns what changes, and a replay writes no recording. | -Keploy also ignores the matching `keploy.yml` settings, such as `test.ignoredTests`, `test.selectedTests` (select test sets with `-t` instead), `test.mustPass`, `test.healthPollTimeout`, `record.passThroughPorts` and `bypassRules`, and says so in a warning when they're set. +Keploy also ignores the matching `keploy.yml` settings, such as `test.ignoredTests`, `test.selectedTests`, `test.mustPass`, `test.healthPollTimeout`, `record.passThroughPorts` and `bypassRules`, and says so in a warning when they're set. For `test.selectedTests`, the warning suggests `-t` instead, but CI's `keploy test` takes no `-t` and replays every test set on disk, so delete a set CI isn't to run. A run with `-t` is never the suite's verdict. ## Related