Skip to content

About

Automated tests to run on product documentation.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

docs-tests

Automated tests for Solo product documentation, extracted out of the doc markdown so that guides stay readable to casual contributors and test scripts are reusable across each product's own CI workflow.

This repo currently holds two things:

  • scripts/ — the extractor/runner tooling (doc_test_extract.py, doc_test_run.py, doc_test_schema_check.py, doc_test_inject_status.py, doc_test_fetch_artifacts.sh, merge_test_results.py, report_summary.py, list_untested_docs.py). This is the canonical copy — agentgateway/website's CI and Makefile check this repo out and invoke these scripts directly (docs-tests/scripts/doc_test_run.py --repo-root ., etc.); it no longer keeps its own copies.
  • products/ — the actual test content, one directory per product. For agentgateway, products/agentgateway/annotations/ holds the test annotations, which mirror each doc page's own path (for example, annotations/assets/agw-docs/pages/traffic-management/rewrite/host.md.yaml) and are applied to the docs content at test time. Kept as a directory layout on main rather than a branch per product × version, and as committed, reviewable, diffable files rather than CI-generated artifacts.

Dependencies

Install yamltest and cloud-provider-kind to run the tests:

npm i -y -g yamltest@latest
pip3 install PyYAML
go install sigs.k8s.io/cloud-provider-kind@latest

How it works

  1. A product's tests.yaml manifest (products/<product>/<mode>/tests.yaml) defines the test scenarios. Each one names its prerequisites and the path selectors to include from a page. A page's front matter test: block is still read for pages the manifest does not cover.
  2. annotations.py attach puts the test markup (paths= tags and hidden {{< doc-test >}} blocks) from products/<product>/annotations/ onto a copy of the content repo's pages. The pages themselves carry no markup.
  3. doc_test_run.py reads the scenarios, chains the selected blocks together, and produces a standalone bash script.
  4. The script is run inside a fresh kind cluster (with cloud-provider-kind for load balancer support).

Test annotations

Content repos are moving to carry no test markup at all. The markup (paths= tags on code blocks, hidden {{< doc-test >}} checks, and front matter test:) moves into this repo as test annotations under products/<product>/annotations/, one YAML file per page, and scripts/annotations.py attaches it to a copy of the pages before a run.

An annotation finds its code block by fingerprint, a hash of the block's text, so pages can move and prose can change freely. Each piece of markup ends a run in one of three states:

State What happened
exact The block's text is unchanged.
close The block between the same unchanged neighbors was edited slightly, and it is the only nearly identical candidate: line similarity at least 0.6, and at least 0.1 ahead of the next candidate.
needs update The block was rewritten or removed. The markup stays in the annotation file and is reported, never attached somewhere else.

Hidden checks move only with the block or sentence they follow. A check that follows a sentence is looked for between the same two code blocks, even when the block before it was close-matched.

Command Use
make annotate PAGE=<page> then make save PAGE=<page> Add or change the tests on one page. annotate writes the page with its markup to out/annotate/; edit the markup there; save writes only the markup back, and refuses if the copy changes the page itself.
annotations.py attach Attach annotations to a checkout, in place or into --out. Removes any inline markup first.
annotations.py refresh After content changes: re-attach, record the new fingerprints, and follow moved pages (git rename detection, then a search by fingerprint). Never drops markup; refuses to write if any would disappear. Inline markup still on a page is ignored: annotations are the source of truth, so refresh never copies a page's markup into them or creates annotation files for pages that gain it. It reports inline front matter test: edits, which need a matching tests.yaml change.
annotations.py pending List the markup still marked needs update, with --json for a machine-readable copy. See Fixing markup that needs update.
annotations.py export / strip Migration only: copy inline markup into annotation files, then remove it from the pages.
annotations.py roundtrip Strip and re-attach every page in memory; every page must come back byte-identical. Runs daily in CI against agentgateway/website.
annotation_replay.py Replays a repo's git history, whose inline markup is the answer key, to measure close matching. Re-run it when matching changes; close WRONG must not rise.
doc_test_run.py --annotation-report <attach report> Skips, with a warning, any scenario that would select markup which did not attach, instead of running it with a step missing. Skipped scenarios go under skipped_needs_update in the results, not under tests.

.github/workflows/refresh-annotations.yml runs refresh daily at 05:00 UTC against agentgateway/website main and commits the result to this repo's main. Every scheduled run posts to #doctopus-tests in the same layout as the doc test results, listing every piece of markup that needs update, with a thread of the day's close matches and moved pages; a failed refresh also posts to #doctopus-builds. Posting needs a DOCS_TESTS_SLACK_BOT_TOKEN secret on this repo; until it is set, the step skips with a warning.

For agentgateway/website, every section has moved: scenario definitions live in the tests.yaml manifests, so their annotations were exported with --front-matter none. A manifest key must be unique, but a scenario name need not be: set name: when two pages in one mode declare the same name. Pages that deliberately have no test are listed under the manifest's skip: key, the manifest form of front matter test: skip; the runner and list_untested_docs.py count them as covered. A skip: entry can also be dir: <folder>/, which covers every page under that folder. Use it for generated pages, such as the agctl CLI reference, where new pages appear without anyone editing the manifest.

Referencing external test content in this repo

Note

No current docs page uses the file="..." attribute or the assert: list described in this section. Test content now lives in the annotation files under products/<product>/annotations/. The extractor still supports both mechanisms, so the paths in these examples are placeholders, not real files.

The hidden {{< doc-test >}} shortcode (used for setup/assertions that must run during tests but never appear in rendered HTML) accepts an optional file="..." attribute. When present, the extractor reads the block's content from that path resolved against a docs-tests checkout, instead of requiring an inline body between the opening and closing shortcode tags:

{{< doc-test paths="rewrite" file="products/<product>/<path>/rewrite.sh" >}}{{< /doc-test >}}

The referenced file (products/<product>/<path>/rewrite.sh in this example) holds exactly the content that used to sit inline between the shortcode tags — nothing else changes about how the block is selected or emitted; it flows through the same paths= selection and script assembly as an inline block.

Resolving the docs-tests checkout root:

  • doc_test_extract.py and doc_test_run.py both accept --docs-tests-root <path>, or the DOCS_TESTS_ROOT environment variable.
  • If neither is set, it defaults to the docs-tests checkout the script itself runs from, so the manifests in products/ are always read.
  • In CI, add a checkout step for docs-tests and pass its path through either mechanism — see scripts/CI_WORKFLOW_SKETCH.md for the concrete workflow change.
  • A missing or misspelled file= path fails the extractor immediately with a FileNotFoundError naming the source doc, line number, and the resolved path that didn't exist — it never silently drops the block.

This is unrelated to the <!-- doc-test-include file="..." --> HTML comment some of the extractor's code still supports — that mechanism runs an external bun test <file> as a separate subprocess (built for a different kind of test) and is not used for this external-content use case.

Referencing external test content from front matter instead

The file="..." shortcode above still leaves one line of test scaffolding sitting in the page body for every hidden block. A step in a page's test: front matter can name the same external content directly instead, via an assert: list, so the page body carries nothing but the real content and the paths="X" tag on the visible block:

test:
  host-rewrite:
    type: functional
    steps:
    - file: ${versionRoot}/quickstart/install.md
      path: experimental
    - file: ${versionRoot}/traffic-management/rewrite/host.md
      path: host-rewrite
      assert:
      - products/<product>/<path>/host-rewrite-wait.sh
      - products/<product>/<path>/host-rewrite-warmup.sh
      - products/<product>/<path>/host-rewrite-assert.sh

Each entry in assert: is a docs-tests-relative path, run in list order. This replaces what would otherwise be three separate {{< doc-test paths="host-rewrite" file="..." >}}{{< /doc-test >}} lines physically sitting in the page. Both mechanisms are supported and can coexist across different pages — assert: is the newer, cleaner form; file="..." on an inline shortcode still works for pages that haven't been converted.

Ordering: assert: doesn't say where in the page these files logically belong, so the extractor anchors each entry to the first block (visible or hidden) in that same file whose paths= matches the step's path: — the same anchor a reader would expect from where the inline shortcode used to sit — and runs the whole assert: list immediately after it, before anything else in the file. This matters because a single paths= value is often reused later in the same page for something unrelated (a Cleanup section's kubectl delete, say); anchoring to the first matching block, not just appending at the end of the file, keeps assertions running before cleanup rather than after it.

Only scenarios that actually execute something need an assert: list — a type: schema step never runs anything, so it has no equivalent.

Path selectors

A path is a string label attached to a fenced code block or hidden command block. It controls which blocks are included in which test scenario.

Tagging visible code blocks

Add ,{paths="<name>}" to the fenced code language line:

```sh,{paths="install-httpbin"}
kubectl apply -f https://raw.githubusercontent.com/.../httpbin.yaml
```

A block may belong to multiple paths:

```sh,{paths="standard,experimental"}
helm upgrade -i --create-namespace ...
```

Only sh/bash/shell/yaml/yml blocks are extracted.

Tagging hidden command blocks

Use the {{< doc-test >}} Hugo shortcode for commands that must run during tests but must not appear in the website HTML (waits, retries, cleanup). The shortcode template outputs nothing, so the content is completely absent from rendered pages:

{{< doc-test paths="install-httpbin" >}}
YAMLTest -f - <<'EOF'
- name: wait for httpbin deployment
  wait:
    ...
EOF
{{< /doc-test >}}

The paths= attribute works identically to fenced blocks. As above, add file="..." instead of an inline body to pull the content from this repo's products/ tree.

Front matter test metadata

On the page being tested, add a test: key to the YAML front matter. Each child key is a named test scenario. Each entry in the list is a file+path pair — a source file and the path selector to pull from it.

---
title: CORS
test:
  cors-in-httproute:
  - file: content/docs/kubernetes/main/quickstart/install.md
    path: experimental
  - file: content/docs/kubernetes/main/setup/gateway.md
    path: all
  - file: content/docs/kubernetes/main/install/sample-app.md
    path: install-httpbin
  - file: content/docs/kubernetes/main/security/cors.md
    path: cors-in-httproute

  cors-in-agentgatewaypolicy:
  - file: content/docs/kubernetes/main/quickstart/install.md
    path: standard
  - file: content/docs/kubernetes/main/setup/gateway.md
    path: all
  - file: content/docs/kubernetes/main/install/sample-app.md
    path: install-httpbin
  - file: content/docs/kubernetes/main/security/cors.md
    path: cors-in-agentgatewaypolicy
---

Multiple scenarios on the same page each get their own kind cluster and generated script.

Typing scenarios

A scenario can declare a type: alongside its file/path entries, renamed steps: under it:

test:
  rewrite:
    type: functional
    steps:
    - file: content/docs/kubernetes/main/quickstart/install.md
      path: experimental
    - file: content/docs/kubernetes/main/traffic-management/transformations/rewrite.md
      path: rewrite

A bare list (no type:/steps: wrapper) is still accepted and treated as functional — every scenario written before this typing existed already applies real config and asserts on a real response, which is what functional means.

Type What it checks Needs Blocks the PR?
schema The doc's own examples validate against the real schema for the version the page documents: custom resources and Gateway API kinds against their CRDs, core Kubernetes kinds through kubeconform, and standalone config files against agentgateway's schema/config.json (renamed/removed/mistyped fields, wrong types) Nothing — no cluster, no execution (doc_test_schema_check.py) Yes, in its own job
functional Real behavior in a real cluster (apply config, assert on a real response) A kind cluster, no vendor credentials Yes

schema needs no prerequisite chain — since nothing executes, only the step that shows the custom resource itself matters:

test:
  rewrite-schema:
    type: schema
    steps:
    - file: content/docs/kubernetes/main/traffic-management/transformations/rewrite.md
      path: rewrite

schema and functional are the only types; any other type: is an error. A page that depends on an external service (an LLM provider, an identity provider, a cloud account) is tested as functional against an in-cluster stand-in: the httpbun mock LLM, an in-cluster Keycloak, a patched provider host. What the stand-in can't reach (a real Entra tenant, a browser login) stays untested, and the scenario's comments say so.

Declaring more than one type on the same scenario

type: also accepts a list, so one steps: chain gets validated more than one way without copy-pasting the whole scenario into a same-named -schema sibling just to add a second type:

test:
  rewrite:
    type: [schema, functional]
    steps:
    - file: content/docs/kubernetes/main/quickstart/install.md
      path: experimental
    - file: content/docs/kubernetes/main/traffic-management/transformations/rewrite.md
      path: rewrite

This produces two independent test cases, rewrite::schema and rewrite::functional, sharing the same steps: — the same shape as hand-writing two separate scenarios, minus the duplication. --test rewrite selects both; --test rewrite::schema selects only the schema one. A single-type scenario is unaffected: its name, generated filenames, and report key stay exactly as before — the name::type suffix only appears once a scenario declares more than one type.

Add schema this way only when the scenario shows something the checker has a schema for: a custom resource whose CRD it was given (agentgateway's own kinds, and Gateway API kinds), a core Kubernetes kind (when kubeconform is available), or, on a standalone page, a whole config file. A scenario in which nothing was validated fails rather than passing vacuously, so schema stays opt-in per scenario, not automatic. Helm values and CRD CEL rules are not checked; see #19.

Running a scenario on macOS or Windows

Scenarios run on Linux, in a kind cluster, unless their tests.yaml entry sets runner:. Set it only for a page that has to run on another OS, such as an install guide for macOS or Windows:

  migrate-install-macos:
    runner: macos        # or windows; linux is the default
    page: documentation/migrate/install/macos/_index.md
    path: migrate-install-macos
  • A macos or windows scenario runs its script directly, with no cluster. Hosted macOS and Windows runners have no Docker, so these scenarios cannot use needs: prerequisites that install agentgateway in Kubernetes.
  • --list-tests reports each scenario's runner, and the consuming site's workflow sends it to a job on that OS (macos-latest or windows-latest). The script still runs in bash; on Windows that is Git Bash, named through DOC_TEST_BASH.
  • A local run skips scenarios for another OS with a SKIPPED (runner) warning. --generate-only still generates them, so you can read the script on any machine.

For a page written in PowerShell, such as a Windows install guide, also set shell: pwsh:

  migrate-install-windows:
    runner: windows
    shell: pwsh
    page: documentation/migrate/install/windows/_index.md
    path: migrate-install-windows
  • The scenario is generated as one .ps1 script and run with pwsh -File, so a variable set in one block is still set in the next.
  • Its page blocks are powershell (or pwsh, ps1) fences. Write its hidden doc-test blocks in PowerShell too.
  • The script stops on the first error, including a native command's non-zero exit. To fail a check, throw.

Tracing prerequisites

Every guide has a Before you begin section that lists prerequisites. Follow the chain from the feature guide back to the install guide:

feature page (e.g. cors.md)
  └── sample-app.md           (httpbin installed + HTTPRoute ready)
        └── gateway.md        (Gateway created + LB address exported)
              └── helm.md     (CRDs + controller installed)

For each hop:

  1. Open the file and find its ## Before you begin section.
  2. Follow the linked page.
  3. Identify which code blocks are relevant and what path label they carry (or add one if missing).
  4. Add that file+path as a source entry above the current one in the test: front matter.

The extractor follows {{< reuse "..." >}} and internal links automatically, so you don't need to inline snippet contents — just reference the top-level content file.

Choosing the right path

  • Use an existing path label if one already exists on the blocks you need.
  • The path all is a conventional catch-all for blocks that are included in every scenario from that file.
  • For tabbed content (Standard / Experimental installs), separate paths (standard, experimental) let you pick the right tab per scenario.
  • A code block with no paths= is skipped by default (skip_tabs_without_paths: true).

Adding a path to an existing block

If a block you need has no path, add one:

```sh {paths="install-httpbin"}
kubectl apply -f ...
```

If the same block already belongs to another path and you need to add yours:

```sh {paths="standard,my-new-path"}
...
```

Waiting for resources with YAMLTest

Use YAMLTest -f - <<'EOF' ... EOF inside a {{< doc-test >}} shortcode block immediately after the kubectl apply it depends on. The wait test type polls a Kubernetes resource until a JSONPath condition is met.

Wait for a Deployment to be ready

{{< doc-test paths="all" >}}
YAMLTest -f - <<'EOF'
- name: wait for agentgateway-proxy deployment to be ready
  wait:
    target:
      kind: Deployment
      metadata:
        namespace: agentgateway-system
        name: agentgateway-proxy
    jsonPath: "$.status.availableReplicas"
    jsonPathExpectation:
      comparator: greaterThan
      value: 0
    polling:
      timeoutSeconds: 300
      intervalSeconds: 5
EOF
{{< /doc-test >}}

Wait for a Service to get a load balancer address and export it

{{< doc-test paths="all" >}}
YAMLTest -f - <<'EOF'
- name: wait for agentgateway-proxy service LB address
  wait:
    target:
      kind: Service
      metadata:
        namespace: agentgateway-system
        name: agentgateway-proxy
    jsonPath: "$.status.loadBalancer.ingress[0].ip"
    jsonPathExpectation:
      comparator: exists
    polling:
      timeoutSeconds: 300
      intervalSeconds: 5
  setVars:
    INGRESS_GW_ADDRESS:
      value: true
EOF
{{< /doc-test >}}

setVars exports the jsonPath-matched value as an environment variable for downstream steps. It is a sibling of wait: (not nested inside it).

Wait for an HTTPRoute condition

{{< doc-test paths="install-httpbin" >}}
YAMLTest -f - <<'EOF'
- name: wait for httpbin HTTPRoute to be accepted
  wait:
    target:
      kind: HTTPRoute
      metadata:
        namespace: httpbin
        name: httpbin
    jsonPath: "$.status.parents[0].conditions[?(@.type=='Accepted')].status"
    jsonPathExpectation:
      comparator: equals
      value: "True"
    polling:
      timeoutSeconds: 300
      intervalSeconds: 5
EOF
{{< /doc-test >}}

Comparators

Comparator Meaning
equals Exact string/number match
greaterThan Numeric greater-than
exists Field is present and non-empty
contains String contains substring

Testing a feature with YAMLTest HTTP assertions

After all resources are ready and INGRESS_GW_ADDRESS is exported, add an HTTP test inside a hidden block on the feature page itself:

{{< doc-test paths="cors-in-httproute,cors-in-agentgatewaypolicy" >}}
YAMLTest -f - <<'EOF'
- name: CORS preflight returns expected headers
  http:
    url: "http://${INGRESS_GW_ADDRESS}:80/get"
    method: OPTIONS
    headers:
      host: www.example.com
      Origin: https://example.com
  source:
    type: local
  expect:
    statusCode: 200
    headers:
      - name: access-control-allow-origin
        comparator: equals
        value: https://example.com
      - name: access-control-allow-methods
        comparator: contains
        value: GET
      - name: access-control-max-age
        comparator: equals
        value: "86400"
EOF
{{< /doc-test >}}
  • source.type: local sends the request from the local machine (default).
  • Use source.type: pod with a pod selector to send the request from inside the cluster.
  • The headers list under expect checks response headers (case-insensitive name matching).
  • Use retries: on a test entry to retry on transient failures (e.g. a race condition where a response code flips briefly). Do not use retries: to work around a new-hostname ECONNRESET — see the Troubleshooting section.

Running the tests

Generate scripts only (no cluster)

python3 scripts/doc_test_run.py --generate-only

Scripts are written to out/tests/generated/.

Run all tests

Requires kind and cloud-provider-kind in PATH.

python3 scripts/doc_test_run.py

Each test scenario:

  1. Creates a kind cluster named doc-test-<scenario>.
  2. Starts cloud-provider-kind in the background (provides LoadBalancer IPs).
  3. Runs the generated bash script.
  4. Deletes the cluster.
  5. Writes results to out/tests/generated/test-results.yaml.

Run a single test scenario

Point directly to a file and (optionally) a named scenario. This generates the script, creates a kind cluster, starts cloud-provider-kind, runs the test, and cleans up — all in one command:

# Run one specific scenario
python3 scripts/doc_test_run.py \
  --file content/docs/kubernetes/main/security/cors.md \
  --test cors-in-httproute

# Run all scenarios defined in a single file
python3 scripts/doc_test_run.py \
  --file content/docs/kubernetes/main/security/cors.md

To only generate the script without running (useful for inspection):

python3 scripts/doc_test_run.py \
  --file content/docs/kubernetes/main/security/cors.md \
  --test cors-in-httproute \
  --generate-only
bash out/tests/generated/<script-name>.sh

Key CLI options

Flag Default Description
--file — Path to a single markdown file to generate/run tests for
--test — Name of a specific test scenario within --file
--docs-glob content/docs/**/*.md Glob to discover pages with test: metadata (ignored when --file is set)
--product kubernetes Context product used for conditional-text resolution
--docs-tests-root sibling docs-tests dir Checkout root for {{< doc-test file="..." >}} external content (or DOCS_TESTS_ROOT env var)
--generated-dir out/tests/generated Output directory for scripts and manifests
--generate-only false Skip cluster creation and execution
--verbose true Stream all command output

The version context (used to resolve {{< version include-if="..." >}} blocks) is inferred automatically from the source file paths — e.g. a source under kubernetes/latest/ resolves to version token latest, and kubernetes/main/ to main.

Extractor rules

doc_test_extract.py processes source files before emitting the script:

  • {{< reuse "..." >}} — inlined recursively from assets/.
  • {{< version include-if="..." >}} — resolved against the inferred version token; non-matching blocks are dropped.
  • {{< conditional-text include-if="..." >}} — resolved against the product context; non-matching blocks are dropped.
  • Indentation is stripped from fenced block content so heredocs work correctly in bash.
  • Duplicate blocks (same content) are emitted only once.
  • Blocks without a paths= attribute are skipped.
  • {{< doc-test file="..." >}} — content is read from the docs-tests checkout instead of the shortcode body (see above).
  • A step's assert: list — synthesizes the same kind of hidden block directly from front matter, anchored to the first block in that file sharing the step's paths= selector (see above).

Checklist for adding a test to a new page

All markup goes in this repo. Never add paths=, {{< doc-test >}}, or front matter test: to a page in the content repo.

  1. Trace prerequisites — follow "Before you begin" links back to helm.md.
  2. Open the page's annotation with make annotate PRODUCT=<product> PAGE=<page>. For a thin wrapper page, annotate the reused assets file, not the wrapper.
  3. Verify path labels on the code blocks you need; add paths="..." where missing, in the out/annotate/ copy.
  4. Add wait blocks after each kubectl apply that creates something tests depend on.
  5. Export INGRESS_GW_ADDRESS — it flows from gateway.md via setVars.
  6. Add the feature assertion as a {{< doc-test >}} block in the out/annotate/ copy, then run make save with the same arguments.
  7. Add the scenario to products/<product>/<mode>/tests.yaml, with its prerequisites under needs: and the page under source: (or page:).
  8. Regenerate from the content repo with make test-generate, which attaches your annotations first. Confirm everything attached, and inspect the script for unresolved shortcodes or missing commands.
  9. Run locally with make test-run DOC_TESTS_ARGS='--file <page> --test <scenario>' from the content repo. If the change goes with a content PR, push it here on a branch with the same name as the content PR's branch, so the content PR's run uses it.

Keeping tests in sync with doc changes

Content changes every day, and the tests here were written against the content as it was. Two things keep them lined up:

  • The daily refresh (.github/workflows/refresh-annotations.yml, see Test annotations) follows edited and moved blocks, so most content edits need no test change at all. It only moves markup; it never decides whether the content or the test is correct.
  • A docs-tests branch with the same name as the content PR's branch. When a content change needs a test change, push the test change to a branch here with the same name as the content PR's branch. The content repo's doc-tests workflow checks out that branch instead of main, so the PR's own run tests the new content with the new test. The job summary says when it did. Merge the content PR first, then the docs-tests PR.

Telling what is broken

A run can report two different problems, and they mean different things.

1. The test could not find its block (needs update). The content changed enough that an annotation no longer attaches.

  • Small edits are followed automatically (the close state), and the test still runs.
  • Bigger edits leave the markup unattached. Any scenario that uses it is skipped with a warning, not run with a step missing. The PR's job summary lists it, the daily refresh marks it as needs update and counts it in its Slack post, and the page loses its Verified badge (test_status: partial) until the annotation is updated. The refresh never updates it; a daily fixer job drafts the update as a pull request here, for a person to review (see Fixing markup that needs update).
  • This usually means someone edited the content on purpose and the test has to catch up. It does not by itself mean anything is wrong with the content.
  • Watch for: a broken content edit that is also big enough to land here. Its test is skipped rather than run, so nothing fails; only the skip warning shows it.

2. The test found its block and failed. The markup attached, the scenario ran, and an assertion or command failed. Any of these can be the cause:

Cause Where the fix goes
The content is wrong, for example a typo in a command or a renamed field The content repo
The test is wrong, for example a stale assertion or a missing wait This repo
The product's behavior changed (the daily run installs the latest dev build) The content repo if the change is intended, the product repo if it is a bug
The environment was flaky, for example a slow load balancer or a rate-limited registry Usually a re-run; a wait or retry here if it repeats

Compare what changed since the last passing run: commits to the page in the content repo, commits to its annotation or tests.yaml entry here, and the product version. Whichever one moved is usually the cause. The daily fixer job sorts each nightly failure into these causes and drafts a fix for the first two (see Fixing a failing doc test).

Fixing markup that needs update

A person or the daily fixer job follows the same steps. The fixer is an agent that runs after the refresh, does this for one product, and opens a pull request here. Nothing it writes reaches main until someone reviews and merges that pull request.

  1. List the work.

    python3 scripts/annotations.py pending --annotations products/<product>/annotations

  2. Open the page with annotations.py annotate, or make annotate PAGE=<page>. The copy in out/annotate/ carries the markup that still attaches. The pieces that do not attach are in the annotation file, under tags or hidden, with status: needs-update.

  3. Find out what changed. Read the page's history in the content repo, for example git log -p -- <page>, to see what happened to the block or sentence the markup followed.

  4. Decide, for each piece:

    What happened to the step What to do
    Still on the page, rewritten or moved Put the tag on the new block, or the hidden check after the new block or sentence. Keep the check's contents as they are, unless a value that the page uses changed, such as a resource name, key, port, or path. Then change only that value.
    Gone from the page Remove the markup. Record it as removed, with the reason.
    You cannot tell Leave it. Record it as unresolved, with the reason.

  5. Save with annotations.py save (or make save). save refuses a copy whose prose or code differs from the page. It also keeps the old needs-update entries, so delete the ones you replaced from the annotation file.

  6. Confirm it attaches.

    python3 scripts/annotations.py attach --repo <content repo> \
  --annotations products/<product>/annotations --page <page> \
  --out out/attached --report out/attach-report.json

    The page must report no needs update, apart from the pieces you recorded as unresolved.

Rules:

  • Do not copy test markup from the page. Some content pages still carry inline markup from before the move. Annotations here are the only source of truth, and an inline copy can undo a fix that was made here. Work from the annotation file and the page's content.
  • Re-anchor; do not rewrite. A check that worked before the page changed is still right, even when the page's inline copy of it differs. If the check needs more than the page's new values, record it as unresolved, and a person rewrites it.
  • Change annotation files only. A fix that needs a tests.yaml change, a new scenario, or a content change is unresolved. Say what it needs in the reason.
  • For a product whose pages are not public, previews stay off, and a check never repeats the page's prose. This repository is public.

scripts/annotation_fix_check.py holds the fixer to these rules. It compares the annotation files to the commit the agent started from and re-attaches every page. It fails when any of the following is true:

  • Markup still does not attach, or has disappeared, and is not recorded.
  • Anything outside the product's annotations changed.
  • A check's contents changed by more than small edits (less than 85% alike, by character), or a check is new.
  • With --private, a check repeats page text.

The fixer records its work in a JSON summary with fixed, unresolved, and removed lists. The comment at the top of the script describes the format.

Keeping a scenario's type in step with its page

A rewrite can change which test type fits a page. The stale test still passes, so nothing else notices. scripts/doc_test_type_check.py compares each scenario's type: with the page it tests, looking only at the page itself, not its needs:.

Finding Severity What --apply does
schema-empty: schema, but the page shows no resource or config file. The check passes only on its prerequisites' resources. Error Drops schema. A scenario left with no type moves to skip:.
functional-empty: functional, but the page has no visible yaml or shell block left Error Drops functional, keeping or adding schema if the page shows resources
skip-checkable: under skip:, but the page shows resources Error Adds a <page>-schema scenario and removes the skip: entry
schema-unreadable: schema, but no resource on the page parses as YAML, often \$ in a heredoc Warning Nothing
could-be-functional: schema only or skip:, but the page has commands to run Warning Nothing. A functional test needs prerequisites and assertions, so a person writes it.

A finding only counts if it holds in every version root the page is in, so a main-only rewrite waits for the release. Run the check after annotations.py attach:

python3 docs-tests/scripts/doc_test_type_check.py --repo-root . \
  --product agentgateway --config-schema agentgateway-main/schema/config.json \
  --summary "$GITHUB_STEP_SUMMARY"

For a site made of {{< rebase >}} shells, such as agentgateway-enterprise, pass the same --upstream-root, --upstream-versions, and --condition values as the schema check. Only the versions in the upstream map are checked. Pass the config schema the site's own schema check uses, here the enterprise config.json. With the open source one, enterprise-only keys make whole config files look like fragments.

Errors listed in products/<product>/type-check-baseline.yaml are reported as baselined and don't fail the run. Use the baseline for a finding --apply would get wrong. For example, install/argocd.md shows an Argo CD Application, which the schema check has no schema for. A run logs baseline entries that no longer match anything. Remove those entries, or regenerate the file with --write-baseline.

--apply edits tests.yaml line by line, so comments stay. A comment right above a removed entry goes with it. The script re-parses the result and compares it with the intended change before it writes anything. Before you merge, run the schema check on the edited manifests: a new skip-checkable scenario can still fail on a kind that has no schema.

Fixing a failing doc test

The same daily fixer job also reads the last nightly run's failures and drafts fixes for them. A person can follow the same steps. Nothing it writes is published until the failing scenario passes on a cluster with the fix applied, and nothing reaches any main branch until someone reviews and merges the pull request it opens.

  1. List the work. scripts/collect_failures.py turns a nightly run's results into a short list. Each entry has the scenario (key, page, test), the step that broke (step: its kind, page, line and command), and the last part of its output. When the page is a shell that rebases another site's page, upstream_page and step.upstream_file name that page. A content fix goes there, not in the shell.

  2. Read the failure, then the pages. Start from the step's command and output. Open the step's page and the scenario's page, and follow each {{< reuse "..." >}} to the file it pulls in. The page you edit is usually a reused file under assets/, not the page the result names. Read the page's annotation here too (products/<product>/annotations/<page>.yaml): a hidden check that broke lives there, not in the page.

  3. Decide what kind of failure it is.

    Kind What it looks like What to do
    content The page tells the reader to do something that does not work: a wrong name, field, flag, path, or value. Fix the page or the file it reuses.
    test The page is right, but the test is not: a stale assertion, a missing wait, a hidden check with the wrong name. Fix the annotation here.
    environment Something outside the page and the test: a timeout pulling an image, a rate limit, a missing secret, a cluster that never came up. Report it.
    product The page and the test agree, and the product does something else. Report it. A person decides whether the page or the product changes.
    unknown You cannot tell from what you have. Report it, saying what you would need.

    A step that broke because an earlier step of the same scenario did not do its job is that earlier step's failure. Fix the earlier one.

  4. Fix it the way a writer would. Change as little as you can.

    • Where the site already has a snippet for a value that differs between sites (a GatewayClass, a namespace, a deployment name), use it instead of the literal: {{< reuse "agw-docs/snippets/<name>.md" >}}.
    • Where it has none and the value differs between a site and a site that rebases it, add one in both: the upstream snippet with the upstream value, and a file at the same path in the consumer's assets/ with its value. The consumer's copy shadows the upstream one in its build and in its tests.
    • A check here that must work on more than one site reuses the same snippet, or reads the value from the cluster, rather than naming one site's value. {{< reuse "..." >}} inside a hidden check is expanded at test time, with each site's own copy of the snippet.
    • Editing a page moves its markup's anchors. A tag follows its block's text, and a hidden check follows the line or block before it, so changing that text can leave markup that no longer attaches. After every page edit, run the attach command from step 6 of Fixing markup that needs update on the page. Re-anchor anything that no longer attaches with annotate and save (steps 2 and 5 there), keeping its contents. The check refuses markup you left unattached.

  5. Never put test markup in a content page. No paths= tags, no {{< doc-test >}} blocks, no front matter test: key. Test markup lives here only. Some pages still carry inline markup from before the move; the run removes it before it tests, so editing it changes nothing.

  6. Keep the private side out of public repos. A consumer's pages, its product name, and its values never appear in a public repo's files or in the text that goes with them, apart from the consumer's own annotations here, which follow the private rules in Fixing markup that needs update.

  7. Record each failure in the summary's failures list, one row each:

    {"key": "<key from the list>", "kind": "content", "fix": true,
 "reason": "why it failed and what you changed",
 "public_reason": "the same, in words fit for the public repo's pull request",
 "files": ["<every file you changed for it, workspace-relative>"]}

    public_reason is needed whenever the row changes a public content repo. It is the only text of the row that reaches that repo.

scripts/failure_fix_check.py holds the fixer to these rules before anything runs on a cluster. It fails when any of the following is true:

  • A failure has no row, or a row has no kind or reason.
  • An environment, product, or unknown row changes a file.
  • A file changed that no row lists, or that is outside the paths a fix may change: content/docs/ and assets/ upstream, assets/<product>-docs/ in a consumer, and annotation files here. tests.yaml is not among them.
  • An annotation changed for a failure belongs to a page that the failure's scenario does not run.
  • A content page gains test markup.
  • A line added to a public content repo, or a public_reason, names the private side.
  • An added line reads a secret or the environment, calls gh, or names a web host that its repo does not already name. These lines run on a cluster before anyone reviews them.

A check edited to fix a failure may change more than a re-anchor would, so annotation_fix_check.py --exempt-pages does not hold it to the small-edit rule. The cluster run holds it instead: the workflow re-runs every failure with a fix, and drops the fix of any that still fails.

A second attempt

A re-run often fails later in the scenario than the nightly did, because the fix got past the step that broke first. So when the re-run of a fix fails, the fixer tries once more, for that failure only:

  • Every checkout already holds the first attempt's changes.
  • The failure's entry in the list carries previous_attempt: the kind and reason the first attempt gave, the files it changed, and the step and output where the re-run failed (rerun_step, rerun_output). For a fix that broke the open source site's own test of the same page, broke_open_source names that test.
  • Read the re-run's failure as a new one, and decide its kind again (step 3). Fix it on top of the first attempt only when it is content or test. Otherwise record it with "fix": false, and the first attempt's changes are taken back with it.
  • The row's reason and public_reason describe the whole fix, both attempts. files lists only what this attempt changed; the workflow adds the first attempt's.

The second attempt is re-run the same way, and a fix that still fails is not published.

Displaying test status on doc pages

Doc pages with passing tests display a "Verified" badge below the page title.

End to end

The following diagram shows how a test result in this repo becomes a badge on a published page. It uses kgateway's site repo (kgateway-dev/kgateway.dev) and Cloudflare Pages as the example. agentgateway follows the same steps with its own site repo and host.

flowchart LR
    subgraph dt["docs-tests repo"]
        direction TB
        ann["products/PRODUCT/annotations/<br/>and tests.yaml"]
        refresh["Refresh test annotations workflow<br/>daily 05:00 UTC"]
        refresh --> ann
    end

    subgraph site["Site repo, GitHub Actions"]
        direction TB
        run["Doc tests workflow on main<br/>daily 06:30 UTC, or started by hand"]
        attach["annotations.py attach<br/>adds test markup to a copy of the pages"]
        kind["doc_test_run.py builds a script<br/>and runs it in a fresh kind cluster"]
        artifact[("doc-test-results artifact<br/>test-results.yaml")]
        run --> attach --> kind --> artifact
    end

    dt -->|"cloned at test time"| site

    subgraph build["Site build, Cloudflare Pages"]
        direction TB
        make["make fetch-test-artifacts-build"]
        fetch["doc_test_fetch_artifacts.sh<br/>finds the latest scheduled or manual run on main"]
        inject["doc_test_inject_status.py<br/>sets test_status in each tested page's front matter"]
        hugo["hugo --gc --minify"]
        make --> fetch --> inject --> hugo
    end

    site -->|"download needs GITHUB_TOKEN"| build
    build --> page["Published page<br/>docs-theme-extras shows the Verified badge<br/>when test_status is passed"]
Loading

A few things the diagram implies:

  • Only runs on main count. The fetch script ignores pull_request runs, so the badge appears only after a scheduled or manual run on main has finished.
  • A build reads results only when it runs. A new passing test run does not change pages that are already published. The badge updates on the next site build.
  • The badge never fails the build. If GITHUB_TOKEN is missing, or no run or artifact is found, the build goes ahead without badges and prints a warning.

Pull requests

The following diagram shows what runs for a pull request in the site repo and for one in this repo. It uses the same kgateway example.

flowchart LR
    subgraph sitepr["PR in the site repo"]
        direction TB
        edit["Change a tested page<br/>or the Doc tests workflow"]
        prtest["Doc tests workflow runs on the PR<br/>with docs-tests main"]
        check["Pass or fail check on the PR<br/>results in the job summary"]
        edit --> prtest --> check
    end

    subgraph preview["Preview build, Cloudflare Pages"]
        direction TB
        pfetch["doc_test_fetch_artifacts.sh<br/>skips PR runs"]
        pbadge["Badges show the results of the<br/>latest scheduled or manual run on main"]
        pfetch --> pbadge
    end

    sitepr -->|"push to the PR branch"| preview

    subgraph dtpr["PR in docs-tests"]
        direction TB
        dedit["Change annotations, tests.yaml,<br/>or scripts"]
        unit["Tests workflow<br/>Python unit tests only"]
        dedit --> unit
    end

    subgraph manual["Optional, in the site repo"]
        direction TB
        dispatch["Start Doc tests by hand on main<br/>with docs_tests_ref set to the PR branch"]
        counts["Results saved as doc-test-results-trial,<br/>so they never reach the badge"]
        dispatch --> counts
    end

    dtpr -->|"to try the change on real pages"| manual
Loading

A few things the diagram implies:

  • The test run on a site PR never sets the badge. It only adds a pass or fail check to the PR. The preview for the PR still shows the badges from main.
  • Doc tests run on a site PR only when it touches a tested page. The workflow's paths: filter lists those pages. Widen the filter when a new scenario is added here.
  • A docs-tests PR does not run any doc tests. To try an annotation change on real pages, start the site's Doc tests workflow by hand with docs_tests_ref set to the PR branch. That run uploads its results as doc-test-results-trial instead of doc-test-results, so they never reach the badge.

How it works

  1. Test results are written to out/tests/generated/test-results.yaml after tests run.
  2. doc_test_inject_status.py reads the results and adds a test_status field to each tested document's front matter:
    • test_status: passed — all tests for the page passed
    • test_status: failed — one or more tests failed (no badge displayed)
    • test_status: partial — no tests failed, but a scenario was skipped because its test annotations did not attach (no badge displayed)
  3. docs-theme-extras renders a green "Verified" badge under the title of any page with test_status: passed. The badge is built into the module's docs page layout from v0.3.18-beta.2 on, so a consuming site needs no layout file or CSS for it, only the inject step. See Doc-test badge for details, including what to delete from a site that had its own copy of the badge.

Makefile targets (in the consuming repo)

Target Description
make deps Install Python dependencies (PyYAML)
make test-generate Generate doc test scripts without running them
make test-run Run all doc tests
make test-artifacts-fetch Fetch test artifacts from the latest main branch workflow run
make test-status Inject test status into markdown files
make fetch-test-artifacts-build Fetch artifacts, inject status, and build Hugo site
make fetch-test-artifacts-serve Fetch artifacts, inject status, and serve Hugo site locally
make test-run-build Run tests, inject status, and build Hugo site
make test-run-serve Run tests, inject status, and serve Hugo site locally

Running locally

To preview the "Verified" badges locally:

# Option 1: Fetch results from CI and serve
make fetch-test-artifacts-serve

# Option 2: Run tests locally and serve
make test-run-serve

To manually inject test status after running tests:

make test-status

This updates the markdown files in content/docs/ with the test status. The badge will appear when you run Hugo.

Fetching test artifacts

The test-artifacts-fetch target downloads test results from the most recent completed workflow run on the main branch. This requires a GITHUB_TOKEN environment variable with actions:read scope:

export GITHUB_TOKEN=<your-token>
make test-artifacts-fetch

CLI options for inject script

Flag Default Description
--repo-root . Repository root directory
--results-file out/tests/generated/test-results.yaml Path to test results file
--dry-run false Preview changes without modifying files
--quiet false Suppress verbose output

Troubleshooting

If you run into issues with installing yamltest, include the --force flag.

On macOS, you might need to run either the python3 scripts/doc_test_run.py command with sudo, or run sudo cloud-provider-kind --gateway-channel=disabled in a separate tab before running the tests. In macOS, the cloud-provider-kind tool to get a LoadBalancer IP requires elevated permissions.

Common issues

File paths differ between latest and main

When copying a test chain from main to latest (or vice versa), update every file: path in the front matter. A main chain references files under content/docs/kubernetes/main/, while latest uses content/docs/kubernetes/latest/. Using the wrong version directory causes the extractor to pull blocks from a different version's content, or fail silently if the file doesn't exist.

Wrong prerequisite file paths

The install prerequisite should point to content/docs/kubernetes/<version>/quickstart/install.md, not content/docs/kubernetes/<version>/install/helm.md or similar. Check the "Before you begin" section of the guide you're testing and follow the links to confirm the exact paths rather than guessing from memory.

Test fails immediately with "kubectl port-forward" error

Tests that contain kubectl port-forward in the generated script are automatically failed without running. Port-forwarding requires a persistent background process that doesn't work in the automated test environment. Replace any port-forward-based verification with a YAMLTest HTTP assertion using ${INGRESS_GW_ADDRESS} instead.

Wait assertions pass but HTTP test hangs then fails with read ECONNRESET

When a test creates a new HTTPRoute with a hostname that was not previously registered, agentgateway-proxy (Rust/hyper) goes through two phases before it can serve the new route. Accepted=True and ResolvedRefs=True on the HTTPRoute only reflect control plane state (~50ms) — they do not guarantee the data plane is ready.

  • Phase 1 (~120s): Proxy is unaware of the new hostname; every connection is immediately reset (< 1ms). curl --max-time 5 iterations cost ~2s each (instant failure + 2s sleep).
  • Phase 2 (last few seconds): Proxy receives xDS update and holds connections while applying it (4–26s per connection), then resets them.

Adding retries: 3 alone (without a warmup loop) multiplies the Phase 2 hang — observed total: 4 × 107s ≈ 429s.

Fix: use both a curl warmup loop (covers Phase 1) and retries: 1 on the first HTTP test entry (covers Phase 2):

{{< doc-test paths="<scenario-name>" >}}
for i in $(seq 1 60); do
  curl -s --max-time 5 -o /dev/null "http://${INGRESS_GW_ADDRESS}:80/get" -H "host: <new-hostname>" && break
  sleep 2
done
{{< /doc-test >}}

Then on the first YAMLTest HTTP assertion entry, add retries: 1. Once curl gets any HTTP response (even 404), Phase 1 is over. retries: 1 absorbs Phase 2. Max poll window: ~420 seconds.

This only applies when the feature page creates an HTTPRoute with a new hostname not in the prereq chain. Tests that update an existing prereq-chain HTTPRoute (same name/namespace) are not affected.

read ECONNRESET persists beyond the warmup window (cloud-provider-kind LB failure)

If the test already has the warmup loop + retries: 1 pattern but the HTTP assertion still fails with read ECONNRESET after 200+ seconds (the warmup curl loop never breaks out), the issue is likely a cloud-provider-kind LoadBalancer networking failure — not a data-plane warmup problem.

How to distinguish from the data-plane warmup issue: Check the controller logs in the diagnostic artifacts at out/tests/generated/context/<scenario>/pods/<controller-pod>-controller-logs.log. Look for XDS: Pushing entries with clients:1 and RDS push responses for your routes:

{"msg":"XDS: Pushing","component":"krtxds","clients":1,"version":"..."}
{"msg":"push response","component":"krtxds","type":"RDS","resources":1,...}

If the controller pushed routes to the proxy successfully, the proxy IS configured — the problem is at the network level between the LB IP and the pod.

Root cause: cloud-provider-kind assigns a LoadBalancer IP to the Service, but traffic from that IP is not properly forwarded to the pod through the Kind Docker network. The LB IP is reachable (TCP connect succeeds) but the proxy resets the connection because it never receives the forwarded packets.

Typical diagnostic evidence:

  • All pods Running with 0 restarts
  • LB IP assigned (e.g. 172.18.0.x) and Service shows 80:<nodePort>/TCP
  • HTTPRoutes show Accepted=True
  • Controller logs show successful xDS pushes with clients:1
  • Proxy log shows started bind bind="80/agentgateway-system/agentgateway-proxy"
  • curl gets ECONNRESET for the entire test duration (200+ seconds)

Resolution: This is a transient infrastructure issue. Re-running the test typically resolves it. On macOS, ensure cloud-provider-kind has proper permissions (sudo). If it recurs frequently, check Docker resource allocation (memory/CPU) for the Kind cluster.

/expect: unknown property "bodyJsonPath" errors

This error almost always means the expect block has bad indentation. bodyJsonPath must be a direct child of expect:, not nested under statusCode or headers. Double-check that all keys under expect: are at the same indentation level:

  expect:
    statusCode: 200
    bodyJsonPath:
      - path: "$.choices[0].message.content"
        comparator: contains
        value: "hello"

About

Automated tests to run on product documentation.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages