Lab is a flexible test runner designed specifically for Ferret scripts. It helps automate testing for web scraping, browser automation, API integration, and regression scenarios written in Ferret Query Language (FQL).
📢 Notice: This branch contains the upcoming Lab for Ferret v2. For the stable v1 release, please visit CLI v1.
🚀 Perfect for:
- End-to-end web application testing
- Web scraping validation and monitoring
- API integration testing
- Browser automation testing
- Regression testing for web applications
Read the introductory blog post about Lab here.
- Features
- Installation
- Quick Start
- Test Suites
- Advanced Usage
- Configuration Reference
- Architecture
- Development
- Best Practices
- Troubleshooting
- Contributing
- License
- Parallel execution - Run multiple tests concurrently for faster feedback
- Configurable concurrency - Control the number of simultaneous test executions
- Test retry mechanism - Automatically retry failed tests with customizable attempts
- Batch execution - Run tests multiple times with configurable intervals
- Built-in Ferret runtime - Execute tests using the embedded Ferret engine
- Remote HTTP runtime - Connect to remote Ferret services over HTTP/HTTPS
- Wire runtime - Execute tests through a hosted Universal Ferret API over gRPC
- Multi-runtime testing - Test against different Ferret versions or runtime configurations
- Local filesystem - Execute scripts from local directories
- Git repositories - Fetch and run tests directly from Git repositories over HTTP/HTTPS
- HTTP sources - Download and execute scripts from web URLs
- Glob pattern matching - Select multiple files using wildcard patterns
- Built-in HTTP server - Serve static files for testing web applications
- Dedicated
servecommand - Run local static and mock services without executing tests - Multiple static endpoints - Host different directories at separate URLs
- Custom aliases - Name content endpoints for better organization
- Dynamic port allocation - Automatically find available ports
- OpenAPI-compatible specs - Serve lightweight REST mocks from YAML or JSON files
- Lab mock extensions - Define responses with operation-level
x-lab-mock - Dynamic templates - Use request method, path params, query params, headers, and JSON bodies in responses
- Stable local API URLs - Expose mock endpoints to FQL scripts under
@lab.mock.<alias>
- Multiple output formats - Console and simple reporters are available
- Detailed test results - Execution metrics and timing information
- Wait conditions - Wait for external services before running tests
- Environment variable support - Configure tests via environment variables
Download the latest pre-built binaries from the releases page.
The recommended local install location is ~/.local/bin, which does not require elevated permissions.
curl -L https://github.com/MontFerret/lab/releases/latest/download/lab_linux_amd64.tar.gz | tar xz
mkdir -p ~/.local/bin
mv lab ~/.local/bin/Make sure ~/.local/bin is in your PATH:
export PATH="$HOME/.local/bin:$PATH"To make the change permanent, add the line above to your shell profile, such as ~/.bashrc, ~/.zshrc, or equivalent.
curl -L https://github.com/MontFerret/lab/releases/latest/download/lab_darwin_amd64.tar.gz | tar xz
mkdir -p ~/.local/bin
mv lab ~/.local/bin/curl -L https://github.com/MontFerret/lab/releases/latest/download/lab_darwin_arm64.tar.gz | tar xz
mkdir -p ~/.local/bin
mv lab ~/.local/bin/Make sure ~/.local/bin is in your PATH:
export PATH="$HOME/.local/bin:$PATH"If you prefer installing Lab system-wide, move the binary to /usr/local/bin:
sudo mv lab /usr/local/bin/This step may require administrator privileges.
Download the .zip file from the releases page, extract lab.exe, and place it in a directory included in your PATH.
The install script detects your operating system and architecture, downloads the matching binary, and installs it into ~/.local/bin by default.
curl -fsSL https://raw.githubusercontent.com/MontFerret/lab/main/install.sh | shTo inspect the script before running it:
curl -fsSL https://raw.githubusercontent.com/MontFerret/lab/main/install.sh -o install.sh
less install.sh
sh install.shTo install into a custom directory, set LAB_INSTALL_DIR:
LAB_INSTALL_DIR="$HOME/bin" sh install.shInstalling into system directories, such as /usr/local/bin, may require elevated permissions.
Run Lab in a container without installing it locally:
# Pull the latest image
docker pull montferret/lab:latest
# Run a simple test
docker run --rm -v $(pwd):/workspace montferret/lab:latest run /workspace/tests/
# With custom options
docker run --rm -v $(pwd):/workspace montferret/lab:latest \
run --concurrency=4 --reporter=simple /workspace/tests/The container entrypoint is Lab-aware, so docker run <image> run ... and docker run <image> version invoke the Lab binary directly, while raw commands such as docker run <image> /bin/sh -c 'echo ok' still pass through to the container shell.
Docker Compose Example:
version: '3.8'
services:
lab:
image: montferret/lab:latest
volumes:
- ./tests:/workspace/tests
- ./static:/workspace/static
command: ["run", "--serve", "/workspace/static", "/workspace/tests/"]For development or custom builds:
# Prerequisites: Go 1.23+ required
git clone https://github.com/MontFerret/lab.git
cd lab
go build -o lab .
# Or use the Makefile
make buildlab version
lab --helpThe simplest way to run Ferret scripts with Lab:
lab run is required for script execution. Root invocation such as lab tests/ or lab -f tests/ is not supported.
# Execute a single FQL script
lab run myscript.fql
# Run all FQL scripts in a directory
lab run myscripts/
# Run with increased concurrency
lab run --concurrency=4 myscripts/
# Run tests multiple times
lab run --times=3 myscript.fqlUse lab version --runtime=... when you want to inspect the version reported by a specific HTTP or Wire runtime.
Create a simple test file example.fql:
LET doc = DOCUMENT("https://www.github.com", {
driver: "cdp",
userAgent: "Lab Test Runner"
})
// Wait for page to load
WAIT_ELEMENT(doc, "header")
// Extract page title
LET title = doc.title
// Return result
RETURN {
url: doc.url,
title: title,
hasGitHubLogo: ELEMENT_EXISTS(doc, "[aria-label*='GitHub']")
}Run it:
lab run example.fqlFor browser automation, you'll need a Chrome or Chromium instance running in headless mode:
# Start Chrome in headless mode in a separate terminal
google-chrome --headless --remote-debugging-port=9222
# Run your tests using the default CDP address
lab run --cdp=http://127.0.0.1:9222 browser-tests/
# Or use a custom CDP address
lab run --cdp=http://localhost:9223 browser-tests/$ lab run example.fql
✓ example.fql (1.23s)
└─ Assertions: 1 passed, 0 failed
Tests: 1 passed, 0 failed
Time: 1.23s
Lab supports test suites defined in YAML format, enabling more complex scenarios with assertions, parameters, setup, and cleanup steps.
Use a plain .fql file for normal successful execution and in-query assertions. Use YAML when the test needs a richer execution-level expectation, such as requiring the query to return an error.
query:
text: |
LET doc = DOCUMENT("https://github.com/", { driver: "cdp" })
HOVER(doc, ".HeaderMenu-details")
CLICK(doc, ".HeaderMenu a")
WAIT_NAVIGATION(doc)
WAIT_ELEMENT(doc, '.IconNav')
FOR el IN ELEMENTS(doc, '.IconNav a')
RETURN TRIM(el.innerText)
assert:
text: RETURN T::NOT::EMPTY(@lab.data.query.result)Save as github-test.yaml and run:
lab run github-test.yamlUse expect.error when a query must fail. An empty error object accepts any error returned by the runtime:
query:
text: |
RETURN 1 / 0
expect:
error: {}Add contains to require a stable part of the error message:
query:
text: |
RETURN 1 / 0
expect:
error:
contains: "division by zero"Message matching prevents an unrelated runtime failure from accidentally satisfying the test. Only contains is supported inside expect.error; unknown fields fail during suite construction instead of falling back to an unqualified error expectation. A suite using expect.error must not define assert, because the expected query failure produces no result for an assertion script.
The legacy .fail.fql filename convention remains supported: it passes when execution returns any error and fails when execution succeeds. It is deprecated; prefer a YAML suite with expect.error for new negative tests.
Keep your FQL scripts separate and reference them in test suites.
navigation.fql:
LET doc = DOCUMENT(@url, { driver: "cdp" })
WAIT_ELEMENT(doc, "body")
RETURN doc.titlesuite.yaml:
query:
ref: ./scripts/navigation.fql
params:
url: "https://example.com"
assert:
text: |
RETURN T::NOT::EMPTY(@lab.data.query.result)
AND T::CONTAINS(@lab.data.query.result, "Example")name: "E-commerce User Journey"
description: "Test complete user purchase flow"
setup:
text: |
LET baseUrl = "https://demo-shop.example.com"
RETURN { baseUrl }
query:
text: |
LET doc = DOCUMENT(@lab.data.setup.result.baseUrl, { driver: "cdp" })
// Navigate to product
CLICK(doc, ".product-item:first-child a")
WAIT_NAVIGATION(doc)
// Add to cart
CLICK(doc, ".add-to-cart")
WAIT_ELEMENT(doc, ".cart-confirmation")
// Go to checkout
CLICK(doc, ".checkout-btn")
WAIT_NAVIGATION(doc)
RETURN {
currentUrl: doc.url,
cartItems: LENGTH(ELEMENTS(doc, ".cart-item")),
totalPrice: INNER_TEXT(doc, ".total-price")
}
assert:
text: |
LET result = @lab.data.query.result
RETURN T::CONTAINS(result.currentUrl, "checkout")
AND result.cartItems > 0
AND T::NOT::EMPTY(result.totalPrice)
cleanup:
text: |
// Clear cart or perform cleanup
RETURN "Cleanup completed"Create reusable test suites with parameters:
query:
text: |
LET doc = DOCUMENT(@testUrl, {
driver: "cdp",
timeout: @pageTimeout
})
WAIT_ELEMENT(doc, @selector)
RETURN {
title: doc.title,
elementExists: ELEMENT_EXISTS(doc, @selector)
}
assert:
text: |
LET result = @lab.data.query.result
RETURN result.elementExists == trueRun with parameters:
lab run --param=testUrl:"https://example.com" \
--param=pageTimeout:5000 \
--param=selector:"h1" \
test-suite.yamlUse --param-bind when a script should keep a production-oriented parameter
while Lab supplies its value. The target is a dot-separated user parameter path;
the source starts with @ and must already exist when Lab finishes setup:
lab run tests/ \
--param='environment:{"api":{"baseUrl":"https://example.com"}}' \
--param-bind config.api.baseUrl=@environment.api.baseUrlBindings preserve the source value's type. Duplicate or overlapping binding
targets, conflicts with --param, malformed paths, and missing sources are
configuration errors rather than runtime precedence rules. Paths are
case-sensitive and dot-separated; each segment starts with a letter or
underscore and may then contain letters, numbers, underscores, or hyphens.
Bracket expressions are not accepted, and the lab target namespace is
reserved.
Use external data sources for broader test coverage:
query:
text: |
LET testData = [
{ url: "https://site1.com", expectedTitle: "Site 1" },
{ url: "https://site2.com", expectedTitle: "Site 2" }
]
FOR test IN testData
LET doc = DOCUMENT(test.url, { driver: "cdp" })
WAIT_ELEMENT(doc, "title")
RETURN {
url: test.url,
expectedTitle: test.expectedTitle,
actualTitle: doc.title,
matches: doc.title == test.expectedTitle
}
assert:
text: |
FOR result IN @lab.data.query.result
FILTER result.matches != true
RETURN false
RETURN trueLab supports multiple source locations for maximum flexibility.
# Single file
lab run /path/to/test.fql
# Directory with glob patterns
lab run "tests/**/*.fql"
lab run tests/integration/
# Multiple paths
lab run --files=tests/unit/ --files=tests/integration/ --files=scripts/smoke.fqlFetch and execute tests directly from Git repositories:
# HTTPS Git repository
lab run git+https://github.com/username/test-repo.git//tests/
# HTTP Git repository
lab run git+http://git.example.com/tests.git//integration/
# Specific branch or tag
lab run git+https://github.com/username/tests.git@v1.2.0//suite.yaml
# Private repositories using token-based authentication
lab run git+https://username:token@github.com/private/repo.git//tests/Download scripts from web URLs:
# Direct script URL
lab run https://raw.githubusercontent.com/user/repo/main/test.fql
# Multiple HTTP sources
lab run https://example.com/tests/suite1.yaml https://example.com/tests/suite2.yamlLab can serve local directories over HTTP during test execution. Bind a generated endpoint to an ordinary script parameter when the script should remain unaware of Lab.
Use lab serve --static when you want to expose local directories over HTTP without running a test suite.
# Serve a single directory
lab serve --static ./website
# Serve multiple directories with aliases
lab serve --static ./frontend@app --static ./mockdata@apilab run tests/ \
--serve ./tests/fixtures@fixtures \
--param-bind baseUrl=@lab.static.fixtures \
--policy-http-allow-localhostFQL script:
LET response = IO::NET::HTTP::GET(@baseUrl + "/products.json")Lab resolves the binding after the fixture server has selected its port and before test execution starts. Scripts that deliberately depend on Lab can continue to use @lab.static.<alias> directly.
LET response = IO::NET::HTTP::GET(@lab.static.fixtures + "/products.json")lab run --policy-http-allow-localhost --serve ./app --serve ./api-mocks tests/FQL script:
LET appPage = DOCUMENT(@lab.static.app, { driver: "cdp" })
LET apiData = IO::NET::HTTP::GET(@lab.static["api-mocks"] + "/users.json")lab run --policy-http-allow-localhost --serve ./frontend@app --serve ./mockdata@api tests/FQL script:
LET homePage = DOCUMENT(@lab.static.app + "/index.html", { driver: "cdp" })
LET userData = IO::NET::HTTP::GET(@lab.static.api + "/user/123.json")lab run \
--policy-http-allow-localhost \
--serve ./dist@webapp \
--serve ./test-fixtures@fixtures \
--serve ./mock-apis@mocks \
--concurrency=3 \
tests/e2e/When Lab runs against a remote runtime or inside a containerized environment, use --serve-host to advertise a client-reachable host instead of loopback. Use --serve-bind when you need to control the listener interface explicitly.
lab run \
--runtime=https://ferret.example.com \
--serve ./dist@app \
--serve-host host.docker.internal \
--serve-bind 0.0.0.0 \
tests/If --serve-host is set without --serve-bind, Lab automatically binds static servers to all interfaces: 0.0.0.0 for IPv4 and hostnames, or :: for IPv6 literals.
Lab can serve local REST mock APIs from OpenAPI-compatible YAML or JSON specs during test execution. Mock endpoints are available in FQL scripts under @lab.mock.<alias>.
Mock specs use operation-level x-lab-mock blocks:
openapi: 3.1.0
info:
title: Users API
version: 1.0.0
paths:
/users/{id}:
get:
x-lab-mock:
status: 200
body:
id: "{{ .Path.id }}"
name: "User {{ .Path.id }}"Run tests with the mock API:
lab run --policy-http-allow-localhost --mock ./users.yaml@api tests/Ferret's built-in HTTP client blocks localhost by default. Tests that access @lab.static or @lab.mock through IO::NET::HTTP must opt in with --policy-http-allow-localhost (or LAB_POLICY_HTTP_ALLOW_LOCALHOST=true). Starting a local service does not broaden the HTTP policy automatically.
Serve the mock API without running tests:
lab serve --mock ./users.yaml@apiStatic and mock services can be served together:
lab serve --static ./dist@app --mock ./users.yaml@apiFQL script:
LET payload = ENCODING::JSON_PARSE(TO_STRING(IO::NET::HTTP::GET(@lab.mock.api + "/users/123")))
RETURN T::EQ(payload.id, "123")Mock API entries use the same binding syntax as static serving: <path>, <path>:<port>, <path>@<alias>, and <path>@<alias>:<port>. --serve-bind and --serve-host also apply to mock API servers.
The built-in runtime exposes FQL filesystem functions through a sandbox rooted at Lab's current working directory. Use --policy-fs-root to select a narrower relative or absolute root, and add --policy-fs-read-only to permit reads while rejecting writes, directory changes, and removals.
lab run \
--policy-fs-root=./fixtures \
--policy-fs-read-only \
tests/The equivalent environment variables are LAB_POLICY_FS_ROOT and LAB_POLICY_FS_READ_ONLY.
Lab's built-in, HTTP, and Wire adapters implement api.Runtime from the Universal Ferret API. Lab uses one-shot Run by default. With --times > 1 or --attempts > 1, it automatically reuses successfully compiled plans within each test case when the runtime supports them.
Every invocation uses fresh parameters, sessions, and outputs. Query and assertion plans compile on first use; changed referenced script names or content select a new plan. HTTP transparently falls back to one-shot execution.
Lab closes outputs and sessions after each invocation and all cached plans before reporting the case, without closing the shared runtime. Cleanup failures remain test failures even if a later retry succeeds and cannot satisfy expected-error tests. Go callers using compilation directly own and close their plans and sessions.
Lab can execute tests against remote Ferret instances instead of using the built-in runtime.
# Connect to remote Ferret service
lab run --runtime=https://ferret.example.com/api tests/
# With custom headers and path
lab run \
--runtime=https://ferret.example.com \
--runtime-option='headers:{"Authorization":"Bearer token123"}' \
--runtime-option='path:"/v1/execute"' \
--param='value:"shared"' \
tests/When the runtime URL already includes a path, Lab sends run requests to that exact path. The optional runtime-option=path value overrides the run endpoint only. lab version --runtime=... uses the runtime URL path and appends /info (/api/info for a runtime URL ending in /api).
Lab rejects --policy-fs-* and --policy-http-* options for HTTP runtimes because their request protocol has no policy contract. The built-in runtime supports these options.
The HTTP runtime sends POST requests with:
{
"text": "FQL script content",
"params": {
"key": "value"
}
}A successful HTTP run returns a one-shot output that streams the response body. Collection preserves the response content type and any bytes received before a read or cleanup error. Suites decode completed query output as JSON. The HTTP protocol supports execution and version reporting; UAPI compilation, filesystem-root, and output-codec options are rejected without sending requests.
Wire exposes the Universal Ferret API over gRPC. The runtime target includes the complete connection endpoint:
lab run --runtime=wire+tcp://127.0.0.1:50051 tests/
lab run \
--runtime=wire+tcp://127.0.0.1:50051 \
--runtime-option='connect-timeout:"10s"' \
--param='value:"shared"' \
tests/
lab version \
--runtime=wire+tcp://127.0.0.1:50051 \
--runtime-option='connect-timeout:"10s"'
LAB_RUNTIME=wire+tcp://127.0.0.1:50051 \
LAB_RUNTIME_OPTION='connect-timeout:"10s"' \
lab run tests/Start a Wire host separately. The target must be wire+tcp://127.0.0.1:<port> with a port from 1 to 65535. Hostnames, IPv6, other addresses, paths, credentials, queries, and fragments are not supported. Plaintext loopback transport is intended for trusted local development, including a host exposed through a local port mapping. The URI scheme adds no encryption or authentication.
The connect-timeout runtime option defaults to "5s" and must be a positive duration string. It bounds connection and handshake setup; --timeout continues to bound each test. Both run and version accept the same runtime target and options.
Lab preserves source identity/content, uses Wire's one-shot Runtime.Run by default, and reuses compiled plans for repetitions or retries. Compiled sessions receive the same nested YAML parameter conversion and validation as one-shot execution. The host must support chunked output framing (Wire v1.0.0-alpha.3). Lab consumes each output before marking an attempt complete, so a terminal failure after the first chunk still fails the attempt. lab version reports the hosted api.Runtime.Version exactly, captured during the handshake.
Lab owns the gRPC transport and closes the logical Wire runtime before that transport. Setup failures also release the transport. Transport loss requires a new Lab invocation; the adapter does not redial or switch runtimes.
The host configures Ferret and its execution policies. Wire therefore rejects Lab's --policy-fs-* and --policy-http-* options. Only connect-timeout is a Wire adapter option. FQL inputs, including headers, cookies, path, and flags, use --param, bindings, or YAML query/assertion parameters. HTTP/Worker support remains separate.
Test against multiple runtime versions:
# Test with built-in runtime
lab run tests/ > builtin-results.txt
# Test with remote runtime
lab run --runtime=https://ferret-v0.17.example.com tests/ > remote-v0.17-results.txt
# Compare results
diff builtin-results.txt remote-v0.17-results.txt# Run up to 8 tests simultaneously
lab run --concurrency=8 tests/
# Balance speed and resource usage
lab run --concurrency=4 --timeout=60 large-test-suite/# Run each test 3 times for reliability testing
lab run --times=3 tests/flaky/
# Retry failed tests up to 2 additional times
lab run --attempts=3 tests/
# Add delay between test cycles
lab run --times=5 --times-interval=10 stress-tests/# Wait for services to be available before running tests
lab run \
--wait=http://127.0.0.1:9222/json/version \
--wait=postgres://localhost:5432/testdb \
--wait-timeout=30 \
tests/integration/These flags apply to lab run.
| Flag | Short | Environment Variable | Default | Description |
|---|---|---|---|---|
--files |
-f |
LAB_FILES |
- | Location of FQL script files to run |
--timeout |
-t |
LAB_TIMEOUT |
30 |
Test timeout in seconds |
--cdp |
- | LAB_CDP |
http://127.0.0.1:9222 |
Chrome DevTools Protocol address |
--reporter |
- | LAB_REPORTER |
console |
Output reporter: console, simple |
--runtime |
-r |
LAB_RUNTIME |
Built-in | Complete runtime target: builtin, HTTP/HTTPS URL, or wire+tcp://127.0.0.1:<port> |
--runtime-option |
- | LAB_RUNTIME_OPTION |
- | Repeatable adapter configuration: name:<JSON value> |
--concurrency |
-c |
LAB_CONCURRENCY |
1 |
Number of parallel test executions |
--times |
- | LAB_TIMES |
1 |
Number of times to run each test |
--attempts |
-a |
LAB_ATTEMPTS |
1 |
Number of retry attempts for failed tests |
--times-interval |
- | LAB_TIMES_INTERVAL |
0 |
Interval between test cycles in seconds |
--serve |
- | LAB_SERVE |
- | Served directory mapping exposed over HTTP |
--mock |
- | LAB_MOCK |
- | OpenAPI mock API spec exposed over HTTP |
--serve-bind |
- | LAB_SERVE_BIND |
- | Host to bind local servers to, without port |
--serve-host |
- | LAB_SERVE_HOST |
- | Host to advertise local server URLs, without port |
--param |
-p |
LAB_PARAM |
- | Query parameters for tests |
--param-bind |
- | LAB_PARAM_BIND |
- | Bind a user parameter path to an existing parameter reference |
--wait |
-w |
LAB_WAIT |
- | Wait for resource availability |
--wait-timeout |
--wt |
LAB_WAIT_TIMEOUT |
5 |
Wait timeout in seconds |
--wait-attempts |
- | LAB_WAIT_ATTEMPTS |
5 |
Number of wait attempts |
--policy-fs-root |
- | LAB_POLICY_FS_ROOT |
Current working directory | Filesystem root for the built-in runtime |
--policy-fs-read-only |
- | LAB_POLICY_FS_READ_ONLY |
false |
Make the built-in runtime filesystem read-only |
--policy-http-allowed-schemes |
- | LAB_POLICY_HTTP_ALLOWED_SCHEMES |
http,https |
Allowed outbound HTTP URL schemes |
--policy-http-allowed-methods |
- | LAB_POLICY_HTTP_ALLOWED_METHODS |
GET,HEAD,POST,PUT,PATCH,DELETE,OPTIONS |
Allowed outbound HTTP methods |
--policy-http-allowed-hosts |
- | LAB_POLICY_HTTP_ALLOWED_HOSTS |
- | Allowed exact hosts or host:port values |
--policy-http-blocked-hosts |
- | LAB_POLICY_HTTP_BLOCKED_HOSTS |
- | Blocked exact hosts or host:port values |
--policy-http-allow-localhost |
- | LAB_POLICY_HTTP_ALLOW_LOCALHOST |
false |
Allow localhost and loopback addresses |
--policy-http-allow-private-networks |
- | LAB_POLICY_HTTP_ALLOW_PRIVATE_NETWORKS |
false |
Allow private network addresses |
--policy-http-allow-link-local |
- | LAB_POLICY_HTTP_ALLOW_LINK_LOCAL |
false |
Allow link-local addresses |
--policy-http-default-headers |
- | LAB_POLICY_HTTP_DEFAULT_HEADERS |
- | Default request headers as a JSON string map |
--policy-http-blocked-request-headers |
- | LAB_POLICY_HTTP_BLOCKED_REQUEST_HEADERS |
- | Blocked request header names |
--policy-http-timeout |
- | LAB_POLICY_HTTP_TIMEOUT |
30s |
Overall HTTP timeout |
--policy-http-no-timeout |
- | LAB_POLICY_HTTP_NO_TIMEOUT |
false |
Explicitly disable the overall HTTP timeout |
--policy-http-max-request-size |
- | LAB_POLICY_HTTP_MAX_REQUEST_SIZE |
16777216 |
Maximum request body size in bytes |
--policy-http-unlimited-request-size |
- | LAB_POLICY_HTTP_UNLIMITED_REQUEST_SIZE |
false |
Explicitly disable the request body limit |
--policy-http-max-response-size |
- | LAB_POLICY_HTTP_MAX_RESPONSE_SIZE |
16777216 |
Maximum response body size in bytes |
--policy-http-unlimited-response-size |
- | LAB_POLICY_HTTP_UNLIMITED_RESPONSE_SIZE |
false |
Explicitly disable the response body limit |
--policy-http-max-response-header-size |
- | LAB_POLICY_HTTP_MAX_RESPONSE_HEADER_SIZE |
1048576 |
Maximum response header size in bytes |
--policy-http-follow-redirects |
- | LAB_POLICY_HTTP_FOLLOW_REDIRECTS |
true |
Follow HTTP redirects |
--policy-http-max-redirects |
- | LAB_POLICY_HTTP_MAX_REDIRECTS |
10 |
Maximum redirects to follow |
These flags apply to lab serve.
| Flag | Environment Variable | Default | Description |
|---|---|---|---|
--static |
LAB_STATIC |
- | Served directory mapping exposed over HTTP |
--mock |
LAB_MOCK |
- | OpenAPI mock API spec exposed over HTTP |
--serve-bind |
LAB_SERVE_BIND |
- | Host to bind local servers to, without port |
--serve-host |
LAB_SERVE_HOST |
- | Host to advertise local server URLs, without port |
Set environment variables for consistent configuration across environments:
# Basic configuration
export LAB_TIMEOUT=60
export LAB_CONCURRENCY=4
export LAB_REPORTER=simple
# CDP configuration
export LAB_CDP=http://chrome-headless:9222
# Runtime configuration
export LAB_RUNTIME=https://ferret-api.example.com
export LAB_RUNTIME_OPTION='headers:{"API-Key":"secret123"}'
# Run tests
lab run tests/#!/bin/bash
# ci-test.sh
# Set CI-friendly defaults
export LAB_TIMEOUT=120
export LAB_CONCURRENCY=2
export LAB_REPORTER=simple
export LAB_ATTEMPTS=3
# Wait for services
lab run \
--wait=http://app:3000/health \
--wait=postgres://db:5432/testdb \
--wait-timeout=60 \
tests/integration/#!/bin/bash
# dev-test.sh
export LAB_CDP=http://localhost:9222
export LAB_TIMEOUT=30
export LAB_CONCURRENCY=1
# Serve local assets and run tests
lab run \
--serve ./dist@app \
--serve ./fixtures@data \
tests/dev/#!/bin/bash
# load-test.sh
lab run \
--concurrency=20 \
--times=100 \
--times-interval=1 \
--timeout=10 \
tests/performance/Configure remote Ferret runtime behavior:
# HTTP runtime with custom headers
lab run \
--runtime=https://ferret.api.com \
--runtime-option='headers:{"Authorization":"Bearer token"}' \
--runtime-option='path:"/v2/execute"' \
tests/Runtime options configure adapters only. Unsupported names, wrong-adapter options, invalid types, and invalid values fail before external setup. An omitted or empty runtime target, or builtin, selects the built-in runtime. Unknown targets and schemes fail; a bare wire or tcp:// target does not select Wire.
| Adapter | Option | JSON value |
|---|---|---|
| Built-in | None | Adapter options are rejected. |
| HTTP/HTTPS | headers |
Object of strings, added to request headers. |
| HTTP/HTTPS | cookies |
Object of strings, sent as request cookies. |
| HTTP/HTTPS | path |
String URL reference overriding execution only; empty retains the runtime URL path. |
| Wire TCP | connect-timeout |
Positive duration string; default "5s", connection and handshake only. |
Values use the existing name:<JSON value> syntax. Quote the entire shell argument to preserve JSON strings, spaces, colons, and punctuation. Repeat the flag for multiple options, or separate entries with commas outside JSON objects, arrays, and strings. LAB_RUNTIME_OPTION accepts the same comma-separated entries. Any explicit --runtime-option replaces the environment option list. Later duplicate keys replace the whole previous value, including objects; they do not merge.
FQL inputs use the existing parameter pipeline across all adapters:
lab run tests/
lab run --param='value:"shared"' tests/
lab version
lab version \
--runtime=https://ferret.example.com/api \
--runtime-option='headers:{"Authorization":"Bearer token123"}'--param values use last-key-wins handling, and CLI flags replace values from LAB_PARAM. Bindings resolve after local service setup from one parameter snapshot; duplicate, overlapping, or reserved targets fail. YAML query and assertion parameters override the common user values independently. Assertion @lab.data.query retains the query result and effective query parameters. @lab remains reserved for Lab system values. Ordinary FQL parameters named headers or path do not configure HTTP, and adapter options are never injected as FQL inputs.
This intentionally replaces the earlier v2 runtime interface. Removed flags fail argument validation, and obsolete environment variables no longer configure Lab.
| Previous usage | Replacement |
|---|---|
--runtime=wire --runtime-endpoint=tcp://host:port |
--runtime=wire+tcp://host:port (existing loopback restriction applies) |
--runtime-connect-timeout=10s |
--runtime-option='connect-timeout:"10s"' |
--runtime-param='headers:{...}' |
--runtime-option='headers:{...}' |
--runtime-param='path:"/v1/execute"' |
--runtime-option='path:"/v1/execute"' |
FQL input through --runtime-param |
Existing --param, bindings, or suite/query parameter facilities |
LAB_RUNTIME_PARAM containing adapter settings |
LAB_RUNTIME_OPTION |
LAB_RUNTIME_ENDPOINT |
Include the endpoint in LAB_RUNTIME=wire+tcp://127.0.0.1:<port> |
LAB_RUNTIME_CONNECT_TIMEOUT=10s |
LAB_RUNTIME_OPTION='connect-timeout:"10s"' |
--rp |
Removed; use --runtime-option for adapter configuration |
The separate runtime FQL defaults layer has been removed. A configuration combining runtime defaults and per-run values may need to be expressed using existing CLI, binding, or query/assertion parameter facilities; there is no guaranteed lossless one-for-one migration of that layer. No new defaults layer is introduced. External CLI binary execution remains unsupported.
Lab is built with a modular architecture that separates source resolution, test orchestration, runtime execution, local static and mock serving, and reporting.
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Test Sources │ │ Test Runner │ │ Ferret │
│ │ │ │ │ Runtime │
│ • File System │───▶│ • Orchestration │───▶│ │
│ • Git Repos │ │ • Parallelism │ │ • Built-in │
│ • HTTP URLs │ │ • Retry Logic │ │ • Remote HTTP │
└─────────────────┘ │ • Reporting │ │ • External Bin │
└─────────────────┘ └─────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Static Server │ │ API Mock Server │
│ │ │ │
│ • Served Dirs │ │ • OpenAPI Specs │
│ • @lab.static │ │ • x-lab-mock │
│ • Auto Ports │ │ • @lab.mock │
└─────────────────┘ └─────────────────┘
Handles fetching test files from various locations:
- FileSystem Source - Local directory and file access with glob pattern support
- Git Source - Clone and fetch files from Git repositories over HTTP/HTTPS
- HTTP Source - Download scripts from web URLs
- Aggregate Source - Combine multiple source types
Manages Ferret script execution:
- Built-in Runtime - Owns Ferret through
uapi.New, preserving embedded version injection and execution policies - Remote Runtime - Communicates with remote Ferret services over HTTP
- Wire Runtime - Uses the hosted Universal Ferret API over gRPC
Orchestrates test execution:
- Parallel Processing - Manages concurrent test execution
- Retry Mechanism - Handles failed test retries
- Resource Management - Controls timeouts and resource allocation
- Lifecycle Management - Handles setup, execution, assertion, and cleanup phases
Built-in HTTP server for local static assets:
- Multiple endpoints - Serve multiple directories simultaneously
- Dynamic ports - Allocate free ports automatically
- Alias support - Assign stable names to served directories
OpenAPI-compatible mock REST API server:
- Spec-driven routes - Register endpoints from
x-lab-mockoperations - Template responses - Render body values and raw templates from request context
- Local lifecycle - Reuses shared local server binding and endpoint management
Shared local HTTP server runtime:
- Host settings - Normalizes bind and advertised hosts
- Entry parsing - Handles path, alias, and port bindings
- Lifecycle management - Starts, stops, and reports endpoint URLs
Output formatting and result presentation:
- Console Reporter - Rich output for interactive use
- Simple Reporter - Plain text output suitable for CI/CD
Test suite definition and validation:
- YAML Parser - Parse test suite definitions
- Parameter Injection - Handle user/system parameters and setup-time parameter rebinding
- Assertion Engine - Validate test results
- Input Processing - Parse command-line arguments and environment variables
- Source Resolution - Fetch test files from configured sources
- Local Service Initialization - Start static and mock servers, if configured
- Runtime Setup - Initialize Ferret runtime, built-in, HTTP remote, or Wire
- Test Discovery - Find and parse test files and suites
- Parallel Execution - Run tests according to concurrency settings
- Result Collection - Gather execution results and timing data
- Reporting - Format and output results via the selected reporter
- Cleanup - Stop local servers and release resources
- Modularity - Each component has a focused responsibility
- Extensibility - New source types, runtimes, and reporters can be added without changing the whole system
- Performance - Parallel execution and resource-conscious orchestration
- Reliability - Retry support, wait conditions, and explicit timeouts
- Flexibility - Works with local, remote, and containerized Ferret runtimes
Prerequisites:
- Go 1.23 or later
- Git
Build Steps:
# Clone the repository
git clone https://github.com/MontFerret/lab.git
cd lab
# Install development tools
make install-tools
# Build the project
make build
# Or manually
go build -o bin/lab -ldflags "-X main.version=dev" ./main.goDevelopment Workflow:
# Run tests
make test
# Or:
go test ./...
# Format code
make fmt
# Lint code
make lint
# Run all checks
make build# Run unit tests
go test -v ./...
# Run specific package tests
go test -v ./sources/...
go test -v ./runtime/...
# Run tests with coverage
make coverlab/
├── main.go # Application entry point
├── cmd/ # CLI command implementations
├── localserver/ # Shared local HTTP server lifecycle
├── mockserver/ # OpenAPI-compatible mock API server
├── staticserver/ # Static file server
├── reporters/ # Output formatters
├── runner/ # Test execution orchestration
├── runtime/ # Ferret runtime implementations
├── sources/ # Test file source handlers
├── testing/ # Test suite definitions
├── assets/ # Documentation assets
├── Dockerfile # Container build definition
├── Makefile # Build automation
└── README.md # Project documentation
- Implement the
Sourceinterface insources/. - Add URL scheme handling in
sources/source.go. - Add tests in
sources/.
- Implement the Universal API's
api.Runtimeinpkg/runtime/, including explicit unsupported-operation errors where the integration lacks a capability. - Add runtime type detection in
pkg/runtime/runtime.go. - Add configuration handling.
- Implement the
Reporterinterface inreporters/. - Register the reporter in CLI flags.
- Add output format tests.
tests/
├── unit/ # Unit tests for individual components
│ ├── api/
│ └── ui/
├── integration/ # Integration tests
│ ├── user-flows/
│ └── data-validation/
├── e2e/ # End-to-end tests
│ ├── critical-path/
│ └── smoke/
├── fixtures/ # Test data and assets
│ ├── pages/
│ └── data/
└── scripts/ # Reusable FQL scripts
├── common/
└── helpers/
- Use descriptive test names, such as
user-registration-flow.yaml. - Prefix test types when useful, such as
smoke-,regression-, orload-. - Use kebab-case for files, such as
checkout-process.fql.
name: "User Authentication Flow"
description: "Verify user login, logout, and session management"
setup:
text: |
// Clear any existing sessions
// Set up test data
query:
text: |
// Main test logic with clear comments
assert:
text: |
// Specific, meaningful assertions
cleanup:
text: |
// Clean up test data# Local development: low concurrency
lab run --concurrency=2 tests/
# CI environments: medium concurrency
lab run --concurrency=4 tests/
# Dedicated test infrastructure: higher concurrency
lab run --concurrency=8 tests/- Use appropriate timeouts for different test types.
- Implement cleanup in test suites.
- Monitor memory usage with large test suites.
- Use the static server for shared static assets.
# Run faster tests first
lab run tests/smoke/ && lab run tests/integration/ && lab run tests/e2e/
# Use separate paths for different test categories
lab run --timeout=60 tests/critical/
lab run --timeout=300 --concurrency=1 tests/extended/- Never commit sensitive data in test files.
- Use environment variables for credentials.
- Sanitize test outputs that might contain secrets.
- Use separate test environments for security testing.
# Good: use environment variables
export TEST_API_KEY="your-key-here"
lab run --param=apiKey:$TEST_API_KEY tests/
# Bad: hardcode secrets in scripts
# LET apiKey = "secret-key-123"Error: Failed to connect to CDP at http://127.0.0.1:9222
Solutions:
-
Start Chrome in headless mode:
google-chrome --headless --remote-debugging-port=9222 --no-sandbox
-
Check if Chrome is running:
curl http://127.0.0.1:9222/json/version
-
Use a custom CDP address:
lab run --cdp=http://localhost:9223 tests/
Error: Test timed out after 30 seconds
Solutions:
-
Increase timeout:
lab run --timeout=60 tests/
-
Optimize test scripts:
-- Add explicit waits WAIT_ELEMENT(doc, ".loading", { displayed: false }) -- Use shorter timeouts for quick checks WAIT_ELEMENT(doc, ".button", { timeout: 5000 })
Error: Failed to clone repository
Solutions:
-
Check repository URL:
git clone https://github.com/user/repo.git
-
Use token-based authentication for private repositories:
lab run git+https://username:token@github.com/private/repo.git//tests/
-
Use SSH for private repositories:
lab run git+ssh://git@github.com/private/repo.git//tests/
Error: failed to start static file server on port 8080
Solutions:
-
Lab normally finds free ports automatically, but you can specify a port if needed:
lab run --serve ./static@app:8081 tests/
-
Check for port conflicts:
netstat -tlnp | grep :8080
- Reduce concurrency with
--concurrency=2. - Implement cleanup in tests.
- Use a separately hosted HTTP or Wire runtime for memory-intensive tests.
- Enable parallel execution with
--concurrency=4. - Use the static server for local assets.
- Optimize FQL scripts.
- Profile tests to identify bottlenecks.
# Enable detailed logging, if supported by the current build
export LOG_LEVEL=debug
lab run tests/
# Use the simple reporter for cleaner output
lab run --reporter=simple tests/# Test one file at a time
lab run specific-test.fql
# Run with retries disabled
lab run --attempts=1 problematic-test.fql# Test FQL syntax with Ferret CLI
ferret -q "RETURN 1"We welcome contributions to Lab. To get started:
- Fork the repository on GitHub.
- Create a feature branch:
git checkout -b feature/awesome-feature. - Make your changes and add tests.
- Run the test suite:
make test. - Commit your changes:
git commit -am 'Add awesome feature'. - Push to the branch:
git push origin feature/awesome-feature. - Submit a pull request.
- Write tests for new features.
- Follow Go conventions and formatting with
make fmt. - Pass linting checks with
make lint. - Update documentation for user-facing changes.
- Keep commits atomic and write clear commit messages.
When reporting bugs, please include:
- Lab version from
lab version - Operating system and version
- Go version, if building from source
- Complete command that failed
- Full error message and stack trace
- Minimal reproduction case
Before requesting features:
- Check existing issues and discussions.
- Describe the use case and problem being solved.
- Consider whether it fits Lab's scope.
- Share examples of the desired behavior when possible.
- Discussion - Major features should be discussed in issues first.
- Implementation - Write code with tests and documentation.
- Review - Submit a pull request for code review.
- Testing - Ensure all CI checks pass.
- Merge - A maintainer will merge when ready.
Lab is licensed under the Apache License 2.0.
Happy Testing! 🚀
For more information about Ferret and FQL, visit the Ferret documentation.
Join the community on Discord for support and discussions.