From 004d28318eb553d7e89a067f532ba2bfb9b3d3ad Mon Sep 17 00:00:00 2001 From: georgiedekker Date: Sat, 14 Nov 2026 23:53:01 +0100 Subject: [PATCH] docs(readme): make every claim true and current Co-Authored-By: Claude Code --- README.md | 275 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 195 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index 494a999..c86812c 100644 --- a/README.md +++ b/README.md @@ -1,41 +1,65 @@ # python-test-executor -Executes arbitrary Python code and analyzes it for quality issues. Part of the **code-graph-helper** microservices system, this service receives Python source code via HTTP, runs it in a subprocess with a configurable timeout, and returns the output. It also includes a result processor that parses output from static analysis tools (ruff, ty, ruff format, bandit, eslint, prettier, vulture, radon) into structured quality issue reports. +[![CI](https://github.com/nonagenticai/python-test-executor/actions/workflows/ci.yml/badge.svg)](https://github.com/nonagenticai/python-test-executor/actions/workflows/ci.yml) +[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE) -**Port:** 8087 +Executes Python code over HTTP and, when test code is supplied, runs it under pytest. Part of +the **code-graph-helper** microservices system, it receives Python source via HTTP, runs it in a +subprocess with a configurable timeout, and returns the captured output and exit code. It also +contains a library component that parses output from eight static-analysis tools into structured +quality-issue objects. -## Architecture +| | | +|---|---| +| **Runtime** | Python 3.14 (`python:3.14-slim`; `pyproject.toml` requires `>=3.14`) | +| **Depends on** | Nothing at runtime: no database, no Redis connection, no outbound service calls | +| **Runs on** | A container listening on `8087`. CI has no deploy job; the service is retired from the estate cluster. | -``` - ┌─────────────────────────┐ - POST /execute ──────► │ FastAPI (test_service) │ - │ - write code to /tmp │ - │ - py_compile check │ - │ - subprocess exec │ - │ - timeout + cleanup │ - └─────────────────────────┘ +## Quick start - ResultProcessor (library, not exposed via API) - - parses stdout/stderr from: ruff, ty, ruff-format, - bandit, eslint, vulture, radon, prettier - - returns list[QualityServiceIssue] +```bash +git clone https://github.com/nonagenticai/python-test-executor.git +cd python-test-executor +docker compose up -d +curl http://localhost:8087/health ``` -**Key components:** +## Usage -| File | Purpose | -|------|---------| -| `src/test_service.py` | FastAPI app — `/health`, `/execute`, `/` endpoints | -| `src/result_processor.py` | Parses output from 9 quality/linting tools into `QualityServiceIssue` objects | -| `src/quality_models.py` | Pydantic model for `QualityServiceIssue` (tool, severity, message, line, column, rule_id, fixable) | +`/` and `/health` are open. Every other route — including `/execute`, `/docs`, `/redoc` and +`/openapi.json` — requires `X-Tenant-Id`, the access marker the orchestrator injects; without it +the service answers `401 {"detail":"X-Tenant-Id header required"}`. This is a lightweight +marker, **not** authentication and not data tenancy — the real isolation control is the cluster +NetworkPolicy that admits only the orchestrator. + +```bash +curl -X POST http://localhost:8087/execute \ + -H 'Content-Type: application/json' \ + -H 'X-Tenant-Id: coding-testing' \ + -d '{"code": "print(\"hello world\")", "timeout": 30}' +``` + +```json +{ + "success": true, + "output": "hello world\n", + "execution_time": 0.05, + "test_id": "0193...", + "exit_code": 0, + "error": null, + "tests_run": 0, + "tests_passed": 0, + "tests_failed": 0, + "no_tests_run": false +} +``` ## API ### `GET /` -Service info and available endpoints. +Service info and the endpoint list. No header required. -**Response:** ```json { "service": "Python Test Executor", @@ -47,9 +71,8 @@ Service info and available endpoints. ### `GET /health` -Health check with uptime and memory usage. +Health check with uptime and memory usage. No header required. -**Response:** ```json { "status": "healthy", @@ -61,53 +84,57 @@ Health check with uptime and memory usage. ### `POST /execute` -Execute Python code and return the result. - -**Request:** -```json -{ - "code": "print('hello world')", - "timeout": 30 -} -``` +Execute Python code. Requires `X-Tenant-Id`. | Field | Type | Default | Description | |-------|------|---------|-------------| -| `code` | string | required | Python source code to execute | +| `code` | string | required | Python source code | +| `test_code` | string \| null | `null` | Optional pytest tests. When supplied, `code` and `test_code` are concatenated into one module and run under pytest; a run that collects zero tests is **not** a pass. When omitted, `code` is executed as a plain script (success = exit code 0). | | `timeout` | int | 30 | Max execution time in seconds | -**Response:** (canonical executor contract — matches the peer go/js/java/rust/csharp/cpp executors) -```json -{ - "success": true, - "output": "hello world\n", - "execution_time": 0.05, - "test_id": "0193...", - "exit_code": 0, - "error": null -} -``` +**Response** (canonical executor contract — matches the peer go/js/java/rust/csharp/cpp +executors): + +| Field | Meaning | +|-------|---------| +| `success` | Exit code 0 (and, in pytest mode, at least one test collected) | +| `output` | Combined stdout/stderr, or the timeout/syntax message | +| `execution_time` | Wall-clock seconds (float) | +| `test_id` | uuid7 of this execution | +| `exit_code` | Subprocess return code; `124` on timeout | +| `error` | `null` on success, else `syntax_error`, `timeout`, `no_tests_collected`, `tests_failed`, `execution_failed` or `internal_error` | +| `tests_run` / `tests_passed` / `tests_failed` | Parsed from the pytest summary (0 in script mode) | +| `no_tests_run` | True when `test_code` was provided but pytest collected zero tests | + +**Script mode (no `test_code`):** -`exit_code` is the subprocess return code (`124` on timeout); `error` is a short failure -category (`syntax_error` / `execution_failed` / `internal_error`) or `null` on success. +1. Writes the code to `/tmp/test_code_.py` +2. Compiles it with `py_compile` to catch syntax errors early +3. Runs `sys.executable ` +4. Kills the process and returns exit code `124` on timeout +5. Deletes the temp file -**Execution flow:** -1. Writes code to a temp file (`/tmp/test_code_.py`) -2. Compiles with `py_compile` to catch syntax errors early -3. Runs as a subprocess (`sys.executable `) with the configured timeout -4. Kills the process on timeout -5. Cleans up the temp file -6. Returns success/failure, stdout, and wall-clock execution time +**Pytest mode (`test_code` supplied):** -### `GET /docs` +1. Writes `code` + `test_code` to `/tmp/test_submission_.py` +2. Compiles the combined module with `py_compile` +3. Runs `sys.executable -m pytest -q -p no:cacheprovider` +4. Maps pytest exit code `5` (no tests collected) to `error: "no_tests_collected"`, + any other non-zero code to `"tests_failed"`, and a timeout to `"timeout"` / `124` +5. Deletes the temp file -Auto-generated OpenAPI (Swagger) docs. +A syntax error returns HTTP 200 with `success: false`, `exit_code: 1` and +`error: "syntax_error"`; an unexpected exception returns `error: "internal_error"`. -## ResultProcessor (Library Component) +### `GET /docs`, `/redoc`, `/openapi.json` -Not exposed via HTTP. Used internally (or by importing services) to parse tool output into structured issues. +FastAPI's generated documentation and schema. These are **not** exempt from the tenant header, +so open them with `-H 'X-Tenant-Id: …'` (for example through a browser extension or `curl`). -**Supported tools:** +## ResultProcessor (library component) + +Not exposed via HTTP. Parses tool output into `QualityServiceIssue` objects. It handles +**eight** tools: | Tool | Input format | What it detects | |------|-------------|-----------------| @@ -116,13 +143,15 @@ Not exposed via HTTP. Used internally (or by importing services) to parse tool o | ruff-format | Text (`ruff format --check`) | Formatting issues; exit 2 = `tool-failure` | | bandit | JSON (`-f json`) | Security vulnerabilities | | eslint | JSON | JS/TS lint issues | +| prettier | Text | Formatting issues | | vulture | Text | Dead/unused code | | radon | JSON (`-j`) | Cyclomatic complexity > 10 | -| prettier | Text | Formatting issues | -`parse_tool_output` requires `exit_code`: ruff / ruff-format / ty exit >= 2 is reported as a failure, never as clean or as findings (ty can exit 2 while printing valid JSON). +`parse_tool_output` requires `exit_code`: ruff / ruff-format / ty exit >= 2 is reported as a +failure, never as clean or as findings (ty can exit 2 while printing valid JSON). Each issue is a `QualityServiceIssue`: + ```python class QualityServiceIssue(BaseModel): tool: str @@ -134,15 +163,52 @@ class QualityServiceIssue(BaseModel): fixable: bool ``` +## Architecture + +``` + ┌─────────────────────────┐ + POST /execute ──────► │ FastAPI (test_service) │ + │ - X-Tenant-Id guard │ + │ - write code to /tmp │ + │ - py_compile check │ + │ - subprocess exec │ + │ (script or pytest) │ + │ - timeout + cleanup │ + └─────────────────────────┘ + + ResultProcessor (library, not exposed via API) + - parses stdout/stderr from: ruff, ty, bandit, eslint, + ruff-format, prettier, vulture, radon + - returns list[QualityServiceIssue] +``` + +**Key components:** + +| File | Purpose | +|------|---------| +| `src/test_service.py` | FastAPI app — `/`, `/health`, `/execute`, tenant guard, script/pytest execution | +| `src/result_processor.py` | Parses output from 8 quality/linting tools into `QualityServiceIssue` objects | +| `src/quality_models.py` | Pydantic model for `QualityServiceIssue` (tool, severity, message, line, column, rule_id, fixable) | + ## Configuration -| Variable | Default | Description | -|----------|---------|-------------| -| `PORT` | 8087 | HTTP listen port | -| `TEST_MODE` | `true` | Set in Docker; not currently used by app logic | -| `PYTHONPATH` | `/app:/app/src` | Set in Docker to resolve imports | +The application reads **no** environment variables: there is no `os.getenv` or `os.environ` +call in `src/`. The listen port is fixed by the uvicorn command line — `--port 8087` in the +Dockerfile `CMD`, and `uvicorn.run(port=8087)` in the `__main__` block. + +These variables are set by the image/compose file but are not read by application code: + +| Variable | Default | Meaning | +|----------|---------|---------| +| `PORT` | `8087` | Set by `docker-compose.yml`; **not read** — the port comes from the uvicorn CLI | +| `TEST_MODE` | `true` | Set in Docker/compose; not used by app logic | +| `PYTHONPATH` | `/app:/app/src` | Set in the image so `test_service:app` resolves; read by Python's import system, not by app code | +| `PYTHONDONTWRITEBYTECODE` / `PYTHONUNBUFFERED` | `1` | Set in the image for container-friendly execution | -No database. No Redis connection at runtime (redis is a dependency but not used by the current endpoints). +`X-Tenant-Id` is a request header, not an environment variable; see **Usage**. + +No database. Redis is declared as a dependency but is not imported by any endpoint (see +**Known Issues**). ## Docker @@ -151,16 +217,23 @@ docker compose up -d # start on port 8087 docker compose down -v # stop ``` -The container image ships pytest + hypothesis (from `uv.lock`) and the analysis CLIs ruff 0.17.3, ty 0.0.92 and bandit 1.9.4 (isolated `uv tool` installs, outside the locked venv). This makes them available for code executed inside the container. +The image is `python:3.14-slim` with `uv` 0.13.7 and installs exactly what `uv.lock` pins +(`uv sync --frozen --no-dev`). pytest + hypothesis are runtime dependencies because +`/execute` shells out to `sys.executable -m pytest`. The analysis CLIs ruff 0.17.3, ty 0.0.92 +and bandit 1.9.4 are installed with `uv tool install` into their own venvs outside the locked +venv, so they are available to code executed in the container without disturbing the lock. -The Docker socket is mounted read-only (`/var/run/docker.sock:/var/run/docker.sock:ro`) for potential container-in-container use. +The Docker socket is mounted read-only (`/var/run/docker.sock:/var/run/docker.sock:ro`) in +`docker-compose.yml` for potential container-in-container use. -## Dependencies on Other Services +## Dependencies -- **coding-testing** (port 8689) — the testing orchestrator that delegates Python execution to this service +- **coding-testing** (port 8689) — the testing orchestrator that delegates Python execution to + this service - **coding-gateway** (port 8687) — routes external requests through to coding-testing -No outbound calls to other services from within this codebase. +No outbound calls to other services from within this codebase (verified: `src/` imports no +HTTP or Redis client). ## Development @@ -168,20 +241,62 @@ No outbound calls to other services from within this codebase. uv sync --dev # install deps uv run ruff check . # lint uv run ruff format --check . # format check -uv run pytest # run tests +uv run ty check src # type check +uv run pytest # unit tests (integration deselected by addopts) +# integration tests hit a REAL service at http://localhost:8087 (start it first): +uv run pytest tests/integration/ -v -m integration --no-cov --override-ini="addopts=" uv run uvicorn src.test_service:app --port 8087 # run locally ``` +## Security + +Report vulnerabilities privately through the repository's Security tab or the organisation +policy at [`SECURITY.md`](https://github.com/nonagenticai/.github/blob/main/SECURITY.md). Do not +open a public issue for a security problem. + +The `X-Tenant-Id` check is an access marker that keeps accidental callers out; it is not +authentication, and it does not verify a tenant. Do not rely on it for isolation — restrict +network reachability with a NetworkPolicy. Submitted code is executed in the same container as +the service, so the container and its network policy are the security boundary. + +## Contributing + +Issues and pull requests are welcome. See +[`CONTRIBUTING.md`](https://github.com/nonagenticai/.github/blob/main/CONTRIBUTING.md) for the +organisation-wide rules; this repository's own commands are in **Development** above. + ## CI/CD -GitHub Actions on `nonagenticai/python-test-executor`, via the public `ci-workflows` archetype on GitHub-hosted runners. On every PR and on `main`: +GitHub Actions on `nonagenticai/python-test-executor`, via the public `ci-workflows` archetype +on GitHub-hosted runners. On every PR and on `main`: + 1. Secret scan and private-infrastructure scan -2. ruff (check + format), ty, the pytest unit suite, `uv build`, bandit and pip-audit -3. Image build and a `/health` smoke against it -4. From `main`: push `sha-` + `latest` to `ghcr.io/nonagenticai/python-test-executor` with a SLSA provenance attestation +2. `uv lock --check`, dependency install, ruff (check + format), ty, the pytest unit suite, + bandit and pip-audit +3. Image build and a `/health` smoke test against it +4. From `main`: push `sha-` + `latest` to + `ghcr.io/nonagenticai/python-test-executor` with a SLSA provenance attestation -There is no deploy job: this service is retired from the estate's cluster, and the repository is published so its tests, image build and job contract stay reproducible. +There is no deploy job: this service is retired from the estate's cluster, and the repository is +published so its tests, image build and job contract stay reproducible. ## Known Issues -- **Unused redis dependency** — redis is listed as a dependency but not imported or used by any endpoint. Left in place for potential future use by the result processor. +- **Unused redis dependency** — redis is listed in `pyproject.toml` but not imported by any + endpoint. Left in place for potential future use by the result processor. + +## Licensing + +The code here is Apache-2.0 and needs no licence key to build, test or run. This service +performs **no licence enforcement**: it reads no licence environment variable and never calls a +licence server, so a clone with no licensing infrastructure behaves exactly like the deployed +instance. + +The estate's optional licence gate is external and lives outside this repository: an Envoy +`ext_authz` filter consults [`license-authz`](https://github.com/nonagenticai/license-authz) only +for hosts listed in its `LICENSE_GATED_APP_KEY` table. The default is **no gate** — a host that +is not in that table is never licence-checked — and `python-test-executor` is not in it. + +## License + +Apache-2.0 — see [`LICENSE`](LICENSE). This repository has no `NOTICE` file.