diff --git a/AGENTS.md b/AGENTS.md index 245be2db7..c7d4c32e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -71,6 +71,17 @@ with an explicit cache, repository, commit and required-file list. Require exit model coverage. See `skills/manage-model-cache/` and `docs/model-workflows/shared-cache.md` for commands and failure branches. +## Golden path: single-node model serving + +Follow `docs/model-workflows/vllm-quickstart.md` inside an approved, isolated GPU +allocation. Use the pinned local snapshot, one GPU, and a loopback-only endpoint. +After startup, run `scripts/validation/validate_vllm.py --json` with the expected +served model name and the cache validator's explicit inputs. This submits one +small completion request; it does not start a server or download models. Require +exit `0` and `ok: true`, but do not treat this as revision attestation, GPU-use +proof, or a performance result. Never run it against someone else's service +without permission. No live testing is implied by offline fixtures. + ## Rules for operating this repository 1. **Validate, don't assume.** Run the doctor before deploying and the diff --git a/docs/model-workflows/shared-cache.md b/docs/model-workflows/shared-cache.md index 33fa0682a..dae262854 100644 --- a/docs/model-workflows/shared-cache.md +++ b/docs/model-workflows/shared-cache.md @@ -75,9 +75,12 @@ python3 -m venv "$HOME/.venvs/model-cache" ``` From a shell running as the designated writer, use a restrictive umask and source -only the cache environment. This example model is public; review its license, -file list and size before downloading. The commit hash, not `main` or a tag, is -the reproducibility boundary: +only the cache environment. The example model, `openai-community/gpt2` at +`607a30d783dfa663caf39e06633721c8d4cfcd7e`, is listed as **MIT** in its +[pinned model card](https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md) +(verified 2026-10-07). Review the license terms, file list and size before +downloading; public access does not replace that review. The commit hash, not +`main` or a tag, is the reproducibility boundary: ```bash umask 0027 @@ -140,6 +143,9 @@ site-enforced egress restrictions. See [air-gapped deployment](../../skills/depl for the separate package/image mirroring requirements. The writer's pinned Python packages must also be mirrored before using it offline. +For a one-GPU serving example using this exact cache layout, see the +[vLLM quickstart and JSON validator](vllm-quickstart.md). + ## Failure handling and lifecycle - Missing mount: stop and have the storage operator restore the approved mount. diff --git a/docs/model-workflows/vllm-quickstart.md b/docs/model-workflows/vllm-quickstart.md new file mode 100644 index 000000000..527d97b17 --- /dev/null +++ b/docs/model-workflows/vllm-quickstart.md @@ -0,0 +1,209 @@ +# Single-node vLLM quickstart + +Serve a pinned model from the [shared model cache](shared-cache.md), then check +one completion using a standard-library-only JSON validator. This is an opt-in, +one-GPU example, not a cluster service deployment or performance benchmark. It +uses open-source vLLM and a public model; no paid endpoint or registry entitlement +is needed. Review the model license before use. + +## Prerequisites and boundaries + +- An approved, otherwise idle Linux GPU allocation with one GPU visible to the + workload. Respect scheduler allocation and device isolation; do not start this + in a login shell outside your allocation. Do not override `CUDA_VISIBLE_DEVICES` + to reach other GPUs. This guide does not allocate resources or install drivers. +- A driver compatible with the selected vLLM wheel's CUDA runtime. The tagged + v0.31.0 NVIDIA guide describes CUDA 12.9 wheels and compute capability 7.5+. + Check the upstream installation guide for your GPU/driver before installing; + a working `nvidia-smi` alone does not establish wheel compatibility. +- Python **3.12** with venv support, the Python 3.12 development headers and a C + compiler (on Ubuntu 24.04: `sudo apt install python3.12-dev build-essential`; + vLLM's Triton helper compiles at server start and fails with `Python.h: No such + file` without them), available disk space for the serving packages, + and a fresh private environment separate from both Ansible and the cache writer. + Python 3.12 lies in both the tagged guide's 3.10–3.13 range and the package's + broader `>=3.10,<3.15` metadata. This is not a claim that every listed Python + version or GPU combination has been tested here. +- A fully populated, immutable snapshot from the shared-cache guide. The writer + downloads `config.json`, `tokenizer.json`, and `model.safetensors` for the GPT-2 + revision below. The serving user must be able to read them. Pause cache writes + while testing; only trusted operators may write to the cache or its parent. +- Run with the consumer's real UID/group and mount layout. In a container, mount + the **whole Hub cache** read-only at the same path so relative blob links work. + Do not mount only `snapshots/`. The reader must not write to the shared cache. + Keep `HF_HOME`, `HF_TOKEN_PATH`, Xet state, and vLLM's compilation cache private + on local storage; never copy authentication into the shared cache. +- This demo binds only to loopback. Other local users can still reach the server: + use an isolated test allocation, no sensitive prompts, and no public exposure. + Do not change the bind address to share it. Authentication, TLS, remote access, + production isolation, multi-node serving, and scheduler manifests are outside + this example. The validator deliberately has no authentication options. + +The commands below run **on the allocated serving node**, from the repository +root. Package installation can download large GPU dependencies. For disconnected +systems, prepare an approved wheel mirror first; offline model loading does not +make package installation offline. + +## 1. Prepare the private serving environment + +```bash +python3.12 -m venv "$HOME/.venvs/deepops-vllm" +"$HOME/.venvs/deepops-vllm/bin/python" -m pip install -r examples/vllm/requirements.txt +"$HOME/.venvs/deepops-vllm/bin/python" -m pip check +"$HOME/.venvs/deepops-vllm/bin/vllm" --version +``` + +The example pins vLLM **0.31.0**, licensed under **Apache-2.0** (verified +2026-10-07 against the [tagged license](https://github.com/vllm-project/vllm/blob/v0.31.0/LICENSE) +and [package metadata](https://pypi.org/project/vllm/0.31.0/)). This is the serving +package's license, not the model's or every dependency's. Transitive packages +are not a full lockfile: record `pip freeze` with validation evidence, and use +your site's reviewed wheel set for repeatable deployment. Do not mix an existing +PyTorch installation into this environment. Do not upgrade the cache writer's pinned CLI to fix a serving +package problem. vLLM loads a local snapshot, not a mutable Hub repository name. + +## 2. Check the cache before starting the server + +The example model, `openai-community/gpt2` at +`607a30d783dfa663caf39e06633721c8d4cfcd7e`, is listed as **MIT** in its +[pinned model card](https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md) +(verified 2026-10-07). Review its license terms before use; re-check the model's +license and access requirements when selecting another model or revision. + +```bash +export HF_HUB_CACHE=/shared/hf-cache/hub +export MODEL_ID=openai-community/gpt2 +export MODEL_REVISION=607a30d783dfa663caf39e06633721c8d4cfcd7e +export MODEL_PATH="$HF_HUB_CACHE/models--openai-community--gpt2/snapshots/$MODEL_REVISION" +python3 scripts/validation/validate_model_cache.py --json \ + --cache-dir "$HF_HUB_CACHE" --repo-id "$MODEL_ID" --revision "$MODEL_REVISION" \ + --require-file config.json --require-file tokenizer.json --require-file model.safetensors +``` + +Stop unless exit status is `0` and `ok` is `true`. For a different model, review +its complete tokenizer/configuration/weight file list, including every indexed +shard. The validator only checks files you name. Never enable remote model code +as a workaround; this example does not use `--trust-remote-code`. + +## 3. Serve one GPU in the foreground + +In the same allocated shell, start the server and keep its log private: + +```bash +HF_HUB_OFFLINE=1 HF_HUB_DISABLE_IMPLICIT_TOKEN=1 \ + "$HOME/.venvs/deepops-vllm/bin/vllm" serve "$MODEL_PATH" \ + --tokenizer "$MODEL_PATH" \ + --served-model-name deepops-gpt2 \ + --host 127.0.0.1 --port 8000 \ + --tensor-parallel-size 1 --dtype half \ + --load-format safetensors --generation-config vllm \ + --max-model-len 512 --max-num-seqs 1 \ + --gpu-memory-utilization 0.5 --enforce-eager +``` + +Wait for server startup to finish before validating. These conservative settings +bound the demo, not total GPU memory or runtime. Startup can still fail on an +unsupported GPU, unavailable memory, incompatible driver, or incomplete files. +Do not remove scheduler isolation or alter drivers to make this example pass. + +`HF_HUB_OFFLINE=1` is scoped to the serving process. It blocks Hub requests but +is not a firewall; use site-enforced egress restrictions when required. Do not +export it globally in clients that need to call a local inference endpoint. +The local model path avoids reader-side Hub downloads and cache updates. Private +framework caches may still be created on the serving node. + +This demo expects no server API-key configuration. If site policy requires +authentication, stop and use its approved authenticated client instead; do not +remove site controls to run this validator. + +## 4. Validate from another shell on the same node + +Use a shell within the same approved allocation and namespace. This makes one +small inference request (up to eight output tokens); it is **not** a read-only +health check and must not be run on someone else's service without permission. + +```bash +python3 scripts/validation/validate_vllm.py --json \ + --base-url http://127.0.0.1:8000 --model deepops-gpt2 \ + --cache-dir /shared/hf-cache/hub \ + --repo-id openai-community/gpt2 \ + --revision 607a30d783dfa663caf39e06633721c8d4cfcd7e \ + --require-file config.json --require-file tokenizer.json --require-file model.safetensors +``` + +It checks, in order: + +1. The existing cache validator's immutable revision and file-readability rules. +2. `GET /health` returns HTTP 200. +3. `GET /v1/models` advertises the requested served name. +4. `POST /v1/completions` returns the expected model, nonempty text, and a positive + count of generated tokens no greater than the eight requested. + +GPT-2 is a base text model, so this uses **completions**, not chat or a chat +template. Generated text is intentionally not printed or checked for factual +accuracy. Neither server responses nor error bodies are copied into the result. + +### JSON and exit contract + +- Exit `0`: all four checks passed and `ok: true`. +- Exit `1`: a validation/input-value failure and `ok: false`. +- Exit `2`: command syntax error (usage on stderr, no JSON promised). +- `--json` writes one JSON object to stdout. Without it, output is a short summary. +- Schema version `1` fields: `schema_version`, `ok`, `checks` (boolean `cache`, + `health`, `model`, `completion`), `cache` (the cache validator's result or null), + and `errors` (objects with `stage` and `message`). A false check may be failed + **or skipped**: execution stops at the first error, whose stage identifies it. +- Only literal loopback HTTP addresses with an explicit port are accepted, + including `http://[::1]:8000`. No `/v1` suffix, credentials, hostname resolution, + proxy routing, or redirects. The default is `http://127.0.0.1:8000`. +- No retries. `--timeout` defaults to a 30-second socket timeout (maximum 120), + with an elapsed-time guard checked between response-body reads. A final blocked + read can add up to one socket timeout; this is not a hard process deadline. + CI runners should also impose their own overall command deadline, including + the cache check: an unresponsive shared filesystem can block local file reads. + Each response is capped at 1 MiB. + +Success proves a small completion from the named local endpoint plus readability +of the listed snapshot files. **It cannot attest that the server loaded that +revision**, used a GPU, avoided all network traffic, or achieved useful quality, +throughput or concurrency. Preserve the server invocation, package versions and +private startup logs for independent verification. An unrelated server using the +same alias is not distinguished by this API check. + +Stop only the foreground server you started with Ctrl-C when finished. Confirm +it has exited before releasing the allocation. Do not kill unrelated processes, +prune the shared cache, or remove snapshots that other jobs may use. + +## Failure handling + +| Failing stage | Action | +|---|---| +| `arguments` | Use a bare loopback origin, served model name and finite timeout. | +| `cache` | Stop; repair missing files as the designated writer or restore the approved mount/reader permissions. Never loosen permissions globally. | +| `health` | Check the private startup log and listening port. Wait for startup, then explicitly rerun once ready. A 401 means use the site's authenticated workflow instead. | +| `model` | Compare `--served-model-name` with `--model`; do not accept a different server/model as success. | +| `completion` | Check the private server log, full file list, model support and available GPU memory. An empty or malformed completion is failure, even when health passed. | + +## Development and evidence limits + +```bash +python3 -m unittest discover -s scripts/validation/tests -p 'test_vllm.py' +``` + +Tests use synthetic local HTTP servers and cache files, not vLLM or real weights. +They exercise positive/negative JSON results, request ordering, malformed +responses, timeouts, bounded output, proxy/redirect refusal and cache failures. +They do not certify the pinned serving environment. Real cache reuse, read-only +mounts, offline startup and GPU inference need a separately approved live test. + +NVIDIA NIM is an optional alternative, documented by its own +[getting-started guide](https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html). +It is not installed, required, or validated by this example. + +## Primary sources (checked 2026-10-05) + +- [vLLM 0.31.0 package metadata](https://pypi.org/project/vllm/0.31.0/). +- Tagged [GPU requirements](https://github.com/vllm-project/vllm/blob/v0.31.0/docs/getting_started/installation/gpu.md) + and [NVIDIA installation details](https://github.com/vllm-project/vllm/blob/v0.31.0/docs/getting_started/installation/gpu.cuda.inc.md). +- Tagged [OpenAI-compatible server guide](https://github.com/vllm-project/vllm/blob/v0.31.0/docs/serving/online_serving/openai_compatible_server.md). +- [Shared-cache sources and writer pin](shared-cache.md#primary-sources-verified-2026-10-05). diff --git a/examples/vllm/requirements.txt b/examples/vllm/requirements.txt new file mode 100644 index 000000000..5bcdd83f2 --- /dev/null +++ b/examples/vllm/requirements.txt @@ -0,0 +1,2 @@ +# Install in a private Python 3.12 environment, separate from Ansible and the cache writer. +vllm==0.31.0 diff --git a/playbooks/model-cache.yml b/playbooks/model-cache.yml index c95d5fd44..8c5918749 100644 --- a/playbooks/model-cache.yml +++ b/playbooks/model-cache.yml @@ -41,6 +41,8 @@ fail_msg: The configured mount must already be mounted as NFS; this playbook never mounts storage. - name: Inspect cache directories without following symlinks + become: true + become_user: "{{ model_cache_owner }}" stat: path: "{{ item }}" follow: false diff --git a/scripts/validation/tests/test_vllm.py b/scripts/validation/tests/test_vllm.py new file mode 100644 index 000000000..f9bef8433 --- /dev/null +++ b/scripts/validation/tests/test_vllm.py @@ -0,0 +1,254 @@ +"""Offline HTTP fixtures: never start vLLM, download weights, or use a GPU.""" +import contextlib +import http.server +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import threading +import time +import unittest + +SCRIPT = Path(__file__).resolve().parents[1] / "validate_vllm.py" +REVISION = "a" * 40 + + +@contextlib.contextmanager +def endpoint(responses, drip_interval=0): + requests = [] + + class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + self.respond() + + def do_POST(self): + self.respond() + + def respond(self): + body = self.rfile.read(int(self.headers.get("Content-Length", 0))) + requests.append((self.command, self.path, json.loads(body) if body else None)) + status, value, headers, delay = responses[self.path] + time.sleep(delay) + data = value if isinstance(value, bytes) else json.dumps(value).encode() + self.send_response(status) + for key, value in headers.items(): + self.send_header(key, value) + self.end_headers() + try: + if drip_interval and self.path == "/health": + for byte in data: + self.wfile.write(bytes([byte])) + self.wfile.flush() + time.sleep(drip_interval) + else: + self.wfile.write(data) + except (BrokenPipeError, ConnectionResetError): + pass + + def log_message(self, *args): + pass + + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}) + thread.start() + try: + yield "http://127.0.0.1:" + str(server.server_port), requests + finally: + server.shutdown() + server.server_close() + thread.join() + + +class VllmTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.cache = Path(self.tmp.name) + self.snapshot = self.cache / "models--example--model" / "snapshots" / REVISION + self.snapshot.mkdir(parents=True) + for name in ("config.json", "tokenizer.json", "model.safetensors"): + (self.snapshot / name).write_text("fixture only") + self.responses = { + "/health": (200, b"", {}, 0), + "/v1/models": (200, {"object": "list", "data": [{"id": "test-model", "object": "model"}]}, {}, 0), + "/v1/completions": (200, { + "id": "cmpl-test", "object": "text_completion", "model": "test-model", + "choices": [{"index": 0, "text": "generated fixture", "finish_reason": "length"}], + "usage": {"prompt_tokens": 5, "completion_tokens": 8, "total_tokens": 13}, + }, {}, 0), + } + + def run_validator(self, url, *extra, env=None): + args = [sys.executable, str(SCRIPT), "--json", "--base-url", url, + "--model", "test-model", "--cache-dir", str(self.cache), + "--repo-id", "example/model", "--revision", REVISION] + for name in ("config.json", "tokenizer.json", "model.safetensors"): + args += ["--require-file", name] + run = subprocess.run(args + list(extra), capture_output=True, text=True, + timeout=5, env=env) + self.assertEqual(run.stderr, "", run.stderr) + self.assertIn(run.returncode, (0, 1), run.stderr) + result = json.loads(run.stdout) + self.assertEqual(result["schema_version"], 1) + self.assertEqual(result["ok"], run.returncode == 0) + return result + + def test_success_requires_cache_health_model_and_completion(self): + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url) + self.assertTrue(result["ok"]) + self.assertEqual(result["checks"], {"cache": True, "health": True, + "model": True, "completion": True}) + self.assertEqual(result["errors"], []) + self.assertEqual([(r[0], r[1]) for r in requests], [ + ("GET", "/health"), ("GET", "/v1/models"), ("POST", "/v1/completions")]) + self.assertEqual(requests[-1][2], {"model": "test-model", "prompt": "The capital of France is", + "max_tokens": 8, "temperature": 0, "stream": False}) + self.assertNotIn("generated fixture", json.dumps(result)) + + def test_missing_cache_file_prevents_all_http(self): + (self.snapshot / "model.safetensors").unlink() + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "cache") + self.assertEqual(requests, []) + + def test_mutable_revision_prevents_all_http(self): + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url, "--revision", "main") + self.assertFalse(result["ok"]) + self.assertEqual(requests, []) + + def test_health_failure_stops_before_inference(self): + self.responses["/health"] = (503, b"private server details", {}, 0) + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "health") + self.assertEqual(len(requests), 1) + self.assertNotIn("private server details", json.dumps(result)) + + def test_health_requires_200_not_other_success_status(self): + self.responses["/health"] = (204, b"", {}, 0) + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "health") + self.assertEqual(len(requests), 1) + + def test_model_must_be_advertised_and_models_shape_valid(self): + for value in ({"data": [{"id": "other"}]}, {"data": []}, [], None, + {"data": "test-model"}, {"data": [None]}, b"not json"): + with self.subTest(value=value): + self.responses["/v1/models"] = (200, value, {}, 0) + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "model") + self.assertEqual(len(requests), 2) + + def test_completion_must_prove_nonempty_output_for_expected_model(self): + good = self.responses["/v1/completions"][1] + bad = [None, [], {}, {**good, "model": "other"}, {**good, "choices": []}, + {**good, "choices": [None]}, {**good, "choices": [{"text": " "}]}, + {**good, "choices": [{"text": 7}]}, + {**good, "usage": {"completion_tokens": 0}}, + {**good, "usage": {"completion_tokens": 9}}, + {**good, "usage": {"completion_tokens": True}}, + {**good, "usage": None}, b"broken json"] + for value in bad: + with self.subTest(value=value): + self.responses["/v1/completions"] = (200, value, {}, 0) + with endpoint(self.responses) as (url, _): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "completion") + + def test_redirect_never_followed(self): + with endpoint(self.responses) as (destination, redirected): + self.responses["/health"] = (302, b"", {"Location": destination + "/health"}, 0) + with endpoint(self.responses) as (url, _): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(redirected, []) + + def test_proxies_are_ignored(self): + with endpoint(self.responses) as (proxy, proxied): + env = {**os.environ, "http_proxy": proxy, "HTTP_PROXY": proxy, + "https_proxy": proxy, "HTTPS_PROXY": proxy, "ALL_PROXY": proxy, + "no_proxy": "", "NO_PROXY": ""} + with endpoint(self.responses) as (url, _): + result = self.run_validator(url, env=env) + self.assertTrue(result["ok"]) + self.assertEqual(proxied, []) + + def test_only_bare_loopback_http_origins_accepted(self): + for url in ("http://example.com:8000", "https://127.0.0.1:8000", "file:///etc/passwd", + "http://user:example@127.0.0.1:8000", "http://127.0.0.1:8000/path", + "http://127.0.0.1:8000/?x=1", "http://127.0.0.1:8000/#fragment", + "http://localhost:8000", "http://127.0.0.1:99999"): + with self.subTest(url=url): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "arguments") + self.assertNotIn("user:example", json.dumps(result)) + + def test_timeout_is_bounded_and_machine_readable(self): + self.responses["/health"] = (200, b"", {}, 0.2) + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url, "--timeout", "0.05") + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "health") + self.assertEqual(len(requests), 1) + + def test_dripping_body_hits_elapsed_guard_not_socket_timeout(self): + # Every byte arrives before the socket timeout, but the full body takes + # longer than the elapsed budget. Without that guard all checks pass. + self.responses["/health"] = (200, b"x" * 20, {}, 0) + with endpoint(self.responses, drip_interval=0.02) as (url, requests): + result = self.run_validator(url, "--timeout", "0.1") + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "health") + self.assertIn("time limit", result["errors"][0]["message"]) + self.assertEqual(len(requests), 1) + + def test_invalid_timeouts_rejected_before_http(self): + for timeout in ("0", "-1", "nan", "inf", "121"): + with self.subTest(timeout=timeout): + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url, "--timeout", timeout) + self.assertFalse(result["ok"]) + self.assertEqual(requests, []) + + def test_oversized_response_rejected(self): + # Valid JSON plus whitespace stays valid even if a broken reader + # truncates the padding. A missing size guard must not pass this test + # merely because malformed JSON failed independently. + data = json.dumps({"data": [{"id": "test-model"}]}).encode() + b" " * (1024 * 1024) + self.responses["/v1/models"] = (200, data, {}, 0) + with endpoint(self.responses) as (url, requests): + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "model") + self.assertIn("exceeds 1 MiB", result["errors"][0]["message"]) + self.assertEqual(len(requests), 2) + + def test_connection_refused_is_json_failure(self): + with endpoint(self.responses) as (url, _): + pass + result = self.run_validator(url) + self.assertFalse(result["ok"]) + self.assertEqual(result["errors"][0]["stage"], "health") + + def test_unknown_cli_flag_is_usage_error(self): + run = subprocess.run([sys.executable, str(SCRIPT), "--unknown"], + capture_output=True, text=True) + self.assertEqual(run.returncode, 2) + self.assertIn("usage:", run.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/validation/validate_vllm.py b/scripts/validation/validate_vllm.py new file mode 100644 index 000000000..57b367f6b --- /dev/null +++ b/scripts/validation/validate_vllm.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +"""Check a local vLLM server with one small inference request; never launch it.""" +import argparse +import http.client +import ipaddress +import json +import math +import re +import sys +import time +import urllib.error +import urllib.parse +import urllib.request + +from validate_model_cache import validate as validate_cache + + +class ValidationError(Exception): + """A failed readiness check, safe to report without server response data.""" + + +class NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + raise ValidationError("HTTP redirects are not allowed") + + +def local_origin(value): + try: + parsed = urllib.parse.urlsplit(value) + if (parsed.scheme != "http" or not parsed.hostname + or not ipaddress.ip_address(parsed.hostname).is_loopback + or "@" in parsed.netloc + or parsed.path not in ("", "/") or parsed.query or parsed.fragment + or parsed.port is None or not 1 <= parsed.port <= 65535): + raise ValueError + return "http://" + parsed.netloc + except ValueError: + raise ValidationError("base URL must be an HTTP loopback IP with an explicit port and no credentials, path, query, or fragment") from None + + +def request(opener, origin, path, timeout, payload=None, expect_json=True): + data = None if payload is None else json.dumps(payload).encode("utf-8") + req = urllib.request.Request(origin + path, data=data, + headers={"Content-Type": "application/json", "Accept": "application/json"}) + try: + deadline = time.monotonic() + timeout + with opener.open(req, timeout=timeout) as response: + if response.status != 200: + raise ValidationError("endpoint did not return HTTP 200") + # Bound memory and check elapsed time between socket reads, including + # peers that drip bytes without triggering a socket inactivity timeout. + body = bytearray() + while True: + if time.monotonic() >= deadline: + raise ValidationError("endpoint response exceeded the time limit") + chunk = response.read1(min(65536, 1024 * 1024 + 1 - len(body))) + body.extend(chunk) + if len(body) > 1024 * 1024: + raise ValidationError("endpoint response exceeds 1 MiB") + if not chunk: + break + return json.loads(body) if expect_json else None + except urllib.error.HTTPError as exc: + status = exc.code + exc.close() + raise ValidationError("endpoint returned HTTP " + str(status)) from None + except (OSError, urllib.error.URLError, http.client.HTTPException): + raise ValidationError("endpoint unreachable, timed out, or connection failed") from None + except (ValueError, UnicodeError, RecursionError): + raise ValidationError("endpoint returned invalid JSON") from None + + +def validate(args): + result = { + "schema_version": 1, + "ok": False, + "checks": {"cache": False, "health": False, "model": False, "completion": False}, + "cache": None, + "errors": [], + } + stage = "arguments" + try: + origin = local_origin(args.base_url) + if not math.isfinite(args.timeout) or not 0 < args.timeout <= 120: + raise ValidationError("timeout must be greater than zero and at most 120 seconds") + if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:/-]{0,255}", args.model): + raise ValidationError("model must be a nonempty served model name (maximum 256 characters)") + stage = "cache" + result["cache"] = validate_cache(args.cache_dir, args.repo_id, args.revision, args.require_file) + if not result["cache"]["ok"]: + raise ValidationError("required pinned cache files failed validation; see cache.errors") + result["checks"][stage] = True + # Never inherit proxy routing or forward a local request via a redirect. + opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), NoRedirect()) + stage = "health" + request(opener, origin, "/health", args.timeout, expect_json=False) + result["checks"][stage] = True + stage = "model" + models = request(opener, origin, "/v1/models", args.timeout) + if (not isinstance(models, dict) or not isinstance(models.get("data"), list) + or not all(isinstance(item, dict) for item in models["data"]) + or not any(item.get("id") == args.model for item in models["data"])): + raise ValidationError("expected served model is absent or model-list response is malformed") + result["checks"][stage] = True + stage = "completion" + completion = request(opener, origin, "/v1/completions", args.timeout, { + "model": args.model, "prompt": "The capital of France is", + "max_tokens": 8, "temperature": 0, "stream": False, + }) + if not isinstance(completion, dict) or completion.get("model") != args.model: + raise ValidationError("completion response does not identify the expected model") + choices = completion.get("choices") + usage = completion.get("usage") + if (not isinstance(choices, list) or not choices or not isinstance(choices[0], dict) + or not isinstance(choices[0].get("text"), str) or not choices[0]["text"].strip() + or not isinstance(usage, dict) or type(usage.get("completion_tokens")) is not int + or not 0 < usage["completion_tokens"] <= 8): + raise ValidationError("completion must contain nonempty text and 1 to 8 generated tokens") + result["checks"][stage] = True + result["ok"] = True + except ValidationError as exc: + result["errors"].append({"stage": stage, "message": str(exc)}) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False) + parser.add_argument("--base-url", default="http://127.0.0.1:8000", + help="Loopback HTTP origin, without /v1; proxies and redirects disabled") + parser.add_argument("--model", required=True, help="Expected --served-model-name") + parser.add_argument("--cache-dir", required=True, help="Explicit shared HF_HUB_CACHE") + parser.add_argument("--repo-id", required=True) + parser.add_argument("--revision", required=True, help="Full immutable model commit hash") + parser.add_argument("--require-file", required=True, action="append", + help="Required snapshot file; repeat for all configuration, tokenizer and weight files") + parser.add_argument("--timeout", type=float, default=30, help="Socket timeout and response-body time budget, 0 < seconds <= 120") + parser.add_argument("--json", action="store_true", help="Emit schema-versioned JSON without generated text") + args = parser.parse_args() + result = validate(args) + if args.json: + print(json.dumps(result, sort_keys=True)) + elif result["ok"]: + print("PASS: cache, health, model and completion checks passed") + else: + print("FAIL: " + "; ".join(error["stage"] + ": " + error["message"] for error in result["errors"])) + return 0 if result["ok"] else 1 + + +if __name__ == "__main__": + sys.exit(main())