Inferrail

Inferrail is a KV cache aware router and control plane for self hosted LLM inference. It sits between OpenAI compatible clients and a fleet of vLLM or SGLang engines, and sends every request to the replica that already holds most of its prompt in cache, without letting any replica overload.

The goal is lower tail latency and fewer GPUs for the same traffic, proven with benchmarks anyone can rerun.

Status: v0.1.11, realistic simulation and the control plane. The gateway proxies chat and completions to a fleet of vLLM or SGLang engines, tokenizes prompts at the edge with token ids identical to vLLM's, predicts for every replica how much of each prompt it already holds and routes on those predictions: cost_model sends each request to the replica with the lowest estimated time to first token, inside load guards that keep any replica from melting. The simulator now schedules the way vLLM does, in continuous batching steps over one pool of KV blocks with chunked prefill and preemption, and publishes KV events in vLLM's own format. One command runs the full benchmark matrix and builds its report. Routing changes while the gateway runs, without dropping a request, from the config file, SIGHUP or the admin API, with A/B splits between policies and candidate policies in shadow. See Roadmap.

Why this exists

Every LLM request has two phases. Prefill reads the whole prompt and builds a key value cache; decode then produces tokens one at a time from that cache. Engines like vLLM keep KV blocks in GPU memory after a request finishes, so a later request with the same prefix skips most of its prefill.

That reuse only happens when the second request lands on the same replica. Round robin and least connections ignore this: with N replicas a repeated prefix finds a warm cache about 1/N of the time, and every miss pays for prefill again in GPU time and time to first token (TTFT).

Prefix reuse is the normal case, not the exception:

• shared system prompts of thousands of tokens sent by every user,
• multi turn chat, where turn ten resends turns one to nine,
• agent loops that resend a growing context on every tool call,
• RAG over popular documents, and few shot evaluation batches.

The naive fixes fail in opposite directions. Hashing on the prefix piles a hot system prompt onto one replica and melts it; pure load balancing throws the cache away. Inferrail treats routing as a cost estimate instead: for each replica, how long until this request's first token, given what is cached there and what is queued there?

What is in v0.1.11

Piece	What it does
inferrail-gateway	OpenAI compatible proxy on hyper: chat and completions, streaming pass through, model list, request IDs, retries before the first byte, health checks, outlier ejection, breakers, DNS discovery, graceful shutdown, edge tokenization on a bounded worker pool, cache predictions per replica, a virtual prefill queue per replica, session keys, live config from the file, SIGHUP or the admin API, A/B splits, shadow policies with a replayable log, /admin/explain, Prometheus metrics on an admin port, OpenTelemetry traces and a sampled decision log
inferrail-policy	The RoutingPolicy trait and seven policies: the round robin, random, least request and power of two choices baselines, prefix_hash on rendezvous hashing, the weighted kv_aware score, and cost_model, the lowest estimated time to first token with bounded load, KV pressure, hot prefix replication and session affinity
inferrail-calibrate	Measures an idle engine's prefill with prompts of known cached and uncached length, timed by the engine's own histogram, and fits the cost model's parameters into a config fragment
inferrail-tokenize	Chat templates and Hugging Face tokenizers with token ids identical to vLLM v0.31.0 on three model families, a cache of repeated prompt pieces, the simulator's tokenizer, and chained XXH3 128 bit block hashes salted by model, adapter and cache_salt
inferrail-index	The KV index in approximate mode: a sharded map from block hash to replicas and a shadow LRU per replica sized to its KV capacity, with longest prefix lookup in microseconds
inferrail-events	vLLM v0.31.0's KV event batches encoded and decoded byte for byte, and ZMTP 3.0 without libzmq: a publisher shaped like vLLM's with sequence numbers and replay, and SUB and REQ clients
inferrail-load	Reads vLLM and SGLang /metrics pages, scrapes every replica ten times a second and publishes load snapshots that are never used once stale
inferrail-control	Lock free replica registry with request, prompt token and queued prefill counters, health state with exponential ejection backoff, the active health checker and DNS discovery
inferrail-core	Lossless OpenAI wire types, an incremental SSE decoder, and validated YAML configuration that knows which sections can change live
inferrail-sim	A CPU mock of vLLM v0.31.0 with the same HTTP API, streaming bytes and Prometheus metrics: continuous batching with chunked prefill over one ref counted pool of KV blocks, prefill and decode slowing each other down, preemption, a calibrated latency model and KV events
bench/	The benchmark harness (inferrail-bench): open loop workloads W1 to W6, W8, mix and hot with Poisson or Gamma arrivals and a diurnal ramp, TTFT, TPOT and ITL measurement, prediction accuracy against the engines' own counts, replica shares, benchmark matrices from one sweep file and a report with knee charts and milestone verdicts, plus the OpenAI SDK suites
deploy/compose, dashboards/	Gateway, simulators, Prometheus, Grafana and Jaeger in one make stack, with a dashboard that now shows config reloads, traffic splits and shadow policies
CI	fmt, clippy with warnings denied, nextest, doc tests, ruff, pytest, cargo audit, cargo machete, and the official OpenAI SDK and the harness against the simulator and the gateway on every push

Quick start

Requirements: Rust (the pinned 1.98.1 toolchain installs itself through rustup), uv for the Python side, and optionally cargo-nextest.

Start three simulated engines and a gateway in front of them:

cargo build --release -p inferrail-sim -p inferrail-gateway -p inferrail-calibrate
for port in 8001 8002 8003; do
  ./target/release/inferrail-sim --port $port --enable-prompt-tokens-details &
done
./target/release/inferrail-gateway --config deploy/local.yaml

Talk to the gateway like any vLLM server. sim is an alias the gateway resolves to inferrail/sim-8b:

curl -si localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "sim",
       "messages": [{"role": "user", "content": "Hello"}],
       "max_tokens": 16}'

The response headers name the replica that served it (x-inferrail-replica), the request ID the engine used (x-request-id), the prompt's length in tokens (x-inferrail-prompt-tokens) and, when the KV index made a prediction, how many of them it predicted that replica had cached (x-inferrail-cached-tokens); with --enable-prompt-tokens-details the simulator reports the real count in usage.prompt_tokens_details.cached_tokens. The official SDK works by changing nothing but the base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
stream = client.chat.completions.create(
    model="sim",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=16,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    print(chunk)

Ask the gateway why it would send a request where it would, without sending it: the admin port answers with every replica's queue, predicted cache and estimated time to first token, the winner and the term that decided it.

curl -s localhost:9090/admin/explain/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "sim", "messages": [{"role": "user", "content": "Hello"}]}'

Change how it routes while it runs: edit deploy/local.yaml and the gateway picks it up within two seconds, or patch the config in effect over the admin port. This sends a tenth of the traffic to kv_aware and watches least_request in shadow:

curl -s -X PATCH localhost:9090/admin/config \
  -H 'content-type: application/json' \
  -d '{"routing": {"split": {"cost_model": 90, "kv_aware": 10},
                   "shadow": ["least_request"]}}'
curl -s localhost:9090/admin/shadow

Measure an engine's prefill and get the cost model for it, ready to paste under routing or a model's entry:

./target/release/inferrail-calibrate --url http://127.0.0.1:8001

Kill one of the simulators while traffic is flowing: only responses it was already sending can break. The gateway retries every request that had not started on another replica, ejects the dead one within milliseconds and readmits it once it is back. inferrail-gateway --check --config <file> validates a config and exits.

The gateway's metrics are on the admin port:

curl -s localhost:9090/metrics | grep inferrail_replica_engine

To see everything on a dashboard, bring up the full stack with Docker (gateway, three simulators, Prometheus, Grafana and Jaeger):

make stack        # Grafana on :3000, gateway on :8080, Jaeger on :16686
make stack-down

To measure the gateway like a benchmark would, drive it open loop:

cd bench
uv run inferrail-bench run --base-url http://localhost:8080 --model sim \
  --workload w1 --rate 20 --requests 600 --out results/try/w1.json
uv run inferrail-bench plot results/try/w1.json --out report/try

The gateway

Routes. POST /v1/chat/completions and POST /v1/completions are proxied; /v1/models and /health are answered locally to GET and HEAD. Other paths get a 404 and wrong methods a 405 with Allow, in the OpenAI error shape vLLM uses. A rejected request that holds its body back behind Expect: 100-continue is answered at once, without being asked to upload. HTTP/1.1 and h2c are accepted from clients; engines are reached over pooled HTTP/1.1 keep alive connections, or h2c with upstream.http2.

Pass through. The request body is read once, within server.request_timeout (a body that stops arriving gets a 408), and forwarded byte for byte; only the model field is parsed. When a client asks for a gateway alias, or omits the model while the gateway serves exactly one, only those bytes are rewritten so the engine sees a name it serves. Responses stream back frame by frame as they arrive, with headers intact except hop by hop ones, and the gateway holds no per token state. Every response carries x-request-id (the client's own when well formed, which vLLM also uses as the response id), and proxied ones name their replica in x-inferrail-replica. Request bodies are never logged unless logging.log_prompts is on.

Models. GET /v1/models lists the configured models and aliases in config order, with the metadata the engines published (context length, root, permissions). The health checker refreshes those cards in the background, so listing never calls an engine.

Retries. A request moves to another replica only before the first response byte reaches the client: when the connection fails, the replica closes before answering, it answers 502, 503 or 504, or it fails its health checks while the request is still waiting for an answer. Untried replicas come first, attempts stop at retry.max_attempts with exponential backoff, and everything shares the server.request_timeout deadline. A stream that breaks after it started is never replayed. The reasoning is in ADR 003.

Health and ejection. Every replica is probed at /health each health.interval and leaves rotation after unhealthy_threshold failed probes. Independently, unhealthy_threshold consecutive failed requests eject it for ejection_backoff, doubling on every repeat up to max_ejection_backoff. Failed requests never eject more than max_ejection_percent (half by default) of the replicas of a model that pass their health checks, so requests that make every engine fail cannot empty the pool. Under load a crashed replica is out of rotation in milliseconds; it comes back once probes pass and its window ends.

Breakers. max_concurrency caps requests in flight to a replica and max_pending caps those still waiting for a response head. Caps never overshoot, even under races. A full replica is skipped and a full fleet gets an immediate 503 with Retry-After, so a replica that hangs stops absorbing traffic instead of making every client wait.

Discovery. Replicas come from the config, from DNS groups, or both. Each address a group name resolves to becomes a replica named <group>-<ip>-<port>. Groups are resolved before the first request and then every refresh; members that stay keep their health and counters, members that leave finish their streams, and a failed lookup keeps the last known set.

Cancellation and shutdown. A client that disconnects, whether streaming, queued in the engine or waiting for a full response, frees the engine's slot within 100 ms. Idle client connections are closed after server.header_read_timeout. On SIGTERM the gateway stops accepting, lets running streams finish for up to server.shutdown_timeout, then closes what is left; a request a client sends on an idle connection at that very moment can be lost, so put the gateway behind something that retries, or drain it from the load balancer first.

Load signals

The gateway knows two things about each replica's load. Its own counters are always current: requests in flight, requests still waiting for a response head, and prompt tokens in flight and still waiting for their first byte (counted for models with a tokenizer, estimated at four bytes of request body per token for the rest). The engine's own view comes from its Prometheus page: running and waiting requests, KV cache usage, prefix cache counters, preemptions and KV capacity.

Engines. vLLM v0.31.0 and SGLang v0.5.21 are read natively; the engine is recognized by its metric prefix, data parallel ranks are summed, and SGLang's per priority breakdowns are skipped. A page that lacks a required family is reported as unreadable rather than taken as an idle engine, so a renamed metric after an engine upgrade shows up in the scrape counters instead of quietly skewing routing.

Freshness. Every replica is scraped in parallel every load.scrape_interval (100 ms), each scrape bounded by load.scrape_timeout and a 4 MiB page limit. A report older than load.max_age (250 ms) is treated as missing: routing degrades to the gateway's own counts instead of trusting stale numbers. A dead, hung or garbage spewing engine is logged once when it stops answering and once when it recovers, and never delays the other scrapes. Under 1,000 requests per second the oldest report any reader saw was 102 ms; with 64 replicas scraped at once it was 112 ms. See ADR 005.

Prompts and the KV index

To route on cache state the gateway has to know a prompt's tokens exactly as the engine will see them, cut them into the blocks the engine caches, and remember where each block went.

Tokenization. Each model names a tokenizer: a model directory, a repository already in the local Hugging Face cache (org/name or org/name@revision), or builtin:sim for the simulator. Nothing is downloaded. Chat requests are prepared the way vLLM v0.31.0 prepares them, rendered with the model's chat template on minijinja set up as transformers sets up Jinja2, and tokenized with the pipeline transformers 5.17 builds, which for LlamaTokenizer differs from tokenizer.json. chat_template, chat_template_content_format and chat_template_kwargs mirror the vLLM flags of the same names. What a template sees of a request is read with the rules of Python's json.loads, template keywords reach the template when Jinja2's own analysis says it reads them, and printed values and case changes follow CPython 3.12. Golden fixtures for Qwen3, Granite 3.3 and Hermes 2 Pro, generated by running vLLM's request handling through transformers, match token for token. See ADR 008.

Speed. Repeated prompt pieces, such as shared system prompts and earlier turns, are tokenized once per model and kept up to tokenizer.cache_tokens. An 8,000 token chat takes 4.5 ms cold and 0.1 ms when its system prompt was seen before. Bodies over 1 KiB go to a pool of tokenizer.workers threads; a request that finds the queue full, or waits longer than tokenizer.max_wait, is routed without a prediction instead of waiting.

Block hashes. Full blocks of block_size tokens are hashed with XXH3 128 in a chain, so a block's hash names its whole prefix; a trailing partial block is never hashed. The chain is salted by the model, the LoRA adapter and vLLM's per request cache_salt. Engine hash values are never used. Hashing costs about 5.5 ns per block.

The index. Every routed prompt is recorded on the replica it went to, in a shadow LRU as large as that replica's KV cache: kv_capacity_blocks when set, else what its engine reports, else index.default_capacity_blocks. What an attempt recorded is taken back when the engine refuses it, answers with an error or fails before answering, since it never computed the prompt. A shared map from block hash to replicas answers, for every candidate, how many leading blocks of a prompt it holds: 2.4 µs for 512 blocks over 64 replicas. Predictions stop one block short of a fully cached prompt, as engines always compute the last token. Replicas that leave the fleet are forgotten, and an engine whose prefix cache counters go backwards is treated as restarted with an empty cache. See ADR 009.

Degrading. A model without a tokenizer, or whose tokenizer fails to load, routes as before; startup and --check say so. A request the tokenizer cannot follow (images, a template in the request, a batch of prompts) is routed without a prediction and counted by reason.

Observability

Metrics. GET /metrics on server.admin_listen (127.0.0.1:9090 by default) serves OpenMetrics, apart from the client port:

Metric	Labels	Meaning
inferrail_requests_total	model, replica, status	inference requests by outcome: a status class, aborted when the engine broke a started response, cancelled when the client left
inferrail_ttft_seconds	model, policy, replica	arrival to first byte of a streamed response
inferrail_request_duration_seconds	model, policy, replica	arrival to the end of the response
inferrail_routing_decisions_total	model, policy, replica	the policy's choices, one per attempt
inferrail_routing_decision_seconds	policy	time to choose and take a lease
inferrail_upstream_retries_total, inferrail_client_disconnects_total	model	retries before the first byte, abandoned responses
inferrail_request_errors_total	model, reason	errors the gateway answered itself
inferrail_tokenized_requests_total	model, result	prompts tokenized, or why a request got no prediction: invalid, unsupported, template, busy, slow, unavailable
inferrail_tokenize_seconds	model	time to tokenize and hash a prompt, waiting for a worker included
inferrail_prompt_tokens_total, inferrail_predicted_cached_tokens_total	model	prompt tokens and the cached tokens predicted on the chosen replica, per decision
inferrail_tokenize_cache_lookups_total	model, result	repeated prompt pieces served from the tokenization cache
inferrail_index_entries, inferrail_index_bytes, inferrail_index_resets_total	replica for resets	blocks the index holds, its memory, engines found restarted
inferrail_estimated_ttft_seconds	model, policy	the cost model's estimate for the chosen replica, at every decision, to set against inferrail_ttft_seconds
inferrail_routing_guards_total	model, guard	decisions where bounded_load or kv_pressure set replicas aside, or kv_waived when every replica was under KV pressure
inferrail_routing_steered_total	model, reason	decisions decided by session affinity, won by a hot_prefix owner only through the credit for a prefix it did not hold yet, or routed by the no_prefix fallback
inferrail_ejections_total, inferrail_load_scrapes_total	replica (and result)	passive ejections, engine scrapes
inferrail_config_reloads_total, inferrail_config_generation	result for reloads	reload attempts from the file, SIGHUP or the admin API, applied, unchanged or rejected, and the config generation in effect
inferrail_model_policy, inferrail_model_policy_percent	model, policy	the policies routing each model, and each one's percent of its traffic
inferrail_shadow_decisions_total	model, policy, agreed	requests a shadow policy chose a replica for, by whether it agreed with the routing policy
inferrail_shadow_estimated_ttft_seconds_total	model, policy, pick	the cost model's estimated TTFT summed over shadowed requests, for the routing policy's pick (active) and the shadow's (shadow)
inferrail_shadow_dropped_total		requests left out of the shadow because its worker was behind
inferrail_replica_*	replica	health, ejection, in flight requests and prompt tokens, queued prefill, the engine's running, waiting, KV usage and prefix cache counters, the age of its report, and the blocks and capacity the index assumes

Replica gauges are read from the fleet when Prometheus scrapes, series of replicas that leave are removed, and recording one streamed request costs about 160 ns.

Traces. With observability.otlp_endpoint set, every request becomes a server span, and every attempt a route span (policy, candidates offered, winner, its in flight count and engine load, decision time, prompt tokens and the winner's predicted cached tokens) and an upstream client span, exported over OTLP HTTP and sampled by trace_sample_ratio. A bare collector address such as http://collector:4318 gets the standard /v1/traces path. A client's W3C traceparent is continued, and the engine receives the gateway's span as its parent. Route spans also carry the winner's estimated time to first token and what steered the decision. A warning is logged when export starts failing and a note when it recovers. Without an endpoint the spans are never created.

Decision log. With observability.decision_log.sample_rate above zero, sampled requests write one JSON line per routing decision to decision_log.path: the policy, the attempt, the prompt's tokens, the winner, its estimated time to first token, what steered it and what the guards set aside, and every candidate's state, load, queued prefill and predicted cached tokens. Requests are sampled by id, so all attempts of a request are kept together. A writer thread drains a bounded queue; if the disk falls behind, records are dropped and counted, never waited for. Prompts are never logged. See ADR 006.

Dashboard. make stack builds the gateway and simulator image and brings up three simulators, the gateway, Prometheus, Grafana and Jaeger, published on localhost only. The provisioned dashboard (dashboards/inferrail.json) shows TTFT per replica and per policy from both the gateway and the engines, in flight requests, engine queues and KV usage per replica, each policy's traffic share, prefix cache hit rates, decision time, retries, ejections and errors, and the predicted hit rate next to the engines' own, tokenization outcomes and time, the blocks the index holds per replica, the cost model's estimated TTFT next to the measured one, queued prefill per replica, how often the load guards and steering act, and the control plane: the config generation and reloads, each model's split, and every shadow policy's agreement and estimated gain. The compose gateway runs least_request and prefix_hash in shadow so those panels have data.

Routing

Every policy implements one RoutingPolicy trait and is chosen per model in the config, so baselines and the main policy run in the same binary on the same candidates. The gateway only ever offers a policy replicas that serve the model, are healthy and have room. Each comes with its requests in flight, the prompt tokens the KV index predicts it has cached, the prefill work queued on it, its running requests and, while the engine's report is fresh, its KV cache usage. The policy also sees the prompt's length and block hashes and the conversation it belongs to. See ADR 004 and ADR 010.

Policy	Rule
round_robin	next replica in turn
random	uniform choice
least_request	fewest requests in flight, ties broken at random
p2c	two random replicas, the less loaded wins
prefix_hash	rendezvous hash on the prompt's first prefix_hash.blocks blocks, whatever the load; power of two choices for prompts without a full block
kv_aware	best cache_weight * cached / prompt - load_weight * in_flight / busiest, ties to fewer requests in flight
cost_model	lowest estimated time to first token among the replicas the load guards allow

The cost model. For each candidate cost_model estimates

TTFT = W + P(u, c) + X
P(u, c) = a*u + b*u*(c + u/2) + d

where u is the prompt tokens the replica would compute, c those it has cached, and a, b, d come from routing.cost_model or the model's own cost_model. W is the prefill already queued there plus decode_interference_secs per running request. The gateway keeps that queue itself: every routed request adds its estimated prefill to its replica's queue, which drains in real time and counts as empty once every request on the replica has had its first byte, so a replica of a fresh fleet costs nothing to wait for and one with three long prompts ahead costs their prefill. X prices cached tokens held in CPU or disk tiers by their load time and only counts when loading beats recomputing; it is zero until the index tracks tiers. Ties go to fewer requests in flight, then to chance.

Load guards. cost_model never picks a replica that

• would hold more than ceil((1 + epsilon) * (in_flight + 1) / n) requests, twice its share with the default epsilon of 1.0 (bounded load, after consistent hashing with bounded loads, so some replica always qualifies),
• reports more than kappa (0.92) of its KV cache in use, for a prompt with more than short_prompt_tokens (256) tokens to compute there, unless every replica does.

A root prefix (its first hot_prefix.blocks blocks) whose prefill demand, counted in a decaying count min sketch over hot_prefix.window, needs more than hot_prefix.utilization of a replica gets ceil(demand / utilization) owners, priced as if they held the longest prefix an owner holds that is that hot itself, so the cost model shares it among them by their queues and a conversation's history counts only where it is. With session_affinity, a request naming its conversation in x-session-id or the user field goes back to the conversation's last replica while the guards allow it. Why epsilon is 1.0 rather than the 0.25 the design started from is in ADR 011, with the measurements.

Explaining a decision. POST /admin/explain/v1/chat/completions (or /v1/completions) on the admin port takes a request as a client would send it and answers with what the model's policy would do now: the prompt's length and blocks, every replica serving the model, whether it was offered and why not, its load, queued prefill, predicted cache, any guard that set it aside and its estimate, the winner, and the term (queue, prefill or tier_load) that put it furthest ahead of the runner up. Nothing is routed and no state changes; bodies are read as on the client port, within the same timeout and size limit.

{
  "policy": "cost_model",
  "prompt_tokens": 625,
  "winner": "r0",
  "decided_by": "prefill",
  "replicas": [
    {"replica": "r0", "cached_tokens": 592, "queued_secs": 0.0,
     "estimate": {"queue_secs": 0.0, "prefill_secs": 0.0084, "tier_secs": 0.0, "ttft_secs": 0.0084}},
    {"replica": "r1", "cached_tokens": 0, "queued_secs": 0.0,
     "estimate": {"queue_secs": 0.0, "prefill_secs": 0.0681, "tier_secs": 0.0, "ttft_secs": 0.0681}}
  ]
}

Calibration. inferrail-calibrate --url <engine> sends an idle engine prompts of known cached and uncached length, one at a time, the cached part sent alone first so the probe finds it cached. Each probe's TTFT comes from the engine's own histogram (vLLM and SGLang export one), so network and HTTP time stay out of the fit and are reported apart, and its cached tokens from the engine's own prefix cache count, so an engine that cached less than planned is fitted on what it computed. Every run sends fresh prompts from its own seed, recorded in the output. A least squares fit that keeps every term non negative gives a, b and d, printed or written with --out as a cost_model fragment. Against the simulator it recovers all three to nine significant digits.

Cost. Randomized policies are seeded from routing.seed, so a fixed seed replays the same choices. A full cost_model decision for an 8,000 token prompt, the index lookup of every candidate included, takes 7 µs at p99 over 64 replicas and 20 µs over 256 (benchmark A5, target 200 µs); a baseline choice takes nanoseconds.

Control plane

Live config. routing, each model's policy, split, shadow, cost_model and aliases, the static replicas, retry and logging.log_prompts change while the gateway runs, without dropping a request. The gateway rereads its config file every server.reload_interval (2 s, off at 0) and applies it when it differs from the config in effect; SIGHUP rereads it at once. A request reads one snapshot of the config and its policies, so a reload never changes a decision halfway and its retries use the policy that routed it. Policies whose settings did not change keep their state (a round robin rotation, a hot prefix sketch, a session table), unchanged replicas keep their health and counters, and requests on a removed replica finish there. A change to anything built at startup (listeners, the upstream client, discovery, health checks, load scraping, telemetry, tokenizers, the index or the model list) is refused whole, naming the sections, and the running config stays. The reasoning is in ADR 015.

Admin API. On the admin port, next to /metrics and /admin/explain:

Route	What it does
GET /admin/config	the config in effect and its generation
PATCH /admin/config	applies a JSON merge patch to the config in effect, leaving the file alone, so the file's next edit replaces the patch
POST /admin/reload	rereads the config file now
GET /admin/index	KV index entries, bytes per entry, and each replica's predicted blocks against its capacity
GET /admin/replicas	health, ejection, requests in flight, queued prefill, and each engine's last load report with its age
GET /admin/shadow	each shadow policy's decisions, agreement with the routing policy and estimated gain

Every reload attempt counts in inferrail_config_reloads_total{result}, and inferrail_config_generation names the config in effect. Admin routes have no authentication yet, so keep server.admin_listen on localhost or a private network.

A/B splits. routing.split, or a model's own split, gives each policy a percent of the traffic. A request's arm follows from its conversation key, or its request ID when it names none, so a conversation stays with one policy and a retry keeps its arm. Each arm has its own policy state; every metric already carries the policy label, responses name the policy in x-inferrail-policy, the harness records it per request, and inferrail_model_policy_percent reports the shares. /admin/explain?policy= explains any arm.

Shadow policies. routing.shadow, or a model's shadow, lists policies that choose a replica for every routed request without routing it. They see exactly the candidates the routing policy saw, in the same state, and run on one worker thread off the request path; when it falls behind, requests are left out of the shadow and counted in inferrail_shadow_dropped_total, never delayed. The model's cost model prices each shadow pick next to the routing pick, giving agreement, the share of requests priced faster or slower, and the mean estimated gain on /admin/shadow and in metrics. That is the cost model's own view, which favors cost_model by construction, so a shadow's gain is a reason to try it in a split, not proof. With observability.shadow_log every config generation and every shadow decision, with what the shadows saw, is written as JSON lines; inferrail-gateway --replay-shadow <log> rebuilds each shadow from the logged configs, replays the requests in order and checks every pick, so the log is evidence anyone can rerun.

The simulator

Real GPUs are rented, not owned, so most development and every CI run happen against inferrail-sim. It has to be faithful enough that a router tuned on it behaves the same on vLLM.

Wire format. Chat and completion responses, stream chunks, the usage chunk, data: [DONE], model cards and error bodies match vLLM v0.31.0 byte for byte, including which fields are omitted and which are explicit null. Request validation follows vLLM as well; for example a non empty stream_options without stream: true is a 400. Parity tests compare the simulator against fixtures derived from that version's source.

Endpoints. POST /v1/chat/completions, POST /v1/completions, GET /v1/models, GET /health, GET /metrics, GET /version and POST /reset_prefix_cache. An x-request-id header becomes the response id, as in vLLM.

Scheduler. One task runs the engine in steps, as vLLM's V1 scheduler does. Each step has a token budget (--max-num-batched-tokens, 2048): running requests first, in the order they started, each decoding one token or computing the next chunk of its prompt, then waiting requests in arrival order while sequences and budget last. A waiting request that does not fit blocks the ones behind it. A running request that needs a block when none is free preempts the newest running request, which gives its blocks back and is recomputed later, counted in vllm:num_preemptions. Steps are planned on a virtual timeline, each starting when the previous one ends, so timer overshoot never accumulates. See ADR 012.

Latency model. A step costs its prefill chunks plus its decode batch. Prefill of u uncached tokens attending to c cached ones costs

P(u, c) = a*u + b*u*(c + u/2) + d

the same shape as the router's cost model. The chunks of one prompt add up to exactly P(u, c), so calibration recovers the parameters and an idle engine's TTFT is still P to the nanosecond. Decoding n sequences costs decode_token_secs + decode_seq_secs * (n - 1) per step (20 ms and 0.1 ms by default), so prefill stretches the decode steps it shares and a big batch decodes slower. TTFT and prefill time histograms are stamped when the model produces the first token, as vLLM stamps them in its engine core. The defaults approximate an 8B model on one 24 GB GPU.

KV cache. Prompts are cut into full blocks, hashed as a chain so each hash names its whole prefix, and held in one ref counted pool. Running requests hold blocks by reference, so a shared prefix counts once in KV usage; a freed block keeps its hash and serves later hits until the pool hands it out again. Free blocks leave in vLLM's order: unhashed ones first, then cached ones least recently freed, a request's tail before its head. Blocks are cached when their computation is scheduled, generated ones included, and a fully cached prompt still recomputes its last block. --no-enable-prefix-caching keeps nothing.

KV events. With --kv-events-config '{"enable_kv_cache_events": true}' every step's stores, evictions and clears go out as one numbered batch on a ZeroMQ PUB socket (tcp://*:5557 unless endpoint says otherwise), in vLLM v0.31.0's msgspec encoding byte for byte, with stored blocks carrying their token ids and parent hash. A replay_endpoint answers requests for the last buffer_steps batches. inferrail-events speaks ZMTP 3.0 itself, so nothing links libzmq, and pyzmq subscribers read the stream unchanged. See ADR 013.

Tokenizer. A deterministic stand in whose token boundaries depend only on the text to their left, so shared text means shared tokens. Generated text retokenizes to the same ids, so multi turn reuse works as it does on a real engine. It lives in inferrail-tokenize, so a gateway configured with tokenizer: builtin:sim tokenizes exactly as its simulators do.

Metrics. Every default vLLM family the simulator can model: running and waiting requests, KV usage, prefix cache queries and hits, token counters, finish reasons, TTFT, inter token latency, end to end, queue, prefill and decode time histograms, with vLLM's names, help text, labels, buckets and prometheus_client text format.

Cancellation. A client that disconnects while its request waits withdraws it at once; one that is running leaves at the next step boundary, as vLLM aborts between steps. Either way its sequence slot and KV blocks are released and it is counted as an abort.

Useful flags (all names follow vLLM where one exists):

Flag	Default	Meaning
--served-model-name	model path	names clients may request
--block-size	16	tokens per KV block
--num-gpu-blocks-override	4096	KV cache capacity in blocks
--max-num-seqs	256	running sequence cap
--max-num-batched-tokens	2048	tokens one step may compute; longer prompts prefill in chunks
--max-model-len	32768	context length
--no-enable-prefix-caching	caching on	keep no finished blocks
--enable-prompt-tokens-details	off	report cached tokens in usage
--kv-events-config	off	publish KV events, as vLLM's flag of the same name
--prefill-token-us, --prefill-pair-ns, --prefill-fixed-ms	100, 3.3, 5	a, b and d above
--decode-token-ms	20	decode step time for one sequence
--decode-seq-us	100	time each further sequence adds to a decode step
--zero-latency	off	answer as fast as possible, for overhead benchmarks

The simulator refuses what vLLM refuses: a KV pool too small for one request of --max-model-len, or a token budget below --max-num-seqs. Not modeled yet: KV blocks offloaded to CPU memory or disk, adapters, speculative decoding and engines spread over several GPUs. Requests that need n > 1, logprobs, echo or best_of get a 400 naming the parameter instead of an answer that silently differs from vLLM; stop strings are ignored.

Benchmark harness

bench/ holds inferrail-bench, which measures policies the way the benchmark plan requires (see ADR 007):

• Open loop. Requests arrive on a seeded Poisson schedule and each is sent at its time whatever the earlier ones are doing. TTFT and end to end latency count from the scheduled time, and the gap between scheduled and actual send time is recorded, so a server that falls behind cannot hide its queueing.
• Workloads. W1: a handful of 2k to 8k token system prompts shared by every user, with 50 to 500 token questions. W2: multi turn chats whose history is resent each turn, each user thinking 10 to 20 s before the next question, generated or replayed from a ShareGPT file, each request naming its conversation in x-session-id. W3: agent loops that resend a context growing by a tool call and its result each step, over shared tool definitions. W4: RAG, one to four documents drawn by Zipf weight from a corpus of 2,000 after a short shared instruction. W5: few shot evaluation, short questions after a shared block of worked examples. W6: prompts that share nothing, the control where cache affinity must not hurt. W8: shared prompts arriving in Gamma bursts over a diurnal ramp. mix: W1 and W6 requests arriving together, half each, so a policy must weigh the cache one kind finds against the prefill the other queues. hot: W1 with its system prompts drawn by Zipf weight (exponent 1.1), so a few are hot.
• Arrivals. Poisson by default; any workload can take Gamma gaps of a chosen coefficient of variation (--arrivals gamma --cv 4) or a diurnal ramp (--ramp diurnal) that keeps the mean rate.
• Measurements. TTFT, time per output token, inter token latency, end to end latency, throughput, goodput against the default SLO (TTFT at most 2 s, TPOT at most 100 ms), the prefix hit rate from usage and from the engines' own counters (read from vLLM or SGLang over the same requests, after the warmup), errors, send lag, requests per replica and the busiest replica's share. Time per output token counts tokens from usage, not stream chunks.
• Prediction accuracy. When the gateway tokenized a request, its predicted prompt and cached tokens are compared with the engine's usage: the share of exact predictions, of over and under predictions, the mean error in tokens and the predicted hit rate. Harness gateways tokenize with builtin:sim.
• Fair comparisons. compare and sweep give every policy the same requests and arrival times and a cold gateway and simulators per run, discard a warmup, and repeat with new seeds.
• Sweeps. inferrail-bench sweep runs a benchmark matrix from one TOML file: shared settings and a list of points, each a workload with its rates and overrides. Policies run innermost, so drift over a long sweep falls on every policy alike; --jobs runs several fleets at once, and a stopped sweep resumes, skipping runs whose result file matches their specification. inferrail-bench report turns result files alone into the milestone verdict, the benchmark targets the simulator speaks to, a headline per point, knee and goodput charts and a table per rate. See ADR 014.

make sweep       # bench/sweeps/sim.toml: 372 runs, then the report
make routing     # every policy on W1, W2, W6, mix and hot, 3 repetitions
make baselines   # the four cache blind baselines on W1, W2 and W6
make report      # charts and reports from bench/results

The committed sweep in bench/results/sweep runs every policy on every workload on three continuous batching simulators with the default latency model of an 8B model on one 24 GB GPU: 372 runs and 146,099 requests from one clean commit, none failed. Main rates sit near 70% of what a cache blind fleet sustains: W1 with 32 system prompts at 6 req/s, W2 at 15, W3 at 3, W4 at 9, W5 at 12, W6 at 9, mix at 7, hot at 10 and W8 at a mean of 5 over its diurnal ramp. Medians over three repetitions, p99 TTFT in ms:

Policy	W1	W2	W3	W4	W5	W6	mix	hot	W8 CV 2	W8 CV 4
round_robin	3,498	490	8,778	593	83	1,470	4,492	3,613	6,113	16,514
random	6,431	1,598	14,625	983	123	3,154	6,250	3,132	8,869	16,015
least_request	4,135	756	8,030	761	91	1,416	4,277	3,489	6,584	14,936
p2c	4,949	726	8,957	843	97	1,837	4,070	2,430	8,523	13,570
prefix_hash	1,112	655	79,786	889	117	3,507	3,075	1,245	1,357	4,588
kv_aware	1,044	552	2,443	761	94	1,406	1,321	1,121	1,126	1,410
cost_model	994	409	1,524	512	92	909	1,200	989	1,012	1,398
cost_model hit rate	0.88	0.23	0.88	0.25	0.94	0	0.44	0.87	0.85	0.87

What the numbers say:

• The milestone. cost_model has a lower p99 TTFT than least_request on W1 to W3 without losing goodput: 994 against 4,135 ms on shared system prompts, 409 against 756 ms on conversations and 1,524 against 8,030 ms on agent loops. It keeps 88% of W1's and W3's prompt tokens cached where the cache blind policies keep 39 to 57%.
• Capacity. On W1, round robin and least_request fall apart past 6 req/s: at 8 their goodput drops to 0.3 and 0.2 req/s with p99 TTFT over 10 s. cost_model and kv_aware keep goodput at the offered rate up to 10 req/s, the highest rate swept, with p99 under 1.1 s, so the same three replicas carry at least two thirds more shared prompt traffic when routed on the cache.
• Agents and bursts. On W3 cost_model cuts least_request's p99 by 81%, while prefix_hash sends every agent sharing tool definitions to one replica and reaches 80 s. Under Gamma bursts with a CV of 4 over a diurnal ramp (W8), the cache blind policies keep at most a sixth of cost_model's goodput.
• No regression. With nothing to share (W6) cost_model still has the lowest p99, 36% below least_request, because queued prefill is the better load signal. On W5, where every policy finds 94% of prompt tokens cached, all are within noise and round robin has the lowest p99 (83 against 92 ms).
• Targets. Against the benchmark plan: an 88% engine hit rate on W1 (target 80%), p99 TTFT 72% below round robin on W1 (target 40%), and goodput 44% above least_request at the median point (target 30%).
• Where it loses. Past W3's knee kv_aware beats cost_model: at 4 req/s its p99 is 4.1 s against 15.4 s, as cost_model's hit rate falls from 88% to 73%, and 56% at 5 req/s, where kv_aware keeps 78% and 68%. W2 at 25 req/s, kept as an overload point, saturates every policy (p99 16 to 20 s).
• Predictions. The approximate index predicted the engine's cached tokens exactly on 94% of requests and over predicted on 4%: generated blocks and concurrent prefills now shape the simulator's cache in ways the router's own decisions do not show. The exact index, fed by the KV events the simulator now publishes, closes that gap.

The full report, with the milestone verdict, the targets, knee and goodput charts and every table, is in bench/report/sweep/report.md. The earlier comparisons on the FIFO simulator, routing and baselines, stay in the repository but are not comparable with these numbers.

Configuration

The gateway reads one YAML file. Only models and at least one replica or DNS group are required; everything else has a safe default, unknown keys are rejected, and every problem is reported at once with its path.

server:
  reload_interval: 2s
models:
  - name: meta-llama/Llama-3.1-8B-Instruct
    aliases: [llama-8b]
    policy: least_request
    tokenizer: /models/Llama-3.1-8B-Instruct
    chat_template_kwargs:
      enable_thinking: false
replicas:
  - id: gpu-0
    url: http://10.0.0.10:8000
    models: [meta-llama/Llama-3.1-8B-Instruct]
    max_concurrency: 256
    max_pending: 32
    kv_capacity_blocks: 18000
discovery:
  dns:
    - name: llama-pool
      url: http://vllm.inference.svc.cluster.local:8000
      models: [meta-llama/Llama-3.1-8B-Instruct]
      refresh: 2s
routing:
  default_policy: cost_model
  split: {cost_model: 90, kv_aware: 10}
  shadow: [least_request]
  epsilon: 1.0
  kappa: 0.92
  session_affinity: true
  seed: 42
  cost_model:
    prefill_token_secs: 0.0001
    prefill_pair_secs: 0.0000000033
    prefill_fixed_secs: 0.005
health:
  interval: 1s
  timeout: 500ms
  unhealthy_threshold: 3
  ejection_backoff: 5s
retry:
  max_attempts: 3
  backoff: 25ms
load:
  scrape_interval: 100ms
  max_age: 250ms
tokenizer:
  workers: 4
  max_wait: 50ms
index:
  default_capacity_blocks: 16384
observability:
  otlp_endpoint: http://localhost:4318/v1/traces
  trace_sample_ratio: 0.1
  decision_log:
    path: /var/log/inferrail/decisions.jsonl
    sample_rate: 0.01
  shadow_log: /var/log/inferrail/shadow.jsonl

See deploy/inferrail.example.yaml for every option with its default, and deploy/local.yaml for the quick start fleet.

Testing

make check runs formatting, clippy with warnings denied, ruff, every Rust test and the Python tests. make audit checks dependencies against the RustSec advisory database and for unused crates. Beyond unit tests the suite includes:

• Golden parity: token ids for 648 requests across Qwen3, Granite 3.3 and Hermes 2 Pro equal what vLLM v0.31.0 computes with transformers 5.17, with tools, multi turn conversations, reasoning, documents, content parts, special tokens in text and continued messages, cold and warm, with and without the tokenization cache.
• Template compatibility: the variables a template reads, found as Jinja2 finds them on real chat templates, numbers read as json.loads reads them, and repr, title and capitalize checked against CPython 3.12, each with the expectations Python itself produced.
• Property tests for SSE decoding under any chunking, tokenizer prefix stability, block hash chaining (a changed token changes its block and every later one, never an earlier one; a prefix hashes the same alone or inside a longer prompt), the KV index against a naive reference model after any sequence of records, records taken back, clears and departures (a reported prefix of depth d means blocks 0 to d minus 1 are held and block d is not), arbitrary request bodies never panicking the tokenizers, the prefix cache LRU, latency monotonicity, config round trips, uniform policy choices under fixed seeds, least request always picking a minimum, power of two choices never picking a unique maximum, fleet membership, replica health and lease token counters against naive reference models, the request splice changing nothing but the model, metrics pages that never crash the parser, and rendered pages that always parse back to the reference totals.
• Routing invariants: across any random sequence of arrivals and departures, with sessions and hot prefixes in play, cost_model never gives a replica more than ceil((1 + epsilon) * average) requests; some replica is always within the cap; explaining never changes the next choice, and whenever it names a winner, routing picks that replica with the same steer and guards; more cache never costs more and a tier hit never costs more than ignoring it; the count min sketch never underestimates; the calibration fit recovers random parameters through noise. The worked example of the design (a cached replica behind a long queue loses to an idle cold one, and wins once the cold one is busier) is a test.
• Fleet simulations (crates/inferrail-policy/tests/fleet.rs): every policy routing the same seeded traffic over replicas that prefill at the simulator's speed and evict their prefix caches. The cache aware policies keep at least 80% of W1's prompt tokens cached where round robin keeps 37%; cost_model has a 26% lower p99 TTFT than kv_aware on a mix of shared and unique prompts; under Zipf traffic it puts at most 29% of requests on one of four replicas where prefix_hash puts 39%; and a hot system prompt that sends prefix_hash past 20 s at p99 leaves it under 0.3 s. Multi turn chats over one hot system prompt keep their hit rate and p99 with hot prefix replication on, as with it off.
• Gateway invariants: no policy ever routes to an unhealthy replica or one that does not serve the model; bytes the engine sends reach the client unchanged under any chunking; request bodies reach the engine byte for byte; no request, however malformed, gets a gateway failure; a request succeeds whenever a healthy replica is within its attempts; every request is counted in exactly one metrics outcome; metrics of departed replicas never outlive them, even when requests race their removal.
• Simulator scheduling: random schedules with shared prompts, preemption and cancellations keep the sequence cap and arrival order, and leave no block held once they drain; the block pool matches a naive reference model under any sequence of operations; a long prompt's chunks add up to its idle P(u, c) within a microsecond; TTFT bends at the engine's capacity and more load never lowers it; decode steps stretch with the batch and with prefill; and a dry pool preempts but finishes every request.
• KV events: batches encode byte for byte as vLLM's msgspec does, checked against fixtures it produced; pyzmq subscribes to the simulator; and a mirror built from the stream alone equals the engine's cache after concurrent traffic, eviction and resets, with every stored block's hash rebuilt from the token ids it carries.
• Control plane: 400 requests, half streamed, through 30 reloads that switch policy, epsilon and the replica set every 100 ms all complete; unchanged policies keep their state; restart only changes are refused whole; a 75/25 split routes within statistical bounds and keeps 40 conversations on one arm each; 300 requests over two config generations with every policy in shadow replay offline to all 1,800 picks, and a tampered line is caught.
• Fuzzing: random wire payloads must round trip, random engine schedules must keep FIFO admission and clean accounting, random HTTP requests must get responses that obey the vLLM contract, and hostile YAML fails fast.
• Integration tests against three in process simulators that can be killed for real or made to refuse, close, hang, delay or fail on demand: routing, retries, ejection and readmission, DNS joins and leaves, breakers, disconnects, graceful shutdown, header timeouts and log hygiene; load snapshots that stay under 250 ms old, engines that die or serve hostile metrics pages; every metric and its labels (a snapshot test), span attributes and trace propagation through an in memory exporter, real OTLP export from the binary, and the decision log.
• Consistency tests: every dashboard query names a metric the gateway or vLLM exports, and the compose stack and its configs are valid.
• Prediction tests: predicted cached tokens equal what three simulators really cached on every one of 300 requests under eviction pressure, through streams and completions, with capacity learned from engine metrics, after an engine restart, and when a tokenizer cannot load.
• Routing tests against in process simulators: the cache aware policies serve 92 to 95% of shared prompt tokens from cache where round robin serves 37%; sessions stick to their replica until it is overloaded and then follow the new one; queued prefill is estimated while it waits and drains after; /admin/explain output is a snapshot, sends nothing to any engine, and marks replicas that were unhealthy or full.
• Calibration tests: inferrail-calibrate against simulators with two different parameter sets recovers a, b and d within 5%, with every probe's engine time equal to the latency model to a microsecond, and does so again on a second run against the same simulator and against one that caches nothing.
• Harness tests: workload shapes and determinism, Poisson statistics, TTFT, TPOT and ITL against servers with known timing, coordinated omission against a server that falls behind, a blocked client loop showing up as lag, prediction accuracy, results, reports and a full comparison on real binaries.
• Sweep tests: sweep files and their errors, run order, resuming by specification, parallel jobs, a report rebuilt byte for byte from result files, and the same point run twice on fresh fleets agreeing within 3% on mean TTFT, end to end latency, TPOT, hit rate and goodput.
• SDK tests (make e2e): the official OpenAI Python SDK, sync and async, against the simulator binary and through the gateway binary in front of three simulators.
• Stress tests (make stress), on an Apple silicon laptop with client, gateway and simulators sharing one machine:

Scenario	Result
2,000 concurrent streams through the gateway over 3 simulators	all complete in 0.3 to 0.7 s, zero errors, even spread
Open loop at 2,000 and 5,000 req/s, 10,000 requests each, with metrics and load scraping on	the gateway adds about 0.1 ms at p50 and under 0.25 ms at p99
The same at 5,000 req/s with the decision log at full sampling and traces exported to an unreachable collector	the gateway adds 0.17 ms at p50 and 0.36 ms at p99; 15,000 decisions logged, none dropped
A simulator killed under 1,000 req/s, then restarted	zero errors for requests sent after the kill, ejected within 10 ms, back in rotation within 60 ms
A replica that hangs, with requests waiting on it	they move to another replica as soon as probes mark it down
Requests that make every engine fail	at most half the replicas are ejected; the pool never empties
A replica flapping between healthy and 503 every 100 ms at 1,000 req/s	zero client errors
1,000 streams with 741 random disconnects	every disconnect aborted in the engine, nothing leaked
3,000 hostile requests	zero gateway failures
35,000 request soak through the gateway	resident memory grew under 10 MB
640 MiB streamed to 20 clients that stop reading	about 20 MB held in total
An 8 MiB request body sent in one byte chunks	about 11 MB held while it arrives
1,000 concurrent streams through the async SDK, direct and via the gateway	all complete with exact usage
2,000 concurrent SSE streams straight to one simulator	all complete in 0.6 s, zero errors
35,000 request simulator soak	resident memory grew under 20 MB
Engine load reports under 1,000 req/s through 3 simulators	every reader saw a report under 102 ms old (p99 100 ms); never stale
64 replicas scraped every 100 ms under 2,000 req/s	oldest report 112 ms, zero errors
/metrics scraped 100 times a second during 2,000 concurrent streams	every stream completes; counters and histograms consistent on every scrape
20 replicas serving 50 MB or endless metrics pages for 5 s	the healthy replica's report never older than 104 ms; each scrape capped at 4 MiB and 100 ms
8,000 replicas churned through discovery	the metrics page never exceeds 40 KB and renders in 0.05 ms
The harness driving zero latency simulators at 300 req/s	every request sent within 2.2 ms of its schedule at p99
Open loop at 2,000 and 5,000 req/s with tokenization and the index on, 200 word system prompts	the gateway adds 0.3 to 0.5 ms at p99
The same with 2,000 word system prompts at 1,000 and 2,000 req/s	the gateway adds 0.3 to 0.5 ms at p99
3,000 requests at 400 req/s over 3 simulators	every prompt and cached token prediction exact
400 concurrent 20,000 word prompts through one tokenizer worker	all served; those that could not wait were routed without a prediction
40,000 distinct prompts soaked through the index	it stays within the replicas' KV capacity, RSS grew 43 MB
12 threads recording, 6 looking up and 2 forgetting for 5 s	the shared map equals the shadow caches exactly afterwards; lookup p99 31 µs
A million blocks over 64 replicas	148 bytes per block, 512 block lookups at 7 µs p99
16 threads sharing one tokenizer and its cache	20 to 33 million tokens per second, every id exact
4 MB messages, 50,000 messages, 1,000 tools, nested arguments	each tokenized or refused within half a second
9,621 more golden requests	every token id matches
A full cost_model decision for an 8,000 token prompt, index lookup included, 20,000 times	p99 6.5 µs at 8 replicas, 7 µs at 64, 20 µs at 256
16 threads sharing one policy of each kind, sessions and hot prefixes in play	800,000 decisions each, every one valid
Open loop at 2,000 and 5,000 req/s over 8 replicas with each cache aware policy, three runs	decisions under 50 µs at p99; the gateway adds 0.2 to 0.4 ms at p50, and its p99, 0.9 to 3.4 ms with client, gateway and eight simulators on one laptop, varies as much between runs as between policies
4,000 requests at 400 req/s over 4 simulators, Zipf over 16 system prompts, 300 sessions, explain requests mixed in	the busiest replica took 25.9 to 27.3%; 97 to 98% of each session's turns stayed on its replica; zero errors
10,000 requests at 1,000 req/s through a simulator publishing KV events, mirrored by a subscriber	6,348 batches without a gap; the mirror equals the engine's cache and the last 1,000 batches replay exactly
Open loop at 5,000 req/s with six policies in shadow and the shadow log on	no measurable cost at p50 or p99, nothing dropped; the 18.5 MiB log replays offline to every pick in 0.06 s
239 config patches every 20 ms, with live policies, splits, shadows and replicas, and 130,000 admin reads, under 2,000 req/s	zero errors; each patch applied in 0.08 ms at p50 and 0.2 ms at p99

The suite runs clean ten times in a row with every core saturated.

Repository layout

crates/
  inferrail-core/        wire types, SSE codec, configuration
  inferrail-gateway/     the router: proxy, retries, health, metrics, traces
  inferrail-policy/      routing policy trait and baseline policies
  inferrail-control/     replica registry, health, ejection, DNS discovery
  inferrail-load/        engine metrics parsing, scraping, load snapshots
  inferrail-sim/         mock vLLM engine (binary and library)
  inferrail-tokenize/    chat templates, tokenizers, block hashing
  inferrail-index/       KV block index
  inferrail-events/      vLLM KV event codec and ZMTP publisher and clients
  inferrail-calibrate/   engine profiler that fits the cost model
bench/                   benchmark harness and the OpenAI SDK suites
  sweeps/                benchmark matrices, one TOML file each
  results/               raw results of committed runs
  report/                charts and reports generated from them
dashboards/              Grafana dashboard JSON
deploy/                  example, local and compose configurations
tests/fixtures/          OpenAI, vLLM and SGLang payloads, tokenizer goldens
docs/adr/                architecture decision records

Roadmap

The aim is a router that serves the same traffic at the same latency target with fewer GPUs than a cache blind balancer, while adding under 1 ms of its own latency. Every number below is a target to prove or disprove with committed, rerunnable benchmarks, not a claim.

• Gateway and baselines. Shipped in v0.1.2: OpenAI compatible proxy, zero copy streaming, retries before the first byte, health checks, outlier ejection, breakers, DNS discovery, cancellation, and four baseline policies behind one trait.
• Load signals and observability. Shipped in v0.1.4: vLLM and SGLang metrics scraped ten times a second and merged with the gateway's own counts, Prometheus metrics, OpenTelemetry traces with routing attributes, a sampled decision log, a Grafana dashboard, and a benchmark harness that compares the baselines on the simulator.
• Tokens, hashes and the approximate index. Shipped in v0.1.6: chat templates and tokenizers with token ids identical to vLLM's, chained XXH3 128 bit block hashes salted by model, adapter and cache_salt, and a KV index built from the router's own decisions that predicts every replica's cached prefix.
• The routing brain. Shipped in v0.1.8: prefix_hash and kv_aware as the cache aware baselines, and cost_model, estimated TTFT per replica as queued prefill plus prefill of uncached tokens plus tier load time, with bounded load, KV pressure guards, hot prefix replication and optional session affinity, inferrail-calibrate to fit it to an engine, and /admin/explain to show why a replica won.
• Realistic simulation and the control plane. Shipped in v0.1.11: continuous batching with chunked prefill, prefill and decode interference, one pool of KV blocks with preemption and KV events in vLLM's format in the simulator; workloads W3, W4, W5 and W8, bursty and diurnal arrivals, and one command for the full sweep and its report; live config, the admin API, A/B splits and shadow policies.
• Exact index. An exact mode fed by engine KV events over ZMQ, with gap detection and fallback to the approximate mode.
• Production controls. SLO aware admission, token based rate limits, API keys and TLS.
• Advanced regimes. Disaggregated prefill and decode pools, tiered KV cache across GPU, CPU and disk, and Kubernetes packaging.
• Proof. A benchmark report against cache blind baselines on synthetic workloads and replayed production traces, with targets such as p99 router overhead under 1 ms, 80% or better prefix hit rate on shared prompts, 40% lower p99 TTFT than round robin, and at least one GPU saved per four.

License

MIT. See LICENSE.