llmleaf

llmleaf is a llm proxy. It proxies different llm providers and their slighty different apis and converts it to a single api surface (enhanced openai-compatible or anthropic).

The origin of this project is that other AI gateways focus on being all encompassing and fast. I wanted a project that is slim, focused and near native performance instead.

Goals:

• fast
• efficient
• light-weight
• extensible

Features

• Stable API endpoints for AI generation (OpenAI compatible, OpenRouter like and Anthropic Message API compatbile)
• Endpoint types: chat, embeddings, rerank, decisions, text-to-speech, speech-to-text, realtime (WebSocket), batch jobs
• Supported modalities: image, audio, video and of course text (though mainly image and text support is focused on)
• Health-aware AI endpoint switchover
• No shared state between instances, which simplifies hosting multiple instances greatly
• Prompt caching
• Reasoning effort selection
• Web socket support (also for SSE and even simple request-response providers)
• Great model list endpoint with all the information you need even from providers that don't support this!

Please use web sockets, it fixes latency and prompt caching issues.

Supported providers

• Native dialects: Anthropic, Google Gemini, Vertex AI, Cohere, Ollama, LM Studio.
• Decisions: TypeSafe JEV (typesafe, alias jev) and OpenRouter.
• Another llmleaf node (llmleaf): chains nodes, e.g. a regional node in front of a central gateway. Every modality maps through, and realtime sessions use the upstream's WebSocket with an HTTP chat fallback.
• OpenAI-wire family: OpenAI, Meta Model API (Muse), OpenRouter, Requesty, Groq, DeepSeek, xAI (Grok), Mistral, Together, Fireworks, Perplexity, Cerebras, Z.AI (GLM), Moonshot (Kimi), MiniMax, Amazon Bedrock, Hugging Face Inference Providers, DeepInfra, Cloudflare Workers AI, OCI Generative AI, Databricks Model Serving, NVIDIA NIM, Baidu AI Cloud Qianfan, Azure OpenAI. Moonshot additionally gets a dedicated provider layer that rewrites tool JSON schemas into the upstream's restricted "moonshot flavored JSON schema" (standard Pydantic/zod output otherwise 400s). Subscription plans ride dedicated kinds where the vendor gives them their own endpoint: zai-coding (GLM Coding Plan, /api/coding/paas/v4) and kimi-coding (Kimi for Coding, api.kimi.com/coding/v1); MiniMax's Token Plan shares the standard endpoint, so minimax-token-plan is an alias of minimax (only the key differs).
• echo for local testing.

Quick start

# Run with the embedded dev config (echo provider, key `local-dev:s3cret`)
cargo run -p llmleaf

# …or point at your own config
cargo run -p llmleaf -- llmleaf.toml

Copy llmleaf.example.toml, fill in provider credentials (use env:VAR indirection — secrets never live in the file), and pass it as the argument. Container image: docker buildx bake image (listens on :8080). Send a request:

curl localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $(printf 'local-dev:s3cret' | base64)" \
  -d '{"model":"demo","messages":[{"role":"user","content":"hi"}]}'

Base64 the id:password credential with no trailing newline — use printf (or base64 -w0), not echo. A stray newline is encoded into the value, so the decoded password becomes pw\n and fails the hash check → 401 unknown api key, even when the configured pw_hash is correct.

See llmleaf.example.toml for the full configuration surface (providers, routes, keys, control plane).

API surface

Consumer endpoints (OpenAI-compatible unless noted):

Endpoint	Purpose
POST /v1/chat/completions	Chat (SSE streaming)
POST /v1/messages	Anthropic Messages dialect
POST /v1/responses	OpenAI Responses dialect (encrypted stateless replay and proxied store/previous_response_id; GET remains a 404-by-design stub)
POST /v1/embeddings	Embeddings
POST /v1/rerank	Rerank (Cohere/Jina/OpenRouter dialect)
POST /v1/decisions, POST /api/alpha/decisions, POST /v1/systemone	Decisions (OpenRouter and TypeSafe JEV dialect)
POST /v1/audio/speech, GET /v1/audio/voices	Text-to-speech
POST /v1/audio/transcriptions	Speech-to-text
GET /v1/realtime	OpenAI Realtime (WebSocket)
POST /v1/batches, GET /v1/batches/{id}[/results]	Batch jobs (ids HMAC-signed + owner-bound with [server].batch_id_secret)
GET /v1/models, GET /v1/openapi.json, GET /healthz	Discovery & health

Read-only admin (optional token): GET /admin/routes, /admin/health, /admin/keys. Official client SDKs for 6 languages live in clients/.

Native API compaction is opt-in. GET /v1/models includes supports_compaction for every entry. The flag uses the primary provider's model capabilities and configured API. Without an override, it is false when support is unconfirmed, including when the provider catalog cannot be fetched.

For an OpenAI-compatible provider whose API supports native Responses compaction, declare it with settings = { chat_api = "responses", supports_compaction = true }. This overrides the catalog flag for that provider's language models, including configured routes whose upstream catalog is unavailable. Set supports_compaction = false to report no support. Omit the setting to use detected support. Compaction still requires an explicit request setting and an upstream that implements it.

For Claude, send compaction: {"type":"summarize"} to /v1/messages for on-demand compaction, or context_management: {"edits":[{"type":"compact_20260112"}]} for threshold compaction. llmleaf sends the required beta header and preserves returned compaction blocks and signatures. For on-demand compaction, replace the summarized transcript with the returned block, keep the same system prompt and tools, then append the next turn. See the Claude compaction docs.

For OpenAI Responses, send context_management: [{"type":"compaction","compact_threshold":200000}] to /v1/responses. Replay the returned encrypted compaction items with their IDs, or continue with previous_response_id. llmleaf preserves those items in collected and streaming responses. This supports inline compaction. The standalone /v1/responses/compact endpoint is not exposed. See the OpenAI compaction docs.

The chat-completions surface accepts these native request fields for routes that support them. It returns compaction blocks in the message.compaction array or streamed delta.compaction array. Replay those blocks as message.compaction on the next request.

Decisions

Send model, state, and named questions to /v1/decisions. The OpenRouter path /api/alpha/decisions and JEV path /v1/systemone accept the same body. Questions can use noul, choice, or score, including structured instructions and criteria. Responses preserve answers, probabilities, confidence, and provider-reported usage.

The jev route in llmleaf.example.toml uses OpenRouter. To call TypeSafe directly, add a provider with kind = "typesafe", credential = "env:TYPESAFE_API_KEY", and route to it with upstream model jev-latest. The jev provider kind is an alias for typesafe. Decisions use the same key permissions and fallback rules as chat. Set LLMLEAF_API_KEY to a consumer bearer token allowed to use the jev route.

curl localhost:8080/v1/decisions \
  -H "Authorization: Bearer $LLMLEAF_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"jev","state":"The export button does nothing.","questions":{"bug":{"type":"noul","instructions":"Does this report broken software?"}}}'

Use GET /v1/models?type=decisions to list models identified as producing decisions. Their architecture.output_modalities contains "decisions", and their architecture.modality is "text->decisions". Configured aliases use metadata from their primary upstream target. Models without known metadata are excluded from this filter, including pinned TypeSafe versions absent from its catalog.

API references: OpenRouter decisions and TypeSafe JEV.

For an example with all three question types, run cargo run -p llmleaf --example decisions -- --state "The export button does nothing." against a server with the jev route configured.

Architecture

Two strictly separated planes. The core (data plane) is a Compio/Cyper server: ingress, control-plane background I/O, and provider HTTP run on Compio. The separate llmleaf-web control-plane app intentionally remains Tokio/Axum; its narrow compatibility bridges never put a Tokio reactor or I/O loop in the data plane. The control plane is reached only outbound — the core pulls identity/verdicts/topology and pushes usage, never the reverse. A pulled topology ([control.topology]) lets the controller also serve provider and route configuration, diffed against the previous pull on every refresh so resources are added, updated, and removed incrementally on top of the immutable config file. See SOUL.md for the full design constitution. To build a compatible controller, see the external control-plane implementation guide.

flowchart LR
  Cons["Consumers<br/>OpenAI · OpenRouter · Anthropic"] --> Surf["Compat surfaces"]
  subgraph Core["llmleaf core — data plane"]
    direction LR
    Surf --> Auth["authenticate"] --> In["map in"] --> Route["route + fallback"] --> Stream["stream"] --> Out["map out"] --> Ev["emit events"]
  end
  Route --> Prov["Providers<br/>compiled-in traits · WASM plugins"]
  Prov --> Up["LLM providers"]
  Ctrl[["Control plane (outbound)"]]
  Auth -. "pull identity / verdicts" .-> Ctrl
  Route -. "pull topology (providers + routes)" .-> Ctrl
  Ev -. "push usage" .-> Ctrl

Loading

AI Disclosure

This project is being developed with AI assistance.

License

Copyright (C) 2026 Fionn Langhans fionnlanghans@codefionn.eu.

llmleaf and its clients are dual-licensed under either the MIT License or the Apache License 2.0, at your option.