steady-proxy

steady-proxy is a local reverse proxy for AI agents and API clients. It sits between your client and API gateway, buffering streaming responses so dropped connections, truncated streams, rate limits, and gateway 5xx errors can be retried before the client sees a failed turn.

Agents / SDKs / applications → steady-proxy (127.0.0.1:8789) → your gateway

Key features

• Transactional buffering: Holds SSE responses until completion (or a configurable time window) so broken streams can be discarded and retried cleanly.
• Retry normalization: Converts recoverable HTTP and pre-commit stream failures into standard 503 responses with Retry-After (plus optional in-proxy retries). Final upstream answers (auth, billing, validation, and other 4xx) are relayed unchanged.
• Stream validation and repair: Validates Anthropic event order and tool JSON, coalesces fragmented input_json_delta chunks, and merges bursty Responses text deltas on replay.
• Claude refusal fallback: Tries an ordered model list before committing the response (default: claude-fable-latest, claude-opus-latest, claude-sonnet-latest; can be disabled). If the chain is exhausted, delivers the original refusal instead of restarting client retries.
• Workflow watchdog support: Uses a short buffering window for Claude Code Workflow agents so their content-stall timer does not trip.
• Diagnostics: Logs one-line request outcomes and optionally archives full request/response payloads for debugging.

Supported routes

API	Route	Default behavior
Anthropic Messages	POST /v1/messages (stream: true)	Buffers up to 4 minutes (240000 ms) until message_stop, then replays. Workflow agents switch to live streaming after 10 seconds once content is ready.
OpenAI Responses	POST /v1/responses (stream: true)	Catches start-of-stream failures before commit, then streams live on first output. Set PROXY_RESPONSES_EARLY_COMMIT=0 for 4-minute full buffering.
Gemini GenerateContent	POST /{version}/models/{model}:streamGenerateContent?alt=sse (including Vertex AI publisher and express paths)	Buffers up to 4 minutes through clean EOF after all requested candidates finish.
TypeSafe System One	POST /v1/systemone	Single-shot JSON decisions with native errors, TypeSafe retry policy, and model/token logging.
Other routes	/v1/chat/completions, generateContent, countTokens, embedContent, non-SSE streamGenerateContent, /v1/models, non-streaming calls	Forwarded single-shot; recoverable HTTP and connection errors still normalize for client retry.

See Architecture for client compatibility details.

Quick start

Requires Docker with Compose v2 (or Go 1.23+ to build from source).

git clone https://github.com/Xllent-AI/steady-proxy
cd steady-proxy
cp .env.example .env
cp docker-compose.yml.example docker-compose.yml

Set your gateway URL in .env:

PROXY_UPSTREAM_URL=https://your-gateway.example.com

Use the gateway root (or required prefix) without a trailing /v1—the proxy appends the incoming request path (/v1/messages, /v1/responses, etc.) verbatim.

Start the proxy and verify it responds:

docker compose up -d --build
curl -fsS http://127.0.0.1:8789/__version

/__version checks local proxy health only. The proxy forwards your client's existing API keys and auth headers to the gateway.

Prebuilt container image

Linux amd64 and arm64 images are published to GitHub Container Registry:

docker pull ghcr.io/xllent-ai/steady-proxy:latest

latest tracks main; version tags (such as v0.1.1) pin a release. To run the prebuilt image with Compose, set image: ghcr.io/xllent-ai/steady-proxy:latest in docker-compose.yml, remove the build: block, and run docker compose pull && docker compose up -d. See Releasing.

Client setup

Claude Code

Merge this env block into ~/.claude/settings.json alongside your existing settings and credentials:

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:8789",
    "CLAUDE_CODE_CONNECT_TIMEOUT_MS": "660000",
    "API_TIMEOUT_MS": "720000",
    "API_FORCE_IDLE_TIMEOUT": "0"
  }
}

Because the proxy sends no HTTP headers while buffering, client timeouts must exceed the 240,000 ms default buffering window and the idle watchdog must be disabled. The conservative client settings above also accommodate an explicit 600,000 ms window. If you change PROXY_KEEPALIVE_MS, keep these client timeouts above it; see Timeouts.

Codex

Update your active provider section in ~/.codex/config.toml (keep /v1 in base_url and keep your auth settings):

[model_providers.your_provider]
base_url = "http://127.0.0.1:8789/v1"
wire_api = "responses"
stream_idle_timeout_ms = 660000
request_max_retries = 10
stream_max_retries = 10

To enable full 4-minute buffering for Responses and up to 3 in-proxy retries per client attempt, add to .env and run docker compose up -d:

PROXY_RESPONSES_EARLY_COMMIT=0
PROXY_TRANSACTIONAL_LOCAL_RETRIES=3

Full buffering delays output until the turn completes (or the window expires). Local retries apply across Messages, Responses, and Gemini SSE within the active buffering window; see Retry budgets.

Gemini

Point your Gemini SDK or client base URL at http://127.0.0.1:8789 and set PROXY_UPSTREAM_URL to your Gemini gateway (or https://generativelanguage.googleapis.com). The proxy forwards x-goog-api-key, Bearer tokens, and query parameters unchanged.

Example streaming call:

curl -N --max-time 720 \
  "http://127.0.0.1:8789/v1beta/models/${GEMINI_MODEL}:streamGenerateContent?alt=sse" \
  -H "x-goog-api-key: ${GEMINI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"Say hello."}]}]}'

Set client timeouts above PROXY_GEMINI_BUFFER_MS (default 240,000 ms) with margin and configure bounded retries on HTTP 503 honoring Retry-After. Failed upstream attempts and local-retry waits count toward that silent window; for a client whose first-byte timeout is shorter, lower PROXY_GEMINI_BUFFER_MS below it (see Match the client to the window). If a Gemini stream fails after committing to live output, the proxy aborts the HTTP transport so the client does not mistake a truncated stream for clean EOF.

TypeSafe System One

Set your TypeSafe SDK's TYPESAFE_BASE_URL to http://127.0.0.1:8789, keeping PROXY_UPSTREAM_URL pointed at your gateway (or https://api.typesafe.ai for direct TypeSafe access). Use the model ID your upstream accepts, unchanged; this example uses the gateway alias xllent/decider-latest:

curl http://127.0.0.1:8789/v1/systemone \
  -H "Authorization: Bearer ${TYPESAFE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"model":"xllent/decider-latest","state":"I was charged twice.","questions":{"billing":{"type":"noul","instructions":"Is this about billing?"}}}'

System One returns JSON, not SSE: do not add stream: true. Choice, Score, Noul, Box, assets, compressed bodies, multipart requests, and extension fields pass through unchanged. Each client call makes one upstream attempt; SSE buffering, local retries, and Claude model fallback do not apply. Configure bounded client retries on HTTP 503; the proxy honors TypeSafe's retry-after-ms before Retry-After and uses native detail errors. GET /v1/models remains a shared single-shot route.

See the TypeSafe API contract and retry behavior.

Other API clients

Point Messages clients at http://127.0.0.1:8789/v1/messages and Responses clients at http://127.0.0.1:8789/v1/responses (watch whether your SDK appends /v1 automatically). Enable stream: true, set client timeouts above the buffering window, and retry HTTP 503 with Retry-After. See Client compatibility.

Verify traffic

Send a test prompt and tail the proxy logs:

docker compose logs -f proxy

Each request logs a single line starting with OK (delivered), RETRY (client asked to retry), WARN (model fallback), FAIL (terminal error), or DROP (stream failed after headers were already sent). See Reading the logs.

Security and operational notes

• Bind to loopback only: The proxy has no authentication of its own; anyone who can reach its port can relay requests through your gateway. Compose binds 127.0.0.1:8789 by default.
• Payload archives are unencrypted: PROXY_REQUEST_LOG_DIR is off by default. When enabled, it writes full prompts and responses to disk without encryption or automatic retention; see Data handling.
• Retries repeat upstream work: Buffering hides failed attempts from the client, but cannot undo side effects already executed by server-side or remote tools. Ensure remote tools are idempotent; see Retry limits and side effects.

Documentation

Guide	Contents
Architecture	Buffering lifecycle, commit points, retry budgets, refusal fallback, and Workflow agents.
Configuration	Environment variables, client timeout matching, and upstream gateway headers.
Operations	Compose workflow, log format, payload archiving, and systemd service setup.
Troubleshooting	Symptom lookup, error classification rules, and timeout debugging.
Development	Building from source, running unit and live tests, and code layout.
Releasing	Local release packaging, tag publishing, and GHCR container images.

License

MIT © Xllent AI.