Skip to content

feat(lore-0023): rate limits per route and client, client IP from a configured header - #60

Open
karolko9 wants to merge 13 commits into
developfrom
lore-0023-rate-limiting-and-http-hardening
Open

karolko9 wants to merge 13 commits into
developfrom
lore-0023-rate-limiting-and-http-hardening

Conversation

@karolko9

@karolko9 karolko9 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Rate limits on /verify, /settle and /discovery/*, keyed by the client IP from a configured header (CF-Connecting-IP behind Cloudflare Tunnel), plus a 400 for bodies PostgreSQL can't store. Task: lore/1-tasks/active/0023_FEATURE_rate-limiting-and-http-hardening.md. Its "Decisions so far" table has 13 rows, each with its code and test.

 POST /verify, POST /settle
+  rateLimit(route)                     # before the body is read: 400 and 413 count
+    client = header CLIENT_IP_HEADER   # socket: the connection's address; X-Forwarded-For never read
+    not one IP address → 500, no store hit
+             IPv6 grouped per /56
+    if client blocked in process until resetAt → 429
+    hit = RateLimitStore.hit("<route>:ip:<client>", window, now)   # store error → 500
+    if hit.count > limit → remember until resetAt, log rate_limited once → 429 + Retry-After
   express.json(BODY_LIMIT_BYTES)
   parsePaymentRequest
-    nestsDeeper(body)
+    refusedBody(body)                  # too deep, or a NUL / lone surrogate in any string or key
+GET /discovery/*  → rateLimit("discovery")
 GET /supported    → no limit
Input Before After
601st /verify from one client within 60 s 200 429 {"error":"too_many_requests"}, Retry-After: <s>
601st /settle from the same client 200 429 (its own budget, 600)
/verify over budget, then /settle — /settle still answered (one route only)
121st GET /discovery/resources 501 429
GET /supported many times 200 200 (no change)
X-Forwarded-For: 1.2.3.4 rotating, CLIENT_IP_HEADER=socket not limited one client (the connection's address)
CLIENT_IP_HEADER unset started startup refused: "CLIENT_IP_HEADER is required"
CF-Connecting-IP: 203.0.113.7, 203.0.113.8 (sent twice) or 8 000 characters 200 500, client_ip_invalid logged without the value, no store hit
CF-Connecting-IP: 2001:db8:0:1::1, then 2001:db8:0:ff::5 not limited same client (same /56)
CLIENT_IP_HEADER=cf-connecting-ip, request without the header 200 500, client_ip_missing logged
rate-limit store down 200 500 internal_error, http_error logged
/settle body with payTo: "a\u0000b" or "a\ud800b" 500 (store write) 400 bad_request, nothing settled
body with "😀" (a surrogate pair) 200 200 (no change)
TRUSTED_PROXY_HOPS=1 in the env parsed startup refused: "is replaced by CLIENT_IP_HEADER"
RATE_LIMIT_VERIFY=100, RATE_LIMIT_SETTLE=99 accepted refused: settle must be ≥ verify
NETWORK=stellar:pubnet refused (0023, 0024, 0025) refused (0024, 0025, 0050)

The rule the rows follow: every request to a limited route counts against the client's budget for that route, and the client is the configured header's IP, never X-Forwarded-For.

Defaults: 600 /verify and 600 /settle per 60 s (were 120 and 60), 120 /discovery/*. The seller's server calls both routes for all of its buyers, and @x402/express settles after the seller's handler has run, so a budget per IP is a budget per seller. A 429 on /settle throws away work the seller already did. Fees stay bounded by the spend budgets (AD-18).

Evidence

  • Before: rate-limit.test.ts (10 cases), the NUL and surrogate cases in verify-route.test.ts and settle-route.test.ts, and the new config.test.ts cases: 19 red before the code (vitest run: 8 failed in config, 11 in the route and limiter files).
    After: TEST_DATABASE_URL=… vitest run in apps/facilitator: 395 tests passed, PostgreSQL suite included (PostgreSQL 14, local). tsc --noEmit, eslint . and prettier --check . are clean.

  • The client IP through a real tunnel (quick tunnel, cloudflared 2026.10.3, an echo server behind it):

    Sent Arrived
    no X-Forwarded-For cf-connecting-ip: <real>, x-forwarded-for: <real>
    X-Forwarded-For: 1.2.3.4, 5.6.7.8 x-forwarded-for: 1.2.3.4, 5.6.7.8,<real>, cf-connecting-ip: <real>
    CF-Connecting-IP: 9.9.9.9 refused by Cloudflare: error code: 1000
  • A flood against PostgreSQL (local, hit pool of 3, budget 10, one client over 100 connections, 10 other clients on their own):

    Burst Store hits Other clients
    20 000 34 all answered, ≤ 312 ms
    20 000, in-process filter off 20 010 all answered, ≤ 114 ms

    No errors either way. The filter takes the flood's writes off the database; it isn't needed locally to keep other clients answered.

Considered and rejected

  • Express's numeric trust proxy (TRUSTED_PROXY_HOPS=1): measured to give the same IP through the tunnel. Dropped for the header that ADR 0011 names in the open PR docs(lore-0039): host the facilitator on the company Hetzner server #24 (on develop it still describes an ALB with TRUSTED_PROXY_HOPS=1, which this PR refuses: merge docs(lore-0039): host the facilitator on the company Hetzner server #24 first), because a wrong count either merges every client (0) or trusts a forged entry (2).
  • An API key as a limiter key (task AC 3): no registry of issued keys exists, and a raw caller key gives each key its own counter, which bypasses the limit. Moved to 0060.
  • Passing requests when the store fails: the limit would vanish during an outage, and /settle fails without the database anyway.
  • express-rate-limit's middleware: its Store needs an adapter with decrement and resetKey, one instance per limiter, its headers off and its logger rerouted. That is more code than the counting it would replace. Only its ipKeyGenerator is used.
  • Keeping 120 / 60: below one paid request a second per seller, with /settle refusing what /verify passed.

Merge Danger

Door: two-way

Revert restores the old behaviour. The rate_limits table already exists (0022); no migration.

Blast Radius: facilitator

  • An operator env with TRUSTED_PROXY_HOPS set refuses to start, by design. deploy/testnet.env.example now sets CLIENT_IP_HEADER=cf-connecting-ip; mainnet.env.example has it blank, since mainnet's proxy is open (0038) and pubnet is refused.
  • CLIENT_IP_HEADER is required in every env: cf-connecting-ip behind Cloudflare, socket with no proxy. Both env examples carry it; mainnet's is left blank until 0038 picks its proxy.
  • The rate_limited log line carries the client IP, once per client and window.

🤖 Generated with Claude Code

karolko9 and others added 13 commits October 23, 2026 11:10
PostgreSQL refuses a NUL in text and jsonb, and jsonb a lone
surrogate, so either one in a /settle body failed the store write
with a 500 before money moved. Both are now a 400 on /verify and
/settle, wherever they sit in the body, keys included. A test also
pins that /verify sends no CORS headers, as /settle already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Behind Cloudflare Tunnel the client IP is CF-Connecting-IP, which
Cloudflare sets on every request and refuses from a client
(measured through a quick tunnel). CLIENT_IP_HEADER names it;
unset, the client is the socket address.

The seller's server calls /verify and /settle for all of its
buyers, and settles after serving, so the budgets are per seller:
600 each by default, and /settle's never below /verify's. The
pubnet refusal now waits for 0024, 0025 and 0050.

BREAKING CHANGE: TRUSTED_PROXY_HOPS is removed and refused at
startup; set CLIENT_IP_HEADER instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/verify, /settle and /discovery/* each get a budget per client and
window, counted in RateLimitStore before the body is read; past it
the route answers 429 with Retry-After. An IPv6 client counts per
/56. A store error is a 500. A client over budget is answered from
the process until its window ends, so a flood doesn't write to the
store on every request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The facilitator README says who may call which route and what the
limits are; the spine and the M1 epics take the client IP header
in place of proxy hops. The task records where each decision lives
in the code and the flood measurement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows are aligned to the clock, so a minute boundary between two
requests of a test reset the count and turned an expected 429 into
a 400. An hour's window makes that all but impossible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A client can send twice its budget across a window's end, and every
blocked client is released at once. Decision 6 no longer names an
option of the middleware this branch dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The header's value went into the store key unchecked: a header sent
twice arrived joined, and a long value failed the key index with a
500 after a store round trip. Anything but one address is now a 500
with client_ip_invalid, without the value and without a store hit.

Requests already in flight when a client ran out of budget each
logged rate_limited; now only the first does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Header lookups ignore case, so lowercasing the name did nothing, and
a wrong name already fails loudly on the first request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cloudflare gives every cross-zone Worker request the same
CF-Connecting-IP, so sellers on Workers share one rate-limit
budget. Task 0060 records it as the reason to land API keys before
the public testnet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Left unset behind the tunnel, every client was cloudflared's address
and shared one budget, with nothing logged. The setting is now
required: a header name, or socket for the connection's own address
with no proxy in front, as in local runs and tests.

BREAKING CHANGE: CLIENT_IP_HEADER must be set; use socket where no
proxy sits in front of the service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iting-and-http-hardening

# Conflicts:
#	docs/architecture/m1-spine.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant