EventRelay

EventRelay is a production-minded webhook ingestion and delivery platform built with FastAPI, PostgreSQL, and Redis. It demonstrates the reliability patterns that turn a basic HTTP endpoint into an operable backend service: HMAC verification, idempotent ingestion, durable queueing, exponential retries, dead-letter handling, redrive, authentication, structured logs, metrics, migrations, tests, and containerized local development.

Portfolio project by Hari Om Tiwari. No proprietary employer code or data is included.

Why this project exists

Third-party webhooks fail in ordinary ways: duplicate deliveries, timeouts, temporary downstream outages, and malformed or forged payloads. EventRelay accepts each valid event once, stores it before queueing, delivers it asynchronously, and preserves failed work for inspection and recovery.

flowchart LR
    P[Webhook producer] -->|HMAC + idempotency key| A[FastAPI ingestion API]
    A -->|commit first| D[(PostgreSQL)]
    A -->|event id| S[(Redis Stream)]
    O[Outbox recovery scan] --> S
    S --> W[Async worker]
    W -->|signed delivery| C[Configured consumer]
    W -->|exponential backoff| R[(Redis retry set)]
    R --> S
    W -->|max attempts| Q[(Dead-letter stream)]
    X[Admin redrive API] --> S
    A --> M[Prometheus metrics]
    W --> M

Loading

Reliability and security decisions

• Commit before enqueue: an accepted event is persisted before Redis is called. If Redis is unavailable, the worker's pending-event scan repairs the handoff.
• Idempotency: (source, idempotency_key) is unique in the database, so repeated webhook requests return the original event instead of creating duplicate work.
• HMAC verification: inbound and outbound bodies are signed with SHA-256 and compared in constant time.
• At-least-once processing: Redis Streams provides durable consumer-group delivery. Downstream consumers should also use the included event_id as an idempotency key.
• Bounded retries: transient failures use exponential backoff. Events that exhaust the configured attempts enter a dead-letter stream and can be redriven through an authenticated endpoint.
• Fixed delivery destination: the worker sends only to the configured DELIVERY_URL, avoiding an arbitrary-URL SSRF surface.
• Operational visibility: JSON logs include event and attempt context. Prometheus metrics expose accepted events, duplicates, attempts by result, and delivery latency.

Quick start with Docker

Requirements: Docker Engine with Compose.

cp .env.example .env
docker compose up --build

The API is available at http://localhost:8000, interactive documentation at http://localhost:8000/docs, health at /healthz, readiness at /readyz, and Prometheus metrics at /metrics.

The included delivery receiver can simulate transient downstream failures. This command creates an event that fails twice and succeeds on its third attempt:

python - <<'PY'
import hashlib, hmac, json, uuid
import httpx

secret = "replace-with-a-long-random-secret"
body = json.dumps(
    {"order_id": "ord_demo", "demo_failures_before_success": 2},
    separators=(",", ":"),
).encode()
signature = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
response = httpx.post(
    "http://localhost:8000/v1/webhooks/demo",
    content=body,
    headers={
        "Content-Type": "application/json",
        "Idempotency-Key": str(uuid.uuid4()),
        "X-Webhook-Signature": f"sha256={signature}",
    },
)
print(response.status_code, response.json())
PY

Inspect the event with the event_id returned above:

curl -H "X-Admin-Key: replace-with-a-long-random-admin-key" \
  http://localhost:8000/v1/events/EVENT_ID

Local development

python -m venv .venv
.venv/Scripts/activate  # Windows
python -m pip install -e ".[dev]"
pytest -q
ruff check .
mypy app

The default configuration uses SQLite for a quick local API run. Docker Compose uses PostgreSQL and Redis, which is the representative production-style setup.

API surface

Method	Route	Purpose	Authentication
POST	/v1/webhooks/{source}	Verify, deduplicate, persist, and enqueue a webhook	HMAC signature
GET	/v1/events/{event_id}	Inspect delivery state and failure context	Admin API key
POST	/v1/events/{event_id}/redrive	Reset and enqueue a dead-letter event	Admin API key
GET	/healthz	Process liveness	None
GET	/readyz	Database and Redis readiness	None
GET	/metrics	Prometheus exposition	None in demo; protect in production

Testing strategy

The test suite uses an in-memory SQLite database, a fake Redis implementation, and HTTPX transports. It verifies:

• HMAC round trips and tamper detection
• rejection of invalid signatures
• idempotent duplicate ingestion
• admin endpoint protection
• retry followed by successful delivery

Run a repeatable local ingestion benchmark after starting the stack:

python scripts/benchmark.py --requests 500 --concurrency 25 \
  --secret replace-with-a-long-random-secret

The benchmark prints observed throughput, mean latency, and p95 latency without claiming results that were not reproduced on your machine.

Production extensions

The repository deliberately keeps the demo focused. A real deployment would add per-tenant secrets, a managed secret store, OpenTelemetry traces, pending-message claiming for crashed Redis consumers, rate limits, retention policies, TLS termination, network isolation for /metrics, and deployment manifests for the chosen cloud.

License

MIT