diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c7a1f53..7d5e59ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,9 +7,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added - Replicator docs (hugo site) to repository. [replicator#90](https://github.com/EventStore/replicator/pull/90) - Replicator docs workflows. [replicator#90](https://github.com/EventStore/replicator/pull/90) +- OAuth 2.0 authentication for the gRPC reader and sink (client credentials or token file), configured independently per side, with automatic token refresh and recovery. Helm values for extra env, volumes, service account and pod labels. ### Changed - Updated workflows to add clear naming separation between replicator (app) and replicator (docs) workflows. [replicator#90](https://github.com/EventStore/replicator/pull/90) - Updated hugo site to use hugo modules, removed blog content. [replicator#90](https://github.com/EventStore/replicator/pull/90) +- When environment variables are printed at startup, the values of connection strings, auth settings (except `auth.type`) and any value containing credentials (a URI with user info such as a MongoDB checkpoint connection string, or `DefaultUserCredentials=`) are masked as `***`. +- The realtime subscription used for scavenging resubscribes reliably after drops, and its metadata cache is rebuilt after a gap. +- Replication no longer hangs on shutdown when the writer has stopped; the metrics reporter keeps running after a failed position read. - +### Fixed +- With `scavenge: true`, replication no longer stalls on a stream that has metadata but no events, such as the system streams KurrentDB 26 creates at startup (`$connectors-mngt/state-projection`): the scavenge filter treats it as empty instead of failing with StreamNotFound. Streams without metadata no longer log a warning per event. Auth failures on these reads still stop replication when OAuth is enabled. (DEV-1903) diff --git a/charts/replicator/templates/statefulset.yaml b/charts/replicator/templates/statefulset.yaml index 947b113c..1ed825e7 100644 --- a/charts/replicator/templates/statefulset.yaml +++ b/charts/replicator/templates/statefulset.yaml @@ -22,11 +22,25 @@ spec: app: {{ template "replicator.name" . }} release: {{ .Release.Name }} tier: web +{{- with .Values.podLabels }} +{{ toYaml . | indent 8 }} +{{- end }} spec: +{{- with .Values.serviceAccountName }} + serviceAccountName: {{ . }} +{{- end }} containers: - name: {{ .Chart.Name }} image: {{ template "replicator.image" . }} imagePullPolicy: {{ .Values.image.pullPolicy | quote }} +{{- with .Values.extraEnv }} + env: +{{ toYaml . | indent 10 }} +{{- end }} +{{- with .Values.extraEnvFrom }} + envFrom: +{{ toYaml . | indent 10 }} +{{- end }} livenessProbe: httpGet: path: /health @@ -59,6 +73,9 @@ spec: - name: partitioner-js mountPath: /app{{ include "replicator.sink.partitioner.filepath" . }} subPath: {{ include "replicator.sink.partitioner.filename" . }} +{{- end }} +{{- with .Values.extraVolumeMounts }} +{{ toYaml . | indent 8 }} {{- end }} resources: {{ toYaml .Values.resources | indent 10 }} @@ -86,3 +103,6 @@ spec: - name: {{ template "replicator.name" . }} persistentVolumeClaim: claimName: {{ template "replicator.fullname" . }} +{{- with .Values.extraVolumes }} +{{ toYaml . | indent 6 }} +{{- end }} diff --git a/charts/replicator/values.yaml b/charts/replicator/values.yaml index 0245cbca..63aa161c 100644 --- a/charts/replicator/values.yaml +++ b/charts/replicator/values.yaml @@ -21,12 +21,24 @@ replicator: reader: connectionString: protocol: tcp + # auth: # optional, gRPC only. See docs: deployment/authentication + # type: oauthClientCredentials + # tokenEndpoint: https://login.microsoftonline.com//oauth2/v2.0/token + # clientId: + # clientSecretFile: /var/run/secrets/replicator/client-secret # do not put clientSecret here + # scope: api:///.default sink: connectionString: protocol: grpc partitionCount: 1 partitioner: bufferSize: 1000 + # auth: # optional, gRPC only. See docs: deployment/authentication + # type: oauthClientCredentials + # tokenEndpoint: https://login.microsoftonline.com//oauth2/v2.0/token + # clientId: + # clientSecretFile: /var/run/secrets/replicator/client-secret # do not put clientSecret here + # scope: api:///.default scavenge: true filters: [] transform: null @@ -36,3 +48,16 @@ replicator: checkpointAfter: 1000 # restartOnFailure: false # runContinuously: true + +# Service account for the pod (e.g. one federated with Azure Workload Identity). +serviceAccountName: "" +# Extra labels on the pod template, e.g. azure.workload.identity/use: "true". +podLabels: {} +# Extra environment variables. Use these (not replicator.*.auth in this file) for secrets, e.g. +# - name: REPLICATOR_SINK_AUTH_CLIENTSECRET +# valueFrom: { secretKeyRef: { name: replicator-oauth, key: client-secret } } +extraEnv: [] +extraEnvFrom: [] +# Extra volumes and mounts, e.g. a Secret holding clientSecretFile or a token file written by a sidecar. +extraVolumes: [] +extraVolumeMounts: [] diff --git a/compose/oauth/.env.example b/compose/oauth/.env.example new file mode 100644 index 00000000..ab945d73 --- /dev/null +++ b/compose/oauth/.env.example @@ -0,0 +1,5 @@ +# Copy to .env and fill in. .env is git-ignored. +# OAuth authentication in KurrentDB requires a license key. +KURRENTDB_LICENSE_KEY= +# Native arm64 images make startup much faster on Apple Silicon; use an amd64 tag elsewhere. +KURRENTDB_IMAGE=kurrentplatform/kurrentdb:26.1.1-experimental-arm64-10.0-noble diff --git a/compose/oauth/.gitignore b/compose/oauth/.gitignore new file mode 100644 index 00000000..faac6d8d --- /dev/null +++ b/compose/oauth/.gitignore @@ -0,0 +1,5 @@ +.env +certs/ +tokens/ +scenarios/*/checkpoint +scenarios/*/*.log diff --git a/compose/oauth/README.md b/compose/oauth/README.md new file mode 100644 index 00000000..f6f6cae7 --- /dev/null +++ b/compose/oauth/README.md @@ -0,0 +1,113 @@ +# Replicator OAuth: local end-to-end test bed + +Runs a test identity provider and three KurrentDB nodes in Docker, then runs Replicator from this branch on the host against them. It follows KurrentDB's "Testing with a local identity server" guide (`docs/server/security/user-authentication.md` in the KurrentDB repo) and uses the same IdentityServer 4 image as KurrentDB's own OAuth tests. + +| Service | Host port | Auth | +|:--|:--|:--| +| `idsrv4` (IdentityServer 4) | `5080` (http), `5001` (https) | issues tokens; client `replicator` / secret `replicator-secret` gets role `$admins`, 5-minute tokens | +| `kurrentdb-basic` | `2113` | basic, `admin:changeit` | +| `kurrentdb-oauth` | `2114` | OAuth only | +| `kurrentdb-oauth2` | `2115` | OAuth only | + +All nodes use TLS with a test CA generated into `certs/`. Replicator trusts it through `tlsCaFile` in the connection strings. Data is in memory, so restarting a node empties it. + +## Setup + +1. OAuth in KurrentDB needs a license key. Copy `.env.example` to `.env` and set `KURRENTDB_LICENSE_KEY`. `.env` is git-ignored. +2. Generate certificates and start everything: + + ```bash + ./scripts/gen-certs.sh + docker compose up -d + ``` + +3. Check the OAuth nodes picked up the IdP. Their logs should show "OAuthAuthentication ... plugin enabled" and "Issuer signing keys have been retrieved": + + ```bash + docker compose logs kurrentdb-oauth | grep -i -E "oauth|signing keys|licen" + ``` + +## Scenarios + +Each scenario folder holds a Replicator `config/appsettings.yaml`. Run it with `./scripts/run-replicator.sh `, which builds Replicator in Release and runs it with the scenario folder as the working directory, with the HTTP API on `localhost:5005`. Delete `scenarios//checkpoint` to start a scenario from scratch. + +Helpers: +- `./scripts/write-events.sh [delay-seconds]` appends events over HTTP. +- `./scripts/stream-head.sh ` prints the last event number. +- `./scripts/get-token.sh` prints a token for the `replicator` client. + +Use a fresh stream name for each run. + +### 1. Basic source → OAuth sink (client credentials, secret file) + +```bash +./scripts/write-events.sh basic s1-a 100 +./scripts/run-replicator.sh 1-basic-to-oauth # in another terminal +./scripts/stream-head.sh oauth s1-a # expect 99 +``` + +To test across token expiry, keep writing while Replicator runs for at least 3 token lifetimes (5 minutes each): + +```bash +./scripts/write-events.sh basic s1-long 1000 1 +``` + +Expected: Replicator logs "access token acquired" about every 2.5 minutes, no write failures, and the target stream head matches the source. + +### 2. OAuth source (client credentials) → basic sink + +```bash +./scripts/write-events.sh oauth s2-a 100 +./scripts/run-replicator.sh 2-oauth-to-basic +./scripts/stream-head.sh basic s2-a # expect 99 +``` + +For the token-expiry run, write slowly for at least 15 minutes (`./scripts/write-events.sh oauth s2-long 1000 1`). Then record which of these happened: +- (a) No reader restart or subscription drop happened at token expiry. +- (b) The read and/or the realtime subscription were ended at expiry and restarted with a fresh token. + +Either way the target must end up with every event. + +### 3. OAuth source (client credentials) → OAuth sink (token file, rotated) + +```bash +./scripts/refresh-token-file.sh 60 & # keeps tokens/sink.token fresh +./scripts/write-events.sh oauth s3-a 100 +./scripts/run-replicator.sh 3-oauth-to-oauth-tokenfile +./scripts/stream-head.sh oauth2 s3-a # expect 99 +``` + +### 4. Token endpoint outage + +While scenario 1 or 2 is replicating, run: + +```bash +docker compose stop idsrv4 +``` + +Wait past a token lifetime (more than 5 minutes), then: + +```bash +docker compose start idsrv4 +``` + +Expected: rate-limited warnings, replication pauses, then resumes on its own with no process restart. + +### 5. Rejected token / misconfiguration + +- **Wrong audience.** Change `Audience` in `kurrentdb/oauth.conf` and run `docker compose up -d --force-recreate kurrentdb-oauth`. Each rejection fetches a new token, because idsrv4 issues a new value every time, so retries settle at one every 30 s. Expected: KurrentDB rejects the token as Unauthenticated. Replicator logs a warning, quarantines the token, re-tests it once a minute, and recovers once the audience is fixed. +- **Missing role.** Remove the `Claims` from the `replicator` client in `idsrv4/idsrv4.conf.json`, then recreate `idsrv4` and restart the OAuth node so it fetches idsrv4's new signing key. + - Sink: with KurrentDB's default ACLs, any authenticated identity can append to user streams, so plain writes still succeed. Metadata writes and deletes need `$admins`. + - Reader: reading `$all` needs `$admins`. Replicator retries the subscription with backoff and logs PermissionDenied warnings, and replication doesn't start until the role is restored. + +## Known issues + +- Recreating `idsrv4` generates a new signing key. Restart the OAuth nodes afterwards so they fetch it; otherwise every token is rejected as Unauthenticated. + +## Teardown + +```bash +docker compose down +``` + +`certs/`, `tokens/` and checkpoints are git-ignored; delete them to start clean. diff --git a/compose/oauth/docker-compose.yml b/compose/oauth/docker-compose.yml new file mode 100644 index 00000000..fa6ab355 --- /dev/null +++ b/compose/oauth/docker-compose.yml @@ -0,0 +1,70 @@ +# Local end-to-end test bed for Replicator OAuth authentication. +# See README.md in this folder. Replicator itself runs on the host from this branch. + +x-kurrentdb-env: &kurrentdb-env + KURRENTDB_CLUSTER_SIZE: 1 + KURRENTDB_MEM_DB: "true" + KURRENTDB_RUN_PROJECTIONS: None + KURRENTDB_NODE_IP: 0.0.0.0 + KURRENTDB_REPLICATION_IP: 0.0.0.0 + KURRENTDB_ENABLE_ATOM_PUB_OVER_HTTP: "true" + KURRENTDB_CERTIFICATE_FILE: /certs/node/node.crt + KURRENTDB_CERTIFICATE_PRIVATE_KEY_FILE: /certs/node/node.key + KURRENTDB_TRUSTED_ROOT_CERTIFICATES_PATH: /certs/ca + KURRENTDB_CERTIFICATE_RESERVED_NODE_COMMON_NAME: kurrentdb-node + +x-oauth-env: &oauth-env + <<: *kurrentdb-env + KURRENTDB_CONFIG: /etc/kurrentdb/oauth.conf + KURRENTDB__LICENSING__LICENSE_KEY: ${KURRENTDB_LICENSE_KEY:-} + +services: + # IdentityServer 4 test IdP, as used by KurrentDB's own OAuth tests and docs. + # HTTP on host port 5080 (Replicator only allows plain http for loopback token endpoints), + # HTTPS on 5001 for KurrentDB's discovery/key fetches inside the network. + idsrv4: + image: ghcr.io/kurrent-io/idsrv4/idsrv4:latest + environment: + DOTNET_URLS: http://+:5000;https://+:5001 + ports: + - "5080:5000" + - "5001:5001" + volumes: + - ./idsrv4/users.conf.json:/etc/idsrv4/users.conf:ro + - ./idsrv4/idsrv4.conf.json:/etc/idsrv4/idsrv4.conf:ro + + # Basic auth (admin:changeit), no license needed. + kurrentdb-basic: + image: ${KURRENTDB_IMAGE:-kurrentplatform/kurrentdb:26.1.1-experimental-arm64-10.0-noble} + environment: + <<: *kurrentdb-env + ports: + - "2113:2113" + volumes: + - ./certs:/certs:ro + + # OAuth-only node #1. + kurrentdb-oauth: + image: ${KURRENTDB_IMAGE:-kurrentplatform/kurrentdb:26.1.1-experimental-arm64-10.0-noble} + environment: + <<: *oauth-env + ports: + - "2114:2113" + volumes: + - ./certs:/certs:ro + - ./kurrentdb/oauth.conf:/etc/kurrentdb/oauth.conf:ro + depends_on: + - idsrv4 + + # OAuth-only node #2 (target for the OAuth-to-OAuth scenario). + kurrentdb-oauth2: + image: ${KURRENTDB_IMAGE:-kurrentplatform/kurrentdb:26.1.1-experimental-arm64-10.0-noble} + environment: + <<: *oauth-env + ports: + - "2115:2113" + volumes: + - ./certs:/certs:ro + - ./kurrentdb/oauth.conf:/etc/kurrentdb/oauth.conf:ro + depends_on: + - idsrv4 diff --git a/compose/oauth/idsrv4/idsrv4.conf.json b/compose/oauth/idsrv4/idsrv4.conf.json new file mode 100644 index 00000000..d056959c --- /dev/null +++ b/compose/oauth/idsrv4/idsrv4.conf.json @@ -0,0 +1,43 @@ +{ + "IdentityResources": [ + { "Name": "openid", "DisplayName": "Your user identifier", "Required": true, "UserClaims": ["sub", "role"] }, + { "Name": "profile", "DisplayName": "User profile", "UserClaims": ["name", "preferred_username"] } + ], + "ApiResources": [ + { "Name": "kurrentdb", "Scopes": ["streams"] } + ], + "ApiScopes": [ + { "Name": "streams", "UserClaims": ["http://schemas.microsoft.com/ws/2008/06/identity/claims/role", "role"] } + ], + "Clients": [ + { + "ClientId": "kurrentdb-client", + "AllowedGrantTypes": ["password", "authorization_code"], + "ClientSecrets": [ { "Value": "K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=" } ], + "AllowedScopes": ["streams", "openid", "profile"], + "RedirectUris": [ + "https://localhost:2114/signin-oidc", "https://localhost:2114/oauth/callback", + "https://localhost:2115/signin-oidc", "https://localhost:2115/oauth/callback" + ], + "AlwaysIncludeUserClaimsInIdToken": true, + "RequireConsent": false, + "AlwaysSendClientClaims": true, + "AllowOfflineAccess": true, + "RequireClientSecret": false, + "AllowAccessTokensViaBrowser": true + }, + { + "ClientId": "replicator", + "AllowedGrantTypes": ["client_credentials"], + "ClientSecrets": [ { "Value": "KAt78nlfli+lNvMiwB6+C9QvQTpwYPuQ4luobTdP+2M=" } ], + "AllowedScopes": ["streams"], + "Claims": [ + { "Type": "role", "Value": "$admins" }, + { "Type": "http://schemas.microsoft.com/ws/2008/06/identity/claims/role", "Value": "$admins" } + ], + "ClientClaimsPrefix": "", + "AlwaysSendClientClaims": true, + "AccessTokenLifetime": 300 + } + ] +} diff --git a/compose/oauth/idsrv4/users.conf.json b/compose/oauth/idsrv4/users.conf.json new file mode 100644 index 00000000..f78ef472 --- /dev/null +++ b/compose/oauth/idsrv4/users.conf.json @@ -0,0 +1,11 @@ +[ + { + "subjectId": "1", + "username": "admin", + "password": "password", + "claims": [ + { "type": "role", "value": "$admins" }, + { "type": "role", "value": "$ops" } + ] + } +] diff --git a/compose/oauth/kurrentdb/oauth.conf b/compose/oauth/kurrentdb/oauth.conf new file mode 100644 index 00000000..871d09dd --- /dev/null +++ b/compose/oauth/kurrentdb/oauth.conf @@ -0,0 +1,12 @@ +--- +AuthenticationType: oauth +LogFailedAuthenticationAttempts: true +OAuth: + Issuer: https://idsrv4:5001/ + Audience: kurrentdb + ClientId: kurrentdb-client + ClientSecret: K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols= + # Test-only: idsrv4 uses a self-signed dev certificate, and tokens fetched via + # http://localhost:5080 carry that issuer rather than https://idsrv4:5001. + Insecure: true + DisableIssuerValidation: true diff --git a/compose/oauth/scenarios/1-basic-to-oauth/config/appsettings.yaml b/compose/oauth/scenarios/1-basic-to-oauth/config/appsettings.yaml new file mode 100644 index 00000000..15568790 --- /dev/null +++ b/compose/oauth/scenarios/1-basic-to-oauth/config/appsettings.yaml @@ -0,0 +1,18 @@ +# Scenario 1: basic-auth source -> OAuth sink (client credentials, secret file) +replicator: + reader: + protocol: grpc + connectionString: "esdb://admin:changeit@localhost:2113?tls=true&tlsCaFile=../../certs/ca/ca.crt" + sink: + protocol: grpc + connectionString: "esdb://localhost:2114?tls=true&tlsCaFile=../../certs/ca/ca.crt" + auth: + type: oauthClientCredentials + tokenEndpoint: http://localhost:5080/connect/token + clientId: replicator + clientSecretFile: ../../secrets/replicator-client-secret + scope: streams + scavenge: true + checkpoint: + type: file + path: ./checkpoint diff --git a/compose/oauth/scenarios/2-oauth-to-basic/config/appsettings.yaml b/compose/oauth/scenarios/2-oauth-to-basic/config/appsettings.yaml new file mode 100644 index 00000000..fa4808a4 --- /dev/null +++ b/compose/oauth/scenarios/2-oauth-to-basic/config/appsettings.yaml @@ -0,0 +1,18 @@ +# Scenario 2: OAuth source (client credentials) -> basic-auth sink +replicator: + reader: + protocol: grpc + connectionString: "esdb://localhost:2114?tls=true&tlsCaFile=../../certs/ca/ca.crt" + auth: + type: oauthClientCredentials + tokenEndpoint: http://localhost:5080/connect/token + clientId: replicator + clientSecretFile: ../../secrets/replicator-client-secret + scope: streams + sink: + protocol: grpc + connectionString: "esdb://admin:changeit@localhost:2113?tls=true&tlsCaFile=../../certs/ca/ca.crt" + scavenge: true + checkpoint: + type: file + path: ./checkpoint diff --git a/compose/oauth/scenarios/3-oauth-to-oauth-tokenfile/config/appsettings.yaml b/compose/oauth/scenarios/3-oauth-to-oauth-tokenfile/config/appsettings.yaml new file mode 100644 index 00000000..65913a7b --- /dev/null +++ b/compose/oauth/scenarios/3-oauth-to-oauth-tokenfile/config/appsettings.yaml @@ -0,0 +1,22 @@ +# Scenario 3: OAuth source (client credentials) -> OAuth sink (token file kept fresh by scripts/refresh-token-file.sh) +replicator: + reader: + protocol: grpc + connectionString: "esdb://localhost:2114?tls=true&tlsCaFile=../../certs/ca/ca.crt" + auth: + type: oauthClientCredentials + tokenEndpoint: http://localhost:5080/connect/token + clientId: replicator + clientSecretFile: ../../secrets/replicator-client-secret + scope: streams + sink: + protocol: grpc + connectionString: "esdb://localhost:2115?tls=true&tlsCaFile=../../certs/ca/ca.crt" + auth: + type: oauthTokenFile + tokenFile: ../../tokens/sink.token + tokenFileReloadSeconds: 30 + scavenge: true + checkpoint: + type: file + path: ./checkpoint diff --git a/compose/oauth/scripts/gen-certs.sh b/compose/oauth/scripts/gen-certs.sh new file mode 100755 index 00000000..5fbbf09e --- /dev/null +++ b/compose/oauth/scripts/gen-certs.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Generates a test CA and one node certificate shared by all KurrentDB nodes (SANs cover every hostname used). +set -euo pipefail +cd "$(dirname "$0")/.." +if [ -f certs/node/node.crt ]; then echo "certs/ already exists; delete it to regenerate"; exit 0; fi +mkdir -p certs +docker run --rm -v "$PWD/certs:/certs" alpine:3 sh -euc ' + apk add -q --no-progress openssl + cd /certs && mkdir -p ca node + openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=Replicator OAuth E2E CA" -keyout ca/ca.key -out ca/ca.crt 2>/dev/null + openssl req -newkey rsa:2048 -nodes -subj "/CN=kurrentdb-node" -keyout node/node.key -out node/node.csr 2>/dev/null + printf "subjectAltName=DNS:localhost,DNS:kurrentdb-node,DNS:kurrentdb-basic,DNS:kurrentdb-oauth,DNS:kurrentdb-oauth2,IP:127.0.0.1\nextendedKeyUsage=serverAuth,clientAuth\nkeyUsage=digitalSignature,keyEncipherment\n" > node/ext.cnf + openssl x509 -req -in node/node.csr -CA ca/ca.crt -CAkey ca/ca.key -CAcreateserial -days 365 -extfile node/ext.cnf -out node/node.crt 2>/dev/null + rm node/node.csr node/ext.cnf + chmod 644 ca/ca.crt node/node.crt node/node.key +' +echo "certs written to $(pwd)/certs" diff --git a/compose/oauth/scripts/get-token.sh b/compose/oauth/scripts/get-token.sh new file mode 100755 index 00000000..a27f7b43 --- /dev/null +++ b/compose/oauth/scripts/get-token.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# Prints an access token for the "replicator" client (client credentials, role $admins). +set -euo pipefail +cd "$(dirname "$0")/.." +curl -fsS http://localhost:5080/connect/token \ + -d grant_type=client_credentials -d client_id=replicator \ + --data-urlencode "client_secret=$(cat secrets/replicator-client-secret)" -d scope=streams \ + | jq -r .access_token diff --git a/compose/oauth/scripts/refresh-token-file.sh b/compose/oauth/scripts/refresh-token-file.sh new file mode 100755 index 00000000..b67f77ad --- /dev/null +++ b/compose/oauth/scripts/refresh-token-file.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Keeps tokens/sink.token fresh (writes a new token every INTERVAL seconds, default 60), as an external +# token agent would. Writes to a temp file and renames, so Replicator never reads a half-written file. +set -euo pipefail +cd "$(dirname "$0")/.." +mkdir -p tokens +interval="${1:-60}" +while true; do + token="$(./scripts/get-token.sh)" + printf '%s' "$token" > tokens/sink.token.tmp && mv tokens/sink.token.tmp tokens/sink.token + echo "$(date '+%H:%M:%S') wrote tokens/sink.token" + sleep "$interval" +done diff --git a/compose/oauth/scripts/run-replicator.sh b/compose/oauth/scripts/run-replicator.sh new file mode 100755 index 00000000..8b9684c8 --- /dev/null +++ b/compose/oauth/scripts/run-replicator.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Builds Replicator from this branch (Release; Debug runs yarn) and runs it with a scenario's config. +# Usage: run-replicator.sh e.g. 1-basic-to-oauth +set -euo pipefail +here="$(cd "$(dirname "$0")/.." && pwd)" +repo="$(cd "$here/../.." && pwd)" +scenario="$here/scenarios/$1" +[ -d "$scenario" ] || { echo "no scenario $1"; exit 1; } +dotnet build "$repo/src/replicator" -c Release -p:UseAppHost=false -v q -nologo +cd "$scenario" +export ASPNETCORE_URLS="${ASPNETCORE_URLS:-http://localhost:5005}" +export DOTNET_ROLL_FORWARD=Major +exec dotnet "$repo/src/replicator/bin/Release/net9.0/replicator.dll" diff --git a/compose/oauth/scripts/stream-head.sh b/compose/oauth/scripts/stream-head.sh new file mode 100755 index 00000000..b336f7b0 --- /dev/null +++ b/compose/oauth/scripts/stream-head.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Prints the last event number of a stream (or "none"). Usage: stream-head.sh +set -euo pipefail +cd "$(dirname "$0")/.." +node="$1"; stream="$2" +case "$node" in + basic) port=2113; auth=(-u admin:changeit) ;; + oauth) port=2114; auth=(-H "Authorization: Bearer $(./scripts/get-token.sh)") ;; + oauth2) port=2115; auth=(-H "Authorization: Bearer $(./scripts/get-token.sh)") ;; +esac +code=$(curl -sS -o /tmp/stream-head.$$ -w '%{http_code}' --cacert certs/ca/ca.crt "${auth[@]}" \ + -H "Accept: application/vnd.eventstore.atom+json" "https://localhost:$port/streams/$stream/head/backward/1?embed=body") +if [ "$code" = 404 ]; then echo none; else jq -r '.entries[0].eventNumber // .entries[0].positionEventNumber' /tmp/stream-head.$$; fi +rm -f /tmp/stream-head.$$ diff --git a/compose/oauth/scripts/write-events.sh b/compose/oauth/scripts/write-events.sh new file mode 100755 index 00000000..0134ab65 --- /dev/null +++ b/compose/oauth/scripts/write-events.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Appends events to a stream over HTTP. Usage: write-events.sh [delay-seconds] +set -euo pipefail +cd "$(dirname "$0")/.." +node="$1"; stream="$2"; count="$3"; delay="${4:-0}" +case "$node" in + basic) port=2113; auth=(-u admin:changeit) ;; + oauth) port=2114; auth=(-H "Authorization: Bearer $(./scripts/get-token.sh)") ;; + oauth2) port=2115; auth=(-H "Authorization: Bearer $(./scripts/get-token.sh)") ;; + *) echo "unknown node $node"; exit 1 ;; +esac +for i in $(seq 1 "$count"); do + id="$(uuidgen | tr 'A-Z' 'a-z')" + curl -fsS --cacert certs/ca/ca.crt "${auth[@]}" \ + -H "Content-Type: application/vnd.eventstore.events+json" \ + -d "[{\"eventId\":\"$id\",\"eventType\":\"TestEvent\",\"data\":{\"n\":$i}}]" \ + "https://localhost:$port/streams/$stream" >/dev/null + # Re-fetch the token periodically for long runs (tokens live 5 minutes) + if [ "$node" != basic ] && [ $((i % 100)) -eq 0 ]; then auth=(-H "Authorization: Bearer $(./scripts/get-token.sh)"); fi + [ "$delay" != 0 ] && sleep "$delay" +done +echo "wrote $count events to $stream on $node" diff --git a/compose/oauth/secrets/replicator-client-secret b/compose/oauth/secrets/replicator-client-secret new file mode 100644 index 00000000..f2a33bcf --- /dev/null +++ b/compose/oauth/secrets/replicator-client-secret @@ -0,0 +1 @@ +replicator-secret \ No newline at end of file diff --git a/docs/package.json b/docs/package.json index 23532733..09da5e7b 100644 --- a/docs/package.json +++ b/docs/package.json @@ -19,5 +19,12 @@ "sharp": "^0.32.5", "tailwindcss": "^4.1.4" }, - "packageManager": "pnpm@10.10.0+sha512.d615db246fe70f25dcfea6d8d73dee782ce23e2245e3c4f6f888249fb568149318637dca73c2c5c8ef2a4ca0d5657fb9567188bfab47f566d1ee6ce987815c39" -} \ No newline at end of file + "packageManager": "pnpm@10.10.0+sha512.d615db246fe70f25dcfea6d8d73dee782ce23e2245e3c4f6f888249fb568149318637dca73c2c5c8ef2a4ca0d5657fb9567188bfab47f566d1ee6ce987815c39", + "pnpm": { + "onlyBuiltDependencies": [ + "esbuild", + "sharp", + "workerd" + ] + } +} diff --git a/docs/src/content/docs/deployment/authentication.mdx b/docs/src/content/docs/deployment/authentication.mdx new file mode 100644 index 00000000..e929ef48 --- /dev/null +++ b/docs/src/content/docs/deployment/authentication.mdx @@ -0,0 +1,118 @@ +--- +title: Authentication +description: Authenticate the gRPC reader and sink with basic credentials or OAuth 2.0 access tokens. +sidebar: + order: 2 +--- + +Replicator authenticates to each KurrentDB cluster separately. The reader (source) and the sink (target) each have their own optional `auth` section, so any combination works: basic on one side and OAuth on the other, OAuth on both with different identity providers, or basic on both. + +OAuth requires a Replicator release that includes OAuth support. It is supported for the `grpc` protocol only. It requires TLS (`tls=true`, the default), and the connection string must not contain `user:pass@`. + +## Auth types + +| `auth.type` | Use when | +|:--|:--| +| `connectionString` (default) | Basic credentials in the connection string, or no authentication. Behaves exactly as before. | +| `oauthClientCredentials` | Replicator gets tokens from your identity provider with the OAuth 2.0 client credentials grant and refreshes them itself. | +| `oauthTokenFile` | Something else (a sidecar, Vault agent, CI job) keeps a valid access token in a file; Replicator reads it. | + +## Options + +| Option | Type | Description | +|:--|:--|:--| +| `tokenEndpoint` | `oauthClientCredentials` | Token endpoint URL. Must be `https` (plain `http` only for loopback addresses, such as `localhost` and `127.0.0.1`). | +| `clientId` | `oauthClientCredentials` | OAuth client ID of the Replicator identity. | +| `clientSecret` | `oauthClientCredentials` | Client secret. Set it through the `REPLICATOR__AUTH_CLIENTSECRET` environment variable, not in the YAML file. | +| `clientSecretFile` | `oauthClientCredentials` | Path to a file with the client secret, e.g. a mounted Kubernetes Secret. | +| `clientAssertionFile` | `oauthClientCredentials` | Path to a signed JWT client assertion (RFC 7523), e.g. the Azure Workload Identity token file. Re-read on every token request. | +| `clientAuthentication` | `oauthClientCredentials` | `post` (default) sends the secret as form fields; `basic` sends it in an HTTP Basic header. | +| `scope` | `oauthClientCredentials` | Scopes to request. For Entra ID: `api:///.default`. | +| `additionalParameters` | `oauthClientCredentials` | Extra token request fields, e.g. `audience` (Auth0, Okta) or `resource` (Entra v1, ADFS). Keys are lower-cased. | +| `defaultTokenLifetimeSeconds` | `oauthClientCredentials` | Lifetime to assume if your provider omits `expires_in`. Required in that case. | +| `refreshBeforeExpirySeconds` | `oauthClientCredentials` | Refresh this many seconds before expiry. Default `300`, capped at half the token lifetime. | +| `tokenFile` | `oauthTokenFile` | Path to a file containing only the access token. | +| `tokenFileReloadSeconds` | `oauthTokenFile` | How often to re-read the file. Default `30`. | + +Exactly one of `clientSecret`, `clientSecretFile` and `clientAssertionFile` must be set. Every option can also be set with an environment variable, for example `REPLICATOR_SINK_AUTH_TOKENENDPOINT` or `REPLICATOR_SINK_AUTH_ADDITIONALPARAMETERS_AUDIENCE`. The exception is an `additionalParameters` key that contains an underscore (such as `requested_token_use`): environment variable names use underscores as separators, so set those keys in the YAML file. + +## KurrentDB permissions + +Replication reads `$all` and stream metadata on the source, and on the target it writes to any stream, sets stream metadata and deletes streams. Map the Replicator identity's role claim to `$admins` in the KurrentDB OAuth configuration on each cluster (or grant equivalent ACLs). + +If the reader's token cannot read a stream's metadata (`PermissionDenied`), Replicator stops replicating with an error rather than copying events it cannot check against scavenge rules. Fix the role mapping and restart Replicator. + +## How tokens are handled + +- Every gRPC call carries a current token. Tokens are refreshed before they expire, without restarting Replicator. +- If the identity provider is unreachable, or KurrentDB rejects a token, replication pauses with a warning (logged at most once a minute) and resumes on its own once a valid token is available. A token KurrentDB rejected is not sent again for 60 seconds, and after that only one call re-tests it. +- The long-running `$all` read and the realtime subscription are authorized when they start. If KurrentDB ends them when the token expires, Replicator restarts them from the last checkpoint with a fresh token. +- Tokens, secrets and assertions are never logged. + +## Example: Entra ID with a client secret + +1. Register an application for KurrentDB. Expose an Application ID URI (e.g. `api://kurrentdb`) and create an app role (e.g. `Replicator`) that your KurrentDB OAuth configuration maps to `$admins`. +2. Register an application for Replicator, create a client secret, and grant it the `Replicator` app role on the KurrentDB application (admin consent required). +3. Configure the side that talks to the OAuth-enabled cluster: + +```yaml +replicator: + reader: + protocol: grpc + connectionString: "esdb://source.example.com:2113?tls=true" + auth: + type: oauthClientCredentials + tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + clientId: "" + clientSecretFile: /var/run/secrets/replicator/client-secret + scope: "api://kurrentdb/.default" + sink: + protocol: grpc + connectionString: "esdb://admin:changeit@target.example.com:2113?tls=true" +``` + +## Example: Entra ID with AKS Workload Identity (no secret) + +Federate the Replicator application with the Kubernetes service account, then point `clientAssertionFile` at the projected token: + +```yaml +serviceAccountName: replicator-wi +podLabels: + azure.workload.identity/use: "true" +replicator: + sink: + protocol: grpc + connectionString: "esdb://target.example.com:2113?tls=true" + auth: + type: oauthClientCredentials + tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + clientId: "" + clientAssertionFile: /var/run/secrets/azure/tokens/azure-identity-token + scope: "api://kurrentdb/.default" +``` + +## Example: other providers + +Keycloak, Okta, Auth0 and other OAuth 2.0 servers work the same way. Many need an `audience`: + +```yaml + auth: + type: oauthClientCredentials + tokenEndpoint: "https://idp.example.com/oauth2/token" + clientId: replicator + clientSecretFile: /var/run/secrets/replicator/client-secret + additionalParameters: + audience: kurrentdb +``` + +Token file written by another process: + +```yaml + auth: + type: oauthTokenFile + tokenFile: /var/run/secrets/kurrentdb/token +``` + +## Helm + +Put secrets in Kubernetes Secrets and inject them with `extraEnv`, `extraEnvFrom`, `extraVolumes` and `extraVolumeMounts`; don't put `clientSecret` in your values file, because it ends up in a ConfigMap. See [Kubernetes](../kubernetes/). diff --git a/docs/src/content/docs/deployment/configuration.mdx b/docs/src/content/docs/deployment/configuration.mdx index ff8aaa18..cd5fb795 100644 --- a/docs/src/content/docs/deployment/configuration.mdx +++ b/docs/src/content/docs/deployment/configuration.mdx @@ -20,11 +20,13 @@ Available configuration options are: | `reader.connectionString` | Connection string for the [source](../../features/readers/) cluster or instance | | `reader.protocol` | Reader protocol (`tcp` or `grpc`) | | `reader.pageSize` | Reader page size (only applicable for TCP protocol | +| `reader.auth.*` | Optional reader [authentication](../authentication/) (gRPC only): `type` is `connectionString` (default), `oauthClientCredentials` or `oauthTokenFile` | | `sink.connectionString` | Connection string for the [target](../../features/sinks/) cluster or instance | | `sink.protocol` | Writer protocol (`tcp` or `grpc`) | | `sink.partitionCount` | Number of [partitioned](../../features/writers/) concurrent writers | | `sink.partitioner` | Custom JavaScript [partitioner](../../features/writers/) | | `sink.bufferSize` | Size of the sink buffer, `1000` events by default | +| `sink.auth.*` | Optional sink [authentication](../authentication/) (gRPC only), configured independently of the reader | | `scavenge` | Enable real-time [scavenge](../../features/scavenge/) | | `runContinuously` | Set to `false` if you want Replicator to stop when it reaches the end of `$all` stream. Default is `true`, so the replication continues until you stop it explicitly. | | `filters` | Add one or more of provided [filters](../../features/filters/) | diff --git a/docs/src/content/docs/deployment/docker.mdx b/docs/src/content/docs/deployment/docker.mdx index 06b905f2..05dfff58 100755 --- a/docs/src/content/docs/deployment/docker.mdx +++ b/docs/src/content/docs/deployment/docker.mdx @@ -31,4 +31,29 @@ When you start all the component using `docker-compose up`, you'd be able to che Watch out for the replication gap and ensure that it decreases. +## OAuth authentication + +Every `auth` option can be set with an environment variable named `REPLICATOR__AUTH_