From 2efafa6d6963a1c0838ac49365ac35bd2ed2702a Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:04:44 -0400 Subject: [PATCH 01/67] docs: spec for OAuth authentication of gRPC reader and sink Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 235 ++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-03-oauth-authentication-design.md diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md new file mode 100644 index 00000000..4afa5ba4 --- /dev/null +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -0,0 +1,235 @@ +# OAuth authentication for KurrentDB gRPC reader and sink + +Date: 2026-10-03 +Status: Draft (for review) + +## Problem + +A customer runs KurrentDB with OAuth authentication (Microsoft Entra ID as the identity provider). With OAuth enabled, KurrentDB no longer accepts basic username/password credentials, so Replicator cannot read from or write to that cluster: today the only way to authenticate the gRPC client is `user:pass@` in the connection string. + +Replicator is a headless service, so it must authenticate as a machine identity (no user, no browser). + +## Goals + +1. Replicator authenticates to a KurrentDB cluster using an OAuth 2.0 access token, for the gRPC protocol. +2. Provider-agnostic: works with any OAuth 2.0 authorization server that supports the client credentials grant (Entra ID, Keycloak, Okta, Auth0, …). Entra ID is the first target and is documented explicitly; nothing in code is Entra-specific. +3. Two token sources: + - **Client credentials** — Replicator obtains and refreshes tokens itself (RFC 6749 §4.4), authenticating to the token endpoint with a client secret or a client assertion (RFC 7523; covers Entra workload identity federation). + - **Token file** — an external process keeps a valid access token in a file; Replicator reads it. +4. Reader (source) and sink (target) are configured completely independently. Any combination is valid: basic + OAuth, OAuth + basic, OAuth + OAuth with different providers/clients, basic + basic. +5. Tokens are refreshed before expiry without restarting Replicator, including across long-running reads and the realtime subscription. +6. Secrets and tokens are never logged. + +## Non-goals + +- OAuth for the TCP protocol (EventStore TCP clients; KurrentDB's OAuth applies to gRPC only), Kafka sink, or the HTTP transform. +- Interactive flows (authorization code, device code), refresh tokens, mTLS client authentication to the token endpoint, OIDC discovery (`authority` lookup). The token endpoint URL is configured directly. +- Custom CA trust for the token endpoint (system trust store is used; can be added later). +- Upgrading from `EventStore.Client.Grpc.Streams` 23.3.8 to `KurrentDB.Client`. +- Securing Replicator's own HTTP API. + +## Background: how the gRPC client applies credentials + +Verified by decompiling `EventStore.Client` 23.3.8 (`EventStoreCallOptions`, `EventStoreClientOperationOptions`): + +- Every call (unary and streaming, including `ReadAllAsync` and `SubscribeToAllAsync`) builds `CallOptions` with `CallCredentials.FromInterceptor(...)` **iff** `userCredentials ?? settings.DefaultCredentials` is non-null. +- That interceptor sets the `authorization` header to the result of `await settings.OperationOptions.GetAuthenticationHeaderValue(credentials, ct)`. The default implementation returns `credentials.ToString()` (`Basic …` or `Bearer …`). +- `GetAuthenticationHeaderValue` is a public, settable `Func>`, evaluated **per call, asynchronously**. + +This is the integration seam: set `DefaultCredentials` to a sentinel bearer credential (so the client attaches call credentials) and replace `GetAuthenticationHeaderValue` with a function that returns `"Bearer " + await tokenSource.GetAccessToken(ct)`. No gRPC interceptor, custom `HttpMessageHandler`, or change to the reader/writer code is needed, and the client's TLS handling (`tls`, `tlsVerifyCert`, `tlsCaFile`) stays untouched. + +`CallCredentials` are only sent over TLS channels by Grpc.Net.Client. OAuth therefore requires `tls=true` (the default); see Validation. + +Existing call sites (`GrpcEventReader`, `GrpcEventWriter`, `Realtime`, `ConnectionExtensions`, `ScavengedEventsFilter`) pass no per-call credentials, so they all fall back to `DefaultCredentials` and pick up the token automatically. + +## Configuration + +A new optional `auth` section under both `replicator.reader` and `replicator.sink`. Each side is bound and validated independently; absence of the section means today's behaviour. + +```yaml +replicator: + reader: + protocol: grpc + connectionString: "esdb://source.example.com:2113?tls=true" + auth: + type: oauthClientCredentials + tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + clientId: "" + clientSecretFile: /var/run/secrets/replicator/reader-client-secret + scope: "api:///.default" + sink: + protocol: grpc + connectionString: "esdb://admin:changeit@target.example.com:2113?tls=true" + # no auth section: basic credentials from the connection string, as today +``` + +### `auth.type` + +| Value | Meaning | +|---|---| +| `connectionString` (default) | Current behaviour. Credentials, if any, come from the connection string (`user:pass@`). | +| `oauthClientCredentials` | Replicator fetches and caches access tokens from `tokenEndpoint` using the client credentials grant. | +| `oauthTokenFile` | Replicator reads the access token from `tokenFile`. | + +Matching is case-insensitive. An unknown value fails startup. + +### Options for `oauthClientCredentials` + +| Option | Required | Description | +|---|---|---| +| `tokenEndpoint` | yes | Absolute token endpoint URL. Must be `https`, except `http` is allowed for loopback hosts (local testing). | +| `clientId` | yes | OAuth client id. | +| `clientSecret` | one of three | Client secret value. Intended for env var injection (`REPLICATOR_READER_AUTH_CLIENTSECRET`). | +| `clientSecretFile` | one of three | Path to a file containing the client secret (e.g. mounted Kubernetes Secret). Read once at startup; trailing whitespace trimmed. | +| `clientAssertionFile` | one of three | Path to a file containing a signed JWT client assertion (RFC 7523), e.g. the Azure Workload Identity projected token (`AZURE_FEDERATED_TOKEN_FILE`). Re-read on every token request because the platform rotates it. | +| `clientAuthentication` | no | How a client secret is sent: `post` (default; `client_id`/`client_secret` form fields) or `basic` (HTTP Basic header, RFC 6749 §2.3.1). Ignored with `clientAssertionFile`. | +| `scope` | no | Space-separated scopes sent as `scope`. For Entra ID: `api:///.default`. | +| `additionalParameters` | no | Map of extra form fields added to the token request, e.g. `audience` (Auth0, Okta) or `resource` (Entra v1 / ADFS). Must not override `grant_type`, `client_id`, `client_secret`, `client_assertion`, `client_assertion_type`, or `scope`; doing so fails startup. | +| `refreshBeforeExpirySeconds` | no | Refresh the token when it has fewer than this many seconds left. Default `300`. Effective value is capped at half the token lifetime, so short-lived tokens still get reused. | + +Exactly one of `clientSecret`, `clientSecretFile`, `clientAssertionFile` must be set. + +### Options for `oauthTokenFile` + +| Option | Required | Description | +|---|---|---| +| `tokenFile` | yes | Path to a file containing the raw access token (no `Bearer ` prefix; trailing whitespace trimmed). | +| `tokenFileReloadSeconds` | no | How often the file is re-read. Default `30`. | + +### Environment variables + +The existing `EnvConfigProvider` maps `REPLICATOR_A_B_C` to `Replicator:A:B:C`, so all options are settable via env vars without code changes, e.g. `REPLICATOR_SINK_AUTH_TYPE=oauthClientCredentials`, `REPLICATOR_SINK_AUTH_CLIENTSECRET=…`. `additionalParameters` entries map as `REPLICATOR_SINK_AUTH_ADDITIONALPARAMETERS_AUDIENCE=…`. + +## Design + +### Components + +All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single job and is testable in isolation. + +| Unit | Responsibility | +|---|---| +| `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | +| `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`. | +| `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests) and a `TimeProvider`. | +| `TokenFileSource` | Reads and caches the token file; reloads on interval. Takes a `TimeProvider`. | +| `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue`. Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | + +### Wiring + +- `replicator.Settings.EsdbSettings` gains `AuthSettings Auth { get; init; }` (bound from `auth`). `AuthSettings` is a settings-binding record in the host project; `Startup` maps it to `GrpcAuthOptions`. +- `GrpcConfigurator` gets a constructor `GrpcConfigurator(GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth)` (mirroring how `TcpConfigurator` receives `pageSize`). `ConfigureReader` applies `readerAuth`; `ConfigureWriter` applies `sinkAuth`. The `IConfigurator` interface is unchanged. +- Inside `ConfigureEventStoreGrpc`: `EventStoreClientSettings.Create(connectionString)` → validate → if type is not `connectionString`, create the token source and `GrpcAuthentication.Apply` → build client. Each side gets its own token source instance; nothing is shared between reader and sink. +- `Startup` rejects a non-default `auth.type` when that side's `protocol` is not `grpc`. + +### Client credentials token flow + +Request: `POST tokenEndpoint`, `application/x-www-form-urlencoded`: + +- `grant_type=client_credentials` +- `scope` if configured +- client authentication: + - secret + `post`: `client_id`, `client_secret` form fields + - secret + `basic`: `Authorization: Basic base64(urlencode(client_id):urlencode(client_secret))` + - assertion: `client_id`, `client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer`, `client_assertion=` +- `additionalParameters` + +Response: JSON with `access_token` (required), `token_type` (if present must be `Bearer`, case-insensitive), `expires_in` (seconds; if absent, assume 3600 and log a warning once). + +Caching and refresh: + +- Cached `(token, expiresAt)`. `expiresAt = requestStartTime + expires_in` (request start, not response time, to stay conservative). +- `GetAccessToken` returns the cached token if `now < expiresAt - refreshWindow`, where `refreshWindow = min(refreshBeforeExpirySeconds, lifetime / 2)`. +- Otherwise one caller performs the refresh (guarded by `SemaphoreSlim`); concurrent callers await the same refresh. +- If a refresh fails while the cached token is still valid (`now < expiresAt - 30s`), log a warning and return the cached token; the next call retries. If there is no usable token, throw `OAuthTokenException` (message includes HTTP status and the OAuth `error` / `error_description` fields when present, never the request body or secrets). +- Token requests have a 30-second timeout. No retry loop inside the source: a token failure surfaces as a failed gRPC call (the exception thrown from `GetAuthenticationHeaderValue` fails the call), and existing error handling applies unchanged: + - Reader: `ReaderPipe` logs the error and stops; `Replicator` restarts the read from the last checkpoint after `RestartDelayInSeconds` (when `RunContinuously`). + - Sink: `SinkPipe`'s retry policy (10 incremental retries, ~0.5 s total) applies, then the failure propagates exactly as a KurrentDB write failure does today. + - Realtime subscription: `Realtime.HandleDrop` resubscribes. + +A token-endpoint outage therefore behaves the same as an outage of the corresponding KurrentDB cluster does today. Making sink retries outage-tolerant is out of scope. + +Tokens are lazily fetched on first use, so a token-endpoint outage at startup does not crash DI construction. + +### Token file flow + +- On first call and whenever `now - lastReadAt >= tokenFileReloadSeconds`, read the file, trim, cache. +- Empty or missing file: if a previously read token exists, log a warning and keep using it; otherwise throw `OAuthTokenException`. +- No JWT parsing; the external process owns validity. + +### Long-lived calls + +Credentials are evaluated when a call starts. `ReadAllAsync` and `SubscribeToAllAsync` are long-lived server-streaming calls; if KurrentDB terminates a stream when its token expires, the existing paths recover with a fresh token: + +- `Realtime.HandleDrop` resubscribes (new call → new header). +- A failed `ReadAllAsync` enumeration is logged by `ReaderPipe`, and `Replicator` restarts the read from the last checkpoint after `RestartDelayInSeconds`. + +No change to these paths is planned. Whether KurrentDB actually terminates streams on token expiry is to be confirmed in the manual end-to-end test (see Testing); if it causes noisy restarts, a follow-up can proactively recycle the read before expiry. + +### Validation (startup, fail fast) + +Collected per side and reported together with the side name (`reader` / `sink`), e.g. `Invalid reader auth configuration: clientId is required; tls must be enabled for OAuth`. + +- `type` is not `connectionString` and `protocol` is not `grpc` → error. +- OAuth type and the connection string contains credentials (`DefaultCredentials != null` after parsing) → error (ambiguous; remove `user:pass@`). +- OAuth type and `ConnectivitySettings.Insecure` (`tls=false`) → error (bearer tokens are not sent over insecure channels). +- `oauthClientCredentials`: `tokenEndpoint` absolute URL with `https` (or `http` + loopback); `clientId` non-empty; exactly one of `clientSecret` / `clientSecretFile` / `clientAssertionFile`; `clientSecretFile` exists and is non-empty; `clientAssertionFile` exists (content checked at request time since it rotates); `clientAuthentication` ∈ {`post`, `basic`}; `refreshBeforeExpirySeconds` ≥ 0; `additionalParameters` does not contain reserved keys. +- `oauthTokenFile`: `tokenFile` set; `tokenFileReloadSeconds` > 0. File existence is not required at startup (the sidecar may write it after the pod starts). +- OAuth-only options set while `type` is `connectionString` → warning (likely misconfiguration), not error. + +### Logging and secrets + +- Never log tokens, client secrets, client assertions, or token-request bodies. +- Info: token acquired for ``, with expiry time and token endpoint host. Debug: cache hits/refresh decisions. +- `EnvConfigProvider` currently prints every `REPLICATOR_*` variable value to stdout. Change it to redact values whose final key segment is `ClientSecret` or `Password`, and to mask the user-info part of keys ending in `ConnectionString` (`esdb://admin:***@host`). This fixes an existing leak of basic-auth passwords and prevents the new secret from leaking. + +## Helm chart + +Needed so secrets and identity can be injected without putting them in the configmap: + +- `extraEnv` (list of `EnvVar`) and `extraEnvFrom` (list of `EnvFromSource`) on the container. +- `extraVolumes` / `extraVolumeMounts` for secret or token files. +- `serviceAccountName` and `podLabels` (Azure Workload Identity needs a federated service account and the `azure.workload.identity/use: "true"` pod label). +- `replicator.reader.auth` / `replicator.sink.auth` pass through to the generated `appsettings.yaml` like other keys; values docs warn against putting `clientSecret` there. + +All new values default to empty, so existing installs render identically. + +## Documentation + +- New page `docs/src/content/docs/deployment/authentication.mdx`: auth types, option reference, KurrentDB role requirements, and three worked examples: + 1. Entra ID with client secret (app registration for KurrentDB exposing an app ID URI and an app role mapped to `$admins`; app registration for Replicator granted that role; `scope: api:///.default`). + 2. Entra ID with AKS Workload Identity (`clientAssertionFile: /var/run/secrets/azure/tokens/azure-identity-token`, federated credential, Helm `serviceAccountName` + `podLabels`). + 3. Generic provider (Keycloak/Okta) with `additionalParameters.audience`, and a token-file example. +- `deployment/configuration.mdx`: add the `auth.*` rows to the options table and link to the new page. +- `features/readers.mdx` / `features/sinks.mdx`: one-line note that gRPC supports OAuth. +- `CHANGELOG.md` entry. + +Required permissions are called out: reading `$all`, reading stream metadata, writing to arbitrary streams, setting metadata and deleting streams require the token's role claim to map to `$admins` (or ACLs granting equivalent rights) on each cluster. + +## Testing + +Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMessageHandler` and `FakeTimeProvider`: + +- `ClientCredentialsTokenSource` + - request shape for each client authentication mode (post, basic, assertion), with `scope` and `additionalParameters` + - caches token; refreshes inside the refresh window; caps window at half lifetime + - concurrent callers trigger exactly one token request + - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying `error`/`error_description` + - missing `access_token`, non-Bearer `token_type`, missing `expires_in` default + - assertion file re-read per request + - exception messages and logs contain no secret/assertion/token +- `TokenFileSource`: reload interval, trimming, missing/empty file with and without previous token. +- `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. +- `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) +- `EnvConfigProvider` redaction. +- Configuration binding: YAML with reader basic + sink OAuth binds to independent `GrpcAuthOptions`, and vice versa. + +Existing container-based tests keep running unchanged (`connectionString` default). + +Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth plugin needs a licence and a real IdP): KurrentDB with OAuth + Entra ID as sink, basic-auth KurrentDB/EventStoreDB as reader; replicate, run past a token expiry, confirm continued replication and observe whether the subscription/read is terminated at expiry. + +## Risks + +- **Stream termination at token expiry** may cause restart churn on long reads; mitigated by existing restart logic and checked in the manual test. +- **Sentinel credential reliance** on `GetAuthenticationHeaderValue` being honoured: it is public API on the current client; a future move to `KurrentDB.Client` must re-verify it. Covered by the end-to-end header test so an upgrade that breaks it fails CI. +- **Role mapping** is the most likely customer misconfiguration (token accepted but `PermissionDenied`); the docs cover it and `PermissionDenied` errors are already logged by the reader/writer. From f6a9aeb5089a8edf88e8d44d26816e370634d9e5 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:09:10 -0400 Subject: [PATCH 02/67] docs: address spec review round 1 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 68 +++++++++++++------ 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 4afa5ba4..d9ae00c1 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -85,6 +85,7 @@ Matching is case-insensitive. An unknown value fails startup. | `clientAuthentication` | no | How a client secret is sent: `post` (default; `client_id`/`client_secret` form fields) or `basic` (HTTP Basic header, RFC 6749 §2.3.1). Ignored with `clientAssertionFile`. | | `scope` | no | Space-separated scopes sent as `scope`. For Entra ID: `api:///.default`. | | `additionalParameters` | no | Map of extra form fields added to the token request, e.g. `audience` (Auth0, Okta) or `resource` (Entra v1 / ADFS). Must not override `grant_type`, `client_id`, `client_secret`, `client_assertion`, `client_assertion_type`, or `scope`; doing so fails startup. | +| `defaultTokenLifetimeSeconds` | no | Lifetime to assume when the token response has no `expires_in`. If unset and the provider omits `expires_in`, token acquisition fails with an error explaining this option. Must be > 0. | | `refreshBeforeExpirySeconds` | no | Refresh the token when it has fewer than this many seconds left. Default `300`. Effective value is capped at half the token lifetime, so short-lived tokens still get reused. | Exactly one of `clientSecret`, `clientSecretFile`, `clientAssertionFile` must be set. @@ -113,13 +114,14 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single | `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`. | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests) and a `TimeProvider`. | | `TokenFileSource` | Reads and caches the token file; reloads on interval. Takes a `TimeProvider`. | +| `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue`. Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | ### Wiring - `replicator.Settings.EsdbSettings` gains `AuthSettings Auth { get; init; }` (bound from `auth`). `AuthSettings` is a settings-binding record in the host project; `Startup` maps it to `GrpcAuthOptions`. - `GrpcConfigurator` gets a constructor `GrpcConfigurator(GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth)` (mirroring how `TcpConfigurator` receives `pageSize`). `ConfigureReader` applies `readerAuth`; `ConfigureWriter` applies `sinkAuth`. The `IConfigurator` interface is unchanged. -- Inside `ConfigureEventStoreGrpc`: `EventStoreClientSettings.Create(connectionString)` → validate → if type is not `connectionString`, create the token source and `GrpcAuthentication.Apply` → build client. Each side gets its own token source instance; nothing is shared between reader and sink. +- Inside `ConfigureEventStoreGrpc`: `EventStoreClientSettings.Create(connectionString)` → validate → if type is not `connectionString`, create the token source and `GrpcAuthentication.Apply` → build client. `GrpcEventReader` and `GrpcEventWriter` receive the side's `IAccessTokenSource?` (null for `connectionString`) for `TokenGate`. Each side gets its own token source instance; nothing is shared between reader and sink. - `Startup` rejects a non-default `auth.type` when that side's `protocol` is not `grpc`. ### Client credentials token flow @@ -134,20 +136,24 @@ Request: `POST tokenEndpoint`, `application/x-www-form-urlencoded`: - assertion: `client_id`, `client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer`, `client_assertion=` - `additionalParameters` -Response: JSON with `access_token` (required), `token_type` (if present must be `Bearer`, case-insensitive), `expires_in` (seconds; if absent, assume 3600 and log a warning once). +Response handling (RFC 6749 §5.1): + +- HTTP 200 with JSON body. `access_token` is required and non-empty. `token_type` is required and must equal `Bearer` (case-insensitive); any other value or absence is rejected. +- `expires_in` must be a positive integer number of seconds (JSON number or numeric string, as some providers send it), at most 86 400 (24 h; larger values are clamped and logged once). Zero, negative, fractional, malformed or overflowing values are rejected. +- If `expires_in` is absent, the configured `defaultTokenLifetimeSeconds` is used. If that is not configured either, the response is rejected with an error telling the operator to set it. There is no silent default. +- Non-2xx responses are errors; the OAuth `error` / `error_description` fields are included in the exception message when the body is a JSON error response. +- Redirects are not followed (`AllowAutoRedirect = false`). Any 3xx response is an error naming the configured endpoint, because following a 307/308 would replay the client secret or assertion to another origin. +- Each request has a 30-second timeout. Caching and refresh: -- Cached `(token, expiresAt)`. `expiresAt = requestStartTime + expires_in` (request start, not response time, to stay conservative). +- Cached `(token, expiresAt)`. `expiresAt = requestStartTime + lifetime` (request start, not response time, to stay conservative). - `GetAccessToken` returns the cached token if `now < expiresAt - refreshWindow`, where `refreshWindow = min(refreshBeforeExpirySeconds, lifetime / 2)`. -- Otherwise one caller performs the refresh (guarded by `SemaphoreSlim`); concurrent callers await the same refresh. -- If a refresh fails while the cached token is still valid (`now < expiresAt - 30s`), log a warning and return the cached token; the next call retries. If there is no usable token, throw `OAuthTokenException` (message includes HTTP status and the OAuth `error` / `error_description` fields when present, never the request body or secrets). -- Token requests have a 30-second timeout. No retry loop inside the source: a token failure surfaces as a failed gRPC call (the exception thrown from `GetAuthenticationHeaderValue` fails the call), and existing error handling applies unchanged: - - Reader: `ReaderPipe` logs the error and stops; `Replicator` restarts the read from the last checkpoint after `RestartDelayInSeconds` (when `RunContinuously`). - - Sink: `SinkPipe`'s retry policy (10 incremental retries, ~0.5 s total) applies, then the failure propagates exactly as a KurrentDB write failure does today. - - Realtime subscription: `Realtime.HandleDrop` resubscribes. - -A token-endpoint outage therefore behaves the same as an outage of the corresponding KurrentDB cluster does today. Making sink retries outage-tolerant is out of scope. +- Otherwise a refresh is needed. Refreshes are single-flight via a shared in-flight `Task`: the first caller starts the request and stores the task; every caller that arrives while it is running awaits the same task and observes the same result or exception. The stored task is cleared when it completes. +- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`), log a warning and return the cached token. +- If there is no usable token, throw `OAuthTokenException` (message includes HTTP status and OAuth `error` / `error_description` when present; never the request body, secret, assertion or token). +- Failure cooldown: after a failed refresh, calls within the next 5 seconds do not start a new request. They return the cached token if it is usable, otherwise rethrow the last failure immediately. This bounds the token-endpoint request rate during an outage to one per 5 seconds per side, and keeps queued callers from each waiting up to 30 seconds in turn. +- `GetAccessToken` honours its `CancellationToken` for the caller's wait. The shared request itself is cancelled only by its 30-second timeout or disposal, so one caller cancelling does not fail the others. Tokens are lazily fetched on first use, so a token-endpoint outage at startup does not crash DI construction. @@ -157,14 +163,27 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do - Empty or missing file: if a previously read token exists, log a warning and keep using it; otherwise throw `OAuthTokenException`. - No JWT parsing; the external process owns validity. -### Long-lived calls +### Failure handling and recovery + +Note from the decompiled client: `GetAuthenticationHeaderValue` is always invoked with `CancellationToken.None`, so any token fetch that happens inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires tokens *before* starting gRPC calls wherever a caller can wait, and treats the header hook as a cache read in the normal case. + +**Sink (writer).** Today, after `SinkPipe` exhausts its short retry policy (10 incremental retries, ~0.5 s), the writer task faults and `Replicator` does not observe it, so the channels fill and replication stalls permanently. That is an existing behaviour for KurrentDB outages and stays out of scope, but token-endpoint outages must not trigger it. When the sink uses OAuth, `GrpcEventWriter` receives the side's `IAccessTokenSource` and, before every write, calls `TokenGate.WaitForToken(source, cancellationToken)`: + +- Calls `GetAccessToken(ct)`. On `OAuthTokenException` it logs a warning (first failure, then at most once a minute while it continues) and retries with exponential backoff (1 s, 2 s, 4 s … capped at 30 s) until a token is obtained or `ct` is cancelled. +- On success after failures, logs that sink authentication recovered. +- The write then runs; the header hook finds the freshly cached token. + +The write blocks while the token endpoint is down, which applies backpressure through the existing bounded channels. When the endpoint recovers, replication resumes with no restart. Non-token failures are unchanged. + +**Reader.** A token failure fails the `ReadAllAsync` call. `ReaderPipe` logs it and stops, and `Replicator` restarts the read from the last checkpoint after `RestartDelayInSeconds` (when `RunContinuously`). No change is needed. The reader also calls `TokenGate.WaitForToken` once at the start of `ReadEvents` so a startup outage is logged clearly and waited out instead of turned into a restart loop. -Credentials are evaluated when a call starts. `ReadAllAsync` and `SubscribeToAllAsync` are long-lived server-streaming calls; if KurrentDB terminates a stream when its token expires, the existing paths recover with a fresh token: +**Realtime subscription.** `Realtime.Start` currently sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored. This is a latent bug that token failures make likely, and it is fixed as part of this work: -- `Realtime.HandleDrop` resubscribes (new call → new header). -- A failed `ReadAllAsync` enumeration is logged by `ReaderPipe`, and `Replicator` restarts the read from the last checkpoint after `RestartDelayInSeconds`. +- `_started` is set only after `SubscribeToAllAsync` succeeds; on failure it stays `false`, and the exception propagates to the caller (`ReadEvents`, which is already covered by the restart loop). +- `HandleDrop` (any reason except `Disposed`) starts an observed resubscribe loop: `WaitForToken`, then subscribe, retrying with the same backoff on any failure, logging each failure. A `CancellationTokenSource` owned by `Realtime` cancels it on dispose. At most one resubscribe loop runs at a time. +- `Realtime` gets an internal constructor that takes a `Func>` subscribe delegate so the loop is unit-testable without a server. -No change to these paths is planned. Whether KurrentDB actually terminates streams on token expiry is to be confirmed in the manual end-to-end test (see Testing); if it causes noisy restarts, a follow-up can proactively recycle the read before expiry. +**Long-lived calls and token expiry.** Credentials are evaluated when a call starts. If KurrentDB terminates `ReadAllAsync` or the subscription when the token expires, the paths above recover with a fresh token. Whether it does is confirmed in the manual end-to-end test. If it causes frequent restarts, proactively recycling the read before expiry is a follow-up. ### Validation (startup, fail fast) @@ -173,7 +192,7 @@ Collected per side and reported together with the side name (`reader` / `sink`), - `type` is not `connectionString` and `protocol` is not `grpc` → error. - OAuth type and the connection string contains credentials (`DefaultCredentials != null` after parsing) → error (ambiguous; remove `user:pass@`). - OAuth type and `ConnectivitySettings.Insecure` (`tls=false`) → error (bearer tokens are not sent over insecure channels). -- `oauthClientCredentials`: `tokenEndpoint` absolute URL with `https` (or `http` + loopback); `clientId` non-empty; exactly one of `clientSecret` / `clientSecretFile` / `clientAssertionFile`; `clientSecretFile` exists and is non-empty; `clientAssertionFile` exists (content checked at request time since it rotates); `clientAuthentication` ∈ {`post`, `basic`}; `refreshBeforeExpirySeconds` ≥ 0; `additionalParameters` does not contain reserved keys. +- `oauthClientCredentials`: `tokenEndpoint` absolute URL with `https` (or `http` + loopback); `clientId` non-empty; exactly one of `clientSecret` / `clientSecretFile` / `clientAssertionFile`; `clientSecretFile` exists and is non-empty; `clientAssertionFile` exists (content checked at request time since it rotates); `clientAuthentication` ∈ {`post`, `basic`}; `refreshBeforeExpirySeconds` ≥ 0; `defaultTokenLifetimeSeconds` > 0 when set; `additionalParameters` does not contain reserved keys. - `oauthTokenFile`: `tokenFile` set; `tokenFileReloadSeconds` > 0. File existence is not required at startup (the sidecar may write it after the pod starts). - OAuth-only options set while `type` is `connectionString` → warning (likely misconfiguration), not error. @@ -181,7 +200,7 @@ Collected per side and reported together with the side name (`reader` / `sink`), - Never log tokens, client secrets, client assertions, or token-request bodies. - Info: token acquired for ``, with expiry time and token endpoint host. Debug: cache hits/refresh decisions. -- `EnvConfigProvider` currently prints every `REPLICATOR_*` variable value to stdout. Change it to redact values whose final key segment is `ClientSecret` or `Password`, and to mask the user-info part of keys ending in `ConnectionString` (`esdb://admin:***@host`). This fixes an existing leak of basic-auth passwords and prevents the new secret from leaking. +- `EnvConfigProvider` currently prints every `REPLICATOR_*` variable value to stdout, which already leaks basic-auth passwords (both `esdb://user:pass@` and the TCP form `DefaultUserCredentials=admin:changeit`). Change it to print `***` instead of the value for any key that ends with `ConnectionString` or contains an `Auth` segment (except `…:Auth:Type`). Key names are still printed so operators can see what was set. ## Helm chart @@ -213,15 +232,21 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - `ClientCredentialsTokenSource` - request shape for each client authentication mode (post, basic, assertion), with `scope` and `additionalParameters` - caches token; refreshes inside the refresh window; caps window at half lifetime - - concurrent callers trigger exactly one token request + - concurrent callers trigger exactly one token request, both when it succeeds and when it fails (all callers see the same exception) + - failure cooldown: calls within 5 s of a failure make no request and rethrow; after the cooldown a new request is made + - one caller cancelling does not cancel the shared request for others - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying `error`/`error_description` - - missing `access_token`, non-Bearer `token_type`, missing `expires_in` default + - response validation: missing/empty `access_token`; missing or non-Bearer `token_type`; `expires_in` zero, negative, fractional, malformed, numeric string, > 24 h (clamped); missing `expires_in` with and without `defaultTokenLifetimeSeconds` + - 3xx response is an error and the redirect target receives no request - assertion file re-read per request - exception messages and logs contain no secret/assertion/token +- `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. +- Sink recovery acceptance test: `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds; the write blocks during failures, then is sent with the new token; cancelling during the outage ends the wait promptly. +- `Realtime`: a failed initial subscribe leaves it restartable; a drop followed by failing resubscribes (token source down) keeps retrying, then subscribes once the source recovers; dispose stops the loop; never two concurrent loops. - `TokenFileSource`: reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. - `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) -- `EnvConfigProvider` redaction. +- `EnvConfigProvider` redaction: gRPC connection string with user-info, TCP connection string with `DefaultUserCredentials`, `Auth:ClientSecret`, `Auth:AdditionalParameters:*` are masked; `Auth:Type` and unrelated keys are printed. - Configuration binding: YAML with reader basic + sink OAuth binds to independent `GrpcAuthOptions`, and vice versa. Existing container-based tests keep running unchanged (`connectionString` default). @@ -230,6 +255,7 @@ Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth p ## Risks +- **Sink stall on KurrentDB outage** (pre-existing, unchanged): a sink write failure that is not a token failure still exhausts `SinkPipe` retries and stalls replication. Out of scope; noted for a follow-up. - **Stream termination at token expiry** may cause restart churn on long reads; mitigated by existing restart logic and checked in the manual test. - **Sentinel credential reliance** on `GetAuthenticationHeaderValue` being honoured: it is public API on the current client; a future move to `KurrentDB.Client` must re-verify it. Covered by the end-to-end header test so an upgrade that breaks it fails CI. - **Role mapping** is the most likely customer misconfiguration (token accepted but `PermissionDenied`); the docs cover it and `PermissionDenied` errors are already logged by the reader/writer. From 6b0bb055090d81b9d0c7e311b0530ed2252b8ec2 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:12:22 -0400 Subject: [PATCH 03/67] docs: address spec review round 2 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 47 +++++++++++++------ 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index d9ae00c1..654f6483 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -115,13 +115,15 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests) and a `TimeProvider`. | | `TokenFileSource` | Reads and caches the token file; reloads on interval. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | +| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `ValueTask EnsureToken(CancellationToken ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token; a no-op when the side uses `connectionString`. `GrpcAuthContext.None` is the no-auth instance. | +| `AuthFailure` | `static bool IsAuthFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` or `PermissionDenied`. | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue`. Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | ### Wiring - `replicator.Settings.EsdbSettings` gains `AuthSettings Auth { get; init; }` (bound from `auth`). `AuthSettings` is a settings-binding record in the host project; `Startup` maps it to `GrpcAuthOptions`. -- `GrpcConfigurator` gets a constructor `GrpcConfigurator(GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth)` (mirroring how `TcpConfigurator` receives `pageSize`). `ConfigureReader` applies `readerAuth`; `ConfigureWriter` applies `sinkAuth`. The `IConfigurator` interface is unchanged. -- Inside `ConfigureEventStoreGrpc`: `EventStoreClientSettings.Create(connectionString)` → validate → if type is not `connectionString`, create the token source and `GrpcAuthentication.Apply` → build client. `GrpcEventReader` and `GrpcEventWriter` receive the side's `IAccessTokenSource?` (null for `connectionString`) for `TokenGate`. Each side gets its own token source instance; nothing is shared between reader and sink. +- `GrpcConfigurator` gets a constructor `GrpcConfigurator(GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth, CancellationToken shutdown)` (mirroring how `TcpConfigurator` receives `pageSize`). `Startup` registers it with a factory that passes `IHostApplicationLifetime.ApplicationStopping` as `shutdown`, which fires before hosted services are stopped. `ConfigureReader` applies `readerAuth`; `ConfigureWriter` applies `sinkAuth`. The `IConfigurator` interface is unchanged. +- Inside `ConfigureEventStoreGrpc`: `EventStoreClientSettings.Create(connectionString)` → validate → if type is not `connectionString`, create the token source and `GrpcAuthentication.Apply` → build client. `GrpcEventReader` (and through it `Realtime` and `ScavengedEventsFilter`) and `GrpcEventWriter` receive the side's `GrpcAuthContext`. Each side gets its own token source instance; nothing is shared between reader and sink. - `Startup` rejects a non-default `auth.type` when that side's `protocol` is not `grpc`. ### Client credentials token flow @@ -165,23 +167,33 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do ### Failure handling and recovery -Note from the decompiled client: `GetAuthenticationHeaderValue` is always invoked with `CancellationToken.None`, so any token fetch that happens inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires tokens *before* starting gRPC calls wherever a caller can wait, and treats the header hook as a cache read in the normal case. +`GetAuthenticationHeaderValue` is always invoked by the client with `CancellationToken.None` (seen in the decompiled `EventStoreCallOptions`), so a token fetch inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires tokens *before* every gRPC call Replicator makes, via `GrpcAuthContext.EnsureToken`, which waits out token-endpoint outages and is cancelled by application shutdown. In the normal case the header hook is then a cache hit. The hook still fetches if the cache is empty (a narrow race at expiry), and a failure there fails that call and is handled by the paths below. -**Sink (writer).** Today, after `SinkPipe` exhausts its short retry policy (10 incremental retries, ~0.5 s), the writer task faults and `Replicator` does not observe it, so the channels fill and replication stalls permanently. That is an existing behaviour for KurrentDB outages and stays out of scope, but token-endpoint outages must not trigger it. When the sink uses OAuth, `GrpcEventWriter` receives the side's `IAccessTokenSource` and, before every write, calls `TokenGate.WaitForToken(source, cancellationToken)`: +Every wait introduced here is bound to the application shutdown token, so nothing outlives shutdown. No new disposal plumbing on `IEventReader` / `IEventWriter` is needed. -- Calls `GetAccessToken(ct)`. On `OAuthTokenException` it logs a warning (first failure, then at most once a minute while it continues) and retries with exponential backoff (1 s, 2 s, 4 s … capped at 30 s) until a token is obtained or `ct` is cancelled. -- On success after failures, logs that sink authentication recovered. -- The write then runs; the header hook finds the freshly cached token. +**Sink (writer).** Today, after `SinkPipe` exhausts its short retry policy (10 incremental retries, ~0.5 s), the writer shovel task faults and `Replicator` does not observe it, so the channels fill and replication stalls permanently. That stays as-is for KurrentDB outages (out of scope), but token-endpoint outages must not trigger it. -The write blocks while the token endpoint is down, which applies backpressure through the existing bounded channels. When the endpoint recovers, replication resumes with no restart. Non-token failures are unchanged. +- `GrpcEventWriter.WriteEvent` calls `auth.EnsureToken(cancellationToken)` before each append, delete or metadata write. `PreparePipe` creates every `SinkContext` with `CancellationToken.None`, so the effective cancellation comes from the shutdown token linked inside `GrpcAuthContext`. +- While the token endpoint is down, the write waits with backoff (1 s, 2 s, 4 s … capped at 30 s) and logs a warning on the first failure and then at most once a minute. The bounded channels apply backpressure to the reader. When the endpoint recovers, the write proceeds with no restart, and recovery is logged. +- On shutdown during an outage, the wait throws `OperationCanceledException`. `ChannelExtensions.Shovel` already treats that as a normal stop, so the writer task completes. +- `Replicator`'s post-read drain loop (`while (sinkChannel.Reader.Count > 0)`) would then spin forever because nothing consumes the channel. It is changed to also exit when the writer task has completed, logging how many events were left unwritten; they are re-read from the last checkpoint on the next start. This is a one-line condition change and also fixes the same hang for any other writer termination. -**Reader.** A token failure fails the `ReadAllAsync` call. `ReaderPipe` logs it and stops, and `Replicator` restarts the read from the last checkpoint after `RestartDelayInSeconds` (when `RunContinuously`). No change is needed. The reader also calls `TokenGate.WaitForToken` once at the start of `ReadEvents` so a startup outage is logged clearly and waited out instead of turned into a restart loop. +**Reader.** `GrpcEventReader` calls `auth.EnsureToken(cancellationToken)` before starting `ReadAllAsync`. A token failure in the middle of the long-running read (after the call started) fails the enumeration. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, which goes through `EnsureToken` again. No pipeline change is needed. -**Realtime subscription.** `Realtime.Start` currently sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored. This is a latent bug that token failures make likely, and it is fixed as part of this work: +**Reader auxiliary calls (scavenge filter).** `ScavengedEventsFilter` issues `GetStreamMetadataAsync` and `ReadStreamAsync` calls during the read. Today `StreamMetaCache.GetOrAddStreamMeta` turns *any* exception into `null`, and the filter treats `null` as "keep the event". If an auth failure took that path with `scavenge: true` (the Helm default), deleted or expired events would be replicated. -- `_started` is set only after `SubscribeToAllAsync` succeeds; on failure it stays `false`, and the exception propagates to the caller (`ReadEvents`, which is already covered by the restart loop). -- `HandleDrop` (any reason except `Disposed`) starts an observed resubscribe loop: `WaitForToken`, then subscribe, retrying with the same backoff on any failure, logging each failure. A `CancellationTokenSource` owned by `Realtime` cancels it on dispose. At most one resubscribe loop runs at a time. -- `Realtime` gets an internal constructor that takes a `Func>` subscribe delegate so the loop is unit-testable without a server. +- `ScavengedEventsFilter` calls `auth.EnsureToken(CancellationToken.None)` before each metadata or stream-size call. The shutdown token linked inside the context still makes it cancellable. During a token-endpoint outage the filter waits instead of falling through. +- `StreamMetaCache.GetOrAddStreamMeta` rethrows when `AuthFailure.IsAuthFailure(e)` is true, instead of returning `null`. Other exceptions keep today's behaviour. An auth failure there (a race at expiry, or the server rejecting the token) then fails the prepare step: `PreparePipe` retries it 10 times, and if it keeps failing the prepare shovel faults and replication stalls with the error logged. That fails closed: scavenged events are never copied because of an auth problem. A persistent server-side rejection is an operator configuration error (role mapping) and is documented as such. +- `GetOrAddStreamSize` already propagates exceptions, so it fails closed through the same path. + +**Realtime subscription.** Today `Realtime.Start` sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored, and a reader restart during a pending resubscribe could open a second subscription. This latent bug becomes likely with tokens, so `Realtime` is reworked around one serialized operation: + +- State: the current `StreamSubscription?` and a single in-flight `Task? _subscribing`, both guarded by a lock. +- `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if a subscription is active, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. +- `SubscribeLoop()` repeats `auth.EnsureToken`, then `SubscribeToAllAsync`, retrying with the same capped backoff and logging each failure, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. +- `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers therefore share one attempt, and there is never more than one subscription. +- `HandleDrop` (any reason except `Disposed`) clears the stored subscription under the lock and calls `EnsureSubscribed(CancellationToken.None)` without awaiting it. The loop is still bound to shutdown, and any exception other than cancellation is caught and logged inside the loop, so nothing is unobserved. +- `Realtime` gets an internal constructor taking a `Func>` subscribe delegate, `GrpcAuthContext` and `TimeProvider`, so all of this is unit-testable without a server. **Long-lived calls and token expiry.** Credentials are evaluated when a call starts. If KurrentDB terminates `ReadAllAsync` or the subscription when the token expires, the paths above recover with a fresh token. Whether it does is confirmed in the manual end-to-end test. If it causes frequent restarts, proactively recycling the read before expiry is a follow-up. @@ -241,8 +253,12 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - assertion file re-read per request - exception messages and logs contain no secret/assertion/token - `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. -- Sink recovery acceptance test: `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds; the write blocks during failures, then is sent with the new token; cancelling during the outage ends the wait promptly. -- `Realtime`: a failed initial subscribe leaves it restartable; a drop followed by failing resubscribes (token source down) keeps retrying, then subscribes once the source recovers; dispose stops the loop; never two concurrent loops. +- `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. +- `AuthFailure.IsAuthFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException`, `Unauthenticated`, `PermissionDenied`, and negative cases. +- `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. +- Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case triggers shutdown during the outage: `Replicate` returns within a bounded time (no drain-loop hang), the remaining count is logged, and the checkpoint is not advanced past unwritten events. +- `StreamMetaCache` / `ScavengedEventsFilter`: an auth failure from the metadata call propagates instead of returning `null`, so the event is not kept; a non-auth failure still returns `null` (unchanged); `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. - `TokenFileSource`: reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. - `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) @@ -256,6 +272,7 @@ Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth p ## Risks - **Sink stall on KurrentDB outage** (pre-existing, unchanged): a sink write failure that is not a token failure still exhausts `SinkPipe` retries and stalls replication. Out of scope; noted for a follow-up. +- **Fail-closed stall on persistent auth rejection**: a token the server rejects (bad role mapping) stalls replication in the scavenge filter or sink with errors logged, rather than replicating incorrectly. Intended; documented in troubleshooting. - **Stream termination at token expiry** may cause restart churn on long reads; mitigated by existing restart logic and checked in the manual test. - **Sentinel credential reliance** on `GetAuthenticationHeaderValue` being honoured: it is public API on the current client; a future move to `KurrentDB.Client` must re-verify it. Covered by the end-to-end header test so an upgrade that breaks it fails CI. - **Role mapping** is the most likely customer misconfiguration (token accepted but `PermissionDenied`); the docs cover it and `PermissionDenied` errors are already logged by the reader/writer. From 4ff10bdc9cba7092f2cd3c6de605f7624fe9c8ea Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:14:48 -0400 Subject: [PATCH 04/67] docs: address spec review round 3 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 654f6483..9e9c5837 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -116,7 +116,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single | `TokenFileSource` | Reads and caches the token file; reloads on interval. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | | `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `ValueTask EnsureToken(CancellationToken ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token; a no-op when the side uses `connectionString`. `GrpcAuthContext.None` is the no-auth instance. | -| `AuthFailure` | `static bool IsAuthFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` or `PermissionDenied`. | +| `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is deliberately excluded: it is an authorization decision (possibly per-stream ACL), not a token problem, and keeps today's handling. | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue`. Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | ### Wiring @@ -176,14 +176,17 @@ Every wait introduced here is bound to the application shutdown token, so nothin - `GrpcEventWriter.WriteEvent` calls `auth.EnsureToken(cancellationToken)` before each append, delete or metadata write. `PreparePipe` creates every `SinkContext` with `CancellationToken.None`, so the effective cancellation comes from the shutdown token linked inside `GrpcAuthContext`. - While the token endpoint is down, the write waits with backoff (1 s, 2 s, 4 s … capped at 30 s) and logs a warning on the first failure and then at most once a minute. The bounded channels apply backpressure to the reader. When the endpoint recovers, the write proceeds with no restart, and recovery is logged. - On shutdown during an outage, the wait throws `OperationCanceledException`. `ChannelExtensions.Shovel` already treats that as a normal stop, so the writer task completes. -- `Replicator`'s post-read drain loop (`while (sinkChannel.Reader.Count > 0)`) would then spin forever because nothing consumes the channel. It is changed to also exit when the writer task has completed, logging how many events were left unwritten; they are re-read from the last checkpoint on the next start. This is a one-line condition change and also fixes the same hang for any other writer termination. +- `Replicator`'s post-read drain loop (`while (sinkChannel.Reader.Count > 0)`) would then spin forever because nothing consumes the channel. It is changed to also exit when the writer task has completed, logging how many events were left unwritten; they are re-read from the last checkpoint on the next start (the checkpoint is only stored after a successful write, in `SinkPipe`). +- The prepare shovel can also be blocked: `PreparePipe` pushes into the bounded sink channel via `sinkChannel.Writer.WriteAsync(ctx, ctx.CancellationToken)`, and that token is `CancellationToken.None`. If the sink channel filled while the writer waited for a token, the prepare shovel stays blocked after the writer stops, and `Replicator` awaits `prepareTask` forever. `Replicator` is changed to create `writerCts` before the pipes and pass `writerCts.Token` to that channel write instead of `ctx.CancellationToken`. `Stop()` already cancels `writerCts`, so the blocked write throws `OperationCanceledException`, which `PreparePipe` already swallows, and the prepare shovel exits through its own (already cancelled) token. In a normal shutdown with a healthy writer nothing changes: the channel drains before `writerCts` is cancelled. -**Reader.** `GrpcEventReader` calls `auth.EnsureToken(cancellationToken)` before starting `ReadAllAsync`. A token failure in the middle of the long-running read (after the call started) fails the enumeration. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, which goes through `EnsureToken` again. No pipeline change is needed. +**Reader.** `GrpcEventReader` calls `auth.EnsureToken(cancellationToken)` before starting `ReadAllAsync`, and before the backwards `ReadAllAsync` in `GetLastPosition` (used by the metrics reporter). A token failure in the middle of the long-running read (after the call started) fails the enumeration. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, which goes through `EnsureToken` again. No pipeline change is needed. + +**Metrics reporter.** `Replicator.Report` calls `reader.GetLastPosition` every `ReportMetricsFrequencyInSeconds` and catches only `OperationCanceledException`, so any other exception ends the reporter for the rest of the run. With the gate above, a token outage makes `GetLastPosition` wait instead of throwing. In addition, `Report` is changed to catch and log (warning, rate-limited to once a minute) any other exception per iteration and continue, so a transient failure, auth or not, no longer stops metrics permanently. **Reader auxiliary calls (scavenge filter).** `ScavengedEventsFilter` issues `GetStreamMetadataAsync` and `ReadStreamAsync` calls during the read. Today `StreamMetaCache.GetOrAddStreamMeta` turns *any* exception into `null`, and the filter treats `null` as "keep the event". If an auth failure took that path with `scavenge: true` (the Helm default), deleted or expired events would be replicated. - `ScavengedEventsFilter` calls `auth.EnsureToken(CancellationToken.None)` before each metadata or stream-size call. The shutdown token linked inside the context still makes it cancellable. During a token-endpoint outage the filter waits instead of falling through. -- `StreamMetaCache.GetOrAddStreamMeta` rethrows when `AuthFailure.IsAuthFailure(e)` is true, instead of returning `null`. Other exceptions keep today's behaviour. An auth failure there (a race at expiry, or the server rejecting the token) then fails the prepare step: `PreparePipe` retries it 10 times, and if it keeps failing the prepare shovel faults and replication stalls with the error logged. That fails closed: scavenged events are never copied because of an auth problem. A persistent server-side rejection is an operator configuration error (role mapping) and is documented as such. +- When the reader uses OAuth, `StreamMetaCache.GetOrAddStreamMeta` rethrows exceptions where `AuthFailure.IsTokenFailure(e)` is true, instead of returning `null`. `StreamMetaCache` gets a constructor flag `failClosedOnTokenFailure`, set only for OAuth readers. For `connectionString` readers (basic auth or anonymous), and for every other exception including `PermissionDenied`, today's warn-and-keep behaviour is unchanged, so existing deployments behave exactly as before. An auth failure there (a race at expiry, or the server rejecting the token) then fails the prepare step: `PreparePipe` retries it 10 times, and if it keeps failing the prepare shovel faults and replication stalls with the error logged. That fails closed: scavenged events are never copied because of an auth problem. A persistent server-side rejection is an operator configuration error (role mapping) and is documented as such. - `GetOrAddStreamSize` already propagates exceptions, so it fails closed through the same path. **Realtime subscription.** Today `Realtime.Start` sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored, and a reader restart during a pending resubscribe could open a second subscription. This latent bug becomes likely with tokens, so `Realtime` is reworked around one serialized operation: @@ -254,10 +257,11 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - exception messages and logs contain no secret/assertion/token - `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. - `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. -- `AuthFailure.IsAuthFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException`, `Unauthenticated`, `PermissionDenied`, and negative cases. +- `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. -- Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case triggers shutdown during the outage: `Replicate` returns within a bounded time (no drain-loop hang), the remaining count is logged, and the checkpoint is not advanced past unwritten events. -- `StreamMetaCache` / `ScavengedEventsFilter`: an auth failure from the metadata call propagates instead of returning `null`, so the event is not kept; a non-auth failure still returns `null` (unchanged); `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. +- Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. +- `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnTokenFailure`, a token failure (`OAuthTokenException`, `Unauthenticated`) from the metadata call propagates instead of returning `null`, so the event is not kept; `PermissionDenied` and other failures still return `null`; without the flag (basic-auth reader), `Unauthenticated` still returns `null` exactly as today; `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. - `TokenFileSource`: reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. @@ -272,7 +276,7 @@ Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth p ## Risks - **Sink stall on KurrentDB outage** (pre-existing, unchanged): a sink write failure that is not a token failure still exhausts `SinkPipe` retries and stalls replication. Out of scope; noted for a follow-up. -- **Fail-closed stall on persistent auth rejection**: a token the server rejects (bad role mapping) stalls replication in the scavenge filter or sink with errors logged, rather than replicating incorrectly. Intended; documented in troubleshooting. +- **Fail-closed stall on persistent token rejection** (OAuth readers only): a token the server rejects as unauthenticated stalls replication in the scavenge filter or sink with errors logged, rather than replicating incorrectly. Intended; documented in troubleshooting. - **Stream termination at token expiry** may cause restart churn on long reads; mitigated by existing restart logic and checked in the manual test. - **Sentinel credential reliance** on `GetAuthenticationHeaderValue` being honoured: it is public API on the current client; a future move to `KurrentDB.Client` must re-verify it. Covered by the end-to-end header test so an upgrade that breaks it fails CI. - **Role mapping** is the most likely customer misconfiguration (token accepted but `PermissionDenied`); the docs cover it and `PermissionDenied` errors are already logged by the reader/writer. From 43a2c69face080d978f4d6cafe16dccbb49736e4 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:14:58 -0400 Subject: [PATCH 05/67] docs: clarify fail-closed scope Co-Authored-By: Claude Opus 5.5 --- .../superpowers/specs/2026-10-03-oauth-authentication-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 9e9c5837..bf9a565a 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -186,7 +186,7 @@ Every wait introduced here is bound to the application shutdown token, so nothin **Reader auxiliary calls (scavenge filter).** `ScavengedEventsFilter` issues `GetStreamMetadataAsync` and `ReadStreamAsync` calls during the read. Today `StreamMetaCache.GetOrAddStreamMeta` turns *any* exception into `null`, and the filter treats `null` as "keep the event". If an auth failure took that path with `scavenge: true` (the Helm default), deleted or expired events would be replicated. - `ScavengedEventsFilter` calls `auth.EnsureToken(CancellationToken.None)` before each metadata or stream-size call. The shutdown token linked inside the context still makes it cancellable. During a token-endpoint outage the filter waits instead of falling through. -- When the reader uses OAuth, `StreamMetaCache.GetOrAddStreamMeta` rethrows exceptions where `AuthFailure.IsTokenFailure(e)` is true, instead of returning `null`. `StreamMetaCache` gets a constructor flag `failClosedOnTokenFailure`, set only for OAuth readers. For `connectionString` readers (basic auth or anonymous), and for every other exception including `PermissionDenied`, today's warn-and-keep behaviour is unchanged, so existing deployments behave exactly as before. An auth failure there (a race at expiry, or the server rejecting the token) then fails the prepare step: `PreparePipe` retries it 10 times, and if it keeps failing the prepare shovel faults and replication stalls with the error logged. That fails closed: scavenged events are never copied because of an auth problem. A persistent server-side rejection is an operator configuration error (role mapping) and is documented as such. +- When the reader uses OAuth, `StreamMetaCache.GetOrAddStreamMeta` rethrows exceptions where `AuthFailure.IsTokenFailure(e)` is true, instead of returning `null`. `StreamMetaCache` gets a constructor flag `failClosedOnTokenFailure`, set only for OAuth readers. For `connectionString` readers (basic auth or anonymous), and for every other exception including `PermissionDenied`, today's warn-and-keep behaviour is unchanged, so existing deployments behave exactly as before. An auth failure there (a race at expiry, or the server rejecting the token) then fails the prepare step: `PreparePipe` retries it 10 times, and if it keeps failing the prepare shovel faults and replication stalls with the error logged. That fails closed: scavenged events are never copied because of an auth problem. A persistent `Unauthenticated` rejection is an operator configuration error (wrong audience, issuer or signing keys in the KurrentDB OAuth config) and is documented as such. `PermissionDenied` from missing role mapping keeps today's warn-and-keep behaviour on metadata reads and surfaces as write/read errors elsewhere, as it does for basic auth today. - `GetOrAddStreamSize` already propagates exceptions, so it fails closed through the same path. **Realtime subscription.** Today `Realtime.Start` sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored, and a reader restart during a pending resubscribe could open a second subscription. This latent bug becomes likely with tokens, so `Realtime` is reworked around one serialized operation: From 544968dab0d255615063d0d7ac2c0a1126357897 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:17:26 -0400 Subject: [PATCH 06/67] docs: address spec review round 4 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 64 ++++++++++++------- 1 file changed, 41 insertions(+), 23 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index bf9a565a..f85d6212 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -111,12 +111,12 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`. | -| `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests) and a `TimeProvider`. | -| `TokenFileSource` | Reads and caches the token file; reloads on interval. Takes a `TimeProvider`. | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)` and `void Invalidate()`, which marks the cached token stale so the next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. | +| `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | +| `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate()`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | -| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `ValueTask EnsureToken(CancellationToken ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token; a no-op when the side uses `connectionString`. `GrpcAuthContext.None` is the no-auth instance. | -| `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is deliberately excluded: it is an authorization decision (possibly per-stream ACL), not a token problem, and keeps today's handling. | +| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `EnsureToken(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token (no-op without OAuth); `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"); without OAuth it just invokes `call`. | +| `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is not a token failure (retrying with a new token does not help), and is handled separately where it matters (scavenge filter). | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue`. Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | ### Wiring @@ -155,7 +155,7 @@ Caching and refresh: - If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`), log a warning and return the cached token. - If there is no usable token, throw `OAuthTokenException` (message includes HTTP status and OAuth `error` / `error_description` when present; never the request body, secret, assertion or token). - Failure cooldown: after a failed refresh, calls within the next 5 seconds do not start a new request. They return the cached token if it is usable, otherwise rethrow the last failure immediately. This bounds the token-endpoint request rate during an outage to one per 5 seconds per side, and keeps queued callers from each waiting up to 30 seconds in turn. -- `GetAccessToken` honours its `CancellationToken` for the caller's wait. The shared request itself is cancelled only by its 30-second timeout or disposal, so one caller cancelling does not fail the others. +- `GetAccessToken` honours its `CancellationToken` for the caller's wait. The shared request runs under its own `CancellationTokenSource` linked to the application shutdown token and the 30-second timeout, never to an individual caller's token, so one caller cancelling does not fail the others, and shutdown aborts an in-flight request. Tokens are lazily fetched on first use, so a token-endpoint outage at startup does not crash DI construction. @@ -167,33 +167,48 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do ### Failure handling and recovery -`GetAuthenticationHeaderValue` is always invoked by the client with `CancellationToken.None` (seen in the decompiled `EventStoreCallOptions`), so a token fetch inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires tokens *before* every gRPC call Replicator makes, via `GrpcAuthContext.EnsureToken`, which waits out token-endpoint outages and is cancelled by application shutdown. In the normal case the header hook is then a cache hit. The hook still fetches if the cache is empty (a narrow race at expiry), and a failure there fails that call and is handled by the paths below. +`GetAuthenticationHeaderValue` is always invoked by the client with `CancellationToken.None` (seen in the decompiled `EventStoreCallOptions`), so a token fetch inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires tokens *before* every gRPC call Replicator makes, and treats the header hook as a cache read in the normal case. The hook still fetches if the cache is empty (a narrow race), and a failure there fails that call like any other token failure below. -Every wait introduced here is bound to the application shutdown token, so nothing outlives shutdown. No new disposal plumbing on `IEventReader` / `IEventWriter` is needed. +Two kinds of token failure are handled the same way: -**Sink (writer).** Today, after `SinkPipe` exhausts its short retry policy (10 incremental retries, ~0.5 s), the writer shovel task faults and `Replicator` does not observe it, so the channels fill and replication stalls permanently. That stays as-is for KurrentDB outages (out of scope), but token-endpoint outages must not trigger it. +- the token cannot be obtained (`OAuthTokenException`), or +- KurrentDB rejects the token it was sent (`RpcException` with `Unauthenticated`), e.g. a token file not yet re-read after rotation, or a token revoked early. -- `GrpcEventWriter.WriteEvent` calls `auth.EnsureToken(cancellationToken)` before each append, delete or metadata write. `PreparePipe` creates every `SinkContext` with `CancellationToken.None`, so the effective cancellation comes from the shutdown token linked inside `GrpcAuthContext`. -- While the token endpoint is down, the write waits with backoff (1 s, 2 s, 4 s … capped at 30 s) and logs a warning on the first failure and then at most once a minute. The bounded channels apply backpressure to the reader. When the endpoint recovers, the write proceeds with no restart, and recovery is logged. +Both are usually transient, so neither may permanently stop replication. Every wait introduced here is bound to the application shutdown token, and the shared token request is too, so nothing outlives shutdown. No new disposal plumbing on `IEventReader` / `IEventWriter` is needed. + +**Token-aware calls.** `GrpcAuthContext.Run(call, ct)` is used for every bounded (non-streaming-for-the-life-of-the-run) gRPC operation: appends, deletes, metadata writes, metadata reads, stream-size reads, `GetLastPosition` and subscribe. It loops: + +1. `EnsureToken(ct)`: waits for a token with capped backoff (1 s, 2 s, 4 s … 30 s), cancellable by `ct` and shutdown. +2. Invoke `call`. +3. If the call throws and `AuthFailure.IsTokenFailure(e)` is true: `Invalidate()` the token source, log a warning (first failure, then at most once a minute; recovery is logged once), wait the next backoff step, and go to 1. +4. Any other exception propagates unchanged, so existing behaviour for non-auth errors is preserved. + +The loop has no attempt limit: a persistent rejection (for example a wrong audience in KurrentDB's OAuth config) shows up as a repeating warning and stalled progress, and replication resumes on its own once the token or the server config is fixed. Retrying is safe: a call rejected as `Unauthenticated` was not executed by the server. + +**Sink (writer).** Today, after `SinkPipe` exhausts its short retry policy (10 incremental retries, ~0.5 s), the writer shovel task faults and `Replicator` does not observe it, so the channels fill and replication stalls permanently. That stays as-is for non-auth KurrentDB failures (out of scope), but token failures must not trigger it. + +- `GrpcEventWriter` runs each append, delete and metadata write through `auth.Run(...)`. `PreparePipe` creates every `SinkContext` with `CancellationToken.None`, so the effective cancellation comes from the shutdown token linked inside `GrpcAuthContext`. +- During a token outage or rejection the write waits; the bounded channels apply backpressure to the reader. When tokens work again, the write proceeds with no restart. - On shutdown during an outage, the wait throws `OperationCanceledException`. `ChannelExtensions.Shovel` already treats that as a normal stop, so the writer task completes. - `Replicator`'s post-read drain loop (`while (sinkChannel.Reader.Count > 0)`) would then spin forever because nothing consumes the channel. It is changed to also exit when the writer task has completed, logging how many events were left unwritten; they are re-read from the last checkpoint on the next start (the checkpoint is only stored after a successful write, in `SinkPipe`). -- The prepare shovel can also be blocked: `PreparePipe` pushes into the bounded sink channel via `sinkChannel.Writer.WriteAsync(ctx, ctx.CancellationToken)`, and that token is `CancellationToken.None`. If the sink channel filled while the writer waited for a token, the prepare shovel stays blocked after the writer stops, and `Replicator` awaits `prepareTask` forever. `Replicator` is changed to create `writerCts` before the pipes and pass `writerCts.Token` to that channel write instead of `ctx.CancellationToken`. `Stop()` already cancels `writerCts`, so the blocked write throws `OperationCanceledException`, which `PreparePipe` already swallows, and the prepare shovel exits through its own (already cancelled) token. In a normal shutdown with a healthy writer nothing changes: the channel drains before `writerCts` is cancelled. +- The prepare shovel can also be blocked: `PreparePipe` pushes into the bounded sink channel via `sinkChannel.Writer.WriteAsync(ctx, ctx.CancellationToken)`, and that token is `CancellationToken.None`. If the sink channel filled while the writer waited, the prepare shovel stays blocked after the writer stops, and `Replicator` awaits `prepareTask` forever. `Replicator` is changed to create `writerCts` before the pipes and pass `writerCts.Token` to that channel write instead of `ctx.CancellationToken`. `Stop()` already cancels `writerCts`, so the blocked write throws `OperationCanceledException`, which `PreparePipe` already swallows, and the prepare shovel exits through its own (already cancelled) token. In a normal shutdown with a healthy writer nothing changes: the channel drains before `writerCts` is cancelled. -**Reader.** `GrpcEventReader` calls `auth.EnsureToken(cancellationToken)` before starting `ReadAllAsync`, and before the backwards `ReadAllAsync` in `GetLastPosition` (used by the metrics reporter). A token failure in the middle of the long-running read (after the call started) fails the enumeration. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, which goes through `EnsureToken` again. No pipeline change is needed. +**Reader.** `GrpcEventReader` calls `auth.EnsureToken(cancellationToken)` before starting the long-running `ReadAllAsync`. A token failure during that read (at start, or if the server ends the stream) fails the enumeration. `ReadEvents` catches it, calls `Invalidate()` if it is a token failure, and rethrows. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, going through `EnsureToken` again with a fresh token. No pipeline change is needed. `GetLastPosition` (a bounded backwards read) goes through `auth.Run`. -**Metrics reporter.** `Replicator.Report` calls `reader.GetLastPosition` every `ReportMetricsFrequencyInSeconds` and catches only `OperationCanceledException`, so any other exception ends the reporter for the rest of the run. With the gate above, a token outage makes `GetLastPosition` wait instead of throwing. In addition, `Report` is changed to catch and log (warning, rate-limited to once a minute) any other exception per iteration and continue, so a transient failure, auth or not, no longer stops metrics permanently. +**Metrics reporter.** `Replicator.Report` calls `reader.GetLastPosition` every `ReportMetricsFrequencyInSeconds` and catches only `OperationCanceledException`, so any other exception ends the reporter for the rest of the run. `GetLastPosition` now waits out token failures. In addition, `Report` is changed to catch and log (warning, rate-limited to once a minute) any other exception per iteration and continue, so a transient failure, auth or not, no longer stops metrics permanently. -**Reader auxiliary calls (scavenge filter).** `ScavengedEventsFilter` issues `GetStreamMetadataAsync` and `ReadStreamAsync` calls during the read. Today `StreamMetaCache.GetOrAddStreamMeta` turns *any* exception into `null`, and the filter treats `null` as "keep the event". If an auth failure took that path with `scavenge: true` (the Helm default), deleted or expired events would be replicated. +**Reader auxiliary calls (scavenge filter).** `ScavengedEventsFilter` issues `GetStreamMetadataAsync` and `ReadStreamAsync` calls during the read. Today `StreamMetaCache.GetOrAddStreamMeta` turns *any* exception into `null`, and the filter treats `null` as "keep the event". If an auth problem took that path with `scavenge: true` (the Helm default), deleted or expired events would be replicated. -- `ScavengedEventsFilter` calls `auth.EnsureToken(CancellationToken.None)` before each metadata or stream-size call. The shutdown token linked inside the context still makes it cancellable. During a token-endpoint outage the filter waits instead of falling through. -- When the reader uses OAuth, `StreamMetaCache.GetOrAddStreamMeta` rethrows exceptions where `AuthFailure.IsTokenFailure(e)` is true, instead of returning `null`. `StreamMetaCache` gets a constructor flag `failClosedOnTokenFailure`, set only for OAuth readers. For `connectionString` readers (basic auth or anonymous), and for every other exception including `PermissionDenied`, today's warn-and-keep behaviour is unchanged, so existing deployments behave exactly as before. An auth failure there (a race at expiry, or the server rejecting the token) then fails the prepare step: `PreparePipe` retries it 10 times, and if it keeps failing the prepare shovel faults and replication stalls with the error logged. That fails closed: scavenged events are never copied because of an auth problem. A persistent `Unauthenticated` rejection is an operator configuration error (wrong audience, issuer or signing keys in the KurrentDB OAuth config) and is documented as such. `PermissionDenied` from missing role mapping keeps today's warn-and-keep behaviour on metadata reads and surfaces as write/read errors elsewhere, as it does for basic auth today. +- Both auxiliary calls go through `auth.Run(..., CancellationToken.None)`; the shutdown token linked inside the context makes them cancellable. Token failures are therefore waited out and retried inside the call and never reach `StreamMetaCache` (except as `OperationCanceledException` at shutdown). +- For OAuth readers, `StreamMetaCache.GetOrAddStreamMeta` must never turn an auth problem into "keep the event". It gets a constructor flag `failClosedOnAuthErrors`, set only for OAuth readers. With the flag, an exception where `IsTokenFailure(e)` is true, or an `RpcException` with `PermissionDenied`, is rethrown instead of returning `null`. A `PermissionDenied` on stream metadata means the token's role cannot see that stream's metadata, so the filter cannot decide safely. The prepare step then fails: `PreparePipe` retries 10 times, then the prepare shovel faults and replication stalls with the error logged. That is fail-closed, and it is a configuration error (the replicator identity must be mapped to `$admins` or have metadata read on all streams) that the docs call out. Restart Replicator after fixing roles. +- Without the flag (`connectionString` readers: basic auth or anonymous), today's warn-and-keep behaviour is unchanged for every exception, so existing deployments behave exactly as before. - `GetOrAddStreamSize` already propagates exceptions, so it fails closed through the same path. **Realtime subscription.** Today `Realtime.Start` sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored, and a reader restart during a pending resubscribe could open a second subscription. This latent bug becomes likely with tokens, so `Realtime` is reworked around one serialized operation: - State: the current `StreamSubscription?` and a single in-flight `Task? _subscribing`, both guarded by a lock. - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if a subscription is active, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. -- `SubscribeLoop()` repeats `auth.EnsureToken`, then `SubscribeToAllAsync`, retrying with the same capped backoff and logging each failure, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. +- `SubscribeLoop()` calls `SubscribeToAllAsync` through `auth.Run` (which handles token failures and invalidation), and additionally retries any other failure with the same capped backoff, logging each, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers therefore share one attempt, and there is never more than one subscription. - `HandleDrop` (any reason except `Disposed`) clears the stored subscription under the lock and calls `EnsureSubscribed(CancellationToken.None)` without awaiting it. The loop is still bound to shutdown, and any exception other than cancellation is caught and logged inside the loop, so nothing is unobserved. - `Realtime` gets an internal constructor taking a `Func>` subscribe delegate, `GrpcAuthContext` and `TimeProvider`, so all of this is unit-testable without a server. @@ -249,21 +264,23 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - caches token; refreshes inside the refresh window; caps window at half lifetime - concurrent callers trigger exactly one token request, both when it succeeds and when it fails (all callers see the same exception) - failure cooldown: calls within 5 s of a failure make no request and rethrow; after the cooldown a new request is made - - one caller cancelling does not cancel the shared request for others + - one caller cancelling does not cancel the shared request for others; cancelling the shutdown token aborts an in-flight request + - `Invalidate()` forces the next call to refresh, still single-flight and subject to cooldown - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying `error`/`error_description` - response validation: missing/empty `access_token`; missing or non-Bearer `token_type`; `expires_in` zero, negative, fractional, malformed, numeric string, > 24 h (clamped); missing `expires_in` with and without `defaultTokenLifetimeSeconds` - 3xx response is an error and the redirect target receives no request - assertion file re-read per request - exception messages and logs contain no secret/assertion/token - `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. -- `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. +- `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. `Run`: retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. - `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. +- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. -- `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnTokenFailure`, a token failure (`OAuthTokenException`, `Unauthenticated`) from the metadata call propagates instead of returning `null`, so the event is not kept; `PermissionDenied` and other failures still return `null`; without the flag (basic-auth reader), `Unauthenticated` still returns `null` exactly as today; `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. -- `TokenFileSource`: reload interval, trimming, missing/empty file with and without previous token. +- `TokenFileSource`: `Invalidate()` forces an immediate re-read; reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. - `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) - `EnvConfigProvider` redaction: gRPC connection string with user-info, TCP connection string with `DefaultUserCredentials`, `Auth:ClientSecret`, `Auth:AdditionalParameters:*` are masked; `Auth:Type` and unrelated keys are printed. @@ -276,7 +293,8 @@ Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth p ## Risks - **Sink stall on KurrentDB outage** (pre-existing, unchanged): a sink write failure that is not a token failure still exhausts `SinkPipe` retries and stalls replication. Out of scope; noted for a follow-up. -- **Fail-closed stall on persistent token rejection** (OAuth readers only): a token the server rejects as unauthenticated stalls replication in the scavenge filter or sink with errors logged, rather than replicating incorrectly. Intended; documented in troubleshooting. +- **Persistent token rejection** (`Unauthenticated`, e.g. wrong audience): replication pauses with repeating warnings and resumes by itself once fixed. Intended; documented in troubleshooting. +- **Fail-closed stall on metadata `PermissionDenied`** (OAuth readers only): stalls replication with errors logged rather than copying events the filter cannot check; needs a restart after fixing roles. Intended; documented. - **Stream termination at token expiry** may cause restart churn on long reads; mitigated by existing restart logic and checked in the manual test. - **Sentinel credential reliance** on `GetAuthenticationHeaderValue` being honoured: it is public API on the current client; a future move to `KurrentDB.Client` must re-verify it. Covered by the end-to-end header test so an upgrade that breaks it fails CI. - **Role mapping** is the most likely customer misconfiguration (token accepted but `PermissionDenied`); the docs cover it and `PermissionDenied` errors are already logged by the reader/writer. From 8b2586a2ea60abd68abbfe1ba87cf4406898dab9 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:19:54 -0400 Subject: [PATCH 07/67] docs: address spec review round 5 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index f85d6212..23488303 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -111,7 +111,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)` and `void Invalidate()`, which marks the cached token stale so the next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)` and `void Invalidate()`. `Invalidate()` marks the current cached token as **rejected**: it is dropped from the cache and is never returned again, including by the stale-token fallbacks below, until a *different* token value is obtained. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate()`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | @@ -143,7 +143,7 @@ Response handling (RFC 6749 §5.1): - HTTP 200 with JSON body. `access_token` is required and non-empty. `token_type` is required and must equal `Bearer` (case-insensitive); any other value or absence is rejected. - `expires_in` must be a positive integer number of seconds (JSON number or numeric string, as some providers send it), at most 86 400 (24 h; larger values are clamped and logged once). Zero, negative, fractional, malformed or overflowing values are rejected. - If `expires_in` is absent, the configured `defaultTokenLifetimeSeconds` is used. If that is not configured either, the response is rejected with an error telling the operator to set it. There is no silent default. -- Non-2xx responses are errors; the OAuth `error` / `error_description` fields are included in the exception message when the body is a JSON error response. +- Non-2xx responses are errors, reported as described under "no usable token" below (status and sanitised `error` code only). - Redirects are not followed (`AllowAutoRedirect = false`). Any 3xx response is an error naming the configured endpoint, because following a 307/308 would replay the client secret or assertion to another origin. - Each request has a 30-second timeout. @@ -152,8 +152,9 @@ Caching and refresh: - Cached `(token, expiresAt)`. `expiresAt = requestStartTime + lifetime` (request start, not response time, to stay conservative). - `GetAccessToken` returns the cached token if `now < expiresAt - refreshWindow`, where `refreshWindow = min(refreshBeforeExpirySeconds, lifetime / 2)`. - Otherwise a refresh is needed. Refreshes are single-flight via a shared in-flight `Task`: the first caller starts the request and stores the task; every caller that arrives while it is running awaits the same task and observes the same result or exception. The stored task is cleared when it completes. -- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`), log a warning and return the cached token. -- If there is no usable token, throw `OAuthTokenException` (message includes HTTP status and OAuth `error` / `error_description` when present; never the request body, secret, assertion or token). +- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`) and has not been rejected via `Invalidate()`, log a warning and return the cached token. After `Invalidate()` there is no cached token to fall back to, so a failed refresh throws. +- If the token endpoint returns the same token value that was rejected (some providers cache issued tokens), it is accepted and cached again. `GrpcAuthContext.Run`'s backoff bounds how often it is retried; it recovers when the provider issues a new token or the server-side problem is fixed. +- If there is no usable token, throw `OAuthTokenException`. Its message contains only the HTTP status, the OAuth `error` code (when it is a single RFC 6749 `error` token of printable ASCII, max 64 chars; otherwise omitted) and the token endpoint host. Provider-supplied `error_description` / `error_uri` and the response body are never included in messages or logs, because a server may echo submitted credentials in them. The full `error_description` is available only through a separate opt-in: logged at Debug level after redacting every occurrence of the client secret, the client assertion and any `access_token` value seen so far (replaced by `***`). - Failure cooldown: after a failed refresh, calls within the next 5 seconds do not start a new request. They return the cached token if it is usable, otherwise rethrow the last failure immediately. This bounds the token-endpoint request rate during an outage to one per 5 seconds per side, and keeps queued callers from each waiting up to 30 seconds in turn. - `GetAccessToken` honours its `CancellationToken` for the caller's wait. The shared request runs under its own `CancellationTokenSource` linked to the application shutdown token and the 30-second timeout, never to an individual caller's token, so one caller cancelling does not fail the others, and shutdown aborts an in-flight request. @@ -162,7 +163,8 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do ### Token file flow - On first call and whenever `now - lastReadAt >= tokenFileReloadSeconds`, read the file, trim, cache. -- Empty or missing file: if a previously read token exists, log a warning and keep using it; otherwise throw `OAuthTokenException`. +- Empty or missing file: if a previously read token exists and has not been rejected via `Invalidate()`, log a warning and keep using it; otherwise throw `OAuthTokenException`. +- After `Invalidate()`, a re-read that returns the same (rejected) value is treated as "no new token yet": `GetAccessToken` throws `OAuthTokenException` ("token file still contains a token the server rejected"), so `Run` backs off instead of resending a known-bad token. Any different value is accepted. - No JWT parsing; the external process owns validity. ### Failure handling and recovery @@ -266,7 +268,9 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - failure cooldown: calls within 5 s of a failure make no request and rethrow; after the cooldown a new request is made - one caller cancelling does not cancel the shared request for others; cancelling the shutdown token aborts an in-flight request - `Invalidate()` forces the next call to refresh, still single-flight and subject to cooldown - - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying `error`/`error_description` + - after `Invalidate()`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token; the cached token before invalidation is never returned again + - an error response whose `error_description`, `error_uri` and body echo the client secret and client assertion: exception message and all logs at Info/Warning contain neither; Debug output contains them only as `***`; a non-conforming `error` value is omitted + - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying the HTTP status and `error` code - response validation: missing/empty `access_token`; missing or non-Bearer `token_type`; `expires_in` zero, negative, fractional, malformed, numeric string, > 24 h (clamped); missing `expires_in` with and without `defaultTokenLifetimeSeconds` - 3xx response is an error and the redirect target receives no request - assertion file re-read per request @@ -275,12 +279,12 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. `Run`: retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. - `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. -- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. +- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. -- `TokenFileSource`: `Invalidate()` forces an immediate re-read; reload interval, trimming, missing/empty file with and without previous token. +- `TokenFileSource`: `Invalidate()` forces an immediate re-read; after `Invalidate()`, a missing/empty file or a file still holding the rejected value throws instead of returning the rejected token, and a different value is accepted; reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. - `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) - `EnvConfigProvider` redaction: gRPC connection string with user-info, TCP connection string with `DefaultUserCredentials`, `Auth:ClientSecret`, `Auth:AdditionalParameters:*` are masked; `Auth:Type` and unrelated keys are printed. From 406f5c5649fcb2b2057581a2bd99fed4c2167578 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:21:34 -0400 Subject: [PATCH 08/67] docs: address spec review round 6 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 23488303..5875d1f2 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -111,7 +111,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)` and `void Invalidate()`. `Invalidate()` marks the current cached token as **rejected**: it is dropped from the cache and is never returned again, including by the stale-token fallbacks below, until a *different* token value is obtained. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)` and `void Invalidate()`. `Invalidate()` marks the current cached token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate()`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | @@ -152,9 +152,8 @@ Caching and refresh: - Cached `(token, expiresAt)`. `expiresAt = requestStartTime + lifetime` (request start, not response time, to stay conservative). - `GetAccessToken` returns the cached token if `now < expiresAt - refreshWindow`, where `refreshWindow = min(refreshBeforeExpirySeconds, lifetime / 2)`. - Otherwise a refresh is needed. Refreshes are single-flight via a shared in-flight `Task`: the first caller starts the request and stores the task; every caller that arrives while it is running awaits the same task and observes the same result or exception. The stored task is cleared when it completes. -- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`) and has not been rejected via `Invalidate()`, log a warning and return the cached token. After `Invalidate()` there is no cached token to fall back to, so a failed refresh throws. -- If the token endpoint returns the same token value that was rejected (some providers cache issued tokens), it is accepted and cached again. `GrpcAuthContext.Run`'s backoff bounds how often it is retried; it recovers when the provider issues a new token or the server-side problem is fixed. -- If there is no usable token, throw `OAuthTokenException`. Its message contains only the HTTP status, the OAuth `error` code (when it is a single RFC 6749 `error` token of printable ASCII, max 64 chars; otherwise omitted) and the token endpoint host. Provider-supplied `error_description` / `error_uri` and the response body are never included in messages or logs, because a server may echo submitted credentials in them. The full `error_description` is available only through a separate opt-in: logged at Debug level after redacting every occurrence of the client secret, the client assertion and any `access_token` value seen so far (replaced by `***`). +- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`), log a warning and return the cached token. After `Invalidate()` the token is no longer cached, so there is nothing to fall back to and a failed refresh throws. +- If there is no usable token, throw `OAuthTokenException`. Its message contains only the HTTP status, the token endpoint host and the OAuth `error` code **only if it is in a fixed allowlist** (`invalid_request`, `invalid_client`, `invalid_grant`, `unauthorized_client`, `unsupported_grant_type`, `invalid_scope`, `server_error`, `temporarily_unavailable`); any other value is reported as `non-standard error (omitted)`. Integer values of Entra ID's `error_codes` array (e.g. `7000215`) are included, since integers cannot carry credentials. Provider-supplied `error_description` / `error_uri` and the response body are never included in messages or logs, because a server may echo submitted credentials in them. The full `error_description` is available only through a separate opt-in: logged at Debug level after redacting every occurrence of the client secret, the client assertion and any `access_token` value seen so far (replaced by `***`). - Failure cooldown: after a failed refresh, calls within the next 5 seconds do not start a new request. They return the cached token if it is usable, otherwise rethrow the last failure immediately. This bounds the token-endpoint request rate during an outage to one per 5 seconds per side, and keeps queued callers from each waiting up to 30 seconds in turn. - `GetAccessToken` honours its `CancellationToken` for the caller's wait. The shared request runs under its own `CancellationTokenSource` linked to the application shutdown token and the 30-second timeout, never to an individual caller's token, so one caller cancelling does not fail the others, and shutdown aborts an in-flight request. @@ -164,7 +163,16 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do - On first call and whenever `now - lastReadAt >= tokenFileReloadSeconds`, read the file, trim, cache. - Empty or missing file: if a previously read token exists and has not been rejected via `Invalidate()`, log a warning and keep using it; otherwise throw `OAuthTokenException`. -- After `Invalidate()`, a re-read that returns the same (rejected) value is treated as "no new token yet": `GetAccessToken` throws `OAuthTokenException` ("token file still contains a token the server rejected"), so `Run` backs off instead of resending a known-bad token. Any different value is accepted. +- After `Invalidate()`, a re-read that returns the rejected value is subject to the rejected-token quarantine below. + +### Rejected tokens + +Shared by both sources, so behaviour is the same for client credentials (where some providers re-issue the same cached token) and token files (where the file may not have rotated yet): + +- `Invalidate()` records the rejected token value and the time, `rejectedAt`. +- For 60 seconds after `rejectedAt` (the quarantine), a newly acquired value equal to the rejected one is treated as "no new token yet": `GetAccessToken` throws `OAuthTokenException` ("the token source returned a token the server just rejected"), and `Run` backs off instead of resending a known-bad token. The stale-token fallbacks never return the rejected value during the quarantine either. +- After the quarantine the same value may be returned again. This is a bounded probe: if the server problem was fixed (for example its OAuth configuration), the probe succeeds and replication resumes without waiting for rotation. If it is rejected again, `Invalidate()` starts a new quarantine. The rejected value is therefore sent at most once per minute per side. +- Any different value is accepted immediately and clears the rejection record. - No JWT parsing; the external process owns validity. ### Failure handling and recovery @@ -268,7 +276,9 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - failure cooldown: calls within 5 s of a failure make no request and rethrow; after the cooldown a new request is made - one caller cancelling does not cancel the shared request for others; cancelling the shutdown token aborts an in-flight request - `Invalidate()` forces the next call to refresh, still single-flight and subject to cooldown - - after `Invalidate()`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token; the cached token before invalidation is never returned again + - after `Invalidate()`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token + - quarantine: the endpoint re-issuing the rejected value within 60 s throws; after 60 s the same value is returned (probe); a different value is returned immediately + - `error` equal to the submitted client secret is not in the allowlist and is reported as `non-standard error (omitted)`; allowlisted codes and integer `error_codes` are reported - an error response whose `error_description`, `error_uri` and body echo the client secret and client assertion: exception message and all logs at Info/Warning contain neither; Debug output contains them only as `***`; a non-conforming `error` value is omitted - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying the HTTP status and `error` code - response validation: missing/empty `access_token`; missing or non-Bearer `token_type`; `expires_in` zero, negative, fractional, malformed, numeric string, > 24 h (clamped); missing `expires_in` with and without `defaultTokenLifetimeSeconds` @@ -279,12 +289,12 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. `Run`: retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. - `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. -- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. +- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. A third case leaves A in the file and switches the stub to accept A after the rejection (a server-side fix): after the 60 s quarantine (advanced with `FakeTimeProvider`) A is sent once more and the write succeeds. - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. -- `TokenFileSource`: `Invalidate()` forces an immediate re-read; after `Invalidate()`, a missing/empty file or a file still holding the rejected value throws instead of returning the rejected token, and a different value is accepted; reload interval, trimming, missing/empty file with and without previous token. +- `TokenFileSource`: `Invalidate()` forces an immediate re-read; after `Invalidate()`, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. - `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) - `EnvConfigProvider` redaction: gRPC connection string with user-info, TCP connection string with `DefaultUserCredentials`, `Auth:ClientSecret`, `Auth:AdditionalParameters:*` are masked; `Auth:Type` and unrelated keys are printed. @@ -297,7 +307,7 @@ Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth p ## Risks - **Sink stall on KurrentDB outage** (pre-existing, unchanged): a sink write failure that is not a token failure still exhausts `SinkPipe` retries and stalls replication. Out of scope; noted for a follow-up. -- **Persistent token rejection** (`Unauthenticated`, e.g. wrong audience): replication pauses with repeating warnings and resumes by itself once fixed. Intended; documented in troubleshooting. +- **Persistent token rejection** (`Unauthenticated`, e.g. wrong audience): replication pauses with repeating warnings and resumes by itself once fixed, whether the fix is a new token or a server-side change (detected by the once-a-minute probe). Intended; documented in troubleshooting. - **Fail-closed stall on metadata `PermissionDenied`** (OAuth readers only): stalls replication with errors logged rather than copying events the filter cannot check; needs a restart after fixing roles. Intended; documented. - **Stream termination at token expiry** may cause restart churn on long reads; mitigated by existing restart logic and checked in the manual test. - **Sentinel credential reliance** on `GetAuthenticationHeaderValue` being honoured: it is public API on the current client; a future move to `KurrentDB.Client` must re-verify it. Covered by the end-to-end header test so an upgrade that breaks it fails CI. From cf6b198ad914ca9b84e9b570ade807392a08e57f Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:22:44 -0400 Subject: [PATCH 09/67] docs: address spec review round 7 Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 5875d1f2..50c1367e 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -111,7 +111,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)` and `void Invalidate()`. `Invalidate()` marks the current cached token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate()` and `void ReportAccepted(string token)` (called by `Run` after a successful call; ends a rejected-token probe). `Invalidate()` marks the current cached token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate()`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | @@ -171,7 +171,12 @@ Shared by both sources, so behaviour is the same for client credentials (where s - `Invalidate()` records the rejected token value and the time, `rejectedAt`. - For 60 seconds after `rejectedAt` (the quarantine), a newly acquired value equal to the rejected one is treated as "no new token yet": `GetAccessToken` throws `OAuthTokenException` ("the token source returned a token the server just rejected"), and `Run` backs off instead of resending a known-bad token. The stale-token fallbacks never return the rejected value during the quarantine either. -- After the quarantine the same value may be returned again. This is a bounded probe: if the server problem was fixed (for example its OAuth configuration), the probe succeeds and replication resumes without waiting for rotation. If it is rejected again, `Invalidate()` starts a new quarantine. The rejected value is therefore sent at most once per minute per side. +- After the quarantine the same value may be used again by **one probe at a time**. The first `GetAccessToken` after the quarantine takes a per-source probe lease and receives the value; every other caller keeps getting the quarantine `OAuthTokenException` (and backs off in `Run`) while the lease is held. The lease ends when: + - the probing call succeeds: `Run` calls `source.ReportAccepted(token)`, which clears the rejection record so all callers get the token again (server-side fix detected without rotation); + - the probing call is rejected: `Invalidate()` starts a new 60 s quarantine; + - or after 60 s with neither, e.g. the prober was cancelled or the long-running `ReadAllAsync` took the lease (it does not go through `Run`). Expiry then counts as a new quarantine. + The rejected value is therefore sent by at most one call per minute per side, even with concurrent writers, the prepare pipe's concurrent filter calls and the metrics reporter. +- Since the header hook fetches through the same `GetAccessToken`, a gRPC call whose hook hits the quarantine fails as a token failure and is retried by `Run`, so it cannot bypass the lease. - Any different value is accepted immediately and clears the rejection record. - No JWT parsing; the external process owns validity. @@ -278,6 +283,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - `Invalidate()` forces the next call to refresh, still single-flight and subject to cooldown - after `Invalidate()`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token - quarantine: the endpoint re-issuing the rejected value within 60 s throws; after 60 s the same value is returned (probe); a different value is returned immediately + - probe lease: with 10 concurrent callers at the quarantine boundary exactly one receives the rejected value and the rest throw; `ReportAccepted` releases the token to all; `Invalidate` during the probe restarts quarantine; an unreported lease expires after 60 s - `error` equal to the submitted client secret is not in the allowlist and is reported as `non-standard error (omitted)`; allowlisted codes and integer `error_codes` are reported - an error response whose `error_description`, `error_uri` and body echo the client secret and client assertion: exception message and all logs at Info/Warning contain neither; Debug output contains them only as `***`; a non-conforming `error` value is omitted - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying the HTTP status and `error` code @@ -286,7 +292,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - assertion file re-read per request - exception messages and logs contain no secret/assertion/token - `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. -- `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. `Run`: retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. +- `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. `Run`: calls `ReportAccepted` after success; retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. - `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. - Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. A third case leaves A in the file and switches the stub to accept A after the rejection (a server-side fix): after the 60 s quarantine (advanced with `FakeTimeProvider`) A is sent once more and the write succeeds. From dd5b004bc187aabd42e3dd5be390d4da71e404ba Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:24:33 -0400 Subject: [PATCH 10/67] docs: address spec review round 8 (per-call credentials) Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 68 +++++++++++-------- 1 file changed, 40 insertions(+), 28 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 50c1367e..0adea4ee 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -36,7 +36,14 @@ Verified by decompiling `EventStore.Client` 23.3.8 (`EventStoreCallOptions`, `Ev - That interceptor sets the `authorization` header to the result of `await settings.OperationOptions.GetAuthenticationHeaderValue(credentials, ct)`. The default implementation returns `credentials.ToString()` (`Basic …` or `Bearer …`). - `GetAuthenticationHeaderValue` is a public, settable `Func>`, evaluated **per call, asynchronously**. -This is the integration seam: set `DefaultCredentials` to a sentinel bearer credential (so the client attaches call credentials) and replace `GetAuthenticationHeaderValue` with a function that returns `"Bearer " + await tokenSource.GetAccessToken(ct)`. No gRPC interceptor, custom `HttpMessageHandler`, or change to the reader/writer code is needed, and the client's TLS handling (`tls`, `tlsVerifyCert`, `tlsCaFile`) stays untouched. +Every client method Replicator uses (`AppendToStreamAsync`, `DeleteAsync`, `SetStreamMetadataAsync`, `GetStreamMetadataAsync`, `ReadStreamAsync`, `ReadAllAsync`, `SubscribeToAllAsync`) also takes an optional per-call `userCredentials`, which takes precedence over `DefaultCredentials`. + +This is the integration seam, in two layers: + +- **Per-call token (primary).** Replicator acquires the token itself before each call and passes `userCredentials: new UserCredentials(token)` (a bearer credential). The header then carries exactly that token, and the code that made the call knows which token was sent. That is what lets it report acceptance or rejection of a specific token value. +- **Fallback hook.** `DefaultCredentials` is set to a sentinel bearer credential and `GetAuthenticationHeaderValue` is replaced. For the sentinel it returns `"Bearer " + await tokenSource.GetAccessToken(None)`. For any other credential it returns `credentials.ToString()` unchanged. This only covers a call site that forgets to pass per-call credentials, so the request is never sent unauthenticated. + +No gRPC interceptor or custom `HttpMessageHandler` is needed, and the client's TLS handling (`tls`, `tlsVerifyCert`, `tlsCaFile`) stays untouched. `CallCredentials` are only sent over TLS channels by Grpc.Net.Client. OAuth therefore requires `tls=true` (the default); see Validation. @@ -111,13 +118,13 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate()` and `void ReportAccepted(string token)` (called by `Run` after a successful call; ends a rejected-token probe). `Invalidate()` marks the current cached token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(string token)` and `void ReportAccepted(string token)`. The latter two name the exact token value a call sent. `Invalidate` for a value that is no longer current (a newer token was already obtained) is ignored. `ReportAccepted` ends a rejected-token probe. `Invalidate(token)` marks that token as **rejected** and drops it from the cache if it is the cached one. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | -| `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate()`. Takes a `TimeProvider`. | +| `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | -| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `EnsureToken(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token (no-op without OAuth); `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"); without OAuth it just invokes `call`. | +| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `ValueTask AcquireCredentials(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token and returns `new UserCredentials(token)` (returns `null` without OAuth, so the client keeps using the connection string's credentials exactly as today); `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"). Every call site passes the credentials it is handed as the client method's `userCredentials` argument. | | `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is not a token failure (retrying with a new token does not help), and is handled separately where it matters (scavenge filter). | -| `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue`. Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | +| `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue` with the fallback hook (sentinel → fetch from source; anything else → `credentials.ToString()`). Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | ### Wiring @@ -152,7 +159,7 @@ Caching and refresh: - Cached `(token, expiresAt)`. `expiresAt = requestStartTime + lifetime` (request start, not response time, to stay conservative). - `GetAccessToken` returns the cached token if `now < expiresAt - refreshWindow`, where `refreshWindow = min(refreshBeforeExpirySeconds, lifetime / 2)`. - Otherwise a refresh is needed. Refreshes are single-flight via a shared in-flight `Task`: the first caller starts the request and stores the task; every caller that arrives while it is running awaits the same task and observes the same result or exception. The stored task is cleared when it completes. -- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`), log a warning and return the cached token. After `Invalidate()` the token is no longer cached, so there is nothing to fall back to and a failed refresh throws. +- If a refresh fails while the cached token is still usable (`now < expiresAt - 30s`), log a warning and return the cached token. After `Invalidate` the token is no longer cached, so there is nothing to fall back to and a failed refresh throws. - If there is no usable token, throw `OAuthTokenException`. Its message contains only the HTTP status, the token endpoint host and the OAuth `error` code **only if it is in a fixed allowlist** (`invalid_request`, `invalid_client`, `invalid_grant`, `unauthorized_client`, `unsupported_grant_type`, `invalid_scope`, `server_error`, `temporarily_unavailable`); any other value is reported as `non-standard error (omitted)`. Integer values of Entra ID's `error_codes` array (e.g. `7000215`) are included, since integers cannot carry credentials. Provider-supplied `error_description` / `error_uri` and the response body are never included in messages or logs, because a server may echo submitted credentials in them. The full `error_description` is available only through a separate opt-in: logged at Debug level after redacting every occurrence of the client secret, the client assertion and any `access_token` value seen so far (replaced by `***`). - Failure cooldown: after a failed refresh, calls within the next 5 seconds do not start a new request. They return the cached token if it is usable, otherwise rethrow the last failure immediately. This bounds the token-endpoint request rate during an outage to one per 5 seconds per side, and keeps queued callers from each waiting up to 30 seconds in turn. - `GetAccessToken` honours its `CancellationToken` for the caller's wait. The shared request runs under its own `CancellationTokenSource` linked to the application shutdown token and the 30-second timeout, never to an individual caller's token, so one caller cancelling does not fail the others, and shutdown aborts an in-flight request. @@ -162,27 +169,27 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do ### Token file flow - On first call and whenever `now - lastReadAt >= tokenFileReloadSeconds`, read the file, trim, cache. -- Empty or missing file: if a previously read token exists and has not been rejected via `Invalidate()`, log a warning and keep using it; otherwise throw `OAuthTokenException`. -- After `Invalidate()`, a re-read that returns the rejected value is subject to the rejected-token quarantine below. +- Empty or missing file: if a previously read token exists and has not been rejected via `Invalidate`, log a warning and keep using it; otherwise throw `OAuthTokenException`. +- After `Invalidate`, a re-read that returns the rejected value is subject to the rejected-token quarantine below. ### Rejected tokens Shared by both sources, so behaviour is the same for client credentials (where some providers re-issue the same cached token) and token files (where the file may not have rotated yet): -- `Invalidate()` records the rejected token value and the time, `rejectedAt`. +- `Invalidate(token)` records the rejected token value and the time, `rejectedAt`. - For 60 seconds after `rejectedAt` (the quarantine), a newly acquired value equal to the rejected one is treated as "no new token yet": `GetAccessToken` throws `OAuthTokenException` ("the token source returned a token the server just rejected"), and `Run` backs off instead of resending a known-bad token. The stale-token fallbacks never return the rejected value during the quarantine either. - After the quarantine the same value may be used again by **one probe at a time**. The first `GetAccessToken` after the quarantine takes a per-source probe lease and receives the value; every other caller keeps getting the quarantine `OAuthTokenException` (and backs off in `Run`) while the lease is held. The lease ends when: - the probing call succeeds: `Run` calls `source.ReportAccepted(token)`, which clears the rejection record so all callers get the token again (server-side fix detected without rotation); - - the probing call is rejected: `Invalidate()` starts a new 60 s quarantine; - - or after 60 s with neither, e.g. the prober was cancelled or the long-running `ReadAllAsync` took the lease (it does not go through `Run`). Expiry then counts as a new quarantine. + - the probing call is rejected: `Invalidate(token)` starts a new 60 s quarantine; + - or after 60 s with neither, e.g. the prober was cancelled before sending. Expiry then counts as a new quarantine. The rejected value is therefore sent by at most one call per minute per side, even with concurrent writers, the prepare pipe's concurrent filter calls and the metrics reporter. -- Since the header hook fetches through the same `GetAccessToken`, a gRPC call whose hook hits the quarantine fails as a token failure and is retried by `Run`, so it cannot bypass the lease. +- The token is handed to the call that holds the lease and travels in that call's `userCredentials`; nothing else calls `GetAccessToken` for it. The sentinel fallback hook goes through `GetAccessToken` like any other caller, so even a forgotten call site cannot bypass the lease. - Any different value is accepted immediately and clears the rejection record. - No JWT parsing; the external process owns validity. ### Failure handling and recovery -`GetAuthenticationHeaderValue` is always invoked by the client with `CancellationToken.None` (seen in the decompiled `EventStoreCallOptions`), so a token fetch inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires tokens *before* every gRPC call Replicator makes, and treats the header hook as a cache read in the normal case. The hook still fetches if the cache is empty (a narrow race), and a failure there fails that call like any other token failure below. +`GetAuthenticationHeaderValue` is always invoked by the client with `CancellationToken.None` (seen in the decompiled `EventStoreCallOptions`), so a token fetch inside the gRPC call path cannot be cancelled by the caller. The design therefore acquires the token *before* every gRPC call, cancellably, and passes it as per-call `userCredentials`. The header hook then just formats that credential and never fetches. Only the sentinel fallback fetches, and no Replicator call site relies on it. Two kinds of token failure are handled the same way: @@ -193,10 +200,13 @@ Both are usually transient, so neither may permanently stop replication. Every w **Token-aware calls.** `GrpcAuthContext.Run(call, ct)` is used for every bounded (non-streaming-for-the-life-of-the-run) gRPC operation: appends, deletes, metadata writes, metadata reads, stream-size reads, `GetLastPosition` and subscribe. It loops: -1. `EnsureToken(ct)`: waits for a token with capped backoff (1 s, 2 s, 4 s … 30 s), cancellable by `ct` and shutdown. -2. Invoke `call`. -3. If the call throws and `AuthFailure.IsTokenFailure(e)` is true: `Invalidate()` the token source, log a warning (first failure, then at most once a minute; recovery is logged once), wait the next backoff step, and go to 1. -4. Any other exception propagates unchanged, so existing behaviour for non-auth errors is preserved. +1. `creds = AcquireCredentials(ct)`: waits for a token with capped backoff (1 s, 2 s, 4 s … 30 s), cancellable by `ct` and shutdown. Remember the token value `t` it carries. +2. Invoke `call(creds, ct)`; the call passes `creds` as `userCredentials`, so the request header is exactly `Bearer t`. +3. On success: `source.ReportAccepted(t)`. +4. If the call throws and `AuthFailure.IsTokenFailure(e)` is true: `source.Invalidate(t)`, log a warning (first failure, then at most once a minute; recovery is logged once), wait the next backoff step, and go to 1. +5. Any other exception propagates unchanged, so existing behaviour for non-auth errors is preserved. + +Because the token travels with the call, the probe lease (see "Rejected tokens") is consumed by exactly the call that sends the probe, and acceptance or rejection is attributed to the exact value sent. The loop has no attempt limit: a persistent rejection (for example a wrong audience in KurrentDB's OAuth config) shows up as a repeating warning and stalled progress, and replication resumes on its own once the token or the server config is fixed. Retrying is safe: a call rejected as `Unauthenticated` was not executed by the server. @@ -208,13 +218,13 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - `Replicator`'s post-read drain loop (`while (sinkChannel.Reader.Count > 0)`) would then spin forever because nothing consumes the channel. It is changed to also exit when the writer task has completed, logging how many events were left unwritten; they are re-read from the last checkpoint on the next start (the checkpoint is only stored after a successful write, in `SinkPipe`). - The prepare shovel can also be blocked: `PreparePipe` pushes into the bounded sink channel via `sinkChannel.Writer.WriteAsync(ctx, ctx.CancellationToken)`, and that token is `CancellationToken.None`. If the sink channel filled while the writer waited, the prepare shovel stays blocked after the writer stops, and `Replicator` awaits `prepareTask` forever. `Replicator` is changed to create `writerCts` before the pipes and pass `writerCts.Token` to that channel write instead of `ctx.CancellationToken`. `Stop()` already cancels `writerCts`, so the blocked write throws `OperationCanceledException`, which `PreparePipe` already swallows, and the prepare shovel exits through its own (already cancelled) token. In a normal shutdown with a healthy writer nothing changes: the channel drains before `writerCts` is cancelled. -**Reader.** `GrpcEventReader` calls `auth.EnsureToken(cancellationToken)` before starting the long-running `ReadAllAsync`. A token failure during that read (at start, or if the server ends the stream) fails the enumeration. `ReadEvents` catches it, calls `Invalidate()` if it is a token failure, and rethrows. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, going through `EnsureToken` again with a fresh token. No pipeline change is needed. `GetLastPosition` (a bounded backwards read) goes through `auth.Run`. +**Reader.** `GrpcEventReader` calls `auth.AcquireCredentials(cancellationToken)` before starting the long-running `ReadAllAsync` and passes them as `userCredentials`. Once the first event (or end of stream) is received, it calls `ReportAccepted(t)`. A token failure during that read (at start, or if the server ends the stream) fails the enumeration. `ReadEvents` catches it, calls `Invalidate(t)` if it is a token failure, and rethrows. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, acquiring credentials again. No pipeline change is needed. `GetLastPosition` (a bounded backwards read) goes through `auth.Run`. **Metrics reporter.** `Replicator.Report` calls `reader.GetLastPosition` every `ReportMetricsFrequencyInSeconds` and catches only `OperationCanceledException`, so any other exception ends the reporter for the rest of the run. `GetLastPosition` now waits out token failures. In addition, `Report` is changed to catch and log (warning, rate-limited to once a minute) any other exception per iteration and continue, so a transient failure, auth or not, no longer stops metrics permanently. **Reader auxiliary calls (scavenge filter).** `ScavengedEventsFilter` issues `GetStreamMetadataAsync` and `ReadStreamAsync` calls during the read. Today `StreamMetaCache.GetOrAddStreamMeta` turns *any* exception into `null`, and the filter treats `null` as "keep the event". If an auth problem took that path with `scavenge: true` (the Helm default), deleted or expired events would be replicated. -- Both auxiliary calls go through `auth.Run(..., CancellationToken.None)`; the shutdown token linked inside the context makes them cancellable. Token failures are therefore waited out and retried inside the call and never reach `StreamMetaCache` (except as `OperationCanceledException` at shutdown). +- Both auxiliary calls go through `auth.Run(...)` with `CancellationToken.None` as the caller token; the shutdown token linked inside the context makes them cancellable. Token failures are therefore waited out and retried inside the call and never reach `StreamMetaCache` (except as `OperationCanceledException` at shutdown). - For OAuth readers, `StreamMetaCache.GetOrAddStreamMeta` must never turn an auth problem into "keep the event". It gets a constructor flag `failClosedOnAuthErrors`, set only for OAuth readers. With the flag, an exception where `IsTokenFailure(e)` is true, or an `RpcException` with `PermissionDenied`, is rethrown instead of returning `null`. A `PermissionDenied` on stream metadata means the token's role cannot see that stream's metadata, so the filter cannot decide safely. The prepare step then fails: `PreparePipe` retries 10 times, then the prepare shovel faults and replication stalls with the error logged. That is fail-closed, and it is a configuration error (the replicator identity must be mapped to `$admins` or have metadata read on all streams) that the docs call out. Restart Replicator after fixing roles. - Without the flag (`connectionString` readers: basic auth or anonymous), today's warn-and-keep behaviour is unchanged for every exception, so existing deployments behave exactly as before. - `GetOrAddStreamSize` already propagates exceptions, so it fails closed through the same path. @@ -280,10 +290,10 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - concurrent callers trigger exactly one token request, both when it succeeds and when it fails (all callers see the same exception) - failure cooldown: calls within 5 s of a failure make no request and rethrow; after the cooldown a new request is made - one caller cancelling does not cancel the shared request for others; cancelling the shutdown token aborts an in-flight request - - `Invalidate()` forces the next call to refresh, still single-flight and subject to cooldown - - after `Invalidate()`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token + - `Invalidate(t)` forces the next call to refresh, still single-flight and subject to cooldown + - after `Invalidate(t)`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token - quarantine: the endpoint re-issuing the rejected value within 60 s throws; after 60 s the same value is returned (probe); a different value is returned immediately - - probe lease: with 10 concurrent callers at the quarantine boundary exactly one receives the rejected value and the rest throw; `ReportAccepted` releases the token to all; `Invalidate` during the probe restarts quarantine; an unreported lease expires after 60 s + - probe lease: with 10 concurrent callers at the quarantine boundary exactly one receives the rejected value and the rest throw; `ReportAccepted` releases the token to all; `Invalidate` during the probe restarts quarantine; `Invalidate` of a stale value (newer token already cached) is ignored; an unreported lease expires after 60 s - `error` equal to the submitted client secret is not in the allowlist and is reported as `non-standard error (omitted)`; allowlisted codes and integer `error_codes` are reported - an error response whose `error_description`, `error_uri` and body echo the client secret and client assertion: exception message and all logs at Info/Warning contain neither; Debug output contains them only as `***`; a non-conforming `error` value is omitted - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying the HTTP status and `error` code @@ -292,17 +302,19 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - assertion file re-read per request - exception messages and logs contain no secret/assertion/token - `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. -- `GrpcAuthContext`: no-op for `None`; cancelled by either the caller token or the shutdown token. `Run`: calls `ReportAccepted` after success; retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. +- `GrpcAuthContext`: `AcquireCredentials` returns `null` for `None`; cancelled by either the caller token or the shutdown token. `Run`: calls `ReportAccepted(t)` after success with the exact value sent; retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. - `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. -- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate()` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. A third case leaves A in the file and switches the stub to accept A after the rejection (a server-side fix): after the 60 s quarantine (advanced with `FakeTimeProvider`) A is sent once more and the write succeeds. -- Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that gates each write on `GrpcAuthContext.EnsureToken` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. +- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate(A)` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. A third case leaves A in the file and switches the stub to accept A after the rejection (a server-side fix): after the 60 s quarantine (advanced with `FakeTimeProvider`) A is sent once more and the write succeeds. +- Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that runs each write through `GrpcAuthContext.Run` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. -- `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; `EnsureToken` is awaited before each auxiliary call, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; each auxiliary call acquires credentials through `auth.Run`, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. -- `TokenFileSource`: `Invalidate()` forces an immediate re-read; after `Invalidate()`, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. +- `TokenFileSource`: `Invalidate(t)` forces an immediate re-read; after it, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. -- `GrpcAuthentication.Apply` end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) +- Per-call credentials through the real client: `GrpcEventWriter` (append, delete, set-metadata), `GrpcEventReader.GetLastPosition`, `ScavengedEventsFilter` metadata and stream-size reads, and the start of `ReadAllAsync` each send `authorization: Bearer `; a test hook that asserts the sentinel fallback is never invoked on these paths. +- Probe end-to-end: quarantine token A, stub that rejects A, then switches to accepting A. After 60 s (FakeTimeProvider), with 5 concurrent writes queued: exactly one HTTP request carries `Bearer A`, it succeeds, `ReportAccepted` releases A, and the other writes then send `Bearer A` and succeed. +- `GrpcAuthentication.Apply` fallback end-to-end through the real client: create an `EventStoreClient` for `esdb://localhost:2113?tls=true` with `CreateHttpMessageHandler` set to a capturing stub handler, invoke `AppendToStreamAsync` and `ReadAllAsync`, assert the captured request carries `authorization: Bearer ` and that a token change is reflected on the next call. (The stub returns a gRPC error response; the test only inspects the request.) - `EnvConfigProvider` redaction: gRPC connection string with user-info, TCP connection string with `DefaultUserCredentials`, `Auth:ClientSecret`, `Auth:AdditionalParameters:*` are masked; `Auth:Type` and unrelated keys are printed. - Configuration binding: YAML with reader basic + sink OAuth binds to independent `GrpcAuthOptions`, and vice versa. From 67e2e0b86297e23839699788039ba338502da5f3 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:24:40 -0400 Subject: [PATCH 11/67] docs: invalidate subscription token on auth drop Co-Authored-By: Claude Opus 5.5 --- docs/superpowers/specs/2026-10-03-oauth-authentication-design.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 0adea4ee..9a2f926e 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -235,6 +235,7 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if a subscription is active, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. - `SubscribeLoop()` calls `SubscribeToAllAsync` through `auth.Run` (which handles token failures and invalidation), and additionally retries any other failure with the same capped backoff, logging each, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers therefore share one attempt, and there is never more than one subscription. +- The token used for the active subscription is stored with it. `HandleDrop` with a token-failure exception calls `Invalidate(t)` for that token first. - `HandleDrop` (any reason except `Disposed`) clears the stored subscription under the lock and calls `EnsureSubscribed(CancellationToken.None)` without awaiting it. The loop is still bound to shutdown, and any exception other than cancellation is caught and logged inside the loop, so nothing is unobserved. - `Realtime` gets an internal constructor taking a `Func>` subscribe delegate, `GrpcAuthContext` and `TimeProvider`, so all of this is unit-testable without a server. From ccab7ca9bac4c78d1eb857efabed1167ac69643e Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:27:33 -0400 Subject: [PATCH 12/67] docs: address spec review round 9 (lease generations) Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 37 ++++++++++--------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 9a2f926e..f64951e5 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -118,11 +118,11 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(string token)` and `void ReportAccepted(string token)`. The latter two name the exact token value a call sent. `Invalidate` for a value that is no longer current (a newer token was already obtained) is ignored. `ReportAccepted` ends a rejected-token probe. `Invalidate(token)` marks that token as **rejected** and drops it from the cache if it is the cached one. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(AccessTokenLease lease)` and `void ReportAccepted(AccessTokenLease lease)`. `AccessTokenLease` is `(string Value, long Generation)`. The source increments `Generation` on every state change: a new token acquired or file re-read, a quarantine started, a probe lease granted. A report is applied only if `lease.Generation` equals the current generation; otherwise it is stale (it belongs to an earlier acquisition or probe, even if the token value is identical) and is ignored, logged at Debug. `ReportAccepted` ends a rejected-token probe. `Invalidate(lease)` (if current) marks that token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | -| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `ValueTask AcquireCredentials(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token and returns `new UserCredentials(token)` (returns `null` without OAuth, so the client keeps using the connection string's credentials exactly as today); `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"). Every call site passes the credentials it is handed as the client method's `userCredentials` argument. | +| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `ValueTask AcquireCredentials(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token and returns `CallAuth(UserCredentials? Credentials, AccessTokenLease? Lease)` (both `null` without OAuth, so the client keeps using the connection string's credentials exactly as today), plus `ReportAccepted(CallAuth)` / `ReportFailure(CallAuth, Exception)` for long-lived calls that cannot use `Run`; `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"). Every call site passes the credentials it is handed as the client method's `userCredentials` argument. | | `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is not a token failure (retrying with a new token does not help), and is handled separately where it matters (scavenge filter). | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue` with the fallback hook (sentinel → fetch from source; anything else → `credentials.ToString()`). Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | @@ -176,11 +176,11 @@ Tokens are lazily fetched on first use, so a token-endpoint outage at startup do Shared by both sources, so behaviour is the same for client credentials (where some providers re-issue the same cached token) and token files (where the file may not have rotated yet): -- `Invalidate(token)` records the rejected token value and the time, `rejectedAt`. +- `Invalidate(lease)` (if current) records the rejected token value and the time `rejectedAt`, and bumps the generation. - For 60 seconds after `rejectedAt` (the quarantine), a newly acquired value equal to the rejected one is treated as "no new token yet": `GetAccessToken` throws `OAuthTokenException` ("the token source returned a token the server just rejected"), and `Run` backs off instead of resending a known-bad token. The stale-token fallbacks never return the rejected value during the quarantine either. - After the quarantine the same value may be used again by **one probe at a time**. The first `GetAccessToken` after the quarantine takes a per-source probe lease and receives the value; every other caller keeps getting the quarantine `OAuthTokenException` (and backs off in `Run`) while the lease is held. The lease ends when: - - the probing call succeeds: `Run` calls `source.ReportAccepted(token)`, which clears the rejection record so all callers get the token again (server-side fix detected without rotation); - - the probing call is rejected: `Invalidate(token)` starts a new 60 s quarantine; + - the probing call succeeds: `Run` calls `source.ReportAccepted(lease)` with the probe's lease (the lease grant bumped the generation, so only the probe's own report matches), which clears the rejection record so all callers get the token again (server-side fix detected without rotation); + - the probing call is rejected: `Invalidate(lease)` starts a new 60 s quarantine; - or after 60 s with neither, e.g. the prober was cancelled before sending. Expiry then counts as a new quarantine. The rejected value is therefore sent by at most one call per minute per side, even with concurrent writers, the prepare pipe's concurrent filter calls and the metrics reporter. - The token is handed to the call that holds the lease and travels in that call's `userCredentials`; nothing else calls `GetAccessToken` for it. The sentinel fallback hook goes through `GetAccessToken` like any other caller, so even a forgotten call site cannot bypass the lease. @@ -200,13 +200,13 @@ Both are usually transient, so neither may permanently stop replication. Every w **Token-aware calls.** `GrpcAuthContext.Run(call, ct)` is used for every bounded (non-streaming-for-the-life-of-the-run) gRPC operation: appends, deletes, metadata writes, metadata reads, stream-size reads, `GetLastPosition` and subscribe. It loops: -1. `creds = AcquireCredentials(ct)`: waits for a token with capped backoff (1 s, 2 s, 4 s … 30 s), cancellable by `ct` and shutdown. Remember the token value `t` it carries. -2. Invoke `call(creds, ct)`; the call passes `creds` as `userCredentials`, so the request header is exactly `Bearer t`. -3. On success: `source.ReportAccepted(t)`. -4. If the call throws and `AuthFailure.IsTokenFailure(e)` is true: `source.Invalidate(t)`, log a warning (first failure, then at most once a minute; recovery is logged once), wait the next backoff step, and go to 1. +1. `lease = source.GetAccessToken` via `TokenGate` (capped backoff 1 s, 2 s, 4 s … 30 s, cancellable by `ct` and shutdown). `creds = new UserCredentials(lease.Value)`. +2. Invoke `call(creds, ct)`; the call passes `creds` as `userCredentials`, so the request header is exactly `Bearer `. +3. On success: `source.ReportAccepted(lease)`. +4. If the call throws and `AuthFailure.IsTokenFailure(e)` is true: `source.Invalidate(lease)`, log a warning (first failure, then at most once a minute; recovery is logged once), wait the next backoff step, and go to 1. 5. Any other exception propagates unchanged, so existing behaviour for non-auth errors is preserved. -Because the token travels with the call, the probe lease (see "Rejected tokens") is consumed by exactly the call that sends the probe, and acceptance or rejection is attributed to the exact value sent. +Because the token travels with the call and results are reported with the lease that call received, the probe lease (see "Rejected tokens") is consumed by exactly the call that sends the probe. Acceptance or rejection is attributed to the exact acquisition it belongs to. A late result from an older call, even one that carried the same token value, cannot clear a newer quarantine or reject a newer accepted probe. The loop has no attempt limit: a persistent rejection (for example a wrong audience in KurrentDB's OAuth config) shows up as a repeating warning and stalled progress, and replication resumes on its own once the token or the server config is fixed. Retrying is safe: a call rejected as `Unauthenticated` was not executed by the server. @@ -218,7 +218,7 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - `Replicator`'s post-read drain loop (`while (sinkChannel.Reader.Count > 0)`) would then spin forever because nothing consumes the channel. It is changed to also exit when the writer task has completed, logging how many events were left unwritten; they are re-read from the last checkpoint on the next start (the checkpoint is only stored after a successful write, in `SinkPipe`). - The prepare shovel can also be blocked: `PreparePipe` pushes into the bounded sink channel via `sinkChannel.Writer.WriteAsync(ctx, ctx.CancellationToken)`, and that token is `CancellationToken.None`. If the sink channel filled while the writer waited, the prepare shovel stays blocked after the writer stops, and `Replicator` awaits `prepareTask` forever. `Replicator` is changed to create `writerCts` before the pipes and pass `writerCts.Token` to that channel write instead of `ctx.CancellationToken`. `Stop()` already cancels `writerCts`, so the blocked write throws `OperationCanceledException`, which `PreparePipe` already swallows, and the prepare shovel exits through its own (already cancelled) token. In a normal shutdown with a healthy writer nothing changes: the channel drains before `writerCts` is cancelled. -**Reader.** `GrpcEventReader` calls `auth.AcquireCredentials(cancellationToken)` before starting the long-running `ReadAllAsync` and passes them as `userCredentials`. Once the first event (or end of stream) is received, it calls `ReportAccepted(t)`. A token failure during that read (at start, or if the server ends the stream) fails the enumeration. `ReadEvents` catches it, calls `Invalidate(t)` if it is a token failure, and rethrows. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, acquiring credentials again. No pipeline change is needed. `GetLastPosition` (a bounded backwards read) goes through `auth.Run`. +**Reader.** `GrpcEventReader` calls `auth.AcquireCredentials(cancellationToken)` before starting the long-running `ReadAllAsync` and passes them as `userCredentials`. Once the first event (or end of stream) is received, it calls `auth.ReportAccepted(callAuth)`. A token failure during that read (at start, or if the server ends the stream) fails the enumeration. `ReadEvents` catches it, calls `auth.ReportFailure(callAuth, e)` (which invalidates the lease if `e` is a token failure), and rethrows. `ReaderPipe` logs it and stops, and `Replicator` restarts from the last checkpoint after `RestartDelayInSeconds`, acquiring credentials again. No pipeline change is needed. `GetLastPosition` (a bounded backwards read) goes through `auth.Run`. **Metrics reporter.** `Replicator.Report` calls `reader.GetLastPosition` every `ReportMetricsFrequencyInSeconds` and catches only `OperationCanceledException`, so any other exception ends the reporter for the rest of the run. `GetLastPosition` now waits out token failures. In addition, `Report` is changed to catch and log (warning, rate-limited to once a minute) any other exception per iteration and continue, so a transient failure, auth or not, no longer stops metrics permanently. @@ -235,7 +235,7 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if a subscription is active, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. - `SubscribeLoop()` calls `SubscribeToAllAsync` through `auth.Run` (which handles token failures and invalidation), and additionally retries any other failure with the same capped backoff, logging each, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers therefore share one attempt, and there is never more than one subscription. -- The token used for the active subscription is stored with it. `HandleDrop` with a token-failure exception calls `Invalidate(t)` for that token first. +- The `CallAuth` used for the active subscription is stored with it. `HandleDrop` with an exception calls `auth.ReportFailure(callAuth, exception)` first. - `HandleDrop` (any reason except `Disposed`) clears the stored subscription under the lock and calls `EnsureSubscribed(CancellationToken.None)` without awaiting it. The loop is still bound to shutdown, and any exception other than cancellation is caught and logged inside the loop, so nothing is unobserved. - `Realtime` gets an internal constructor taking a `Func>` subscribe delegate, `GrpcAuthContext` and `TimeProvider`, so all of this is unit-testable without a server. @@ -291,10 +291,11 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - concurrent callers trigger exactly one token request, both when it succeeds and when it fails (all callers see the same exception) - failure cooldown: calls within 5 s of a failure make no request and rethrow; after the cooldown a new request is made - one caller cancelling does not cancel the shared request for others; cancelling the shutdown token aborts an in-flight request - - `Invalidate(t)` forces the next call to refresh, still single-flight and subject to cooldown - - after `Invalidate(t)`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token + - `Invalidate(lease)` forces the next call to refresh, still single-flight and subject to cooldown + - after `Invalidate(lease)`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token - quarantine: the endpoint re-issuing the rejected value within 60 s throws; after 60 s the same value is returned (probe); a different value is returned immediately - - probe lease: with 10 concurrent callers at the quarantine boundary exactly one receives the rejected value and the rest throw; `ReportAccepted` releases the token to all; `Invalidate` during the probe restarts quarantine; `Invalidate` of a stale value (newer token already cached) is ignored; an unreported lease expires after 60 s + - out-of-order reports with a re-issued value: call 1 gets A (gen 1); A is rejected via another call (quarantine, gen 2); after the quarantine the probe gets A (gen 3) and is accepted. A late `Invalidate` from call 1 (gen 1) does not re-quarantine A. Conversely, a late `ReportAccepted` from gen 1 arriving during the gen-2 quarantine does not clear it. + - probe lease: with 10 concurrent callers at the quarantine boundary exactly one receives the rejected value and the rest throw; `ReportAccepted` releases the token to all; `Invalidate` during the probe restarts quarantine; `Invalidate`/`ReportAccepted` with a stale generation are ignored; an unreported lease expires after 60 s - `error` equal to the submitted client secret is not in the allowlist and is reported as `non-standard error (omitted)`; allowlisted codes and integer `error_codes` are reported - an error response whose `error_description`, `error_uri` and body echo the client secret and client assertion: exception message and all logs at Info/Warning contain neither; Debug output contains them only as `***`; a non-conforming `error` value is omitted - refresh failure with valid cached token returns cached token; with expired token throws `OAuthTokenException` carrying the HTTP status and `error` code @@ -303,15 +304,15 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - assertion file re-read per request - exception messages and logs contain no secret/assertion/token - `TokenGate`: backoff sequence and cap, cancellation, rate-limited logging, recovery log. -- `GrpcAuthContext`: `AcquireCredentials` returns `null` for `None`; cancelled by either the caller token or the shutdown token. `Run`: calls `ReportAccepted(t)` after success with the exact value sent; retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. +- `GrpcAuthContext`: `AcquireCredentials` returns `null` for `None`; cancelled by either the caller token or the shutdown token. `Run`: calls `ReportAccepted(lease)` after success with the lease the call used; retries `OAuthTokenException` and `Unauthenticated` with invalidation and backoff until success; propagates `PermissionDenied` and other exceptions immediately; shutdown ends the loop. - `AuthFailure.IsTokenFailure`: direct `OAuthTokenException`, wrapped in `RpcException.Status.DebugException` and `AggregateException`, `Unauthenticated`; negative cases including `PermissionDenied`. - `GrpcEventWriter` over a capturing-handler client with a token source that fails N times then succeeds: the write waits during failures, then is sent with the new token. Cancelling the shutdown token during the outage ends the wait with `OperationCanceledException`. -- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate(A)` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. A third case leaves A in the file and switches the stub to accept A after the rejection (a server-side fix): after the 60 s quarantine (advanced with `FakeTimeProvider`) A is sent once more and the write succeeds. +- Rejected-token recovery: `GrpcEventWriter` with a `TokenFileSource` whose file holds token A, over a capturing stub handler that records requests and answers with gRPC status `Unauthenticated` while it sees `Bearer A`. The test rotates the file to token B. Expected: the write is retried, the file is re-read via `Invalidate(leaseA)` without waiting for the reload interval, and the next request carries `Bearer B`. A second case removes the file after the rejection, then writes token B later: no request carries `Bearer A` after the rejection, and the write succeeds with B. A third case leaves A in the file and switches the stub to accept A after the rejection (a server-side fix): after the 60 s quarantine (advanced with `FakeTimeProvider`) A is sent once more and the write succeeds. - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that runs each write through `GrpcAuthContext.Run` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; each auxiliary call acquires credentials through `auth.Run`, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. -- `TokenFileSource`: `Invalidate(t)` forces an immediate re-read; after it, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. +- `TokenFileSource`: `Invalidate(lease)` forces an immediate re-read; after it, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. - Per-call credentials through the real client: `GrpcEventWriter` (append, delete, set-metadata), `GrpcEventReader.GetLastPosition`, `ScavengedEventsFilter` metadata and stream-size reads, and the start of `ReadAllAsync` each send `authorization: Bearer `; a test hook that asserts the sentinel fallback is never invoked on these paths. - Probe end-to-end: quarantine token A, stub that rejects A, then switches to accepting A. After 60 s (FakeTimeProvider), with 5 concurrent writes queued: exactly one HTTP request carries `Bearer A`, it succeeds, `ReportAccepted` releases A, and the other writes then send `Bearer A` and succeed. From c124bd7ba96677ef47b27f52a8dab9a552dbe2fc Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:29:14 -0400 Subject: [PATCH 13/67] docs: address spec review round 10 (generation bump rules) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index f64951e5..f19e5d7a 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -118,7 +118,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(AccessTokenLease lease)` and `void ReportAccepted(AccessTokenLease lease)`. `AccessTokenLease` is `(string Value, long Generation)`. The source increments `Generation` on every state change: a new token acquired or file re-read, a quarantine started, a probe lease granted. A report is applied only if `lease.Generation` equals the current generation; otherwise it is stale (it belongs to an earlier acquisition or probe, even if the token value is identical) and is ignored, logged at Debug. `ReportAccepted` ends a rejected-token probe. `Invalidate(lease)` (if current) marks that token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(AccessTokenLease lease)` and `void ReportAccepted(AccessTokenLease lease)`. `AccessTokenLease` is `(string Value, long Generation)`. The source increments `Generation` when the cached token *value* changes, and on every rejection-state transition: a quarantine starts, a probe lease is granted, a probe is accepted, or a lease expires. A refresh or file re-read that yields the *same* value does **not** increment it. So a rejection of a value that is still current is never lost to a routine reload, while a report from before a quarantine or probe is recognisably stale. A report is applied only if `lease.Generation` equals the current generation (which also implies `lease.Value` equals the current value); otherwise it is ignored and logged at Debug. `ReportAccepted` ends a rejected-token probe. `Invalidate(lease)` (if current) marks that token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | @@ -294,6 +294,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - `Invalidate(lease)` forces the next call to refresh, still single-flight and subject to cooldown - after `Invalidate(lease)`, a failing token endpoint causes `GetAccessToken` to throw rather than fall back to the rejected token - quarantine: the endpoint re-issuing the rejected value within 60 s throws; after 60 s the same value is returned (probe); a different value is returned immediately + - same-value reload does not lose a rejection: call 1 gets A (gen 1); the reload timer re-reads unchanged A (still gen 1) and a client-credentials refresh re-issues A (still gen 1); call 1 is then rejected, so `Invalidate` applies, quarantine starts, and no call is handed A until the quarantine ends (verified for both sources) - out-of-order reports with a re-issued value: call 1 gets A (gen 1); A is rejected via another call (quarantine, gen 2); after the quarantine the probe gets A (gen 3) and is accepted. A late `Invalidate` from call 1 (gen 1) does not re-quarantine A. Conversely, a late `ReportAccepted` from gen 1 arriving during the gen-2 quarantine does not clear it. - probe lease: with 10 concurrent callers at the quarantine boundary exactly one receives the rejected value and the rest throw; `ReportAccepted` releases the token to all; `Invalidate` during the probe restarts quarantine; `Invalidate`/`ReportAccepted` with a stale generation are ignored; an unreported lease expires after 60 s - `error` equal to the submitted client secret is not in the allowlist and is reported as `non-standard error (omitted)`; allowlisted codes and integer `error_codes` are reported From 6d5805c76a9eb4f04064c043ebaa1ed8b212de31 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:30:53 -0400 Subject: [PATCH 14/67] docs: address spec review round 11 (CallAuth to Run callbacks) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index f19e5d7a..950bc0e0 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -122,7 +122,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | -| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `ValueTask AcquireCredentials(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token and returns `CallAuth(UserCredentials? Credentials, AccessTokenLease? Lease)` (both `null` without OAuth, so the client keeps using the connection string's credentials exactly as today), plus `ReportAccepted(CallAuth)` / `ReportFailure(CallAuth, Exception)` for long-lived calls that cannot use `Run`; `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"). Every call site passes the credentials it is handed as the client method's `userCredentials` argument. | +| `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `ValueTask AcquireCredentials(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token and returns `CallAuth(UserCredentials? Credentials, AccessTokenLease? Lease)` (both `null` without OAuth, so the client keeps using the connection string's credentials exactly as today), plus `ReportAccepted(CallAuth)` / `ReportFailure(CallAuth, Exception)` for long-lived calls that cannot use `Run`; `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"). Every call site passes `callAuth.Credentials` as the client method's `userCredentials` argument. Call sites that must remember which lease a result belongs to (the subscription) keep the `CallAuth` they were handed. | | `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is not a token failure (retrying with a new token does not help), and is handled separately where it matters (scavenge filter). | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue` with the fallback hook (sentinel → fetch from source; anything else → `credentials.ToString()`). Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | @@ -201,7 +201,7 @@ Both are usually transient, so neither may permanently stop replication. Every w **Token-aware calls.** `GrpcAuthContext.Run(call, ct)` is used for every bounded (non-streaming-for-the-life-of-the-run) gRPC operation: appends, deletes, metadata writes, metadata reads, stream-size reads, `GetLastPosition` and subscribe. It loops: 1. `lease = source.GetAccessToken` via `TokenGate` (capped backoff 1 s, 2 s, 4 s … 30 s, cancellable by `ct` and shutdown). `creds = new UserCredentials(lease.Value)`. -2. Invoke `call(creds, ct)`; the call passes `creds` as `userCredentials`, so the request header is exactly `Bearer `. +2. Invoke `call(callAuth, ct)` where `callAuth = CallAuth(creds, lease)`; the call passes `callAuth.Credentials` as `userCredentials`, so the request header is exactly `Bearer `. 3. On success: `source.ReportAccepted(lease)`. 4. If the call throws and `AuthFailure.IsTokenFailure(e)` is true: `source.Invalidate(lease)`, log a warning (first failure, then at most once a minute; recovery is logged once), wait the next backoff step, and go to 1. 5. Any other exception propagates unchanged, so existing behaviour for non-auth errors is preserved. @@ -233,7 +233,7 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - State: the current `StreamSubscription?` and a single in-flight `Task? _subscribing`, both guarded by a lock. - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if a subscription is active, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. -- `SubscribeLoop()` calls `SubscribeToAllAsync` through `auth.Run` (which handles token failures and invalidation), and additionally retries any other failure with the same capped backoff, logging each, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. +- `SubscribeLoop()` calls `SubscribeToAllAsync` through `auth.Run`; the callback stores the `CallAuth` it receives together with the returned `StreamSubscription`, so the lease of the token actually sent is known for later drops (which handles token failures and invalidation), and additionally retries any other failure with the same capped backoff, logging each, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers therefore share one attempt, and there is never more than one subscription. - The `CallAuth` used for the active subscription is stored with it. `HandleDrop` with an exception calls `auth.ReportFailure(callAuth, exception)` first. - `HandleDrop` (any reason except `Disposed`) clears the stored subscription under the lock and calls `EnsureSubscribed(CancellationToken.None)` without awaiting it. The loop is still bound to shutdown, and any exception other than cancellation is caught and logged inside the loop, so nothing is unobserved. @@ -312,6 +312,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that runs each write through `GrpcAuthContext.Run` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; each auxiliary call acquires credentials through `auth.Run`, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. - `TokenFileSource`: `Invalidate(lease)` forces an immediate re-read; after it, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. - `GrpcAuthOptionsValidator`: every rule above, including the combined-errors message. From 0883dbddab21b664c2b095ecfb60766a0a7e12f1 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:32:34 -0400 Subject: [PATCH 15/67] docs: address spec review round 12 (subscription attempts) Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 950bc0e0..33b8c9be 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -231,13 +231,17 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie **Realtime subscription.** Today `Realtime.Start` sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored, and a reader restart during a pending resubscribe could open a second subscription. This latent bug becomes likely with tokens, so `Realtime` is reworked around one serialized operation: -- State: the current `StreamSubscription?` and a single in-flight `Task? _subscribing`, both guarded by a lock. -- `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if a subscription is active, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. -- `SubscribeLoop()` calls `SubscribeToAllAsync` through `auth.Run`; the callback stores the `CallAuth` it receives together with the returned `StreamSubscription`, so the lease of the token actually sent is known for later drops (which handles token failures and invalidation), and additionally retries any other failure with the same capped backoff, logging each, until it succeeds or the shutdown token is cancelled. On success it stores the subscription and clears `_subscribing`. -- `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers therefore share one attempt, and there is never more than one subscription. -- The `CallAuth` used for the active subscription is stored with it. `HandleDrop` with an exception calls `auth.ReportFailure(callAuth, exception)` first. -- `HandleDrop` (any reason except `Disposed`) clears the stored subscription under the lock and calls `EnsureSubscribed(CancellationToken.None)` without awaiting it. The loop is still bound to shutdown, and any exception other than cancellation is caught and logged inside the loop, so nothing is unobserved. -- `Realtime` gets an internal constructor taking a `Func>` subscribe delegate, `GrpcAuthContext` and `TimeProvider`, so all of this is unit-testable without a server. +- State, guarded by one lock: the current published `Attempt?` and a single in-flight `Task? _subscribing`. +- Each subscribe attempt is represented by an `Attempt` object, created inside the `auth.Run` callback *before* `SubscribeToAllAsync` is called. It holds the `CallAuth` the callback received, a `Dropped` flag, and later the `StreamSubscription`. The `subscriptionDropped` callback passed to `SubscribeToAllAsync` is a closure over that `Attempt`, so a drop is always correlated with the attempt (and lease) that produced it, however early it fires. +- `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if an attempt is published and not dropped, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. +- `SubscribeLoop()` runs attempts through `auth.Run`, which handles token failures and invalidation, and additionally retries any other failure with the same capped backoff, logging each, until an attempt is published or the shutdown token is cancelled. When `SubscribeToAllAsync` returns, under the lock: if `attempt.Dropped` is already set (the drop beat the registration), the returned subscription is disposed, not published, and the loop backs off and tries again. Otherwise the attempt is published and `_subscribing` is cleared. +- `HandleDrop(attempt, reason, exception)` (any reason except `Disposed`): first `auth.ReportFailure(attempt.CallAuth, exception)`. Then, under the lock, set `attempt.Dropped`. + - If `attempt` is the published one, unpublish it and call `EnsureSubscribed(CancellationToken.None)` without awaiting it. + - If it is still pending, do nothing more: the running loop sees `Dropped` and retries. + - If it is neither (an old attempt), ignore it. + The loop is bound to shutdown, and any exception other than cancellation is caught and logged inside it, so nothing is unobserved. +- `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers share one attempt, and there is never more than one live subscription. +- `Realtime` gets an internal constructor taking a subscribe delegate `Func, CancellationToken, Task>`, `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing, is unit-testable without a server. **Long-lived calls and token expiry.** Credentials are evaluated when a call starts. If KurrentDB terminates `ReadAllAsync` or the subscription when the token expires, the paths above recover with a fresh token. Whether it does is confirmed in the manual end-to-end test. If it causes frequent restarts, proactively recycling the read before expiry is a follow-up. @@ -312,6 +316,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that runs each write through `GrpcAuthContext.Run` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; each auxiliary call acquires credentials through `auth.Run`, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- `Realtime` early drop: the test delegate invokes the drop callback (reason `ServerError`, `Unauthenticated`) before its returned task completes. The attempt's lease is reported as failed, the returned subscription is disposed and never published, and the loop resubscribes; afterwards exactly one live subscription is published. - `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. - `TokenFileSource`: `Invalidate(lease)` forces an immediate re-read; after it, a missing/empty file or a file still holding the rejected value throws during the quarantine, the same value is probed again after it, and a different value is accepted at once; reload interval, trimming, missing/empty file with and without previous token. From 6e325c9afa5170f56ed2943a005fce779c05a76c Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:32:48 -0400 Subject: [PATCH 16/67] docs: early-drop attempt does not report acceptance Co-Authored-By: Claude Opus 5.5 --- .../superpowers/specs/2026-10-03-oauth-authentication-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 33b8c9be..44e778b5 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -234,7 +234,7 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - State, guarded by one lock: the current published `Attempt?` and a single in-flight `Task? _subscribing`. - Each subscribe attempt is represented by an `Attempt` object, created inside the `auth.Run` callback *before* `SubscribeToAllAsync` is called. It holds the `CallAuth` the callback received, a `Dropped` flag, and later the `StreamSubscription`. The `subscriptionDropped` callback passed to `SubscribeToAllAsync` is a closure over that `Attempt`, so a drop is always correlated with the attempt (and lease) that produced it, however early it fires. - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if an attempt is published and not dropped, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. -- `SubscribeLoop()` runs attempts through `auth.Run`, which handles token failures and invalidation, and additionally retries any other failure with the same capped backoff, logging each, until an attempt is published or the shutdown token is cancelled. When `SubscribeToAllAsync` returns, under the lock: if `attempt.Dropped` is already set (the drop beat the registration), the returned subscription is disposed, not published, and the loop backs off and tries again. Otherwise the attempt is published and `_subscribing` is cleared. +- `SubscribeLoop()` runs attempts through `auth.Run`, which handles token failures and invalidation, and additionally retries any other failure with the same capped backoff, logging each, until an attempt is published or the shutdown token is cancelled. When `SubscribeToAllAsync` returns, under the lock: if `attempt.Dropped` is already set (the drop beat the registration), the returned subscription is disposed and not published. The callback then throws an internal `SubscriptionDroppedEarlyException`, so `Run` does not call `ReportAccepted` for a lease whose subscription already failed, and the loop backs off and tries again (a token failure was already reported by `HandleDrop`). Otherwise the attempt is published and `_subscribing` is cleared. - `HandleDrop(attempt, reason, exception)` (any reason except `Disposed`): first `auth.ReportFailure(attempt.CallAuth, exception)`. Then, under the lock, set `attempt.Dropped`. - If `attempt` is the published one, unpublish it and call `EnsureSubscribed(CancellationToken.None)` without awaiting it. - If it is still pending, do nothing more: the running loop sees `Dropped` and retries. From c2bd0e8e2baa9ab4ac08356a900ca64372c75b8c Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:34:09 -0400 Subject: [PATCH 17/67] docs: address spec review round 13 (long-lived calls, OAuth reader E2E) Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 44e778b5..fd4a62d4 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -17,7 +17,7 @@ Replicator is a headless service, so it must authenticate as a machine identity - **Client credentials** — Replicator obtains and refreshes tokens itself (RFC 6749 §4.4), authenticating to the token endpoint with a client secret or a client assertion (RFC 7523; covers Entra workload identity federation). - **Token file** — an external process keeps a valid access token in a file; Replicator reads it. 4. Reader (source) and sink (target) are configured completely independently. Any combination is valid: basic + OAuth, OAuth + basic, OAuth + OAuth with different providers/clients, basic + basic. -5. Tokens are refreshed before expiry without restarting Replicator, including across long-running reads and the realtime subscription. +5. Replication continues across token expiry without restarting Replicator. Every new gRPC call carries a current token. A long-running call (the `$all` read and the realtime subscription) is authorized by KurrentDB when it starts. Replicator relies on that server-side contract: if the server keeps the stream open past the token's expiry, the stream simply continues, which is correct and needs no refresh. If the server ends it, Replicator restarts it with a fresh token. Either outcome must leave replication running with no lost or duplicated events beyond today's at-least-once semantics. 6. Secrets and tokens are never logged. ## Non-goals @@ -243,7 +243,12 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers share one attempt, and there is never more than one live subscription. - `Realtime` gets an internal constructor taking a subscribe delegate `Func, CancellationToken, Task>`, `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing, is unit-testable without a server. -**Long-lived calls and token expiry.** Credentials are evaluated when a call starts. If KurrentDB terminates `ReadAllAsync` or the subscription when the token expires, the paths above recover with a fresh token. Whether it does is confirmed in the manual end-to-end test. If it causes frequent restarts, proactively recycling the read before expiry is a follow-up. +**Long-lived calls and token expiry.** Credentials are evaluated when a call starts, and gRPC has no way to swap the header of an open stream. There are two possible server behaviours, and both are handled: + +- **Stream kept open past expiry:** the call stays authorized as it was at start. Replication continues. Nothing to do: the token is not re-sent on that stream, and every other call (writes, metadata reads, `GetLastPosition`, a later restart) uses a current token. Proactive recycling would only add restarts and is not done. +- **Stream ended at expiry** (any status, typically `Unauthenticated`): the `$all` read takes the reader restart path (checkpointed, at-least-once as today), and the subscription takes the `HandleDrop` path. Both acquire a fresh token. + +The manual end-to-end test records which behaviour KurrentDB has, so the docs can state it. No design change depends on the answer. ### Validation (startup, fail fast) @@ -329,7 +334,12 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes Existing container-based tests keep running unchanged (`connectionString` default). -Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth plugin needs a licence and a real IdP): KurrentDB with OAuth + Entra ID as sink, basic-auth KurrentDB/EventStoreDB as reader; replicate, run past a token expiry, confirm continued replication and observe whether the subscription/read is terminated at expiry. +Manual end-to-end (documented in the PR, not automated — the KurrentDB OAuth plugin needs a licence and a real IdP). Use an Entra ID app with a short access-token lifetime (token lifetime policy, or a Keycloak realm with a 5-minute lifetime for the generic case), and a continuous writer appending to the source throughout: + +1. **OAuth sink, basic reader:** replicate for at least 3 token lifetimes. Expected: the sink log shows token refreshes with no write failures, and the target event count tracks the source. +2. **OAuth reader, basic sink:** replicate for at least 3 token lifetimes without interruption, covering both the `$all` read and the realtime subscription. Expected, and recorded in the PR: either (a) no reader restart or subscription drop across expiries, and events keep flowing; or (b) a reader restart and/or subscription drop at each expiry, followed by resumption from the checkpoint with a fresh token. In both cases the target ends with every source event, with no gap. +3. **OAuth on both sides with different client IDs** (`oauthClientCredentials` on the reader, `oauthTokenFile` on the sink): replicate past expiry, then rotate the sink token file and verify the switch to the new token. +4. **Token endpoint outage:** block the endpoint for longer than a token lifetime, then restore it. Expected: warnings, paused progress, automatic resumption, no restart of the process. ## Risks From 5a1a5f254bec56eeed043d3aee7ee902e74745bb Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:35:40 -0400 Subject: [PATCH 18/67] docs: address spec review round 14 (gap-safe resubscribe) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index fd4a62d4..63be99ca 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -240,8 +240,9 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - If it is still pending, do nothing more: the running loop sees `Dropped` and retries. - If it is neither (an old attempt), ignore it. The loop is bound to shutdown, and any exception other than cancellation is caught and logged inside it, so nothing is unobserved. +- **Gap-safe resubscription.** Today the subscription starts from `FromAll.End`, and `HandleEvent` keeps `StreamMetaCache` (metadata, last event numbers) current, which the scavenge filter trusts. A resubscription from `End` would silently skip updates made while it was down, leaving the cache stale. `Realtime` therefore records the `$all` position of every event `HandleEvent` processes (including the skipped system events), in `_lastPosition`. The first subscription uses `FromAll.End`. Every later attempt uses `FromAll.After(_lastPosition)` when a position is known, so the missed updates are replayed into the cache before it is current again. If nothing was received yet, it uses `End` as today, because there is no earlier state to be stale. `HandleEvent` is idempotent for replays: `UpdateStreamMeta` already ignores older versions, and `UpdateStreamLastEventNumber` only moves forward. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers share one attempt, and there is never more than one live subscription. -- `Realtime` gets an internal constructor taking a subscribe delegate `Func, CancellationToken, Task>`, `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing, is unit-testable without a server. +- `Realtime` gets an internal constructor taking a subscribe delegate `Func, Action, CallAuth, CancellationToken, Task>` (start position, event handler, drop callback, auth), `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing and start positions, is unit-testable without a server. **Long-lived calls and token expiry.** Credentials are evaluated when a call starts, and gRPC has no way to swap the header of an open stream. There are two possible server behaviours, and both are handled: @@ -321,6 +322,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that runs each write through `GrpcAuthContext.Run` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; each auxiliary call acquires credentials through `auth.Run`, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. +- `Realtime` gap safety, with `scavenge: true`: receive events up to position P, drop the subscription, and while it is down write a metadata change (`$maxCount`) and new events to a stream. The resubscribe delegate is called with `FromAll.After(P)`, the replayed events update `StreamMetaCache`, and `ScavengedEventsFilter` then decides using the new metadata and last event number. The first subscription uses `End`. - `Realtime` early drop: the test delegate invokes the drop callback (reason `ServerError`, `Unauthenticated`) before its returned task completes. The attempt's lease is reported as failed, the returned subscription is disposed and never published, and the loop resubscribes; afterwards exactly one live subscription is published. - `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. From d488bb6514ace248df2a64b21114471d0ff624f5 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:36:54 -0400 Subject: [PATCH 19/67] docs: address spec review round 15 (cache live state) Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-oauth-authentication-design.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 63be99ca..962b52de 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -240,9 +240,14 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - If it is still pending, do nothing more: the running loop sees `Dropped` and retries. - If it is neither (an old attempt), ignore it. The loop is bound to shutdown, and any exception other than cancellation is caught and logged inside it, so nothing is unobserved. -- **Gap-safe resubscription.** Today the subscription starts from `FromAll.End`, and `HandleEvent` keeps `StreamMetaCache` (metadata, last event numbers) current, which the scavenge filter trusts. A resubscription from `End` would silently skip updates made while it was down, leaving the cache stale. `Realtime` therefore records the `$all` position of every event `HandleEvent` processes (including the skipped system events), in `_lastPosition`. The first subscription uses `FromAll.End`. Every later attempt uses `FromAll.After(_lastPosition)` when a position is known, so the missed updates are replayed into the cache before it is current again. If nothing was received yet, it uses `End` as today, because there is no earlier state to be stale. `HandleEvent` is idempotent for replays: `UpdateStreamMeta` already ignores older versions, and `UpdateStreamLastEventNumber` only moves forward. +- **Cache coherence across subscription gaps.** `HandleEvent` keeps `StreamMetaCache` (metadata, last event numbers) current, and the scavenge filter trusts it. While the subscription is down, updates are missed, so cached entries may be stale. Replaying from a saved position is not enough: the cache can also be filled by direct reads before any event arrives, and replay is asynchronous to filtering. Instead, `StreamMetaCache` gets an explicit "live" state owned by `Realtime`: + - `MarkNotLive()` is called by `HandleDrop` before anything else. While not live, `GetOrAddStreamMeta` / `GetOrAddStreamSize` ignore cached entries, always fetch the current value directly (through `auth.Run`, as above), and do not store it. Filtering during the gap is therefore based on current server state. + - `MarkLive()` is called by the subscribe loop only after `SubscribeToAllAsync` has returned (the server has confirmed the subscription) and the attempt is published. It atomically clears both dictionaries, bumps a cache epoch and re-enables caching. From then on, every update reaches the cache through the live subscription, and every entry is re-fetched fresh on first use. + - A fetch that started under an older epoch does not store its result (checked when it completes), so a read issued during the gap cannot reinsert a pre-gap value after `MarkLive`. + - The resubscription starts from `FromAll.End`, as today. No position replay or catch-up barrier is needed, because nothing cached before or during the gap survives. + - Before the first subscription the cache starts not-live, and the first successful subscribe marks it live. `ReadEvents` already awaits `Start()` before reading, so initial behaviour is unchanged. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers share one attempt, and there is never more than one live subscription. -- `Realtime` gets an internal constructor taking a subscribe delegate `Func, Action, CallAuth, CancellationToken, Task>` (start position, event handler, drop callback, auth), `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing and start positions, is unit-testable without a server. +- `Realtime` gets an internal constructor taking a subscribe delegate `Func, Action, CallAuth, CancellationToken, Task>` (event handler, drop callback, auth), `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing and cache state, is unit-testable without a server. **Long-lived calls and token expiry.** Credentials are evaluated when a call starts, and gRPC has no way to swap the header of an open stream. There are two possible server behaviours, and both are handled: @@ -322,7 +327,10 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - Pipeline acceptance test through `Replicator.Replicate` with a fake reader producing events and a test `IEventWriter` that runs each write through `GrpcAuthContext.Run` like `GrpcEventWriter`, using a token source that fails, then recovers: all events are written after recovery with no restart. A second case uses a sink `BufferSize` of 1 and a prepare buffer larger than the event count, so that the sink channel is full and the prepare shovel is blocked on `WriteAsync` when shutdown is triggered during the outage. `Replicate` must return within a bounded time (no drain-loop or `prepareTask` hang), the unwritten count is logged, and the checkpoint is not advanced past unwritten events. - Metrics reporter: `GetLastPosition` waits during a token outage and resumes after recovery; a non-cancellation exception from `GetLastPosition` is logged and the next iteration still runs. - `StreamMetaCache` / `ScavengedEventsFilter`: with `failClosedOnAuthErrors`, `PermissionDenied` and token failures propagate instead of returning `null`, so the event is never kept on an auth problem; other failures still return `null`; without the flag (basic-auth reader), every failure including `Unauthenticated` and `PermissionDenied` still returns `null` exactly as today; a transient `Unauthenticated` on the metadata call is retried inside `auth.Run` and the filter then decides correctly; each auxiliary call acquires credentials through `auth.Run`, so a token outage that starts after the read began makes the filter wait, then filter correctly after recovery. -- `Realtime` gap safety, with `scavenge: true`: receive events up to position P, drop the subscription, and while it is down write a metadata change (`$maxCount`) and new events to a stream. The resubscribe delegate is called with `FromAll.After(P)`, the replayed events update `StreamMetaCache`, and `ScavengedEventsFilter` then decides using the new metadata and last event number. The first subscription uses `End`. +- `StreamMetaCache` live state: while not live, reads always fetch and never store; `MarkLive` clears both dictionaries; a fetch started before `MarkLive` that completes after it does not store its value. +- `Realtime` cache coherence, with `scavenge: true`: + (a) The cache is hydrated by direct reads with zero subscription events. The subscription drops, and `$maxCount` changes during the outage. Filtering during the outage uses the new value (direct fetch), and after resubscription the cache holds the new value, not the pre-drop one. + (b) After resubscribing, the test delays the first event callbacks. Filtering in that window uses freshly fetched values, never pre-drop cache entries. - `Realtime` early drop: the test delegate invokes the drop callback (reason `ServerError`, `Unauthenticated`) before its returned task completes. The attempt's lease is reported as failed, the returned subscription is disposed and never published, and the loop resubscribes; afterwards exactly one live subscription is published. - `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. From 78739c606a219ef8787a19be385f72f0b4759d25 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:38:12 -0400 Subject: [PATCH 20/67] docs: address spec review round 16 (classify drops before mutating) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 962b52de..534ba8cd 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -235,13 +235,13 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - Each subscribe attempt is represented by an `Attempt` object, created inside the `auth.Run` callback *before* `SubscribeToAllAsync` is called. It holds the `CallAuth` the callback received, a `Dropped` flag, and later the `StreamSubscription`. The `subscriptionDropped` callback passed to `SubscribeToAllAsync` is a closure over that `Attempt`, so a drop is always correlated with the attempt (and lease) that produced it, however early it fires. - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if an attempt is published and not dropped, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. - `SubscribeLoop()` runs attempts through `auth.Run`, which handles token failures and invalidation, and additionally retries any other failure with the same capped backoff, logging each, until an attempt is published or the shutdown token is cancelled. When `SubscribeToAllAsync` returns, under the lock: if `attempt.Dropped` is already set (the drop beat the registration), the returned subscription is disposed and not published. The callback then throws an internal `SubscriptionDroppedEarlyException`, so `Run` does not call `ReportAccepted` for a lease whose subscription already failed, and the loop backs off and tries again (a token failure was already reported by `HandleDrop`). Otherwise the attempt is published and `_subscribing` is cleared. -- `HandleDrop(attempt, reason, exception)` (any reason except `Disposed`): first `auth.ReportFailure(attempt.CallAuth, exception)`. Then, under the lock, set `attempt.Dropped`. - - If `attempt` is the published one, unpublish it and call `EnsureSubscribed(CancellationToken.None)` without awaiting it. - - If it is still pending, do nothing more: the running loop sees `Dropped` and retries. - - If it is neither (an old attempt), ignore it. +- `HandleDrop(attempt, reason, exception)`: if `reason` is `Disposed`, return. Otherwise, under the Realtime lock, classify the attempt **before mutating any state**: + - **Old** (neither the published attempt nor the pending one): ignore it entirely. No token report, no cache change. A delayed callback from a superseded attempt cannot affect the current subscription. + - **Published:** set `Dropped`, unpublish it, call `cache.MarkNotLive()` and remember to report. Outside the lock: `auth.ReportFailure(attempt.CallAuth, exception)`, then `EnsureSubscribed(CancellationToken.None)` without awaiting. + - **Pending:** set `Dropped` and remember to report. Outside the lock: `auth.ReportFailure(...)`. The running loop sees `Dropped` and retries. The cache is still not-live, because it is only marked live on publish. The loop is bound to shutdown, and any exception other than cancellation is caught and logged inside it, so nothing is unobserved. - **Cache coherence across subscription gaps.** `HandleEvent` keeps `StreamMetaCache` (metadata, last event numbers) current, and the scavenge filter trusts it. While the subscription is down, updates are missed, so cached entries may be stale. Replaying from a saved position is not enough: the cache can also be filled by direct reads before any event arrives, and replay is asynchronous to filtering. Instead, `StreamMetaCache` gets an explicit "live" state owned by `Realtime`: - - `MarkNotLive()` is called by `HandleDrop` before anything else. While not live, `GetOrAddStreamMeta` / `GetOrAddStreamSize` ignore cached entries, always fetch the current value directly (through `auth.Run`, as above), and do not store it. Filtering during the gap is therefore based on current server state. + - `MarkNotLive()` is called by `HandleDrop` only for the published attempt (see above). While not live, `GetOrAddStreamMeta` / `GetOrAddStreamSize` ignore cached entries, always fetch the current value directly (through `auth.Run`, as above), and do not store it. Filtering during the gap is therefore based on current server state. - `MarkLive()` is called by the subscribe loop only after `SubscribeToAllAsync` has returned (the server has confirmed the subscription) and the attempt is published. It atomically clears both dictionaries, bumps a cache epoch and re-enables caching. From then on, every update reaches the cache through the live subscription, and every entry is re-fetched fresh on first use. - A fetch that started under an older epoch does not store its result (checked when it completes), so a read issued during the gap cannot reinsert a pre-gap value after `MarkLive`. - The resubscription starts from `FromAll.End`, as today. No position replay or catch-up barrier is needed, because nothing cached before or during the gap survives. @@ -331,6 +331,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes - `Realtime` cache coherence, with `scavenge: true`: (a) The cache is hydrated by direct reads with zero subscription events. The subscription drops, and `$maxCount` changes during the outage. Filtering during the outage uses the new value (direct fetch), and after resubscription the cache holds the new value, not the pre-drop one. (b) After resubscribing, the test delays the first event callbacks. Filtering in that window uses freshly fetched values, never pre-drop cache entries. +- `Realtime` stale callback: attempt A is published, then dropped (a non-token reason), and replacement B is published with the same token generation. Then A's drop callback is delivered again, delayed. B stays published, its token is not invalidated or quarantined, the cache stays live, and no resubscribe starts. - `Realtime` early drop: the test delegate invokes the drop callback (reason `ServerError`, `Unauthenticated`) before its returned task completes. The attempt's lease is reported as failed, the returned subscription is disposed and never published, and the loop resubscribes; afterwards exactly one live subscription is published. - `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. From 2049633a352c99e53905258c80c1be9fde8dc17a Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:39:07 -0400 Subject: [PATCH 21/67] docs: address spec review round 17 (pending attempt identity) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 534ba8cd..85bece39 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -231,14 +231,14 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie **Realtime subscription.** Today `Realtime.Start` sets `_started = true` before `SubscribeToAllAsync` succeeds, and `HandleDrop` resubscribes via an unobserved `Task.Run(Start)`. A failed resubscribe leaves `_started` stuck at `true`, so the subscription is never restored, and a reader restart during a pending resubscribe could open a second subscription. This latent bug becomes likely with tokens, so `Realtime` is reworked around one serialized operation: -- State, guarded by one lock: the current published `Attempt?` and a single in-flight `Task? _subscribing`. -- Each subscribe attempt is represented by an `Attempt` object, created inside the `auth.Run` callback *before* `SubscribeToAllAsync` is called. It holds the `CallAuth` the callback received, a `Dropped` flag, and later the `StreamSubscription`. The `subscriptionDropped` callback passed to `SubscribeToAllAsync` is a closure over that `Attempt`, so a drop is always correlated with the attempt (and lease) that produced it, however early it fires. +- State, guarded by one lock: `_published` (`Attempt?`), `_pending` (`Attempt?`, the single attempt currently being subscribed) and a single in-flight `Task? _subscribing` (the loop that may run several attempts in sequence). +- Each subscribe attempt is represented by an `Attempt` object, created inside the `auth.Run` callback *before* `SubscribeToAllAsync` is called, and assigned to `_pending` under the lock, replacing any previous pending attempt. When an attempt ends (published, failed, or dropped early), the loop clears `_pending` under the lock if it still refers to that attempt. It holds the `CallAuth` the callback received, a `Dropped` flag, and later the `StreamSubscription`. The `subscriptionDropped` callback passed to `SubscribeToAllAsync` is a closure over that `Attempt`, so a drop is always correlated with the attempt (and lease) that produced it, however early it fires. - `EnsureSubscribed(CancellationToken ct)` is the only way to subscribe. Under the lock: if an attempt is published and not dropped, return. If `_subscribing` is running, await it (with `ct` applied via `WaitAsync`). Otherwise start `_subscribing = SubscribeLoop()` and await it. - `SubscribeLoop()` runs attempts through `auth.Run`, which handles token failures and invalidation, and additionally retries any other failure with the same capped backoff, logging each, until an attempt is published or the shutdown token is cancelled. When `SubscribeToAllAsync` returns, under the lock: if `attempt.Dropped` is already set (the drop beat the registration), the returned subscription is disposed and not published. The callback then throws an internal `SubscriptionDroppedEarlyException`, so `Run` does not call `ReportAccepted` for a lease whose subscription already failed, and the loop backs off and tries again (a token failure was already reported by `HandleDrop`). Otherwise the attempt is published and `_subscribing` is cleared. - `HandleDrop(attempt, reason, exception)`: if `reason` is `Disposed`, return. Otherwise, under the Realtime lock, classify the attempt **before mutating any state**: - - **Old** (neither the published attempt nor the pending one): ignore it entirely. No token report, no cache change. A delayed callback from a superseded attempt cannot affect the current subscription. - - **Published:** set `Dropped`, unpublish it, call `cache.MarkNotLive()` and remember to report. Outside the lock: `auth.ReportFailure(attempt.CallAuth, exception)`, then `EnsureSubscribed(CancellationToken.None)` without awaiting. - - **Pending:** set `Dropped` and remember to report. Outside the lock: `auth.ReportFailure(...)`. The running loop sees `Dropped` and retries. The cache is still not-live, because it is only marked live on publish. + - **Old** (neither `_published` nor `_pending`, compared by reference): ignore it entirely. No token report, no cache change. A delayed callback from a superseded attempt cannot affect the current subscription. + - **Published** (`attempt == _published`): set `Dropped`, unpublish it, call `cache.MarkNotLive()` and remember to report. Outside the lock: `auth.ReportFailure(attempt.CallAuth, exception)`, then `EnsureSubscribed(CancellationToken.None)` without awaiting. + - **Pending** (`attempt == _pending`): set `Dropped` and remember to report. Outside the lock: `auth.ReportFailure(...)`. The running loop sees `Dropped` and retries. The cache is still not-live, because it is only marked live on publish. The loop is bound to shutdown, and any exception other than cancellation is caught and logged inside it, so nothing is unobserved. - **Cache coherence across subscription gaps.** `HandleEvent` keeps `StreamMetaCache` (metadata, last event numbers) current, and the scavenge filter trusts it. While the subscription is down, updates are missed, so cached entries may be stale. Replaying from a saved position is not enough: the cache can also be filled by direct reads before any event arrives, and replay is asynchronous to filtering. Instead, `StreamMetaCache` gets an explicit "live" state owned by `Realtime`: - `MarkNotLive()` is called by `HandleDrop` only for the published attempt (see above). While not live, `GetOrAddStreamMeta` / `GetOrAddStreamSize` ignore cached entries, always fetch the current value directly (through `auth.Run`, as above), and do not store it. Filtering during the gap is therefore based on current server state. @@ -332,6 +332,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes (a) The cache is hydrated by direct reads with zero subscription events. The subscription drops, and `$maxCount` changes during the outage. Filtering during the outage uses the new value (direct fetch), and after resubscription the cache holds the new value, not the pre-drop one. (b) After resubscribing, the test delays the first event callbacks. Filtering in that window uses freshly fetched values, never pre-drop cache entries. - `Realtime` stale callback: attempt A is published, then dropped (a non-token reason), and replacement B is published with the same token generation. Then A's drop callback is delivered again, delayed. B stays published, its token is not invalidated or quarantined, the cache stays live, and no resubscribe starts. +- `Realtime` late drop during a retry: attempt A fails, and the same loop starts attempt B (now `_pending`) with the same token generation. A's delayed drop callback then arrives. It is classified as old: B is not marked dropped, its token is not reported, and B publishes normally. A drop for B itself is classified as pending and handled. - `Realtime` early drop: the test delegate invokes the drop callback (reason `ServerError`, `Unauthenticated`) before its returned task completes. The attempt's lease is reported as failed, the returned subscription is disposed and never published, and the loop resubscribes; afterwards exactly one live subscription is published. - `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. From 12ee317344c621f400990166d96268c922c3989d Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:40:10 -0400 Subject: [PATCH 22/67] docs: address spec review round 18 (report inside lock) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 85bece39..0889e417 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -118,7 +118,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single |---|---| | `GrpcAuthOptions` (record) | Plain options for one side: `Type` enum plus the fields above. No binding logic. | | `GrpcAuthOptionsValidator` | Validates a `GrpcAuthOptions` against its parsed `EventStoreClientSettings`; returns all errors (not just the first). | -| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(AccessTokenLease lease)` and `void ReportAccepted(AccessTokenLease lease)`. `AccessTokenLease` is `(string Value, long Generation)`. The source increments `Generation` when the cached token *value* changes, and on every rejection-state transition: a quarantine starts, a probe lease is granted, a probe is accepted, or a lease expires. A refresh or file re-read that yields the *same* value does **not** increment it. So a rejection of a value that is still current is never lost to a routine reload, while a report from before a quarantine or probe is recognisably stale. A report is applied only if `lease.Generation` equals the current generation (which also implies `lease.Value` equals the current value); otherwise it is ignored and logged at Debug. `ReportAccepted` ends a rejected-token probe. `Invalidate(lease)` (if current) marks that token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | +| `IAccessTokenSource` | `ValueTask GetAccessToken(CancellationToken ct)`, `void Invalidate(AccessTokenLease lease)` and `void ReportAccepted(AccessTokenLease lease)`. The two report methods are synchronous, do no I/O, hold only the source's own short lock and never call out, so callers may invoke them while holding their own locks. `AccessTokenLease` is `(string Value, long Generation)`. The source increments `Generation` when the cached token *value* changes, and on every rejection-state transition: a quarantine starts, a probe lease is granted, a probe is accepted, or a lease expires. A refresh or file re-read that yields the *same* value does **not** increment it. So a rejection of a value that is still current is never lost to a routine reload, while a report from before a quarantine or probe is recognisably stale. A report is applied only if `lease.Generation` equals the current generation (which also implies `lease.Value` equals the current value); otherwise it is ignored and logged at Debug. `ReportAccepted` ends a rejected-token probe. `Invalidate(lease)` (if current) marks that token as **rejected** and drops it from the cache. The next `GetAccessToken` refreshes (client credentials) or re-reads the file (token file), subject to the same single-flight and cooldown rules. Rejected-token quarantine applies to both sources (see "Rejected tokens"). | | `ClientCredentialsTokenSource` | Token request, response parsing, caching, single-flight refresh. Takes an `HttpMessageHandler` (injectable for tests), a `TimeProvider` and the application shutdown token. | | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | @@ -237,8 +237,9 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - `SubscribeLoop()` runs attempts through `auth.Run`, which handles token failures and invalidation, and additionally retries any other failure with the same capped backoff, logging each, until an attempt is published or the shutdown token is cancelled. When `SubscribeToAllAsync` returns, under the lock: if `attempt.Dropped` is already set (the drop beat the registration), the returned subscription is disposed and not published. The callback then throws an internal `SubscriptionDroppedEarlyException`, so `Run` does not call `ReportAccepted` for a lease whose subscription already failed, and the loop backs off and tries again (a token failure was already reported by `HandleDrop`). Otherwise the attempt is published and `_subscribing` is cleared. - `HandleDrop(attempt, reason, exception)`: if `reason` is `Disposed`, return. Otherwise, under the Realtime lock, classify the attempt **before mutating any state**: - **Old** (neither `_published` nor `_pending`, compared by reference): ignore it entirely. No token report, no cache change. A delayed callback from a superseded attempt cannot affect the current subscription. - - **Published** (`attempt == _published`): set `Dropped`, unpublish it, call `cache.MarkNotLive()` and remember to report. Outside the lock: `auth.ReportFailure(attempt.CallAuth, exception)`, then `EnsureSubscribed(CancellationToken.None)` without awaiting. - - **Pending** (`attempt == _pending`): set `Dropped` and remember to report. Outside the lock: `auth.ReportFailure(...)`. The running loop sees `Dropped` and retries. The cache is still not-live, because it is only marked live on publish. + - **Published** (`attempt == _published`): still under the lock, call `auth.ReportFailure(attempt.CallAuth, exception)`, set `Dropped`, unpublish it and call `cache.MarkNotLive()`. Then, outside the lock, call `EnsureSubscribed(CancellationToken.None)` without awaiting. + - **Pending** (`attempt == _pending`): under the lock, `auth.ReportFailure(...)` and set `Dropped`. The running loop sees `Dropped` and retries. The cache is still not-live, because it is only marked live on publish. + - Reporting inside the lock means no other caller can observe "nothing published, nothing pending" before the rejected token is quarantined, so a concurrent foreground `Start` can never be handed the rejected token. This is safe because `ReportFailure` / `Invalidate` / `ReportAccepted` are synchronous, do no I/O and never call back into `Realtime`. Lock order is always Realtime lock, then token-source lock. The token source never takes the Realtime lock. The loop is bound to shutdown, and any exception other than cancellation is caught and logged inside it, so nothing is unobserved. - **Cache coherence across subscription gaps.** `HandleEvent` keeps `StreamMetaCache` (metadata, last event numbers) current, and the scavenge filter trusts it. While the subscription is down, updates are missed, so cached entries may be stale. Replaying from a saved position is not enough: the cache can also be filled by direct reads before any event arrives, and replay is asynchronous to filtering. Instead, `StreamMetaCache` gets an explicit "live" state owned by `Realtime`: - `MarkNotLive()` is called by `HandleDrop` only for the published attempt (see above). While not live, `GetOrAddStreamMeta` / `GetOrAddStreamSize` ignore cached entries, always fetch the current value directly (through `auth.Run`, as above), and do not store it. Filtering during the gap is therefore based on current server state. @@ -333,6 +334,7 @@ Unit tests (TUnit, `test/Kurrent.Replicator.Tests/Auth/`), using a stub `HttpMes (b) After resubscribing, the test delays the first event callbacks. Filtering in that window uses freshly fetched values, never pre-drop cache entries. - `Realtime` stale callback: attempt A is published, then dropped (a non-token reason), and replacement B is published with the same token generation. Then A's drop callback is delivered again, delayed. B stays published, its token is not invalidated or quarantined, the cache stays live, and no resubscribe starts. - `Realtime` late drop during a retry: attempt A fails, and the same loop starts attempt B (now `_pending`) with the same token generation. A's delayed drop callback then arrives. It is classified as old: B is not marked dropped, its token is not reported, and B publishes normally. A drop for B itself is classified as pending and handled. +- `Realtime` drop/Start interleaving: a published subscription with token A (current generation) drops with `Unauthenticated`. A foreground `Start` is triggered concurrently from another thread, synchronised so that it runs immediately after `HandleDrop` releases the lock. The new attempt is never handed A: it receives the quarantine failure and backs off. A test-only hook inside the lock asserts that `Invalidate` has already been applied at the point `_published` becomes null. - `Realtime` early drop: the test delegate invokes the drop callback (reason `ServerError`, `Unauthenticated`) before its returned task completes. The attempt's lease is reported as failed, the returned subscription is disposed and never published, and the loop resubscribes; afterwards exactly one live subscription is published. - `Realtime` lease attribution: subscribe with A (gen 1); the source then moves to token B (gen 2) for other callers; the subscription drops with `Unauthenticated`. `HandleDrop` reports failure with the stored gen-1 lease, which is stale and ignored, so B is not invalidated, and the resubscribe uses B. In a second case the source is still on A (gen 1) when the drop arrives: A is invalidated and quarantined, and the resubscribe waits for a different token. - `Realtime` via the subscribe delegate: a failed initial subscribe leaves it retryable; a drop followed by failing resubscribes keeps retrying, then subscribes once the source recovers; a foreground `Start` during a pending resubscribe shares the same attempt and the delegate is called once per attempt (never two live subscriptions); cancelling the shutdown token ends the loop. From 8318cdef13de0d2b64fe0e2ab0354a9bf0de9b03 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 08:46:01 -0400 Subject: [PATCH 23/67] docs: spec status and small consistency fixes Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-03-oauth-authentication-design.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md index 0889e417..5888c3da 100644 --- a/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md +++ b/docs/superpowers/specs/2026-10-03-oauth-authentication-design.md @@ -1,7 +1,7 @@ # OAuth authentication for KurrentDB gRPC reader and sink Date: 2026-10-03 -Status: Draft (for review) +Status: Reviewed (Codex spec-review flow, clean at round 19) ## Problem @@ -47,7 +47,7 @@ No gRPC interceptor or custom `HttpMessageHandler` is needed, and the client's T `CallCredentials` are only sent over TLS channels by Grpc.Net.Client. OAuth therefore requires `tls=true` (the default); see Validation. -Existing call sites (`GrpcEventReader`, `GrpcEventWriter`, `Realtime`, `ConnectionExtensions`, `ScavengedEventsFilter`) pass no per-call credentials, so they all fall back to `DefaultCredentials` and pick up the token automatically. +Existing call sites (`GrpcEventReader`, `GrpcEventWriter`, `Realtime`, `ConnectionExtensions`, `ScavengedEventsFilter`) pass no per-call credentials today. This work changes every one of them to pass per-call credentials. The fallback hook only protects against a call site being missed. ## Configuration @@ -123,7 +123,7 @@ All new code in `src/Kurrent.Replicator.KurrentDb/Auth/`. Each unit has a single | `TokenFileSource` | Reads and caches the token file; reloads on interval or on `Invalidate`. Takes a `TimeProvider`. | | `TokenGate` | `WaitForToken(IAccessTokenSource, CancellationToken)`: retries token acquisition with capped exponential backoff and rate-limited logging until it succeeds or is cancelled. Takes a `TimeProvider` for testing. | | `GrpcAuthContext` | Per-side handle passed to reader/writer code: the side's `IAccessTokenSource?` plus the application shutdown token. `GrpcAuthContext.None` is the no-auth instance. Two operations: `ValueTask AcquireCredentials(ct)` runs `TokenGate.WaitForToken` with `ct` linked to the shutdown token and returns `CallAuth(UserCredentials? Credentials, AccessTokenLease? Lease)` (both `null` without OAuth, so the client keeps using the connection string's credentials exactly as today), plus `ReportAccepted(CallAuth)` / `ReportFailure(CallAuth, Exception)` for long-lived calls that cannot use `Run`; `Run(Func> call, ct)` executes one bounded gRPC operation with token recovery (see "Token-aware calls"). Every call site passes `callAuth.Credentials` as the client method's `userCredentials` argument. Call sites that must remember which lease a result belongs to (the subscription) keep the `CallAuth` they were handed. | -| `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). `PermissionDenied` is not a token failure (retrying with a new token does not help), and is handled separately where it matters (scavenge filter). | +| `AuthFailure` | `static bool IsTokenFailure(Exception)`: true if the exception chain (`InnerException`, `AggregateException` members, `RpcException.Status.DebugException`) contains `OAuthTokenException`, the client's `NotAuthenticatedException` (which it converts `Unauthenticated` into), or an `RpcException` with `StatusCode.Unauthenticated` (token missing, expired or rejected). A companion `IsPermissionDenied` matches the client's `AccessDeniedException` or `RpcException` `PermissionDenied`. `PermissionDenied` is not a token failure (retrying with a new token does not help), and is handled separately where it matters (scavenge filter). | | `GrpcAuthentication` | `static void Apply(EventStoreClientSettings settings, IAccessTokenSource source)` — sets the sentinel `DefaultCredentials` and replaces `OperationOptions.GetAuthenticationHeaderValue` with the fallback hook (sentinel → fetch from source; anything else → `credentials.ToString()`). Also `IAccessTokenSource? Create(GrpcAuthOptions)` factory. | ### Wiring @@ -248,7 +248,7 @@ The loop has no attempt limit: a persistent rejection (for example a wrong audie - The resubscription starts from `FromAll.End`, as today. No position replay or catch-up barrier is needed, because nothing cached before or during the gap survives. - Before the first subscription the cache starts not-live, and the first successful subscribe marks it live. `ReadEvents` already awaits `Start()` before reading, so initial behaviour is unchanged. - `Start()` (called from `ReadEvents`) becomes `EnsureSubscribed(cancellationToken)`. Concurrent foreground and recovery callers share one attempt, and there is never more than one live subscription. -- `Realtime` gets an internal constructor taking a subscribe delegate `Func, Action, CallAuth, CancellationToken, Task>` (event handler, drop callback, auth), `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing and cache state, is unit-testable without a server. +- `Realtime` gets an internal constructor taking a subscribe delegate `Func, Action, CallAuth, CancellationToken, Task>` (the production delegate returns the `StreamSubscription`, which is `IDisposable`; `StreamSubscription` itself cannot be constructed in tests) (event handler, drop callback, auth), `GrpcAuthContext` and `TimeProvider`, so all of this, including drop timing and cache state, is unit-testable without a server. **Long-lived calls and token expiry.** Credentials are evaluated when a call starts, and gRPC has no way to swap the header of an open stream. There are two possible server behaviours, and both are handled: From ddf5620889a212074917a8296e71960c52defced Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:02:56 -0400 Subject: [PATCH 24/67] docs: implementation plan for OAuth authentication Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-10-03-oauth-authentication.md | 5044 +++++++++++++++++ 1 file changed, 5044 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-03-oauth-authentication.md diff --git a/docs/superpowers/plans/2026-10-03-oauth-authentication.md b/docs/superpowers/plans/2026-10-03-oauth-authentication.md new file mode 100644 index 00000000..6e6035f8 --- /dev/null +++ b/docs/superpowers/plans/2026-10-03-oauth-authentication.md @@ -0,0 +1,5044 @@ +# OAuth Authentication for KurrentDB gRPC Reader and Sink — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let Replicator authenticate its gRPC reader and sink to KurrentDB with OAuth 2.0 access tokens (client-credentials grant or a token file), configured independently per side, refreshing and recovering from token failures without restarts. + +**Architecture:** A per-side `IAccessTokenSource` (client credentials or token file) built on a shared `TokenState` that owns generations, rejected-token quarantine and the probe lease. A per-side `GrpcAuthContext` acquires a token *before* every gRPC call and passes it as the client's per-call `userCredentials`; `Run` retries token failures with backoff and reports acceptance/rejection by lease. A sentinel `DefaultCredentials` plus the client's public `GetAuthenticationHeaderValue` hook is a fallback only. `Realtime`, `StreamMetaCache`, `ScavengedEventsFilter`, `GrpcEventReader`, `GrpcEventWriter` and `Replicator` are adjusted so token failures never stall or corrupt replication. + +**Tech Stack:** .NET 9 (C#, nullable enabled), `EventStore.Client.Grpc.Streams` 23.3.8, Grpc.Net.Client, LibLog → Serilog, TUnit 0.19 + Microsoft Testing Platform, `Microsoft.Extensions.TimeProvider.Testing` (`FakeTimeProvider`), Helm, Astro/Starlight docs. + +**Spec:** `docs/superpowers/specs/2026-10-03-oauth-authentication-design.md` (reviewed clean by the Codex spec-review flow). Read it before starting; this plan argues from it and references its sections. + +## Global Constraints + +- Target framework stays `net9.0`; do not upgrade `EventStore.Client.Grpc.Streams` (stays `23.3.8`) or migrate to `KurrentDB.Client`. +- All new production code lives in `src/Kurrent.Replicator.KurrentDb/Auth/`, namespace `Kurrent.Replicator.KurrentDb.Auth`, except host settings (`src/replicator/Settings/`) and the listed edits to existing files. +- Config keys (camelCase in YAML, case-insensitive): `auth.type` ∈ `connectionString` (default) | `oauthClientCredentials` | `oauthTokenFile`; `tokenEndpoint`, `clientId`, `clientSecret`, `clientSecretFile`, `clientAssertionFile`, `clientAuthentication` (`post` default | `basic`), `scope`, `additionalParameters`, `defaultTokenLifetimeSeconds`, `refreshBeforeExpirySeconds` (default `300`), `tokenFile`, `tokenFileReloadSeconds` (default `30`). +- Fixed constants: token request timeout 30 s; failure cooldown 5 s; stale-token fallback margin 30 s; rejected-token quarantine 60 s; probe lease timeout 60 s; `expires_in` clamp 86 400 s; retry backoff 1 s, 2 s, 4 s … capped at 30 s; rate-limited warnings at most once per 60 s. +- Reserved `additionalParameters` keys: `grant_type`, `client_id`, `client_secret`, `client_assertion`, `client_assertion_type`, `scope`. +- Allowlisted OAuth `error` codes: `invalid_request`, `invalid_client`, `invalid_grant`, `unauthorized_client`, `unsupported_grant_type`, `invalid_scope`, `server_error`, `temporarily_unavailable`; anything else is reported as `non-standard error (omitted)`. +- Never log or put into exception messages: access tokens, client secrets, client assertions, token-request bodies, `error_description`, `error_uri` or raw token-endpoint response bodies (Debug-level `error_description` only after redaction to `***`). +- `connectionString` mode must behave exactly as today, apart from the `Realtime` bug fix, `StreamMetaCache` live-state and `Replicator` shutdown/drain fixes, which apply to all modes. +- Logging in new `Auth` classes resolves the logger per call (`static ILog Log => LogProvider.GetLogger(typeof(X))`), so tests that swap `Serilog.Log.Logger` capture output. +- Commit after each task with messages ending in `Co-Authored-By: Claude Opus 5.5 `. + +### Running unit tests locally + +The test project targets net9; this machine has only .NET 10 runtimes, and the net9 apphost is missing. Build without an apphost and roll forward: + +```bash +dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q +``` + +```bash +DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/AuthFailureTests/*" +``` + +Replace `AuthFailureTests` with the test class named in each step. Use `"/*/*/*/*"` for everything. The existing container tests (`ValuePartitionerTests`, `ChaserCheckpointSeedingTests`) need Docker; exclude them locally with `--treenode-filter "/*/Kurrent.Replicator.Tests.Auth/*/*"` (all new tests live in namespace `Kurrent.Replicator.Tests.Auth`). On a machine with a net9 runtime, `dotnet test` works too. + +## Review Focus + +1. **Token value containing characters that are legal in tokens but awkward in headers/forms** (e.g. `+`, `/`, `=`, `%`, spaces in secrets). Expect secrets to round-trip exactly through `post` and `basic` client authentication. Pinned in Task 4 (`Basic_auth_form_encodes_reserved_characters`, `Post_auth_encodes_reserved_characters`). +2. **Token file written non-atomically** (empty file or a trailing newline mid-rotation). Expect the previous token to keep working and the new one to be picked up on the next read. Pinned in Task 3 (`Empty_file_mid_rotation_keeps_previous_token`, `Trailing_newline_is_trimmed`). +3. **Clock jumps / very short token lifetimes** (`expires_in` of 1–60 s). Expect no tight refresh loop: refresh window capped at half the lifetime, and single-flight still holds. Pinned in Task 5 (`Very_short_lifetime_does_not_refresh_on_every_call`). +4. **Both sides pointing at the same cluster with different auth** (reader basic, sink OAuth to the same host). Expect fully independent clients and headers. Pinned in Task 11 (`Reader_basic_and_sink_oauth_send_independent_headers`). +5. **Environment-variable-only configuration** (no YAML `auth` section, everything via `REPLICATOR_SINK_AUTH_*`, including `ADDITIONALPARAMETERS_AUDIENCE`). Expect binding identical to YAML. Pinned in Task 13 (`Env_vars_bind_auth_including_additional_parameters`). + +--- + +## File Structure + +**Create (production, `src/Kurrent.Replicator.KurrentDb/Auth/`):** +- `AssemblyInfo.cs` (in `src/Kurrent.Replicator.KurrentDb/`) — `InternalsVisibleTo("Kurrent.Replicator.Tests")`. +- `OAuthTokenException.cs` — the single token-failure exception type. +- `AccessTokenLease.cs` — `AccessTokenLease` record struct and `IAccessTokenSource`. +- `AuthFailure.cs` — classify exceptions (`IsTokenFailure`, `IsPermissionDenied`). +- `GrpcAuthOptions.cs` — options record + enums. +- `RateLimitedWarning.cs` — "log at most once per interval" helper. +- `TokenState.cs` — generation, quarantine, probe lease (shared by both sources). +- `TokenFileSource.cs` — `oauthTokenFile`. +- `TokenEndpointClient.cs` — one token request: form building, response parsing, sanitised errors. +- `ClientCredentialsTokenSource.cs` — caching, refresh window, single-flight, cooldown, fallback. +- `TokenGate.cs` — wait-for-token with backoff. +- `GrpcAuthContext.cs` — `CallAuth`, `AcquireCredentials`, `Run`, `ReportAccepted`, `ReportFailure`. +- `GrpcAuthOptionsValidator.cs` — startup validation + warnings. +- `GrpcAuthentication.cs` — sentinel + fallback hook, token-source factory. + +**Modify (production):** +- `src/Kurrent.Replicator.KurrentDb/Kurrent.Replicator.KurrentDb.csproj` — `Using` for the Auth namespace. +- `src/Kurrent.Replicator.KurrentDb/Configurator.cs` — per-side auth. +- `src/Kurrent.Replicator.KurrentDb/GrpcEventWriter.cs` — calls through `auth.Run`. +- `src/Kurrent.Replicator.KurrentDb/GrpcEventReader.cs` — per-call creds, `GetLastPosition` via `Run`, wiring. +- `src/Kurrent.Replicator.KurrentDb/Realtime.cs` — rewritten (attempt model). +- `src/Kurrent.Replicator.KurrentDb/StreamMetaCache.cs` — fail-closed flag, live state, epoch. +- `src/Kurrent.Replicator.KurrentDb/EventFilters.cs` — `ScavengedEventsFilter` through `auth.Run`. +- `src/Kurrent.Replicator.KurrentDb/ConnectionExtensions.cs` — credentials + cancellation parameters. +- `src/Kurrent.Replicator/Replicator.cs` — prepare→sink write token, drain loop, resilient reporter. +- `src/replicator/Settings/ReplicatorSettings.cs`, new `src/replicator/Settings/AuthSettings.cs` — binding + mapping. +- `src/replicator/Settings/EnvConfigProvider.cs` — redaction. +- `src/replicator/Startup.cs` — wiring. +- `charts/replicator/templates/statefulset.yaml`, `charts/replicator/values.yaml`. +- Docs: `docs/src/content/docs/deployment/authentication.mdx` (new), `deployment/configuration.mdx`, `features/readers.mdx`, `features/sinks.mdx`, `CHANGELOG.md`. + +**Tests (`test/Kurrent.Replicator.Tests/Auth/`, namespace `Kurrent.Replicator.Tests.Auth`):** +- `Support/` — `LogCapture.cs`, `TimeDriver.cs`, `ControllableTokenSource.cs`, `FakeKurrentDbHandler.cs`, `StubTokenEndpoint.cs`, `TestEvents.cs`. +- One test file per unit: `AuthFailureTests.cs`, `TokenStateTests.cs`, `TokenFileSourceTests.cs`, `TokenEndpointClientTests.cs`, `ClientCredentialsTokenSourceTests.cs`, `TokenGateTests.cs`, `GrpcAuthContextTests.cs`, `GrpcAuthOptionsValidatorTests.cs`, `GrpcAuthenticationTests.cs`, `GrpcEventWriterAuthTests.cs`, `StreamMetaCacheTests.cs`, `ScavengedEventsFilterAuthTests.cs`, `RealtimeTests.cs`, `GrpcEventReaderAuthTests.cs`, `GrpcConfiguratorTests.cs`, `ReplicatorShutdownTests.cs`, `EnvConfigProviderTests.cs`, `AuthSettingsBindingTests.cs`. + +--- + +### Task 1: Foundations — test support, exception, lease contract, options record, failure classification + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/AssemblyInfo.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/OAuthTokenException.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/AccessTokenLease.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/AuthFailure.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthOptions.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/RateLimitedWarning.cs` +- Modify: `src/Kurrent.Replicator.KurrentDb/Kurrent.Replicator.KurrentDb.csproj` +- Modify: `test/Kurrent.Replicator.Tests/Kurrent.Replicator.Tests.csproj` +- Create: `test/Kurrent.Replicator.Tests/Auth/Support/LogCapture.cs` +- Create: `test/Kurrent.Replicator.Tests/Auth/Support/TimeDriver.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/AuthFailureTests.cs` + +**Interfaces:** +- Produces: + - `public sealed class OAuthTokenException(string message, Exception? inner = null) : Exception` + - `public readonly record struct AccessTokenLease(string Value, long Generation)` + - `public interface IAccessTokenSource { ValueTask GetAccessToken(CancellationToken ct); void Invalidate(AccessTokenLease lease); void ReportAccepted(AccessTokenLease lease); }` + - `public static class AuthFailure { bool IsTokenFailure(Exception); bool IsPermissionDenied(Exception); }` + - `public enum GrpcAuthType { ConnectionString, OAuthClientCredentials, OAuthTokenFile }`, `public enum ClientAuthenticationMethod { Post, Basic }`, `public sealed record GrpcAuthOptions` (properties listed in code below; `static GrpcAuthOptions Default`, `bool IsOAuth`) + - `public sealed class RateLimitedWarning(TimeProvider time, TimeSpan interval) { bool ShouldLog(); }` + - Test support: `LogCapture` (`[NotInParallel("global-logger")]` users), `TimeDriver.Drive(Task, FakeTimeProvider, TimeSpan? step = null)` / `Drive`. + +- [ ] **Step 1: Add assembly attribute, global using, test packages** + +`src/Kurrent.Replicator.KurrentDb/AssemblyInfo.cs` (the project sets `GenerateAssemblyInfo=false`, so the MSBuild `InternalsVisibleTo` item would be ignored): + +```csharp +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("Kurrent.Replicator.Tests")] +``` + +In `src/Kurrent.Replicator.KurrentDb/Kurrent.Replicator.KurrentDb.csproj`, add to the existing `` with `` items: + +```xml + + +``` + +In `test/Kurrent.Replicator.Tests/Kurrent.Replicator.Tests.csproj`, add to the package ``: + +```xml + +``` + +- [ ] **Step 2: Create the contract types** + +`src/Kurrent.Replicator.KurrentDb/Auth/OAuthTokenException.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +/// +/// A token could not be obtained, or the token source refuses to hand one out (quarantine, probe in progress). +/// Messages never contain tokens, secrets, assertions or provider-supplied free text. +/// +public sealed class OAuthTokenException(string message, Exception? inner = null) : Exception(message, inner); +``` + +`src/Kurrent.Replicator.KurrentDb/Auth/AccessTokenLease.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +/// A token value handed to exactly one call, tagged with the source generation it was issued under. +public readonly record struct AccessTokenLease(string Value, long Generation) { + public override string ToString() => $"AccessTokenLease(Generation={Generation})"; +} + +public interface IAccessTokenSource { + ValueTask GetAccessToken(CancellationToken ct); + + /// KurrentDB rejected the token carried by this lease. Synchronous, no I/O, never calls out. + void Invalidate(AccessTokenLease lease); + + /// A call carrying this lease succeeded. Synchronous, no I/O, never calls out. + void ReportAccepted(AccessTokenLease lease); +} +``` + +Note: `ToString` is overridden so a lease logged by accident never prints the token. + +`src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthOptions.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +public enum GrpcAuthType { ConnectionString, OAuthClientCredentials, OAuthTokenFile } + +public enum ClientAuthenticationMethod { Post, Basic } + +public sealed record GrpcAuthOptions { + public static GrpcAuthOptions Default { get; } = new(); + + public GrpcAuthType Type { get; init; } = GrpcAuthType.ConnectionString; + + // oauthClientCredentials + public string? TokenEndpoint { get; init; } + public string? ClientId { get; init; } + public string? ClientSecret { get; init; } + public string? ClientSecretFile { get; init; } + public string? ClientAssertionFile { get; init; } + public ClientAuthenticationMethod ClientAuthentication { get; init; } = ClientAuthenticationMethod.Post; + public string? Scope { get; init; } + public IReadOnlyDictionary AdditionalParameters { get; init; } = new Dictionary(); + public int? DefaultTokenLifetimeSeconds { get; init; } + public int RefreshBeforeExpirySeconds { get; init; } = 300; + + // oauthTokenFile + public string? TokenFile { get; init; } + public int TokenFileReloadSeconds { get; init; } = 30; + + public bool IsOAuth => Type != GrpcAuthType.ConnectionString; + + public override string ToString() => $"GrpcAuthOptions(Type={Type})"; // never print secrets +} +``` + +`src/Kurrent.Replicator.KurrentDb/Auth/RateLimitedWarning.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +/// Returns true the first time and then at most once per interval. +public sealed class RateLimitedWarning(TimeProvider time, TimeSpan interval) { + readonly object _lock = new(); + DateTimeOffset? _last; + + public bool ShouldLog() { + lock (_lock) { + var now = time.GetUtcNow(); + + if (_last is { } last && now - last < interval) return false; + + _last = now; + + return true; + } + } + + public void Reset() { + lock (_lock) _last = null; + } +} +``` + +- [ ] **Step 3: Create test support** + +`test/Kurrent.Replicator.Tests/Auth/Support/LogCapture.cs`: + +```csharp +using System.Collections.Concurrent; +using Kurrent.Replicator.Tests.Logging; +using Serilog; +using Serilog.Core; +using Serilog.Events; + +namespace Kurrent.Replicator.Tests.Auth.Support; + +/// +/// Captures everything logged through Serilog's global logger while alive. Tests using it must be +/// [NotInParallel("global-logger")] because Log.Logger is process-global. +/// +public sealed class LogCapture : ILogEventSink, IDisposable { + readonly ConcurrentQueue _events = new(); + readonly ILogger _previous = Log.Logger; + + public LogCapture() { + Log.Logger = new LoggerConfiguration() + .MinimumLevel.Verbose() + .WriteTo.Sink(this) + .WriteTo.TestOutput() + .CreateLogger(); + } + + public void Emit(LogEvent logEvent) => _events.Enqueue(logEvent); + + public IReadOnlyList<(LogEventLevel Level, string Text)> Events + => _events.Select(e => (e.Level, e.RenderMessage() + " " + e.Exception)).ToList(); + + public IEnumerable TextAtOrAbove(LogEventLevel level) => Events.Where(e => e.Level >= level).Select(e => e.Text); + + public string AllText => string.Join("\n", Events.Select(e => e.Text)); + + public void Dispose() => Log.Logger = _previous; +} +``` + +`test/Kurrent.Replicator.Tests/Auth/Support/TimeDriver.cs`: + +```csharp +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth.Support; + +public static class TimeDriver { + /// Advances fake time in steps until the task completes (or the step budget runs out). + public static async Task Drive(Task task, FakeTimeProvider time, TimeSpan? step = null, int maxSteps = 5000) { + await Drive((Task)task, time, step, maxSteps); + + return await task; + } + + public static async Task Drive(Task task, FakeTimeProvider time, TimeSpan? step = null, int maxSteps = 5000) { + for (var i = 0; i < maxSteps && !task.IsCompleted; i++) { + time.Advance(step ?? TimeSpan.FromSeconds(1)); + await Task.Delay(2); + } + + await task.WaitAsync(TimeSpan.FromSeconds(10)); + } + + /// Advances fake time until the condition holds. + public static async Task Until(Func condition, FakeTimeProvider time, TimeSpan? step = null, int maxSteps = 5000) { + for (var i = 0; i < maxSteps && !condition(); i++) { + time.Advance(step ?? TimeSpan.FromSeconds(1)); + await Task.Delay(2); + } + + if (!condition()) throw new TimeoutException("Condition not reached"); + } +} +``` + +- [ ] **Step 4: Write the failing AuthFailure tests** + +`test/Kurrent.Replicator.Tests/Auth/AuthFailureTests.cs`: + +```csharp +using EventStore.Client; +using Grpc.Core; +using Kurrent.Replicator.KurrentDb.Auth; + +namespace Kurrent.Replicator.Tests.Auth; + +public class AuthFailureTests { + static RpcException Rpc(StatusCode code, Exception? debug = null) => new(new Status(code, "x", debug)); + + [Test] + public async Task Token_failures_are_recognised() { + await Assert.That(AuthFailure.IsTokenFailure(new OAuthTokenException("x"))).IsTrue(); + await Assert.That(AuthFailure.IsTokenFailure(Rpc(StatusCode.Unauthenticated))).IsTrue(); + await Assert.That(AuthFailure.IsTokenFailure(new NotAuthenticatedException("x", Rpc(StatusCode.Unauthenticated)))).IsTrue(); + await Assert.That(AuthFailure.IsTokenFailure(Rpc(StatusCode.Internal, new OAuthTokenException("x")))).IsTrue(); + await Assert.That(AuthFailure.IsTokenFailure(new AggregateException(new InvalidOperationException(), new OAuthTokenException("x")))).IsTrue(); + await Assert.That(AuthFailure.IsTokenFailure(new InvalidOperationException("outer", new OAuthTokenException("x")))).IsTrue(); + } + + [Test] + public async Task Non_token_failures_are_not_token_failures() { + await Assert.That(AuthFailure.IsTokenFailure(Rpc(StatusCode.PermissionDenied))).IsFalse(); + await Assert.That(AuthFailure.IsTokenFailure(new AccessDeniedException("x", Rpc(StatusCode.PermissionDenied)))).IsFalse(); + await Assert.That(AuthFailure.IsTokenFailure(Rpc(StatusCode.Unavailable))).IsFalse(); + await Assert.That(AuthFailure.IsTokenFailure(new OperationCanceledException())).IsFalse(); + } + + [Test] + public async Task Permission_denied_is_recognised() { + await Assert.That(AuthFailure.IsPermissionDenied(Rpc(StatusCode.PermissionDenied))).IsTrue(); + await Assert.That(AuthFailure.IsPermissionDenied(new AccessDeniedException("x", Rpc(StatusCode.PermissionDenied)))).IsTrue(); + await Assert.That(AuthFailure.IsPermissionDenied(Rpc(StatusCode.Unauthenticated))).IsFalse(); + } + + [Test] + public async Task Self_referencing_chains_terminate() { + var agg = new AggregateException(new AggregateException(new AggregateException(new InvalidOperationException()))); + await Assert.That(AuthFailure.IsTokenFailure(agg)).IsFalse(); + } +} +``` + +- [ ] **Step 5: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: build FAILS with `The name 'AuthFailure' does not exist`. + +- [ ] **Step 6: Implement AuthFailure** + +`src/Kurrent.Replicator.KurrentDb/Auth/AuthFailure.cs`: + +```csharp +using Grpc.Core; + +namespace Kurrent.Replicator.KurrentDb.Auth; + +public static class AuthFailure { + /// Token missing, expired or rejected: retrying with a (new) token may help. + public static bool IsTokenFailure(Exception e) + => Chain(e).Any(x => x is OAuthTokenException or NotAuthenticatedException or RpcException { StatusCode: StatusCode.Unauthenticated }); + + /// Authorization decision: a new token does not help. + public static bool IsPermissionDenied(Exception e) + => Chain(e).Any(x => x is AccessDeniedException or RpcException { StatusCode: StatusCode.PermissionDenied }); + + static IEnumerable Chain(Exception root) { + var pending = new Stack(); + pending.Push(root); + var visited = 0; + + while (pending.Count > 0 && visited++ < 64) { + var e = pending.Pop(); + + yield return e; + + if (e is AggregateException agg) { + foreach (var inner in agg.InnerExceptions) pending.Push(inner); + } + else if (e.InnerException is { } inner) { + pending.Push(inner); + } + + if (e is RpcException rpc && rpc.Status.DebugException is { } debug) pending.Push(debug); + } + } +} +``` + +- [ ] **Step 7: Run tests to verify they pass** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/AuthFailureTests/*"` +Expected: 4 tests PASS. + +- [ ] **Step 8: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb test/Kurrent.Replicator.Tests +git commit -m "feat(auth): token-failure classification and auth contracts + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 2: TokenState — generations, quarantine, probe lease + +The heart of the rejected-token rules ("Rejected tokens" in the spec). Both token sources delegate to it. + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/TokenState.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/TokenStateTests.cs` + +**Interfaces:** +- Consumes: `AccessTokenLease`, `OAuthTokenException` (Task 1). +- Produces: `internal sealed class TokenState(TimeProvider time)` with: + - `AccessTokenLease Accept(string value)` — call with every value a source is about to hand out (fresh fetch, file read, or cache hit). Returns the lease or throws `OAuthTokenException` (quarantined / probe held). + - `bool Invalidate(AccessTokenLease lease)` — true if applied (current generation). + - `bool ReportAccepted(AccessTokenLease lease)` — true if applied. + - `bool IsRejected(string value)` — the value is under an active rejection record. + - `bool HasRejection` — any rejection record active. + - `long Generation`. + - `static readonly TimeSpan Quarantine = 60 s`, `ProbeLeaseTimeout = 60 s`. + +- [ ] **Step 1: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/TokenStateTests.cs`: + +```csharp +using Kurrent.Replicator.KurrentDb.Auth; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class TokenStateTests { + readonly FakeTimeProvider _time = new(); + TokenState NewState() => new(_time); + + [Test] + public async Task Same_value_does_not_bump_generation() { + var s = NewState(); + var a1 = s.Accept("A"); + var a2 = s.Accept("A"); + await Assert.That(a2.Generation).IsEqualTo(a1.Generation); + } + + [Test] + public async Task New_value_bumps_generation() { + var s = NewState(); + var a = s.Accept("A"); + var b = s.Accept("B"); + await Assert.That(b.Generation).IsGreaterThan(a.Generation); + } + + [Test] + public async Task Invalidate_current_starts_quarantine() { + var s = NewState(); + var a = s.Accept("A"); + await Assert.That(s.Invalidate(a)).IsTrue(); + await Assert.That(() => s.Accept("A")).Throws(); + _time.Advance(TimeSpan.FromSeconds(59)); + await Assert.That(() => s.Accept("A")).Throws(); + } + + [Test] + public async Task Different_value_is_accepted_immediately_and_clears_rejection() { + var s = NewState(); + s.Invalidate(s.Accept("A")); + var b = s.Accept("B"); + await Assert.That(b.Value).IsEqualTo("B"); + await Assert.That(s.HasRejection).IsFalse(); + } + + [Test] + public async Task Same_value_reload_does_not_lose_rejection() { + var s = NewState(); + var call1 = s.Accept("A"); + s.Accept("A"); // routine reload / reissue of the same value + s.Accept("A"); + await Assert.That(s.Invalidate(call1)).IsTrue(); + await Assert.That(() => s.Accept("A")).Throws(); + } + + [Test] + public async Task After_quarantine_exactly_one_probe_gets_the_value() { + var s = NewState(); + s.Invalidate(s.Accept("A")); + _time.Advance(TimeSpan.FromSeconds(61)); + + var results = Enumerable.Range(0, 10).Select(_ => { + try { return (AccessTokenLease?)s.Accept("A"); } catch (OAuthTokenException) { return null; } + }).ToList(); + + await Assert.That(results.Count(r => r is not null)).IsEqualTo(1); + } + + [Test] + public async Task Accepted_probe_releases_value_to_everyone() { + var s = NewState(); + s.Invalidate(s.Accept("A")); + _time.Advance(TimeSpan.FromSeconds(61)); + var probe = s.Accept("A"); + await Assert.That(s.ReportAccepted(probe)).IsTrue(); + await Assert.That(s.Accept("A").Value).IsEqualTo("A"); + await Assert.That(s.HasRejection).IsFalse(); + } + + [Test] + public async Task Rejected_probe_restarts_quarantine() { + var s = NewState(); + s.Invalidate(s.Accept("A")); + _time.Advance(TimeSpan.FromSeconds(61)); + var probe = s.Accept("A"); + await Assert.That(s.Invalidate(probe)).IsTrue(); + _time.Advance(TimeSpan.FromSeconds(30)); + await Assert.That(() => s.Accept("A")).Throws(); + } + + [Test] + public async Task Unreported_probe_lease_expires_and_counts_as_new_quarantine() { + var s = NewState(); + s.Invalidate(s.Accept("A")); + _time.Advance(TimeSpan.FromSeconds(61)); + var probe = s.Accept("A"); + _time.Advance(TimeSpan.FromSeconds(61)); // lease expired -> new quarantine starts now + await Assert.That(() => s.Accept("A")).Throws(); + await Assert.That(s.ReportAccepted(probe)).IsFalse(); // stale after expiry + _time.Advance(TimeSpan.FromSeconds(61)); + await Assert.That(s.Accept("A").Value).IsEqualTo("A"); // next probe + } + + [Test] + public async Task Stale_reports_are_ignored() { + var s = NewState(); + var call1 = s.Accept("A"); // gen 1 + s.Invalidate(s.Accept("A")); // another call rejects -> quarantine, gen 2 + await Assert.That(s.ReportAccepted(call1)).IsFalse(); // late success from gen 1 does not clear quarantine + await Assert.That(() => s.Accept("A")).Throws(); + + _time.Advance(TimeSpan.FromSeconds(61)); + var probe = s.Accept("A"); // gen 3 + s.ReportAccepted(probe); // gen 4 + await Assert.That(s.Invalidate(call1)).IsFalse(); // late rejection from gen 1 ignored + await Assert.That(s.Accept("A").Value).IsEqualTo("A"); + } + + [Test] + public async Task IsRejected_reports_the_quarantined_value() { + var s = NewState(); + s.Invalidate(s.Accept("A")); + await Assert.That(s.IsRejected("A")).IsTrue(); + await Assert.That(s.IsRejected("B")).IsFalse(); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, `TokenState` not found. + +- [ ] **Step 3: Implement TokenState** + +`src/Kurrent.Replicator.KurrentDb/Auth/TokenState.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +/// +/// Generation, rejected-token quarantine and probe lease, shared by both token sources. +/// All members are synchronous and hold only this object's lock. +/// +sealed class TokenState(TimeProvider time) { + public static readonly TimeSpan Quarantine = TimeSpan.FromSeconds(60); + public static readonly TimeSpan ProbeLeaseTimeout = TimeSpan.FromSeconds(60); + + readonly object _lock = new(); + + string? _current; + long _generation; + string? _rejected; + DateTimeOffset _rejectedAt; + bool _probeHeld; + DateTimeOffset _probeGrantedAt; + + public long Generation { get { lock (_lock) return _generation; } } + + public bool HasRejection { get { lock (_lock) return _rejected != null; } } + + public bool IsRejected(string value) { + lock (_lock) return _rejected == value; + } + + public AccessTokenLease Accept(string value) { + lock (_lock) { + var now = time.GetUtcNow(); + ExpireProbe(now); + + if (_rejected != null && value == _rejected) { + if (now - _rejectedAt < Quarantine) + throw new OAuthTokenException("The token source returned a token the server just rejected; waiting for a new token"); + + if (_probeHeld) + throw new OAuthTokenException("A previously rejected token is being re-tested by another call; waiting for the result"); + + _probeHeld = true; + _probeGrantedAt = now; + _current = value; + _generation++; + + return new(value, _generation); + } + + if (_rejected != null) { + _rejected = null; + _probeHeld = false; + } + + if (value != _current) { + _current = value; + _generation++; + } + + return new(value, _generation); + } + } + + public bool Invalidate(AccessTokenLease lease) { + lock (_lock) { + if (lease.Generation != _generation) return false; + + _rejected = lease.Value; + _rejectedAt = time.GetUtcNow(); + _probeHeld = false; + _current = null; + _generation++; + + return true; + } + } + + public bool ReportAccepted(AccessTokenLease lease) { + lock (_lock) { + if (lease.Generation != _generation) return false; + + if (_probeHeld && _rejected == lease.Value) { + _rejected = null; + _probeHeld = false; + _generation++; + } + + return true; + } + } + + void ExpireProbe(DateTimeOffset now) { + if (!_probeHeld || now - _probeGrantedAt < ProbeLeaseTimeout) return; + + _probeHeld = false; + _rejectedAt = now; // expiry counts as a new quarantine + _current = null; + _generation++; + } +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/TokenStateTests/*"` +Expected: 11 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/Auth/TokenState.cs test/Kurrent.Replicator.Tests/Auth/TokenStateTests.cs +git commit -m "feat(auth): token generations, quarantine and probe lease + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 3: TokenFileSource + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/TokenFileSource.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/TokenFileSourceTests.cs` + +**Interfaces:** +- Consumes: `TokenState`, `IAccessTokenSource`, `OAuthTokenException`. +- Produces: `public sealed class TokenFileSource(string path, TimeSpan reloadInterval, TimeProvider time, string side) : IAccessTokenSource`. + +Behaviour (spec "Token file flow" + "Rejected tokens"): read on first call and when `now - lastReadAt >= reloadInterval`; also on every call while a rejection record is active (so a rotated file is picked up immediately after a rejection); trim; missing/empty → keep previous token unless it was rejected, else throw. + +- [ ] **Step 1: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/TokenFileSourceTests.cs`: + +```csharp +using Kurrent.Replicator.KurrentDb.Auth; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class TokenFileSourceTests { + readonly FakeTimeProvider _time = new(); + readonly string _path = Path.Combine(Path.GetTempPath(), $"replicator-token-{Guid.NewGuid():N}"); + + TokenFileSource NewSource(int reloadSeconds = 30) => new(_path, TimeSpan.FromSeconds(reloadSeconds), _time, "sink"); + + [After(Test)] + public void Cleanup() { + if (File.Exists(_path)) File.Delete(_path); + } + + [Test] + public async Task Reads_and_trims() { + await File.WriteAllTextAsync(_path, " A \n"); + var lease = await NewSource().GetAccessToken(default); + await Assert.That(lease.Value).IsEqualTo("A"); + } + + [Test] + public async Task Trailing_newline_is_trimmed() { + await File.WriteAllTextAsync(_path, "eyJhbGciOi.abc.def\r\n"); + await Assert.That((await NewSource().GetAccessToken(default)).Value).IsEqualTo("eyJhbGciOi.abc.def"); + } + + [Test] + public async Task Missing_file_without_previous_token_throws() { + await Assert.That(async () => await NewSource().GetAccessToken(default)).Throws(); + } + + [Test] + public async Task Does_not_reread_before_interval() { + await File.WriteAllTextAsync(_path, "A"); + var source = NewSource(); + await source.GetAccessToken(default); + await File.WriteAllTextAsync(_path, "B"); + _time.Advance(TimeSpan.FromSeconds(29)); + await Assert.That((await source.GetAccessToken(default)).Value).IsEqualTo("A"); + _time.Advance(TimeSpan.FromSeconds(1)); + await Assert.That((await source.GetAccessToken(default)).Value).IsEqualTo("B"); + } + + [Test] + public async Task Empty_file_mid_rotation_keeps_previous_token() { + await File.WriteAllTextAsync(_path, "A"); + var source = NewSource(); + await source.GetAccessToken(default); + await File.WriteAllTextAsync(_path, ""); + _time.Advance(TimeSpan.FromSeconds(30)); + await Assert.That((await source.GetAccessToken(default)).Value).IsEqualTo("A"); + File.Delete(_path); + _time.Advance(TimeSpan.FromSeconds(30)); + await Assert.That((await source.GetAccessToken(default)).Value).IsEqualTo("A"); + } + + [Test] + public async Task Invalidate_forces_immediate_reread_and_accepts_new_value() { + await File.WriteAllTextAsync(_path, "A"); + var source = NewSource(); + var a = await source.GetAccessToken(default); + await File.WriteAllTextAsync(_path, "B"); + source.Invalidate(a); + await Assert.That((await source.GetAccessToken(default)).Value).IsEqualTo("B"); + } + + [Test] + public async Task After_invalidate_missing_or_same_value_throws_during_quarantine_then_probes() { + await File.WriteAllTextAsync(_path, "A"); + var source = NewSource(); + source.Invalidate(await source.GetAccessToken(default)); + + await Assert.That(async () => await source.GetAccessToken(default)).Throws(); // still A + File.Delete(_path); + await Assert.That(async () => await source.GetAccessToken(default)).Throws(); // missing, no fallback to A + + await File.WriteAllTextAsync(_path, "A"); + _time.Advance(TimeSpan.FromSeconds(61)); + await Assert.That((await source.GetAccessToken(default)).Value).IsEqualTo("A"); // probe + } + + [Test] + public async Task Same_value_reload_does_not_lose_rejection() { + await File.WriteAllTextAsync(_path, "A"); + var source = NewSource(); + var call1 = await source.GetAccessToken(default); + _time.Advance(TimeSpan.FromSeconds(30)); + await source.GetAccessToken(default); // timer re-read of unchanged A + source.Invalidate(call1); + await Assert.That(async () => await source.GetAccessToken(default)).Throws(); + } + + [Test] + public async Task Honours_cancellation() { + await File.WriteAllTextAsync(_path, "A"); + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + await Assert.That(async () => await NewSource().GetAccessToken(cts.Token)).Throws(); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, `TokenFileSource` not found. + +- [ ] **Step 3: Implement** + +`src/Kurrent.Replicator.KurrentDb/Auth/TokenFileSource.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +public sealed class TokenFileSource(string path, TimeSpan reloadInterval, TimeProvider time, string side) : IAccessTokenSource { + static ILog Log => LogProvider.GetLogger(typeof(TokenFileSource)); + + readonly TokenState _state = new(time); + readonly object _lock = new(); + + string? _cached; + DateTimeOffset? _lastReadAt; + bool _forceReload = true; + + public ValueTask GetAccessToken(CancellationToken ct) { + ct.ThrowIfCancellationRequested(); + string? value; + + lock (_lock) { + var now = time.GetUtcNow(); + + if (_forceReload || _state.HasRejection || _lastReadAt is null || now - _lastReadAt >= reloadInterval) { + var read = TryRead(); + _lastReadAt = now; + _forceReload = false; + + if (read != null) { + _cached = read; + } + else if (_cached != null && !_state.IsRejected(_cached)) { + Log.Warn("{Side}: token file {Path} is missing or empty; keeping the previously read token", side, path); + } + else { + _cached = null; + } + } + + value = _cached; + } + + if (value == null) throw new OAuthTokenException($"{side}: token file {path} is missing or empty"); + + return ValueTask.FromResult(_state.Accept(value)); + } + + public void Invalidate(AccessTokenLease lease) { + if (!_state.Invalidate(lease)) { + Log.Debug("{Side}: ignoring stale token rejection report", side); + + return; + } + + lock (_lock) { + _forceReload = true; + if (_cached == lease.Value) _cached = null; + } + } + + public void ReportAccepted(AccessTokenLease lease) => _state.ReportAccepted(lease); + + string? TryRead() { + try { + var text = File.ReadAllText(path).Trim(); + + return text.Length == 0 ? null : text; + } catch (IOException) { + return null; + } catch (UnauthorizedAccessException) { + return null; + } + } +} +``` + +Note: `FileNotFoundException` and `DirectoryNotFoundException` derive from `IOException`. + +Note on the "missing after rejection" case: `Invalidate` nulls `_cached`, so a missing file afterwards yields `value == null` and throws. A file still holding the rejected value reaches `_state.Accept`, which throws during quarantine and grants the probe after it. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/TokenFileSourceTests/*"` +Expected: 9 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/Auth/TokenFileSource.cs test/Kurrent.Replicator.Tests/Auth/TokenFileSourceTests.cs +git commit -m "feat(auth): token file source + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 4: TokenEndpointClient — one token request, strict parsing, sanitised errors + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/TokenEndpointClient.cs` +- Create: `test/Kurrent.Replicator.Tests/Auth/Support/StubTokenEndpoint.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/TokenEndpointClientTests.cs` + +**Interfaces:** +- Consumes: `GrpcAuthOptions`, `ClientAuthenticationMethod`, `OAuthTokenException`. +- Produces: + - `internal sealed record TokenResponse(string AccessToken, TimeSpan Lifetime)` + - `internal sealed class TokenEndpointClient : IDisposable` with ctor `(GrpcAuthOptions options, HttpMessageHandler handler, string side)`, `Task RequestToken(CancellationToken ct)`, `string EndpointHost`, `static SocketsHttpHandler CreateDefaultHandler()` (with `AllowAutoRedirect = false`). + - `StubTokenEndpoint : HttpMessageHandler` (tests): `Func Respond`, `List<(HttpRequestMessage Request, string Body)> Requests`, helpers `static HttpResponseMessage Json(HttpStatusCode, string)`, `static string Token(string value, object? expiresIn = 3600, string? tokenType = "Bearer")`, `TaskCompletionSource? Gate` (when set, requests wait for it). + +- [ ] **Step 1: Create the stub token endpoint** + +`test/Kurrent.Replicator.Tests/Auth/Support/StubTokenEndpoint.cs`: + +```csharp +using System.Collections.Concurrent; +using System.Net; +using System.Text; +using System.Text.Json; + +namespace Kurrent.Replicator.Tests.Auth.Support; + +public sealed class StubTokenEndpoint : HttpMessageHandler { + public Func Respond { get; set; } = (_, _) => Json(HttpStatusCode.OK, Token("A")); + + public ConcurrentQueue<(HttpRequestMessage Request, string Body)> Requests { get; } = new(); + + /// When set, every request waits for it before responding. + public TaskCompletionSource? Gate { get; set; } + + public int Count => Requests.Count; + + protected override async Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) { + var body = request.Content == null ? "" : await request.Content.ReadAsStringAsync(cancellationToken); + Requests.Enqueue((request, body)); + + if (Gate is { } gate) await gate.Task.WaitAsync(cancellationToken); + + return Respond(request, body); + } + + public static HttpResponseMessage Json(HttpStatusCode status, string json) + => new(status) { Content = new StringContent(json, Encoding.UTF8, "application/json") }; + + public static string Token(string value, object? expiresIn = 3600, string? tokenType = "Bearer") { + var dict = new Dictionary { ["access_token"] = value }; + if (tokenType != null) dict["token_type"] = tokenType; + if (expiresIn != null) dict["expires_in"] = expiresIn; + + return JsonSerializer.Serialize(dict); + } + + public static Dictionary Form(string body) + => body.Split('&', StringSplitOptions.RemoveEmptyEntries) + .Select(p => p.Split('=', 2)) + .ToDictionary(p => WebUtility.UrlDecode(p[0]), p => WebUtility.UrlDecode(p.Length > 1 ? p[1] : "")); +} +``` + +- [ ] **Step 2: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/TokenEndpointClientTests.cs`: + +```csharp +using System.Net; +using System.Text; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; + +namespace Kurrent.Replicator.Tests.Auth; + +public class TokenEndpointClientTests { + readonly StubTokenEndpoint _stub = new(); + + static GrpcAuthOptions Options(Func? configure = null) { + var o = new GrpcAuthOptions { + Type = GrpcAuthType.OAuthClientCredentials, + TokenEndpoint = "https://idp.example.com/oauth2/token", + ClientId = "replicator", + ClientSecret = "s3cr3t", + Scope = "api://kurrentdb/.default" + }; + + return configure?.Invoke(o) ?? o; + } + + TokenEndpointClient Client(GrpcAuthOptions o) => new(o, _stub, "sink"); + + [Test] + public async Task Post_auth_sends_form_with_secret() { + var o = Options(x => x with { AdditionalParameters = new Dictionary { ["audience"] = "kdb" } }); + var r = await Client(o).RequestToken(default); + + await Assert.That(r.AccessToken).IsEqualTo("A"); + await Assert.That(r.Lifetime).IsEqualTo(TimeSpan.FromSeconds(3600)); + var (req, body) = _stub.Requests.Single(); + var form = StubTokenEndpoint.Form(body); + await Assert.That(req.Method).IsEqualTo(HttpMethod.Post); + await Assert.That(req.Content!.Headers.ContentType!.MediaType).IsEqualTo("application/x-www-form-urlencoded"); + await Assert.That(form["grant_type"]).IsEqualTo("client_credentials"); + await Assert.That(form["client_id"]).IsEqualTo("replicator"); + await Assert.That(form["client_secret"]).IsEqualTo("s3cr3t"); + await Assert.That(form["scope"]).IsEqualTo("api://kurrentdb/.default"); + await Assert.That(form["audience"]).IsEqualTo("kdb"); + await Assert.That(req.Headers.Authorization).IsNull(); + } + + [Test] + public async Task Post_auth_encodes_reserved_characters() { + await Client(Options(x => x with { ClientSecret = "a+b/c=d%e f&g" })).RequestToken(default); + var form = StubTokenEndpoint.Form(_stub.Requests.Single().Body); + await Assert.That(form["client_secret"]).IsEqualTo("a+b/c=d%e f&g"); + } + + [Test] + public async Task Basic_auth_form_encodes_reserved_characters() { + await Client(Options(x => x with { ClientAuthentication = ClientAuthenticationMethod.Basic, ClientId = "my app", ClientSecret = "a+b:c" })).RequestToken(default); + var (req, body) = _stub.Requests.Single(); + var form = StubTokenEndpoint.Form(body); + await Assert.That(form.ContainsKey("client_secret")).IsFalse(); + await Assert.That(form.ContainsKey("client_id")).IsFalse(); + await Assert.That(req.Headers.Authorization!.Scheme).IsEqualTo("Basic"); + var decoded = Encoding.UTF8.GetString(Convert.FromBase64String(req.Headers.Authorization.Parameter!)); + await Assert.That(decoded).IsEqualTo("my+app:a%2Bb%3Ac"); // RFC 6749 2.3.1: form-urlencode each part + } + + [Test] + public async Task Assertion_is_reread_on_every_request() { + var file = Path.GetTempFileName(); + await File.WriteAllTextAsync(file, "jwt-1\n"); + var client = Client(Options(x => x with { ClientSecret = null, ClientAssertionFile = file })); + await client.RequestToken(default); + await File.WriteAllTextAsync(file, "jwt-2"); + await client.RequestToken(default); + + var forms = _stub.Requests.Select(r => StubTokenEndpoint.Form(r.Body)).ToList(); + await Assert.That(forms[0]["client_assertion"]).IsEqualTo("jwt-1"); + await Assert.That(forms[1]["client_assertion"]).IsEqualTo("jwt-2"); + await Assert.That(forms[0]["client_assertion_type"]).IsEqualTo("urn:ietf:params:oauth:client-assertion-type:jwt-bearer"); + await Assert.That(forms[0]["client_id"]).IsEqualTo("replicator"); + await Assert.That(forms[0].ContainsKey("client_secret")).IsFalse(); + File.Delete(file); + } + + [Test] + public async Task Secret_file_is_read_once_and_trimmed() { + var file = Path.GetTempFileName(); + await File.WriteAllTextAsync(file, "from-file\n"); + var client = Client(Options(x => x with { ClientSecret = null, ClientSecretFile = file })); + await File.WriteAllTextAsync(file, "changed"); + await client.RequestToken(default); + await Assert.That(StubTokenEndpoint.Form(_stub.Requests.Single().Body)["client_secret"]).IsEqualTo("from-file"); + File.Delete(file); + } + + [Test] + [Arguments("{\"token_type\":\"Bearer\",\"expires_in\":3600}")] // no access_token + [Arguments("{\"access_token\":\"\",\"token_type\":\"Bearer\",\"expires_in\":3600}")] // empty + [Arguments("{\"access_token\":\"A\",\"expires_in\":3600}")] // no token_type + [Arguments("{\"access_token\":\"A\",\"token_type\":\"mac\",\"expires_in\":3600}")] // wrong type + [Arguments("{\"access_token\":\"A\",\"token_type\":\"Bearer\",\"expires_in\":0}")] + [Arguments("{\"access_token\":\"A\",\"token_type\":\"Bearer\",\"expires_in\":-5}")] + [Arguments("{\"access_token\":\"A\",\"token_type\":\"Bearer\",\"expires_in\":1.5}")] + [Arguments("{\"access_token\":\"A\",\"token_type\":\"Bearer\",\"expires_in\":\"soon\"}")] + [Arguments("{\"access_token\":\"A\",\"token_type\":\"Bearer\",\"expires_in\":99999999999999999999}")] + [Arguments("{\"access_token\":\"A\",\"token_type\":\"Bearer\"}")] // no expires_in, no default + [Arguments("not json")] + public async Task Invalid_responses_are_rejected(string json) { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.OK, json); + await Assert.That(async () => await Client(Options()).RequestToken(default)).Throws(); + } + + [Test] + public async Task Numeric_string_expires_in_and_case_insensitive_bearer_are_accepted() { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.OK, StubTokenEndpoint.Token("A", "120", "bearer")); + var r = await Client(Options()).RequestToken(default); + await Assert.That(r.Lifetime).IsEqualTo(TimeSpan.FromSeconds(120)); + } + + [Test] + public async Task Missing_expires_in_uses_configured_default() { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.OK, StubTokenEndpoint.Token("A", expiresIn: null)); + var r = await Client(Options(x => x with { DefaultTokenLifetimeSeconds = 900 })).RequestToken(default); + await Assert.That(r.Lifetime).IsEqualTo(TimeSpan.FromSeconds(900)); + } + + [Test] + public async Task Lifetime_over_24h_is_clamped() { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.OK, StubTokenEndpoint.Token("A", 200000)); + var r = await Client(Options()).RequestToken(default); + await Assert.That(r.Lifetime).IsEqualTo(TimeSpan.FromSeconds(86400)); + } + + [Test] + public async Task Redirect_is_an_error_and_not_followed() { + _stub.Respond = (_, _) => new HttpResponseMessage(HttpStatusCode.TemporaryRedirect) { Headers = { Location = new Uri("https://evil.example.com/steal") } }; + var ex = await Assert.That(async () => await Client(Options()).RequestToken(default)).Throws(); + await Assert.That(ex!.Message).Contains("redirect"); + await Assert.That(_stub.Count).IsEqualTo(1); + await Assert.That(TokenEndpointClient.CreateDefaultHandler().AllowAutoRedirect).IsFalse(); + } + + [Test] + public async Task Allowlisted_error_and_integer_error_codes_are_reported() { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.BadRequest, "{\"error\":\"invalid_client\",\"error_codes\":[7000215],\"error_description\":\"AADSTS7000215: Invalid client secret\"}"); + var ex = await Assert.That(async () => await Client(Options()).RequestToken(default)).Throws(); + await Assert.That(ex!.Message).Contains("400"); + await Assert.That(ex.Message).Contains("invalid_client"); + await Assert.That(ex.Message).Contains("7000215"); + await Assert.That(ex.Message).Contains("idp.example.com"); + await Assert.That(ex.Message).DoesNotContain("AADSTS7000215: Invalid"); + } + + [Test] + public async Task Error_echoing_the_secret_is_omitted() { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.Unauthorized, "{\"error\":\"s3cr3t\",\"error_description\":\"bad s3cr3t\",\"error_uri\":\"https://x/s3cr3t\"}"); + var ex = await Assert.That(async () => await Client(Options()).RequestToken(default)).Throws(); + await Assert.That(ex!.Message).DoesNotContain("s3cr3t"); + await Assert.That(ex.Message).Contains("non-standard error (omitted)"); + } +} +``` + +- [ ] **Step 3: Add the log-redaction test (separate class because it swaps the global logger)** + +Append to the same file: + +```csharp +[NotInParallel("global-logger")] +public class TokenEndpointClientLoggingTests { + [Test] + public async Task Provider_text_is_never_logged_above_debug_and_is_redacted_at_debug() { + using var logs = new LogCapture(); + var assertionFile = Path.GetTempFileName(); + await File.WriteAllTextAsync(assertionFile, "assertion-xyz"); + + var stub = new StubTokenEndpoint { + Respond = (_, _) => StubTokenEndpoint.Json( + HttpStatusCode.BadRequest, + "{\"error\":\"invalid_client\",\"error_description\":\"echo assertion-xyz\",\"error_uri\":\"https://x/assertion-xyz\"}" + ) + }; + + var options = new GrpcAuthOptions { + Type = GrpcAuthType.OAuthClientCredentials, TokenEndpoint = "https://idp.example.com/t", ClientId = "c", ClientAssertionFile = assertionFile + }; + + var ex = await Assert.That(async () => await new TokenEndpointClient(options, stub, "sink").RequestToken(default)).Throws(); + + await Assert.That(ex!.ToString()).DoesNotContain("assertion-xyz"); + await Assert.That(string.Join("\n", logs.TextAtOrAbove(Serilog.Events.LogEventLevel.Information))).DoesNotContain("assertion-xyz"); + await Assert.That(logs.AllText).DoesNotContain("assertion-xyz"); + await Assert.That(logs.AllText).Contains("echo ***"); + File.Delete(assertionFile); + } +} +``` + +Add `using Serilog.Events;` is not needed because the level is fully qualified above. + +- [ ] **Step 4: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, `TokenEndpointClient` not found. + +- [ ] **Step 5: Implement** + +`src/Kurrent.Replicator.KurrentDb/Auth/TokenEndpointClient.cs`: + +```csharp +using System.Globalization; +using System.Net; +using System.Net.Http.Headers; +using System.Text; +using System.Text.Json; + +namespace Kurrent.Replicator.KurrentDb.Auth; + +sealed record TokenResponse(string AccessToken, TimeSpan Lifetime) { + public override string ToString() => $"TokenResponse(Lifetime={Lifetime})"; +} + +sealed class TokenEndpointClient : IDisposable { + static ILog Log => LogProvider.GetLogger(typeof(TokenEndpointClient)); + + const string AssertionType = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"; + static readonly TimeSpan MaxLifetime = TimeSpan.FromSeconds(86400); + + static readonly HashSet AllowedErrors = [ + "invalid_request", "invalid_client", "invalid_grant", "unauthorized_client", + "unsupported_grant_type", "invalid_scope", "server_error", "temporarily_unavailable" + ]; + + readonly GrpcAuthOptions _options; + readonly string _side; + readonly HttpClient _http; + readonly Uri _endpoint; + readonly string? _secret; + readonly HashSet _seenTokens = []; + string? _lastAssertion; + bool _clampWarned; + + public TokenEndpointClient(GrpcAuthOptions options, HttpMessageHandler handler, string side) { + _options = options; + _side = side; + _endpoint = new Uri(options.TokenEndpoint!, UriKind.Absolute); + _http = new HttpClient(handler, disposeHandler: false) { Timeout = Timeout.InfiniteTimeSpan }; + _secret = options.ClientSecret ?? (options.ClientSecretFile != null ? File.ReadAllText(options.ClientSecretFile).Trim() : null); + } + + public string EndpointHost => _endpoint.Host; + + public static SocketsHttpHandler CreateDefaultHandler() => new() { AllowAutoRedirect = false }; + + public async Task RequestToken(CancellationToken ct) { + using var request = await BuildRequest(ct).ConfigureAwait(false); + using var response = await _http.SendAsync(request, HttpCompletionOption.ResponseContentRead, ct).ConfigureAwait(false); + + var status = (int)response.StatusCode; + + if (status is >= 300 and < 400) + throw new OAuthTokenException($"{_side}: token endpoint {EndpointHost} returned HTTP {status} (redirect); redirects are not followed"); + + var body = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false); + + if (!response.IsSuccessStatusCode) throw ErrorResponse(status, body); + + return Parse(body); + } + + async Task BuildRequest(CancellationToken ct) { + var form = new List> { new("grant_type", "client_credentials") }; + + if (!string.IsNullOrWhiteSpace(_options.Scope)) form.Add(new("scope", _options.Scope)); + + var request = new HttpRequestMessage(HttpMethod.Post, _endpoint); + + if (_options.ClientAssertionFile != null) { + var assertion = (await File.ReadAllTextAsync(_options.ClientAssertionFile, ct).ConfigureAwait(false)).Trim(); + + if (assertion.Length == 0) throw new OAuthTokenException($"{_side}: client assertion file {_options.ClientAssertionFile} is empty"); + + _lastAssertion = assertion; + form.Add(new("client_id", _options.ClientId!)); + form.Add(new("client_assertion_type", AssertionType)); + form.Add(new("client_assertion", assertion)); + } + else if (_options.ClientAuthentication == ClientAuthenticationMethod.Basic) { + var raw = $"{WebUtility.UrlEncode(_options.ClientId)}:{WebUtility.UrlEncode(_secret)}"; + request.Headers.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(Encoding.UTF8.GetBytes(raw))); + } + else { + form.Add(new("client_id", _options.ClientId!)); + form.Add(new("client_secret", _secret!)); + } + + foreach (var (key, value) in _options.AdditionalParameters) form.Add(new(key, value)); + + request.Content = new FormUrlEncodedContent(form); + + return request; + } + + TokenResponse Parse(string body) { + JsonDocument doc; + + try { + doc = JsonDocument.Parse(body); + } catch (JsonException) { + throw new OAuthTokenException($"{_side}: token endpoint {EndpointHost} returned a response that is not valid JSON"); + } + + using (doc) { + var root = doc.RootElement; + + if (root.ValueKind != JsonValueKind.Object) throw Invalid("the response is not a JSON object"); + + if (!root.TryGetProperty("access_token", out var tokenEl) || tokenEl.ValueKind != JsonValueKind.String || string.IsNullOrEmpty(tokenEl.GetString())) + throw Invalid("access_token is missing or empty"); + + if (!root.TryGetProperty("token_type", out var typeEl) || typeEl.ValueKind != JsonValueKind.String || + !string.Equals(typeEl.GetString(), "Bearer", StringComparison.OrdinalIgnoreCase)) + throw Invalid("token_type must be Bearer"); + + var token = tokenEl.GetString()!; + lock (_seenTokens) _seenTokens.Add(token); + + return new(token, Lifetime(root)); + } + } + + TimeSpan Lifetime(JsonElement root) { + if (!root.TryGetProperty("expires_in", out var el)) { + if (_options.DefaultTokenLifetimeSeconds is { } fallback) return TimeSpan.FromSeconds(fallback); + + throw Invalid("expires_in is missing; set auth.defaultTokenLifetimeSeconds to the lifetime your provider documents"); + } + + long seconds; + + if (el.ValueKind == JsonValueKind.Number) { + if (!el.TryGetInt64(out seconds)) throw Invalid("expires_in is not a whole number of seconds"); + } + else if (el.ValueKind == JsonValueKind.String) { + if (!long.TryParse(el.GetString(), NumberStyles.None, CultureInfo.InvariantCulture, out seconds)) throw Invalid("expires_in is not a whole number of seconds"); + } + else { + throw Invalid("expires_in is not a number"); + } + + if (seconds <= 0) throw Invalid("expires_in must be positive"); + + if (seconds > MaxLifetime.TotalSeconds) { + if (!_clampWarned) { + _clampWarned = true; + Log.Warn("{Side}: token lifetime {Seconds}s exceeds 24h; treating it as 24h", _side, seconds); + } + + return MaxLifetime; + } + + return TimeSpan.FromSeconds(seconds); + } + + OAuthTokenException Invalid(string reason) => new($"{_side}: token endpoint {EndpointHost} returned an invalid token response: {reason}"); + + OAuthTokenException ErrorResponse(int status, string body) { + string? error = null; + string? description = null; + var codes = new List(); + + try { + using var doc = JsonDocument.Parse(body); + var root = doc.RootElement; + + if (root.ValueKind == JsonValueKind.Object) { + if (root.TryGetProperty("error", out var e) && e.ValueKind == JsonValueKind.String) error = e.GetString(); + if (root.TryGetProperty("error_description", out var d) && d.ValueKind == JsonValueKind.String) description = d.GetString(); + + if (root.TryGetProperty("error_codes", out var c) && c.ValueKind == JsonValueKind.Array) { + foreach (var item in c.EnumerateArray()) + if (item.ValueKind == JsonValueKind.Number && item.TryGetInt64(out var code)) codes.Add(code); + } + } + } catch (JsonException) { + // body is not JSON: report status only + } + + var message = new StringBuilder($"{_side}: token request to {EndpointHost} failed: HTTP {status}"); + + if (error != null) message.Append(", error=").Append(AllowedErrors.Contains(error) ? error : "non-standard error (omitted)"); + if (codes.Count > 0) message.Append(", error_codes=").Append(string.Join(",", codes)); + + if (description != null && Log.IsDebugEnabled()) Log.Debug("{Side}: token endpoint error_description: {Description}", _side, Redact(description)); + + return new(message.ToString()); + } + + string Redact(string text) { + var secrets = new List(); + if (!string.IsNullOrEmpty(_secret)) secrets.Add(_secret); + if (!string.IsNullOrEmpty(_lastAssertion)) secrets.Add(_lastAssertion); + lock (_seenTokens) secrets.AddRange(_seenTokens); + + return secrets.Where(s => s.Length > 0).OrderByDescending(s => s.Length).Aggregate(text, (t, s) => t.Replace(s, "***", StringComparison.Ordinal)); + } + + public void Dispose() => _http.Dispose(); +} +``` + +Note: the expected `Basic` value in the test (`my+app:a%2Bb%3Ac`) is what `WebUtility.UrlEncode` produces (`+` for space, upper-case hex). RFC 6749 §2.3.1 requires `application/x-www-form-urlencoded` encoding of each part. + +- [ ] **Step 6: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/TokenEndpointClient*/*"` +Expected: all PASS (including the 11 parameterised invalid-response cases). + +- [ ] **Step 7: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/Auth/TokenEndpointClient.cs test/Kurrent.Replicator.Tests/Auth +git commit -m "feat(auth): token endpoint client with strict parsing and sanitised errors + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 5: ClientCredentialsTokenSource — cache, refresh window, single-flight, cooldown, fallback + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/ClientCredentialsTokenSource.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/ClientCredentialsTokenSourceTests.cs` + +**Interfaces:** +- Consumes: `TokenEndpointClient`, `TokenResponse`, `TokenState`, `GrpcAuthOptions`. +- Produces: `public sealed class ClientCredentialsTokenSource : IAccessTokenSource, IDisposable` with ctor `(GrpcAuthOptions options, string side, HttpMessageHandler handler, TimeProvider time, CancellationToken shutdown)`; constants `RequestTimeout = 30 s`, `FailureCooldown = 5 s`, `FallbackMargin = 30 s`. + +Rules (spec "Caching and refresh"): cached if `now < expiresAt - min(refreshBefore, lifetime/2)`; shared in-flight task (`Task.Run`, treat completed task as none); cooldown 5 s after failure; fallback to cached token if `now < expiresAt - 30 s` and not invalidated; shared request cancelled by shutdown or 30 s timeout, never by a caller. + +- [ ] **Step 1: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/ClientCredentialsTokenSourceTests.cs`: + +```csharp +using System.Net; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class ClientCredentialsTokenSourceTests { + readonly FakeTimeProvider _time = new(); + readonly StubTokenEndpoint _stub = new(); + readonly CancellationTokenSource _shutdown = new(); + + static readonly GrpcAuthOptions Options = new() { + Type = GrpcAuthType.OAuthClientCredentials, TokenEndpoint = "https://idp.example.com/t", ClientId = "c", ClientSecret = "s" + }; + + ClientCredentialsTokenSource NewSource(GrpcAuthOptions? o = null) => new(o ?? Options, "sink", _stub, _time, _shutdown.Token); + + void Respond(params string[] tokens) { + var i = 0; + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.OK, StubTokenEndpoint.Token(tokens[Math.Min(i++, tokens.Length - 1)])); + } + + void Fail() => _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.ServiceUnavailable, "{\"error\":\"temporarily_unavailable\"}"); + + [Test] + public async Task Caches_until_refresh_window() { + Respond("A", "B"); + var s = NewSource(); + await Assert.That((await s.GetAccessToken(default)).Value).IsEqualTo("A"); + _time.Advance(TimeSpan.FromSeconds(3299)); // 3600 - 300 - 1 + await Assert.That((await s.GetAccessToken(default)).Value).IsEqualTo("A"); + _time.Advance(TimeSpan.FromSeconds(1)); + await Assert.That((await s.GetAccessToken(default)).Value).IsEqualTo("B"); + await Assert.That(_stub.Count).IsEqualTo(2); + } + + [Test] + public async Task Very_short_lifetime_does_not_refresh_on_every_call() { + _stub.Respond = (_, _) => StubTokenEndpoint.Json(HttpStatusCode.OK, StubTokenEndpoint.Token("A", 60)); + var s = NewSource(); // refreshBefore 300 > lifetime: window capped at 30s + await s.GetAccessToken(default); + _time.Advance(TimeSpan.FromSeconds(29)); + await s.GetAccessToken(default); + await s.GetAccessToken(default); + await Assert.That(_stub.Count).IsEqualTo(1); + } + + [Test] + public async Task Concurrent_callers_share_one_successful_request() { + Respond("A"); + _stub.Gate = new TaskCompletionSource(); + var s = NewSource(); + var calls = Enumerable.Range(0, 10).Select(_ => s.GetAccessToken(default).AsTask()).ToList(); + await Task.Delay(50); + _stub.Gate.SetResult(); + var leases = await Task.WhenAll(calls); + await Assert.That(_stub.Count).IsEqualTo(1); + await Assert.That(leases.Select(l => l.Value).Distinct().Single()).IsEqualTo("A"); + } + + [Test] + public async Task Concurrent_callers_share_one_failing_request_and_its_exception() { + Fail(); + _stub.Gate = new TaskCompletionSource(); + var s = NewSource(); + var calls = Enumerable.Range(0, 10).Select(_ => s.GetAccessToken(default).AsTask()).ToList(); + await Task.Delay(50); + _stub.Gate.SetResult(); + + foreach (var call in calls) await Assert.That(async () => await call).Throws(); + await Assert.That(_stub.Count).IsEqualTo(1); + } + + [Test] + public async Task Cooldown_suppresses_requests_for_five_seconds() { + Fail(); + var s = NewSource(); + await Assert.That(async () => await s.GetAccessToken(default)).Throws(); + _time.Advance(TimeSpan.FromSeconds(4)); + await Assert.That(async () => await s.GetAccessToken(default)).Throws(); + await Assert.That(_stub.Count).IsEqualTo(1); + _time.Advance(TimeSpan.FromSeconds(1)); + Respond("A"); + await Assert.That((await s.GetAccessToken(default)).Value).IsEqualTo("A"); + await Assert.That(_stub.Count).IsEqualTo(2); + } + + [Test] + public async Task Refresh_failure_falls_back_to_usable_cached_token() { + Respond("A"); + var s = NewSource(); + await s.GetAccessToken(default); + _time.Advance(TimeSpan.FromSeconds(3400)); // inside refresh window, 200s before expiry + Fail(); + await Assert.That((await s.GetAccessToken(default)).Value).IsEqualTo("A"); + } + + [Test] + public async Task Refresh_failure_with_expired_token_throws_with_status_and_code() { + Respond("A"); + var s = NewSource(); + await s.GetAccessToken(default); + _time.Advance(TimeSpan.FromSeconds(3580)); // within 30s of expiry: not usable + Fail(); + var ex = await Assert.That(async () => await s.GetAccessToken(default)).Throws(); + await Assert.That(ex!.Message).Contains("503"); + await Assert.That(ex.Message).Contains("temporarily_unavailable"); + } + + [Test] + public async Task Invalidate_forces_refresh_and_disables_fallback() { + Respond("A"); + var s = NewSource(); + var a = await s.GetAccessToken(default); + s.Invalidate(a); + Fail(); + await Assert.That(async () => await s.GetAccessToken(default)).Throws(); + await Assert.That(_stub.Count).IsEqualTo(2); + } + + [Test] + public async Task Reissued_rejected_token_is_quarantined_then_probed() { + Respond("A"); + var s = NewSource(); + s.Invalidate(await s.GetAccessToken(default)); + await Assert.That(async () => await s.GetAccessToken(default)).Throws(); // endpoint re-issues A + _time.Advance(TimeSpan.FromSeconds(61)); + await Assert.That((await s.GetAccessToken(default)).Value).IsEqualTo("A"); + } + + [Test] + public async Task Same_value_refresh_does_not_lose_rejection() { + Respond("A"); + var s = NewSource(); + var call1 = await s.GetAccessToken(default); + _time.Advance(TimeSpan.FromSeconds(3300)); // refresh re-issues A, same generation + var again = await s.GetAccessToken(default); + await Assert.That(again.Generation).IsEqualTo(call1.Generation); + s.Invalidate(call1); + await Assert.That(async () => await s.GetAccessToken(default)).Throws(); + } + + [Test] + public async Task Caller_cancellation_does_not_cancel_shared_request() { + Respond("A"); + _stub.Gate = new TaskCompletionSource(); + var s = NewSource(); + using var cts = new CancellationTokenSource(); + var cancelled = s.GetAccessToken(cts.Token).AsTask(); + var other = s.GetAccessToken(default).AsTask(); + await Task.Delay(20); + await cts.CancelAsync(); + await Assert.That(async () => await cancelled).Throws(); + _stub.Gate.SetResult(); + await Assert.That((await other).Value).IsEqualTo("A"); + await Assert.That(_stub.Count).IsEqualTo(1); + } + + [Test] + public async Task Shutdown_aborts_in_flight_request() { + Respond("A"); + _stub.Gate = new TaskCompletionSource(); // never released + var s = NewSource(); + var call = s.GetAccessToken(default).AsTask(); + await Task.Delay(20); + await _shutdown.CancelAsync(); + await Assert.That(async () => await call.WaitAsync(TimeSpan.FromSeconds(5))).Throws(); + } + + [Test] + public async Task Request_times_out_after_30_seconds() { + Respond("A"); + _stub.Gate = new TaskCompletionSource(); // never released + var s = NewSource(); + var call = s.GetAccessToken(default).AsTask(); + await Task.Delay(20); + _time.Advance(TimeSpan.FromSeconds(31)); + var ex = await Assert.That(async () => await call.WaitAsync(TimeSpan.FromSeconds(5))).Throws(); + await Assert.That(ex!.Message).Contains("timed out"); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, `ClientCredentialsTokenSource` not found. + +- [ ] **Step 3: Implement** + +`src/Kurrent.Replicator.KurrentDb/Auth/ClientCredentialsTokenSource.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +public sealed class ClientCredentialsTokenSource : IAccessTokenSource, IDisposable { + static ILog Log => LogProvider.GetLogger(typeof(ClientCredentialsTokenSource)); + + public static readonly TimeSpan RequestTimeout = TimeSpan.FromSeconds(30); + public static readonly TimeSpan FailureCooldown = TimeSpan.FromSeconds(5); + public static readonly TimeSpan FallbackMargin = TimeSpan.FromSeconds(30); + + sealed record Cached(string Value, DateTimeOffset ExpiresAt, TimeSpan Lifetime); + + readonly GrpcAuthOptions _options; + readonly string _side; + readonly TimeProvider _time; + readonly CancellationToken _shutdown; + readonly TokenEndpointClient _client; + readonly TokenState _state; + readonly object _lock = new(); + + Cached? _cached; + Task? _inflight; + DateTimeOffset? _lastFailureAt; + OAuthTokenException? _lastFailure; + + public ClientCredentialsTokenSource(GrpcAuthOptions options, string side, HttpMessageHandler handler, TimeProvider time, CancellationToken shutdown) { + _options = options; + _side = side; + _time = time; + _shutdown = shutdown; + _client = new TokenEndpointClient(options, handler, side); + _state = new TokenState(time); + } + + public async ValueTask GetAccessToken(CancellationToken ct) { + ct.ThrowIfCancellationRequested(); + Task task; + + lock (_lock) { + var now = _time.GetUtcNow(); + + if (_cached is { } c && now < c.ExpiresAt - RefreshWindow(c)) return _state.Accept(c.Value); + + if (_inflight is null || _inflight.IsCompleted) { + if (_lastFailureAt is { } failedAt && now - failedAt < FailureCooldown) { + if (Usable(now) is { } fallback) return _state.Accept(fallback); + + throw new OAuthTokenException(_lastFailure!.Message, _lastFailure); + } + + _inflight = Task.Run(Refresh, CancellationToken.None); + } + + task = _inflight; + } + + try { + var token = await task.WaitAsync(ct).ConfigureAwait(false); + + return _state.Accept(token.Value); + } catch (OAuthTokenException) { + string? fallback; + lock (_lock) fallback = Usable(_time.GetUtcNow()); + + if (fallback == null) throw; + + Log.Warn("{Side}: token refresh failed; using the current token until it nears expiry", _side); + + return _state.Accept(fallback); + } + } + + public void Invalidate(AccessTokenLease lease) { + if (!_state.Invalidate(lease)) { + Log.Debug("{Side}: ignoring stale token rejection report", _side); + + return; + } + + lock (_lock) { + if (_cached?.Value == lease.Value) _cached = null; + } + } + + public void ReportAccepted(AccessTokenLease lease) => _state.ReportAccepted(lease); + + TimeSpan RefreshWindow(Cached c) { + var configured = TimeSpan.FromSeconds(Math.Max(0, _options.RefreshBeforeExpirySeconds)); + var half = c.Lifetime / 2; + + return configured < half ? configured : half; + } + + string? Usable(DateTimeOffset now) + => _cached is { } c && now < c.ExpiresAt - FallbackMargin && !_state.IsRejected(c.Value) ? c.Value : null; + + async Task Refresh() { + using var timeout = new CancellationTokenSource(RequestTimeout, _time); + using var linked = CancellationTokenSource.CreateLinkedTokenSource(_shutdown, timeout.Token); + var started = _time.GetUtcNow(); + + try { + var response = await _client.RequestToken(linked.Token).ConfigureAwait(false); + var cached = new Cached(response.AccessToken, started + response.Lifetime, response.Lifetime); + + lock (_lock) { + _cached = cached; + _lastFailureAt = null; + _lastFailure = null; + } + + Log.Info("{Side}: access token acquired from {Host}, expires at {ExpiresAt:O}", _side, _client.EndpointHost, cached.ExpiresAt); + + return cached; + } catch (OperationCanceledException) when (_shutdown.IsCancellationRequested) { + throw; + } catch (Exception e) { + var failure = e switch { + OAuthTokenException o => o, + OperationCanceledException => new OAuthTokenException($"{_side}: token request to {_client.EndpointHost} timed out after {RequestTimeout.TotalSeconds:0}s", e), + _ => new OAuthTokenException($"{_side}: token request to {_client.EndpointHost} failed: {e.GetType().Name}", e) + }; + + lock (_lock) { + _lastFailureAt = _time.GetUtcNow(); + _lastFailure = failure; + } + + throw failure; + } + } + + public void Dispose() => _client.Dispose(); +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/ClientCredentialsTokenSourceTests/*"` +Expected: 13 tests PASS. If `Request_times_out_after_30_seconds` hangs, check that the timeout CTS is built with `_time` (`new CancellationTokenSource(TimeSpan, TimeProvider)`), so `FakeTimeProvider.Advance` fires it. + +- [ ] **Step 5: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/Auth/ClientCredentialsTokenSource.cs test/Kurrent.Replicator.Tests/Auth/ClientCredentialsTokenSourceTests.cs +git commit -m "feat(auth): client credentials token source + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- +### Task 6: TokenGate and GrpcAuthContext + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/TokenGate.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthContext.cs` +- Create: `test/Kurrent.Replicator.Tests/Auth/Support/ControllableTokenSource.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/TokenGateTests.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/GrpcAuthContextTests.cs` + +**Interfaces:** +- Consumes: `IAccessTokenSource`, `AccessTokenLease`, `AuthFailure`, `RateLimitedWarning`, `TokenState` (test support only). +- Produces: + - `public sealed class TokenGate(TimeProvider time, string side)` with `ValueTask WaitForToken(IAccessTokenSource source, CancellationToken ct)` and `static TimeSpan Backoff(int attempt)` (1, 2, 4, 8, 16, 30, 30 … seconds). + - `public readonly record struct CallAuth(UserCredentials? Credentials, AccessTokenLease? Lease)`. + - `public sealed class GrpcAuthContext` with ctor `(IAccessTokenSource? source, CancellationToken shutdown, TimeProvider time, string side)`, `static GrpcAuthContext None`, `bool Enabled`, `CancellationToken Shutdown`, `string Side`, `TimeProvider Time`, `ValueTask AcquireCredentials(CancellationToken ct)`, `void ReportAccepted(CallAuth)`, `void ReportFailure(CallAuth, Exception?)`, `Task Run(Func> call, CancellationToken ct)`, `Task Run(Func call, CancellationToken ct)`. + - Test support `ControllableTokenSource(TimeProvider time) : IAccessTokenSource` — `string? Value` (null → throws `OAuthTokenException`), `ConcurrentQueue<(AccessTokenLease Lease, bool Applied)> Invalidated`, `ConcurrentQueue Accepted`, `int Calls`, `long Generation`, `Action? OnInvalidate`. + +`Run` passes the caller's `ct` (not the shutdown-linked token) to `call`, so a graceful shutdown never cancels an in-flight write; only waits (token acquisition, backoff) are bound to shutdown. + +- [ ] **Step 1: Create the controllable token source** + +`test/Kurrent.Replicator.Tests/Auth/Support/ControllableTokenSource.cs`: + +```csharp +using System.Collections.Concurrent; +using Kurrent.Replicator.KurrentDb.Auth; + +namespace Kurrent.Replicator.Tests.Auth.Support; + +/// A token source whose current value the test controls, with the real TokenState rules. +public sealed class ControllableTokenSource(TimeProvider time) : IAccessTokenSource { + readonly TokenState _state = new(time); + int _calls; + + public volatile string? Value; + + public ConcurrentQueue<(AccessTokenLease Lease, bool Applied)> Invalidated { get; } = new(); + public ConcurrentQueue Accepted { get; } = new(); + public Action? OnInvalidate { get; set; } + + public int Calls => Volatile.Read(ref _calls); + public long Generation => _state.Generation; + + public ValueTask GetAccessToken(CancellationToken ct) { + Interlocked.Increment(ref _calls); + ct.ThrowIfCancellationRequested(); + var value = Value ?? throw new OAuthTokenException("test: token unavailable"); + + return ValueTask.FromResult(_state.Accept(value)); + } + + public void Invalidate(AccessTokenLease lease) { + var applied = _state.Invalidate(lease); + Invalidated.Enqueue((lease, applied)); + OnInvalidate?.Invoke(lease); + } + + public void ReportAccepted(AccessTokenLease lease) { + Accepted.Enqueue(lease); + _state.ReportAccepted(lease); + } +} +``` + +- [ ] **Step 2: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/TokenGateTests.cs`: + +```csharp +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; +using Serilog.Events; + +namespace Kurrent.Replicator.Tests.Auth; + +public class TokenGateTests { + readonly FakeTimeProvider _time = new(); + + [Test] + public async Task Backoff_doubles_and_caps_at_30_seconds() { + var delays = Enumerable.Range(0, 8).Select(i => TokenGate.Backoff(i).TotalSeconds).ToArray(); + await Assert.That(delays).IsEquivalentTo(new double[] { 1, 2, 4, 8, 16, 30, 30, 30 }); + } + + [Test] + public async Task Waits_until_source_recovers() { + var source = new ControllableTokenSource(_time); + var wait = new TokenGate(_time, "sink").WaitForToken(source, default).AsTask(); + await TimeDriver.Until(() => source.Calls >= 3, _time); + await Assert.That(wait.IsCompleted).IsFalse(); + source.Value = "A"; + var lease = await TimeDriver.Drive(wait, _time); + await Assert.That(lease.Value).IsEqualTo("A"); + } + + [Test] + public async Task Cancellation_ends_the_wait() { + var source = new ControllableTokenSource(_time); + using var cts = new CancellationTokenSource(); + var wait = new TokenGate(_time, "sink").WaitForToken(source, cts.Token).AsTask(); + await TimeDriver.Until(() => source.Calls >= 2, _time); + await cts.CancelAsync(); + await Assert.That(async () => await wait).Throws(); + } +} + +[NotInParallel("global-logger")] +public class TokenGateLoggingTests { + [Test] + public async Task Warnings_are_rate_limited_and_recovery_is_logged_once() { + using var logs = new LogCapture(); + var time = new FakeTimeProvider(); + var side = $"side-{Guid.NewGuid():N}"; + var source = new ControllableTokenSource(time); + var wait = new TokenGate(time, side).WaitForToken(source, default).AsTask(); + + await TimeDriver.Until(() => time.GetUtcNow() - time.Start >= TimeSpan.FromSeconds(200), time, TimeSpan.FromSeconds(5)); + source.Value = "A"; + await TimeDriver.Drive(wait, time); + + var warnings = logs.Events.Count(e => e.Level == LogEventLevel.Warning && e.Text.Contains(side)); + await Assert.That(warnings).IsLessThanOrEqualTo(5); // first + at most once per minute over ~200s + await Assert.That(warnings).IsGreaterThanOrEqualTo(1); + await Assert.That(logs.Events.Count(e => e.Level == LogEventLevel.Information && e.Text.Contains(side) && e.Text.Contains("acquired again"))).IsEqualTo(1); + } +} +``` + +Note: `FakeTimeProvider.Start` is the provider's initial time. + +`test/Kurrent.Replicator.Tests/Auth/GrpcAuthContextTests.cs`: + +```csharp +using EventStore.Client; +using Grpc.Core; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class GrpcAuthContextTests { + readonly FakeTimeProvider _time = new(); + readonly CancellationTokenSource _shutdown = new(); + readonly ControllableTokenSource _source; + readonly GrpcAuthContext _ctx; + + public GrpcAuthContextTests() { + _source = new(_time) { Value = "A" }; + _ctx = new(_source, _shutdown.Token, _time, "sink"); + } + + static NotAuthenticatedException Unauthenticated() => new("no", new RpcException(new Status(StatusCode.Unauthenticated, "no"))); + + [Test] + public async Task None_passes_no_credentials_and_calls_once() { + var calls = 0; + CallAuth seen = default; + var result = await GrpcAuthContext.None.Run((a, _) => { calls++; seen = a; return Task.FromResult(42); }, default); + await Assert.That(result).IsEqualTo(42); + await Assert.That(calls).IsEqualTo(1); + await Assert.That(seen.Credentials).IsNull(); + await Assert.That((await GrpcAuthContext.None.AcquireCredentials(default)).Credentials).IsNull(); + await Assert.That(GrpcAuthContext.None.Enabled).IsFalse(); + } + + [Test] + public async Task AcquireCredentials_returns_bearer_with_lease() { + var auth = await _ctx.AcquireCredentials(default); + await Assert.That(auth.Credentials!.ToString()).IsEqualTo("Bearer A"); + await Assert.That(auth.Lease!.Value.Value).IsEqualTo("A"); + } + + [Test] + public async Task Run_reports_acceptance_with_the_lease_the_call_used() { + AccessTokenLease? used = null; + await _ctx.Run((a, _) => { used = a.Lease; return Task.CompletedTask; }, default); + await Assert.That(_source.Accepted.Single()).IsEqualTo(used!.Value); + } + + [Test] + public async Task Run_passes_the_caller_token_to_the_call() { + using var caller = new CancellationTokenSource(); + CancellationToken seen = default; + await _ctx.Run((_, c) => { seen = c; return Task.CompletedTask; }, caller.Token); + await Assert.That(seen).IsEqualTo(caller.Token); + } + + [Test] + public async Task Run_retries_token_rejection_with_invalidation_and_new_token() { + var headers = new List(); + + var run = _ctx.Run((a, _) => { + headers.Add(a.Credentials!.ToString()); + if (headers.Count == 1) { + _source.Value = "B"; + throw Unauthenticated(); + } + + return Task.FromResult(1); + }, default); + + await TimeDriver.Drive(run, _time); + await Assert.That(headers).IsEquivalentTo(new[] { "Bearer A", "Bearer B" }); + await Assert.That(_source.Invalidated.Single().Applied).IsTrue(); + await Assert.That(_source.Invalidated.Single().Lease.Value).IsEqualTo("A"); + } + + [Test] + public async Task Run_propagates_permission_denied_and_other_errors_immediately() { + var calls = 0; + var denied = new AccessDeniedException("no", new RpcException(new Status(StatusCode.PermissionDenied, "no"))); + await Assert.That(async () => await _ctx.Run((_, _) => { calls++; throw denied; }, default)).Throws(); + await Assert.That(async () => await _ctx.Run((_, _) => { calls++; throw new InvalidOperationException(); }, default)).Throws(); + await Assert.That(calls).IsEqualTo(2); + await Assert.That(_source.Invalidated).IsEmpty(); + } + + [Test] + public async Task Shutdown_ends_the_retry_loop() { + _source.Value = null; + var run = _ctx.Run((_, _) => Task.FromResult(1), default); + await TimeDriver.Until(() => _source.Calls >= 2, _time); + await _shutdown.CancelAsync(); + await Assert.That(async () => await run).Throws(); + } + + [Test] + public async Task ReportFailure_only_invalidates_on_token_failures() { + var auth = await _ctx.AcquireCredentials(default); + _ctx.ReportFailure(auth, new InvalidOperationException()); + _ctx.ReportFailure(auth, null); + await Assert.That(_source.Invalidated).IsEmpty(); + _ctx.ReportFailure(auth, Unauthenticated()); + await Assert.That(_source.Invalidated.Single().Applied).IsTrue(); + } +} +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, `TokenGate` / `GrpcAuthContext` / `CallAuth` not found. + +- [ ] **Step 4: Implement TokenGate** + +`src/Kurrent.Replicator.KurrentDb/Auth/TokenGate.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +public sealed class TokenGate(TimeProvider time, string side) { + static ILog Log => LogProvider.GetLogger(typeof(TokenGate)); + + static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(30); + + readonly RateLimitedWarning _warn = new(time, TimeSpan.FromSeconds(60)); + + public static TimeSpan Backoff(int attempt) { + var seconds = attempt >= 5 ? MaxBackoff.TotalSeconds : Math.Pow(2, attempt); + + return TimeSpan.FromSeconds(Math.Min(seconds, MaxBackoff.TotalSeconds)); + } + + public async ValueTask WaitForToken(IAccessTokenSource source, CancellationToken ct) { + var failures = 0; + + while (true) { + ct.ThrowIfCancellationRequested(); + + try { + var lease = await source.GetAccessToken(ct).ConfigureAwait(false); + + if (failures > 0) { + Log.Info("{Side}: access token acquired again after {Failures} failed attempts", side, failures); + _warn.Reset(); + } + + return lease; + } catch (OAuthTokenException e) { + if (_warn.ShouldLog()) Log.Warn("{Side}: no access token available ({Reason}); retrying with backoff", side, e.Message); + + await Task.Delay(Backoff(failures++), time, ct).ConfigureAwait(false); + } + } + } +} +``` + +- [ ] **Step 5: Implement GrpcAuthContext** + +`src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthContext.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +/// Credentials for one gRPC call, plus the lease they were issued under. Both null without OAuth. +public readonly record struct CallAuth(UserCredentials? Credentials, AccessTokenLease? Lease); + +public sealed class GrpcAuthContext { + static ILog Log => LogProvider.GetLogger(typeof(GrpcAuthContext)); + + public static GrpcAuthContext None { get; } = new(null, CancellationToken.None, TimeProvider.System, "none"); + + readonly IAccessTokenSource? _source; + readonly TokenGate _gate; + readonly RateLimitedWarning _warn; + + public GrpcAuthContext(IAccessTokenSource? source, CancellationToken shutdown, TimeProvider time, string side) { + _source = source; + Shutdown = shutdown; + Time = time; + Side = side; + _gate = new TokenGate(time, side); + _warn = new RateLimitedWarning(time, TimeSpan.FromSeconds(60)); + } + + public bool Enabled => _source != null; + public CancellationToken Shutdown { get; } + public TimeProvider Time { get; } + public string Side { get; } + + public async ValueTask AcquireCredentials(CancellationToken ct) { + if (_source == null) return default; + + using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct, Shutdown); + var lease = await _gate.WaitForToken(_source, linked.Token).ConfigureAwait(false); + + return new(new UserCredentials(lease.Value), lease); + } + + public void ReportAccepted(CallAuth auth) { + if (_source != null && auth.Lease is { } lease) _source.ReportAccepted(lease); + } + + public void ReportFailure(CallAuth auth, Exception? error) { + if (_source != null && auth.Lease is { } lease && error != null && AuthFailure.IsTokenFailure(error)) _source.Invalidate(lease); + } + + public async Task Run(Func> call, CancellationToken ct) { + if (_source == null) return await call(default, ct).ConfigureAwait(false); + + using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct, Shutdown); + var failures = 0; + + while (true) { + var auth = await AcquireCredentials(linked.Token).ConfigureAwait(false); + + try { + var result = await call(auth, ct).ConfigureAwait(false); + ReportAccepted(auth); + + if (failures > 0) { + Log.Info("{Side}: KurrentDB accepted the access token again after {Failures} failed attempts", Side, failures); + _warn.Reset(); + } + + return result; + } catch (Exception e) when (AuthFailure.IsTokenFailure(e) && !linked.IsCancellationRequested) { + ReportFailure(auth, e); + + if (_warn.ShouldLog()) Log.Warn("{Side}: gRPC call failed with a token error ({Error}); retrying with backoff", Side, e.Message); + + await Task.Delay(TokenGate.Backoff(failures++), Time, linked.Token).ConfigureAwait(false); + } + } + } + + public Task Run(Func call, CancellationToken ct) + => Run(async (a, c) => { + await call(a, c).ConfigureAwait(false); + + return true; + }, + ct + ); +} +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/TokenGate*/*"` and the same with `GrpcAuthContextTests`. +Expected: all PASS. + +- [ ] **Step 7: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/Auth test/Kurrent.Replicator.Tests/Auth +git commit -m "feat(auth): token gate and per-call auth context + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 7: Options validation, fallback hook, source factory, fake KurrentDB handler + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthOptionsValidator.cs` +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthentication.cs` +- Create: `test/Kurrent.Replicator.Tests/Auth/Support/FakeKurrentDbHandler.cs` +- Create: `test/Kurrent.Replicator.Tests/Auth/Support/TestEvents.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/GrpcAuthOptionsValidatorTests.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/GrpcAuthenticationTests.cs` + +**Interfaces:** +- Consumes: `GrpcAuthOptions`, `IAccessTokenSource`, `ClientCredentialsTokenSource`, `TokenFileSource`, `TokenEndpointClient.CreateDefaultHandler()`. +- Produces: + - `public static class GrpcAuthOptionsValidator` — `IReadOnlyList Validate(GrpcAuthOptions, EventStoreClientSettings)`, `IReadOnlyList Warnings(GrpcAuthOptions)`, `string? ValidateProtocol(GrpcAuthOptions, string? protocol)`, `void EnsureValid(string side, GrpcAuthOptions, EventStoreClientSettings)` (throws `InvalidOperationException("Invalid {side} auth configuration: e1; e2")`). + - `public static class GrpcAuthentication` — `FallbackUsage Apply(EventStoreClientSettings, IAccessTokenSource)`, `IAccessTokenSource? CreateSource(GrpcAuthOptions, string side, TimeProvider, CancellationToken shutdown)`; `public sealed class FallbackUsage { int Count }`. + - Test support `FakeKurrentDbHandler : HttpMessageHandler` — `ConcurrentQueue Requests` (`record Seen(string Path, string? Authorization)`), `Func> Respond`, constants `AppendPath`, `ReadPath`, `DeletePath`, helpers `TrailersOnly(StatusCode)`, `AppendSuccess()`, `static EventStoreClient Client(FakeKurrentDbHandler, string connectionString = "esdb://localhost:2113?tls=true", Action? configure = null)`. + - Test support `TestEvents` — `Proposed(stream)`, `Meta(stream)`, `Delete(stream)`, `Original(stream, eventNumber)`. + +- [ ] **Step 1: Create the fake KurrentDB handler and test events** + +`test/Kurrent.Replicator.Tests/Auth/Support/FakeKurrentDbHandler.cs`: + +```csharp +using System.Buffers.Binary; +using System.Collections.Concurrent; +using System.Net; +using System.Net.Http.Headers; +using EventStore.Client; +using Grpc.Core; + +namespace Kurrent.Replicator.Tests.Auth.Support; + +/// +/// Stands in for KurrentDB at the HTTP/2 level: records the path and authorization header of every gRPC call +/// and answers with a configurable gRPC response. Server-feature discovery is answered with Unimplemented so the +/// client uses the classic Append RPC. +/// +public sealed class FakeKurrentDbHandler : HttpMessageHandler { + public const string AppendPath = "/event_store.client.streams.Streams/Append"; + public const string ReadPath = "/event_store.client.streams.Streams/Read"; + public const string DeletePath = "/event_store.client.streams.Streams/Delete"; + + public sealed record Seen(string Path, string? Authorization); + + public ConcurrentQueue Requests { get; } = new(); + + /// Default: every call fails with Unavailable (enough for header-capture tests). + public Func> Respond { get; set; } = _ => Task.FromResult(TrailersOnly(StatusCode.Unavailable)); + + public IReadOnlyList AuthorizationsFor(string path) => Requests.Where(r => r.Path == path).Select(r => r.Authorization).ToList(); + + protected override async Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) { + var path = request.RequestUri!.AbsolutePath; + + if (path.StartsWith("/event_store.client.server_features.", StringComparison.Ordinal)) return TrailersOnly(StatusCode.Unimplemented); + + if (request.Content != null) await request.Content.ReadAsByteArrayAsync(cancellationToken); + + var auth = request.Headers.TryGetValues("authorization", out var values) ? values.FirstOrDefault() : null; + var seen = new Seen(path, auth); + Requests.Enqueue(seen); + + return await Respond(seen); + } + + public static HttpResponseMessage TrailersOnly(StatusCode code) { + var response = new HttpResponseMessage(HttpStatusCode.OK) { Version = new Version(2, 0), Content = new ByteArrayContent([]) }; + response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/grpc"); + response.Headers.Add("grpc-status", ((int)code).ToString()); + response.Headers.Add("grpc-message", code.ToString()); + + return response; + } + + /// AppendResp { success { current_revision = 0, position { commit = 1, prepare = 1 } } } + public static HttpResponseMessage AppendSuccess() => Unary([0x0A, 0x08, 0x08, 0x00, 0x1A, 0x04, 0x08, 0x01, 0x10, 0x01]); + + static HttpResponseMessage Unary(byte[] message) { + var frame = new byte[5 + message.Length]; + BinaryPrimitives.WriteUInt32BigEndian(frame.AsSpan(1), (uint)message.Length); + message.CopyTo(frame, 5); + + var response = new HttpResponseMessage(HttpStatusCode.OK) { Version = new Version(2, 0), Content = new ByteArrayContent(frame) }; + response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/grpc"); + response.TrailingHeaders.Add("grpc-status", "0"); + + return response; + } + + /// Append succeeds for the listed bearer tokens and fails Unauthenticated for anything else; other calls: Unavailable / Unauthenticated. + public static Func> AcceptBearer(params string[] tokens) + => seen => { + var ok = tokens.Any(t => seen.Authorization == $"Bearer {t}"); + + return Task.FromResult( + !ok ? TrailersOnly(StatusCode.Unauthenticated) : + seen.Path == AppendPath ? AppendSuccess() : + TrailersOnly(StatusCode.Unavailable) + ); + }; + + public static EventStoreClient Client(FakeKurrentDbHandler handler, string connectionString = "esdb://localhost:2113?tls=true", Action? configure = null) { + var settings = EventStoreClientSettings.Create(connectionString); + settings.CreateHttpMessageHandler = () => handler; + configure?.Invoke(settings); + + return new EventStoreClient(settings); + } +} +``` + +`test/Kurrent.Replicator.Tests/Auth/Support/TestEvents.cs`: + +```csharp +using System.Diagnostics; +using Kurrent.Replicator.Shared; +using Kurrent.Replicator.Shared.Contracts; + +namespace Kurrent.Replicator.Tests.Auth.Support; + +public static class TestEvents { + static EventDetails Details(string stream) => new(stream, Guid.NewGuid(), "TestEvent", ContentTypes.Json); + + static readonly TracingMetadata Tracing = new(ActivityTraceId.CreateRandom(), ActivitySpanId.CreateRandom()); + + public static ProposedEvent Proposed(string stream, long eventNumber = 0) + => new(Details(stream), "{}"u8.ToArray(), null, new LogPosition(eventNumber, (ulong)eventNumber + 1), eventNumber); + + public static ProposedMetaEvent Meta(string stream) + => new(Details(stream), new StreamMetadata(10, null, null, null, null), new LogPosition(0, 1), 0); + + public static ProposedDeleteStream Delete(string stream) => new(Details(stream), new LogPosition(0, 1), 0); + + public static OriginalEvent Original(string stream, long eventNumber, DateTimeOffset? created = null) + => new(created ?? DateTimeOffset.UtcNow, Details(stream), "{}"u8.ToArray(), null, new LogPosition(eventNumber, (ulong)eventNumber + 1), eventNumber, Tracing); +} +``` + +- [ ] **Step 2: Write the failing validator tests** + +`test/Kurrent.Replicator.Tests/Auth/GrpcAuthOptionsValidatorTests.cs`: + +```csharp +using EventStore.Client; +using Kurrent.Replicator.KurrentDb.Auth; + +namespace Kurrent.Replicator.Tests.Auth; + +public class GrpcAuthOptionsValidatorTests { + static readonly EventStoreClientSettings Tls = EventStoreClientSettings.Create("esdb://localhost:2113?tls=true"); + + static GrpcAuthOptions Cc(Func? f = null) { + var o = new GrpcAuthOptions { Type = GrpcAuthType.OAuthClientCredentials, TokenEndpoint = "https://idp/t", ClientId = "c", ClientSecret = "s" }; + + return f?.Invoke(o) ?? o; + } + + [Test] + public async Task Connection_string_mode_is_always_valid() { + var basic = EventStoreClientSettings.Create("esdb://admin:changeit@localhost:2113?tls=false"); + await Assert.That(GrpcAuthOptionsValidator.Validate(GrpcAuthOptions.Default, basic)).IsEmpty(); + } + + [Test] + public async Task Valid_client_credentials_pass() => await Assert.That(GrpcAuthOptionsValidator.Validate(Cc(), Tls)).IsEmpty(); + + [Test] + [Arguments("esdb://admin:changeit@localhost:2113?tls=true", "credentials")] + [Arguments("esdb://localhost:2113?tls=false", "tls")] + public async Task Connection_string_conflicts_are_errors(string cs, string expected) { + var errors = GrpcAuthOptionsValidator.Validate(Cc(), EventStoreClientSettings.Create(cs)); + await Assert.That(string.Join(";", errors)).Contains(expected); + } + + [Test] + public async Task Client_credentials_rules() { + async Task Expect(GrpcAuthOptions o, string fragment) + => await Assert.That(string.Join(";", GrpcAuthOptionsValidator.Validate(o, Tls))).Contains(fragment); + + await Expect(Cc(o => o with { TokenEndpoint = null }), "tokenEndpoint is required"); + await Expect(Cc(o => o with { TokenEndpoint = "http://idp.example.com/t" }), "https"); + await Expect(Cc(o => o with { TokenEndpoint = "/relative" }), "https"); + await Expect(Cc(o => o with { ClientId = " " }), "clientId is required"); + await Expect(Cc(o => o with { ClientSecret = null }), "exactly one of"); + await Expect(Cc(o => o with { ClientSecretFile = "/x" }), "exactly one of"); + await Expect(Cc(o => o with { ClientSecret = null, ClientSecretFile = "/does/not/exist" }), "clientSecretFile"); + await Expect(Cc(o => o with { ClientSecret = null, ClientAssertionFile = "/does/not/exist" }), "clientAssertionFile"); + await Expect(Cc(o => o with { RefreshBeforeExpirySeconds = -1 }), "refreshBeforeExpirySeconds"); + await Expect(Cc(o => o with { DefaultTokenLifetimeSeconds = 0 }), "defaultTokenLifetimeSeconds"); + await Expect(Cc(o => o with { AdditionalParameters = new Dictionary { ["Scope"] = "x", ["audience"] = "y" } }), "Scope"); + } + + [Test] + public async Task Loopback_http_is_allowed() { + await Assert.That(GrpcAuthOptionsValidator.Validate(Cc(o => o with { TokenEndpoint = "http://127.0.0.1:8080/t" }), Tls)).IsEmpty(); + await Assert.That(GrpcAuthOptionsValidator.Validate(Cc(o => o with { TokenEndpoint = "http://localhost:8080/t" }), Tls)).IsEmpty(); + } + + [Test] + public async Task Empty_secret_file_is_an_error() { + var file = Path.GetTempFileName(); + await File.WriteAllTextAsync(file, " \n"); + var errors = GrpcAuthOptionsValidator.Validate(Cc(o => o with { ClientSecret = null, ClientSecretFile = file }), Tls); + await Assert.That(string.Join(";", errors)).Contains("clientSecretFile"); + File.Delete(file); + } + + [Test] + public async Task Token_file_rules() { + var ok = new GrpcAuthOptions { Type = GrpcAuthType.OAuthTokenFile, TokenFile = "/not/yet/written" }; + await Assert.That(GrpcAuthOptionsValidator.Validate(ok, Tls)).IsEmpty(); // existence not required at startup + await Assert.That(string.Join(";", GrpcAuthOptionsValidator.Validate(ok with { TokenFile = null }, Tls))).Contains("tokenFile is required"); + await Assert.That(string.Join(";", GrpcAuthOptionsValidator.Validate(ok with { TokenFileReloadSeconds = 0 }, Tls))).Contains("tokenFileReloadSeconds"); + } + + [Test] + public async Task EnsureValid_reports_all_errors_with_side() { + var ex = await Assert.That(() => GrpcAuthOptionsValidator.EnsureValid("reader", Cc(o => o with { ClientId = null, TokenEndpoint = null }), Tls)) + .Throws(); + await Assert.That(ex!.Message).StartsWith("Invalid reader auth configuration: "); + await Assert.That(ex.Message).Contains("clientId is required"); + await Assert.That(ex.Message).Contains("tokenEndpoint is required"); + } + + [Test] + public async Task Protocol_must_be_grpc_for_oauth() { + await Assert.That(GrpcAuthOptionsValidator.ValidateProtocol(Cc(), "tcp")).IsNotNull(); + await Assert.That(GrpcAuthOptionsValidator.ValidateProtocol(Cc(), "GRPC")).IsNull(); + await Assert.That(GrpcAuthOptionsValidator.ValidateProtocol(GrpcAuthOptions.Default, "tcp")).IsNull(); + } + + [Test] + public async Task Oauth_options_with_connection_string_type_warn() { + var warnings = GrpcAuthOptionsValidator.Warnings(GrpcAuthOptions.Default with { ClientId = "c", TokenFile = "/t" }); + await Assert.That(string.Join(";", warnings)).Contains("clientId"); + await Assert.That(string.Join(";", warnings)).Contains("tokenFile"); + await Assert.That(GrpcAuthOptionsValidator.Warnings(Cc())).IsEmpty(); + } +} +``` + +- [ ] **Step 3: Write the failing GrpcAuthentication tests** + +`test/Kurrent.Replicator.Tests/Auth/GrpcAuthenticationTests.cs`: + +```csharp +using EventStore.Client; +using Grpc.Core; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class GrpcAuthenticationTests { + static EventData Event() => new(Uuid.NewUuid(), "T", "{}"u8.ToArray()); + + [Test] + public async Task Fake_handler_append_succeeds_with_basic_credentials() { + // Sanity check of the hand-encoded AppendResp: if this fails, fix FakeKurrentDbHandler before anything else. + var handler = new FakeKurrentDbHandler { Respond = _ => Task.FromResult(FakeKurrentDbHandler.AppendSuccess()) }; + var client = FakeKurrentDbHandler.Client(handler, "esdb://admin:changeit@localhost:2113?tls=true"); + var result = await client.AppendToStreamAsync("s", StreamState.Any, [Event()]); + await Assert.That(result.LogPosition.CommitPosition).IsEqualTo(1UL); + await Assert.That(handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath).Single()).StartsWith("Basic "); + } + + [Test] + public async Task Fallback_hook_fetches_from_source_when_no_per_call_credentials() { + var source = new ControllableTokenSource(new FakeTimeProvider()) { Value = "A" }; + var handler = new FakeKurrentDbHandler { Respond = _ => Task.FromResult(FakeKurrentDbHandler.AppendSuccess()) }; + FallbackUsage usage = null!; + var client = FakeKurrentDbHandler.Client(handler, configure: s => usage = GrpcAuthentication.Apply(s, source)); + + await client.AppendToStreamAsync("s", StreamState.Any, [Event()]); + source.Value = "B"; + await client.AppendToStreamAsync("s", StreamState.Any, [Event()]); + + await Assert.That(handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath)).IsEquivalentTo(new[] { "Bearer A", "Bearer B" }); + await Assert.That(usage.Count).IsEqualTo(2); + } + + [Test] + public async Task Per_call_credentials_bypass_the_fallback() { + var source = new ControllableTokenSource(new FakeTimeProvider()) { Value = "A" }; + var handler = new FakeKurrentDbHandler { Respond = _ => Task.FromResult(FakeKurrentDbHandler.AppendSuccess()) }; + FallbackUsage usage = null!; + var client = FakeKurrentDbHandler.Client(handler, configure: s => usage = GrpcAuthentication.Apply(s, source)); + + await client.AppendToStreamAsync("s", StreamState.Any, [Event()], userCredentials: new UserCredentials("X")); + await client.AppendToStreamAsync("s", StreamState.Any, [Event()], userCredentials: new UserCredentials("u", "p")); + + var headers = handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath); + await Assert.That(headers[0]).IsEqualTo("Bearer X"); + await Assert.That(headers[1]).StartsWith("Basic "); + await Assert.That(usage.Count).IsEqualTo(0); + } + + [Test] + public async Task Fallback_applies_to_streaming_reads() { + var source = new ControllableTokenSource(new FakeTimeProvider()) { Value = "A" }; + var handler = new FakeKurrentDbHandler(); + var client = FakeKurrentDbHandler.Client(handler, configure: s => GrpcAuthentication.Apply(s, source)); + + await Assert.That(async () => await client.ReadAllAsync(Direction.Forwards, Position.Start, 1).ToArrayAsync()).Throws(); + await Assert.That(handler.AuthorizationsFor(FakeKurrentDbHandler.ReadPath).First()).IsEqualTo("Bearer A"); + } + + [Test] + public async Task CreateSource_matches_the_type() { + var time = TimeProvider.System; + await Assert.That(GrpcAuthentication.CreateSource(GrpcAuthOptions.Default, "sink", time, default)).IsNull(); + await Assert.That(GrpcAuthentication.CreateSource(new() { Type = GrpcAuthType.OAuthTokenFile, TokenFile = "/t" }, "sink", time, default)).IsTypeOf(); + + var cc = new GrpcAuthOptions { Type = GrpcAuthType.OAuthClientCredentials, TokenEndpoint = "https://idp/t", ClientId = "c", ClientSecret = "s" }; + await Assert.That(GrpcAuthentication.CreateSource(cc, "sink", time, default)).IsTypeOf(); + } +} +``` + +`ToArrayAsync` on `ReadStreamResult` comes from `System.Linq.Async`, which the client already references transitively; if the compiler cannot find it, add `` to the test project. + +- [ ] **Step 4: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, `GrpcAuthOptionsValidator` / `GrpcAuthentication` not found. + +- [ ] **Step 5: Implement the validator** + +`src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthOptionsValidator.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +public static class GrpcAuthOptionsValidator { + static readonly HashSet Reserved = new(StringComparer.OrdinalIgnoreCase) { + "grant_type", "client_id", "client_secret", "client_assertion", "client_assertion_type", "scope" + }; + + public static IReadOnlyList Validate(GrpcAuthOptions o, EventStoreClientSettings settings) { + var errors = new List(); + + if (!o.IsOAuth) return errors; + + if (settings.DefaultCredentials != null) + errors.Add("the connection string contains credentials (user:pass@); remove them when using OAuth"); + + if (settings.ConnectivitySettings.Insecure) + errors.Add("tls must be enabled for OAuth (bearer tokens are not sent over insecure channels)"); + + if (o.Type == GrpcAuthType.OAuthClientCredentials) ValidateClientCredentials(o, errors); + else ValidateTokenFile(o, errors); + + return errors; + } + + static void ValidateClientCredentials(GrpcAuthOptions o, List errors) { + if (string.IsNullOrWhiteSpace(o.TokenEndpoint)) { + errors.Add("tokenEndpoint is required"); + } + else if (!Uri.TryCreate(o.TokenEndpoint, UriKind.Absolute, out var uri) || + !(uri.Scheme == Uri.UriSchemeHttps || (uri.Scheme == Uri.UriSchemeHttp && uri.IsLoopback))) { + errors.Add("tokenEndpoint must be an absolute https URL (http is only allowed for loopback hosts)"); + } + + if (string.IsNullOrWhiteSpace(o.ClientId)) errors.Add("clientId is required"); + + var credentialCount = new[] { o.ClientSecret, o.ClientSecretFile, o.ClientAssertionFile }.Count(x => !string.IsNullOrEmpty(x)); + + if (credentialCount != 1) errors.Add("exactly one of clientSecret, clientSecretFile or clientAssertionFile must be set"); + + if (!string.IsNullOrEmpty(o.ClientSecretFile) && (!File.Exists(o.ClientSecretFile) || File.ReadAllText(o.ClientSecretFile).Trim().Length == 0)) + errors.Add($"clientSecretFile {o.ClientSecretFile} does not exist or is empty"); + + if (!string.IsNullOrEmpty(o.ClientAssertionFile) && !File.Exists(o.ClientAssertionFile)) + errors.Add($"clientAssertionFile {o.ClientAssertionFile} does not exist"); + + if (o.RefreshBeforeExpirySeconds < 0) errors.Add("refreshBeforeExpirySeconds must be >= 0"); + + if (o.DefaultTokenLifetimeSeconds is <= 0) errors.Add("defaultTokenLifetimeSeconds must be > 0"); + + var reserved = o.AdditionalParameters.Keys.Where(Reserved.Contains).ToList(); + + if (reserved.Count > 0) errors.Add($"additionalParameters must not override {string.Join(", ", reserved)}"); + } + + static void ValidateTokenFile(GrpcAuthOptions o, List errors) { + if (string.IsNullOrWhiteSpace(o.TokenFile)) errors.Add("tokenFile is required"); + + if (o.TokenFileReloadSeconds <= 0) errors.Add("tokenFileReloadSeconds must be > 0"); + } + + public static IReadOnlyList Warnings(GrpcAuthOptions o) { + if (o.IsOAuth) return []; + + var set = new List(); + if (o.TokenEndpoint != null) set.Add("tokenEndpoint"); + if (o.ClientId != null) set.Add("clientId"); + if (o.ClientSecret != null) set.Add("clientSecret"); + if (o.ClientSecretFile != null) set.Add("clientSecretFile"); + if (o.ClientAssertionFile != null) set.Add("clientAssertionFile"); + if (o.Scope != null) set.Add("scope"); + if (o.AdditionalParameters.Count > 0) set.Add("additionalParameters"); + if (o.DefaultTokenLifetimeSeconds != null) set.Add("defaultTokenLifetimeSeconds"); + if (o.TokenFile != null) set.Add("tokenFile"); + + return set.Count == 0 ? [] : [$"auth options {string.Join(", ", set)} are ignored because auth.type is connectionString"]; + } + + public static string? ValidateProtocol(GrpcAuthOptions o, string? protocol) + => o.IsOAuth && !string.Equals(protocol, "grpc", StringComparison.OrdinalIgnoreCase) + ? $"auth.type {o.Type} is only supported with protocol grpc (configured: {protocol ?? "none"})" + : null; + + public static void EnsureValid(string side, GrpcAuthOptions o, EventStoreClientSettings settings) { + var errors = Validate(o, settings); + + if (errors.Count > 0) throw new InvalidOperationException($"Invalid {side} auth configuration: {string.Join("; ", errors)}"); + } +} +``` + +- [ ] **Step 6: Implement GrpcAuthentication** + +`src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthentication.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +public sealed class FallbackUsage { + int _count; + + public int Count => Volatile.Read(ref _count); + + internal void Increment() => Interlocked.Increment(ref _count); +} + +public static class GrpcAuthentication { + static ILog Log => LogProvider.GetLogger(typeof(GrpcAuthentication)); + + /// Marks "no per-call credentials were passed"; never sent as-is. + internal static readonly UserCredentials Sentinel = new("replicator-oauth-sentinel"); + + public static FallbackUsage Apply(EventStoreClientSettings settings, IAccessTokenSource source) { + var usage = new FallbackUsage(); + settings.DefaultCredentials = Sentinel; + + settings.OperationOptions.GetAuthenticationHeaderValue = async (credentials, ct) => { + if (!ReferenceEquals(credentials, Sentinel)) return credentials.ToString(); + + usage.Increment(); + Log.Debug("gRPC call made without per-call credentials; using the fallback token"); + var lease = await source.GetAccessToken(ct).ConfigureAwait(false); + + return $"Bearer {lease.Value}"; + }; + + return usage; + } + + public static IAccessTokenSource? CreateSource(GrpcAuthOptions options, string side, TimeProvider time, CancellationToken shutdown) + => options.Type switch { + GrpcAuthType.ConnectionString => null, + GrpcAuthType.OAuthClientCredentials => new ClientCredentialsTokenSource(options, side, TokenEndpointClient.CreateDefaultHandler(), time, shutdown), + GrpcAuthType.OAuthTokenFile => new TokenFileSource(options.TokenFile!, TimeSpan.FromSeconds(options.TokenFileReloadSeconds), time, side), + _ => throw new ArgumentOutOfRangeException(nameof(options), options.Type, "Unknown auth type") + }; +} +``` + +- [ ] **Step 7: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/GrpcAuth*/*"` +Expected: all PASS. If `Fake_handler_append_succeeds_with_basic_credentials` fails, print `handler.Requests` to see which RPC paths the client used (for example, a `BatchAppend` path means feature discovery was not answered as Unimplemented) and fix the fake before continuing. Tasks 8 and 11 depend on it. + +- [ ] **Step 8: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/Auth test/Kurrent.Replicator.Tests/Auth +git commit -m "feat(auth): options validation, fallback header hook, source factory + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 8: GrpcEventWriter through `auth.Run` + +**Files:** +- Modify: `src/Kurrent.Replicator.KurrentDb/GrpcEventWriter.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/GrpcEventWriterAuthTests.cs` + +**Interfaces:** +- Consumes: `GrpcAuthContext.Run`, `CallAuth`, `FakeKurrentDbHandler`, `ControllableTokenSource`, `TokenFileSource`, `TestEvents`, `TimeDriver`. +- Produces: `public GrpcEventWriter(EventStoreClient client, GrpcAuthContext auth)`; the existing `GrpcEventWriter(EventStoreClient client)` stays (delegates to `GrpcAuthContext.None`), so the container tests compile unchanged. + +- [ ] **Step 1: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/GrpcEventWriterAuthTests.cs`: + +```csharp +using Grpc.Core; +using Kurrent.Replicator.KurrentDb; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class GrpcEventWriterAuthTests { + readonly FakeTimeProvider _time = new(); + readonly CancellationTokenSource _shutdown = new(); + readonly FakeKurrentDbHandler _handler = new(); + readonly string _file = Path.Combine(Path.GetTempPath(), $"replicator-writer-{Guid.NewGuid():N}"); + + [After(Test)] + public void Cleanup() { + if (File.Exists(_file)) File.Delete(_file); + } + + (GrpcEventWriter Writer, FallbackUsage Usage) Writer(IAccessTokenSource source) { + FallbackUsage usage = null!; + var client = FakeKurrentDbHandler.Client(_handler, configure: s => usage = GrpcAuthentication.Apply(s, source)); + + return (new GrpcEventWriter(client, new GrpcAuthContext(source, _shutdown.Token, _time, "sink")), usage); + } + + [Test] + public async Task Every_write_kind_sends_its_own_bearer_token() { + var source = new ControllableTokenSource(_time) { Value = "A" }; + var (writer, usage) = Writer(source); + _handler.Respond = s => Task.FromResult(s.Path == FakeKurrentDbHandler.AppendPath ? FakeKurrentDbHandler.AppendSuccess() : FakeKurrentDbHandler.TrailersOnly(StatusCode.Unavailable)); + + await writer.WriteEvent(TestEvents.Proposed("s"), default); + await writer.WriteEvent(TestEvents.Meta("s"), default); // metadata is an Append to $$s + await Assert.That(async () => await writer.WriteEvent(TestEvents.Delete("s"), default)).Throws(); + + await Assert.That(_handler.Requests.Select(r => r.Authorization).Distinct().Single()).IsEqualTo("Bearer A"); + await Assert.That(_handler.AuthorizationsFor(FakeKurrentDbHandler.DeletePath)).IsNotEmpty(); + await Assert.That(usage.Count).IsEqualTo(0); + } + + [Test] + public async Task Write_waits_out_a_token_outage_then_sends_the_new_token() { + var source = new ControllableTokenSource(_time); + var (writer, _) = Writer(source); + _handler.Respond = FakeKurrentDbHandler.AcceptBearer("A"); + + var write = writer.WriteEvent(TestEvents.Proposed("s"), default); + await TimeDriver.Until(() => source.Calls >= 3, _time); + await Assert.That(write.IsCompleted).IsFalse(); + await Assert.That(_handler.Requests).IsEmpty(); + + source.Value = "A"; + await TimeDriver.Drive(write, _time); + await Assert.That(_handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath).Single()).IsEqualTo("Bearer A"); + } + + [Test] + public async Task Shutdown_during_outage_ends_the_write_with_cancellation() { + var source = new ControllableTokenSource(_time); + var (writer, _) = Writer(source); + var write = writer.WriteEvent(TestEvents.Proposed("s"), CancellationToken.None); + await TimeDriver.Until(() => source.Calls >= 2, _time); + await _shutdown.CancelAsync(); + await Assert.That(async () => await write.WaitAsync(TimeSpan.FromSeconds(5))).Throws(); + } + + [Test] + public async Task Rejected_file_token_is_replaced_by_rotated_file_without_waiting_for_reload() { + await File.WriteAllTextAsync(_file, "A"); + var source = new TokenFileSource(_file, TimeSpan.FromHours(1), _time, "sink"); + var (writer, _) = Writer(source); + _handler.Respond = FakeKurrentDbHandler.AcceptBearer("B"); + + var write = writer.WriteEvent(TestEvents.Proposed("s"), default); + await TimeDriver.Until(() => _handler.Requests.Count >= 1, _time); + await File.WriteAllTextAsync(_file, "B"); + await TimeDriver.Drive(write, _time); + + var headers = _handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath); + await Assert.That(headers.First()).IsEqualTo("Bearer A"); + await Assert.That(headers.Last()).IsEqualTo("Bearer B"); + await Assert.That(headers.Count(h => h == "Bearer A")).IsEqualTo(1); + } + + [Test] + public async Task Removed_file_after_rejection_never_resends_the_rejected_token() { + await File.WriteAllTextAsync(_file, "A"); + var source = new TokenFileSource(_file, TimeSpan.FromSeconds(30), _time, "sink"); + var (writer, _) = Writer(source); + _handler.Respond = FakeKurrentDbHandler.AcceptBearer("B"); + + var write = writer.WriteEvent(TestEvents.Proposed("s"), default); + await TimeDriver.Until(() => _handler.Requests.Count >= 1, _time); + File.Delete(_file); + _time.Advance(TimeSpan.FromSeconds(120)); // past quarantine: file is missing, so nothing to probe + await Task.Delay(50); + await File.WriteAllTextAsync(_file, "B"); + await TimeDriver.Drive(write, _time); + + var headers = _handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath); + await Assert.That(headers.Count(h => h == "Bearer A")).IsEqualTo(1); + await Assert.That(headers.Last()).IsEqualTo("Bearer B"); + } + + [Test] + public async Task Server_side_fix_is_detected_by_the_probe_after_quarantine() { + await File.WriteAllTextAsync(_file, "A"); + var source = new TokenFileSource(_file, TimeSpan.FromSeconds(30), _time, "sink"); + var (writer, _) = Writer(source); + var acceptA = false; + _handler.Respond = s => Task.FromResult(Volatile.Read(ref acceptA) ? FakeKurrentDbHandler.AppendSuccess() : FakeKurrentDbHandler.TrailersOnly(StatusCode.Unauthenticated)); + + var write = writer.WriteEvent(TestEvents.Proposed("s"), default); + await TimeDriver.Until(() => _handler.Requests.Count >= 1, _time); + Volatile.Write(ref acceptA, true); + await TimeDriver.Drive(write, _time); + + await Assert.That(_handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath)).IsEquivalentTo(new[] { "Bearer A", "Bearer A" }); + } + + [Test] + public async Task Only_one_concurrent_write_probes_a_quarantined_token() { + await File.WriteAllTextAsync(_file, "A"); + var source = new TokenFileSource(_file, TimeSpan.FromSeconds(30), _time, "sink"); + var (writer, _) = Writer(source); + + var acceptA = false; + var probeHold = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var afterFlip = 0; + + _handler.Respond = async s => { + if (!Volatile.Read(ref acceptA)) return FakeKurrentDbHandler.TrailersOnly(StatusCode.Unauthenticated); + + if (Interlocked.Increment(ref afterFlip) == 1) await probeHold.Task; + + return FakeKurrentDbHandler.AppendSuccess(); + }; + + var first = writer.WriteEvent(TestEvents.Proposed("s"), default); // gets rejected -> quarantine + await TimeDriver.Until(() => _handler.Requests.Count >= 1, _time); + Volatile.Write(ref acceptA, true); + + var others = Enumerable.Range(0, 5).Select(i => writer.WriteEvent(TestEvents.Proposed("s", i + 1), default)).ToList(); + + await TimeDriver.Until(() => Volatile.Read(ref afterFlip) >= 1, _time); + for (var i = 0; i < 40; i++) { _time.Advance(TimeSpan.FromSeconds(1)); await Task.Delay(5); } // others keep backing off + await Assert.That(Volatile.Read(ref afterFlip)).IsEqualTo(1); + + probeHold.SetResult(); + await TimeDriver.Drive(Task.WhenAll(others.Append(first)), _time); + await Assert.That(Volatile.Read(ref afterFlip)).IsEqualTo(6); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL, no `GrpcEventWriter(EventStoreClient, GrpcAuthContext)` constructor. + +- [ ] **Step 3: Implement** + +In `src/Kurrent.Replicator.KurrentDb/GrpcEventWriter.cs`, change the class header and the three operations (everything else unchanged): + +```csharp +public class GrpcEventWriter(EventStoreClient client, GrpcAuthContext auth) : IEventWriter { + static readonly ILog Log = LogProvider.GetCurrentClassLogger(); + + public GrpcEventWriter(EventStoreClient client) : this(client, GrpcAuthContext.None) { } +``` + +```csharp + var result = await auth.Run( + (a, c) => client.AppendToStreamAsync( + proposedEvent.EventDetails.Stream, + StreamState.Any, + [Map(p)], + userCredentials: a.Credentials, + cancellationToken: c + ), + cancellationToken + ) + .ConfigureAwait(false); +``` + +```csharp + var result = await auth.Run( + (a, c) => client.DeleteAsync(stream, StreamState.Any, userCredentials: a.Credentials, cancellationToken: c), + cancellationToken + ) + .ConfigureAwait(false); +``` + +```csharp + var result = await auth.Run( + (a, c) => client.SetStreamMetadataAsync( + meta.EventDetails.Stream, + StreamState.Any, + new( + meta.Data.MaxCount, + meta.Data.MaxAge, + ValueOrNull(meta.Data.TruncateBefore, x => new StreamPosition((ulong)x!)), + meta.Data.CacheControl, + ValueOrNull( + meta.Data.StreamAcl, + x => new StreamAcl(x.ReadRoles, x.WriteRoles, x.DeleteRoles, x.MetaReadRoles, x.MetaWriteRoles) + ) + ), + userCredentials: a.Credentials, + cancellationToken: c + ), + cancellationToken + ) + .ConfigureAwait(false); +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/GrpcEventWriterAuthTests/*"` +Expected: 7 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb/GrpcEventWriter.cs test/Kurrent.Replicator.Tests/Auth/GrpcEventWriterAuthTests.cs +git commit -m "feat(auth): sink writes carry per-call tokens and recover from token failures + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 9: StreamMetaCache live state and fail-closed scavenge filter + +**Files:** +- Modify: `src/Kurrent.Replicator.KurrentDb/StreamMetaCache.cs` +- Modify: `src/Kurrent.Replicator.KurrentDb/EventFilters.cs` +- Modify: `src/Kurrent.Replicator.KurrentDb/ConnectionExtensions.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/StreamMetaCacheTests.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/ScavengedEventsFilterAuthTests.cs` + +**Interfaces:** +- Consumes: `AuthFailure`, `GrpcAuthContext.Run`, `CallAuth`. +- Produces: + - `class StreamMetaCache(bool failClosedOnAuthErrors = false)` — new `bool IsLive`, `void MarkNotLive()`, `void MarkLive()`; existing `GetOrAddStreamMeta`, `GetOrAddStreamSize`, `UpdateStreamMeta`, `UpdateStreamLastEventNumber` keep their signatures. Starts **not live**. + - `class ScavengedEventsFilter` — ctor `(EventStoreClient client, StreamMetaCache cache, GrpcAuthContext auth)` and internal ctor `(Func> readMeta, Func> readSize, StreamMetaCache cache, GrpcAuthContext auth)`; `ValueTask Filter(BaseOriginalEvent)` unchanged. + - `ConnectionExtensions.GetStreamSize(this EventStoreClient, string stream, UserCredentials? credentials = null, CancellationToken ct = default)` and `GetStreamMeta(...)` with the same extra parameters. + +- [ ] **Step 1: Write the failing cache tests** + +`test/Kurrent.Replicator.Tests/Auth/StreamMetaCacheTests.cs`: + +```csharp +using EventStore.Client; +using Grpc.Core; +using Kurrent.Replicator.KurrentDb; +using Kurrent.Replicator.KurrentDb.Auth; + +namespace Kurrent.Replicator.Tests.Auth; + +public class StreamMetaCacheTests { + static readonly StreamMeta Meta = new(false, null, null, 0); + + static Exception Denied() => new AccessDeniedException("x", new RpcException(new Status(StatusCode.PermissionDenied, "x"))); + static Exception Unauthenticated() => new NotAuthenticatedException("x", new RpcException(new Status(StatusCode.Unauthenticated, "x"))); + + [Test] + public async Task Not_live_never_caches() { + var cache = new StreamMetaCache(); + var calls = 0; + await cache.GetOrAddStreamMeta("s", _ => { calls++; return Task.FromResult(Meta); }); + await cache.GetOrAddStreamMeta("s", _ => { calls++; return Task.FromResult(Meta); }); + await Assert.That(cache.IsLive).IsFalse(); + await Assert.That(calls).IsEqualTo(2); + } + + [Test] + public async Task Live_caches_and_MarkLive_clears() { + var cache = new StreamMetaCache(); + cache.MarkLive(); + var calls = 0; + await cache.GetOrAddStreamMeta("s", _ => { calls++; return Task.FromResult(Meta); }); + await cache.GetOrAddStreamMeta("s", _ => { calls++; return Task.FromResult(Meta); }); + await Assert.That(calls).IsEqualTo(1); + + await cache.GetOrAddStreamSize("s", _ => { calls++; return Task.FromResult(new StreamSize(5)); }); + cache.MarkNotLive(); + cache.MarkLive(); + await cache.GetOrAddStreamMeta("s", _ => { calls++; return Task.FromResult(Meta); }); + await cache.GetOrAddStreamSize("s", _ => { calls++; return Task.FromResult(new StreamSize(5)); }); + await Assert.That(calls).IsEqualTo(4); + } + + [Test] + public async Task Fetch_started_before_MarkLive_is_not_stored() { + var cache = new StreamMetaCache(); + var slow = new TaskCompletionSource(); + var fetch = cache.GetOrAddStreamMeta("s", _ => slow.Task); + cache.MarkLive(); + slow.SetResult(Meta with { IsDeleted = true }); + await fetch; + + var fresh = await cache.GetOrAddStreamMeta("s", _ => Task.FromResult(Meta)); + await Assert.That(fresh!.IsDeleted).IsFalse(); + } + + [Test] + public async Task Without_flag_every_failure_returns_null_as_today() { + var cache = new StreamMetaCache(); + await Assert.That(await cache.GetOrAddStreamMeta("s", _ => throw Denied())).IsNull(); + await Assert.That(await cache.GetOrAddStreamMeta("s", _ => throw Unauthenticated())).IsNull(); + await Assert.That(await cache.GetOrAddStreamMeta("s", _ => throw new InvalidOperationException())).IsNull(); + } + + [Test] + public async Task With_flag_auth_failures_propagate_and_others_return_null() { + var cache = new StreamMetaCache(failClosedOnAuthErrors: true); + await Assert.That(async () => await cache.GetOrAddStreamMeta("s", _ => throw Denied())).Throws(); + await Assert.That(async () => await cache.GetOrAddStreamMeta("s", _ => throw Unauthenticated())).Throws(); + await Assert.That(async () => await cache.GetOrAddStreamMeta("s", _ => throw new OAuthTokenException("x"))).Throws(); + await Assert.That(async () => await cache.GetOrAddStreamMeta("s", _ => throw new OperationCanceledException())).Throws(); + await Assert.That(await cache.GetOrAddStreamMeta("s", _ => throw new InvalidOperationException())).IsNull(); + } +} +``` + +- [ ] **Step 2: Write the failing filter tests** + +`test/Kurrent.Replicator.Tests/Auth/ScavengedEventsFilterAuthTests.cs`: + +```csharp +using EventStore.Client; +using Grpc.Core; +using Kurrent.Replicator.KurrentDb; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class ScavengedEventsFilterAuthTests { + readonly FakeTimeProvider _time = new(); + readonly ControllableTokenSource _source; + readonly GrpcAuthContext _auth; + + public ScavengedEventsFilterAuthTests() { + _source = new(_time) { Value = "A" }; + _auth = new(_source, CancellationToken.None, _time, "reader"); + } + + static readonly StreamMeta Deleted = new(true, null, null, 0); + static readonly StreamMeta Plain = new(false, null, null, 0); + + static Task NoSize(string s, CallAuth a, CancellationToken c) => Task.FromResult(new StreamSize(0)); + + [Test] + public async Task Oauth_permission_denied_fails_closed() { + var filter = new ScavengedEventsFilter( + (_, _, _) => throw new AccessDeniedException("x", new RpcException(new Status(StatusCode.PermissionDenied, "x"))), + NoSize, new StreamMetaCache(true), _auth + ); + await Assert.That(async () => await filter.Filter(TestEvents.Original("s", 0))).Throws(); + } + + [Test] + public async Task Basic_auth_permission_denied_keeps_today_behaviour() { + var filter = new ScavengedEventsFilter( + (_, _, _) => throw new AccessDeniedException("x", new RpcException(new Status(StatusCode.PermissionDenied, "x"))), + NoSize, new StreamMetaCache(), GrpcAuthContext.None + ); + await Assert.That(await filter.Filter(TestEvents.Original("s", 0))).IsTrue(); + } + + [Test] + public async Task Transient_unauthenticated_is_retried_and_the_filter_then_decides_correctly() { + var calls = 0; + var filter = new ScavengedEventsFilter( + (_, _, _) => ++calls == 1 + ? throw new NotAuthenticatedException("x", new RpcException(new Status(StatusCode.Unauthenticated, "x"))) + : Task.FromResult(Deleted), + NoSize, new StreamMetaCache(true), _auth + ); + _source.OnInvalidate = _ => _source.Value = "B"; + + var keep = await TimeDriver.Drive(filter.Filter(TestEvents.Original("s", 0)).AsTask(), _time); + await Assert.That(keep).IsFalse(); + await Assert.That(calls).IsEqualTo(2); + } + + [Test] + public async Task Token_outage_after_read_began_makes_the_filter_wait() { + var filter = new ScavengedEventsFilter((_, _, _) => Task.FromResult(Deleted), NoSize, new StreamMetaCache(true), _auth); + await Assert.That(await filter.Filter(TestEvents.Original("s", 0))).IsFalse(); + + _source.Value = null; // outage + var pending = filter.Filter(TestEvents.Original("s", 1)).AsTask(); + await TimeDriver.Until(() => _source.Calls >= 4, _time); + await Assert.That(pending.IsCompleted).IsFalse(); + _source.Value = "B"; + await Assert.That(await TimeDriver.Drive(pending, _time)).IsFalse(); + } + + [Test] + public async Task Metadata_and_size_reads_carry_per_call_bearer() { + var handler = new FakeKurrentDbHandler(); + FallbackUsage usage = null!; + var client = FakeKurrentDbHandler.Client(handler, configure: s => usage = GrpcAuthentication.Apply(s, _source)); + var filter = new ScavengedEventsFilter(client, new StreamMetaCache(true), _auth); + + await filter.Filter(TestEvents.Original("s", 0)); // Unavailable is not an auth error -> null meta -> keep + await Assert.That(handler.AuthorizationsFor(FakeKurrentDbHandler.ReadPath).Single()).IsEqualTo("Bearer A"); + await Assert.That(_source.Accepted).IsEmpty(); + await Assert.That(usage.Count).IsEqualTo(0); // per-call credentials, not the sentinel fallback + } +} +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL (`MarkLive`, new `StreamMetaCache` ctor, new `ScavengedEventsFilter` ctors missing). + +- [ ] **Step 4: Implement StreamMetaCache** + +Replace the top of `src/Kurrent.Replicator.KurrentDb/StreamMetaCache.cs` (class declaration through `GetOrAddStreamSize`), keeping `UpdateStreamMeta`, `UpdateStreamLastEventNumber`, `IsStreamDeleted` and the `StreamSize` / `StreamMeta` records unchanged: + +```csharp +using System.Collections.Concurrent; +using Kurrent.Replicator.Shared.Logging; + +namespace Kurrent.Replicator.KurrentDb; + +/// +/// Metadata and last-event-number cache used by the scavenge filter. Only trusted while "live", i.e. while the +/// realtime subscription is delivering updates; otherwise every read goes to the server. +/// +class StreamMetaCache(bool failClosedOnAuthErrors = false) { + static readonly ILog Log = LogProvider.GetCurrentClassLogger(); + + readonly ConcurrentDictionary _streamsSize = new(); + readonly ConcurrentDictionary _streamsMeta = new(); + readonly object _lock = new(); + + bool _live; + long _epoch; + + public bool IsLive { + get { lock (_lock) return _live; } + } + + public void MarkNotLive() { + lock (_lock) _live = false; + } + + public void MarkLive() { + lock (_lock) { + _streamsMeta.Clear(); + _streamsSize.Clear(); + _epoch++; + _live = true; + } + } + + public async Task GetOrAddStreamMeta(string stream, Func> getMeta) { + try { + return await GetOrFetch(_streamsMeta, stream, getMeta).ConfigureAwait(false); + } catch (Exception e) when (!FailClosed(e)) { + Log.Warn(e, "Unable to read metadata for stream {Stream}", stream); + + return null; + } + } + + public Task GetOrAddStreamSize(string stream, Func> getSize) + => GetOrFetch(_streamsSize, stream, getSize); + + bool FailClosed(Exception e) + => failClosedOnAuthErrors && (e is OperationCanceledException || AuthFailure.IsTokenFailure(e) || AuthFailure.IsPermissionDenied(e)); + + async Task GetOrFetch(ConcurrentDictionary dict, string key, Func> fetch) { + long epoch; + + lock (_lock) { + epoch = _epoch; + + if (_live && dict.TryGetValue(key, out var cached)) return cached; + } + + var value = await fetch(key).ConfigureAwait(false); + + lock (_lock) { + if (_live && _epoch == epoch) dict.TryAdd(key, value); + } + + return value; + } +``` + +Remove the now-unused `using Kurrent.Replicator.Shared.Extensions;`. + +- [ ] **Step 5: Implement ConnectionExtensions and ScavengedEventsFilter** + +`src/Kurrent.Replicator.KurrentDb/ConnectionExtensions.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb; + +static class ConnectionExtensions { + public static async Task GetStreamSize(this EventStoreClient client, string stream, UserCredentials? credentials = null, CancellationToken ct = default) { + var read = client.ReadStreamAsync(Direction.Backwards, stream, StreamPosition.End, 1, userCredentials: credentials, cancellationToken: ct); + var last = await read.ToArrayAsync(ct).ConfigureAwait(false); + + return new(last[0].OriginalEventNumber.ToInt64()); + } + + public static async Task GetStreamMeta(this EventStoreClient client, string stream, UserCredentials? credentials = null, CancellationToken ct = default) { + var streamMeta = await client.GetStreamMetadataAsync(stream, userCredentials: credentials, cancellationToken: ct).ConfigureAwait(false); + + var streamDeleted = streamMeta.StreamDeleted || streamMeta.Metadata.TruncateBefore == StreamPosition.End; + + return new( + streamDeleted, + streamMeta.Metadata.MaxAge, + streamMeta.Metadata.MaxCount, + streamMeta.MetastreamRevision!.Value.ToInt64() + ); + } +} +``` + +`src/Kurrent.Replicator.KurrentDb/EventFilters.cs`: + +```csharp +using Kurrent.Replicator.Shared.Contracts; + +namespace Kurrent.Replicator.KurrentDb; + +class ScavengedEventsFilter( + Func> readMeta, + Func> readSize, + StreamMetaCache cache, + GrpcAuthContext auth + ) { + public ScavengedEventsFilter(EventStoreClient client, StreamMetaCache cache, GrpcAuthContext auth) + : this( + (s, a, c) => client.GetStreamMeta(s, a.Credentials, c), + (s, a, c) => client.GetStreamSize(s, a.Credentials, c), + cache, + auth + ) { } + + public async ValueTask Filter(BaseOriginalEvent originalEvent) { + var stream = originalEvent.EventDetails.Stream; + + var meta = await cache.GetOrAddStreamMeta(stream, s => auth.Run((a, c) => readMeta(s, a, c), CancellationToken.None)).ConfigureAwait(false); + + return meta == null || !meta.IsDeleted && !TtlExpired() && !await OverMaxCount().ConfigureAwait(false); + + bool TtlExpired() => meta.MaxAge.HasValue && originalEvent.Created < DateTime.Now - meta.MaxAge; + + // add the check timestamp, so we can check again if we get newer events (edge case) + async Task OverMaxCount() { + if (!meta.MaxCount.HasValue) + return false; + + var streamSize = await cache.GetOrAddStreamSize(stream, s => auth.Run((a, c) => readSize(s, a, c), CancellationToken.None)).ConfigureAwait(false); + + return originalEvent.LogPosition.EventNumber < streamSize.LastEventNumber - meta.MaxCount; + } + } +} +``` + +`GrpcEventReader` still constructs `new(client, metaCache)`; it is updated in Task 11. To keep the build green now, change that line in `GrpcEventReader`'s constructor to `_filter = new(client, metaCache, GrpcAuthContext.None);`, and add `metaCache.MarkLive();` right after `_realtime = new(client, metaCache);` so the cache behaves as before until Task 10/11 wire the live state properly. + +- [ ] **Step 6: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/StreamMetaCacheTests/*"` and the same for `ScavengedEventsFilterAuthTests`. +Expected: all PASS. + +- [ ] **Step 7: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb test/Kurrent.Replicator.Tests/Auth +git commit -m "feat(auth): fail-closed scavenge filter and live-state metadata cache + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- +### Task 10: Realtime — serialized attempts, drop classification, cache coherence + +Rewrites `Realtime` per the spec's "Realtime subscription" section. Read that section in full before starting; every bullet maps to code below. + +**Files:** +- Modify (rewrite): `src/Kurrent.Replicator.KurrentDb/Realtime.cs` +- Modify: `src/Kurrent.Replicator.KurrentDb/GrpcEventReader.cs` (constructor + `Start` call only) +- Test: `test/Kurrent.Replicator.Tests/Auth/RealtimeTests.cs` + +**Interfaces:** +- Consumes: `GrpcAuthContext` (`Run`, `ReportFailure`, `Shutdown`, `Time`), `CallAuth`, `StreamMetaCache.MarkLive/MarkNotLive`, `TokenGate.Backoff`. +- Produces: + - `class Realtime` — public ctor `(EventStoreClient client, StreamMetaCache cache, GrpcAuthContext auth)`; internal ctor `(Realtime.Subscribe subscribe, StreamMetaCache cache, GrpcAuthContext auth)`; `Task Start(CancellationToken ct)`; `internal Action? OnUnpublishedUnderLock` (test hook); `internal bool IsPublished`. + - `internal delegate Task Subscribe(Func onEvent, Action onDropped, CallAuth auth, CancellationToken ct)` nested in `Realtime`. + - `internal sealed class SubscriptionDroppedEarlyException : Exception`. + +- [ ] **Step 1: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/RealtimeTests.cs`: + +```csharp +using System.Collections.Concurrent; +using EventStore.Client; +using Grpc.Core; +using Kurrent.Replicator.KurrentDb; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class RealtimeTests { + sealed class FakeSubscription : IDisposable { + public bool Disposed; + public void Dispose() => Disposed = true; + } + + sealed class Call { + public required Func OnEvent; + public required Action OnDropped; + public required CallAuth Auth; + public readonly TaskCompletionSource Result = new(TaskCreationOptions.RunContinuationsAsynchronously); + public readonly FakeSubscription Subscription = new(); + public string? Token => Auth.Lease?.Value; + public void Succeed() => Result.SetResult(Subscription); + public void Fail() => Result.SetException(new RpcException(new Status(StatusCode.Unavailable, "down"))); + } + + readonly FakeTimeProvider _time = new(); + readonly CancellationTokenSource _shutdown = new(); + readonly ControllableTokenSource _source; + readonly ConcurrentQueue _calls = new(); + readonly StreamMetaCache _cache = new(failClosedOnAuthErrors: true); + readonly Realtime _realtime; + + public RealtimeTests() { + _source = new(_time) { Value = "A" }; + var auth = new GrpcAuthContext(_source, _shutdown.Token, _time, "reader"); + + _realtime = new Realtime( + (onEvent, onDropped, a, _) => { + var call = new Call { OnEvent = onEvent, OnDropped = onDropped, Auth = a }; + _calls.Enqueue(call); + + return call.Result.Task; + }, + _cache, + auth + ); + } + + Call Nth(int n) => _calls.ElementAt(n); + + async Task WaitForCall(int n) { + await TimeDriver.Until(() => _calls.Count > n, _time); + + return Nth(n); + } + + async Task Publish(int n) { + var start = _realtime.Start(default); + (await WaitForCall(n)).Succeed(); + await TimeDriver.Drive(start, _time); + } + + static NotAuthenticatedException Unauthenticated() => new("x", new RpcException(new Status(StatusCode.Unauthenticated, "x"))); + + [Test] + public async Task First_subscribe_marks_cache_live() { + await Assert.That(_cache.IsLive).IsFalse(); + await Publish(0); + await Assert.That(_cache.IsLive).IsTrue(); + await Assert.That(_realtime.IsPublished).IsTrue(); + } + + [Test] + public async Task Failed_initial_subscribe_is_retried() { + var start = _realtime.Start(default); + (await WaitForCall(0)).Fail(); + (await WaitForCall(1)).Succeed(); + await TimeDriver.Drive(start, _time); + await Assert.That(_calls.Count).IsEqualTo(2); + } + + [Test] + public async Task Drop_then_failing_resubscribes_keep_retrying_until_success() { + await Publish(0); + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, new RpcException(new Status(StatusCode.Unavailable, "x"))); + await Assert.That(_cache.IsLive).IsFalse(); + (await WaitForCall(1)).Fail(); + (await WaitForCall(2)).Fail(); + (await WaitForCall(3)).Succeed(); + await TimeDriver.Until(() => _realtime.IsPublished, _time); + await Assert.That(_cache.IsLive).IsTrue(); + } + + [Test] + public async Task Foreground_start_during_pending_resubscribe_shares_the_attempt() { + await Publish(0); + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, null); + var pending = await WaitForCall(1); + var start = _realtime.Start(default); + await Task.Delay(20); + await Assert.That(_calls.Count).IsEqualTo(2); + pending.Succeed(); + await TimeDriver.Drive(start, _time); + await Assert.That(_calls.Count).IsEqualTo(2); + } + + [Test] + public async Task Shutdown_ends_the_loop() { + var start = _realtime.Start(default); + (await WaitForCall(0)).Fail(); + await _shutdown.CancelAsync(); + await Assert.That(async () => await start.WaitAsync(TimeSpan.FromSeconds(5))).Throws(); + } + + [Test] + public async Task Stale_callback_from_an_old_attempt_is_ignored() { + await Publish(0); + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, null); // non-token drop + (await WaitForCall(1)).Succeed(); + await TimeDriver.Until(() => _realtime.IsPublished, _time); + + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, Unauthenticated()); // delayed duplicate from A + await Task.Delay(20); + + await Assert.That(_realtime.IsPublished).IsTrue(); + await Assert.That(_cache.IsLive).IsTrue(); + await Assert.That(_source.Invalidated).IsEmpty(); + await Assert.That(_calls.Count).IsEqualTo(2); + } + + [Test] + public async Task Late_drop_of_failed_attempt_does_not_affect_the_pending_one() { + var start = _realtime.Start(default); + var a = await WaitForCall(0); + a.Fail(); + var b = await WaitForCall(1); // now pending + a.OnDropped(SubscriptionDroppedReason.ServerError, Unauthenticated()); + await Assert.That(_source.Invalidated).IsEmpty(); + b.Succeed(); + await TimeDriver.Drive(start, _time); + await Assert.That(_realtime.IsPublished).IsTrue(); + } + + [Test] + public async Task Drop_of_pending_attempt_is_handled() { + var start = _realtime.Start(default); + var a = await WaitForCall(0); + a.OnDropped(SubscriptionDroppedReason.ServerError, Unauthenticated()); // early drop + await Assert.That(_source.Invalidated.Single().Applied).IsTrue(); + _source.Value = "B"; + a.Succeed(); // the subscribe call still returns + await Assert.That(a.Subscription.Disposed).IsTrue(); + var b = await WaitForCall(1); + await Assert.That(b.Token).IsEqualTo("B"); + b.Succeed(); + await TimeDriver.Drive(start, _time); + await Assert.That(_realtime.IsPublished).IsTrue(); + await Assert.That(_source.Accepted.Count(l => l.Value == "A")).IsEqualTo(0); + } + + [Test] + public async Task Rejected_token_is_quarantined_before_anyone_can_observe_unpublished_state() { + await Publish(0); + var invalidatedWhenUnpublished = false; + _realtime.OnUnpublishedUnderLock = () => invalidatedWhenUnpublished = _source.Invalidated.Any(i => i.Applied); + + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, Unauthenticated()); + var start = _realtime.Start(default); // concurrent foreground start + + await Assert.That(invalidatedWhenUnpublished).IsTrue(); + for (var i = 0; i < 30; i++) { _time.Advance(TimeSpan.FromSeconds(1)); await Task.Delay(2); } // < quarantine + await Assert.That(_calls.Count).IsEqualTo(1); // A never handed out again + + _source.Value = "B"; + var b = await WaitForCall(1); + await Assert.That(b.Token).IsEqualTo("B"); + b.Succeed(); + await TimeDriver.Drive(start, _time); + } + + [Test] + public async Task Drop_reports_with_the_subscription_lease_not_the_current_one() { + await Publish(0); // subscribed with A + _source.Value = "B"; + await _source.GetAccessToken(default); // source moves to B (new generation) + + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, Unauthenticated()); + await Assert.That(_source.Invalidated.Single().Applied).IsFalse(); // stale lease, B untouched + + var next = await WaitForCall(1); + await Assert.That(next.Token).IsEqualTo("B"); + } + + [Test] + public async Task Cache_is_coherent_across_a_subscription_gap() { + var serverMeta = new StreamMeta(false, null, null, 0); + var metaReads = 0; + var filter = new ScavengedEventsFilter( + (_, _, _) => { metaReads++; return Task.FromResult(serverMeta); }, + (_, _, _) => Task.FromResult(new StreamSize(10)), + _cache, + new GrpcAuthContext(_source, _shutdown.Token, _time, "reader") + ); + + await Publish(0); + await Assert.That(await filter.Filter(TestEvents.Original("s", 5))).IsTrue(); // cached: no maxCount + await filter.Filter(TestEvents.Original("s", 5)); + await Assert.That(metaReads).IsEqualTo(1); + + Nth(0).OnDropped(SubscriptionDroppedReason.ServerError, null); + serverMeta = serverMeta with { MaxCount = 1 }; // changed while the subscription is down + + await Assert.That(await filter.Filter(TestEvents.Original("s", 5))).IsFalse(); // direct read sees the change + await Assert.That(metaReads).IsEqualTo(2); + + (await WaitForCall(1)).Succeed(); // resubscribed; no events delivered yet + await TimeDriver.Until(() => _realtime.IsPublished, _time); + await Assert.That(await filter.Filter(TestEvents.Original("s", 5))).IsFalse(); // fresh fetch, not pre-gap cache + await Assert.That(metaReads).IsEqualTo(3); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL (`Realtime` has no matching constructor / `IsPublished` / `OnUnpublishedUnderLock`). + +- [ ] **Step 3: Implement Realtime** + +Replace `src/Kurrent.Replicator.KurrentDb/Realtime.cs` entirely: + +```csharp +using System.Text.Json; +using Kurrent.Replicator.KurrentDb.Internals; + +namespace Kurrent.Replicator.KurrentDb; + +sealed class SubscriptionDroppedEarlyException() : Exception("The subscription dropped before it was registered"); + +/// +/// Keeps StreamMetaCache current from a $all subscription. One serialized subscribe loop; drops are attributed to +/// the attempt (and token lease) that produced them. See the spec, "Realtime subscription". +/// +class Realtime { + static ILog Log => LogProvider.GetLogger(typeof(Realtime)); + + internal delegate Task Subscribe( + Func onEvent, + Action onDropped, + CallAuth auth, + CancellationToken ct + ); + + sealed class Attempt(CallAuth auth) { + public CallAuth Auth { get; } = auth; + public bool Dropped { get; set; } + public IDisposable? Subscription { get; set; } + } + + readonly Subscribe _subscribe; + readonly StreamMetaCache _cache; + readonly GrpcAuthContext _auth; + readonly object _lock = new(); + + Attempt? _published; + Attempt? _pending; + Task? _subscribing; + + internal Action? OnUnpublishedUnderLock { get; set; } + + internal bool IsPublished { + get { lock (_lock) return _published is { Dropped: false }; } + } + + public Realtime(EventStoreClient client, StreamMetaCache cache, GrpcAuthContext auth) + : this( + async (onEvent, onDropped, a, ct) => await client.SubscribeToAllAsync( + FromAll.End, + (_, evt, _) => onEvent(evt), + subscriptionDropped: (_, reason, ex) => onDropped(reason, ex), + userCredentials: a.Credentials, + cancellationToken: ct + ) + .ConfigureAwait(false), + cache, + auth + ) { } + + internal Realtime(Subscribe subscribe, StreamMetaCache cache, GrpcAuthContext auth) { + _subscribe = subscribe; + _cache = cache; + _auth = auth; + } + + public Task Start(CancellationToken ct) => EnsureSubscribed(ct); + + Task EnsureSubscribed(CancellationToken ct) { + Task loop; + + lock (_lock) { + if (_published is { Dropped: false }) return Task.CompletedTask; + + if (_subscribing is null || _subscribing.IsCompleted) _subscribing = Task.Run(SubscribeLoop, CancellationToken.None); + + loop = _subscribing; + } + + return loop.WaitAsync(ct); + } + + async Task SubscribeLoop() { + var failures = 0; + + while (true) { + try { + await _auth.Run(SubscribeOnce, _auth.Shutdown).ConfigureAwait(false); + + return; + } catch (OperationCanceledException) when (_auth.Shutdown.IsCancellationRequested) { + throw; + } catch (Exception e) { + Log.Warn(e, "Realtime subscription failed; retrying"); + await Task.Delay(TokenGate.Backoff(failures++), _auth.Time, _auth.Shutdown).ConfigureAwait(false); + } + } + } + + async Task SubscribeOnce(CallAuth auth, CancellationToken ct) { + var attempt = new Attempt(auth); + + lock (_lock) _pending = attempt; + + IDisposable subscription; + + try { + subscription = await _subscribe(HandleEvent, (reason, ex) => HandleDrop(attempt, reason, ex), auth, ct).ConfigureAwait(false); + } catch { + lock (_lock) { + if (ReferenceEquals(_pending, attempt)) _pending = null; + } + + throw; + } + + lock (_lock) { + if (ReferenceEquals(_pending, attempt)) _pending = null; + + if (attempt.Dropped) { + subscription.Dispose(); + + throw new SubscriptionDroppedEarlyException(); + } + + attempt.Subscription = subscription; + _published = attempt; + _cache.MarkLive(); + } + + Log.Info("Realtime subscription started"); + + return true; + } + + void HandleDrop(Attempt attempt, SubscriptionDroppedReason reason, Exception? exception) { + if (reason == SubscriptionDroppedReason.Disposed) return; + + bool resubscribe; + + lock (_lock) { + if (ReferenceEquals(attempt, _published)) { + _auth.ReportFailure(attempt.Auth, exception); + attempt.Dropped = true; + _published = null; + _cache.MarkNotLive(); + OnUnpublishedUnderLock?.Invoke(); + resubscribe = true; + } + else if (ReferenceEquals(attempt, _pending)) { + _auth.ReportFailure(attempt.Auth, exception); + attempt.Dropped = true; + resubscribe = false; + } + else { + return; // superseded attempt: ignore entirely + } + } + + Log.Warn(exception, "Realtime subscription dropped: {Reason}", reason); + + if (!resubscribe) return; + + _ = EnsureSubscribed(CancellationToken.None) + .ContinueWith( + t => { + if (t.Exception?.GetBaseException() is not OperationCanceledException) + Log.Error(t.Exception, "Realtime resubscription stopped"); + }, + TaskContinuationOptions.OnlyOnFaulted + ); + } + + Task HandleEvent(ResolvedEvent re) { + if (IsSystemEvent()) + return Task.CompletedTask; + + if (IsMetadataUpdate()) { + var stream = re.OriginalStreamId[2..]; + var meta = JsonSerializer.Deserialize(re.Event.Data.Span, MetaSerialization.StreamMetadataJsonSerializerOptions); + _cache.UpdateStreamMeta(stream, meta, re.OriginalEventNumber.ToInt64()); + } + else { + _cache.UpdateStreamLastEventNumber(re.OriginalStreamId, re.OriginalEventNumber.ToInt64()); + } + + return Task.CompletedTask; + + bool IsSystemEvent() => re.Event.EventType.StartsWith('$') && re.Event.EventType != Predefined.MetadataEventType; + + bool IsMetadataUpdate() => re.Event.EventType == Predefined.MetadataEventType; + } +} +``` + +Keep the body of `HandleEvent` identical to today's (copy it from the current file if it differs from the above in any detail). + +- [ ] **Step 4: Adjust GrpcEventReader to the new constructor** + +In `src/Kurrent.Replicator.KurrentDb/GrpcEventReader.cs`: +- constructor: replace `_realtime = new(client, metaCache);` with `_realtime = new(client, metaCache, GrpcAuthContext.None);` and delete the temporary `metaCache.MarkLive();` line added in Task 9 (Realtime now marks the cache live on publish). +- `ReadEvents`: replace `await _realtime.Start();` with `await _realtime.Start(cancellationToken).ConfigureAwait(false);`. + +- [ ] **Step 5: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/RealtimeTests/*"` +Expected: 11 tests PASS. Also rerun `"/*/Kurrent.Replicator.Tests.Auth/*/*"` to confirm nothing regressed. + +- [ ] **Step 6: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb test/Kurrent.Replicator.Tests/Auth/RealtimeTests.cs +git commit -m "fix(realtime): serialized resubscription with drop attribution and cache coherence + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 11: GrpcEventReader per-call credentials and per-side GrpcConfigurator + +**Files:** +- Modify: `src/Kurrent.Replicator.KurrentDb/GrpcEventReader.cs` +- Modify: `src/Kurrent.Replicator.KurrentDb/Configurator.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/GrpcEventReaderAuthTests.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/GrpcConfiguratorTests.cs` + +**Interfaces:** +- Consumes: everything from Tasks 1–10. +- Produces: + - `GrpcEventReader` ctors: `(EventStoreClient client)` (unchanged behaviour), `(EventStoreClient client, GrpcAuthContext auth)`, `internal (EventStoreClient client, GrpcAuthContext auth, Func realtimeFactory)`. + - `GrpcConfigurator` ctors: `()`, `(GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth, CancellationToken shutdown)`, `internal (GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth, CancellationToken shutdown, Action? configureSettings, TimeProvider time)`. + +- [ ] **Step 1: Write the failing reader tests** + +`test/Kurrent.Replicator.Tests/Auth/GrpcEventReaderAuthTests.cs`: + +```csharp +using Grpc.Core; +using Kurrent.Replicator.KurrentDb; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Shared; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class GrpcEventReaderAuthTests { + readonly FakeTimeProvider _time = new(); + readonly FakeKurrentDbHandler _handler = new(); + readonly string _file = Path.Combine(Path.GetTempPath(), $"replicator-reader-{Guid.NewGuid():N}"); + + [After(Test)] + public void Cleanup() { + if (File.Exists(_file)) File.Delete(_file); + } + + FallbackUsage _usage = null!; + + GrpcEventReader Reader(IAccessTokenSource source) { + var client = FakeKurrentDbHandler.Client(_handler, configure: s => _usage = GrpcAuthentication.Apply(s, source)); + var auth = new GrpcAuthContext(source, CancellationToken.None, _time, "reader"); + + // realtime subscription that succeeds immediately, so ReadEvents gets to the $all read + return new GrpcEventReader(client, auth, cache => new Realtime((_, _, _, _) => Task.FromResult(new MemoryStream()), cache, auth)); + } + + [Test] + public async Task Read_all_carries_the_token_and_a_rejection_invalidates_it() { + await File.WriteAllTextAsync(_file, "A"); + var source = new TokenFileSource(_file, TimeSpan.FromHours(1), _time, "reader"); + var reader = Reader(source); + _handler.Respond = _ => Task.FromResult(FakeKurrentDbHandler.TrailersOnly(StatusCode.Unauthenticated)); + + await Assert.That(async () => await reader.ReadEvents(LogPosition.Start, _ => ValueTask.CompletedTask, default)).Throws(); + await Assert.That(_handler.AuthorizationsFor(FakeKurrentDbHandler.ReadPath).Single()).IsEqualTo("Bearer A"); + + await File.WriteAllTextAsync(_file, "B"); + _handler.Respond = _ => Task.FromResult(FakeKurrentDbHandler.TrailersOnly(StatusCode.Unavailable)); + await Assert.That(async () => await reader.ReadEvents(LogPosition.Start, _ => ValueTask.CompletedTask, default)).Throws(); + await Assert.That(_handler.AuthorizationsFor(FakeKurrentDbHandler.ReadPath).Last()).IsEqualTo("Bearer B"); // re-read immediately after rejection + await Assert.That(_usage.Count).IsEqualTo(0); + } + + [Test] + public async Task GetLastPosition_retries_a_rejected_token_with_the_new_one() { + await File.WriteAllTextAsync(_file, "A"); + var source = new TokenFileSource(_file, TimeSpan.FromHours(1), _time, "reader"); + var reader = Reader(source); + _handler.Respond = s => { + if (s.Authorization == "Bearer A") File.WriteAllText(_file, "B"); // rotate while rejecting A + + return Task.FromResult(FakeKurrentDbHandler.TrailersOnly(s.Authorization == "Bearer A" ? StatusCode.Unauthenticated : StatusCode.Unavailable)); + }; + + var call = reader.GetLastPosition(default); + await Assert.That(async () => await TimeDriver.Drive(call, _time)).Throws(); // Unavailable with B is not a token error + await Assert.That(_handler.AuthorizationsFor(FakeKurrentDbHandler.ReadPath)).IsEquivalentTo(new[] { "Bearer A", "Bearer B" }); + await Assert.That(_usage.Count).IsEqualTo(0); + } +} +``` + +Note: `MemoryStream` is just a convenient `IDisposable` stand-in for the subscription. + +- [ ] **Step 2: Write the failing configurator tests** + +`test/Kurrent.Replicator.Tests/Auth/GrpcConfiguratorTests.cs`: + +```csharp +using Kurrent.Replicator.KurrentDb; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class GrpcConfiguratorTests { + readonly string _file = Path.Combine(Path.GetTempPath(), $"replicator-cfg-{Guid.NewGuid():N}"); + + [After(Test)] + public void Cleanup() { + if (File.Exists(_file)) File.Delete(_file); + } + + [Test] + public async Task Reader_basic_and_sink_oauth_send_independent_headers() { + await File.WriteAllTextAsync(_file, "SINK-TOKEN"); + var handler = new FakeKurrentDbHandler { Respond = s => Task.FromResult(s.Path == FakeKurrentDbHandler.AppendPath ? FakeKurrentDbHandler.AppendSuccess() : FakeKurrentDbHandler.TrailersOnly(Grpc.Core.StatusCode.Unavailable)) }; + var sinkAuth = new GrpcAuthOptions { Type = GrpcAuthType.OAuthTokenFile, TokenFile = _file }; + + var configurator = new GrpcConfigurator(GrpcAuthOptions.Default, sinkAuth, CancellationToken.None, s => s.CreateHttpMessageHandler = () => handler, new FakeTimeProvider()); + + var writer = configurator.ConfigureWriter("esdb://target.example.com:2113?tls=true"); + var reader = configurator.ConfigureReader("esdb://admin:changeit@target.example.com:2113?tls=true"); + + await writer.WriteEvent(TestEvents.Proposed("s"), default); + await Assert.That(async () => await reader.GetLastPosition(default)).Throws(); + + await Assert.That(handler.AuthorizationsFor(FakeKurrentDbHandler.AppendPath).Single()).IsEqualTo("Bearer SINK-TOKEN"); + await Assert.That(handler.AuthorizationsFor(FakeKurrentDbHandler.ReadPath).Single()).StartsWith("Basic "); + } + + [Test] + public async Task Invalid_side_configuration_names_the_side() { + var oauth = new GrpcAuthOptions { Type = GrpcAuthType.OAuthTokenFile, TokenFile = _file }; + var configurator = new GrpcConfigurator(oauth, GrpcAuthOptions.Default, CancellationToken.None); + + var ex = await Assert.That(() => configurator.ConfigureReader("esdb://admin:changeit@localhost:2113?tls=true")).Throws(); + await Assert.That(ex!.Message).StartsWith("Invalid reader auth configuration"); + + await Assert.That(() => configurator.ConfigureWriter("esdb://admin:changeit@localhost:2113?tls=false")).ThrowsNothing(); // sink is basic + } +} +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL (missing constructors). + +- [ ] **Step 4: Implement GrpcEventReader changes** + +In `src/Kurrent.Replicator.KurrentDb/GrpcEventReader.cs`: + +Fields and constructors: + +```csharp + readonly EventStoreClient _client; + readonly GrpcAuthContext _auth; + readonly ScavengedEventsFilter _filter; + readonly Realtime _realtime; + + public GrpcEventReader(EventStoreClient client) : this(client, GrpcAuthContext.None) { } + + public GrpcEventReader(EventStoreClient client, GrpcAuthContext auth) : this(client, auth, cache => new Realtime(client, cache, auth)) { } + + internal GrpcEventReader(EventStoreClient client, GrpcAuthContext auth, Func realtimeFactory) { + _log = LogProvider.GetCurrentClassLogger(); + _debugLog = _log.IsDebugEnabled() ? _log : null; + + var metaCache = new StreamMetaCache(failClosedOnAuthErrors: auth.Enabled); + _client = client; + _auth = auth; + _filter = new(client, metaCache, auth); + _realtime = realtimeFactory(metaCache); + } +``` + +`ReadEvents` — acquire credentials before the long-running read, report on first response, report failures: + +```csharp + public async Task ReadEvents(LogPosition fromLogPosition, Func next, CancellationToken cancellationToken) { + var sequence = 0; + var lastPosition = 0L; + + _log.Info("Starting gRPC reader"); + + await _realtime.Start(cancellationToken).ConfigureAwait(false); + + var (_, eventPosition) = fromLogPosition; + + var callAuth = await _auth.AcquireCredentials(cancellationToken).ConfigureAwait(false); + var accepted = false; + + try { + var read = _client.ReadAllAsync( + Direction.Forwards, + new(eventPosition, eventPosition), + userCredentials: callAuth.Credentials, + cancellationToken: cancellationToken + ); + + var enumerator = read.GetAsyncEnumerator(cancellationToken); + + do { + using var activity = new Activity("read"); + activity.Start(); + + var hasValue = await Metrics.MeasureValueTask( + () => enumerator.MoveNextAsync(cancellationToken), + ReplicationMetrics.ReadsHistogram, + ReplicationMetrics.ReadErrorsCount + ) + .ConfigureAwait(false); + + if (!accepted) { + _auth.ReportAccepted(callAuth); + accepted = true; + } + + if (!hasValue) break; + + // Keep the existing loop body from `var evt = enumerator.Current;` through + // `await next(originalEvent).ConfigureAwait(false);` exactly as it is today. + } while (true); + } catch (Exception e) { + _auth.ReportFailure(callAuth, e); + + throw; + } + + _log.Info("Reached the end of the stream at {Position}", lastPosition); + } +``` + +`GetLastPosition` through `Run`: + +```csharp + public Task GetLastPosition(CancellationToken cancellationToken) + => _auth.Run( + async (a, c) => { + var events = await _client + .ReadAllAsync(Direction.Backwards, Position.End, 1, userCredentials: a.Credentials, cancellationToken: c) + .ToArrayAsync(c) + .ConfigureAwait(false); + + return (long?)events[0].OriginalPosition?.CommitPosition; + }, + cancellationToken + ); +``` + +- [ ] **Step 5: Implement GrpcConfigurator** + +Replace `src/Kurrent.Replicator.KurrentDb/Configurator.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb; + +public class GrpcConfigurator : IConfigurator { + static ILog Log => LogProvider.GetLogger(typeof(GrpcConfigurator)); + + readonly GrpcAuthOptions _readerAuth; + readonly GrpcAuthOptions _sinkAuth; + readonly CancellationToken _shutdown; + readonly Action? _configureSettings; + readonly TimeProvider _time; + + public GrpcConfigurator() : this(GrpcAuthOptions.Default, GrpcAuthOptions.Default, CancellationToken.None) { } + + public GrpcConfigurator(GrpcAuthOptions readerAuth, GrpcAuthOptions sinkAuth, CancellationToken shutdown) + : this(readerAuth, sinkAuth, shutdown, null, TimeProvider.System) { } + + internal GrpcConfigurator( + GrpcAuthOptions readerAuth, + GrpcAuthOptions sinkAuth, + CancellationToken shutdown, + Action? configureSettings, + TimeProvider time + ) { + _readerAuth = readerAuth; + _sinkAuth = sinkAuth; + _shutdown = shutdown; + _configureSettings = configureSettings; + _time = time; + } + + public string Protocol => "grpc"; + + public IEventReader ConfigureReader(string connectionString) { + var (client, auth) = Configure(connectionString, _readerAuth, "reader", follower: true); + + return new GrpcEventReader(client, auth); + } + + public IEventWriter ConfigureWriter(string connectionString) { + var (client, auth) = Configure(connectionString, _sinkAuth, "sink", follower: false); + + return new GrpcEventWriter(client, auth); + } + + (EventStoreClient Client, GrpcAuthContext Auth) Configure(string connectionString, GrpcAuthOptions options, string side, bool follower) { + var settings = EventStoreClientSettings.Create(connectionString); + + if (follower) settings.ConnectivitySettings.NodePreference = NodePreference.Follower; + + _configureSettings?.Invoke(settings); + + GrpcAuthOptionsValidator.EnsureValid(side, options, settings); + + foreach (var warning in GrpcAuthOptionsValidator.Warnings(options)) Log.Warn("{Side}: {Warning}", side, warning); + + var source = GrpcAuthentication.CreateSource(options, side, _time, _shutdown); + + if (source != null) { + GrpcAuthentication.Apply(settings, source); + Log.Info("{Side}: using {AuthType} authentication", side, options.Type); + } + + return (new EventStoreClient(settings), new GrpcAuthContext(source, _shutdown, _time, side)); + } +} +``` + +`connectionString` sides also get a context with the real shutdown token (`source == null`), so the `Realtime` retry loop stops at shutdown for basic-auth readers too. + +- [ ] **Step 6: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/Kurrent.Replicator.Tests.Auth/*/*"` +Expected: all Auth tests PASS. Also build the whole solution: `dotnet build Kurrent.Replicator.slnx -p:UseAppHost=false -v q` → 0 errors (Startup still calls `new GrpcConfigurator()` via DI; that is changed in Task 13). + +- [ ] **Step 7: Commit** + +```bash +git add src/Kurrent.Replicator.KurrentDb test/Kurrent.Replicator.Tests/Auth +git commit -m "feat(auth): per-side gRPC configurator and reader per-call credentials + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 12: Replicator pipeline — cancellable prepare→sink handoff, drain exit, resilient metrics reporter + +**Files:** +- Modify: `src/Kurrent.Replicator/Replicator.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/ReplicatorShutdownTests.cs` + +**Interfaces:** +- Consumes: `Replicator.Replicate(...)` (signature unchanged), `GrpcAuthContext`, `ControllableTokenSource`, `TestEvents`. +- Produces: behaviour only — no API change. + +Changes (spec "Sink (writer)" and "Metrics reporter"): +1. Create `writerCts` before the pipes; the prepare pipe's send delegate becomes `ctx => sinkChannel.Writer.WriteAsync(ctx, writerCts.Token)`. +2. Drain loop: `while (sinkChannel.Reader.Count > 0 && !writerTask.IsCompleted)`; afterwards, if the writer has stopped with items left, log a warning with the count. +3. On stopping: cancel `writerCts` **before** `sinkChannel.Writer.Complete()`, so a prepare write blocked on the full channel ends with `OperationCanceledException` (which `PreparePipe` swallows) rather than `ChannelClosedException`. +4. `Report`: catch non-cancellation exceptions per iteration, log at most once a minute, continue. + +- [ ] **Step 1: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/ReplicatorShutdownTests.cs`: + +```csharp +using System.Collections.Concurrent; +using Kurrent.Replicator.KurrentDb.Auth; +using Kurrent.Replicator.Prepare; +using Kurrent.Replicator.Shared; +using Kurrent.Replicator.Shared.Contracts; +using Kurrent.Replicator.Sink; +using Kurrent.Replicator.Tests.Auth.Support; +using Microsoft.Extensions.Time.Testing; + +namespace Kurrent.Replicator.Tests.Auth; + +public class ReplicatorShutdownTests { + sealed class FakeReader(int count, bool blockAfter) : IEventReader { + public int PositionCalls; + public Func? Position { get; init; } + + public string Protocol => "fake"; + + public async Task ReadEvents(LogPosition fromLogPosition, Func next, CancellationToken cancellationToken) { + for (var i = 0; i < count; i++) await next(TestEvents.Original("s", i)); + + if (blockAfter) await Task.Delay(Timeout.Infinite, cancellationToken); + } + + public Task GetLastPosition(CancellationToken cancellationToken) { + var n = Interlocked.Increment(ref PositionCalls); + + return Task.FromResult(Position?.Invoke(n) ?? count); + } + + public ValueTask Filter(BaseOriginalEvent originalEvent) => ValueTask.FromResult(true); + } + + sealed class AuthGatedWriter(GrpcAuthContext auth) : IEventWriter { + public readonly ConcurrentQueue Written = new(); + + public Task Start() => Task.CompletedTask; + + public Task WriteEvent(BaseProposedEvent proposedEvent, CancellationToken cancellationToken) + => auth.Run((_, _) => { + Written.Enqueue(proposedEvent.SourceLogPosition.EventNumber); + + return Task.FromResult(1L); + }, + cancellationToken + ); + } + + sealed class RecordingCheckpointStore : ICheckpointStore { + public readonly ConcurrentQueue Stored = new(); + + public ValueTask HasStoredCheckpoint(CancellationToken ct) => ValueTask.FromResult(false); + public ValueTask LoadCheckpoint(CancellationToken ct) => ValueTask.FromResult(LogPosition.Start); + public ValueTask Flush(CancellationToken ct) => ValueTask.CompletedTask; + + public ValueTask StoreCheckpoint(LogPosition logPosition, CancellationToken ct) { + Stored.Enqueue(logPosition); + + return ValueTask.CompletedTask; + } + } + + [Test] + public async Task Sink_token_outage_pauses_replication_and_it_resumes_without_restart() { + var time = new FakeTimeProvider(); + var source = new ControllableTokenSource(time); // no token yet + var writer = new AuthGatedWriter(new GrpcAuthContext(source, CancellationToken.None, time, "sink")); + var store = new RecordingCheckpointStore(); + + var run = Replicator.Replicate( + new FakeReader(5, blockAfter: false), + writer, + new SinkPipeOptions(1, 10), + new PreparePipelineOptions(null, null, 1, 10), + new NoCheckpointSeeder(), + store, + new ReplicatorOptions(false, false, TimeSpan.Zero, TimeSpan.FromSeconds(1)), + CancellationToken.None + ); + + await TimeDriver.Until(() => source.Calls >= 5, time); + await Assert.That(writer.Written).IsEmpty(); + source.Value = "A"; + await TimeDriver.Drive(run, time); + + await Assert.That(writer.Written.Count).IsEqualTo(5); + } + + [Test] + public async Task Shutdown_during_outage_with_full_sink_channel_returns_promptly() { + var time = new FakeTimeProvider(); + var shutdown = new CancellationTokenSource(); + var stopping = new CancellationTokenSource(); + var source = new ControllableTokenSource(time); + var writer = new AuthGatedWriter(new GrpcAuthContext(source, shutdown.Token, time, "sink")); + var store = new RecordingCheckpointStore(); + + var run = Replicator.Replicate( + new FakeReader(10, blockAfter: true), + writer, + new SinkPipeOptions(1, 1), // sink buffer of 1: fills immediately + new PreparePipelineOptions(null, null, 1, 100), // prepare buffer larger than the event count + new NoCheckpointSeeder(), + store, + new ReplicatorOptions(false, true, TimeSpan.Zero, TimeSpan.FromSeconds(1)), + stopping.Token + ); + + await TimeDriver.Until(() => source.Calls >= 3, time); + await Task.Delay(200); // let the prepare shovel block on the full sink channel + + await shutdown.CancelAsync(); // ApplicationStopping + await stopping.CancelAsync(); // hosted service stop + + await run.WaitAsync(TimeSpan.FromSeconds(15)); + await Assert.That(writer.Written).IsEmpty(); + await Assert.That(store.Stored).IsEmpty(); + } + + [Test] + public async Task Metrics_reporter_survives_a_failing_position_read() { + using var stopping = new CancellationTokenSource(); + var reader = new FakeReader(0, blockAfter: true) { Position = n => n == 1 ? throw new InvalidOperationException("boom") : 1 }; + + var run = Replicator.Replicate( + reader, + new AuthGatedWriter(GrpcAuthContext.None), + new SinkPipeOptions(), + new PreparePipelineOptions(null, null), + new NoCheckpointSeeder(), + new RecordingCheckpointStore(), + new ReplicatorOptions(false, true, TimeSpan.Zero, TimeSpan.FromMilliseconds(20)), + stopping.Token + ); + + await Task.Delay(500); + await Assert.That(Volatile.Read(ref reader.PositionCalls)).IsGreaterThanOrEqualTo(3); + await stopping.CancelAsync(); + await run.WaitAsync(TimeSpan.FromSeconds(15)); + } +} +``` + +Note: `FakeReader.Position` throws inside `GetLastPosition` for the first call. `Task.FromResult(Position?.Invoke(n) ...)` evaluates the delegate synchronously, so the exception surfaces from `GetLastPosition` exactly as a failing gRPC read would. + +- [ ] **Step 2: Run to verify failure** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/ReplicatorShutdownTests/*" --timeout 120s` +Expected: `Shutdown_during_outage_with_full_sink_channel_returns_promptly` FAILS with a timeout after 15 s (drain-loop hang), and `Metrics_reporter_survives_a_failing_position_read` FAILS (only 1 call). The first test may already pass. + +- [ ] **Step 3: Implement** + +In `src/Kurrent.Replicator/Replicator.cs`: + +Move `var writerCts = new CancellationTokenSource();` to directly after `var sinkChannel = ...;` and change the prepare pipe: + +```csharp + var writerCts = new CancellationTokenSource(); + + var readerPipe = new ReaderPipe( + reader, + checkpointStore, + ctx => prepareChannel.Writer.WriteAsync(ctx, ctx.CancellationToken) + ); + + // writerCts (not ctx.CancellationToken, which is None) so a hand-off blocked on a full sink channel ends at shutdown + var preparePipe = new PreparePipe( + preparePipeOptions.Filter, + preparePipeOptions.Transform, + ctx => sinkChannel.Writer.WriteAsync(ctx, writerCts.Token) + ); + var sinkPipe = new SinkPipe(writer, sinkPipeOptions, checkpointStore); +``` + +Drain loop and stop ordering: + +```csharp + while (sinkChannel.Reader.Count > 0 && !writerTask.IsCompleted) { + await checkpointStore.Flush(CancellationToken.None).ConfigureAwait(false); + Log.Info("Waiting for the sink pipe to exhaust ({Left} left)...", sinkChannel.Reader.Count); + await Task.Delay(1000, CancellationToken.None).ConfigureAwait(false); + } + + if (writerTask.IsCompleted && sinkChannel.Reader.Count > 0) { + Log.Warn( + "Writer stopped with {Count} events not written; they will be read again from the last checkpoint", + sinkChannel.Reader.Count + ); + } + + await Flush().ConfigureAwait(false); + + if (stopping) { + await writerCts.CancelAsync(); + sinkChannel.Writer.Complete(); + + break; + } +``` + +`Report`: + +```csharp + async Task Report() { + DateTimeOffset? lastWarning = null; + + while (!stoppingToken.IsCancellationRequested) { + try { + var position = await reader.GetLastPosition(stoppingToken).ConfigureAwait(false); + + if (position.HasValue) { + ReplicationMetrics.LastSourcePosition.Set(position.Value); + } + } catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) { + break; + } catch (Exception e) { + var now = DateTimeOffset.UtcNow; + + if (lastWarning is null || now - lastWarning > TimeSpan.FromMinutes(1)) { + lastWarning = now; + Log.Warn(e, "Unable to read the source position for metrics; will retry"); + } + } + + try { + await Task.Delay(replicatorOptions.ReportMetricsFrequency, stoppingToken).ConfigureAwait(false); + } catch (OperationCanceledException) { + break; + } + } + + Log.Info("Reporting stopped"); + } +``` + +`Stop()` still calls `writerCts.Cancel()`; cancelling twice is harmless. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/ReplicatorShutdownTests/*" --timeout 120s` +Expected: 3 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/Kurrent.Replicator/Replicator.cs test/Kurrent.Replicator.Tests/Auth/ReplicatorShutdownTests.cs +git commit -m "fix(replicator): no shutdown hang on stalled writer; reporter survives read errors + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- +### Task 13: Host configuration — binding, per-side wiring, environment-variable redaction + +The host project (`src/replicator`) runs `yarn install` on Debug builds, so the test project must not reference it. Binding and redaction logic therefore live in library projects, and the host only wires them up. + +**Files:** +- Create: `src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthSettings.cs` +- Create: `src/Kurrent.Replicator.Shared/ConfigRedaction.cs` +- Modify: `src/replicator/Settings/ReplicatorSettings.cs` +- Modify: `src/replicator/Settings/EnvConfigProvider.cs` +- Modify: `src/replicator/Startup.cs` +- Modify: `test/Kurrent.Replicator.Tests/Kurrent.Replicator.Tests.csproj` +- Test: `test/Kurrent.Replicator.Tests/Auth/AuthSettingsBindingTests.cs` +- Test: `test/Kurrent.Replicator.Tests/Auth/ConfigRedactionTests.cs` + +**Interfaces:** +- Consumes: `GrpcAuthOptions`, `GrpcAuthOptionsValidator.ValidateProtocol`, `GrpcConfigurator(GrpcAuthOptions, GrpcAuthOptions, CancellationToken)`. +- Produces: + - `public sealed record GrpcAuthSettings` (raw strings/ints bound from `auth`) with `GrpcAuthOptions ToOptions(string side)` (throws `InvalidOperationException("Invalid {side} auth configuration: ...")` for unknown `type` / `clientAuthentication`; lower-cases `additionalParameters` keys, last one wins). + - `public static class ConfigRedaction { string? Display(string configKey, string? value); }` in namespace `Kurrent.Replicator.Shared`. + - `EsdbSettings.Auth` (`GrpcAuthSettings`, default `new()`). + +- [ ] **Step 1: Add test packages** + +In `test/Kurrent.Replicator.Tests/Kurrent.Replicator.Tests.csproj`, add: + +```xml + + +``` + +- [ ] **Step 2: Write the failing tests** + +`test/Kurrent.Replicator.Tests/Auth/AuthSettingsBindingTests.cs`: + +```csharp +using Kurrent.Replicator.KurrentDb.Auth; +using Microsoft.Extensions.Configuration; + +namespace Kurrent.Replicator.Tests.Auth; + +public class AuthSettingsBindingTests { + static GrpcAuthOptions Bind(Dictionary values, string side) { + var config = new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + var raw = config.GetSection($"Replicator:{side}:Auth").Get() ?? new GrpcAuthSettings(); + + return raw.ToOptions(side.ToLowerInvariant()); + } + + [Test] + public async Task Missing_section_means_connection_string() { + var options = Bind(new(), "Reader"); + await Assert.That(options.Type).IsEqualTo(GrpcAuthType.ConnectionString); + await Assert.That(options.IsOAuth).IsFalse(); + } + + [Test] + public async Task Reader_basic_and_sink_oauth_bind_independently() { + var values = new Dictionary { + ["Replicator:Reader:ConnectionString"] = "esdb://admin:changeit@source:2113", + ["Replicator:Sink:Auth:Type"] = "OAuthClientCredentials", + ["Replicator:Sink:Auth:TokenEndpoint"] = "https://login.microsoftonline.com/t/oauth2/v2.0/token", + ["Replicator:Sink:Auth:ClientId"] = "app", + ["Replicator:Sink:Auth:ClientSecretFile"] = "/secrets/sink", + ["Replicator:Sink:Auth:ClientAuthentication"] = "Basic", + ["Replicator:Sink:Auth:Scope"] = "api://kdb/.default", + ["Replicator:Sink:Auth:RefreshBeforeExpirySeconds"] = "120" + }; + + var reader = Bind(values, "Reader"); + var sink = Bind(values, "Sink"); + + await Assert.That(reader.Type).IsEqualTo(GrpcAuthType.ConnectionString); + await Assert.That(sink.Type).IsEqualTo(GrpcAuthType.OAuthClientCredentials); + await Assert.That(sink.ClientAuthentication).IsEqualTo(ClientAuthenticationMethod.Basic); + await Assert.That(sink.ClientSecretFile).IsEqualTo("/secrets/sink"); + await Assert.That(sink.RefreshBeforeExpirySeconds).IsEqualTo(120); + await Assert.That(sink.TokenFileReloadSeconds).IsEqualTo(30); + } + + [Test] + public async Task Reader_oauth_and_sink_basic_bind_independently() { + var values = new Dictionary { + ["Replicator:Reader:Auth:Type"] = "oauthTokenFile", + ["Replicator:Reader:Auth:TokenFile"] = "/var/run/token", + ["Replicator:Sink:Auth:Type"] = "connectionString" + }; + + await Assert.That(Bind(values, "Reader").TokenFile).IsEqualTo("/var/run/token"); + await Assert.That(Bind(values, "Sink").IsOAuth).IsFalse(); + } + + [Test] + public async Task Env_vars_bind_auth_including_additional_parameters() { + // EnvConfigProvider turns REPLICATOR_A_B into the key REPLICATOR:A:B (case preserved, lookups case-insensitive) + var env = new Dictionary { + ["REPLICATOR_SINK_AUTH_TYPE"] = "oauthClientCredentials", + ["REPLICATOR_SINK_AUTH_TOKENENDPOINT"] = "https://idp/t", + ["REPLICATOR_SINK_AUTH_CLIENTID"] = "c", + ["REPLICATOR_SINK_AUTH_CLIENTSECRET"] = "s", + ["REPLICATOR_SINK_AUTH_ADDITIONALPARAMETERS_AUDIENCE"] = "kurrentdb" + }; + var values = env.ToDictionary(kv => kv.Key.Replace("_", ":"), kv => kv.Value); + + var sink = Bind(values, "Sink"); + await Assert.That(sink.Type).IsEqualTo(GrpcAuthType.OAuthClientCredentials); + await Assert.That(sink.ClientSecret).IsEqualTo("s"); + await Assert.That(sink.AdditionalParameters["audience"]).IsEqualTo("kurrentdb"); + } + + [Test] + public async Task Unknown_values_fail_with_the_side() { + var badType = new Dictionary { ["Replicator:Reader:Auth:Type"] = "oauth" }; + var ex = await Assert.That(() => Bind(badType, "Reader")).Throws(); + await Assert.That(ex!.Message).StartsWith("Invalid reader auth configuration"); + + var badMethod = new Dictionary { + ["Replicator:Sink:Auth:Type"] = "oauthClientCredentials", ["Replicator:Sink:Auth:ClientAuthentication"] = "jwt" + }; + await Assert.That(() => Bind(badMethod, "Sink")).Throws(); + } + + [Test] + public async Task ToString_never_prints_secrets() { + var raw = new GrpcAuthSettings { Type = "oauthClientCredentials", ClientSecret = "s3cr3t" }; + await Assert.That(raw.ToString()).DoesNotContain("s3cr3t"); + await Assert.That(raw.ToOptions("sink").ToString()).DoesNotContain("s3cr3t"); + } +} +``` + +`test/Kurrent.Replicator.Tests/Auth/ConfigRedactionTests.cs`: + +```csharp +using Kurrent.Replicator.Shared; + +namespace Kurrent.Replicator.Tests.Auth; + +public class ConfigRedactionTests { + [Test] + [Arguments("REPLICATOR:READER:CONNECTIONSTRING", "esdb://admin:changeit@host:2113")] + [Arguments("REPLICATOR:READER:CONNECTIONSTRING", "GossipSeeds=a:2113; DefaultUserCredentials=admin:changeit;")] + [Arguments("REPLICATOR:SINK:AUTH:CLIENTSECRET", "s3cr3t")] + [Arguments("REPLICATOR:SINK:AUTH:ADDITIONALPARAMETERS:AUDIENCE", "kdb")] + [Arguments("REPLICATOR:CHECKPOINT:PATH", "mongodb://u:p@mongo")] + public async Task Sensitive_values_are_masked(string key, string value) { + if (key.EndsWith("PATH")) { + await Assert.That(ConfigRedaction.Display(key, value)).IsEqualTo(value); // not a connection string key: printed as before + return; + } + + await Assert.That(ConfigRedaction.Display(key, value)).IsEqualTo("***"); + } + + [Test] + public async Task Auth_type_and_unrelated_keys_are_printed() { + await Assert.That(ConfigRedaction.Display("REPLICATOR:SINK:AUTH:TYPE", "oauthTokenFile")).IsEqualTo("oauthTokenFile"); + await Assert.That(ConfigRedaction.Display("REPLICATOR:SINK:PARTITIONCOUNT", "4")).IsEqualTo("4"); + await Assert.That(ConfigRedaction.Display("REPLICATOR:SINK:AUTHORITY", "x")).IsEqualTo("x"); // segment match, not substring + } +} +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` +Expected: FAIL (`GrpcAuthSettings`, `ConfigRedaction` not found). + +- [ ] **Step 4: Implement GrpcAuthSettings** + +`src/Kurrent.Replicator.KurrentDb/Auth/GrpcAuthSettings.cs`: + +```csharp +namespace Kurrent.Replicator.KurrentDb.Auth; + +/// Raw `auth` section for one side, exactly as bound from YAML or environment variables. +public sealed record GrpcAuthSettings { + public string? Type { get; init; } + public string? TokenEndpoint { get; init; } + public string? ClientId { get; init; } + public string? ClientSecret { get; init; } + public string? ClientSecretFile { get; init; } + public string? ClientAssertionFile { get; init; } + public string? ClientAuthentication { get; init; } + public string? Scope { get; init; } + public Dictionary AdditionalParameters { get; init; } = new(); + public int? DefaultTokenLifetimeSeconds { get; init; } + public int? RefreshBeforeExpirySeconds { get; init; } + public string? TokenFile { get; init; } + public int? TokenFileReloadSeconds { get; init; } + + public GrpcAuthOptions ToOptions(string side) { + var type = (Type ?? "connectionString").Trim().ToLowerInvariant() switch { + "connectionstring" => GrpcAuthType.ConnectionString, + "oauthclientcredentials" => GrpcAuthType.OAuthClientCredentials, + "oauthtokenfile" => GrpcAuthType.OAuthTokenFile, + _ => throw new InvalidOperationException( + $"Invalid {side} auth configuration: unknown auth.type '{Type}' (expected connectionString, oauthClientCredentials or oauthTokenFile)" + ) + }; + + var method = (ClientAuthentication ?? "post").Trim().ToLowerInvariant() switch { + "post" => ClientAuthenticationMethod.Post, + "basic" => ClientAuthenticationMethod.Basic, + _ => throw new InvalidOperationException( + $"Invalid {side} auth configuration: unknown auth.clientAuthentication '{ClientAuthentication}' (expected post or basic)" + ) + }; + + var parameters = new Dictionary(); + foreach (var (key, value) in AdditionalParameters) parameters[key.ToLowerInvariant()] = value; + + return new GrpcAuthOptions { + Type = type, + TokenEndpoint = NullIfEmpty(TokenEndpoint), + ClientId = NullIfEmpty(ClientId), + ClientSecret = NullIfEmpty(ClientSecret), + ClientSecretFile = NullIfEmpty(ClientSecretFile), + ClientAssertionFile = NullIfEmpty(ClientAssertionFile), + ClientAuthentication = method, + Scope = NullIfEmpty(Scope), + AdditionalParameters = parameters, + DefaultTokenLifetimeSeconds = DefaultTokenLifetimeSeconds, + RefreshBeforeExpirySeconds = RefreshBeforeExpirySeconds ?? 300, + TokenFile = NullIfEmpty(TokenFile), + TokenFileReloadSeconds = TokenFileReloadSeconds ?? 30 + }; + } + + static string? NullIfEmpty(string? value) => string.IsNullOrEmpty(value) ? null : value; + + public override string ToString() => $"GrpcAuthSettings(Type={Type})"; +} +``` + +- [ ] **Step 5: Implement ConfigRedaction and use it in EnvConfigProvider** + +`src/Kurrent.Replicator.Shared/ConfigRedaction.cs`: + +```csharp +namespace Kurrent.Replicator.Shared; + +/// Decides what may be printed for a configuration value. +public static class ConfigRedaction { + public static string? Display(string configKey, string? value) { + var segments = configKey.Split(':'); + + if (segments[^1].Equals("ConnectionString", StringComparison.OrdinalIgnoreCase)) return "***"; + + var authIndex = Array.FindIndex(segments, s => s.Equals("Auth", StringComparison.OrdinalIgnoreCase)); + + if (authIndex < 0) return value; + + var isAuthType = authIndex == segments.Length - 2 && segments[^1].Equals("Type", StringComparison.OrdinalIgnoreCase); + + return isAuthType ? value : "***"; + } +} +``` + +`src/replicator/Settings/EnvConfigProvider.cs` — replace the provider and record: + +```csharp +public class EnvConfigProvider : ConfigurationProvider { + public override void Load() { + var envVars = Environment.GetEnvironmentVariables(); + + var vars = envVars.Cast() + .Select(x => new EnvVar(x.Key.ToString()!, x.Value?.ToString())) + .Where(x => x.Key.StartsWith("REPLICATOR_") && x.Value != null) + .ToList(); + + foreach (var v in vars) Console.WriteLine($"{v.ConfigKey} = {ConfigRedaction.Display(v.ConfigKey, v.Value)}"); + + Data = vars.ToDictionary(x => x.ConfigKey, x => x.Value, StringComparer.OrdinalIgnoreCase); + } + + record EnvVar(string Key, string? Value) { + public string ConfigKey => Key.Replace("_", ":"); + } +} +``` + +Add `using Kurrent.Replicator.Shared;` at the top of the file. + +- [ ] **Step 6: Wire the host** + +`src/replicator/Settings/ReplicatorSettings.cs` — add `using Kurrent.Replicator.KurrentDb.Auth;` and extend `EsdbSettings`: + +```csharp +public record EsdbSettings { + public string ConnectionString { get; init; } + public string Protocol { get; init; } + public int PageSize { get; init; } = 1024; + public GrpcAuthSettings Auth { get; init; } = new(); +} +``` + +`src/replicator/Startup.cs` — add `using Kurrent.Replicator.KurrentDb.Auth;` and replace the gRPC configurator registration: + +```csharp + var readerAuth = AuthFor("reader", replicatorOptions.Reader); + var sinkAuth = AuthFor("sink", replicatorOptions.Sink); + + services.AddSingleton(_ => new(replicatorOptions.Reader.PageSize)); + services.AddSingleton( + sp => new(readerAuth, sinkAuth, sp.GetRequiredService().ApplicationStopping) + ); + services.AddSingleton(_ => new(replicatorOptions.Sink.Router)); +``` + +and add the helper at the bottom of the `Startup` class: + +```csharp + static GrpcAuthOptions AuthFor(string side, EsdbSettings settings) { + var options = (settings.Auth ?? new GrpcAuthSettings()).ToOptions(side); + + if (GrpcAuthOptionsValidator.ValidateProtocol(options, settings.Protocol) is { } error) + throw new InvalidOperationException($"Invalid {side} auth configuration: {error}"); + + return options; + } +``` + +- [ ] **Step 7: Run tests and build the host** + +Run: build, then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/*/AuthSettingsBindingTests/*"` and `ConfigRedactionTests`. +Expected: all PASS. + +Run: `dotnet build src/replicator -c Release -p:UseAppHost=false -v q` (Release skips the yarn install target) +Expected: 0 errors. + +Manual smoke test of fail-fast validation (no KurrentDB needed): + +```bash +mkdir -p /tmp/replicator-smoke/config && printf 'replicator:\n reader:\n protocol: tcp\n connectionString: "ConnectTo=tcp://localhost:1113"\n auth:\n type: oauthTokenFile\n tokenFile: /tmp/t\n sink:\n protocol: grpc\n connectionString: "esdb://localhost:2113"\n' > /tmp/replicator-smoke/config/appsettings.yaml +``` + +```bash +cd /tmp/replicator-smoke && DOTNET_ROLL_FORWARD=Major dotnet /Users/tony/dev/replicator/.claude/worktrees/replicator-oauth-auth-e66296/src/replicator/bin/Release/net9.0/replicator.dll +``` + +Expected: the process exits with `Invalid reader auth configuration: auth.type OAuthTokenFile is only supported with protocol grpc (configured: tcp)`. + +- [ ] **Step 8: Commit** + +```bash +git add src test/Kurrent.Replicator.Tests +git commit -m "feat(auth): per-side auth configuration, host wiring, env var redaction + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 14: Helm chart — inject secrets, files and workload identity + +**Files:** +- Modify: `charts/replicator/values.yaml` +- Modify: `charts/replicator/templates/statefulset.yaml` + +**Interfaces:** +- Produces Helm values: `serviceAccountName`, `podLabels`, `extraEnv`, `extraEnvFrom`, `extraVolumes`, `extraVolumeMounts` (all empty by default). `replicator.reader.auth` / `replicator.sink.auth` already pass through `toYaml .Values.replicator` in `configmap.yaml`. + +- [ ] **Step 1: Capture today's rendering** + +```bash +helm template r charts/replicator > /tmp/replicator-before.yaml +``` + +- [ ] **Step 2: Add values** + +Append to `charts/replicator/values.yaml` (top level): + +```yaml +# Service account for the pod (e.g. one federated with Azure Workload Identity). +serviceAccountName: "" +# Extra labels on the pod template, e.g. azure.workload.identity/use: "true". +podLabels: {} +# Extra environment variables. Use these (not replicator.*.auth in this file) for secrets, e.g. +# - name: REPLICATOR_SINK_AUTH_CLIENTSECRET +# valueFrom: { secretKeyRef: { name: replicator-oauth, key: client-secret } } +extraEnv: [] +extraEnvFrom: [] +# Extra volumes and mounts, e.g. a Secret holding clientSecretFile or a token file written by a sidecar. +extraVolumes: [] +extraVolumeMounts: [] +``` + +and document the `auth` block under `replicator.reader` / `replicator.sink` as a comment: + +```yaml +replicator: + reader: + connectionString: + protocol: tcp + # auth: # optional, gRPC only. See docs: deployment/authentication + # type: oauthClientCredentials + # tokenEndpoint: https://login.microsoftonline.com//oauth2/v2.0/token + # clientId: + # clientSecretFile: /var/run/secrets/replicator/client-secret # do not put clientSecret here + # scope: api:///.default +``` + +(Add the same commented block under `sink`.) + +- [ ] **Step 3: Template the pod spec** + +In `charts/replicator/templates/statefulset.yaml`: + +Pod labels — after `tier: web` in `spec.template.metadata.labels`: + +```yaml +{{- with .Values.podLabels }} +{{ toYaml . | indent 8 }} +{{- end }} +``` + +Service account — directly under `spec.template.spec:` (before `containers:`): + +```yaml +{{- with .Values.serviceAccountName }} + serviceAccountName: {{ . }} +{{- end }} +``` + +Container env — after `imagePullPolicy:`: + +```yaml +{{- with .Values.extraEnv }} + env: +{{ toYaml . | indent 10 }} +{{- end }} +{{- with .Values.extraEnvFrom }} + envFrom: +{{ toYaml . | indent 10 }} +{{- end }} +``` + +Volume mounts — at the end of the container's `volumeMounts:` list (after the partitioner block): + +```yaml +{{- with .Values.extraVolumeMounts }} +{{ toYaml . | indent 8 }} +{{- end }} +``` + +Volumes — at the end of `volumes:` (after the PVC volume): + +```yaml +{{- with .Values.extraVolumes }} +{{ toYaml . | indent 6 }} +{{- end }} +``` + +- [ ] **Step 4: Verify rendering** + +```bash +helm template r charts/replicator > /tmp/replicator-after.yaml && diff /tmp/replicator-before.yaml /tmp/replicator-after.yaml && echo "defaults unchanged" +``` + +Expected: `defaults unchanged`. + +```bash +helm template r charts/replicator --set serviceAccountName=replicator-wi --set podLabels."azure\.workload\.identity/use"=true --set 'extraEnv[0].name=REPLICATOR_SINK_AUTH_TYPE' --set 'extraEnv[0].value=oauthClientCredentials' --set 'extraVolumes[0].name=oauth' --set 'extraVolumes[0].secret.secretName=replicator-oauth' --set 'extraVolumeMounts[0].name=oauth' --set 'extraVolumeMounts[0].mountPath=/var/run/secrets/replicator' --set replicator.sink.auth.type=oauthTokenFile --set replicator.sink.auth.tokenFile=/var/run/secrets/replicator/token +``` + +Expected: valid YAML containing `serviceAccountName: replicator-wi`, the pod label, the `env` entry, the extra volume and mount, and an `auth:` block under `sink:` in the ConfigMap. Pipe it through `kubectl apply --dry-run=client -f -` if `kubectl` is available. + +- [ ] **Step 5: Commit** + +```bash +git add charts/replicator +git commit -m "feat(helm): extra env, volumes, service account and pod labels for OAuth + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 15: Documentation and changelog + +**Files:** +- Create: `docs/src/content/docs/deployment/authentication.mdx` +- Modify: `docs/src/content/docs/deployment/configuration.mdx` +- Modify: `docs/src/content/docs/features/readers.mdx` +- Modify: `docs/src/content/docs/features/sinks.mdx` +- Modify: `CHANGELOG.md` + +- [ ] **Step 1: Write the authentication page** + +`docs/src/content/docs/deployment/authentication.mdx`: + +````mdx +--- +title: Authentication +description: Authenticate the gRPC reader and sink with basic credentials or OAuth 2.0 access tokens. +sidebar: + order: 2 +--- + +Replicator authenticates to each KurrentDB cluster separately. The reader (source) and the sink (target) each have their own optional `auth` section, so any combination works: basic on one side and OAuth on the other, OAuth on both with different identity providers, or basic on both. + +OAuth is supported for the `grpc` protocol only. It requires TLS (`tls=true`, the default), and the connection string must not contain `user:pass@`. + +## Auth types + +| `auth.type` | Use when | +|:--|:--| +| `connectionString` (default) | Basic credentials in the connection string, or no authentication. Behaves exactly as before. | +| `oauthClientCredentials` | Replicator gets tokens from your identity provider with the OAuth 2.0 client credentials grant and refreshes them itself. | +| `oauthTokenFile` | Something else (a sidecar, Vault agent, CI job) keeps a valid access token in a file; Replicator reads it. | + +## Options + +| Option | Type | Description | +|:--|:--|:--| +| `tokenEndpoint` | `oauthClientCredentials` | Token endpoint URL. Must be `https` (plain `http` only for `localhost`). | +| `clientId` | `oauthClientCredentials` | OAuth client ID of the Replicator identity. | +| `clientSecret` | `oauthClientCredentials` | Client secret. Set it through the `REPLICATOR__AUTH_CLIENTSECRET` environment variable, not in the YAML file. | +| `clientSecretFile` | `oauthClientCredentials` | Path to a file with the client secret, e.g. a mounted Kubernetes Secret. | +| `clientAssertionFile` | `oauthClientCredentials` | Path to a signed JWT client assertion (RFC 7523), e.g. the Azure Workload Identity token file. Re-read on every token request. | +| `clientAuthentication` | `oauthClientCredentials` | `post` (default) sends the secret as form fields; `basic` sends it in an HTTP Basic header. | +| `scope` | `oauthClientCredentials` | Scopes to request. For Entra ID: `api:///.default`. | +| `additionalParameters` | `oauthClientCredentials` | Extra token request fields, e.g. `audience` (Auth0, Okta) or `resource` (Entra v1, ADFS). Keys are lower-cased. | +| `defaultTokenLifetimeSeconds` | `oauthClientCredentials` | Lifetime to assume if your provider omits `expires_in`. Required in that case. | +| `refreshBeforeExpirySeconds` | `oauthClientCredentials` | Refresh this many seconds before expiry. Default `300`, capped at half the token lifetime. | +| `tokenFile` | `oauthTokenFile` | Path to a file containing only the access token. | +| `tokenFileReloadSeconds` | `oauthTokenFile` | How often to re-read the file. Default `30`. | + +Exactly one of `clientSecret`, `clientSecretFile` and `clientAssertionFile` must be set. Every option can also be set with an environment variable, for example `REPLICATOR_SINK_AUTH_TOKENENDPOINT` or `REPLICATOR_SINK_AUTH_ADDITIONALPARAMETERS_AUDIENCE`. + +## KurrentDB permissions + +Replication reads `$all` and stream metadata on the source, and on the target it writes to any stream, sets stream metadata and deletes streams. Map the Replicator identity's role claim to `$admins` in the KurrentDB OAuth configuration on each cluster (or grant equivalent ACLs). + +If the reader's token cannot read a stream's metadata (`PermissionDenied`), Replicator stops replicating with an error rather than copying events it cannot check against scavenge rules. Fix the role mapping and restart Replicator. + +## How tokens are handled + +- Every gRPC call carries a current token. Tokens are refreshed before they expire, without restarting Replicator. +- If the identity provider is unreachable, or KurrentDB rejects a token, replication pauses with a warning (logged at most once a minute) and resumes on its own once a valid token is available. A token KurrentDB rejected is not sent again for 60 seconds, and after that only one call re-tests it. +- The long-running `$all` read and the realtime subscription are authorized when they start. If KurrentDB ends them when the token expires, Replicator restarts them from the last checkpoint with a fresh token. +- Tokens, secrets and assertions are never logged. + +## Example: Entra ID with a client secret + +1. Register an application for KurrentDB. Expose an Application ID URI (e.g. `api://kurrentdb`) and create an app role (e.g. `Replicator`) that your KurrentDB OAuth configuration maps to `$admins`. +2. Register an application for Replicator, create a client secret, and grant it the `Replicator` app role on the KurrentDB application (admin consent required). +3. Configure the side that talks to the OAuth-enabled cluster: + +```yaml +replicator: + reader: + protocol: grpc + connectionString: "esdb://source.example.com:2113?tls=true" + auth: + type: oauthClientCredentials + tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + clientId: "" + clientSecretFile: /var/run/secrets/replicator/client-secret + scope: "api://kurrentdb/.default" + sink: + protocol: grpc + connectionString: "esdb://admin:changeit@target.example.com:2113?tls=true" +``` + +## Example: Entra ID with AKS Workload Identity (no secret) + +Federate the Replicator application with the Kubernetes service account, then point `clientAssertionFile` at the projected token: + +```yaml +serviceAccountName: replicator-wi +podLabels: + azure.workload.identity/use: "true" +replicator: + sink: + protocol: grpc + connectionString: "esdb://target.example.com:2113?tls=true" + auth: + type: oauthClientCredentials + tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + clientId: "" + clientAssertionFile: /var/run/secrets/azure/tokens/azure-identity-token + scope: "api://kurrentdb/.default" +``` + +## Example: other providers + +Keycloak, Okta, Auth0 and other OAuth 2.0 servers work the same way. Many need an `audience`: + +```yaml + auth: + type: oauthClientCredentials + tokenEndpoint: "https://idp.example.com/oauth2/token" + clientId: replicator + clientSecretFile: /var/run/secrets/replicator/client-secret + additionalParameters: + audience: kurrentdb +``` + +Token file written by another process: + +```yaml + auth: + type: oauthTokenFile + tokenFile: /var/run/secrets/kurrentdb/token +``` + +## Helm + +Put secrets in Kubernetes Secrets and inject them with `extraEnv`, `extraEnvFrom`, `extraVolumes` and `extraVolumeMounts`; don't put `clientSecret` in your values file, because it ends up in a ConfigMap. See [Kubernetes](../kubernetes/). +```` + +- [ ] **Step 2: Update the configuration reference** + +In `docs/src/content/docs/deployment/configuration.mdx`, add rows after `reader.pageSize` and after `sink.bufferSize`: + +```markdown +| `reader.auth.*` | Optional reader [authentication](../authentication/) (gRPC only): `type` is `connectionString` (default), `oauthClientCredentials` or `oauthTokenFile` | +``` + +```markdown +| `sink.auth.*` | Optional sink [authentication](../authentication/) (gRPC only), configured independently of the reader | +``` + +- [ ] **Step 3: One-line notes in features** + +Append to `docs/src/content/docs/features/readers.mdx` and `features/sinks.mdx`: + +```markdown +The gRPC reader can authenticate with OAuth 2.0 access tokens instead of basic credentials. See [Authentication](../../deployment/authentication/). +``` + +(Use "gRPC sink" in `sinks.mdx`.) + +- [ ] **Step 4: Changelog** + +Under `## [Unreleased]` in `CHANGELOG.md`, add to `### Added`: + +```markdown +- OAuth 2.0 authentication for the gRPC reader and sink (client credentials or token file), configured independently per side, with automatic token refresh and recovery. Helm values for extra env, volumes, service account and pod labels. +``` + +and to `### Changed`: + +```markdown +- Connection strings and auth settings are no longer printed when environment variables are loaded. +- The realtime subscription used for scavenging resubscribes reliably after drops, and its metadata cache is rebuilt after a gap. +- Replication no longer hangs on shutdown when the writer has stopped; the metrics reporter keeps running after a failed position read. +``` + +- [ ] **Step 5: Build the docs** + +```bash +cd docs && pnpm install --frozen-lockfile && pnpm build +``` + +Expected: build succeeds and `dist/deployment/authentication/index.html` exists. If `pnpm` is not available, skip and say so in the PR description. + +- [ ] **Step 6: Commit** + +```bash +git add docs/src CHANGELOG.md +git commit -m "docs: OAuth authentication guide and configuration reference + +Co-Authored-By: Claude Opus 5.5 " +``` + +--- + +### Task 16: Full verification + +- [ ] **Step 1: Whole solution builds** + +Run: `dotnet build Kurrent.Replicator.slnx -c Release -p:UseAppHost=false -v q` +Expected: 0 errors, no new warnings in `Kurrent.Replicator.KurrentDb`. + +- [ ] **Step 2: All new tests** + +Run: `dotnet build test/Kurrent.Replicator.Tests -p:UseAppHost=false -v q` then `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/Kurrent.Replicator.Tests.Auth/*/*" --timeout 300s` +Expected: all PASS. Run it three times to catch flaky timing tests; fix any flake rather than retrying. + +- [ ] **Step 3: Existing container tests (needs Docker)** + +Run: `DOTNET_ROLL_FORWARD=Major dotnet exec test/Kurrent.Replicator.Tests/bin/Debug/net9.0/Kurrent.Replicator.Tests.dll --treenode-filter "/*/Kurrent.Replicator.Tests/*/*"` +Expected: `ValuePartitionerTests` and `ChaserCheckpointSeedingTests` PASS unchanged (they exercise basic-auth gRPC reads/writes, the reworked `Realtime`, `StreamMetaCache` and `Replicator`). If Docker is unavailable, note it for CI. + +- [ ] **Step 4: Secret hygiene sweep** + +```bash +git diff master --stat && git diff master -- src | grep -nE 'Log\.(Info|Warn|Error|Debug).*\b(ClientSecret|_secret|assertion|AccessToken|lease\.Value|Value\))' || echo "no suspicious log arguments" +``` + +Expected: `no suspicious log arguments` (review any hit manually). + +- [ ] **Step 5: Record the manual end-to-end plan** + +The spec's "Manual end-to-end" scenarios (OAuth sink, OAuth reader across ≥3 token lifetimes, OAuth on both sides with token-file rotation, token endpoint outage) need a licensed KurrentDB with the OAuth plugin and a real IdP. They are run before release and recorded in the PR description, including which long-lived-stream behaviour KurrentDB showed at token expiry. From 5c8e2bf53618b7fe0053a0872410ab000c996316 Mon Sep 17 00:00:00 2001 From: realtonyyoung <6655045+realtonyyoung@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:34:11 -0400 Subject: [PATCH 25/67] docs: spec and plan cover Kubernetes and Docker OAuth docs Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-10-03-oauth-authentication.md | 99 ++++++++++++++++++- .../2026-10-03-oauth-authentication-design.md | 2 + 2 files changed, 100 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-10-03-oauth-authentication.md b/docs/superpowers/plans/2026-10-03-oauth-authentication.md index 6e6035f8..eff940a5 100644 --- a/docs/superpowers/plans/2026-10-03-oauth-authentication.md +++ b/docs/superpowers/plans/2026-10-03-oauth-authentication.md @@ -4830,6 +4830,8 @@ Co-Authored-By: Claude Opus 5.5 " - Modify: `docs/src/content/docs/deployment/configuration.mdx` - Modify: `docs/src/content/docs/features/readers.mdx` - Modify: `docs/src/content/docs/features/sinks.mdx` +- Modify: `docs/src/content/docs/deployment/kubernetes.mdx` +- Modify: `docs/src/content/docs/deployment/docker.mdx` - Modify: `CHANGELOG.md` - [ ] **Step 1: Write the authentication page** @@ -4979,6 +4981,101 @@ The gRPC reader can authenticate with OAuth 2.0 access tokens instead of basic c (Use "gRPC sink" in `sinks.mdx`.) +- [ ] **Step 3a: Kubernetes page — Helm values and an OAuth example** + +In `docs/src/content/docs/deployment/kubernetes.mdx`, add these rows to the "Available options" table (after `replicator.transform.bufferSize`): + +```markdown +| `replicator.reader.auth.*` | Reader [authentication](../authentication/) (gRPC only), e.g. `type: oauthClientCredentials` | nil | +| `replicator.sink.auth.*` | Sink [authentication](../authentication/) (gRPC only), configured independently of the reader | nil | +| `serviceAccountName` | Service account for the pod (e.g. federated for Azure Workload Identity) | `""` | +| `podLabels` | Extra pod labels, e.g. `azure.workload.identity/use: "true"` | `{}` | +| `extraEnv` | Extra container environment variables (use for secrets such as `REPLICATOR_SINK_AUTH_CLIENTSECRET`) | `[]` | +| `extraEnvFrom` | Extra `envFrom` sources (Secrets, ConfigMaps) | `[]` | +| `extraVolumes` | Extra pod volumes (e.g. a Secret with a client secret or token file) | `[]` | +| `extraVolumeMounts` | Extra container volume mounts | `[]` | +``` + +Then add a subsection after "Provide configuration": + +````mdx +### OAuth authentication + +When a cluster uses OAuth (for example with Microsoft Entra ID), configure that side's `auth` section and inject the secret from a Kubernetes Secret. Never put `clientSecret` in `values.yml`: the `replicator` block is rendered into a ConfigMap. + +```bash +kubectl create secret generic replicator-oauth --from-literal=client-secret='' +``` + +```yaml +replicator: + reader: + protocol: grpc + connectionString: "esdb://admin:changeit@source.example.com:2113?tls=true" + sink: + protocol: grpc + connectionString: "esdb://[cloudclusterid].mesdb.eventstore.cloud:2113?tls=true" + auth: + type: oauthClientCredentials + tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + clientId: "" + clientSecretFile: /var/run/secrets/replicator/client-secret + scope: "api://kurrentdb/.default" +extraVolumes: + - name: replicator-oauth + secret: + secretName: replicator-oauth +extraVolumeMounts: + - name: replicator-oauth + mountPath: /var/run/secrets/replicator + readOnly: true +``` + +Alternatively, pass the secret as an environment variable instead of a file: + +```yaml +extraEnv: + - name: REPLICATOR_SINK_AUTH_CLIENTSECRET + valueFrom: + secretKeyRef: + name: replicator-oauth + key: client-secret +``` + +With Azure Workload Identity there is no secret at all. Set `serviceAccountName` to the federated service account, add the `azure.workload.identity/use: "true"` pod label, and use `clientAssertionFile: /var/run/secrets/azure/tokens/azure-identity-token` instead of `clientSecretFile`. See [Authentication](../authentication/) for every option. +```` + +- [ ] **Step 3b: Docker page — environment variables** + +In `docs/src/content/docs/deployment/docker.mdx`, add a subsection at the end: + +````mdx +## OAuth authentication + +Every `auth` option can be set with an environment variable named `REPLICATOR__AUTH_