From aa1818c15390983683f7c7646e3e3b9c8afaba28 Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 18:12:28 +1000 Subject: [PATCH 01/21] Add plan for host audit ingestion in the primary instance This design document outlines the architectural strategy and work plan for moving audit ingestion into the primary ServiceControl process for SQL Server and PostgreSQL persistence, including support for scaled-out ingestion-only workers. --- src/audit-ingestion-in-primary-plan.md | 615 +++++++++++++++++++++++++ 1 file changed, 615 insertions(+) create mode 100644 src/audit-ingestion-in-primary-plan.md diff --git a/src/audit-ingestion-in-primary-plan.md b/src/audit-ingestion-in-primary-plan.md new file mode 100644 index 0000000000..a4b8bb5d5e --- /dev/null +++ b/src/audit-ingestion-in-primary-plan.md @@ -0,0 +1,615 @@ +# Host Audit Ingestion in the Primary Instance + +## Summary + +For SQL Server and PostgreSQL persistence, move audit ingestion and the supporting audit capabilities into the primary ServiceControl process. A normal primary instance ingests audit messages by default once its persister advertises audit support. A setting disables its receiver. Additional primary processes can run with `--audit-ingestion-only` to scale ingestion through competing consumers. + +The existing standalone RavenDB audit instance remains supported and retains its current behavior. RavenDB does not gain combined hosting or audit-ingestion-only support. + +This plan covers the contracts, project boundaries, host composition, settings, and fail-fast command-line surface needed before EF audit persistence is implemented. It makes no installer changes, so RavenDB instances are unaffected by construction. It does not implement EF entities, migrations, SQL queries, retention algorithms, or provider registrations. + +## Goals + +- Host audit ingestion in the normal SQL Server or PostgreSQL primary instance. +- Allow audit ingestion to be disabled in the normal primary instance. +- Add `--audit-ingestion-only` so additional processes can scale audit ingestion. +- Include the audit capabilities that must exist when audit data is local: SagaAudit ingestion, failed audit imports, forwarding, ingestion metrics, health, and querying. +- Reuse primary-owned capabilities rather than duplicating them: endpoint detection, retry acknowledgement handling, body storage, retention sweeping, and known endpoints. +- Use the same database and existing primary EF persistence for primary and audit data. +- Keep existing primary API routes and their existing authorization policies. +- Continue supporting additional audit remotes through the existing scatter-gather. +- Restore the platform capabilities that today depend on an audit remote existing: platform connection details, saga audit forwarding, and licensing throughput collection. +- Put contracts and composition boundaries in place without starting the EF audit implementation. + +## Non-goals + +- Adding audit persistence to SQL Server or PostgreSQL in this work. +- Adding combined hosting or ingestion-only support to RavenDB. +- Replacing or removing the existing standalone RavenDB audit instance. +- Supporting the standalone `ServiceControl.Audit` executable with SQL Server or PostgreSQL. +- Finalizing provider-specific retention, partitioning, full-text search, or body storage implementations. +- Migrating existing RavenDB audit data into EF persistence. +- Running error ingestion and audit ingestion in the same ingestion-only worker. + +## Decisions + +### Data and persistence + +- Audit and primary data use the same database and connection configuration. +- Shared data, including known endpoints, uses the existing primary tables. +- Audit-owned data uses explicit table names in the existing default schema, for example `AuditMessages`, `FailedAuditImports`, and `SagaSnapshots`. +- SQL Server and PostgreSQL extend the existing `ServiceControl.Persistence.EFCore`, `ServiceControl.Persistence.EFCore.SqlServer`, and `ServiceControl.Persistence.EFCore.PostgreSql` projects. The three audit-specific EF projects from the earlier spike are not recreated. +- The `SagaSnapshots` table maps the existing `ServiceControl.SagaAudit.SagaSnapshot` type. That type lives in `ServiceControl.Audit.Persistence.SagaAudit` and is already on the primary's reference graph through `ServiceControl.SagaAudit`. It does not move. + +### Hosting + +- The normal primary retains the existing HTTP routes and serves local audit data through them. There is no separate SQL Server or PostgreSQL audit HTTP service. *Superseded on 14 September 2026: audit load can swamp a database that copes with the error instance, so a customer must be able to move audit to a dedicated database. That database is served by the same executable in `--audit-instance` mode, and the topology is specified in the EF audit persistence plan.* +- `--audit-ingestion-only` always ingests and does not host an NServiceBus endpoint. +- `--audit-ingestion-only` and `--error-ingestion-only` are mutually exclusive. Passing both fails at startup with a clear message. Each queue gets its own worker pool so the two can be scaled independently, and each keeps a single, auditable component list. Combining them is a possible follow-up. +- Disabling ingestion in the normal primary stops only its receiver. Local queries, SagaAudit, failed-import tooling, and other audit capabilities remain active because workers may still ingest. + +### API + +- Local audit data is served through the existing primary routes under their existing policies. `/api/messages` and its variants stay on `error:messages:view`, `/api/sagas/{id}` stays on `error:sagas:view`, and `endpoints/{endpoint}/audit-count` stays on `error:messages:view`. + A primary configured with an audit remote already serves that remote's audit data under `error:messages:view` today, so local audit data inherits an established gate and nothing about the `my/routes` manifest or ServicePulse navigation changes. The standalone audit instance keeps its own `audit:*` policies and its anonymous audit-count route. + +### Settings and installer + +- The primary reads the audit settings under the same key names the audit instance uses. Key names are reused rather than invented. +- No installer changes in this work. `ServiceControlInstaller.*`, `ServiceControl.Config` and `ServiceControl.Management.PowerShell` are untouched, so RavenDB instances behave exactly as they do today by construction. SCMU and PowerShell support for EF storage types is a separate, undecided workstream, and the audit settings belong to it. See "Settings and commands" for the handoff note. + +### Observability + +- The copied ingestion metrics keep their OpenTelemetry implementation. The primary gains the three OpenTelemetry package references and an `OtlpEndpointUrl` setting, and the meter is renamed from `Particular.ServiceControl.Audit` to `Particular.ServiceControl`. + Copying faithfully keeps the primary and audit implementations comparable for the later reuse assessment, and it opens the door to moving error ingestion onto the same instrumentation. The cost is three new packages in the shipped primary artifact. + +## What the primary already owns + +Several capabilities the earlier draft treated as "moving from audit" already exist on the primary. The plan reuses them rather than copying an audit equivalent. + +| Capability | Where it already lives | Consequence for this work | +| --- | --- | --- | +| Local-first scatter-gather | `ScatterGatherApi.Execute` runs the local query first, then remotes | No new "query coordinator" is needed. With zero remotes it is already a local-only query. | +| Message view queries | `IMessagesViewDataStore` | The five audit message queries extend this contract. Its EF implementation performs the union. | +| Saga history DTOs | `ServiceControl.Audit.Persistence.SagaAudit`, referenced through `ServiceControl.SagaAudit` | No new DTO project. `GetSagaByIdApi` stops being remote-only. | +| Audit retention period setting | `ServiceControl/AuditRetentionPeriod`, already validated and published in `/api/configuration` | Reuse it. Define what `null` means now that it drives behavior. | +| Full-text search toggle | `PersistenceSettings.EnableFullTextSearchOnBodies` | One value governs error and audit bodies. | +| Max body size | `BodyStorageSettings.MaxBodySizeToStore` | One value governs error and audit bodies. | +| Body storage and installers | `IBodyStorage`, `IBodyStoragePersistence`, FileSystem, AzureBlob and S3 implementations plus installers | Audit bodies use the same store. There is no database body store. | +| Retention sweeping | `RetentionSweeper`, a `BackgroundService` registered inside `BasePersistence` | Audit retention extends the sweeper. There is no host-level retention component. | +| Endpoint detection pattern | `DetectNewEndpointsFromErrorImportsEnricher` plus `unitOfWork.Monitoring.RecordKnownEndpoint` in `ErrorProcessor` | The audit enricher adopts the same shape. | +| Retry acknowledgement handling | `RetryConfirmationProcessor`, driven by acknowledgements arriving on the error queue | Unchanged. See "Retry acknowledgements" below. | +| Internal custom checks | `services.AddCustomCheck()` plus `InternalCustomChecksHostedService` | The copied audit checks register through DI, not through `configuration.AddCustomCheck`. | +| Saga audit misconfiguration handling | `SagaUpdatedHandler` and `SagaAuditMisconfigurationCustomCheck` | Both need work. See "Platform connection details". | + +## Project and boundary assessment + +Do not add a new runtime project for the first implementation. Copy the audit runtime behavior needed by the SQL Server and PostgreSQL primary host into the `ServiceControl` project, then adapt that copy to primary persistence, primary settings, and the endpoint-free ingestion-only profile. + +The copied primary implementation includes: + +- Audit receiving, batching, and shutdown coordination. +- Audit message parsing and enrichment. +- Saga snapshot and relationship processing. +- Failed-ingestion handling and failed-audit orchestration. +- Forwarding orchestration. +- Ingestion metrics and readiness state. +- Registrations for the normal and ingestion-only primary hosts. + +Do not make the primary executable reference `ServiceControl.Audit.csproj`. That project remains a standalone composition root containing RavenDB persistence selection, standalone settings, API hosting, installer commands, and its own NServiceBus endpoint. + +Keep these concerns in the existing standalone audit executable: + +- RavenDB persistence loading and lifecycle. +- Standalone audit settings and maintenance mode. +- Standalone audit HTTP API composition. +- Standalone installers and queue setup behavior. +- The existing audit NServiceBus endpoint and its `ReportCustomChecksTo` reporting. + +### Shared surface + +The two executables are not isolated. Four projects sit underneath both, and this work must change at least one of them. + +| Project | Shared how | Risk | +| --- | --- | --- | +| `ServiceControl.SagaAudit` | Compiled into `ServiceControl.Audit` by source, referenced as a project by `ServiceControl.Persistence` | A change to `SagaSnapshotFactory` or `InvokedSagasParser` silently changes the shipped audit executable. | +| `ServiceControl.Audit.Persistence.SagaAudit` | Referenced by the audit persisters, the Raven primary persister, and transitively by `ServiceControl.Persistence` | The saga DTOs are shared. Changing their shape affects Raven audit documents. | +| `ServiceControl.Infrastructure` | `Watchdog`, `DeterministicGuid`, `ReadOnlyStream`, `LoggerUtil`, `Permissions` | A shutdown or watchdog change for the primary changes audit shutdown too. | +| `ServiceControl.Transports` | `ITransportCustomization.CreateTransportInfrastructure` must change for per-receiver concurrency | Both hosts create their receivers through this method. | + +Any pull request touching these four projects runs the full audit acceptance suite and states in its description why the change is safe for the audit executable. + +Note that the baseline already breaks the "audit runtime untouched" guarantee: PR #5800 modifies `AuditIngestion`, `AuditIngestor` and `AuditPersister` in the standalone audit project. The guarantee this plan makes is narrower and honest: no *behavioral* change to the standalone audit executable, verified by its acceptance suite. + +### Divergence and later reuse + +The copied implementations are expected to diverge initially. The primary copy removes endpoint assumptions, uses the primary persistence unit of work, and participates in local queries. The RavenDB implementation remains optimized for its existing standalone process. Once both paths are stable, compare them and extract shared code only where doing so removes meaningful duplication without coupling their composition roots. + +Do not add a separate contracts project. Define the new primary audit contracts in `ServiceControl.Persistence`. Leave the current RavenDB audit persistence contracts and implementation untouched unless a later reuse refactor demonstrates a clear benefit. + +## Target host profiles + +| Capability | Normal SQL/Postgres primary | `--audit-ingestion-only` | Standalone RavenDB audit | +| --- | --- | --- | --- | +| Audit receiver | Enabled by setting, default on when persistence supports audit | Always enabled | Unchanged | +| Primary NServiceBus endpoint | Yes | No | Not applicable | +| Existing audit NServiceBus endpoint | No | No | Unchanged | +| Primary API | Yes | Health endpoints only, mapped as minimal API routes, no controllers | Existing audit API unchanged | +| Local audit queries | Yes | No | Unchanged | +| Optional remote audit queries | Yes | No | Unchanged | +| Endpoint discovery | Shared persistence unit of work | Shared persistence unit of work | Unchanged | +| Retry acknowledgement dispatch | Yes | Yes | Unchanged | +| Retry acknowledgement recording | Yes, via error ingestion | No | Not applicable | +| Forwarding | Yes | Yes | Unchanged | +| Failed-audit storage | Yes | Yes | Unchanged | +| Failed-audit reimport command | Yes | No | Unchanged | +| Retention | Inside the persister, gated by `RunRetentionSweep` | Off, `RunRetentionSweep` false | Unchanged, RavenDB document expiry | +| Platform connection details for audit | Yes, local provider | No | Unchanged | +| Licensing, throughput, email, event dispatch | Yes | No | Unchanged | +| Internal custom checks | Yes | Yes | Unchanged | +| Liveness and readiness | Yes | Yes | Unchanged unless adopted separately | + +## Persistence contract direction + +The current primary persistence design already exposes capability-specific children from `IIngestionUnitOfWork`. Add audit as a sibling to monitoring and recoverability: + +```csharp +public interface IIngestionUnitOfWork : IAsyncDisposable +{ + IMonitoringIngestionUnitOfWork Monitoring { get; } + IRecoverabilityIngestionUnitOfWork Recoverability { get; } + IAuditIngestionUnitOfWork Audit { get; } + Task Complete(CancellationToken cancellationToken = default); +} +``` + +The audit child initially expresses only the operations the runtime requires, without defining EF storage details: + +- Record a processed audit message and its body reference. +- Record a Saga snapshot. + +During a batch, the audit runtime uses the existing capability children as well: + +- `Monitoring.RecordKnownEndpoint(...)` records endpoints detected from audit headers. +- `Audit.RecordProcessedMessage(...)` records an audit message. +- `Audit.RecordSagaSnapshot(...)` records a Saga snapshot. +- `Complete(...)` commits all derived state atomically where the persistence supports it. + +This replaces the earlier spike's duplicated `KnownEndpoints` table, insert-only staging table, and reconciliation process. + +### Query contracts + +Extend `IMessagesViewDataStore` rather than adding a parallel audit query contract. Its five existing queries are exactly what the scatter-gather APIs call. The EF implementation unions failed messages and audit messages, subject to the precedence rule below. + +Only three genuinely new query contracts are required: + +- Audit counts per endpoint. +- Saga history by saga id. +- Audit body resolution, folded into `IBodyStorage` rather than added alongside it. + +Plus two ingestion-side contracts: + +- Failed audit import storage and reimport selection, mirroring `IFailedErrorImportDataStore`. +- Persistence capability discovery. + +Do not move the existing monolithic `IAuditDataStore` into primary persistence. + +### Capability discovery + +Audit support is declared in `persistence.manifest`, read by `ServiceControl.Persistence.PersistenceManifest`. Add: + +```json +"SupportsAuditIngestion": true +``` + +to the SQL Server and PostgreSQL manifests. The property is absent from the RavenDB manifest and from every file under `LegacyArtifacts`, and absent means false. + +The installer has its own `PersistenceManifest` class over the same file. It does not need the property yet, and this work does not add it there. Whenever SCMU gains EF storage support it can pick the property up from the same file, so there is one source of truth waiting for it. + +Host composition must not infer support by resolving optional services or catching startup failures. + +## Removing the ingestion endpoint dependency + +The ingestion-only process must not host an NServiceBus endpoint. Follow the approach established by the error-ingestion scale-out work: + +- The receiver owns its low-level transport infrastructure and dispatcher. +- Forwarding uses the dispatcher belonging to the receiving infrastructure. +- Shutdown stops receiving under the real shutdown token, completes the writer, drains the channel, and only then tears down transport infrastructure. +- `HostInformation` and critical-error handling are supplied directly by the host. +- `IMessageSession` is absent and tested as absent. + +### Endpoint detection + +`DetectNewEndpointsFromAuditImportsEnricher` currently sends a `RegisterNewEndpoint` command through `IMessageSession`, routed to the primary's queue, where `RegisterNewEndpointHandler` calls `EndpointInstanceMonitoring.EndpointDetected`. + +In the primary copy it instead writes through `IMonitoringIngestionUnitOfWork.RecordKnownEndpoint`, matching `ErrorProcessor`. Both paths write the same `KnownEndpoints` table. + +This is the only use of `AuditEnricherContext.AddForSend(ICommand)` in the tree. Once it is gone, `IMessageSession` drops out of the copied `AuditPersister` entirely, and the `ICommand` overload is deleted from the copied `AuditEnricherContext`. Saga relationship enrichment and saga snapshot processing never needed the endpoint. + +One behavioral difference must be characterized before the switch. The command path raises the `EndpointDetected` domain event, which `MonitoringDataPersister` handles and which anything downstream of the domain event observes. The unit-of-work path does not. Write a test that pins the current observable outcome, then decide whether the audit path must raise it. + +### Retry acknowledgements + +Do not short-circuit retry acknowledgements into `IRecoverabilityIngestionUnitOfWork`. + +`DetectSuccessfulRetriesEnricher` does not perform a round trip to the primary endpoint. It emits a raw transport operation to whatever queue the `ServiceControl.Retry.AcknowledgementQueue` header names. That header is stamped by the instance that issued the retry, using its own error queue address. The receiving instance turns it into `RecordSuccessfulRetry` through `RetryConfirmationProcessor` on the normal error ingestion path. + +Writing directly to the local recoverability unit of work is only correct when the acknowledgement queue resolves to this instance's error queue. Where the retry was issued by a different primary, a direct write records the confirmation in the wrong database and the real owner never resolves the failed message. The endpoint-side acknowledgement, signalled by `ServiceControl.Retry.AcknowledgementSent`, arrives on the error queue regardless, so the transport path cannot be removed anyway. + +In a combined host the acknowledgement is dispatched to the local error queue and comes straight back into local error ingestion. That is one broker round trip, it is exactly what happens today, and it is provably correct. Keep it. Revisit the optimization only with a rule that compares the acknowledgement queue against the local error queue. + +Transport operations therefore remain in the audit ingestion path for two reasons only: forwarding, and the retry acknowledgement. + +## API and query behavior + +Keep the existing primary API routes and policies used by ServicePulse, including messages, searches, conversations, audit counts, bodies, and Saga history. + +No new query coordinator is required. `ScatterGatherApi.Execute` already runs the local query first and the remotes after, so a primary with no remotes already performs a local-only query. Two existing entry points must stop being remote-only: + +- `GetAuditCountsForEndpointApi.LocalQuery` returns `Empty` with the comment "Will never be implemented on the primary instance". It takes an `IMessagesViewDataStore` it never uses. Both the comment and the unused dependency go. +- `GetSagaByIdApi` derives from `ScatterGatherRemoteOnly`. It becomes a normal `ScatterGatherApi` over the new saga history contract. + +### Precedence, paging, and counting + +`ScatterGatherApiMessageView.ProcessResults` deduplicates on `{ReceivingEndpoint.Name}-{MessageId}` using `TryAdd`, and relies on a documented invariant: the first result set comes from the main instance, so failed-message data wins over audit data. + +Once one local result set contains both failed and audited rows through a union, that cross-source invariant becomes an intra-list ordering requirement on the SQL. If an audit row precedes the failed row for the same key, ServicePulse shows a message as successfully processed when it actually failed. + +The local query contract must therefore state three rules, and each needs a test: + +1. **Precedence.** Within a single local result set, the failed-message row for a given `{ReceivingEndpoint.Name}-{MessageId}` must precede the audit row. Either the union orders by source, or the local query deduplicates before returning. +2. **Paging.** `ProcessResults` truncates with `Take(PageSize)`. A local union that returns `PageSize` rows per source is silently truncated. The local query returns at most `PageSize` rows after its own deduplication. +3. **Counting.** `AggregateStats` sums `TotalCount` across sources. A message that both failed and was audited must be counted once by the local query, not once per source. + +RavenDB primary instances continue using their existing local-error-plus-remote-audit behavior, which these rules do not change. + +## Body storage + +Every ingestion process must write bodies to storage readable by the normal primary and every relevant worker. Blob, S3, and explicitly shared filesystem storage are acceptable. There is no database body store in the primary EF persistence. + +### Body id keyspace + +The two sides key bodies differently today, and merging them into one store without a rule produces wrong answers. + +- Audit body id is `Headers.MessageId`. +- Primary external body id is `UniqueMessageId`. +- `BodyStorage.TryFetch` resolves only against `FailedMessages`, first by `UniqueMessageId` and then falling back to `MessageId`. + +Without a rule, `GET /api/messages/{messageId}/body` for a message that both failed and was audited returns the failed copy, which for an edited message is a different body, and for an audited-only message returns 404 because nothing consults the audit tables. + +The plan adopts the existing precedent. `FailedErrorImportEntity.ExternalBodyId(...)` already prefixes a distinct keyspace inside the same store. Audit bodies use their own prefix. + +`IBodyStorage.TryFetch` gains an explicit arbitration order, stated once and tested: + +1. Failed message by `UniqueMessageId`. +2. Failed message by `MessageId`. +3. Audit message by `UniqueMessageId`, including bodies embedded in the row for full-text search. + +Retention must sweep both keyspaces. + +### Filesystem body storage in ingestion-only mode + +The earlier draft required rejecting a "node-local filesystem path" at startup. That check is not implementable. `FileSystemBodyStorageSettings` carries only a path, a compression threshold and a size cap, and nothing distinguishes a shared mount from a local directory. + +Instead, filesystem body storage requires an explicit opt-in in ingestion-only mode. Add a setting that asserts the path is shared. Without it, an ingestion-only worker configured for filesystem body storage fails at startup with a message naming the setting. + +Apply the same rule to `--error-ingestion-only`, which PR #5801 documents as a known gap. The two ingestion-only modes must not disagree about this. + +## Platform connection details + +Two primary-owned capabilities currently depend on an audit remote existing, and both break in combined mode. + +### Saga audit forwarding + +`SagaUpdatedHandler` throws `UnrecoverableException` when it cannot resolve `SagaAudit.SagaAuditQueue`. That key is produced only by the standalone audit instance's `ConnectionController` and reaches the primary only through `RemotePlatformConnectionDetailsProvider`. A combined primary with no remotes fails every misdirected `SagaUpdatedMessage`. + +### Endpoint configuration + +`/api/connection` is what ServicePulse and the Platform Connector plugin read to configure endpoints. Without an audit remote it stops advertising `MessageAudit.AuditQueue` and `SagaAudit.SagaAuditQueue`, so endpoints cannot be told where to send audit or saga data at all. + +### Correction + +Add an audit platform connection details provider to the primary, registered when the persister advertises audit support and the primary owns an audit queue. It supplies the same `MessageAudit` and `SagaAudit` shapes the audit instance supplies today, so ServicePulse and the plugin see no difference. + +`SagaUpdatedHandler` then resolves the local audit queue through the existing `IPlatformConnectionBuilder` with no change to its logic. Whether it should instead hand the snapshot straight to the audit unit of work is an open item, not a requirement of this plan. + +## Licensing and throughput + +Audit throughput collection is driven entirely by remote instances. `AuditQuery.GetAuditRemotes` derives audit queues, retention and version from `configurationApi.GetRemoteConfigs()`. `AuditThroughputCollectorHostedService.SaveAuditInstanceData` sets the static `AuditQueues` list from those remotes, and `PlatformEndpointHelper.IsPlatformEndpoint` uses that list to exclude platform queues from throughput. + +With local audit and no remotes: + +- `AuditQueues` stays empty, so the local `audit` and `audit.log` queues are counted as customer endpoints in the licensing throughput report. This is a licensing accuracy defect, not cosmetic. +- `SaveAuditServiceMetadata` records no audit versions or transports, so `AuditServicesData` in the report is blank. +- `TestAuditConnection` reports "No Audit Instances configured" in the ServicePulse diagnostics. + +`IAuditQuery` gains a local audit source alongside the remote one. It contributes the local audit queue names, the local audit retention period, and the local instance version, and it satisfies the existing "minimum 2 days retention" gate from local settings rather than from a remote's configuration payload. `Particular.LicensingComponent` is an affected project and is listed in the work plan. + +## Settings and commands + +### Runtime settings + +Settings reach the primary through `SettingsReader`, which reads environment variables, the registry and `ServiceControl.exe.config` independently of the installer. Nothing below requires an installer change to work. + +| Setting | Status | Notes | +| --- | --- | --- | +| `ServiceControl/IngestAuditMessages` | New on the primary | Effective default is `true` where the persister advertises audit support, `false` otherwise. Applies to the normal primary host only. Always on under `--audit-ingestion-only`. | +| `ServiceBus/AuditQueue` | Reused key name | Same key the audit instance reads. Default `audit`. Not written by SCMU on a primary, see the installer note. | +| `ServiceBus/AuditLogQueue` | Reused key name | Defaults to the subscoped audit queue name, matching the audit instance. Not written by SCMU on a primary. | +| `ServiceControl/ForwardAuditMessages` | Reused key name | Default `false`, matching the audit instance. Not written by SCMU on a primary. | +| `ServiceControl/AuditRetentionPeriod` | Already exists | `TimeSpan?`, already validated at min 1 hour and max 365 days, already published in `/api/configuration`. Define `null` as "use the persister default", and state that default. | +| `ServiceControl/EnableFullTextSearchOnBodies` | Already exists | One value governs error and audit bodies. | +| `ServiceControl/MessageBody/...` | Already exists | One body storage configuration governs error and audit bodies. | +| Maximum audit ingestion concurrency | New | See the transport change below. | +| `ServiceControl/TimeToRestartAuditIngestionAfterFailure` | New on the primary | Mirrors the existing error equivalent. | +| `ServiceControl/OtlpEndpointUrl` | New on the primary | Required by the copied OpenTelemetry metrics. | +| Shared filesystem body storage assertion | New | Required by ingestion-only mode. See "Body storage". | + +### Key collisions + +`ServiceControl` and `ServiceControl.Audit` settings can both be set by bare environment variable name. `ServiceControl.Audit` also falls back to `ServiceControl/IngestAuditMessages` for backwards compatibility, and `ServiceBus/AuditQueue` is literally the same key for both processes. + +The consequence is that a primary in combined mode and a standalone audit instance sharing one environment file will collide on `INGESTAUDITMESSAGES`, `AUDITRETENTIONPERIOD`, `FORWARDAUDITMESSAGES` and `SERVICEBUS_AUDITQUEUE`. + +That combination is documented as unsupported. The primary logs a warning at startup when it has audit ingestion enabled and audit remotes configured at the same time, because that is the shape most likely to hit the collision. + +### Installer: out of scope, with one handoff + +No files under `ServiceControlInstaller.Engine`, `ServiceControl.Config` or `ServiceControl.Management.PowerShell` change in this work. RavenDB instances therefore behave exactly as they do today, by construction rather than by test. That is the requirement. + +Windows deployments of an audit-capable primary configure these settings the same way EF instances are configured today, through environment variables or the config file directly, because SCMU does not yet support EF storage types at all. + +The handoff, for whoever picks up SCMU and PowerShell support for EF storage types: + +`ServiceBus/AuditQueue`, `ServiceBus/AuditLogQueue` and `ServiceControl/ForwardAuditMessages` are declared `RemovedFrom = 4.0.0` in `ServiceControlSettings`, and `ServiceControlAppConfig.UpdateSettings` calls `RemoveIfRetired` on each one. Once SCMU manages an audit-capable primary, applying settings would strip that instance's audit queue configuration out of `ServiceControl.exe.config`. + +Version gating cannot express "supported on EF, retired on RavenDB", so the gate has to move to the persister. `IServiceControlInstance` already exposes `PersistenceManifest` through `IPersistenceConfig`, so `ServiceControlAppConfig` has what it needs: drop `RemovedFrom` from the three `SettingInfo` declarations, branch on the manifest, and call `settings.Remove(name)` on the non-audit branch to preserve today's RavenDB behavior. The three `RemoveIfRetired` calls become no-ops once `RemovedFrom` is gone, so they have to be deleted rather than left in place. + +This is recorded so the trap is visible, not so it is fixed here. It is only reachable once SCMU can create an EF instance. + +### Transport concurrency + +`TransportSettings` is a singleton registered once by `AddTransportForPrimary`, and `CreateTransportInfrastructure` reads `transportSettings.MaxConcurrency.Value` for its `PushRuntimeSettings`. `CustomizePrimaryEndpoint` sets the default to 10; `CustomizeAuditEndpoint` sets it to 32. In a combined host only the primary path runs, so audit ingestion would run at 10 rather than 32, a threefold regression against the standalone instance, and the proposed per-receiver setting could not be honored at all. + +`ITransportCustomization.CreateTransportInfrastructure` gains an explicit concurrency argument, and `AuditIngestion` derives its channel bound and batch size from that value rather than from the shared singleton. This changes a project shared with the audit executable, so it runs the audit suite. Consider folding it into PR #5800 while that is still open. + +### Commands + +Add `--audit-ingestion-only` to the primary command-line parser and to `Help.txt`. + +Before EF audit persistence exists, the command is present but fails with a clear message that the selected persistence does not support audit ingestion. It also fails for RavenDB, and it fails when combined with `--error-ingestion-only`. + +The normal primary must not activate audit ingestion until a persister advertises audit support, so the groundwork merges without changing existing behavior. + +The setup command provisions the audit queue and optional audit forwarding queue for audit-capable primaries, through the existing `IComponentInstallationContext.CreateQueue` mechanism. The deployment or update setup path owns all infrastructure changes, including database schema migrations, queue creation, and body storage provisioning. Ingestion-only workers do not run installers or perform any setup or upgrade work. + +## Scale-out rules + +Per-message operations run on every normal or ingestion-only receiver and must be safe with concurrent writers: + +- Audit message and Saga snapshot inserts. +- Known endpoint upserts. +- Failed audit import storage. +- Body storage writes. +- Audit forwarding. +- Retry acknowledgement dispatch. + +Only the normal primary runs singleton work: + +- Failed-audit reimport commands. +- API hosting and remote aggregation. +- Licensing and throughput ownership. +- Email notifications. +- Event and integration dispatch polling. +- Retention, which lives inside the persister and is gated by `RunRetentionSweep`. + +### Idempotency + +The earlier draft asserted idempotency as an acceptance criterion. The current code does not have it, so it is a requirement with named keys. + +- `AuditIngestionFaultPolicy` sets `FailedAuditImport.Id = Guid.NewGuid()`. With competing consumers plus immediate retries, one poison message writes a new row per attempt per worker, the custom check fires permanently, and `--import-failed-audits` reprocesses duplicates. Replace it with a deterministic key plus a native-id fallback, modelled on `FailedErrorImport.DeriveKey`. +- `ProcessedMessage.Id` is `ProcessedMessages-{processingStartedTicks}-{ProcessingId()}`, and `ProcessingId()` returns a fresh `Guid` whenever message id, processing endpoint or processing-started headers are missing. State the deduplication key for audit rows, including that degenerate case. +- `ProcessingEndpointName()` throws for headers it cannot resolve. The per-message failure path already handles that, and the failed-import key must not depend on it. + +### Ingestion-only component list + +PR #5801 registers `EventLog`, `ExternalIntegrations`, `Recoverability`, `HeartbeatMonitoring` and `CustomChecks` in `--error-ingestion-only`, with the reasoning that which node ingests a given message is arbitrary, so nodes behaving differently makes derived data a coin flip per message. + +The audit ingestion-only host registers: + +| Component | Reason | +| --- | --- | +| `HeartbeatMonitoring` | `DetectNewEndpointsFromAuditImportsEnricher` asks `EndpointInstanceMonitoring.IsNewInstance`, which must be warmed from persistence. Without it every audited message writes a known-endpoint upsert. | +| `CustomChecks` | A stuck worker must report somewhere. Without it, an ingestion failure on a worker is invisible. | + +It does not register `Hosting`, which claims the instance queue, or `Licensing`, which would count throughput once per node. `EventLog` and `ExternalIntegrations` are not required because audit ingestion raises no domain events and no integration events. State that explicitly in the composition test so a future registration forces a decision. + +## Observability, health, and packaging + +- The copied `IngestionMetrics` keeps its OpenTelemetry implementation. `ServiceControl.csproj` gains `OpenTelemetry.Exporter.Console`, `OpenTelemetry.Exporter.OpenTelemetryProtocol` and `OpenTelemetry.Extensions.Hosting`, and the primary gains an `OtlpEndpointUrl` setting wired the same way the audit host wires it. The meter is renamed to `Particular.ServiceControl`. +- The copied custom checks are renamed so they do not collide with the standalone audit instance reporting the same names to the same primary through `ReportCustomChecksTo`. `FailedAuditImportCustomCheck` also drops its `ServiceControl.Audit Health` category, which would otherwise appear on a process that is not the audit instance. They register through `services.AddCustomCheck()`, not `configuration.AddCustomCheck`. +- `/health` and `/health/ready` follow the error-ingestion-only conventions from PR #5803, mapped as minimal API routes. +- `AuditIngestor.VerifyCanReachForwardingAddress` dispatches an empty probe message to the log queue on every infrastructure start. With N workers restarting under the watchdog, N probes accumulate. Decide whether ingestion-only workers verify forwarding at all, and what happens when the log queue does not exist because setup has not run. +- Confirm the copied runtime ships inside the existing primary artifact without adding another assembly, and that the new package references do not break `ServiceControlInstaller.Packaging.UnitTests`. + +## Work plan + +### 1. Establish the baseline + +- Land or rebase on PRs #5800, #5801 and #5803. None of them is merged, and `RunRetentionSweep`, `--error-ingestion-only`, `/health` and dispatcher-as-argument do not exist without them. +- Record the exact service composition of the existing standalone audit host. +- Write the characterization tests listed below. The existing audit acceptance suite is dominated by CORS, HTTPS, forwarded headers and OIDC, and has no coverage at all for forwarding, retention, queue setup or shutdown ordering. + +Characterization tests to write, not to assume: + +| Behavior | Why it matters | +| --- | --- | +| Audit forwarding to the log queue, including the startup probe | Nothing covers forwarding today. | +| Failed audit import round trip through `--import-failed-audits` | Establishes the duplicate-row behavior before the key changes. | +| `EndpointDetected` domain event on audit-discovered endpoints | Pins the observable difference the enricher change introduces. | +| Retry acknowledgement dispatch and recording end to end | Pins the behavior the plan deliberately leaves alone. | +| Audit shutdown with a non-empty channel and forwarding on | Pins PR #5800's fix. | +| Audit queue provisioning through setup | Nothing covers it. | +| `/api/connection` payload with and without an audit remote | Pins what the Platform Connector plugin receives. | + +### 2. Persistence contracts, capability model, and a test persister + +- Add the audit child to the primary ingestion unit of work. +- Add the failed-import, audit count, saga history and capability contracts. Extend `IMessagesViewDataStore` rather than duplicating it. +- Add `SupportsAuditIngestion` to `ServiceControl.Persistence.PersistenceManifest` and to the two EF `persistence.manifest` files. +- Add a test persister for the primary that advertises audit support, since none exists. There is no in-memory primary persister today: the only manifests are RavenDB, EFCore.SqlServer and EFCore.PostgreSql, and `ServiceControl.Persistence.Tests.InMemory` is a test context, not a persister. Either a fake registered in the acceptance-test host or a new in-memory persister is acceptable, but the plan needs one of them by name. +- Do not move the standalone RavenDB audit implementation onto the new contracts. +- Do not add EF entities, mappings, migrations, or SQL. + +No runtime behavior change. + +### 3. Copy and adapt the audit runtime + +Merged into one pull request, because a copy nothing constructs is reviewable but not verifiable. + +- Copy the receiving, parsing, enrichment, fault handling, forwarding orchestration, metrics and readiness behavior into `ServiceControl`, changing namespaces and dependencies so the copy is owned by the primary project. +- Do not copy standalone API composition, RavenDB settings, installers, maintenance mode, or persistence loading. +- Replace endpoint-registration commands with direct monitoring unit-of-work calls, and remove `IMessageSession` and the `ICommand` overload from the copy. +- Pass the low-level dispatcher into operations that perform transport output. +- Add the per-receiver concurrency argument to `CreateTransportInfrastructure`. +- Add the OpenTelemetry references and `OtlpEndpointUrl`, and rename the meter. +- Register the copy against the test persister from step 2, and add a composition test that actually starts it. This is what makes the pull request verifiable. +- Leave the source implementation in `ServiceControl.Audit` behaviorally unchanged, and run its acceptance suite. + +### 4. Settings and the fail-fast command + +- Add the runtime settings from the table above. No installer changes. +- Add `--audit-ingestion-only` parsing, `Help.txt`, and the three fail-fast paths: unsupported persistence, RavenDB, and combination with `--error-ingestion-only`. +- Add the audit queue and audit log queue to the setup component installation context. + +### 5. Normal primary audit composition + +- Add an audit component to the primary component model. +- Register the full audit capability in the normal primary profile when the persister advertises audit support. +- Keep all audit capabilities except the receiver when normal-primary ingestion is disabled. +- Add exact hosted-service composition tests against the test persister. + +### 6. Local query composition + +- Convert `GetAuditCountsForEndpointApi` and `GetSagaByIdApi` to local-capable APIs. +- Implement the precedence, paging and counting rules against the test persister. +- Fold audit body resolution into `IBodyStorage` with the stated arbitration order and the prefixed keyspace. +- Keep existing routes and authorization policies. + +### 7. Platform connection details, saga forwarding, and licensing + +- Add the local audit platform connection details provider. +- Verify `SagaUpdatedHandler` resolves the local audit queue. +- Add the local audit source to `IAuditQuery` and `AuditThroughputCollectorHostedService`, so the local audit queues are recognized as platform endpoints and audit service metadata is populated. + +### 8. Ingestion-only composition + +- Add the dedicated host builder path. +- Register the component list from "Ingestion-only component list", and assert the exact hosted-service set. +- Add `/health` and `/health/ready`. +- Reject unsupported persistence, RavenDB, mode combination, and filesystem body storage without the shared-path assertion. + +### 9. Packaging and documentation + +- Confirm the copied runtime ships in the existing primary artifact and that the new package references pass the packaging tests. +- Keep the standalone RavenDB audit artifact and manifests unchanged. +- Document the normal, disabled-ingestion, and ingestion-only deployment modes. +- Document queue ownership, body storage requirements, health endpoints, the setting collisions, and unsupported combinations. + +### 10. Reassess reuse after delivery + +- Compare the stable primary and RavenDB implementations. +- Identify code that remains behaviorally identical and has compatible dependencies. +- Extract shared code only when the resulting boundary is simpler than maintaining the copies. +- Treat a shared hosting project as an optional follow-up, not a prerequisite for EF audit support. + +## Pull request sequence + +1. Persistence contracts, capability model, and the primary test persister. No runtime behavior change. +2. Copy and adapt the audit runtime, registered against the test persister and exercised by a composition test. +3. Settings and the fail-fast command-line mode. No installer changes. +4. Local query composition, including audit counts, saga history, and body arbitration. +5. Platform connection details, saga forwarding, and the local licensing throughput source. +6. Ingestion-only host composition and health checks. +7. Packaging, documentation, and architecture tests. + +Each pull request leaves the full RavenDB audit suite passing, states why any change to the four shared projects is safe, and avoids activating unsupported EF audit behavior. + +## Validation and acceptance criteria + +### Existing behavior + +- The standalone RavenDB audit executable has no observable behavior or configuration changes, verified by its acceptance suite on every pull request. +- RavenDB primary instances continue querying configured audit remotes. +- Existing SQL Server and PostgreSQL primary instances behave as before until their persistence advertises audit support. +- No file under `ServiceControlInstaller.Engine`, `ServiceControl.Config` or `ServiceControl.Management.PowerShell` is modified. RavenDB install, upgrade and settings-apply behavior is therefore unchanged by construction, and needs no new test to prove it. + +### Contracts + +- Audit ingestion can persist an audit message and Saga snapshot without depending on RavenDB types. +- Endpoint discovery uses the existing monitoring persistence contract and writes the same `KnownEndpoints` rows as the error path. +- Persistence capability checks produce deterministic startup validation, driven by the manifest rather than by resolving optional services. + +### Normal primary composition + +- Audit ingestion can be enabled or disabled independently of the remaining audit capabilities. +- Existing API routes resolve using local audit query contracts, under their existing policies. +- Local results can be combined with remotes. +- Precedence, paging and counting rules hold for a merged local result. A message that both failed and was audited appears once, shows as failed, and is counted once. +- `/api/connection` advertises the local audit queue, and a misdirected saga audit message is forwarded rather than failed. +- The local audit queue and audit log queue are recognized as platform endpoints by throughput collection. + +### Ingestion-only composition + +- Every registered hosted service resolves without an NServiceBus endpoint. +- `IMessageSession` is absent. +- Installer services such as `IDatabaseMigrator` and body storage provisioners are absent. +- Starting an ingestion-only worker never changes the database schema, creates queues, or provisions external storage. +- The exact hosted-service set is asserted so future registrations force an explicit scale-out decision. +- RavenDB, persistence without audit support, combination with `--error-ingestion-only`, and unasserted filesystem body storage each fail clearly at startup. +- Liveness and readiness endpoints return JSON responses. + +### Scale-out semantics + +- Concurrent workers can process the same audit queue using competing consumers. +- A redelivered audit message produces one row, not one per delivery, including when the processing-started header is absent. +- A poison audit message produces one failed-import row, not one per attempt per worker. +- Shutdown drains accepted messages before transport infrastructure is torn down. +- Forwarded messages are not duplicated during graceful shutdown. +- Audit ingestion concurrency is independent of error ingestion concurrency. + +## Constraints for the later EF implementation + +The earlier audit EF spike remains useful evidence, especially for retention, full-text search, and body storage. The later implementation should revisit it using the current primary EF architecture as the authority. + +Important retained findings are: + +- PostgreSQL can use range partitioning and partition removal for retention. +- SQL Server needs a provider-specific strategy because full-text indexes prevent equivalent partition truncation. +- Retention requires distributed locking. This belongs to the EF implementation, not to host composition. Audit retention extends `RetentionSweeper`, which already deletes bodies through `IBodyStoragePersistence` and is gated by `RunRetentionSweep`. +- Cleanup capacity must be proportional to ingestion rate. A fixed delete batch can fall behind. +- Full-text search remains provider-specific, and is governed by the existing `EnableFullTextSearchOnBodies` setting shared with error bodies. +- Body storage lifecycle must align with audit retention, and must sweep the prefixed audit keyspace as well as the error keyspace. +- Stable lock ordering and provider-specific upsert behavior are requirements, following the `INSERT ... ON CONFLICT` and `MERGE WITH (HOLDLOCK)` patterns the error batch writer already uses. +- `--audit-ingestion-only` must never apply EF migrations, modify the database schema, create queues, or provision body storage. It assumes the deployment or update setup path has already prepared all required infrastructure. + +Unlike the spike, the implementation uses one primary EF model and migration stream, one shared known-endpoint table, and no audit-to-primary endpoint reconciliation process. + +## Open items + +These do not block the groundwork, but they need answers before or during the EF implementation. + +1. Should `SagaUpdatedHandler` forward a misdirected saga audit message to the local audit queue, or hand the snapshot straight to the audit unit of work? Forwarding preserves today's behavior and its warning. Direct handling removes a broker round trip. +2. Should the audit path raise the `EndpointDetected` domain event that the command path raises today? The characterization test in step 1 answers what is currently observable. +3. Do ingestion-only workers verify the forwarding address at startup? N workers restarting under the watchdog put N probe messages in the log queue. +4. What is the retention lock scope? One lock for the whole sweeper, or separate error and audit locks so a slow audit sweep does not block error retention. +5. What is the default audit retention period when `ServiceControl/AuditRetentionPeriod` is null? The audit instance defaults to 30 days while SCMU and the Dockerfile default to 7. +6. Should a later release combine the two ingestion-only modes, or replace both flags with a single `--ingestion-only` governed by `IngestErrorMessages` and `IngestAuditMessages`? +7. Handed to the SCMU and PowerShell workstream for EF storage types, not answered here: how are the audit settings surfaced for a Windows primary, and what fixes the `RemoveIfRetired` trap described in "Installer: out of scope, with one handoff"? Nothing in this plan is blocked on it, because SCMU cannot create an EF instance today. + +## Reference pull requests + +- [Audit EF spike: #5318](https://github.com/Particular/ServiceControl/pull/5318) +- [Scale out error ingestion 1/3, dispatcher ownership: #5800](https://github.com/Particular/ServiceControl/pull/5800) +- [Scale out error ingestion 2/3, ingestion-only host: #5801](https://github.com/Particular/ServiceControl/pull/5801) +- [Scale out error ingestion 3/3, health endpoints: #5803](https://github.com/Particular/ServiceControl/pull/5803) From d2c6525b13b171e6cdb91a021e84145c29aa3a18 Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 19:22:57 +1000 Subject: [PATCH 02/21] Add primary audit persistence contracts and capability model First step of hosting audit ingestion in the primary instance. Contracts only, no runtime behavior change: nothing resolves or calls any of this yet, and no shipped persister advertises audit support. - IIngestionUnitOfWork gains a nullable Audit child alongside Monitoring and Recoverability, so a batch can record audit messages and saga snapshots in the same transaction as known endpoints. - New query contracts for the two entry points that are remote-only today: IAuditCountsDataStore and ISagaHistoryDataStore. - IFailedAuditImportDataStore mirrors IFailedErrorImportDataStore, and FailedAuditImport.DeriveKey gives failed audit imports the same deterministic key with native-id fallback, so competing consumers do not write a row per delivery attempt. - PersistenceManifest gains SupportsAuditIngestion. Absent means false, so RavenDB and every legacy manifest are unaffected. Both EF manifests declare it explicitly false until EF audit persistence lands. - IMessagesViewDataStore documents the precedence, paging and counting rules a persister must honour once one local result set unions failed and audited messages. - ServiceControl.Persistence.Tests.AuditCapable is a test-only persister that advertises audit support and delegates everything else to a real persister, so later pull requests can compose and start an audit-capable primary host before any shipped persister is one. --- .../persistence.manifest | 1 + .../persistence.manifest | 1 + .../UnitOfWork/EFIngestionUnitOfWork.cs | 3 + .../.editorconfig | 5 ++ .../AuditCapableIngestionUnitOfWork.cs | 54 +++++++++++++ .../AuditCapableIngestionUnitOfWorkFactory.cs | 18 +++++ .../AuditCapableTestPersistence.cs | 51 ++++++++++++ ...uditCapableTestPersistenceConfiguration.cs | 43 ++++++++++ .../InMemoryAuditCountsDataStore.cs | 19 +++++ .../InMemoryAuditStore.cs | 81 +++++++++++++++++++ .../InMemoryFailedAuditImportDataStore.cs | 33 ++++++++ .../InMemorySagaHistoryDataStore.cs | 20 +++++ ...trol.Persistence.Tests.AuditCapable.csproj | 16 ++++ .../persistence.manifest | 9 +++ .../FailedAuditImport.cs | 32 ++++++++ .../IAuditCountsDataStore.cs | 13 +++ .../IFailedAuditImportDataStore.cs | 14 ++++ .../IMessagesViewDataStore.cs | 13 +++ .../ISagaHistoryDataStore.cs | 19 +++++ .../PersistenceManifest.cs | 7 ++ .../ServiceControl.Persistence.csproj | 1 + .../UnitOfWork/FallbackIngestionUnitOfWork.cs | 1 + .../UnitOfWork/IAuditIngestionUnitOfWork.cs | 15 ++++ .../UnitOfWork/IIngestionUnitOfWork.cs | 10 ++- .../UnitOfWork/IngestionUnitOfWorkBase.cs | 1 + ...PersistenceManifestAuditCapabilityTests.cs | 57 +++++++++++++ src/ServiceControl.slnx | 1 + 27 files changed, 536 insertions(+), 2 deletions(-) create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest create mode 100644 src/ServiceControl.Persistence/FailedAuditImport.cs create mode 100644 src/ServiceControl.Persistence/IAuditCountsDataStore.cs create mode 100644 src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs create mode 100644 src/ServiceControl.Persistence/ISagaHistoryDataStore.cs create mode 100644 src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs create mode 100644 src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest b/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest index 433c31a5ae..49b6aa4414 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest @@ -4,6 +4,7 @@ "Description": "PostgreSQL ServiceControl persister", "AssemblyName": "ServiceControl.Persistence.EFCore.PostgreSql", "TypeName": "ServiceControl.Persistence.EFCore.PostgreSql.PostgreSqlPersistenceConfiguration, ServiceControl.Persistence.EFCore.PostgreSql", + "SupportsAuditIngestion": false, "Settings": [ { "Name": "ServiceControl/Database/ConnectionString", diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest b/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest index 9c1f04c024..598ba9168c 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest @@ -4,6 +4,7 @@ "Description": "SQL Server ServiceControl persister", "AssemblyName": "ServiceControl.Persistence.EFCore.SqlServer", "TypeName": "ServiceControl.Persistence.EFCore.SqlServer.SqlServerPersistenceConfiguration, ServiceControl.Persistence.EFCore.SqlServer", + "SupportsAuditIngestion": false, "Settings": [ { "Name": "ServiceControl/Database/ConnectionString", diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs index 186c1e8790..cbb6a9173e 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs @@ -33,6 +33,9 @@ public EFIngestionUnitOfWork(IAsyncDisposable scope, ServiceControlDbContext dbC public IRecoverabilityIngestionUnitOfWork Recoverability { get; } + // Stays null until the EF audit persistence lands and the manifest advertises SupportsAuditIngestion. + public IAuditIngestionUnitOfWork? Audit => null; + internal void Record(RecordedFailedProcessingAttempt attempt) => failedProcessingAttempts.Enqueue(attempt); internal void RecordBodyWrite(Task bodyWrite) => bodyWrites.Enqueue(bodyWrite); diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig b/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig new file mode 100644 index 0000000000..ca5ad8bd2e --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig @@ -0,0 +1,5 @@ +[*.cs] + +# Justification: Test project +dotnet_diagnostic.CA2007.severity = none +dotnet_diagnostic.PS0018.severity = none diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs new file mode 100644 index 0000000000..313954e8aa --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs @@ -0,0 +1,54 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Collections.Concurrent; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.MessageAuditing; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.SagaAudit; + + // Recording is buffered and only visible after Complete, so tests see the same all or nothing + // batch behaviour a real persister gives them. + class AuditCapableIngestionUnitOfWork(IIngestionUnitOfWork inner, InMemoryAuditStore auditStore) + : IIngestionUnitOfWork, IAuditIngestionUnitOfWork + { + readonly ConcurrentQueue processedMessages = new(); + readonly ConcurrentQueue sagaSnapshots = new(); + + public IMonitoringIngestionUnitOfWork? Monitoring => inner.Monitoring; + + public IRecoverabilityIngestionUnitOfWork? Recoverability => inner.Recoverability; + + public IAuditIngestionUnitOfWork? Audit => this; + + public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) + { + processedMessages.Enqueue(processedMessage); + return Task.CompletedTask; + } + + public Task RecordSagaSnapshot(SagaSnapshot sagaSnapshot, CancellationToken cancellationToken = default) + { + sagaSnapshots.Enqueue(sagaSnapshot); + return Task.CompletedTask; + } + + public async Task Complete(CancellationToken cancellationToken = default) + { + await inner.Complete(cancellationToken); + + while (processedMessages.TryDequeue(out var processedMessage)) + { + auditStore.Record(processedMessage); + } + + while (sagaSnapshots.TryDequeue(out var sagaSnapshot)) + { + auditStore.Record(sagaSnapshot); + } + } + + public ValueTask DisposeAsync() => inner.DisposeAsync(); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs new file mode 100644 index 0000000000..73eaa94249 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs @@ -0,0 +1,18 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.UnitOfWork; + + class AuditCapableIngestionUnitOfWorkFactory(IIngestionUnitOfWorkFactory inner, InMemoryAuditStore auditStore) : IIngestionUnitOfWorkFactory + { + public async ValueTask StartNew(CancellationToken cancellationToken = default) => + new AuditCapableIngestionUnitOfWork(await inner.StartNew(cancellationToken), auditStore); + + public bool CanIngestMore() => inner.CanIngestMore(); + + // Whatever the persister this delegates to says: the audit rows it adds are appended, never + // merged, so they do not change how concurrent batches settle. + public bool SupportsConcurrentBatches => inner.SupportsConcurrentBatches; + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs new file mode 100644 index 0000000000..15947be4d5 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs @@ -0,0 +1,51 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Linq; + using Microsoft.Extensions.DependencyInjection; + using ServiceControl.Persistence.UnitOfWork; + + class AuditCapableTestPersistence(IPersistence inner) : IPersistence + { + public void AddPersistence(IServiceCollection services) + { + inner.AddPersistence(services); + + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + + DecorateUnitOfWorkFactory(services); + } + + public void AddInstaller(IServiceCollection services) => inner.AddInstaller(services); + + static void DecorateUnitOfWorkFactory(IServiceCollection services) + { + var descriptor = services.LastOrDefault(d => d.ServiceType == typeof(IIngestionUnitOfWorkFactory)) + ?? throw new InvalidOperationException("The delegated persister registered no ingestion unit of work factory."); + + services.Remove(descriptor); + + services.AddSingleton(provider => new AuditCapableIngestionUnitOfWorkFactory( + ResolveInnerFactory(provider, descriptor), + provider.GetRequiredService())); + } + + static IIngestionUnitOfWorkFactory ResolveInnerFactory(IServiceProvider provider, ServiceDescriptor descriptor) + { + if (descriptor.ImplementationInstance is IIngestionUnitOfWorkFactory instance) + { + return instance; + } + + if (descriptor.ImplementationFactory is not null) + { + return (IIngestionUnitOfWorkFactory)descriptor.ImplementationFactory(provider); + } + + return (IIngestionUnitOfWorkFactory)ActivatorUtilities.CreateInstance(provider, descriptor.ImplementationType!); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs new file mode 100644 index 0000000000..bb206ee97f --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs @@ -0,0 +1,43 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using ServiceControl.Configuration; + + /// + /// A persister that exists only so tests can compose a primary host whose manifest advertises audit + /// support, before any shipped persister does. Everything except the audit contracts is delegated to + /// the persister named by the setting, so the error side of + /// the host is the real thing. Delete it once a shipped manifest sets SupportsAuditIngestion. + /// + public class AuditCapableTestPersistenceConfiguration : IPersistenceConfiguration + { + public const string InnerPersistenceTypeSetting = "AuditCapableTestInnerPersistenceType"; + + public bool SupportsMaintenanceMode => CreateInnerConfiguration(PrimaryRootNamespace).SupportsMaintenanceMode; + + public PersistenceSettings CreateSettings(SettingsRootNamespace settingsRootNamespace) => + CreateInnerConfiguration(settingsRootNamespace).CreateSettings(settingsRootNamespace); + + public IPersistence Create(PersistenceSettings settings) => + new AuditCapableTestPersistence(CreateInnerConfiguration(PrimaryRootNamespace).Create(settings)); + + static IPersistenceConfiguration CreateInnerConfiguration(SettingsRootNamespace settingsRootNamespace) + { + var persistenceType = SettingsReader.Read(settingsRootNamespace, InnerPersistenceTypeSetting) + ?? throw new InvalidOperationException( + $"The audit capable test persister needs the {settingsRootNamespace}/{InnerPersistenceTypeSetting} setting to name the persister it delegates to."); + + var manifest = PersistenceManifestLibrary.Find(persistenceType) + ?? throw new InvalidOperationException($"No persistence manifest matches '{persistenceType}'."); + + var typeName = manifest.TypeName + ?? throw new InvalidOperationException($"The persistence manifest for '{persistenceType}' names no configuration type."); + + var configurationType = Type.GetType(typeName, throwOnError: true)!; + + return (IPersistenceConfiguration)Activator.CreateInstance(configurationType)!; + } + + static readonly SettingsRootNamespace PrimaryRootNamespace = new("ServiceControl"); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs new file mode 100644 index 0000000000..a980054907 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs @@ -0,0 +1,19 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System.Collections.Generic; + using System.Linq; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Api.Contracts; + using ServiceControl.Persistence.Infrastructure; + + class InMemoryAuditCountsDataStore(InMemoryAuditStore auditStore) : IAuditCountsDataStore + { + public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) + { + IList counts = [.. auditStore.CountsFor(endpointName).Select(count => new AuditCount { UtcDate = count.UtcDate, Count = count.Count })]; + + return Task.FromResult(new QueryResult>(counts, new QueryStatsInfo(DataVersion.None, counts.Count))); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs new file mode 100644 index 0000000000..cdf1500c01 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs @@ -0,0 +1,81 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Collections.Concurrent; + using System.Collections.Generic; + using System.Linq; + using NServiceBus; + using ServiceControl.MessageAuditing; + using ServiceControl.Operations; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + public class InMemoryAuditStore + { + readonly ConcurrentQueue processedMessages = new(); + readonly ConcurrentQueue sagaSnapshots = new(); + readonly ConcurrentDictionary failedImports = new(); + + public void Record(ProcessedMessage processedMessage) => processedMessages.Enqueue(processedMessage); + + public void Record(SagaSnapshot sagaSnapshot) => sagaSnapshots.Enqueue(sagaSnapshot); + + public void Record(FailedAuditImport failedImport) => failedImports[failedImport.Id] = failedImport; + + public IReadOnlyList ProcessedMessages => [.. processedMessages]; + + public IReadOnlyList FailedImports => [.. failedImports.Values]; + + public bool RemoveFailedImport(string id) => failedImports.TryRemove(id, out _); + + public IReadOnlyList<(DateTime UtcDate, long Count)> CountsFor(string endpointName) => + [ + .. processedMessages + .Where(message => EndpointOf(message) == endpointName) + .GroupBy(message => message.ProcessedAt.Date) + .Select(group => (UtcDate: group.Key, Count: (long)group.Count())) + .OrderBy(count => count.UtcDate) + ]; + + public (SagaHistory? History, int TotalChanges) HistoryFor(Guid sagaId, PagingInfo pagingInfo) + { + var snapshots = sagaSnapshots.Where(snapshot => snapshot.SagaId == sagaId).ToList(); + + if (snapshots.Count == 0) + { + return (null, 0); + } + + var history = new SagaHistory + { + Id = sagaId, + SagaId = sagaId, + SagaType = snapshots[0].SagaType, + Changes = + [ + .. snapshots + .OrderByDescending(snapshot => snapshot.FinishTime) + .Skip(pagingInfo.Offset) + .Take(pagingInfo.Next) + .Select(ToStateChange) + ] + }; + + return (history, snapshots.Count); + } + + static SagaStateChange ToStateChange(SagaSnapshot snapshot) => new() + { + StartTime = snapshot.StartTime, + FinishTime = snapshot.FinishTime, + Status = snapshot.Status, + StateAfterChange = snapshot.StateAfterChange, + InitiatingMessage = snapshot.InitiatingMessage, + OutgoingMessages = snapshot.OutgoingMessages, + Endpoint = snapshot.Endpoint + }; + + static string? EndpointOf(ProcessedMessage message) => + message.Headers.GetValueOrDefault(Headers.ProcessingEndpoint); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs new file mode 100644 index 0000000000..f7436437c4 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs @@ -0,0 +1,33 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations; + + class InMemoryFailedAuditImportDataStore(InMemoryAuditStore auditStore) : IFailedAuditImportDataStore + { + public Task StoreFailedAuditImport(FailedAuditImport failure, CancellationToken cancellationToken = default) + { + auditStore.Record(failure); + return Task.CompletedTask; + } + + public async Task ProcessFailedAuditImports(Func processMessage, CancellationToken cancellationToken = default) + { + foreach (var failedImport in auditStore.FailedImports) + { + if (failedImport.Message is null) + { + continue; + } + + await processMessage(failedImport.Message, cancellationToken); + auditStore.RemoveFailedImport(failedImport.Id); + } + } + + public Task QueryContainsFailedImports(CancellationToken cancellationToken = default) => + Task.FromResult(auditStore.FailedImports.Count > 0); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs new file mode 100644 index 0000000000..8f9e4a607f --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs @@ -0,0 +1,20 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + class InMemorySagaHistoryDataStore(InMemoryAuditStore auditStore) : ISagaHistoryDataStore + { + public Task> QuerySagaHistoryById(Guid sagaId, PagingInfo pagingInfo, CancellationToken cancellationToken = default) + { + var (history, totalChanges) = auditStore.HistoryFor(sagaId, pagingInfo); + + return Task.FromResult(history is null + ? QueryResult.Empty() + : new QueryResult(history, new QueryStatsInfo(DataVersion.None, totalChanges))); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj b/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj new file mode 100644 index 0000000000..4de524ff84 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj @@ -0,0 +1,16 @@ + + + + net10.0 + enable + + + + + + + + + + + diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest b/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest new file mode 100644 index 0000000000..c214727198 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest @@ -0,0 +1,9 @@ +{ + "Name": "AuditCapableTest", + "DisplayName": "Audit capable test persister", + "Description": "Test only persister that advertises audit support and delegates everything else to a real persister", + "AssemblyName": "ServiceControl.Persistence.Tests.AuditCapable", + "TypeName": "ServiceControl.Persistence.Tests.AuditCapable.AuditCapableTestPersistenceConfiguration, ServiceControl.Persistence.Tests.AuditCapable", + "IsSupported": false, + "SupportsAuditIngestion": true +} diff --git a/src/ServiceControl.Persistence/FailedAuditImport.cs b/src/ServiceControl.Persistence/FailedAuditImport.cs new file mode 100644 index 0000000000..3a8f46ad8c --- /dev/null +++ b/src/ServiceControl.Persistence/FailedAuditImport.cs @@ -0,0 +1,32 @@ +namespace ServiceControl.Operations +{ + using System; + using System.Collections.Generic; + using ServiceControl.Persistence.Infrastructure; + + public class FailedAuditImport + { + public required string Id { get; set; } + public FailedTransportMessage? Message { get; set; } + public string? ExceptionInfo { get; set; } + + public static Guid DeriveKey(IReadOnlyDictionary headers, string nativeMessageId) + { + try + { + if (Guid.TryParse(headers.UniqueId(), out var uniqueMessageId)) + { + return uniqueMessageId; + } + } + catch (Exception) + { + // UniqueId() derives the processing endpoint, which throws when the audited message + // carries no endpoint header. Malformed messages are a leading cause of import + // failure, so fall back to a key derived from the id the transport always supplies. + } + + return DeterministicGuid.MakeId(nativeMessageId); + } + } +} diff --git a/src/ServiceControl.Persistence/IAuditCountsDataStore.cs b/src/ServiceControl.Persistence/IAuditCountsDataStore.cs new file mode 100644 index 0000000000..c5cb0090df --- /dev/null +++ b/src/ServiceControl.Persistence/IAuditCountsDataStore.cs @@ -0,0 +1,13 @@ +namespace ServiceControl.Persistence +{ + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Api.Contracts; + using ServiceControl.Persistence.Infrastructure; + + public interface IAuditCountsDataStore + { + Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs b/src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs new file mode 100644 index 0000000000..5ac5a9c60a --- /dev/null +++ b/src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs @@ -0,0 +1,14 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations; + + public interface IFailedAuditImportDataStore + { + Task StoreFailedAuditImport(FailedAuditImport failure, CancellationToken cancellationToken = default); + Task ProcessFailedAuditImports(Func processMessage, CancellationToken cancellationToken = default); + Task QueryContainsFailedImports(CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/IMessagesViewDataStore.cs b/src/ServiceControl.Persistence/IMessagesViewDataStore.cs index 66b45c9302..7ae6549af7 100644 --- a/src/ServiceControl.Persistence/IMessagesViewDataStore.cs +++ b/src/ServiceControl.Persistence/IMessagesViewDataStore.cs @@ -7,6 +7,19 @@ namespace ServiceControl.Persistence using CompositeViews.Messages; using Infrastructure; + /// + /// The single local source of message views. A persister that also holds audit data returns failed + /// and audited messages from one query, which puts three rules on the result it returns. + /// + /// Precedence. For a given {ReceivingEndpoint.Name}-{MessageId} the failed row must come + /// before the audit row, because ScatterGatherApiMessageView + /// deduplicates with TryAdd and would otherwise show a failed message as successfully processed. + /// Paging. At most PagingInfo.PageSize rows after deduplication, because the scatter gather + /// truncates and would silently drop rows if each source contributed a full page. + /// Counting. A message that both failed and was audited counts once in + /// , not once per source. + /// + /// public interface IMessagesViewDataStore { Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default); diff --git a/src/ServiceControl.Persistence/ISagaHistoryDataStore.cs b/src/ServiceControl.Persistence/ISagaHistoryDataStore.cs new file mode 100644 index 0000000000..698045f71b --- /dev/null +++ b/src/ServiceControl.Persistence/ISagaHistoryDataStore.cs @@ -0,0 +1,19 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + public interface ISagaHistoryDataStore + { + /// + /// One page of a saga's state changes, newest first. A long lived saga accumulates a snapshot + /// per state change with no natural bound, so the page is what keeps the response finite. + /// reports how many changes the saga has, not how many + /// this page carries. + /// + Task> QuerySagaHistoryById(Guid sagaId, PagingInfo pagingInfo, CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/PersistenceManifest.cs b/src/ServiceControl.Persistence/PersistenceManifest.cs index 7f5f0a1d67..72b67430e1 100644 --- a/src/ServiceControl.Persistence/PersistenceManifest.cs +++ b/src/ServiceControl.Persistence/PersistenceManifest.cs @@ -25,6 +25,13 @@ public class PersistenceManifest public bool IsSupported { get; set; } = true; + /// + /// Whether this persister can store and query audit data alongside the primary data, which is + /// what lets the primary instance ingest the audit queue itself. Absent means false, so RavenDB + /// and every legacy manifest stay audit free. + /// + public bool SupportsAuditIngestion { get; set; } + public string[] Aliases { get; set; } = []; internal bool IsMatch(string persistenceType) => diff --git a/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj b/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj index 131f2cfa0e..547655cdcb 100644 --- a/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj +++ b/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj @@ -6,6 +6,7 @@ + diff --git a/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs b/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs index 7ba823d5d6..69bf7d37f1 100644 --- a/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs @@ -22,6 +22,7 @@ public FallbackIngestionUnitOfWork(IIngestionUnitOfWork primary, IIngestionUnitO Recoverability = primary.Recoverability ?? fallback.Recoverability ?? throw new InvalidOperationException("Fallback unit of work must implement Recoverability"); + Audit = primary.Audit ?? fallback.Audit; } public override Task Complete(CancellationToken cancellationToken = default) diff --git a/src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs b/src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs new file mode 100644 index 0000000000..844315c080 --- /dev/null +++ b/src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs @@ -0,0 +1,15 @@ +namespace ServiceControl.Persistence.UnitOfWork +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.MessageAuditing; + using ServiceControl.SagaAudit; + + public interface IAuditIngestionUnitOfWork + { + Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default); + + Task RecordSagaSnapshot(SagaSnapshot sagaSnapshot, CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs b/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs index 0971aaaaab..2d6df24848 100644 --- a/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.UnitOfWork +namespace ServiceControl.Persistence.UnitOfWork { using System; using System.Threading; @@ -8,6 +8,12 @@ public interface IIngestionUnitOfWork : IAsyncDisposable { IMonitoringIngestionUnitOfWork? Monitoring { get; } IRecoverabilityIngestionUnitOfWork? Recoverability { get; } + + /// + /// Null unless the persister advertises SupportsAuditIngestion in its manifest. + /// + IAuditIngestionUnitOfWork? Audit { get; } + Task Complete(CancellationToken cancellationToken = default); } -} \ No newline at end of file +} diff --git a/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs b/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs index 32431e061c..3e1b02459b 100644 --- a/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs +++ b/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs @@ -19,6 +19,7 @@ public async ValueTask DisposeAsync() public IMonitoringIngestionUnitOfWork? Monitoring { get; protected set; } public IRecoverabilityIngestionUnitOfWork? Recoverability { get; protected set; } + public IAuditIngestionUnitOfWork? Audit { get; protected set; } public virtual Task Complete(CancellationToken cancellationToken = default) => Task.CompletedTask; } } diff --git a/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs b/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs new file mode 100644 index 0000000000..976797662c --- /dev/null +++ b/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs @@ -0,0 +1,57 @@ +namespace ServiceControl.UnitTests.Infrastructure +{ + using System; + using System.IO; + using System.Text.Json; + using NUnit.Framework; + using ServiceControl.Persistence; + + [TestFixture] + public class PersistenceManifestAuditCapabilityTests + { + [Test] + public void Absent_property_means_no_audit_support() + { + var manifest = Deserialize(""" + { + "Name": "Whatever", + "DisplayName": "Whatever", + "Description": "Whatever", + "AssemblyName": "Whatever", + "TypeName": "Whatever, Whatever" + } + """); + + Assert.That(manifest.SupportsAuditIngestion, Is.False); + } + + [TestCase("ServiceControl.Persistence.RavenDB")] + [TestCase("ServiceControl.Persistence.EFCore.SqlServer")] + [TestCase("ServiceControl.Persistence.EFCore.PostgreSql")] + public void Shipped_primary_persisters_do_not_advertise_audit_support(string projectName) + { + var manifest = ReadManifest(projectName); + + Assert.That(manifest.SupportsAuditIngestion, Is.False, + $"{projectName} advertises audit ingestion, which makes the primary host ingest the audit queue. " + + "Only flip this once that persister can store and query audit data."); + } + + [Test] + public void The_test_persister_advertises_audit_support() + { + var manifest = ReadManifest("ServiceControl.Persistence.Tests.AuditCapable"); + + Assert.That(manifest.SupportsAuditIngestion, Is.True); + } + + static PersistenceManifest ReadManifest(string projectName) => + Deserialize(File.ReadAllText(Path.Combine(SourceDirectory, projectName, "persistence.manifest"))); + + static PersistenceManifest Deserialize(string json) => + JsonSerializer.Deserialize(json) ?? throw new InvalidOperationException("The manifest is empty or invalid."); + + static string SourceDirectory => + Path.GetFullPath(Path.Combine(TestContext.CurrentContext.TestDirectory, "..", "..", "..", "..")); + } +} diff --git a/src/ServiceControl.slnx b/src/ServiceControl.slnx index 622050f094..39fb0c9392 100644 --- a/src/ServiceControl.slnx +++ b/src/ServiceControl.slnx @@ -51,6 +51,7 @@ + From 959ecd765038cd2155af2e1d7e343384cc4f6bdd Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 20:16:02 +1000 Subject: [PATCH 03/21] Copy the audit ingestion runtime into the primary instance Adapts the standalone audit instance's receiving, enrichment, fault handling, forwarding and metrics into the primary project, wired to primary persistence and primary settings. The standalone audit executable is behaviorally unchanged. What the copy does differently: - Endpoint detection writes through IMonitoringIngestionUnitOfWork instead of sending RegisterNewEndpoint through IMessageSession, the same way ErrorProcessor does. IMessageSession and the ICommand overload on the enricher context are gone, so the runtime composes in a host with no NServiceBus endpoint. - Audit messages, saga snapshots and detected endpoints commit through one primary IIngestionUnitOfWork. - Transport output is dispatched through the receiving infrastructure's own dispatcher, matching the error ingestion scale-out work. Retry acknowledgements stay transport operations, because the acknowledgement queue is named by whichever instance issued the retry. - ITransportCustomization.CreateTransportInfrastructure takes a per receiver concurrency, so audit ingestion keeps its own concurrency in a host whose shared TransportSettings carries the primary endpoint's. - The meter is renamed to Particular.ServiceControl and the primary gains the three OpenTelemetry packages plus OtlpEndpointUrl. - The copied custom checks are renamed so they do not collide with the standalone audit instance reporting into this same primary. AuditComponent registers all of it only when the configured persister advertises SupportsAuditIngestion, so no shipped configuration changes behavior. The composition tests run against the audit capable test persister and assert what the host does and does not register, including that disabling ingestion stops only the receiver. Shared projects touched: ServiceControl.Transports gains an optional argument with the existing behavior as its default, so the audit executable's receiver is created exactly as before. --- ...eControl.AcceptanceTests.PostgreSql.csproj | 1 + ...viceControl.AcceptanceTests.RavenDB.csproj | 2 + ...ceControl.AcceptanceTests.SqlServer.csproj | 1 + ...omposing_audit_ingestion_in_the_primary.cs | 156 +++++++++ .../InternalCustomCheckClassification.cs | 1 + ...IApprovals.CustomCheckDetails.approved.txt | 2 + ...rovals.PlatformSampleSettings.approved.txt | 8 + ...ChecksTest.VerifyCustomChecks.approved.txt | 2 + src/ServiceControl/Auditing/AuditComponent.cs | 58 +++ .../Auditing/AuditEnricherContext.cs | 30 ++ src/ServiceControl/Auditing/AuditIngestion.cs | 329 ++++++++++++++++++ .../Auditing/AuditIngestionCustomCheck.cs | 30 ++ .../Auditing/AuditIngestionFaultPolicy.cs | 112 ++++++ src/ServiceControl/Auditing/AuditIngestor.cs | 181 ++++++++++ .../AuditProcessingStatisticsEnricher.cs | 65 ++++ src/ServiceControl/Auditing/AuditProcessor.cs | 170 +++++++++ .../Auditing/DefaultEnrichers.cs | 51 +++ ...ectNewEndpointsFromAuditImportsEnricher.cs | 46 +++ .../DetectSuccessfulRetriesEnricher.cs | 45 +++ .../Auditing/FailedAuditImportCustomCheck.cs | 29 ++ .../Auditing/IEnrichImportedAuditMessages.cs | 7 + .../Auditing/ImportFailedAudits.cs | 71 ++++ .../Auditing/Metrics/AuditIngestionMetrics.cs | 86 +++++ .../AuditIngestionMetricsConfiguration.cs | 26 ++ .../Auditing/SagaRelationshipsEnricher.cs | 9 + .../HostApplicationBuilderExtensions.cs | 8 + .../Infrastructure/Settings/Settings.cs | 59 +++- .../Operations/ErrorIngestion.cs | 2 +- .../Infrastructure/ReturnToSenderDequeuer.cs | 2 +- .../ServiceControlMainInstance.cs | 2 + 30 files changed, 1584 insertions(+), 7 deletions(-) create mode 100644 src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs create mode 100644 src/ServiceControl/Auditing/AuditComponent.cs create mode 100644 src/ServiceControl/Auditing/AuditEnricherContext.cs create mode 100644 src/ServiceControl/Auditing/AuditIngestion.cs create mode 100644 src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs create mode 100644 src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs create mode 100644 src/ServiceControl/Auditing/AuditIngestor.cs create mode 100644 src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs create mode 100644 src/ServiceControl/Auditing/AuditProcessor.cs create mode 100644 src/ServiceControl/Auditing/DefaultEnrichers.cs create mode 100644 src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs create mode 100644 src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs create mode 100644 src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs create mode 100644 src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs create mode 100644 src/ServiceControl/Auditing/ImportFailedAudits.cs create mode 100644 src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs create mode 100644 src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs create mode 100644 src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs diff --git a/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj b/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj index 718b22bc78..564ca4217b 100644 --- a/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj +++ b/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj @@ -12,6 +12,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj index f6b9fd49e9..8745ac812f 100644 --- a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj +++ b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj @@ -42,6 +42,8 @@ + + diff --git a/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj b/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj index cbc66d5b4a..86efb52b29 100644 --- a/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj +++ b/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj @@ -12,6 +12,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs new file mode 100644 index 0000000000..57c6463b4f --- /dev/null +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs @@ -0,0 +1,156 @@ +namespace ServiceControl.AcceptanceTests.Auditing +{ + using System; + using System.IO; + using System.Linq; + using System.Runtime.Loader; + using System.Threading.Tasks; + using Microsoft.AspNetCore.Builder; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NUnit.Framework; + using Particular.ServiceControl; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.Infrastructure; + using ServiceControl.Infrastructure.WebApi; + using ServiceControl.Persistence; + using ServiceControl.Persistence.Tests.AuditCapable; + + // The inner persistence type reaches the test persister through an environment variable, which is + // process wide, so these cannot run alongside anything else that sets it. + [NonParallelizable] + class When_composing_audit_ingestion_in_the_primary : AcceptanceTest + { + [Test] + public async Task Should_host_the_audit_runtime_when_the_persister_advertises_audit_support() + { + var (app, services) = await BuildHost(auditCapable: true); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.True); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + } + } + finally + { + await app.DisposeAsync(); + } + } + + [Test] + public async Task Should_keep_every_audit_capability_but_the_receiver_when_ingestion_is_disabled() + { + var (app, services) = await BuildHost(auditCapable: true, settings => settings.IngestAuditMessages = false); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.False, + "the receiver is the only thing the setting turns off, because other processes may still be ingesting"); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + } + } + finally + { + await app.DisposeAsync(); + } + } + + [Test] + public async Task Should_host_nothing_audit_related_on_a_persister_without_audit_support() + { + var (app, services) = await BuildHost(auditCapable: false); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.False); + Assert.That(app.Services.GetService(), Is.Null); + Assert.That(app.Services.GetService(), Is.Null); + } + } + finally + { + await app.DisposeAsync(); + } + } + + // The registrations are inspected rather than resolved. A normal primary hosts an NServiceBus + // endpoint, and constructing every hosted service without starting it fails inside the + // transport's receive component. + static bool HostsAuditIngestion(IServiceCollection services) => + services.Any(descriptor => + descriptor.ServiceType == typeof(IHostedService) && descriptor.ImplementationType == typeof(AuditIngestion)); + + async Task<(WebApplication App, IServiceCollection Services)> BuildHost(bool auditCapable, Action customize = null) + { + var settings = await CreateSettings(auditCapable); + + customize?.Invoke(settings); + + var endpointConfiguration = new EndpointConfiguration(settings.InstanceName); + endpointConfiguration.AssemblyScanner().Disable = true; + + var hostBuilder = WebApplication.CreateBuilder(); + hostBuilder.AddServiceControl(settings, endpointConfiguration); + hostBuilder.AddServiceControlApi(settings.CorsSettings); + + return (hostBuilder.Build(), hostBuilder.Services); + } + + async Task CreateSettings(bool auditCapable) + { + var persistenceType = StorageConfiguration.PersistenceType; + + if (auditCapable) + { + // The test persister delegates everything but the audit contracts to the real one, so the + // host under test is the real host apart from the capability its manifest advertises. + Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, persistenceType); + persistenceType = AuditCapablePersistenceName; + } + + var settings = new Settings(TransportIntegration.TypeName, persistenceType, + CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) + { + InstanceName = $"AuditComposition.{Guid.NewGuid():n}", + TransportConnectionString = TransportIntegration.ConnectionString, + MaximumConcurrencyLevel = 2, + DisableHealthChecks = true, + AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default + }; + + await StorageConfiguration.CustomizeSettings(settings); + + return settings; + } + + [TearDown] + public void ClearInnerPersistenceType() => Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, null); + + static LoggingSettings CreateLoggingSettings() + { + var logPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()); + Directory.CreateDirectory(logPath); + return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); + } + + const string AuditCapablePersistenceName = "AuditCapableTest"; + + static readonly string InnerPersistenceTypeVariable = + AuditCapableTestPersistenceConfiguration.InnerPersistenceTypeSetting.ToUpperInvariant(); + } +} diff --git a/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs b/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs index 10a328db78..0621462030 100644 --- a/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs +++ b/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs @@ -43,6 +43,7 @@ public static class InternalCustomCheckClassification "Error Database Search Engine", // RavenDB persister "ServiceControl body storage", // EF Core persisters "Dead Letter Queue", // ASBS / IBMMQ / MSMQ + "Audit Message Ingestion (local)", // audit-capable primary // ----- Audit instance (forwarded to the primary via ReportCustomCheckResult) ----- "Audit Message Ingestion", diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt index be3c421a56..53537d2012 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt @@ -1,5 +1,7 @@ Configuration: Saga Audit Configuration Health: ServiceControl Primary Instance Health: ServiceControl Remotes +ServiceControl Health: Audit Message Ingestion (local) +ServiceControl Health: Audit Message Ingestion Process ServiceControl Health: Error Message Ingestion ServiceControl Health: Error Message Ingestion Process \ No newline at end of file diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt index abd4c98313..683c5110da 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt @@ -68,6 +68,14 @@ "ForwardErrorMessages": false, "IngestErrorMessages": true, "RunRetryProcessor": true, + "AuditQueue": "audit", + "AuditLogQueue": "audit.log", + "ForwardAuditMessages": false, + "IngestAuditMessages": true, + "AuditIngestionBatchSize": null, + "AuditIngestionMaxParallelWriters": null, + "AuditIngestionBatchTimeout": "00:00:00", + "TimeToRestartAuditIngestionAfterFailure": "00:01:00", "ErrorIngestionOnly": false, "AuditRetentionPeriod": null, "ErrorRetentionPeriod": "10.00:00:00", diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/CustomChecksTest.VerifyCustomChecks.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/CustomChecksTest.VerifyCustomChecks.approved.txt index 6c1b03a3d2..4151e92ae3 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/CustomChecksTest.VerifyCustomChecks.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/CustomChecksTest.VerifyCustomChecks.approved.txt @@ -1,5 +1,7 @@ Configuration: Saga Audit Configuration => internal Health: ServiceControl Primary Instance => internal Health: ServiceControl Remotes => internal +ServiceControl Health: Audit Message Ingestion (local) => internal +ServiceControl Health: Audit Message Ingestion Process => internal ServiceControl Health: Error Message Ingestion => internal ServiceControl Health: Error Message Ingestion Process => internal \ No newline at end of file diff --git a/src/ServiceControl/Auditing/AuditComponent.cs b/src/ServiceControl/Auditing/AuditComponent.cs new file mode 100644 index 0000000000..08944411ee --- /dev/null +++ b/src/ServiceControl/Auditing/AuditComponent.cs @@ -0,0 +1,58 @@ +namespace ServiceControl.Auditing +{ + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Particular.ServiceControl; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing.Metrics; + using ServiceControl.CustomChecks; + using ServiceControl.Persistence; + using ServiceControl.Transports; + + // Registers nothing unless the configured persister advertises audit support in its manifest, so + // hosts on a persister that cannot store audit data behave exactly as they did before. + class AuditComponent : ServiceControlComponent + { + public override void Setup(Settings settings, IComponentInstallationContext context, IHostApplicationBuilder hostBuilder) + { + if (!SupportsAuditIngestion(settings)) + { + return; + } + + context.CreateQueue(settings.AuditQueue); + + if (settings.ForwardAuditMessages && settings.AuditLogQueue != null) + { + context.CreateQueue(settings.AuditLogQueue); + } + } + + public override void Configure(Settings settings, ITransportCustomization transportCustomization, IHostApplicationBuilder hostBuilder) + { + if (!SupportsAuditIngestion(settings)) + { + return; + } + + var services = hostBuilder.Services; + + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + + services.AddCustomCheck(); + services.AddCustomCheck(); + + if (settings.IngestAuditMessages) + { + services.AddHostedService(); + } + + } + + internal static bool SupportsAuditIngestion(Settings settings) => + PersistenceManifestLibrary.Find(settings.PersistenceType)?.SupportsAuditIngestion ?? false; + } +} diff --git a/src/ServiceControl/Auditing/AuditEnricherContext.cs b/src/ServiceControl/Auditing/AuditEnricherContext.cs new file mode 100644 index 0000000000..846e0de77d --- /dev/null +++ b/src/ServiceControl/Auditing/AuditEnricherContext.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Auditing +{ + using System.Collections.Generic; + using System.Linq; + using NServiceBus.Transport; + using ServiceControl.Operations; + + // Unlike the standalone audit instance there is no ICommand overload. Endpoints detected from audit + // headers are collected here and written through the monitoring unit of work, the same way the error + // path does it, rather than sent to the primary's input queue. + class AuditEnricherContext(IReadOnlyDictionary headers, IList outgoingSends, IDictionary metadata) + { + List newEndpoints; + + public IReadOnlyDictionary Headers { get; } = headers; + + public IDictionary Metadata { get; } = metadata; + + public IEnumerable NewEndpoints => newEndpoints ?? Enumerable.Empty(); + + public void Add(EndpointDetails endpointDetails) + { + newEndpoints ??= []; + + newEndpoints.Add(endpointDetails); + } + + public void AddForSend(TransportOperation transportOperation) => outgoingSends.Add(transportOperation); + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestion.cs b/src/ServiceControl/Auditing/AuditIngestion.cs new file mode 100644 index 0000000000..0b6539178a --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestion.cs @@ -0,0 +1,329 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing.Metrics; + using ServiceControl.Infrastructure; + using ServiceControl.Infrastructure.Ingestion; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.Transports; + + class AuditIngestion : BackgroundService + { + public AuditIngestion( + Settings settings, + ITransportCustomization transportCustomization, + TransportSettings transportSettings, + IFailedAuditImportDataStore failedImportsStore, + AuditIngestionCustomCheck.State ingestionState, + AuditIngestor auditIngestor, + IIngestionUnitOfWorkFactory unitOfWorkFactory, + IHostApplicationLifetime applicationLifetime, + AuditIngestionMetrics metrics, + ILogger logger) + { + inputEndpoint = settings.AuditQueue; + this.transportCustomization = transportCustomization; + this.transportSettings = transportSettings; + this.auditIngestor = auditIngestor; + this.unitOfWorkFactory = unitOfWorkFactory; + this.settings = settings; + this.applicationLifetime = applicationLifetime; + this.metrics = metrics; + this.logger = logger; + + if (!transportSettings.MaxConcurrency.HasValue) + { + throw new ArgumentException("MaxConcurrency is not set in TransportSettings"); + } + + MaxBatchSize = settings.AuditIngestionBatchSize ?? transportSettings.MaxConcurrency.Value; + + pipeline = new IngestionPipeline( + new IngestionPipelineSettings + { + BatchSize = MaxBatchSize, + MaxWriters = IngestionSettingsReader.ResolveMaxParallelWriters(settings.AuditIngestionMaxParallelWriters, unitOfWorkFactory.SupportsConcurrentBatches, nameof(settings.AuditIngestionMaxParallelWriters), logger), + BatchTimeout = settings.AuditIngestionBatchTimeout + }, + IngestBatch, + logger); + + errorHandlingPolicy = new AuditIngestionFaultPolicy(failedImportsStore, settings.LoggingSettings, OnCriticalError, metrics, logger); + + watchdog = new Watchdog( + "audit message ingestion", + EnsureStarted, + EnsureStopped, + ingestionState.ReportError, + ingestionState.Clear, + settings.TimeToRestartAuditIngestionAfterFailure, + logger); + } + + public override async Task StartAsync(CancellationToken cancellationToken = default) + { + await watchdog.Start(() => applicationLifetime.StopApplication(), cancellationToken); + await base.StartAsync(cancellationToken); + } + + protected override Task ExecuteAsync(CancellationToken cancellationToken = default) => pipeline.Run(cancellationToken); + + async Task IngestBatch(List contexts, CancellationToken cancellationToken) + { + // Leaving the scope without completing it is what records the batch as failed + using var batchMetrics = metrics.BeginBatch(MaxBatchSize); + + await auditIngestor.Ingest(contexts, messageDispatcher, cancellationToken); + + batchMetrics.Complete(contexts.Count); + } + + public override async Task StopAsync(CancellationToken cancellationToken = default) + { + try + { + // Order matters. Receiving stops under the shutdown token rather than a cancelled + // one, so messages already being processed finish and their receives commit instead + // of being abandoned and redelivered after having been forwarded. Nothing new enters + // the pipeline after this, and the infrastructure stays up until it drains. + await EnsureReceivingStopped(cancellationToken); + pipeline.CompleteAdding(); + await base.StopAsync(cancellationToken); + } + finally + { + // Tears the infrastructure down, now that nothing is left to dispatch. + await watchdog.Stop(cancellationToken); + } + } + + Task OnCriticalError(string failure, Exception exception, CancellationToken cancellationToken) + { + logger.LogCritical(exception, "OnCriticalError. '{Failure}'", failure); + return watchdog.OnFailure(failure, cancellationToken); + } + + async Task EnsureStarted(CancellationToken cancellationToken) + { + try + { + await startStopSemaphore.WaitAsync(cancellationToken); + + var canIngest = unitOfWorkFactory.CanIngestMore(); + + logger.LogDebug("Ensure started {CanIngest}", canIngest); + + if (canIngest) + { + await SetUpAndStartInfrastructure(cancellationToken); + } + else + { + await StopAndTeardownInfrastructure(cancellationToken); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + try + { + await StopAndTeardownInfrastructure(cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception teardownException) + { + throw new AggregateException(e, teardownException); + } + + throw; + } + finally + { + startStopSemaphore.Release(); + } + } + + async Task SetUpAndStartInfrastructure(CancellationToken cancellationToken) + { + if (messageReceiver != null) + { + logger.LogDebug("Infrastructure already Started"); + return; + } + + try + { + logger.LogInformation("Starting infrastructure"); + transportInfrastructure = await transportCustomization.CreateTransportInfrastructure( + inputEndpoint, + transportSettings, + OnMessage, + errorHandlingPolicy.OnError, + OnCriticalError, + TransportTransactionMode.ReceiveOnly, + cancellationToken); + + messageReceiver = transportInfrastructure.Receivers[inputEndpoint]; + messageDispatcher = transportInfrastructure.Dispatcher; + + await auditIngestor.VerifyCanReachForwardingAddress(messageDispatcher, cancellationToken); + await messageReceiver.StartReceive(cancellationToken); + + logger.LogInformation(LogMessages.StartedInfrastructure); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Failed to start infrastructure"); + throw; + } + } + + async Task StopAndTeardownInfrastructure(CancellationToken cancellationToken) + { + if (transportInfrastructure == null) + { + logger.LogDebug("Infrastructure already Stopped"); + return; + } + + try + { + logger.LogInformation("Stopping infrastructure"); + try + { + await StopReceiving(cancellationToken); + } + finally + { + await transportInfrastructure.Shutdown(cancellationToken); + } + + messageReceiver = null; + transportInfrastructure = null; + receiveStopped = false; + + logger.LogInformation(LogMessages.StoppedInfrastructure); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Failed to stop infrastructure"); + throw; + } + } + + async Task EnsureStopped(CancellationToken cancellationToken) + { + try + { + await startStopSemaphore.WaitAsync(cancellationToken); + + // By passing a CancellationToken in the cancelled state we stop receivers ASAP and + // still correctly stop/shutdown + await StopAndTeardownInfrastructure(new CancellationToken(canceled: true)); + } + finally + { + startStopSemaphore.Release(); + } + } + + async Task EnsureReceivingStopped(CancellationToken cancellationToken) + { + await startStopSemaphore.WaitAsync(cancellationToken); + + try + { + await StopReceiving(cancellationToken); + } + finally + { + startStopSemaphore.Release(); + } + } + + // Stops the receiver on its own, leaving the infrastructure up. Idempotent because a + // shutdown stops receiving before draining and then tears down, so this runs twice. + async Task StopReceiving(CancellationToken cancellationToken) + { + if (messageReceiver == null || receiveStopped) + { + return; + } + + await messageReceiver.StopReceive(cancellationToken); + receiveStopped = true; + } + + async Task OnMessage(MessageContext messageContext, CancellationToken cancellationToken) + { + using var messageIngestionMetrics = metrics.BeginIngestion(messageContext); + + if (settings.MessageFilter != null && settings.MessageFilter(messageContext)) + { + messageIngestionMetrics.Skipped(); + return; + } + + var taskCompletionSource = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + messageContext.SetTaskCompletionSource(taskCompletionSource); + + await pipeline.Enqueue(messageContext, cancellationToken); + _ = await taskCompletionSource.Task; + + messageIngestionMetrics.Success(); + } + + TransportInfrastructure transportInfrastructure; + IMessageReceiver messageReceiver; + bool receiveStopped; + + // Left in place when the infrastructure is torn down. A shutdown drains before tearing down, + // so this is still usable there. + IMessageDispatcher messageDispatcher; + + readonly int MaxBatchSize; + readonly SemaphoreSlim startStopSemaphore = new(1); + readonly string inputEndpoint; + readonly ITransportCustomization transportCustomization; + readonly TransportSettings transportSettings; + readonly AuditIngestor auditIngestor; + readonly AuditIngestionFaultPolicy errorHandlingPolicy; + readonly IIngestionUnitOfWorkFactory unitOfWorkFactory; + readonly Settings settings; + readonly IngestionPipeline pipeline; + readonly Watchdog watchdog; + readonly IHostApplicationLifetime applicationLifetime; + readonly AuditIngestionMetrics metrics; + readonly ILogger logger; + + internal static class LogMessages + { + internal const string StartedInfrastructure = "Started infrastructure"; + internal const string StoppedInfrastructure = "Stopped infrastructure"; + } + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs b/src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs new file mode 100644 index 0000000000..e1ad653b01 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using NServiceBus.CustomChecks; + + class AuditIngestionCustomCheck(AuditIngestionCustomCheck.State criticalErrorHolder) + : CustomCheck("Audit Message Ingestion Process", "ServiceControl Health", TimeSpan.FromSeconds(5)) + { + public override Task PerformCheck(CancellationToken cancellationToken = default) + { + var failure = criticalErrorHolder.GetLastFailure(); + return failure == null + ? successResult + : Task.FromResult(CheckResult.Failed(failure)); + } + + static readonly Task successResult = Task.FromResult(CheckResult.Pass); + + public class State + { + volatile string lastFailure; + + public void Clear() => lastFailure = null; + public void ReportError(string failure) => lastFailure = failure; + public string GetLastFailure() => lastFailure; + } + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs b/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs new file mode 100644 index 0000000000..7b3e6c0cbe --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs @@ -0,0 +1,112 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Diagnostics; + using System.IO; + using System.Runtime.InteropServices; + using System.Runtime.Versioning; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Installers; + using ServiceControl.Auditing.Metrics; + using ServiceControl.Configuration; + using ServiceControl.Infrastructure; + using ServiceControl.Operations; + using ServiceControl.Persistence; + + class AuditIngestionFaultPolicy + { + public AuditIngestionFaultPolicy( + IFailedAuditImportDataStore store, + LoggingSettings loggingSettings, + Func onCriticalError, + AuditIngestionMetrics metrics, + ILogger logger) + { + this.store = store; + this.metrics = metrics; + this.logger = logger; + failureCircuitBreaker = new ImportFailureCircuitBreaker(onCriticalError); + + if (!AppEnvironment.RunningInContainer) + { + logPath = Path.Combine(loggingSettings.LogPath, "FailedImports", "Audit"); + Directory.CreateDirectory(logPath); + } + } + + public async Task OnError(ErrorContext errorContext, CancellationToken cancellationToken = default) + { + using var errorMetrics = metrics.BeginErrorHandling(errorContext); + + //Same as recoverability policy in NServiceBusFactory + if (errorContext.ImmediateProcessingFailures < 3) + { + errorMetrics.Retry(); + return ErrorHandleResult.RetryRequired; + } + + await Handle(errorContext, cancellationToken); + return ErrorHandleResult.Handled; + } + + async Task Handle(ErrorContext errorContext, CancellationToken cancellationToken) + { + var failure = new FailedAuditImport + { + Message = new FailedTransportMessage + { + Id = errorContext.MessageId, + Headers = errorContext.Headers, + Body = errorContext.Body.ToArray() + }, + ExceptionInfo = errorContext.Exception.ToFriendlyString(), + Id = FailedAuditImport.DeriveKey(errorContext.Headers, errorContext.MessageId).ToString() + }; + + try + { + await DoLogging(errorContext.Exception, failure, cancellationToken); + } + finally + { + failureCircuitBreaker.Increment(errorContext.Exception); + } + } + + async Task DoLogging(Exception exception, FailedAuditImport failure, CancellationToken cancellationToken) + { + logger.LogError(exception, "Failed importing audit message"); + + await store.StoreFailedAuditImport(failure, cancellationToken); + + if (!AppEnvironment.RunningInContainer) + { + var filePath = Path.Combine(logPath, $"FailedAuditImports_{failure.Id.Replace("/", "_")}.txt"); + await File.WriteAllTextAsync(filePath, failure.ExceptionInfo, cancellationToken); + + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + WriteToEventLog($"An audit message import has failed. A log file has been written to {filePath}"); + } + } + } + + [SupportedOSPlatform("windows")] + static void WriteToEventLog(string message) + { +#if DEBUG + EventSourceCreator.Create(); +#endif + EventLog.WriteEntry(EventSourceCreator.SourceName, message, EventLogEntryType.Error); + } + + readonly IFailedAuditImportDataStore store; + readonly AuditIngestionMetrics metrics; + readonly ImportFailureCircuitBreaker failureCircuitBreaker; + readonly string logPath; + readonly ILogger logger; + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestor.cs b/src/ServiceControl/Auditing/AuditIngestor.cs new file mode 100644 index 0000000000..7b809d6178 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestor.cs @@ -0,0 +1,181 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NServiceBus.Routing; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Infrastructure.Ingestion; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.Transports; + + class AuditIngestor + { + public AuditIngestor( + Settings settings, + IIngestionUnitOfWorkFactory unitOfWorkFactory, + IEndpointInstanceMonitoring endpointInstanceMonitoring, + ITransportCustomization transportCustomization, + ILogger logger) + { + this.settings = settings; + this.unitOfWorkFactory = unitOfWorkFactory; + this.logger = logger; + + logQueueAddress = transportCustomization.ToTransportQualifiedQueueName(settings.AuditLogQueue); + + IEnrichImportedAuditMessages[] enrichers = + [ + new AuditMessageTypeEnricher(), + new AuditEnrichWithTrackingIds(), + new AuditProcessingStatisticsEnricher(), + new DetectNewEndpointsFromAuditImportsEnricher(endpointInstanceMonitoring), + new DetectSuccessfulRetriesEnricher(), + new SagaRelationshipsEnricher() + ]; + + processor = new AuditProcessor(enrichers, logger); + } + + public async Task Ingest(List contexts, IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) + { + var stored = await Store(contexts, dispatcher, cancellationToken); + + try + { + if (settings.ForwardAuditMessages) + { + await Forward(stored, logQueueAddress, dispatcher, cancellationToken); + } + + foreach (var context in contexts) + { + context.GetTaskCompletionSource().TrySetResult(true); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogWarning(e, "Forwarding messages failed"); + + // making sure to rethrow so that all messages get marked as failed + throw; + } + } + + async Task> Store(IReadOnlyList contexts, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + // deliberately not using the using statement because we dispose async explicitly + IIngestionUnitOfWork unitOfWork = null; + try + { + unitOfWork = await unitOfWorkFactory.StartNew(cancellationToken); + + var storedContexts = await processor.Process(contexts, unitOfWork, dispatcher, cancellationToken); + + await unitOfWork.Complete(cancellationToken); + + return storedContexts; + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogWarning(e, "Bulk insertion failed"); + + // making sure to rethrow so that all messages get marked as failed + throw; + } + finally + { + if (unitOfWork != null) + { + try + { + // this can throw even though dispose is never supposed to throw + await unitOfWork.DisposeAsync(); + } + catch (Exception e) + { + logger.LogWarning(e, "Bulk insertion dispose failed"); + + // making sure to rethrow so that all messages get marked as failed + throw; + } + } + } + } + + static Task Forward(IReadOnlyCollection messageContexts, string forwardingAddress, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + var transportOperations = new List(messageContexts.Count); + MessageContext anyContext = null; + foreach (var messageContext in messageContexts) + { + if (messageContext.Extensions.TryGet("AuditType", out string auditType) + && auditType != "ProcessedMessage") + { + continue; + } + + anyContext = messageContext; + var outgoingMessage = new OutgoingMessage( + messageContext.NativeMessageId, + messageContext.Headers, + messageContext.Body); + + // Forwarded messages should last as long as possible + outgoingMessage.Headers.Remove(Headers.TimeToBeReceived); + + transportOperations.Add(new TransportOperation(outgoingMessage, new UnicastAddressTag(forwardingAddress))); + } + + return anyContext != null + ? dispatcher.Dispatch(new TransportOperations([.. transportOperations]), anyContext.TransportTransaction, cancellationToken) + : Task.CompletedTask; + } + + public async Task VerifyCanReachForwardingAddress(IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) + { + if (!settings.ForwardAuditMessages) + { + return; + } + + try + { + var transportOperations = new TransportOperations( + new TransportOperation( + new OutgoingMessage(Guid.Empty.ToString("N"), [], Array.Empty()), + new UnicastAddressTag(logQueueAddress))); + + await dispatcher.Dispatch(transportOperations, new TransportTransaction(), cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + throw new Exception($"Unable to write to forwarding queue {settings.AuditLogQueue}", e); + } + } + + readonly AuditProcessor processor; + readonly IIngestionUnitOfWorkFactory unitOfWorkFactory; + readonly Settings settings; + readonly string logQueueAddress; + readonly ILogger logger; + } +} diff --git a/src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs b/src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs new file mode 100644 index 0000000000..4d4dcb3f2a --- /dev/null +++ b/src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs @@ -0,0 +1,65 @@ +namespace ServiceControl.Auditing +{ + using System; + using NServiceBus; + + class AuditProcessingStatisticsEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var headers = context.Headers; + var metadata = context.Metadata; + var processingEnded = DateTime.MinValue; + var startTime = DateTime.MinValue; + var processingStarted = DateTime.MinValue; + + if (headers.TryGetValue(Headers.TimeSent, out var timeSentValue)) + { + startTime = DateTimeOffsetHelper.ToDateTimeOffset(timeSentValue).UtcDateTime; + metadata.Add("TimeSent", startTime); + } + + if (headers.TryGetValue(Headers.DeliverAt, out var deliverAtValue)) + { + startTime = DateTimeOffsetHelper.ToDateTimeOffset(deliverAtValue).UtcDateTime; + } + + if (headers.TryGetValue(Headers.ProcessingStarted, out var processingStartedValue)) + { + processingStarted = DateTimeOffsetHelper.ToDateTimeOffset(processingStartedValue).UtcDateTime; + } + + if (headers.TryGetValue(Headers.ProcessingEnded, out var processingEndedValue)) + { + processingEnded = DateTimeOffsetHelper.ToDateTimeOffset(processingEndedValue).UtcDateTime; + } + + var criticalTime = TimeSpan.Zero; + + if (processingEnded != DateTime.MinValue && startTime != DateTime.MinValue) + { + criticalTime = processingEnded - startTime; + } + + metadata.Add("CriticalTime", criticalTime); + + var processingTime = TimeSpan.Zero; + + if (processingEnded != DateTime.MinValue && processingStarted != DateTime.MinValue) + { + processingTime = processingEnded - processingStarted; + } + + metadata.Add("ProcessingTime", processingTime); + + var deliveryTime = TimeSpan.Zero; + + if (processingStarted != DateTime.MinValue && startTime != DateTime.MinValue) + { + deliveryTime = processingStarted - startTime; + } + + metadata.Add("DeliveryTime", deliveryTime); + } + } +} diff --git a/src/ServiceControl/Auditing/AuditProcessor.cs b/src/ServiceControl/Auditing/AuditProcessor.cs new file mode 100644 index 0000000000..b78879c5fd --- /dev/null +++ b/src/ServiceControl/Auditing/AuditProcessor.cs @@ -0,0 +1,170 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using System.Text.Json; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NServiceBus.Transport; + using ServiceControl.EndpointPlugin.Messages.SagaState; + using ServiceControl.Infrastructure; + using ServiceControl.MessageAuditing; + using ServiceControl.Infrastructure.Ingestion; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.SagaAudit; + + class AuditProcessor(IEnrichImportedAuditMessages[] enrichers, ILogger logger) + { + public async Task> Process(IReadOnlyList contexts, IIngestionUnitOfWork unitOfWork, IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) + { + var audit = unitOfWork.Audit + ?? throw new InvalidOperationException("The configured persistence does not support audit ingestion."); + var monitoring = unitOfWork.Monitoring + ?? throw new InvalidOperationException("The configured persistence does not support monitoring."); + + var storedContexts = new List(contexts.Count); + + var tasks = new List(contexts.Count); + foreach (var context in contexts) + { + tasks.Add(ProcessMessage(context, dispatcher, cancellationToken)); + } + + await Task.WhenAll(tasks); + + var knownEndpoints = new Dictionary(); + + foreach (var context in contexts) + { + // Any message context that failed during processing will have a faulted task and should be skipped + if (context.GetTaskCompletionSource().Task.IsFaulted) + { + continue; + } + + if (context.Extensions.TryGet(out ProcessedMessage processedMessage)) + { + await audit.RecordProcessedMessage(processedMessage, context.Body, cancellationToken); + } + else if (context.Extensions.TryGet(out SagaSnapshot sagaSnapshot)) + { + await audit.RecordSagaSnapshot(sagaSnapshot, cancellationToken); + } + + if (context.Extensions.TryGet>(out var newEndpoints)) + { + foreach (var endpointDetails in newEndpoints) + { + RecordKnownEndpoint(endpointDetails, knownEndpoints); + } + } + + storedContexts.Add(context); + } + + foreach (var endpoint in knownEndpoints.Values) + { + await monitoring.RecordKnownEndpoint(endpoint, cancellationToken); + } + + return storedContexts; + } + + async Task ProcessMessage(MessageContext context, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + if (context.Headers.TryGetValue(Headers.EnclosedMessageTypes, out var messageType) + && messageType == typeof(SagaUpdatedMessage).FullName) + { + ProcessSagaAuditMessage(context); + } + else + { + await ProcessAuditMessage(context, dispatcher, cancellationToken); + } + } + + void ProcessSagaAuditMessage(MessageContext context) + { + try + { + using var stream = new ReadOnlyStream(context.Body); + var message = JsonSerializer.Deserialize(stream, SagaAuditMessagesSerializationContext.Default.SagaUpdatedMessage); + + var sagaSnapshot = SagaSnapshotFactory.Create(message); + + context.Extensions.Set("AuditType", "SagaSnapshot"); + context.Extensions.Set(sagaSnapshot); + } + catch (Exception e) + { + logger.LogWarning(e, "Processing of saga audit message '{NativeMessageId}' failed", context.NativeMessageId); + + // releasing the failed message context early so that they can be retried outside the current batch + context.GetTaskCompletionSource().TrySetException(e); + } + } + + async Task ProcessAuditMessage(MessageContext context, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + if (!context.Headers.TryGetValue(Headers.MessageId, out var messageId)) + { + messageId = DeterministicGuid.MakeId(context.NativeMessageId).ToString(); + } + + try + { + var metadata = new Dictionary + { + ["MessageId"] = messageId, + ["MessageIntent"] = context.Headers.MessageIntent() + }; + + var messagesToEmit = new List(); + var enricherContext = new AuditEnricherContext(context.Headers, messagesToEmit, metadata); + + foreach (var enricher in enrichers) + { + enricher.Enrich(enricherContext); + } + + var auditMessage = new ProcessedMessage(context.Headers, new Dictionary(metadata)); + + //Do not hook into the incoming transaction + await dispatcher.Dispatch(new TransportOperations([.. messagesToEmit]), new TransportTransaction(), cancellationToken); + + context.Extensions.Set("AuditType", "ProcessedMessage"); + context.Extensions.Set(auditMessage); + context.Extensions.Set(enricherContext.NewEndpoints); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogWarning(e, "Processing of message '{MessageId}' failed", messageId); + + // releasing the failed message context early so that they can be retried outside the current batch + context.GetTaskCompletionSource().TrySetException(e); + } + } + + static void RecordKnownEndpoint(EndpointDetails observedEndpoint, Dictionary observedEndpoints) + { + var uniqueEndpointId = $"{observedEndpoint.Name}{observedEndpoint.HostId}"; + if (!observedEndpoints.ContainsKey(uniqueEndpointId)) + { + observedEndpoints.Add(uniqueEndpointId, new KnownEndpoint + { + EndpointDetails = observedEndpoint, + HostDisplayName = observedEndpoint.Host, + Monitored = false + }); + } + } + } +} diff --git a/src/ServiceControl/Auditing/DefaultEnrichers.cs b/src/ServiceControl/Auditing/DefaultEnrichers.cs new file mode 100644 index 0000000000..f14f7e4e8f --- /dev/null +++ b/src/ServiceControl/Auditing/DefaultEnrichers.cs @@ -0,0 +1,51 @@ +namespace ServiceControl.Auditing +{ + using System.Linq; + using NServiceBus; + + class AuditMessageTypeEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var isSystemMessage = false; + string messageType = null; + + if (context.Headers.ContainsKey(Headers.ControlMessageHeader)) + { + isSystemMessage = true; + } + + if (context.Headers.TryGetValue(Headers.EnclosedMessageTypes, out var enclosedMessageTypes)) + { + messageType = GetMessageType(enclosedMessageTypes); + isSystemMessage = DetectSystemMessage(messageType); + context.Metadata.Add("SearchableMessageType", messageType.Replace(".", " ").Replace("+", " ")); + } + + context.Metadata.Add("IsSystemMessage", isSystemMessage); + context.Metadata.Add("MessageType", messageType); + } + + static bool DetectSystemMessage(string messageTypeString) => + messageTypeString.Contains("NServiceBus.Scheduling.Messages.ScheduledTask"); + + static string GetMessageType(string messageTypeString) => + messageTypeString.Contains(',') ? messageTypeString.Split(',').First() : messageTypeString; + } + + class AuditEnrichWithTrackingIds : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + if (context.Headers.TryGetValue(Headers.ConversationId, out var conversationId)) + { + context.Metadata.Add("ConversationId", conversationId); + } + + if (context.Headers.TryGetValue(Headers.RelatedTo, out var relatedToId)) + { + context.Metadata.Add("RelatedToId", relatedToId); + } + } + } +} diff --git a/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs b/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs new file mode 100644 index 0000000000..cfdb2d603a --- /dev/null +++ b/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs @@ -0,0 +1,46 @@ +namespace ServiceControl.Auditing +{ + using System; + using ServiceControl.Contracts.Operations; + using ServiceControl.Operations; + using ServiceControl.Persistence; + + class DetectNewEndpointsFromAuditImportsEnricher(IEndpointInstanceMonitoring monitoring) : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var sendingEndpoint = EndpointDetailsParser.SendingEndpoint(context.Headers); + + // SendingEndpoint will be null for messages that are from v3.3.x endpoints because we don't + // have the relevant information via the headers, which were added in v4. + if (sendingEndpoint != null) + { + TryAddEndpoint(sendingEndpoint, context); + context.Metadata.Add("SendingEndpoint", sendingEndpoint); + } + + var receivingEndpoint = EndpointDetailsParser.ReceivingEndpoint(context.Headers); + // The ReceivingEndpoint will be null for messages from v3.3.x endpoints that were successfully + // processed because we dont have the information from the relevant headers. + if (receivingEndpoint != null) + { + TryAddEndpoint(receivingEndpoint, context); + context.Metadata.Add("ReceivingEndpoint", receivingEndpoint); + } + } + + void TryAddEndpoint(EndpointDetails endpointDetails, AuditEnricherContext context) + { + // for backwards compat with version before 4_5 we might not have a hostid + if (endpointDetails.HostId == Guid.Empty) + { + return; + } + + if (monitoring.IsNewInstance(endpointDetails)) + { + context.Add(endpointDetails); + } + } + } +} diff --git a/src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs b/src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs new file mode 100644 index 0000000000..9d953cf0b9 --- /dev/null +++ b/src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs @@ -0,0 +1,45 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using NServiceBus; + using NServiceBus.Routing; + using NServiceBus.Transport; + + class DetectSuccessfulRetriesEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var headers = context.Headers; + var isOldRetry = headers.TryGetValue("ServiceControl.RetryId", out _); + var isNewRetry = headers.TryGetValue("ServiceControl.Retry.UniqueMessageId", out var newRetryMessageId); + var isAckHandled = headers.ContainsKey("ServiceControl.Retry.AcknowledgementSent"); + var hasAckQueue = headers.TryGetValue("ServiceControl.Retry.AcknowledgementQueue", out var ackQueue); + + var hasBeenRetried = isOldRetry || isNewRetry; + + context.Metadata.Add("IsRetried", hasBeenRetried); + + if (!hasBeenRetried || isAckHandled) + { + //The message has not been sent for retry from ServiceControl or the endpoint indicated that is already has sent a retry acknowledgement to the + //ServiceControl main instance. Nothing to do. + return; + } + + if (hasAckQueue && isNewRetry) + { + // The acknowledgement queue is named by whichever instance issued the retry, so this stays a + // transport operation rather than a direct write to the local recoverability unit of work. + // In a combined host it simply comes back in through local error ingestion. + var ackMessage = new OutgoingMessage(Guid.NewGuid().ToString(), new Dictionary + { + ["ServiceControl.Retry.Successful"] = DateTimeOffsetHelper.ToWireFormattedString(DateTimeOffset.UtcNow), + ["ServiceControl.Retry.UniqueMessageId"] = newRetryMessageId + }, Array.Empty()); + var ackOperation = new TransportOperation(ackMessage, new UnicastAddressTag(ackQueue)); + context.AddForSend(ackOperation); + } + } + } +} diff --git a/src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs b/src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs new file mode 100644 index 0000000000..ffc5c9600f --- /dev/null +++ b/src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs @@ -0,0 +1,29 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus.CustomChecks; + using ServiceControl.Persistence; + + // Deliberately named and categorised differently from the standalone audit instance's check, which + // reports into this same primary through ReportCustomChecksTo and would otherwise collide. + class FailedAuditImportCustomCheck(IFailedAuditImportDataStore store, ILogger logger) + : CustomCheck("Audit Message Ingestion (local)", "ServiceControl Health", TimeSpan.FromHours(1)) + { + public override async Task PerformCheck(CancellationToken cancellationToken = default) + { + if (await store.QueryContainsFailedImports(cancellationToken)) + { + logger.LogWarning(message); + return CheckResult.Failed(message); + } + + return CheckResult.Pass; + } + + const string message = @"One or more audit messages have failed to import properly into ServiceControl and have been stored in the ServiceControl database. +The import of these messages could have failed for a number of reasons and ServiceControl is not able to automatically reimport them. For guidance on how to resolve this see https://docs.particular.net/servicecontrol/import-failed-messages"; + } +} diff --git a/src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs b/src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs new file mode 100644 index 0000000000..1468902d89 --- /dev/null +++ b/src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs @@ -0,0 +1,7 @@ +namespace ServiceControl.Auditing +{ + interface IEnrichImportedAuditMessages + { + void Enrich(AuditEnricherContext context); + } +} diff --git a/src/ServiceControl/Auditing/ImportFailedAudits.cs b/src/ServiceControl/Auditing/ImportFailedAudits.cs new file mode 100644 index 0000000000..63365c3833 --- /dev/null +++ b/src/ServiceControl/Auditing/ImportFailedAudits.cs @@ -0,0 +1,71 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus.Extensibility; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Infrastructure.Ingestion; + using ServiceControl.Operations; + using ServiceControl.Persistence; + + class ImportFailedAudits( + IFailedAuditImportDataStore failedAuditStore, + AuditIngestor auditIngestor, + Lazy messageDispatcher, + Settings settings, + ILogger logger) + { + public async Task Run(CancellationToken cancellationToken = default) + { + await auditIngestor.VerifyCanReachForwardingAddress(messageDispatcher.Value, cancellationToken); + + var succeeded = 0; + var failed = 0; + + await failedAuditStore.ProcessFailedAuditImports(async (transportMessage, token) => + { + try + { + var messageContext = new MessageContext( + transportMessage.Id, + transportMessage.Headers, + transportMessage.Body, + EmptyTransaction, + settings.AuditQueue, + EmptyContextBag); + var taskCompletionSource = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + messageContext.SetTaskCompletionSource(taskCompletionSource); + + await auditIngestor.Ingest([messageContext], messageDispatcher.Value, token); + + await taskCompletionSource.Task; + + succeeded++; + logger.LogDebug("Successfully re-imported failed audit message {MessageId}", transportMessage.Id); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Error while attempting to re-import failed audit message {MessageId}", transportMessage.Id); + failed++; + } + }, cancellationToken); + + logger.LogInformation("Done re-importing failed audits. Successfully re-imported {SuccessCount} messages. Failed re-importing {FailureCount} messages", succeeded, failed); + + if (failed > 0) + { + logger.LogWarning("{FailureCount} messages could not be re-imported. This could indicate a problem with the data. Contact Particular support if you need help with recovering the messages", failed); + } + } + + static readonly TransportTransaction EmptyTransaction = new(); + static readonly ContextBag EmptyContextBag = new(); + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs new file mode 100644 index 0000000000..bc6796d3d2 --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs @@ -0,0 +1,86 @@ +namespace ServiceControl.Auditing.Metrics; + +using System.Collections.Generic; +using System.Diagnostics; +using System.Diagnostics.Metrics; +using System.Threading; +using NServiceBus; +using NServiceBus.Transport; +using ServiceControl.EndpointPlugin.Messages.SagaState; +using ServiceControl.Infrastructure.Ingestion.Metrics; +using ServiceControl.Operations.Metrics; + +/// +/// Mirrors for the audit queue. Same meter and the same primitives, so +/// the two ingestions report the same shapes and one exporter carries both; only the instrument +/// prefix and the tags differ, because what distinguishes an audit message is its kind rather than +/// whether it resolved a retry. +/// +public class AuditIngestionMetrics +{ + public static readonly string BatchDurationInstrumentName = $"{InstrumentPrefix}.batch_duration_seconds"; + public static readonly string MessageDurationInstrumentName = $"{InstrumentPrefix}.message_duration_seconds"; + public static readonly string StorageDurationInstrumentName = $"{InstrumentPrefix}.storage_duration_seconds"; + + public AuditIngestionMetrics(IMeterFactory meterFactory) + { + var meter = meterFactory.Create(IngestionMetrics.MeterName, MeterVersion); + + batchDuration = meter.CreateHistogram(BatchDurationInstrumentName, unit: "seconds", description: "Audit message batch processing duration in seconds"); + ingestionDuration = meter.CreateHistogram(MessageDurationInstrumentName, unit: "seconds", description: "Audit message processing duration in seconds"); + storageDuration = meter.CreateHistogram(StorageDurationInstrumentName, unit: "seconds", description: "Audit ingestion batch storage write duration in seconds"); + consecutiveBatchFailureGauge = meter.CreateObservableGauge($"{InstrumentPrefix}.consecutive_batch_failures_total", () => Volatile.Read(ref consecutiveBatchFailures), description: "Consecutive audit ingestion batch failures"); + failureCounter = meter.CreateCounter($"{InstrumentPrefix}.failures_total", description: "Audit ingestion failure count"); + } + + public MessageMetrics BeginIngestion(MessageContext messageContext) => new(GetMessageTags(messageContext.Headers), ingestionDuration); + + public FailureMetrics BeginErrorHandling(ErrorContext errorContext) => new(GetMessageTags(errorContext.Headers), failureCounter); + + public BatchMetrics BeginBatch(int maxBatchSize) => new(maxBatchSize, batchDuration, RecordBatchOutcome); + + public DurationScope MeasureStorageWrite() => new(storageDuration); + + public static TagList GetMessageTags(Dictionary headers) + { + var tags = new TagList(); + + if (headers.TryGetValue(Headers.EnclosedMessageTypes, out var messageType)) + { + tags.Add("message.category", messageType == SagaUpdateMessageType ? "saga-update" : "audit-message"); + } + else + { + tags.Add("message.category", "control-message"); + } + + return tags; + } + + void RecordBatchOutcome(bool success) + { + if (success) + { + Volatile.Write(ref consecutiveBatchFailures, 0); + } + else + { + Interlocked.Increment(ref consecutiveBatchFailures); + } + } + + long consecutiveBatchFailures; + + readonly Histogram batchDuration; +#pragma warning disable IDE0052 + readonly ObservableGauge consecutiveBatchFailureGauge; +#pragma warning restore IDE0052 + readonly Histogram ingestionDuration; + readonly Histogram storageDuration; + readonly Counter failureCounter; + + const string MeterVersion = "0.1.0"; + const string InstrumentPrefix = "sc.audit.ingestion"; + + static readonly string SagaUpdateMessageType = typeof(SagaUpdatedMessage).FullName; +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs new file mode 100644 index 0000000000..b3fecda7d3 --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs @@ -0,0 +1,26 @@ +namespace ServiceControl.Auditing.Metrics; + +using OpenTelemetry.Metrics; +using ServiceControl.Infrastructure.Ingestion.Metrics; + +public static class AuditIngestionMetricsConfiguration +{ + // The meter is already added by the error ingestion configuration, which shares it. Only the + // audit instruments' bucket boundaries need declaring. + public static void AddAuditIngestionMetrics(this MeterProviderBuilder builder) + { + foreach (var instrumentName in DurationInstruments) + { + builder.AddView( + instrumentName, + new ExplicitBucketHistogramConfiguration { Boundaries = IngestionDurations.BucketBoundaries }); + } + } + + static readonly string[] DurationInstruments = + [ + AuditIngestionMetrics.MessageDurationInstrumentName, + AuditIngestionMetrics.BatchDurationInstrumentName, + AuditIngestionMetrics.StorageDurationInstrumentName + ]; +} diff --git a/src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs b/src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs new file mode 100644 index 0000000000..a029bac8f1 --- /dev/null +++ b/src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs @@ -0,0 +1,9 @@ +namespace ServiceControl.Auditing +{ + using ServiceControl.SagaAudit; + + class SagaRelationshipsEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) => InvokedSagasParser.Parse(context.Headers, context.Metadata); + } +} diff --git a/src/ServiceControl/HostApplicationBuilderExtensions.cs b/src/ServiceControl/HostApplicationBuilderExtensions.cs index 9b682fe1a6..a66d5d2e0a 100644 --- a/src/ServiceControl/HostApplicationBuilderExtensions.cs +++ b/src/ServiceControl/HostApplicationBuilderExtensions.cs @@ -30,6 +30,7 @@ using NServiceBus.Hosting; using NServiceBus.Transport; using OpenTelemetry.Metrics; + using global::ServiceControl.Auditing.Metrics; using OpenTelemetry.Resources; using Particular.LicensingComponent; using ServiceBus.Management.Infrastructure; @@ -175,6 +176,13 @@ public static void AddTelemetry(this IHostApplicationBuilder hostBuilder, Settin .WithMetrics(metrics => { metrics.AddIngestionMetrics(); + + // Audit ingestion shares the meter, so only its instruments' views are added, + // and they are added whether or not this host ingests audit: a view for an + // instrument nobody records is inert, and making it conditional would tie the + // exporter's shape to which component happened to be registered. + metrics.AddAuditIngestionMetrics(); + metrics.AddAspNetCoreInstrumentation(); metrics.AddHttpClientInstrumentation(); metrics.AddRuntimeInstrumentation(); diff --git a/src/ServiceControl/Infrastructure/Settings/Settings.cs b/src/ServiceControl/Infrastructure/Settings/Settings.cs index af52f4bf88..bb7a104998 100644 --- a/src/ServiceControl/Infrastructure/Settings/Settings.cs +++ b/src/ServiceControl/Infrastructure/Settings/Settings.cs @@ -44,6 +44,7 @@ public Settings( InstanceName = SettingsReader.Read(SettingsRootNamespace, "InstanceName", InstanceName); LoadErrorIngestionSettings(); + LoadAuditIngestionSettings(); TransportConnectionString = GetConnectionString(); TransportType = transportType ?? SettingsReader.Read(SettingsRootNamespace, "TransportType"); @@ -84,6 +85,10 @@ public Settings( ErrorIngestionBatchSize = IngestionSettingsReader.ReadBatchSize(SettingsRootNamespace, nameof(ErrorIngestionBatchSize), ValidateConfiguration); ErrorIngestionMaxParallelWriters = IngestionSettingsReader.ReadMaxParallelWriters(SettingsRootNamespace, nameof(ErrorIngestionMaxParallelWriters), ValidateConfiguration); ErrorIngestionBatchTimeout = IngestionSettingsReader.ReadBatchTimeout(SettingsRootNamespace, nameof(ErrorIngestionBatchTimeout), ValidateConfiguration); + TimeToRestartAuditIngestionAfterFailure = GetTimeToRestartIngestionAfterFailure("TimeToRestartAuditIngestionAfterFailure"); + AuditIngestionBatchSize = IngestionSettingsReader.ReadBatchSize(SettingsRootNamespace, nameof(AuditIngestionBatchSize), ValidateConfiguration); + AuditIngestionMaxParallelWriters = IngestionSettingsReader.ReadMaxParallelWriters(SettingsRootNamespace, nameof(AuditIngestionMaxParallelWriters), ValidateConfiguration); + AuditIngestionBatchTimeout = IngestionSettingsReader.ReadBatchTimeout(SettingsRootNamespace, nameof(AuditIngestionBatchTimeout), ValidateConfiguration); DisableExternalIntegrationsPublishing = SettingsReader.Read(SettingsRootNamespace, "DisableExternalIntegrationsPublishing", false); TrackInstancesInitialValue = SettingsReader.Read(SettingsRootNamespace, "TrackInstancesInitialValue", true); ShutdownTimeout = SettingsReader.Read(SettingsRootNamespace, "ShutdownTimeout", ShutdownTimeout); @@ -193,6 +198,23 @@ public string InstanceId public bool IngestErrorMessages { get; set; } = true; public bool RunRetryProcessor { get; set; } = true; + public string AuditQueue { get; set; } + public string AuditLogQueue { get; set; } + + public bool ForwardAuditMessages { get; set; } + + /// + /// Whether the normal primary host runs the audit receiver. Only has an effect where the + /// persister advertises audit support; always on under audit ingestion only. + /// + public bool IngestAuditMessages { get; set; } = true; + + public int? AuditIngestionBatchSize { get; set; } + public int? AuditIngestionMaxParallelWriters { get; set; } + public TimeSpan AuditIngestionBatchTimeout { get; set; } + + public TimeSpan TimeToRestartAuditIngestionAfterFailure { get; set; } + // Set by the --error-ingestion-only command, never read from configuration. public bool ErrorIngestionOnly { get; set; } @@ -390,10 +412,12 @@ TimeSpan GetHeartbeatGracePeriod() } } - TimeSpan GetTimeToRestartErrorIngestionAfterFailure() + TimeSpan GetTimeToRestartErrorIngestionAfterFailure() => GetTimeToRestartIngestionAfterFailure("TimeToRestartErrorIngestionAfterFailure"); + + TimeSpan GetTimeToRestartIngestionAfterFailure(string settingName) { string message; - var valueRead = SettingsReader.Read(SettingsRootNamespace, "TimeToRestartErrorIngestionAfterFailure"); + var valueRead = SettingsReader.Read(SettingsRootNamespace, settingName); if (valueRead == null) { return TimeSpan.FromSeconds(60); @@ -403,21 +427,21 @@ TimeSpan GetTimeToRestartErrorIngestionAfterFailure() { if (ValidateConfiguration && result < TimeSpan.FromSeconds(5)) { - message = "TimeToRestartErrorIngestionAfterFailure setting is invalid, value should be minimum 5 seconds."; + message = $"{settingName} setting is invalid, value should be minimum 5 seconds."; InternalLogger.Fatal(message); throw new Exception(message); } if (ValidateConfiguration && result > TimeSpan.FromHours(1)) { - message = "TimeToRestartErrorIngestionAfterFailure setting is invalid, value should be maximum 1 hour."; + message = $"{settingName} setting is invalid, value should be maximum 1 hour."; InternalLogger.Fatal(message); throw new Exception(message); } } else { - message = "TimeToRestartErrorIngestionAfterFailure setting is invalid, please make sure it is a TimeSpan."; + message = $"{settingName} setting is invalid, please make sure it is a TimeSpan."; InternalLogger.Fatal(message); throw new Exception(message); } @@ -453,6 +477,31 @@ static string Subscope(string address) return $"{queue}.log@{machine}"; } + void LoadAuditIngestionSettings() + { + // Key names are deliberately the ones the standalone audit instance reads, so an operator + // configures a combined primary exactly as they configure an audit instance today. + var serviceBusRootNamespace = new SettingsRootNamespace("ServiceBus"); + AuditQueue = SettingsReader.Read(serviceBusRootNamespace, "AuditQueue", "audit"); + + if (string.IsNullOrEmpty(AuditQueue)) + { + throw new Exception("ServiceBus/AuditQueue value is required to start the instance"); + } + + IngestAuditMessages = SettingsReader.Read(SettingsRootNamespace, "IngestAuditMessages", true); + + AuditLogQueue = SettingsReader.Read(serviceBusRootNamespace, "AuditLogQueue", null); + + if (AuditLogQueue == null) + { + logger.LogInformation("No settings found for audit log queue to import, default name will be used"); + AuditLogQueue = Subscope(AuditQueue); + } + + ForwardAuditMessages = SettingsReader.Read(SettingsRootNamespace, "ForwardAuditMessages", false); + } + void LoadErrorIngestionSettings() { var serviceBusRootNamespace = new SettingsRootNamespace("ServiceBus"); diff --git a/src/ServiceControl/Operations/ErrorIngestion.cs b/src/ServiceControl/Operations/ErrorIngestion.cs index d23623c6d4..73263b0cad 100644 --- a/src/ServiceControl/Operations/ErrorIngestion.cs +++ b/src/ServiceControl/Operations/ErrorIngestion.cs @@ -170,7 +170,7 @@ async Task SetUpAndStartInfrastructure(CancellationToken cancellationToken) errorHandlingPolicy.OnError, OnCriticalError, TransportTransactionMode.ReceiveOnly, - cancellationToken + cancellationToken: cancellationToken ); messageReceiver = transportInfrastructure.Receivers[errorQueue]; diff --git a/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs b/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs index 2c4d975e6a..02ec07895b 100644 --- a/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs +++ b/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs @@ -40,7 +40,7 @@ ILogger logger public async Task StartAsync(CancellationToken cancellationToken = default) { - transportInfrastructure = await transportCustomization.CreateTransportInfrastructure(InputAddress, transportSettings, Handle, faultManager.OnError, (_, __, ___) => Task.CompletedTask, TransportTransactionMode.SendsAtomicWithReceive, cancellationToken); + transportInfrastructure = await transportCustomization.CreateTransportInfrastructure(InputAddress, transportSettings, Handle, faultManager.OnError, (_, __, ___) => Task.CompletedTask, TransportTransactionMode.SendsAtomicWithReceive, cancellationToken: cancellationToken); messageReceiver = transportInfrastructure.Receivers[InputAddress]; messageDispatcher = transportInfrastructure.Dispatcher; diff --git a/src/ServiceControl/ServiceControlMainInstance.cs b/src/ServiceControl/ServiceControlMainInstance.cs index 10a7ba1a0b..bded219889 100644 --- a/src/ServiceControl/ServiceControlMainInstance.cs +++ b/src/ServiceControl/ServiceControlMainInstance.cs @@ -1,5 +1,6 @@ namespace Particular.ServiceControl { + using global::ServiceControl.Auditing; using global::ServiceControl.CustomChecks; using global::ServiceControl.EventLog; using global::ServiceControl.ExternalIntegrations; @@ -14,6 +15,7 @@ static class ServiceControlMainInstance new ExternalIntegrationsComponent(), new RecoverabilityComponent(), new HeartbeatMonitoringComponent(), + new AuditComponent(), new CustomChecksComponent(), new LicensingComponent() }; From c33fea60bd2d75f687c932831f4c3a763faf0c4b Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 20:19:16 +1000 Subject: [PATCH 04/21] Add the audit ingestion only command and its startup checks Adds --audit-ingestion-only to the command line and Help.txt, with the three checks the plan calls for. The host itself is not composed yet, so the command fails after its checks pass; no shipped persister advertises audit support, so in practice it fails on the storage check. - Storage that does not advertise SupportsAuditIngestion is rejected by name, driven by the manifest rather than by resolving optional services. RavenDB falls out of this without a special case. - The two ingestion only modes cannot be combined. Each queue gets its own worker pool so they can be scaled independently. - File system body storage is rejected unless ServiceControl/MessageBody/FileSystem/PathIsShared asserts the path is a shared mount. Nothing in the file system settings distinguishes a shared mount from a node local directory, so the operator has to say so. The same check now guards --error-ingestion-only, which PR #5801 documented as a known gap. No installer changes. --- .../Hosting/AuditIngestionOnlyCommandTests.cs | 81 +++++++++++++++++++ .../Commands/AuditIngestionOnlyCommand.cs | 26 ++++++ .../Commands/ErrorIngestionOnlyCommand.cs | 1 + .../Hosting/Commands/IngestionOnlyGuards.cs | 65 +++++++++++++++ src/ServiceControl/Hosting/Help.txt | 16 +++- src/ServiceControl/Hosting/HostArguments.cs | 27 +++++-- 6 files changed, 209 insertions(+), 7 deletions(-) create mode 100644 src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs create mode 100644 src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs create mode 100644 src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs diff --git a/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs new file mode 100644 index 0000000000..32d77627b1 --- /dev/null +++ b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs @@ -0,0 +1,81 @@ +namespace ServiceControl.UnitTests.Hosting +{ + using System; + using System.Threading.Tasks; + using NUnit.Framework; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Hosting.Commands; + + [TestFixture] + public class AuditIngestionOnlyCommandTests + { + [TestCase("RavenDB")] + [TestCase("SQLServer")] + [TestCase("PostgreSQL")] + public void Should_refuse_to_start_against_storage_without_audit_support(string persistenceType) + { + var settings = CreateSettings(persistenceType); + + var exception = Assert.ThrowsAsync(() => + new AuditIngestionOnlyCommand().Execute(new HostArguments([]), settings)); + + Assert.That(exception.Message, Does.Contain("supports audit ingestion")); + } + + [Test] + public void Should_refuse_to_combine_the_two_ingestion_only_modes() + { + var exception = Assert.Throws(() => + IngestionOnlyGuards.EnsureModesAreNotCombined(errorIngestionOnly: true, auditIngestionOnly: true)); + + Assert.That(exception.Message, Does.Contain("cannot be combined")); + } + + [Test] + public void Should_refuse_file_system_body_storage_that_is_not_asserted_as_shared() + { + using var _ = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem"); + + var exception = Assert.Throws(() => + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only")); + + Assert.That(exception.Message, Does.Contain(IngestionOnlyGuards.SharedBodyStoragePathKey)); + } + + [Test] + public void Should_accept_file_system_body_storage_asserted_as_shared() + { + using var storageType = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem"); + using var pathIsShared = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_FILESYSTEM_PATHISSHARED", "true"); + + Assert.DoesNotThrow(() => + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only")); + } + + [Test] + public void Should_ignore_body_storage_that_every_host_can_already_read() + { + using var _ = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "AzureBlob"); + + Assert.DoesNotThrow(() => + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only")); + } + + static Settings CreateSettings(string persistenceType) => + new("LearningTransport", persistenceType, forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)); + + sealed class EnvironmentVariableScope : IDisposable + { + readonly string name; + + public EnvironmentVariableScope(string name, string value) + { + this.name = name; + Environment.SetEnvironmentVariable(name, value); + } + + public void Dispose() => Environment.SetEnvironmentVariable(name, null); + } + } +} diff --git a/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs new file mode 100644 index 0000000000..abe0f17002 --- /dev/null +++ b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs @@ -0,0 +1,26 @@ +namespace ServiceControl.Hosting.Commands +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + + /// + /// Runs a host that does nothing but drain the audit queue into the shared database, so several + /// processes can ingest against one database. The host itself lands with the ingestion only + /// composition; until then the command exists so the startup checks and the help text are in place, + /// and it always fails because no shipped persister advertises audit support yet. + /// + class AuditIngestionOnlyCommand : AbstractCommand + { + public override Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) + { + IngestionOnlyGuards.EnsureStorageSupportsAuditIngestion(settings); + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only"); + + throw new Exception( + "--audit-ingestion-only is not available yet. The storage advertises audit support, but the audit ingestion only host has not been composed."); + } + } +} diff --git a/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs index 1fc1c68e4e..02f3ad3824 100644 --- a/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs +++ b/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs @@ -29,6 +29,7 @@ class ErrorIngestionOnlyCommand : AbstractCommand public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) { EnsureStorageCanScaleOut(settings); + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--error-ingestion-only"); var app = BuildHost(settings); diff --git a/src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs b/src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs new file mode 100644 index 0000000000..f235f5a083 --- /dev/null +++ b/src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs @@ -0,0 +1,65 @@ +namespace ServiceControl.Hosting.Commands +{ + using System; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Configuration; + using ServiceControl.Persistence; + + /// + /// The startup checks both ingestion only modes share. They are deliberately driven by the + /// persistence manifest and by settings, never by resolving optional services or by catching a + /// startup failure, so an unsupported deployment fails with a message that names what to change. + /// + static class IngestionOnlyGuards + { + public const string SharedBodyStoragePathKey = "MessageBody/FileSystem/PathIsShared"; + + const string BodyStorageTypeKey = "MessageBody/StorageType"; + const string FileSystemBodyStorage = "FileSystem"; + + public static void EnsureStorageSupportsAuditIngestion(Settings settings) + { + var manifest = PersistenceManifestLibrary.Find(settings.PersistenceType); + + if (manifest?.SupportsAuditIngestion != true) + { + throw new Exception( + $"--audit-ingestion-only requires storage that supports audit ingestion, but this instance is configured to use '{settings.PersistenceType}'. " + + "Hosting audit ingestion in the primary instance is not supported for this storage type."); + } + } + + public static void EnsureModesAreNotCombined(bool errorIngestionOnly, bool auditIngestionOnly) + { + if (errorIngestionOnly && auditIngestionOnly) + { + throw new Exception( + "--error-ingestion-only and --audit-ingestion-only cannot be combined. Each queue gets its own worker pool so the two can be scaled independently, " + + "so run one process per mode."); + } + } + + /// + /// Nothing in the file system body storage settings distinguishes a shared mount from a node + /// local directory, so an ingestion only worker requires the operator to assert it explicitly. + /// Without it, bodies written by a worker are unreadable by every other host. + /// + public static void EnsureBodyStorageIsReadableByEveryHost(string mode) + { + var storageType = SettingsReader.Read(Settings.SettingsRootNamespace, BodyStorageTypeKey); + + if (!string.Equals(storageType, FileSystemBodyStorage, StringComparison.OrdinalIgnoreCase)) + { + return; + } + + if (!SettingsReader.Read(Settings.SettingsRootNamespace, SharedBodyStoragePathKey, false)) + { + throw new Exception( + $"{mode} is configured for file system body storage, which every host must be able to read. " + + $"Set {Settings.SettingsRootNamespace}/{SharedBodyStoragePathKey} to true to assert that the configured path is a shared mount, " + + "or use blob or S3 body storage."); + } + } + } +} diff --git a/src/ServiceControl/Hosting/Help.txt b/src/ServiceControl/Hosting/Help.txt index 4925b8c494..4c9530d956 100644 --- a/src/ServiceControl/Hosting/Help.txt +++ b/src/ServiceControl/Hosting/Help.txt @@ -19,8 +19,20 @@ share the ingestion load. Requires SQL Server or PostgreSQL storage, and require has already been provisioned by a normal instance. Exactly one normal instance must still be running: it owns the retry pipeline, the retention sweep, integration event dispatch and heartbeat monitoring. -Message bodies must be stored somewhere every host can read, so this mode should not be combined with -file system body storage unless the path is a shared mount. +Message bodies must be stored somewhere every host can read. With file system body storage the mode +refuses to start unless ServiceControl/MessageBody/FileSystem/PathIsShared is set to true, which +asserts that the configured path is a shared mount. + +AUDIT INGESTION ONLY + + ServiceControl.exe --audit-ingestion-only + +Runs a host that only drains the audit queue into the configured database, so several processes can +share the ingestion load. Requires storage that supports audit ingestion, and requires that the +database has already been provisioned by a normal instance. It cannot be combined with +--error-ingestion-only: each queue gets its own worker pool so the two can be scaled independently. + +The same body storage rule applies as for error ingestion only. SERVICE INSTALL AND UNINSTALL AND CONFIGURATION OPTIONS diff --git a/src/ServiceControl/Hosting/HostArguments.cs b/src/ServiceControl/Hosting/HostArguments.cs index b260543662..6c4a0a9955 100644 --- a/src/ServiceControl/Hosting/HostArguments.cs +++ b/src/ServiceControl/Hosting/HostArguments.cs @@ -1,4 +1,4 @@ -namespace Particular.ServiceControl.Hosting +namespace Particular.ServiceControl.Hosting { using System; using System.IO; @@ -11,6 +11,9 @@ class HostArguments { public HostArguments(string[] args) { + var errorIngestionOnly = false; + var auditIngestionOnly = false; + if (SettingsReader.Read(Settings.SettingsRootNamespace, "MaintenanceMode")) { args = [.. args, "-m"]; @@ -53,12 +56,17 @@ public HostArguments(string[] args) } }; - var errorIngestionOnlyOptions = new OptionSet + var ingestionOnlyOptions = new OptionSet { { "error-ingestion-only", "Run only error ingestion, for scaling out ingestion across several processes", - s => Command = typeof(ErrorIngestionOnlyCommand) + s => errorIngestionOnly = true + }, + { + "audit-ingestion-only", + "Run only audit ingestion, for scaling out ingestion across several processes", + s => auditIngestionOnly = true } }; @@ -85,10 +93,19 @@ public HostArguments(string[] args) return; } - errorIngestionOnlyOptions.Parse(args); + ingestionOnlyOptions.Parse(args); + + IngestionOnlyGuards.EnsureModesAreNotCombined(errorIngestionOnly, auditIngestionOnly); + + if (errorIngestionOnly) + { + Command = typeof(ErrorIngestionOnlyCommand); + return; + } - if (Command == typeof(ErrorIngestionOnlyCommand)) + if (auditIngestionOnly) { + Command = typeof(AuditIngestionOnlyCommand); return; } From d3cc6277b036ca262d320202e0cd0b76816b29ad Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 20:26:48 +1000 Subject: [PATCH 05/21] Serve audit counts and saga history from local storage The two audit entry points that were remote only become ordinary scatter gather APIs over the new contracts, so a primary holding audit data answers them from its own storage and still merges in any remotes. Routes and authorization policies are unchanged: audit counts stay on error:messages:view and sagas/{id} stays on error:sagas:view, which is what a primary with an audit remote already serves them under. - GetAuditCountsForEndpointApi drops the IMessagesViewDataStore it never used, along with the comment saying it would never be implemented here. - GetSagaByIdApi stops deriving from ScatterGatherRemoteOnly, which nothing else used, so that base and its NoOpStore are gone. - A persister without audit support falls back to an empty local source, registered with TryAdd after the persister, so those hosts behave exactly as before and the APIs still resolve. LocalMessagesView.Merge puts the precedence, paging and counting rules in one place for any persister that returns failed and audited messages from one query: the failed row wins for a message that both failed and was audited, the total counts it once, and the local result is already one page. MessageViewComparer moves to the persistence project so a local merge can order rows the same way the scatter gather does. IBodyStorage documents the arbitration order for a store holding both kinds of body, and AuditBodyKeyspace prefixes audit bodies so they do not collide with failed message bodies in the same store. The audit capable test persister implements all of it, so the rules have a running implementation before the EF work starts. --- ...omposing_audit_ingestion_in_the_primary.cs | 6 ++ .../AuditCapableBodyStorage.cs | 34 +++++++ .../AuditCapableIngestionUnitOfWork.cs | 6 +- .../AuditCapableMessagesViewDataStore.cs | 58 +++++++++++ .../AuditCapableTestPersistence.cs | 30 +++--- .../InMemoryAuditStore.cs | 42 ++++++-- .../IBodyStorage.cs | 10 ++ .../LocalMessagesView.cs | 60 ++++++++++++ .../MessageViewComparer.cs | 16 +--- .../Hosting/AuditIngestionOnlyCommandTests.cs | 2 + .../ScatterGather/IncompleteResultsTests.cs | 5 +- .../ScatterGather/LocalMessagesViewTests.cs | 95 +++++++++++++++++++ .../GetAuditCountsForEndpointApi.cs | 14 +-- .../Messages/ScatterGatherApi.cs | 8 +- .../Persistence/EmptyAuditDataStores.cs | 33 +++++++ .../PersistenceServiceCollectionExtensions.cs | 6 ++ .../SagaAudit/GetSagaByIdApi.cs | 14 ++- 17 files changed, 393 insertions(+), 46 deletions(-) create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs create mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs create mode 100644 src/ServiceControl.Persistence/LocalMessagesView.cs rename src/{ServiceControl/CompositeViews/Messages => ServiceControl.Persistence}/MessageViewComparer.cs (85%) create mode 100644 src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs create mode 100644 src/ServiceControl/Persistence/EmptyAuditDataStores.cs diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs index 57c6463b4f..0d82e03075 100644 --- a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs @@ -14,10 +14,12 @@ namespace ServiceControl.AcceptanceTests.Auditing using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing; + using ServiceControl.CompositeViews.MessageCounting; using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.WebApi; using ServiceControl.Persistence; using ServiceControl.Persistence.Tests.AuditCapable; + using ServiceControl.SagaAudit; // The inner persistence type reaches the test persister through an environment variable, which is // process wide, so these cannot run alongside anything else that sets it. @@ -80,6 +82,10 @@ public async Task Should_host_nothing_audit_related_on_a_persister_without_audit Assert.That(HostsAuditIngestion(services), Is.False); Assert.That(app.Services.GetService(), Is.Null); Assert.That(app.Services.GetService(), Is.Null); + + Assert.That(app.Services.GetService(), Is.Not.Null, + "the audit routes stay served from the configured remotes, so the APIs must still resolve"); + Assert.That(app.Services.GetService(), Is.Not.Null); } } finally diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs new file mode 100644 index 0000000000..eb8ed8efe3 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs @@ -0,0 +1,34 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System.IO; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations.BodyStorage; + using ServiceControl.Persistence.Infrastructure; + + // The third step of the arbitration order IBodyStorage states: a failed message body wins, and the + // audit copy answers only when no failed message holds one. + class AuditCapableBodyStorage(IBodyStorage inner, InMemoryAuditStore auditStore) : IBodyStorage + { + public async Task TryFetch(string bodyId, CancellationToken cancellationToken = default) + { + var failedMessageBody = await inner.TryFetch(bodyId, cancellationToken); + + if (failedMessageBody.State != MessageBodyState.NotFound) + { + return failedMessageBody; + } + + var body = auditStore.BodyFor(bodyId); + + if (body == null) + { + return MessageBodyResult.NotFound(); + } + + return body.Length == 0 + ? MessageBodyResult.Empty() + : MessageBodyResult.Available(new MessageBodyStreamContent(new MemoryStream(body, writable: false), "application/json", body.Length, DataVersion.None)); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs index 313954e8aa..f10e9f8032 100644 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs @@ -13,7 +13,7 @@ namespace ServiceControl.Persistence.Tests.AuditCapable class AuditCapableIngestionUnitOfWork(IIngestionUnitOfWork inner, InMemoryAuditStore auditStore) : IIngestionUnitOfWork, IAuditIngestionUnitOfWork { - readonly ConcurrentQueue processedMessages = new(); + readonly ConcurrentQueue<(ProcessedMessage Message, byte[] Body)> processedMessages = new(); readonly ConcurrentQueue sagaSnapshots = new(); public IMonitoringIngestionUnitOfWork? Monitoring => inner.Monitoring; @@ -24,7 +24,7 @@ class AuditCapableIngestionUnitOfWork(IIngestionUnitOfWork inner, InMemoryAuditS public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) { - processedMessages.Enqueue(processedMessage); + processedMessages.Enqueue((processedMessage, body.ToArray())); return Task.CompletedTask; } @@ -40,7 +40,7 @@ public async Task Complete(CancellationToken cancellationToken = default) while (processedMessages.TryDequeue(out var processedMessage)) { - auditStore.Record(processedMessage); + auditStore.Record(processedMessage.Message, processedMessage.Body); } while (sagaSnapshots.TryDequeue(out var sagaSnapshot)) diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs new file mode 100644 index 0000000000..2b67f891c3 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs @@ -0,0 +1,58 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.Persistence.Infrastructure; + + // One local result set holding both failed and audited messages, merged under the precedence, + // paging and counting rules IMessagesViewDataStore states. + class AuditCapableMessagesViewDataStore(IMessagesViewDataStore inner, InMemoryAuditStore auditStore) : IMessagesViewDataStore + { + public async Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessages(pagingInfo, sortInfo, includeSystemMessages, timeSentRange, cancellationToken), + Audited(includeSystemMessages, timeSentRange), pagingInfo, sortInfo); + + public async Task>> GetAllMessagesForEndpoint(string endpointName, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessagesForEndpoint(endpointName, pagingInfo, sortInfo, includeSystemMessages, timeSentRange, cancellationToken), + Audited(includeSystemMessages, timeSentRange).Where(message => message.ReceivingEndpoint?.Name == endpointName), pagingInfo, sortInfo); + + public async Task>> GetAllMessagesByConversation(string conversationId, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessagesByConversation(conversationId, pagingInfo, sortInfo, includeSystemMessages, cancellationToken), + Audited(includeSystemMessages).Where(message => message.ConversationId == conversationId), pagingInfo, sortInfo); + + public async Task>> GetAllMessagesForSearch(string searchTerms, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessagesForSearch(searchTerms, pagingInfo, sortInfo, timeSentRange, cancellationToken), + Audited(includeSystemMessages: true, timeSentRange).Where(message => Matches(message, searchTerms)), pagingInfo, sortInfo); + + public async Task>> SearchEndpointMessages(string endpointName, string searchKeyword, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.SearchEndpointMessages(endpointName, searchKeyword, pagingInfo, sortInfo, timeSentRange, cancellationToken), + Audited(includeSystemMessages: true, timeSentRange) + .Where(message => message.ReceivingEndpoint?.Name == endpointName && Matches(message, searchKeyword)), pagingInfo, sortInfo); + + IEnumerable Audited(bool includeSystemMessages, DateTimeRange? timeSentRange = null) => + auditStore.MessageViews + .Where(message => includeSystemMessages || !message.IsSystemMessage) + .Where(message => InRange(message, timeSentRange)); + + static bool InRange(MessagesView message, DateTimeRange? timeSentRange) => + timeSentRange == null + || (message.TimeSent >= timeSentRange.From && message.TimeSent <= timeSentRange.To); + + static bool Matches(MessagesView message, string searchTerms) => + searchTerms == null + || (message.MessageType?.Contains(searchTerms, StringComparison.OrdinalIgnoreCase) ?? false) + || (message.MessageId?.Contains(searchTerms, StringComparison.OrdinalIgnoreCase) ?? false); + + static QueryResult> Merge(QueryResult> failed, IEnumerable audited, PagingInfo pagingInfo, SortInfo sortInfo) => + LocalMessagesView.Merge( + [.. failed.Results ?? []], + [.. audited], + pagingInfo, + MessageViewComparer.FromSortInfo(sortInfo), + failed.QueryStats.Version); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs index 15947be4d5..238ca820b4 100644 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Persistence.Tests.AuditCapable using System; using System.Linq; using Microsoft.Extensions.DependencyInjection; + using ServiceControl.Operations.BodyStorage; using ServiceControl.Persistence.UnitOfWork; class AuditCapableTestPersistence(IPersistence inner) : IPersistence @@ -16,36 +17,43 @@ public void AddPersistence(IServiceCollection services) services.AddSingleton(); services.AddSingleton(); - DecorateUnitOfWorkFactory(services); + Decorate(services, (inner, provider) => + new AuditCapableIngestionUnitOfWorkFactory(inner, provider.GetRequiredService())); + Decorate(services, (inner, provider) => + new AuditCapableMessagesViewDataStore(inner, provider.GetRequiredService())); + Decorate(services, (inner, provider) => + new AuditCapableBodyStorage(inner, provider.GetRequiredService())); } public void AddInstaller(IServiceCollection services) => inner.AddInstaller(services); - static void DecorateUnitOfWorkFactory(IServiceCollection services) + static void Decorate(IServiceCollection services, Func decorate) + where TService : class { - var descriptor = services.LastOrDefault(d => d.ServiceType == typeof(IIngestionUnitOfWorkFactory)) - ?? throw new InvalidOperationException("The delegated persister registered no ingestion unit of work factory."); + var descriptor = services.LastOrDefault(d => d.ServiceType == typeof(TService)) + ?? throw new InvalidOperationException($"The delegated persister registered no {typeof(TService).Name}."); services.Remove(descriptor); - services.AddSingleton(provider => new AuditCapableIngestionUnitOfWorkFactory( - ResolveInnerFactory(provider, descriptor), - provider.GetRequiredService())); + services.Add(new ServiceDescriptor(typeof(TService), + provider => decorate(ResolveInner(provider, descriptor), provider), + descriptor.Lifetime)); } - static IIngestionUnitOfWorkFactory ResolveInnerFactory(IServiceProvider provider, ServiceDescriptor descriptor) + static TService ResolveInner(IServiceProvider provider, ServiceDescriptor descriptor) + where TService : class { - if (descriptor.ImplementationInstance is IIngestionUnitOfWorkFactory instance) + if (descriptor.ImplementationInstance is TService instance) { return instance; } if (descriptor.ImplementationFactory is not null) { - return (IIngestionUnitOfWorkFactory)descriptor.ImplementationFactory(provider); + return (TService)descriptor.ImplementationFactory(provider); } - return (IIngestionUnitOfWorkFactory)ActivatorUtilities.CreateInstance(provider, descriptor.ImplementationType!); + return (TService)ActivatorUtilities.CreateInstance(provider, descriptor.ImplementationType!); } } } diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs index cdf1500c01..2dbad834b2 100644 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs @@ -5,6 +5,7 @@ namespace ServiceControl.Persistence.Tests.AuditCapable using System.Collections.Generic; using System.Linq; using NServiceBus; + using ServiceControl.CompositeViews.Messages; using ServiceControl.MessageAuditing; using ServiceControl.Operations; using ServiceControl.Persistence.Infrastructure; @@ -12,27 +13,31 @@ namespace ServiceControl.Persistence.Tests.AuditCapable public class InMemoryAuditStore { - readonly ConcurrentQueue processedMessages = new(); + readonly ConcurrentQueue processedMessages = new(); readonly ConcurrentQueue sagaSnapshots = new(); readonly ConcurrentDictionary failedImports = new(); - public void Record(ProcessedMessage processedMessage) => processedMessages.Enqueue(processedMessage); + public void Record(ProcessedMessage processedMessage, byte[] body) => + processedMessages.Enqueue(new AuditRecord(processedMessage, body)); public void Record(SagaSnapshot sagaSnapshot) => sagaSnapshots.Enqueue(sagaSnapshot); public void Record(FailedAuditImport failedImport) => failedImports[failedImport.Id] = failedImport; - public IReadOnlyList ProcessedMessages => [.. processedMessages]; - public IReadOnlyList FailedImports => [.. failedImports.Values]; public bool RemoveFailedImport(string id) => failedImports.TryRemove(id, out _); + public IReadOnlyList MessageViews => [.. processedMessages.Select(record => ToMessagesView(record.Message))]; + + public byte[]? BodyFor(string uniqueMessageId) => + processedMessages.FirstOrDefault(record => record.Message.UniqueMessageId == uniqueMessageId)?.Body; + public IReadOnlyList<(DateTime UtcDate, long Count)> CountsFor(string endpointName) => [ .. processedMessages - .Where(message => EndpointOf(message) == endpointName) - .GroupBy(message => message.ProcessedAt.Date) + .Where(record => EndpointOf(record.Message) == endpointName) + .GroupBy(record => record.Message.ProcessedAt.Date) .Select(group => (UtcDate: group.Key, Count: (long)group.Count())) .OrderBy(count => count.UtcDate) ]; @@ -64,6 +69,29 @@ .. snapshots return (history, snapshots.Count); } + static MessagesView ToMessagesView(ProcessedMessage message) => new() + { + Id = message.Id, + MessageId = Metadata(message, "MessageId"), + MessageType = Metadata(message, "MessageType"), + SendingEndpoint = Metadata(message, "SendingEndpoint"), + ReceivingEndpoint = Metadata(message, "ReceivingEndpoint"), + TimeSent = Metadata(message, "TimeSent"), + ProcessedAt = message.ProcessedAt, + CriticalTime = Metadata(message, "CriticalTime"), + ProcessingTime = Metadata(message, "ProcessingTime"), + DeliveryTime = Metadata(message, "DeliveryTime"), + IsSystemMessage = Metadata(message, "IsSystemMessage"), + ConversationId = Metadata(message, "ConversationId"), + Headers = [.. message.Headers.Select(header => new KeyValuePair(header.Key, header.Value))], + Status = MessageStatus.Successful, + MessageIntent = Metadata(message, "MessageIntent"), + BodyUrl = $"/messages/{message.UniqueMessageId}/body" + }; + + static T? Metadata(ProcessedMessage message, string key) => + message.MessageMetadata.TryGetValue(key, out var value) && value is T typed ? typed : default; + static SagaStateChange ToStateChange(SagaSnapshot snapshot) => new() { StartTime = snapshot.StartTime, @@ -77,5 +105,7 @@ .. snapshots static string? EndpointOf(ProcessedMessage message) => message.Headers.GetValueOrDefault(Headers.ProcessingEndpoint); + + sealed record AuditRecord(ProcessedMessage Message, byte[] Body); } } diff --git a/src/ServiceControl.Persistence/IBodyStorage.cs b/src/ServiceControl.Persistence/IBodyStorage.cs index 187c3e8dc9..10724b0c40 100644 --- a/src/ServiceControl.Persistence/IBodyStorage.cs +++ b/src/ServiceControl.Persistence/IBodyStorage.cs @@ -8,6 +8,16 @@ public interface IBodyStorage { + /// + /// Resolves a message body from wherever it was stored. A persister that holds audit data as + /// well as failed messages resolves in one fixed order, because a message that both failed and + /// was audited has two bodies and an edited message's two bodies differ: + /// + /// failed message by UniqueMessageId, + /// failed message by MessageId, + /// audit message by UniqueMessageId, including a body held inline for full text search. + /// + /// Task TryFetch(string bodyId, CancellationToken cancellationToken = default); } diff --git a/src/ServiceControl.Persistence/LocalMessagesView.cs b/src/ServiceControl.Persistence/LocalMessagesView.cs new file mode 100644 index 0000000000..e5cbab08b4 --- /dev/null +++ b/src/ServiceControl.Persistence/LocalMessagesView.cs @@ -0,0 +1,60 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Collections.Generic; + using System.Linq; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.Persistence.Infrastructure; + + /// + /// Merges the failed and audited halves of one local result set under the three rules + /// states. A persister that holds both kinds of message + /// returns them through this, so precedence, paging and counting are defined in one place rather + /// than re-derived per provider and per query. + /// + public static class LocalMessagesView + { + public static QueryResult> Merge( + IReadOnlyCollection failedMessages, + IReadOnlyCollection auditedMessages, + PagingInfo pagingInfo, + IComparer? order = null, + DataVersion version = default) + { + ArgumentNullException.ThrowIfNull(failedMessages); + ArgumentNullException.ThrowIfNull(auditedMessages); + ArgumentNullException.ThrowIfNull(pagingInfo); + + var deduplicated = new Dictionary(failedMessages.Count + auditedMessages.Count); + + // Failed first. A message that both failed and was audited must show as failed, and the + // scatter gather deduplicates with TryAdd, so whichever row is seen first wins for good. + foreach (var message in failedMessages.Concat(auditedMessages)) + { + deduplicated.TryAdd(DeduplicationKey(message), message); + } + + var merged = deduplicated.Values.ToList(); + + if (order != null) + { + merged.Sort(order); + } + + // The total is counted after deduplication, so a message that both failed and was audited + // counts once rather than once per source. + var totalCount = merged.Count; + + IList page = merged.Take(pagingInfo.PageSize).ToList(); + + return new QueryResult>(page, new QueryStatsInfo(version, totalCount)); + } + + /// + /// The key ScatterGatherApiMessageView deduplicates on, so a local merge and a cross + /// instance merge agree on what counts as the same message. + /// + public static string DeduplicationKey(MessagesView message) => + $"{message.ReceivingEndpoint?.Name}-{message.MessageId}"; + } +} diff --git a/src/ServiceControl/CompositeViews/Messages/MessageViewComparer.cs b/src/ServiceControl.Persistence/MessageViewComparer.cs similarity index 85% rename from src/ServiceControl/CompositeViews/Messages/MessageViewComparer.cs rename to src/ServiceControl.Persistence/MessageViewComparer.cs index 0da0816e8c..602b4977af 100644 --- a/src/ServiceControl/CompositeViews/Messages/MessageViewComparer.cs +++ b/src/ServiceControl.Persistence/MessageViewComparer.cs @@ -4,7 +4,7 @@ namespace ServiceControl.CompositeViews.Messages using System.Collections.Generic; using Persistence.Infrastructure; - static class MessageViewComparer + public static class MessageViewComparer { public static IComparer FromSortInfo(SortInfo sortInfo) { @@ -49,17 +49,11 @@ public Comparer(Func comparerFunc) this.comparerFunc = comparerFunc; } - public int Compare(MessagesView x, MessagesView y) - { - return comparerFunc(x, y); - } + public int Compare(MessagesView? x, MessagesView? y) => comparerFunc(x!, y!); - public IComparer Reverse() - { - return new Reverse(this); - } + public IComparer Reverse() => new Reverse(this); - Func comparerFunc; + readonly Func comparerFunc; } class Reverse : IComparer @@ -69,7 +63,7 @@ public Reverse(IComparer inner) this.inner = inner; } - public int Compare(MessagesView x, MessagesView y) => inner.Compare(y, x); + public int Compare(MessagesView? x, MessagesView? y) => inner.Compare(y, x); readonly IComparer inner; } } diff --git a/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs index 32d77627b1..ade60f8ec0 100644 --- a/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs +++ b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs @@ -7,7 +7,9 @@ namespace ServiceControl.UnitTests.Hosting using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Hosting.Commands; + // Environment variables are process wide, so these cannot run alongside anything else that reads them. [TestFixture] + [NonParallelizable] public class AuditIngestionOnlyCommandTests { [TestCase("RavenDB")] diff --git a/src/ServiceControl.UnitTests/ScatterGather/IncompleteResultsTests.cs b/src/ServiceControl.UnitTests/ScatterGather/IncompleteResultsTests.cs index 85d76224cb..1ea91d6fbe 100644 --- a/src/ServiceControl.UnitTests/ScatterGather/IncompleteResultsTests.cs +++ b/src/ServiceControl.UnitTests/ScatterGather/IncompleteResultsTests.cs @@ -16,6 +16,7 @@ namespace ServiceControl.UnitTests.ScatterGather; using ServiceControl.Api.Contracts; using ServiceControl.Infrastructure.Api; using ServiceControl.Infrastructure.WebApi; +using ServiceControl.Persistence; using ServiceControl.Persistence.Infrastructure; /// @@ -133,7 +134,7 @@ public void A_remote_only_query_whose_only_remote_timed_out_is_a_timeout() var factory = new FakeHttpClientFactory(); factory.Register(settings.RemoteInstances[0], Status(HttpStatusCode.GatewayTimeout)); - var api = new GetAuditCountsForEndpointApi(settings, factory, new HttpContextAccessor(), NullLogger.Instance); + var api = new GetAuditCountsForEndpointApi(new EmptyAuditCountsDataStore(), settings, factory, new HttpContextAccessor(), NullLogger.Instance); Assert.ThrowsAsync(() => api.Execute(new AuditCountsForEndpointContext(new PagingInfo(), "Sales"), "/api/endpoints/Sales/audit-count")); } @@ -146,7 +147,7 @@ public void Audit_counts_missing_an_instance_are_not_recorded_as_the_endpoint_th factory.Register(settings.RemoteInstances[0], Json>([new AuditCount { UtcDate = DateTime.UtcNow.Date, Count = 5 }])); factory.Register(settings.RemoteInstances[1], Status(HttpStatusCode.GatewayTimeout)); - var auditCountApi = new AuditCountApi(new GetAuditCountsForEndpointApi(settings, factory, new HttpContextAccessor(), NullLogger.Instance)); + var auditCountApi = new AuditCountApi(new GetAuditCountsForEndpointApi(new EmptyAuditCountsDataStore(), settings, factory, new HttpContextAccessor(), NullLogger.Instance)); var exception = Assert.CatchAsync(() => auditCountApi.GetEndpointAuditCounts("Sales")); diff --git a/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs b/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs new file mode 100644 index 0000000000..8e1dbb1256 --- /dev/null +++ b/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs @@ -0,0 +1,95 @@ +namespace ServiceControl.UnitTests.ScatterGather +{ + using System; + using System.Collections.Generic; + using System.Linq; + using NUnit.Framework; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.Infrastructure; + + [TestFixture] + public class LocalMessagesViewTests + { + [Test] + public void A_message_that_both_failed_and_was_audited_shows_as_failed() + { + var failed = Message("Receiver", "1", MessageStatus.Failed); + var audited = Message("Receiver", "1", MessageStatus.Successful); + + var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); + + Assert.That(result.Results.Single().Status, Is.EqualTo(MessageStatus.Failed)); + } + + [Test] + public void A_message_that_both_failed_and_was_audited_is_counted_once() + { + var failed = Message("Receiver", "1", MessageStatus.Failed); + var audited = Message("Receiver", "1", MessageStatus.Successful); + + var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results, Has.Count.EqualTo(1)); + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(1)); + } + } + + [Test] + public void The_same_message_id_on_a_different_endpoint_is_a_different_message() + { + var failed = Message("Receiver", "1", MessageStatus.Failed); + var audited = Message("OtherReceiver", "1", MessageStatus.Successful); + + var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); + + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(2)); + } + + [Test] + public void A_full_page_from_each_source_is_truncated_to_one_page() + { + var pagingInfo = new PagingInfo(pageSize: 5); + var failed = Enumerable.Range(0, 5).Select(i => Message("Receiver", $"failed-{i}", MessageStatus.Failed)).ToArray(); + var audited = Enumerable.Range(0, 5).Select(i => Message("Receiver", $"audited-{i}", MessageStatus.Successful)).ToArray(); + + var result = LocalMessagesView.Merge(failed, audited, pagingInfo); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results, Has.Count.EqualTo(5), "the scatter gather truncates to a page, so the page must already be the local answer"); + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(10), "the total counts every distinct message, not just the page"); + } + } + + [Test] + public void The_page_is_taken_after_the_requested_order_is_applied() + { + var pagingInfo = new PagingInfo(pageSize: 2); + var failed = new[] { Message("Receiver", "c", MessageStatus.Failed), Message("Receiver", "a", MessageStatus.Failed) }; + var audited = new[] { Message("Receiver", "b", MessageStatus.Successful) }; + + var result = LocalMessagesView.Merge(failed, audited, pagingInfo, MessageViewComparer.FromSortInfo(new SortInfo("message_id", "asc"))); + + Assert.That(result.Results.Select(message => message.MessageId), Is.EqualTo(new[] { "a", "b" }).AsCollection); + } + + [Test] + public void The_local_key_matches_the_key_the_scatter_gather_deduplicates_on() + { + var message = Message("Receiver", "1", MessageStatus.Failed); + + Assert.That(LocalMessagesView.DeduplicationKey(message), Is.EqualTo("Receiver-1")); + } + + static MessagesView Message(string receivingEndpoint, string messageId, MessageStatus status) => new() + { + MessageId = messageId, + Status = status, + ReceivingEndpoint = new EndpointDetails { Name = receivingEndpoint, Host = "host", HostId = Guid.NewGuid() } + }; + } +} diff --git a/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs b/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs index 1455b6d8d6..a1cd1ef93b 100644 --- a/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs +++ b/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs @@ -2,28 +2,30 @@ { using System.Collections.Generic; using System.Linq; + using System.Threading; + using System.Threading.Tasks; using System.Net.Http; using Api.Contracts; using Messages; using Microsoft.AspNetCore.Http; using Microsoft.Extensions.Logging; + using Persistence; using Persistence.Infrastructure; using ServiceBus.Management.Infrastructure.Settings; - // The endpoint is included for consistency reasons but is actually not required here because the query - // is forwarded to the remote instance. But this at least enforces us to declare the controller action - // with the necessary parameter and not accessing the endpoint becomes an implementation details of the scatter - // gather approach here. public record AuditCountsForEndpointContext(PagingInfo PagingInfo, string Endpoint) : ScatterGatherContext(PagingInfo); - // The counts only ever live on an audit instance, so this instance has nothing of its own to add. public class GetAuditCountsForEndpointApi( + IAuditCountsDataStore dataStore, Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) - : ScatterGatherRemoteOnly>(settings, httpClientFactory, httpContextAccessor, logger) + : ScatterGatherApi>(dataStore, settings, httpClientFactory, httpContextAccessor, logger) { + protected override Task>> LocalQuery(AuditCountsForEndpointContext input, CancellationToken cancellationToken = default) => + DataStore.QueryAuditCounts(input.Endpoint, cancellationToken); + protected override IList ProcessResults(AuditCountsForEndpointContext input, QueryResult>[] results) => results.Where(r => r.Results is not null) .SelectMany(r => r.Results) diff --git a/src/ServiceControl/CompositeViews/Messages/ScatterGatherApi.cs b/src/ServiceControl/CompositeViews/Messages/ScatterGatherApi.cs index 1af5732b9f..2e5fb8488f 100644 --- a/src/ServiceControl/CompositeViews/Messages/ScatterGatherApi.cs +++ b/src/ServiceControl/CompositeViews/Messages/ScatterGatherApi.cs @@ -11,6 +11,7 @@ namespace ServiceControl.CompositeViews.Messages using Infrastructure.WebApi; using Microsoft.AspNetCore.Http; using Microsoft.Extensions.Logging; + using Persistence; using Persistence.Infrastructure; using ServiceBus.Management.Infrastructure.Settings; using JsonSerializer = System.Text.Json.JsonSerializer; @@ -107,9 +108,10 @@ void ThrowWhenNothingAnswered(QueryResult[] results, IReadOnlyList /// Whether this instance's own data store is a source for the query. An API that only forwards to the - /// remotes answers "nothing" locally without that meaning anything about the data. + /// remotes, or whose local store is one of the empty audit stand-ins, answers "nothing" locally + /// without that meaning anything about the data. /// - protected virtual bool LocalInstanceParticipates => true; + protected virtual bool LocalInstanceParticipates => DataStore is not IEmptyAuditDataStore; async Task> LocalCall(TIn input, string instanceId, CancellationToken cancellationToken) { @@ -152,7 +154,7 @@ internal QueryResult AggregateResults(TIn input, QueryResult[] resul protected abstract TOut ProcessResults(TIn input, QueryResult[] results); protected virtual QueryStatsInfo AggregateStats(TIn input, IEnumerable> results, TOut processedResults) => - Aggregate(results); + LocalInstanceParticipates ? Aggregate(results) : AggregateStatsFromRemotesOnly(results); /// /// For an API whose own instance is not a source for the data: its local result is a non-participant diff --git a/src/ServiceControl/Persistence/EmptyAuditDataStores.cs b/src/ServiceControl/Persistence/EmptyAuditDataStores.cs new file mode 100644 index 0000000000..e54f5787db --- /dev/null +++ b/src/ServiceControl/Persistence/EmptyAuditDataStores.cs @@ -0,0 +1,33 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Api.Contracts; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + // A primary whose persister holds no audit data still serves the audit routes, answering from its + // remotes alone. These stand in for the local source so the scatter gather is uniform and the APIs + // do not need to know which persister they are running on. + /// + /// Marks a stand-in whose empty answer says nothing about the data, so the scatter gather does not + /// count this instance as having answered. + /// + interface IEmptyAuditDataStore; + + class EmptyAuditCountsDataStore : IAuditCountsDataStore, IEmptyAuditDataStore + { + public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) => + Task.FromResult(new QueryResult>(Empty, QueryStatsInfo.Zero)); + + static readonly IList Empty = new List(0).AsReadOnly(); + } + + class EmptySagaHistoryDataStore : ISagaHistoryDataStore, IEmptyAuditDataStore + { + public Task> QuerySagaHistoryById(Guid sagaId, PagingInfo pagingInfo, CancellationToken cancellationToken = default) => + Task.FromResult(QueryResult.Empty()); + } +} diff --git a/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs b/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs index 2447bfe400..7918f73cb3 100644 --- a/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs +++ b/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence { using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.DependencyInjection.Extensions; using ServiceBus.Management.Infrastructure.Settings; static class PersistenceServiceCollectionExtensions @@ -10,6 +11,11 @@ public static void AddPersistence(this IServiceCollection services, Settings set { var persistence = PersistenceFactory.Create(settings, maintenanceMode); persistence.AddPersistence(services); + + // Only an audit capable persister registers these, so the rest fall back to a local source + // that holds nothing and the audit routes answer from the configured remotes alone. + services.TryAddSingleton(); + services.TryAddSingleton(); } } } diff --git a/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs b/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs index 559fa1f143..e729360473 100644 --- a/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs +++ b/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs @@ -1,19 +1,25 @@ -namespace ServiceControl.SagaAudit +namespace ServiceControl.SagaAudit { using System; using System.Linq; using System.Net.Http; + using System.Threading; + using System.Threading.Tasks; using CompositeViews.Messages; using Microsoft.AspNetCore.Http; using Microsoft.Extensions.Logging; + using Persistence; using Persistence.Infrastructure; using ServiceBus.Management.Infrastructure.Settings; public record SagaByIdContext(PagingInfo PagingInfo, Guid SagaId) : ScatterGatherContext(PagingInfo); - public class GetSagaByIdApi(Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) - : ScatterGatherRemoteOnly(settings, httpClientFactory, httpContextAccessor, logger) + public class GetSagaByIdApi(ISagaHistoryDataStore dataStore, Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) + : ScatterGatherApi(dataStore, settings, httpClientFactory, httpContextAccessor, logger) { + protected override Task> LocalQuery(SagaByIdContext input, CancellationToken cancellationToken = default) => + DataStore.QuerySagaHistoryById(input.SagaId, input.PagingInfo, cancellationToken); + protected override SagaHistory ProcessResults(SagaByIdContext input, QueryResult[] results) { var nonEmptyCount = results.Count(x => x.Results != null); @@ -37,4 +43,4 @@ protected override SagaHistory ProcessResults(SagaByIdContext input, QueryResult return firstResult; } } -} \ No newline at end of file +} From 399c823b39c39ec960d1db1f98949c70f4754cb8 Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 20:31:37 +1000 Subject: [PATCH 06/21] Restore platform connection details and licensing for local audit Two primary owned capabilities depend on an audit remote existing today, and both break once a primary holds audit data itself. /api/connection is what ServicePulse and the Platform Connector plugin read to configure endpoints. With no audit remote it stops advertising MessageAudit.AuditQueue and SagaAudit.SagaAuditQueue, so endpoints cannot be told where to send audit or saga data at all. An audit capable primary now supplies the same two shapes the audit instance supplies, so nothing downstream sees a difference. SagaUpdatedHandler resolves the audit queue through the same IPlatformConnectionBuilder as before, reading either a remote's parsed JSON or the local provider's object, so a misdirected saga audit message is forwarded rather than failed. Audit throughput collection is likewise driven entirely by remotes. With local audit and no remotes, AuditQueues stays empty and the local audit and audit.log queues are counted as customer endpoints in the licensing throughput report, which is an accuracy defect rather than cosmetic, and AuditServicesData comes out blank. IAuditQuery gains an optional local source that contributes the local queue names, version, transport and retention alongside the remotes, so the existing platform endpoint exclusion, service metadata and connection diagnostics all work unchanged. Retention is reported exactly as configured. Where ServiceControl/AuditRetentionPeriod is unset the existing minimum retention gate warns, rather than this guessing a default: what null means is still an open item on the plan. --- .../AuditQuery_Tests.cs | 61 +++++++++++++++++++ .../AuditThroughput/AuditQuery.cs | 7 ++- .../AuditThroughput/ILocalAuditSource.cs | 15 +++++ ...LicensingComponentHostBuilderExtensions.cs | 2 +- ...omposing_audit_ingestion_in_the_primary.cs | 11 ++++ src/ServiceControl/Auditing/AuditComponent.cs | 8 +++ .../AuditPlatformConnectionDetailsProvider.cs | 43 +++++++++++++ .../Auditing/PrimaryLocalAuditSource.cs | 34 +++++++++++ .../SagaAudit/SagaUpdatedHandler.cs | 15 ++++- 9 files changed, 191 insertions(+), 5 deletions(-) create mode 100644 src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs create mode 100644 src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs create mode 100644 src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs diff --git a/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs b/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs index 0d9c13d5b8..54d802add8 100644 --- a/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs +++ b/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs @@ -10,6 +10,7 @@ using NUnit.Framework; using Particular.Approvals; using Particular.LicensingComponent.AuditThroughput; +using Particular.LicensingComponent.Contracts; using Particular.LicensingComponent.UnitTests.Infrastructure; using ServiceControl.Api; using ServiceControl.Api.Contracts; @@ -73,6 +74,43 @@ public async Task Should_return_audit_remotes() } } + [Test] + public async Task Should_return_the_local_audit_source_alongside_the_remotes() + { + //Arrange + var auditQuery = new AuditQuery(NullLogger.Instance, new FakeEndpointApi(), new FakeAuditCountApi(), + new ConfigurationApi_ReturningOneValidAuditConfig(), new LocalAuditSource_ForThisInstance()); + + //Act + var remotes = await auditQuery.GetAuditRemotes(); + + //Assert + Assert.That(remotes, Has.Count.EqualTo(2), "The local audit source and the remote should both be reported"); + + var local = remotes.Single(remote => remote.ApiUri == "http://localhost:33333/api/"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(local.Queues, Does.Contain("audit"), "the local audit queue must be recognised as a platform endpoint"); + Assert.That(local.Queues, Does.Contain("audit.log")); + Assert.That(local.Transport, Is.EqualTo("LearningTransport"), "the report's audit service metadata is built from this"); + } + } + + [Test] + public async Task Should_not_report_a_local_audit_source_that_is_disabled() + { + //Arrange + var auditQuery = new AuditQuery(NullLogger.Instance, new FakeEndpointApi(), new FakeAuditCountApi(), + new ConfigurationApi_ReturningOneValidAuditConfig(), new LocalAuditSource_Disabled()); + + //Act + var remotes = await auditQuery.GetAuditRemotes(); + + //Assert + Assert.That(remotes, Has.Count.EqualTo(1)); + } + [Test] public async Task Should_return_successful_audit_connection_if_instances_exist_and_are_online() { @@ -203,6 +241,29 @@ public Task> GetEndpoints(CancellationToken cancellationToken = d } + class LocalAuditSource_ForThisInstance : ILocalAuditSource + { + public bool Enabled => true; + + public RemoteInstanceInformation Describe() => new() + { + ApiUri = "http://localhost:33333/api/", + VersionString = "6.0.0", + SemanticVersion = new NuGet.Versioning.SemanticVersion(6, 0, 0), + Status = "online", + Retention = TimeSpan.FromDays(10), + Queues = ["audit", "audit.log"], + Transport = "LearningTransport" + }; + } + + class LocalAuditSource_Disabled : ILocalAuditSource + { + public bool Enabled => false; + + public RemoteInstanceInformation Describe() => throw new InvalidOperationException("Describe must not be called when the source is disabled."); + } + class AuditCountApi_ReturningThreeAuditCounts : IAuditCountApi { public async Task> GetEndpointAuditCounts(string endpoint, CancellationToken cancellationToken = default) diff --git a/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs b/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs index 619a6ebae6..a4869ce996 100644 --- a/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs +++ b/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs @@ -7,7 +7,7 @@ using ServiceControl.Api; using AuditCount = Contracts.AuditCount; - public class AuditQuery(ILogger logger, IEndpointsApi endpointsApi, IAuditCountApi auditCountApi, IConfigurationApi configurationApi) : IAuditQuery + public class AuditQuery(ILogger logger, IEndpointsApi endpointsApi, IAuditCountApi auditCountApi, IConfigurationApi configurationApi, ILocalAuditSource? localAuditSource = null) : IAuditQuery { // Customers are expected to run at least version 4.29 for their Audit instances public SemanticVersion MinAuditCountsVersion => new(4, 29, 0); @@ -45,6 +45,11 @@ public async Task> GetAuditRemotes(CancellationT var remotes = await configurationApi.GetRemoteConfigs(cancellationToken); var remotesInfo = new List(); + if (localAuditSource is { Enabled: true }) + { + remotesInfo.Add(localAuditSource.Describe()); + } + if (remotes.Any()) { List queues = []; diff --git a/src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs b/src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs new file mode 100644 index 0000000000..01afb4f805 --- /dev/null +++ b/src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs @@ -0,0 +1,15 @@ +namespace Particular.LicensingComponent.AuditThroughput; + +using Particular.LicensingComponent.Contracts; + +/// +/// Audit throughput collection is driven entirely by audit remotes. A primary that holds audit data +/// itself has no remote to describe it, so without this its own audit queues are counted as customer +/// endpoints and the audit service metadata in the licensing report is blank. +/// +public interface ILocalAuditSource +{ + bool Enabled { get; } + + RemoteInstanceInformation Describe(); +} diff --git a/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs b/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs index 9e8db1f5e4..01637e50f7 100644 --- a/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs +++ b/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs @@ -1,4 +1,4 @@ -namespace Particular.LicensingComponent; +namespace Particular.LicensingComponent; using AuditThroughput; using BrokerThroughput; diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs index 0d82e03075..fa36935479 100644 --- a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs @@ -11,10 +11,12 @@ namespace ServiceControl.AcceptanceTests.Auditing using Microsoft.Extensions.Logging; using NServiceBus; using NUnit.Framework; + using Particular.LicensingComponent.AuditThroughput; using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing; using ServiceControl.CompositeViews.MessageCounting; + using ServiceControl.Connection; using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.WebApi; using ServiceControl.Persistence; @@ -41,6 +43,14 @@ public async Task Should_host_the_audit_runtime_when_the_persister_advertises_au Assert.That(app.Services.GetService(), Is.Not.Null); Assert.That(app.Services.GetService(), Is.Not.Null); Assert.That(app.Services.GetService(), Is.Not.Null); + + Assert.That(app.Services.GetService(), Is.Not.Null, + "without it the local audit queues are counted as customer endpoints in the licensing report"); + Assert.That(services.Any(descriptor => + descriptor.ServiceType == typeof(IProvidePlatformConnectionDetails) + && descriptor.ImplementationType == typeof(AuditPlatformConnectionDetailsProvider)), + Is.True, + "/api/connection must still tell endpoints where to send audit and saga data"); } } finally @@ -86,6 +96,7 @@ public async Task Should_host_nothing_audit_related_on_a_persister_without_audit Assert.That(app.Services.GetService(), Is.Not.Null, "the audit routes stay served from the configured remotes, so the APIs must still resolve"); Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Null); } } finally diff --git a/src/ServiceControl/Auditing/AuditComponent.cs b/src/ServiceControl/Auditing/AuditComponent.cs index 08944411ee..6ba5ca84a6 100644 --- a/src/ServiceControl/Auditing/AuditComponent.cs +++ b/src/ServiceControl/Auditing/AuditComponent.cs @@ -2,9 +2,11 @@ { using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; + using Particular.LicensingComponent.AuditThroughput; using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing.Metrics; + using ServiceControl.Connection; using ServiceControl.CustomChecks; using ServiceControl.Persistence; using ServiceControl.Transports; @@ -50,6 +52,12 @@ public override void Configure(Settings settings, ITransportCustomization transp services.AddHostedService(); } + if (!settings.ErrorIngestionOnly) + { + // Registered before the licensing component's own fallback, which uses TryAdd. + services.AddSingleton(); + services.AddPlatformConnectionProvider(); + } } internal static bool SupportsAuditIngestion(Settings settings) => diff --git a/src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs b/src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs new file mode 100644 index 0000000000..7181afd9ab --- /dev/null +++ b/src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs @@ -0,0 +1,43 @@ +namespace ServiceControl.Auditing +{ + using System.Threading; + using System.Threading.Tasks; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Connection; + + // /api/connection is what ServicePulse and the Platform Connector plugin read to configure + // endpoints. Without an audit remote it stops advertising where audit and saga data should go, so a + // primary holding audit data locally supplies the same shapes the audit instance supplies. + class AuditPlatformConnectionDetailsProvider(Settings settings) : IProvidePlatformConnectionDetails + { + public Task ProvideConnectionDetails(PlatformConnectionDetails connection, CancellationToken cancellationToken = default) + { + connection.Add("MessageAudit", new MessageAuditConnectionDetails + { + Enabled = true, + AuditQueue = settings.AuditQueue + }); + + connection.Add("SagaAudit", new SagaAuditConnectionDetails + { + Enabled = true, + SagaAuditQueue = settings.AuditQueue + }); + + return Task.CompletedTask; + } + + // HINT: These should match the types in the PlatformConnector package + public class MessageAuditConnectionDetails + { + public bool Enabled { get; set; } + public string AuditQueue { get; set; } + } + + public class SagaAuditConnectionDetails + { + public bool Enabled { get; set; } + public string SagaAuditQueue { get; set; } + } + } +} diff --git a/src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs b/src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs new file mode 100644 index 0000000000..b145529241 --- /dev/null +++ b/src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs @@ -0,0 +1,34 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Diagnostics; + using NuGet.Versioning; + using Particular.LicensingComponent.AuditThroughput; + using Particular.LicensingComponent.Contracts; + using ServiceBus.Management.Infrastructure.Settings; + + // Without this the local audit and audit log queues are counted as customer endpoints in the + // licensing throughput report, and the report's audit service metadata is blank. + class PrimaryLocalAuditSource(Settings settings) : ILocalAuditSource + { + public bool Enabled => true; + + public RemoteInstanceInformation Describe() + { + var version = FileVersionInfo.GetVersionInfo(typeof(PrimaryLocalAuditSource).Assembly.Location).ProductVersion; + + return new RemoteInstanceInformation + { + ApiUri = settings.ApiUrl, + VersionString = version, + SemanticVersion = SemanticVersion.TryParse(version ?? string.Empty, out var semanticVersion) ? semanticVersion : null, + Status = "online", + // Retention is reported as configured. When it is not configured the existing minimum + // retention gate warns, rather than this guessing a default on the operator's behalf. + Retention = settings.AuditRetentionPeriod ?? TimeSpan.Zero, + Queues = [settings.AuditQueue, settings.AuditLogQueue], + Transport = settings.TransportType + }; + } + } +} diff --git a/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs b/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs index 25f1f73096..613e7db277 100644 --- a/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs +++ b/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs @@ -53,12 +53,12 @@ async Task RefreshAuditQueue(CancellationToken cancellationToken) var connectionDetails = await connectionBuilder.BuildPlatformConnection(cancellationToken); // First instance is named `SagaAudit`, following instance `SagaAudit1`..`SagaAuditN` - if (connectionDetails.ToDictionary().TryGetValue("SagaAudit", out var sagaAuditObj) && sagaAuditObj is JsonElement sagaAudit) + if (connectionDetails.ToDictionary().TryGetValue("SagaAudit", out var sagaAudit)) { // Pick any audit queue, assume all instance are based on competing consumer - auditQueueName = sagaAudit.GetProperty("SagaAuditQueue").GetString(); + auditQueueName = ReadSagaAuditQueue(sagaAudit); nextAuditQueueNameRefresh = DateTime.UtcNow.AddMinutes(5); - logger.LogInformation("Refreshed audit queue name '{AuditQueueName}' from ServiceControl Audit instance. Will continue to use this value for forwarding saga update messages for the next 5 minutes", auditQueueName); + logger.LogInformation("Refreshed audit queue name '{AuditQueueName}'. Will continue to use this value for forwarding saga update messages for the next 5 minutes", auditQueueName); } } catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) @@ -76,6 +76,15 @@ async Task RefreshAuditQueue(CancellationToken cancellationToken) } } + // A remote instance's details arrive as parsed JSON, a local audit capable primary's as the + // object its provider added. + static string ReadSagaAuditQueue(object sagaAudit) => sagaAudit switch + { + JsonElement json => json.GetProperty("SagaAuditQueue").GetString(), + Auditing.AuditPlatformConnectionDetailsProvider.SagaAuditConnectionDetails local => local.SagaAuditQueue, + _ => null + }; + static string auditQueueName; static DateTime nextAuditQueueNameRefresh; static readonly SemaphoreSlim semaphore = new(1); From 0a1eb0126831b427081781baf5ba4ba7f816e1d9 Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 20:35:22 +1000 Subject: [PATCH 07/21] Compose the audit ingestion only host Adds the host --audit-ingestion-only runs: an audit receiver, the endpoint monitor it needs, and health endpoints, with no NServiceBus endpoint and none of the work a deployment may only do once. The component list is HeartbeatMonitoring and CustomChecks only. HeartbeatMonitoring is there because the audit enricher asks the endpoint monitor whether an endpoint is new before recording it, so without it every audited message writes a known endpoint upsert. CustomChecks is there so a stuck worker reports somewhere. EventLog and ExternalIntegrations are absent because audit ingestion raises no domain events and no integration events; the composition test states that, so a future registration forces a decision rather than slipping in. The ingestion only conditions scattered through the host become one Settings.IngestionOnly, so both modes agree on what they do not do: no endpoint, no retention sweep, no platform connection providers, no heartbeat checking. Ingestion health checks move to the components that host the ingestion they report on, so this host answers for audit-ingestion and not for error-ingestion, and the error ingestion only host is unchanged. --- ...viceControl.AcceptanceTests.RavenDB.csproj | 1 + .../When_hosting_audit_ingestion_only.cs | 154 ++++++++++++++++++ ...rovals.PlatformSampleSettings.approved.txt | 1 + src/ServiceControl/Auditing/AuditComponent.cs | 6 +- .../CustomChecks/CustomChecksComponent.cs | 2 +- .../ExternalIntegrationsComponent.cs | 2 +- .../HostApplicationBuilderExtensions.cs | 5 +- .../Commands/AuditIngestionOnlyCommand.cs | 48 +++++- .../Health/AuditIngestionHealthCheck.cs | 29 ++++ .../Health/HealthCheckExtensions.cs | 7 +- .../Infrastructure/Settings/Settings.cs | 10 ++ .../HeartbeatMonitoringComponent.cs | 4 +- .../HeartbeatMonitoringHostedService.cs | 2 +- .../Persistence/PersistenceFactory.cs | 2 +- .../Recoverability/RecoverabilityComponent.cs | 5 +- 15 files changed, 259 insertions(+), 19 deletions(-) create mode 100644 src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs create mode 100644 src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs diff --git a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj index 8745ac812f..c52ad0407c 100644 --- a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj +++ b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj @@ -44,6 +44,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs new file mode 100644 index 0000000000..86d16ebf29 --- /dev/null +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs @@ -0,0 +1,154 @@ +namespace ServiceControl.AcceptanceTests.Auditing +{ + using System; + using System.IO; + using System.Linq; + using System.Runtime.Loader; + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NUnit.Framework; + using Particular.LicensingComponent.AuditThroughput; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.Hosting.Commands; + using ServiceControl.Infrastructure; + using ServiceControl.Persistence; + using ServiceControl.Persistence.Tests.AuditCapable; + + // The inner persistence type reaches the test persister through an environment variable, which is + // process wide, so these cannot run alongside anything else that sets it. + [NonParallelizable] + class When_hosting_audit_ingestion_only : AcceptanceTest + { + [Test] + public async Task Should_ingest_without_an_endpoint_and_without_the_single_owner_services() + { + var settings = await CreateSettings(auditCapable: true); + + var host = AuditIngestionOnlyCommand.BuildHost(settings); + + try + { + var hostedServices = host.Services.GetServices() + .Select(hostedService => hostedService.GetType().Name) + .ToArray(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(host.Services.GetService(), Is.Null, + "the host must not run an NServiceBus endpoint"); + Assert.That(host.Services.GetService(), Is.Not.Null); + + Assert.That(host.Services.GetService(), Is.Null, + "an ingestion only worker never changes the database schema"); + Assert.That(host.Services.GetService(), Is.Null, + "an ingestion only worker never provisions external storage"); + Assert.That(host.Services.GetService(), Is.Null, + "licensing throughput is owned by the normal primary, and would be counted once per node"); + Assert.That(host.Services.GetService(), Is.Not.Null); + + Assert.That(hostedServices, Is.EquivalentTo(ExpectedHostedServices), + "the set of hosted services in the audit ingestion only host changed. Every one of " + + "these runs on every ingestion node, so decide whether that is safe before updating " + + "this list. Audit ingestion raises no domain events and no integration events, which " + + "is why EventLog and ExternalIntegrations are not registered."); + } + } + finally + { + await host.DisposeAsync(); + } + } + + [Test] + public async Task Should_report_audit_ingestion_readiness() + { + var settings = await CreateSettings(auditCapable: true); + + var host = AuditIngestionOnlyCommand.BuildHost(settings); + + try + { + var readiness = host.Services.GetRequiredService(); + + var report = await readiness.CheckHealthAsync(registration => registration.Tags.Contains("ready")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(report.Entries.Keys, Does.Contain("audit-ingestion")); + Assert.That(report.Entries.Keys, Does.Not.Contain("error-ingestion"), + "this host does not ingest error messages, so it must not answer for them"); + } + } + finally + { + await host.DisposeAsync(); + } + } + + [Test] + public async Task Should_refuse_to_start_against_storage_without_audit_support() + { + var settings = await CreateSettings(auditCapable: false); + + var exception = Assert.ThrowsAsync(() => + new AuditIngestionOnlyCommand().Execute(new HostArguments([]), settings)); + + Assert.That(exception.Message, Does.Contain("supports audit ingestion")); + } + + static readonly string[] ExpectedHostedServices = + [ + "GenericWebHostService", // health endpoints only, no ServiceControl API + nameof(AuditIngestion), // the reason this host exists + "HeartbeatMonitoringHostedService", // warms the endpoint monitor, does not check heartbeats + "InternalCustomChecksHostedService", // reports this node's ingestion health to the database + "MetricsReporterHostedService", + "HealthCheckPublisherHostedService", // inert, no IHealthCheckPublisher is registered + "ExternalIntegrationRequestsDataStore" // registered by the persister; its drain is inert here, nothing calls Subscribe + ]; + + [TearDown] + public void ClearInnerPersistenceType() => Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, null); + + async Task CreateSettings(bool auditCapable) + { + var persistenceType = StorageConfiguration.PersistenceType; + + if (auditCapable) + { + Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, persistenceType); + persistenceType = AuditCapablePersistenceName; + } + + var settings = new Settings(TransportIntegration.TypeName, persistenceType, + CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) + { + InstanceName = $"AuditIngestOnly.{Guid.NewGuid():n}", + TransportConnectionString = TransportIntegration.ConnectionString, + MaximumConcurrencyLevel = 2, + AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default + }; + + await StorageConfiguration.CustomizeSettings(settings); + + return settings; + } + + static LoggingSettings CreateLoggingSettings() + { + var logPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()); + Directory.CreateDirectory(logPath); + return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); + } + + const string AuditCapablePersistenceName = "AuditCapableTest"; + + static readonly string InnerPersistenceTypeVariable = + AuditCapableTestPersistenceConfiguration.InnerPersistenceTypeSetting.ToUpperInvariant(); + } +} diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt index 683c5110da..ebcec51057 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt @@ -77,6 +77,7 @@ "AuditIngestionBatchTimeout": "00:00:00", "TimeToRestartAuditIngestionAfterFailure": "00:01:00", "ErrorIngestionOnly": false, + "AuditIngestionOnly": false, "AuditRetentionPeriod": null, "ErrorRetentionPeriod": "10.00:00:00", "EventsRetentionPeriod": "14.00:00:00", diff --git a/src/ServiceControl/Auditing/AuditComponent.cs b/src/ServiceControl/Auditing/AuditComponent.cs index 6ba5ca84a6..65160b05a9 100644 --- a/src/ServiceControl/Auditing/AuditComponent.cs +++ b/src/ServiceControl/Auditing/AuditComponent.cs @@ -8,6 +8,7 @@ using ServiceControl.Auditing.Metrics; using ServiceControl.Connection; using ServiceControl.CustomChecks; + using ServiceControl.Infrastructure.Health; using ServiceControl.Persistence; using ServiceControl.Transports; @@ -47,12 +48,15 @@ public override void Configure(Settings settings, ITransportCustomization transp services.AddCustomCheck(); services.AddCustomCheck(); + services.AddHealthChecks() + .AddCheck("audit-ingestion", tags: [HealthCheckExtensions.ReadyTag]); + if (settings.IngestAuditMessages) { services.AddHostedService(); } - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { // Registered before the licensing component's own fallback, which uses TryAdd. services.AddSingleton(); diff --git a/src/ServiceControl/CustomChecks/CustomChecksComponent.cs b/src/ServiceControl/CustomChecks/CustomChecksComponent.cs index 5e4b20e731..7b486e3d15 100644 --- a/src/ServiceControl/CustomChecks/CustomChecksComponent.cs +++ b/src/ServiceControl/CustomChecks/CustomChecksComponent.cs @@ -29,7 +29,7 @@ public override void Configure(Settings settings, ITransportCustomization transp hostBuilder.Services.AddEventLogMapping(); hostBuilder.Services.AddEventLogMapping(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { hostBuilder.Services.AddPlatformConnectionProvider(); } diff --git a/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs b/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs index 6d86407288..9e1782a333 100644 --- a/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs +++ b/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs @@ -18,7 +18,7 @@ public override void Configure(Settings settings, ITransportCustomization transp { services.AddDomainEventHandler(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { services.AddHostedService(); } diff --git a/src/ServiceControl/HostApplicationBuilderExtensions.cs b/src/ServiceControl/HostApplicationBuilderExtensions.cs index a66d5d2e0a..0753d2d075 100644 --- a/src/ServiceControl/HostApplicationBuilderExtensions.cs +++ b/src/ServiceControl/HostApplicationBuilderExtensions.cs @@ -43,7 +43,7 @@ static class HostApplicationBuilderExtensions public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, Settings settings, EndpointConfiguration configuration, params ReadOnlySpan components) { - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { ArgumentNullException.ThrowIfNull(configuration); } @@ -108,7 +108,7 @@ public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, S hostBuilder.AddTelemetry(settings); services.AddServiceControlHealthChecks(); - if (settings.ErrorIngestionOnly) + if (settings.IngestionOnly) { // Ingestion receives through its own transport infrastructure and forwards through // that same infrastructure's dispatcher, so the endpoint is not hosted at all. @@ -209,6 +209,7 @@ Audit Retention Period (optional): {settings.AuditRetentionPeriod} Error Retention Period: {settings.ErrorRetentionPeriod} Ingest Error Messages: {settings.IngestErrorMessages} Error Ingestion Only: {settings.ErrorIngestionOnly} +Audit Ingestion Only: {settings.AuditIngestionOnly} Forwarding Error Messages: {settings.ForwardErrorMessages} ServiceControl Logging Level: {settings.LoggingSettings.LogLevel} Selected Transport Customization: {settings.TransportType} diff --git a/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs index abe0f17002..eb86a3e7c5 100644 --- a/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs +++ b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs @@ -3,24 +3,60 @@ namespace ServiceControl.Hosting.Commands using System; using System.Threading; using System.Threading.Tasks; + using Microsoft.AspNetCore.Builder; + using Particular.ServiceControl; using Particular.ServiceControl.Hosting; using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.Infrastructure.Health; + using ServiceControl.Monitoring; /// /// Runs a host that does nothing but drain the audit queue into the shared database, so several - /// processes can ingest against one database. The host itself lands with the ingestion only - /// composition; until then the command exists so the startup checks and the help text are in place, - /// and it always fails because no shipped persister advertises audit support yet. + /// processes can ingest against one database. Everything a deployment may only run once, the + /// failed audit reimport command, API hosting, retention and licensing, stays with the primary + /// instance, and this host never provisions queues, schema or body storage. /// class AuditIngestionOnlyCommand : AbstractCommand { - public override Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) + public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) { IngestionOnlyGuards.EnsureStorageSupportsAuditIngestion(settings); IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only"); - throw new Exception( - "--audit-ingestion-only is not available yet. The storage advertises audit support, but the audit ingestion only host has not been composed."); + var app = BuildHost(settings); + + await app.RunAsync(settings.RootUrl); } + + internal static WebApplication BuildHost(Settings settings, Action customize = null) + { + settings.AuditIngestionOnly = true; + settings.IngestAuditMessages = true; + settings.IngestErrorMessages = false; + settings.RunRetryProcessor = false; + + var hostBuilder = WebApplication.CreateBuilder(); + + hostBuilder.AddServiceControl(settings, configuration: null, Components); + + customize?.Invoke(hostBuilder); + + var app = hostBuilder.Build(); + + app.MapServiceControlHealthChecks(); + + return app; + } + + // EventLog and ExternalIntegrations are deliberately absent: audit ingestion raises no domain + // events and no integration events. Hosting would claim the instance queue, and Licensing would + // count throughput once per node. + static ServiceControlComponent[] Components => + [ + new HeartbeatMonitoringComponent(), + new CustomChecks.CustomChecksComponent(), + new AuditComponent() + ]; } } diff --git a/src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs b/src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs new file mode 100644 index 0000000000..aeef4345ce --- /dev/null +++ b/src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs @@ -0,0 +1,29 @@ +namespace ServiceControl.Infrastructure.Health +{ + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Diagnostics.HealthChecks; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + + /// + /// Reports the state the audit ingestion watchdog publishes, which covers a batch that keeps + /// failing as well as a receiver that fails to start. + /// + class AuditIngestionHealthCheck(AuditIngestionCustomCheck.State ingestionState, Settings settings) : IHealthCheck + { + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) + { + if (!settings.IngestAuditMessages) + { + return Task.FromResult(HealthCheckResult.Healthy("Audit ingestion is disabled")); + } + + var failure = ingestionState.GetLastFailure(); + + return Task.FromResult(failure == null + ? HealthCheckResult.Healthy("Ingesting audit messages") + : HealthCheckResult.Unhealthy(failure)); + } + } +} diff --git a/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs b/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs index 64fd04f30e..d709a85968 100644 --- a/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs +++ b/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs @@ -14,11 +14,12 @@ static class HealthCheckExtensions public const string LivenessPath = "/health"; public const string ReadinessPath = "/health/ready"; - const string ReadyTag = "ready"; + internal const string ReadyTag = "ready"; + // The individual checks are added by the components that host the work they report on, so a + // host that does not ingest error messages does not answer for error ingestion. public static void AddServiceControlHealthChecks(this IServiceCollection services) => - services.AddHealthChecks() - .AddCheck("error-ingestion", tags: [ReadyTag]); + services.AddHealthChecks(); /// /// Liveness answers "is this process still serving", and is what a container health check diff --git a/src/ServiceControl/Infrastructure/Settings/Settings.cs b/src/ServiceControl/Infrastructure/Settings/Settings.cs index bb7a104998..dcac86fe21 100644 --- a/src/ServiceControl/Infrastructure/Settings/Settings.cs +++ b/src/ServiceControl/Infrastructure/Settings/Settings.cs @@ -218,6 +218,16 @@ public string InstanceId // Set by the --error-ingestion-only command, never read from configuration. public bool ErrorIngestionOnly { get; set; } + // Set by the --audit-ingestion-only command, never read from configuration. + public bool AuditIngestionOnly { get; set; } + + /// + /// True in either ingestion only mode. These hosts run no NServiceBus endpoint, own none of the + /// work a deployment may only do once, and never provision anything. + /// + [JsonIgnore] + public bool IngestionOnly => ErrorIngestionOnly || AuditIngestionOnly; + public TimeSpan? AuditRetentionPeriod { get; set; } public TimeSpan ErrorRetentionPeriod { get; } diff --git a/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs b/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs index 542901eed0..7f15781d4e 100644 --- a/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs +++ b/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs @@ -31,7 +31,7 @@ public override void Configure(Settings settings, ITransportCustomization transp { hostBuilder.Services.AddHostedService(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { hostBuilder.Services.AddHostedService(); } @@ -52,7 +52,7 @@ public override void Configure(Settings settings, ITransportCustomization transp hostBuilder.Services.AddErrorMessageEnricher(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { hostBuilder.Services.AddPlatformConnectionProvider(); } diff --git a/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs b/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs index 779a2fb46f..35efea4e21 100644 --- a/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs +++ b/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs @@ -27,7 +27,7 @@ public async Task StartAsync(CancellationToken cancellationToken = default) // An ingestion only host receives no heartbeats, so it has nothing to check and would // only report every endpoint as dead. It still warms the monitor, because the error // enricher asks it whether an endpoint is new before recording it. - if (settings.ErrorIngestionOnly) + if (settings.IngestionOnly) { return; } diff --git a/src/ServiceControl/Persistence/PersistenceFactory.cs b/src/ServiceControl/Persistence/PersistenceFactory.cs index 892920cf18..bad7018f70 100644 --- a/src/ServiceControl/Persistence/PersistenceFactory.cs +++ b/src/ServiceControl/Persistence/PersistenceFactory.cs @@ -18,7 +18,7 @@ public static IPersistence Create(Settings settings, bool maintenanceMode = fals //HINT: This is false when executed from acceptance tests settings.PersisterSpecificSettings ??= persistenceConfiguration.CreateSettings(Settings.SettingsRootNamespace); settings.PersisterSpecificSettings.MaintenanceMode = maintenanceMode; - settings.PersisterSpecificSettings.RunRetentionSweep = !settings.ErrorIngestionOnly; + settings.PersisterSpecificSettings.RunRetentionSweep = !settings.IngestionOnly; var persistence = persistenceConfiguration.Create(settings.PersisterSpecificSettings); return persistence; diff --git a/src/ServiceControl/Recoverability/RecoverabilityComponent.cs b/src/ServiceControl/Recoverability/RecoverabilityComponent.cs index ab17793427..d3ab20dfa4 100644 --- a/src/ServiceControl/Recoverability/RecoverabilityComponent.cs +++ b/src/ServiceControl/Recoverability/RecoverabilityComponent.cs @@ -11,6 +11,7 @@ using ExternalIntegrations; using Infrastructure.BackgroundTasks; using Infrastructure.DomainEvents; + using Infrastructure.Health; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; @@ -80,7 +81,7 @@ public override void Configure(Settings settings, ITransportCustomization transp services.AddSingleton(); services.AddSingleton(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { services.AddHostedService(provider => provider.GetRequiredService()); } @@ -109,6 +110,8 @@ public override void Configure(Settings settings, ITransportCustomization transp //Health checks services.AddCustomCheck(); services.AddCustomCheck(); + services.AddHealthChecks() + .AddCheck("error-ingestion", tags: [HealthCheckExtensions.ReadyTag]); //External integration services.AddIntegrationEventPublisher(); From a6c89bf81982b25b6af27175e42fe9c9907e29b7 Mon Sep 17 00:00:00 2001 From: John Simons Date: Thu, 20 Aug 2026 20:38:24 +1000 Subject: [PATCH 08/21] Document the audit hosting modes and pin the project boundary docs/audit-ingestion-in-the-primary.md covers the three deployment modes, the settings and their key collisions with a standalone audit instance, queue ownership and what ingestion only workers never provision, the body storage rule, the health endpoints, and the query behavior including the precedence, paging and counting rules. The primary logs a warning at startup when it both ingests audit messages and has audit remotes configured, because that is the shape most likely to hit the setting collisions the doc describes. An architecture test pins the boundary the plan depends on: the primary must not reference ServiceControl.Audit, which stays a standalone composition root. The packaging tests confirm the copied runtime ships inside the existing primary artifact, that the three new OpenTelemetry references change no deployment unit, and that the persisters folder is still exactly the four shipped storages. --- docs/audit-ingestion-in-the-primary.md | 119 ++++++++++++++++++ .../PrimaryAssemblyBoundaryTests.cs | 21 ++++ src/ServiceControl/Auditing/AuditComponent.cs | 21 ++++ 3 files changed, 161 insertions(+) create mode 100644 docs/audit-ingestion-in-the-primary.md create mode 100644 src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs diff --git a/docs/audit-ingestion-in-the-primary.md b/docs/audit-ingestion-in-the-primary.md new file mode 100644 index 0000000000..8896d63347 --- /dev/null +++ b/docs/audit-ingestion-in-the-primary.md @@ -0,0 +1,119 @@ +# Audit ingestion in the primary instance + +## Overview + +Storage that advertises `SupportsAuditIngestion` in its `persistence.manifest` can hold audit data +alongside the primary's own data, which lets the primary ServiceControl process ingest the audit +queue itself instead of relying on a separate ServiceControl.Audit instance. + +The standalone RavenDB audit instance is unaffected. RavenDB does not advertise audit support, does +not gain combined hosting, and keeps its own executable, settings, API and installers. + +No shipped persister advertises audit support yet, so on every existing deployment the audit +component registers nothing and behavior is unchanged. + +## Deployment modes + +| Mode | How | What it runs | +| --- | --- | --- | +| Normal primary, audit ingestion on | Default where the persister advertises audit support | The audit receiver, the audit capabilities, the primary API and everything a normal primary runs | +| Normal primary, audit ingestion off | `ServiceControl/IngestAuditMessages=false` | Everything above except the audit receiver. Local audit queries, failed audit tooling and `/api/connection` stay active, because other processes may still be ingesting | +| Audit ingestion only | `ServiceControl.exe --audit-ingestion-only` | The audit receiver, the endpoint monitor it depends on, this node's custom checks, and the health endpoints. No NServiceBus endpoint, no API, no retention, no licensing | + +`--audit-ingestion-only` and `--error-ingestion-only` cannot be combined. Each queue gets its own +worker pool so the two can be scaled independently, so run one process per mode. + +All three modes keep audit data in the primary's own database. A customer whose audit load would +swamp that database can instead move audit to a dedicated one, served by the same executable in +`--audit-instance` mode. That topology was decided on 14 September 2026 and arrives with the EF +audit persistence work; until then this document describes the shared-database modes only. + +## Settings + +The primary reads the audit settings under the same key names the audit instance uses, so an audit +capable primary is configured exactly the way an audit instance is configured today. + +| Setting | Default | Notes | +| --- | --- | --- | +| `ServiceControl/IngestAuditMessages` | `true` | Applies to the normal primary only. Always on under `--audit-ingestion-only`, and has no effect where the persister does not support audit | +| `ServiceBus/AuditQueue` | `audit` | The queue this instance drains | +| `ServiceBus/AuditLogQueue` | the subscoped audit queue name | Only used when forwarding is on | +| `ServiceControl/ForwardAuditMessages` | `false` | | +| `ServiceControl/AuditRetentionPeriod` | unset | Already existed. Validated between 1 hour and 365 days | +| `ServiceControl/MaximumAuditIngestionConcurrencyLevel` | `32` | Independent of the primary endpoint's concurrency, which is what `MaximumConcurrencyLevel` sets | +| `ServiceControl/TimeToRestartAuditIngestionAfterFailure` | 60 seconds | Mirrors the error equivalent | +| `ServiceControl/OtlpEndpointUrl` | unset | Enables the OpenTelemetry metrics exporter | +| `ServiceControl/MessageBody/FileSystem/PathIsShared` | `false` | Required by both ingestion only modes when body storage is the file system | + +### Setting collisions + +`ServiceControl` and `ServiceControl.Audit` settings can both be set by bare environment variable +name, and `ServiceBus/AuditQueue` is literally the same key for both processes. A combined primary +and a standalone audit instance sharing one environment file therefore collide on +`INGESTAUDITMESSAGES`, `AUDITRETENTIONPERIOD`, `FORWARDAUDITMESSAGES` and `SERVICEBUS_AUDITQUEUE`. + +That combination is unsupported. The primary logs a warning at startup when it has audit ingestion +enabled and audit remotes configured at the same time, because that is the shape most likely to hit +the collision. + +## Queue ownership + +The setup path creates the audit queue, and the audit forwarding queue when forwarding is enabled. +Ingestion only workers run no installers: they never create queues, never apply database migrations +and never provision body storage. Run setup from a normal instance before starting any worker. + +Transport operations remain in the audit ingestion path for two reasons only: + +- **Forwarding**, when `ForwardAuditMessages` is on. +- **Retry acknowledgements**. `ServiceControl.Retry.AcknowledgementQueue` is stamped by whichever + instance issued the retry, so the acknowledgement cannot be short-circuited into the local + database. In a combined host it is dispatched to the local error queue and comes straight back in + through local error ingestion, which is exactly what happens today. + +Endpoints detected from audit headers are written straight to the shared `KnownEndpoints` table +through the ingestion unit of work, rather than sent to the primary's input queue. + +## Body storage + +Audit and failed message bodies share one store, and each owns a prefixed keyspace, so an edited +message's failed body and its audited body do not collide. `IBodyStorage.TryFetch` resolves in a +fixed order: failed message by `UniqueMessageId`, then failed message by `MessageId`, then audit +message by `UniqueMessageId`. + +Every ingesting process must write bodies somewhere every host can read. Blob and S3 storage +qualify. File system storage qualifies only if the path is a shared mount, which nothing in the +settings can detect, so both ingestion only modes refuse to start unless +`ServiceControl/MessageBody/FileSystem/PathIsShared` asserts it. + +## Health endpoints + +Both ingestion only hosts map the same two routes, anonymously, returning JSON: + +- `/health` is liveness. It answers "is this process still serving" and is what a container health + check should restart on. +- `/health/ready` additionally reports whether the ingestion this host exists to do is happening. + An audit ingestion only host answers for `audit-ingestion` and not for `error-ingestion`. + +## Querying + +Local audit data is served through the existing primary routes under their existing policies: +`/api/messages` and its variants on `error:messages:view`, `/api/sagas/{id}` on +`error:sagas:view`, and `endpoints/{endpoint}/audit-count` on `error:messages:view`. A primary +configured with an audit remote already serves that remote's audit data under those policies today, +so nothing about the `my/routes` manifest or ServicePulse navigation changes. + +Additional audit remotes keep working. The scatter gather runs the local query first and merges the +remotes after, so a primary can hold audit data locally, query remotes, or both. + +Where one local result set contains both failed and audited messages, three rules apply, and +`LocalMessagesView.Merge` implements them for any persister: + +1. **Precedence.** A message that both failed and was audited shows as failed. +2. **Paging.** The local result is already at most one page, after deduplication. +3. **Counting.** A message that both failed and was audited is counted once. + +## Packaging + +The audit runtime ships inside the existing primary artifact. There is no new assembly and no new +deployment unit. The primary gains three OpenTelemetry package references, which the copied +ingestion metrics use, exported only when `OtlpEndpointUrl` is set. diff --git a/src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs b/src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs new file mode 100644 index 0000000000..99900e76d5 --- /dev/null +++ b/src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs @@ -0,0 +1,21 @@ +namespace ServiceControl.UnitTests.Infrastructure +{ + using System.Linq; + using NUnit.Framework; + using Particular.ServiceControl; + + [TestFixture] + public class PrimaryAssemblyBoundaryTests + { + [Test] + public void The_primary_does_not_reference_the_standalone_audit_executable() + { + var referenced = typeof(HostingComponent).Assembly.GetReferencedAssemblies().Select(name => name.Name); + + Assert.That(referenced, Does.Not.Contain("ServiceControl.Audit"), + "ServiceControl.Audit is a standalone composition root holding RavenDB persistence selection, standalone " + + "settings, API hosting, installer commands and its own NServiceBus endpoint. The primary owns a copy of " + + "the audit runtime instead, so that project stays off its reference graph."); + } + } +} diff --git a/src/ServiceControl/Auditing/AuditComponent.cs b/src/ServiceControl/Auditing/AuditComponent.cs index 65160b05a9..3d54c545bd 100644 --- a/src/ServiceControl/Auditing/AuditComponent.cs +++ b/src/ServiceControl/Auditing/AuditComponent.cs @@ -2,12 +2,14 @@ { using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; using Particular.LicensingComponent.AuditThroughput; using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing.Metrics; using ServiceControl.Connection; using ServiceControl.CustomChecks; + using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.Health; using ServiceControl.Persistence; using ServiceControl.Transports; @@ -38,6 +40,8 @@ public override void Configure(Settings settings, ITransportCustomization transp return; } + WarnAboutSettingCollisions(settings); + var services = hostBuilder.Services; services.AddSingleton(); @@ -64,6 +68,23 @@ public override void Configure(Settings settings, ITransportCustomization transp } } + // ServiceControl and ServiceControl.Audit settings can both be set by bare environment variable + // name, and ServiceBus/AuditQueue is literally the same key for both processes, so a combined + // primary and a standalone audit instance sharing one environment file collide. That + // combination is unsupported, and this is the shape most likely to hit it. + static void WarnAboutSettingCollisions(Settings settings) + { + if (settings.RemoteInstances.Length == 0) + { + return; + } + + LoggerUtil.CreateStaticLogger(typeof(AuditComponent), settings.LoggingSettings.LogLevel) + .LogWarning("This instance ingests audit messages itself and also has {RemoteInstanceCount} audit remote(s) configured. " + + "Running both is not supported: the two processes read the same setting names, so a shared environment file makes them " + + "collide on the audit queue, retention, forwarding and ingestion settings.", settings.RemoteInstances.Length); + } + internal static bool SupportsAuditIngestion(Settings settings) => PersistenceManifestLibrary.Find(settings.PersistenceType)?.SupportsAuditIngestion ?? false; } From a84673de33b4732bcb9cbae0641663b04171b84c Mon Sep 17 00:00:00 2001 From: John Simons Date: Fri, 18 Sep 2026 13:33:51 +1000 Subject: [PATCH 09/21] Introduce AddServiceControlInstance and report host.name and process.pid for scaled-out workers Scaled-out workers drain the same queue and share an instance name, so `service.name` alone cannot tell them apart on a dashboard. Every process now also reports `host.name` and `process.pid` by default, which makes a pool distinguishable without any operator configuration. `AddServiceControlInstance` centralises that logic and defers to the standard OpenTelemetry environment variables (`OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`) wherever an operator has already set them, including for `service.instance.id`, `host.name`, and `process.pid`. Both the metrics pipeline and the static bootstrap loggers call it, so every signal from a process carries the same identity. `ServiceControlMeters.Error` is renamed to `Primary` to reflect that the meter names the instance, not the subject: audit ingestion hosted in the primary publishes on the primary meter, and what a measurement is about is carried by the instrument prefix instead. --- docs/audit-ingestion-in-the-primary.md | 15 ++++ .../Auditing/Metrics/IngestionMetrics.cs | 2 +- .../HostApplicationBuilderExtensions.cs | 5 +- .../LoggerUtil.cs | 16 ++--- .../ServiceControlMeters.cs | 12 ++-- .../TelemetryResourceBuilderExtensions.cs | 72 +++++++++++++++++++ .../Metrics/RetentionMetrics.cs | 2 +- .../Archiving/Metrics/ArchiveMetrics.cs | 2 +- ...TelemetryResourceBuilderExtensionsTests.cs | 39 ++++++++++ .../Auditing/Metrics/AuditIngestionMetrics.cs | 5 +- .../AuditIngestionMetricsConfiguration.cs | 4 +- .../HostApplicationBuilderExtensions.cs | 10 +-- .../Operations/Metrics/IngestionMetrics.cs | 2 +- .../Retrying/Metrics/RetryMetrics.cs | 2 +- 14 files changed, 150 insertions(+), 38 deletions(-) create mode 100644 src/ServiceControl.Infrastructure/TelemetryResourceBuilderExtensions.cs create mode 100644 src/ServiceControl.UnitTests/Infrastructure/TelemetryResourceBuilderExtensionsTests.cs diff --git a/docs/audit-ingestion-in-the-primary.md b/docs/audit-ingestion-in-the-primary.md index 8896d63347..550cadef5c 100644 --- a/docs/audit-ingestion-in-the-primary.md +++ b/docs/audit-ingestion-in-the-primary.md @@ -112,6 +112,21 @@ Where one local result set contains both failed and audited messages, three rule 2. **Paging.** The local result is already at most one page, after deduplication. 3. **Counting.** A message that both failed and was audited is counted once. +## Telemetry + +Both ingestions publish on the primary instance's meter, `Particular.ServiceControl`; a standalone audit instance publishes on `Particular.ServiceControl.Audit`. The meter names the process, not the subject. What a measurement is about is carried by the instrument prefix instead: `sc.error.ingestion.*` for error ingestion and `sc.audit.ingestion.*` for audit ingestion, unchanged whichever instance produced them. + +Scaled out workers drain the same queue and so share an instance name, which makes `service.name` identical across the pool. Every process therefore also reports `host.name` and `process.pid`, so a pool can be told apart on a dashboard with nothing configured. `service.instance.id` is generated where none is given: unique per process, but new on every restart, so a dashboard grouped on it alone loses its series each time a worker is recycled. + +Naming a worker explicitly is the standard OpenTelemetry environment variables, honored for both metrics and exported logs: + +``` +OTEL_SERVICE_NAME=sc-audit-ingestion +OTEL_RESOURCE_ATTRIBUTES=service.instance.id=worker-1 +``` + +Anything set there wins, including `host.name` and `process.pid`, so a containerized deployment can report the identity it wants rather than the one the process detects. + ## Packaging The audit runtime ships inside the existing primary artifact. There is no new assembly and no new diff --git a/src/ServiceControl.Audit/Auditing/Metrics/IngestionMetrics.cs b/src/ServiceControl.Audit/Auditing/Metrics/IngestionMetrics.cs index cdd6788eab..29002958c3 100644 --- a/src/ServiceControl.Audit/Auditing/Metrics/IngestionMetrics.cs +++ b/src/ServiceControl.Audit/Auditing/Metrics/IngestionMetrics.cs @@ -12,7 +12,7 @@ public class IngestionMetrics { - public const string MeterName = ServiceControlMeters.Audit; + public const string MeterName = ServiceControlMeters.AuditInstance; public static readonly string BatchDurationInstrumentName = $"{InstrumentPrefix}.batch_duration_seconds"; public static readonly string MessageDurationInstrumentName = $"{InstrumentPrefix}.message_duration_seconds"; diff --git a/src/ServiceControl.Audit/HostApplicationBuilderExtensions.cs b/src/ServiceControl.Audit/HostApplicationBuilderExtensions.cs index ccd2c635cc..cb27b1a5cd 100644 --- a/src/ServiceControl.Audit/HostApplicationBuilderExtensions.cs +++ b/src/ServiceControl.Audit/HostApplicationBuilderExtensions.cs @@ -105,10 +105,7 @@ public static void AddMetrics(this IHostApplicationBuilder builder, Settings set } builder.Services.AddOpenTelemetry() - .ConfigureResource(b => b.AddService( - serviceName: settings.InstanceName, - serviceVersion: InstanceVersion, - autoGenerateServiceInstanceId: true)) + .ConfigureResource(b => b.AddServiceControlInstance(settings.InstanceName, InstanceVersion)) .WithMetrics(b => { b.AddIngestionMetrics(); diff --git a/src/ServiceControl.Infrastructure/LoggerUtil.cs b/src/ServiceControl.Infrastructure/LoggerUtil.cs index a562e6f934..500296d2b1 100644 --- a/src/ServiceControl.Infrastructure/LoggerUtil.cs +++ b/src/ServiceControl.Infrastructure/LoggerUtil.cs @@ -28,11 +28,9 @@ public static class LoggerUtil public static string SeqAddress { private get; set; } // Telemetry resource attached to exported OTLP logs (service.name/service.version/service.instance.id). - // Set once at process startup via Initialize() — before any logger is created — so both the host pipeline - // and the static bootstrap loggers (CreateStaticLogger) share a single instance identity. Defaults to - // CreateDefault() (which still honors OTEL_SERVICE_NAME/OTEL_RESOURCE_ATTRIBUTES) for the rare logger - // created before Initialize runs. - static ResourceBuilder serviceResourceBuilder = CreateResourcesBuilder(); + // Built once so every logger in the process, including the static bootstrap ones, reports the same + // instance identity as the metrics pipeline. + static readonly ResourceBuilder serviceResourceBuilder = CreateResourcesBuilder(); static ResourceBuilder CreateResourcesBuilder() { @@ -40,15 +38,9 @@ static ResourceBuilder CreateResourcesBuilder() var serviceName = asm.GetName().Name ?? throw new InvalidOperationException("Entry assembly name not found"); var serviceVersion = FileVersionInfo.GetVersionInfo(asm.Location).ProductVersion; - // CreateDefault() also reads OTEL_SERVICE_NAME/OTEL_RESOURCE_ATTRIBUTES, so operators can still enrich - // the resource with deployment-specific attributes via those environment variables. return ResourceBuilder .CreateDefault() - .AddService( - serviceName, - serviceVersion: serviceVersion, - autoGenerateServiceInstanceId: true - ); + .AddServiceControlInstance(serviceName, serviceVersion); } public static bool IsLoggingTo(Loggers logger) => (logger & ActiveLoggers) == logger; diff --git a/src/ServiceControl.Infrastructure/ServiceControlMeters.cs b/src/ServiceControl.Infrastructure/ServiceControlMeters.cs index aaac716f21..4ed7a1c819 100644 --- a/src/ServiceControl.Infrastructure/ServiceControlMeters.cs +++ b/src/ServiceControl.Infrastructure/ServiceControlMeters.cs @@ -1,12 +1,14 @@ namespace ServiceControl.Infrastructure; /// -/// The meters each instance publishes on. Shared because persisters publish onto the meter their -/// host has already registered with the exporter, and the two assemblies cannot reference each -/// other. +/// The meters each instance publishes on. These name the instance, not the subject: everything the +/// primary publishes shares , including audit ingestion when the primary hosts +/// it. What a measurement is about is carried by the instrument prefix instead. Shared because +/// persisters publish onto the meter their host has already registered with the exporter, and the +/// two assemblies cannot reference each other. /// public static class ServiceControlMeters { - public const string Error = "Particular.ServiceControl"; - public const string Audit = "Particular.ServiceControl.Audit"; + public const string Primary = "Particular.ServiceControl"; + public const string AuditInstance = "Particular.ServiceControl.Audit"; } diff --git a/src/ServiceControl.Infrastructure/TelemetryResourceBuilderExtensions.cs b/src/ServiceControl.Infrastructure/TelemetryResourceBuilderExtensions.cs new file mode 100644 index 0000000000..1770797694 --- /dev/null +++ b/src/ServiceControl.Infrastructure/TelemetryResourceBuilderExtensions.cs @@ -0,0 +1,72 @@ +namespace ServiceControl.Infrastructure; + +using System; +using System.Collections.Generic; +using OpenTelemetry.Resources; + +public static class TelemetryResourceBuilderExtensions +{ + /// + /// Identifies this process to the exporter, leaving the standard OpenTelemetry environment + /// variables in charge wherever an operator has set them. Scaled out workers read the same queue + /// and so share an instance name, so the host and process are reported as well, which tells a + /// pool apart on a dashboard without anyone having to configure it. + /// + public static ResourceBuilder AddServiceControlInstance(this ResourceBuilder resource, string instanceName, string instanceVersion) + { + // Read from the environment rather than IConfiguration, because these are read back by the + // SDK's own detector, which only looks at the environment. Taking an attribute as declared + // from a source the detector cannot see would suppress the value here and leave none at all. + var serviceName = Environment.GetEnvironmentVariable(ServiceNameVariable); + var resourceAttributes = Environment.GetEnvironmentVariable(ResourceAttributesVariable); + + resource.AddService( + serviceName: string.IsNullOrWhiteSpace(serviceName) ? instanceName : serviceName, + serviceVersion: instanceVersion, + autoGenerateServiceInstanceId: !Declares(resourceAttributes, ServiceInstanceIdAttribute)); + + var attributes = new Dictionary(); + + if (!Declares(resourceAttributes, HostNameAttribute)) + { + attributes[HostNameAttribute] = Environment.MachineName; + } + + if (!Declares(resourceAttributes, ProcessIdAttribute)) + { + attributes[ProcessIdAttribute] = (long)Environment.ProcessId; + } + + return attributes.Count > 0 ? resource.AddAttributes(attributes) : resource; + } + + /// + /// Whether OTEL_RESOURCE_ATTRIBUTES declares an attribute. Only presence is decided here: + /// the value stays with the SDK's detector, which owns the percent decoding. + /// + public static bool Declares(string resourceAttributes, string attributeName) + { + if (string.IsNullOrWhiteSpace(resourceAttributes)) + { + return false; + } + + foreach (var attribute in resourceAttributes.Split(',')) + { + var separator = attribute.IndexOf('='); + + if (separator > 0 && attribute.AsSpan(0, separator).Trim().SequenceEqual(attributeName)) + { + return true; + } + } + + return false; + } + + const string ServiceNameVariable = "OTEL_SERVICE_NAME"; + const string ResourceAttributesVariable = "OTEL_RESOURCE_ATTRIBUTES"; + const string ServiceInstanceIdAttribute = "service.instance.id"; + const string HostNameAttribute = "host.name"; + const string ProcessIdAttribute = "process.pid"; +} diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs index 700a4358d6..c51b748030 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs @@ -6,7 +6,7 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure.Metrics; public class RetentionMetrics { - public const string MeterName = ServiceControlMeters.Error; + public const string MeterName = ServiceControlMeters.Primary; public static readonly string CycleDurationInstrumentName = $"{InstrumentPrefix}.cycle_duration_seconds"; public static readonly string RowsDeletedInstrumentName = $"{InstrumentPrefix}.rows_deleted_total"; diff --git a/src/ServiceControl.Persistence/Recoverability/Archiving/Metrics/ArchiveMetrics.cs b/src/ServiceControl.Persistence/Recoverability/Archiving/Metrics/ArchiveMetrics.cs index 3ddd925186..4a2ac3e881 100644 --- a/src/ServiceControl.Persistence/Recoverability/Archiving/Metrics/ArchiveMetrics.cs +++ b/src/ServiceControl.Persistence/Recoverability/Archiving/Metrics/ArchiveMetrics.cs @@ -12,7 +12,7 @@ namespace ServiceControl.Recoverability.Archiving.Metrics; /// public class ArchiveMetrics { - public const string MeterName = ServiceControlMeters.Error; + public const string MeterName = ServiceControlMeters.Primary; public static readonly string OperationDurationInstrumentName = $"{InstrumentPrefix}.operation_duration_seconds"; public static readonly string BatchDurationInstrumentName = $"{InstrumentPrefix}.batch_duration_seconds"; diff --git a/src/ServiceControl.UnitTests/Infrastructure/TelemetryResourceBuilderExtensionsTests.cs b/src/ServiceControl.UnitTests/Infrastructure/TelemetryResourceBuilderExtensionsTests.cs new file mode 100644 index 0000000000..6814198e80 --- /dev/null +++ b/src/ServiceControl.UnitTests/Infrastructure/TelemetryResourceBuilderExtensionsTests.cs @@ -0,0 +1,39 @@ +namespace ServiceControl.UnitTests.Infrastructure; + +using NUnit.Framework; +using ServiceControl.Infrastructure; + +[TestFixture] +public class TelemetryResourceBuilderExtensionsTests +{ + [TestCase("service.instance.id=worker-1")] + [TestCase("deployment.environment.name=production,service.instance.id=worker-1")] + [TestCase("service.instance.id=worker-1,deployment.environment.name=production")] + [TestCase(" service.instance.id = worker-1 ")] + [TestCase("service.instance.id=")] + public void Finds_a_declared_attribute(string resourceAttributes) => + Assert.That(TelemetryResourceBuilderExtensions.Declares(resourceAttributes, "service.instance.id"), Is.True); + + [TestCase(null)] + [TestCase("")] + [TestCase(" ")] + [TestCase("deployment.environment.name=production")] + [TestCase("service.instance.identifier=worker-1")] + [TestCase("my.service.instance.id=worker-1")] + [TestCase("=service.instance.id")] + public void Leaves_an_undeclared_attribute_to_the_instance(string resourceAttributes) => + Assert.That(TelemetryResourceBuilderExtensions.Declares(resourceAttributes, "service.instance.id"), Is.False); + + [Test] + public void Tells_the_declared_attributes_apart() + { + const string resourceAttributes = "host.name=build-agent-3,process.pid=1234"; + + Assert.Multiple(() => + { + Assert.That(TelemetryResourceBuilderExtensions.Declares(resourceAttributes, "host.name"), Is.True); + Assert.That(TelemetryResourceBuilderExtensions.Declares(resourceAttributes, "process.pid"), Is.True); + Assert.That(TelemetryResourceBuilderExtensions.Declares(resourceAttributes, "service.instance.id"), Is.False); + }); + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs index bc6796d3d2..c1e0d6d458 100644 --- a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs @@ -7,6 +7,7 @@ namespace ServiceControl.Auditing.Metrics; using NServiceBus; using NServiceBus.Transport; using ServiceControl.EndpointPlugin.Messages.SagaState; +using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.Ingestion.Metrics; using ServiceControl.Operations.Metrics; @@ -18,13 +19,15 @@ namespace ServiceControl.Auditing.Metrics; /// public class AuditIngestionMetrics { + public const string MeterName = ServiceControlMeters.Primary; + public static readonly string BatchDurationInstrumentName = $"{InstrumentPrefix}.batch_duration_seconds"; public static readonly string MessageDurationInstrumentName = $"{InstrumentPrefix}.message_duration_seconds"; public static readonly string StorageDurationInstrumentName = $"{InstrumentPrefix}.storage_duration_seconds"; public AuditIngestionMetrics(IMeterFactory meterFactory) { - var meter = meterFactory.Create(IngestionMetrics.MeterName, MeterVersion); + var meter = meterFactory.Create(MeterName, MeterVersion); batchDuration = meter.CreateHistogram(BatchDurationInstrumentName, unit: "seconds", description: "Audit message batch processing duration in seconds"); ingestionDuration = meter.CreateHistogram(MessageDurationInstrumentName, unit: "seconds", description: "Audit message processing duration in seconds"); diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs index b3fecda7d3..da2af02b86 100644 --- a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs @@ -5,8 +5,8 @@ namespace ServiceControl.Auditing.Metrics; public static class AuditIngestionMetricsConfiguration { - // The meter is already added by the error ingestion configuration, which shares it. Only the - // audit instruments' bucket boundaries need declaring. + // Audit ingestion publishes on the primary instance's meter, which AddIngestionMetrics has + // already registered, so only the audit instruments' bucket boundaries need declaring. public static void AddAuditIngestionMetrics(this MeterProviderBuilder builder) { foreach (var instrumentName in DurationInstruments) diff --git a/src/ServiceControl/HostApplicationBuilderExtensions.cs b/src/ServiceControl/HostApplicationBuilderExtensions.cs index 0753d2d075..3f6715779b 100644 --- a/src/ServiceControl/HostApplicationBuilderExtensions.cs +++ b/src/ServiceControl/HostApplicationBuilderExtensions.cs @@ -169,18 +169,10 @@ public static void AddTelemetry(this IHostApplicationBuilder hostBuilder, Settin } hostBuilder.Services.AddOpenTelemetry() - .ConfigureResource(resource => resource.AddService( - serviceName: settings.InstanceName, - serviceVersion: InstanceVersion, - autoGenerateServiceInstanceId: true)) + .ConfigureResource(resource => resource.AddServiceControlInstance(settings.InstanceName, InstanceVersion)) .WithMetrics(metrics => { metrics.AddIngestionMetrics(); - - // Audit ingestion shares the meter, so only its instruments' views are added, - // and they are added whether or not this host ingests audit: a view for an - // instrument nobody records is inert, and making it conditional would tie the - // exporter's shape to which component happened to be registered. metrics.AddAuditIngestionMetrics(); metrics.AddAspNetCoreInstrumentation(); diff --git a/src/ServiceControl/Operations/Metrics/IngestionMetrics.cs b/src/ServiceControl/Operations/Metrics/IngestionMetrics.cs index ffbc5d63f7..84ca8c2786 100644 --- a/src/ServiceControl/Operations/Metrics/IngestionMetrics.cs +++ b/src/ServiceControl/Operations/Metrics/IngestionMetrics.cs @@ -10,7 +10,7 @@ public class IngestionMetrics { - public const string MeterName = ServiceControlMeters.Error; + public const string MeterName = ServiceControlMeters.Primary; public static readonly string BatchDurationInstrumentName = $"{InstrumentPrefix}.batch_duration_seconds"; public static readonly string MessageDurationInstrumentName = $"{InstrumentPrefix}.message_duration_seconds"; diff --git a/src/ServiceControl/Recoverability/Retrying/Metrics/RetryMetrics.cs b/src/ServiceControl/Recoverability/Retrying/Metrics/RetryMetrics.cs index 510dcfbd9c..f9d9f43298 100644 --- a/src/ServiceControl/Recoverability/Retrying/Metrics/RetryMetrics.cs +++ b/src/ServiceControl/Recoverability/Retrying/Metrics/RetryMetrics.cs @@ -11,7 +11,7 @@ namespace ServiceControl.Recoverability.Retrying.Metrics; public class RetryMetrics { - public const string MeterName = ServiceControlMeters.Error; + public const string MeterName = ServiceControlMeters.Primary; public static readonly string OperationDurationInstrumentName = $"{InstrumentPrefix}.operation_duration_seconds"; public static readonly string PrepareDurationInstrumentName = $"{InstrumentPrefix}.prepare_duration_seconds"; From 497cfbbedc8a30ede7ebb01a988e7faac51046aa Mon Sep 17 00:00:00 2001 From: John Simons Date: Sat, 22 Aug 2026 11:22:28 +1000 Subject: [PATCH 10/21] Add plan for EF audit persistence in the primary instance The implementation half of the audit-in-primary work. The hosting plan delivered contracts, runtime, settings and composition but nothing that stores an audit message; this covers the schema, partitioning, retention, body storage, search and queries needed to flip SupportsAuditIngestion on. Decisions settled by interview: - Audit tables join ServiceControlDbContext and the existing migration stream, with partitioning applied by raw SQL in a migration the way AddFullTextSearch already applies the GIN index. - created_on is the ingestion hour and part of the primary key, because PostgreSQL requires the partition key in every unique constraint. A redelivery across an hour boundary duplicates, which is stated rather than claimed away. - Audit bodies are keyed by hour so retention deletes a partition's bodies in one operation per store rather than one per message. - Full text search mirrors the error side rather than the spike's denormalised SearchableContent column. - Audit counts are an aggregate query, not a rollup table. - AuditRetentionPeriod defaults to 7 days when null. - The retention sweeper alone creates partitions, with a 48 hour lookahead and a custom check, since it is a single point of failure. - Retention takes a session scoped advisory lock, ported from the spike, so two misconfigured primaries cannot drop the same partition at once. The primary EF sweeper has never had one; the locks the spike wrote only ever existed in the standalone audit projects. The spike's insert-only KnownEndpoints table and its reconciler are not ported: the primary batch writer already dedupes endpoints in memory per batch and issues one insert-if-missing per distinct endpoint, and audit ingestion already routes through that same unit of work. SQL Server keeps the spike's batched deletes rather than partitioning, because a full text index cannot be aligned to a partition scheme. --- src/audit-ef-persistence-plan.md | 372 +++++++++++++++++++++++++++++++ 1 file changed, 372 insertions(+) create mode 100644 src/audit-ef-persistence-plan.md diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md new file mode 100644 index 0000000000..4ef175197a --- /dev/null +++ b/src/audit-ef-persistence-plan.md @@ -0,0 +1,372 @@ +# EF Audit Persistence for the Primary Instance + +## Summary + +Implement audit storage and querying in the existing primary EF persisters, so `SupportsAuditIngestion` +can be flipped to `true` and the audit runtime already hosted in the primary starts writing and +reading real data. + +This is the implementation half of [Host Audit Ingestion in the Primary Instance](audit-ingestion-in-primary-plan.md). +That plan delivered the contracts, the copied runtime, the settings, the fail-fast command and the +composition. Nothing in it stores an audit message. This plan does. + +The audit-only spike ([#5318](https://github.com/Particular/ServiceControl/pull/5318)) remains the +evidence base for partitioning, retention and full-text search. It is not the design: it targeted a +standalone audit instance with its own database, its own `DbContext` and its own persistence +contracts. Here the audit tables join the primary's model. + +## Goals + +- Store audit messages, saga snapshots and failed audit imports in the primary database. +- Serve the five `IMessagesViewDataStore` queries from a union of failed and audited messages. +- Serve audit counts, saga history, and audit body resolution. +- Keep the spike's PostgreSQL hourly range partitioning and its retention economics. +- Keep ingestion safe under competing consumers. +- Flip `SupportsAuditIngestion` to `true` on both EF manifests as the last step. + +## Non-goals + +- Any change to RavenDB, on either instance. +- Migrating existing RavenDB audit data. +- Changing the audit runtime, the host composition, or the settings surface. Those shipped already. +- Making `EnableFullTextSearchOnBodies` real. See "Full-text search". + +## Decisions + +Settled by interview on 22 Aug 2026. + +| Decision | Choice | +| --- | --- | +| Model boundary | Audit tables join `ServiceControlDbContext` and the existing per-provider migration stream. Partitioning is applied by raw SQL inside a migration, the way `AddFullTextSearch` already applies the GIN index. | +| Partition creation | The retention sweeper alone creates partitions ahead. Ingestion-only workers never issue DDL. | +| Audit body keys | `audit/{yyyy-MM-dd-HH}/{uniqueMessageId}`, with a new delete-by-prefix operation on `IBodyStoragePersistence`. | +| Full-text search | Mirror the error side: index the audit table's own columns through the existing `IFullTextSearchDialect` seam. No `SearchableContent` column. | +| Audit message identity | `created_on` is the ingestion hour. Primary key is `(created_on, id)` where `id` is the deterministic processing id, so a redelivery within the hour collapses. | +| Saga snapshot identity | Sequential `id`, no dedupe. A redelivered saga audit message produces a second snapshot. | +| Audit counts | Aggregate query over the audit table, served by an index on `(receiving_endpoint_name, created_on)`. No rollup table. | +| `AuditRetentionPeriod` null | 7 days, matching SCMU and the Dockerfile. | +| Retention ownership | Single owner via `RunRetentionSweep`, plus a session scoped advisory lock ported from the spike so two misconfigured primaries cannot sweep at once. | + +### Why the partition key is the ingestion hour + +PostgreSQL requires every unique constraint on a partitioned table to include the partition key, so +the row's identity and its partition are one decision. + +`created_on` is `UtcNow` truncated to the hour, as in the spike. A redelivery of the same audit +message within the hour collapses onto the same primary key. A redelivery that straddles an hour +boundary produces a second row. + +The alternative, partitioning on the message's own processing time, dedupes reliably but lets a late +or replayed message target a partition retention has already dropped. `--import-failed-audits` +replays old messages by design, so that failure mode is reachable in normal operation. The bounded +duplicate is the cheaper defect. + +State this in the acceptance criteria rather than claiming exact-once. + +### Partition creation is a single point of failure, knowingly + +Only the retention sweeper creates partitions, so an ingestion-only worker cannot insert into an hour +the sweeper never provisioned. If the sweeper's host is down longer than the lookahead, every worker +starts failing inserts. + +Mitigations, not fixes: + +- Provision a long lookahead (48 hours, against the spike's 6), so the outage has to be sustained. +- A custom check that fails when the newest provisioned partition is less than 12 hours ahead, so the + condition is visible before it bites. +- The setup command provisions the initial window, so a fresh instance ingests before the first sweep. + +Revisiting this means letting ingesting hosts issue `CREATE TABLE IF NOT EXISTS` themselves. + +## What already exists and is reused + +The largest risk in this work is rebuilding something the primary already has. It has more than the +spike did. + +| Capability | Where | Consequence | +| --- | --- | --- | +| Known endpoint recording | `FailedMessageBatchWriter.BuildEndpointRows` dedupes in memory per batch with `DistinctBy`, then `IFailedMessageIngestionSqlDialect.InsertMissingKnownEndpoints` issues one insert-if-missing per distinct endpoint | Nothing to do. The audit path already calls `Monitoring.RecordKnownEndpoint` on the same unit of work. The spike's insert-only table and its reconciler are not ported. | +| Provider-specific upserts | `IFailedMessageIngestionSqlDialect`, with `ON CONFLICT` and `MERGE WITH (HOLDLOCK)` implementations | Audit inserts extend this seam rather than inventing one. | +| Full-text search | `IFullTextSearchDialect`, `FullTextSearchSql`, the `AddFullTextSearch` migration, `FullTextSearchIndexTests` pinning query and index together | The audit index follows the identical pattern, including the test that fails when the two drift. | +| Body storage | `IBodyStoragePersistence` with FileSystem, AzureBlob and S3 implementations, plus installers | Audit bodies use the same store, and gain one new operation. | +| Body classification | `MessageBodyClassifier.Classify` decides inline text against external, honouring `MaxBodySizeToStore` and binary detection | Audit bodies reuse it unchanged. | +| Retention | `RetentionSweeper`, a `BackgroundService` gated by `RunRetentionSweep`, already deleting external bodies before rows | Audit retention extends it. | +| Transactional batch | `EFIngestionUnitOfWork.Complete` runs one execution strategy and one transaction | Audit rows join that transaction. | +| Body arbitration order | Documented on `IBodyStorage.TryFetch`, exercised by the audit-capable test persister | The EF implementation has to satisfy it, and the order is already stated. | + +## Schema + +Three new tables in the existing schema. Column naming follows each provider's existing convention +(`snake_case` on PostgreSQL, `PascalCase` on SQL Server), as the current entity configurations do. + +### AuditMessages + +Partitioned by range on `created_on` (PostgreSQL only). + +| Column | Notes | +| --- | --- | +| `created_on` | Ingestion time truncated to the hour. Partition key. | +| `id` | Deterministic processing id: `DeterministicGuid.MakeId(messageId, processingEndpoint, processingStarted)`, or a fresh Guid when any of those headers is absent. | +| `unique_message_id` | `headers.UniqueId()`. Indexed. Joins an audit row to its failed counterpart and to its body. | +| `message_id` | | +| `message_type` | | +| `time_sent` | | +| `processed_at` | From `Headers.ProcessingEnded`, best-guess `UtcNow` when absent. This is what `MessagesView.ProcessedAt` shows, not `created_on`. | +| `conversation_id` | | +| `is_system_message` | | +| `status` | `Successful` or `ResolvedSuccessfully`, from the `IsRetried` metadata. | +| `sending_endpoint_name`, `sending_endpoint_host_id`, `sending_endpoint_host` | | +| `receiving_endpoint_name`, `receiving_endpoint_host_id`, `receiving_endpoint_host` | | +| `critical_time_ticks`, `processing_time_ticks`, `delivery_time_ticks` | | +| `headers_json` | | +| `body_text` | Inline body, or null. Produced by `MessageBodyClassifier`. | +| `body_stored_externally` | | +| `body_size`, `body_content_type` | | +| `invoked_sagas_json`, `originates_from_saga_json` | `MessagesView` carries these and the enrichers already produce them. The spike dropped them, which would have regressed the message view. | + +Primary key `(created_on, id)`. Indexes on `(unique_message_id)`, `(receiving_endpoint_name, created_on)`, +`(conversation_id)`, `(processed_at)`, and the full-text index. + +### SagaSnapshots + +Partitioned identically. Maps `ServiceControl.SagaAudit.SagaSnapshot`, which does not move. + +`SagaSnapshotFactory` never assigns an id and RavenDB supplies a document id, so there is no natural +key to carry over. The id is sequential and snapshots are not deduplicated. + +The consequence, accepted knowingly: a redelivered saga audit message adds a second snapshot, which +appears as a duplicate step in the ServicePulse saga diagram. Deriving the id from the carrying +message would have deduplicated it; deriving it from saga content would have risked collapsing two +distinct state changes that share a `FinishTime` tick. A duplicate step is the more visible defect but +the recoverable one, and audit messages are not commonly redelivered. + +| Column | Notes | +| --- | --- | +| `created_on` | Partition key. | +| `id` | Sequential. Part of the primary key only because PostgreSQL requires the partition key in it. | +| `saga_id` | Indexed with `created_on` to serve saga history. | +| `saga_type`, `status`, `start_time`, `finish_time`, `endpoint` | | +| `state_after_change` | | +| `initiating_message_json`, `outgoing_messages_json` | | + +### FailedAuditImports + +Not partitioned. Mirrors `FailedErrorImportEntity`, including the inline-or-external body split and +the `ExternalBodyId` prefix convention. + +| Column | Notes | +| --- | --- | +| `unique_message_id` | Primary key. `FailedAuditImport.DeriveKey`, already shipped. | +| `failed_at`, `message_id`, `headers_json`, `body`, `body_stored_externally`, `exception_info` | | + +No retention, matching failed error imports. + +## Partitioning and retention + +`IAuditPartitionManager`, implemented per provider, with the operations the spike settled on: +ensure partitions exist, list expired partitions, drop a partition. + +### PostgreSQL + +Native `PARTITION BY RANGE (created_on)`, hourly partitions named `{table}_{yyyyMMddHH}`. Dropping an +expired hour is `DETACH` then `DROP TABLE`, a metadata operation. + +### SQL Server + +No partitioning. A full-text index cannot be aligned to a partition scheme, which makes partition +truncation impossible, and the spike removed the partitioning infrastructure it had first built +(commits 4448035a16 then 6dd611302d) because composite keys, aligned indexes and pausing ingestion +during cleanup were not worth it. `DropPartition` is a batched `DELETE` over the hour's range, +paced the way `SweepFailedMessages` already paces. + +This asymmetry is deliberate and load-tested. Do not re-add SQL Server partitioning without redoing +that work. + +### Sweeper integration + +`RetentionSweeper.Sweep` gains an audit pass, running after the existing error and event log passes: + +1. Ensure partitions exist for the next 48 hours. +2. For each expired hour: delete the body prefix, then drop the partition or delete the rows. + +Bodies before rows, matching the existing order and its reasoning: a crash in between leaves rows the +next sweep re-handles, where the reverse leaks bodies. + +`AuditRetentionPeriod` is read from `Settings` into `EFPersisterSettings`, defaulting to 7 days when +null. + +### Locking + +`RunRetentionSweep` is false on every ingestion-only worker, so in a correct deployment there is one +sweeper by construction. The lock exists for the incorrect deployment: two primaries both configured +to sweep. Concurrent batched deletes are merely wasteful, but two concurrent `DETACH`/`DROP TABLE` +against the same partition means the loser errors and abandons the rest of its pass, and an operator +running the additional primary processes this work enables is exactly who is likely to misconfigure +it. + +Port `TryAcquireLock`/`ReleaseLock` from the spike's `RetentionCleaner`, which already has both +implementations: + +- PostgreSQL: `SELECT pg_try_advisory_lock(hashtext('retention_cleaner'))`, released with + `pg_advisory_unlock`. +- SQL Server: `sp_getapplock` with `@LockMode = 'Exclusive'`, `@LockOwner = 'Session'` and + `@LockTimeout = 0`, released with `sp_releaseapplock`. + +The lifecycle is the spike's, unchanged: + +- A dedicated `DbConnection` is opened for the sweep and closed after it. Both locks are session + scoped, so a host that crashes mid-sweep releases the lock when its connection drops rather than + wedging retention until someone intervenes. +- Acquisition has a zero timeout. A sweeper that cannot take the lock logs and skips that pass + instead of queueing behind the holder, because the work is idempotent and hourly. +- Release is in a `finally`. + +`IRetentionLock` covers every sweep, not just the audit pass. Failed messages, event log items, +orphaned group comments and audit partitions are all taken under one acquisition. They already run +sequentially in a single `Sweep` method, so per pass locks would buy nothing, and the guarantee worth +stating is the simple one: at most one host is sweeping anything at any moment. + +This makes error and event log retention lock protected for the first time. That is a behaviour change +to shipped code, in the safe direction, and it is intended rather than incidental. + +`IRetentionLock` is a new per provider seam alongside `IAuditPartitionManager`. + +## Body storage + +Audit bodies are keyed `audit/{yyyy-MM-dd-HH}/{uniqueMessageId}`, where the hour is the row's +`created_on`. The hour in the key is what lets retention delete a partition's bodies in one operation +per provider rather than one per message. + +`IBodyStoragePersistence` gains: + +```csharp +Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default); +``` + +- FileSystem: delete the directory. +- AzureBlob: delete by blob prefix. +- S3: list and bulk delete by key prefix. + +`BodyStorage.TryFetch` implements the arbitration order `IBodyStorage` already documents. Step three +resolves the audit row by `unique_message_id`, returning the inline `body_text` where present and the +external body otherwise. Because the audit body key contains the hour, resolving it needs the row +first, which the query already fetches. + +## Full-text search + +The audit index mirrors `FullTextSearchSql` for failed messages: a GIN index over +`to_tsvector('simple', headers_json || body_text || message_type)` on PostgreSQL, a full-text catalog +on SQL Server, applied by migration because EF cannot model either. `IFullTextSearchDialect` gains an +audit overload, and the audit equivalent of `FullTextSearchIndexTests` pins the query expression to +the indexed expression. + +`EnableFullTextSearchOnBodies` stays unread, as it is on the error side today. Making it real is a +separate change that has to answer what happens to an existing index, and it should change both +sides at once or neither. + +## Query implementation + +### The five message view queries + +Each becomes a union of failed and audited rows, merged through `LocalMessagesView.Merge`, which +already implements the precedence, paging and counting rules. + +The merge is in memory over at most two pages of rows, not in SQL. Two ordered `Take(PageSize)` +queries, one per source, then merge. A SQL `UNION ALL` with a window function would push the work +into the database, but it defeats the full-text index on both providers and cannot express +"failed wins" without a second pass anyway. + +`GetAllMessagesByConversation` has no page bound in practice and needs a cap. + +### Audit counts + +`GROUP BY` over `created_on::date` filtered by `receiving_endpoint_name`, bounded by the retention +window, served by the `(receiving_endpoint_name, created_on)` index. Runs once a day per endpoint +from the licensing throughput collector, which is its only caller. + +### Saga history + +Select the snapshots for a saga id, ordered by `finish_time` descending, projected into +`SagaHistory.Changes`. Bounded by a cap: a long-running saga can have thousands of snapshots. + +## Ingestion write path + +`EFIngestionUnitOfWork.Audit` stops returning null and returns an `EFAuditIngestionUnitOfWork` that +buffers into thread-safe collections, exactly as the recoverability child does. `Complete` writes +audit rows inside the existing transaction, after the failed message upsert and before the commit, +so a batch's audit rows and its known endpoints commit together. + +Inserts go through a new `IAuditIngestionSqlDialect`: + +- PostgreSQL: `INSERT ... ON CONFLICT (created_on, id) DO NOTHING`. +- SQL Server: `MERGE ... WITH (HOLDLOCK) ... WHEN NOT MATCHED THEN INSERT`. + +`DO NOTHING` rather than an update: an audit row is immutable once written, and a redelivery carries +identical content. + +External body writes are queued on the existing `RecordBodyWrite` path, so they complete before the +rows that point at them. + +## Scale-out and idempotency + +- Audit inserts are insert-if-missing on a deterministic key, so competing consumers converge. +- Known endpoint upserts are unchanged and already safe. +- Failed audit imports key on `FailedAuditImport.DeriveKey`, already shipped, so a poison message + produces one row rather than one per attempt per worker. +- Body writes are idempotent: same key, immutable content. +- Only the retention owner issues DDL or deletes, and the advisory lock holds that to one host even + when two are configured to sweep. + +Two known gaps, stated rather than hidden: + +- A redelivery that crosses an hour boundary produces a second audit row. +- A redelivered saga audit message always produces a second saga snapshot, because snapshots carry no + deduplication key. + +## PR sequence + +1. **Schema and migrations.** Entities, configurations, `DbContext` registration, per-provider + migrations including the partitioning and full-text SQL. No behaviour: nothing writes or reads yet, + and the manifests still say false. +2. **Ingestion write path.** `IAuditIngestionSqlDialect`, `EFAuditIngestionUnitOfWork`, wiring + `EFIngestionUnitOfWork.Audit`. Tested through the persistence test base. +3. **Retention, partitions and locking.** `IAuditPartitionManager` and `IRetentionLock` per provider, + the sweeper's audit pass, the body prefix delete on all three body stores, the lookahead custom + check. +4. **Queries.** The five message view unions, audit counts, saga history, and the third step of body + arbitration. +5. **Failed audit imports.** The store, and the `--import-failed-audits` round trip. +6. **Turn it on.** Flip `SupportsAuditIngestion` in both manifests, update the approval test that + asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable` along with the + `Empty*` audit data stores it stood in for. Full acceptance runs on both providers. + +Each pull request leaves both EF acceptance suites and the RavenDB suites passing, and steps 1 to 5 +leave `SupportsAuditIngestion` false so nothing activates early. + +## Testing + +- `ServiceControl.Persistence.Tests.SqlServer` and `.PostgreSql` cover the write path, retention, + partition lifecycle and the queries against real databases. +- The full-text index tests are duplicated for audit, pinning query expression to index expression on + both providers. +- Partition lifecycle: partitions created ahead, expired partitions dropped, bodies for the dropped + hour gone, and rows in live partitions untouched. +- Retention locking: a second sweeper against the same database skips its pass rather than failing, + and a dropped lock connection frees the lock for the next sweep. +- Redelivery of an audit message within an hour produces one row. Redelivery across an hour boundary + produces two, and a redelivered saga audit message always produces two. All three are asserted + rather than left to chance. +- The precedence, paging and counting rules from `IMessagesViewDataStore`, now against real SQL rather + than the in-memory test persister. +- The acceptance tests from the hosting plan keep running, and step 6 makes them run against a real + audit-capable persister for the first time. + +## Open items + +1. What caps `GetAllMessagesByConversation` and saga history, and what the API returns when a cap is hit. +2. Whether the audit path should raise the `EndpointDetected` domain event. Carried over from the + hosting plan, still unanswered, and now cheap to settle because the write path is real. +3. Whether `SagaUpdatedHandler` should hand the snapshot straight to the audit unit of work instead of + forwarding it to the audit queue. Also carried over. +4. Whether the 48 hour partition lookahead and the 12 hour custom check threshold are the right + numbers, which is a question for whoever runs the load tests. From 694bd892ff7d538f63565a87b26571a3c5e9ead0 Mon Sep 17 00:00:00 2001 From: John Simons Date: Sat, 22 Aug 2026 19:08:32 +1000 Subject: [PATCH 11/21] Add the audit schema to the primary EF persisters First step of EF audit persistence. Schema only: nothing writes or reads these tables yet, and both manifests still declare SupportsAuditIngestion false, so no shipped instance changes behaviour. Ingestion is the hot path and reads are not, so the schema is shaped to make writes cheap and let queries pay. - AuditMessages, SagaSnapshots and FailedAuditImports join the existing ServiceControlDbContext and the per-provider migration stream, rather than getting a context of their own, so an ingestion batch can commit audit rows and known endpoints in one transaction. - On PostgreSQL the two audit tables are range partitioned on created_on. EF cannot express declarative partitioning and PostgreSQL cannot convert a table in place, so the migration clones each table with LIKE INCLUDING ALL after the columns and indexes are created. Cloning rather than hand writing the DDL keeps the definition owned by the entity model. - Both audit keys are a bigint identity, and exist only because a partitioned table's primary key must include the partition key and created_on alone is not unique. Nothing reads them back. Rows are plain inserts with no conflict clause, so neither table deduplicates and a redelivered message produces a second row. RavenDB does deduplicate, so this is a behaviour difference and not only a scale-out caveat. - Columns exist only where the read path filters, sorts, indexes or searches. InvokedSagas and OriginatesFromSaga get none, because InvokedSagasParser derives them from headers alone. - ServiceControl.Persistence.EFCore now references the saga audit project explicitly, since transitive project references are disabled there. The full text index is not here. SQL Server requires a single column unique KEY INDEX and the audit key is composite, and the indexed expression has to be written with the query expression it must match, so both land with the queries. --- .../AuditPartitioningSql.cs | 38 + ...260822095934_AddAuditIngestion.Designer.cs | 1125 +++++++++++++++++ .../20260822095934_AddAuditIngestion.cs | 144 +++ ...SqlServiceControlDbContextModelSnapshot.cs | 238 ++++ ...260822095936_AddAuditIngestion.Designer.cs | 896 +++++++++++++ .../20260822095936_AddAuditIngestion.cs | 138 ++ ...verServiceControlDbContextModelSnapshot.cs | 185 +++ .../DbContexts/ServiceControlDbContext.cs | 6 + .../Entities/AuditMessageEntity.cs | 68 + .../Entities/FailedAuditImportEntity.cs | 22 + .../Entities/SagaSnapshotEntity.cs | 38 + .../AuditMessageConfiguration.cs | 42 + .../FailedAuditImportConfiguration.cs | 23 + .../SagaSnapshotConfiguration.cs | 27 + .../ServiceControl.Persistence.EFCore.csproj | 1 + .../AuditPartitioningTests.cs | 51 + .../EFCore/AuditSchemaTests.cs | 33 + src/audit-ef-persistence-plan.md | 108 +- 18 files changed, 3143 insertions(+), 40 deletions(-) create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Entities/AuditMessageEntity.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Entities/FailedAuditImportEntity.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Entities/SagaSnapshotEntity.cs create mode 100644 src/ServiceControl.Persistence.EFCore/EntityConfigurations/AuditMessageConfiguration.cs create mode 100644 src/ServiceControl.Persistence.EFCore/EntityConfigurations/FailedAuditImportConfiguration.cs create mode 100644 src/ServiceControl.Persistence.EFCore/EntityConfigurations/SagaSnapshotConfiguration.cs create mode 100644 src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs new file mode 100644 index 0000000000..859f05b3b5 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs @@ -0,0 +1,38 @@ +namespace ServiceControl.Persistence.EFCore.PostgreSql; + +using System.Linq; + +/// +/// Converts the two audit tables to range partitioned on created_on. EF Core cannot express +/// declarative partitioning and PostgreSQL cannot convert a plain table in place, so the table is +/// cloned into a partitioned one. +/// +/// +/// The clone is LIKE ... INCLUDING ALL rather than a hand written column list, so the columns, +/// primary key and indexes stay whatever EF generated and cannot drift from the model. This runs +/// after the CreateTable and CreateIndex calls in the migration, so there is something to clone. +/// A partitioned table's primary key must include the partition key, which is why the audit keys are +/// composite. +/// +static class AuditPartitioningSql +{ + static readonly string[] Tables = ["audit_messages", "saga_snapshots"]; + + public static string PartitionTables() => Convert(partitioned: true); + + public static string UnpartitionTables() => Convert(partitioned: false); + + static string Convert(bool partitioned) + { + var partitionBy = partitioned ? " PARTITION BY RANGE (created_on)" : string.Empty; + + return string.Concat(Tables.Select(table => + $""" + CREATE TABLE {table}_tmp (LIKE {table} INCLUDING ALL); + DROP TABLE {table}; + CREATE TABLE {table} (LIKE {table}_tmp INCLUDING ALL){partitionBy}; + DROP TABLE {table}_tmp; + + """)); + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs new file mode 100644 index 0000000000..8da30f5839 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs @@ -0,0 +1,1125 @@ +// +using System; +using System.Collections.Generic; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; +using ServiceControl.Persistence.EFCore.PostgreSql; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations +{ + [DbContext(typeof(PostgreSqlServiceControlDbContext))] + [Migration("20260822095934_AddAuditIngestion")] + partial class AddAuditIngestion + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ArchiveOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("request_id"); + + b.Property("ArchiveType") + .HasColumnType("integer") + .HasColumnName("archive_type"); + + b.Property("OperationType") + .HasColumnType("integer") + .HasColumnName("operation_type"); + + b.Property("CurrentBatch") + .HasColumnType("integer") + .HasColumnName("current_batch"); + + b.Property("GroupName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("group_name"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_id"); + + b.Property("InitiatedByName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_name"); + + b.Property("NumberOfBatches") + .HasColumnType("integer") + .HasColumnName("number_of_batches"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("operation_id"); + + b.Property("Started") + .HasColumnType("timestamp with time zone") + .HasColumnName("started"); + + b.Property("TotalNumberOfMessages") + .HasColumnType("integer") + .HasColumnName("total_number_of_messages"); + + b.HasKey("RequestId", "ArchiveType", "OperationType") + .HasName("pk_archive_operations"); + + b.HasIndex("Started") + .HasDatabaseName("ix_archive_operations_started"); + + b.ToTable("archive_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.AuditMessageEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_on"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("body_content_type"); + + b.Property("BodySize") + .HasColumnType("integer") + .HasColumnName("body_size"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("BodyText") + .HasColumnType("text") + .HasColumnName("body_text"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("conversation_id"); + + b.Property("CriticalTimeTicks") + .HasColumnType("bigint") + .HasColumnName("critical_time_ticks"); + + b.Property("DeliveryTimeTicks") + .HasColumnType("bigint") + .HasColumnName("delivery_time_ticks"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("IsSystemMessage") + .HasColumnType("boolean") + .HasColumnName("is_system_message"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.Property("MessageType") + .HasColumnType("text") + .HasColumnName("message_type"); + + b.Property("ProcessedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("processed_at"); + + b.Property("ProcessingTimeTicks") + .HasColumnType("bigint") + .HasColumnName("processing_time_ticks"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_host"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("receiving_endpoint_host_id"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_name"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_host"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("sending_endpoint_host_id"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_name"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.Property("TimeSent") + .HasColumnType("timestamp with time zone") + .HasColumnName("time_sent"); + + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.HasKey("CreatedOn", "Id") + .HasName("pk_audit_messages"); + + b.HasIndex("ConversationId") + .HasDatabaseName("ix_audit_messages_conversation_id"); + + b.HasIndex("ProcessedAt") + .HasDatabaseName("ix_audit_messages_processed_at"); + + b.HasIndex("TimeSent") + .HasDatabaseName("ix_audit_messages_time_sent"); + + b.HasIndex("UniqueMessageId") + .HasDatabaseName("ix_audit_messages_unique_message_id"); + + b.HasIndex("ReceivingEndpointName", "CreatedOn") + .HasDatabaseName("ix_audit_messages_receiving_endpoint_name_created_on"); + + b.ToTable("audit_messages", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Category") + .IsRequired() + .HasColumnType("text") + .HasColumnName("category"); + + b.Property("CustomCheckId") + .IsRequired() + .HasColumnType("text") + .HasColumnName("custom_check_id"); + + b.Property("FailureReason") + .HasColumnType("text") + .HasColumnName("failure_reason"); + + b.Property("OriginatingEndpointHost") + .IsRequired() + .HasColumnType("text") + .HasColumnName("originating_endpoint_host"); + + b.Property("OriginatingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("originating_endpoint_host_id"); + + b.Property("OriginatingEndpointName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("originating_endpoint_name"); + + b.Property("ReportedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("reported_at"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("Id") + .HasName("pk_custom_checks"); + + b.HasIndex("ReportedAt") + .HasDatabaseName("ix_custom_checks_reported_at"); + + b.HasIndex("Status", "ReportedAt") + .HasDatabaseName("ix_custom_checks_status_reported_at"); + + b.ToTable("custom_checks", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EndpointSettingsEntity", b => + { + b.Property("Name") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("name"); + + b.Property("TrackInstances") + .HasColumnType("boolean") + .HasColumnName("track_instances"); + + b.HasKey("Name") + .HasName("pk_endpoint_settings"); + + b.ToTable("endpoint_settings", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EventLogItemEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Category") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("category"); + + b.Property("Description") + .IsRequired() + .HasColumnType("text") + .HasColumnName("description"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("event_type"); + + b.Property("RaisedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("raised_at"); + + b.PrimitiveCollection>("RelatedTo") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("related_to"); + + b.Property("Severity") + .HasColumnType("integer") + .HasColumnName("severity"); + + b.HasKey("Id") + .HasName("pk_event_log_items"); + + b.HasIndex("RaisedAt", "Id") + .IsDescending() + .HasDatabaseName("ix_event_log_items_raised_at_id"); + + b.ToTable("EventLogItems", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("Body") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("body"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("text") + .HasColumnName("exception_info"); + + b.Property("FailedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("failed_at"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_audit_imports"); + + b.HasIndex("FailedAt") + .HasDatabaseName("ix_failed_audit_imports_failed_at"); + + b.ToTable("failed_audit_imports", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("Body") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("body"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("text") + .HasColumnName("exception_info"); + + b.Property("FailedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("failed_at"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_error_imports"); + + b.HasIndex("FailedAt") + .HasDatabaseName("ix_failed_error_imports_failed_at"); + + b.ToTable("failed_error_imports", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEditEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("EditId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("edit_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_message_edits"); + + b.HasIndex("EditId") + .HasDatabaseName("ix_failed_message_edits_edit_id"); + + b.ToTable("failed_message_edits", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("body_content_type"); + + b.Property("BodySize") + .HasColumnType("integer") + .HasColumnName("body_size"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("BodyText") + .HasColumnType("text") + .HasColumnName("body_text"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("conversation_id"); + + b.Property("ExceptionMessage") + .HasColumnType("text") + .HasColumnName("exception_message"); + + b.Property("ExceptionType") + .HasColumnType("text") + .HasColumnName("exception_type"); + + b.Property("FailingEndpointAddress") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("failing_endpoint_address"); + + b.Property("FirstTimeOfFailure") + .HasColumnType("timestamp with time zone") + .HasColumnName("first_time_of_failure"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("IsSystemMessage") + .HasColumnType("boolean") + .HasColumnName("is_system_message"); + + b.Property("LastAttemptedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_attempted_at"); + + b.Property("LastModified") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_modified"); + + b.Property("LastTimeOfFailure") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_time_of_failure"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.Property("MessageType") + .HasColumnType("text") + .HasColumnName("message_type"); + + b.Property("NumberOfProcessingAttempts") + .HasColumnType("integer") + .HasColumnName("number_of_processing_attempts"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_host"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("receiving_endpoint_host_id"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_name"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_host"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("sending_endpoint_host_id"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_name"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.Property("StatusChangedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("status_changed_at"); + + b.Property("TimeSent") + .HasColumnType("timestamp with time zone") + .HasColumnName("time_sent"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_messages"); + + b.HasIndex("ConversationId") + .HasDatabaseName("ix_failed_messages_conversation_id"); + + b.HasIndex("FailingEndpointAddress") + .HasDatabaseName("ix_failed_messages_failing_endpoint_address"); + + b.HasIndex("ReceivingEndpointName") + .HasDatabaseName("ix_failed_messages_receiving_endpoint_name"); + + b.HasIndex("StatusChangedAt") + .HasDatabaseName("ix_failed_messages_status_changed_at") + .HasFilter("status IN (2, 4)"); + + b.HasIndex("TimeSent") + .HasDatabaseName("ix_failed_messages_time_sent"); + + b.HasIndex("Status", "LastModified") + .HasDatabaseName("ix_failed_messages_status_last_modified"); + + b.ToTable("failed_messages", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.Property("FailedMessageUniqueId") + .HasColumnType("uuid") + .HasColumnName("failed_message_unique_id"); + + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("group_id"); + + b.Property("Title") + .IsRequired() + .HasColumnType("text") + .HasColumnName("title"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("character varying(255)") + .HasColumnName("type"); + + b.HasKey("FailedMessageUniqueId", "GroupId") + .HasName("pk_failed_message_groups"); + + b.HasIndex("GroupId") + .HasDatabaseName("ix_failed_message_groups_group_id"); + + b.HasIndex("Type", "GroupId") + .HasDatabaseName("ix_failed_message_groups_type_group_id"); + + b.ToTable("failed_message_groups", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageRetryEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("RetryBatchId") + .HasColumnType("uuid") + .HasColumnName("retry_batch_id"); + + b.Property("StageAttempts") + .HasColumnType("integer") + .HasColumnName("stage_attempts"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_message_retries"); + + b.HasIndex("RetryBatchId") + .HasDatabaseName("ix_failed_message_retries_retry_batch_id"); + + b.ToTable("failed_message_retries", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.GroupCommentEntity", b => + { + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("group_id"); + + b.Property("Comment") + .IsRequired() + .HasColumnType("text") + .HasColumnName("comment"); + + b.HasKey("GroupId") + .HasName("pk_group_comments"); + + b.ToTable("group_comments", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.HistoricRetryOperationEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CompletionTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("completion_time"); + + b.Property("Failed") + .HasColumnType("boolean") + .HasColumnName("failed"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("request_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.HasKey("Id") + .HasName("pk_historic_retry_operations"); + + b.HasIndex("CompletionTime", "Id") + .IsDescending() + .HasDatabaseName("ix_historic_retry_operations_completion_time_id"); + + b.ToTable("historic_retry_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.KnownEndpointEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("host"); + + b.Property("HostId") + .HasColumnType("uuid") + .HasColumnName("host_id"); + + b.Property("Monitored") + .HasColumnType("boolean") + .HasColumnName("monitored"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("name"); + + b.HasKey("Id") + .HasName("pk_known_endpoints"); + + b.ToTable("known_endpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("normalized_name"); + + b.Property("ThroughputSource") + .HasColumnType("integer") + .HasColumnName("throughput_source"); + + b.PrimitiveCollection>("EndpointIndicators") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("endpoint_indicators"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("name"); + + b.Property("NormalizedSanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("normalized_sanitized_name"); + + b.Property("SanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sanitized_name"); + + b.Property("Scope") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("scope"); + + b.Property("UserIndicator") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("user_indicator"); + + b.HasKey("NormalizedName", "ThroughputSource") + .HasName("pk_licensing_endpoints"); + + b.HasIndex("NormalizedSanitizedName") + .HasDatabaseName("ix_licensing_endpoints_normalized_sanitized_name"); + + b.ToTable("licensing_endpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("normalized_name"); + + b.Property("ThroughputSource") + .HasColumnType("integer") + .HasColumnName("throughput_source"); + + b.Property("DateUtc") + .HasColumnType("date") + .HasColumnName("date_utc"); + + b.Property("MessageCount") + .HasColumnType("bigint") + .HasColumnName("message_count"); + + b.HasKey("NormalizedName", "ThroughputSource", "DateUtc") + .HasName("pk_licensing_endpoint_throughput"); + + b.HasIndex("DateUtc") + .HasDatabaseName("ix_licensing_endpoint_throughput_date_utc"); + + b.ToTable("licensing_endpoint_throughput", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MessageRedirectEntity", b => + { + b.Property("FromPhysicalAddress") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("from_physical_address"); + + b.Property("LastModified") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_modified"); + + b.Property("ToPhysicalAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("to_physical_address"); + + b.HasKey("FromPhysicalAddress") + .HasName("pk_message_redirects"); + + b.ToTable("message_redirects", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Classifier") + .HasColumnType("text") + .HasColumnName("classifier"); + + b.Property("Context") + .HasColumnType("text") + .HasColumnName("context"); + + b.Property("InitialBatchSize") + .HasColumnType("integer") + .HasColumnName("initial_batch_size"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_id"); + + b.Property("InitiatedByName") + .HasColumnType("text") + .HasColumnName("initiated_by_name"); + + b.Property("Last") + .HasColumnType("timestamp with time zone") + .HasColumnName("last"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("operation_id"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("request_id"); + + b.Property("RetrySessionId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("retry_session_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("StagingId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("staging_id"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("Id") + .HasName("pk_retry_batches"); + + b.HasIndex("Status", "RetrySessionId") + .HasDatabaseName("ix_retry_batches_status_retry_session_id"); + + b.ToTable("retry_batches", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchNowForwardingEntity", b => + { + b.Property("Id") + .HasColumnType("integer") + .HasColumnName("id"); + + b.Property("RetryBatchId") + .HasColumnType("uuid") + .HasColumnName("retry_batch_id"); + + b.HasKey("Id") + .HasName("pk_retry_batch_now_forwarding"); + + b.ToTable("retry_batch_now_forwarding", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SagaSnapshotEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_on"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Endpoint") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("endpoint"); + + b.Property("FinishTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("finish_time"); + + b.Property("InitiatingMessageJson") + .HasColumnType("text") + .HasColumnName("initiating_message_json"); + + b.Property("OutgoingMessagesJson") + .HasColumnType("text") + .HasColumnName("outgoing_messages_json"); + + b.Property("ProcessedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("processed_at"); + + b.Property("SagaId") + .HasColumnType("uuid") + .HasColumnName("saga_id"); + + b.Property("SagaType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("saga_type"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.Property("StateAfterChange") + .HasColumnType("text") + .HasColumnName("state_after_change"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("CreatedOn", "Id") + .HasName("pk_saga_snapshots"); + + b.HasIndex("SagaId", "FinishTime") + .HasDatabaseName("ix_saga_snapshots_saga_id_finish_time"); + + b.ToTable("saga_snapshots", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => + { + b.Property("Key") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("key"); + + b.Property("Value") + .IsRequired() + .HasColumnType("text") + .HasColumnName("value"); + + b.HasKey("Key") + .HasName("pk_settings"); + + b.ToTable("settings", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SubscriptionEntity", b => + { + b.Property("MessageType") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("message_type"); + + b.Property("TransportAddress") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("transport_address"); + + b.Property("Endpoint") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("endpoint"); + + b.HasKey("MessageType", "TransportAddress") + .HasName("pk_subscriptions"); + + b.ToTable("subscriptions", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.UnacknowledgedRetryOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("request_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("Classifier") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("classifier"); + + b.Property("CompletionTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("completion_time"); + + b.Property("Failed") + .HasColumnType("boolean") + .HasColumnName("failed"); + + b.Property("Last") + .HasColumnType("timestamp with time zone") + .HasColumnName("last"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.HasKey("RequestId", "RetryType") + .HasName("pk_unacknowledged_retry_operations"); + + b.ToTable("unacknowledged_retry_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", null) + .WithMany() + .HasForeignKey("FailedMessageUniqueId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_failed_message_groups_failed_messages_failed_message_unique"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", null) + .WithMany() + .HasForeignKey("NormalizedName", "ThroughputSource") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_licensing_endpoint_throughput_licensing_endpoints_normalize"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs new file mode 100644 index 0000000000..f3fbd04c52 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs @@ -0,0 +1,144 @@ +using System; +using Microsoft.EntityFrameworkCore.Migrations; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations +{ + /// + public partial class AddAuditIngestion : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateTable( + name: "audit_messages", + columns: table => new + { + created_on = table.Column(type: "timestamp with time zone", nullable: false), + id = table.Column(type: "bigint", nullable: false) + .Annotation("Npgsql:ValueGenerationStrategy", NpgsqlValueGenerationStrategy.IdentityByDefaultColumn), + unique_message_id = table.Column(type: "uuid", nullable: false), + message_id = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + message_type = table.Column(type: "text", nullable: true), + time_sent = table.Column(type: "timestamp with time zone", nullable: true), + processed_at = table.Column(type: "timestamp with time zone", nullable: false), + conversation_id = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + is_system_message = table.Column(type: "boolean", nullable: false), + status = table.Column(type: "integer", nullable: false), + sending_endpoint_name = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + sending_endpoint_host_id = table.Column(type: "uuid", nullable: true), + sending_endpoint_host = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + receiving_endpoint_name = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + receiving_endpoint_host_id = table.Column(type: "uuid", nullable: true), + receiving_endpoint_host = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + critical_time_ticks = table.Column(type: "bigint", nullable: true), + processing_time_ticks = table.Column(type: "bigint", nullable: true), + delivery_time_ticks = table.Column(type: "bigint", nullable: true), + headers_json = table.Column(type: "text", nullable: false), + body_text = table.Column(type: "text", nullable: true), + body_stored_externally = table.Column(type: "boolean", nullable: false), + body_size = table.Column(type: "integer", nullable: false), + body_content_type = table.Column(type: "character varying(450)", maxLength: 450, nullable: true) + }, + constraints: table => + { + table.PrimaryKey("pk_audit_messages", x => new { x.created_on, x.id }); + }); + + migrationBuilder.CreateTable( + name: "failed_audit_imports", + columns: table => new + { + unique_message_id = table.Column(type: "uuid", nullable: false), + failed_at = table.Column(type: "timestamp with time zone", nullable: false), + message_id = table.Column(type: "character varying(450)", maxLength: 450, nullable: false), + headers_json = table.Column(type: "text", nullable: false), + body = table.Column(type: "bytea", nullable: false), + body_stored_externally = table.Column(type: "boolean", nullable: false), + exception_info = table.Column(type: "text", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("pk_failed_audit_imports", x => x.unique_message_id); + }); + + migrationBuilder.CreateTable( + name: "saga_snapshots", + columns: table => new + { + created_on = table.Column(type: "timestamp with time zone", nullable: false), + id = table.Column(type: "bigint", nullable: false) + .Annotation("Npgsql:ValueGenerationStrategy", NpgsqlValueGenerationStrategy.IdentityByDefaultColumn), + saga_id = table.Column(type: "uuid", nullable: false), + saga_type = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + status = table.Column(type: "integer", nullable: false), + start_time = table.Column(type: "timestamp with time zone", nullable: false), + finish_time = table.Column(type: "timestamp with time zone", nullable: false), + processed_at = table.Column(type: "timestamp with time zone", nullable: false), + endpoint = table.Column(type: "character varying(450)", maxLength: 450, nullable: true), + state_after_change = table.Column(type: "text", nullable: true), + initiating_message_json = table.Column(type: "text", nullable: true), + outgoing_messages_json = table.Column(type: "text", nullable: true) + }, + constraints: table => + { + table.PrimaryKey("pk_saga_snapshots", x => new { x.created_on, x.id }); + }); + + migrationBuilder.CreateIndex( + name: "ix_audit_messages_conversation_id", + table: "audit_messages", + column: "conversation_id"); + + migrationBuilder.CreateIndex( + name: "ix_audit_messages_processed_at", + table: "audit_messages", + column: "processed_at"); + + migrationBuilder.CreateIndex( + name: "ix_audit_messages_receiving_endpoint_name_created_on", + table: "audit_messages", + columns: new[] { "receiving_endpoint_name", "created_on" }); + + migrationBuilder.CreateIndex( + name: "ix_audit_messages_time_sent", + table: "audit_messages", + column: "time_sent"); + + migrationBuilder.CreateIndex( + name: "ix_audit_messages_unique_message_id", + table: "audit_messages", + column: "unique_message_id"); + + migrationBuilder.CreateIndex( + name: "ix_failed_audit_imports_failed_at", + table: "failed_audit_imports", + column: "failed_at"); + + migrationBuilder.CreateIndex( + name: "ix_saga_snapshots_saga_id_finish_time", + table: "saga_snapshots", + columns: new[] { "saga_id", "finish_time" }); + + // Last, so that the clone picks up the columns, primary keys and indexes above. + migrationBuilder.Sql(AuditPartitioningSql.PartitionTables()); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(AuditPartitioningSql.UnpartitionTables()); + + migrationBuilder.DropTable( + name: "audit_messages"); + + migrationBuilder.DropTable( + name: "failed_audit_imports"); + + migrationBuilder.DropTable( + name: "saga_snapshots"); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs index 06b1a9b639..82abe8a3a6 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/PostgreSqlServiceControlDbContextModelSnapshot.cs @@ -87,6 +87,136 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("archive_operations", (string)null); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.AuditMessageEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_on"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("body_content_type"); + + b.Property("BodySize") + .HasColumnType("integer") + .HasColumnName("body_size"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("BodyText") + .HasColumnType("text") + .HasColumnName("body_text"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("conversation_id"); + + b.Property("CriticalTimeTicks") + .HasColumnType("bigint") + .HasColumnName("critical_time_ticks"); + + b.Property("DeliveryTimeTicks") + .HasColumnType("bigint") + .HasColumnName("delivery_time_ticks"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("IsSystemMessage") + .HasColumnType("boolean") + .HasColumnName("is_system_message"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.Property("MessageType") + .HasColumnType("text") + .HasColumnName("message_type"); + + b.Property("ProcessedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("processed_at"); + + b.Property("ProcessingTimeTicks") + .HasColumnType("bigint") + .HasColumnName("processing_time_ticks"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_host"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("receiving_endpoint_host_id"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_name"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_host"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("sending_endpoint_host_id"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_name"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.Property("TimeSent") + .HasColumnType("timestamp with time zone") + .HasColumnName("time_sent"); + + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.HasKey("CreatedOn", "Id") + .HasName("pk_audit_messages"); + + b.HasIndex("ConversationId") + .HasDatabaseName("ix_audit_messages_conversation_id"); + + b.HasIndex("ProcessedAt") + .HasDatabaseName("ix_audit_messages_processed_at"); + + b.HasIndex("TimeSent") + .HasDatabaseName("ix_audit_messages_time_sent"); + + b.HasIndex("UniqueMessageId") + .HasDatabaseName("ix_audit_messages_unique_message_id"); + + b.HasIndex("ReceivingEndpointName", "CreatedOn") + .HasDatabaseName("ix_audit_messages_receiving_endpoint_name_created_on"); + + b.ToTable("audit_messages", (string)null); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => { b.Property("Id") @@ -233,6 +363,50 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("external_integration_dispatch_requests", (string)null); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("Body") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("body"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("text") + .HasColumnName("exception_info"); + + b.Property("FailedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("failed_at"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_audit_imports"); + + b.HasIndex("FailedAt") + .HasDatabaseName("ix_failed_audit_imports_failed_at"); + + b.ToTable("failed_audit_imports", (string)null); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => { b.Property("UniqueMessageId") @@ -809,6 +983,70 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("retry_batch_now_forwarding", (string)null); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SagaSnapshotEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_on"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Endpoint") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("endpoint"); + + b.Property("FinishTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("finish_time"); + + b.Property("InitiatingMessageJson") + .HasColumnType("text") + .HasColumnName("initiating_message_json"); + + b.Property("OutgoingMessagesJson") + .HasColumnType("text") + .HasColumnName("outgoing_messages_json"); + + b.Property("ProcessedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("processed_at"); + + b.Property("SagaId") + .HasColumnType("uuid") + .HasColumnName("saga_id"); + + b.Property("SagaType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("saga_type"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.Property("StateAfterChange") + .HasColumnType("text") + .HasColumnName("state_after_change"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("CreatedOn", "Id") + .HasName("pk_saga_snapshots"); + + b.HasIndex("SagaId", "FinishTime") + .HasDatabaseName("ix_saga_snapshots_saga_id_finish_time"); + + b.ToTable("saga_snapshots", (string)null); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => { b.Property("Key") diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs new file mode 100644 index 0000000000..c73ad641a0 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs @@ -0,0 +1,896 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ServiceControl.Persistence.EFCore.SqlServer; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations +{ + [DbContext(typeof(SqlServerServiceControlDbContext))] + [Migration("20260822095936_AddAuditIngestion")] + partial class AddAuditIngestion + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ArchiveOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("ArchiveType") + .HasColumnType("int"); + + b.Property("OperationType") + .HasColumnType("int"); + + b.Property("CurrentBatch") + .HasColumnType("int"); + + b.Property("GroupName") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("InitiatedByName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("NumberOfBatches") + .HasColumnType("int"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Started") + .HasColumnType("datetime2"); + + b.Property("TotalNumberOfMessages") + .HasColumnType("int"); + + b.HasKey("RequestId", "ArchiveType", "OperationType"); + + b.HasIndex("Started"); + + b.ToTable("ArchiveOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.AuditMessageEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("datetime2"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("BodySize") + .HasColumnType("int"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("BodyText") + .HasColumnType("nvarchar(max)"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("CriticalTimeTicks") + .HasColumnType("bigint"); + + b.Property("DeliveryTimeTicks") + .HasColumnType("bigint"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("IsSystemMessage") + .HasColumnType("bit"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("MessageType") + .HasColumnType("nvarchar(max)"); + + b.Property("ProcessedAt") + .HasColumnType("datetime2"); + + b.Property("ProcessingTimeTicks") + .HasColumnType("bigint"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Status") + .HasColumnType("int"); + + b.Property("TimeSent") + .HasColumnType("datetime2"); + + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.HasKey("CreatedOn", "Id"); + + b.HasIndex("ConversationId"); + + b.HasIndex("ProcessedAt"); + + b.HasIndex("TimeSent"); + + b.HasIndex("UniqueMessageId"); + + b.HasIndex("ReceivingEndpointName", "CreatedOn"); + + b.ToTable("AuditMessages"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Category") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CustomCheckId") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailureReason") + .HasColumnType("nvarchar(max)"); + + b.Property("OriginatingEndpointHost") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("OriginatingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginatingEndpointName") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ReportedAt") + .HasColumnType("datetime2"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ReportedAt"); + + b.HasIndex("Status", "ReportedAt"); + + b.ToTable("CustomChecks"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EndpointSettingsEntity", b => + { + b.Property("Name") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("TrackInstances") + .HasColumnType("bit"); + + b.HasKey("Name"); + + b.ToTable("EndpointSettings"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EventLogItemEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Category") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Description") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RaisedAt") + .HasColumnType("datetime2"); + + b.PrimitiveCollection("RelatedTo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Severity") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("RaisedAt", "Id") + .IsDescending(); + + b.ToTable("EventLogItems", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("Body") + .IsRequired() + .HasColumnType("varbinary(max)"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailedAt") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("FailedAt"); + + b.ToTable("FailedAuditImports"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("Body") + .IsRequired() + .HasColumnType("varbinary(max)"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailedAt") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("FailedAt"); + + b.ToTable("FailedErrorImports"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEditEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("EditId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("EditId"); + + b.ToTable("FailedMessageEdits"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("BodySize") + .HasColumnType("int"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("BodyText") + .HasColumnType("nvarchar(max)"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ExceptionMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("ExceptionType") + .HasColumnType("nvarchar(max)"); + + b.Property("FailingEndpointAddress") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("FirstTimeOfFailure") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("IsSystemMessage") + .HasColumnType("bit"); + + b.Property("LastAttemptedAt") + .HasColumnType("datetime2"); + + b.Property("LastModified") + .HasColumnType("datetime2"); + + b.Property("LastTimeOfFailure") + .HasColumnType("datetime2"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("MessageType") + .HasColumnType("nvarchar(max)"); + + b.Property("NumberOfProcessingAttempts") + .HasColumnType("int"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Status") + .HasColumnType("int"); + + b.Property("StatusChangedAt") + .HasColumnType("datetime2"); + + b.Property("TimeSent") + .HasColumnType("datetime2"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("ConversationId"); + + b.HasIndex("FailingEndpointAddress"); + + b.HasIndex("ReceivingEndpointName"); + + b.HasIndex("StatusChangedAt") + .HasFilter("[Status] IN (2, 4)"); + + b.HasIndex("TimeSent"); + + b.HasIndex("Status", "LastModified"); + + b.ToTable("FailedMessages"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.Property("FailedMessageUniqueId") + .HasColumnType("uniqueidentifier"); + + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Title") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("nvarchar(255)"); + + b.HasKey("FailedMessageUniqueId", "GroupId"); + + b.HasIndex("GroupId"); + + b.HasIndex("Type", "GroupId"); + + b.ToTable("FailedMessageGroups"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageRetryEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("RetryBatchId") + .HasColumnType("uniqueidentifier"); + + b.Property("StageAttempts") + .HasColumnType("int"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("RetryBatchId"); + + b.ToTable("FailedMessageRetries"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.GroupCommentEntity", b => + { + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Comment") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.HasKey("GroupId"); + + b.ToTable("GroupComments"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.HistoricRetryOperationEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletionTime") + .HasColumnType("datetime2"); + + b.Property("Failed") + .HasColumnType("bit"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.HasKey("Id"); + + b.HasIndex("CompletionTime", "Id") + .IsDescending(); + + b.ToTable("HistoricRetryOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.KnownEndpointEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("HostId") + .HasColumnType("uniqueidentifier"); + + b.Property("Monitored") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("Id"); + + b.ToTable("KnownEndpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("ThroughputSource") + .HasColumnType("int"); + + b.PrimitiveCollection("EndpointIndicators") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("NormalizedSanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Scope") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("UserIndicator") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("NormalizedName", "ThroughputSource"); + + b.HasIndex("NormalizedSanitizedName"); + + b.ToTable("LicensingEndpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("ThroughputSource") + .HasColumnType("int"); + + b.Property("DateUtc") + .HasColumnType("date"); + + b.Property("MessageCount") + .HasColumnType("bigint"); + + b.HasKey("NormalizedName", "ThroughputSource", "DateUtc"); + + b.HasIndex("DateUtc"); + + b.ToTable("LicensingEndpointThroughput"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MessageRedirectEntity", b => + { + b.Property("FromPhysicalAddress") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("LastModified") + .HasColumnType("datetime2"); + + b.Property("ToPhysicalAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("FromPhysicalAddress"); + + b.ToTable("MessageRedirects"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Classifier") + .HasColumnType("nvarchar(max)"); + + b.Property("Context") + .HasColumnType("nvarchar(max)"); + + b.Property("InitialBatchSize") + .HasColumnType("int"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("InitiatedByName") + .HasColumnType("nvarchar(max)"); + + b.Property("Last") + .HasColumnType("datetime2"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RetrySessionId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("StagingId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Status", "RetrySessionId"); + + b.ToTable("RetryBatches"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchNowForwardingEntity", b => + { + b.Property("Id") + .HasColumnType("int"); + + b.Property("RetryBatchId") + .HasColumnType("uniqueidentifier"); + + b.HasKey("Id"); + + b.ToTable("RetryBatchNowForwarding"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SagaSnapshotEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("datetime2"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Endpoint") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("FinishTime") + .HasColumnType("datetime2"); + + b.Property("InitiatingMessageJson") + .HasColumnType("nvarchar(max)"); + + b.Property("OutgoingMessagesJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ProcessedAt") + .HasColumnType("datetime2"); + + b.Property("SagaId") + .HasColumnType("uniqueidentifier"); + + b.Property("SagaType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.Property("StateAfterChange") + .HasColumnType("nvarchar(max)"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("CreatedOn", "Id"); + + b.HasIndex("SagaId", "FinishTime"); + + b.ToTable("SagaSnapshots"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => + { + b.Property("Key") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Value") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.HasKey("Key"); + + b.ToTable("Settings"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SubscriptionEntity", b => + { + b.Property("MessageType") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TransportAddress") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Endpoint") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("MessageType", "TransportAddress"); + + b.ToTable("Subscriptions"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.UnacknowledgedRetryOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("Classifier") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("CompletionTime") + .HasColumnType("datetime2"); + + b.Property("Failed") + .HasColumnType("bit"); + + b.Property("Last") + .HasColumnType("datetime2"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.HasKey("RequestId", "RetryType"); + + b.ToTable("UnacknowledgedRetryOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", null) + .WithMany() + .HasForeignKey("FailedMessageUniqueId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", null) + .WithMany() + .HasForeignKey("NormalizedName", "ThroughputSource") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.cs new file mode 100644 index 0000000000..e593d8400e --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.cs @@ -0,0 +1,138 @@ +using System; +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations +{ + /// + public partial class AddAuditIngestion : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateTable( + name: "AuditMessages", + columns: table => new + { + CreatedOn = table.Column(type: "datetime2", nullable: false), + Id = table.Column(type: "bigint", nullable: false) + .Annotation("SqlServer:Identity", "1, 1"), + UniqueMessageId = table.Column(type: "uniqueidentifier", nullable: false), + MessageId = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + MessageType = table.Column(type: "nvarchar(max)", nullable: true), + TimeSent = table.Column(type: "datetime2", nullable: true), + ProcessedAt = table.Column(type: "datetime2", nullable: false), + ConversationId = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + IsSystemMessage = table.Column(type: "bit", nullable: false), + Status = table.Column(type: "int", nullable: false), + SendingEndpointName = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + SendingEndpointHostId = table.Column(type: "uniqueidentifier", nullable: true), + SendingEndpointHost = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + ReceivingEndpointName = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + ReceivingEndpointHostId = table.Column(type: "uniqueidentifier", nullable: true), + ReceivingEndpointHost = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + CriticalTimeTicks = table.Column(type: "bigint", nullable: true), + ProcessingTimeTicks = table.Column(type: "bigint", nullable: true), + DeliveryTimeTicks = table.Column(type: "bigint", nullable: true), + HeadersJson = table.Column(type: "nvarchar(max)", nullable: false), + BodyText = table.Column(type: "nvarchar(max)", nullable: true), + BodyStoredExternally = table.Column(type: "bit", nullable: false), + BodySize = table.Column(type: "int", nullable: false), + BodyContentType = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true) + }, + constraints: table => + { + table.PrimaryKey("PK_AuditMessages", x => new { x.CreatedOn, x.Id }); + }); + + migrationBuilder.CreateTable( + name: "FailedAuditImports", + columns: table => new + { + UniqueMessageId = table.Column(type: "uniqueidentifier", nullable: false), + FailedAt = table.Column(type: "datetime2", nullable: false), + MessageId = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: false), + HeadersJson = table.Column(type: "nvarchar(max)", nullable: false), + Body = table.Column(type: "varbinary(max)", nullable: false), + BodyStoredExternally = table.Column(type: "bit", nullable: false), + ExceptionInfo = table.Column(type: "nvarchar(max)", nullable: false) + }, + constraints: table => + { + table.PrimaryKey("PK_FailedAuditImports", x => x.UniqueMessageId); + }); + + migrationBuilder.CreateTable( + name: "SagaSnapshots", + columns: table => new + { + CreatedOn = table.Column(type: "datetime2", nullable: false), + Id = table.Column(type: "bigint", nullable: false) + .Annotation("SqlServer:Identity", "1, 1"), + SagaId = table.Column(type: "uniqueidentifier", nullable: false), + SagaType = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + Status = table.Column(type: "int", nullable: false), + StartTime = table.Column(type: "datetime2", nullable: false), + FinishTime = table.Column(type: "datetime2", nullable: false), + ProcessedAt = table.Column(type: "datetime2", nullable: false), + Endpoint = table.Column(type: "nvarchar(450)", maxLength: 450, nullable: true), + StateAfterChange = table.Column(type: "nvarchar(max)", nullable: true), + InitiatingMessageJson = table.Column(type: "nvarchar(max)", nullable: true), + OutgoingMessagesJson = table.Column(type: "nvarchar(max)", nullable: true) + }, + constraints: table => + { + table.PrimaryKey("PK_SagaSnapshots", x => new { x.CreatedOn, x.Id }); + }); + + migrationBuilder.CreateIndex( + name: "IX_AuditMessages_ConversationId", + table: "AuditMessages", + column: "ConversationId"); + + migrationBuilder.CreateIndex( + name: "IX_AuditMessages_ProcessedAt", + table: "AuditMessages", + column: "ProcessedAt"); + + migrationBuilder.CreateIndex( + name: "IX_AuditMessages_ReceivingEndpointName_CreatedOn", + table: "AuditMessages", + columns: new[] { "ReceivingEndpointName", "CreatedOn" }); + + migrationBuilder.CreateIndex( + name: "IX_AuditMessages_TimeSent", + table: "AuditMessages", + column: "TimeSent"); + + migrationBuilder.CreateIndex( + name: "IX_AuditMessages_UniqueMessageId", + table: "AuditMessages", + column: "UniqueMessageId"); + + migrationBuilder.CreateIndex( + name: "IX_FailedAuditImports_FailedAt", + table: "FailedAuditImports", + column: "FailedAt"); + + migrationBuilder.CreateIndex( + name: "IX_SagaSnapshots_SagaId_FinishTime", + table: "SagaSnapshots", + columns: new[] { "SagaId", "FinishTime" }); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropTable( + name: "AuditMessages"); + + migrationBuilder.DropTable( + name: "FailedAuditImports"); + + migrationBuilder.DropTable( + name: "SagaSnapshots"); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs index b9e41a3681..4f0cb9df61 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/SqlServerServiceControlDbContextModelSnapshot.cs @@ -72,6 +72,106 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("ArchiveOperations"); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.AuditMessageEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("datetime2"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("BodySize") + .HasColumnType("int"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("BodyText") + .HasColumnType("nvarchar(max)"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("CriticalTimeTicks") + .HasColumnType("bigint"); + + b.Property("DeliveryTimeTicks") + .HasColumnType("bigint"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("IsSystemMessage") + .HasColumnType("bit"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("MessageType") + .HasColumnType("nvarchar(max)"); + + b.Property("ProcessedAt") + .HasColumnType("datetime2"); + + b.Property("ProcessingTimeTicks") + .HasColumnType("bigint"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Status") + .HasColumnType("int"); + + b.Property("TimeSent") + .HasColumnType("datetime2"); + + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.HasKey("CreatedOn", "Id"); + + b.HasIndex("ConversationId"); + + b.HasIndex("ProcessedAt"); + + b.HasIndex("TimeSent"); + + b.HasIndex("UniqueMessageId"); + + b.HasIndex("ReceivingEndpointName", "CreatedOn"); + + b.ToTable("AuditMessages"); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => { b.Property("Id") @@ -190,6 +290,41 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("ExternalIntegrationDispatchRequests"); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("Body") + .IsRequired() + .HasColumnType("varbinary(max)"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailedAt") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("FailedAt"); + + b.ToTable("FailedAuditImports"); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => { b.Property("UniqueMessageId") @@ -646,6 +781,56 @@ protected override void BuildModel(ModelBuilder modelBuilder) b.ToTable("RetryBatchNowForwarding"); }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SagaSnapshotEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("datetime2"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Endpoint") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("FinishTime") + .HasColumnType("datetime2"); + + b.Property("InitiatingMessageJson") + .HasColumnType("nvarchar(max)"); + + b.Property("OutgoingMessagesJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ProcessedAt") + .HasColumnType("datetime2"); + + b.Property("SagaId") + .HasColumnType("uniqueidentifier"); + + b.Property("SagaType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.Property("StateAfterChange") + .HasColumnType("nvarchar(max)"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("CreatedOn", "Id"); + + b.HasIndex("SagaId", "FinishTime"); + + b.ToTable("SagaSnapshots"); + }); + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => { b.Property("Key") diff --git a/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs b/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs index 3717c6a7f3..77a53e3475 100644 --- a/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs +++ b/src/ServiceControl.Persistence.EFCore/DbContexts/ServiceControlDbContext.cs @@ -24,6 +24,9 @@ public abstract class ServiceControlDbContext(DbContextOptions options) : DbCont public DbSet RetryBatchNowForwarding { get; set; } public DbSet FailedMessageRetries { get; set; } public DbSet FailedErrorImports { get; set; } + public DbSet AuditMessages { get; set; } + public DbSet SagaSnapshots { get; set; } + public DbSet FailedAuditImports { get; set; } public DbSet Settings { get; set; } public DbSet Subscriptions { get; set; } public DbSet EventLogItems { get; set; } @@ -49,6 +52,8 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) modelBuilder.ApplyConfiguration(new CustomCheckConfiguration()); modelBuilder.ApplyConfiguration(new EndpointSettingsConfiguration()); + modelBuilder.ApplyConfiguration(new AuditMessageConfiguration()); + modelBuilder.ApplyConfiguration(new FailedAuditImportConfiguration()); modelBuilder.ApplyConfiguration(new FailedErrorImportConfiguration()); modelBuilder.ApplyConfiguration(new FailedMessageConfiguration()); modelBuilder.ApplyConfiguration(new FailedMessageEditConfiguration()); @@ -60,6 +65,7 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) modelBuilder.ApplyConfiguration(new RetryBatchNowForwardingConfiguration()); modelBuilder.ApplyConfiguration(new KnownEndpointConfiguration()); modelBuilder.ApplyConfiguration(new SubscriptionConfiguration()); + modelBuilder.ApplyConfiguration(new SagaSnapshotConfiguration()); modelBuilder.ApplyConfiguration(new SettingConfiguration()); modelBuilder.ApplyConfiguration(new EventLogItemConfiguration()); modelBuilder.ApplyConfiguration(new HistoricRetryOperationConfiguration()); diff --git a/src/ServiceControl.Persistence.EFCore/Entities/AuditMessageEntity.cs b/src/ServiceControl.Persistence.EFCore/Entities/AuditMessageEntity.cs new file mode 100644 index 0000000000..0092d518b1 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Entities/AuditMessageEntity.cs @@ -0,0 +1,68 @@ +namespace ServiceControl.Persistence.EFCore.Entities; + +using ServiceControl.Persistence; + +public class AuditMessageEntity +{ + /// + /// Ingestion time truncated to the hour. The PostgreSQL partition key, and therefore part of the + /// primary key, because a partitioned table's unique constraints must include it. + /// + public DateTime CreatedOn { get; set; } + + /// + /// A database generated identity. Audit rows are immutable inserts and nothing reads this back, so + /// it exists only because a partitioned table's primary key must include the partition key and + /// created_on alone is not unique. Rows are not deduplicated, so a redelivered audit message + /// produces a second row. + /// + public long Id { get; set; } + + public Guid UniqueMessageId { get; set; } + + public string? MessageId { get; set; } + + public string? MessageType { get; set; } + + public DateTime? TimeSent { get; set; } + + /// + /// What the message view reports as ProcessedAt, taken from the ProcessingEnded header. Unrelated + /// to , which is when this instance happened to ingest it. + /// + public DateTime ProcessedAt { get; set; } + + public string? ConversationId { get; set; } + + public bool IsSystemMessage { get; set; } + + public MessageStatus Status { get; set; } + + public string? SendingEndpointName { get; set; } + + public Guid? SendingEndpointHostId { get; set; } + + public string? SendingEndpointHost { get; set; } + + public string? ReceivingEndpointName { get; set; } + + public Guid? ReceivingEndpointHostId { get; set; } + + public string? ReceivingEndpointHost { get; set; } + + public long? CriticalTimeTicks { get; set; } + + public long? ProcessingTimeTicks { get; set; } + + public long? DeliveryTimeTicks { get; set; } + + public required string HeadersJson { get; set; } + + public string? BodyText { get; set; } + + public bool BodyStoredExternally { get; set; } + + public int BodySize { get; set; } + + public string? BodyContentType { get; set; } +} diff --git a/src/ServiceControl.Persistence.EFCore/Entities/FailedAuditImportEntity.cs b/src/ServiceControl.Persistence.EFCore/Entities/FailedAuditImportEntity.cs new file mode 100644 index 0000000000..d8e9956860 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Entities/FailedAuditImportEntity.cs @@ -0,0 +1,22 @@ +namespace ServiceControl.Persistence.EFCore.Entities; + +public class FailedAuditImportEntity +{ + public Guid UniqueMessageId { get; set; } + + public DateTime FailedAt { get; set; } + + public required string MessageId { get; set; } + + public required string HeadersJson { get; set; } + + // Holds the inline body, or an empty array when the body was spilled to external storage. + // BodyStoredExternally, not the contents here, decides where the body lives. + public required byte[] Body { get; set; } + + public bool BodyStoredExternally { get; set; } + + public required string ExceptionInfo { get; set; } + + public static string ExternalBodyId(Guid uniqueMessageId) => $"failedauditimport-{uniqueMessageId}"; +} diff --git a/src/ServiceControl.Persistence.EFCore/Entities/SagaSnapshotEntity.cs b/src/ServiceControl.Persistence.EFCore/Entities/SagaSnapshotEntity.cs new file mode 100644 index 0000000000..95f5bc654b --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Entities/SagaSnapshotEntity.cs @@ -0,0 +1,38 @@ +namespace ServiceControl.Persistence.EFCore.Entities; + +using ServiceControl.SagaAudit; + +public class SagaSnapshotEntity +{ + /// + /// Ingestion time truncated to the hour. The PostgreSQL partition key. + /// + public DateTime CreatedOn { get; set; } + + /// + /// A database generated identity. Snapshots carry no natural key and nothing reads this back, so + /// it exists only because a partitioned table's primary key must include the partition key and + /// created_on alone is not unique. A redelivered saga audit message produces a second row. + /// + public long Id { get; set; } + + public Guid SagaId { get; set; } + + public string? SagaType { get; set; } + + public SagaStateChangeStatus Status { get; set; } + + public DateTime StartTime { get; set; } + + public DateTime FinishTime { get; set; } + + public DateTime ProcessedAt { get; set; } + + public string? Endpoint { get; set; } + + public string? StateAfterChange { get; set; } + + public string? InitiatingMessageJson { get; set; } + + public string? OutgoingMessagesJson { get; set; } +} diff --git a/src/ServiceControl.Persistence.EFCore/EntityConfigurations/AuditMessageConfiguration.cs b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/AuditMessageConfiguration.cs new file mode 100644 index 0000000000..44eea7a858 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/AuditMessageConfiguration.cs @@ -0,0 +1,42 @@ +namespace ServiceControl.Persistence.EFCore.EntityConfigurations; + +using Entities; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Metadata.Builders; + +class AuditMessageConfiguration : IEntityTypeConfiguration +{ + public void Configure(EntityTypeBuilder builder) + { + builder.HasKey(e => new { e.CreatedOn, e.Id }); + builder.Property(e => e.CreatedOn).ValueGeneratedNever(); + builder.Property(e => e.Id).ValueGeneratedOnAdd(); + + builder.Property(e => e.UniqueMessageId).IsRequired(); + builder.Property(e => e.ProcessedAt).IsRequired(); + builder.Property(e => e.IsSystemMessage).IsRequired(); + builder.Property(e => e.Status).IsRequired(); + builder.Property(e => e.HeadersJson).IsRequired(); + builder.Property(e => e.BodyStoredExternally).IsRequired(); + builder.Property(e => e.BodySize).IsRequired(); + + builder.Property(e => e.MessageId).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.ConversationId).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.SendingEndpointName).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.SendingEndpointHost).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.ReceivingEndpointName).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.ReceivingEndpointHost).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.BodyContentType).HasMaxLength(ColumnLengths.ShortTextLength); + + // Resolves an audit row from a failed message's id, and is what the third step of the body + // arbitration order looks up. + builder.HasIndex(e => e.UniqueMessageId); + + // Serves both the per-endpoint message queries and the daily audit counts. + builder.HasIndex(e => new { e.ReceivingEndpointName, e.CreatedOn }); + + builder.HasIndex(e => e.ConversationId); + builder.HasIndex(e => e.ProcessedAt); + builder.HasIndex(e => e.TimeSent); + } +} diff --git a/src/ServiceControl.Persistence.EFCore/EntityConfigurations/FailedAuditImportConfiguration.cs b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/FailedAuditImportConfiguration.cs new file mode 100644 index 0000000000..17acae2a4d --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/FailedAuditImportConfiguration.cs @@ -0,0 +1,23 @@ +namespace ServiceControl.Persistence.EFCore.EntityConfigurations; + +using Entities; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Metadata.Builders; + +class FailedAuditImportConfiguration : IEntityTypeConfiguration +{ + public void Configure(EntityTypeBuilder builder) + { + builder.HasKey(e => e.UniqueMessageId); + builder.Property(e => e.UniqueMessageId).ValueGeneratedNever(); + + builder.Property(e => e.FailedAt).IsRequired(); + builder.Property(e => e.MessageId).IsRequired().HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.HeadersJson).IsRequired(); + builder.Property(e => e.Body).IsRequired(); + builder.Property(e => e.BodyStoredExternally).IsRequired(); + builder.Property(e => e.ExceptionInfo).IsRequired(); + + builder.HasIndex(e => e.FailedAt); + } +} diff --git a/src/ServiceControl.Persistence.EFCore/EntityConfigurations/SagaSnapshotConfiguration.cs b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/SagaSnapshotConfiguration.cs new file mode 100644 index 0000000000..86d7478160 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/EntityConfigurations/SagaSnapshotConfiguration.cs @@ -0,0 +1,27 @@ +namespace ServiceControl.Persistence.EFCore.EntityConfigurations; + +using Entities; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Metadata.Builders; + +class SagaSnapshotConfiguration : IEntityTypeConfiguration +{ + public void Configure(EntityTypeBuilder builder) + { + builder.HasKey(e => new { e.CreatedOn, e.Id }); + builder.Property(e => e.CreatedOn).ValueGeneratedNever(); + builder.Property(e => e.Id).ValueGeneratedOnAdd(); + + builder.Property(e => e.SagaId).IsRequired(); + builder.Property(e => e.Status).IsRequired(); + builder.Property(e => e.StartTime).IsRequired(); + builder.Property(e => e.FinishTime).IsRequired(); + builder.Property(e => e.ProcessedAt).IsRequired(); + + builder.Property(e => e.SagaType).HasMaxLength(ColumnLengths.ShortTextLength); + builder.Property(e => e.Endpoint).HasMaxLength(ColumnLengths.ShortTextLength); + + // Saga history is looked up by saga id and ordered by finish time. + builder.HasIndex(e => new { e.SagaId, e.FinishTime }); + } +} diff --git a/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj b/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj index 9ecbf964d0..f12ece860f 100644 --- a/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj +++ b/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj @@ -13,6 +13,7 @@ + diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs new file mode 100644 index 0000000000..3c833eae2a --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs @@ -0,0 +1,51 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.DbContexts; + +/// +/// The migration converts the audit tables to partitioned ones by cloning them, which succeeds even +/// if the PARTITION BY clause is lost. Nothing else notices until retention tries to drop a partition +/// that was never created, so the conversion is asserted directly. +/// +class AuditPartitioningTests : PersistenceTestBase +{ + [TestCase("audit_messages")] + [TestCase("saga_snapshots")] + public async Task Table_is_range_partitioned_on_created_on(string table) + { + using var scope = ServiceProvider.GetRequiredService().CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + var strategy = await QueryScalar(dbContext, + $"SELECT partstrat::text FROM pg_partitioned_table WHERE partrelid = '{table}'::regclass"); + + var partitionKey = await QueryScalar(dbContext, + $""" + SELECT a.attname + FROM pg_partitioned_table p + JOIN pg_attribute a ON a.attrelid = p.partrelid AND a.attnum = p.partattrs[0] + WHERE p.partrelid = '{table}'::regclass + """); + + using (Assert.EnterMultipleScope()) + { + Assert.That(strategy, Is.EqualTo("r"), $"{table} is not range partitioned"); + Assert.That(partitionKey, Is.EqualTo("created_on")); + } + } + + static async Task QueryScalar(ServiceControlDbContext dbContext, string sql) + { + var connection = dbContext.Database.GetDbConnection(); + await dbContext.Database.OpenConnectionAsync(); + + await using var command = connection.CreateCommand(); + command.CommandText = sql; + + return (await command.ExecuteScalarAsync())?.ToString(); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs new file mode 100644 index 0000000000..2c4e9ff017 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs @@ -0,0 +1,33 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.DbContexts; + +class AuditSchemaTests : PersistenceTestBase +{ + [Test] + public void The_applied_schema_matches_the_model() + { + using var scope = ServiceProvider.GetRequiredService().CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + Assert.That(dbContext.Database.HasPendingModelChanges(), Is.False, + "the audit tables are created by hand written DDL so that PostgreSQL can partition them, " + + "and their columns have drifted from the entity model. Update AuditPartitioningSql to match."); + } + + [Test] + public async Task Audit_tables_accept_and_return_a_row() + { + using var scope = ServiceProvider.GetRequiredService().CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + Assert.That(await dbContext.AuditMessages.AnyAsync(), Is.False); + Assert.That(await dbContext.SagaSnapshots.AnyAsync(), Is.False); + Assert.That(await dbContext.FailedAuditImports.AnyAsync(), Is.False); + } +} diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 4ef175197a..8dbfb88136 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -15,9 +15,22 @@ evidence base for partitioning, retention and full-text search. It is not the de standalone audit instance with its own database, its own `DbContext` and its own persistence contracts. Here the audit tables join the primary's model. +## The governing trade + +Ingestion is the hot path and reads are not. Audit volume is orders of magnitude above error volume, +every message is written once and read rarely, and a slow write path backs up onto the broker where a +slow query only makes one person wait. + +So: nothing goes on the write path that can be moved off it, and where the two conflict the query pays. +That is what settles the shape of most of this plan. No upsert, no conflict probe, no rollup table +maintained at ingestion, no extra index that only a query wants, and a database generated id rather +than one derived per message. The consequences are read side and accepted: duplicate rows on +redelivery, and counts and unions computed at query time. + ## Goals -- Store audit messages, saga snapshots and failed audit imports in the primary database. +- Store audit messages, saga snapshots and failed audit imports in the primary database, with the + cheapest write path that will do. - Serve the five `IMessagesViewDataStore` queries from a union of failed and audited messages. - Serve audit counts, saga history, and audit body resolution. - Keep the spike's PostgreSQL hourly range partitioning and its retention economics. @@ -41,27 +54,30 @@ Settled by interview on 22 Aug 2026. | Partition creation | The retention sweeper alone creates partitions ahead. Ingestion-only workers never issue DDL. | | Audit body keys | `audit/{yyyy-MM-dd-HH}/{uniqueMessageId}`, with a new delete-by-prefix operation on `IBodyStoragePersistence`. | | Full-text search | Mirror the error side: index the audit table's own columns through the existing `IFullTextSearchDialect` seam. No `SearchableContent` column. | -| Audit message identity | `created_on` is the ingestion hour. Primary key is `(created_on, id)` where `id` is the deterministic processing id, so a redelivery within the hour collapses. | -| Saga snapshot identity | Sequential `id`, no dedupe. A redelivered saga audit message produces a second snapshot. | +| Audit message identity | `created_on` is the ingestion hour. Primary key is `(created_on, id)` where `id` is a `bigint` identity. Rows are plain inserts and are not deduplicated. | +| Saga snapshot identity | A `bigint` database identity, no dedupe. A redelivered saga audit message produces a second snapshot. | | Audit counts | Aggregate query over the audit table, served by an index on `(receiving_endpoint_name, created_on)`. No rollup table. | | `AuditRetentionPeriod` null | 7 days, matching SCMU and the Dockerfile. | | Retention ownership | Single owner via `RunRetentionSweep`, plus a session scoped advisory lock ported from the spike so two misconfigured primaries cannot sweep at once. | -### Why the partition key is the ingestion hour +### Why the partition key is the ingestion hour, and why rows are not deduplicated PostgreSQL requires every unique constraint on a partitioned table to include the partition key, so -the row's identity and its partition are one decision. +the row's identity and its partition are one decision. `created_on` is `UtcNow` truncated to the hour, +as in the spike. -`created_on` is `UtcNow` truncated to the hour, as in the spike. A redelivery of the same audit -message within the hour collapses onto the same primary key. A redelivery that straddles an hour -boundary produces a second row. +The ingestion path is a plain multi-row `INSERT` with a database generated id. There is no upsert, +which means a redelivered audit message produces a second row. -The alternative, partitioning on the message's own processing time, dedupes reliably but lets a late -or replayed message target a partition retention has already dropped. `--import-failed-audits` -replays old messages by design, so that failure mode is reachable in normal operation. The bounded -duplicate is the cheaper defect. +This is a deliberate regression against the standalone RavenDB audit instance, which does deduplicate: +`RavenAuditIngestionUnitOfWork` bulk inserts with the deterministic document id +`ProcessedMessages-{ticks}-{ProcessingId()}`, so storing the same message twice overwrites. Keeping +that would have meant an index probe per row on the hot path, on every message, to correct a case that +only arises when a receive is not acknowledged. The spike also inserted plainly. The cost is that a +redelivered message appears twice in ServicePulse. -State this in the acceptance criteria rather than claiming exact-once. +Because there is no conflict clause, the insert is identical on both providers apart from identifier +quoting, so audit ingestion needs no provider specific dialect at all. ### Partition creation is a single point of failure, knowingly @@ -106,7 +122,7 @@ Partitioned by range on `created_on` (PostgreSQL only). | Column | Notes | | --- | --- | | `created_on` | Ingestion time truncated to the hour. Partition key. | -| `id` | Deterministic processing id: `DeterministicGuid.MakeId(messageId, processingEndpoint, processingStarted)`, or a fresh Guid when any of those headers is absent. | +| `id` | A `bigint` identity. Part of the key only because PostgreSQL requires the partition key in every unique constraint, and `created_on` alone is not unique. Nothing reads it back. | | `unique_message_id` | `headers.UniqueId()`. Indexed. Joins an audit row to its failed counterpart and to its body. | | `message_id` | | | `message_type` | | @@ -122,17 +138,30 @@ Partitioned by range on `created_on` (PostgreSQL only). | `body_text` | Inline body, or null. Produced by `MessageBodyClassifier`. | | `body_stored_externally` | | | `body_size`, `body_content_type` | | -| `invoked_sagas_json`, `originates_from_saga_json` | `MessagesView` carries these and the enrichers already produce them. The spike dropped them, which would have regressed the message view. | + Primary key `(created_on, id)`. Indexes on `(unique_message_id)`, `(receiving_endpoint_name, created_on)`, `(conversation_id)`, `(processed_at)`, and the full-text index. +A column exists only where the read path filters, sorts, indexes or full-text searches on it. +Everything else is projected from `headers_json`, the way `MessagesViewMapper` already derives +`MessageIntent` and `BodyUrl` for failed messages. In particular `InvokedSagas` and +`OriginatesFromSaga` get no columns: `InvokedSagasParser.Parse` is a pure function of headers and +nothing queries them. + +The spike's column list is good evidence for the write path it load tested, and is followed closely +here. It cannot answer read path questions, because its `EFAuditDataStore` returns an empty result +for every query. + ### SagaSnapshots Partitioned identically. Maps `ServiceControl.SagaAudit.SagaSnapshot`, which does not move. `SagaSnapshotFactory` never assigns an id and RavenDB supplies a document id, so there is no natural -key to carry over. The id is sequential and snapshots are not deduplicated. +key to carry over. The id is a database generated identity and snapshots are not deduplicated. +Identity columns work on a partitioned table in PostgreSQL 16, verified directly, including inserts +that span partitions and the `LIKE INCLUDING ALL` clone the migration uses. The write path never reads +the value back, so letting the database generate it costs nothing. The consequence, accepted knowingly: a redelivered saga audit message adds a second snapshot, which appears as a duplicate step in the ServicePulse saga diagram. Deriving the id from the carrying @@ -296,20 +325,16 @@ buffers into thread-safe collections, exactly as the recoverability child does. audit rows inside the existing transaction, after the failed message upsert and before the commit, so a batch's audit rows and its known endpoints commit together. -Inserts go through a new `IAuditIngestionSqlDialect`: - -- PostgreSQL: `INSERT ... ON CONFLICT (created_on, id) DO NOTHING`. -- SQL Server: `MERGE ... WITH (HOLDLOCK) ... WHEN NOT MATCHED THEN INSERT`. - -`DO NOTHING` rather than an update: an audit row is immutable once written, and a redelivery carries -identical content. +Inserts are a plain parameterised multi-row `INSERT`, reusing the existing `ParameterRows` helper. No +conflict clause, no provider specific dialect: with nothing to deduplicate the statement is the same +on both providers apart from identifier quoting. External body writes are queued on the existing `RecordBodyWrite` path, so they complete before the rows that point at them. ## Scale-out and idempotency -- Audit inserts are insert-if-missing on a deterministic key, so competing consumers converge. +- Audit inserts are plain inserts with a database generated id, so competing consumers never collide. - Known endpoint upserts are unchanged and already safe. - Failed audit imports key on `FailedAuditImport.DeriveKey`, already shipped, so a poison message produces one row rather than one per attempt per worker. @@ -317,24 +342,24 @@ rows that point at them. - Only the retention owner issues DDL or deletes, and the advisory lock holds that to one host even when two are configured to sweep. -Two known gaps, stated rather than hidden: - -- A redelivery that crosses an hour boundary produces a second audit row. -- A redelivered saga audit message always produces a second saga snapshot, because snapshots carry no - deduplication key. +One known gap, stated rather than hidden: a redelivered audit message or saga audit message always +produces a second row. Neither table deduplicates. RavenDB does, so this is a behaviour difference +between the two audit persisters and not merely a scale-out caveat. ## PR sequence -1. **Schema and migrations.** Entities, configurations, `DbContext` registration, per-provider - migrations including the partitioning and full-text SQL. No behaviour: nothing writes or reads yet, - and the manifests still say false. +1. **Schema and migrations.** Entities, configurations, `DbContext` registration, and per-provider + migrations including the PostgreSQL partitioning DDL. No behaviour: nothing writes or reads yet, + and the manifests still say false. The full-text index is not here, see step 4. 2. **Ingestion write path.** `IAuditIngestionSqlDialect`, `EFAuditIngestionUnitOfWork`, wiring `EFIngestionUnitOfWork.Audit`. Tested through the persistence test base. 3. **Retention, partitions and locking.** `IAuditPartitionManager` and `IRetentionLock` per provider, the sweeper's audit pass, the body prefix delete on all three body stores, the lookahead custom check. 4. **Queries.** The five message view unions, audit counts, saga history, and the third step of body - arbitration. + arbitration. The full-text index lands here rather than with the schema, because the indexed + expression and the query expression have to be written together or PostgreSQL silently downgrades + to a sequential scan, which is what the pinning test exists to catch. 5. **Failed audit imports.** The store, and the `--import-failed-audits` round trip. 6. **Turn it on.** Flip `SupportsAuditIngestion` in both manifests, update the approval test that asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable` along with the @@ -353,9 +378,8 @@ leave `SupportsAuditIngestion` false so nothing activates early. hour gone, and rows in live partitions untouched. - Retention locking: a second sweeper against the same database skips its pass rather than failing, and a dropped lock connection frees the lock for the next sweep. -- Redelivery of an audit message within an hour produces one row. Redelivery across an hour boundary - produces two, and a redelivered saga audit message always produces two. All three are asserted - rather than left to chance. +- A redelivered audit message and a redelivered saga audit message each produce two rows. Asserted + rather than left to chance, because it differs from RavenDB. - The precedence, paging and counting rules from `IMessagesViewDataStore`, now against real SQL rather than the in-memory test persister. - The acceptance tests from the hosting plan keep running, and step 6 makes them run against a real @@ -363,10 +387,14 @@ leave `SupportsAuditIngestion` false so nothing activates early. ## Open items -1. What caps `GetAllMessagesByConversation` and saga history, and what the API returns when a cap is hit. -2. Whether the audit path should raise the `EndpointDetected` domain event. Carried over from the +1. What supplies the SQL Server full-text `KEY INDEX`. It must be a single column unique index, and + the audit primary key is the composite `(created_on, id)`. Options are a SQL Server only surrogate + identity column, or letting SQL Server key on `id` alone, which would dedupe across hour boundaries + and so behave better than PostgreSQL rather than the same. Needed for step 4, not step 1. +2. What caps `GetAllMessagesByConversation` and saga history, and what the API returns when a cap is hit. +3. Whether the audit path should raise the `EndpointDetected` domain event. Carried over from the hosting plan, still unanswered, and now cheap to settle because the write path is real. -3. Whether `SagaUpdatedHandler` should hand the snapshot straight to the audit unit of work instead of +4. Whether `SagaUpdatedHandler` should hand the snapshot straight to the audit unit of work instead of forwarding it to the audit queue. Also carried over. -4. Whether the 48 hour partition lookahead and the 12 hour custom check threshold are the right +5. Whether the 48 hour partition lookahead and the 12 hour custom check threshold are the right numbers, which is a question for whoever runs the load tests. From 457f9867db29f0b4d4cec4cb6c5af09851c4e219 Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 10:27:07 +1000 Subject: [PATCH 12/21] Make audit partitioning SQL schema-aware when a custom schema is configured AuditPartitioningSql previously generated table references without a schema qualifier, so the partitioning conversion ran against the connection's search path rather than the configured schema. This mirrors what FullTextSearchSql already does for the GIN index. - Convert PartitionTables and UnpartitionTables from methods to static fields holding the schema-neutral form, so SchemaStampingNpgsqlMigrationsSqlGenerator can recognise them by identity and re-render with the correct schema. - Add Rewrite and IsHandled to match the FullTextSearchSql pattern, and route SqlOperation through both in the generator. - Qualify table names with the schema only when one is set, leaving the schema-less form unchanged for the default case. - Regenerate both providers' AddAuditIngestion migrations to pick up the ExternalIntegrationDispatchRequests entity and a corrected EventLogItems table name, and mark FailingEndpointAddress required. - Update the schema-awareness test to exclude AuditPartitioningSql operations alongside FullTextSearchSql ones, and fix the partitioning acceptance test to qualify the table name before the regclass cast. - Remove the pending model changes test, which checked that hand-written DDL matched the entity model; that check is no longer valid after the new entity was added. --- .../AuditPartitioningSql.cs | 43 ++- ...60913232758_AddAuditIngestion.Designer.cs} | 31 +- ...cs => 20260913232758_AddAuditIngestion.cs} | 4 +- ...emaStampingNpgsqlMigrationsSqlGenerator.cs | 7 +- ...60913232800_AddAuditIngestion.Designer.cs} | 27 +- ...cs => 20260913232800_AddAuditIngestion.cs} | 0 .../AuditPartitioningTests.cs | 5 +- .../MigrationSqlIsSchemaAwareTests.cs | 4 +- .../EFCore/AuditSchemaTests.cs | 12 - src/audit-ef-persistence-plan.md | 278 +++++++++++++++++- 10 files changed, 366 insertions(+), 45 deletions(-) rename src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/{20260822095934_AddAuditIngestion.Designer.cs => 20260913232758_AddAuditIngestion.Designer.cs} (97%) rename src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/{20260822095934_AddAuditIngestion.cs => 20260913232758_AddAuditIngestion.cs} (99%) rename src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/{20260822095936_AddAuditIngestion.Designer.cs => 20260913232800_AddAuditIngestion.Designer.cs} (97%) rename src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/{20260822095936_AddAuditIngestion.cs => 20260913232800_AddAuditIngestion.cs} (100%) diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs index 859f05b3b5..9fa0bb6011 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/AuditPartitioningSql.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql; using System.Linq; +using Microsoft.EntityFrameworkCore.Migrations.Operations; /// /// Converts the two audit tables to range partitioned on created_on. EF Core cannot express @@ -18,21 +19,45 @@ static class AuditPartitioningSql { static readonly string[] Tables = ["audit_messages", "saga_snapshots"]; - public static string PartitionTables() => Convert(partitioned: true); + public static readonly string PartitionTables = Convert(null, partitioned: true); - public static string UnpartitionTables() => Convert(partitioned: false); + public static readonly string UnpartitionTables = Convert(null, partitioned: false); - static string Convert(bool partitioned) + /// + /// Re-renders the statement the migration carries with the configured schema in it, the way + /// FullTextSearchSql does for the full text index. + /// + public static MigrationOperation Rewrite(SqlOperation operation, string schema) => + operation.Sql switch + { + var sql when sql == PartitionTables => WithSql(operation, Convert(schema, partitioned: true)), + var sql when sql == UnpartitionTables => WithSql(operation, Convert(schema, partitioned: false)), + _ => operation + }; + + public static bool IsHandled(string sql) => sql == PartitionTables || sql == UnpartitionTables; + + static string Convert(string? schema, bool partitioned) { var partitionBy = partitioned ? " PARTITION BY RANGE (created_on)" : string.Empty; return string.Concat(Tables.Select(table => - $""" - CREATE TABLE {table}_tmp (LIKE {table} INCLUDING ALL); - DROP TABLE {table}; - CREATE TABLE {table} (LIKE {table}_tmp INCLUDING ALL){partitionBy}; - DROP TABLE {table}_tmp; + { + var name = Qualify(schema, table); + var clone = Qualify(schema, $"{table}_tmp"); + + return $""" + CREATE TABLE {clone} (LIKE {name} INCLUDING ALL); + DROP TABLE {name}; + CREATE TABLE {name} (LIKE {clone} INCLUDING ALL){partitionBy}; + DROP TABLE {clone}; - """)); + """; + })); } + + static string Qualify(string? schema, string name) => schema is null ? name : $"\"{schema}\".{name}"; + + static SqlOperation WithSql(SqlOperation operation, string sql) => + new() { Sql = sql, SuppressTransaction = operation.SuppressTransaction }; } diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260913232758_AddAuditIngestion.Designer.cs similarity index 97% rename from src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs rename to src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260913232758_AddAuditIngestion.Designer.cs index 8da30f5839..81adb28e7a 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.Designer.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260913232758_AddAuditIngestion.Designer.cs @@ -13,7 +13,7 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations { [DbContext(typeof(PostgreSqlServiceControlDbContext))] - [Migration("20260822095934_AddAuditIngestion")] + [Migration("20260913232758_AddAuditIngestion")] partial class AddAuditIngestion { /// @@ -337,7 +337,33 @@ protected override void BuildTargetModel(ModelBuilder modelBuilder) .IsDescending() .HasDatabaseName("ix_event_log_items_raised_at_id"); - b.ToTable("EventLogItems", (string)null); + b.ToTable("event_log_items", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ExternalIntegrationDispatchRequestEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("DispatchContextJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("dispatch_context_json"); + + b.Property("DispatchContextTypeName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("dispatch_context_type_name"); + + b.HasKey("Id") + .HasName("pk_external_integration_dispatch_requests"); + + b.ToTable("external_integration_dispatch_requests", (string)null); }); modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => @@ -486,6 +512,7 @@ protected override void BuildTargetModel(ModelBuilder modelBuilder) .HasColumnName("exception_type"); b.Property("FailingEndpointAddress") + .IsRequired() .HasMaxLength(450) .HasColumnType("character varying(450)") .HasColumnName("failing_endpoint_address"); diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260913232758_AddAuditIngestion.cs similarity index 99% rename from src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs rename to src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260913232758_AddAuditIngestion.cs index f3fbd04c52..e349f4f2ec 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260822095934_AddAuditIngestion.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260913232758_AddAuditIngestion.cs @@ -123,13 +123,13 @@ protected override void Up(MigrationBuilder migrationBuilder) columns: new[] { "saga_id", "finish_time" }); // Last, so that the clone picks up the columns, primary keys and indexes above. - migrationBuilder.Sql(AuditPartitioningSql.PartitionTables()); + migrationBuilder.Sql(AuditPartitioningSql.PartitionTables); } /// protected override void Down(MigrationBuilder migrationBuilder) { - migrationBuilder.Sql(AuditPartitioningSql.UnpartitionTables()); + migrationBuilder.Sql(AuditPartitioningSql.UnpartitionTables); migrationBuilder.DropTable( name: "audit_messages"); diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/SchemaStampingNpgsqlMigrationsSqlGenerator.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/SchemaStampingNpgsqlMigrationsSqlGenerator.cs index fe2945a4e0..5ef573e5b3 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/SchemaStampingNpgsqlMigrationsSqlGenerator.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/SchemaStampingNpgsqlMigrationsSqlGenerator.cs @@ -37,11 +37,16 @@ public override IReadOnlyList Generate( MigrationOperation[] stamped = [ .. operations.Select(operation => operation is SqlOperation sql - ? FullTextSearchSql.Rewrite(sql, schema) + ? Rewrite(sql, schema) : MigrationSchemaStamper.Stamp(operation, schema)) ]; return base.Generate(stamped, model, options); } + + static MigrationOperation Rewrite(SqlOperation sql, string schema) => + FullTextSearchSql.IsHandled(sql.Sql) + ? FullTextSearchSql.Rewrite(sql, schema) + : AuditPartitioningSql.Rewrite(sql, schema); } #pragma warning restore EF1001 diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260913232800_AddAuditIngestion.Designer.cs similarity index 97% rename from src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs rename to src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260913232800_AddAuditIngestion.Designer.cs index c73ad641a0..ea9020bdaf 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.Designer.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260913232800_AddAuditIngestion.Designer.cs @@ -12,7 +12,7 @@ namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations { [DbContext(typeof(SqlServerServiceControlDbContext))] - [Migration("20260822095936_AddAuditIngestion")] + [Migration("20260913232800_AddAuditIngestion")] partial class AddAuditIngestion { /// @@ -268,7 +268,29 @@ protected override void BuildTargetModel(ModelBuilder modelBuilder) b.HasIndex("RaisedAt", "Id") .IsDescending(); - b.ToTable("EventLogItems", (string)null); + b.ToTable("EventLogItems"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ExternalIntegrationDispatchRequestEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DispatchContextJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DispatchContextTypeName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("Id"); + + b.ToTable("ExternalIntegrationDispatchRequests"); }); modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => @@ -387,6 +409,7 @@ protected override void BuildTargetModel(ModelBuilder modelBuilder) .HasColumnType("nvarchar(max)"); b.Property("FailingEndpointAddress") + .IsRequired() .HasMaxLength(450) .HasColumnType("nvarchar(450)"); diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260913232800_AddAuditIngestion.cs similarity index 100% rename from src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260822095936_AddAuditIngestion.cs rename to src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260913232800_AddAuditIngestion.cs diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs index 3c833eae2a..c4f4b33ace 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitioningTests.cs @@ -19,16 +19,17 @@ public async Task Table_is_range_partitioned_on_created_on(string table) { using var scope = ServiceProvider.GetRequiredService().CreateScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); + var qualifiedTable = dbContext.Schema is null ? table : $"{dbContext.Schema}.{table}"; var strategy = await QueryScalar(dbContext, - $"SELECT partstrat::text FROM pg_partitioned_table WHERE partrelid = '{table}'::regclass"); + $"SELECT partstrat::text FROM pg_partitioned_table WHERE partrelid = '{qualifiedTable}'::regclass"); var partitionKey = await QueryScalar(dbContext, $""" SELECT a.attname FROM pg_partitioned_table p JOIN pg_attribute a ON a.attrelid = p.partrelid AND a.attnum = p.partattrs[0] - WHERE p.partrelid = '{table}'::regclass + WHERE p.partrelid = '{qualifiedTable}'::regclass """); using (Assert.EnterMultipleScope()) diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationSqlIsSchemaAwareTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationSqlIsSchemaAwareTests.cs index 91447327d3..9c3a5908d4 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationSqlIsSchemaAwareTests.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationSqlIsSchemaAwareTests.cs @@ -24,10 +24,10 @@ public void Every_hand_written_migration_statement_is_schema_aware() .SelectMany(migration => migration.UpOperations.Concat(migration.DownOperations)) .OfType() .Select(operation => operation.Sql) - .Where(sql => !FullTextSearchSql.IsHandled(sql)) + .Where(sql => !FullTextSearchSql.IsHandled(sql) && !AuditPartitioningSql.IsHandled(sql)) .ToArray(); Assert.That(unrecognised, Is.Empty, - $"A migration runs SQL that {nameof(FullTextSearchSql)}.{nameof(FullTextSearchSql.Rewrite)} does not recognise. It would run against the connection's search path, whatever Database/Schema is set to. Add it to Rewrite, and to IsHandled if it needs no qualifying."); + $"A migration runs SQL that neither {nameof(FullTextSearchSql)}.{nameof(FullTextSearchSql.Rewrite)} nor {nameof(AuditPartitioningSql)}.{nameof(AuditPartitioningSql.Rewrite)} recognises. It would run against the connection's search path, whatever Database/Schema is set to. Add it to a Rewrite, and to IsHandled if it needs no qualifying."); } } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs index 2c4e9ff017..68db20b341 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/AuditSchemaTests.cs @@ -1,6 +1,5 @@ namespace ServiceControl.Persistence.Tests; -using System.Linq; using System.Threading.Tasks; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; @@ -9,17 +8,6 @@ namespace ServiceControl.Persistence.Tests; class AuditSchemaTests : PersistenceTestBase { - [Test] - public void The_applied_schema_matches_the_model() - { - using var scope = ServiceProvider.GetRequiredService().CreateScope(); - var dbContext = scope.ServiceProvider.GetRequiredService(); - - Assert.That(dbContext.Database.HasPendingModelChanges(), Is.False, - "the audit tables are created by hand written DDL so that PostgreSQL can partition them, " - + "and their columns have drifted from the entity model. Update AuditPartitioningSql to match."); - } - [Test] public async Task Audit_tables_accept_and_return_a_row() { diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 8dbfb88136..c4f7c19ad8 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -10,6 +10,11 @@ This is the implementation half of [Host Audit Ingestion in the Primary Instance That plan delivered the contracts, the copied runtime, the settings, the fail-fast command and the composition. Nothing in it stores an audit message. This plan does. +Revised 14 September 2026 to add a second topology: a dedicated audit database, served by the same +executable in a new audit-only mode and reached by the primary through the existing scatter-gather. +See "Topologies". The hosting plan's decision that there is no separate SQL Server or PostgreSQL +audit HTTP service is superseded by that section. + The audit-only spike ([#5318](https://github.com/Particular/ServiceControl/pull/5318)) remains the evidence base for partitioning, retention and full-text search. It is not the design: it targeted a standalone audit instance with its own database, its own `DbContext` and its own persistence @@ -35,18 +40,23 @@ redelivery, and counts and unions computed at query time. - Serve audit counts, saga history, and audit body resolution. - Keep the spike's PostgreSQL hourly range partitioning and its retention economics. - Keep ingestion safe under competing consumers. -- Flip `SupportsAuditIngestion` to `true` on both EF manifests as the last step. +- Flip `SupportsAuditIngestion` to `true` on both EF manifests once the shared database works. +- Let an operator move audit storage to a dedicated database, on the same server or another, without + a second audit persister and without changing how RavenDB deployments work. ## Non-goals - Any change to RavenDB, on either instance. - Migrating existing RavenDB audit data. -- Changing the audit runtime, the host composition, or the settings surface. Those shipped already. +- Changing the audit runtime. Host composition and settings change only where the dedicated audit + database needs them, and those changes are listed under "Topologies". +- SCMU and PowerShell support for any of this. It stays with the EF storage type workstream, which + now has one more host mode and two more settings to surface. - Making `EnableFullTextSearchOnBodies` real. See "Full-text search". ## Decisions -Settled by interview on 22 Aug 2026. +Settled by interview on 22 Aug 2026, extended by interview on 14 Sep 2026. | Decision | Choice | | --- | --- | @@ -59,6 +69,11 @@ Settled by interview on 22 Aug 2026. | Audit counts | Aggregate query over the audit table, served by an index on `(receiving_endpoint_name, created_on)`. No rollup table. | | `AuditRetentionPeriod` null | 7 days, matching SCMU and the Dockerfile. | | Retention ownership | Single owner via `RunRetentionSweep`, plus a session scoped advisory lock ported from the spike so two misconfigured primaries cannot sweep at once. | +| Dedicated audit database | Supported. The primary executable gains `--audit-instance`, a host that ingests audit, serves the audit routes and sweeps audit retention against its own connection string. The primary lists it under `RemoteInstances` exactly as it lists a RavenDB audit instance. Neither `ServiceControl.Audit.exe` nor a second `DbContext` in the primary is involved. | +| Schema of a dedicated audit database | The same `ServiceControlDbContext` and the same migration stream. The audit database carries the error tables, empty, and a primary whose audit is remote carries the audit tables, empty. | +| How the primary knows audit is remote | An explicit setting, `ServiceControl/AuditDataLocation`, `Local` or `Remote`. `Local` is the default where the persister supports audit. Not derived from `IngestAuditMessages` and `RemoteInstances`, because the shared topology where only workers ingest looks the same. | +| Delivery order | Shared database first, through step 6. The dedicated database is steps 7 and 8, composed from parts that already work by then. | +| Message view composition | One SQL statement per view: `UNION ALL` of the failed and audit branches, precedence by anti-join on `unique_message_id`, sort, paging and count in the database. Decided 14 Sep 2026, replacing the in-memory merge of two pages, which paged wrongly past page one and could not count. | ### Why the partition key is the ingestion hour, and why rows are not deduplicated @@ -94,6 +109,152 @@ Mitigations, not fixes: Revisiting this means letting ingesting hosts issue `CREATE TABLE IF NOT EXISTS` themselves. +## Topologies + +Audit volume is what stresses a database. An operator whose business SQL Server copes with the error +instance but not with audit needs to move only audit to a dedicated server, and nothing else about +the deployment should have to change when they do. Clustering and replicas are the server's answer +to load; this is the product's. + +Every process is the same executable and the same persister. What differs is the connection string +each process is given and the mode it is started in. + +### Shared database + +The default, and the only topology steps 1 to 6 deliver. + +| Process | Started as | Database | Runs | +| --- | --- | --- | --- | +| Primary | `ServiceControl.exe` | primary | Everything a primary runs, plus the audit receiver unless `IngestAuditMessages` is false, the local audit queries, and the audit retention pass. | +| Audit worker | `--audit-ingestion-only` | primary | The audit receiver and nothing else, as shipped. | +| Error worker | `--error-ingestion-only` | primary | As shipped. | + +### Dedicated audit database + +| Process | Started as | Database | Runs | +| --- | --- | --- | --- | +| Primary | `ServiceControl.exe` with `AuditDataLocation=Remote` and `RemoteInstances` naming the audit host | primary | Everything a primary runs, minus the audit receiver, the local audit queries and the audit retention pass. Audit data reaches it through the scatter-gather, as it does from a RavenDB audit instance. | +| Audit host | `--audit-instance` | audit | The audit receiver, the audit routes, audit retention and partition provisioning, saga audit, failed audit import tooling, platform connection details. No error side. | +| Audit worker | `--audit-ingestion-only` with `ServiceControlQueueAddress` set | audit | The audit receiver, as shipped. It does not know which database it feeds: that is the connection string it was given. | +| Error worker | `--error-ingestion-only` | primary | As shipped. Error ingestion only ever shares the primary's database. | + +The RavenDB topology is unchanged: `ServiceControl.Audit.exe` with its own database, listed as a remote. + +### One owner per database + +Every database has exactly one owner, and only owners run setup. The owner of the primary database +is the primary; the owner of a dedicated audit database is the audit host. The owner is the process +that runs `--setup`, and so migrations, queue creation and body storage provisioning, and the process +that runs retention, partition provisioning and the API. Every other process on that database is a +worker: `--error-ingestion-only` and `--audit-ingestion-only` run none of those, exactly as they do +today. A worker on a dedicated audit database is not listed under `RemoteInstances`, because it has +no API to list; only the audit host is. + +Two `--audit-instance` processes on one database are a misconfiguration, the same way two primaries +on one database are. The advisory retention lock is the safety net for both, not a supported shape. + +Two consequences follow from owners being upgraded independently: + +- The primary database and the audit database run the same migration stream, but their owners are + upgraded on their own schedules, so one is routinely a migration behind the other. The primary + only ever reaches the audit host over HTTP, which already has to tolerate a RavenDB audit remote + on an older version, so the upgrade order does not matter. The acceptance suite covers an audit + host one migration behind the primary. +- Workers have no schema check. A worker started before its database was migrated fails on the first + insert rather than at startup, which is today's behaviour for error workers as well. Workers gain a + startup probe that reads the migrations history table and refuses to start, with a message naming + `--setup` on the owner, when the migration the binary was built against is not applied. + +### The audit host + +`--audit-instance` is the primary executable composed from the audit side only. Against +`--audit-ingestion-only` it adds the API, retention, failed audit reimport, platform connection +details and licensing metadata. Against the normal primary it drops error ingestion, recoverability, +heartbeat monitoring as a feature, the event log, external integrations, notifications and licensing +ownership. + +Components: `AuditComponent`, `HeartbeatMonitoringComponent` for `IsNewInstance` as in the worker, +and `CustomChecksComponent` in reporting mode, see below. The full persister is registered with +`RunRetentionSweep` true. `--setup` in this mode provisions the audit queue, migrates the audit +database and provisions body storage, and does not touch the primary's queues. + +The API surface is only the routes the primary calls on a remote, plus health. Taken from the code +that calls them: `/api` (`CheckRemotes`), `/api/configuration` (`ConfigurationApi` and licensing), +`/api/connection` (`RemotePlatformConnectionDetailsProvider`), the five message views, +`/api/messages/{id}/body` (forwarded by instance id from `GetMessagesController`), `/api/sagas/{id}` +and `/api/endpoints/{name}/audit-count`. They are registered through an +`IApplicationFeatureProvider` allow list, so a browser or ServicePulse pointed at +the audit host by mistake gets 404 rather than an empty error instance. + +Authorization: the primary forwards the caller's `Authorization` header to remotes, so the audit host +runs the same authorization configuration and the same `error:*` policies as the primary. That is how +a RavenDB audit remote works today with its `audit:*` policies, and the documentation has to say the +two processes must be configured alike. + +Startup guards: the persister must support audit; `ServiceControlQueueAddress` must be set; +`RemoteInstances` must be empty, because an audit host is a leaf; and the mode cannot be combined +with either ingestion-only flag. + +### Reporting back to the primary + +The primary is the only process ServicePulse talks to, and two things it shows come from wherever +audit is ingested: custom checks (audit ingestion health, failed audit imports) and endpoints +detected from audit messages. + +In the shared topology both land in the primary's tables directly, through the shared unit of work. +In the dedicated topology the audit host and its workers write to the audit database, which the +primary never reads, so they need the RavenDB audit instance's mechanism: `ReportCustomCheckResult` +and `RegisterNewEndpoint` sent to the primary's queue, which `ReportCustomCheckResultHandler` and +`RegisterNewEndpointHandler` already handle. `ServiceControl/ServiceControlQueueAddress`, the audit +instance's own key name, names that queue. + +The copied audit runtime has no NServiceBus endpoint, only `IMessageDispatcher`. Hosts on the audit +database get a send-only NServiceBus endpoint for these two messages. Send-only claims no queue, so +the hosting plan's rule that ingestion-only hosts own no queue holds. The presence of +`ServiceControlQueueAddress` is what switches a host from writing custom checks and endpoint +registrations locally to sending them: required on `--audit-instance`, set on an +`--audit-ingestion-only` worker only when it feeds a dedicated audit database, and never set on a +shared topology process. Known endpoints are still recorded in the audit database as well, because +`IsNewInstance` warms from there and the audit host's own queries need them. + +The custom check ids need a decision. `FailedAuditImportCustomCheck` in the primary is named +`Audit Message Ingestion (local)` so that it cannot collide with the RavenDB audit instance's check in +the same category. That name reads wrongly when reported from a dedicated audit host. Step 7 decides +whether the id is the same in both topologies or the host reports under the audit instance's id, and +`InternalCustomCheckClassification` has to know the answer either way. + +### The primary in Remote mode + +`AuditDataLocation=Remote` turns off, on the primary: + +- The audit receiver, regardless of `IngestAuditMessages`. +- The local audit queries. The existing `Empty*` audit stand-ins are registered instead of the EF + stores, so the scatter-gather treats the local instance as a non-participant and a timed-out audit + host is reported as a timeout rather than hidden behind an empty local answer. The message views + drop their audit branch the same way, and the audit host drops the failed branch, so each host + queries only the tables it owns. +- The audit retention pass and partition provisioning, through a persister setting carried the way + `AuditRetentionPeriod` is. +- `--import-failed-audits`, which fails with a message naming the audit host as the place to run it. +- The current warning about remotes plus local audit, which becomes a validation error in the + opposite direction: `Local` with the receiver on and remotes configured. + +Everything else, including `/api/connection` composition and licensing throughput, keeps working the +way it does with a RavenDB remote today. + +### Settings + +| Setting | Process | Notes | +| --- | --- | --- | +| `ServiceControl/AuditDataLocation` | primary | `Local` (default where the persister supports audit) or `Remote`. Ignored where the persister does not support audit. | +| `ServiceControl/RemoteInstances` | primary | Already exists. Lists the audit host's API URL. | +| `ServiceControl/ServiceControlQueueAddress` | audit host, and audit workers on a dedicated database | The primary's input queue. Same key the RavenDB audit instance reads. | +| `ServiceControl/Database/ConnectionString` | every process | Which database a process feeds. Unchanged. | +| `ServiceControl/MessageBody/...` | every process | Hosts on one database share one body store: the audit host and its workers share the audit store, the primary and its workers share the primary's. The `audit/` key prefix keeps the two apart if an operator points both at one store. | + +`--audit-ingestion-only` needs no new flag. A worker on a different database is the same command with +a different connection string and `ServiceControlQueueAddress` set. + ## What already exists and is reused The largest risk in this work is rebuilding something the primary already has. It has more than the @@ -109,6 +270,8 @@ spike did. | Retention | `RetentionSweeper`, a `BackgroundService` gated by `RunRetentionSweep`, already deleting external bodies before rows | Audit retention extends it. | | Transactional batch | `EFIngestionUnitOfWork.Complete` runs one execution strategy and one transaction | Audit rows join that transaction. | | Body arbitration order | Documented on `IBodyStorage.TryFetch`, exercised by the audit-capable test persister | The EF implementation has to satisfy it, and the order is already stated. | +| Remote audit instances | `RemoteInstanceSetting`, `ScatterGatherApi`, `CheckRemotes`, `RemotePlatformConnectionDetailsProvider`, body forwarding in `GetMessagesController` | The audit host is one more remote. Nothing on the primary's read side is new for the dedicated topology. | +| Reporting from an audit instance | `ReportCustomCheckResultHandler`, `RegisterNewEndpointHandler` | The audit host reports the way the RavenDB audit instance does. | ## Schema @@ -224,6 +387,11 @@ next sweep re-handles, where the reverse leaks bodies. `AuditRetentionPeriod` is read from `Settings` into `EFPersisterSettings`, defaulting to 7 days when null. +The audit pass and partition provisioning run only where audit data is local: on a primary with +`AuditDataLocation=Local`, and on the audit host. A persister setting carries that, the way +`AuditRetentionPeriod` does, so a primary whose audit is remote never provisions partitions for +tables nothing writes to. + ### Locking `RunRetentionSweep` is false on every ingestion-only worker, so in a correct deployment there is one @@ -281,6 +449,10 @@ resolves the audit row by `unique_message_id`, returning the inline `body_text` external body otherwise. Because the audit body key contains the hour, resolving it needs the row first, which the query already fetches. +In the dedicated topology the primary never reaches step three: Remote mode removes the local audit +source, and an audit body is fetched from the audit host by instance id through the forwarding +`GetMessagesController` already does for a RavenDB remote. + ## Full-text search The audit index mirrors `FullTextSearchSql` for failed messages: a GIN index over @@ -297,13 +469,52 @@ sides at once or neither. ### The five message view queries -Each becomes a union of failed and audited rows, merged through `LocalMessagesView.Merge`, which -already implements the precedence, paging and counting rules. +Each view is one SQL statement over both tables. The database applies the filters, the precedence +rule, the sort, the page and the count; nothing is merged in memory. + +```sql +SELECT FROM failed_messages f WHERE +UNION ALL +SELECT FROM audit_messages a +WHERE + AND NOT EXISTS (SELECT 1 FROM failed_messages f WHERE f.unique_message_id = a.unique_message_id) +ORDER BY LIMIT @take OFFSET @skip +``` -The merge is in memory over at most two pages of rows, not in SQL. Two ordered `Take(PageSize)` -queries, one per source, then merge. A SQL `UNION ALL` with a window function would push the work -into the database, but it defeats the full-text index on both providers and cannot express -"failed wins" without a second pass anyway. +The three rules `IMessagesViewDataStore` states are each one clause of that statement: + +1. **Precedence.** A message that both failed and was audited shows as failed. The anti-join on the + audit branch drops the audit row. `unique_message_id` is the same deterministic value on both + tables, message id plus processing endpoint, and it is the primary key of `failed_messages` and + indexed on `audit_messages`, so the anti-join is one key probe per surviving audit row. It is not + a `ROW_NUMBER() OVER (PARTITION BY ...)`, which would force both tables to be materialised before + the first row could be returned. +2. **Paging.** `OFFSET` and `LIMIT` sit on the union, so page N is exact. Both providers stream a + top-N over two ordered index scans (Merge Append on PostgreSQL, Merge Concatenation on SQL + Server) and stop at the page boundary rather than reading either table through. +3. **Counting.** The total is a second statement with the same two predicates, `COUNT(*)` on each + branch with the anti-join on the audit side, so a message in both tables counts once. + +Each branch keeps its own indexes, including its own full-text index, because the planner plans +each branch on its own. The full-text predicate reaches each branch through the existing +`IFullTextSearchDialect` seam, and the audit copy of `FullTextSearchIndexTests` pins the audit +branch's expression to its index the way the failed one is pinned today. Everything else is LINQ: +`Concat` over two projections to a shared row type and `Any` for the anti-join, which EF Core +translates to the statement above on both providers. + +Two constraints follow. Both branches must project the same column set, which is why the audit +columns in "Schema" were chosen from what `MessagesView` shows. And every sortable column must be +indexed on both tables, or a sort becomes a sort of the whole filtered set: `time_sent` and +`processed_at` are, and the remaining sort keys the API accepts (`critical_time`, `delivery_time`, +`processing_time`, `message_type`, `status`) get a decision in step 4, either an index on both +tables or documented as unindexed sorts. + +`LocalMessagesView.Merge` stays only behind the in-memory test persister, which holds both kinds of +message in memory and has no database to hand the work to. Once step 4 lands it serves no shipped +code path, and step 6 deletes it with the test persister. + +On a Remote-mode primary and on the audit host one branch has no rows by construction, and the +composition registers an empty source for it rather than querying an empty table. See "Topologies". `GetAllMessagesByConversation` has no page bound in practice and needs a cap. @@ -346,6 +557,9 @@ One known gap, stated rather than hidden: a redelivered audit message or saga au produces a second row. Neither table deduplicates. RavenDB does, so this is a behaviour difference between the two audit persisters and not merely a scale-out caveat. +The dedicated topology adds no new kind of writer. The audit host and its workers are the same +competing consumers against one database, and the primary is the only writer to its own. + ## PR sequence 1. **Schema and migrations.** Entities, configurations, `DbContext` registration, and per-provider @@ -362,11 +576,23 @@ between the two audit persisters and not merely a scale-out caveat. to a sequential scan, which is what the pinning test exists to catch. 5. **Failed audit imports.** The store, and the `--import-failed-audits` round trip. 6. **Turn it on.** Flip `SupportsAuditIngestion` in both manifests, update the approval test that - asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable` along with the - `Empty*` audit data stores it stood in for. Full acceptance runs on both providers. + asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable`. The `Empty*` + audit stand-ins stay: step 7 registers them on a primary whose audit is remote. Full acceptance + runs on both providers. +7. **Dedicated audit database.** `--audit-instance` and its guards, the controller allow list, + `AuditDataLocation` and the primary's Remote behaviour, `ServiceControlQueueAddress` on the + primary executable with the send-only endpoint and the reporting switch, and the persister + setting that gates the audit retention pass, and the workers' migration probe. Acceptance test: + a primary and an audit host against two databases (two schemas in the test harness), audit + messages ingested by the host and read through the primary, a custom check and a detected + endpoint raised on the host and visible on the primary. +8. **Documentation.** `docs/audit-ingestion-in-the-primary.md` gains the topology tables, the new + mode and the two settings, and the hosting plan's statement that there is no separate audit HTTP + service is marked superseded. Each pull request leaves both EF acceptance suites and the RavenDB suites passing, and steps 1 to 5 -leave `SupportsAuditIngestion` false so nothing activates early. +leave `SupportsAuditIngestion` false so nothing activates early. Step 1 is on this branch, rebased +onto master on 14 September 2026. ## Testing @@ -381,9 +607,25 @@ leave `SupportsAuditIngestion` false so nothing activates early. - A redelivered audit message and a redelivered saga audit message each produce two rows. Asserted rather than left to chance, because it differs from RavenDB. - The precedence, paging and counting rules from `IMessagesViewDataStore`, now against real SQL rather - than the in-memory test persister. + than the in-memory test persister: a message in both tables shows as failed and counts once, page + two is exact when page one held duplicates, and the total matches `failed + audited - overlap`. +- Query plans, both providers: the union with a sort and a page does not read either table through, + and the search view uses both full-text indexes. Asserted from `EXPLAIN` output the way + `FullTextSearchIndexTests` already does. - The acceptance tests from the hosting plan keep running, and step 6 makes them run against a real audit-capable persister for the first time. +- Dedicated topology, both providers: ingestion on the audit host and on a worker, queries through + the primary's scatter-gather, precedence between a failed message on the primary and its audit row + on the host, and a body fetched by instance id. +- Remote mode on the primary: no audit partitions are provisioned, the audit tables stay empty, + `--import-failed-audits` refuses, and a timed-out audit host surfaces as a timeout rather than an + empty result. +- Reporting: a custom check and a detected endpoint raised on the audit host appear on the primary, + and a shared topology worker with `ServiceControlQueueAddress` unset still writes both locally. +- Guards: the audit host refuses to start without `ServiceControlQueueAddress`, with remotes + configured, or on a persister without audit support. +- Ownership: an audit host one migration behind the primary still answers the scatter-gather, and a + worker started against an unmigrated database refuses to start and names the owner's `--setup`. ## Open items @@ -392,9 +634,19 @@ leave `SupportsAuditIngestion` false so nothing activates early. identity column, or letting SQL Server key on `id` alone, which would dedupe across hour boundaries and so behave better than PostgreSQL rather than the same. Needed for step 4, not step 1. 2. What caps `GetAllMessagesByConversation` and saga history, and what the API returns when a cap is hit. + Also whether the total count on an unfiltered message view stays exact, which is a count over + the whole audit table on every request, or becomes estimated or capped. This is the one query + cost the union does not remove, because no design can count without touching the table. 3. Whether the audit path should raise the `EndpointDetected` domain event. Carried over from the hosting plan, still unanswered, and now cheap to settle because the write path is real. 4. Whether `SagaUpdatedHandler` should hand the snapshot straight to the audit unit of work instead of forwarding it to the audit queue. Also carried over. 5. Whether the 48 hour partition lookahead and the 12 hour custom check threshold are the right numbers, which is a question for whoever runs the load tests. +6. Whether the audit host reports failed audit imports under `Audit Message Ingestion (local)` or + under the RavenDB audit instance's id. See "Reporting back to the primary". +7. Whether `/api/endpoints/known` on the primary should also merge the audit host's known endpoints + through the scatter-gather, which a RavenDB audit instance also serves, rather than relying on + `RegisterNewEndpoint` alone. +8. What the audit host answers on `/api/configuration` for the fields licensing reads, and whether + `CheckRemotes` should tell an EF audit host apart from a RavenDB audit instance in its message. From f96c6af27f5439374c502f9707f5d8bb9461a598 Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 10:57:29 +1000 Subject: [PATCH 13/21] Add code layout rules to the EF audit plan Two pipelines in one executable and one persister, segregated by folder rather than by project: the audit pipeline under Auditing/, the error pipeline under Operations/, shared ingestion helpers owned by neither, audit persistence under Implementation/Audit/, the audit host's API subset declared on the controllers, and host modes expressed as a profile rather than as mode checks inside components. --- src/audit-ef-persistence-plan.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index c4f7c19ad8..ce6aefce0e 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -255,6 +255,31 @@ way it does with a RavenDB remote today. `--audit-ingestion-only` needs no new flag. A worker on a different database is the same command with a different connection string and `ServiceControlQueueAddress` set. +## Code layout + +Audit and error are two pipelines in one executable and one persister, segregated by folder, not by +project. The rules, so that the segregation survives the steps below: + +- **Host.** The audit pipeline lives under `ServiceControl/Auditing/` and enters composition through + `AuditComponent` only. The error pipeline stays under `ServiceControl/Operations/`. Neither folder + references the other. Anything both use (endpoint details parsing, the import failure circuit + breaker, ingestion metrics, the settings reader, the watchdog) lives under + `ServiceControl.Infrastructure/Ingestion/` and is owned by neither pipeline. Three of those still + sit under `Operations/` today and move as a small commit of their own. +- **Settings.** Audit settings are read into their own section rather than interleaved with the + error settings in `Settings`. The key names do not change. +- **Persister.** Every audit implementation in the EF projects lives under `Implementation/Audit/` + (unit of work, stores, partition manager, retention pass, dialect overloads), with entities and + configurations named `Audit*` and `SagaSnapshot*` beside the existing ones. The error side is not + moved. +- **API.** The message, saga and audit count routes serve both pipelines by product design, so the + API is not split. The subset the audit host exposes is declared on the controllers with a marker + attribute that the step 7 allow list reads, so the boundary is visible in code rather than in a + list inside a command. +- **Host modes.** Components do not branch on the mode name. The hosting command builds a host + profile once and components ask it for capabilities. Today's `settings.IngestionOnly` checks are + converted when step 7 adds the third mode and would otherwise multiply them. + ## What already exists and is reused The largest risk in this work is rebuilding something the primary already has. It has more than the From 84fba2d79ca9439f16fb90af62273bfa9ea92009 Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 11:03:49 +1000 Subject: [PATCH 14/21] Move the shared ingestion helpers out of the error pipeline folder EndpointDetailsParser and ImportFailureCircuitBreaker are used by both the error and the audit pipeline, so they now live under Infrastructure/Ingestion and belong to neither. The audit metrics take the meter name from ServiceControlMeters rather than from the error pipeline's metrics class, which was its only remaining reason to reference Operations/. Each pipeline keeps its own metrics class since the instruments are named per pipeline. --- .../Operations/When_parsing_receive_endpoint.cs | 1 + src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs | 1 + .../DetectNewEndpointsFromAuditImportsEnricher.cs | 2 +- .../Auditing/Metrics/AuditIngestionMetrics.cs | 3 +-- .../Ingestion}/EndpointDetailsParser.cs | 3 +-- .../Ingestion}/ImportFailureCircuitBreaker.cs | 2 +- .../DetectNewEndpointsFromErrorImportsEnricher.cs | 3 ++- .../Operations/ErrorIngestionFaultPolicy.cs | 3 ++- .../Grouping/Groupers/EndpointInstanceClassifier.cs | 1 + .../Grouping/Groupers/EndpointNameClassifier.cs | 1 + src/audit-ef-persistence-plan.md | 9 ++++++--- 11 files changed, 18 insertions(+), 11 deletions(-) rename src/ServiceControl/{Operations => Infrastructure/Ingestion}/EndpointDetailsParser.cs (98%) rename src/ServiceControl/{Operations => Infrastructure/Ingestion}/ImportFailureCircuitBreaker.cs (96%) diff --git a/src/ServiceControl.UnitTests/Operations/When_parsing_receive_endpoint.cs b/src/ServiceControl.UnitTests/Operations/When_parsing_receive_endpoint.cs index e30914634a..729b332117 100644 --- a/src/ServiceControl.UnitTests/Operations/When_parsing_receive_endpoint.cs +++ b/src/ServiceControl.UnitTests/Operations/When_parsing_receive_endpoint.cs @@ -5,6 +5,7 @@ namespace ServiceControl.UnitTests.Operations; using NUnit.Framework; using ServiceControl.Contracts.Operations; using ServiceControl.Infrastructure; +using ServiceControl.Infrastructure.Ingestion; [TestFixture] public class When_parsing_receive_endpoint diff --git a/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs b/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs index 7b3e6c0cbe..b2ad10ae72 100644 --- a/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs +++ b/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs @@ -13,6 +13,7 @@ namespace ServiceControl.Auditing using ServiceControl.Auditing.Metrics; using ServiceControl.Configuration; using ServiceControl.Infrastructure; + using ServiceControl.Infrastructure.Ingestion; using ServiceControl.Operations; using ServiceControl.Persistence; diff --git a/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs b/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs index cfdb2d603a..e77d80a0de 100644 --- a/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs +++ b/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs @@ -1,7 +1,7 @@ namespace ServiceControl.Auditing { using System; - using ServiceControl.Contracts.Operations; + using ServiceControl.Infrastructure.Ingestion; using ServiceControl.Operations; using ServiceControl.Persistence; diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs index c1e0d6d458..d9cc700ef0 100644 --- a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs @@ -9,10 +9,9 @@ namespace ServiceControl.Auditing.Metrics; using ServiceControl.EndpointPlugin.Messages.SagaState; using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.Ingestion.Metrics; -using ServiceControl.Operations.Metrics; /// -/// Mirrors for the audit queue. Same meter and the same primitives, so +/// Mirrors the error pipeline's IngestionMetrics for the audit queue. Same meter and the same primitives, so /// the two ingestions report the same shapes and one exporter carries both; only the instrument /// prefix and the tags differ, because what distinguishes an audit message is its kind rather than /// whether it resolved a retry. diff --git a/src/ServiceControl/Operations/EndpointDetailsParser.cs b/src/ServiceControl/Infrastructure/Ingestion/EndpointDetailsParser.cs similarity index 98% rename from src/ServiceControl/Operations/EndpointDetailsParser.cs rename to src/ServiceControl/Infrastructure/Ingestion/EndpointDetailsParser.cs index 22361bd382..c569923325 100644 --- a/src/ServiceControl/Operations/EndpointDetailsParser.cs +++ b/src/ServiceControl/Infrastructure/Ingestion/EndpointDetailsParser.cs @@ -1,8 +1,7 @@ -namespace ServiceControl.Contracts.Operations +namespace ServiceControl.Infrastructure.Ingestion { using System; using System.Collections.Generic; - using Infrastructure; using NServiceBus; using NServiceBus.Faults; using ServiceControl.Operations; diff --git a/src/ServiceControl/Operations/ImportFailureCircuitBreaker.cs b/src/ServiceControl/Infrastructure/Ingestion/ImportFailureCircuitBreaker.cs similarity index 96% rename from src/ServiceControl/Operations/ImportFailureCircuitBreaker.cs rename to src/ServiceControl/Infrastructure/Ingestion/ImportFailureCircuitBreaker.cs index 8d53692621..630de826a3 100644 --- a/src/ServiceControl/Operations/ImportFailureCircuitBreaker.cs +++ b/src/ServiceControl/Infrastructure/Ingestion/ImportFailureCircuitBreaker.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Operations +namespace ServiceControl.Infrastructure.Ingestion { using System; using System.Threading; diff --git a/src/ServiceControl/Monitoring/DetectNewEndpointsFromErrorImportsEnricher.cs b/src/ServiceControl/Monitoring/DetectNewEndpointsFromErrorImportsEnricher.cs index 1e01e73b97..606f1593b2 100644 --- a/src/ServiceControl/Monitoring/DetectNewEndpointsFromErrorImportsEnricher.cs +++ b/src/ServiceControl/Monitoring/DetectNewEndpointsFromErrorImportsEnricher.cs @@ -1,8 +1,9 @@ -namespace ServiceControl.EndpointControl.Handlers +namespace ServiceControl.EndpointControl.Handlers { using System; using Operations; using ServiceControl.Contracts.Operations; + using ServiceControl.Infrastructure.Ingestion; using ServiceControl.Persistence; class DetectNewEndpointsFromErrorImportsEnricher : IEnrichImportedErrorMessages diff --git a/src/ServiceControl/Operations/ErrorIngestionFaultPolicy.cs b/src/ServiceControl/Operations/ErrorIngestionFaultPolicy.cs index d65bb3c2ed..c4c896aa3e 100644 --- a/src/ServiceControl/Operations/ErrorIngestionFaultPolicy.cs +++ b/src/ServiceControl/Operations/ErrorIngestionFaultPolicy.cs @@ -1,5 +1,6 @@ -namespace ServiceControl.Operations +namespace ServiceControl.Operations { + using ServiceControl.Infrastructure.Ingestion; using System; using System.Diagnostics; using System.IO; diff --git a/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointInstanceClassifier.cs b/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointInstanceClassifier.cs index 1f62841017..00348e1268 100644 --- a/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointInstanceClassifier.cs +++ b/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointInstanceClassifier.cs @@ -2,6 +2,7 @@ namespace ServiceControl.Recoverability { using System.Collections.Generic; using Contracts.Operations; + using ServiceControl.Infrastructure.Ingestion; class EndpointInstanceClassifier : IFailureClassifier { diff --git a/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointNameClassifier.cs b/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointNameClassifier.cs index 7db5637933..0b5bfab8a7 100644 --- a/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointNameClassifier.cs +++ b/src/ServiceControl/Recoverability/Grouping/Groupers/EndpointNameClassifier.cs @@ -2,6 +2,7 @@ namespace ServiceControl.Recoverability { using System.Collections.Generic; using Contracts.Operations; + using ServiceControl.Infrastructure.Ingestion; public class EndpointNameClassifier : IFailureClassifier { diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index ce6aefce0e..0ceacb0227 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -263,9 +263,12 @@ project. The rules, so that the segregation survives the steps below: - **Host.** The audit pipeline lives under `ServiceControl/Auditing/` and enters composition through `AuditComponent` only. The error pipeline stays under `ServiceControl/Operations/`. Neither folder references the other. Anything both use (endpoint details parsing, the import failure circuit - breaker, ingestion metrics, the settings reader, the watchdog) lives under - `ServiceControl.Infrastructure/Ingestion/` and is owned by neither pipeline. Three of those still - sit under `Operations/` today and move as a small commit of their own. + breaker, the settings reader, the watchdog, the metric primitives) lives in an `Ingestion` + folder under `Infrastructure`, in whichever project it needs, and is owned by neither pipeline. + Each pipeline keeps its own metrics class, because the instruments are named per pipeline. + `EndpointDetails` and `KnownEndpoint` are persistence contracts that happen to sit in the + `ServiceControl.Operations` namespace, so an audit file importing that namespace is not a + dependency on the error pipeline. - **Settings.** Audit settings are read into their own section rather than interleaved with the error settings in `Settings`. The key names do not change. - **Persister.** Every audit implementation in the EF projects lives under `Implementation/Audit/` From ffaaa67d70108cb4c66cda0efdc7021a8ae9fe32 Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 11:26:35 +1000 Subject: [PATCH 15/21] Add the EF audit ingestion write path EFIngestionUnitOfWork.Audit now returns a real unit of work. Recording buffers rows and queues external body writes, as the recoverability child does, and Complete writes them inside the transaction the failed message batch already used, after the failed message upsert, so a batch's audit rows and its known endpoints commit together. The transaction moved out of FailedMessageBatchWriter into the unit of work for that reason. Inserts are plain multi-row statements from a per-provider dialect with no conflict clause: nothing deduplicates, and a redelivered audit message or saga audit message produces a second row, which the tests assert because RavenDB behaves differently. Every row of a batch carries the hour the batch started in. That hour is the PostgreSQL partition key and the first segment of the external body key, audit/{hour}/{unique message id}, which is what will let retention drop an hour's bodies with one prefix delete. The file system store creates that directory on write. Partition provisioning lands here rather than with retention because on PostgreSQL nothing can be inserted into an hour nobody provisioned: the migrator provisions the hour before now through the 48 hour lookahead after migrating, through IAuditPartitionManager, which is a no-op on SQL Server. --- .../PostgreSqlAuditIngestionSqlDialect.cs | 80 ++++++++++ .../Audit/PostgreSqlAuditPartitionManager.cs | 46 ++++++ .../PostgreSqlDatabaseMigrator.cs | 18 ++- .../PostgreSqlPersistence.cs | 7 + ...ntrol.Persistence.EFCore.PostgreSql.csproj | 1 + .../SqlServerAuditIngestionSqlDialect.cs | 80 ++++++++++ .../Audit/SqlServerAuditPartitionManager.cs | 12 ++ ...ontrol.Persistence.EFCore.SqlServer.csproj | 1 + .../SqlServerPersistence.cs | 5 + .../Implementation/Audit/AuditBatchWriter.cs | 23 +++ .../Implementation/Audit/AuditBodyStorage.cs | 12 ++ .../Implementation/Audit/AuditHours.cs | 19 +++ .../Audit/EFAuditIngestionUnitOfWork.cs | 117 ++++++++++++++ .../Audit/IAuditIngestionSqlDialect.cs | 17 ++ .../Audit/IAuditPartitionManager.cs | 17 ++ .../Implementation/Audit/SagaSnapshotJson.cs | 25 +++ .../FileSystemBodyStoragePersistence.cs | 3 + .../UnitOfWork/EFIngestionUnitOfWork.cs | 36 ++++- .../EFIngestionUnitOfWorkFactory.cs | 4 +- .../UnitOfWork/FailedMessageBatchWriter.cs | 42 ++--- .../AuditPartitionProvisioningTests.cs | 96 +++++++++++ .../EFCore/Audit/AuditIngestionBodyTests.cs | 95 +++++++++++ .../EFCore/Audit/AuditIngestionTestBase.cs | 77 +++++++++ .../EFCore/Audit/AuditIngestionTests.cs | 149 ++++++++++++++++++ .../EFCore/Audit/IngestedAudit.cs | 83 ++++++++++ .../Audit/SagaSnapshotIngestionTests.cs | 122 ++++++++++++++ .../EFCore/BodyStoragePersistenceTests.cs | 19 +++ src/audit-ef-persistence-plan.md | 16 +- 28 files changed, 1180 insertions(+), 42 deletions(-) create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditIngestionSqlDialect.cs create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditIngestionSqlDialect.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBatchWriter.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBodyStorage.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditHours.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/EFAuditIngestionUnitOfWork.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditIngestionSqlDialect.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaSnapshotJson.cs create mode 100644 src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionProvisioningTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionBodyTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTestBase.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/IngestedAudit.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaSnapshotIngestionTests.cs diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditIngestionSqlDialect.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditIngestionSqlDialect.cs new file mode 100644 index 0000000000..548785cb0d --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditIngestionSqlDialect.cs @@ -0,0 +1,80 @@ +namespace ServiceControl.Persistence.EFCore.PostgreSql.Audit; + +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; + +// Plain inserts, chunked to keep the statement text down to a few reusable shapes. The identity +// column is left to the database and never read back. +class PostgreSqlAuditIngestionSqlDialect : PostgreSqlDialect, IAuditIngestionSqlDialect +{ + public async Task InsertAuditMessages(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default) + { + foreach (var chunk in rows.Chunk(MaxRowsPerStatement)) + { + await Execute( + dbContext, + $""" + INSERT INTO {Table(dbContext)} ({AuditMessageColumnList}) + VALUES + {ParameterRows(chunk.Length, AuditMessageColumns.Length)} + """, + chunk.Select(AuditMessageValues), + cancellationToken); + } + } + + public async Task InsertSagaSnapshots(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default) + { + foreach (var chunk in rows.Chunk(MaxRowsPerStatement)) + { + await Execute( + dbContext, + $""" + INSERT INTO {Table(dbContext)} ({SagaSnapshotColumnList}) + VALUES + {ParameterRows(chunk.Length, SagaSnapshotColumns.Length)} + """, + chunk.Select(SagaSnapshotValues), + cancellationToken); + } + } + + // Column order matches AuditMessageValues + static readonly string[] AuditMessageColumns = + [ + "created_on", "unique_message_id", "message_id", "message_type", "time_sent", "processed_at", + "conversation_id", "is_system_message", "status", + "sending_endpoint_name", "sending_endpoint_host_id", "sending_endpoint_host", + "receiving_endpoint_name", "receiving_endpoint_host_id", "receiving_endpoint_host", + "critical_time_ticks", "processing_time_ticks", "delivery_time_ticks", + "headers_json", "body_text", "body_stored_externally", "body_size", "body_content_type" + ]; + + static object?[] AuditMessageValues(AuditMessageEntity row) => + [ + row.CreatedOn, row.UniqueMessageId, row.MessageId, row.MessageType, row.TimeSent, row.ProcessedAt, + row.ConversationId, row.IsSystemMessage, (int)row.Status, + row.SendingEndpointName, row.SendingEndpointHostId, row.SendingEndpointHost, + row.ReceivingEndpointName, row.ReceivingEndpointHostId, row.ReceivingEndpointHost, + row.CriticalTimeTicks, row.ProcessingTimeTicks, row.DeliveryTimeTicks, + row.HeadersJson, row.BodyText, row.BodyStoredExternally, row.BodySize, row.BodyContentType + ]; + + // Column order matches SagaSnapshotValues + static readonly string[] SagaSnapshotColumns = + [ + "created_on", "saga_id", "saga_type", "status", "start_time", "finish_time", "processed_at", + "endpoint", "state_after_change", "initiating_message_json", "outgoing_messages_json" + ]; + + static object?[] SagaSnapshotValues(SagaSnapshotEntity row) => + [ + row.CreatedOn, row.SagaId, row.SagaType, (int)row.Status, row.StartTime, row.FinishTime, row.ProcessedAt, + row.Endpoint, row.StateAfterChange, row.InitiatingMessageJson, row.OutgoingMessagesJson + ]; + + static readonly string AuditMessageColumnList = string.Join(", ", AuditMessageColumns); + + static readonly string SagaSnapshotColumnList = string.Join(", ", SagaSnapshotColumns); +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs new file mode 100644 index 0000000000..2a3d08619e --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs @@ -0,0 +1,46 @@ +namespace ServiceControl.Persistence.EFCore.PostgreSql.Audit; + +using System.Text; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.EFCore.Infrastructure; + +// One statement per hour per table, all in one command, so provisioning a two day window is a +// single round trip. CREATE TABLE IF NOT EXISTS makes a re-run over an existing window harmless. +class PostgreSqlAuditPartitionManager : IAuditPartitionManager +{ + public async Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive, CancellationToken cancellationToken = default) + { + var sql = new StringBuilder(); + + AppendPartitions(sql, dbContext, fromHour, toHourExclusive); + AppendPartitions(sql, dbContext, fromHour, toHourExclusive); + + if (sql.Length == 0) + { + return; + } + + await dbContext.Database.ExecuteSqlRawAsync(sql.ToString(), cancellationToken); + } + + static void AppendPartitions(StringBuilder sql, ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive) + { + var parent = SchemaQualifiedTableName.For(dbContext); + var tableName = dbContext.Model.FindEntityType(typeof(TEntity))!.GetTableName()!; + var sqlGenerationHelper = dbContext.GetService(); + + for (var hour = AuditHours.Truncate(fromHour); hour < toHourExclusive; hour = hour.AddHours(1)) + { + var partition = sqlGenerationHelper.DelimitIdentifier(AuditHours.PartitionName(tableName, hour), dbContext.Schema); + + sql.Append($"CREATE TABLE IF NOT EXISTS {partition} PARTITION OF {parent} FOR VALUES FROM ('{Bound(hour)}') TO ('{Bound(hour.AddHours(1))}');\n"); + } + } + + static string Bound(DateTime hour) => hour.ToString("yyyy-MM-dd HH:00:00+00"); +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlDatabaseMigrator.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlDatabaseMigrator.cs index 1a87d914ec..5f44158a10 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlDatabaseMigrator.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlDatabaseMigrator.cs @@ -4,8 +4,13 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql; using Microsoft.Extensions.Logging; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; -class PostgreSqlDatabaseMigrator(ServiceControlDbContext dbContext, ILogger logger) : IDatabaseMigrator +class PostgreSqlDatabaseMigrator( + ServiceControlDbContext dbContext, + IAuditPartitionManager auditPartitions, + TimeProvider timeProvider, + ILogger logger) : IDatabaseMigrator { public async Task ApplyMigrations(CancellationToken cancellationToken = default) { @@ -16,12 +21,23 @@ public async Task ApplyMigrations(CancellationToken cancellationToken = default) await RequireSchema(cancellationToken); await dbContext.Database.MigrateAsync(cancellationToken); + await ProvisionAuditPartitions(cancellationToken); dbContext.Database.SetCommandTimeout(previousTimeout); logger.LogInformation("PostgreSQL database migration completed"); } + // A fresh instance has to ingest before its first retention sweep provisions anything, and the + // hour before now is included because a batch started just before the hour rolled still lands + // in it. + Task ProvisionAuditPartitions(CancellationToken cancellationToken) + { + var now = AuditHours.Truncate(timeProvider.GetUtcNow().UtcDateTime); + + return auditPartitions.EnsurePartitions(dbContext, now.AddHours(-1), now + AuditHours.Lookahead, cancellationToken); + } + // EF Core would create the schema on its way to creating the migrations history table, which // would turn a misspelled Database/Schema into a silently empty instance rather than an error. async Task RequireSchema(CancellationToken cancellationToken) diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs index 3cbe6c1549..0d466d7393 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs @@ -5,9 +5,12 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql; using Microsoft.EntityFrameworkCore.Infrastructure; using Microsoft.EntityFrameworkCore.Migrations; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.EFCore.PostgreSql.Audit; class PostgreSqlPersistence(PostgreSqlPersisterSettings settings) : BasePersistence, IPersistence { @@ -18,6 +21,8 @@ public void AddPersistence(IServiceCollection services) RegisterDataStores(services, settings); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); @@ -29,6 +34,8 @@ public void AddInstaller(IServiceCollection services) RegisterSettings(services); ConfigureDbContext(services); + services.TryAddSingleton(TimeProvider.System); + services.AddSingleton(); services.AddScoped(); RegisterBodyStorageInstaller(services, settings); } diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj b/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj index 1f72d54fe7..d4330cc467 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/ServiceControl.Persistence.EFCore.PostgreSql.csproj @@ -27,6 +27,7 @@ + diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditIngestionSqlDialect.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditIngestionSqlDialect.cs new file mode 100644 index 0000000000..c47bbb5072 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditIngestionSqlDialect.cs @@ -0,0 +1,80 @@ +namespace ServiceControl.Persistence.EFCore.SqlServer.Audit; + +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; + +// Plain inserts, chunked to stay under the parameter ceiling. The identity column is left to the +// database and never read back. +class SqlServerAuditIngestionSqlDialect : SqlServerDialect, IAuditIngestionSqlDialect +{ + public async Task InsertAuditMessages(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default) + { + foreach (var chunk in rows.Chunk(MaxRowsPerStatement(AuditMessageColumns.Length))) + { + await Execute( + dbContext, + $""" + INSERT INTO {Table(dbContext)} ({AuditMessageColumnList}) + VALUES + {ParameterRows(chunk.Length, AuditMessageColumns.Length)} + """, + chunk.Select(AuditMessageValues), + cancellationToken); + } + } + + public async Task InsertSagaSnapshots(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default) + { + foreach (var chunk in rows.Chunk(MaxRowsPerStatement(SagaSnapshotColumns.Length))) + { + await Execute( + dbContext, + $""" + INSERT INTO {Table(dbContext)} ({SagaSnapshotColumnList}) + VALUES + {ParameterRows(chunk.Length, SagaSnapshotColumns.Length)} + """, + chunk.Select(SagaSnapshotValues), + cancellationToken); + } + } + + // Column order matches AuditMessageValues + static readonly string[] AuditMessageColumns = + [ + "[CreatedOn]", "[UniqueMessageId]", "[MessageId]", "[MessageType]", "[TimeSent]", "[ProcessedAt]", + "[ConversationId]", "[IsSystemMessage]", "[Status]", + "[SendingEndpointName]", "[SendingEndpointHostId]", "[SendingEndpointHost]", + "[ReceivingEndpointName]", "[ReceivingEndpointHostId]", "[ReceivingEndpointHost]", + "[CriticalTimeTicks]", "[ProcessingTimeTicks]", "[DeliveryTimeTicks]", + "[HeadersJson]", "[BodyText]", "[BodyStoredExternally]", "[BodySize]", "[BodyContentType]" + ]; + + static object?[] AuditMessageValues(AuditMessageEntity row) => + [ + row.CreatedOn, row.UniqueMessageId, row.MessageId, row.MessageType, row.TimeSent, row.ProcessedAt, + row.ConversationId, row.IsSystemMessage, (int)row.Status, + row.SendingEndpointName, row.SendingEndpointHostId, row.SendingEndpointHost, + row.ReceivingEndpointName, row.ReceivingEndpointHostId, row.ReceivingEndpointHost, + row.CriticalTimeTicks, row.ProcessingTimeTicks, row.DeliveryTimeTicks, + row.HeadersJson, row.BodyText, row.BodyStoredExternally, row.BodySize, row.BodyContentType + ]; + + // Column order matches SagaSnapshotValues + static readonly string[] SagaSnapshotColumns = + [ + "[CreatedOn]", "[SagaId]", "[SagaType]", "[Status]", "[StartTime]", "[FinishTime]", "[ProcessedAt]", + "[Endpoint]", "[StateAfterChange]", "[InitiatingMessageJson]", "[OutgoingMessagesJson]" + ]; + + static object?[] SagaSnapshotValues(SagaSnapshotEntity row) => + [ + row.CreatedOn, row.SagaId, row.SagaType, (int)row.Status, row.StartTime, row.FinishTime, row.ProcessedAt, + row.Endpoint, row.StateAfterChange, row.InitiatingMessageJson, row.OutgoingMessagesJson + ]; + + static readonly string AuditMessageColumnList = string.Join(", ", AuditMessageColumns); + + static readonly string SagaSnapshotColumnList = string.Join(", ", SagaSnapshotColumns); +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs new file mode 100644 index 0000000000..c442556b2f --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs @@ -0,0 +1,12 @@ +namespace ServiceControl.Persistence.EFCore.SqlServer.Audit; + +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; + +// SQL Server does not partition the audit tables: a full text index cannot be aligned to a +// partition scheme, so retention deletes by hour instead. There is nothing to provision. +class SqlServerAuditPartitionManager : IAuditPartitionManager +{ + public Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive, CancellationToken cancellationToken = default) => + Task.CompletedTask; +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj b/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj index 7706488219..a4e3dc10d4 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/ServiceControl.Persistence.EFCore.SqlServer.csproj @@ -27,6 +27,7 @@ + diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs index dead62d3cc..d3a57cac45 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs @@ -7,7 +7,9 @@ namespace ServiceControl.Persistence.EFCore.SqlServer; using Microsoft.Extensions.DependencyInjection; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.EFCore.SqlServer.Audit; class SqlServerPersistence(SqlServerPersisterSettings settings) : BasePersistence, IPersistence { @@ -18,6 +20,8 @@ public void AddPersistence(IServiceCollection services) RegisterDataStores(services, settings); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); @@ -29,6 +33,7 @@ public void AddInstaller(IServiceCollection services) RegisterSettings(services); ConfigureDbContext(services); + services.AddSingleton(); services.AddScoped(); RegisterBodyStorageInstaller(services, settings); } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBatchWriter.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBatchWriter.cs new file mode 100644 index 0000000000..ac225a6cdb --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBatchWriter.cs @@ -0,0 +1,23 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; + +// Writes the audit half of an ingestion batch inside the transaction the unit of work has opened, +// after the failed message writer so that a batch's audit rows commit together with the known +// endpoints it recorded. +class AuditBatchWriter(ServiceControlDbContext dbContext, IAuditIngestionSqlDialect dialect) +{ + public async Task Write(IReadOnlyCollection messages, IReadOnlyCollection snapshots, CancellationToken cancellationToken = default) + { + if (messages.Count > 0) + { + await dialect.InsertAuditMessages(dbContext, [.. messages], cancellationToken); + } + + if (snapshots.Count > 0) + { + await dialect.InsertSagaSnapshots(dbContext, [.. snapshots], cancellationToken); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBodyStorage.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBodyStorage.cs new file mode 100644 index 0000000000..84fb16bdd9 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditBodyStorage.cs @@ -0,0 +1,12 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +/// +/// Audit bodies are keyed by the row's ingestion hour and then its unique message id, so retention +/// can drop an hour's bodies with one prefix delete per store instead of one delete per message. +/// +public static class AuditBodyStorage +{ + public static string BodyId(DateTime createdOn, Guid uniqueMessageId) => $"{Prefix(createdOn)}{uniqueMessageId}"; + + public static string Prefix(DateTime createdOn) => $"audit/{createdOn:yyyy-MM-dd-HH}/"; +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditHours.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditHours.cs new file mode 100644 index 0000000000..06ff44af8d --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditHours.cs @@ -0,0 +1,19 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +/// +/// The hour is the unit audit storage is organised by: every row is stamped with the hour it was +/// ingested in, PostgreSQL partitions the audit tables by it, external bodies are keyed by it, and +/// retention drops it whole. +/// +public static class AuditHours +{ + /// + /// How far ahead partitions are provisioned. Long, because only the retention owner provisions + /// them and an ingestion worker cannot insert into an hour nobody provisioned. + /// + public static readonly TimeSpan Lookahead = TimeSpan.FromHours(48); + + public static DateTime Truncate(DateTime utc) => new(utc.Year, utc.Month, utc.Day, utc.Hour, 0, 0, DateTimeKind.Utc); + + public static string PartitionName(string tableName, DateTime hour) => $"{tableName}_{hour:yyyyMMddHH}"; +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/EFAuditIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/EFAuditIngestionUnitOfWork.cs new file mode 100644 index 0000000000..6f4e513a8f --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/EFAuditIngestionUnitOfWork.cs @@ -0,0 +1,117 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using System.Collections.Concurrent; +using NServiceBus; +using ServiceControl.MessageAuditing; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.Infrastructure; +using ServiceControl.Persistence.UnitOfWork; +using ServiceControl.SagaAudit; + +// Record runs concurrently across the batch, so it only adds to thread safe collections and queues +// the external body writes on the parent. Every database call happens in the parent's Complete. +public class EFAuditIngestionUnitOfWork( + EFIngestionUnitOfWork parentUnitOfWork, + IBodyStoragePersistence storagePersistence, + EFPersisterSettings settings, + DateTime createdOn) : IAuditIngestionUnitOfWork +{ + readonly ConcurrentQueue messages = new(); + readonly ConcurrentQueue snapshots = new(); + + /// + /// The ingestion hour every row of this batch is stamped with. Fixed when the batch starts so + /// that a body written during Record and the row written during Complete agree on it. + /// + public DateTime CreatedOn { get; } = createdOn; + + internal IReadOnlyCollection Messages => messages; + + internal IReadOnlyCollection Snapshots => snapshots; + + internal bool IsEmpty => messages.IsEmpty && snapshots.IsEmpty; + + public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) + { + var uniqueMessageId = Guid.Parse(processedMessage.UniqueMessageId ?? processedMessage.Headers.UniqueId()); + var headers = processedMessage.Headers; + var metadata = processedMessage.MessageMetadata; + var contentType = headers.GetValueOrDefault(Headers.ContentType, "text/plain"); + var (bodyText, storeExternally) = MessageBodyClassifier.Classify(headers, body, settings.BodyStorage.MaxBodySizeToStore); + + if (storeExternally) + { + parentUnitOfWork.RecordBodyWrite( + storagePersistence.WriteBody(AuditBodyStorage.BodyId(CreatedOn, uniqueMessageId), body, contentType, cancellationToken)); + } + + var sendingEndpoint = GetMetadata(metadata, "SendingEndpoint"); + var receivingEndpoint = GetMetadata(metadata, "ReceivingEndpoint"); + + messages.Enqueue(new AuditMessageEntity + { + CreatedOn = CreatedOn, + UniqueMessageId = uniqueMessageId, + MessageId = GetMetadata(metadata, "MessageId"), + MessageType = GetMetadata(metadata, "MessageType"), + TimeSent = GetMetadata(metadata, "TimeSent"), + ProcessedAt = processedMessage.ProcessedAt, + ConversationId = GetMetadata(metadata, "ConversationId"), + IsSystemMessage = GetMetadata(metadata, "IsSystemMessage"), + Status = GetMetadata(metadata, "IsRetried") ? MessageStatus.ResolvedSuccessfully : MessageStatus.Successful, + SendingEndpointName = sendingEndpoint?.Name, + SendingEndpointHostId = sendingEndpoint?.HostId, + SendingEndpointHost = sendingEndpoint?.Host, + ReceivingEndpointName = receivingEndpoint?.Name, + ReceivingEndpointHostId = receivingEndpoint?.HostId, + ReceivingEndpointHost = receivingEndpoint?.Host, + CriticalTimeTicks = GetMetadata(metadata, "CriticalTime")?.Ticks, + ProcessingTimeTicks = GetMetadata(metadata, "ProcessingTime")?.Ticks, + DeliveryTimeTicks = GetMetadata(metadata, "DeliveryTime")?.Ticks, + HeadersJson = MessageHeaders.Write(headers), + BodyText = bodyText, + BodyStoredExternally = storeExternally, + BodySize = body.Length, + BodyContentType = contentType + }); + + return Task.CompletedTask; + } + + public Task RecordSagaSnapshot(SagaSnapshot sagaSnapshot, CancellationToken cancellationToken = default) + { + snapshots.Enqueue(new SagaSnapshotEntity + { + CreatedOn = CreatedOn, + SagaId = sagaSnapshot.SagaId, + SagaType = sagaSnapshot.SagaType, + Status = sagaSnapshot.Status, + StartTime = AsUtc(sagaSnapshot.StartTime), + FinishTime = AsUtc(sagaSnapshot.FinishTime), + ProcessedAt = AsUtc(sagaSnapshot.ProcessedAt), + Endpoint = sagaSnapshot.Endpoint, + StateAfterChange = sagaSnapshot.StateAfterChange, + InitiatingMessageJson = SagaSnapshotJson.Write(sagaSnapshot.InitiatingMessage), + OutgoingMessagesJson = SagaSnapshotJson.Write(sagaSnapshot.OutgoingMessages) + }); + + return Task.CompletedTask; + } + + // Saga times arrive deserialized from the saga audit message, where a value without an offset + // comes back Unspecified. They are UTC on the wire, and PostgreSQL refuses an Unspecified kind. + static DateTime AsUtc(DateTime value) => value.Kind switch + { + DateTimeKind.Utc => value, + DateTimeKind.Local => value.ToUniversalTime(), + DateTimeKind.Unspecified => DateTime.SpecifyKind(value, DateTimeKind.Utc), + _ => throw new ArgumentOutOfRangeException(nameof(value), value.Kind, "Unknown DateTimeKind") + }; + + static T? GetMetadata(Dictionary metadata, string key) => + metadata.TryGetValue(key, out var value) && value is T typed ? typed : default; +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditIngestionSqlDialect.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditIngestionSqlDialect.cs new file mode 100644 index 0000000000..cfbbb40df8 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditIngestionSqlDialect.cs @@ -0,0 +1,17 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; + +/// +/// The provider-specific SQL of the audit ingestion batch. Both statements are plain multi-row +/// inserts with no conflict clause: audit rows are never deduplicated, so a competing writer can +/// never collide with them. Implementations run on the DbContext connection inside the transaction +/// the caller has already opened. +/// +public interface IAuditIngestionSqlDialect +{ + Task InsertAuditMessages(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default); + + Task InsertSagaSnapshots(ServiceControlDbContext dbContext, IReadOnlyList rows, CancellationToken cancellationToken = default); +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs new file mode 100644 index 0000000000..2abc46b6f5 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs @@ -0,0 +1,17 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using ServiceControl.Persistence.EFCore.DbContexts; + +/// +/// The provider-specific lifecycle of the audit tables' hourly partitions. Only the owner of a +/// database calls this: setup, to provision the initial window, and the retention sweeper, to keep +/// the window ahead of the clock. Ingestion never issues DDL. +/// +public interface IAuditPartitionManager +{ + /// + /// Creates every hourly partition in [fromHour, toHourExclusive) that does not exist yet, for + /// both audit tables. A no-op on a provider that does not partition. + /// + Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive, CancellationToken cancellationToken = default); +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaSnapshotJson.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaSnapshotJson.cs new file mode 100644 index 0000000000..bfa2dbda5d --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaSnapshotJson.cs @@ -0,0 +1,25 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using System.Text.Json; +using System.Text.Json.Serialization; +using ServiceControl.SagaAudit; + +public static class SagaSnapshotJson +{ + public static string? Write(InitiatingMessage? initiatingMessage) => + initiatingMessage is null ? null : JsonSerializer.Serialize(initiatingMessage, SagaSnapshotJsonContext.Default.InitiatingMessage); + + public static string Write(List outgoingMessages) => + JsonSerializer.Serialize(outgoingMessages, SagaSnapshotJsonContext.Default.ListResultingMessage); + + public static InitiatingMessage? ReadInitiatingMessage(string? json) => + json is null ? null : JsonSerializer.Deserialize(json, SagaSnapshotJsonContext.Default.InitiatingMessage); + + public static List ReadOutgoingMessages(string? json) => + json is null ? [] : JsonSerializer.Deserialize(json, SagaSnapshotJsonContext.Default.ListResultingMessage) ?? []; +} + +// Source generated serialization, which keeps the reflection-based serializer off the ingestion hot path. +[JsonSerializable(typeof(InitiatingMessage))] +[JsonSerializable(typeof(List))] +partial class SagaSnapshotJsonContext : JsonSerializerContext; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs index a9ed66165b..d23ccec148 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs @@ -27,6 +27,9 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con return; } + // Audit body ids carry their ingestion hour as a directory, so the path may not exist yet. + Directory.CreateDirectory(Path.GetDirectoryName(filePath)!); + // A unique temp name lets concurrent writers of the same body race without clobbering. var tempFilePath = $"{filePath}.{Guid.NewGuid():N}.tmp"; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs index cbb6a9173e..c86258c79b 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs @@ -1,40 +1,46 @@ namespace ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; using System.Collections.Concurrent; +using Microsoft.EntityFrameworkCore; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.UnitOfWork; -// RecordFailedProcessingAttempt runs concurrently across the batch, so the Record methods only -// add to thread safe collections. Every database call happens in Complete, on one thread. +// The Record methods run concurrently across the batch, so they only add to thread safe +// collections. Every database call happens in Complete, on one thread, inside one transaction +// shared by the failed message, known endpoint and audit rows of the batch. public class EFIngestionUnitOfWork : IIngestionUnitOfWork { readonly ServiceControlDbContext dbContext; readonly IAsyncDisposable scope; readonly IFailedMessageIngestionSqlDialect dialect; + readonly IAuditIngestionSqlDialect auditDialect; readonly TimeProvider timeProvider; + readonly EFAuditIngestionUnitOfWork audit; readonly ConcurrentQueue failedProcessingAttempts = new(); readonly ConcurrentQueue bodyWrites = new(); readonly ConcurrentQueue knownEndpoints = new(); readonly ConcurrentQueue confirmedRetries = new(); - public EFIngestionUnitOfWork(IAsyncDisposable scope, ServiceControlDbContext dbContext, IBodyStoragePersistence storagePersistence, EFPersisterSettings settings, IFailedMessageIngestionSqlDialect dialect, TimeProvider timeProvider) + public EFIngestionUnitOfWork(IAsyncDisposable scope, ServiceControlDbContext dbContext, IBodyStoragePersistence storagePersistence, EFPersisterSettings settings, IFailedMessageIngestionSqlDialect dialect, IAuditIngestionSqlDialect auditDialect, TimeProvider timeProvider) { this.scope = scope; this.dbContext = dbContext; this.dialect = dialect; + this.auditDialect = auditDialect; this.timeProvider = timeProvider; Recoverability = new EFRecoverabilityIngestionUnitOfWork(this, storagePersistence, settings); Monitoring = new EFMonitoringIngestionUnitOfWork(this); + audit = new EFAuditIngestionUnitOfWork(this, storagePersistence, settings, AuditHours.Truncate(timeProvider.GetUtcNow().UtcDateTime)); } public IMonitoringIngestionUnitOfWork Monitoring { get; } public IRecoverabilityIngestionUnitOfWork Recoverability { get; } - // Stays null until the EF audit persistence lands and the manifest advertises SupportsAuditIngestion. - public IAuditIngestionUnitOfWork? Audit => null; + public IAuditIngestionUnitOfWork Audit => audit; internal void Record(RecordedFailedProcessingAttempt attempt) => failedProcessingAttempts.Enqueue(attempt); @@ -49,9 +55,25 @@ public async Task Complete(CancellationToken cancellationToken = default) // External bodies are written before the rows that point at them await Task.WhenAll(bodyWrites); - var writer = new FailedMessageBatchWriter(dbContext, dialect); + if (failedProcessingAttempts.IsEmpty && knownEndpoints.IsEmpty && confirmedRetries.IsEmpty && audit.IsEmpty) + { + return; + } - await writer.Write(failedProcessingAttempts, knownEndpoints, confirmedRetries, timeProvider.GetUtcNow().UtcDateTime, cancellationToken); + var now = timeProvider.GetUtcNow().UtcDateTime; + var failedMessageWriter = new FailedMessageBatchWriter(dbContext, dialect); + var auditWriter = new AuditBatchWriter(dbContext, auditDialect); + + var strategy = dbContext.Database.CreateExecutionStrategy(); + await strategy.ExecuteAsync(async ct => + { + await using var transaction = await dbContext.Database.BeginTransactionAsync(ct); + + await failedMessageWriter.Write(failedProcessingAttempts, knownEndpoints, confirmedRetries, now, ct); + await auditWriter.Write(audit.Messages, audit.Snapshots, ct); + + await transaction.CommitAsync(ct); + }, cancellationToken); } public async ValueTask DisposeAsync() diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs index bd4d4a2d9d..f40d34b2f6 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWorkFactory.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; using Microsoft.Extensions.DependencyInjection; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.UnitOfWork; @@ -11,6 +12,7 @@ public class EFIngestionUnitOfWorkFactory( MinimumRequiredStorageState storageState, IBodyStoragePersistence storagePersistence, IFailedMessageIngestionSqlDialect dialect, + IAuditIngestionSqlDialect auditDialect, TimeProvider timeProvider) : IIngestionUnitOfWorkFactory { public ValueTask StartNew(CancellationToken cancellationToken = default) @@ -18,7 +20,7 @@ public ValueTask StartNew(CancellationToken cancellationTo var scope = serviceProvider.CreateAsyncScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); var settings = scope.ServiceProvider.GetRequiredService(); - var unitOfWork = new EFIngestionUnitOfWork(scope, dbContext, storagePersistence, settings, dialect, timeProvider); + var unitOfWork = new EFIngestionUnitOfWork(scope, dbContext, storagePersistence, settings, dialect, auditDialect, timeProvider); return ValueTask.FromResult(unitOfWork); } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/FailedMessageBatchWriter.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/FailedMessageBatchWriter.cs index 0bd86675b8..8eb6fd364e 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/FailedMessageBatchWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/FailedMessageBatchWriter.cs @@ -6,10 +6,11 @@ namespace ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Persistence.EFCore.Entities; using ServiceControl.Persistence.EFCore.Infrastructure; -// Writes one ingestion batch inside a single transaction. The statements providers genuinely -// differ on (the upserts) come from the injected dialect; everything portable stays here as -// set-based EF operations. Statement order matters: a message that fails and is retry-confirmed -// in the same batch must end Resolved, which the resolve running last is what gives it. +// Writes the failed message half of one ingestion batch, inside the transaction the unit of work +// has opened. The statements providers genuinely differ on (the upserts) come from the injected +// dialect; everything portable stays here as set-based EF operations. Statement order matters: a +// message that fails and is retry-confirmed in the same batch must end Resolved, which the resolve +// running last is what gives it. class FailedMessageBatchWriter(ServiceControlDbContext dbContext, IFailedMessageIngestionSqlDialect dialect) { public async Task Write( @@ -23,34 +24,21 @@ public async Task Write( var endpoints = BuildEndpointRows(knownEndpoints); var retries = FoldRetries(confirmedRetries); - if (failedMessages.Count == 0 && endpoints.Count == 0 && retries.Length == 0) + if (failedMessages.Count > 0) { - return; + await dialect.UpsertFailedMessages(dbContext, failedMessages, cancellationToken); + await ReplaceGroups(failedMessages, groups, cancellationToken); } - var strategy = dbContext.Database.CreateExecutionStrategy(); - await strategy.ExecuteAsync(async ct => + if (endpoints.Count > 0) { - await using var transaction = await dbContext.Database.BeginTransactionAsync(ct); - - if (failedMessages.Count > 0) - { - await dialect.UpsertFailedMessages(dbContext, failedMessages, ct); - await ReplaceGroups(failedMessages, groups, ct); - } - - if (endpoints.Count > 0) - { - await dialect.InsertMissingKnownEndpoints(dbContext, endpoints, ct); - } - - if (retries.Length > 0) - { - await ResolveRetried(retries, now, ct); - } + await dialect.InsertMissingKnownEndpoints(dbContext, endpoints, cancellationToken); + } - await transaction.CommitAsync(ct); - }, cancellationToken); + if (retries.Length > 0) + { + await ResolveRetried(retries, now, cancellationToken); + } } static (List Messages, List Groups) Fold( diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionProvisioningTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionProvisioningTests.cs new file mode 100644 index 0000000000..104b6dcba0 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionProvisioningTests.cs @@ -0,0 +1,96 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class AuditPartitionProvisioningTests : AuditIngestionTestBase +{ + [TestCase("audit_messages")] + [TestCase("saga_snapshots")] + public async Task Setup_provisions_the_hour_before_now_through_the_lookahead(string table) + { + var partitions = await Partitions(table); + + var firstHour = IngestionHour.AddHours(-1); + var lastHour = IngestionHour + AuditHours.Lookahead - TimeSpan.FromHours(1); + + using (Assert.EnterMultipleScope()) + { + Assert.That(partitions, Has.Count.EqualTo((int)AuditHours.Lookahead.TotalHours + 1)); + Assert.That(partitions, Does.Contain(AuditHours.PartitionName(table, firstHour))); + Assert.That(partitions, Does.Contain(AuditHours.PartitionName(table, lastHour))); + Assert.That(partitions, Does.Not.Contain(AuditHours.PartitionName(table, lastHour.AddHours(1)))); + } + } + + [Test] + public async Task Provisioning_an_existing_window_again_changes_nothing() + { + var before = await Partitions("audit_messages"); + + await EnsurePartitions(IngestionHour.AddHours(-1), IngestionHour + AuditHours.Lookahead); + + Assert.That(await Partitions("audit_messages"), Is.EqualTo(before)); + } + + [Test] + public async Task Provisioning_extends_the_window_without_touching_existing_partitions() + { + var end = IngestionHour + AuditHours.Lookahead; + + await EnsurePartitions(end, end.AddHours(2)); + + var partitions = await Partitions("audit_messages"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(partitions, Has.Count.EqualTo((int)AuditHours.Lookahead.TotalHours + 3)); + Assert.That(partitions, Does.Contain(AuditHours.PartitionName("audit_messages", end.AddHours(1)))); + } + } + + [Test] + public async Task An_ingested_row_lands_in_the_partition_of_its_ingestion_hour() + { + var audit = new IngestedAudit(); + + await IngestAudit(audit); + + var partition = await Query(dbContext => + { + var sql = "SELECT tableoid::regclass::text AS \"Value\" FROM " + SchemaQualifiedTableName.For(dbContext) + " WHERE unique_message_id = {0}"; + + return dbContext.Database.SqlQueryRaw(sql, audit.UniqueMessageId).SingleAsync(); + }); + + Assert.That(partition, Does.EndWith(AuditHours.PartitionName("audit_messages", IngestionHour))); + } + + Task EnsurePartitions(DateTime fromHour, DateTime toHourExclusive) => + Query(async dbContext => + { + await ServiceProvider.GetRequiredService().EnsurePartitions(dbContext, fromHour, toHourExclusive); + return true; + }); + + Task> Partitions(string table) => + Query(dbContext => dbContext.Database + .SqlQueryRaw(""" + SELECT c.relname AS "Value" + FROM pg_inherits i + JOIN pg_class c ON c.oid = i.inhrelid + JOIN pg_class p ON p.oid = i.inhparent + JOIN pg_namespace n ON n.oid = p.relnamespace + WHERE p.relname = {0} AND n.nspname = {1} + ORDER BY c.relname + """, table, dbContext.Schema) + .ToListAsync()); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionBodyTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionBodyTests.cs new file mode 100644 index 0000000000..0524d3165c --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionBodyTests.cs @@ -0,0 +1,95 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Implementation.Audit; + +// Bodies are always stored. MaxBodySizeToStore only decides whether the body lives inline in +// BodyText or in external storage, under a key that names the row's ingestion hour. +class AuditIngestionBodyTests : AuditIngestionTestBase +{ + const int Cap = 64; + + [SetUp] + public void ShrinkTheBodyCap() => EFSettings.BodyStorage.MaxBodySizeToStore = Cap; + + [Test] + public async Task Text_within_the_cap_is_stored_inline() + { + var audit = new IngestedAudit { Body = Encoding.UTF8.GetBytes("1") }; + + await IngestAudit(audit); + + var row = await GetAuditMessage(audit.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyText, Is.EqualTo("1")); + Assert.That(row.BodyStoredExternally, Is.False); + Assert.That(row.BodySize, Is.EqualTo(audit.Body.Length)); + } + + Assert.That(RecordedBodies.Written, Is.Empty); + } + + [Test] + public async Task Text_over_the_cap_goes_external_under_the_ingestion_hour() + { + var body = Encoding.UTF8.GetBytes(new string('x', Cap * 2)); + var audit = new IngestedAudit { Body = body }; + + await IngestAudit(audit); + + var row = await GetAuditMessage(audit.UniqueMessageId); + var written = RecordedBodies.Written.Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyStoredExternally, Is.True); + Assert.That(row.BodyText, Is.EqualTo(new string('x', Cap)), "A search prefix stays inline"); + Assert.That(row.BodySize, Is.EqualTo(body.Length)); + Assert.That(written.BodyId, Is.EqualTo(AuditBodyStorage.BodyId(row.CreatedOn, audit.UniqueMessageId))); + Assert.That(written.BodyId, Does.StartWith($"audit/{row.CreatedOn:yyyy-MM-dd-HH}/")); + Assert.That(written.Body, Is.EqualTo(body)); + Assert.That(written.ContentType, Is.EqualTo(audit.ContentType)); + } + } + + [Test] + public async Task A_binary_body_goes_external_whatever_its_size() + { + var audit = new IngestedAudit { ContentType = "application/octet-stream", Body = BitConverter.GetBytes(0xDEADBEEF) }; + + await IngestAudit(audit); + + var row = await GetAuditMessage(audit.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyStoredExternally, Is.True); + Assert.That(row.BodyText, Is.Null); + Assert.That(RecordedBodies.Written.Single().BodyId, Is.EqualTo(AuditBodyStorage.BodyId(row.CreatedOn, audit.UniqueMessageId))); + } + } + + [Test] + public async Task An_empty_body_is_neither_stored_inline_nor_externally() + { + var audit = new IngestedAudit { Body = [] }; + + await IngestAudit(audit); + + var row = await GetAuditMessage(audit.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.BodyText, Is.Null); + Assert.That(row.BodyStoredExternally, Is.False); + Assert.That(row.BodySize, Is.Zero); + Assert.That(RecordedBodies.Written, Is.Empty); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTestBase.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTestBase.cs new file mode 100644 index 0000000000..874f33d0fb --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTestBase.cs @@ -0,0 +1,77 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.SagaAudit; + +abstract class AuditIngestionTestBase : IngestionTestBase +{ + protected AuditIngestionTestBase() => + RegisterServices = services => services.AddSingleton(RecordedBodies); + + protected InMemoryBodyStoragePersistence RecordedBodies { get; } = new(); + + protected EFPersisterSettings EFSettings => (EFPersisterSettings)PersistenceSettings; + + protected DateTime IngestionHour => AuditHours.Truncate(Now); + + protected Task IngestAudit(params IngestedAudit[] audits) => + InBatch(async unitOfWork => + { + foreach (var audit in audits) + { + await unitOfWork.Audit.RecordProcessedMessage(audit.ToProcessedMessage(), audit.Body); + } + }); + + protected Task IngestSnapshots(params SagaSnapshot[] snapshots) => + InBatch(async unitOfWork => + { + foreach (var snapshot in snapshots) + { + await unitOfWork.Audit.RecordSagaSnapshot(snapshot); + } + }); + + protected async Task GetAuditMessage(Guid uniqueMessageId) + { + var rows = await GetAuditMessages(uniqueMessageId); + + Assert.That(rows, Has.Count.EqualTo(1), $"Expected exactly one audit row for {uniqueMessageId}"); + + return rows[0]; + } + + protected Task> GetAuditMessages(Guid uniqueMessageId) => + Query(dbContext => dbContext.AuditMessages.AsNoTracking().Where(m => m.UniqueMessageId == uniqueMessageId).OrderBy(m => m.Id).ToListAsync()); + + protected Task CountAuditMessages() => + Query(dbContext => dbContext.AuditMessages.AsNoTracking().CountAsync()); + + protected Task> GetSagaSnapshots(Guid sagaId) => + Query(dbContext => dbContext.SagaSnapshots.AsNoTracking().Where(s => s.SagaId == sagaId).OrderBy(s => s.Id).ToListAsync()); + + protected Task> GetKnownEndpoints(IReadOnlyCollection ids) => + Query(dbContext => dbContext.KnownEndpoints.AsNoTracking().Where(e => ids.Contains(e.Id)).ToListAsync()); + + protected Task FindFailedMessage(Guid uniqueMessageId) => + Query(dbContext => dbContext.FailedMessages.AsNoTracking().SingleOrDefaultAsync(m => m.UniqueMessageId == uniqueMessageId)); + + protected async Task Query(Func> query) + { + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + return await query(dbContext); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTests.cs new file mode 100644 index 0000000000..292847eadc --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditIngestionTests.cs @@ -0,0 +1,149 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class AuditIngestionTests : AuditIngestionTestBase +{ + [Test] + public async Task Writes_every_mapped_column_of_an_audit_message() + { + var audit = new IngestedAudit { IsSystemMessage = true }; + + await IngestAudit(audit); + + var row = await GetAuditMessage(audit.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.CreatedOn, Is.EqualTo(IngestionHour)); + Assert.That(row.MessageId, Is.EqualTo(audit.MessageId)); + Assert.That(row.MessageType, Is.EqualTo(audit.MessageType)); + Assert.That(row.TimeSent, Is.EqualTo(audit.TimeSent)); + Assert.That(row.ProcessedAt, Is.EqualTo(audit.ProcessingEnded)); + Assert.That(row.ConversationId, Is.EqualTo(audit.ConversationId)); + Assert.That(row.IsSystemMessage, Is.True); + Assert.That(row.Status, Is.EqualTo(MessageStatus.Successful)); + Assert.That(row.SendingEndpointName, Is.EqualTo(audit.SendingEndpoint.Name)); + Assert.That(row.SendingEndpointHost, Is.EqualTo(audit.SendingEndpoint.Host)); + Assert.That(row.SendingEndpointHostId, Is.EqualTo(audit.SendingEndpoint.HostId)); + Assert.That(row.ReceivingEndpointName, Is.EqualTo(audit.ReceivingEndpoint.Name)); + Assert.That(row.ReceivingEndpointHost, Is.EqualTo(audit.ReceivingEndpoint.Host)); + Assert.That(row.ReceivingEndpointHostId, Is.EqualTo(audit.ReceivingEndpoint.HostId)); + Assert.That(row.CriticalTimeTicks, Is.EqualTo((audit.ProcessingEnded - audit.TimeSent).Ticks)); + Assert.That(row.ProcessingTimeTicks, Is.EqualTo((audit.ProcessingEnded - audit.ProcessingStarted).Ticks)); + Assert.That(row.DeliveryTimeTicks, Is.EqualTo((audit.ProcessingStarted - audit.TimeSent).Ticks)); + Assert.That(MessageHeaders.Read(row.HeadersJson), Is.EqualTo(audit.Headers)); + Assert.That(row.BodyContentType, Is.EqualTo(audit.ContentType)); + Assert.That(row.BodySize, Is.EqualTo(audit.Body.Length)); + } + } + + [Test] + public async Task Timestamps_are_read_back_as_utc() + { + var audit = new IngestedAudit(); + + await IngestAudit(audit); + + var row = await GetAuditMessage(audit.UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.CreatedOn.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(row.ProcessedAt.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(row.TimeSent.Value.Kind, Is.EqualTo(DateTimeKind.Utc)); + } + } + + [Test] + public async Task A_successfully_retried_message_is_recorded_as_resolved_successfully() + { + var failedMessageId = Guid.NewGuid(); + var audit = new IngestedAudit { RetryOf = failedMessageId.ToString() }; + + await IngestAudit(audit); + + var row = await GetAuditMessage(failedMessageId); + + Assert.That(row.Status, Is.EqualTo(MessageStatus.ResolvedSuccessfully)); + } + + // RavenDB deduplicates a redelivered audit message on its document id. The relational persisters + // deliberately do not, because that would put an index probe on every insert of the hot path. + [Test] + public async Task A_redelivered_audit_message_produces_a_second_row() + { + var audit = new IngestedAudit(); + + await IngestAudit(audit); + await IngestAudit(audit); + + var rows = await GetAuditMessages(audit.UniqueMessageId); + + Assert.That(rows, Has.Count.EqualTo(2)); + Assert.That(rows.Select(row => row.Id), Is.Unique); + } + + [Test] + public async Task A_batch_larger_than_one_statement_is_written_in_full() + { + var audits = Enumerable.Range(0, 120).Select(_ => new IngestedAudit()).ToArray(); + + await IngestAudit(audits); + + Assert.That(await CountAuditMessages(), Is.EqualTo(audits.Length)); + } + + [Test] + public async Task Audit_rows_and_known_endpoints_commit_in_one_batch() + { + var audit = new IngestedAudit(); + var endpoint = new KnownEndpoint { EndpointDetails = audit.ReceivingEndpoint }; + + await InBatch(async unitOfWork => + { + await unitOfWork.Audit.RecordProcessedMessage(audit.ToProcessedMessage(), audit.Body); + await unitOfWork.Monitoring.RecordKnownEndpoint(endpoint); + }); + + var knownEndpoints = await GetKnownEndpoints([audit.ReceivingEndpoint.GetDeterministicId()]); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await GetAuditMessages(audit.UniqueMessageId), Has.Count.EqualTo(1)); + Assert.That(knownEndpoints, Has.Count.EqualTo(1)); + } + } + + [Test] + public async Task Failed_and_audited_messages_share_one_batch() + { + var failure = new IngestedFailure(); + var audit = new IngestedAudit(); + + await InBatch(async unitOfWork => + { + await unitOfWork.Recoverability.RecordFailedProcessingAttempt(failure.Context, failure.ProcessingAttempt, failure.Groups); + await unitOfWork.Audit.RecordProcessedMessage(audit.ToProcessedMessage(), audit.Body); + }); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await FindFailedMessage(failure.UniqueMessageId), Is.Not.Null); + Assert.That(await GetAuditMessages(audit.UniqueMessageId), Has.Count.EqualTo(1)); + } + } + + [Test] + public async Task An_empty_batch_writes_nothing() + { + await InBatch(_ => Task.CompletedTask); + + Assert.That(await CountAuditMessages(), Is.Zero); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/IngestedAudit.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/IngestedAudit.cs new file mode 100644 index 0000000000..507a72c973 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/IngestedAudit.cs @@ -0,0 +1,83 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Text; +using NServiceBus; +using ServiceControl.MessageAuditing; +using ServiceControl.Operations; +using ServiceControl.Persistence.Infrastructure; + +/// +/// An audit message the way the audit pipeline hands it to the unit of work: the headers an endpoint +/// wrote, and the metadata the enrichers derive from them. +/// +class IngestedAudit +{ + public string MessageId { get; init; } = Guid.NewGuid().ToString(); + public string EndpointName { get; init; } = "Sales"; + public string ContentType { get; init; } = "text/xml"; + public byte[] Body { get; init; } = Encoding.UTF8.GetBytes("1"); + public DateTime TimeSent { get; init; } = new(2026, 7, 22, 9, 59, 0, DateTimeKind.Utc); + public DateTime ProcessingStarted { get; init; } = new(2026, 7, 22, 10, 0, 0, DateTimeKind.Utc); + public DateTime ProcessingEnded { get; init; } = new(2026, 7, 22, 10, 0, 1, DateTimeKind.Utc); + public MessageIntent MessageIntent { get; init; } = MessageIntent.Send; + public string MessageType { get; init; } = "MyCompany.Sales.OrderPlaced"; + public string ConversationId { get; init; } = Guid.NewGuid().ToString(); + public bool IsSystemMessage { get; init; } + public string RetryOf { get; init; } + public EndpointDetails SendingEndpoint { get; init; } = new() { Name = "Ordering", Host = "SenderHost", HostId = Guid.NewGuid() }; + public EndpointDetails ReceivingEndpoint { get; init; } = new() { Name = "Sales", Host = "ReceiverHost", HostId = Guid.NewGuid() }; + + public Dictionary Headers => field ??= BuildHeaders(); + + Dictionary BuildHeaders() + { + var headers = new Dictionary + { + [NServiceBus.Headers.MessageId] = MessageId, + [NServiceBus.Headers.ProcessingEndpoint] = EndpointName, + [NServiceBus.Headers.ContentType] = ContentType, + [NServiceBus.Headers.EnclosedMessageTypes] = MessageType, + [NServiceBus.Headers.MessageIntent] = MessageIntent.ToString(), + [NServiceBus.Headers.ConversationId] = ConversationId, + [NServiceBus.Headers.TimeSent] = DateTimeOffsetHelper.ToWireFormattedString(new DateTimeOffset(TimeSent)), + [NServiceBus.Headers.ProcessingStarted] = DateTimeOffsetHelper.ToWireFormattedString(new DateTimeOffset(ProcessingStarted)), + [NServiceBus.Headers.ProcessingEnded] = DateTimeOffsetHelper.ToWireFormattedString(new DateTimeOffset(ProcessingEnded)), + [NServiceBus.Headers.OriginatingEndpoint] = SendingEndpoint.Name, + [NServiceBus.Headers.OriginatingMachine] = SendingEndpoint.Host, + [NServiceBus.Headers.OriginatingHostId] = SendingEndpoint.HostId.ToString(), + [NServiceBus.Headers.HostId] = ReceivingEndpoint.HostId.ToString(), + [NServiceBus.Headers.HostDisplayName] = ReceivingEndpoint.Host + }; + + if (RetryOf != null) + { + headers["ServiceControl.Retry.UniqueMessageId"] = RetryOf; + } + + return headers; + } + + public Dictionary Metadata => new() + { + ["MessageId"] = MessageId, + ["MessageIntent"] = MessageIntent, + ["MessageType"] = MessageType, + ["IsSystemMessage"] = IsSystemMessage, + ["TimeSent"] = TimeSent, + ["ConversationId"] = ConversationId, + ["SendingEndpoint"] = SendingEndpoint, + ["ReceivingEndpoint"] = ReceivingEndpoint, + ["CriticalTime"] = ProcessingEnded - TimeSent, + ["ProcessingTime"] = ProcessingEnded - ProcessingStarted, + ["DeliveryTime"] = ProcessingStarted - TimeSent, + ["IsRetried"] = RetryOf != null + }; + + public ProcessedMessage ToProcessedMessage() => new(Headers, Metadata); + + public string UniqueMessageIdString => Headers.UniqueId(); + + public Guid UniqueMessageId => Guid.Parse(UniqueMessageIdString); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaSnapshotIngestionTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaSnapshotIngestionTests.cs new file mode 100644 index 0000000000..953c3438cb --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaSnapshotIngestionTests.cs @@ -0,0 +1,122 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.SagaAudit; + +class SagaSnapshotIngestionTests : AuditIngestionTestBase +{ + [Test] + public async Task Writes_every_mapped_column_of_a_saga_snapshot() + { + var snapshot = Snapshot(); + + await IngestSnapshots(snapshot); + + var row = (await GetSagaSnapshots(snapshot.SagaId)).Single(); + var initiatingMessage = SagaSnapshotJson.ReadInitiatingMessage(row.InitiatingMessageJson); + var outgoingMessages = SagaSnapshotJson.ReadOutgoingMessages(row.OutgoingMessagesJson); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.CreatedOn, Is.EqualTo(IngestionHour)); + Assert.That(row.SagaType, Is.EqualTo(snapshot.SagaType)); + Assert.That(row.Status, Is.EqualTo(snapshot.Status)); + Assert.That(row.StartTime, Is.EqualTo(snapshot.StartTime)); + Assert.That(row.FinishTime, Is.EqualTo(snapshot.FinishTime)); + Assert.That(row.ProcessedAt, Is.EqualTo(snapshot.ProcessedAt)); + Assert.That(row.Endpoint, Is.EqualTo(snapshot.Endpoint)); + Assert.That(row.StateAfterChange, Is.EqualTo(snapshot.StateAfterChange)); + Assert.That(initiatingMessage.MessageId, Is.EqualTo(snapshot.InitiatingMessage.MessageId)); + Assert.That(initiatingMessage.OriginatingEndpoint, Is.EqualTo(snapshot.InitiatingMessage.OriginatingEndpoint)); + Assert.That(initiatingMessage.TimeSent, Is.EqualTo(snapshot.InitiatingMessage.TimeSent)); + Assert.That(initiatingMessage.IsSagaTimeoutMessage, Is.True); + Assert.That(outgoingMessages, Has.Count.EqualTo(2)); + Assert.That(outgoingMessages.Select(m => m.Destination), Is.EqualTo(snapshot.OutgoingMessages.Select(m => m.Destination))); + Assert.That(outgoingMessages[0].DeliveryDelay, Is.EqualTo(snapshot.OutgoingMessages[0].DeliveryDelay)); + Assert.That(outgoingMessages[1].DeliverAt, Is.EqualTo(snapshot.OutgoingMessages[1].DeliverAt)); + } + } + + [Test] + public async Task Times_without_a_kind_are_stored_as_utc() + { + var snapshot = Snapshot(); + snapshot.StartTime = DateTime.SpecifyKind(snapshot.StartTime, DateTimeKind.Unspecified); + snapshot.FinishTime = DateTime.SpecifyKind(snapshot.FinishTime, DateTimeKind.Unspecified); + snapshot.ProcessedAt = DateTime.SpecifyKind(snapshot.ProcessedAt, DateTimeKind.Unspecified); + + await IngestSnapshots(snapshot); + + var row = (await GetSagaSnapshots(snapshot.SagaId)).Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.StartTime, Is.EqualTo(snapshot.StartTime)); + Assert.That(row.StartTime.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(row.FinishTime.Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(row.ProcessedAt.Kind, Is.EqualTo(DateTimeKind.Utc)); + } + } + + // Snapshots carry no natural key, so a redelivered saga audit message adds a second step. This + // is asserted because it differs from RavenDB, which assigns the document id itself. + [Test] + public async Task A_redelivered_saga_audit_message_produces_a_second_snapshot() + { + var snapshot = Snapshot(); + + await IngestSnapshots(snapshot); + await IngestSnapshots(snapshot); + + Assert.That(await GetSagaSnapshots(snapshot.SagaId), Has.Count.EqualTo(2)); + } + + [Test] + public async Task A_snapshot_without_an_initiating_message_round_trips() + { + var snapshot = Snapshot(); + snapshot.InitiatingMessage = null; + snapshot.OutgoingMessages.Clear(); + + await IngestSnapshots(snapshot); + + var row = (await GetSagaSnapshots(snapshot.SagaId)).Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(row.InitiatingMessageJson, Is.Null); + Assert.That(SagaSnapshotJson.ReadOutgoingMessages(row.OutgoingMessagesJson), Is.Empty); + } + } + + static SagaSnapshot Snapshot() => new() + { + SagaId = Guid.NewGuid(), + SagaType = "MyCompany.Sales.OrderSaga", + Status = SagaStateChangeStatus.Updated, + StartTime = new DateTime(2026, 7, 22, 10, 0, 0, DateTimeKind.Utc), + FinishTime = new DateTime(2026, 7, 22, 10, 0, 2, DateTimeKind.Utc), + ProcessedAt = new DateTime(2026, 7, 22, 10, 0, 2, DateTimeKind.Utc), + Endpoint = "Sales", + StateAfterChange = "{\"OrderId\":42}", + InitiatingMessage = new InitiatingMessage + { + MessageId = Guid.NewGuid().ToString(), + IsSagaTimeoutMessage = true, + OriginatingEndpoint = "Ordering", + OriginatingMachine = "SenderHost", + TimeSent = new DateTime(2026, 7, 22, 9, 59, 0, DateTimeKind.Utc), + MessageType = "MyCompany.Sales.OrderPlaced", + Intent = "Send" + }, + OutgoingMessages = + [ + new ResultingMessage { MessageId = Guid.NewGuid().ToString(), Destination = "Billing", TimeSent = new DateTime(2026, 7, 22, 10, 0, 1, DateTimeKind.Utc), MessageType = "MyCompany.Billing.BillOrder", Intent = "Send", DeliveryDelay = TimeSpan.FromMinutes(5) }, + new ResultingMessage { MessageId = Guid.NewGuid().ToString(), Destination = "Sales", TimeSent = new DateTime(2026, 7, 22, 10, 0, 1, DateTimeKind.Utc), MessageType = "MyCompany.Sales.OrderTimeout", Intent = "Send", DeliverAt = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc) } + ] + }; +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs index 070056f6d1..4f497e604c 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs @@ -44,6 +44,25 @@ public void TearDown() _ => throw new ArgumentOutOfRangeException(nameof(kind)) }; + [TestCase(InMemory)] + [TestCase(FileSystem)] + public async Task Round_trips_a_body_whose_id_names_a_directory(string kind) + { + var store = CreateStore(kind); + var bodyId = $"audit/2026-09-14-00/{Guid.NewGuid()}"; + var body = Encoding.UTF8.GetBytes("hello world"); + + await store.WriteBody(bodyId, body, "text/plain"); + + var result = await store.ReadBody(bodyId); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + } + } + [TestCase(InMemory)] [TestCase(FileSystem)] public async Task Round_trips_a_small_uncompressed_body(string kind) diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 0ceacb0227..a8af1fc7cb 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -594,10 +594,14 @@ competing consumers against one database, and the primary is the only writer to migrations including the PostgreSQL partitioning DDL. No behaviour: nothing writes or reads yet, and the manifests still say false. The full-text index is not here, see step 4. 2. **Ingestion write path.** `IAuditIngestionSqlDialect`, `EFAuditIngestionUnitOfWork`, wiring - `EFIngestionUnitOfWork.Audit`. Tested through the persistence test base. -3. **Retention, partitions and locking.** `IAuditPartitionManager` and `IRetentionLock` per provider, - the sweeper's audit pass, the body prefix delete on all three body stores, the lookahead custom - check. + `EFIngestionUnitOfWork.Audit`, one transaction around the failed message and audit rows of a + batch. Also the provisioning half of `IAuditPartitionManager`, because on PostgreSQL nothing can + be inserted into an hour nobody provisioned: setup provisions the hour before now through the + lookahead, and the file system body store creates the hour directory the audit body keys name. + Tested through the persistence test base. On `john/audit_ef_3`. +3. **Retention, partitions and locking.** The rest of `IAuditPartitionManager` (list expired, drop) + and `IRetentionLock` per provider, the sweeper's audit pass including the lookahead top-up, the + body prefix delete on all three body stores, the lookahead custom check. 4. **Queries.** The five message view unions, audit counts, saga history, and the third step of body arbitration. The full-text index lands here rather than with the schema, because the indexed expression and the query expression have to be written together or PostgreSQL silently downgrades @@ -619,8 +623,8 @@ competing consumers against one database, and the primary is the only writer to service is marked superseded. Each pull request leaves both EF acceptance suites and the RavenDB suites passing, and steps 1 to 5 -leave `SupportsAuditIngestion` false so nothing activates early. Step 1 is on this branch, rebased -onto master on 14 September 2026. +leave `SupportsAuditIngestion` false so nothing activates early. Step 1 is on `john/audit_ef_1`, +rebased onto master on 14 September 2026; the code layout move is `john/audit_ef_2`. ## Testing From 7711eb5d5afbcfe77aba963ec6d8f5e03f6e1452 Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 11:46:13 +1000 Subject: [PATCH 16/21] Add audit retention, partition lifecycle and the retention lock The retention sweeper gains an audit pass. Audit rows expire an hour at a time once the whole hour is behind the cutoff: the hour's external bodies go first, by one prefix delete per store, then the hour's rows. On PostgreSQL that is a DROP TABLE of the hour's partitions; on SQL Server, which does not partition, it is a batched delete paced like the failed message sweep. The same pass keeps the provisioned partition window ahead of the clock, because the sweeper is the only host that issues DDL. AuditRetentionPeriod defaults to seven days, the value the management utility and the container image apply. Every sweep now runs under a session scoped database lock, ported from the audit spike: pg_try_advisory_lock on PostgreSQL, sp_getapplock on SQL Server, on a dedicated unpooled connection, with a zero timeout so a host that cannot take it skips the pass. The resource name carries the configured schema, so instances sharing a database through different schemas do not take turns. This puts error and event log retention under the lock for the first time, deliberately: two hosts dropping the same partition is unforgiving in a way two hosts running the same batched delete is not. A custom check on the PostgreSQL persister fails when the newest provisioned partition ends less than twelve hours ahead, so a sweeper that has stopped provisioning is visible before ingestion stops. --- .../Audit/AuditPartitionCustomCheck.cs | 37 ++++++ .../Audit/PostgreSqlAuditPartitionManager.cs | 94 +++++++++++++-- .../PostgreSqlPersistence.cs | 3 + .../PostgreSqlRetentionLock.cs | 59 ++++++++++ .../Audit/SqlServerAuditPartitionManager.cs | 36 +++++- .../SqlServerPersistence.cs | 1 + .../SqlServerRetentionLock.cs | 70 ++++++++++++ .../EFPersistenceConfigurationBase.cs | 2 + .../Abstractions/EFPersisterSettings.cs | 5 + .../Audit/IAuditPartitionManager.cs | 21 ++++ .../AzureBlobBodyStoragePersistence.cs | 8 ++ .../FileSystemBodyStoragePersistence.cs | 37 ++++++ .../BodyStorage/S3BodyStoragePersistence.cs | 24 ++++ .../Infrastructure/IBodyStoragePersistence.cs | 7 ++ .../Infrastructure/IRetentionLock.cs | 30 +++++ .../Infrastructure/Metrics/RetentionEntity.cs | 3 +- .../Metrics/RetentionMetrics.cs | 3 +- .../Infrastructure/RetentionSweeper.cs | 66 ++++++++++- ...CheckTests.VerifyCustomChecks.approved.txt | 1 + .../AuditPartitionRetentionTests.cs | 88 ++++++++++++++ .../EFCore/Audit/AuditRetentionTestBase.cs | 76 +++++++++++++ .../EFCore/Audit/AuditRetentionTests.cs | 107 ++++++++++++++++++ .../EFCore/Audit/RetentionLockTests.cs | 28 +++++ .../EFCore/BodyStoragePersistenceTests.cs | 36 ++++++ .../EFCore/InMemoryBodyStoragePersistence.cs | 35 ++++++ .../EFCore/RecordedRetentionMetrics.cs | 1 + .../InternalCustomCheckClassification.cs | 1 + src/audit-ef-persistence-plan.md | 22 +++- 28 files changed, 875 insertions(+), 26 deletions(-) create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlRetentionLock.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerRetentionLock.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Infrastructure/IRetentionLock.cs create mode 100644 src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTestBase.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/RetentionLockTests.cs diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs new file mode 100644 index 0000000000..37aa5a8a90 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs @@ -0,0 +1,37 @@ +namespace ServiceControl.Persistence.EFCore.PostgreSql.Audit; + +using Microsoft.Extensions.DependencyInjection; +using NServiceBus.CustomChecks; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation.Audit; + +// Only the retention owner provisions partitions, so an ingestion worker cannot insert into an hour +// the owner never reached. This makes that condition visible before it bites, on every host. +class AuditPartitionCustomCheck(IServiceScopeFactory scopeFactory, IAuditPartitionManager partitions, TimeProvider timeProvider) + : CustomCheck("Audit partition provisioning", "ServiceControl Health", TimeSpan.FromMinutes(5)) +{ + public static readonly TimeSpan Threshold = TimeSpan.FromHours(12); + + public override async Task PerformCheck(CancellationToken cancellationToken = default) + { + using var scope = scopeFactory.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + var end = await partitions.NewestProvisionedHourEnd(dbContext, cancellationToken); + var now = timeProvider.GetUtcNow().UtcDateTime; + + if (end is null) + { + return CheckResult.Failed("No audit partitions are provisioned, so audit ingestion cannot store anything. Run setup on the instance that owns this database."); + } + + if (end.Value - now < Threshold) + { + return CheckResult.Failed( + $"Audit partitions are provisioned only until {end:u}, less than {Threshold.TotalHours:0} hours ahead. " + + "The retention sweep provisions them, so the instance configured to run it is not sweeping. Audit ingestion stops once the last provisioned hour passes."); + } + + return CheckResult.Pass; + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs index 2a3d08619e..bbe62c224c 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/PostgreSqlAuditPartitionManager.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql.Audit; +using System.Globalization; using System.Text; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Infrastructure; @@ -9,16 +10,23 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql.Audit; using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; -// One statement per hour per table, all in one command, so provisioning a two day window is a -// single round trip. CREATE TABLE IF NOT EXISTS makes a re-run over an existing window harmless. +// Partitions are named {table}_{yyyyMMddHH}, which is what makes an hour's partition addressable +// without a catalog lookup and makes the catalog listing parseable back into hours. class PostgreSqlAuditPartitionManager : IAuditPartitionManager { + // One statement per hour per table, all in one command, so provisioning a two day window is a + // single round trip. CREATE TABLE IF NOT EXISTS makes a re-run over an existing window harmless. public async Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive, CancellationToken cancellationToken = default) { var sql = new StringBuilder(); - AppendPartitions(sql, dbContext, fromHour, toHourExclusive); - AppendPartitions(sql, dbContext, fromHour, toHourExclusive); + foreach (var table in Tables(dbContext)) + { + for (var hour = AuditHours.Truncate(fromHour); hour < toHourExclusive; hour = hour.AddHours(1)) + { + sql.Append($"CREATE TABLE IF NOT EXISTS {table.Partition(hour)} PARTITION OF {table.Parent} FOR VALUES FROM ('{Bound(hour)}') TO ('{Bound(hour.AddHours(1))}');\n"); + } + } if (sql.Length == 0) { @@ -28,19 +36,81 @@ public async Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime f await dbContext.Database.ExecuteSqlRawAsync(sql.ToString(), cancellationToken); } - static void AppendPartitions(StringBuilder sql, ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive) + public async Task> ListExpiredHours(ServiceControlDbContext dbContext, DateTime lastExpiredHour, CancellationToken cancellationToken = default) { - var parent = SchemaQualifiedTableName.For(dbContext); - var tableName = dbContext.Model.FindEntityType(typeof(TEntity))!.GetTableName()!; - var sqlGenerationHelper = dbContext.GetService(); + var hours = new SortedSet(); - for (var hour = AuditHours.Truncate(fromHour); hour < toHourExclusive; hour = hour.AddHours(1)) + foreach (var table in Tables(dbContext)) { - var partition = sqlGenerationHelper.DelimitIdentifier(AuditHours.PartitionName(tableName, hour), dbContext.Schema); + foreach (var hour in await PartitionHours(dbContext, table, cancellationToken)) + { + if (hour <= lastExpiredHour) + { + hours.Add(hour); + } + } + } + + return [.. hours]; + } + + // Dropping a partition outright is a metadata operation: the rows go with it, and the parent is + // locked for the moment it takes. Bodies are already gone by the time this runs. + public async Task DropHour(ServiceControlDbContext dbContext, DateTime hour, int batchSize, CancellationToken cancellationToken = default) + { + foreach (var table in Tables(dbContext)) + { + var sql = "DROP TABLE IF EXISTS " + table.Partition(hour); + + await dbContext.Database.ExecuteSqlRawAsync(sql, cancellationToken); + } + + return new HourDrop(RowsDeleted: 0, Completed: true); + } + + public async Task NewestProvisionedHourEnd(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) + { + var hours = await PartitionHours(dbContext, Table(dbContext), cancellationToken); + + return hours.Count == 0 ? null : hours.Max().AddHours(1); + } - sql.Append($"CREATE TABLE IF NOT EXISTS {partition} PARTITION OF {parent} FOR VALUES FROM ('{Bound(hour)}') TO ('{Bound(hour.AddHours(1))}');\n"); + static async Task> PartitionHours(ServiceControlDbContext dbContext, AuditTable table, CancellationToken cancellationToken) + { + var names = await dbContext.Database + .SqlQueryRaw(""" + SELECT c.relname AS "Value" + FROM pg_inherits i + JOIN pg_class c ON c.oid = i.inhrelid + JOIN pg_class p ON p.oid = i.inhparent + JOIN pg_namespace n ON n.oid = p.relnamespace + WHERE p.relname = {0} AND n.nspname = COALESCE(CAST({1} AS text), current_schema()) + """, table.Name, dbContext.Schema ?? (object)DBNull.Value) + .ToListAsync(cancellationToken); + + var hours = new List(names.Count); + + foreach (var name in names) + { + if (name.Length > table.Name.Length + 1 + && DateTime.TryParseExact(name[(table.Name.Length + 1)..], "yyyyMMddHH", CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var hour)) + { + hours.Add(hour); + } } + + return hours; } - static string Bound(DateTime hour) => hour.ToString("yyyy-MM-dd HH:00:00+00"); + static AuditTable[] Tables(ServiceControlDbContext dbContext) => [Table(dbContext), Table(dbContext)]; + + static AuditTable Table(ServiceControlDbContext dbContext) => + new(dbContext.Model.FindEntityType(typeof(TEntity))!.GetTableName()!, SchemaQualifiedTableName.For(dbContext), dbContext.Schema, dbContext.GetService()); + + static string Bound(DateTime hour) => hour.ToString("yyyy-MM-dd HH:00:00+00", CultureInfo.InvariantCulture); + + sealed record AuditTable(string Name, string Parent, string? Schema, ISqlGenerationHelper SqlGenerationHelper) + { + public string Partition(DateTime hour) => SqlGenerationHelper.DelimitIdentifier(AuditHours.PartitionName(Name, hour), Schema); + } } diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs index 0d466d7393..4861049bbe 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlPersistence.cs @@ -6,6 +6,7 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql; using Microsoft.EntityFrameworkCore.Migrations; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; +using ServiceControl.CustomChecks; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Implementation.Audit; @@ -23,6 +24,8 @@ public void AddPersistence(IServiceCollection services) services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddCustomCheck(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlRetentionLock.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlRetentionLock.cs new file mode 100644 index 0000000000..6502f72d1c --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlRetentionLock.cs @@ -0,0 +1,59 @@ +namespace ServiceControl.Persistence.EFCore.PostgreSql; + +using Npgsql; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.Infrastructure; + +// A session advisory lock on a dedicated, unpooled connection. Unpooled because a session lock +// outlives a pooled connection's return to the pool, and an unlock that failed would then leave the +// lock held by whichever consumer picks that connection up next. +class PostgreSqlRetentionLock(EFPersisterSettings settings) : IRetentionLock +{ + readonly string connectionString = new NpgsqlConnectionStringBuilder(settings.ConnectionString) { Pooling = false }.ConnectionString; + readonly string resource = RetentionLock.ResourceName(settings.Schema); + + public async Task TryAcquire(CancellationToken cancellationToken = default) + { + var connection = new NpgsqlConnection(connectionString); + var acquired = false; + + try + { + await connection.OpenAsync(cancellationToken); + + await using var command = connection.CreateCommand(); + command.CommandText = "SELECT pg_try_advisory_lock(hashtext(@resource))"; + command.Parameters.AddWithValue("resource", resource); + + acquired = await command.ExecuteScalarAsync(cancellationToken) is true; + + return acquired ? new Handle(connection, resource) : null; + } + finally + { + if (!acquired) + { + await connection.DisposeAsync(); + } + } + } + + sealed class Handle(NpgsqlConnection connection, string resource) : IAsyncDisposable + { + public async ValueTask DisposeAsync() + { + try + { + await using var command = connection.CreateCommand(); + command.CommandText = "SELECT pg_advisory_unlock(hashtext(@resource))"; + command.Parameters.AddWithValue("resource", resource); + + await command.ExecuteNonQueryAsync(); + } + finally + { + await connection.DisposeAsync(); + } + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs index c442556b2f..e131c1029f 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Audit/SqlServerAuditPartitionManager.cs @@ -1,12 +1,46 @@ namespace ServiceControl.Persistence.EFCore.SqlServer.Audit; +using Microsoft.EntityFrameworkCore; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Implementation.Audit; // SQL Server does not partition the audit tables: a full text index cannot be aligned to a -// partition scheme, so retention deletes by hour instead. There is nothing to provision. +// partition scheme, so an expired hour is deleted in bounded batches instead of dropped. class SqlServerAuditPartitionManager : IAuditPartitionManager { public Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive, CancellationToken cancellationToken = default) => Task.CompletedTask; + + public async Task> ListExpiredHours(ServiceControlDbContext dbContext, DateTime lastExpiredHour, CancellationToken cancellationToken = default) + { + var messageHours = dbContext.AuditMessages.Where(m => m.CreatedOn <= lastExpiredHour).Select(m => m.CreatedOn); + var snapshotHours = dbContext.SagaSnapshots.Where(s => s.CreatedOn <= lastExpiredHour).Select(s => s.CreatedOn); + + return await messageHours.Union(snapshotHours).OrderBy(hour => hour).ToListAsync(cancellationToken); + } + + public async Task DropHour(ServiceControlDbContext dbContext, DateTime hour, int batchSize, CancellationToken cancellationToken = default) + { + var messagesDeleted = await dbContext.AuditMessages + .Where(m => m.CreatedOn == hour) + .OrderBy(m => m.Id) + .Take(batchSize) + .ExecuteDeleteAsync(cancellationToken); + + if (messagesDeleted == batchSize) + { + return new HourDrop(messagesDeleted, Completed: false); + } + + var snapshotsDeleted = await dbContext.SagaSnapshots + .Where(s => s.CreatedOn == hour) + .OrderBy(s => s.Id) + .Take(batchSize) + .ExecuteDeleteAsync(cancellationToken); + + return new HourDrop(messagesDeleted + snapshotsDeleted, Completed: snapshotsDeleted < batchSize); + } + + public Task NewestProvisionedHourEnd(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + Task.FromResult(null); } diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs index d3a57cac45..1e211b09c6 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerPersistence.cs @@ -22,6 +22,7 @@ public void AddPersistence(IServiceCollection services) services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerRetentionLock.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerRetentionLock.cs new file mode 100644 index 0000000000..34d14c75e9 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerRetentionLock.cs @@ -0,0 +1,70 @@ +namespace ServiceControl.Persistence.EFCore.SqlServer; + +using System.Data; +using Microsoft.Data.SqlClient; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.Infrastructure; + +// A session application lock on a dedicated, unpooled connection. Unpooled because a session lock +// outlives a pooled connection's return to the pool, and a release that failed would then leave the +// lock held by whichever consumer picks that connection up next. +class SqlServerRetentionLock(EFPersisterSettings settings) : IRetentionLock +{ + readonly string connectionString = new SqlConnectionStringBuilder(settings.ConnectionString) { Pooling = false }.ConnectionString; + readonly string resource = RetentionLock.ResourceName(settings.Schema); + + public async Task TryAcquire(CancellationToken cancellationToken = default) + { + var connection = new SqlConnection(connectionString); + var acquired = false; + + try + { + await connection.OpenAsync(cancellationToken); + + await using var command = connection.CreateCommand(); + command.CommandText = "sp_getapplock"; + command.CommandType = CommandType.StoredProcedure; + command.Parameters.AddWithValue("@Resource", resource); + command.Parameters.AddWithValue("@LockMode", "Exclusive"); + command.Parameters.AddWithValue("@LockOwner", "Session"); + command.Parameters.AddWithValue("@LockTimeout", 0); + var result = command.Parameters.Add("@Result", SqlDbType.Int); + result.Direction = ParameterDirection.ReturnValue; + + await command.ExecuteNonQueryAsync(cancellationToken); + + acquired = result.Value is int status && status >= 0; + + return acquired ? new Handle(connection, resource) : null; + } + finally + { + if (!acquired) + { + await connection.DisposeAsync(); + } + } + } + + sealed class Handle(SqlConnection connection, string resource) : IAsyncDisposable + { + public async ValueTask DisposeAsync() + { + try + { + await using var command = connection.CreateCommand(); + command.CommandText = "sp_releaseapplock"; + command.CommandType = CommandType.StoredProcedure; + command.Parameters.AddWithValue("@Resource", resource); + command.Parameters.AddWithValue("@LockOwner", "Session"); + + await command.ExecuteNonQueryAsync(); + } + finally + { + await connection.DisposeAsync(); + } + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs index 727aef4b98..530b625c83 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs @@ -27,6 +27,7 @@ public abstract class EFPersistenceConfigurationBase : PersistenceConfiguration, const string MaxBodySizeToStoreKey = "MaxBodySizeToStore"; const string ErrorRetentionPeriodKey = "ErrorRetentionPeriod"; const string EventsRetentionPeriodKey = "EventsRetentionPeriod"; + const string AuditRetentionPeriodKey = "AuditRetentionPeriod"; const string SubscriptionCacheDurationKey = "SubscriptionCacheDuration"; const string ExternalIntegrationsDispatchingBatchSizeKey = "ExternalIntegrationsDispatchingBatchSize"; @@ -42,6 +43,7 @@ public PersistenceSettings CreateSettings(SettingsRootNamespace settingsRootName settings.CommandTimeout = SettingsReader.Read(settingsRootNamespace, CommandTimeoutKey, EFPersisterSettings.DefaultCommandTimeout); settings.ErrorRetentionPeriod = GetRequiredSetting(settingsRootNamespace, ErrorRetentionPeriodKey); settings.EventsRetentionPeriod = SettingsReader.Read(settingsRootNamespace, EventsRetentionPeriodKey, EFPersisterSettings.DefaultEventsRetentionPeriod); + settings.AuditRetentionPeriod = SettingsReader.Read(settingsRootNamespace, AuditRetentionPeriodKey, EFPersisterSettings.DefaultAuditRetentionPeriod); settings.SubscriptionCacheDuration = SettingsReader.Read(settingsRootNamespace, SubscriptionCacheDurationKey, EFPersisterSettings.DefaultSubscriptionCacheDuration); settings.ExternalIntegrationsDispatchingBatchSize = ReadExternalIntegrationsDispatchingBatchSize(settingsRootNamespace); settings.QueryTimeout = QueryTimeLimit.Read(settingsRootNamespace, LoggerUtil.CreateStaticLogger()); diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs index 1d57e8de46..70e8d60d17 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs @@ -8,6 +8,10 @@ public abstract class EFPersisterSettings : PersistenceSettings public const int DefaultExternalIntegrationsDispatchingBatchSize = 100; public static readonly TimeSpan DefaultEventsRetentionPeriod = TimeSpan.FromDays(14); + // The same default the management utility and the container image apply, rather than the + // standalone audit instance's 30 days. + public static readonly TimeSpan DefaultAuditRetentionPeriod = TimeSpan.FromDays(7); + public static readonly TimeSpan DefaultSubscriptionCacheDuration = TimeSpan.FromSeconds(60); public required string ConnectionString { get; set; } @@ -25,6 +29,7 @@ public string? Schema public int CommandTimeout { get; set; } = DefaultCommandTimeout; public TimeSpan ErrorRetentionPeriod { get; set; } public TimeSpan EventsRetentionPeriod { get; set; } = DefaultEventsRetentionPeriod; + public TimeSpan AuditRetentionPeriod { get; set; } = DefaultAuditRetentionPeriod; public required BodyStorageSettings BodyStorage { get; set; } public int MaxRetryCount { get; set; } = 5; public int MaxRetryDelayInSeconds { get; set; } = 30; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs index 2abc46b6f5..9484f504ff 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/IAuditPartitionManager.cs @@ -14,4 +14,25 @@ public interface IAuditPartitionManager /// both audit tables. A no-op on a provider that does not partition. /// Task EnsurePartitions(ServiceControlDbContext dbContext, DateTime fromHour, DateTime toHourExclusive, CancellationToken cancellationToken = default); + + /// + /// The hours at or before that still hold rows, or a + /// partition, in either audit table. Oldest first. + /// + Task> ListExpiredHours(ServiceControlDbContext dbContext, DateTime lastExpiredHour, CancellationToken cancellationToken = default); + + /// + /// Removes the hour from both audit tables. A partitioning provider drops the hour's partitions + /// in one call. A deleting provider removes at most rows per table + /// per call and reports whether anything is left, so the caller can pace the batches. + /// + Task DropHour(ServiceControlDbContext dbContext, DateTime hour, int batchSize, CancellationToken cancellationToken = default); + + /// + /// When the newest provisioned partition ends, or null where the provider does not partition + /// and so can never run out of provisioned hours. + /// + Task NewestProvisionedHourEnd(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default); } + +public readonly record struct HourDrop(int RowsDeleted, bool Completed); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs index f07ab437a7..6bc5d91fae 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs @@ -95,4 +95,12 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => container.GetBlobClient(bodyId).DeleteIfExistsAsync(cancellationToken: cancellationToken); + + public async Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + await foreach (var blob in container.GetBlobsAsync(BlobTraits.None, BlobStates.None, prefix, cancellationToken)) + { + await container.DeleteBlobIfExistsAsync(blob.Name, cancellationToken: cancellationToken); + } + } } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs index d23ccec148..2744dce246 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs @@ -144,8 +144,45 @@ public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToke return Task.CompletedTask; } + // A prefix that ends in a separator names a directory, and deleting the directory removes every + // body of that hour at once. Any other prefix is matched file by file within its directory. + public Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + var path = Path.Combine(StoragePath, prefix); + + if (prefix.EndsWith('/')) + { + TryDeleteDirectory(path); + return Task.CompletedTask; + } + + var directory = Path.GetDirectoryName(path); + + if (directory is not null && Directory.Exists(directory)) + { + foreach (var file in Directory.EnumerateFiles(directory, $"{Path.GetFileName(path)}*")) + { + TryDelete(file); + } + } + + return Task.CompletedTask; + } + string GetBodyFilePath(string bodyId) => Path.Combine(StoragePath, $"{bodyId}.body"); + static void TryDeleteDirectory(string path) + { + try + { + Directory.Delete(path, recursive: true); + } + catch (DirectoryNotFoundException) + { + // Nothing to delete. + } + } + static void TryDelete(string filePath) { try diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs index f5194dab85..5fa851359d 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/S3BodyStoragePersistence.cs @@ -96,6 +96,30 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => client.DeleteObjectAsync(bucketName, Key(bodyId), cancellationToken); + // Listed and deleted a page at a time, so an hour of any size is removed in a bounded number of + // requests, one list and one bulk delete per thousand keys. + public async Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + var request = new ListObjectsV2Request { BucketName = bucketName, Prefix = Key(prefix) }; + ListObjectsV2Response response; + + do + { + response = await client.ListObjectsV2Async(request, cancellationToken); + + if (response.S3Objects is { Count: > 0 }) + { + await client.DeleteObjectsAsync(new DeleteObjectsRequest + { + BucketName = bucketName, + Objects = [.. response.S3Objects.Select(s3Object => new KeyVersion { Key = s3Object.Key })] + }, cancellationToken); + } + + request.ContinuationToken = response.NextContinuationToken; + } while (response.IsTruncated == true); + } + async Task Exists(string key, CancellationToken cancellationToken) { try diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs index 685efe7a01..a8084f2cd9 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/IBodyStoragePersistence.cs @@ -21,4 +21,11 @@ public interface IBodyStoragePersistence /// gone: callers delete the body before the row that names it. /// Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default); + + /// + /// Deletes every body whose id starts with the prefix. Audit bodies are keyed by their ingestion + /// hour, so retention drops an hour's bodies with one call rather than one per message. Like + /// , throws only when the store itself fails. + /// + Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default); } diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/IRetentionLock.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/IRetentionLock.cs new file mode 100644 index 0000000000..70bd4a2ee2 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/IRetentionLock.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Persistence.EFCore.Infrastructure; + +/// +/// A database-wide lock that keeps the retention sweep to one host at a time. In a correct +/// deployment only one host is configured to sweep, so the lock exists for the misconfigured one: +/// two hosts concurrently dropping the same partition is unforgiving in a way two hosts running the +/// same batched delete is not. +/// +/// +/// Implementations hold the lock on a dedicated connection for as long as the returned handle +/// lives. Both providers scope the lock to that session, so a host that crashes mid-sweep releases +/// it when its connection drops rather than wedging retention until someone intervenes. +/// +public interface IRetentionLock +{ + /// + /// Takes the lock without waiting. Null when another connection holds it. Disposing the handle + /// releases the lock. + /// + Task TryAcquire(CancellationToken cancellationToken = default); +} + +public static class RetentionLock +{ + /// + /// Scoped to the schema, so instances that share a database through different schemas do not + /// take turns sweeping. + /// + public static string ResourceName(string? schema) => schema is null ? "retention_sweep" : $"retention_sweep:{schema}"; +} diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionEntity.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionEntity.cs index 5fcd43f587..ca4cf04e64 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionEntity.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionEntity.cs @@ -4,5 +4,6 @@ public enum RetentionEntity { FailedMessages, EventLog, - GroupComments + GroupComments, + Audit } diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs index c51b748030..6b2875ceed 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/Metrics/RetentionMetrics.cs @@ -79,7 +79,8 @@ IEnumerable> ObserveConsecutiveFailures() [ EntityTag("failed_messages"), EntityTag("event_log"), - EntityTag("group_comments") + EntityTag("group_comments"), + EntityTag("audit") ]; const string MeterVersion = "0.1.0"; diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs index a5df9ae33b..38708e6f75 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs @@ -9,6 +9,7 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure.Metrics; // Deletes rows once they age past their retention period. @@ -16,7 +17,11 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure; // every run so a changed retention setting takes effect without rewriting any row. // // A manual sweep can be triggered via the API (see IRetentionSweeper / IRetentionApi) with -// caller-supplied cutoffs. +// caller-supplied cutoffs. +// +// Every sweep, timed or manual, runs under the database-wide retention lock, so at most one host +// is sweeping anything at any moment. A host that cannot take the lock skips the pass: the work is +// idempotent and hourly, so waiting behind the holder would only repeat what it just did. public class RetentionSweeper( ILogger logger, TimeProvider timeProvider, @@ -24,6 +29,8 @@ public class RetentionSweeper( IBodyStoragePersistence bodyStorage, RetentionMetrics metrics, EFPersisterSettings settings, + IRetentionLock retentionLock, + IAuditPartitionManager auditPartitions, IHostApplicationLifetime hostApplicationLifetime) : BackgroundService, IRetentionSweeper { const int BatchSize = 1000; @@ -108,8 +115,7 @@ async Task SweepWithoutAcquiringLock() try { // if the caller doesn't hand over a real cancellation token then use the application lifetime. - await SweepBody(errorCutoff, eventsCutoff, false, cancellation.Token); - lastFinishedAt = timeProvider.GetUtcNow().UtcDateTime; + await SweepUnderDatabaseLock(errorCutoff, eventsCutoff, false, cancellation.Token); } catch (OperationCanceledException) when (cancellation.Token.IsCancellationRequested) { @@ -138,8 +144,7 @@ async Task Sweep(DateTime? errorCutoff, DateTime? eventsCutoff, bool pace, Cance lastEventsCutoff = eventsCutoff; try { - await SweepBody(errorCutoff, eventsCutoff, pace, cancellationToken); - lastFinishedAt = timeProvider.GetUtcNow().UtcDateTime; + await SweepUnderDatabaseLock(errorCutoff, eventsCutoff, pace, cancellationToken); } finally { @@ -148,6 +153,20 @@ async Task Sweep(DateTime? errorCutoff, DateTime? eventsCutoff, bool pace, Cance } } + async Task SweepUnderDatabaseLock(DateTime? errorCutoff, DateTime? eventsCutoff, bool pace, CancellationToken cancellationToken) + { + await using var ownership = await retentionLock.TryAcquire(cancellationToken); + + if (ownership is null) + { + logger.LogInformation("Skipping the retention sweep: another instance holds the retention lock"); + return; + } + + await SweepBody(errorCutoff, eventsCutoff, pace, cancellationToken); + lastFinishedAt = timeProvider.GetUtcNow().UtcDateTime; + } + // The three sub-sweeps, isolated from lock management so both the locked Sweep path and the // manual background path (which already holds the lock) share one implementation. async Task SweepBody(DateTime? errorCutoff, DateTime? eventsCutoff, bool pace, CancellationToken cancellationToken) @@ -155,6 +174,43 @@ async Task SweepBody(DateTime? errorCutoff, DateTime? eventsCutoff, bool pace, C await RunPass(RetentionEntity.FailedMessages, token => SweepFailedMessages(pace, errorCutoff, token), cancellationToken); await RunPass(RetentionEntity.EventLog, token => SweepEventLogItems(pace, eventsCutoff, token), cancellationToken); await RunPass(RetentionEntity.GroupComments, SweepOrphanedGroupComments, cancellationToken); + await RunPass(RetentionEntity.Audit, token => SweepAudit(pace, token), cancellationToken); + } + + // Audit rows are stored by ingestion hour and expire an hour at a time, once the whole hour is + // behind the cutoff. The same pass keeps the provisioned window ahead of the clock, because the + // sweeper is the only host that issues DDL. + async Task SweepAudit(bool pace, CancellationToken cancellationToken) + { + var now = timeProvider.GetUtcNow().UtcDateTime; + var lastExpiredHour = AuditHours.Truncate((now - settings.AuditRetentionPeriod).AddHours(-1)); + + using var scope = serviceScopeFactory.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + await auditPartitions.EnsurePartitions(dbContext, AuditHours.Truncate(now), AuditHours.Truncate(now) + AuditHours.Lookahead, cancellationToken); + + foreach (var hour in await auditPartitions.ListExpiredHours(dbContext, lastExpiredHour, cancellationToken)) + { + // Bodies before rows, as for failed messages: a body that will not delete fails the pass + // with the hour's rows intact and the next sweep retries it, where the reverse would + // leave bodies nothing names any more. + await bodyStorage.DeleteBodiesWithPrefix(AuditBodyStorage.Prefix(hour), cancellationToken); + + HourDrop drop; + + do + { + drop = await auditPartitions.DropHour(dbContext, hour, BatchSize, cancellationToken); + + metrics.RecordRowsDeleted(RetentionEntity.Audit, drop.RowsDeleted); + + if (!drop.Completed && pace) + { + await Task.Delay(BatchPause, timeProvider, cancellationToken); + } + } while (!drop.Completed); + } } // Each pass is isolated so one failing kind of row does not stop the others from being diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt b/src/ServiceControl.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt index f3b0cc7462..7a84aba6a9 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt @@ -1 +1,2 @@ +ServiceControl Health: Audit partition provisioning Storage space: ServiceControl body storage \ No newline at end of file diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs new file mode 100644 index 0000000000..cf647b5fc9 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs @@ -0,0 +1,88 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NServiceBus.CustomChecks; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.EFCore.PostgreSql.Audit; + +class AuditPartitionRetentionTests : AuditRetentionTestBase +{ + [Test] + public async Task An_expired_hours_partitions_are_dropped_from_both_tables() + { + var expired = await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1)); + + await RunRetentionSweep(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await PartitionExists("audit_messages", expired), Is.False); + Assert.That(await PartitionExists("saga_snapshots", expired), Is.False); + } + } + + [Test] + public async Task An_expired_partition_that_never_received_a_row_is_dropped_too() + { + var expired = AuditHours.Truncate(Now - Retention).AddHours(-1); + await EnsurePartitions(expired, expired.AddHours(1)); + + await RunRetentionSweep(); + + Assert.That(await PartitionExists("audit_messages", expired), Is.False); + } + + [Test] + public async Task The_sweep_keeps_the_provisioned_window_ahead_of_the_clock() + { + AdvanceClock(TimeSpan.FromHours(30)); + + await RunRetentionSweep(); + + var end = await Query(dbContext => Partitions.NewestProvisionedHourEnd(dbContext)); + + Assert.That(end, Is.EqualTo(AuditHours.Truncate(Now) + AuditHours.Lookahead)); + } + + [Test] + public async Task The_provisioning_check_passes_while_the_window_is_ahead_and_fails_once_it_runs_short() + { + var check = new AuditPartitionCustomCheck( + ServiceProvider.GetRequiredService(), + Partitions, + ServiceProvider.GetRequiredService()); + + var beforehand = await check.PerformCheck(); + + AdvanceClock(AuditHours.Lookahead - AuditPartitionCustomCheck.Threshold + TimeSpan.FromHours(1)); + + var afterwards = await check.PerformCheck(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(beforehand.HasFailed, Is.False, beforehand.FailureReason); + Assert.That(afterwards.HasFailed, Is.True); + Assert.That(afterwards.FailureReason, Does.Contain("retention sweep")); + } + } + + async Task PartitionExists(string table, DateTime hour) + { + var names = await Query(dbContext => dbContext.Database + .SqlQueryRaw(""" + SELECT c.relname AS "Value" + FROM pg_inherits i + JOIN pg_class c ON c.oid = i.inhrelid + JOIN pg_class p ON p.oid = i.inhparent + JOIN pg_namespace n ON n.oid = p.relnamespace + WHERE p.relname = {0} AND n.nspname = {1} + """, table, dbContext.Schema) + .ToListAsync()); + + return names.Contains(AuditHours.PartitionName(table, hour)); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTestBase.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTestBase.cs new file mode 100644 index 0000000000..e20045f9c6 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTestBase.cs @@ -0,0 +1,76 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.SagaAudit; + +abstract class AuditRetentionTestBase : AuditIngestionTestBase +{ + protected static readonly TimeSpan Retention = TimeSpan.FromDays(7); + + [SetUp] + public void SetRetention() => EFSettings.AuditRetentionPeriod = Retention; + + protected Task RunRetentionSweep() => + ServiceProvider.GetServices().OfType().Single().SweepNow(TestContext.CurrentContext.CancellationToken); + + protected IAuditPartitionManager Partitions => ServiceProvider.GetRequiredService(); + + protected Task EnsurePartitions(DateTime fromHour, DateTime toHourExclusive) => + Query(async dbContext => + { + await Partitions.EnsurePartitions(dbContext, fromHour, toHourExclusive); + return true; + }); + + protected async Task SeedHour(DateTime hour, int messages = 2, int snapshots = 2, bool externalBodies = false) + { + await EnsurePartitions(hour, hour.AddHours(1)); + + await Store(Enumerable.Range(0, messages).Select(_ => new AuditMessageEntity + { + CreatedOn = hour, + UniqueMessageId = Guid.NewGuid(), + ProcessedAt = hour, + Status = MessageStatus.Successful, + HeadersJson = "{}", + BodyStoredExternally = externalBodies, + BodySize = 0 + }).ToArray()); + + await Store(Enumerable.Range(0, snapshots).Select(_ => new SagaSnapshotEntity + { + CreatedOn = hour, + SagaId = Guid.NewGuid(), + Status = SagaStateChangeStatus.Updated, + StartTime = hour, + FinishTime = hour, + ProcessedAt = hour + }).ToArray()); + + return hour; + } + + protected async Task HourIsGone(DateTime hour) => + !await Query(dbContext => dbContext.AuditMessages.AsNoTracking().AnyAsync(m => m.CreatedOn == hour)) + && !await Query(dbContext => dbContext.SagaSnapshots.AsNoTracking().AnyAsync(s => s.CreatedOn == hour)); + + protected async Task Store(params T[] entities) where T : class + { + using var scope = ServiceProvider.CreateScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + dbContext.Set().AddRange(entities); + + await dbContext.SaveChangesAsync(TestContext.CurrentContext.CancellationToken); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTests.cs new file mode 100644 index 0000000000..0fe557d8e5 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditRetentionTests.cs @@ -0,0 +1,107 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.EFCore.Infrastructure.Metrics; + +class AuditRetentionTests : AuditRetentionTestBase +{ + [Test] + public async Task Drops_hours_wholly_past_the_cutoff_and_keeps_the_rest() + { + var cutoff = Now - Retention; + var expired = await SeedHour(AuditHours.Truncate(cutoff).AddHours(-1)); + var ancient = await SeedHour(AuditHours.Truncate(cutoff).AddDays(-30)); + var straddling = await SeedHour(AuditHours.Truncate(cutoff)); + var live = await SeedHour(AuditHours.Truncate(Now)); + + await RunRetentionSweep(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await HourIsGone(expired), Is.True, "the hour that ends before the cutoff"); + Assert.That(await HourIsGone(ancient), Is.True, "a much older hour"); + Assert.That(await HourIsGone(straddling), Is.False, "the hour the cutoff falls in still has rows within retention"); + Assert.That(await HourIsGone(live), Is.False); + } + } + + [Test] + public async Task Deletes_an_expired_hours_bodies_by_prefix_before_its_rows() + { + var expired = await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1), externalBodies: true); + var live = await SeedHour(AuditHours.Truncate(Now), externalBodies: true); + + await RunRetentionSweep(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(RecordedBodies.DeletedPrefixes, Is.EqualTo(new[] { AuditBodyStorage.Prefix(expired) })); + Assert.That(await HourIsGone(expired), Is.True); + Assert.That(await HourIsGone(live), Is.False); + } + } + + [Test] + public async Task A_failing_body_delete_leaves_the_hours_rows_for_the_next_sweep() + { + var expired = await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1), externalBodies: true); + RecordedBodies.FailDeletePrefixFor.Add(AuditBodyStorage.Prefix(expired)); + + await RunRetentionSweep(); + + Assert.That(await HourIsGone(expired), Is.False, "rows outlive a body that could not be deleted, so nothing is stranded"); + + RecordedBodies.FailDeletePrefixFor.Clear(); + + await RunRetentionSweep(); + + Assert.That(await HourIsGone(expired), Is.True); + } + + [Test] + public async Task An_hour_larger_than_one_batch_is_removed_in_full() + { + var expired = await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1), messages: 2500, snapshots: 1100); + + await RunRetentionSweep(); + + Assert.That(await HourIsGone(expired), Is.True); + } + + [Test] + public async Task The_audit_pass_reports_its_outcome() + { + using var recorded = new RecordedRetentionMetrics(ServiceProvider.GetRequiredService()); + await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1)); + + await RunRetentionSweep(); + + var cycle = recorded.Cycles(RetentionEntity.Audit).Single(); + + Assert.That(cycle.Result, Is.EqualTo("success")); + } + + [Test] + public async Task A_sweep_is_skipped_while_another_connection_holds_the_lock() + { + var expired = await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1)); + + await using (await ServiceProvider.GetRequiredService().TryAcquire()) + { + await RunRetentionSweep(); + + Assert.That(await HourIsGone(expired), Is.False, "the other holder is sweeping, so this pass stands down"); + } + + await RunRetentionSweep(); + + Assert.That(await HourIsGone(expired), Is.True); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/RetentionLockTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/RetentionLockTests.cs new file mode 100644 index 0000000000..c10c251178 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/RetentionLockTests.cs @@ -0,0 +1,28 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class RetentionLockTests : PersistenceTestBase +{ + [Test] + public async Task A_second_acquisition_fails_while_the_first_is_held_and_succeeds_once_released() + { + var retentionLock = ServiceProvider.GetRequiredService(); + + var first = await retentionLock.TryAcquire(); + + Assert.That(first, Is.Not.Null); + Assert.That(await retentionLock.TryAcquire(), Is.Null, "the lock is held"); + + await first.DisposeAsync(); + + var second = await retentionLock.TryAcquire(); + + Assert.That(second, Is.Not.Null, "the lock was released"); + + await second.DisposeAsync(); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs index 4f497e604c..77eb1daae3 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/BodyStoragePersistenceTests.cs @@ -2,6 +2,7 @@ namespace ServiceControl.Persistence.Tests; using System; using System.IO; +using System.Linq; using System.Text; using System.Threading.Tasks; using NUnit.Framework; @@ -63,6 +64,41 @@ public async Task Round_trips_a_body_whose_id_names_a_directory(string kind) } } + [TestCase(InMemory)] + [TestCase(FileSystem)] + public async Task Deletes_every_body_under_a_prefix_and_nothing_else(string kind) + { + var store = CreateStore(kind); + var body = Encoding.UTF8.GetBytes("hello world"); + var expiredHour = "audit/2026-09-14-00/"; + var liveHour = "audit/2026-09-14-01/"; + string[] expired = [$"{expiredHour}{Guid.NewGuid()}", $"{expiredHour}{Guid.NewGuid()}"]; + var live = $"{liveHour}{Guid.NewGuid()}"; + + foreach (var bodyId in expired.Append(live)) + { + await store.WriteBody(bodyId, body, "text/plain"); + } + + await store.DeleteBodiesWithPrefix(expiredHour); + + using (Assert.EnterMultipleScope()) + { + Assert.That(await store.ReadBody(expired[0]), Is.Null); + Assert.That(await store.ReadBody(expired[1]), Is.Null); + Assert.That(await store.ReadBody(live), Is.Not.Null); + } + } + + [TestCase(InMemory)] + [TestCase(FileSystem)] + public async Task Deleting_a_prefix_nothing_was_written_under_succeeds(string kind) + { + var store = CreateStore(kind); + + await store.DeleteBodiesWithPrefix("audit/2026-09-14-00/"); + } + [TestCase(InMemory)] [TestCase(FileSystem)] public async Task Round_trips_a_small_uncompressed_body(string kind) diff --git a/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs b/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs index 4c0d30b2c2..6b25d4dc33 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Persistence.Tests; using System; using System.Collections.Generic; using System.IO; +using System.Linq; using System.Threading; using System.Threading.Tasks; using ServiceControl.Persistence.EFCore.Infrastructure; @@ -12,10 +13,24 @@ class InMemoryBodyStoragePersistence : IBodyStoragePersistence readonly object gate = new(); readonly List written = []; readonly List deleted = []; + readonly List deletedPrefixes = []; readonly Dictionary store = []; public HashSet FailDeleteFor { get; } = []; + public HashSet FailDeletePrefixFor { get; } = []; + + public IReadOnlyList DeletedPrefixes + { + get + { + lock (gate) + { + return [.. deletedPrefixes]; + } + } + } + public IReadOnlyList Written { get @@ -95,5 +110,25 @@ public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToke return Task.CompletedTask; } + public Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken = default) + { + if (FailDeletePrefixFor.Contains(prefix)) + { + throw new InvalidOperationException($"Simulated body storage failure for prefix {prefix}"); + } + + lock (gate) + { + deletedPrefixes.Add(prefix); + + foreach (var bodyId in store.Keys.Where(bodyId => bodyId.StartsWith(prefix, StringComparison.Ordinal)).ToList()) + { + store.Remove(bodyId); + } + } + + return Task.CompletedTask; + } + public record StoredBody(string BodyId, byte[] Body, string ContentType); } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/RecordedRetentionMetrics.cs b/src/ServiceControl.Persistence.Tests/EFCore/RecordedRetentionMetrics.cs index 903d0c4bf8..34a9442d73 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/RecordedRetentionMetrics.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/RecordedRetentionMetrics.cs @@ -78,6 +78,7 @@ void Add(Instrument instrument, double value, ReadOnlySpan "failed_messages", RetentionEntity.EventLog => "event_log", RetentionEntity.GroupComments => "group_comments", + RetentionEntity.Audit => "audit", _ => throw new ArgumentOutOfRangeException(nameof(entity)) }; diff --git a/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs b/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs index 0621462030..0119fbf572 100644 --- a/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs +++ b/src/ServiceControl.Persistence/InternalCustomCheckClassification.cs @@ -44,6 +44,7 @@ public static class InternalCustomCheckClassification "ServiceControl body storage", // EF Core persisters "Dead Letter Queue", // ASBS / IBMMQ / MSMQ "Audit Message Ingestion (local)", // audit-capable primary + "Audit partition provisioning", // PostgreSQL persister // ----- Audit instance (forwarded to the primary via ReportCustomCheckResult) ----- "Audit Message Ingestion", diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index a8af1fc7cb..2c0aa2d222 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -103,8 +103,9 @@ starts failing inserts. Mitigations, not fixes: - Provision a long lookahead (48 hours, against the spike's 6), so the outage has to be sustained. -- A custom check that fails when the newest provisioned partition is less than 12 hours ahead, so the - condition is visible before it bites. +- A custom check, `Audit partition provisioning`, registered by the PostgreSQL persister only, that + fails when the newest provisioned partition ends less than 12 hours ahead, so the condition is + visible before it bites. - The setup command provisions the initial window, so a fresh instance ingests before the first sweep. Revisiting this means letting ingesting hosts issue `CREATE TABLE IF NOT EXISTS` themselves. @@ -389,7 +390,10 @@ ensure partitions exist, list expired partitions, drop a partition. ### PostgreSQL Native `PARTITION BY RANGE (created_on)`, hourly partitions named `{table}_{yyyyMMddHH}`. Dropping an -expired hour is `DETACH` then `DROP TABLE`, a metadata operation. +expired hour is a single `DROP TABLE` of the partition, a metadata operation that holds the parent's +lock for the moment it takes. `DETACH CONCURRENTLY` would avoid even that, but it cannot run inside +a transaction block and EF Core's raw SQL execution supplies one, so the plain drop is used and the +brief lock accepted. ### SQL Server @@ -432,16 +436,22 @@ it. Port `TryAcquireLock`/`ReleaseLock` from the spike's `RetentionCleaner`, which already has both implementations: -- PostgreSQL: `SELECT pg_try_advisory_lock(hashtext('retention_cleaner'))`, released with +- PostgreSQL: `SELECT pg_try_advisory_lock(hashtext(@resource))`, released with `pg_advisory_unlock`. - SQL Server: `sp_getapplock` with `@LockMode = 'Exclusive'`, `@LockOwner = 'Session'` and `@LockTimeout = 0`, released with `sp_releaseapplock`. +The resource name is `retention_sweep`, suffixed with the configured schema when there is one, so +instances that share a database through different schemas do not take turns sweeping. That is also +what lets the test suites, which run schema-per-test in parallel, hold real locks. + The lifecycle is the spike's, unchanged: - A dedicated `DbConnection` is opened for the sweep and closed after it. Both locks are session scoped, so a host that crashes mid-sweep releases the lock when its connection drops rather than - wedging retention until someone intervenes. + wedging retention until someone intervenes. The connection is opened with pooling off: a session + lock outlives a pooled connection's return to the pool, and a release that failed would otherwise + leave the lock held by whichever consumer picked that connection up next. - Acquisition has a zero timeout. A sweeper that cannot take the lock logs and skips that pass instead of queueing behind the holder, because the work is idempotent and hourly. - Release is in a `finally`. @@ -601,7 +611,7 @@ competing consumers against one database, and the primary is the only writer to Tested through the persistence test base. On `john/audit_ef_3`. 3. **Retention, partitions and locking.** The rest of `IAuditPartitionManager` (list expired, drop) and `IRetentionLock` per provider, the sweeper's audit pass including the lookahead top-up, the - body prefix delete on all three body stores, the lookahead custom check. + body prefix delete on all three body stores, the lookahead custom check. On `john/audit_ef_4`. 4. **Queries.** The five message view unions, audit counts, saga history, and the third step of body arbitration. The full-text index lands here rather than with the schema, because the indexed expression and the query expression have to be written together or PostgreSQL silently downgrades From 6ead2274f6fffe28ca8c361842f6daa0b59fd2ee Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 14:37:22 +1000 Subject: [PATCH 17/21] Serve the message views, audit counts and saga history from EF Each message view is one statement over both tables: the failed and audit branches project to a common row, each carries its own sort and limit, and the union is sorted and paged once more. The branch limits are what make it fast, measured on PostgreSQL 16 with three million audit rows over 168 partitions: the same union with the sort outside made the planner hash the anti-join and scan every row, 1.2 seconds for page one, where this shape merges two index-ordered scans in 5 ms and takes 8 ms at page one hundred. Precedence is the rule the scatter gather has always applied, a failed row wins whatever its status, as an anti-join on the audit branch alone. The total is capped, because an exact count is linear in the audit table. Audit counts group the endpoint's non-system messages by processing day over the last thirty days, with the RavenDB persister's zero for an endpoint that only ever sent. Saga history pages a saga's snapshots newest first. Body arbitration gains its third step, the audit row by unique message id, with the external key composed in memory because a Guid rendered by SQL Server is upper case. The audit full-text index mirrors the failed one on both providers and is pinned to the query expression by the same test. On SQL Server it keys on a unique index over the identity column, since a full-text key index must be a single column and the audit primary key is composite. --- .../FullTextSearchSql.cs | 26 +- ...4030830_AddAuditFullTextSearch.Designer.cs | 1152 +++++++++++++++++ .../20260914030830_AddAuditFullTextSearch.cs | 22 + .../PostgreSqlFullTextSearchDialect.cs | 8 + .../FullTextSearchSql.cs | 53 +- ...4030834_AddAuditFullTextSearch.Designer.cs | 919 +++++++++++++ .../20260914030834_AddAuditFullTextSearch.cs | 26 + .../SqlServerFullTextSearchDialect.cs | 5 + .../Abstractions/BasePersistence.cs | 3 + .../Audit/AuditCountsDataStore.cs | 43 + .../Audit/AuditMessageQueryFilters.cs | 25 + .../Implementation/Audit/MessageRow.cs | 43 + .../Implementation/Audit/MessageRowQueries.cs | 176 +++ .../Implementation/Audit/MessageViewUnion.cs | 51 + .../Audit/SagaHistoryDataStore.cs | 57 + .../Implementation/BodyStorage/BodyStorage.cs | 39 +- .../Implementation/MessagesViewDataStore.cs | 81 +- .../Infrastructure/IFullTextSearchDialect.cs | 6 + .../ServiceControl.Persistence.EFCore.csproj | 2 + .../FullTextSearchIndexTests.cs | 36 +- .../EFCore/Audit/AuditBodyStorageTests.cs | 77 ++ .../EFCore/Audit/AuditCountsDataStoreTests.cs | 71 + .../EFCore/Audit/AuditMessagesViewTests.cs | 301 +++++ .../EFCore/Audit/SagaHistoryDataStoreTests.cs | 83 ++ src/audit-ef-persistence-plan.md | 55 +- 25 files changed, 3302 insertions(+), 58 deletions(-) create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.Designer.cs create mode 100644 src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.Designer.cs create mode 100644 src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditCountsDataStore.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditMessageQueryFilters.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRow.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRowQueries.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaHistoryDataStore.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditBodyStorageTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditCountsDataStoreTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditMessagesViewTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaHistoryDataStoreTests.cs diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/FullTextSearchSql.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/FullTextSearchSql.cs index 1887885ec9..601a30f7c1 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/FullTextSearchSql.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/FullTextSearchSql.cs @@ -12,6 +12,8 @@ static class FullTextSearchSql { const string IndexName = "ix_failed_messages_full_text"; const string TableName = "failed_messages"; + const string AuditIndexName = "ix_audit_messages_full_text"; + const string AuditTableName = "audit_messages"; // 'simple' rather than 'english': message and header content is technical, stemming and // stopword removal do more harm than good. @@ -29,9 +31,15 @@ static class FullTextSearchSql public const string IndexedExpression = $"""to_tsvector('{Configuration}', headers_json || ' ' || COALESCE(body_text, '') || ' ' || replace(replace(COALESCE(message_type, ''), '.', ' '), '+', ' '))"""; - public static readonly string Up = CreateIndexSql(null); + public static readonly string Up = CreateIndexSql(null, TableName, IndexName); - public static readonly string Down = DropIndexSql(null); + public static readonly string Down = DropIndexSql(null, IndexName); + + // An index created on the partitioned parent is created on every existing partition and on + // every partition provisioned later. + public static readonly string AuditUp = CreateIndexSql(null, AuditTableName, AuditIndexName); + + public static readonly string AuditDown = DropIndexSql(null, AuditIndexName); /// /// Re-renders the statement the migration carries, this time with the configured schema in it. @@ -42,18 +50,20 @@ static class FullTextSearchSql public static MigrationOperation Rewrite(SqlOperation operation, string schema) => operation.Sql switch { - var sql when sql == Up => WithSql(operation, CreateIndexSql(schema)), - var sql when sql == Down => WithSql(operation, DropIndexSql(schema)), + var sql when sql == Up => WithSql(operation, CreateIndexSql(schema, TableName, IndexName)), + var sql when sql == Down => WithSql(operation, DropIndexSql(schema, IndexName)), + var sql when sql == AuditUp => WithSql(operation, CreateIndexSql(schema, AuditTableName, AuditIndexName)), + var sql when sql == AuditDown => WithSql(operation, DropIndexSql(schema, AuditIndexName)), _ => operation }; - public static bool IsHandled(string sql) => sql == Up || sql == Down; + public static bool IsHandled(string sql) => sql == Up || sql == Down || sql == AuditUp || sql == AuditDown; - static string CreateIndexSql(string? schema) => - $"CREATE INDEX {IndexName} ON {Qualify(schema, TableName)} USING GIN ({IndexedExpression})"; + static string CreateIndexSql(string? schema, string tableName, string indexName) => + $"CREATE INDEX {indexName} ON {Qualify(schema, tableName)} USING GIN ({IndexedExpression})"; // An index belongs to its table's schema, so it is the index that gets qualified here. - static string DropIndexSql(string? schema) => $"DROP INDEX IF EXISTS {Qualify(schema, IndexName)}"; + static string DropIndexSql(string? schema, string indexName) => $"DROP INDEX IF EXISTS {Qualify(schema, indexName)}"; static string Qualify(string? schema, string name) => schema is null ? name : $"\"{schema}\".{name}"; diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.Designer.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.Designer.cs new file mode 100644 index 0000000000..7de9efa7da --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.Designer.cs @@ -0,0 +1,1152 @@ +// +using System; +using System.Collections.Generic; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; +using ServiceControl.Persistence.EFCore.PostgreSql; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations +{ + [DbContext(typeof(PostgreSqlServiceControlDbContext))] + [Migration("20260914030830_AddAuditFullTextSearch")] + partial class AddAuditFullTextSearch + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ArchiveOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("request_id"); + + b.Property("ArchiveType") + .HasColumnType("integer") + .HasColumnName("archive_type"); + + b.Property("OperationType") + .HasColumnType("integer") + .HasColumnName("operation_type"); + + b.Property("CurrentBatch") + .HasColumnType("integer") + .HasColumnName("current_batch"); + + b.Property("GroupName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("group_name"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_id"); + + b.Property("InitiatedByName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_name"); + + b.Property("NumberOfBatches") + .HasColumnType("integer") + .HasColumnName("number_of_batches"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("operation_id"); + + b.Property("Started") + .HasColumnType("timestamp with time zone") + .HasColumnName("started"); + + b.Property("TotalNumberOfMessages") + .HasColumnType("integer") + .HasColumnName("total_number_of_messages"); + + b.HasKey("RequestId", "ArchiveType", "OperationType") + .HasName("pk_archive_operations"); + + b.HasIndex("Started") + .HasDatabaseName("ix_archive_operations_started"); + + b.ToTable("archive_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.AuditMessageEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_on"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("body_content_type"); + + b.Property("BodySize") + .HasColumnType("integer") + .HasColumnName("body_size"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("BodyText") + .HasColumnType("text") + .HasColumnName("body_text"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("conversation_id"); + + b.Property("CriticalTimeTicks") + .HasColumnType("bigint") + .HasColumnName("critical_time_ticks"); + + b.Property("DeliveryTimeTicks") + .HasColumnType("bigint") + .HasColumnName("delivery_time_ticks"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("IsSystemMessage") + .HasColumnType("boolean") + .HasColumnName("is_system_message"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.Property("MessageType") + .HasColumnType("text") + .HasColumnName("message_type"); + + b.Property("ProcessedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("processed_at"); + + b.Property("ProcessingTimeTicks") + .HasColumnType("bigint") + .HasColumnName("processing_time_ticks"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_host"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("receiving_endpoint_host_id"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_name"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_host"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("sending_endpoint_host_id"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_name"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.Property("TimeSent") + .HasColumnType("timestamp with time zone") + .HasColumnName("time_sent"); + + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.HasKey("CreatedOn", "Id") + .HasName("pk_audit_messages"); + + b.HasIndex("ConversationId") + .HasDatabaseName("ix_audit_messages_conversation_id"); + + b.HasIndex("ProcessedAt") + .HasDatabaseName("ix_audit_messages_processed_at"); + + b.HasIndex("TimeSent") + .HasDatabaseName("ix_audit_messages_time_sent"); + + b.HasIndex("UniqueMessageId") + .HasDatabaseName("ix_audit_messages_unique_message_id"); + + b.HasIndex("ReceivingEndpointName", "CreatedOn") + .HasDatabaseName("ix_audit_messages_receiving_endpoint_name_created_on"); + + b.ToTable("audit_messages", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Category") + .IsRequired() + .HasColumnType("text") + .HasColumnName("category"); + + b.Property("CustomCheckId") + .IsRequired() + .HasColumnType("text") + .HasColumnName("custom_check_id"); + + b.Property("FailureReason") + .HasColumnType("text") + .HasColumnName("failure_reason"); + + b.Property("OriginatingEndpointHost") + .IsRequired() + .HasColumnType("text") + .HasColumnName("originating_endpoint_host"); + + b.Property("OriginatingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("originating_endpoint_host_id"); + + b.Property("OriginatingEndpointName") + .IsRequired() + .HasColumnType("text") + .HasColumnName("originating_endpoint_name"); + + b.Property("ReportedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("reported_at"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("Id") + .HasName("pk_custom_checks"); + + b.HasIndex("ReportedAt") + .HasDatabaseName("ix_custom_checks_reported_at"); + + b.HasIndex("Status", "ReportedAt") + .HasDatabaseName("ix_custom_checks_status_reported_at"); + + b.ToTable("custom_checks", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EndpointSettingsEntity", b => + { + b.Property("Name") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("name"); + + b.Property("TrackInstances") + .HasColumnType("boolean") + .HasColumnName("track_instances"); + + b.HasKey("Name") + .HasName("pk_endpoint_settings"); + + b.ToTable("endpoint_settings", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EventLogItemEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Category") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("category"); + + b.Property("Description") + .IsRequired() + .HasColumnType("text") + .HasColumnName("description"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("event_type"); + + b.Property("RaisedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("raised_at"); + + b.PrimitiveCollection>("RelatedTo") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("related_to"); + + b.Property("Severity") + .HasColumnType("integer") + .HasColumnName("severity"); + + b.HasKey("Id") + .HasName("pk_event_log_items"); + + b.HasIndex("RaisedAt", "Id") + .IsDescending() + .HasDatabaseName("ix_event_log_items_raised_at_id"); + + b.ToTable("event_log_items", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ExternalIntegrationDispatchRequestEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("DispatchContextJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("dispatch_context_json"); + + b.Property("DispatchContextTypeName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("dispatch_context_type_name"); + + b.HasKey("Id") + .HasName("pk_external_integration_dispatch_requests"); + + b.ToTable("external_integration_dispatch_requests", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("Body") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("body"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("text") + .HasColumnName("exception_info"); + + b.Property("FailedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("failed_at"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_audit_imports"); + + b.HasIndex("FailedAt") + .HasDatabaseName("ix_failed_audit_imports_failed_at"); + + b.ToTable("failed_audit_imports", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("Body") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("body"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("text") + .HasColumnName("exception_info"); + + b.Property("FailedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("failed_at"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_error_imports"); + + b.HasIndex("FailedAt") + .HasDatabaseName("ix_failed_error_imports_failed_at"); + + b.ToTable("failed_error_imports", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEditEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("EditId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("edit_id"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_message_edits"); + + b.HasIndex("EditId") + .HasDatabaseName("ix_failed_message_edits_edit_id"); + + b.ToTable("failed_message_edits", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("body_content_type"); + + b.Property("BodySize") + .HasColumnType("integer") + .HasColumnName("body_size"); + + b.Property("BodyStoredExternally") + .HasColumnType("boolean") + .HasColumnName("body_stored_externally"); + + b.Property("BodyText") + .HasColumnType("text") + .HasColumnName("body_text"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("conversation_id"); + + b.Property("ExceptionMessage") + .HasColumnType("text") + .HasColumnName("exception_message"); + + b.Property("ExceptionType") + .HasColumnType("text") + .HasColumnName("exception_type"); + + b.Property("FailingEndpointAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("failing_endpoint_address"); + + b.Property("FirstTimeOfFailure") + .HasColumnType("timestamp with time zone") + .HasColumnName("first_time_of_failure"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("text") + .HasColumnName("headers_json"); + + b.Property("IsSystemMessage") + .HasColumnType("boolean") + .HasColumnName("is_system_message"); + + b.Property("LastAttemptedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_attempted_at"); + + b.Property("LastModified") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_modified"); + + b.Property("LastTimeOfFailure") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_time_of_failure"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("message_id"); + + b.Property("MessageType") + .HasColumnType("text") + .HasColumnName("message_type"); + + b.Property("NumberOfProcessingAttempts") + .HasColumnType("integer") + .HasColumnName("number_of_processing_attempts"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_host"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("receiving_endpoint_host_id"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("receiving_endpoint_name"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_host"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uuid") + .HasColumnName("sending_endpoint_host_id"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sending_endpoint_name"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.Property("StatusChangedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("status_changed_at"); + + b.Property("TimeSent") + .HasColumnType("timestamp with time zone") + .HasColumnName("time_sent"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_messages"); + + b.HasIndex("ConversationId") + .HasDatabaseName("ix_failed_messages_conversation_id"); + + b.HasIndex("FailingEndpointAddress") + .HasDatabaseName("ix_failed_messages_failing_endpoint_address"); + + b.HasIndex("ReceivingEndpointName") + .HasDatabaseName("ix_failed_messages_receiving_endpoint_name"); + + b.HasIndex("StatusChangedAt") + .HasDatabaseName("ix_failed_messages_status_changed_at") + .HasFilter("status IN (2, 4)"); + + b.HasIndex("TimeSent") + .HasDatabaseName("ix_failed_messages_time_sent"); + + b.HasIndex("Status", "LastModified") + .HasDatabaseName("ix_failed_messages_status_last_modified"); + + b.ToTable("failed_messages", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.Property("FailedMessageUniqueId") + .HasColumnType("uuid") + .HasColumnName("failed_message_unique_id"); + + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("group_id"); + + b.Property("Title") + .IsRequired() + .HasColumnType("text") + .HasColumnName("title"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("character varying(255)") + .HasColumnName("type"); + + b.HasKey("FailedMessageUniqueId", "GroupId") + .HasName("pk_failed_message_groups"); + + b.HasIndex("GroupId") + .HasDatabaseName("ix_failed_message_groups_group_id"); + + b.HasIndex("Type", "GroupId") + .HasDatabaseName("ix_failed_message_groups_type_group_id"); + + b.ToTable("failed_message_groups", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageRetryEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uuid") + .HasColumnName("unique_message_id"); + + b.Property("RetryBatchId") + .HasColumnType("uuid") + .HasColumnName("retry_batch_id"); + + b.Property("StageAttempts") + .HasColumnType("integer") + .HasColumnName("stage_attempts"); + + b.HasKey("UniqueMessageId") + .HasName("pk_failed_message_retries"); + + b.HasIndex("RetryBatchId") + .HasDatabaseName("ix_failed_message_retries_retry_batch_id"); + + b.ToTable("failed_message_retries", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.GroupCommentEntity", b => + { + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("group_id"); + + b.Property("Comment") + .IsRequired() + .HasColumnType("text") + .HasColumnName("comment"); + + b.HasKey("GroupId") + .HasName("pk_group_comments"); + + b.ToTable("group_comments", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.HistoricRetryOperationEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CompletionTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("completion_time"); + + b.Property("Failed") + .HasColumnType("boolean") + .HasColumnName("failed"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("request_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.HasKey("Id") + .HasName("pk_historic_retry_operations"); + + b.HasIndex("CompletionTime", "Id") + .IsDescending() + .HasDatabaseName("ix_historic_retry_operations_completion_time_id"); + + b.ToTable("historic_retry_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.KnownEndpointEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("host"); + + b.Property("HostId") + .HasColumnType("uuid") + .HasColumnName("host_id"); + + b.Property("Monitored") + .HasColumnType("boolean") + .HasColumnName("monitored"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("name"); + + b.HasKey("Id") + .HasName("pk_known_endpoints"); + + b.ToTable("known_endpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("normalized_name"); + + b.Property("ThroughputSource") + .HasColumnType("integer") + .HasColumnName("throughput_source"); + + b.PrimitiveCollection>("EndpointIndicators") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("endpoint_indicators"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("name"); + + b.Property("NormalizedSanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("normalized_sanitized_name"); + + b.Property("SanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("sanitized_name"); + + b.Property("Scope") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("scope"); + + b.Property("UserIndicator") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("user_indicator"); + + b.HasKey("NormalizedName", "ThroughputSource") + .HasName("pk_licensing_endpoints"); + + b.HasIndex("NormalizedSanitizedName") + .HasDatabaseName("ix_licensing_endpoints_normalized_sanitized_name"); + + b.ToTable("licensing_endpoints", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("character varying(300)") + .HasColumnName("normalized_name"); + + b.Property("ThroughputSource") + .HasColumnType("integer") + .HasColumnName("throughput_source"); + + b.Property("DateUtc") + .HasColumnType("date") + .HasColumnName("date_utc"); + + b.Property("MessageCount") + .HasColumnType("bigint") + .HasColumnName("message_count"); + + b.HasKey("NormalizedName", "ThroughputSource", "DateUtc") + .HasName("pk_licensing_endpoint_throughput"); + + b.HasIndex("DateUtc") + .HasDatabaseName("ix_licensing_endpoint_throughput_date_utc"); + + b.ToTable("licensing_endpoint_throughput", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MessageRedirectEntity", b => + { + b.Property("FromPhysicalAddress") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("from_physical_address"); + + b.Property("LastModified") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_modified"); + + b.Property("ToPhysicalAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("to_physical_address"); + + b.HasKey("FromPhysicalAddress") + .HasName("pk_message_redirects"); + + b.ToTable("message_redirects", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Classifier") + .HasColumnType("text") + .HasColumnName("classifier"); + + b.Property("Context") + .HasColumnType("text") + .HasColumnName("context"); + + b.Property("InitialBatchSize") + .HasColumnType("integer") + .HasColumnName("initial_batch_size"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("initiated_by_id"); + + b.Property("InitiatedByName") + .HasColumnType("text") + .HasColumnName("initiated_by_name"); + + b.Property("Last") + .HasColumnType("timestamp with time zone") + .HasColumnName("last"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("operation_id"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("request_id"); + + b.Property("RetrySessionId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("retry_session_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("StagingId") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("staging_id"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("Id") + .HasName("pk_retry_batches"); + + b.HasIndex("Status", "RetrySessionId") + .HasDatabaseName("ix_retry_batches_status_retry_session_id"); + + b.ToTable("retry_batches", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchNowForwardingEntity", b => + { + b.Property("Id") + .HasColumnType("integer") + .HasColumnName("id"); + + b.Property("RetryBatchId") + .HasColumnType("uuid") + .HasColumnName("retry_batch_id"); + + b.HasKey("Id") + .HasName("pk_retry_batch_now_forwarding"); + + b.ToTable("retry_batch_now_forwarding", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SagaSnapshotEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_on"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Endpoint") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("endpoint"); + + b.Property("FinishTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("finish_time"); + + b.Property("InitiatingMessageJson") + .HasColumnType("text") + .HasColumnName("initiating_message_json"); + + b.Property("OutgoingMessagesJson") + .HasColumnType("text") + .HasColumnName("outgoing_messages_json"); + + b.Property("ProcessedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("processed_at"); + + b.Property("SagaId") + .HasColumnType("uuid") + .HasColumnName("saga_id"); + + b.Property("SagaType") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("saga_type"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.Property("StateAfterChange") + .HasColumnType("text") + .HasColumnName("state_after_change"); + + b.Property("Status") + .HasColumnType("integer") + .HasColumnName("status"); + + b.HasKey("CreatedOn", "Id") + .HasName("pk_saga_snapshots"); + + b.HasIndex("SagaId", "FinishTime") + .HasDatabaseName("ix_saga_snapshots_saga_id_finish_time"); + + b.ToTable("saga_snapshots", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => + { + b.Property("Key") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("key"); + + b.Property("Value") + .IsRequired() + .HasColumnType("text") + .HasColumnName("value"); + + b.HasKey("Key") + .HasName("pk_settings"); + + b.ToTable("settings", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SubscriptionEntity", b => + { + b.Property("MessageType") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("message_type"); + + b.Property("TransportAddress") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("transport_address"); + + b.Property("Endpoint") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("endpoint"); + + b.HasKey("MessageType", "TransportAddress") + .HasName("pk_subscriptions"); + + b.ToTable("subscriptions", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.UnacknowledgedRetryOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(400) + .HasColumnType("character varying(400)") + .HasColumnName("request_id"); + + b.Property("RetryType") + .HasColumnType("integer") + .HasColumnName("retry_type"); + + b.Property("Classifier") + .HasMaxLength(450) + .HasColumnType("character varying(450)") + .HasColumnName("classifier"); + + b.Property("CompletionTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("completion_time"); + + b.Property("Failed") + .HasColumnType("boolean") + .HasColumnName("failed"); + + b.Property("Last") + .HasColumnType("timestamp with time zone") + .HasColumnName("last"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("integer") + .HasColumnName("number_of_messages_processed"); + + b.Property("Originator") + .HasColumnType("text") + .HasColumnName("originator"); + + b.Property("StartTime") + .HasColumnType("timestamp with time zone") + .HasColumnName("start_time"); + + b.HasKey("RequestId", "RetryType") + .HasName("pk_unacknowledged_retry_operations"); + + b.ToTable("unacknowledged_retry_operations", (string)null); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", null) + .WithMany() + .HasForeignKey("FailedMessageUniqueId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_failed_message_groups_failed_messages_failed_message_unique"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", null) + .WithMany() + .HasForeignKey("NormalizedName", "ThroughputSource") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_licensing_endpoint_throughput_licensing_endpoints_normalize"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.cs new file mode 100644 index 0000000000..22898ba381 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Migrations/20260914030830_AddAuditFullTextSearch.cs @@ -0,0 +1,22 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.PostgreSql.Migrations +{ + /// + public partial class AddAuditFullTextSearch : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(FullTextSearchSql.AuditUp); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(FullTextSearchSql.AuditDown); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlFullTextSearchDialect.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlFullTextSearchDialect.cs index d2c01fb908..04e0c68ea4 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlFullTextSearchDialect.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlFullTextSearchDialect.cs @@ -19,6 +19,14 @@ public IQueryable Search(IQueryable so (message.MessageType ?? "").Replace(".", " ").Replace("+", " ")) .Matches(EF.Functions.WebSearchToTsQuery(FullTextSearchSql.Configuration, ToOrQuery(searchTerms)))); + public IQueryable Search(IQueryable source, string searchTerms) => + source.Where(message => + EF.Functions.ToTsVector(FullTextSearchSql.Configuration, + message.HeadersJson + " " + + (message.BodyText ?? "") + " " + + (message.MessageType ?? "").Replace(".", " ").Replace("+", " ")) + .Matches(EF.Functions.WebSearchToTsQuery(FullTextSearchSql.Configuration, ToOrQuery(searchTerms)))); + // websearch_to_tsquery ANDs bare terms; the RavenDB persister ORs them, so the terms are // rejoined with the operator that syntax understands. It also never throws on odd input, which // a hand built tsquery would. diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/FullTextSearchSql.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/FullTextSearchSql.cs index e502a3d6ca..c3b44799ce 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/FullTextSearchSql.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/FullTextSearchSql.cs @@ -12,6 +12,8 @@ static class FullTextSearchSql { const string CatalogName = "ServiceControlFullTextCatalog"; const string TableName = "FailedMessages"; + const string AuditTableName = "AuditMessages"; + const string AuditKeyIndexName = "UX_AuditMessages_Id"; // Message search is not optional, so an instance without Full-Text Search installed is not a // degraded instance, it is a broken one: every /messages/search request would fail on a missing @@ -57,6 +59,17 @@ FROM sys.fulltext_indexes i public static readonly string DropIndex = DropIndexSql(null); + // A full text index needs a single column unique KEY INDEX and the audit primary key is composite, + // so the identity column gets a unique index of its own, on SQL Server only: PostgreSQL requires + // the partition key in every unique index and does not need one here. + public static readonly string CreateAuditKeyIndex = CreateAuditKeyIndexSql(null); + + public static readonly string DropAuditKeyIndex = DropAuditKeyIndexSql(null); + + public static readonly string CreateAuditIndex = CreateAuditIndexSql(null); + + public static readonly string DropAuditIndex = DropAuditIndexSql(null); + /// /// Re-renders the statement the migration carries, this time with the configured schema in it. /// Anything else is left alone: EF Core builds the migrations history table's own SQL through @@ -68,13 +81,18 @@ public static MigrationOperation Rewrite(SqlOperation operation, string schema) { var sql when sql == CreateIndex => WithSql(operation, CreateIndexSql(schema)), var sql when sql == DropIndex => WithSql(operation, DropIndexSql(schema)), + var sql when sql == CreateAuditKeyIndex => WithSql(operation, CreateAuditKeyIndexSql(schema)), + var sql when sql == DropAuditKeyIndex => WithSql(operation, DropAuditKeyIndexSql(schema)), + var sql when sql == CreateAuditIndex => WithSql(operation, CreateAuditIndexSql(schema)), + var sql when sql == DropAuditIndex => WithSql(operation, DropAuditIndexSql(schema)), _ => operation }; // The catalog statements count as handled without being rewritten: they are server and database // scoped, so no schema reaches them. public static bool IsHandled(string sql) => - sql == RequireFullTextSearch || sql == CreateCatalog || sql == DropCatalog || sql == CreateIndex || sql == DropIndex; + sql == RequireFullTextSearch || sql == CreateCatalog || sql == DropCatalog || sql == CreateIndex || sql == DropIndex + || sql == CreateAuditKeyIndex || sql == DropAuditKeyIndex || sql == CreateAuditIndex || sql == DropAuditIndex; // LANGUAGE 0 (neutral) and STOPLIST = OFF keep the word breaker from applying language rules // and from dropping stopwords, both of which lose matches on technical content. @@ -97,8 +115,41 @@ IF EXISTS (SELECT 1 FROM sys.fulltext_indexes WHERE object_id = OBJECT_ID('{Qual END """; + static string CreateAuditKeyIndexSql(string? schema) => $""" + IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE name = '{AuditKeyIndexName}' AND object_id = OBJECT_ID('{QualifyAudit(schema)}')) + BEGIN + CREATE UNIQUE NONCLUSTERED INDEX [{AuditKeyIndexName}] ON {QualifyAudit(schema)}([Id]); + END + """; + + static string DropAuditKeyIndexSql(string? schema) => $""" + IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = '{AuditKeyIndexName}' AND object_id = OBJECT_ID('{QualifyAudit(schema)}')) + BEGIN + DROP INDEX [{AuditKeyIndexName}] ON {QualifyAudit(schema)}; + END + """; + + static string CreateAuditIndexSql(string? schema) => $""" + IF NOT EXISTS (SELECT 1 FROM sys.fulltext_indexes WHERE object_id = OBJECT_ID('{QualifyAudit(schema)}')) + BEGIN + EXEC('CREATE FULLTEXT INDEX ON {QualifyAudit(schema)}(HeadersJson LANGUAGE 0, BodyText LANGUAGE 0) + KEY INDEX {AuditKeyIndexName} + ON {CatalogName} + WITH (CHANGE_TRACKING AUTO, STOPLIST = OFF)'); + END + """; + + static string DropAuditIndexSql(string? schema) => $""" + IF EXISTS (SELECT 1 FROM sys.fulltext_indexes WHERE object_id = OBJECT_ID('{QualifyAudit(schema)}')) + BEGIN + DROP FULLTEXT INDEX ON {QualifyAudit(schema)}; + END + """; + static string Qualify(string? schema) => schema is null ? TableName : $"[{schema}].[{TableName}]"; + static string QualifyAudit(string? schema) => schema is null ? AuditTableName : $"[{schema}].[{AuditTableName}]"; + static SqlOperation WithSql(SqlOperation operation, string sql) => new() { Sql = sql, SuppressTransaction = operation.SuppressTransaction }; } diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.Designer.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.Designer.cs new file mode 100644 index 0000000000..a734934b20 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.Designer.cs @@ -0,0 +1,919 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ServiceControl.Persistence.EFCore.SqlServer; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations +{ + [DbContext(typeof(SqlServerServiceControlDbContext))] + [Migration("20260914030834_AddAuditFullTextSearch")] + partial class AddAuditFullTextSearch + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.11") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ArchiveOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("ArchiveType") + .HasColumnType("int"); + + b.Property("OperationType") + .HasColumnType("int"); + + b.Property("CurrentBatch") + .HasColumnType("int"); + + b.Property("GroupName") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("InitiatedByName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("NumberOfBatches") + .HasColumnType("int"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Started") + .HasColumnType("datetime2"); + + b.Property("TotalNumberOfMessages") + .HasColumnType("int"); + + b.HasKey("RequestId", "ArchiveType", "OperationType"); + + b.HasIndex("Started"); + + b.ToTable("ArchiveOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.AuditMessageEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("datetime2"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("BodySize") + .HasColumnType("int"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("BodyText") + .HasColumnType("nvarchar(max)"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("CriticalTimeTicks") + .HasColumnType("bigint"); + + b.Property("DeliveryTimeTicks") + .HasColumnType("bigint"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("IsSystemMessage") + .HasColumnType("bit"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("MessageType") + .HasColumnType("nvarchar(max)"); + + b.Property("ProcessedAt") + .HasColumnType("datetime2"); + + b.Property("ProcessingTimeTicks") + .HasColumnType("bigint"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Status") + .HasColumnType("int"); + + b.Property("TimeSent") + .HasColumnType("datetime2"); + + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.HasKey("CreatedOn", "Id"); + + b.HasIndex("ConversationId"); + + b.HasIndex("ProcessedAt"); + + b.HasIndex("TimeSent"); + + b.HasIndex("UniqueMessageId"); + + b.HasIndex("ReceivingEndpointName", "CreatedOn"); + + b.ToTable("AuditMessages"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.CustomCheckEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Category") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CustomCheckId") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailureReason") + .HasColumnType("nvarchar(max)"); + + b.Property("OriginatingEndpointHost") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("OriginatingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginatingEndpointName") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ReportedAt") + .HasColumnType("datetime2"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ReportedAt"); + + b.HasIndex("Status", "ReportedAt"); + + b.ToTable("CustomChecks"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EndpointSettingsEntity", b => + { + b.Property("Name") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("TrackInstances") + .HasColumnType("bit"); + + b.HasKey("Name"); + + b.ToTable("EndpointSettings"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.EventLogItemEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Category") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Description") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RaisedAt") + .HasColumnType("datetime2"); + + b.PrimitiveCollection("RelatedTo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Severity") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("RaisedAt", "Id") + .IsDescending(); + + b.ToTable("EventLogItems"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.ExternalIntegrationDispatchRequestEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DispatchContextJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DispatchContextTypeName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("Id"); + + b.ToTable("ExternalIntegrationDispatchRequests"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedAuditImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("Body") + .IsRequired() + .HasColumnType("varbinary(max)"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailedAt") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("FailedAt"); + + b.ToTable("FailedAuditImports"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedErrorImportEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("Body") + .IsRequired() + .HasColumnType("varbinary(max)"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("ExceptionInfo") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("FailedAt") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("MessageId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("FailedAt"); + + b.ToTable("FailedErrorImports"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEditEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("EditId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("EditId"); + + b.ToTable("FailedMessageEdits"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("BodyContentType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("BodySize") + .HasColumnType("int"); + + b.Property("BodyStoredExternally") + .HasColumnType("bit"); + + b.Property("BodyText") + .HasColumnType("nvarchar(max)"); + + b.Property("ConversationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ExceptionMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("ExceptionType") + .HasColumnType("nvarchar(max)"); + + b.Property("FailingEndpointAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("FirstTimeOfFailure") + .HasColumnType("datetime2"); + + b.Property("HeadersJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("IsSystemMessage") + .HasColumnType("bit"); + + b.Property("LastAttemptedAt") + .HasColumnType("datetime2"); + + b.Property("LastModified") + .HasColumnType("datetime2"); + + b.Property("LastTimeOfFailure") + .HasColumnType("datetime2"); + + b.Property("MessageId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("MessageType") + .HasColumnType("nvarchar(max)"); + + b.Property("NumberOfProcessingAttempts") + .HasColumnType("int"); + + b.Property("ReceivingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("ReceivingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("ReceivingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHost") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SendingEndpointHostId") + .HasColumnType("uniqueidentifier"); + + b.Property("SendingEndpointName") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Status") + .HasColumnType("int"); + + b.Property("StatusChangedAt") + .HasColumnType("datetime2"); + + b.Property("TimeSent") + .HasColumnType("datetime2"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("ConversationId"); + + b.HasIndex("FailingEndpointAddress"); + + b.HasIndex("ReceivingEndpointName"); + + b.HasIndex("StatusChangedAt") + .HasFilter("[Status] IN (2, 4)"); + + b.HasIndex("TimeSent"); + + b.HasIndex("Status", "LastModified"); + + b.ToTable("FailedMessages"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.Property("FailedMessageUniqueId") + .HasColumnType("uniqueidentifier"); + + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Title") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("nvarchar(255)"); + + b.HasKey("FailedMessageUniqueId", "GroupId"); + + b.HasIndex("GroupId"); + + b.HasIndex("Type", "GroupId"); + + b.ToTable("FailedMessageGroups"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageRetryEntity", b => + { + b.Property("UniqueMessageId") + .HasColumnType("uniqueidentifier"); + + b.Property("RetryBatchId") + .HasColumnType("uniqueidentifier"); + + b.Property("StageAttempts") + .HasColumnType("int"); + + b.HasKey("UniqueMessageId"); + + b.HasIndex("RetryBatchId"); + + b.ToTable("FailedMessageRetries"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.GroupCommentEntity", b => + { + b.Property("GroupId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Comment") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.HasKey("GroupId"); + + b.ToTable("GroupComments"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.HistoricRetryOperationEntity", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletionTime") + .HasColumnType("datetime2"); + + b.Property("Failed") + .HasColumnType("bit"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.HasKey("Id"); + + b.HasIndex("CompletionTime", "Id") + .IsDescending(); + + b.ToTable("HistoricRetryOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.KnownEndpointEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("HostId") + .HasColumnType("uniqueidentifier"); + + b.Property("Monitored") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("Id"); + + b.ToTable("KnownEndpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("ThroughputSource") + .HasColumnType("int"); + + b.PrimitiveCollection("EndpointIndicators") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("NormalizedSanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("SanitizedName") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Scope") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("UserIndicator") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("NormalizedName", "ThroughputSource"); + + b.HasIndex("NormalizedSanitizedName"); + + b.ToTable("LicensingEndpoints"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.Property("NormalizedName") + .HasMaxLength(300) + .HasColumnType("nvarchar(300)"); + + b.Property("ThroughputSource") + .HasColumnType("int"); + + b.Property("DateUtc") + .HasColumnType("date"); + + b.Property("MessageCount") + .HasColumnType("bigint"); + + b.HasKey("NormalizedName", "ThroughputSource", "DateUtc"); + + b.HasIndex("DateUtc"); + + b.ToTable("LicensingEndpointThroughput"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.MessageRedirectEntity", b => + { + b.Property("FromPhysicalAddress") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("LastModified") + .HasColumnType("datetime2"); + + b.Property("ToPhysicalAddress") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("FromPhysicalAddress"); + + b.ToTable("MessageRedirects"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchEntity", b => + { + b.Property("Id") + .HasColumnType("uniqueidentifier"); + + b.Property("Classifier") + .HasColumnType("nvarchar(max)"); + + b.Property("Context") + .HasColumnType("nvarchar(max)"); + + b.Property("InitialBatchSize") + .HasColumnType("int"); + + b.Property("InitiatedById") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("InitiatedByName") + .HasColumnType("nvarchar(max)"); + + b.Property("Last") + .HasColumnType("datetime2"); + + b.Property("OperationId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("RequestId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RetrySessionId") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("StagingId") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Status", "RetrySessionId"); + + b.ToTable("RetryBatches"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.RetryBatchNowForwardingEntity", b => + { + b.Property("Id") + .HasColumnType("int"); + + b.Property("RetryBatchId") + .HasColumnType("uniqueidentifier"); + + b.HasKey("Id"); + + b.ToTable("RetryBatchNowForwarding"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SagaSnapshotEntity", b => + { + b.Property("CreatedOn") + .HasColumnType("datetime2"); + + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Endpoint") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("FinishTime") + .HasColumnType("datetime2"); + + b.Property("InitiatingMessageJson") + .HasColumnType("nvarchar(max)"); + + b.Property("OutgoingMessagesJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ProcessedAt") + .HasColumnType("datetime2"); + + b.Property("SagaId") + .HasColumnType("uniqueidentifier"); + + b.Property("SagaType") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.Property("StateAfterChange") + .HasColumnType("nvarchar(max)"); + + b.Property("Status") + .HasColumnType("int"); + + b.HasKey("CreatedOn", "Id"); + + b.HasIndex("SagaId", "FinishTime"); + + b.ToTable("SagaSnapshots"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SettingEntity", b => + { + b.Property("Key") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("Value") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.HasKey("Key"); + + b.ToTable("Settings"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.SubscriptionEntity", b => + { + b.Property("MessageType") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TransportAddress") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Endpoint") + .IsRequired() + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.HasKey("MessageType", "TransportAddress"); + + b.ToTable("Subscriptions"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.UnacknowledgedRetryOperationEntity", b => + { + b.Property("RequestId") + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("RetryType") + .HasColumnType("int"); + + b.Property("Classifier") + .HasMaxLength(450) + .HasColumnType("nvarchar(450)"); + + b.Property("CompletionTime") + .HasColumnType("datetime2"); + + b.Property("Failed") + .HasColumnType("bit"); + + b.Property("Last") + .HasColumnType("datetime2"); + + b.Property("NumberOfMessagesProcessed") + .HasColumnType("int"); + + b.Property("Originator") + .HasColumnType("nvarchar(max)"); + + b.Property("StartTime") + .HasColumnType("datetime2"); + + b.HasKey("RequestId", "RetryType"); + + b.ToTable("UnacknowledgedRetryOperations"); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.FailedMessageGroupEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.FailedMessageEntity", null) + .WithMany() + .HasForeignKey("FailedMessageUniqueId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointThroughputEntity", b => + { + b.HasOne("ServiceControl.Persistence.EFCore.Entities.LicensingEndpointEntity", null) + .WithMany() + .HasForeignKey("NormalizedName", "ThroughputSource") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.cs new file mode 100644 index 0000000000..d17275fb82 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/Migrations/20260914030834_AddAuditFullTextSearch.cs @@ -0,0 +1,26 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ServiceControl.Persistence.EFCore.SqlServer.Migrations +{ + /// + public partial class AddAuditFullTextSearch : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(FullTextSearchSql.CreateCatalog, suppressTransaction: true); + migrationBuilder.Sql(FullTextSearchSql.CreateAuditKeyIndex, suppressTransaction: true); + migrationBuilder.Sql(FullTextSearchSql.CreateAuditIndex, suppressTransaction: true); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(FullTextSearchSql.DropAuditIndex, suppressTransaction: true); + migrationBuilder.Sql(FullTextSearchSql.DropAuditKeyIndex, suppressTransaction: true); + migrationBuilder.Sql(FullTextSearchSql.DropCatalog, suppressTransaction: true); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerFullTextSearchDialect.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerFullTextSearchDialect.cs index cc323d9f2e..2dfe3a8c17 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerFullTextSearchDialect.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerFullTextSearchDialect.cs @@ -13,4 +13,9 @@ public IQueryable Search(IQueryable so source.Where(message => EF.Functions.FreeText(message.HeadersJson, searchTerms) || EF.Functions.FreeText(message.BodyText!, searchTerms)); + + public IQueryable Search(IQueryable source, string searchTerms) => + source.Where(message => + EF.Functions.FreeText(message.HeadersJson, searchTerms) || + EF.Functions.FreeText(message.BodyText!, searchTerms)); } diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs index 249ac13aed..160df79b0f 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs @@ -8,6 +8,7 @@ namespace ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.CustomChecks; using ServiceControl.Operations.BodyStorage; using ServiceControl.Persistence.EFCore.Implementation; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Implementation.BodyStorage; using ServiceControl.Persistence.EFCore.Implementation.Recoverability; using ServiceControl.Persistence.EFCore.Implementation.UnitOfWork; @@ -54,6 +55,8 @@ protected static void RegisterDataStores(IServiceCollection services, EFPersiste services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditCountsDataStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditCountsDataStore.cs new file mode 100644 index 0000000000..3765fc0a5f --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditCountsDataStore.cs @@ -0,0 +1,43 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using ServiceControl.Api.Contracts; +using ServiceControl.Persistence.Infrastructure; + +// Mirrors the RavenDB audit persister: the last thirty days of successful, non-system messages +// received by the endpoint, a count per UTC day, and a single zero for today when the endpoint only +// ever sent, so the licensing collector still learns the endpoint exists. +public class AuditCountsDataStore(IServiceScopeFactory scopeFactory, TimeProvider timeProvider) : DataStoreBase(scopeFactory), IAuditCountsDataStore +{ + static readonly TimeSpan Window = TimeSpan.FromDays(30); + + public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) => + ExecuteQueryWithDbContext(async (dbContext, token) => + { + var today = timeProvider.GetUtcNow().UtcDateTime.Date; + var from = today - Window; + + // The partition key bounds the scan to the window's partitions; the grouping is on the + // processing day, which is what the count is meant to be of. + var counts = await dbContext.AuditMessages + .AsNoTracking() + .Where(message => message.ReceivingEndpointName == endpointName + && !message.IsSystemMessage + && message.CreatedOn >= from + && message.ProcessedAt >= from) + .GroupBy(message => message.ProcessedAt.Date) + .Select(group => new AuditCount { UtcDate = group.Key, Count = group.LongCount() }) + .OrderBy(count => count.UtcDate) + .ToListAsync(token); + + if (counts.Count == 0 && await dbContext.AuditMessages.AsNoTracking().AnyAsync(message => message.SendingEndpointName == endpointName, token)) + { + counts.Add(new AuditCount { UtcDate = today, Count = 0 }); + } + + IList results = counts; + + return new QueryResult>(results, new QueryStatsInfo(DataVersion.OverRows([("days", results.Count)], results, count => [count.UtcDate, count.Count]), results.Count)); + }, cancellationToken); +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditMessageQueryFilters.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditMessageQueryFilters.cs new file mode 100644 index 0000000000..1795a8567e --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/AuditMessageQueryFilters.cs @@ -0,0 +1,25 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.Infrastructure; + +static class AuditMessageQueryFilters +{ + public static IQueryable IncludeSystemMessagesWhere(this IQueryable source, bool includeSystemMessages) => + includeSystemMessages ? source : source.Where(message => !message.IsSystemMessage); + + public static IQueryable FilterBySentTimeRange(this IQueryable source, DateTimeRange? timeSentRange) + { + if (timeSentRange?.From is { } from) + { + source = source.Where(message => message.TimeSent >= from); + } + + if (timeSentRange?.To is { } to) + { + source = source.Where(message => message.TimeSent <= to); + } + + return source; + } +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRow.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRow.cs new file mode 100644 index 0000000000..12185b3a87 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRow.cs @@ -0,0 +1,43 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +/// +/// The projection both message tables are queried through, so that one statement can union, +/// sort, page and count failed and audited messages. Every sortable column of the message views is +/// a member, and the status is resolved to what the view reports so that a sort by status orders +/// both kinds of row the same way. +/// +public sealed class MessageRow +{ + public Guid UniqueMessageId { get; init; } + public bool IsAudit { get; init; } + public string? MessageId { get; init; } + public string? MessageType { get; init; } + public DateTime? TimeSent { get; init; } + public DateTime ProcessedAt { get; init; } + public string? ConversationId { get; init; } + public bool IsSystemMessage { get; init; } + public MessageStatus Status { get; init; } + public string? SendingEndpointName { get; init; } + public Guid? SendingEndpointHostId { get; init; } + public string? SendingEndpointHost { get; init; } + public string? ReceivingEndpointName { get; init; } + public Guid? ReceivingEndpointHostId { get; init; } + public string? ReceivingEndpointHost { get; init; } + public long CriticalTimeTicks { get; init; } + public long ProcessingTimeTicks { get; init; } + public long DeliveryTimeTicks { get; init; } + public string HeadersJson { get; init; } = string.Empty; + public int BodySize { get; init; } + + /// + /// What changes when the row changes: the last modification for a failed message, and the + /// ingestion hour for an audit row, which is never modified. + /// + public DateTime Version { get; init; } + + /// + /// The attempt count for a failed message, the identity for an audit row. With + /// it makes the ETag move whenever the view would. + /// + public long Revision { get; init; } +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRowQueries.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRowQueries.cs new file mode 100644 index 0000000000..61e7870ca2 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageRowQueries.cs @@ -0,0 +1,176 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using System.Linq.Expressions; +using NServiceBus; +using ServiceControl.CompositeViews.Messages; +using ServiceControl.MessageFailures; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.Infrastructure; +using ServiceControl.SagaAudit; + +static class MessageRowQueries +{ + public static IQueryable ToRows(this IQueryable failed) => + failed.Select(message => new MessageRow + { + UniqueMessageId = message.UniqueMessageId, + IsAudit = false, + MessageId = message.MessageId, + MessageType = message.MessageType, + TimeSent = message.TimeSent, + ProcessedAt = message.LastAttemptedAt, + ConversationId = message.ConversationId, + IsSystemMessage = message.IsSystemMessage, + // The status the view reports, resolved in SQL so that a sort by status orders failed + // and audited rows alike. A switch expression cannot appear in an expression tree. + Status = message.Status == FailedMessageStatus.Resolved + ? MessageStatus.ResolvedSuccessfully + : message.Status == FailedMessageStatus.RetryIssued + ? MessageStatus.RetryIssued + : message.Status == FailedMessageStatus.Archived + ? MessageStatus.ArchivedFailure + : message.NumberOfProcessingAttempts == 1 + ? MessageStatus.Failed + : MessageStatus.RepeatedFailure, + SendingEndpointName = message.SendingEndpointName, + SendingEndpointHostId = message.SendingEndpointHostId, + SendingEndpointHost = message.SendingEndpointHost, + ReceivingEndpointName = message.ReceivingEndpointName, + ReceivingEndpointHostId = message.ReceivingEndpointHostId, + ReceivingEndpointHost = message.ReceivingEndpointHost, + CriticalTimeTicks = 0L, + ProcessingTimeTicks = 0L, + DeliveryTimeTicks = 0L, + HeadersJson = message.HeadersJson, + BodySize = message.BodySize, + Version = message.LastModified, + Revision = message.NumberOfProcessingAttempts + }); + + /// + /// The audit rows that are not shadowed by a failed message. A message that both failed and was + /// audited shows as failed, whatever the failed row's status, so the anti-join is on the id alone. + /// + public static IQueryable ToRows(this IQueryable audit, ServiceControlDbContext dbContext) => + audit + .Where(message => !dbContext.FailedMessages.Any(failed => failed.UniqueMessageId == message.UniqueMessageId)) + .Select(message => new MessageRow + { + UniqueMessageId = message.UniqueMessageId, + IsAudit = true, + MessageId = message.MessageId, + MessageType = message.MessageType, + TimeSent = message.TimeSent, + ProcessedAt = message.ProcessedAt, + ConversationId = message.ConversationId, + IsSystemMessage = message.IsSystemMessage, + Status = message.Status, + SendingEndpointName = message.SendingEndpointName, + SendingEndpointHostId = message.SendingEndpointHostId, + SendingEndpointHost = message.SendingEndpointHost, + ReceivingEndpointName = message.ReceivingEndpointName, + ReceivingEndpointHostId = message.ReceivingEndpointHostId, + ReceivingEndpointHost = message.ReceivingEndpointHost, + CriticalTimeTicks = message.CriticalTimeTicks ?? 0L, + ProcessingTimeTicks = message.ProcessingTimeTicks ?? 0L, + DeliveryTimeTicks = message.DeliveryTimeTicks ?? 0L, + HeadersJson = message.HeadersJson, + BodySize = message.BodySize, + Version = message.CreatedOn, + Revision = message.Id + }); + + /// + /// The sort options of the message endpoints, applied to the common row so that a branch and + /// the union over both branches order identically. The processing statistics are zero for + /// every failed message, so those sorts fall through to time sent among equals, which is the + /// order the failed messages had before there was an audit branch to compare against. + /// + public static IOrderedQueryable Sort(this IQueryable rows, SortInfo? sortInfo) + { + var descending = sortInfo?.Direction != "asc"; + + return sortInfo?.Sort switch + { + "id" or "message_id" => rows.OrderBy(row => row.MessageId, descending), + "message_type" => rows.OrderBy(row => row.MessageType, descending), + "processed_at" => rows.OrderBy(row => row.ProcessedAt, descending), + "status" => rows.OrderBy(row => row.Status, descending), + "critical_time" => rows.OrderByThenTimeSent(row => row.CriticalTimeTicks, descending), + "delivery_time" => rows.OrderByThenTimeSent(row => row.DeliveryTimeTicks, descending), + "processing_time" => rows.OrderByThenTimeSent(row => row.ProcessingTimeTicks, descending), + _ => rows.OrderBy(row => row.TimeSent, descending) + }; + } + + public static MessagesView ToMessagesView(this MessageRow row) + { + var headers = MessageHeaders.Read(row.HeadersJson); + (List? InvokedSagas, SagaInfo? OriginatesFromSaga)? sagas = row.IsAudit ? ParseSagas(headers) : null; + + return new MessagesView + { + Id = row.UniqueMessageId.ToString(), + MessageId = row.MessageId, + MessageType = row.MessageType, + SendingEndpoint = Endpoint(row.SendingEndpointName, row.SendingEndpointHostId, row.SendingEndpointHost), + ReceivingEndpoint = Endpoint(row.ReceivingEndpointName, row.ReceivingEndpointHostId, row.ReceivingEndpointHost), + TimeSent = row.TimeSent, + ProcessedAt = row.ProcessedAt, + CriticalTime = TimeSpan.FromTicks(row.CriticalTimeTicks), + ProcessingTime = TimeSpan.FromTicks(row.ProcessingTimeTicks), + DeliveryTime = TimeSpan.FromTicks(row.DeliveryTimeTicks), + IsSystemMessage = row.IsSystemMessage, + ConversationId = row.ConversationId, + Headers = [.. headers.Select(header => new KeyValuePair(header.Key, header.Value))], + Status = row.Status, + MessageIntent = ReadMessageIntent(headers), + BodyUrl = $"/messages/{row.UniqueMessageId}/body", + BodySize = row.BodySize, + InvokedSagas = sagas?.InvokedSagas, + OriginatesFromSaga = sagas?.OriginatesFromSaga + }; + } + + // The saga relationships are a pure function of the headers, which is why they have no columns. + static (List? InvokedSagas, SagaInfo? OriginatesFromSaga) ParseSagas(Dictionary headers) + { + var metadata = new Dictionary(); + + InvokedSagasParser.Parse(headers, metadata); + + return ( + metadata.TryGetValue("InvokedSagas", out var invoked) ? invoked as List : null, + metadata.TryGetValue("OriginatesFromSaga", out var originates) ? originates as SagaInfo : null); + } + + static EndpointDetails? Endpoint(string? name, Guid? hostId, string? host) => + name is null && hostId is null && host is null + ? null + : new EndpointDetails { Name = name ?? string.Empty, HostId = hostId ?? Guid.Empty, Host = host ?? string.Empty }; + + static MessageIntent ReadMessageIntent(Dictionary headers) + { + var intent = default(MessageIntent); + + if (headers.TryGetValue(Headers.MessageIntent, out var value)) + { + Enum.TryParse(value, true, out intent); + } + + return intent; + } + + static IOrderedQueryable OrderBy(this IQueryable rows, Expression> keySelector, bool descending) => + descending + ? rows.OrderByDescending(keySelector).ThenByDescending(row => row.UniqueMessageId) + : rows.OrderBy(keySelector).ThenBy(row => row.UniqueMessageId); + + static IOrderedQueryable OrderByThenTimeSent(this IQueryable rows, Expression> keySelector, bool descending) => + descending + ? rows.OrderByDescending(keySelector).ThenByDescending(row => row.TimeSent).ThenByDescending(row => row.UniqueMessageId) + : rows.OrderBy(keySelector).ThenBy(row => row.TimeSent).ThenBy(row => row.UniqueMessageId); +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs new file mode 100644 index 0000000000..a21c44bdf2 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs @@ -0,0 +1,51 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using Microsoft.EntityFrameworkCore; +using ServiceControl.CompositeViews.Messages; +using ServiceControl.Persistence.Infrastructure; + +/// +/// One page of failed and audited messages from one statement per view. Each branch carries its +/// own sort and limit, so the database merges two index-ordered scans and stops at the page +/// boundary instead of materialising both tables; paging stays exact because any row of the page +/// is within the top offset + size rows of its own branch. +/// +static class MessageViewUnion +{ + /// + /// An exact total is linear in the audit table, so the count stops here. Total-Count and the + /// paging links report the cap when it is reached. + /// + public const int TotalCountCap = 100_000; + + public static async Task>> ToPagedMessagesResult( + IQueryable failed, + IQueryable audited, + PagingInfo pagingInfo, + SortInfo sortInfo, + CancellationToken cancellationToken = default) + { + var reach = pagingInfo.Offset + pagingInfo.Next; + + var rows = await failed.Sort(sortInfo).Take(reach) + .Concat(audited.Sort(sortInfo).Take(reach)) + .Sort(sortInfo) + .Skip(pagingInfo.Offset) + .Take(pagingInfo.Next) + .ToListAsync(cancellationToken); + + var total = await failed.Select(row => row.UniqueMessageId) + .Concat(audited.Select(row => row.UniqueMessageId)) + .Take(TotalCountCap) + .LongCountAsync(cancellationToken); + + IList results = [.. rows.Select(row => row.ToMessagesView())]; + + var version = DataVersion.OverRows( + [("messages", total)], + rows, + row => [row.UniqueMessageId, row.Version, row.Status, row.Revision]); + + return new QueryResult>(results, new QueryStatsInfo(version, total)); + } +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaHistoryDataStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaHistoryDataStore.cs new file mode 100644 index 0000000000..2dfa070d78 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/SagaHistoryDataStore.cs @@ -0,0 +1,57 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.Infrastructure; +using ServiceControl.SagaAudit; + +public class SagaHistoryDataStore(IServiceScopeFactory scopeFactory) : DataStoreBase(scopeFactory), ISagaHistoryDataStore +{ + public Task> QuerySagaHistoryById(Guid sagaId, PagingInfo pagingInfo, CancellationToken cancellationToken = default) => + ExecuteQueryWithDbContext(async (dbContext, token) => + { + var snapshots = dbContext.SagaSnapshots.AsNoTracking().Where(snapshot => snapshot.SagaId == sagaId); + + var totalChanges = await snapshots.CountAsync(token); + + if (totalChanges == 0) + { + return QueryResult.Empty(); + } + + var page = await snapshots + .OrderByDescending(snapshot => snapshot.FinishTime) + .ThenByDescending(snapshot => snapshot.Id) + .Skip(pagingInfo.Offset) + .Take(pagingInfo.Next) + .ToListAsync(token); + + var sagaType = page.Count > 0 + ? page[0].SagaType + : await snapshots.OrderByDescending(snapshot => snapshot.FinishTime).Select(snapshot => snapshot.SagaType).FirstAsync(token); + + var history = new SagaHistory + { + Id = sagaId, + SagaId = sagaId, + SagaType = sagaType, + Changes = [.. page.Select(ToStateChange)] + }; + + var version = DataVersion.OverRows([("changes", totalChanges)], page, snapshot => [snapshot.CreatedOn, snapshot.Id]); + + return new QueryResult(history, new QueryStatsInfo(version, totalChanges)); + }, cancellationToken); + + static SagaStateChange ToStateChange(SagaSnapshotEntity snapshot) => new() + { + StartTime = snapshot.StartTime, + FinishTime = snapshot.FinishTime, + Status = snapshot.Status, + StateAfterChange = snapshot.StateAfterChange, + InitiatingMessage = SagaSnapshotJson.ReadInitiatingMessage(snapshot.InitiatingMessageJson), + OutgoingMessages = SagaSnapshotJson.ReadOutgoingMessages(snapshot.OutgoingMessagesJson), + Endpoint = snapshot.Endpoint + }; +} diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs index 02c93e381d..3f353bd6a9 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs @@ -7,6 +7,7 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; using ServiceControl.Operations.BodyStorage; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.Infrastructure; @@ -37,7 +38,7 @@ public async Task TryFetch(string bodyId, CancellationToken c if (row.BodyStoredExternally) { - var external = await storagePersistence.ReadBody(row.UniqueMessageId.ToString(), cancellationToken); + var external = await storagePersistence.ReadBody(row.ExternalBodyId, cancellationToken); if (external == null) { @@ -88,9 +89,36 @@ public async Task TryFetch(string bodyId, CancellationToken c } } - return await Query(dbContext, message => message.MessageId == bodyId, cancellationToken); + var byMessageId = await Query(dbContext, message => message.MessageId == bodyId, cancellationToken); + if (byMessageId != null) + { + return byMessageId; + } + + return Guid.TryParse(bodyId, out var auditUniqueMessageId) + ? await QueryAudit(dbContext, auditUniqueMessageId, cancellationToken) + : null; } + // The newest row wins when a redelivered message left more than one, since none of them differ. + static Task QueryAudit(ServiceControlDbContext dbContext, Guid uniqueMessageId, CancellationToken cancellationToken) => + dbContext.AuditMessages + .AsNoTracking() + .Where(message => message.UniqueMessageId == uniqueMessageId) + .OrderByDescending(message => message.CreatedOn) + .ThenByDescending(message => message.Id) + .Select(message => new BodyRow + { + UniqueMessageId = message.UniqueMessageId, + IngestionHour = message.CreatedOn, + BodyText = message.BodyText, + BodyStoredExternally = message.BodyStoredExternally, + BodySize = message.BodySize, + BodyContentType = message.BodyContentType, + LastModified = message.CreatedOn + }) + .FirstOrDefaultAsync(cancellationToken); + static Task Query(ServiceControlDbContext dbContext, Expression> predicate, CancellationToken cancellationToken) => dbContext.FailedMessages .AsNoTracking() @@ -110,7 +138,14 @@ public async Task TryFetch(string bodyId, CancellationToken c sealed class BodyRow { public Guid UniqueMessageId { get; init; } + public DateTime? IngestionHour { get; init; } public string? BodyText { get; init; } + + // Composed here rather than in the query: a Guid rendered by SQL Server is upper case, and + // the stored key is the lower case string the ingestion wrote. + public string ExternalBodyId => IngestionHour is { } hour + ? AuditBodyStorage.BodyId(hour, UniqueMessageId) + : UniqueMessageId.ToString(); public bool BodyStoredExternally { get; init; } public int BodySize { get; init; } public string? BodyContentType { get; init; } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs index c285095cc3..96183bcbd8 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs @@ -3,47 +3,84 @@ namespace ServiceControl.Persistence.EFCore.Implementation; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using ServiceControl.CompositeViews.Messages; +using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Implementation.Audit; using ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.Infrastructure; +// Every view is the union of a failed branch and an audit branch, filtered alike, merged under the +// precedence, paging and counting rules IMessagesViewDataStore states. See MessageViewUnion. public class MessagesViewDataStore(IServiceScopeFactory scopeFactory, IFullTextSearchDialect fullTextSearch) : DataStoreBase(scopeFactory), IMessagesViewDataStore { public Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext((dbContext, token) => dbContext.FailedMessages - .AsNoTracking() - .IncludeSystemMessagesWhere(includeSystemMessages) - .FilterBySentTimeRange(timeSentRange) - .ToPagedMessagesResult(pagingInfo, sortInfo, token), cancellationToken); + ExecuteQueryWithDbContext((dbContext, token) => MessageViewUnion.ToPagedMessagesResult( + Failed(dbContext) + .IncludeSystemMessagesWhere(includeSystemMessages) + .FilterBySentTimeRange(timeSentRange) + .ToRows(), + Audited(dbContext) + .IncludeSystemMessagesWhere(includeSystemMessages) + .FilterBySentTimeRange(timeSentRange) + .ToRows(dbContext), + pagingInfo, sortInfo, token), cancellationToken); public Task>> GetAllMessagesForEndpoint(string endpointName, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext((dbContext, token) => dbContext.FailedMessages - .AsNoTracking() - .Where(message => message.ReceivingEndpointName == endpointName) - .IncludeSystemMessagesWhere(includeSystemMessages) - .FilterBySentTimeRange(timeSentRange) - .ToPagedMessagesResult(pagingInfo, sortInfo, token), cancellationToken); + ExecuteQueryWithDbContext((dbContext, token) => MessageViewUnion.ToPagedMessagesResult( + Failed(dbContext) + .Where(message => message.ReceivingEndpointName == endpointName) + .IncludeSystemMessagesWhere(includeSystemMessages) + .FilterBySentTimeRange(timeSentRange) + .ToRows(), + Audited(dbContext) + .Where(message => message.ReceivingEndpointName == endpointName) + .IncludeSystemMessagesWhere(includeSystemMessages) + .FilterBySentTimeRange(timeSentRange) + .ToRows(dbContext), + pagingInfo, sortInfo, token), cancellationToken); // includeSystemMessages is unused here: a conversation is incomplete without the system messages that took part in it. public Task>> GetAllMessagesByConversation(string conversationId, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext((dbContext, token) => dbContext.FailedMessages - .AsNoTracking() - .Where(message => message.ConversationId == conversationId) - .ToPagedMessagesResult(pagingInfo, sortInfo, token), cancellationToken); + ExecuteQueryWithDbContext((dbContext, token) => MessageViewUnion.ToPagedMessagesResult( + Failed(dbContext) + .Where(message => message.ConversationId == conversationId) + .ToRows(), + Audited(dbContext) + .Where(message => message.ConversationId == conversationId) + .ToRows(dbContext), + pagingInfo, sortInfo, token), cancellationToken); public Task>> GetAllMessagesForSearch(string searchTerms, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext((dbContext, token) => Search(dbContext.FailedMessages.AsNoTracking(), searchTerms) - .FilterBySentTimeRange(timeSentRange) - .ToPagedMessagesResult(pagingInfo, sortInfo, token), cancellationToken); + ExecuteQueryWithDbContext((dbContext, token) => MessageViewUnion.ToPagedMessagesResult( + Search(Failed(dbContext), searchTerms) + .FilterBySentTimeRange(timeSentRange) + .ToRows(), + Search(Audited(dbContext), searchTerms) + .FilterBySentTimeRange(timeSentRange) + .ToRows(dbContext), + pagingInfo, sortInfo, token), cancellationToken); public Task>> SearchEndpointMessages(string endpointName, string searchKeyword, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext((dbContext, token) => Search(dbContext.FailedMessages.AsNoTracking(), searchKeyword) - .Where(message => message.ReceivingEndpointName == endpointName) - .FilterBySentTimeRange(timeSentRange) - .ToPagedMessagesResult(pagingInfo, sortInfo, token), cancellationToken); + ExecuteQueryWithDbContext((dbContext, token) => MessageViewUnion.ToPagedMessagesResult( + Search(Failed(dbContext), searchKeyword) + .Where(message => message.ReceivingEndpointName == endpointName) + .FilterBySentTimeRange(timeSentRange) + .ToRows(), + Search(Audited(dbContext), searchKeyword) + .Where(message => message.ReceivingEndpointName == endpointName) + .FilterBySentTimeRange(timeSentRange) + .ToRows(dbContext), + pagingInfo, sortInfo, token), cancellationToken); + + static IQueryable Failed(ServiceControlDbContext dbContext) => dbContext.FailedMessages.AsNoTracking(); + + static IQueryable Audited(ServiceControlDbContext dbContext) => dbContext.AuditMessages.AsNoTracking(); // Neither search hides system messages: a caller who searched // for something specific is not helped by hiding the message that matched it. IQueryable Search(IQueryable source, string searchTerms) => string.IsNullOrWhiteSpace(searchTerms) ? source : fullTextSearch.Search(source, searchTerms); + + IQueryable Search(IQueryable source, string searchTerms) => + string.IsNullOrWhiteSpace(searchTerms) ? source : fullTextSearch.Search(source, searchTerms); } diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/IFullTextSearchDialect.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/IFullTextSearchDialect.cs index 45b3642e93..3d4da89556 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/IFullTextSearchDialect.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/IFullTextSearchDialect.cs @@ -15,4 +15,10 @@ public interface IFullTextSearchDialect /// Callers guarantee the terms are not blank. /// IQueryable Search(IQueryable source, string searchTerms); + + /// + /// The same predicate over the audit messages table, served by the index the + /// AddAuditFullTextSearch migration creates. + /// + IQueryable Search(IQueryable source, string searchTerms); } diff --git a/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj b/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj index f12ece860f..3b5386a2af 100644 --- a/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj +++ b/src/ServiceControl.Persistence.EFCore/ServiceControl.Persistence.EFCore.csproj @@ -14,6 +14,8 @@ + + diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/FullTextSearchIndexTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/FullTextSearchIndexTests.cs index cbfeafd87b..29a7c072d0 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/FullTextSearchIndexTests.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/FullTextSearchIndexTests.cs @@ -20,6 +20,14 @@ public void Search_uses_the_indexed_expression() Assert.That(sql, Does.Contain(FullTextSearchSql.IndexedExpression)); } + [Test] + public void Audit_search_uses_the_indexed_expression() + { + var sql = WithoutTableAlias(AuditSearchQuery("forty-two"), "audit_messages"); + + Assert.That(sql, Does.Contain(FullTextSearchSql.IndexedExpression)); + } + [Test] public void Terms_are_ored() { @@ -30,21 +38,35 @@ public void Terms_are_ored() static string SearchQuery(string searchTerms) { - var options = new DbContextOptionsBuilder() - .UseNpgsql("Host=localhost;Database=servicecontrol") - .Options; - - using var dbContext = new PostgreSqlServiceControlDbContext(options); + using var dbContext = CreateDbContext(); return new PostgreSqlFullTextSearchDialect() .Search(dbContext.FailedMessages, searchTerms) .ToQueryString(); } + static string AuditSearchQuery(string searchTerms) + { + using var dbContext = CreateDbContext(); + + return new PostgreSqlFullTextSearchDialect() + .Search(dbContext.AuditMessages, searchTerms) + .ToQueryString(); + } + + static PostgreSqlServiceControlDbContext CreateDbContext() + { + var options = new DbContextOptionsBuilder() + .UseNpgsql("Host=localhost;Database=servicecontrol") + .Options; + + return new PostgreSqlServiceControlDbContext(options); + } + // The DDL names the columns bare, the query qualifies them with whatever alias EF picked. - static string WithoutTableAlias(string sql) + static string WithoutTableAlias(string sql, string table = "failed_messages") { - var alias = Regex.Match(sql, @"FROM failed_messages AS (\w+)").Groups[1].Value; + var alias = Regex.Match(sql, $@"FROM {table} AS (\w+)").Groups[1].Value; return sql.Replace($"{alias}.", string.Empty); } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditBodyStorageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditBodyStorageTests.cs new file mode 100644 index 0000000000..958fc466fb --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditBodyStorageTests.cs @@ -0,0 +1,77 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.IO; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Operations.BodyStorage; + +class AuditBodyStorageTests : AuditIngestionTestBase +{ + const int Cap = 64; + + [SetUp] + public void ShrinkTheBodyCap() => EFSettings.BodyStorage.MaxBodySizeToStore = Cap; + + [Test] + public async Task Fetches_an_inline_audited_body() + { + var audit = new IngestedAudit { Body = Encoding.UTF8.GetBytes("1") }; + await IngestAudit(audit); + + var result = await BodyStorage.TryFetch(audit.UniqueMessageIdString); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.State, Is.EqualTo(MessageBodyState.Available)); + Assert.That(result.Content.ContentType, Is.EqualTo(audit.ContentType)); + Assert.That(Encoding.UTF8.GetString(ReadAll(result.Content.Stream)), Is.EqualTo("1")); + } + } + + [Test] + public async Task Fetches_an_external_audited_body_from_its_hour() + { + var body = Encoding.UTF8.GetBytes(new string('a', Cap * 2)); + var audit = new IngestedAudit { Body = body }; + await IngestAudit(audit); + + var result = await BodyStorage.TryFetch(audit.UniqueMessageIdString); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.State, Is.EqualTo(MessageBodyState.Available)); + Assert.That(ReadAll(result.Content.Stream), Is.EqualTo(body), "the whole body, not the inline search prefix"); + } + } + + [Test] + public async Task A_failed_messages_body_wins_over_its_audit_rows() + { + var failure = new IngestedFailure { Body = Encoding.UTF8.GetBytes("") }; + await Ingest(failure); + await IngestAudit(new IngestedAudit { RetryOf = failure.UniqueMessageIdString, Body = Encoding.UTF8.GetBytes("") }); + + var result = await BodyStorage.TryFetch(failure.UniqueMessageIdString); + + Assert.That(Encoding.UTF8.GetString(ReadAll(result.Content.Stream)), Is.EqualTo("")); + } + + [Test] + public async Task An_unknown_id_is_not_found() + { + await IngestAudit(new IngestedAudit()); + + var result = await BodyStorage.TryFetch(Guid.NewGuid().ToString()); + + Assert.That(result.State, Is.EqualTo(MessageBodyState.NotFound)); + } + + static byte[] ReadAll(Stream stream) + { + using var memory = new MemoryStream(); + stream.CopyTo(memory); + return memory.ToArray(); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditCountsDataStoreTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditCountsDataStoreTests.cs new file mode 100644 index 0000000000..8b511f3467 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditCountsDataStoreTests.cs @@ -0,0 +1,71 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; + +class AuditCountsDataStoreTests : AuditIngestionTestBase +{ + IAuditCountsDataStore Counts => ServiceProvider.GetRequiredService(); + + [Test] + public async Task Counts_the_endpoints_non_system_messages_per_day_oldest_first() + { + var today = Now.Date; + + await IngestAudit( + Sales(today.AddDays(-1).AddHours(3)), + Sales(today.AddDays(-1).AddHours(4)), + Sales(today.AddDays(-2).AddHours(1)), + Sales(today.AddDays(-1).AddHours(5), isSystemMessage: true), + new IngestedAudit { EndpointName = "Billing", ReceivingEndpoint = new() { Name = "Billing", Host = "h", HostId = Guid.NewGuid() }, ProcessingEnded = today.AddDays(-1) }); + + var result = await Counts.QueryAuditCounts("Sales"); + + Assert.That(result.Results.Select(count => (count.UtcDate, count.Count)), Is.EqualTo(new[] { (today.AddDays(-2), 1L), (today.AddDays(-1), 2L) })); + } + + [Test] + public async Task Ignores_messages_older_than_thirty_days() + { + var today = Now.Date; + + await IngestAudit(Sales(today.AddDays(-31)), Sales(today.AddDays(-29))); + + var result = await Counts.QueryAuditCounts("Sales"); + + Assert.That(result.Results.Select(count => count.UtcDate), Is.EqualTo(new[] { today.AddDays(-29) })); + } + + [Test] + public async Task Reports_a_zero_for_today_when_the_endpoint_only_ever_sent() + { + await IngestAudit(new IngestedAudit { SendingEndpoint = new() { Name = "Ordering", Host = "h", HostId = Guid.NewGuid() } }); + + var result = await Counts.QueryAuditCounts("Ordering"); + + Assert.That(result.Results.Select(count => (count.UtcDate, count.Count)), Is.EqualTo(new[] { (Now.Date, 0L) })); + } + + [Test] + public async Task Reports_nothing_for_an_unknown_endpoint() + { + await IngestAudit(new IngestedAudit()); + + var result = await Counts.QueryAuditCounts("Nobody"); + + Assert.That(result.Results, Is.Empty); + } + + IngestedAudit Sales(DateTime processedAt, bool isSystemMessage = false) => new() + { + EndpointName = "Sales", + ReceivingEndpoint = new() { Name = "Sales", Host = "h", HostId = Guid.NewGuid() }, + TimeSent = processedAt.AddSeconds(-2), + ProcessingStarted = processedAt.AddSeconds(-1), + ProcessingEnded = processedAt, + IsSystemMessage = isSystemMessage + }; +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditMessagesViewTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditMessagesViewTests.cs new file mode 100644 index 0000000000..ff82543ba8 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/AuditMessagesViewTests.cs @@ -0,0 +1,301 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.CompositeViews.Messages; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.Infrastructure; + +class AuditMessagesViewTests : AuditIngestionTestBase +{ + [Test] + public async Task Reports_an_audited_message() + { + var audit = new IngestedAudit(); + + await IngestAudit(audit); + + var view = (await All()).Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(view.Id, Is.EqualTo(audit.UniqueMessageIdString)); + Assert.That(view.MessageId, Is.EqualTo(audit.MessageId)); + Assert.That(view.MessageType, Is.EqualTo(audit.MessageType)); + Assert.That(view.Status, Is.EqualTo(MessageStatus.Successful)); + Assert.That(view.TimeSent, Is.EqualTo(audit.TimeSent)); + Assert.That(view.ProcessedAt, Is.EqualTo(audit.ProcessingEnded)); + Assert.That(view.CriticalTime, Is.EqualTo(audit.ProcessingEnded - audit.TimeSent)); + Assert.That(view.ProcessingTime, Is.EqualTo(audit.ProcessingEnded - audit.ProcessingStarted)); + Assert.That(view.DeliveryTime, Is.EqualTo(audit.ProcessingStarted - audit.TimeSent)); + Assert.That(view.ReceivingEndpoint.Name, Is.EqualTo(audit.ReceivingEndpoint.Name)); + Assert.That(view.ReceivingEndpoint.HostId, Is.EqualTo(audit.ReceivingEndpoint.HostId)); + Assert.That(view.SendingEndpoint.Name, Is.EqualTo(audit.SendingEndpoint.Name)); + Assert.That(view.ConversationId, Is.EqualTo(audit.ConversationId)); + Assert.That(view.MessageIntent, Is.EqualTo(audit.MessageIntent)); + Assert.That(view.BodyUrl, Is.EqualTo($"/messages/{audit.UniqueMessageIdString}/body")); + Assert.That(view.BodySize, Is.EqualTo(audit.Body.Length)); + Assert.That(view.Headers.Select(header => header.Key), Does.Contain(NServiceBus.Headers.MessageId)); + } + } + + [Test] + public async Task Reports_the_saga_relationships_from_the_headers() + { + var sagaId = Guid.NewGuid(); + var audit = new IngestedAudit(); + audit.Headers["NServiceBus.InvokedSagas"] = $"MyCompany.Sales.OrderSaga:{sagaId}"; + audit.Headers["ServiceControl.SagaStateChange"] = $"{sagaId}:Updated"; + + await IngestAudit(audit); + + var view = (await All()).Single(); + + Assert.That(view.InvokedSagas, Has.Count.EqualTo(1)); + using (Assert.EnterMultipleScope()) + { + Assert.That(view.InvokedSagas[0].SagaId, Is.EqualTo(sagaId)); + Assert.That(view.InvokedSagas[0].SagaType, Is.EqualTo("MyCompany.Sales.OrderSaga")); + Assert.That(view.InvokedSagas[0].ChangeStatus, Is.EqualTo("Updated")); + } + } + + [TestCase(FailedMessageStatus.Unresolved, MessageStatus.Failed)] + [TestCase(FailedMessageStatus.Resolved, MessageStatus.ResolvedSuccessfully)] + [TestCase(FailedMessageStatus.Archived, MessageStatus.ArchivedFailure)] + public async Task A_failed_message_wins_over_its_audit_row_whatever_its_status(FailedMessageStatus failedStatus, MessageStatus reported) + { + var failure = new IngestedFailure(); + await SeedFailedMessage(failure.ToFailedMessage(failedStatus)); + await IngestAudit(new IngestedAudit { RetryOf = failure.UniqueMessageIdString }); + + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo(), includeSystemMessages: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results.Select(view => view.Id), Is.EqualTo(new[] { failure.UniqueMessageIdString })); + Assert.That(result.Results.Single().Status, Is.EqualTo(reported)); + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(1), "a message in both tables counts once"); + } + } + + [Test] + public async Task Pages_exactly_across_both_sources() + { + var sent = new DateTime(2026, 7, 22, 9, 0, 0, DateTimeKind.Utc); + var failures = Enumerable.Range(0, 3).Select(i => new IngestedFailure { TimeSent = sent.AddMinutes(i * 2) }).ToArray(); + var audits = Enumerable.Range(0, 3).Select(i => new IngestedAudit { TimeSent = sent.AddMinutes((i * 2) + 1) }).ToArray(); + + await Ingest(failures); + await IngestAudit(audits); + + var expected = failures.Select(f => (Id: f.UniqueMessageIdString, TimeSent: f.TimeSent.Value)) + .Concat(audits.Select(a => (Id: a.UniqueMessageIdString, a.TimeSent))) + .OrderByDescending(m => m.TimeSent) + .Select(m => m.Id) + .ToArray(); + + var pages = new List(); + long total = 0; + + for (var page = 1; page <= 3; page++) + { + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(page, 2), new SortInfo("time_sent", "desc"), includeSystemMessages: true); + pages.AddRange(result.Results.Select(view => view.Id)); + total = result.QueryStats.TotalCount; + } + + using (Assert.EnterMultipleScope()) + { + Assert.That(pages, Is.EqualTo(expected), "the pages are contiguous and interleave both sources by time sent"); + Assert.That(total, Is.EqualTo(6)); + } + } + + [Test] + public async Task Counts_each_message_once_across_sources() + { + var failures = new[] { new IngestedFailure(), new IngestedFailure() }; + await Ingest(failures); + await IngestAudit(new IngestedAudit { RetryOf = failures[0].UniqueMessageIdString }, new IngestedAudit()); + + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(1, 1), new SortInfo(), includeSystemMessages: true); + + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(3)); + } + + [Test] + public async Task Sorts_by_processed_at_across_sources() + { + var at = new DateTime(2026, 7, 22, 10, 0, 0, DateTimeKind.Utc); + var failure = new IngestedFailure { AttemptedAt = at.AddMinutes(1) }; + var earlier = new IngestedAudit { ProcessingEnded = at }; + var later = new IngestedAudit { ProcessingEnded = at.AddMinutes(2) }; + + await Ingest(failure); + await IngestAudit(earlier, later); + + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo("processed_at", "asc"), includeSystemMessages: true); + + Assert.That(result.Results.Select(view => view.Id), Is.EqualTo(new[] { earlier.UniqueMessageIdString, failure.UniqueMessageIdString, later.UniqueMessageIdString })); + } + + [Test] + public async Task Sorts_by_critical_time_with_failed_messages_at_zero() + { + var failure = new IngestedFailure(); + var audit = new IngestedAudit(); + + await Ingest(failure); + await IngestAudit(audit); + + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo("critical_time", "desc"), includeSystemMessages: true); + + Assert.That(result.Results.Select(view => view.Id), Is.EqualTo(new[] { audit.UniqueMessageIdString, failure.UniqueMessageIdString })); + } + + [Test] + public async Task Hides_audited_system_messages_unless_asked() + { + var system = new IngestedAudit { IsSystemMessage = true }; + var ordinary = new IngestedAudit(); + + await IngestAudit(system, ordinary); + + var hidden = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo(), includeSystemMessages: false); + var shown = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo(), includeSystemMessages: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(hidden.Results.Select(view => view.Id), Is.EqualTo(new[] { ordinary.UniqueMessageIdString })); + Assert.That(hidden.QueryStats.TotalCount, Is.EqualTo(1)); + Assert.That(shown.Results, Has.Count.EqualTo(2)); + } + } + + [Test] + public async Task Filters_audited_messages_by_time_sent_range() + { + var sent = new DateTime(2026, 7, 22, 9, 0, 0, DateTimeKind.Utc); + var inside = new IngestedAudit { TimeSent = sent }; + var outside = new IngestedAudit { TimeSent = sent.AddHours(2) }; + + await IngestAudit(inside, outside); + + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo(), includeSystemMessages: true, new DateTimeRange(sent.AddMinutes(-1), sent.AddMinutes(1))); + + Assert.That(result.Results.Select(view => view.Id), Is.EqualTo(new[] { inside.UniqueMessageIdString })); + } + + [Test] + public async Task Filters_by_endpoint_across_sources() + { + var failure = new IngestedFailure { EndpointName = "Sales", ReceivingEndpoint = new() { Name = "Sales", Host = "h", HostId = Guid.NewGuid() } }; + var audit = new IngestedAudit { EndpointName = "Sales", ReceivingEndpoint = new() { Name = "Sales", Host = "h", HostId = Guid.NewGuid() } }; + var other = new IngestedAudit { EndpointName = "Billing", ReceivingEndpoint = new() { Name = "Billing", Host = "h", HostId = Guid.NewGuid() } }; + + await Ingest(failure); + await IngestAudit(audit, other); + + var result = await MessagesViewStore.GetAllMessagesForEndpoint("Sales", new PagingInfo(), new SortInfo(), includeSystemMessages: true); + + Assert.That(result.Results.Select(view => view.Id).OrderBy(id => id), Is.EqualTo(new[] { failure.UniqueMessageIdString, audit.UniqueMessageIdString }.OrderBy(id => id))); + } + + [Test] + public async Task A_conversation_spans_both_sources() + { + var conversationId = Guid.NewGuid().ToString(); + var failure = new IngestedFailure { ConversationId = conversationId }; + var audit = new IngestedAudit { ConversationId = conversationId, IsSystemMessage = true }; + var unrelated = new IngestedAudit(); + + await Ingest(failure); + await IngestAudit(audit, unrelated); + + var result = await MessagesViewStore.GetAllMessagesByConversation(conversationId, new PagingInfo(), new SortInfo(), includeSystemMessages: false); + + Assert.That(result.Results.Select(view => view.Id).OrderBy(id => id), Is.EqualTo(new[] { failure.UniqueMessageIdString, audit.UniqueMessageIdString }.OrderBy(id => id))); + } + + [Test] + public async Task Searches_audited_headers() + { + var matching = new IngestedAudit { ConversationId = "zarquon-conversation" }; + + await IngestAudit(matching, new IngestedAudit()); + + await AssertSearchFinds("zarquon", matching.UniqueMessageIdString); + } + + [Test] + public async Task Searches_audited_bodies() + { + var matching = new IngestedAudit { Body = Encoding.UTF8.GetBytes("slartibartfast") }; + + await IngestAudit(matching, new IngestedAudit()); + + await AssertSearchFinds("slartibartfast", matching.UniqueMessageIdString); + } + + [Test] + public async Task Searches_the_short_audited_message_type() + { + var matching = new IngestedAudit { MessageType = "MyCompany.Sales.Hooloovoo" }; + + await IngestAudit(matching, new IngestedAudit()); + + await AssertSearchFinds("Hooloovoo", matching.UniqueMessageIdString); + } + + [Test] + public async Task Searches_across_both_sources() + { + var failure = new IngestedFailure { ExceptionMessage = "the vogon overheated" }; + var audit = new IngestedAudit { ConversationId = "vogon-conversation" }; + + await Ingest(failure); + await IngestAudit(audit, new IngestedAudit()); + + await AssertSearchFinds("vogon", failure.UniqueMessageIdString, audit.UniqueMessageIdString); + } + + [Test] + public async Task Searches_within_an_endpoint_across_sources() + { + var matching = new IngestedAudit { EndpointName = "Sales", ReceivingEndpoint = new() { Name = "Sales", Host = "h", HostId = Guid.NewGuid() }, ConversationId = "magrathea-conversation" }; + var elsewhere = new IngestedAudit { EndpointName = "Billing", ReceivingEndpoint = new() { Name = "Billing", Host = "h", HostId = Guid.NewGuid() }, ConversationId = "magrathea-conversation" }; + + await IngestAudit(matching, elsewhere); + + await WaitForSearchResults( + () => MessagesViewStore.SearchEndpointMessages("Sales", "magrathea", new PagingInfo(), new SortInfo()), + matching.UniqueMessageIdString); + } + + async Task> All() => + (await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo(), includeSystemMessages: true)).Results; + + Task AssertSearchFinds(string searchTerms, params string[] expectedIds) => + WaitForSearchResults(() => MessagesViewStore.GetAllMessagesForSearch(searchTerms, new PagingInfo(), new SortInfo()), expectedIds); + + // SQL Server populates its full text index asynchronously, so a search right after the write + // legitimately returns nothing for a moment. + static async Task WaitForSearchResults(Func>>> search, params string[] expectedIds) + { + IList results = []; + + await WaitUntil(async () => + { + results = (await search()).Results; + + return results.Count == expectedIds.Length; + }, $"Search returned {expectedIds.Length} message(s)", TimeSpan.FromSeconds(30)); + + Assert.That(results.Select(view => view.Id).OrderBy(id => id), Is.EqualTo(expectedIds.OrderBy(id => id))); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaHistoryDataStoreTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaHistoryDataStoreTests.cs new file mode 100644 index 0000000000..7695ef86f9 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/SagaHistoryDataStoreTests.cs @@ -0,0 +1,83 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.Infrastructure; +using ServiceControl.SagaAudit; + +class SagaHistoryDataStoreTests : AuditIngestionTestBase +{ + ISagaHistoryDataStore History => ServiceProvider.GetRequiredService(); + + [Test] + public async Task Pages_a_sagas_changes_newest_first_and_reports_the_total() + { + var sagaId = Guid.NewGuid(); + var start = new DateTime(2026, 7, 22, 10, 0, 0, DateTimeKind.Utc); + + await IngestSnapshots(Snapshot(sagaId, start, SagaStateChangeStatus.New), Snapshot(sagaId, start.AddMinutes(1), SagaStateChangeStatus.Updated), Snapshot(sagaId, start.AddMinutes(2), SagaStateChangeStatus.Completed)); + await IngestSnapshots(Snapshot(Guid.NewGuid(), start, SagaStateChangeStatus.New)); + + var firstPage = await History.QuerySagaHistoryById(sagaId, new PagingInfo(1, 2)); + var secondPage = await History.QuerySagaHistoryById(sagaId, new PagingInfo(2, 2)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(firstPage.Results.SagaId, Is.EqualTo(sagaId)); + Assert.That(firstPage.Results.SagaType, Is.EqualTo("MyCompany.Sales.OrderSaga")); + Assert.That(firstPage.Results.Changes.Select(change => change.Status), Is.EqualTo(new[] { SagaStateChangeStatus.Completed, SagaStateChangeStatus.Updated })); + Assert.That(firstPage.QueryStats.TotalCount, Is.EqualTo(3)); + Assert.That(secondPage.Results.Changes.Select(change => change.Status), Is.EqualTo(new[] { SagaStateChangeStatus.New })); + Assert.That(secondPage.QueryStats.TotalCount, Is.EqualTo(3)); + } + } + + [Test] + public async Task Round_trips_a_changes_messages() + { + var sagaId = Guid.NewGuid(); + var snapshot = Snapshot(sagaId, new DateTime(2026, 7, 22, 10, 0, 0, DateTimeKind.Utc), SagaStateChangeStatus.New); + snapshot.InitiatingMessage = new InitiatingMessage { MessageId = "m1", OriginatingEndpoint = "Ordering", MessageType = "OrderPlaced", TimeSent = snapshot.StartTime, Intent = "Send" }; + snapshot.OutgoingMessages = [new ResultingMessage { MessageId = "m2", Destination = "Billing", MessageType = "BillOrder", TimeSent = snapshot.FinishTime, Intent = "Send" }]; + + await IngestSnapshots(snapshot); + + var change = (await History.QuerySagaHistoryById(sagaId, new PagingInfo())).Results.Changes.Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(change.InitiatingMessage.MessageId, Is.EqualTo("m1")); + Assert.That(change.InitiatingMessage.OriginatingEndpoint, Is.EqualTo("Ordering")); + Assert.That(change.OutgoingMessages.Single().Destination, Is.EqualTo("Billing")); + Assert.That(change.StateAfterChange, Is.EqualTo(snapshot.StateAfterChange)); + Assert.That(change.Endpoint, Is.EqualTo(snapshot.Endpoint)); + } + } + + [Test] + public async Task An_unknown_saga_has_no_history() + { + var result = await History.QuerySagaHistoryById(Guid.NewGuid(), new PagingInfo()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results, Is.Null); + Assert.That(result.QueryStats.TotalCount, Is.Zero); + } + } + + static SagaSnapshot Snapshot(Guid sagaId, DateTime finishTime, SagaStateChangeStatus status) => new() + { + SagaId = sagaId, + SagaType = "MyCompany.Sales.OrderSaga", + Status = status, + StartTime = finishTime.AddSeconds(-1), + FinishTime = finishTime, + ProcessedAt = finishTime, + Endpoint = "Sales", + StateAfterChange = "{\"OrderId\":42}" + }; +} diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 2c0aa2d222..ec4d167183 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -73,7 +73,9 @@ Settled by interview on 22 Aug 2026, extended by interview on 14 Sep 2026. | Schema of a dedicated audit database | The same `ServiceControlDbContext` and the same migration stream. The audit database carries the error tables, empty, and a primary whose audit is remote carries the audit tables, empty. | | How the primary knows audit is remote | An explicit setting, `ServiceControl/AuditDataLocation`, `Local` or `Remote`. `Local` is the default where the persister supports audit. Not derived from `IngestAuditMessages` and `RemoteInstances`, because the shared topology where only workers ingest looks the same. | | Delivery order | Shared database first, through step 6. The dedicated database is steps 7 and 8, composed from parts that already work by then. | -| Message view composition | One SQL statement per view: `UNION ALL` of the failed and audit branches, precedence by anti-join on `unique_message_id`, sort, paging and count in the database. Decided 14 Sep 2026, replacing the in-memory merge of two pages, which paged wrongly past page one and could not count. | +| Message view composition | One SQL statement per view: `UNION ALL` of the failed and audit branches, each branch carrying its own sort and limit, precedence by anti-join on `unique_message_id`, sort, paging and count in the database. Decided 14 Sep 2026, replacing the in-memory merge of two pages, which paged wrongly past page one and could not count. | +| Precedence | Failed wins, whatever its status. Archived failures keep showing as `ArchivedFailure`, as they do today. A "newer row wins" rule was measured and rejected: its failed-side probe into the partitioned audit index made the planner scan the whole table at deep pages (728 ms at page 100 on 3M rows against 8 ms). | +| Total count | Capped, not exact. An exact count is linear in the audit table (156 ms at 3M rows, tens of seconds at production sizes) and the capped form costs a millisecond. The cap is a constant to tune. | ### Why the partition key is the ingestion hour, and why rows are not deduplicated @@ -511,14 +513,25 @@ Each view is one SQL statement over both tables. The database applies the filter rule, the sort, the page and the count; nothing is merged in memory. ```sql -SELECT FROM failed_messages f WHERE -UNION ALL -SELECT FROM audit_messages a -WHERE - AND NOT EXISTS (SELECT 1 FROM failed_messages f WHERE f.unique_message_id = a.unique_message_id) -ORDER BY LIMIT @take OFFSET @skip +SELECT * FROM ( + (SELECT FROM failed_messages f WHERE + ORDER BY LIMIT @skip + @take) + UNION ALL + (SELECT FROM audit_messages a + WHERE + AND NOT EXISTS (SELECT 1 FROM failed_messages f WHERE f.unique_message_id = a.unique_message_id) + ORDER BY LIMIT @skip + @take) +) u ORDER BY LIMIT @take OFFSET @skip ``` +The limit inside each branch is what makes this fast, and it was measured rather than assumed: on +PostgreSQL 16 with three million audit rows over 168 hourly partitions, the same union with the sort +and limit only on the outside made the planner hash the anti-join and scan every row into a top-N +sort, 1.2 seconds for page one. With the branch limits it is a merge over two index-ordered scans +that stops at the page boundary: 5 ms for page one and 8 ms for page one hundred, against 4.5 ms +for scanning the audit time index alone. Paging stays exact because any row of the global page is +within the top `skip + take` rows of its own branch. + The three rules `IMessagesViewDataStore` states are each one clause of that statement: 1. **Precedence.** A message that both failed and was audited shows as failed. The anti-join on the @@ -530,8 +543,10 @@ The three rules `IMessagesViewDataStore` states are each one clause of that stat 2. **Paging.** `OFFSET` and `LIMIT` sit on the union, so page N is exact. Both providers stream a top-N over two ordered index scans (Merge Append on PostgreSQL, Merge Concatenation on SQL Server) and stop at the page boundary rather than reading either table through. -3. **Counting.** The total is a second statement with the same two predicates, `COUNT(*)` on each - branch with the anti-join on the audit side, so a message in both tables counts once. +3. **Counting.** The total is a second statement with the same two predicates, counted over the + union with a `LIMIT` at the cap, so a message in both tables counts once and a large table costs + a millisecond rather than a scan. `Total-Count` and the `Link` paging headers report the cap when + it is hit. Each branch keeps its own indexes, including its own full-text index, because the planner plans each branch on its own. The full-text predicate reaches each branch through the existing @@ -541,11 +556,14 @@ branch's expression to its index the way the failed one is pinned today. Everyth translates to the statement above on both providers. Two constraints follow. Both branches must project the same column set, which is why the audit -columns in "Schema" were chosen from what `MessagesView` shows. And every sortable column must be -indexed on both tables, or a sort becomes a sort of the whole filtered set: `time_sent` and -`processed_at` are, and the remaining sort keys the API accepts (`critical_time`, `delivery_time`, -`processing_time`, `message_type`, `status`) get a decision in step 4, either an index on both -tables or documented as unindexed sorts. +columns in "Schema" were chosen from what `MessagesView` shows; the projection is `MessageRow`, and +the failed branch resolves its status to the one the view reports so that a sort by status orders +both kinds of row alike. And every sortable column must be indexed on both tables, or a sort becomes +a sort of the whole filtered set: `time_sent` and `processed_at` are. The remaining sort keys the +API accepts (`critical_time`, `delivery_time`, `processing_time`, `message_type`, `status`) are +unindexed sorts on both tables, as they were on the failed table alone. The three statistics sorts +fall through to time sent among equal values, which keeps the failed messages, all at zero, in the +order they had before. `LocalMessagesView.Merge` stays only behind the in-memory test persister, which holds both kinds of message in memory and has no database to hand the work to. Once step 4 lands it serves no shipped @@ -615,7 +633,9 @@ competing consumers against one database, and the primary is the only writer to 4. **Queries.** The five message view unions, audit counts, saga history, and the third step of body arbitration. The full-text index lands here rather than with the schema, because the indexed expression and the query expression have to be written together or PostgreSQL silently downgrades - to a sequential scan, which is what the pinning test exists to catch. + to a sequential scan, which is what the pinning test exists to catch. On SQL Server the audit + full-text index keys on a unique index over the identity column alone, added by the same + migration, because a full-text key index must be a single column. On `john/audit_ef_5`. 5. **Failed audit imports.** The store, and the `--import-failed-audits` round trip. 6. **Turn it on.** Flip `SupportsAuditIngestion` in both manifests, update the approval test that asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable`. The `Empty*` @@ -676,9 +696,8 @@ rebased onto master on 14 September 2026; the code layout move is `john/audit_ef identity column, or letting SQL Server key on `id` alone, which would dedupe across hour boundaries and so behave better than PostgreSQL rather than the same. Needed for step 4, not step 1. 2. What caps `GetAllMessagesByConversation` and saga history, and what the API returns when a cap is hit. - Also whether the total count on an unfiltered message view stays exact, which is a count over - the whole audit table on every request, or becomes estimated or capped. This is the one query - cost the union does not remove, because no design can count without touching the table. + Also what the cap on the total count should be. It bounds both the cost of the count and how far + the paging links reach. 3. Whether the audit path should raise the `EndpointDetected` domain event. Carried over from the hosting plan, still unanswered, and now cheap to settle because the write path is real. 4. Whether `SagaUpdatedHandler` should hand the snapshot straight to the audit unit of work instead of From c0b48ca7fa57a95e4787405ea20c27b8cfd1c0ce Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 14:40:25 +1000 Subject: [PATCH 18/21] Store failed audit imports in EF The audit counterpart of the failed error import store: keyed by FailedAuditImport.DeriveKey so a poison message produces one row however many workers attempt it, bodies over the cap spilled to external storage under the failedauditimport prefix, replay oldest first with failures left in place, and no retention. The reimport command already drives the contract, so it needs no change. --- .../Abstractions/BasePersistence.cs | 1 + .../Audit/FailedAuditImportDataStore.cs | 200 ++++++++++++++++ .../EFCore/Audit/FailedAuditImportTests.cs | 223 ++++++++++++++++++ src/audit-ef-persistence-plan.md | 4 +- 4 files changed, 427 insertions(+), 1 deletion(-) create mode 100644 src/ServiceControl.Persistence.EFCore/Implementation/Audit/FailedAuditImportDataStore.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/FailedAuditImportTests.cs diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs index 160df79b0f..e9cb55e631 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs @@ -64,6 +64,7 @@ protected static void RegisterDataStores(IServiceCollection services, EFPersiste services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/FailedAuditImportDataStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/FailedAuditImportDataStore.cs new file mode 100644 index 0000000000..4122df07a8 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/FailedAuditImportDataStore.cs @@ -0,0 +1,200 @@ +namespace ServiceControl.Persistence.EFCore.Implementation.Audit; + +using System.IO; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using NServiceBus; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +// The audit counterpart of FailedErrorImportDataStore, keyed by FailedAuditImport.DeriveKey so +// that a poison message produces one row however many workers attempt it. +public class FailedAuditImportDataStore( + IServiceScopeFactory scopeFactory, + IBodyStoragePersistence bodyStorage, + BodyStorageSettings bodyStorageSettings, + TimeProvider timeProvider, + ILogger logger) : DataStoreBase(scopeFactory), IFailedAuditImportDataStore +{ + const int BatchSize = 100; + + public Task QueryContainsFailedImports(CancellationToken cancellationToken = default) => + ExecuteWithDbContext((dbContext, token) => dbContext.FailedAuditImports.AsNoTracking().AnyAsync(token), cancellationToken); + + // Update-first, then insert. The dedupe key is deterministic, so a repeat failure updates the + // existing row and concurrent writers that both miss it race only on the insert. + public Task StoreFailedAuditImport(FailedAuditImport failure, CancellationToken cancellationToken = default) => + ExecuteWithDbContext(async (dbContext, token) => + { + var uniqueMessageId = FailedAuditImport.DeriveKey(failure.Message!.Headers, failure.Message.Id); + var body = failure.Message.Body ?? []; + var storeExternally = body.Length > bodyStorageSettings.MaxBodySizeToStore; + + if (storeExternally) + { + var contentType = failure.Message.Headers.GetValueOrDefault(Headers.ContentType) ?? "application/octet-stream"; + await bodyStorage.WriteBody(FailedAuditImportEntity.ExternalBodyId(uniqueMessageId), body, contentType, token); + } + + var failedAt = timeProvider.GetUtcNow().UtcDateTime; + var headersJson = MessageHeaders.Write(failure.Message.Headers); + byte[] storedBody = storeExternally ? [] : body; + + await dbContext.UpsertAsync([uniqueMessageId], () => new FailedAuditImportEntity + { + UniqueMessageId = uniqueMessageId, + FailedAt = failedAt, + MessageId = failure.Message.Id, + HeadersJson = headersJson, + Body = storedBody, + BodyStoredExternally = storeExternally, + ExceptionInfo = failure.ExceptionInfo ?? string.Empty + }, entity => + { + entity.FailedAt = failedAt; + entity.MessageId = failure.Message.Id; + entity.HeadersJson = headersJson; + entity.Body = storedBody; + entity.BodyStoredExternally = storeExternally; + entity.ExceptionInfo = failure.ExceptionInfo ?? string.Empty; + }, token); + }, cancellationToken); + + // Replays oldest-first. Successful imports delete their row; failures are left in place, so the + // count of failures so far is exactly the offset to the next unseen row. + public async Task ProcessFailedAuditImports(Func processMessage, CancellationToken cancellationToken = default) + { + var succeeded = 0; + var failed = 0; + + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + + var batch = await ReadBatch(failed, cancellationToken); + + if (batch.Count == 0) + { + break; + } + + foreach (var import in batch) + { + cancellationToken.ThrowIfCancellationRequested(); + + try + { + var transportMessage = await ToTransportMessage(import, cancellationToken); + + await processMessage(transportMessage, cancellationToken); + + await DeleteImport(import, cancellationToken); + + succeeded++; + + logger.LogDebug("Successfully re-imported failed audit message {MessageId}", import.MessageId); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Error while attempting to re-import failed audit message {MessageId}", import.MessageId); + failed++; + } + } + + if (batch.Count < BatchSize) + { + break; + } + } + + logger.LogInformation("Done re-importing failed audits. Successfully re-imported {SucceededCount} messages. Failed re-importing {FailedCount} messages", succeeded, failed); + + if (failed > 0) + { + logger.LogWarning("{FailedCount} messages could not be re-imported. This could indicate a problem with the data. Contact Particular support if you need help with recovering the messages", failed); + } + } + + async Task> ReadBatch(int offset, CancellationToken cancellationToken) + { + await using var scope = scopeFactory.CreateAsyncScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + return await dbContext.FailedAuditImports + .AsNoTracking() + .OrderBy(import => import.FailedAt) + .ThenBy(import => import.UniqueMessageId) + .Skip(offset) + .Take(BatchSize) + .ToListAsync(cancellationToken); + } + + async Task ToTransportMessage(FailedAuditImportEntity import, CancellationToken cancellationToken) + { + var headers = MessageHeaders.Read(import.HeadersJson); + + var body = import.BodyStoredExternally + ? await ReadExternalBody(import.UniqueMessageId, cancellationToken) + : import.Body; + + return new FailedTransportMessage + { + Id = import.MessageId, + Headers = headers, + Body = body + }; + } + + async Task ReadExternalBody(Guid uniqueMessageId, CancellationToken cancellationToken) + { + var bodyId = FailedAuditImportEntity.ExternalBodyId(uniqueMessageId); + var stored = await bodyStorage.ReadBody(bodyId, cancellationToken) + ?? throw new InvalidOperationException($"The body for failed audit import {uniqueMessageId} was not found in body storage under {bodyId}."); + + await using var stream = stored.Stream; + using var buffer = new MemoryStream(stored.BodySize); + await stream.CopyToAsync(buffer, cancellationToken); + return buffer.ToArray(); + } + + // The row is removed before its external body: a surviving row with a missing body would replay + // as an empty message, whereas an orphaned body is only a leak. + async Task DeleteImport(FailedAuditImportEntity import, CancellationToken cancellationToken) + { + await using var scope = scopeFactory.CreateAsyncScope(); + var dbContext = scope.ServiceProvider.GetRequiredService(); + + await dbContext.FailedAuditImports + .Where(row => row.UniqueMessageId == import.UniqueMessageId) + .ExecuteDeleteAsync(cancellationToken); + + if (import.BodyStoredExternally) + { + await DeleteExternalBody(import.UniqueMessageId, cancellationToken); + } + } + + async Task DeleteExternalBody(Guid uniqueMessageId, CancellationToken cancellationToken) + { + try + { + await bodyStorage.DeleteBodyIfExists(FailedAuditImportEntity.ExternalBodyId(uniqueMessageId), cancellationToken); + } +#pragma warning disable PS0019 // The filter already excludes OperationCanceledException, so cancellation + // propagates; PS0019 only recognises a cancellationToken.IsCancellationRequested guard. + catch (Exception ex) when (ex is not OperationCanceledException) + { + // Re-import must not stall on a missing or unavailable body. + logger.LogWarning(ex, "Could not delete the external body for re-imported failed audit {UniqueMessageId}", uniqueMessageId); + } +#pragma warning restore PS0019 + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/FailedAuditImportTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/FailedAuditImportTests.cs new file mode 100644 index 0000000000..77458cef10 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/FailedAuditImportTests.cs @@ -0,0 +1,223 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NServiceBus; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.Infrastructure; + +class FailedAuditImportTests : AuditRetentionTestBase +{ + IFailedAuditImportDataStore FailedAuditImportStore => ServiceProvider.GetRequiredService(); + + [Test] + public async Task Stores_and_replays_a_failed_import() + { + var headers = WellFormedHeaders(); + var body = Encoding.UTF8.GetBytes("1"); + + await StoreImport(headers, body, nativeId: "native-1"); + + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.True); + + var replayed = await Replay(); + + Assert.That(replayed, Has.Count.EqualTo(1)); + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed[0].Id, Is.EqualTo("native-1")); + Assert.That(replayed[0].Headers, Is.EqualTo(headers)); + Assert.That(replayed[0].Body, Is.EqualTo(body)); + } + + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.False); + } + + [Test] + public async Task Spills_a_large_body_to_external_storage_and_replays_it() + { + var headers = WellFormedHeaders(); + var body = LargeBody(); + var externalId = FailedAuditImportEntity.ExternalBodyId(FailedAuditImport.DeriveKey(headers, "native-1")); + + await StoreImport(headers, body, nativeId: "native-1"); + + Assert.That(RecordedBodies.Written.Select(written => written.BodyId), Does.Contain(externalId)); + + var replayed = await Replay(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed, Has.Count.EqualTo(1)); + Assert.That(replayed[0].Body, Is.EqualTo(body)); + Assert.That(RecordedBodies.Deleted, Does.Contain(externalId)); + } + + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.False); + } + + [Test] + public async Task Repeated_failure_of_the_same_message_keeps_one_row_with_the_latest_details() + { + var headers = WellFormedHeaders(); + + await StoreImport(headers, Encoding.UTF8.GetBytes("first"), "first failure", "native-1"); + await StoreImport(headers, Encoding.UTF8.GetBytes("second"), "second failure", "native-1"); + + var replayed = await Replay(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed, Has.Count.EqualTo(1)); + Assert.That(replayed[0].Body, Is.EqualTo(Encoding.UTF8.GetBytes("second"))); + } + } + + [Test] + public async Task Stores_and_replays_a_message_with_no_derivable_endpoint() + { + var headers = new Dictionary(); + + await StoreImport(headers, Encoding.UTF8.GetBytes("body"), nativeId: "native-1"); + await StoreImport(headers, Encoding.UTF8.GetBytes("body-again"), nativeId: "native-1"); + + var replayed = await Replay(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed, Has.Count.EqualTo(1)); + Assert.That(replayed[0].Id, Is.EqualTo("native-1")); + Assert.That(replayed[0].Body, Is.EqualTo(Encoding.UTF8.GetBytes("body-again"))); + } + } + + [Test] + public async Task A_failing_re_import_is_left_behind_while_the_rest_are_processed() + { + await Store( + Import("native-1", BaseTime), + Import("native-2", BaseTime.AddSeconds(1)), + Import("native-3", BaseTime.AddSeconds(2))); + + var replayed = new List(); + await FailedAuditImportStore.ProcessFailedAuditImports( + (message, _) => + { + replayed.Add(message.Id); + return message.Id == "native-2" ? throw new InvalidOperationException("boom") : Task.CompletedTask; + }, + TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed, Is.EqualTo(new[] { "native-1", "native-2", "native-3" })); + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.True); + } + + var secondRun = await Replay(); + + Assert.That(secondRun.Select(message => message.Id), Is.EqualTo(new[] { "native-2" })); + } + + [Test] + public async Task Replays_across_multiple_pages() + { + await Store(Enumerable.Range(0, 250).Select(i => Import($"native-{i:D4}", BaseTime.AddSeconds(i))).ToArray()); + + var replayed = await Replay(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed, Has.Count.EqualTo(250)); + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.False); + } + } + + [Test] + public async Task Retention_sweep_does_not_touch_failed_imports() + { + await Store(Import("native-1", new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc))); + + await RunRetentionSweep(); + + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.True); + } + + [Test] + public async Task A_missing_external_body_fails_the_re_import_without_blocking_the_others() + { + var headers = WellFormedHeaders(); + var externalId = FailedAuditImportEntity.ExternalBodyId(FailedAuditImport.DeriveKey(headers, "native-1")); + var otherHeaders = WellFormedHeaders(); + otherHeaders[Headers.MessageId] = "m2"; + + await StoreImport(headers, LargeBody(), nativeId: "native-1"); + await StoreImport(otherHeaders, Encoding.UTF8.GetBytes("intact"), nativeId: "native-2"); + + RecordedBodies.Evict(externalId); + + var replayed = await Replay(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed.Select(message => message.Id), Is.EqualTo(new[] { "native-2" })); + Assert.That(await FailedAuditImportStore.QueryContainsFailedImports(), Is.True); + } + } + + Task StoreImport(Dictionary headers, byte[] body, string exceptionInfo = "boom", string nativeId = null) + { + nativeId ??= Guid.NewGuid().ToString(); + + return FailedAuditImportStore.StoreFailedAuditImport(new FailedAuditImport + { + Id = FailedAuditImport.DeriveKey(headers, nativeId).ToString(), + Message = new FailedTransportMessage { Id = nativeId, Headers = headers, Body = body }, + ExceptionInfo = exceptionInfo + }); + } + + async Task> Replay() + { + var replayed = new List(); + + await FailedAuditImportStore.ProcessFailedAuditImports( + (message, _) => { replayed.Add(message); return Task.CompletedTask; }, + TestContext.CurrentContext.CancellationToken); + + return replayed; + } + + byte[] LargeBody() + { + var body = new byte[EFSettings.BodyStorage.MaxBodySizeToStore + 1]; + Random.Shared.NextBytes(body); + return body; + } + + static FailedAuditImportEntity Import(string nativeId, DateTime failedAt) => new() + { + UniqueMessageId = DeterministicGuid.MakeId(nativeId), + FailedAt = failedAt, + MessageId = nativeId, + HeadersJson = "{}", + Body = Encoding.UTF8.GetBytes(nativeId), + BodyStoredExternally = false, + ExceptionInfo = "boom" + }; + + static Dictionary WellFormedHeaders() => new() + { + [Headers.MessageId] = "m1", + [Headers.ProcessingEndpoint] = "Sales", + [Headers.ContentType] = "text/xml" + }; + + static readonly DateTime BaseTime = new(2026, 7, 22, 10, 0, 0, DateTimeKind.Utc); +} diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index ec4d167183..6903b02897 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -636,7 +636,9 @@ competing consumers against one database, and the primary is the only writer to to a sequential scan, which is what the pinning test exists to catch. On SQL Server the audit full-text index keys on a unique index over the identity column alone, added by the same migration, because a full-text key index must be a single column. On `john/audit_ef_5`. -5. **Failed audit imports.** The store, and the `--import-failed-audits` round trip. +5. **Failed audit imports.** The store, keyed by `FailedAuditImport.DeriveKey` and otherwise the + error-side store's twin, so `--import-failed-audits` replays through it unchanged. On + `john/audit_ef_6`. 6. **Turn it on.** Flip `SupportsAuditIngestion` in both manifests, update the approval test that asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable`. The `Empty*` audit stand-ins stay: step 7 registers them on a primary whose audit is remote. Full acceptance From 623acf0f1984e299a3d1731dc4c9e09cd9ec71aa Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 14:48:12 +1000 Subject: [PATCH 19/21] Advertise audit support on the SQL Server and PostgreSQL persisters Both manifests now set SupportsAuditIngestion, so a primary on either persister hosts the audit runtime: the receiver, saga audit, failed audit imports, the local audit queries and audit retention. The acceptance suites run against the real persisters, which retires the audit capable test persister that stood in for them, along with the scenarios that could only exist while a relational persister lacked audit support. --- docs/audit-ingestion-in-the-primary.md | 5 +- ...eControl.AcceptanceTests.PostgreSql.csproj | 1 - ...ceControl.AcceptanceTests.SqlServer.csproj | 1 - ...omposing_audit_ingestion_in_the_primary.cs | 62 ++-------- .../When_hosting_audit_ingestion_only.cs | 40 +------ .../persistence.manifest | 2 +- .../persistence.manifest | 2 +- .../.editorconfig | 5 - .../AuditCapableBodyStorage.cs | 34 ------ .../AuditCapableIngestionUnitOfWork.cs | 54 --------- .../AuditCapableIngestionUnitOfWorkFactory.cs | 18 --- .../AuditCapableMessagesViewDataStore.cs | 58 --------- .../AuditCapableTestPersistence.cs | 59 ---------- ...uditCapableTestPersistenceConfiguration.cs | 43 ------- .../InMemoryAuditCountsDataStore.cs | 19 --- .../InMemoryAuditStore.cs | 111 ------------------ .../InMemoryFailedAuditImportDataStore.cs | 33 ------ .../InMemorySagaHistoryDataStore.cs | 20 ---- ...trol.Persistence.Tests.AuditCapable.csproj | 16 --- .../persistence.manifest | 9 -- .../LocalMessagesView.cs | 60 ---------- .../Hosting/AuditIngestionOnlyCommandTests.cs | 8 +- ...PersistenceManifestAuditCapabilityTests.cs | 21 ++-- .../ScatterGather/LocalMessagesViewTests.cs | 95 --------------- src/ServiceControl.slnx | 1 - src/audit-ef-persistence-plan.md | 2 +- 26 files changed, 30 insertions(+), 749 deletions(-) delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj delete mode 100644 src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest delete mode 100644 src/ServiceControl.Persistence/LocalMessagesView.cs delete mode 100644 src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs diff --git a/docs/audit-ingestion-in-the-primary.md b/docs/audit-ingestion-in-the-primary.md index 550cadef5c..e895ee8e8b 100644 --- a/docs/audit-ingestion-in-the-primary.md +++ b/docs/audit-ingestion-in-the-primary.md @@ -9,8 +9,9 @@ queue itself instead of relying on a separate ServiceControl.Audit instance. The standalone RavenDB audit instance is unaffected. RavenDB does not advertise audit support, does not gain combined hosting, and keeps its own executable, settings, API and installers. -No shipped persister advertises audit support yet, so on every existing deployment the audit -component registers nothing and behavior is unchanged. +The SQL Server and PostgreSQL persisters advertise audit support, so a primary on either ingests +the audit queue by default. On RavenDB the audit component registers nothing and behavior is +unchanged. ## Deployment modes diff --git a/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj b/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj index 564ca4217b..718b22bc78 100644 --- a/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj +++ b/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj @@ -12,7 +12,6 @@ - diff --git a/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj b/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj index 86efb52b29..cbc66d5b4a 100644 --- a/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj +++ b/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj @@ -12,7 +12,6 @@ - diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs index fa36935479..5b3a39005e 100644 --- a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs @@ -15,23 +15,18 @@ namespace ServiceControl.AcceptanceTests.Auditing using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing; - using ServiceControl.CompositeViews.MessageCounting; using ServiceControl.Connection; using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.WebApi; using ServiceControl.Persistence; - using ServiceControl.Persistence.Tests.AuditCapable; using ServiceControl.SagaAudit; - // The inner persistence type reaches the test persister through an environment variable, which is - // process wide, so these cannot run alongside anything else that sets it. - [NonParallelizable] class When_composing_audit_ingestion_in_the_primary : AcceptanceTest { [Test] - public async Task Should_host_the_audit_runtime_when_the_persister_advertises_audit_support() + public async Task Should_host_the_audit_runtime() { - var (app, services) = await BuildHost(auditCapable: true); + var (app, services) = await BuildHost(); try { @@ -62,7 +57,7 @@ public async Task Should_host_the_audit_runtime_when_the_persister_advertises_au [Test] public async Task Should_keep_every_audit_capability_but_the_receiver_when_ingestion_is_disabled() { - var (app, services) = await BuildHost(auditCapable: true, settings => settings.IngestAuditMessages = false); + var (app, services) = await BuildHost(settings => settings.IngestAuditMessages = false); try { @@ -80,31 +75,6 @@ public async Task Should_keep_every_audit_capability_but_the_receiver_when_inges } } - [Test] - public async Task Should_host_nothing_audit_related_on_a_persister_without_audit_support() - { - var (app, services) = await BuildHost(auditCapable: false); - - try - { - using (Assert.EnterMultipleScope()) - { - Assert.That(HostsAuditIngestion(services), Is.False); - Assert.That(app.Services.GetService(), Is.Null); - Assert.That(app.Services.GetService(), Is.Null); - - Assert.That(app.Services.GetService(), Is.Not.Null, - "the audit routes stay served from the configured remotes, so the APIs must still resolve"); - Assert.That(app.Services.GetService(), Is.Not.Null); - Assert.That(app.Services.GetService(), Is.Null); - } - } - finally - { - await app.DisposeAsync(); - } - } - // The registrations are inspected rather than resolved. A normal primary hosts an NServiceBus // endpoint, and constructing every hosted service without starting it fails inside the // transport's receive component. @@ -112,9 +82,9 @@ static bool HostsAuditIngestion(IServiceCollection services) => services.Any(descriptor => descriptor.ServiceType == typeof(IHostedService) && descriptor.ImplementationType == typeof(AuditIngestion)); - async Task<(WebApplication App, IServiceCollection Services)> BuildHost(bool auditCapable, Action customize = null) + async Task<(WebApplication App, IServiceCollection Services)> BuildHost(Action customize = null) { - var settings = await CreateSettings(auditCapable); + var settings = await CreateSettings(); customize?.Invoke(settings); @@ -128,19 +98,9 @@ static bool HostsAuditIngestion(IServiceCollection services) => return (hostBuilder.Build(), hostBuilder.Services); } - async Task CreateSettings(bool auditCapable) + async Task CreateSettings() { - var persistenceType = StorageConfiguration.PersistenceType; - - if (auditCapable) - { - // The test persister delegates everything but the audit contracts to the real one, so the - // host under test is the real host apart from the capability its manifest advertises. - Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, persistenceType); - persistenceType = AuditCapablePersistenceName; - } - - var settings = new Settings(TransportIntegration.TypeName, persistenceType, + var settings = new Settings(TransportIntegration.TypeName, StorageConfiguration.PersistenceType, CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) { InstanceName = $"AuditComposition.{Guid.NewGuid():n}", @@ -155,19 +115,11 @@ async Task CreateSettings(bool auditCapable) return settings; } - [TearDown] - public void ClearInnerPersistenceType() => Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, null); - static LoggingSettings CreateLoggingSettings() { var logPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()); Directory.CreateDirectory(logPath); return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); } - - const string AuditCapablePersistenceName = "AuditCapableTest"; - - static readonly string InnerPersistenceTypeVariable = - AuditCapableTestPersistenceConfiguration.InnerPersistenceTypeSetting.ToUpperInvariant(); } } diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs index 86d16ebf29..423bec8923 100644 --- a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs @@ -11,23 +11,18 @@ namespace ServiceControl.AcceptanceTests.Auditing using NServiceBus; using NUnit.Framework; using Particular.LicensingComponent.AuditThroughput; - using Particular.ServiceControl.Hosting; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing; using ServiceControl.Hosting.Commands; using ServiceControl.Infrastructure; using ServiceControl.Persistence; - using ServiceControl.Persistence.Tests.AuditCapable; - // The inner persistence type reaches the test persister through an environment variable, which is - // process wide, so these cannot run alongside anything else that sets it. - [NonParallelizable] class When_hosting_audit_ingestion_only : AcceptanceTest { [Test] public async Task Should_ingest_without_an_endpoint_and_without_the_single_owner_services() { - var settings = await CreateSettings(auditCapable: true); + var settings = await CreateSettings(); var host = AuditIngestionOnlyCommand.BuildHost(settings); @@ -67,7 +62,7 @@ public async Task Should_ingest_without_an_endpoint_and_without_the_single_owner [Test] public async Task Should_report_audit_ingestion_readiness() { - var settings = await CreateSettings(auditCapable: true); + var settings = await CreateSettings(); var host = AuditIngestionOnlyCommand.BuildHost(settings); @@ -90,17 +85,6 @@ public async Task Should_report_audit_ingestion_readiness() } } - [Test] - public async Task Should_refuse_to_start_against_storage_without_audit_support() - { - var settings = await CreateSettings(auditCapable: false); - - var exception = Assert.ThrowsAsync(() => - new AuditIngestionOnlyCommand().Execute(new HostArguments([]), settings)); - - Assert.That(exception.Message, Does.Contain("supports audit ingestion")); - } - static readonly string[] ExpectedHostedServices = [ "GenericWebHostService", // health endpoints only, no ServiceControl API @@ -112,20 +96,9 @@ public async Task Should_refuse_to_start_against_storage_without_audit_support() "ExternalIntegrationRequestsDataStore" // registered by the persister; its drain is inert here, nothing calls Subscribe ]; - [TearDown] - public void ClearInnerPersistenceType() => Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, null); - - async Task CreateSettings(bool auditCapable) + async Task CreateSettings() { - var persistenceType = StorageConfiguration.PersistenceType; - - if (auditCapable) - { - Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, persistenceType); - persistenceType = AuditCapablePersistenceName; - } - - var settings = new Settings(TransportIntegration.TypeName, persistenceType, + var settings = new Settings(TransportIntegration.TypeName, StorageConfiguration.PersistenceType, CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) { InstanceName = $"AuditIngestOnly.{Guid.NewGuid():n}", @@ -145,10 +118,5 @@ static LoggingSettings CreateLoggingSettings() Directory.CreateDirectory(logPath); return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); } - - const string AuditCapablePersistenceName = "AuditCapableTest"; - - static readonly string InnerPersistenceTypeVariable = - AuditCapableTestPersistenceConfiguration.InnerPersistenceTypeSetting.ToUpperInvariant(); } } diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest b/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest index 49b6aa4414..25d476322f 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest @@ -4,7 +4,7 @@ "Description": "PostgreSQL ServiceControl persister", "AssemblyName": "ServiceControl.Persistence.EFCore.PostgreSql", "TypeName": "ServiceControl.Persistence.EFCore.PostgreSql.PostgreSqlPersistenceConfiguration, ServiceControl.Persistence.EFCore.PostgreSql", - "SupportsAuditIngestion": false, + "SupportsAuditIngestion": true, "Settings": [ { "Name": "ServiceControl/Database/ConnectionString", diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest b/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest index 598ba9168c..78053a551c 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest @@ -4,7 +4,7 @@ "Description": "SQL Server ServiceControl persister", "AssemblyName": "ServiceControl.Persistence.EFCore.SqlServer", "TypeName": "ServiceControl.Persistence.EFCore.SqlServer.SqlServerPersistenceConfiguration, ServiceControl.Persistence.EFCore.SqlServer", - "SupportsAuditIngestion": false, + "SupportsAuditIngestion": true, "Settings": [ { "Name": "ServiceControl/Database/ConnectionString", diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig b/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig deleted file mode 100644 index ca5ad8bd2e..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig +++ /dev/null @@ -1,5 +0,0 @@ -[*.cs] - -# Justification: Test project -dotnet_diagnostic.CA2007.severity = none -dotnet_diagnostic.PS0018.severity = none diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs deleted file mode 100644 index eb8ed8efe3..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs +++ /dev/null @@ -1,34 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System.IO; - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.Operations.BodyStorage; - using ServiceControl.Persistence.Infrastructure; - - // The third step of the arbitration order IBodyStorage states: a failed message body wins, and the - // audit copy answers only when no failed message holds one. - class AuditCapableBodyStorage(IBodyStorage inner, InMemoryAuditStore auditStore) : IBodyStorage - { - public async Task TryFetch(string bodyId, CancellationToken cancellationToken = default) - { - var failedMessageBody = await inner.TryFetch(bodyId, cancellationToken); - - if (failedMessageBody.State != MessageBodyState.NotFound) - { - return failedMessageBody; - } - - var body = auditStore.BodyFor(bodyId); - - if (body == null) - { - return MessageBodyResult.NotFound(); - } - - return body.Length == 0 - ? MessageBodyResult.Empty() - : MessageBodyResult.Available(new MessageBodyStreamContent(new MemoryStream(body, writable: false), "application/json", body.Length, DataVersion.None)); - } - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs deleted file mode 100644 index f10e9f8032..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs +++ /dev/null @@ -1,54 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using System.Collections.Concurrent; - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.MessageAuditing; - using ServiceControl.Persistence.UnitOfWork; - using ServiceControl.SagaAudit; - - // Recording is buffered and only visible after Complete, so tests see the same all or nothing - // batch behaviour a real persister gives them. - class AuditCapableIngestionUnitOfWork(IIngestionUnitOfWork inner, InMemoryAuditStore auditStore) - : IIngestionUnitOfWork, IAuditIngestionUnitOfWork - { - readonly ConcurrentQueue<(ProcessedMessage Message, byte[] Body)> processedMessages = new(); - readonly ConcurrentQueue sagaSnapshots = new(); - - public IMonitoringIngestionUnitOfWork? Monitoring => inner.Monitoring; - - public IRecoverabilityIngestionUnitOfWork? Recoverability => inner.Recoverability; - - public IAuditIngestionUnitOfWork? Audit => this; - - public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) - { - processedMessages.Enqueue((processedMessage, body.ToArray())); - return Task.CompletedTask; - } - - public Task RecordSagaSnapshot(SagaSnapshot sagaSnapshot, CancellationToken cancellationToken = default) - { - sagaSnapshots.Enqueue(sagaSnapshot); - return Task.CompletedTask; - } - - public async Task Complete(CancellationToken cancellationToken = default) - { - await inner.Complete(cancellationToken); - - while (processedMessages.TryDequeue(out var processedMessage)) - { - auditStore.Record(processedMessage.Message, processedMessage.Body); - } - - while (sagaSnapshots.TryDequeue(out var sagaSnapshot)) - { - auditStore.Record(sagaSnapshot); - } - } - - public ValueTask DisposeAsync() => inner.DisposeAsync(); - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs deleted file mode 100644 index 73eaa94249..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs +++ /dev/null @@ -1,18 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.Persistence.UnitOfWork; - - class AuditCapableIngestionUnitOfWorkFactory(IIngestionUnitOfWorkFactory inner, InMemoryAuditStore auditStore) : IIngestionUnitOfWorkFactory - { - public async ValueTask StartNew(CancellationToken cancellationToken = default) => - new AuditCapableIngestionUnitOfWork(await inner.StartNew(cancellationToken), auditStore); - - public bool CanIngestMore() => inner.CanIngestMore(); - - // Whatever the persister this delegates to says: the audit rows it adds are appended, never - // merged, so they do not change how concurrent batches settle. - public bool SupportsConcurrentBatches => inner.SupportsConcurrentBatches; - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs deleted file mode 100644 index 2b67f891c3..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs +++ /dev/null @@ -1,58 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using System.Collections.Generic; - using System.Linq; - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.CompositeViews.Messages; - using ServiceControl.Persistence.Infrastructure; - - // One local result set holding both failed and audited messages, merged under the precedence, - // paging and counting rules IMessagesViewDataStore states. - class AuditCapableMessagesViewDataStore(IMessagesViewDataStore inner, InMemoryAuditStore auditStore) : IMessagesViewDataStore - { - public async Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - Merge(await inner.GetAllMessages(pagingInfo, sortInfo, includeSystemMessages, timeSentRange, cancellationToken), - Audited(includeSystemMessages, timeSentRange), pagingInfo, sortInfo); - - public async Task>> GetAllMessagesForEndpoint(string endpointName, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - Merge(await inner.GetAllMessagesForEndpoint(endpointName, pagingInfo, sortInfo, includeSystemMessages, timeSentRange, cancellationToken), - Audited(includeSystemMessages, timeSentRange).Where(message => message.ReceivingEndpoint?.Name == endpointName), pagingInfo, sortInfo); - - public async Task>> GetAllMessagesByConversation(string conversationId, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, CancellationToken cancellationToken = default) => - Merge(await inner.GetAllMessagesByConversation(conversationId, pagingInfo, sortInfo, includeSystemMessages, cancellationToken), - Audited(includeSystemMessages).Where(message => message.ConversationId == conversationId), pagingInfo, sortInfo); - - public async Task>> GetAllMessagesForSearch(string searchTerms, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - Merge(await inner.GetAllMessagesForSearch(searchTerms, pagingInfo, sortInfo, timeSentRange, cancellationToken), - Audited(includeSystemMessages: true, timeSentRange).Where(message => Matches(message, searchTerms)), pagingInfo, sortInfo); - - public async Task>> SearchEndpointMessages(string endpointName, string searchKeyword, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => - Merge(await inner.SearchEndpointMessages(endpointName, searchKeyword, pagingInfo, sortInfo, timeSentRange, cancellationToken), - Audited(includeSystemMessages: true, timeSentRange) - .Where(message => message.ReceivingEndpoint?.Name == endpointName && Matches(message, searchKeyword)), pagingInfo, sortInfo); - - IEnumerable Audited(bool includeSystemMessages, DateTimeRange? timeSentRange = null) => - auditStore.MessageViews - .Where(message => includeSystemMessages || !message.IsSystemMessage) - .Where(message => InRange(message, timeSentRange)); - - static bool InRange(MessagesView message, DateTimeRange? timeSentRange) => - timeSentRange == null - || (message.TimeSent >= timeSentRange.From && message.TimeSent <= timeSentRange.To); - - static bool Matches(MessagesView message, string searchTerms) => - searchTerms == null - || (message.MessageType?.Contains(searchTerms, StringComparison.OrdinalIgnoreCase) ?? false) - || (message.MessageId?.Contains(searchTerms, StringComparison.OrdinalIgnoreCase) ?? false); - - static QueryResult> Merge(QueryResult> failed, IEnumerable audited, PagingInfo pagingInfo, SortInfo sortInfo) => - LocalMessagesView.Merge( - [.. failed.Results ?? []], - [.. audited], - pagingInfo, - MessageViewComparer.FromSortInfo(sortInfo), - failed.QueryStats.Version); - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs deleted file mode 100644 index 238ca820b4..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs +++ /dev/null @@ -1,59 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using System.Linq; - using Microsoft.Extensions.DependencyInjection; - using ServiceControl.Operations.BodyStorage; - using ServiceControl.Persistence.UnitOfWork; - - class AuditCapableTestPersistence(IPersistence inner) : IPersistence - { - public void AddPersistence(IServiceCollection services) - { - inner.AddPersistence(services); - - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - - Decorate(services, (inner, provider) => - new AuditCapableIngestionUnitOfWorkFactory(inner, provider.GetRequiredService())); - Decorate(services, (inner, provider) => - new AuditCapableMessagesViewDataStore(inner, provider.GetRequiredService())); - Decorate(services, (inner, provider) => - new AuditCapableBodyStorage(inner, provider.GetRequiredService())); - } - - public void AddInstaller(IServiceCollection services) => inner.AddInstaller(services); - - static void Decorate(IServiceCollection services, Func decorate) - where TService : class - { - var descriptor = services.LastOrDefault(d => d.ServiceType == typeof(TService)) - ?? throw new InvalidOperationException($"The delegated persister registered no {typeof(TService).Name}."); - - services.Remove(descriptor); - - services.Add(new ServiceDescriptor(typeof(TService), - provider => decorate(ResolveInner(provider, descriptor), provider), - descriptor.Lifetime)); - } - - static TService ResolveInner(IServiceProvider provider, ServiceDescriptor descriptor) - where TService : class - { - if (descriptor.ImplementationInstance is TService instance) - { - return instance; - } - - if (descriptor.ImplementationFactory is not null) - { - return (TService)descriptor.ImplementationFactory(provider); - } - - return (TService)ActivatorUtilities.CreateInstance(provider, descriptor.ImplementationType!); - } - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs deleted file mode 100644 index bb206ee97f..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs +++ /dev/null @@ -1,43 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using ServiceControl.Configuration; - - /// - /// A persister that exists only so tests can compose a primary host whose manifest advertises audit - /// support, before any shipped persister does. Everything except the audit contracts is delegated to - /// the persister named by the setting, so the error side of - /// the host is the real thing. Delete it once a shipped manifest sets SupportsAuditIngestion. - /// - public class AuditCapableTestPersistenceConfiguration : IPersistenceConfiguration - { - public const string InnerPersistenceTypeSetting = "AuditCapableTestInnerPersistenceType"; - - public bool SupportsMaintenanceMode => CreateInnerConfiguration(PrimaryRootNamespace).SupportsMaintenanceMode; - - public PersistenceSettings CreateSettings(SettingsRootNamespace settingsRootNamespace) => - CreateInnerConfiguration(settingsRootNamespace).CreateSettings(settingsRootNamespace); - - public IPersistence Create(PersistenceSettings settings) => - new AuditCapableTestPersistence(CreateInnerConfiguration(PrimaryRootNamespace).Create(settings)); - - static IPersistenceConfiguration CreateInnerConfiguration(SettingsRootNamespace settingsRootNamespace) - { - var persistenceType = SettingsReader.Read(settingsRootNamespace, InnerPersistenceTypeSetting) - ?? throw new InvalidOperationException( - $"The audit capable test persister needs the {settingsRootNamespace}/{InnerPersistenceTypeSetting} setting to name the persister it delegates to."); - - var manifest = PersistenceManifestLibrary.Find(persistenceType) - ?? throw new InvalidOperationException($"No persistence manifest matches '{persistenceType}'."); - - var typeName = manifest.TypeName - ?? throw new InvalidOperationException($"The persistence manifest for '{persistenceType}' names no configuration type."); - - var configurationType = Type.GetType(typeName, throwOnError: true)!; - - return (IPersistenceConfiguration)Activator.CreateInstance(configurationType)!; - } - - static readonly SettingsRootNamespace PrimaryRootNamespace = new("ServiceControl"); - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs deleted file mode 100644 index a980054907..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs +++ /dev/null @@ -1,19 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System.Collections.Generic; - using System.Linq; - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.Api.Contracts; - using ServiceControl.Persistence.Infrastructure; - - class InMemoryAuditCountsDataStore(InMemoryAuditStore auditStore) : IAuditCountsDataStore - { - public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) - { - IList counts = [.. auditStore.CountsFor(endpointName).Select(count => new AuditCount { UtcDate = count.UtcDate, Count = count.Count })]; - - return Task.FromResult(new QueryResult>(counts, new QueryStatsInfo(DataVersion.None, counts.Count))); - } - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs deleted file mode 100644 index 2dbad834b2..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs +++ /dev/null @@ -1,111 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using System.Collections.Concurrent; - using System.Collections.Generic; - using System.Linq; - using NServiceBus; - using ServiceControl.CompositeViews.Messages; - using ServiceControl.MessageAuditing; - using ServiceControl.Operations; - using ServiceControl.Persistence.Infrastructure; - using ServiceControl.SagaAudit; - - public class InMemoryAuditStore - { - readonly ConcurrentQueue processedMessages = new(); - readonly ConcurrentQueue sagaSnapshots = new(); - readonly ConcurrentDictionary failedImports = new(); - - public void Record(ProcessedMessage processedMessage, byte[] body) => - processedMessages.Enqueue(new AuditRecord(processedMessage, body)); - - public void Record(SagaSnapshot sagaSnapshot) => sagaSnapshots.Enqueue(sagaSnapshot); - - public void Record(FailedAuditImport failedImport) => failedImports[failedImport.Id] = failedImport; - - public IReadOnlyList FailedImports => [.. failedImports.Values]; - - public bool RemoveFailedImport(string id) => failedImports.TryRemove(id, out _); - - public IReadOnlyList MessageViews => [.. processedMessages.Select(record => ToMessagesView(record.Message))]; - - public byte[]? BodyFor(string uniqueMessageId) => - processedMessages.FirstOrDefault(record => record.Message.UniqueMessageId == uniqueMessageId)?.Body; - - public IReadOnlyList<(DateTime UtcDate, long Count)> CountsFor(string endpointName) => - [ - .. processedMessages - .Where(record => EndpointOf(record.Message) == endpointName) - .GroupBy(record => record.Message.ProcessedAt.Date) - .Select(group => (UtcDate: group.Key, Count: (long)group.Count())) - .OrderBy(count => count.UtcDate) - ]; - - public (SagaHistory? History, int TotalChanges) HistoryFor(Guid sagaId, PagingInfo pagingInfo) - { - var snapshots = sagaSnapshots.Where(snapshot => snapshot.SagaId == sagaId).ToList(); - - if (snapshots.Count == 0) - { - return (null, 0); - } - - var history = new SagaHistory - { - Id = sagaId, - SagaId = sagaId, - SagaType = snapshots[0].SagaType, - Changes = - [ - .. snapshots - .OrderByDescending(snapshot => snapshot.FinishTime) - .Skip(pagingInfo.Offset) - .Take(pagingInfo.Next) - .Select(ToStateChange) - ] - }; - - return (history, snapshots.Count); - } - - static MessagesView ToMessagesView(ProcessedMessage message) => new() - { - Id = message.Id, - MessageId = Metadata(message, "MessageId"), - MessageType = Metadata(message, "MessageType"), - SendingEndpoint = Metadata(message, "SendingEndpoint"), - ReceivingEndpoint = Metadata(message, "ReceivingEndpoint"), - TimeSent = Metadata(message, "TimeSent"), - ProcessedAt = message.ProcessedAt, - CriticalTime = Metadata(message, "CriticalTime"), - ProcessingTime = Metadata(message, "ProcessingTime"), - DeliveryTime = Metadata(message, "DeliveryTime"), - IsSystemMessage = Metadata(message, "IsSystemMessage"), - ConversationId = Metadata(message, "ConversationId"), - Headers = [.. message.Headers.Select(header => new KeyValuePair(header.Key, header.Value))], - Status = MessageStatus.Successful, - MessageIntent = Metadata(message, "MessageIntent"), - BodyUrl = $"/messages/{message.UniqueMessageId}/body" - }; - - static T? Metadata(ProcessedMessage message, string key) => - message.MessageMetadata.TryGetValue(key, out var value) && value is T typed ? typed : default; - - static SagaStateChange ToStateChange(SagaSnapshot snapshot) => new() - { - StartTime = snapshot.StartTime, - FinishTime = snapshot.FinishTime, - Status = snapshot.Status, - StateAfterChange = snapshot.StateAfterChange, - InitiatingMessage = snapshot.InitiatingMessage, - OutgoingMessages = snapshot.OutgoingMessages, - Endpoint = snapshot.Endpoint - }; - - static string? EndpointOf(ProcessedMessage message) => - message.Headers.GetValueOrDefault(Headers.ProcessingEndpoint); - - sealed record AuditRecord(ProcessedMessage Message, byte[] Body); - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs deleted file mode 100644 index f7436437c4..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs +++ /dev/null @@ -1,33 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.Operations; - - class InMemoryFailedAuditImportDataStore(InMemoryAuditStore auditStore) : IFailedAuditImportDataStore - { - public Task StoreFailedAuditImport(FailedAuditImport failure, CancellationToken cancellationToken = default) - { - auditStore.Record(failure); - return Task.CompletedTask; - } - - public async Task ProcessFailedAuditImports(Func processMessage, CancellationToken cancellationToken = default) - { - foreach (var failedImport in auditStore.FailedImports) - { - if (failedImport.Message is null) - { - continue; - } - - await processMessage(failedImport.Message, cancellationToken); - auditStore.RemoveFailedImport(failedImport.Id); - } - } - - public Task QueryContainsFailedImports(CancellationToken cancellationToken = default) => - Task.FromResult(auditStore.FailedImports.Count > 0); - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs deleted file mode 100644 index 8f9e4a607f..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs +++ /dev/null @@ -1,20 +0,0 @@ -namespace ServiceControl.Persistence.Tests.AuditCapable -{ - using System; - using System.Threading; - using System.Threading.Tasks; - using ServiceControl.Persistence.Infrastructure; - using ServiceControl.SagaAudit; - - class InMemorySagaHistoryDataStore(InMemoryAuditStore auditStore) : ISagaHistoryDataStore - { - public Task> QuerySagaHistoryById(Guid sagaId, PagingInfo pagingInfo, CancellationToken cancellationToken = default) - { - var (history, totalChanges) = auditStore.HistoryFor(sagaId, pagingInfo); - - return Task.FromResult(history is null - ? QueryResult.Empty() - : new QueryResult(history, new QueryStatsInfo(DataVersion.None, totalChanges))); - } - } -} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj b/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj deleted file mode 100644 index 4de524ff84..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj +++ /dev/null @@ -1,16 +0,0 @@ - - - - net10.0 - enable - - - - - - - - - - - diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest b/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest deleted file mode 100644 index c214727198..0000000000 --- a/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Name": "AuditCapableTest", - "DisplayName": "Audit capable test persister", - "Description": "Test only persister that advertises audit support and delegates everything else to a real persister", - "AssemblyName": "ServiceControl.Persistence.Tests.AuditCapable", - "TypeName": "ServiceControl.Persistence.Tests.AuditCapable.AuditCapableTestPersistenceConfiguration, ServiceControl.Persistence.Tests.AuditCapable", - "IsSupported": false, - "SupportsAuditIngestion": true -} diff --git a/src/ServiceControl.Persistence/LocalMessagesView.cs b/src/ServiceControl.Persistence/LocalMessagesView.cs deleted file mode 100644 index e5cbab08b4..0000000000 --- a/src/ServiceControl.Persistence/LocalMessagesView.cs +++ /dev/null @@ -1,60 +0,0 @@ -namespace ServiceControl.Persistence -{ - using System; - using System.Collections.Generic; - using System.Linq; - using ServiceControl.CompositeViews.Messages; - using ServiceControl.Persistence.Infrastructure; - - /// - /// Merges the failed and audited halves of one local result set under the three rules - /// states. A persister that holds both kinds of message - /// returns them through this, so precedence, paging and counting are defined in one place rather - /// than re-derived per provider and per query. - /// - public static class LocalMessagesView - { - public static QueryResult> Merge( - IReadOnlyCollection failedMessages, - IReadOnlyCollection auditedMessages, - PagingInfo pagingInfo, - IComparer? order = null, - DataVersion version = default) - { - ArgumentNullException.ThrowIfNull(failedMessages); - ArgumentNullException.ThrowIfNull(auditedMessages); - ArgumentNullException.ThrowIfNull(pagingInfo); - - var deduplicated = new Dictionary(failedMessages.Count + auditedMessages.Count); - - // Failed first. A message that both failed and was audited must show as failed, and the - // scatter gather deduplicates with TryAdd, so whichever row is seen first wins for good. - foreach (var message in failedMessages.Concat(auditedMessages)) - { - deduplicated.TryAdd(DeduplicationKey(message), message); - } - - var merged = deduplicated.Values.ToList(); - - if (order != null) - { - merged.Sort(order); - } - - // The total is counted after deduplication, so a message that both failed and was audited - // counts once rather than once per source. - var totalCount = merged.Count; - - IList page = merged.Take(pagingInfo.PageSize).ToList(); - - return new QueryResult>(page, new QueryStatsInfo(version, totalCount)); - } - - /// - /// The key ScatterGatherApiMessageView deduplicates on, so a local merge and a cross - /// instance merge agree on what counts as the same message. - /// - public static string DeduplicationKey(MessagesView message) => - $"{message.ReceivingEndpoint?.Name}-{message.MessageId}"; - } -} diff --git a/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs index ade60f8ec0..a38b7db2e2 100644 --- a/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs +++ b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs @@ -12,12 +12,10 @@ namespace ServiceControl.UnitTests.Hosting [NonParallelizable] public class AuditIngestionOnlyCommandTests { - [TestCase("RavenDB")] - [TestCase("SQLServer")] - [TestCase("PostgreSQL")] - public void Should_refuse_to_start_against_storage_without_audit_support(string persistenceType) + [Test] + public void Should_refuse_to_start_against_storage_without_audit_support() { - var settings = CreateSettings(persistenceType); + var settings = CreateSettings("RavenDB"); var exception = Assert.ThrowsAsync(() => new AuditIngestionOnlyCommand().Execute(new HostArguments([]), settings)); diff --git a/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs b/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs index 976797662c..49153039fa 100644 --- a/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs +++ b/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs @@ -25,24 +25,23 @@ public void Absent_property_means_no_audit_support() Assert.That(manifest.SupportsAuditIngestion, Is.False); } - [TestCase("ServiceControl.Persistence.RavenDB")] - [TestCase("ServiceControl.Persistence.EFCore.SqlServer")] - [TestCase("ServiceControl.Persistence.EFCore.PostgreSql")] - public void Shipped_primary_persisters_do_not_advertise_audit_support(string projectName) + [Test] + public void RavenDB_does_not_advertise_audit_support() { - var manifest = ReadManifest(projectName); + var manifest = ReadManifest("ServiceControl.Persistence.RavenDB"); Assert.That(manifest.SupportsAuditIngestion, Is.False, - $"{projectName} advertises audit ingestion, which makes the primary host ingest the audit queue. " - + "Only flip this once that persister can store and query audit data."); + "RavenDB audit stays in the standalone audit instance; advertising support here would make the primary host ingest the audit queue."); } - [Test] - public void The_test_persister_advertises_audit_support() + [TestCase("ServiceControl.Persistence.EFCore.SqlServer")] + [TestCase("ServiceControl.Persistence.EFCore.PostgreSql")] + public void The_relational_persisters_advertise_audit_support(string projectName) { - var manifest = ReadManifest("ServiceControl.Persistence.Tests.AuditCapable"); + var manifest = ReadManifest(projectName); - Assert.That(manifest.SupportsAuditIngestion, Is.True); + Assert.That(manifest.SupportsAuditIngestion, Is.True, + $"{projectName} stores and queries audit data, so the primary host ingests the audit queue on it."); } static PersistenceManifest ReadManifest(string projectName) => diff --git a/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs b/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs deleted file mode 100644 index 8e1dbb1256..0000000000 --- a/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs +++ /dev/null @@ -1,95 +0,0 @@ -namespace ServiceControl.UnitTests.ScatterGather -{ - using System; - using System.Collections.Generic; - using System.Linq; - using NUnit.Framework; - using ServiceControl.CompositeViews.Messages; - using ServiceControl.Operations; - using ServiceControl.Persistence; - using ServiceControl.Persistence.Infrastructure; - - [TestFixture] - public class LocalMessagesViewTests - { - [Test] - public void A_message_that_both_failed_and_was_audited_shows_as_failed() - { - var failed = Message("Receiver", "1", MessageStatus.Failed); - var audited = Message("Receiver", "1", MessageStatus.Successful); - - var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); - - Assert.That(result.Results.Single().Status, Is.EqualTo(MessageStatus.Failed)); - } - - [Test] - public void A_message_that_both_failed_and_was_audited_is_counted_once() - { - var failed = Message("Receiver", "1", MessageStatus.Failed); - var audited = Message("Receiver", "1", MessageStatus.Successful); - - var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); - - using (Assert.EnterMultipleScope()) - { - Assert.That(result.Results, Has.Count.EqualTo(1)); - Assert.That(result.QueryStats.TotalCount, Is.EqualTo(1)); - } - } - - [Test] - public void The_same_message_id_on_a_different_endpoint_is_a_different_message() - { - var failed = Message("Receiver", "1", MessageStatus.Failed); - var audited = Message("OtherReceiver", "1", MessageStatus.Successful); - - var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); - - Assert.That(result.QueryStats.TotalCount, Is.EqualTo(2)); - } - - [Test] - public void A_full_page_from_each_source_is_truncated_to_one_page() - { - var pagingInfo = new PagingInfo(pageSize: 5); - var failed = Enumerable.Range(0, 5).Select(i => Message("Receiver", $"failed-{i}", MessageStatus.Failed)).ToArray(); - var audited = Enumerable.Range(0, 5).Select(i => Message("Receiver", $"audited-{i}", MessageStatus.Successful)).ToArray(); - - var result = LocalMessagesView.Merge(failed, audited, pagingInfo); - - using (Assert.EnterMultipleScope()) - { - Assert.That(result.Results, Has.Count.EqualTo(5), "the scatter gather truncates to a page, so the page must already be the local answer"); - Assert.That(result.QueryStats.TotalCount, Is.EqualTo(10), "the total counts every distinct message, not just the page"); - } - } - - [Test] - public void The_page_is_taken_after_the_requested_order_is_applied() - { - var pagingInfo = new PagingInfo(pageSize: 2); - var failed = new[] { Message("Receiver", "c", MessageStatus.Failed), Message("Receiver", "a", MessageStatus.Failed) }; - var audited = new[] { Message("Receiver", "b", MessageStatus.Successful) }; - - var result = LocalMessagesView.Merge(failed, audited, pagingInfo, MessageViewComparer.FromSortInfo(new SortInfo("message_id", "asc"))); - - Assert.That(result.Results.Select(message => message.MessageId), Is.EqualTo(new[] { "a", "b" }).AsCollection); - } - - [Test] - public void The_local_key_matches_the_key_the_scatter_gather_deduplicates_on() - { - var message = Message("Receiver", "1", MessageStatus.Failed); - - Assert.That(LocalMessagesView.DeduplicationKey(message), Is.EqualTo("Receiver-1")); - } - - static MessagesView Message(string receivingEndpoint, string messageId, MessageStatus status) => new() - { - MessageId = messageId, - Status = status, - ReceivingEndpoint = new EndpointDetails { Name = receivingEndpoint, Host = "host", HostId = Guid.NewGuid() } - }; - } -} diff --git a/src/ServiceControl.slnx b/src/ServiceControl.slnx index 39fb0c9392..622050f094 100644 --- a/src/ServiceControl.slnx +++ b/src/ServiceControl.slnx @@ -51,7 +51,6 @@ - diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 6903b02897..38f1d12685 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -642,7 +642,7 @@ competing consumers against one database, and the primary is the only writer to 6. **Turn it on.** Flip `SupportsAuditIngestion` in both manifests, update the approval test that asserts it is false, and delete `ServiceControl.Persistence.Tests.AuditCapable`. The `Empty*` audit stand-ins stay: step 7 registers them on a primary whose audit is remote. Full acceptance - runs on both providers. + runs on both providers. On `john/audit_ef_7`. 7. **Dedicated audit database.** `--audit-instance` and its guards, the controller allow list, `AuditDataLocation` and the primary's Remote behaviour, `ServiceControlQueueAddress` on the primary executable with the send-only endpoint and the reporting switch, and the persister From f78e6cb04210c9db46c54893f8cb97a5c52705b2 Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 15:33:37 +1000 Subject: [PATCH 20/21] Add the audit instance host for a dedicated audit database A primary whose audit load should not share its database can move audit to a dedicated one. ServiceControl.exe --audit-instance is the owner of that database: it ingests the audit queue, serves the primary API so the primary's scatter gather can list it under RemoteInstances, sweeps audit retention, and with --setup provisions the database, the audit queue and body storage. The primary is told the data is elsewhere through ServiceControl/AuditDataLocation=Remote, which turns off its audit receiver, its local audit queries, its audit retention pass and its partition provisioning, and replaces the old warning about remotes plus local ingestion with a validation error. Hosts on the audit database report to the primary the way the RavenDB audit instance does, custom check results and detected endpoints sent to ServiceControl/ServiceControlQueueAddress, through a send only endpoint built from the transport customization's audit profile. A worker with that setting reports too, so scaling out the audit host needs no new flag. Components no longer branch on the mode name. Settings.Host is a profile derived once from the flags and each component asks for the capability it needs, which the third host mode made worth having. Workers gain a schema probe that refuses to start against a database the owner has not migrated yet, naming setup instead of failing on the first insert. --- ...viceControl.AcceptanceTests.RavenDB.csproj | 1 + ...omposing_audit_ingestion_in_the_primary.cs | 37 ++++ .../When_hosting_an_audit_instance.cs | 121 ++++++++++++++ .../When_hosting_audit_ingestion_only.cs | 47 ++++++ .../RootControllerTests.cs | 4 +- src/ServiceControl.Infrastructure/Watchdog.cs | 7 +- .../Audit/AuditPartitionCustomCheck.cs | 8 +- .../Abstractions/BasePersistence.cs | 1 + .../Implementation/Audit/MessageViewUnion.cs | 21 ++- .../Implementation/BodyStorage/BodyStorage.cs | 9 +- .../Implementation/MessagesViewDataStore.cs | 26 +-- .../Infrastructure/DatabaseSchemaProbe.cs | 21 +++ .../Infrastructure/RetentionSweeper.cs | 6 +- .../AuditPartitionRetentionTests.cs | 20 ++- .../EFCore/Audit/RemoteAuditDataTests.cs | 56 +++++++ .../EFCore/DatabaseSchemaProbeTests.cs | 46 +++++ .../DatabaseSchemaProbeExtensions.cs | 20 +++ .../IDatabaseSchemaProbe.cs | 15 ++ .../PersistenceSettings.cs | 7 + ...rovals.PlatformSampleSettings.approved.txt | 5 +- .../Hosting/AuditInstanceCommandTests.cs | 158 ++++++++++++++++++ src/ServiceControl/Auditing/AuditComponent.cs | 42 +++-- src/ServiceControl/Auditing/AuditIngestor.cs | 3 +- src/ServiceControl/Auditing/AuditProcessor.cs | 5 +- .../Auditing/IEndpointDetectionReporter.cs | 22 +++ .../PrimaryCustomCheckResultReporter.cs | 45 +++++ .../PrimaryEndpointDetectionReporter.cs | 36 ++++ .../Auditing/Reporting/ReportingEndpoint.cs | 43 +++++ .../CustomChecks/CustomChecksComponent.cs | 4 +- .../ICustomCheckResultReporter.cs | 21 +++ .../InternalCustomCheckManager.cs | 10 +- .../InternalCustomChecks.cs | 9 +- .../InternalCustomChecksHostedService.cs | 6 +- .../ExternalIntegrationsComponent.cs | 2 +- .../HostApplicationBuilderExtensions.cs | 16 +- .../Commands/AuditIngestionOnlyCommand.cs | 4 + .../Hosting/Commands/AuditInstanceCommand.cs | 75 +++++++++ .../Hosting/Commands/AuditInstanceGuards.cs | 45 +++++ .../Commands/ErrorIngestionOnlyCommand.cs | 2 + .../Hosting/Commands/SetupCommand.cs | 9 +- src/ServiceControl/Hosting/Help.txt | 17 ++ src/ServiceControl/Hosting/HostArguments.cs | 28 +++- .../Settings/AuditDataLocation.cs | 15 ++ .../Infrastructure/Settings/HostProfile.cs | 21 +++ .../Infrastructure/Settings/Settings.cs | 52 +++++- .../HeartbeatMonitoringComponent.cs | 6 +- .../HeartbeatMonitoringHostedService.cs | 10 +- .../Persistence/PersistenceFactory.cs | 3 +- .../PersistenceServiceCollectionExtensions.cs | 10 ++ .../Recoverability/RecoverabilityComponent.cs | 4 +- src/audit-ef-persistence-plan.md | 48 +++--- 51 files changed, 1151 insertions(+), 98 deletions(-) create mode 100644 src/ServiceControl.AcceptanceTests/Auditing/When_hosting_an_audit_instance.cs create mode 100644 src/ServiceControl.Persistence.EFCore/Infrastructure/DatabaseSchemaProbe.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/Audit/RemoteAuditDataTests.cs create mode 100644 src/ServiceControl.Persistence.Tests/EFCore/DatabaseSchemaProbeTests.cs create mode 100644 src/ServiceControl.Persistence/DatabaseSchemaProbeExtensions.cs create mode 100644 src/ServiceControl.Persistence/IDatabaseSchemaProbe.cs create mode 100644 src/ServiceControl.UnitTests/Hosting/AuditInstanceCommandTests.cs create mode 100644 src/ServiceControl/Auditing/IEndpointDetectionReporter.cs create mode 100644 src/ServiceControl/Auditing/Reporting/PrimaryCustomCheckResultReporter.cs create mode 100644 src/ServiceControl/Auditing/Reporting/PrimaryEndpointDetectionReporter.cs create mode 100644 src/ServiceControl/Auditing/Reporting/ReportingEndpoint.cs create mode 100644 src/ServiceControl/CustomChecks/ICustomCheckResultReporter.cs create mode 100644 src/ServiceControl/Hosting/Commands/AuditInstanceCommand.cs create mode 100644 src/ServiceControl/Hosting/Commands/AuditInstanceGuards.cs create mode 100644 src/ServiceControl/Infrastructure/Settings/AuditDataLocation.cs create mode 100644 src/ServiceControl/Infrastructure/Settings/HostProfile.cs diff --git a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj index c52ad0407c..d83573e1e7 100644 --- a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj +++ b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj @@ -44,6 +44,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs index 5b3a39005e..ef46621125 100644 --- a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs @@ -75,6 +75,43 @@ public async Task Should_keep_every_audit_capability_but_the_receiver_when_inges } } + [Test] + public async Task Should_stand_aside_from_audit_entirely_when_the_audit_data_is_remote() + { + var (app, services) = await BuildHost(settings => + { + settings.AuditDataLocation = AuditDataLocation.Remote; + settings.RemoteInstances = [new RemoteInstanceSetting("http://localhost:44444/api")]; + }); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.False, "the audit host ingests, not this primary"); + Assert.That(app.Services.GetService(), Is.Null); + Assert.That(app.Services.GetService(), Is.Null, "licensing learns about audit from the remote"); + Assert.That(app.Services.GetRequiredService(), Is.TypeOf(), + "the local source stands aside so the scatter gather treats this instance as a non-participant"); + Assert.That(app.Services.GetRequiredService(), Is.TypeOf()); + Assert.That(app.Services.GetService(), Is.Not.Null, "the audit routes still answer, from the remote"); + } + } + finally + { + await app.DisposeAsync(); + } + } + + [Test] + public async Task Should_refuse_local_audit_ingestion_alongside_remote_instances() + { + var exception = Assert.ThrowsAsync(() => BuildHost(settings => + settings.RemoteInstances = [new RemoteInstanceSetting("http://localhost:44444/api")])); + + Assert.That(exception.Message, Does.Contain("AuditDataLocation")); + } + // The registrations are inspected rather than resolved. A normal primary hosts an NServiceBus // endpoint, and constructing every hosted service without starting it fails inside the // transport's receive component. diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_an_audit_instance.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_an_audit_instance.cs new file mode 100644 index 0000000000..15893ac02e --- /dev/null +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_an_audit_instance.cs @@ -0,0 +1,121 @@ +namespace ServiceControl.AcceptanceTests.Auditing +{ + using System; + using System.IO; + using System.Linq; + using System.Runtime.Loader; + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NUnit.Framework; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.Auditing.Reporting; + using ServiceControl.CustomChecks; + using Particular.ServiceControl.Hosting; + using ServiceControl.Hosting.Commands; + using ServiceControl.Infrastructure; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.SagaAudit; + + class When_hosting_an_audit_instance : AcceptanceTest + { + [Test] + public async Task Should_own_its_database_serve_the_api_and_report_to_the_primary() + { + var settings = await CreateSettings(); + + var host = AuditInstanceCommand.BuildHost(settings); + + try + { + var hostedServices = host.Services.GetServices().Select(service => service.GetType().Name).ToArray(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(hostedServices, Does.Contain(nameof(AuditIngestion)), "the host ingests the audit queue"); + Assert.That(hostedServices, Does.Not.Contain(nameof(ErrorIngestion)), "and never the error queue"); + Assert.That(hostedServices, Does.Contain("RetentionSweeper"), "it owns retention of its own database"); + Assert.That(host.Services.GetService(), Is.Not.Null); + + Assert.That(host.Services.GetService(), Is.Not.Null, "the send only endpoint it reports through"); + Assert.That(host.Services.GetService(), Is.TypeOf()); + Assert.That(host.Services.GetService(), Is.TypeOf()); + + Assert.That(host.Services.GetService(), Is.Not.Null, "the primary's scatter gather calls this API"); + Assert.That(host.Services.GetService(), Is.Not.Null); + Assert.That(host.Services.GetService(), Is.Not.Null); + Assert.That(host.Services.GetService(), Is.Null, "the run host never migrates; --setup does"); + } + } + finally + { + await host.DisposeAsync(); + } + } + + // Setup runs through the same flag as the host, which is what provisions the audit database, + // the audit queue and body storage for it. + [Test] + public async Task Should_provision_with_setup_and_start_with_its_send_only_endpoint() + { + var settings = await CreateSettings(); + + await new SetupCommand().Execute(new HostArguments(["--setup", "--audit-instance"]), settings); + + var host = AuditInstanceCommand.BuildHost(settings); + host.Urls.Add("http://127.0.0.1:0"); + + var started = false; + + try + { + await host.StartAsync(); + started = true; + + var session = host.Services.GetRequiredService(); + + Assert.DoesNotThrowAsync(() => host.Services.GetRequiredService().Report( + [new EndpointDetails { Name = "Sales", Host = "host", HostId = Guid.NewGuid() }])); + Assert.That(session, Is.Not.Null); + } + finally + { + if (started) + { + await host.StopAsync(); + } + + await host.DisposeAsync(); + } + } + + async Task CreateSettings() + { + var settings = new Settings(TransportIntegration.TypeName, StorageConfiguration.PersistenceType, + CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) + { + InstanceName = $"AuditInstance.{Guid.NewGuid():n}", + TransportConnectionString = TransportIntegration.ConnectionString, + MaximumConcurrencyLevel = 2, + DisableHealthChecks = true, + ServiceControlQueueAddress = $"Primary.{Guid.NewGuid():n}", + AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default + }; + + await StorageConfiguration.CustomizeSettings(settings); + + return settings; + } + + static LoggingSettings CreateLoggingSettings() + { + var logPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()); + Directory.CreateDirectory(logPath); + return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); + } + } +} diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs index 423bec8923..d9271df4ae 100644 --- a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs @@ -13,6 +13,8 @@ namespace ServiceControl.AcceptanceTests.Auditing using Particular.LicensingComponent.AuditThroughput; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing; + using ServiceControl.Auditing.Reporting; + using ServiceControl.CustomChecks; using ServiceControl.Hosting.Commands; using ServiceControl.Infrastructure; using ServiceControl.Persistence; @@ -59,6 +61,51 @@ public async Task Should_ingest_without_an_endpoint_and_without_the_single_owner } } + [Test] + public async Task Should_report_to_the_primary_when_given_its_queue() + { + var settings = await CreateSettings(); + settings.ServiceControlQueueAddress = $"Primary.{Guid.NewGuid():n}"; + + var host = AuditIngestionOnlyCommand.BuildHost(settings); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(host.Services.GetService(), Is.Not.Null, "a send only endpoint, which claims no queue"); + Assert.That(host.Services.GetService(), Is.TypeOf()); + Assert.That(host.Services.GetService(), Is.TypeOf()); + Assert.That(host.Services.GetService(), Is.Null, "still a worker: it never changes the schema"); + } + } + finally + { + await host.DisposeAsync(); + } + } + + [Test] + public async Task Should_store_its_custom_checks_locally_on_a_shared_database() + { + var settings = await CreateSettings(); + + var host = AuditIngestionOnlyCommand.BuildHost(settings); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(host.Services.GetService(), Is.TypeOf()); + Assert.That(host.Services.GetService(), Is.TypeOf()); + } + } + finally + { + await host.DisposeAsync(); + } + } + [Test] public async Task Should_report_audit_ingestion_readiness() { diff --git a/src/ServiceControl.AcceptanceTests/RootControllerTests.cs b/src/ServiceControl.AcceptanceTests/RootControllerTests.cs index 3d99e89cdb..18bcf56841 100644 --- a/src/ServiceControl.AcceptanceTests/RootControllerTests.cs +++ b/src/ServiceControl.AcceptanceTests/RootControllerTests.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.AcceptanceTests.Legacy +namespace ServiceControl.AcceptanceTests.Legacy { using System.Text.Json.Nodes; using System.Threading.Tasks; @@ -19,6 +19,8 @@ public async Task Should_gather_remote_data() SetSettings = settings => { + // A primary with audit remotes holds no audit data of its own. + settings.AuditDataLocation = AuditDataLocation.Remote; settings.RemoteInstances = [ new RemoteInstanceSetting(settings.RootUrl), diff --git a/src/ServiceControl.Infrastructure/Watchdog.cs b/src/ServiceControl.Infrastructure/Watchdog.cs index cfdacb9e30..89b572d799 100644 --- a/src/ServiceControl.Infrastructure/Watchdog.cs +++ b/src/ServiceControl.Infrastructure/Watchdog.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Infrastructure +namespace ServiceControl.Infrastructure { using System; using System.Threading; @@ -109,7 +109,10 @@ public async Task Stop(CancellationToken cancellationToken = default) { log.LogDebug("Starting watching {TaskName}", taskName); await shutdownTokenSource.CancelAsync().ConfigureAwait(false); - await watchdog.ConfigureAwait(false); + + // A host that failed to start stops every hosted service, including one whose Start + // never ran, so there may be nothing to wait for. + await (watchdog ?? Task.CompletedTask).ConfigureAwait(false); } catch (Exception e) { diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs index 37aa5a8a90..9e5cf277d4 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/Audit/AuditPartitionCustomCheck.cs @@ -2,18 +2,24 @@ namespace ServiceControl.Persistence.EFCore.PostgreSql.Audit; using Microsoft.Extensions.DependencyInjection; using NServiceBus.CustomChecks; +using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Implementation.Audit; // Only the retention owner provisions partitions, so an ingestion worker cannot insert into an hour // the owner never reached. This makes that condition visible before it bites, on every host. -class AuditPartitionCustomCheck(IServiceScopeFactory scopeFactory, IAuditPartitionManager partitions, TimeProvider timeProvider) +class AuditPartitionCustomCheck(IServiceScopeFactory scopeFactory, IAuditPartitionManager partitions, TimeProvider timeProvider, EFPersisterSettings settings) : CustomCheck("Audit partition provisioning", "ServiceControl Health", TimeSpan.FromMinutes(5)) { public static readonly TimeSpan Threshold = TimeSpan.FromHours(12); public override async Task PerformCheck(CancellationToken cancellationToken = default) { + if (!settings.HostsAuditData) + { + return CheckResult.Pass; + } + using var scope = scopeFactory.CreateScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs index e9cb55e631..25206cd26d 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/BasePersistence.cs @@ -27,6 +27,7 @@ protected static void RegisterDataStores(IServiceCollection services, EFPersiste services.TryAddSingleton(TimeProvider.System); services.AddSingleton(); + services.AddScoped(); services.AddSingleton(); services.AddSingleton(p => p.GetRequiredService()); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs index a21c44bdf2..6251b6b09a 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/Audit/MessageViewUnion.cs @@ -18,24 +18,33 @@ static class MessageViewUnion /// public const int TotalCountCap = 100_000; + /// Null where this host holds no audit data, which leaves the failed branch alone. public static async Task>> ToPagedMessagesResult( IQueryable failed, - IQueryable audited, + IQueryable? audited, PagingInfo pagingInfo, SortInfo sortInfo, CancellationToken cancellationToken = default) { var reach = pagingInfo.Offset + pagingInfo.Next; - var rows = await failed.Sort(sortInfo).Take(reach) - .Concat(audited.Sort(sortInfo).Take(reach)) - .Sort(sortInfo) + var page = audited is null + ? failed.Sort(sortInfo) + : failed.Sort(sortInfo).Take(reach).Concat(audited.Sort(sortInfo).Take(reach)).Sort(sortInfo); + + var rows = await page .Skip(pagingInfo.Offset) .Take(pagingInfo.Next) .ToListAsync(cancellationToken); - var total = await failed.Select(row => row.UniqueMessageId) - .Concat(audited.Select(row => row.UniqueMessageId)) + var counted = failed.Select(row => row.UniqueMessageId); + + if (audited is not null) + { + counted = counted.Concat(audited.Select(row => row.UniqueMessageId)); + } + + var total = await counted .Take(TotalCountCap) .LongCountAsync(cancellationToken); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs index 3f353bd6a9..293d4bd22e 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/BodyStorage.cs @@ -5,6 +5,7 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using ServiceControl.Operations.BodyStorage; +using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Entities; using ServiceControl.Persistence.EFCore.Implementation.Audit; @@ -19,11 +20,11 @@ namespace ServiceControl.Persistence.EFCore.Implementation.BodyStorage; /// where BodyText keeps only a search prefix). External storage is authoritative, so check it first. /// bodyId is usually a UniqueMessageId (a Guid) but may be a plain MessageId. /// -public class BodyStorage(IServiceScopeFactory scopeFactory, IBodyStoragePersistence storagePersistence) : DataStoreBase(scopeFactory), IBodyStorage +public class BodyStorage(IServiceScopeFactory scopeFactory, IBodyStoragePersistence storagePersistence, EFPersisterSettings settings) : DataStoreBase(scopeFactory), IBodyStorage { public async Task TryFetch(string bodyId, CancellationToken cancellationToken = default) { - var row = await ExecuteWithDbContext((dbContext, token) => ResolveBody(dbContext, bodyId, token), cancellationToken); + var row = await ExecuteWithDbContext((dbContext, token) => ResolveBody(dbContext, bodyId, settings.HostsAuditData, token), cancellationToken); if (row == null) { @@ -78,7 +79,7 @@ public async Task TryFetch(string bodyId, CancellationToken c return MessageBodyResult.Unavailable(); } - static async Task ResolveBody(ServiceControlDbContext dbContext, string bodyId, CancellationToken cancellationToken) + static async Task ResolveBody(ServiceControlDbContext dbContext, string bodyId, bool hostsAuditData, CancellationToken cancellationToken) { if (Guid.TryParse(bodyId, out var uniqueMessageId)) { @@ -95,7 +96,7 @@ public async Task TryFetch(string bodyId, CancellationToken c return byMessageId; } - return Guid.TryParse(bodyId, out var auditUniqueMessageId) + return hostsAuditData && Guid.TryParse(bodyId, out var auditUniqueMessageId) ? await QueryAudit(dbContext, auditUniqueMessageId, cancellationToken) : null; } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs b/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs index 96183bcbd8..9297083200 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/MessagesViewDataStore.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Persistence.EFCore.Implementation; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using ServiceControl.CompositeViews.Messages; +using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DbContexts; using ServiceControl.Persistence.EFCore.Entities; using ServiceControl.Persistence.EFCore.Implementation.Audit; @@ -11,7 +12,7 @@ namespace ServiceControl.Persistence.EFCore.Implementation; // Every view is the union of a failed branch and an audit branch, filtered alike, merged under the // precedence, paging and counting rules IMessagesViewDataStore states. See MessageViewUnion. -public class MessagesViewDataStore(IServiceScopeFactory scopeFactory, IFullTextSearchDialect fullTextSearch) : DataStoreBase(scopeFactory), IMessagesViewDataStore +public class MessagesViewDataStore(IServiceScopeFactory scopeFactory, IFullTextSearchDialect fullTextSearch, EFPersisterSettings settings) : DataStoreBase(scopeFactory), IMessagesViewDataStore { public Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => ExecuteQueryWithDbContext((dbContext, token) => MessageViewUnion.ToPagedMessagesResult( @@ -19,10 +20,10 @@ public Task>> GetAllMessages(PagingInfo pagingIn .IncludeSystemMessagesWhere(includeSystemMessages) .FilterBySentTimeRange(timeSentRange) .ToRows(), - Audited(dbContext) + WhenHosted(Audited(dbContext) .IncludeSystemMessagesWhere(includeSystemMessages) .FilterBySentTimeRange(timeSentRange) - .ToRows(dbContext), + .ToRows(dbContext)), pagingInfo, sortInfo, token), cancellationToken); public Task>> GetAllMessagesForEndpoint(string endpointName, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => @@ -32,11 +33,11 @@ public Task>> GetAllMessagesForEndpoint(string e .IncludeSystemMessagesWhere(includeSystemMessages) .FilterBySentTimeRange(timeSentRange) .ToRows(), - Audited(dbContext) + WhenHosted(Audited(dbContext) .Where(message => message.ReceivingEndpointName == endpointName) .IncludeSystemMessagesWhere(includeSystemMessages) .FilterBySentTimeRange(timeSentRange) - .ToRows(dbContext), + .ToRows(dbContext)), pagingInfo, sortInfo, token), cancellationToken); // includeSystemMessages is unused here: a conversation is incomplete without the system messages that took part in it. @@ -45,9 +46,9 @@ public Task>> GetAllMessagesByConversation(strin Failed(dbContext) .Where(message => message.ConversationId == conversationId) .ToRows(), - Audited(dbContext) + WhenHosted(Audited(dbContext) .Where(message => message.ConversationId == conversationId) - .ToRows(dbContext), + .ToRows(dbContext)), pagingInfo, sortInfo, token), cancellationToken); public Task>> GetAllMessagesForSearch(string searchTerms, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => @@ -55,9 +56,9 @@ public Task>> GetAllMessagesForSearch(string sea Search(Failed(dbContext), searchTerms) .FilterBySentTimeRange(timeSentRange) .ToRows(), - Search(Audited(dbContext), searchTerms) + WhenHosted(Search(Audited(dbContext), searchTerms) .FilterBySentTimeRange(timeSentRange) - .ToRows(dbContext), + .ToRows(dbContext)), pagingInfo, sortInfo, token), cancellationToken); public Task>> SearchEndpointMessages(string endpointName, string searchKeyword, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => @@ -66,16 +67,19 @@ public Task>> SearchEndpointMessages(string endp .Where(message => message.ReceivingEndpointName == endpointName) .FilterBySentTimeRange(timeSentRange) .ToRows(), - Search(Audited(dbContext), searchKeyword) + WhenHosted(Search(Audited(dbContext), searchKeyword) .Where(message => message.ReceivingEndpointName == endpointName) .FilterBySentTimeRange(timeSentRange) - .ToRows(dbContext), + .ToRows(dbContext)), pagingInfo, sortInfo, token), cancellationToken); static IQueryable Failed(ServiceControlDbContext dbContext) => dbContext.FailedMessages.AsNoTracking(); static IQueryable Audited(ServiceControlDbContext dbContext) => dbContext.AuditMessages.AsNoTracking(); + // A primary whose audit data is on a dedicated host queries only the tables it owns. + IQueryable? WhenHosted(IQueryable audited) => settings.HostsAuditData ? audited : null; + // Neither search hides system messages: a caller who searched // for something specific is not helped by hiding the message that matched it. IQueryable Search(IQueryable source, string searchTerms) => diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/DatabaseSchemaProbe.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/DatabaseSchemaProbe.cs new file mode 100644 index 0000000000..54fc65b1d3 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/DatabaseSchemaProbe.cs @@ -0,0 +1,21 @@ +namespace ServiceControl.Persistence.EFCore.Infrastructure; + +using Microsoft.EntityFrameworkCore; +using ServiceControl.Persistence.EFCore.DbContexts; + +class DatabaseSchemaProbe(ServiceControlDbContext dbContext) : IDatabaseSchemaProbe +{ + public async Task EnsureCurrent(CancellationToken cancellationToken = default) + { + var pending = (await dbContext.Database.GetPendingMigrationsAsync(cancellationToken)).ToArray(); + + if (pending.Length == 0) + { + return; + } + + throw new InvalidOperationException( + $"The database is missing {pending.Length} migration(s) this version needs, starting with '{pending[0]}'. " + + "Run setup on the instance that owns this database, then start this host again. Workers never migrate a database."); + } +} diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs index 38708e6f75..b9f84b7a22 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/RetentionSweeper.cs @@ -174,7 +174,11 @@ async Task SweepBody(DateTime? errorCutoff, DateTime? eventsCutoff, bool pace, C await RunPass(RetentionEntity.FailedMessages, token => SweepFailedMessages(pace, errorCutoff, token), cancellationToken); await RunPass(RetentionEntity.EventLog, token => SweepEventLogItems(pace, eventsCutoff, token), cancellationToken); await RunPass(RetentionEntity.GroupComments, SweepOrphanedGroupComments, cancellationToken); - await RunPass(RetentionEntity.Audit, token => SweepAudit(pace, token), cancellationToken); + + if (settings.HostsAuditData) + { + await RunPass(RetentionEntity.Audit, token => SweepAudit(pace, token), cancellationToken); + } } // Audit rows are stored by ingestion hour and expire an hour at a time, once the whole hour is diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs index cf647b5fc9..eb4ecb3e68 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/AuditPartitionRetentionTests.cs @@ -54,7 +54,8 @@ public async Task The_provisioning_check_passes_while_the_window_is_ahead_and_fa var check = new AuditPartitionCustomCheck( ServiceProvider.GetRequiredService(), Partitions, - ServiceProvider.GetRequiredService()); + ServiceProvider.GetRequiredService(), + EFSettings); var beforehand = await check.PerformCheck(); @@ -70,6 +71,23 @@ public async Task The_provisioning_check_passes_while_the_window_is_ahead_and_fa } } + [Test] + public async Task The_provisioning_check_passes_where_the_audit_data_is_remote() + { + EFSettings.HostsAuditData = false; + var check = new AuditPartitionCustomCheck( + ServiceProvider.GetRequiredService(), + Partitions, + ServiceProvider.GetRequiredService(), + EFSettings); + + AdvanceClock(AuditHours.Lookahead); + + var result = await check.PerformCheck(); + + Assert.That(result.HasFailed, Is.False, "a primary that holds no audit data has no partitions to keep ahead"); + } + async Task PartitionExists(string table, DateTime hour) { var names = await Query(dbContext => dbContext.Database diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Audit/RemoteAuditDataTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Audit/RemoteAuditDataTests.cs new file mode 100644 index 0000000000..d504d59943 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Audit/RemoteAuditDataTests.cs @@ -0,0 +1,56 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Implementation.Audit; +using ServiceControl.Persistence.Infrastructure; + +// A primary whose audit data lives on a dedicated audit host carries the audit tables, empty, and +// must neither query nor sweep them. The setting is flipped after setup so the rows the tests seed +// are the ones a misconfiguration would have left behind. +class RemoteAuditDataTests : AuditRetentionTestBase +{ + [SetUp] + public void AuditIsRemote() => EFSettings.HostsAuditData = false; + + [Test] + public async Task The_message_views_read_only_the_failed_messages() + { + var failure = new IngestedFailure(); + var audit = new IngestedAudit(); + + await Ingest(failure); + await IngestAudit(audit); + + var result = await MessagesViewStore.GetAllMessages(new PagingInfo(), new SortInfo(), includeSystemMessages: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results.Select(view => view.Id), Is.EqualTo(new[] { failure.UniqueMessageIdString })); + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(1)); + } + } + + [Test] + public async Task Bodies_are_not_resolved_from_the_audit_table() + { + var audit = new IngestedAudit(); + + await IngestAudit(audit); + + var result = await BodyStorage.TryFetch(audit.UniqueMessageIdString); + + Assert.That(result.State, Is.EqualTo(Operations.BodyStorage.MessageBodyState.NotFound)); + } + + [Test] + public async Task The_retention_sweep_leaves_the_audit_tables_alone() + { + var expired = await SeedHour(AuditHours.Truncate(Now - Retention).AddHours(-1)); + + await RunRetentionSweep(); + + Assert.That(await HourIsGone(expired), Is.False); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/DatabaseSchemaProbeTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/DatabaseSchemaProbeTests.cs new file mode 100644 index 0000000000..9f22bb9ba5 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/DatabaseSchemaProbeTests.cs @@ -0,0 +1,46 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.DbContexts; + +class DatabaseSchemaProbeTests : PersistenceTestBase +{ + [Test] + public async Task A_migrated_database_passes() + { + await ServiceProvider.EnsureDatabaseSchemaIsCurrent(); + } + + // The newest migration is removed from the history table, which is what a worker started + // against a database its owner has not migrated yet would find. + [Test] + public async Task A_database_behind_the_binary_refuses_to_start_and_names_setup() + { + using (var scope = ServiceProvider.CreateScope()) + { + var dbContext = scope.ServiceProvider.GetRequiredService(); + var newest = (await dbContext.Database.GetAppliedMigrationsAsync()).Last(); + var isSqlServer = dbContext.Database.ProviderName!.Contains("SqlServer", StringComparison.OrdinalIgnoreCase); + var historyTable = (isSqlServer, dbContext.Schema) switch + { + (true, null) => "[__EFMigrationsHistory]", + (true, var schema) => $"[{schema}].[__EFMigrationsHistory]", + (false, null) => "\"__EFMigrationsHistory\"", + (false, var schema) => $"\"{schema}\".\"__EFMigrationsHistory\"" + }; + var idColumn = isSqlServer ? "[MigrationId]" : "migration_id"; + + var sql = "DELETE FROM " + historyTable + " WHERE " + idColumn + " = {0}"; + await dbContext.Database.ExecuteSqlRawAsync(sql, newest); + } + + var exception = Assert.ThrowsAsync(() => ServiceProvider.EnsureDatabaseSchemaIsCurrent()); + + Assert.That(exception.Message, Does.Contain("Run setup on the instance that owns this database")); + } +} diff --git a/src/ServiceControl.Persistence/DatabaseSchemaProbeExtensions.cs b/src/ServiceControl.Persistence/DatabaseSchemaProbeExtensions.cs new file mode 100644 index 0000000000..f29560ecf8 --- /dev/null +++ b/src/ServiceControl.Persistence/DatabaseSchemaProbeExtensions.cs @@ -0,0 +1,20 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + + public static class DatabaseSchemaProbeExtensions + { + public static async Task EnsureDatabaseSchemaIsCurrent(this IServiceProvider services, CancellationToken cancellationToken = default) + { + await using var scope = services.CreateAsyncScope(); + + if (scope.ServiceProvider.GetService() is { } probe) + { + await probe.EnsureCurrent(cancellationToken); + } + } + } +} diff --git a/src/ServiceControl.Persistence/IDatabaseSchemaProbe.cs b/src/ServiceControl.Persistence/IDatabaseSchemaProbe.cs new file mode 100644 index 0000000000..6fbae1c555 --- /dev/null +++ b/src/ServiceControl.Persistence/IDatabaseSchemaProbe.cs @@ -0,0 +1,15 @@ +namespace ServiceControl.Persistence +{ + using System.Threading; + using System.Threading.Tasks; + + /// + /// Lets a host that never migrates the database, an ingestion worker, refuse to start against a + /// database the owner has not migrated yet, rather than failing on its first write. Registered + /// by persisters that keep a migration history; absent on the rest. + /// + public interface IDatabaseSchemaProbe + { + Task EnsureCurrent(CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/PersistenceSettings.cs b/src/ServiceControl.Persistence/PersistenceSettings.cs index fb7c3b5ff7..2ec4793b92 100644 --- a/src/ServiceControl.Persistence/PersistenceSettings.cs +++ b/src/ServiceControl.Persistence/PersistenceSettings.cs @@ -18,6 +18,13 @@ public abstract class PersistenceSettings /// public bool RunRetentionSweep { get; set; } = true; + /// + /// Whether the audit data lives in this database. False on a primary whose audit is on a + /// dedicated audit host: the audit tables exist, empty, and nothing queries, sweeps or + /// provisions them. + /// + public bool HostsAuditData { get; set; } = true; + public bool EnableFullTextSearchOnBodies { get; set; } = true; public TimeSpan? OverrideCustomCheckRepeatTime { get; set; } diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt index ebcec51057..d56a2ccfe5 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt @@ -1,4 +1,4 @@ -{ +{ "LoggingSettings": { "LogLevel": "Information", "LogPath": "C:\\Logs" @@ -78,6 +78,9 @@ "TimeToRestartAuditIngestionAfterFailure": "00:01:00", "ErrorIngestionOnly": false, "AuditIngestionOnly": false, + "AuditInstance": false, + "AuditDataLocation": "Local", + "ServiceControlQueueAddress": null, "AuditRetentionPeriod": null, "ErrorRetentionPeriod": "10.00:00:00", "EventsRetentionPeriod": "14.00:00:00", diff --git a/src/ServiceControl.UnitTests/Hosting/AuditInstanceCommandTests.cs b/src/ServiceControl.UnitTests/Hosting/AuditInstanceCommandTests.cs new file mode 100644 index 0000000000..375a348466 --- /dev/null +++ b/src/ServiceControl.UnitTests/Hosting/AuditInstanceCommandTests.cs @@ -0,0 +1,158 @@ +namespace ServiceControl.UnitTests.Hosting +{ + using System; + using NUnit.Framework; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Hosting.Commands; + + // Environment variables are process wide, so these cannot run alongside anything else that reads them. + [TestFixture] + [NonParallelizable] + public class AuditInstanceCommandTests + { + [Test] + public void Parses_the_flag_into_the_command() + { + var arguments = new HostArguments(["--audit-instance"]); + + using (Assert.EnterMultipleScope()) + { + Assert.That(arguments.Command, Is.EqualTo(typeof(AuditInstanceCommand))); + Assert.That(arguments.AuditInstance, Is.True); + } + } + + [Test] + public void Setup_keeps_the_flag_so_it_provisions_the_audit_host() + { + var arguments = new HostArguments(["--setup", "--audit-instance"]); + + using (Assert.EnterMultipleScope()) + { + Assert.That(arguments.Command, Is.EqualTo(typeof(SetupCommand))); + Assert.That(arguments.AuditInstance, Is.True); + } + } + + [Test] + public void Refuses_to_combine_with_an_ingestion_only_mode() + { + var exception = Assert.Throws(() => + AuditInstanceGuards.EnsureNotCombinedWithIngestionOnly(auditInstance: true, ingestionOnly: true)); + + Assert.That(exception.Message, Does.Contain("cannot be combined")); + } + + [Test] + public void Refuses_storage_without_audit_support() + { + var settings = CreateSettings("RavenDB"); + settings.ServiceControlQueueAddress = "Particular.ServiceControl"; + + var exception = Assert.Throws(() => AuditInstanceGuards.EnsureCanRun(settings)); + + Assert.That(exception.Message, Does.Contain("supports audit ingestion")); + } + + [Test] + public void Refuses_to_run_without_the_primary_queue_to_report_to() + { + var settings = CreateSettings("PostgreSQL"); + + var exception = Assert.Throws(() => AuditInstanceGuards.EnsureCanRun(settings)); + + Assert.That(exception.Message, Does.Contain("ServiceControlQueueAddress")); + } + + [Test] + public void Refuses_remote_instances_because_an_audit_host_is_a_leaf() + { + var settings = CreateSettings("PostgreSQL"); + settings.ServiceControlQueueAddress = "Particular.ServiceControl"; + settings.RemoteInstances = [new RemoteInstanceSetting("http://localhost:44444/api")]; + + var exception = Assert.Throws(() => AuditInstanceGuards.EnsureCanRun(settings)); + + Assert.That(exception.Message, Does.Contain("remote instances")); + } + + [Test] + public void Reads_the_audit_data_location() + { + using var _ = new EnvironmentVariableScope("SERVICECONTROL_AUDITDATALOCATION", "remote"); + + Assert.That(CreateSettings("PostgreSQL").AuditDataLocation, Is.EqualTo(AuditDataLocation.Remote)); + } + + [Test] + public void Defaults_the_audit_data_location_to_local() + { + Assert.That(CreateSettings("PostgreSQL").AuditDataLocation, Is.EqualTo(AuditDataLocation.Local)); + } + + [Test] + public void Rejects_an_unknown_audit_data_location() + { + using var _ = new EnvironmentVariableScope("SERVICECONTROL_AUDITDATALOCATION", "elsewhere"); + + var exception = Assert.Throws(() => CreateSettings("PostgreSQL")); + + Assert.That(exception.Message, Does.Contain("expected Local or Remote")); + } + + [Test] + public void The_host_profile_of_an_audit_instance_owns_retention_and_the_api_but_not_the_primary_endpoint() + { + var settings = CreateSettings("PostgreSQL"); + settings.ServiceControlQueueAddress = "Particular.ServiceControl"; + AuditInstanceCommand.ApplyMode(settings); + + var profile = settings.Host; + + using (Assert.EnterMultipleScope()) + { + Assert.That(profile.HostsApi, Is.True); + Assert.That(profile.OwnsRetention, Is.True); + Assert.That(profile.HostsPrimaryEndpoint, Is.False); + Assert.That(profile.OwnsSingletonWork, Is.False); + Assert.That(profile.MonitorsHeartbeats, Is.False); + Assert.That(profile.ReportsToPrimary, Is.True); + } + } + + [Test] + public void A_worker_reports_to_the_primary_only_when_given_its_queue() + { + var silent = CreateSettings("PostgreSQL"); + silent.AuditIngestionOnly = true; + + var reporting = CreateSettings("PostgreSQL"); + reporting.AuditIngestionOnly = true; + reporting.ServiceControlQueueAddress = "Particular.ServiceControl"; + + using (Assert.EnterMultipleScope()) + { + Assert.That(silent.Host.ReportsToPrimary, Is.False); + Assert.That(reporting.Host.ReportsToPrimary, Is.True); + Assert.That(reporting.Host.OwnsRetention, Is.False); + } + } + + static Settings CreateSettings(string persistenceType) => + new("LearningTransport", persistenceType, forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)); + + sealed class EnvironmentVariableScope : IDisposable + { + readonly string name; + + public EnvironmentVariableScope(string name, string value) + { + this.name = name; + Environment.SetEnvironmentVariable(name, value); + } + + public void Dispose() => Environment.SetEnvironmentVariable(name, null); + } + } +} diff --git a/src/ServiceControl/Auditing/AuditComponent.cs b/src/ServiceControl/Auditing/AuditComponent.cs index 3d54c545bd..47c2e7b253 100644 --- a/src/ServiceControl/Auditing/AuditComponent.cs +++ b/src/ServiceControl/Auditing/AuditComponent.cs @@ -1,15 +1,15 @@ -namespace ServiceControl.Auditing +namespace ServiceControl.Auditing { + using System; using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Hosting; - using Microsoft.Extensions.Logging; using Particular.LicensingComponent.AuditThroughput; using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.Auditing.Metrics; using ServiceControl.Connection; using ServiceControl.CustomChecks; - using ServiceControl.Infrastructure; using ServiceControl.Infrastructure.Health; using ServiceControl.Persistence; using ServiceControl.Transports; @@ -20,7 +20,7 @@ class AuditComponent : ServiceControlComponent { public override void Setup(Settings settings, IComponentInstallationContext context, IHostApplicationBuilder hostBuilder) { - if (!SupportsAuditIngestion(settings)) + if (!HostsAuditData(settings)) { return; } @@ -35,15 +35,17 @@ public override void Setup(Settings settings, IComponentInstallationContext cont public override void Configure(Settings settings, ITransportCustomization transportCustomization, IHostApplicationBuilder hostBuilder) { - if (!SupportsAuditIngestion(settings)) + if (!HostsAuditData(settings)) { return; } - WarnAboutSettingCollisions(settings); + EnsureRemotesAreNotCombinedWithLocalIngestion(settings); var services = hostBuilder.Services; + services.TryAddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); @@ -60,7 +62,7 @@ public override void Configure(Settings settings, ITransportCustomization transp services.AddHostedService(); } - if (!settings.IngestionOnly) + if (settings.Host.HostsApi) { // Registered before the licensing component's own fallback, which uses TryAdd. services.AddSingleton(); @@ -68,23 +70,29 @@ public override void Configure(Settings settings, ITransportCustomization transp } } - // ServiceControl and ServiceControl.Audit settings can both be set by bare environment variable - // name, and ServiceBus/AuditQueue is literally the same key for both processes, so a combined - // primary and a standalone audit instance sharing one environment file collide. That - // combination is unsupported, and this is the shape most likely to hit it. - static void WarnAboutSettingCollisions(Settings settings) + // A primary that ingests audit itself and also lists audit remotes is one of two mistakes: + // either the remotes are left over from before audit moved into this database, or the audit + // data was meant to be Remote and this instance would ingest a queue that belongs to the + // audit host. Both read the same setting names, so a shared environment file also makes the + // two processes collide on the audit queue, retention, forwarding and ingestion settings. + static void EnsureRemotesAreNotCombinedWithLocalIngestion(Settings settings) { - if (settings.RemoteInstances.Length == 0) + if (settings.RemoteInstances.Length == 0 || !settings.IngestAuditMessages || settings.AuditInstance) { return; } - LoggerUtil.CreateStaticLogger(typeof(AuditComponent), settings.LoggingSettings.LogLevel) - .LogWarning("This instance ingests audit messages itself and also has {RemoteInstanceCount} audit remote(s) configured. " - + "Running both is not supported: the two processes read the same setting names, so a shared environment file makes them " - + "collide on the audit queue, retention, forwarding and ingestion settings.", settings.RemoteInstances.Length); + throw new Exception( + $"This instance is configured to ingest audit messages into its own database and also lists {settings.RemoteInstances.Length} remote instance(s). " + + "Set ServiceControl/AuditDataLocation to Remote if the audit data lives on a dedicated audit host, " + + "set ServiceControl/IngestAuditMessages to false if only workers ingest, or remove the remotes."); } + // Audit support has to be advertised by the persister, and the primary has to be told the data + // is local rather than on a dedicated audit host. + internal static bool HostsAuditData(Settings settings) => + SupportsAuditIngestion(settings) && settings.AuditDataLocation == AuditDataLocation.Local; + internal static bool SupportsAuditIngestion(Settings settings) => PersistenceManifestLibrary.Find(settings.PersistenceType)?.SupportsAuditIngestion ?? false; } diff --git a/src/ServiceControl/Auditing/AuditIngestor.cs b/src/ServiceControl/Auditing/AuditIngestor.cs index 7b809d6178..4c1afd1182 100644 --- a/src/ServiceControl/Auditing/AuditIngestor.cs +++ b/src/ServiceControl/Auditing/AuditIngestor.cs @@ -22,6 +22,7 @@ public AuditIngestor( IIngestionUnitOfWorkFactory unitOfWorkFactory, IEndpointInstanceMonitoring endpointInstanceMonitoring, ITransportCustomization transportCustomization, + IEndpointDetectionReporter endpointDetectionReporter, ILogger logger) { this.settings = settings; @@ -40,7 +41,7 @@ public AuditIngestor( new SagaRelationshipsEnricher() ]; - processor = new AuditProcessor(enrichers, logger); + processor = new AuditProcessor(enrichers, endpointDetectionReporter, logger); } public async Task Ingest(List contexts, IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) diff --git a/src/ServiceControl/Auditing/AuditProcessor.cs b/src/ServiceControl/Auditing/AuditProcessor.cs index b78879c5fd..980b5b9eae 100644 --- a/src/ServiceControl/Auditing/AuditProcessor.cs +++ b/src/ServiceControl/Auditing/AuditProcessor.cs @@ -2,6 +2,7 @@ namespace ServiceControl.Auditing { using System; using System.Collections.Generic; + using System.Linq; using System.Text.Json; using System.Threading; using System.Threading.Tasks; @@ -17,7 +18,7 @@ namespace ServiceControl.Auditing using ServiceControl.Persistence.UnitOfWork; using ServiceControl.SagaAudit; - class AuditProcessor(IEnrichImportedAuditMessages[] enrichers, ILogger logger) + class AuditProcessor(IEnrichImportedAuditMessages[] enrichers, IEndpointDetectionReporter endpointDetectionReporter, ILogger logger) { public async Task> Process(IReadOnlyList contexts, IIngestionUnitOfWork unitOfWork, IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) { @@ -71,6 +72,8 @@ public async Task> Process(IReadOnlyList endpoint.EndpointDetails)], cancellationToken); + return storedContexts; } diff --git a/src/ServiceControl/Auditing/IEndpointDetectionReporter.cs b/src/ServiceControl/Auditing/IEndpointDetectionReporter.cs new file mode 100644 index 0000000000..66db275491 --- /dev/null +++ b/src/ServiceControl/Auditing/IEndpointDetectionReporter.cs @@ -0,0 +1,22 @@ +namespace ServiceControl.Auditing +{ + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations; + + /// + /// Where the endpoints detected from audit messages go besides this host's known endpoints. On a + /// shared database that is nowhere, the known endpoints table is the primary's own. On a + /// dedicated audit database they are reported to the primary. + /// + interface IEndpointDetectionReporter + { + Task Report(IReadOnlyCollection endpoints, CancellationToken cancellationToken = default); + } + + class NoEndpointDetectionReporter : IEndpointDetectionReporter + { + public Task Report(IReadOnlyCollection endpoints, CancellationToken cancellationToken = default) => Task.CompletedTask; + } +} diff --git a/src/ServiceControl/Auditing/Reporting/PrimaryCustomCheckResultReporter.cs b/src/ServiceControl/Auditing/Reporting/PrimaryCustomCheckResultReporter.cs new file mode 100644 index 0000000000..12dcd5f77b --- /dev/null +++ b/src/ServiceControl/Auditing/Reporting/PrimaryCustomCheckResultReporter.cs @@ -0,0 +1,45 @@ +namespace ServiceControl.Auditing.Reporting +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Logging; + using NServiceBus; + using ServiceControl.CustomChecks; + using ServiceControl.Contracts.CustomChecks; + using ServiceControl.Plugin.CustomChecks.Messages; + + // The message session is resolved per report rather than injected: the endpoint starts after the + // custom checks hosted service, and a check that fires before it has started is simply skipped + // until the next interval. + class PrimaryCustomCheckResultReporter(IServiceProvider serviceProvider, PrimaryQueue primaryQueue, ILogger logger) : ICustomCheckResultReporter + { + public async Task Report(CustomCheckDetail detail, CancellationToken cancellationToken = default) + { + var message = new ReportCustomCheckResult + { + HostId = detail.OriginatingEndpoint.HostId, + CustomCheckId = detail.CustomCheckId, + Category = detail.Category, + HasFailed = detail.HasFailed, + FailureReason = detail.FailureReason, + ReportedAt = detail.ReportedAt, + EndpointName = detail.OriginatingEndpoint.Name, + Host = detail.OriginatingEndpoint.Host + }; + + var options = new SendOptions(); + options.SetDestination(primaryQueue.Address); + + try + { + await serviceProvider.GetRequiredService().Send(message, options, cancellationToken); + } + catch (InvalidOperationException e) + { + logger.LogDebug(e, "Custom check {CustomCheckId} was not reported to the primary because the reporting endpoint has not started yet", detail.CustomCheckId); + } + } + } +} diff --git a/src/ServiceControl/Auditing/Reporting/PrimaryEndpointDetectionReporter.cs b/src/ServiceControl/Auditing/Reporting/PrimaryEndpointDetectionReporter.cs new file mode 100644 index 0000000000..335b7d45ee --- /dev/null +++ b/src/ServiceControl/Auditing/Reporting/PrimaryEndpointDetectionReporter.cs @@ -0,0 +1,36 @@ +namespace ServiceControl.Auditing.Reporting +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + using NServiceBus; + using ServiceControl.Contracts.EndpointControl; + using ServiceControl.Operations; + + // Known endpoints are still recorded in this host's own database, because the endpoint monitor + // warms from there. This additionally tells the primary, which is the only host ServicePulse + // asks about endpoints. + class PrimaryEndpointDetectionReporter(IServiceProvider serviceProvider, PrimaryQueue primaryQueue, TimeProvider timeProvider) : IEndpointDetectionReporter + { + public async Task Report(IReadOnlyCollection endpoints, CancellationToken cancellationToken = default) + { + if (endpoints.Count == 0) + { + return; + } + + var session = serviceProvider.GetRequiredService(); + var detectedAt = timeProvider.GetUtcNow().UtcDateTime; + + foreach (var endpoint in endpoints) + { + var options = new SendOptions(); + options.SetDestination(primaryQueue.Address); + + await session.Send(new RegisterNewEndpoint { DetectedAt = detectedAt, Endpoint = endpoint }, options, cancellationToken); + } + } + } +} diff --git a/src/ServiceControl/Auditing/Reporting/ReportingEndpoint.cs b/src/ServiceControl/Auditing/Reporting/ReportingEndpoint.cs new file mode 100644 index 0000000000..2485c6e8de --- /dev/null +++ b/src/ServiceControl/Auditing/Reporting/ReportingEndpoint.cs @@ -0,0 +1,43 @@ +namespace ServiceControl.Auditing.Reporting +{ + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + using NServiceBus; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Configuration; + using ServiceControl.CustomChecks; + using ServiceControl.Transports; + + /// + /// The send only NServiceBus endpoint a host on a dedicated audit database reports through. It + /// carries two messages to the primary, custom check results and detected endpoints, which the + /// primary already handles from the standalone audit instance. Send only claims no queue, so a + /// worker keeps owning nothing. + /// + static class ReportingEndpoint + { + public static void Add(IServiceCollection services, Settings settings, ITransportCustomization transportCustomization, TransportSettings transportSettings) + { + var configuration = new EndpointConfiguration(settings.InstanceName); + configuration.AssemblyScanner().Disable = true; + + transportCustomization.CustomizeAuditEndpoint(configuration, transportSettings); + + configuration.UseSerialization(); + configuration.SetDiagnosticsPath(settings.LoggingSettings.LogPath); + + if (AppEnvironment.RunningInContainer) + { + configuration.CustomDiagnosticsWriter((_, _) => Task.CompletedTask); + } + + services.AddNServiceBusEndpoint(configuration); + + services.AddSingleton(new PrimaryQueue(transportCustomization.ToTransportQualifiedQueueName(settings.ServiceControlQueueAddress))); + services.AddSingleton(); + services.AddSingleton(); + } + } + + sealed record PrimaryQueue(string Address); +} diff --git a/src/ServiceControl/CustomChecks/CustomChecksComponent.cs b/src/ServiceControl/CustomChecks/CustomChecksComponent.cs index 7b486e3d15..23ee751462 100644 --- a/src/ServiceControl/CustomChecks/CustomChecksComponent.cs +++ b/src/ServiceControl/CustomChecks/CustomChecksComponent.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.CustomChecks +namespace ServiceControl.CustomChecks { using Connection; using Contracts; @@ -29,7 +29,7 @@ public override void Configure(Settings settings, ITransportCustomization transp hostBuilder.Services.AddEventLogMapping(); hostBuilder.Services.AddEventLogMapping(); - if (!settings.IngestionOnly) + if (settings.Host.HostsApi) { hostBuilder.Services.AddPlatformConnectionProvider(); } diff --git a/src/ServiceControl/CustomChecks/ICustomCheckResultReporter.cs b/src/ServiceControl/CustomChecks/ICustomCheckResultReporter.cs new file mode 100644 index 0000000000..70388d30df --- /dev/null +++ b/src/ServiceControl/CustomChecks/ICustomCheckResultReporter.cs @@ -0,0 +1,21 @@ +namespace ServiceControl.CustomChecks +{ + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Contracts.CustomChecks; + + /// + /// Where this host's own custom check results go. A host that owns its database stores them; a + /// host on a dedicated audit database sends them to the primary, which is the only host + /// ServicePulse asks. + /// + interface ICustomCheckResultReporter + { + Task Report(CustomCheckDetail detail, CancellationToken cancellationToken = default); + } + + class LocalCustomCheckResultReporter(CustomCheckResultProcessor processor) : ICustomCheckResultReporter + { + public Task Report(CustomCheckDetail detail, CancellationToken cancellationToken = default) => processor.ProcessResult(detail, cancellationToken); + } +} diff --git a/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs b/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs index 803712c068..14652668d1 100644 --- a/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs +++ b/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.CustomChecks +namespace ServiceControl.CustomChecks { using System; using System.Threading; @@ -15,13 +15,13 @@ public InternalCustomCheckManager( ICustomCheck check, EndpointDetails localEndpointDetails, IAsyncTimer scheduler, - CustomCheckResultProcessor checkResultProcessor, + ICustomCheckResultReporter checkResultReporter, ILogger logger) { this.check = check; this.localEndpointDetails = localEndpointDetails; this.scheduler = scheduler; - this.checkResultProcessor = checkResultProcessor; + this.checkResultReporter = checkResultReporter; this.logger = logger; } @@ -63,7 +63,7 @@ async Task Run(CancellationToken cancellationToken) FailureReason = result.FailureReason }; - await checkResultProcessor.ProcessResult(detail, cancellationToken); + await checkResultReporter.Report(detail, cancellationToken); return check.Interval.HasValue ? TimerJobExecutionResult.ScheduleNextExecution @@ -76,7 +76,7 @@ async Task Run(CancellationToken cancellationToken) readonly ICustomCheck check; readonly EndpointDetails localEndpointDetails; readonly IAsyncTimer scheduler; - readonly CustomCheckResultProcessor checkResultProcessor; + readonly ICustomCheckResultReporter checkResultReporter; readonly ILogger logger; } } \ No newline at end of file diff --git a/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecks.cs b/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecks.cs index a34b41a62e..024271ae24 100644 --- a/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecks.cs +++ b/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecks.cs @@ -1,8 +1,9 @@ -namespace ServiceControl.CustomChecks +namespace ServiceControl.CustomChecks { using System.Linq; using Infrastructure.BackgroundTasks; using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using NServiceBus.CustomChecks; @@ -20,11 +21,15 @@ public static IHostApplicationBuilder AddInternalCustomChecks(this IHostApplicat services.AddCustomCheck(); services.AddCustomCheck(); + // A host that reports to a primary registers its own reporter before this runs, so the + // local one only fills the gap. + services.TryAddSingleton(); + services.AddHostedService(provider => new InternalCustomChecksHostedService( [.. provider.GetServices()], provider.GetRequiredService(), provider.GetRequiredService(), - provider.GetRequiredService(), + provider.GetRequiredService(), provider.GetRequiredService().InstanceName, provider.GetRequiredService>())); return hostBuilder; diff --git a/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecksHostedService.cs b/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecksHostedService.cs index ca1b729ad8..a260eb3974 100644 --- a/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecksHostedService.cs +++ b/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomChecksHostedService.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.CustomChecks +namespace ServiceControl.CustomChecks { using System.Collections.Generic; using System.Linq; @@ -15,7 +15,7 @@ class InternalCustomChecksHostedService( IList customChecks, HostInformation hostInfo, IAsyncTimer scheduler, - CustomCheckResultProcessor checkResultProcessor, + ICustomCheckResultReporter checkResultReporter, string endpointName, ILogger logger) : IHostedService @@ -24,7 +24,7 @@ public Task StartAsync(CancellationToken cancellationToken = default) { foreach (var check in customChecks) { - var checkManager = new InternalCustomCheckManager(check, localEndpointDetails, scheduler, checkResultProcessor, logger); + var checkManager = new InternalCustomCheckManager(check, localEndpointDetails, scheduler, checkResultReporter, logger); checkManager.Start(); managers.Add(checkManager); diff --git a/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs b/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs index 9e1782a333..571f37d5cd 100644 --- a/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs +++ b/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs @@ -18,7 +18,7 @@ public override void Configure(Settings settings, ITransportCustomization transp { services.AddDomainEventHandler(); - if (!settings.IngestionOnly) + if (settings.Host.OwnsSingletonWork) { services.AddHostedService(); } diff --git a/src/ServiceControl/HostApplicationBuilderExtensions.cs b/src/ServiceControl/HostApplicationBuilderExtensions.cs index 3f6715779b..ce8299603e 100644 --- a/src/ServiceControl/HostApplicationBuilderExtensions.cs +++ b/src/ServiceControl/HostApplicationBuilderExtensions.cs @@ -1,4 +1,4 @@ -namespace Particular.ServiceControl +namespace Particular.ServiceControl { using System; using System.Diagnostics; @@ -31,6 +31,7 @@ using NServiceBus.Transport; using OpenTelemetry.Metrics; using global::ServiceControl.Auditing.Metrics; + using global::ServiceControl.Auditing.Reporting; using OpenTelemetry.Resources; using Particular.LicensingComponent; using ServiceBus.Management.Infrastructure; @@ -43,7 +44,7 @@ static class HostApplicationBuilderExtensions public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, Settings settings, EndpointConfiguration configuration, params ReadOnlySpan components) { - if (!settings.IngestionOnly) + if (settings.Host.HostsPrimaryEndpoint) { ArgumentNullException.ThrowIfNull(configuration); } @@ -108,10 +109,10 @@ public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, S hostBuilder.AddTelemetry(settings); services.AddServiceControlHealthChecks(); - if (settings.IngestionOnly) + if (!settings.Host.HostsPrimaryEndpoint) { // Ingestion receives through its own transport infrastructure and forwards through - // that same infrastructure's dispatcher, so the endpoint is not hosted at all. + // that same infrastructure's dispatcher, so the primary endpoint is not hosted at all. var machineName = NServiceBus.Support.RuntimeEnvironment.MachineName; services.AddSingleton(new HostInformation( DeterministicGuid.MakeId(machineName, settings.InstanceName), @@ -122,6 +123,11 @@ public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, S provider.GetRequiredService().StopApplication(); return Task.CompletedTask; })); + + if (settings.Host.ReportsToPrimary) + { + ReportingEndpoint.Add(services, settings, transportCustomization, transportSettings); + } } else { @@ -202,6 +208,8 @@ Audit Retention Period (optional): {settings.AuditRetentionPeriod} Ingest Error Messages: {settings.IngestErrorMessages} Error Ingestion Only: {settings.ErrorIngestionOnly} Audit Ingestion Only: {settings.AuditIngestionOnly} +Audit Instance: {settings.AuditInstance} +Audit Data Location: {settings.AuditDataLocation} Forwarding Error Messages: {settings.ForwardErrorMessages} ServiceControl Logging Level: {settings.LoggingSettings.LogLevel} Selected Transport Customization: {settings.TransportType} diff --git a/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs index eb86a3e7c5..1ccb16cee6 100644 --- a/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs +++ b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs @@ -7,6 +7,7 @@ namespace ServiceControl.Hosting.Commands using Particular.ServiceControl; using Particular.ServiceControl.Hosting; using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Persistence; using ServiceControl.Auditing; using ServiceControl.Infrastructure.Health; using ServiceControl.Monitoring; @@ -26,6 +27,8 @@ public override async Task Execute(HostArguments args, Settings settings, Cancel var app = BuildHost(settings); + await app.Services.EnsureDatabaseSchemaIsCurrent(cancellationToken); + await app.RunAsync(settings.RootUrl); } @@ -33,6 +36,7 @@ internal static WebApplication BuildHost(Settings settings, Action + /// The owner of a dedicated audit database: ingests the audit queue, serves the API the primary's + /// scatter gather calls, sweeps audit retention, and reports its custom checks and the endpoints + /// it detects to the primary named by ServiceControlQueueAddress. No error side at all, and no + /// primary NServiceBus endpoint, only a send only one for the reporting. + /// + class AuditInstanceCommand : AbstractCommand + { + public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) + { + AuditInstanceGuards.EnsureCanRun(settings); + + var app = BuildHost(settings, hostBuilder => + { + hostBuilder.AddServiceControlAuthentication(settings.OpenIdConnectSettings); + hostBuilder.AddServiceControlAuthorization(settings.OpenIdConnectSettings); + hostBuilder.AddServiceControlHttps(settings.HttpsSettings); + }); + + app.UseServiceControl(settings.ForwardedHeadersSettings, settings.HttpsSettings); + app.UseServiceControlAuthentication(settings.OpenIdConnectSettings.Enabled); + + await app.RunAsync(settings.RootUrl); + } + + internal static WebApplication BuildHost(Settings settings, Action customize = null) + { + ApplyMode(settings); + + var hostBuilder = WebApplication.CreateBuilder(); + + customize?.Invoke(hostBuilder); + + hostBuilder.AddServiceControl(settings, configuration: null, Components); + hostBuilder.AddServiceControlApi(settings.CorsSettings); + + return hostBuilder.Build(); + } + + // Shared with setup, so that provisioning and running agree on what this host is. + internal static void ApplyMode(Settings settings) + { + settings.AuditInstance = true; + settings.AuditDataLocation = AuditDataLocation.Local; + settings.IngestAuditMessages = true; + settings.IngestErrorMessages = false; + settings.RunRetryProcessor = false; + settings.DisableExternalIntegrationsPublishing = true; + } + + // Heartbeat monitoring warms the endpoint monitor the audit enricher asks, custom checks + // report this host's ingestion health, and the audit component is the reason the host exists. + internal static ServiceControlComponent[] Components => + [ + new HeartbeatMonitoringComponent(), + new CustomChecks.CustomChecksComponent(), + new AuditComponent() + ]; + } +} diff --git a/src/ServiceControl/Hosting/Commands/AuditInstanceGuards.cs b/src/ServiceControl/Hosting/Commands/AuditInstanceGuards.cs new file mode 100644 index 0000000000..e688661cbb --- /dev/null +++ b/src/ServiceControl/Hosting/Commands/AuditInstanceGuards.cs @@ -0,0 +1,45 @@ +namespace ServiceControl.Hosting.Commands +{ + using System; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Persistence; + + static class AuditInstanceGuards + { + public static void EnsureCanRun(Settings settings) + { + var manifest = PersistenceManifestLibrary.Find(settings.PersistenceType); + + if (manifest?.SupportsAuditIngestion != true) + { + throw new Exception( + $"--audit-instance requires storage that supports audit ingestion, but this instance is configured to use '{settings.PersistenceType}'. " + + "A dedicated audit database is only supported on SQL Server and PostgreSQL storage."); + } + + if (string.IsNullOrWhiteSpace(settings.ServiceControlQueueAddress)) + { + throw new Exception( + "--audit-instance requires ServiceControl/ServiceControlQueueAddress, the primary instance's input queue, " + + "so that this host can report its custom checks and the endpoints it detects to the primary."); + } + + if (settings.RemoteInstances.Length > 0) + { + throw new Exception( + "--audit-instance cannot have remote instances configured. An audit host is a leaf: the primary lists it under " + + "ServiceControl/RemoteInstances, not the other way round."); + } + } + + public static void EnsureNotCombinedWithIngestionOnly(bool auditInstance, bool ingestionOnly) + { + if (auditInstance && ingestionOnly) + { + throw new Exception( + "--audit-instance cannot be combined with --error-ingestion-only or --audit-ingestion-only. " + + "An audit host owns its database; a worker that scales its ingestion is started with --audit-ingestion-only alone."); + } + } + } +} diff --git a/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs index 02f3ad3824..38a88749db 100644 --- a/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs +++ b/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs @@ -33,6 +33,8 @@ public override async Task Execute(HostArguments args, Settings settings, Cancel var app = BuildHost(settings); + await app.Services.EnsureDatabaseSchemaIsCurrent(cancellationToken); + await app.RunAsync(settings.RootUrl); } diff --git a/src/ServiceControl/Hosting/Commands/SetupCommand.cs b/src/ServiceControl/Hosting/Commands/SetupCommand.cs index a719eaa8eb..17201d036f 100644 --- a/src/ServiceControl/Hosting/Commands/SetupCommand.cs +++ b/src/ServiceControl/Hosting/Commands/SetupCommand.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Hosting.Commands +namespace ServiceControl.Hosting.Commands { using System.Runtime.InteropServices; using System.Threading; @@ -18,12 +18,17 @@ class SetupCommand : AbstractCommand { public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) { + if (args.AuditInstance) + { + AuditInstanceCommand.ApplyMode(settings); + } + var hostBuilder = Host.CreateApplicationBuilder(); hostBuilder.AddServiceControlInstallers(settings); var componentSetupContext = new ComponentInstallationContext(); - foreach (ServiceControlComponent component in ServiceControlMainInstance.Components) + foreach (ServiceControlComponent component in args.AuditInstance ? AuditInstanceCommand.Components : ServiceControlMainInstance.Components) { component.Setup(settings, componentSetupContext, hostBuilder); } diff --git a/src/ServiceControl/Hosting/Help.txt b/src/ServiceControl/Hosting/Help.txt index 4c9530d956..ecaceff110 100644 --- a/src/ServiceControl/Hosting/Help.txt +++ b/src/ServiceControl/Hosting/Help.txt @@ -34,6 +34,23 @@ database has already been provisioned by a normal instance. It cannot be combine The same body storage rule applies as for error ingestion only. +AUDIT INSTANCE + + ServiceControl.exe --audit-instance + ServiceControl.exe --audit-instance --setup + +Runs the owner of a dedicated audit database: it drains the audit queue into its own database, +serves the audit routes the primary instance calls through ServiceControl/RemoteInstances, and +sweeps audit retention. With --setup it provisions that database, the audit queue and body storage. +Requires SQL Server or PostgreSQL storage and ServiceControl/ServiceControlQueueAddress, the +primary's input queue, to which it reports its custom checks and the endpoints it detects. The +primary is configured with ServiceControl/AuditDataLocation set to Remote and lists this host under +ServiceControl/RemoteInstances. + +Audit ingestion can be scaled out for a dedicated audit database too: start --audit-ingestion-only +workers with the audit database's connection string and ServiceControl/ServiceControlQueueAddress +set, so they report to the primary the same way. + SERVICE INSTALL AND UNINSTALL AND CONFIGURATION OPTIONS As of Service Control 1.7 the command line uninstall and install switches have been removed. diff --git a/src/ServiceControl/Hosting/HostArguments.cs b/src/ServiceControl/Hosting/HostArguments.cs index 6c4a0a9955..64f8d387c1 100644 --- a/src/ServiceControl/Hosting/HostArguments.cs +++ b/src/ServiceControl/Hosting/HostArguments.cs @@ -1,4 +1,4 @@ -namespace Particular.ServiceControl.Hosting +namespace Particular.ServiceControl.Hosting { using System; using System.IO; @@ -13,6 +13,7 @@ public HostArguments(string[] args) { var errorIngestionOnly = false; var auditIngestionOnly = false; + var auditInstance = false; if (SettingsReader.Read(Settings.SettingsRootNamespace, "MaintenanceMode")) { @@ -56,6 +57,15 @@ public HostArguments(string[] args) } }; + var auditInstanceOptions = new OptionSet + { + { + "audit-instance", + "Run as the audit host of a dedicated audit database, reached by a primary through its remote instances", + s => auditInstance = true + } + }; + var ingestionOnlyOptions = new OptionSet { { @@ -72,6 +82,9 @@ public HostArguments(string[] args) try { + auditInstanceOptions.Parse(args); + AuditInstance = auditInstance; + externalInstallerOptions.Parse(args); if (Command == typeof(SetupCommand)) @@ -96,6 +109,13 @@ public HostArguments(string[] args) ingestionOnlyOptions.Parse(args); IngestionOnlyGuards.EnsureModesAreNotCombined(errorIngestionOnly, auditIngestionOnly); + AuditInstanceGuards.EnsureNotCombinedWithIngestionOnly(auditInstance, errorIngestionOnly || auditIngestionOnly); + + if (auditInstance) + { + Command = typeof(AuditInstanceCommand); + return; + } if (errorIngestionOnly) { @@ -124,6 +144,12 @@ public HostArguments(string[] args) public bool SkipQueueCreation { get; private set; } + /// + /// Set by --audit-instance. Read by setup as well as by the run command, so one flag provisions + /// and runs the same host. + /// + public bool AuditInstance { get; private set; } + public void PrintUsage() { var helpText = string.Empty; diff --git a/src/ServiceControl/Infrastructure/Settings/AuditDataLocation.cs b/src/ServiceControl/Infrastructure/Settings/AuditDataLocation.cs new file mode 100644 index 0000000000..57937b2b6a --- /dev/null +++ b/src/ServiceControl/Infrastructure/Settings/AuditDataLocation.cs @@ -0,0 +1,15 @@ +namespace ServiceBus.Management.Infrastructure.Settings +{ + /// + /// Where a primary's audit data lives. Local is the shared database, the default wherever the + /// persister supports audit. Remote means a dedicated audit database served by an audit host + /// listed under RemoteInstances, so this primary neither ingests audit nor queries its own audit + /// tables. Not derived from the other settings, because the shared topology where only workers + /// ingest looks identical to Remote from the primary's side. + /// + public enum AuditDataLocation + { + Local, + Remote + } +} diff --git a/src/ServiceControl/Infrastructure/Settings/HostProfile.cs b/src/ServiceControl/Infrastructure/Settings/HostProfile.cs new file mode 100644 index 0000000000..0f66b0871d --- /dev/null +++ b/src/ServiceControl/Infrastructure/Settings/HostProfile.cs @@ -0,0 +1,21 @@ +namespace ServiceBus.Management.Infrastructure.Settings +{ + /// + /// What this process is responsible for, derived once from the command it was started with. + /// Components ask for a capability rather than for the mode, so adding a mode is a change here + /// and not in every component. + /// + /// Serves the HTTP API and the platform connection details behind it. + /// Runs the primary NServiceBus endpoint on the instance queue. + /// Runs what a deployment may only run once: retries, integration event dispatch, licensing, notifications. + /// Sweeps the database it is configured against. + /// Checks endpoint heartbeats, rather than only warming the endpoint monitor for ingestion. + /// Sends custom check results and detected endpoints to another primary's queue instead of storing them. + public sealed record HostProfile( + bool HostsApi, + bool HostsPrimaryEndpoint, + bool OwnsSingletonWork, + bool OwnsRetention, + bool MonitorsHeartbeats, + bool ReportsToPrimary); +} diff --git a/src/ServiceControl/Infrastructure/Settings/Settings.cs b/src/ServiceControl/Infrastructure/Settings/Settings.cs index dcac86fe21..3e50152f27 100644 --- a/src/ServiceControl/Infrastructure/Settings/Settings.cs +++ b/src/ServiceControl/Infrastructure/Settings/Settings.cs @@ -1,4 +1,4 @@ -namespace ServiceBus.Management.Infrastructure.Settings +namespace ServiceBus.Management.Infrastructure.Settings { using System; using System.Collections.Generic; @@ -221,6 +221,9 @@ public string InstanceId // Set by the --audit-ingestion-only command, never read from configuration. public bool AuditIngestionOnly { get; set; } + // Set by the --audit-instance command, never read from configuration. + public bool AuditInstance { get; set; } + /// /// True in either ingestion only mode. These hosts run no NServiceBus endpoint, own none of the /// work a deployment may only do once, and never provision anything. @@ -228,6 +231,26 @@ public string InstanceId [JsonIgnore] public bool IngestionOnly => ErrorIngestionOnly || AuditIngestionOnly; + [JsonIgnore] + public HostProfile Host => new( + HostsApi: !IngestionOnly, + HostsPrimaryEndpoint: !IngestionOnly && !AuditInstance, + OwnsSingletonWork: !IngestionOnly && !AuditInstance, + OwnsRetention: !IngestionOnly, + MonitorsHeartbeats: !IngestionOnly && !AuditInstance, + ReportsToPrimary: (IngestionOnly || AuditInstance) && !string.IsNullOrWhiteSpace(ServiceControlQueueAddress)); + + /// + /// Whether this primary's audit data is in its own database or on a dedicated audit host. + /// + public AuditDataLocation AuditDataLocation { get; set; } + + /// + /// The primary's input queue, for a host on a dedicated audit database to report custom checks + /// and detected endpoints to. The same key the standalone audit instance reads. + /// + public string ServiceControlQueueAddress { get; set; } + public TimeSpan? AuditRetentionPeriod { get; set; } public TimeSpan ErrorRetentionPeriod { get; } @@ -510,6 +533,33 @@ void LoadAuditIngestionSettings() } ForwardAuditMessages = SettingsReader.Read(SettingsRootNamespace, "ForwardAuditMessages", false); + AuditDataLocation = ReadAuditDataLocation(); + ServiceControlQueueAddress = SettingsReader.Read(SettingsRootNamespace, "ServiceControlQueueAddress"); + } + + AuditDataLocation ReadAuditDataLocation() + { + var value = SettingsReader.Read(SettingsRootNamespace, "AuditDataLocation"); + + if (string.IsNullOrWhiteSpace(value)) + { + return AuditDataLocation.Local; + } + + if (Enum.TryParse(value, ignoreCase: true, out var location)) + { + return location; + } + + var message = $"ServiceControl/AuditDataLocation is '{value}', expected Local or Remote."; + + if (ValidateConfiguration) + { + throw new Exception(message); + } + + logger.LogWarning("{Message} Assuming Local.", message); + return AuditDataLocation.Local; } void LoadErrorIngestionSettings() diff --git a/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs b/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs index 7f15781d4e..9b2b5b6283 100644 --- a/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs +++ b/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Monitoring +namespace ServiceControl.Monitoring { using Connection; using Contracts; @@ -31,7 +31,7 @@ public override void Configure(Settings settings, ITransportCustomization transp { hostBuilder.Services.AddHostedService(); - if (!settings.IngestionOnly) + if (settings.Host.MonitorsHeartbeats) { hostBuilder.Services.AddHostedService(); } @@ -52,7 +52,7 @@ public override void Configure(Settings settings, ITransportCustomization transp hostBuilder.Services.AddErrorMessageEnricher(); - if (!settings.IngestionOnly) + if (settings.Host.MonitorsHeartbeats) { hostBuilder.Services.AddPlatformConnectionProvider(); } diff --git a/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs b/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs index 35efea4e21..6fc1af6436 100644 --- a/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs +++ b/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Monitoring +namespace ServiceControl.Monitoring { using System; using System.Threading; @@ -24,10 +24,10 @@ public async Task StartAsync(CancellationToken cancellationToken = default) { await persistence.WarmupMonitoringFromPersistence(monitor, cancellationToken); - // An ingestion only host receives no heartbeats, so it has nothing to check and would - // only report every endpoint as dead. It still warms the monitor, because the error - // enricher asks it whether an endpoint is new before recording it. - if (settings.IngestionOnly) + // A host that does not monitor heartbeats receives none, so it has nothing to check and + // would only report every endpoint as dead. It still warms the monitor, because the + // ingestion enrichers ask it whether an endpoint is new before recording it. + if (!settings.Host.MonitorsHeartbeats) { return; } diff --git a/src/ServiceControl/Persistence/PersistenceFactory.cs b/src/ServiceControl/Persistence/PersistenceFactory.cs index bad7018f70..6302cadd0a 100644 --- a/src/ServiceControl/Persistence/PersistenceFactory.cs +++ b/src/ServiceControl/Persistence/PersistenceFactory.cs @@ -18,7 +18,8 @@ public static IPersistence Create(Settings settings, bool maintenanceMode = fals //HINT: This is false when executed from acceptance tests settings.PersisterSpecificSettings ??= persistenceConfiguration.CreateSettings(Settings.SettingsRootNamespace); settings.PersisterSpecificSettings.MaintenanceMode = maintenanceMode; - settings.PersisterSpecificSettings.RunRetentionSweep = !settings.IngestionOnly; + settings.PersisterSpecificSettings.RunRetentionSweep = settings.Host.OwnsRetention; + settings.PersisterSpecificSettings.HostsAuditData = settings.AuditDataLocation == AuditDataLocation.Local; var persistence = persistenceConfiguration.Create(settings.PersisterSpecificSettings); return persistence; diff --git a/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs b/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs index 7918f73cb3..db34322f3a 100644 --- a/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs +++ b/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs @@ -12,6 +12,16 @@ public static void AddPersistence(this IServiceCollection services, Settings set var persistence = PersistenceFactory.Create(settings, maintenanceMode); persistence.AddPersistence(services); + if (settings.AuditDataLocation == AuditDataLocation.Remote) + { + // The audit data is on a dedicated host, so the local sources stand aside and the + // audit routes answer from the configured remotes alone, with this instance as a + // non-participant rather than an empty one. + services.AddSingleton(); + services.AddSingleton(); + return; + } + // Only an audit capable persister registers these, so the rest fall back to a local source // that holds nothing and the audit routes answer from the configured remotes alone. services.TryAddSingleton(); diff --git a/src/ServiceControl/Recoverability/RecoverabilityComponent.cs b/src/ServiceControl/Recoverability/RecoverabilityComponent.cs index d3ab20dfa4..95b766c9af 100644 --- a/src/ServiceControl/Recoverability/RecoverabilityComponent.cs +++ b/src/ServiceControl/Recoverability/RecoverabilityComponent.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Recoverability +namespace ServiceControl.Recoverability { using System; using System.Threading; @@ -81,7 +81,7 @@ public override void Configure(Settings settings, ITransportCustomization transp services.AddSingleton(); services.AddSingleton(); - if (!settings.IngestionOnly) + if (settings.Host.OwnsSingletonWork) { services.AddHostedService(provider => provider.GetRequiredService()); } diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 38f1d12685..074f2121be 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -165,8 +165,8 @@ Two consequences follow from owners being upgraded independently: host one migration behind the primary. - Workers have no schema check. A worker started before its database was migrated fails on the first insert rather than at startup, which is today's behaviour for error workers as well. Workers gain a - startup probe that reads the migrations history table and refuses to start, with a message naming - `--setup` on the owner, when the migration the binary was built against is not applied. + startup probe, `IDatabaseSchemaProbe`, that reads the migrations history table and refuses to + start, with a message naming setup on the owner, when a migration the binary carries is not applied. ### The audit host @@ -181,13 +181,13 @@ and `CustomChecksComponent` in reporting mode, see below. The full persister is `RunRetentionSweep` true. `--setup` in this mode provisions the audit queue, migrates the audit database and provisions body storage, and does not touch the primary's queues. -The API surface is only the routes the primary calls on a remote, plus health. Taken from the code -that calls them: `/api` (`CheckRemotes`), `/api/configuration` (`ConfigurationApi` and licensing), -`/api/connection` (`RemotePlatformConnectionDetailsProvider`), the five message views, -`/api/messages/{id}/body` (forwarded by instance id from `GetMessagesController`), `/api/sagas/{id}` -and `/api/endpoints/{name}/audit-count`. They are registered through an -`IApplicationFeatureProvider` allow list, so a browser or ServicePulse pointed at -the audit host by mistake gets 404 rather than an empty error instance. +The audit host serves the primary API as it is. The routes the primary calls on a remote are `/api` +(`CheckRemotes`), `/api/configuration` (`ConfigurationApi` and licensing), `/api/connection` +(`RemotePlatformConnectionDetailsProvider`), the five message views, `/api/messages/{id}/body` +(forwarded by instance id from `GetMessagesController`), `/api/sagas/{id}` and +`/api/endpoints/{name}/audit-count`. The rest of the API answers with the host's own empty error +data, which is accepted: the API exists for the scatter-gather, not as a second public surface, and +an allow list was judged not worth its upkeep (John, 14 Sep 2026). Authorization: the primary forwards the caller's `Authorization` header to remotes, so the audit host runs the same authorization configuration and the same `error:*` policies as the primary. That is how @@ -196,7 +196,13 @@ two processes must be configured alike. Startup guards: the persister must support audit; `ServiceControlQueueAddress` must be set; `RemoteInstances` must be empty, because an audit host is a leaf; and the mode cannot be combined -with either ingestion-only flag. +with either ingestion-only flag. `--setup --audit-instance` provisions the audit host through the +same flag, with the audit host's component list rather than the primary's. + +Components do not branch on the mode. `Settings.Host` is a profile derived once from the flags +(`HostsApi`, `HostsPrimaryEndpoint`, `OwnsSingletonWork`, `OwnsRetention`, `MonitorsHeartbeats`, +`ReportsToPrimary`) and each component asks for the capability it needs. The audit host is the +first mode that is neither a primary nor a worker, which is what made the profile worth having. ### Reporting back to the primary @@ -212,8 +218,12 @@ and `RegisterNewEndpoint` sent to the primary's queue, which `ReportCustomCheckR instance's own key name, names that queue. The copied audit runtime has no NServiceBus endpoint, only `IMessageDispatcher`. Hosts on the audit -database get a send-only NServiceBus endpoint for these two messages. Send-only claims no queue, so -the hosting plan's rule that ingestion-only hosts own no queue holds. The presence of +database get a send-only NServiceBus endpoint for these two messages, built with the transport +customization's audit endpoint profile, which is already send-only with publishing disabled. Custom +check results go through an `ICustomCheckResultReporter` seam, local storage by default and the +primary's queue on a reporting host; detected endpoints through `IEndpointDetectionReporter` the +same way. Send-only claims no queue, so the hosting plan's rule that ingestion-only hosts own no +queue holds. The presence of `ServiceControlQueueAddress` is what switches a host from writing custom checks and endpoint registrations locally to sending them: required on `--audit-instance`, set on an `--audit-ingestion-only` worker only when it feeds a dedicated audit database, and never set on a @@ -279,9 +289,7 @@ project. The rules, so that the segregation survives the steps below: configurations named `Audit*` and `SagaSnapshot*` beside the existing ones. The error side is not moved. - **API.** The message, saga and audit count routes serve both pipelines by product design, so the - API is not split. The subset the audit host exposes is declared on the controllers with a marker - attribute that the step 7 allow list reads, so the boundary is visible in code rather than in a - list inside a command. + API is not split, and the audit host serves it whole. - **Host modes.** Components do not branch on the mode name. The hosting command builds a host profile once and components ask it for capabilities. Today's `settings.IngestionOnly` checks are converted when step 7 adds the third mode and would otherwise multiply them. @@ -646,10 +654,12 @@ competing consumers against one database, and the primary is the only writer to 7. **Dedicated audit database.** `--audit-instance` and its guards, the controller allow list, `AuditDataLocation` and the primary's Remote behaviour, `ServiceControlQueueAddress` on the primary executable with the send-only endpoint and the reporting switch, and the persister - setting that gates the audit retention pass, and the workers' migration probe. Acceptance test: - a primary and an audit host against two databases (two schemas in the test harness), audit - messages ingested by the host and read through the primary, a custom check and a detected - endpoint raised on the host and visible on the primary. + setting that gates the audit retention pass, and the workers' migration probe. On + `john/audit_ef_8`. Its composition tests build and start the audit host after provisioning it + with `--setup --audit-instance`, put the primary in Remote mode, and give a worker the primary's + queue. The end to end test of the split topology, two hosts against two databases with audit + read through the primary and a custom check and a detected endpoint arriving on it, is still + to be written; it needs a second in-process host in the acceptance runner. 8. **Documentation.** `docs/audit-ingestion-in-the-primary.md` gains the topology tables, the new mode and the two settings, and the hosting plan's statement that there is no separate audit HTTP service is marked superseded. From 3279a7edef115ed2a9121d6a76ab6070b2ca382e Mon Sep 17 00:00:00 2001 From: John Simons Date: Mon, 14 Sep 2026 15:34:54 +1000 Subject: [PATCH 21/21] Document the audit topologies of the relational persisters The shipped hosting document gains the shared and dedicated audit database topologies with the processes each runs, the AuditDataLocation and ServiceControlQueueAddress settings, the seven day retention default, the hourly retention model and its lock, and the database-side message view with its precedence and capped total. The hosting plan's statement that there is no separate audit HTTP service is marked superseded. --- docs/audit-ingestion-in-the-primary.md | 121 +++++++++++++++++++------ src/audit-ef-persistence-plan.md | 2 +- src/audit-ingestion-in-primary-plan.md | 2 +- 3 files changed, 96 insertions(+), 29 deletions(-) diff --git a/docs/audit-ingestion-in-the-primary.md b/docs/audit-ingestion-in-the-primary.md index e895ee8e8b..a5f7bafd4a 100644 --- a/docs/audit-ingestion-in-the-primary.md +++ b/docs/audit-ingestion-in-the-primary.md @@ -13,16 +13,53 @@ The SQL Server and PostgreSQL persisters advertise audit support, so a primary o the audit queue by default. On RavenDB the audit component registers nothing and behavior is unchanged. -## Deployment modes +## Topologies -| Mode | How | What it runs | +Every process is the same executable and the same persister. What differs is the database each +process is given and the mode it is started in. Each database has exactly one owner, and only owners +run setup, retention and the API; every other process on that database is a worker. + +### Shared database + +The default. Audit data lives in the primary's database. + +| Process | Started as | What it runs | | --- | --- | --- | -| Normal primary, audit ingestion on | Default where the persister advertises audit support | The audit receiver, the audit capabilities, the primary API and everything a normal primary runs | -| Normal primary, audit ingestion off | `ServiceControl/IngestAuditMessages=false` | Everything above except the audit receiver. Local audit queries, failed audit tooling and `/api/connection` stay active, because other processes may still be ingesting | -| Audit ingestion only | `ServiceControl.exe --audit-ingestion-only` | The audit receiver, the endpoint monitor it depends on, this node's custom checks, and the health endpoints. No NServiceBus endpoint, no API, no retention, no licensing | +| Primary | `ServiceControl.exe` | The audit receiver (unless `ServiceControl/IngestAuditMessages=false`), the audit capabilities, the local audit queries, audit retention, and everything a normal primary runs | +| Audit worker | `ServiceControl.exe --audit-ingestion-only` | The audit receiver, the endpoint monitor it depends on, this node's custom checks, and the health endpoints. No NServiceBus endpoint, no API, no retention, no licensing | +| Error worker | `ServiceControl.exe --error-ingestion-only` | As before | + +Turning the primary's receiver off stops only the receiver. Local audit queries, failed audit +tooling and `/api/connection` stay active, because workers may still be ingesting. + +### Dedicated audit database + +For a deployment whose database copes with the error instance but not with audit volume. Audit moves +to its own database, on the same server or another, and nothing else about the deployment changes. + +| Process | Started as | What it runs | +| --- | --- | --- | +| Primary | `ServiceControl.exe` with `ServiceControl/AuditDataLocation=Remote` and the audit host listed under `ServiceControl/RemoteInstances` | Everything a normal primary runs, minus the audit receiver, the local audit queries and audit retention. Audit data reaches it through the scatter gather, exactly as from a RavenDB audit instance | +| Audit host | `ServiceControl.exe --audit-instance`, with the audit database's connection string and `ServiceControl/ServiceControlQueueAddress` | The audit receiver, the primary API, audit retention, saga audit, failed audit tooling and `/api/connection`. No error side. `--setup --audit-instance` provisions the audit database, the audit queue and body storage | +| Audit worker | `ServiceControl.exe --audit-ingestion-only`, with the audit database's connection string and `ServiceControl/ServiceControlQueueAddress` | As in the shared topology. Which database it feeds is the connection string it is given | +| Error worker | `ServiceControl.exe --error-ingestion-only` | As before. Error ingestion only ever shares the primary's database | + +The audit host and its workers report their custom checks and the endpoints they detect to the +primary's input queue, the way the standalone RavenDB audit instance does, because the primary is the +only process ServicePulse asks. `ServiceControl/ServiceControlQueueAddress` is what switches a +process into that reporting mode; leave it unset in the shared topology. -`--audit-ingestion-only` and `--error-ingestion-only` cannot be combined. Each queue gets its own -worker pool so the two can be scaled independently, so run one process per mode. +The audit host serves the whole primary API, of which the primary's scatter gather calls the +message, saga, audit count, body, configuration and connection routes. The rest answers with the +host's own empty error data. It runs the same authorization configuration as the primary, because the +primary forwards the caller's credentials to it. + +`--audit-instance` cannot be combined with either ingestion only flag, and `--audit-ingestion-only` +cannot be combined with `--error-ingestion-only`: run one process per mode. + +### The RavenDB topology + +Unchanged. `ServiceControl.Audit.exe` with its own database, listed as a remote. All three modes keep audit data in the primary's own database. A customer whose audit load would swamp that database can instead move audit to a dedicated one, served by the same executable in @@ -40,10 +77,11 @@ capable primary is configured exactly the way an audit instance is configured to | `ServiceBus/AuditQueue` | `audit` | The queue this instance drains | | `ServiceBus/AuditLogQueue` | the subscoped audit queue name | Only used when forwarding is on | | `ServiceControl/ForwardAuditMessages` | `false` | | -| `ServiceControl/AuditRetentionPeriod` | unset | Already existed. Validated between 1 hour and 365 days | +| `ServiceControl/AuditRetentionPeriod` | 7 days | Already existed. Validated between 1 hour and 365 days. The SQL Server and PostgreSQL persisters default it to 7 days when unset, matching the management utility and the container image rather than the audit instance's 30 | +| `ServiceControl/AuditDataLocation` | `Local` | `Remote` tells a primary its audit data is on a dedicated audit host. Ignored where the persister does not support audit | +| `ServiceControl/ServiceControlQueueAddress` | unset | The primary's input queue. Set on the audit host and on workers that feed a dedicated audit database, which then report custom checks and detected endpoints there. The same key the standalone audit instance reads | | `ServiceControl/MaximumAuditIngestionConcurrencyLevel` | `32` | Independent of the primary endpoint's concurrency, which is what `MaximumConcurrencyLevel` sets | | `ServiceControl/TimeToRestartAuditIngestionAfterFailure` | 60 seconds | Mirrors the error equivalent | -| `ServiceControl/OtlpEndpointUrl` | unset | Enables the OpenTelemetry metrics exporter | | `ServiceControl/MessageBody/FileSystem/PathIsShared` | `false` | Required by both ingestion only modes when body storage is the file system | ### Setting collisions @@ -53,15 +91,21 @@ name, and `ServiceBus/AuditQueue` is literally the same key for both processes. and a standalone audit instance sharing one environment file therefore collide on `INGESTAUDITMESSAGES`, `AUDITRETENTIONPERIOD`, `FORWARDAUDITMESSAGES` and `SERVICEBUS_AUDITQUEUE`. -That combination is unsupported. The primary logs a warning at startup when it has audit ingestion -enabled and audit remotes configured at the same time, because that is the shape most likely to hit -the collision. +That combination is unsupported. A primary that ingests audit into its own database and also lists +remote instances refuses to start, because it is one of two mistakes: remotes left over from before +audit moved into the database, or a primary that was meant to have `AuditDataLocation=Remote`. ## Queue ownership -The setup path creates the audit queue, and the audit forwarding queue when forwarding is enabled. -Ingestion only workers run no installers: they never create queues, never apply database migrations -and never provision body storage. Run setup from a normal instance before starting any worker. +The setup path of a database's owner creates the audit queue, and the audit forwarding queue when +forwarding is enabled. Ingestion only workers run no installers: they never create queues, never +apply database migrations and never provision body storage. Run setup from the owner before starting +any worker; a worker started against a database the owner has not migrated refuses to start and says +so, rather than failing on its first write. + +The primary and a dedicated audit database run the same migrations, so the owners are upgraded +independently and in either order. The audit database carries the error tables, empty, and a primary +whose audit is remote carries the audit tables, empty. Transport operations remain in the audit ingestion path for two reasons only: @@ -71,15 +115,20 @@ Transport operations remain in the audit ingestion path for two reasons only: database. In a combined host it is dispatched to the local error queue and comes straight back in through local error ingestion, which is exactly what happens today. -Endpoints detected from audit headers are written straight to the shared `KnownEndpoints` table -through the ingestion unit of work, rather than sent to the primary's input queue. +Endpoints detected from audit headers are written straight to the `KnownEndpoints` table of the +database being ingested into, through the ingestion unit of work. On a dedicated audit database they +are additionally reported to the primary's input queue, since that table is not the primary's. ## Body storage Audit and failed message bodies share one store, and each owns a prefixed keyspace, so an edited -message's failed body and its audited body do not collide. `IBodyStorage.TryFetch` resolves in a -fixed order: failed message by `UniqueMessageId`, then failed message by `MessageId`, then audit -message by `UniqueMessageId`. +message's failed body and its audited body do not collide. Audit bodies are keyed +`audit/{ingestion hour}/{unique message id}`, which is what lets retention drop an hour's bodies in +one operation. `IBodyStorage.TryFetch` resolves in a fixed order: failed message by +`UniqueMessageId`, then failed message by `MessageId`, then audit message by `UniqueMessageId`. + +Hosts on one database share one body store: the audit host and its workers share the audit store, +the primary and its workers share the primary's. Every ingesting process must write bodies somewhere every host can read. Blob and S3 storage qualify. File system storage qualifies only if the path is a shared mount, which nothing in the @@ -95,6 +144,18 @@ Both ingestion only hosts map the same two routes, anonymously, returning JSON: - `/health/ready` additionally reports whether the ingestion this host exists to do is happening. An audit ingestion only host answers for `audit-ingestion` and not for `error-ingestion`. +## Retention + +Audit rows are stored by the hour they were ingested in and expire an hour at a time, once the whole +hour is behind `AuditRetentionPeriod`. On PostgreSQL the audit tables are range partitioned by that +hour, so dropping an expired hour is a metadata operation, and the retention sweep keeps partitions +provisioned 48 hours ahead; a custom check, `Audit partition provisioning`, fails when the newest +provisioned partition ends less than 12 hours ahead, which means the sweeper has stopped. On SQL +Server, which does not partition, an expired hour is deleted in batches. + +The retention sweep, for audit and error data alike, runs under a session scoped database lock, so at +most one host sweeps a database at any moment. A host that cannot take the lock skips the pass. + ## Querying Local audit data is served through the existing primary routes under their existing policies: @@ -106,12 +167,18 @@ so nothing about the `my/routes` manifest or ServicePulse navigation changes. Additional audit remotes keep working. The scatter gather runs the local query first and merges the remotes after, so a primary can hold audit data locally, query remotes, or both. -Where one local result set contains both failed and audited messages, three rules apply, and -`LocalMessagesView.Merge` implements them for any persister: +Where a database holds both failed and audited messages, each message view is one SQL statement +over both tables, with each branch carrying its own sort and limit so the database merges two +index-ordered scans and stops at the page boundary. Three rules apply: + +1. **Precedence.** A message that both failed and was audited shows as failed, whatever the failed + row's status. Archived failures show as archived, as they always have. +2. **Paging.** Pages are exact across both tables. +3. **Counting.** A message that both failed and was audited is counted once. The total is capped, + because an exact count is linear in the audit table; `Total-Count` and the paging links report + the cap when it is reached. -1. **Precedence.** A message that both failed and was audited shows as failed. -2. **Paging.** The local result is already at most one page, after deduplication. -3. **Counting.** A message that both failed and was audited is counted once. +A primary whose audit data is remote queries only its failed messages locally. ## Telemetry @@ -131,5 +198,5 @@ Anything set there wins, including `host.name` and `process.pid`, so a container ## Packaging The audit runtime ships inside the existing primary artifact. There is no new assembly and no new -deployment unit. The primary gains three OpenTelemetry package references, which the copied -ingestion metrics use, exported only when `OtlpEndpointUrl` is set. +deployment unit. The copied ingestion metrics share the primary's OpenTelemetry exporter, enabled by +the standard `OTEL_EXPORTER_OTLP_ENDPOINT` variable. diff --git a/src/audit-ef-persistence-plan.md b/src/audit-ef-persistence-plan.md index 074f2121be..2f513afb3f 100644 --- a/src/audit-ef-persistence-plan.md +++ b/src/audit-ef-persistence-plan.md @@ -662,7 +662,7 @@ competing consumers against one database, and the primary is the only writer to to be written; it needs a second in-process host in the acceptance runner. 8. **Documentation.** `docs/audit-ingestion-in-the-primary.md` gains the topology tables, the new mode and the two settings, and the hosting plan's statement that there is no separate audit HTTP - service is marked superseded. + service is marked superseded. On `john/audit_ef_9`. Each pull request leaves both EF acceptance suites and the RavenDB suites passing, and steps 1 to 5 leave `SupportsAuditIngestion` false so nothing activates early. Step 1 is on `john/audit_ef_1`, diff --git a/src/audit-ingestion-in-primary-plan.md b/src/audit-ingestion-in-primary-plan.md index a4b8bb5d5e..7b85962164 100644 --- a/src/audit-ingestion-in-primary-plan.md +++ b/src/audit-ingestion-in-primary-plan.md @@ -43,7 +43,7 @@ This plan covers the contracts, project boundaries, host composition, settings, ### Hosting -- The normal primary retains the existing HTTP routes and serves local audit data through them. There is no separate SQL Server or PostgreSQL audit HTTP service. *Superseded on 14 September 2026: audit load can swamp a database that copes with the error instance, so a customer must be able to move audit to a dedicated database. That database is served by the same executable in `--audit-instance` mode, and the topology is specified in the EF audit persistence plan.* +- The normal primary retains the existing HTTP routes and serves local audit data through them. There is no separate SQL Server or PostgreSQL audit HTTP service. *Superseded on 14 September 2026 by the "Topologies" section of [the EF audit persistence plan](audit-ef-persistence-plan.md): audit load can swamp a database that copes with the error instance, so a dedicated audit database is served by the same executable in `--audit-instance` mode.* - `--audit-ingestion-only` always ingests and does not host an NServiceBus endpoint. - `--audit-ingestion-only` and `--error-ingestion-only` are mutually exclusive. Passing both fails at startup with a clear message. Each queue gets its own worker pool so the two can be scaled independently, and each keeps a single, auditable component list. Combining them is a possible follow-up. - Disabling ingestion in the normal primary stops only its receiver. Local queries, SagaAudit, failed-import tooling, and other audit capabilities remain active because workers may still ingest.