[Migration Engine Part 1] Implement the foundation for migration engine by warwickschroeder · Pull Request #5894 · Particular/ServiceControl

warwickschroeder · 2026-09-15T06:36:55Z

What this adds

The foundation for moving an error instance's data from RavenDB to SQL Server or PostgreSQL. Nothing is copied yet, and the migration is not switched on anywhere.

A read-only RavenDB source. RavenMigrationSource opens both RavenDB databases, embedded or external, through a client that refuses anything outside a read allow-list (OnBeforeRequest, non-tracking sessions), so a patch, a bulk insert or a HiLo range throws in-process rather than reaching the database. It can describe itself and count every collection; Count, Read and ReadBody for a category all throw NotSupportedException, so no reader exists yet.
--migration-source-report. Prints the source's RavenDB version, where it is, both database names with the setting each came from, and a row count per collection. It reads the instance's own RavenDB settings; ServiceControl/Migration/SourcePersistenceType defaults to RavenDB, and Settings.MigrationSourcePersistenceType now appears in the approved platform sample settings.
Two persisters in one process. PersistenceFactory.CreateMigrationSource loads the source persister in its own AssemblyLoadContext, beside the configured one, and OpenMigrationSource opens it.
The migration engine, against fakes. Store-neutral seams in ServiceControl.Persistence/DataMigration (IMigrationSource, IMigrationTarget, IMigrationCheckpointStore, the category registry) and MigrationEngine, which copies a category in batches with checkpoint resume, ordering between categories, body read retries, throttling, skip reasons and a halt threshold. A checkpoint carries a Version and the target extends it inside the transaction that writes the rows, so a save that loses a race throws MigrationCheckpointConflictException and leaves the other writer's row standing rather than halting the category. There is no SQL target yet; it runs against in-memory fakes.
Docs and CI. docs/migration/ holds the overview, a system design diagram, and short instructions for what can be run today, linked from the README. .github/workflows/ci.yml gains a Migration test category, which also downloads the RavenDB server.

Tests

ServiceControl.UnitTests/Migration: engine, registry, options and the fakes.
ServiceControl.UnitTests/Hosting: --migration-source-report argument parsing.
ServiceControl.UnitTests/ApprovalFiles: the platform sample settings, which gained the new setting.
ServiceControl.Persistence.Tests.RavenDB/DataMigration: the read-only source lifecycle.
ServiceControl.Migration.Tests (new): the source report command, and both persisters loaded in one process.

…d ServiceControl migration

…n, clarifying strategies, goals, and migration limitations.

…on source reporting - Implemented MigrationEngineFailurePathTests to validate behavior during write failures and halts. - Created MigrationEngineHaltTests to ensure categories halt correctly on systemic failures. - Added MigrationEngineOptionsTests to verify default and environment variable configurations. - Developed MigrationEngineOrderingTests to check execution order of migration categories. - Introduced MigrationEngineResumeTests to confirm no duplicates or gaps after restarts. - Added MigrationEngineRunCategoriesTests to ensure all categories run in specified order. - Implemented MigrationEngineSkipReasonTests to validate skip reasons and their aggregation. - Enhanced MigrationEngineThrottleTests to verify pause behavior for optional categories. - Updated MigrationSourceReportCommand for improved output and clarity. - Modified Help.txt and HostArguments.cs for better command descriptions. - Refined Settings.cs and PersistenceFactory.cs for clearer migration source configuration.

…r SQL migration from RavenDB

rbev

Partial review - adding as comments for now.

rbev · 2026-09-15T08:25:55Z

+| `ServiceControl/RavenDB/ClientCertificatePath` or `ServiceControl/RavenDB/ClientCertificateBase64`, with `ServiceControl/RavenDB/ClientCertificatePassword` | `SERVICECONTROL_RAVENDB_CLIENTCERTIFICATEPATH` and so on | A secured external server's client certificate |
+| `ServiceControl/ErrorRetentionPeriod` | `SERVICECONTROL_ERRORRETENTIONPERIOD` | Required. Don't change it during the move |
+
+`ServiceControl/Migration/SourcePersistenceType` defaults to `RavenDB` and needs no setting.


So given the abstractions does this technically allow two way migration? or will that be missing the migrationsource implementation?

If this defaults to the only permissible value should it even be a configuration knob?

The abstraction is designed to allow for possible "any direction" migrations, but there will not be an implementation for SQL as the source or Raven as the target for this version.

Yep - I'll remove that setting as it is redundant at the moment. I did think about removing this..

rbev · 2026-09-15T09:08:40Z

+
+                var result = await target.Write(category, batchToWrite, checkpointAfterBatch, cancellationToken);
+
+                checkpoint = checkpointAfterBatch with


If the Write call writes checkpointAfterBatch as given (as described in the interface specification) then how do these extra counts that are being added on from the result survive a crash/reboot before the next batch commits?

Fixed in 909e4bf#diff-14df616bb97776e63067960a909ea5a788d33570699a00737b6053d4708b9f34.

I've added a sequence diagram to the overview doc to clearly show what the migration engine does and how the checkpoint/resume functionality works.

Have added more tests for all paths and edge cases of the checkpoint/resume logic

rbev · 2026-09-15T09:10:10Z

+                processedThisRun += bodySkips + result.Copied + result.Skipped + result.AlreadyPresent;
+                skippedThisRun += bodySkips + result.Skipped;
+
+                if (HaltThreshold.Exceeded(skippedThisRun, processedThisRun, options.HaltThresholdPercent, options.HaltThresholdMinimum))


Halt threshold is based on percent of progress rather than percent of total, does this cause a problem if there are a small number of invalid records at the start of the migration that would otherwise be ok?

A small number cannot halt anything, because the floor gates the percentage entirely. But a cluster of more than 100 bad rows near the start does halt a category, which is probably correct.

One bug I did notice though is that legitimate skips (past retention, orphaned group comments, etc) are being counted towards a halt. These types of skips should not count towards it. Will fix.

I've also added a section to the overview doc with a diagram to illustrate this.

Have also added some more tests to test all paths of the halting logic

johnsimons

A few question and comments but overall, lets keep going

johnsimons · 2026-09-16T23:48:11Z

+| `ServiceControl/RavenDB/ConnectionString` | `SERVICECONTROL_RAVENDB_CONNECTIONSTRING` | An external RavenDB server. Leave unset for an embedded database |
+| `ServiceControl/DbPath` | `SERVICECONTROL_DBPATH` | The embedded database's data directory |
+| `ServiceControl/RavenDB/DatabaseName` | `SERVICECONTROL_RAVENDB_DATABASENAME` | The primary database, `primary` by default |
+| `LicensingComponent/RavenDB/ThroughputDatabaseName` | `LICENSINGCOMPONENT_RAVENDB_THROUGHPUTDATABASENAME` | The throughput database, `throughput` by default |
+| `ServiceControl/RavenDB/ClientCertificatePath` or `ServiceControl/RavenDB/ClientCertificateBase64`, with `ServiceControl/RavenDB/ClientCertificatePassword` | `SERVICECONTROL_RAVENDB_CLIENTCERTIFICATEPATH` and so on | A secured external server's client certificate |
+| `ServiceControl/ErrorRetentionPeriod` | `SERVICECONTROL_ERRORRETENTIONPERIOD` | Required. Don't change it during the move |


So are these exclusively being used for the migration? And if so, should then be something like ServiceControl/Migration\....?

These just list the current settings for raven, which the migration engine uses to set its "Source", but yes, any new setting thats migration specific will be "ServiceControl/Migration/xxxxx"

johnsimons · 2026-09-17T00:01:56Z

+- The fifteen RavenDB index definitions, which map to a much smaller set of ordinary SQL indexes, and two of which are dead already
+- The transient in-flight collections, which are empty when nothing is running: `RetryBatches`, `RetryBatchNowForwardings`, `FailedMessageRetries`, `ArchiveOperations` and `UnarchiveOperations`
+- `ArchiveBatches` and `UnarchiveBatches`, which exist only because of how RavenDB works
+- `ConnectedApplications`, which only versions 6.0 and 6.1 wrote and nothing has read since


This is used for when the user is using the MassTransitConnector and this data is passed to ServicePulse to toggle features on/off

johnsimons · 2026-09-17T00:03:06Z

+- The transient in-flight collections, which are empty when nothing is running: `RetryBatches`, `RetryBatchNowForwardings`, `FailedMessageRetries`, `ArchiveOperations` and `UnarchiveOperations`
+- `ArchiveBatches` and `UnarchiveBatches`, which exist only because of how RavenDB works
+- `ConnectedApplications`, which only versions 6.0 and 6.1 wrote and nothing has read since
+- Integration events still waiting to be sent when you switch over are never sent


anyway to mitigate this?

johnsimons · 2026-09-17T00:04:09Z

+### Optional
+
+- Archived and resolved failed messages: the biggest category by far, and most of the copying time
+- The event log, without which the ServicePulse activity feed starts empty


I think I still see beneficial of bringing in at least the last 7 days or something like that

johnsimons · 2026-09-17T00:07:26Z

+
+- Archived and resolved failed messages: the biggest category by far, and most of the copying time
+- The event log, without which the ServicePulse activity feed starts empty
+- Custom checks, which cost almost nothing to skip because every check re-reports on its next interval


is this related to heartbeats?
I am asking becasue there is no mention of those?
One thing to watch out, is an instance is down, we will not get a heartbeat so we want to ensure that we somehow assume that everything is ok.

johnsimons · 2026-09-17T00:08:24Z

+- Archived and resolved failed messages: the biggest category by far, and most of the copying time
+- The event log, without which the ServicePulse activity feed starts empty
+- Custom checks, which cost almost nothing to skip because every check re-reports on its next interval
+- Failed error imports, the record of errors that could not be ingested


I don't know is we can ignore these at all, at the end of the day they are error messages that could not be ingested, we may need to notify the customer about them!

Probably worth just marking them as errors in the pre-flight and force them to resolve before migrate!

johnsimons · 2026-09-17T00:09:31Z

+- The event log, without which the ServicePulse activity feed starts empty
+- Custom checks, which cost almost nothing to skip because every check re-reports on its next interval
+- Failed error imports, the record of errors that could not be ingested
+- Group comments, **copied last of everything**, after archived and resolved messages. A comment survives only once the failed messages its group is built from have arrived, so on a large archive the comments are the last thing to appear. An empty comment field partway through a migration is the copy still running, not data loss


This is important to bring in, as soon as we create the group, becasue there may be info there about informing things like "do not retry this group...."

johnsimons · 2026-09-17T00:09:49Z

+- Custom checks, which cost almost nothing to skip because every check re-reports on its next interval
+- Failed error imports, the record of errors that could not be ingested
+- Group comments, **copied last of everything**, after archived and resolved messages. A comment survives only once the failed messages its group is built from have arrived, so on a large archive the comments are the last thing to appear. An empty comment field partway through a migration is the copy still running, not data loss
+- Failed message edits


What is this?

ServiceControl/src/ServiceControl.Persistence.RavenDB/Editing/EditFailedMessagesDataStore.cs

Line 9 in 7dd5105

class EditFailedMessagesDataStore(IRavenSessionProvider sessionProvider, ExpirationManager expirationManager) : IEditFailedMessagesDataStore

I've expanded the dot point to add more clarity.

- Introduced `BenignSkipped` property in `MigrationWriteResult` to track rows that the target would have deleted anyway. - Updated `MigrationEngine` logic to ensure benign skips do not count towards the halt threshold. - Added tests to verify that benign skips are handled correctly and do not cause category halts. - Updated skip reasons to include `PastRetention` for better clarity on skipped rows. - Created approval files for category selection tests to ensure correct order and configuration.

- Improved error logging in MigrationEngine to provide clearer context on exceptions. - Added functionality to InMemoryMigrationSource to simulate shutdown behavior during body reads. - Enhanced InMemoryMigrationTarget to allow custom exceptions for failure scenarios. - Introduced TimerRecordingTimeProvider to facilitate testing of timer-related functionality. - Expanded HaltThresholdTests to cover edge cases involving mixed benign and fault skips. - Updated MigrationEngineBodyRetryTests to ensure proper handling of shutdowns during body reads. - Added tests to verify behavior when categories are empty or already completed. - Implemented checks for checkpoint conflicts and cancellation scenarios in MigrationEngineFailurePathTests. - Enhanced MigrationEngineHaltTests to evaluate mixed skip scenarios and their impact on halting. - Improved MigrationEngineOrderingTests to ensure blocked categories behave correctly. - Added tests to MigrationEngineResumeTests to verify that restart behavior maintains original start time. - Updated MigrationEngineRunCategoriesTests to handle cases where no categories are selected. - Enhanced MigrationEngineSkipReasonTests to ensure accurate recording of multiple skip reasons. - Added tests to MigrationEngineThrottleTests to verify behavior during pauses and shutdowns.

warwickschroeder added 7 commits September 11, 2026 08:58

Add migration overview documentation for RavenDB to SQL transition

c878f32

Refine migration overview documentation for RavenDB to SQL transition

c0ccc15

Clarify documentation on RavenDB source requirements for containerize…

2e57527

…d ServiceControl migration

Enhance migration overview documentation for RavenDB to SQL transitio…

f76ce9f

…n, clarifying strategies, goals, and migration limitations.

Implement the foundation for migration engine

b045ec3

Clarify migration overview by detailing additional skip conditions fo…

d25e12b

…r SQL migration from RavenDB

internalautomation Bot assigned warwickschroeder Sep 15, 2026

warwickschroeder mentioned this pull request Sep 15, 2026

[DRAFT] Raven to SQL migration engine #5851

Closed

PR review fixes

b58e559

warwickschroeder requested review from abparticular, johnsimons and rbev September 15, 2026 08:17

rbev reviewed Sep 15, 2026

View reviewed changes

warwickschroeder added this pull request to stack #5898 September 16, 2026 04:25

warwickschroeder mentioned this pull request Sep 16, 2026

[Migration Engine Part 2] Add a migration checkpoint table to SQL Server and PostgreSQL #5897

Draft

warwickschroeder changed the title ~~Implement the foundation for migration engine~~ [Migration Engine Part 2] Implement the foundation for migration engine Sep 16, 2026

warwickschroeder changed the title ~~[Migration Engine Part 2] Implement the foundation for migration engine~~ [Migration Engine Part 1] Implement the foundation for migration engine Sep 16, 2026

rbev reviewed Sep 16, 2026

View reviewed changes

Comment thread src/ServiceControl.UnitTests/Migration/MigrationEngineCategorySelectionTests.cs Outdated

Comment thread src/ServiceControl.UnitTests/Migration/MigrationEngineCategorySelectionTests.cs Outdated

PR review

909e4bf

johnsimons approved these changes Sep 17, 2026

View reviewed changes

warwickschroeder added 2 commits September 17, 2026 10:16


		var result = await target.Write(category, batchToWrite, checkpointAfterBatch, cancellationToken);

		checkpoint = checkpointAfterBatch with

Conversation

warwickschroeder commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Tests

Uh oh!

rbev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

warwickschroeder Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

warwickschroeder Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

johnsimons left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

warwickschroeder commented Sep 15, 2026 •

edited

Loading

warwickschroeder Sep 15, 2026 •

edited

Loading

warwickschroeder Sep 17, 2026 •

edited

Loading