From bcacf0ada7fc022786e0ba80737d7921e32c9360 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 9 Oct 2026 11:51:14 +0300 Subject: [PATCH 1/3] RDSC-4929: Clarify SQL Server schema migration and capture history limits --- .../data-pipelines/prepare-dbs/sql-server.md | 22 +++++-------------- .../integrate/redis-data-integration/faq.md | 2 ++ 2 files changed, 8 insertions(+), 16 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md index 54a932525a..091677b1d2 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md @@ -759,16 +759,9 @@ GO ## Handling changes to the schema -RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. For example, -if you add a new column to a table you are capturing then RDI will generate errors -instead of capturing the changes correctly. See Debezium's -[SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution) -docs for more information. +RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. The existing capture instance continues to use its captured column structure. For example, adding a column can leave RDI streaming changes without that column instead of reporting an error. Other schema changes can produce incorrect events or errors. Create a new capture instance to capture the updated schema. See Debezium's [SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution) documentation for more information. -If you have administrator privileges, you can follow the steps below to update RDI after -a schema change and resume CDC. See the -[online schema updates](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#online-schema-updates) -documentation for further details. +If you have administrator privileges, use this online procedure to update the capture instance while Debezium is running. During the interval between changing the source schema and creating the new capture instance, events still use the old captured column structure. A short interval or low write traffic does not guarantee correct capture. If you cannot tolerate events that omit the new column, use Debezium's [offline schema update procedure](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#offline-schema-updates). It suspends application writes and drains pending events before stopping Debezium and updating the schema and capture instance. ```checklist {id="sqlserver-schema-changes" nointeractive="true" } - [ ] [Make your changes to the source table schema](#make-your-changes-to-the-source-table-schema) @@ -796,9 +789,9 @@ documentation for further details. ``` 1. - When Debezium starts streaming from the new capture table, drop the old capture table by running - the `sys.sp_cdc_disable_table` stored procedure with the parameter `@capture_instance` set to the old - capture instance name, `dbo_MyTable`: + When Debezium starts streaming from the new capture table and you no longer need the old captured changes, drop the old capture table by running the `sys.sp_cdc_disable_table` stored procedure with the parameter `@capture_instance` set to the old capture instance name, `dbo_MyTable`: + + Disabling the old instance deletes its change table and captured history. Rewinding an offset cannot recover changes that existed only in that capture table, even if SQL Server still retains them in the transaction log. If you need those changes for recovery or replay, retain the old instance until they are no longer needed. Retaining the instance does not prevent CDC cleanup from removing expired changes. Check the [CDC retention and capture instance limits](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server). SQL Server permits only two capture instances per source table, so retaining the old instance also affects the next schema migration. ```sql EXEC sys.sp_cdc_disable_table @@ -809,7 +802,4 @@ documentation for further details. ``` > [!NOTE] -> RDI will *not* correctly capture changes that happen in the time gap between changing -> the source schema (step 1 above) and updating the value of `@capture_instance` (step 2). -> Try to keep the gap as short as possible or perform the update at a time when you expect -> few changes to the data. +> Verify the values of the new or changed columns in the captured events after the migration. Continued streaming or a matching row count does not prove that the updated columns were captured. diff --git a/content/integrate/redis-data-integration/faq.md b/content/integrate/redis-data-integration/faq.md index 0ef4e0fe5f..98e7fcfdc7 100644 --- a/content/integrate/redis-data-integration/faq.md +++ b/content/integrate/redis-data-integration/faq.md @@ -100,6 +100,8 @@ and changes to column names unless you configure it for a specific set of column Bear in mind that the Redis keys in the target database will change to reflect the new or renamed tables and columns. +SQL Server is an exception to automatic column handling. You must create a new CDC capture instance after a schema change. Until you do, events can continue to use the old captured column structure and omit a newly added column. Follow [Handling changes to the SQL Server schema](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md#handling-changes-to-the-schema) for the migration procedure and its capture history limits. + ## Should I be concerned when the log says RDI is out of memory? {#rdi-oom} Sometimes the Debezium log will contain a message saying that RDI is out of From 8e49d6692b0de6dd0b072d3b2fce3200bcfadef7 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 9 Oct 2026 12:04:37 +0300 Subject: [PATCH 2/3] RDSC-4929: Document offline SQL Server schema updates through RDI --- .../data-pipelines/prepare-dbs/sql-server.md | 40 ++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md index 091677b1d2..d05b3d9db0 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md @@ -761,7 +761,7 @@ GO RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. The existing capture instance continues to use its captured column structure. For example, adding a column can leave RDI streaming changes without that column instead of reporting an error. Other schema changes can produce incorrect events or errors. Create a new capture instance to capture the updated schema. See Debezium's [SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution) documentation for more information. -If you have administrator privileges, use this online procedure to update the capture instance while Debezium is running. During the interval between changing the source schema and creating the new capture instance, events still use the old captured column structure. A short interval or low write traffic does not guarantee correct capture. If you cannot tolerate events that omit the new column, use Debezium's [offline schema update procedure](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#offline-schema-updates). It suspends application writes and drains pending events before stopping Debezium and updating the schema and capture instance. +If you have administrator privileges, use this online procedure to update the capture instance while RDI is running. During the interval between changing the source schema and creating the new capture instance, events still use the old captured column structure. A short interval or low write traffic does not guarantee correct capture. If you cannot tolerate events that omit the new column, [update the schema offline](#update-the-schema-offline). ```checklist {id="sqlserver-schema-changes" nointeractive="true" } - [ ] [Make your changes to the source table schema](#make-your-changes-to-the-source-table-schema) @@ -803,3 +803,41 @@ If you have administrator privileges, use this online procedure to update the ca > [!NOTE] > Verify the values of the new or changed columns in the captured events after the migration. Continued streaming or a matching row count does not prove that the updated columns were captured. + +### Update the schema offline + +Use this procedure to adapt Debezium's [offline schema update procedure](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#offline-schema-updates) to RDI. Keep the pipeline running and stop only the SQL Server source, so the processor and other sources remain available. Replace `` with your pipeline name and `` with the SQL Server source name from `config.yaml`. + +1. Suspend application writes to the tables captured by the SQL Server source. + +1. Wait for the final committed changes to reach the target Redis database. Use [`redis-di describe`](/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md) to check that the source's tables have zero pending records, no errors, and no new rejected records. Also verify the expected final changes in the target. Zero pending records alone does not prove that SQL Server's capture job has captured the final writes. + + ```bash + redis-di describe + ``` + +1. Stop the SQL Server source with [`redis-di stop`](/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md). Wait for the command to complete successfully before changing the schema. + + ```bash + redis-di stop --source + ``` + +1. [Apply the schema changes](#make-your-changes-to-the-source-table-schema) while application writes remain suspended. + +1. [Create a new capture instance](#create-a-new-capture-table-for-the-updated-source-table) with a unique name. Keep the old instance until RDI has switched to the new one and you no longer need its captured history. + +1. If your pipeline selects specific columns or references changed column names in transformations, update its configuration and [deploy it](/content/integrate/redis-data-integration/data-pipelines/deploy.md) while application writes remain suspended. + +1. Restart the SQL Server source with [`redis-di start`](/content/integrate/redis-data-integration/reference/cli/redis-di-start.md). + + ```bash + redis-di start --source + ``` + +1. Check `redis-di describe ` for source connectivity and errors before resuming application writes. Starting a source does not start a stopped pipeline. + +1. Resume application writes. + +1. Verify that changes made after the migration include the new or changed column values in the target as intended by your transformation jobs. + +1. [Drop the old capture instance](#drop-the-old-capture-table) only after RDI streams from the new instance and you no longer need the old captured history. From 7f3440eef11880d32b1eb66f9f5101551d7391d3 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Fri, 9 Oct 2026 12:14:18 +0300 Subject: [PATCH 3/3] RDSC-4929: Clarify source collector wording and remove generic warning --- .../data-pipelines/prepare-dbs/sql-server.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md index d05b3d9db0..bc96f3e722 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md @@ -759,7 +759,7 @@ GO ## Handling changes to the schema -RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. The existing capture instance continues to use its captured column structure. For example, adding a column can leave RDI streaming changes without that column instead of reporting an error. Other schema changes can produce incorrect events or errors. Create a new capture instance to capture the updated schema. See Debezium's [SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution) documentation for more information. +RDI can't adapt automatically when you change the schema of a CDC table in SQL Server. The existing capture instance continues to use its captured column structure. For example, adding a column can leave RDI streaming changes without that column instead of reporting an error. Create a new capture instance to capture the updated schema. See Debezium's [SQL Server schema evolution](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-schema-evolution) documentation for more information. If you have administrator privileges, use this online procedure to update the capture instance while RDI is running. During the interval between changing the source schema and creating the new capture instance, events still use the old captured column structure. A short interval or low write traffic does not guarantee correct capture. If you cannot tolerate events that omit the new column, [update the schema offline](#update-the-schema-offline). @@ -806,9 +806,9 @@ If you have administrator privileges, use this online procedure to update the ca ### Update the schema offline -Use this procedure to adapt Debezium's [offline schema update procedure](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#offline-schema-updates) to RDI. Keep the pipeline running and stop only the SQL Server source, so the processor and other sources remain available. Replace `` with your pipeline name and `` with the SQL Server source name from `config.yaml`. +Use this procedure to adapt Debezium's [offline schema update procedure](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#offline-schema-updates) to RDI. Keep the pipeline running and stop only the SQL Server source collector, so the processor and other source collectors remain available. Replace `` with your pipeline name and `` with the SQL Server source name from `config.yaml`. -1. Suspend application writes to the tables captured by the SQL Server source. +1. Suspend application writes to the tables captured by the SQL Server source collector. 1. Wait for the final committed changes to reach the target Redis database. Use [`redis-di describe`](/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md) to check that the source's tables have zero pending records, no errors, and no new rejected records. Also verify the expected final changes in the target. Zero pending records alone does not prove that SQL Server's capture job has captured the final writes. @@ -816,7 +816,7 @@ Use this procedure to adapt Debezium's [offline schema update procedure](https:/ redis-di describe ``` -1. Stop the SQL Server source with [`redis-di stop`](/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md). Wait for the command to complete successfully before changing the schema. +1. Stop the SQL Server source collector with [`redis-di stop`](/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md). Wait for the command to complete successfully before changing the schema. ```bash redis-di stop --source @@ -828,13 +828,13 @@ Use this procedure to adapt Debezium's [offline schema update procedure](https:/ 1. If your pipeline selects specific columns or references changed column names in transformations, update its configuration and [deploy it](/content/integrate/redis-data-integration/data-pipelines/deploy.md) while application writes remain suspended. -1. Restart the SQL Server source with [`redis-di start`](/content/integrate/redis-data-integration/reference/cli/redis-di-start.md). +1. Restart the SQL Server source collector with [`redis-di start`](/content/integrate/redis-data-integration/reference/cli/redis-di-start.md). ```bash redis-di start --source ``` -1. Check `redis-di describe ` for source connectivity and errors before resuming application writes. Starting a source does not start a stopped pipeline. +1. Check `redis-di describe ` for source connectivity and errors before resuming application writes. Starting a source collector does not start a stopped pipeline. 1. Resume application writes.