Repository navigation
Clarify RDI snapshot modes and missed deletes - #4238
Open
ZdravkoDonev-redis wants to merge 4 commits into
Open
ZdravkoDonev-redis wants to merge 4 commits into
ZdravkoDonev-redis wants to merge 4 commits into
Conversation
Contributor
Contributor
🧠 Redis MemoryFound 8 related items from repository history:
Memory updated at f6d2bb1 |
andy-stark-redis
approved these changes
Oct 12, 2026
andy-stark-redis
left a comment
andy-stark-redis
left a comment
Contributor
There was a problem hiding this comment.
Some suggestions (mainly adding the word "mode" after always, initial, etc) and I'd prefer not to use that new shortcode unless you really feel it's necessary. Otherwise looks good, so I'll approve to avoid holding you up.
| @@ -62,11 +62,11 @@ When extending an older single-source pipeline, preserve the original source nam | |||
| 1. On **Dashboard**, open the source card's **More actions** menu. | |||
| 1. Select **Stop source**, then confirm with **Stop source**. | |||
|
|
|||
| Stopping one source pauses data capture from that source without stopping the other sources. Starting it again resumes from its saved position. To resume a source, open its **More actions** menu and select **Start source**, then confirm with **Start source**. A source processes data only while the pipeline is running. See [How does stopping sources affect billing?](/content/operate/rc/rdi/faq.md#stopping-and-billing). | |||
| Stopping one source pauses data capture from that source without stopping the other sources. Starting a Debezium source again follows its [snapshot mode](/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) and the availability of its saved position. To resume a source, open its **More actions** menu and select **Start source**, then confirm with **Start source**. A source processes data only while the pipeline is running. See [How does stopping sources affect billing?](/content/operate/rc/rdi/faq.md#stopping-and-billing). | |||
Collaborator
There was a problem hiding this comment.
If we are mentioning "Debezium" explicitly here, perhaps we should also mention the behavior for other non-Debezium sources e.g. spanner and snowflake.
| position, then streams changes. A restart with a completed snapshot and a valid saved | ||
| position resumes streaming. Deployment or restart alone does not require a new snapshot. | ||
| If the saved position is no longer available, `initial` does not automatically | ||
| re-snapshot to recover, and the collector can report an error. |
Collaborator
There was a problem hiding this comment.
Suggested change
| re-snapshot to recover, and the collector can report an error. | |
| re-snapshot to recover, and the collector stops and reports an error. |
| position resumes streaming. Deployment or restart alone does not require a new snapshot. | ||
| If the saved position is no longer available, `initial` does not automatically | ||
| re-snapshot to recover, and the collector can report an error. | ||
|
|
Collaborator
There was a problem hiding this comment.
Consider adding that "A restart in the middle of the snapshot causes the snapshot to start again. This can also result in a snapshot gap, where deleted records remain in the target"
Comment on lines
+36
to
+37
| An interrupted initial snapshot can restart when the collector starts again, including | ||
| with `initial` and `initial_only`. |
Collaborator
There was a problem hiding this comment.
Check my comments regarding a restart during the snapshot above.
Comment on lines
+12
to
+15
| With `when_needed` mode, the collector also takes a new snapshot when it detects that the | ||
| saved position is no longer available. This can reload all selected data after an | ||
| outage without a manual reset. The snapshot captures current rows, rather than | ||
| reconstructing every missed change. |
Collaborator
There was a problem hiding this comment.
Perhaps a warning should be added here that using when_needed may cause a long-running snapshot in recoverable situations, e.g., restoring the missing offset on the server. If when_needed is used, this should be combined with proper monitoring and alerting to inform the system operator about the increased latency and possible stale records.
| configuration, so you can start it again later from where it left off. | ||
| configuration. A Debezium source restarts according to its | ||
| [snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) | ||
| and the availability of its saved source position. |
Collaborator
There was a problem hiding this comment.
Suggested change
| and the availability of its saved source position. | |
| and the availability of its saved source position. | |
| If you keep the source stopped for long periods, the stored offset may be removed, making it impossible to resume streaming. |
| | `initial` (default) | Snapshot when no offset exists, then stream changes. | Resume after a completed snapshot if the position is valid. No automatic snapshot recovery for unavailable log history. | Snapshot, then stream changes. | | ||
| | `when_needed` | Snapshot when no offset exists, then stream changes. | Resume if the position is valid. Snapshot again if the connector detects an unavailable position. | Snapshot, then stream changes. | | ||
| | `always` | Snapshot on every collector start, then stream changes. | Take a new snapshot even if the position is valid. | Snapshot, then stream changes. | | ||
| | `initial_only` | Take an initial snapshot without streaming subsequent changes. | Restart an interrupted initial snapshot; do not capture subsequent changes after it completes. | Snapshot without ongoing change capture. | |
Collaborator
There was a problem hiding this comment.
It may be worth noting that using initial_only will cause the collector to enter a crash loop upon completion, causing the source to appear as "not connected" in monitoring.
| @@ -459,7 +459,7 @@ redis.wait.retry.delay.ms: 1000 | |||
| <a name="sourcesadvancedsource"></a> | |||
| #### sources\.advanced\.source: Advanced source settings | |||
|
|
|||
| Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**<br/><br/>For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.<br/><br/>**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.** | |||
| Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**<br/><br/>For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.<br/><br/>Set `snapshot.mode` as a free-form source property to control data snapshots. The default is `initial`; available modes and restart behavior depend on the connector. A reset clears the saved position without changing the mode. See [Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode), including the [missed-delete timeline](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#missed-deletes).<br/><br/>**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.** | |||
Collaborator
There was a problem hiding this comment.
As we don't support DB2, we should not have it in this list.
Comment on lines
+66
to
+71
| With `initial` mode, the collector can fail rather than start a new snapshot when the saved | ||
| position is unavailable. With `when_needed` mode, it can automatically snapshot the current | ||
| selected data and resume streaming. A new snapshot after a restart can therefore | ||
| indicate that the source history was no longer available. See | ||
| [Choose a snapshot mode](/content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md#choose-a-snapshot-mode) | ||
| for the connector-specific behavior. |
Collaborator
There was a problem hiding this comment.
the collector can fail rather than start a new snapshot when the saved position is unavailable. - This leaves it to the reader's interpretation whether this is normal behavior, a problem caused by RDI, or something else.
I recommend adding a note explaining that this is an exceptional situation that may be caused by one of the following (but not only):
- collector being stopped for a long time, and the change log is removed from the server
- change in the configuration causing the collector to connect to another server with same data but different offsets
- changelog being prematurely removed from the server as a part of a cleanup process, while RDI is operational
- offset missing on the server after a crash and restore
| which is expected on first deployment or after a reset. | ||
|
|
||
| With `initial` mode, the collector can fail rather than start a new snapshot when the saved | ||
| position is unavailable. With `when_needed` mode, it can automatically snapshot the current |
Collaborator
There was a problem hiding this comment.
Suggested change
| position is unavailable. With `when_needed` mode, it can automatically snapshot the current | |
| position is unavailable. With `when_needed` mode, it will automatically snapshot the current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a saved source log position expires,
when_neededcan take a new snapshot and resume collection while leaving deleted rows in Redis. The current docs do not explain this gap or the connector differences between snapshot modes.This change compares all six listed modes for Debezium sources and adds a shared Mermaid timeline to the latest and v1.19.1 configuration guides. It updates the configuration references, deployment/reset guidance, troubleshooting, multiple-source guidance, and Cloud pages. Reset and target cleanup remain separate. The embed shortcode now enables Mermaid rendering in the parent page.
The Cloud FAQ includes a six-event timeline with colored markers and alternating labels on wide screens. It switches to a vertical timeline when the content area is narrow. The diagram uses semantic HTML and CSS, with no JavaScript dependency.
The Architecture guarantee remains unchanged, with an open product-review note about expired history and missed deletes.
Validation:
no_data.Companion schema-description PR: https://github.com/RedisLabs/redis-data-integration/pull/2886
Context: Slack discussion.
Note
Low Risk
Documentation-only changes with no runtime or configuration behavior changes in this repository.
Overview
Documents Debezium
snapshot.modefor RDI CDC sources and the stale-target gap when recovery snapshots cannot replay expired source log history (especially missed deletes underwhen_needed).Adds shared embeds (
rdi-snapshot-modes.md,rdi-snapshot-gap.mdwith a Mermaid sequence diagram) and a Choose a snapshot mode section in the latest and v1.19.1 pipeline guides, including a six-mode comparison table and Debezium 3.5 connector notes. Theembed-mdshortcode now propagateshasMermaidso diagrams render on parent pages.Reframes operational docs so
redis-di reset, stop/start, multi-source lifecycle, and Redis Cloud flows depend on configured snapshot mode: reset clears saved positions without changing mode or flushing Redis; reload and restart behavior vary by mode (initial_only,no_data,always, etc.). Architecture adds a note that at-least-once applies only to captured events. New troubleshooting covers unavailable source log history and links to missed-delete recovery guidance. Config reference and CLI pages cross-link the new section.Reviewed by Cursor Bugbot for commit f6d2bb1. Bugbot is set up for automated code reviews on this repo. Configure here.