Document Cloud RDI workspace capacity planning and pipeline scaling

content/embeds/rc-rdi-create-rdi-workspace.md

@@ -1,42 +1,58 @@
 To create a Data Integration workspace for an existing [Pro subscription](/content/operate/rc/databases/create-database/create-pro-database-new.md):
 
-1. From the Redis Cloud console, select **Data Integration** from the left-hand menu. If you don't have any workspaces yet, select **Create workspace** to go to the **Create workspace** page.
+1. In the Redis Cloud console, select **Data Integration**.
+
+2. Select **Create workspace** if you have no workspaces yet.
 
     ![The create workspace button.](/images/rc/rdi/rdi-create-workspace-button.png)
     {width="200px"}
 
-    If you already have a workspace deployed, you'll see your current workspaces. Select **New workspace** to go to the **Create workspace** page.
+    If you already have a workspace, select **New workspace**.
 
     ![The new workspace button.](/images/rc/rdi/rdi-new-workspace-button.png)
     {width="150px"}
 
-    You can also go to the **Data Integration** tab from your subscription or database page and select **Create workspace** to go to the **Create workspace** page for your subscription.
+    You can also open **Data Integration** from a subscription or database page.
+    Select **Create workspace** to use that subscription.
 
     ![The create workspace button.](/images/rc/rdi/rdi-create-workspace-button.png)
     {width="200px"}
 
-2. Select your Pro subscription from the list if it's not already selected.
+3. Select your Pro subscription if it is not already selected.
 
     ![The select pro subscription drop down.](/images/rc/rdi/rdi-create-workspace-select-subscription.png)
     {width="80%"}
 
-3. A **Data Integration subnet (CIDR)** is automatically generated for you. Each RDI workspace uses a dedicated `/22` CIDR.
+4. Review **Data Integration subnet (CIDR)**. The console suggests a dedicated
+   `/22` Classless Inter-Domain Routing (CIDR) range for the workspace.
+
+    [Plan network capacity for all pipelines](/content/operate/rc/rdi/scale-pipeline.md#plan-workspace-network-capacity)
+    before you create the workspace. Choose a `/21` or larger range if you plan
+    more than 5 sources or more than 5 processor replicas in total.
+    These counts are a conservative planning trigger, not a capacity limit.
+    You cannot enlarge the workspace CIDR after creation.
 
-    For AWS, the RDI workspace CIDR must:
+    For Amazon Web Services (AWS), the RDI workspace CIDR must:
 
-    - Be in the same [RFC 1918 private address range](https://datatracker.ietf.org/doc/html/rfc1918#section-3) as the subscription VPC's primary CIDR: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
+    - Be in the same [RFC 1918 private address range](https://datatracker.ietf.org/doc/html/rfc1918#section-3) as the subscription virtual private cloud (VPC)'s primary CIDR: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
     - Not overlap with existing subscription, peering, transit gateway (TGW), application, database, or RDI workspace CIDR ranges.
 
-    For example, if the subscription VPC's primary CIDR is `10.238.252.0/24`, then `192.168.0.0/22` is invalid because it is in a different RFC 1918 range. An unused range such as `10.239.0.0/22` is valid.
+    For example, a subscription VPC has the primary CIDR `10.238.252.0/24`.
+    The range `192.168.0.0/22` is invalid because it is in a different
+    RFC 1918 range. An unused range such as `10.239.0.0/22` is valid.
 
-    If the automatic suggestion is missing or unsuitable, select another unused `/22` CIDR in the same private range. For more information, see [VPC CIDR block association restrictions](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html#vpc-resize).
+    If the suggestion is missing or unsuitable, select another unused range.
+    It must meet your capacity needs and use the same private address range.
+    See [VPC CIDR block association restrictions](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html#vpc-resize).
 
     ![The select pro subscription drop down.](/images/rc/rdi/rdi-create-workspace-cidr.png)
     {width="80%"}
 
-4. Select **Create workspace** to create your workspace.
+5. Select **Create workspace**.
 
     ![The create workspace button.](/images/rc/rdi/rdi-create-workspace-button.png)
     {width="200px"}
 
-Your workspace will be created in the background. You can select **Create pipeline** to [create your pipeline](/content/operate/rc/rdi/define.md) while the workspace is provisioning, or you can select **Create pipeline later** to go back to the Redis Cloud console.
+Redis Cloud creates the workspace in the background. Select **Create pipeline**
+to [create your pipeline](/content/operate/rc/rdi/define.md) while it provisions.
+Select **Create pipeline later** to return to the Redis Cloud console.

content/operate/rc/rdi/_index.md

@@ -114,6 +114,9 @@ Once your data pipeline is defined, you can [view and edit](/content/operate/rc/
 
 For complete production setups, including SQL Server failover handling, see [Production use cases](/content/operate/rc/rdi/use-cases/_index.md).
 
+To plan workspace capacity and scale a pipeline, see [Scale a Cloud RDI pipeline]({{<
+relref "/operate/rc/rdi/scale-pipeline" >}}).
+
 ## Billing and common questions
 
 See the [RDI Cloud FAQ](/content/operate/rc/rdi/faq.md) for billing examples, reset and flush behavior, and working with multiple sources.

content/operate/rc/rdi/faq.md

@@ -57,6 +57,32 @@ Stop the affected source before deleting its records and allow its pending recor
 
 No. Deleting a source removes its pipeline configuration and internal RDI state. Records it already wrote to the target remain. You must remove or reassign transformation jobs that refer to the source before deleting it. See [Remove a source](/content/operate/rc/rdi/view-edit.md#remove-source).
 
+## Capacity and network planning
+
+### What CIDR range should I use for Cloud RDI?
+
+Plan the Classless Inter-Domain Routing (CIDR) range for all pipelines in the
+workspace, including future sources and processor replicas. You cannot enlarge
+it after creation. See [Plan workspace network capacity](/content/operate/rc/rdi/scale-pipeline.md#plan-workspace-network-capacity)
+for the `/22`, `/21`, and `/20` capacity table and its configuration.
+
+### How do I change a workspace CIDR that is too small? {#change-workspace-cidr}
+
+You cannot change or enlarge the CIDR of an existing workspace in place.
+To use a larger range, [delete the existing workspace](/content/operate/rc/rdi/create-workspace.md#delete-workspace),
+[create a workspace](/content/operate/rc/rdi/create-workspace.md#create-a-data-integration-workspace)
+with the larger CIDR, and redeploy your pipelines.
+
+Before you delete anything, record your pipeline configuration, source settings,
+transformation jobs, and secret references. Recreation interrupts ingestion
+and removes the saved pipeline state. Redeployed sources take new initial
+snapshots. Records already written to the target database remain there.
+
+For help planning the recreation and connectivity changes, open a
+[Redis support ticket](https://redis.io/support/) before deleting anything. Include
+your subscription and workspace identifiers, current CIDR, and planned totals for
+sources and processor replicas across all pipelines.
+
 ## Upgrades and maintenance
 
 ### What happens during an RDI Cloud upgrade?
@@ -109,3 +135,28 @@ Yes. RDI usage is based on the running collectors and processor replicas, not th
 Creating a workspace or saving a setup draft does not start RDI usage billing. Billing starts when a pipeline is deployed. Stopping the pipeline reduces usage to the workspace charge. Deleting the deployed pipeline ends new RDI usage when no deployed pipelines remain in the workspace; usage already recorded for the hour can still be billed.
 
 Delete an unused pipeline and then [delete its workspace](/content/operate/rc/rdi/create-workspace.md#delete-workspace) when you no longer need RDI. This does not delete the target Redis database or stop its separate charges.
+
+## Processor scaling
+
+### Does Cloud RDI automatically scale the processor?
+
+No. Set the desired processor replica count with
+`advanced.resources.taskManager.replicas`. See [Increase processor capacity](/content/operate/rc/rdi/scale-pipeline.md#increase-processor-capacity).
+
+### How do I increase processing capacity for a pipeline?
+
+First [identify the bottleneck](/content/operate/rc/rdi/scale-pipeline.md#identify-the-bottleneck).
+Then tune the collector, RDI database, or processor. Check shared
+workspace network capacity before adding sources or processor replicas.
+
+### Can I see that the processor has scaled in the console?
+
+Yes. The **Dashboard** shows running and configured processor replicas next to
+the processor status. See [Confirm the applied capacity](/content/operate/rc/rdi/scale-pipeline.md#confirm-the-applied-capacity)
+for how to read the count and handle a persistent mismatch.
+
+### Can I use billing to confirm a scaling change?
+
+No. Billing does not show scaling changes in real time. Use the Dashboard or
+the [Redis Cloud application programming interface (API)](/content/operate/rc/rdi/scale-pipeline.md#confirm-the-applied-capacity)
+to confirm the running count.

content/operate/rc/rdi/scale-pipeline.md

@@ -0,0 +1,207 @@
+---
+Title: Scale a Cloud RDI pipeline
+aliases:
+    - /operate/rc/rdi/scale-processor/
+    - /operate/rc/rdi/scale-processor
+    - /operate/rc/databases/rdi/scale-processor/
+    - /operate/rc/databases/rdi/scale-processor
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Increase collector, RDI database, and processor capacity for a Cloud RDI pipeline.
+hideListLinks: true
+weight: 5
+---
+
+Every Redis Data Integration (RDI) pipeline in Redis Cloud uses the Flink
+processor. The collector, RDI database, processor, or target database can
+limit pipeline capacity. Identify the bottleneck before you change a setting.
+
+Cloud RDI does not automatically add or remove processor replicas
+(TaskManagers) based on load, pending records, throughput, or backpressure.
+Set the number of processor replicas when you need more processing capacity.
+
+## Plan workspace network capacity
+
+Choose the workspace Classless Inter-Domain Routing (CIDR) range for all
+pipelines you plan to run. All sources and processor replicas share the
+workspace's network capacity. Each source uses a collector. Each TaskManager
+is a processor replica. Adding either uses more workspace addresses.
+
+The console suggests a `/22` range. If you expect more than 5 sources or more
+than 5 processor replicas in total across the workspace, choose a `/21` or
+larger range when you create it. For further growth, consider a `/20` or larger
+range. A `/21` has twice the address space of a `/22`; a `/20` has twice that
+of a `/21`. Doubling the address space does not guarantee twice as many
+sources or processor replicas.
+
+This is conservative planning guidance, not a guaranteed capacity limit.
+Leave room for planned growth, replacement resources, and maintenance.
+The range must also meet the requirements of your connectivity method and
+must not overlap with connected networks.
+
+### Processor capacity by CIDR
+
+Use this table as a reference for processor address capacity. The reference
+configuration uses RDI 2.0.0 on Amazon Web Services (AWS), with three
+availability zones. It has one pipeline, no sources, and two control-plane
+nodes. Each processor requests one central processing unit (CPU) and 5 GiB
+of memory. Temporary workspace scaling overrides allow the configured counts
+in the table. The allocation window is 10 minutes per scaling request.
+
+Configured replicas are the requested count. Ready replicas are running
+processors that pass readiness checks. Both columns count processors only.
+They exclude source collectors and control-plane nodes. These resources
+also use workspace addresses.
+
+| Workspace CIDR | Configured replicas | Ready replicas | Capacity constraint |
+| --- | ---: | ---: | --- |
+| `/22` | 15 | 14 | Address blocks exhausted in the assigned subnet. Another zone retained a block. |
+| `/21` | 50 | 32 | Address blocks exhausted in all worker subnets. Allocation requests were also throttled. |
+| `/20` | 50 | 50 | Capacity for at least 50 replicas in this configuration. The upper limit is not established. |
+
+The `/22` and `/21` figures use gradual scaling. The `/20` figure uses a
+direct request for 50 replicas.
+
+The counts of 14 and 32 are not fixed workspace limits. Address distribution
+across zones and allocation throttling also affect scaling. The table does
+not define supported replica limits, throughput, or combined source and
+processor capacity. Reserve addresses for collectors, other pipelines,
+growth, and maintenance.
+
+You cannot enlarge an existing workspace CIDR. A larger range requires
+[workspace recreation](/content/operate/rc/rdi/faq.md#change-workspace-cidr). See
+[Create a workspace](/content/operate/rc/rdi/create-workspace.md) for CIDR
+selection and connectivity requirements.
+
+## Identify the bottleneck
+
+Use the **Dashboard** to compare **Throughput**, **Pending**, **Processor load**,
+and **RDI database load** before you change capacity:
+
+- If ingestion from a source is slow and the processor has spare capacity,
+  check the source's diagnostics in **Metrics** before tuning collector
+  properties.
+- If the processor stays at capacity while pending records grow, consider
+  increasing processor replicas.
+- If the processor is backpressured, check target database performance and
+  connectivity before adding processor replicas.
+- If the RDI database stays full or reports out-of-memory (OOM) errors, check
+  ingestion and processing rates, database memory, and throughput. OOM alone
+  does not identify which capacity setting to change.
+
+Change one setting at a time and compare the same representative workload.
+Confirm that the change improves throughput or reduces pending records before
+making another increase.
+
+## Increase collector ingestion capacity
+
+Tune collector properties when the collector cannot ingest data from the
+source quickly enough. These properties are specific to each source. In the
+Cloud console, select the source in **Configuration**, then select **Edit
+collector properties**.
+
+The console exposes these commonly tuned Debezium properties when they apply to
+the selected source:
+
+| Collector section | Property | Effect |
+| --- | --- | --- |
+| Source | `snapshot.max.threads` | Increases the threads used for the initial snapshot. |
+| Source | `snapshot.fetch.size` | Increases the rows fetched in each snapshot batch. |
+| Source | `max.batch.size` | Increases the records processed in each batch. |
+| Source | `max.queue.size` | Increases the records buffered in collector memory. |
+| Source | `poll.interval.ms` | Sets the wait before the collector polls for change data capture (CDC) changes. |
+| Source | `record.processing.threads` | Increases the threads that process captured records. |
+| Sink | `redis.batch.size` | Increases the records written to the RDI database in each batch. |
+| Sink | `redis.flush.interval.ms` | Reduces the maximum wait before the collector flushes a batch. |
+
+Use only properties shown for the selected source. The available properties can
+vary by source database.
+
+Cloud RDI collectors have 2 CPUs and 8 GB of
+random access memory (RAM). You cannot select a different collector size.
+Test each change with a representative workload before you use it in production.
+
+Larger batches and queues use more collector memory. More snapshot and
+record-processing threads share the same 2 CPUs. A shorter poll interval can
+reduce CDC latency but can increase load on the source database. Change one
+property at a time and use the Dashboard to compare throughput, pending
+records, and processor load.
+
+## Increase RDI database throughput
+
+Each pipeline has an RDI database. When database throughput limits ingestion
+or processing, choose a higher throughput value from its **Performance**
+settings.
+
+1. In the Cloud console, open the RDI database for the pipeline.
+1. Edit the database **Performance** settings.
+1. Increase **Throughput** to the next available value.
+1. Check the pipeline Dashboard before making another increase.
+
+Stop increasing throughput when the pipeline no longer improves. Check
+processor and target performance too. More database throughput does not
+resolve a bottleneck in another component.
+
+## Increase processor capacity
+
+1. From the Cloud RDI **Pipelines** list, select the pipeline.
+1. Select the **Settings** tab.
+1. Select **Edit**.
+1. Add or update the `advanced.resources.taskManager.replicas` property with
+   the number of processor replicas you need. Use a positive whole number
+   accepted by the console and check [workspace network capacity](#plan-workspace-network-capacity)
+   before you increase it.
+1. Select **Save changes**.
+1. Select **Apply and restart** to apply the change. This temporarily
+   interrupts processing.
+
+This property sets the desired processor replica count. After the restart,
+check the running count and pipeline performance. More replicas do not
+guarantee higher throughput.
+
+For application programming interface (API) configuration, the same property
+is represented as follows:
+
+```yaml
+processors:
+  advanced:
+    resources:
+      taskManager:
+        replicas: 3
+```
+
+## Confirm the applied capacity
+
+The **Dashboard** shows the processor replica count next to the processor
+status. It shows the running count and configured count in the format
+`running / configured replicas`. For example, `3 / 3 replicas` means three
+TaskManagers are running. The configured count is also three. If only one
+count is available, the Dashboard labels it as running or configured.
+
+The running count can be lower while a restart is in progress. If it remains
+below the configured count after the restart, check pipeline and source status
+for errors and contact [Redis support](https://redis.io/support/). Network
+capacity can prevent additional replicas from starting. Increasing the
+configured count again does not resolve a network capacity shortage.
+
+The console metrics show data-stream record counts and pending records. They do
+not include a TaskManager replica-count metric.
+
+For programmatic confirmation, use the [Redis Cloud API](/content/operate/rc/api/api-reference.md#tag/Data-Integration)
+to send a `GET` request to
+`/v1/subscriptions/<subscription-id>/data-integration-workspace/rdi/api/v2/pipelines/<pipeline-name>/status`.
+Inspect the `flink-processor` entry in `components`. Its `replicas` value is
+the number of ready processor replicas. For example:
+
+```json
+{
+  "name": "flink-processor",
+  "replicas": 3
+}
+```
+
+Billing is not a real-time way to confirm that a scaling change has completed.
+Use the Dashboard or the status API instead.

content/operate/rc/rdi/view-edit.md

@@ -159,7 +159,7 @@ The **Settings** tab contains the default data structure (**Hash** or **JSON**)
 ![Pipeline-wide Settings tab with default data structure and processor properties.](/images/rc/rdi/rdi-2-settings.png)
 {width="100%"}
 
-Select **Edit** to change these settings, then **Save changes** and **Apply and restart**. RDI Cloud uses the Flink processor. Review [processor properties](/content/integrate/redis-data-integration/reference/config-yaml-reference.md#processors-data-processing-configuration) before changing them.
+Select **Edit** to change these settings. Select **Save changes**, then **Apply and restart**. Every Cloud RDI pipeline uses the Flink processor. Review [processor properties](/content/integrate/redis-data-integration/reference/config-yaml-reference.md#processors-data-processing-configuration) before changing them. To change processor capacity, see [Scale a Cloud RDI pipeline](/content/operate/rc/rdi/scale-pipeline.md).
 
 ![The processor advanced properties editor with key and value fields.](/images/rc/rdi/rdi-processor-advanced-properties.png)
 {width="80%"}