Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions content/operate/rc/rdi/create-workspace.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub

{{< embed-md "rc-rdi-create-rdi-workspace.md" >}}

{{<image filename="images/rc/rdi/rdi-workspace-add-pipeline.png" alt="The Add pipeline control is available while the workspace is being created." width=80% >}}
![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png)
{width="80%"}

## View workspace status and details

Expand All @@ -30,11 +31,13 @@ You can view your workspace in one of the following ways:
- From the Redis Cloud console, go to the **Data integration** page, or
- From your subscription, select the **Data Integration** tab

{{<image filename="images/rc/rdi/rdi-2-workspace-sources.png" alt="Workspace pipeline list showing a streaming pipeline and its Sources column." width=100% >}}
![Workspace pipeline list showing a streaming pipeline and its Sources column.](/images/rc/rdi/rdi-2-workspace-sources.png)
{width="100%"}

There, you'll see your workspace and its pipelines. The **Sources** column lists each pipeline's source names and database types. A source with unfinished configuration is marked **Pending setup**. Open its draft to continue setup. To see your workspace details, including the deployment CIDR and region information, select **Workspace actions > Workspace details**.

{{<image filename="images/rc/rdi/rdi-2-workspace-actions.png" alt="Workspace actions menu with Workspace details and Delete workspace." width=240px >}}
![Workspace actions menu with Workspace details and Delete workspace.](/images/rc/rdi/rdi-2-workspace-actions.png)
{width="240px"}

## Delete workspace

Expand All @@ -44,4 +47,5 @@ Make sure to [delete your data pipeline]({{<relref "/operate/rc/rdi/view-edit#de

To delete your workspace, select **Workspace actions > Delete workspace** from your workspace.

{{<image filename="images/rc/rdi/rdi-2-workspace-actions.png" alt="Workspace actions menu with Workspace details and Delete workspace." width=240px >}}
![Workspace actions menu with Workspace details and Delete workspace.](/images/rc/rdi/rdi-2-workspace-actions.png)
{width="240px"}
51 changes: 34 additions & 17 deletions content/operate/rc/rdi/define.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ After you have [prepared each source database]({{<relref "/operate/rc/rdi/setup"

In the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab and select **Add pipeline**. You can also open the workspace from the **Data Integration** page or your subscription's **Data Integration** tab. To continue an existing draft, open its actions menu and select **Resume pipeline setup**.

{{<image filename="images/rc/rdi/rdi-workspace-add-pipeline.png" alt="The Add pipeline control is available while the workspace is being created." width=80% >}}
![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png)
{width="80%"}

To create a pipeline:

Expand All @@ -32,12 +33,14 @@ To create a pipeline:

1. Select the target from the **Target database** list. All sources write to this database.

{{<image filename="images/rc/rdi/rdi-choose-target.png" alt="The target database list in pipeline Settings." width=80% >}}
![The target database list in pipeline Settings.](/images/rc/rdi/rdi-choose-target.png)
{width="80%"}

1. Select **Hash** or **JSON** as the **Default data structure**. Transformation jobs can override how individual records are written.
1. If needed, configure **Processor properties**. These apply to the whole pipeline, not to an individual source. See the [processor configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors-data-processing-configuration" >}}).

{{<image filename="images/rc/rdi/rdi-processor-advanced-properties.png" alt="The processor advanced properties editor with key and value fields." width=80% >}}
![The processor advanced properties editor with key and value fields.](/images/rc/rdi/rdi-processor-advanced-properties.png)
{width="80%"}

1. Select **Continue**.

Expand Down Expand Up @@ -70,19 +73,22 @@ Choose **AWS Private Link** or **Public Endpoint** for the selected source, acco

- For **AWS Private Link**, enter the **Private Link service name** from your [endpoint service]({{< relref "/operate/rc/rdi/setup#set-up-connectivity" >}}). Select **Connect to Private Link** and wait for connectivity to complete. If the connection fails, check the service name and its allowed principal.

{{<image filename="images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png" alt="AWS Private Link connectivity with the service name and Connect to Private Link control." width=80% >}}
![AWS Private Link connectivity with the service name and Connect to Private Link control.](/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png)
{width="80%"}

- For **Public Endpoint**, enter the source IP address or hostname. Add the Redis Cloud outbound IP address shown in the console to your source database's allowlist.

{{<image filename="images/rc/rdi/rdi-source-configuration-source-connectivity-public.png" alt="Public endpoint connectivity with source hostname and Redis Cloud outbound IP addresses." width=80% >}}
![Public endpoint connectivity with source hostname and Redis Cloud outbound IP addresses.](/images/rc/rdi/rdi-source-configuration-source-connectivity-public.png)
{width="80%"}

Configure connectivity for each source separately. Sources in the same pipeline can use different connectivity methods.

### Secrets

Enter the Amazon Resource Name (ARN) of the selected source's [database credentials secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in **Credentials Secret ARN**.

{{<image filename="images/rc/rdi/rdi-source-configuration-secrets.png" alt="The Credentials Secret ARN field, transit security options, and Validate control." width=80% >}}
![The Credentials Secret ARN field, transit security options, and Validate control.](/images/rc/rdi/rdi-source-configuration-secrets.png)
{width="80%"}

For Snowflake, select the authentication method matching your secret:

Expand All @@ -93,11 +99,13 @@ Under **Transit security**, select the mode required by your source:

- **TLS** (Transport Layer Security): Provide the CA certificate secret ARN when your source requires it.

{{<image filename="images/rc/rdi/rdi-define-tls.png" alt="TLS transit security with the CA Certificate Secret ARN field." width=80% >}}
![TLS transit security with the CA Certificate Secret ARN field.](/images/rc/rdi/rdi-define-tls.png)
{width="80%"}

- **mTLS** (mutual TLS): Provide the CA certificate, client certificate, and client private key secret ARNs. Also provide the client key passphrase secret ARN if the key is encrypted.

{{<image filename="images/rc/rdi/rdi-define-mtls.png" alt="mTLS transit security with certificate, private key, and optional password secret ARN fields." width=80% >}}
![mTLS transit security with certificate, private key, and optional password secret ARN fields.](/images/rc/rdi/rdi-define-mtls.png)
{width="80%"}

Select **Validate** to check access to the selected source's secrets. Repeat this for each source. The AWS secret contents and permissions are described in [Share source database credentials]({{< relref "/operate/rc/rdi/setup#share-source-database-credentials" >}}).

Expand All @@ -111,13 +119,16 @@ Enter the selected source's database settings. The fields depend on the database
- **PDB**: The Oracle pluggable database.
- **Connection string**: The MongoDB connection information.

{{<image filename="images/rc/rdi/rdi-2-source-configuration.png" alt="Source-specific database, port, and collector properties." width=80% >}}
![Source-specific database, port, and collector properties.](/images/rc/rdi/rdi-2-source-configuration.png)
{width="80%"}

Use **Collector properties** for additional source and sink settings. These settings apply to the selected source. See the [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration" >}}).

{{<image filename="images/rc/rdi/rdi-source-configuration-collector-properties.png" alt="The Edit advanced properties control under Collector properties." width=500px >}}
![The Edit advanced properties control under Collector properties.](/images/rc/rdi/rdi-source-configuration-collector-properties.png)
{width="500px"}

{{<image filename="images/rc/rdi/rdi-advanced-properties.png" alt="The advanced properties dialog with separate collector source and sink properties." width=80% >}}
![The advanced properties dialog with separate collector source and sink properties.](/images/rc/rdi/rdi-advanced-properties.png)
{width="80%"}

## Select data {#dataset}

Expand All @@ -130,17 +141,21 @@ Do not write data directly to keys managed by RDI. Changes from another applicat
1. Select a schema in **Schemas** to see its tables.
1. Select the tables to ingest in **Tables**.

{{<image filename="images/rc/rdi/rdi-dataset-schema-selected.png" alt="Selecting a schema shows its tables for ingestion." width=75% >}}
![Selecting a schema shows its tables for ingestion.](/images/rc/rdi/rdi-dataset-schema-selected.png)
{width="75%"}

1. Select a table to see its **Columns**, then select the columns to ingest.

{{<image filename="images/rc/rdi/rdi-select-columns.png" alt="Selecting a table shows its columns and the columns selected for ingestion." width=75% >}}
![Selecting a table shows its columns and the columns selected for ingestion.](/images/rc/rdi/rdi-select-columns.png)
{width="75%"}

1. If a table has no unique key, use the key control next to the column that identifies each record. Review any missing-key warnings before continuing.

{{<image filename="images/rc/rdi/rdi-dataset-missing-unique-key.png" alt="The missing unique key warning with the affected table and its columns." width=75% >}}
![The missing unique key warning with the affected table and its columns.](/images/rc/rdi/rdi-dataset-missing-unique-key.png)
{width="75%"}

{{<image filename="images/rc/rdi/rdi-unique-key-selected.png" alt="The key control beside the column used to identify a record." width=500px >}}
![The key control beside the column used to identify a record.](/images/rc/rdi/rdi-unique-key-selected.png)
{width="500px"}

1. Repeat for the other sources, then select **Continue**.

Expand All @@ -154,7 +169,8 @@ Transformation jobs are optional. Without a matching job, RDI writes records usi
1. Check the **Source name** assignment for each job. In a multi-source pipeline, `source.server_name` identifies the source the job reads from. Select the source name you chose during setup.
1. Review each job's validation status and correct errors.

{{<image filename="images/rc/rdi/rdi-2-transformation-jobs.png" alt="Example transformation jobs with their source assignments and validation status." width=100% >}}
![Example transformation jobs with their source assignments and validation status.](/images/rc/rdi/rdi-2-transformation-jobs.png)
{width="100%"}

1. Select **Continue to review & deploy**.

Expand All @@ -166,6 +182,7 @@ Review every source's connection, selected data, and transformation jobs, togeth

Select **Deploy pipeline** to start the pipeline. Each source performs its initial snapshot and then captures ongoing changes. Snapshot duration depends on the amount of selected data and source performance.

{{<image filename="images/rc/rdi/rdi-confirm-deploy.png" alt="The Deploy pipeline button." width=175px >}}
![The Deploy pipeline button.](/images/rc/rdi/rdi-confirm-deploy.png)
{width="175px"}

Open the pipeline's [Dashboard and Metrics tabs]({{<relref "/operate/rc/rdi/view-edit">}}) to follow progress for each source.
27 changes: 18 additions & 9 deletions content/operate/rc/rdi/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,19 +44,22 @@ This example creates one PostgreSQL source. You can [add more sources]({{< relre

1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab and select **Add pipeline**.

{{<image filename="images/rc/rdi/rdi-workspace-add-pipeline.png" alt="The Add pipeline control is available while the workspace is being created." width=80% >}}
![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png)
{width="80%"}

1. In **Settings**, select your target database and choose **Hash** or **JSON** as the default data structure, then select **Continue**.
1. In **Add sources**, select **PostgreSQL**.
1. Enter a **Source name** of your choice, for example `inventory-postgres`.
1. Select **Continue** to open **Configure source**.
1. Under **Source connectivity**, copy the **Role ARN**. Use it as `redis_privatelink_arn` in the Terraform configuration.

{{<image filename="images/rc/rdi/rdi-setup-connectivity-arn.png" alt="The source connectivity Role ARN and availability zones." width=80% >}}
![The source connectivity Role ARN and availability zones.](/images/rc/rdi/rdi-setup-connectivity-arn.png)
{width="80%"}

1. Under **Secrets**, copy the **Role ARN**. Use it as `redis_secrets_arn` in the Terraform configuration.

{{<image filename="images/rc/rdi/rdi-credentials-arn.png" alt="The Role ARN in the Secrets section." width=80% >}}
![The Role ARN in the Secrets section.](/images/rc/rdi/rdi-credentials-arn.png)
{width="80%"}

1. Select **Save & exit** while you create the source resources.

Expand Down Expand Up @@ -117,32 +120,38 @@ If you lose any outputs, run `terraform output` to view them again.
1. Under **Source connectivity**, enter the Terraform `vpc_endpoint_service_name` output as the **Private Link service name**.
1. Select **Connect to Private Link** and wait for connectivity to complete.

{{<image filename="images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png" alt="AWS Private Link connectivity with the service name and Connect to Private Link control." width=80% >}}
![AWS Private Link connectivity with the service name and Connect to Private Link control.](/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png)
{width="80%"}

1. Under **Secrets**, enter the `secret_arn` output as **Credentials Secret ARN**.
1. Select **Validate** to check access to the secret.

{{<image filename="images/rc/rdi/rdi-source-configuration-secrets.png" alt="The Credentials Secret ARN field, transit security options, and Validate control." width=80% >}}
![The Credentials Secret ARN field, transit security options, and Validate control.](/images/rc/rdi/rdi-source-configuration-secrets.png)
{width="80%"}

1. Under **Source configuration**, enter the `database` and `port` Terraform outputs in the matching fields.
1. Select **Test source** and correct any validation errors, then select **Continue**.
1. In **Select data**, select the schemas, tables, and columns to ingest. Review the selected key for each table.

{{<image filename="images/rc/rdi/rdi-dataset-schema-selected.png" alt="Selecting a schema shows its tables for ingestion." width=75% >}}
![Selecting a schema shows its tables for ingestion.](/images/rc/rdi/rdi-dataset-schema-selected.png)
{width="75%"}

{{<image filename="images/rc/rdi/rdi-select-columns.png" alt="Selecting a table shows its columns and the columns selected for ingestion." width=75% >}}
![Selecting a table shows its columns and the columns selected for ingestion.](/images/rc/rdi/rdi-select-columns.png)
{width="75%"}

1. Select **Continue** to open **Add transformations**. For this example, you can keep the default mapping without adding jobs.
1. Select **Continue to review & deploy**.
1. Review the source and target, then select **Deploy pipeline**.

{{<image filename="images/rc/rdi/rdi-confirm-deploy.png" alt="The Deploy pipeline button." width=175px >}}
![The Deploy pipeline button.](/images/rc/rdi/rdi-confirm-deploy.png)
{width="175px"}

The source first imports its selected data, then captures ongoing changes. Open the pipeline's **Dashboard** or **Metrics** tab to follow its progress.

The following example shows the metrics for one selected source in a pipeline with multiple sources.

{{<image filename="images/rc/rdi/rdi-2-metrics.png" alt="Metrics for one selected source in a pipeline, including snapshot progress and per-table record counts." width=80% >}}
![Metrics for one selected source in a pipeline, including snapshot progress and per-table record counts.](/images/rc/rdi/rdi-2-metrics.png)
{width="80%"}

See [View and edit data pipeline]({{<relref "/operate/rc/rdi/view-edit">}}) for source actions, dataset changes, and monitoring.

Expand Down
12 changes: 8 additions & 4 deletions content/operate/rc/rdi/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,14 +143,16 @@ Before you can set up your source connectivity and secrets, you need the AWS Acc
1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab.
1. Select **Add pipeline**, or resume an existing draft. To add a source to a running pipeline, select **Add source** on its **Dashboard**.

{{<image filename="images/rc/rdi/rdi-workspace-add-pipeline.png" alt="The Add pipeline control is available while the workspace is being created." width=80% >}}
![The Add pipeline control is available while the workspace is being created.](/images/rc/rdi/rdi-workspace-add-pipeline.png)
{width="80%"}

1. For a new pipeline, complete **Settings**, including the target database, and select **Continue**.
1. In **Add sources**, select the source type and enter a unique **Source name**. This name identifies the source in the pipeline configuration and transformation jobs. See [Add sources]({{< relref "/operate/rc/rdi/define#pipeline-setup" >}}) for naming rules.
1. Select **Continue** to open **Configure source**.
1. Under **Source connectivity**, copy the **Role ARN** and extract its AWS account ID.

{{<image filename="images/rc/rdi/rdi-setup-connectivity-arn.png" alt="The source connectivity Role ARN and availability zones." width=80% >}}
![The source connectivity Role ARN and availability zones.](/images/rc/rdi/rdi-setup-connectivity-arn.png)
{width="80%"}

The account ID is the number after `arn:aws:iam::`. For example, `arn:aws:iam::123456789012:role/redis-data-pipeline` contains account ID `123456789012`.

Expand All @@ -171,11 +173,13 @@ The following diagrams show the network setup for the different database setups:

- Database hosted on an AWS EC2 instance:

{{<image filename="images/rc/rdi/rdi-setup-diagram-ec2.png" alt="The network setup for a database hosted on an AWS EC2 instance." width=80% >}}
![The network setup for a database hosted on an AWS EC2 instance.](/images/rc/rdi/rdi-setup-diagram-ec2.png)
{width="80%"}

- Database hosted on AWS RDS or AWS Aurora:

{{<image filename="images/rc/rdi/rdi-setup-diagram-aurora.png" alt="The network setup for a database hosted on AWS RDS or AWS Aurora." width=80% >}}
![The network setup for a database hosted on AWS RDS or AWS Aurora.](/images/rc/rdi/rdi-setup-diagram-aurora.png)
{width="80%"}

Select the steps for your database setup.

Expand Down
Loading
Loading