RDSC-6142: Document external RDI collectors for proof of concept projects

[ZdravkoDonev-redis]

Document external collectors for RDI proof of concept projects

Readers can currently select type: external without an input contract or operating guidance. This change adds Use an external collector under Data pipelines → Prepare source databases, with navigation weight 2. It explains the recommended PoC scope, production downsides, integration options, stream/event contract, configuration, transformation job, and a complete Python writer. It also explains the differences in source controls and visibility and links the guide from current source guidance, existing mentions, and relevant CLI references.

The recommendation does not declare external collectors unsupported. The same pipeline configuration and deployment command apply to self-managed VM and Kubernetes installations. Cloud setup and the 1.19.1 snapshot are unchanged.

Version requirement: The guide now requires self-managed RDI 2.2.0 or later, which includes the connectionless external-source deployment fix. Earlier versions reject connectionless external sources during deployment validation. The validation evidence below records the versions actually tested; it does not claim a new RDI 2.2.0 qualification.

Validation

• The exact inline configuration and job pass the RDI 2.0.0 configuration and job schemas and the released API's common validation. The inline Python writer matches the script used in runtime tests.
• A disposable Redis Enterprise instance and the released redis/rdi-flink-processor:2.0.0 image passed read, create, update, and delete operations, transformation/key routing, the resulting hash after a repeated create, malformed JSON and missing key/value rejection to dead-letter queues, schema/payload wrappers with schema: null, and discovery/routing of another source stream. These are processor data-path checks, not a complete installation/deployment qualification or general delivery-guarantee tests.
• Stopping the test processor left the external writer able to add events while target results stayed unchanged. After processor restart, buffered events and new updates were processed.
• Follow-up validation reproduced the deployment regression in the full released validate_config service path with the documented job, both with and without table validation. Disabling table validation does not bypass the issue.
• The released redis/rdi-api:2.0.0 service functions reject per-source external start, stop, and reset with HTTP 400, and metadata discovery with HTTP 423. Kubernetes access was mocked; these are service checks, not an end-to-end CLI/control-plane qualification. Whole-pipeline stop/reset remained accepted at that layer.
• The focused RDI Hugo build passes with the CI Hugo version 0.143.1. Navigation order, added rendered references/anchors, and code blocks were checked. Browser inspection covered the lead recommendation, navigation, input contract, and example layout. git diff --check passes.

Limits and review notes

• A complete Kubernetes installation and whole-pipeline CLI stop/start/reset/removal were not exercised. The installed host CLI could not run on this architecture. Runtime checks used the released processor directly, with checkpointing and properties aligned to the released chart. No runtime or schema changes are included.
• The writer's runtime checks used the isolated database's non-TLS endpoint. TLS and mutual TLS configuration branches are included but were not tested end to end.
• The full site build is blocked by missing generated client example files (jupyter-example) outside this change. The focused RDI build passes.
• The configuration schema currently marks external with x-internal-values: ["external"]. This change leaves the marker, generated schema visibility, and runtime behavior unchanged. Review this deliberate public guidance against that marker before publication.
• Input stream ownership was checked separately: an external source has no collector-owned input stream manifest. The guide therefore does not promise that reset or source removal deletes externally written streams.

Parallel qualification evidence

Reviewed RDSC-6142 and RDI PR #2875 at 928f605ac3a0647087f94f7bbfad8e1cf665d84f. Its Helm/Flink/single-source restrictions are smoke-runner constraints, not new product restrictions. The ticket and PR report a fresh-machine Helm installation with Redis TLS/mTLS, connectionless deployment, no managed source collector, exact target checks for all four operations, and actual whole-pipeline stop/start with queued delivery. That qualification used a development build with the API fix applied, not the exact PR commit or the released 2.0.0 images. Exact-commit smoke CI was pending at review time; the implementation PR is open. This is reference evidence and was not rerun in this docs task.

The public guide now explains manual VM configuration after installation, separates writer credentials from target secrets, and creates the pipeline before setting target secrets on installations without a default pipeline. It does not link private implementation evidence from the public pages. Full VM qualification remains unconfirmed.

---

Note

Low Risk
Documentation-only changes to RDI integration content; no application code, APIs, or runtime behavior modified.

Overview
Adds a new Use an external collector guide under Prepare source databases, framing type: external as a proof-of-concept path (not recommended for production) and documenting the Debezium-compatible Redis stream contract, PoC setup (config.yaml, jobs, deploy/secrets flow), a sample Python XADD writer, verification steps, and how operations differ from RDI-managed collectors.

Existing RDI pages and CLI references are updated to link to the guide and clarify behavior: PoC callouts on overview/architecture/pipeline entry points, target-only secrets and deploy ordering for external sources, expanded external source config notes, per-source start/stop/reset limitations (including --source), no redis-di scaffold template, and release-note pointers to the new doc.

^{Reviewed by Cursor Bugbot for commit 15d02a2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Staging links:
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/architecture/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/data-pipelines/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/data-pipelines/deploy/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/data-pipelines/multiple-sources/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/data-pipelines/pipeline-config/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/data-pipelines/prepare-dbs/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/data-pipelines/prepare-dbs/external-sources/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/reference/cli/redis-di-reset/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/reference/cli/redis-di-scaffold/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/reference/cli/redis-di-start/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/reference/cli/redis-di-stop/
https://redis.io/docs/staging/docs/rdi-external-collectors-poc/integrate/redis-data-integration/release-notes/rdi-1-8-0/

[github-actions]

🧠 Redis Memory

Found 8 related items from repository history:

• PR — Improve RDI pipeline configuration guide

  What changed

• PR — RDSC-6038: Add RDI 2.0.0 release notes

  Adds the September 2026 RDI 2.0.0 release notes and changes rdi_current_version from 1.19.1 to 2.0.0. The site currently has no 2.0.0

• PR — RDSC-6045: Document draining streams before Flink processor migration

  Switching an existing pipeline from the classic processor to the Flink processor while input streams still contain records can leave records

• PR — DOC-6833: Fix four RDI Cloud doc issues from #rdi-cloud

  • Remove Active-Active from the 'multiple targets' use case sentence; add it as an explicit unsupported target topology under 'When not to u

• PR — RDSC-6038: Update Redis Cloud RDI documentation for 2.0.0

  Updates the Cloud RDI guides for the RDI 2.0.0 console. The current guides describe one source and the older five-step setup. The updated gu

• Commit — Document external RDI collectors for proof of concept projects

  Document external RDI collectors for proof of concept projects

• Commit — RDSC-6142: Clarify external source setup and deployment limitation

  RDSC-6142: Clarify external source setup and deployment limitation

• Commit — RDSC-6142: Move external collector guide under source preparation

  RDSC-6142: Move external collector guide under source preparation

Memory updated at 15d02a2