Skip to content

feat: cross-engine case-insensitive data diff, column-level row-diff, and HTML report viewer - #128

Merged
iamvirul merged 8 commits into
mainfrom
feat/cross-engine-case-insensitive-data-diff
Sep 25, 2026
Merged

iamvirul merged 8 commits into
mainfrom
feat/cross-engine-case-insensitive-data-diff

Conversation

@iamvirul

@iamvirul iamvirul commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Description

This PR introduces cross-engine case-insensitive table discovery and row hashing, a new column-level row inspection engine (row-diff command and diff --row-diff flag), interactive HTML report viewer enhancements, and security vulnerability fixes for Go 1.25.13 and golang.org/x/text.

It specifically addresses real-world migration verification issues (such as PostgreSQL $\leftrightarrow$ Oracle pipelines) where default catalog case folding (lowercase um_user_attribute in PostgreSQL vs. UPPERCASE UM_USER_ATTRIBUTE in Oracle) previously resulted in false-negative "No differences found" results.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Documentation update
  • Test addition/update

Changes Made

1. Cross-Engine Case-Insensitive Discovery & Dialect Handling

  • Canonical Identifier Normalization: Table and column identifiers are matched case-insensitively using schema.CanonicalIdent across all database engines.
  • Dialect-Preserving Execution: PostgreSQL uses lowercase quoting ("customers"), Oracle uses UPPERCASE ("CUSTOMERS"), MySQL uses backticks (`customers`), and MSSQL uses brackets ([customers]).
  • Selective Cross-Engine Row Fetching: Added resolve.FetchConflictRowsCrossEngine with independent source/target connection drivers.

2. Column-Level Row Inspection Engine (row-diff)

  • New CLI Command: Added deepdiffdb row-diff with flags for --table, --key, --status (updated, added, removed, all), --limit, and --format (table or json).
  • Full Diff Integration: Added --row-diff flag to deepdiffdb diff to display side-by-side differing columns in terminal output.
  • Terminal & Report Formatting: Differences are highlighted with an asterisk (*) and saved to structured row_diff.json and human-readable row_diff.txt.

3. Interactive HTML Report Viewer Enhancements

  • Data Changes Tab: Opens as the default active view when data drift exists, featuring expandable rows that embed the complete side-by-side column comparison tables.
  • Dedicated Row Data Tab: Card view of every differing row across all tables with real-time table filtering and counters.
  • Conflicts Tab: Enhanced conflict cards with side-by-side column difference tables and status badges.
  • URL Hash Navigation: Supports #rowdiff, #data, and #conflicts deep linking on page load.

4. Documentation Site & Sample Project

  • Feature Documentation: Added website/docs/features/cross-engine-diff.md detailing cross-engine migration verification challenges, pipeline mismatch screenshots, and terminal illustrations.
  • Command Documentation: Added website/docs/commands/row-diff.md and updated website/docs/commands/diff.md.
  • Live HTML Sample Report: Updated website/static/samples/report.html and website/docs/features/html-reports.md.
  • Sample 18: Created samples/18-cross-engine-row-diff/ with Docker Compose for PostgreSQL 16 $\leftrightarrow$ Oracle XE 21c, initialization scripts (um_user, um_user_attribute), configuration, Makefile, and pre-generated diff outputs.
  • Changelog: Documented all changes and security updates in CHANGELOG.md under ## [Unreleased].

5. Security Vulnerability Fixes

  • Updated go.mod to Go 1.25.13 and ci.yml matrix/steps to 1.25.13, resolving 8 Go standard library CVEs.
  • Upgraded golang.org/x/text to v0.39.0 (resolves GO-2026-5970).
  • Updated .github/workflows/security.yml to use go-version-file: go.mod instead of hardcoded older Go versions.

Testing

  • Unit tests added/updated (tests/resolve/rowdiff_test.go, tests/resolve/fetch_test.go, tests/html/generator_test.go)
  • Manual testing performed with live Docker containers
  • Tested with PostgreSQL 16 (test-pg-1)
  • Tested with Oracle XE 21c (test-ora-1)
  • Tested with SQLite
  • Security vulnerability check verified via govulncheck ./... (0 vulnerabilities)

Test Results

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
ok  	github.com/iamvirul/deepdiff-db/tests/resolve	1.077s
ok  	github.com/iamvirul/deepdiff-db/tests/html	0.595s

Checklist

  • Code follows the project's style guidelines
  • Self-review completed
  • Comments added for complex code
  • Documentation updated (if needed)
  • No new warnings generated
  • Tests pass locally
  • All existing tests pass
  • Changes are backward compatible (if applicable)

Additional Notes

Tested against real cross-engine dataset (PostgreSQL $\leftrightarrow$ Oracle) with table name case mismatches (customers vs. CUSTOMERS, um_user vs. UM_USER). Verified that single-row inspection via --table and --key executes targeted queries without full table scans.

Summary by CodeRabbit

  • New Features
    • Added row-diff to inspect column-level differences in updated, added, and removed rows, with table, key, and status filters and text or JSON output.
    • HTML reports now include expandable row comparisons, a filterable Row Data tab, and column-level details for conflicts.
    • Cross-engine comparisons now match table and column names regardless of case. Added a PostgreSQL-to-Oracle migration-verification sample.
  • Documentation
    • Added command and feature guides for row-level and cross-engine comparisons, plus instructions for the new sample.
  • Chores
    • Updated Go and related security dependencies.

@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 145d2a3c-2801-4b27-a117-eb11bdb1613a

📥 Commits

Reviewing files that changed from the base of the PR and between ae4b79b and 4bc5c0b.

⛔ Files ignored due to path filters (3)
  • go.sum is excluded by !**/*.sum
  • website/static/img/cross-engine-pipeline-mismatch.jpg is excluded by !**/*.jpg
  • website/static/img/row-diff-terminal.svg is excluded by !**/*.svg
📒 Files selected for processing (27)
  • .github/workflows/ci.yml
  • .github/workflows/security.yml
  • CHANGELOG.md
  • cmd/deepdiffdb/main.go
  • go.mod
  • internal/cli/display.go
  • internal/content/resolve/fetch.go
  • internal/content/resolve/rowdiff.go
  • internal/report/html/generator.go
  • internal/report/html/template.go
  • internal/report/html/types.go
  • samples/18-cross-engine-row-diff/Makefile
  • samples/18-cross-engine-row-diff/README.md
  • samples/18-cross-engine-row-diff/deepdiffdb.config.yaml
  • samples/18-cross-engine-row-diff/docker-compose.yml
  • samples/18-cross-engine-row-diff/init-scripts/01-postgres-init.sql
  • samples/18-cross-engine-row-diff/init-scripts/02-oracle-init.sql
  • tests/resolve/fetch_test.go
  • tests/resolve/rowdiff_test.go
  • website/docs/commands/diff.md
  • website/docs/commands/index.md
  • website/docs/commands/row-diff.md
  • website/docs/features/cross-engine-diff.md
  • website/docs/features/html-reports.md
  • website/docs/samples/index.md
  • website/sidebars.js
  • website/static/samples/report.html
 __________________________________________
< Every day I'm shufflin'... through code. >
 ------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@iamvirul
iamvirul merged commit f472ed0 into main Sep 25, 2026
17 checks passed
@iamvirul iamvirul mentioned this pull request Sep 25, 2026
15 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant