feat: Support difference_cols for upsert update detection

[hedger9487]

Closes #3598

Rationale & Motivation

During Table.upsert(), matched rows are compared cell-by-cell in pure Python (upsert_util.get_rows_to_update) to determine whether any non-key columns have changed, avoiding unnecessary rewrite IO.

However, for wide tables (e.g. 50–200 columns), comparing every single non-key column across matched rows incurs substantial CPU overhead:
In production, the vast majority of matched rows are unchanged. Determining that a row has not changed requires scanning all $M$ non-key columns ($O(N \times M)$).

In workflows where users maintain dedicated change-tracking columns in their tables (such as an upstream row hash, update timestamp, or version column), or only care about changes to specific business fields, comparing every other column is unnecessary.

This PR introduces the optional difference_cols parameter to allow users to specify which non-key columns determine whether a matched row is considered changed. Changes to columns outside difference_cols are intentionally ignored for update detection.

---

Key Invariants & Design

1. Detection Authority with Full-Row Writes:
   difference_cols only restricts which columns are inspected to determine if a matched row has changed. When a row is detected as changed, all columns of the source row are written, preserving complete row data integrity.

2. Fail-Fast Validation (validate_difference_cols):

   • Defensively rejects empty lists (difference_cols=[]) with: "difference_cols must contain at least one column; use None to compare all non-key columns".
   • Rejects non-existent columns with deterministic error messages.
   • Rejects columns that overlap with join_cols, since join columns are guaranteed to match for matched rows and therefore cannot provide meaningful change detection.
   • Validated upfront in Transaction.upsert to fail fast before performing storage or network operations.

3. Backward Compatibility:

   • Defaults to difference_cols=None, which preserves the existing change-detection behavior of checking all non-key columns.
   • Existing callers that do not provide difference_cols require no changes.

---

Benchmark Results (10 Measured Trials)

Measured using in-memory Arrow tables isolated to get_rows_to_update.
Note: The benchmark measures change-detection overhead only and excludes catalog, filesystem, and Parquet I/O.

• Hardware / OS: macOS (Apple Silicon ARM64)
• Environment: Python 3.14.7, PyArrow 25.0.1
• Dataset: 10,000 rows $\times$ 200 string columns (95% unchanged rows, 5% changed rows; 3 warmup iterations, 10 measured trials)
• Control: For the changed rows, col_0 was modified so that the same 500 rows were identified as changed under both configurations.

Metric	Baseline (All 200 columns)	With difference_cols=["col_0"]	Improvement
Execution Time (Mean)	34.1514s ($\pm 0.3602$s)	0.6358s ($\pm 0.0027$s)	53.71x faster
Execution Time (Median)	34.1264s	0.6358s	53.68x faster (98.1% reduction)
Changed Rows Detected	500 rows	500 rows	Identical set detected

---

Proposed Changes

1. pyiceberg/table/upsert_util.py

• Added validate_difference_cols(column_names, join_cols, difference_cols): validates non-key constraint, non-emptiness, and column existence.
• Updated get_rows_to_update: uses difference_cols when provided; otherwise falls back to existing list(all_columns - join_cols_set).

2. pyiceberg/table/__init__.py

• Added difference_cols: list[str] | None = None parameter and docstrings to Table.upsert and Transaction.upsert.
• Added upfront validate_difference_cols before remote data scan execution.

3. tests/table/test_upsert.py

Added 9 comprehensive unit and integration tests:

• test_get_rows_to_update_with_difference_cols: Verifies that only difference_cols are used for change detection and that changed rows retain all source columns.
• test_get_rows_to_update_difference_cols_validation: Verifies ValueError on empty list, unknown column, and join key overlap.
• test_get_rows_to_update_with_multiple_difference_cols: Verifies composite difference columns (["col1", "col2"]).
• test_get_rows_to_update_difference_cols_with_nulls: Verifies None $\leftrightarrow$ value transitions.
• test_get_rows_to_update_difference_cols_ignore_other_columns: Verifies that changes to non-difference columns are ignored for update detection (NOOP).
• test_upsert_with_difference_cols: End-to-end catalog upsert test verifying rows skipped, updated, and inserted with actual Parquet files, confirming full-row writes.
• test_upsert_schema_evolution_with_difference_cols: Verifies interaction between table schema evolution and difference_cols.
• test_upsert_with_invalid_difference_cols: Verifies fail-fast rejection via table.upsert.
• test_upsert_transaction_with_difference_cols: Verifies transaction support.

---

Verification Checklist

• uv run prek run -a passes cleanly.
• All tests in tests/table/test_upsert.py pass.
• All tests in tests/table/ pass locally.
• Added 9 comprehensive unit and integration tests covering validation, nulls, ignored columns, transactions, and schema evolution.
• Rigorous statistical benchmark demonstrates 53.68x speedup on wide tables.
• Backward compatible by default.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.