Fix upsert after schema evolution (#3105)

[hedger9487]

Closes #3105

Rationale for this change

When an Iceberg table undergoes schema evolution (e.g. update_schema().add_column(...) or update_schema().union_by_name(...)), calling table.upsert() previously failed with:

ValueError: Target schema's field names are not matching the table's field names: ['id', 'name', 'age', 'city', 'ping'], ['id', 'name', 'age', 'city']

Root Cause

In pyiceberg/table/upsert_util.py (get_rows_to_update), Step 1 previously cast the entire source_table (which has the new schema including evolved columns) to target_table.schema (which only contains columns present in older data files). PyArrow requires column names to match exactly when casting an entire table, raising ValueError.

Fix

1. In Step 1, cast only the join_cols to the target table's join schema (join_schema = pa.schema([target_table.schema.field(col) for col in join_cols])).
2. In Step 4, safely retrieve non-key values from target_row as None if the column was added during schema evolution and absent in target_table.
3. Add regression tests covering upsert after add_column, union_by_name, multiple schema evolutions with composite keys, and no-op null comparisons, while explicitly verifying that underlying Parquet data files are properly replaced (Copy-on-Write) and snapshot operations include OVERWRITE and APPEND.

Are these changes tested?

Yes:

• Added test_upsert_after_schema_evolution (verifies file replacement and snapshot operations)
• Added test_upsert_after_schema_evolution_union_by_name
• Added test_upsert_after_multiple_schema_evolutions_with_composite_keys
• Added test_upsert_after_schema_evolution_noop_and_nulls
• Verified mutation test fails without the fix and passes with the fix.
• All 12 pre-commit linters and 471 table tests pass cleanly.

Are there any user-facing changes?

No.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.