[cdc] Fix computed column args upper-cased in case-insensitive mode - #9717
Conversation
sortComputedColumnArgs upper-cased the whole --computed_column argument when the catalog is case-insensitive. date_format(ts,yyyy-MM-dd) was built with the pattern YYYY-MM-DD (week year, day of year) and wrote partitions like dt=2023-03-82, cast literals changed case, the field reference only matched upper-case source columns, and a computed column referencing another one failed the type lookup. Lower-case only the keys used for the dependency sort, keep names and arguments as typed, and match the referenced field in the record ignoring case when the catalog is case-insensitive.
JingsongLi
left a comment
There was a problem hiding this comment.
This fixes a concrete CDC correctness problem: the case-insensitive startup path must not change date-format literals or lose a source column because its name uses a different case. I traced the existing-table/schema-retrieval fallback through dependency ordering, type lookup and the record parsers. Keeping literal text intact while normalizing lookup keys is consistent with those callers; exact record-key matches also correctly preserve NULL instead of falling through to a different key.
No blocking issue found. The five ComputedColumnUtilsTest cases pass with the changed classes on an isolated JDK 8 classpath. I inspected the Kafka integration regression but did not rerun the Kafka/Flink cluster tests.
Purpose
fix #9716
When the catalog is case-insensitive,
ComputedColumnUtils.sortComputedColumnArgs(introduced in #5972, shipped in 1.3.0) upper-cases the whole--computed_columnargument before parsing it. Three things break:dt=date_format(create_time,yyyy-MM-dd)is built withYYYY-MM-DD(week year, day of year): a record withcreate_time = 2023-03-23 10:15:00lands indt=2023-03-82with no error.cast(hello, STRING)becomesHELLO.Cannot write null to non-null column(dt).Referenced field '_year' is not in given fields: the type is registered under the upper-cased name whileReferencedFieldlooks it up lower-cased.This is the path where the Paimon table already exists and the schema cannot be read from the source (Kafka/Pulsar with an empty topic at startup), the only caller that passes
caseSensitivetobuildComputedColumns. Hive catalogs are case-insensitive by default, JDBC catalogs always. The Javadoc ofbuildComputedColumnsalready says field names are not changed at building phase; #5972 broke that.Fix:
sortComputedColumnArgslower-cases only the keys used for the dependency sort (the formReferencedFielduses) and keeps names and arguments as typed.buildComputedColumnsregisters a computed column's type under the lower-cased name.ComputedColumn.evalFromRecord(Map)matches the referenced field by exact name, then ignoring case when the catalog is case-insensitive. The record's column case is only known at runtime, and without this step removing the upper-casing would break a lower-case reference against upper-case source columns, which works today. The four parsers call it; MySQL, Postgres and MongoDB always build withcaseSensitive=true, so nothing changes for them.Tests
ComputedColumnUtilsTest: literals kept as typed, cross-reference between computed columns written in different case,evalFromRecordfor both catalog modes. The first two fail on master.KafkaCanalSyncTableActionITCase#testComputedColumnWithCaseInsensitiveadds_DATE_STR=date_format(_DATE,yyyy-MM-dd); thetriggerSchemaRetrievalException=truecase times out on master and passes here.case-sensitive=falsecatalog,kafka_sync_tablestarted on an empty topic, one canal-json record. Master writesdt=2023-03-82(upper-case source columns) or restarts on every record withCannot write null to non-null column(dt)(lower-case source columns). With the fix both writedt=2023-03-23.