Skip to content

[cdc] Fix computed column args upper-cased in case-insensitive mode - #9717

Merged
JingsongLi merged 1 commit into
apache:masterfrom
NestDream:fix-cdc-computed-column-case
Sep 11, 2026
Merged

[cdc] Fix computed column args upper-cased in case-insensitive mode#9717
JingsongLi merged 1 commit into
apache:masterfrom
NestDream:fix-cdc-computed-column-case

Conversation

@NestDream

Copy link
Copy Markdown
Contributor

Purpose

fix #9716

When the catalog is case-insensitive, ComputedColumnUtils.sortComputedColumnArgs (introduced in #5972, shipped in 1.3.0) upper-cases the whole --computed_column argument before parsing it. Three things break:

  • Literals change case. dt=date_format(create_time,yyyy-MM-dd) is built with YYYY-MM-DD (week year, day of year): a record with create_time = 2023-03-23 10:15:00 lands in dt=2023-03-82 with no error. cast(hello, STRING) becomes HELLO.
  • The field reference is upper-cased and the parsers look it up in the source record by exact name, so it only matches upper-case source columns. With lower-case columns the computed value is null; for a partition key the job fails on every record with Cannot write null to non-null column(dt).
  • A computed column referencing another one fails with Referenced field '_year' is not in given fields: the type is registered under the upper-cased name while ReferencedField looks it up lower-cased.

This is the path where the Paimon table already exists and the schema cannot be read from the source (Kafka/Pulsar with an empty topic at startup), the only caller that passes caseSensitive to buildComputedColumns. Hive catalogs are case-insensitive by default, JDBC catalogs always. The Javadoc of buildComputedColumns already says field names are not changed at building phase; #5972 broke that.

Fix:

  • sortComputedColumnArgs lower-cases only the keys used for the dependency sort (the form ReferencedField uses) and keeps names and arguments as typed. buildComputedColumns registers a computed column's type under the lower-cased name.
  • ComputedColumn.evalFromRecord(Map) matches the referenced field by exact name, then ignoring case when the catalog is case-insensitive. The record's column case is only known at runtime, and without this step removing the upper-casing would break a lower-case reference against upper-case source columns, which works today. The four parsers call it; MySQL, Postgres and MongoDB always build with caseSensitive=true, so nothing changes for them.

Tests

  • ComputedColumnUtilsTest: literals kept as typed, cross-reference between computed columns written in different case, evalFromRecord for both catalog modes. The first two fail on master.
  • KafkaCanalSyncTableActionITCase#testComputedColumnWithCaseInsensitive adds _DATE_STR=date_format(_DATE,yyyy-MM-dd); the triggerSchemaRetrievalException=true case times out on master and passes here.
  • The existing computed column ITCases for Kafka, MySQL, Postgres and MongoDB pass locally.
  • Reproduced on a Flink 1.20.1 standalone cluster with Kafka: table created first in a case-sensitive=false catalog, kafka_sync_table started on an empty topic, one canal-json record. Master writes dt=2023-03-82 (upper-case source columns) or restarts on every record with Cannot write null to non-null column(dt) (lower-case source columns). With the fix both write dt=2023-03-23.

sortComputedColumnArgs upper-cased the whole --computed_column argument
when the catalog is case-insensitive. date_format(ts,yyyy-MM-dd) was
built with the pattern YYYY-MM-DD (week year, day of year) and wrote
partitions like dt=2023-03-82, cast literals changed case, the field
reference only matched upper-case source columns, and a computed column
referencing another one failed the type lookup.

Lower-case only the keys used for the dependency sort, keep names and
arguments as typed, and match the referenced field in the record
ignoring case when the catalog is case-insensitive.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes a concrete CDC correctness problem: the case-insensitive startup path must not change date-format literals or lose a source column because its name uses a different case. I traced the existing-table/schema-retrieval fallback through dependency ordering, type lookup and the record parsers. Keeping literal text intact while normalizing lookup keys is consistent with those callers; exact record-key matches also correctly preserve NULL instead of falling through to a different key.

No blocking issue found. The five ComputedColumnUtilsTest cases pass with the changed classes on an isolated JDK 8 classpath. I inspected the Kafka integration regression but did not rerun the Kafka/Flink cluster tests.

@JingsongLi
JingsongLi merged commit a4c7882 into apache:master Sep 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] CDC computed column arguments are upper-cased when the catalog is case-insensitive

2 participants