Search before asking
Paimon version
master (d77efe0). Also 1.3.x and 1.4.x, introduced by #5972.
Compute Engine
Flink 1.20.1, kafka_sync_table with canal-json.
Minimal reproduce step
- Catalog with
case-sensitive=false (the Hive catalog default). Create the table first: orders (id INT, create_time TIMESTAMP(3), dt STRING, PRIMARY KEY (id, dt) NOT ENFORCED) PARTITIONED BY (dt).
- Start
kafka_sync_table on an empty canal-json topic with --computed_column 'dt=date_format(create_time,yyyy-MM-dd)'. The table exists and no record is available, so the job builds the computed column from the Paimon schema with caseSensitive=false.
- Produce one record with
create_time = 2023-03-23 10:15:00.
What doesn't meet your expectations?
The record lands in partition dt=2023-03-82 instead of dt=2023-03-23, with no error. sortComputedColumnArgs upper-cases the whole argument, so the pattern becomes YYYY-MM-DD (week year, day of year).
Two more effects of the same upper-casing: with lower-case source column names the job restarts on every record with Cannot write null to non-null column(dt), because the upper-cased reference does not match the record; and a computed column referencing another computed column fails with Referenced field '...' is not in given fields.
Anything else?
Introduced by #5972, which upper-cases the whole argument before parsing it.
Are you willing to submit a PR?
Search before asking
Paimon version
master (d77efe0). Also 1.3.x and 1.4.x, introduced by #5972.
Compute Engine
Flink 1.20.1, kafka_sync_table with canal-json.
Minimal reproduce step
case-sensitive=false(the Hive catalog default). Create the table first:orders (id INT, create_time TIMESTAMP(3), dt STRING, PRIMARY KEY (id, dt) NOT ENFORCED) PARTITIONED BY (dt).kafka_sync_tableon an empty canal-json topic with--computed_column 'dt=date_format(create_time,yyyy-MM-dd)'. The table exists and no record is available, so the job builds the computed column from the Paimon schema withcaseSensitive=false.create_time = 2023-03-23 10:15:00.What doesn't meet your expectations?
The record lands in partition
dt=2023-03-82instead ofdt=2023-03-23, with no error.sortComputedColumnArgsupper-cases the whole argument, so the pattern becomesYYYY-MM-DD(week year, day of year).Two more effects of the same upper-casing: with lower-case source column names the job restarts on every record with
Cannot write null to non-null column(dt), because the upper-cased reference does not match the record; and a computed column referencing another computed column fails withReferenced field '...' is not in given fields.Anything else?
Introduced by #5972, which upper-cases the whole argument before parsing it.
Are you willing to submit a PR?