feat(datafusion): support the $audit_log system table by jerry-024 · Pull Request #787 · apache/paimon-rust

jerry-024 · 2026-09-03T07:24:52Z

Purpose

Add Paimon Java-compatible support for querying the $audit_log system table from DataFusion and the core read API.

The table exposes each row's physical row kind and, when enabled in the persisted table options, its sequence number. It preserves current-state merge semantics for primary-key tables instead of dropping winning retract rows.

Brief change log

Register <table>$audit_log in the DataFusion system-table loader.
Expose rowkind as the first column and optional _SEQUENCE_NUMBER after it.
Support projection, predicate pushdown with exact residual filtering, limits, runtime filters, and time travel.
Add current-state audit reads for append-only and primary-key tables.
Support deduplicate, first-row, partial-update, and aggregation merge paths, including ignore-delete behavior.
Preserve Java-compatible deletion-vector visibility: rows hidden by deletion vectors are not reconstructed as retract records.
Reject dynamically enabling table-read.sequence-number.enabled, matching Paimon Java; persisted table configuration remains supported.
Bound merge fan-in and share the Parquet read budget across audit input paths.

Tests

cargo +1.94.0 test -p paimon --test audit_log_table_test - 20 passed
cargo +1.94.0 test -p paimon-datafusion --test system_tables - 22 passed
cargo +1.94.0 clippy -p paimon-datafusion --lib --test system_tables -- -D warnings
cargo fmt --all
git diff --check

Coverage includes append-only and primary-key row kinds, sequence numbers, merge engines, ignore-delete, projection/filter/limit behavior, dynamic time travel, fail-closed authorization behavior, deletion vectors, and Java-compatible dynamic-option validation.

Query authorization compatibility

This PR is safe to merge as an incremental implementation:

With query authorization disabled, $audit_log works normally.
With query-auth.enabled = true, whether persisted on the table or enabled through session options, Rust rejects reads from the table and its system tables, including $audit_log.
This fail-closed behavior does not depend on whether the current user has a row filter or column masking rule, so no unfiltered data is exposed.
In deployments such as DLF where query authorization is enabled by default, $audit_log remains unavailable until the authorization integration is implemented.

One Java compatibility gap remains: Java applies row filters and column masking before returning $audit_log, while Rust currently rejects the query. Full parity can be implemented after #513 lands, together with its audit-log integration, and does not need to block #787.

API and Format

Adds the $audit_log system-table API and core audit-read support. No existing API is removed or changed incompatibly.

No storage-format change is introduced; existing Paimon data files, manifests, deletion vectors, and table options remain compatible.

Documentation

Updated docs/src/sql.md with $audit_log schema, query examples, row-kind semantics, sequence-number behavior, and deletion-vector visibility.

…em-table * upstream/main: fix(spec): align index manifest field order with Java (apache#797) fix(variant): sort object keys by UTF-8 bytes like Java (apache#789) feat(file_index): integrate pruning into raw data-file reads (apache#783) fix(spec): reserve 16 bytes for non-compact decimal in BinaryRow (apache#791) build: prepare the 0.4.0 release (apache#790) feat(datafusion): support the $consumers system table (apache#782) build: raise MSRV to Rust 1.94 (apache#788) feat(blob): support MAP<K, BLOB> reads in Go (apache#784) # Conflicts: # crates/paimon/src/table/table_read.rs

shyjsarah

I found three issues that should be addressed before merging: one credential-exposure risk in the query-auth system-table policy, one metadata-authorization regression, and one correctness issue in first-row audit scans.

shyjsarah · 2026-09-09T07:29:05Z

+    "file_key_ranges",
+    "binlog",
+    "statistics",
+];


Critical: This selective denylist makes $options and $schemas readable for a query-auth.enabled table. Both providers expose persisted schema options without redaction, and those options may contain object-store credentials such as secret keys, account keys, SAS tokens, or security tokens. The new test only verifies that dynamic session options are not persisted; it does not cover credentials already stored in the table schema. A restricted user could retrieve those credentials and access the backing storage directly, bypassing row filters and column masks. Please keep $options and $schemas fail-closed until sensitive option values are centrally redacted, and add a regression test using a credential persisted in the table WITH options.

shyjsarah · 2026-09-09T07:29:05Z

+    "file_key_ranges",
+    "binlog",
+    "statistics",
+];


Major: This also allows $partitions, $manifests, and $table_indexes for query-auth.enabled tables. These providers expose partition values or partition statistics without applying the base-table row filter or column masks. This weakens the previous fail-closed behavior and may disclose protected tenant or partition values. Please keep these system tables blocked until their output can be filtered or masked consistently with the base table.

shyjsarah · 2026-09-09T07:29:05Z

-            None,
-            self.case_sensitive,
-        )?))
+        if audit_log {


Major: bucket_round_robin distributes individual splits across DataFusion execution partitions, but first-row merging groups splits only within each partition's execute call. Splits belonging to the same (partition, bucket) group can therefore be processed independently and each emit a winner, resulting in duplicate keys or an incorrect first row. Please group splits by (partition, bucket) before assigning them to execution partitions. As a safe fallback, first-row audit scans could use a single execution partition. A DataFusion-level regression test with two same-bucket splits and target_partitions >= 2 would cover this boundary.

shyjsarah

+1

JingsongLi

Core support for retaining winning retract rows is necessary for $audit_log. I found two issues in the new read path and suggest consolidating the three TableRead audit methods, as detailed inline.

Validation: all 19 existing core audit tests passed locally. An additional first-row test using real Parquet rows and matching value statistics reproduced the incorrect result described below.

JingsongLi · 2026-09-10T06:29:30Z

+                        .collect(),
+                    read_batch_size: core_options.read_batch_size()?,
+                    keep_delete: true,
+                    merge_splits: merge_engine == MergeEngine::FirstRow,


[P2] Make scan pruning aware of first-row audit merging

This introduces a merge read path for first-row, but TableScan::stats_pruning_predicates still exempts first-row from key-only pruning. With with_scan_all_files(), two L0 files containing (id=1, value=10) followed by (id=1, value=20), and a value=20 predicate, file statistics can prune the first file before this merge runs. The audit then incorrectly returns (1,20); merging first would select (1,10), so the filtered result should be empty.

I reproduced this with real Parquet data and matching value statistics. Rust's current KV writer leaves value statistics empty, which is why the existing test does not expose it. This affects the L0-inclusive core path exercised by the new first-row test. Please make planning aware of this merge mode, retain only merge-safe key predicates for pruning, and apply non-key predicates after merging.

JingsongLi · 2026-09-10T06:29:30Z

+                read_type.clone(),
+                self.data_predicates.clone(),
+            )
+            .with_batch_size(Some(core_options.read_batch_size()?))


[P2] Preserve FileIndex configuration in the audit raw reader

This separate DataFileReader construction omits .with_file_index_read_enabled(core_options.file_index_read_enabled()). The reader defaults to false, whereas the ordinary new_data_file_reader() propagates the table option, whose default is true. As a result, audit reads of raw-convertible PK splits silently skip FileIndex pruning even when enabled, causing unnecessary data-file reads for indexed predicates.

Please share the reader construction/configuration with the ordinary path, or at least propagate this option here and cover it with an indexed raw-split test.

JingsongLi · 2026-09-10T06:29:30Z

+    }
+
+    /// As [`Self::to_audit_log_arrow_for_splits`], omitting unrequested system columns.
+    pub fn to_projected_audit_log_arrow_for_splits(


Could we consolidate the three TableRead audit methods into one?

There is already to_audit_log_arrow(&IncrementalPlan), and this PR adds both to_audit_log_arrow_for_splits and to_projected_audit_log_arrow_for_splits. These vary along two independent dimensions: read mode and projection.

A single to_audit_log_arrow could accept an input enum such as Current(&[DataSplit]) | Incremental(&IncrementalPlan), with standard From conversions to preserve existing ordinary calls. Keep current-state merging, delta/changelog reads, and before/after diff semantics explicit internally.

Projection should use the existing read-type mechanism, including the requested system fields and their order, rather than a separate public method with include_rowkind/include_sequence booleans. This needs to preserve the distinction between an unspecified projection (the existing default system columns) and an explicit projection, including zero columns.

The implementation can retain private mode-specific helpers while sharing reader configuration and output projection.

JingsongLi

Rechecked 7c305ef. The audit API consolidation and FileIndex configuration change address the earlier feedback, and the original first-row reproducer now passes. I reproduced two remaining issues, detailed inline. All 19 existing core audit tests and the additional default/explicit/empty/Diff projection checks passed locally.

JingsongLi · 2026-09-11T05:34:13Z

        if has_primary_keys
            && (!deletion_vectors_enabled || deletion_vectors_merge_on_read)
-            && !first_row
+            && (!first_row || self.scan_all_files)


[P2] Apply merge-safe pruning to audit scans with deletion vectors

new_audit_scan() now includes L0 files and merges visible key versions, but the preceding DV condition still exempts tables with deletion-vectors.enabled=true and deletion-vectors.merge-on-read=false from key-only pruning. Consequently, non-key value statistics can discard the winning version before the audit reader merges the remaining files.

I reproduced this with a deduplicate table containing two uncompacted L0 files: (id=1, value=10) followed by (id=1, value=20), with value statistics matching the actual Parquet rows. An unfiltered audit returns (1,20), but adding value=10 prunes the newer file and incorrectly returns (1,10) instead of an empty result.

Please make audit scans that merge key versions use merge-safe key predicates regardless of this DV exemption, and retain non-key predicates for post-merge filtering. Add coverage for this DV-enabled, merge-on-read-disabled case with populated value statistics.

JingsongLi · 2026-09-11T05:34:13Z

+        read_type.retain(|field| {
+            !matches!(
+                field.id(),
+                crate::spec::ROW_KIND_FIELD_ID | crate::spec::SEQUENCE_NUMBER_FIELD_ID
+            )
+        });


[P2] Preserve explicit system-field projections for ordinary reads

new_read() is also used by ordinary to_arrow() calls, so removing _SEQUENCE_NUMBER here changes their output even when no audit read is requested. On a PK table with sequence-number reads enabled, with_read_type([_SEQUENCE_NUMBER, id]).new_read().to_arrow(...) now returns only [id].

I verified this with a single raw-convertible data file: the same test passes against the original PR base 78994db0 and fails against 7c305efe, where the sequence column is silently omitted.

Please preserve the original read type when constructing a normal TableRead, and derive the user-only physical projection inside the audit-specific read paths. The new audit projection handling should not alter ordinary read projections.

JingsongLi

The changes are too extensive; is it possible to decouple the audit code from the standard scan and read operations?

jerry-024 added 2 commits September 3, 2026 13:04

feat: support audit log system table

8fc48ac

fix(datafusion): align audit dynamic options with Java

f50fa9a

jerry-024 marked this pull request as draft September 3, 2026 07:25

jerry-024 changed the title ~~feat: support audit log system table~~ feat(datafusion): support the $audit_log system table Sep 3, 2026

jerry-024 added 2 commits September 3, 2026 17:13

fix(datafusion): align system table authorization with Java

4376087

fix(datafusion): retain system table type validation

e16c7cb

jerry-024 marked this pull request as ready for review September 8, 2026 01:40

jerry-024 marked this pull request as draft September 8, 2026 01:40

jerry-024 added 3 commits September 8, 2026 14:31

perf(table): batch interleave diff output

46752b2

perf(table): deduplicate pinned diff batches

c98dbf3

jerry-024 force-pushed the feat/audit-log-system-table branch from 9dfc122 to 28c3db0 Compare September 9, 2026 05:57

jerry-024 marked this pull request as ready for review September 9, 2026 06:56

shyjsarah reviewed Sep 9, 2026

View reviewed changes

fix(datafusion): harden audit log planning and auth

d793270

shyjsarah approved these changes Sep 10, 2026

View reviewed changes

JingsongLi reviewed Sep 10, 2026

View reviewed changes

jerry-024 added 4 commits September 10, 2026 18:01

fix(table): align audit log read planning

a9e4e99

fix(table): harden audit log scan planning

cfc40aa

fix(table): make audit splits independently mergeable

0e86915

docs(table): clarify first-row audit split path

7c305ef

JingsongLi reviewed Sep 11, 2026

View reviewed changes

fix(table): preserve audit projection and pruning semantics

3af12be

JingsongLi requested changes Sep 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(datafusion): support the $audit_log system table - #787

feat(datafusion): support the $audit_log system table#787
jerry-024 wants to merge 13 commits into
apache:mainfrom
jerry-024:feat/audit-log-system-table

jerry-024 commented Sep 3, 2026 •

edited

Loading

Uh oh!

shyjsarah left a comment

Uh oh!

shyjsarah Sep 9, 2026

Uh oh!

shyjsarah Sep 9, 2026

Uh oh!

shyjsarah Sep 9, 2026

Uh oh!

shyjsarah left a comment

Uh oh!

JingsongLi left a comment

Uh oh!

JingsongLi Sep 10, 2026

Uh oh!

JingsongLi Sep 10, 2026

Uh oh!

JingsongLi Sep 10, 2026

Uh oh!

JingsongLi left a comment

Uh oh!

JingsongLi Sep 11, 2026

Uh oh!

JingsongLi Sep 11, 2026

Uh oh!

JingsongLi left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jerry-024 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

Query authorization compatibility

API and Format

Documentation

Uh oh!

shyjsarah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shyjsarah left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jerry-024 commented Sep 3, 2026 •

edited

Loading