feat(auth): authorize query-auth reads and carry the grant on the split

[plusplusjiajia]

Purpose

A query-auth.enabled table makes the server return a per-user row filter and column masking that the client is expected to apply. This client cannot, so it refuses to read such a table at all — even for a user the server reports as unrestricted. This slice authorizes at scan-plan time and carries the result to the read, so that user can read. A user with rules is refused as before.

Brief change log

TableScan::plan authorizes once and stamps every split, as Java wraps each one in a QueryAuthSplit; to_arrow then requires every split to carry an unrestricted grant issued to this handle. Whether the table is query-auth comes from the server, not the option cached at load — every entry that reads without planning asks too.

Five refusals are deliberate: a restricted grant, at planning, since a plan already carries row counts and bounds; a time-travelled, branch or decorated handle, which reads files the server did not rule on; a read type the current schema does not contain, catching an older nested shape while a projection still reads; old-file statistics for a dropped column; and a reserved system column, which the server's own check
would reject.

Known limitation

Authorization is a plan-time decision and a plan is the capability recording it. A split kept from before the option was enabled, or built by hand, is read on the caller's word — as in Java, where unwrapQueryAuthSplit returns no result for a plain split. Callers must re-plan after an authorization change. A read-boundary guard could not change this: Table::file_io() and DataSplit::data_file_path()
are public.

Divergence from Java

Java serializes the auth result with the split; here it is runtime-only, so plan and read must share a process. Java's QueryAuthSplit trusts whatever it is handed, but to_arrow, Table::new and RESTEnv are public here, so the grant records a session only the catalog mints, and the auth call and the planning that follows are bracketed by a uuid and schema-id freshness check.

Tests

A unit test per guard, and mock-server tests for the outcomes: an unrestricted user reads real rows, a restricted one is refused at planning, and a re-created, evolved, assembled or decorated handle is refused.
Each was verified to fail with its guard removed.

API and Format

No format change. One new public method, Table::ensure_read_authorized, for DataFusion's system tables, which read metadata without planning. ReadBuilder::new_read no longer refuses at construction; that moved to to_arrow. Planning an ordinary REST table costs one extra get_table; a query-auth table costs four round-trips.

[JingsongLi]

[P1] Apply the live server check to direct search APIs too

This helper closes the stale-handle gap for TableScan, but the direct scored/search entry points still call only CoreOptions::ensure_read_authorized() on the schema cached when the handle was loaded. In particular, BatchVectorSearchBuilder::execute reads the snapshot/index manifest directly, and VectorSearchBuilder::execute_scored, FullTextSearchBuilder::execute_scored, and HybridSearchBuilder::execute_scored reach those direct paths without an authorized TableScan.

Therefore: load a REST table while query auth is false, enable restricted query auth on the server, then reuse the handle for one of these searches. The cached guard passes and row IDs/scores derived from protected data are returned without the auth exchange. Please route every out-of-band search entry through this async server-state check and reject when query auth is enabled (these paths cannot apply masking/filtering), with stale-handle regressions analogous to the new scan test.

[plusplusjiajia]

@JingsongLi Good catch — real hole. Fixed and widened past the searches: fourteen sites now ask the server via Table::ensure_read_authorized_live, and delegated searches ask once rather than once per route.

[JingsongLi]

[P1] Validate the full nested field shape, not only the top-level pair

Both this guard and the old-file check above compare only the top-level (id, name). That leaves a concrete disclosure path after nested schema evolution: suppose the authorized current schema contains profile ROW<public>, while a live older file has the same top-level field id/name but profile ROW<public, secret>. A caller can use the public ReadBuilder::with_read_type with that old Row type; this check passes, and data_file_reader::prune_data_type recursively selects the requested old child by id, so profile.secret is decoded even though it is absent from the schema/column set the server authorized. The planning check at lines 87-92 also passes the old file for the same reason.

Please validate canonical fields recursively (including Row children and nested Array/Map/Multiset element types, allowing only explicitly safe evolution), and make the old-file containment check recursive too. An end-to-end test with a dropped nested field and a crafted old read type should be rejected.

[plusplusjiajia]

@JingsongLi Good catch. Both checks now compare id, name and full data type; DataType derives structural equality, so nested shapes are covered recursively. End-to-end test as you asked: a Row column read with a read type that keeps the authorized (id, name) but carries an extra nested child — refused.

[JingsongLi]

[P1] Do not accept an unmarked split from a stale REST handle

When a REST handle was loaded while query auth was false, core_options.query_auth_enabled() stays false. If the server later enables restricted query auth, a caller that bypasses TableScan can still pass an old unmarked plan—or a split made through the public DataSplitBuilder—to this synchronous to_arrow boundary. required is false, so the method returns rows without any auth RPC or grant. This contradicts the PR’s stated stale-handle guarantee; the new test covers planning after the toggle, but not this public read path.

Please make REST reads require evidence that the split was planned after a live server-state check even when that check said query auth was disabled (for example, a catalog-session-bound checked-plan capability distinct from query_auth_required), or make materialization able to re-authorize. Unmarked splits should remain acceptable only where no REST authorization boundary exists. Add a regression for: load with false, retain/build an unmarked split, enable a restricted response, then call to_arrow and require refusal.

[plusplusjiajia]

@JingsongLi The gap is real, but I could not find a fix worth its cost, so I documented it here and in the PR description.

I implemented your suggestion first. It broke four legitimate flows on ordinary, non-query-auth REST tables — including DataFusion's register_cow_target_table and the documented C paimon_plan_from_split_bytes round trip — because it required enumerating every path that rebuilds a planned split, and I missed several. It also proved nothing: DataSplit is publicly serde-serializable, so {query_auth_checked: true, query_auth_required: false} skips both /auth and grant validation.

And closing it buys little: Table::file_io() and DataSplit::data_file_path() are public, so the same caller can read the bytes without touching TableRead. A real boundary means binding data tokens to an authorization epoch at the data plane — server-side work not available to us. Java stops here too.

So the contract is plan-time: callers must re-plan after an authorization change. A split from a query-auth table still fails closed if it lost its grant.

Do you have a better idea?

[JingsongLi]

[P1] Re-authorize before the writer performs lazy reads

This live check runs only when commit starts, but TableWrite has already read table state by then. TableWrite::new still checks only the schema cached on the handle; on the first normal PK write, create_kv_writer calls scan_partition_sequence_numbers, which reads the latest snapshot/manifests, and a dynamic-bucket write additionally runs DynamicBucketAssigner::ensure_index_entries_loaded plus HashIndexFile::read. Thus a handle loaded while query-auth.enabled=false can be reused after the server enables restricted query auth: write_arrow_batch and prepare_commit read protected metadata/index contents before this check eventually rejects the commit. The dynamic path can even emit a replacement hash-index file containing hashes restored from the protected index, so rejecting only at commit does not undo the disclosure. Please put a live authorization check before the first async writer read (or add an auth-aware async initialization) and cover the stale-handle write path.

[plusplusjiajia]

@JingsongLi You're right that the commit check comes too late. write_arrow_batch and prepare_commit now ask before the writer's first lazy read, so the snapshot scan and the hash-index load sit behind it.

[JingsongLi]

[P1] Do not trust a disabled answer from a replacement table

When the cached schema is false, this accepts the current name’s schema without checking that it still has rest_env’s UUID. A stale handle can therefore miss the very transition this helper is meant to catch: load table A with auth disabled, enable restricted query auth for A, then drop/re-create the name as table B with auth disabled. current_table() now reports B’s false, ensure_read_authorized_live succeeds, and direct vector/full-text/index paths read A’s still-reachable files through the stale handle without any auth exchange. The comment says the live answer only strengthens, but a false from a different UUID does not constrain A at all. Please validate the UUID before accepting a live false (schema freshness can remain a separate policy) and add a stale-false/recreated-table regression.

[plusplusjiajia]

@JingsongLi Agreed — a bare false proved nothing. It is now trusted only from the uuid this handle was loaded with; a different one errors and asks for a re-load, a missing one reads as true.

[JingsongLi]

[P1] Check the live branch schema, not the base-table name

For a table obtained through copy_with_branch, self.schema and the managers point at the branch, but rest_env.current_table() still queries the base identifier stored when the original handle was loaded. If the branch handle cached query-auth.enabled=false, the branch is later changed to restricted auth, and the base schema remains false, this method returns false; authorize_read(false) then exits before its branch_reference refusal and the scan reads the branch without an auth exchange. The current branch tests only start with a locally true option, so they do not cover this stale-false case. Please query branch_identifier(self.branch()) for live state (or conservatively refuse REST branch handles) and add a branch-specific toggle regression.

[plusplusjiajia]

@JingsongLi This was asking about the wrong table. It now asks about the branch, as db.t$branch_x, built from the base name so a handle already loaded as a branch doesn't double the decoration, with main mapping back.