[fupelaqu]

Summary

ARRAY<X> columns, derived-column typing and struct order in core (SCHEMA-6, after #455). A column declared ARRAY<X> reads back as ARRAY<X> (DESCRIBE, schema, type checks) and its values are lists. A column not declared a list that holds several values in a document now fails the statement and names the ALTER TABLE … SET DATA TYPE ARRAY<X> that declares it a list. That ALTER changes only _meta and runs in place. Derived-table and CTE columns are typed from their body, ARRAY_AGG is ARRAY<elem>, and sub-fields keep their declared order.

Closes #419

Covers #419's list read-back, derived / CTE typing, object lists merged field by field, sub-field order, and list value loss (a declared list reads lists, and an undeclared one fails with its remedy).

What changes

• Read-back and DDL: type names parse array<…> recursively; an inner list or ANY reads ANY. CHAR reads back CHAR. ARRAY<VARBINARY> maps to binary, and ARRAY<GEO_POINT> parses in DDL. One check refuses a list of lists in CREATE TABLE, ADD COLUMN, SET DATA TYPE, SET FIELD and a CTAS column.
• Declaring a field a list: SET DATA TYPE ARRAY<X> on a field of X's Elasticsearch type is not a type change. The _meta data_type / field_order keys are safe, so the update runs in place (no reindex, same uuid). Changing a list's element type, or turning a scalar into a list of another Elasticsearch type, stays Impossible.
• Several values in a scalar column fail. The message names the column, the _id, the hit's index (the ES 8 / 9 envelope now carries _index) and an ALTER that parses: SET DATA TYPE ARRAY<X>, or SET FIELD with the sub-field's own options. It is raised on the top-hits path too (FIRST_VALUE / LAST_VALUE / ARRAY_AGG). WHERE-subquery inner statements are unaffected. Main's text (ALTER COLUMN c TYPE ARRAY<...>) did not parse and named the pattern.
• A list of objects in a STRUCT column still answers its list. obj.a under it collects the elements' values in order (it was NULL), also beside obj or *.
• Element view: one function reads an ARRAY<X> column (X scalar) as X where Elasticsearch works per element: comparisons and their queries, the Painless type, GROUP BY key columns, MIN / MAX / SUM / AVG, function argument types. Declared positions keep ARRAY<X>: schema, DESCRIBE, CTAS, FIRST / LAST / ARRAY_AGG. PARTITION BY over a list reads as on 0.24.0. Where a script reads the whole value (ingest, transform, computed-column validation), it stays ARRAY<X>, so CREATE and ALTER judge a computed column alike, with 0.24.0's processor bytes.
• After the rebase onto feat(core): computed-only projections without _source, DuckDB numeric widths (SCHEMA-5) #455: ROUND / FLOOR / CEIL over ARRAY<X> report, render and guard overflow as over X. A quoted decimal beside ARRAY<INT> is sent rounded as beside INT (arr = '1.5' sends 2) in comparisons, IN, BETWEEN and HAVING key filters.
• ARRAY_AGG is declared ARRAY<elem> for describe, set-operation slots and CTAS (_meta ARRAY<KEYWORD>, mapping keyword).
• Struct order: _meta records field_order. Sub-fields read in that order, with unlisted ones after them. SET FIELD rewrites a sub-field in place.
• Derived / CTE typing: DerivedTable.attach types every derived body (CTE references included) by the projection-item rule. SearchApi.attachDerived (private[elastic]) serves it to the relational engine. It executes nothing. Core's refusal to run derived statements is unchanged, and plain update never types a derived branch.
• Docs: DDL and DQL pages.

Release notes

• ARRAY<X> columns report ARRAY<X> in DESCRIBE, the schema and JDBC metadata (were ANY). Values are lists; one value is a list of one.
• Several values in a column not declared ARRAY<X> fail and name the in-place ALTER. This includes 0.24.0 CTAS / view columns built from ARRAY_AGG (declared as the element): run the ALTER, or redeploy the view. A list of objects in a STRUCT column keeps answering, so a STRUCT column may hold a List. Its obj.a reads Elasticsearch's values (was NULL) and fails like a scalar when several.
• ARRAY_AGG is ARRAY<elem>. CTAS _meta follows (mapping unchanged).
• CHAR reads back CHAR (it read ANY).
• Sub-fields in declared order for tables created from 0.25.0. Older tables keep mapping order.
• ARRAY<VARBINARY> maps to binary (was object). ARRAY<GEO_POINT> is accepted in DDL. Lists of lists are refused.
• A list column used as a scalar is typed as its element, and the element's refusals apply. ad + 1 over ARRAY<DATE> (a Painless error before) now answers as over DATE.
• An ALTER judges computed columns as CREATE does (newly refused statements below). An ALTER adding c DATE SCRIPT AS (CAST(<list> AS DATE)) over ARRAY<KEYWORD|TIMESTAMP|INT> now writes CREATE's processor, which copies the value. 0.24.0's ALTER wrote .toLocalDate(), which cannot run on an ingest String or list, so it stored nothing.
• Fix: an ALTER on a table holding an ARRAY<X> column no longer re-maps it to object. On 0.24.0 and main, lists then failed to index; with documents, the reindex rollback failed and the index was deleted.
• No downstream rebuild: no case class changes arity. HitReading (private[client], not a case class) gains two defaulted constructor parameters.

Answered before, refused now (accepted)

ES 8.18.3 gate against v0.24.0; main answers the first six rows as v0.24.0 does. The computed-column rows were re-measured on the final bytes (verdicts and processor bytes, no cluster). Columns: arr ARRAY<INT>, arrd ARRAY<DOUBLE>, arrk ARRAY<KEYWORD>, ad ARRAY<DATE>, ats ARRAY<TIMESTAMP>, n INT, d DATE.

Statement	v0.24.0	This PR
Several values in a column not declared a list (15 cells): plain SELECT, SELECT *, LIMIT, scroll, index pattern, UNION ALL, window base, FIRST_VALUE, ARRAY_AGG; obj.a alone, beside obj, beside *; nested items.q; reading a CTAS and an INSERT … SELECT copy	the list as stored in a KEYWORD column; obj.a, items.q NULL	fails, naming column, _id, index and SET DATA TYPE ARRAY<KEYWORD> / SET FIELD a ARRAY<INT> / SET FIELD q ARRAY<INT>
arr ∪ d, arr ∪ n, arrk ∪ s, arrk ∪ d	one column of mixed classes (List, Integer, LocalDate, String)	refused by the set-operation rule (#420)
ARRAY_AGG(k) … GROUP BY g UNION ALL SELECT k …	lists beside strings	refused (ARRAY<KEYWORD> beside KEYWORD)
c ARRAY<ARRAY<INT>> by CREATE TABLE, ADD COLUMN, SET FIELD	an object field; indexing a list then failed	refused at validation, naming the column
CREATE TABLE … AS SELECT g, ARRAY_AGG(arr) AS l … GROUP BY g	0 rows written, 5 failures, l mapped object	refused at validation, naming l
arrk + 1	a1, b1, c1 (concatenation)	refused, as over KEYWORD
ALTER … ADD COLUMN, a computed column over a list (33 cells, below)	accepted (list read ANY), but the ALTER re-mapped the list to object; with documents the index was deleted	refused, naming the column, exactly as v0.24.0's CREATE refuses it

The 33 computed-column cells:

• numeric (14): arr + 1, arr * 2, arr - n, arr % 2, arr / 2, ABS(arr), SIGN(arr), SQRT(arr), POWER(arr, 2), LOG(arr), ROUND(arrd), ROUND(arrd, 1), FLOOR(arrd), CEIL(arrd);
• temporal (14): YEAR(ad), EXTRACT(YEAR FROM ad), DATE_ADD(ad, INTERVAL 1 DAY), DATE_SUB(ad, INTERVAL 1 DAY), ad + INTERVAL 1 DAY, ad * 2, DATE_TRUNC(ad, MONTH), LAST_DAY(ad), DATEDIFF(ad, d), DATEDIFF(d, ad), DATE_FORMAT(ad, '%Y'), MONTH(ats), ats + INTERVAL 1 HOUR, DATE_TRUNC(ats, DAY);
• NULLIF of the list with a value (4): NULLIF(arr, 0), NULLIF(arr, arr), NULLIF(n, arr), NULLIF(arrk, 'a');
• date arithmetic (1): ad - d.

Acceptance evidence

Check	Evidence
Round trip for the 18 X types: DESCRIBE, mapping, SHOW CREATE re-parse, lists; on a raw-mapping index, the failure, then the in-place ALTER (same uuid), then the lists	ArrayRoundTripSpec, RUN on ES 8.18.3 and 6.8.23
Failure text, hit and top-hits paths	MultiValuedFailSpec (unit); ES 8.18.3 gate
A list column used as X behaves as a scalar X column	ES 8.18.3: 80 statements, 5 element types, verdicts equal 80/80, no list from a computed item. Final bytes: request body identical to main for scalar and untyped columns (900/900); list = scalar on 73/75 (2 ORDER BY: unmapped_type only, as on 0.24.0). ES 8.18.3: '1.5' in =, IN, BETWEEN, >, HAVING gives the scalar's rows; ROUND(arr, -1) = ROUND(n, -1) (Long)
Computed columns over lists	132 SCRIPT AS shapes: ALTER verdict = CREATE verdict 132/132; CREATE verdict + bytes = v0.24.0 117/117; ALTER bytes = v0.24.0's 114/117 (3 = the CAST AS DATE rows); transform 53/53, UPDATE … SET 7/7 unchanged. RUN ES 8.18.3: the processors write what v0.24.0's write
Derived / CTE typing vs DuckDB 1.5.5 DESCRIBE	DerivedColumnTypeMatrixSpec (DuckDB answers embedded, divergences listed with reasons); JavaClientSchemaConsumerSpec ES 8.18.3 13/13, incl. #455's tests
Struct order, meta-only ALTER, lists of lists	StructFieldOrderSpec, ArrayDeclarationDiffSpec, ArrayTypeMappingSpec; ES 8.18.3 / 6.8.23 legs
Nothing else answered before is refused	ES 8.18.3 gate on v0.24.0, main, branch (263 to 277 cells each)
Final bytes	sql/test 2246/0, core/test 1429/0, bridges es6 360/0, es8 353/0, es9 353/0, sbt "+ compile" OK; ES 8.18.3 full es8java/test 879/0 (3 canceled: legacy templates, ES < 7.8); ES 6.8.23 67/0
CI lint line (headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck)	green on the committed bytes

Final bytes against main, one forked JVM per run, main and branch alternated twice per item (main, branch, branch, main). Each figure is the median of the per-run medians (18 timed blocks after 60 warm-up blocks). The 16-CPU machine was not quiet: the 1-minute load was 6 to 12 during the runs, so differences of a few percent are within noise.

Item	main	branch	branch / main
Plain SELECT of 10,000 scalar rows (control), ns/row	298	310	1.04 (first pair 1.00, second pair 1.08)
SELECT of ARRAY columns, 10,000 rows, ns/row	241	213	0.88
Schema built from a mapping (cache miss), µs	13.7	14.7	1.07
First read in a fresh JVM, ms	52.9	53.9	1.02
ES 8 response envelope, 10,000 hits, ns/hit (the branch also carries each hit's index)	45.3	31.1	0.69

For the ES 8 envelope, three runs disturbed by load (two on main, one on the branch) are left out; with them the medians are not comparable.

Downstream

Against a throwaway local publish, extensions, JDBC and arrow (main) compile and test-compile with only the edits #453 / #454 already require.

• arrow: DeclaredLeg* 11/0, 7/0, 5/0, 5/0; DuckDBJoinExecutorSpec 84/0. ES 8 Flight spec: 108 passed, 3 failed, the same 3 as on main 6a06d48.
• JDBC: unit 85/0. ES 8: getColumns reports ARRAY columns (DATA_TYPE 2003), and getObject returns List values.
• extensions: ES 8: a view over ARRAY<KEYWORD> / ARRAY<INT> sources deploys. It describes as lists, returns the lists and re-deploys.

Adoption items, not this PR's:

• JDBC ResultSetMetaData infers ARRAY<ANY> from the values, while getColumns reports ARRAY<KEYWORD>.
• A multi-valued document in a scalar column reaches JDBC callers as IllegalStateException, not SQLException (core's message, unchanged). The driver's streaming path has had this gap on main since the typed-rows work.
• Extensions refuse ARRAY_AGG in a view's transform by their own rule.
• Three arrow Flight assertions pinned on earlier behaviour fail on main too.

Recorded, not fixed

• NULLIF over ARRAY<DATE|TIMESTAMP> against a DATE value (CAST(ts AS DATE), CURRENT_DATE), and its mirror, keeps main's comparison and fails loudly as on main.
• Residuals over list columns, bytes equal to 0.24.0's: GREATEST / LEAST (no output cast), simple CASE c WHEN over ARRAY<DOUBLE|REAL> (compares as strings), ARRAY<TIMESTAMP|DATETIME> + INTERVAL vs CURRENT_DATE (truncates to a date).
• ARRAY_AGG(<expression>) does not parse (grammar unchanged). Its type over ROUND / DATEDIFF is pinned at the typing function.
• Seen by the gate, identical on main: ARRAY_AGG(x) OVER (PARTITION BY g) is NULL. A global ARRAY_AGG(k) answers 3 of 5 values. COUNT(*) OVER (PARTITION BY <list>) is mostly NULL. BOOLEAN BETWEEN throws.

Test plan

• sql/test 2246/0, core/test 1429/0, bridges es6 360/0, es8 353/0, es9 353/0, sbt "+ compile" (2.12 + 2.13)
• ES 8.18.3: full es8java/test, 879 passed, 0 failed (3 version-gated cancels)
• ES 6.8.23: array round trip, set-operation types and client specs, 67/0
• ES 8.18.3 gate on v0.24.0, main and this branch; computed columns re-measured on the final bytes
• Consumer smoke: extensions, JDBC, arrow (compile, targeted specs, ES 8 legs)
• CI lint line on the committed bytes
• CI: all ES legs

🤖 Generated with Claude Code