Skip to content

feat(core): computed-only projections without _source, DuckDB numeric widths (SCHEMA-5) - #455

Merged
fupelaqu merged 3 commits into
mainfrom
feature/SCHEMA-5
Nov 4, 2026
Merged

fupelaqu merged 3 commits into
mainfrom
feature/SCHEMA-5

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Computed-only projections and numeric widths in core (SCHEMA-5, after #453 and #454). A SELECT whose items are all computed (SELECT 2 AS k FROM t, SELECT n + 1 AS v FROM t) no longer fetches _source ("_source": false), so its rows hold only the selected columns on every path. The computed columns core still declared as an unsized number (DATEDIFF family, ROUND, EXP / LOG / LOG10 / SQRT / POW) declare DuckDB 1.5.5's type, and DATE_TRUNC over a date declares TIMESTAMP. A quoted number beside a whole type reads as DuckDB casts it.

Closes #418

#418 asked for one plain value per SELECT item and for _source: false when no item is a plain column (SELECT 2 AS k FROM t returned every source field beside k). The one-plain-value half shipped with #453 and #454; this PR ships the _source half and the widths of the columns such a statement declares.

What changes

  • One predicate for "computed-only": a statement with script fields and no plain column, *, _id, nested or UNNEST item, aggregate, GROUP BY, window, or DELETE / UPDATE by query. Both bridges (ES 6 and ES 7/8/9) send _source: false from it; every other request keeps its bytes. The _msearch legs and the per-leg UNION ALL route reuse the conversion.
  • Window base request: the base of a computed-only window statement asks _source.includes for its PARTITION BY columns, in the same parse that adds the instants' docvalue_fields. A base with a plain column keeps its includes.
  • INSERT ... SELECT: a computed-only SELECT writes only its projected columns. One that projects none of the target's PRIMARY KEY / PARTITION BY columns is refused before any write.
  • DATEDIFF / DATE_DIFF / TIMESTAMPDIFF declare BIGINT. EXP / LOG / LOG10 / SQRT / POW declare DOUBLE (values unchanged). ROUND declares its operand's type (REAL for REAL, DOUBLE for DOUBLE, the integer type for an integer); a negative scale over TINYINT / SMALLINT / INT declares BIGINT, so the result never overflows. DATE_TRUNC over a date operand declares TIMESTAMP, in a query only. Every consumer of the declared type follows: describe, set-operation slots, GROUP BY key levels, MIN / MAX / SUM, per-group calculations, window values, CTAS / materialized-view mappings, date arithmetic, GREATEST / LEAST, conditional arms.
  • Negative-scale ROUND renders DuckDB's form, signum(x) * round(|x| / 10^-s) * 10^-s. Over a BIGINT whose rounded value leaves the BIGINT range the query fails (Overflow in ROUND of integer).
  • Quoted numbers (one function): beside TINYINT / SMALLINT / INT / BIGINT, DATEDIFF, ROUND over an integer or a date part, a decimal text reads as its integer rounded half away from zero ('1.5' = 2, '-2.5' = -3, '1e3' = 1000), in comparisons, IN, BETWEEN, NULLIF, CASE, HAVING key filters and arithmetic, on either side of a scripted comparison. Past the BIGINT range after rounding, DOUBLE / REAL operands, whole-number texts and non-numeric texts ('abc', 'NaN') are unchanged.
  • Docs: functions_math.md and the type pages describe the new widths and the BIGINT overflow rule.

Release notes

  • Computed-only statements fetch no _source: their rows hold only their columns, schema-less included. INSERT ... SELECT writes only those columns. An entity's unselected Option field reads None (a defaulted one its default); searchAsUnchecked / scrollAsUnchecked fail on an unselected required field. A schema-less window row carries only its partition columns as extras.
  • DATEDIFF family is BIGINT (Long): its GROUP BY key is Long (was Double 3.0); MIN / MAX over it map long in CTAS / MV (was refused); SUM over it is BIGINT (was DOUBLE), so its CTAS / MV column is long (was double) and its window value and JDBC / Arrow type move the same way. d + DATEDIFF(a, b) over a DATE d is a DATE in whole days (was a TIMESTAMP).
  • ROUND takes its operand's type: over DOUBLE it answers Double (was Long at scale 0), over REAL Float. GROUP BY ROUND(n) keys are Long (were 3.0). CTAS / MV and MIN / MAX map integer / long / float / double (was refused); SUM(ROUND(whole)) is BIGINT, long (was double). d + ROUND(n) over a DATE is a DATE. A negative scale over TINYINT / SMALLINT / INT answers a Long and DuckDB's value: ROUND(n, -5) for n = 423789293 is 423800000 (main: 423799999.99999994), and ROUND(CAST(2147483647 AS INT), -1) is 2147483650 (main: the double 2.14748365E9; DuckDB wraps to -2147483646 because it keeps INTEGER). An untyped whole literal inside a function is BIGINT, so ROUND(2147483647, -1) answers 2147483650 as a Long where DuckDB labels it INTEGER.
  • EXP / LOG / LOG10 / SQRT / POW are DOUBLE: CTAS / MV maps them double (was refused). Values unchanged.
  • DATE_TRUNC over a date operand is TIMESTAMP.
  • Quoted numbers beside a whole type read as DuckDB reads them (see What changes). Main refused them by name, or compared the fraction (n = '1.5' matched no row, HAVING k = '1.5' no group), and rendered a literal on the left of a scripted comparison as text.
  • An ALTER re-creates some ingest processors whose bytes moved, with the value unchanged: date arithmetic over a negative-scale ROUND; n + '1.5' (now rounded); a wrapped DATEDIFF or whole-day ROUND term (d + ABS(n)-style wrappers: only Math.round(x * 86400000.0) becomes x * 86400000L); GREATEST / LEAST over DATEDIFF or an integer ROUND, bare or wrapped (now exact beside a BIGINT past 2^53, where main went through Math.max(double, double)).
  • Answered before, refused now: the four rows below, accepted by name.
  • No downstream rebuild: no case-class arity changes.

Answered before, refused now (accepted)

Statement Main answered This PR
INSERT ... SELECT of computed columns only that projects none of the target's PRIMARY KEY / PARTITION BY columns wrote documents carrying the leaked source fields refused before any write
searchAsUnchecked / scrollAsUnchecked of a SELECT that does not project a required field filled the field from the leaked _source fails the conversion (an Option field reads None)
ABS(n) + '1.5' over an integer column refused by name Elasticsearch script compile error, as main's ABS(n) + '1' already is
negative-scale ROUND over a BIGINT whose rounded value leaves the BIGINT range an inexact double the query fails (Overflow in ROUND of integer)

Acceptance evidence

Check Evidence
Computed-only statements send _source: false, every other request keeps its bytes SQLQuerySpec in both bridges (moved pins are all computed-only row statements); ES 8.18.3 and 6.8.23 area legs
One column per item with its DuckDB type, one class per row, on one-shot, async, scroll, _msearch and per-leg routes ComputedProjectionSpec on ES 8.18.3 and 6.8.23
Window statements unchanged window oracle and the unselected-partition test on ES 8.18.3 and 6.8.23
Widths in every position (plain item, UNION ALL leg, GROUP BY key with HAVING, MIN / MAX / SUM, per-group calculation, window value, CTAS mapping) ComputedProjectionSpec and NumericWidthReportedTypeSpec (21/0)
Negative-scale ROUND values and BIGINT overflow ROUND(n, -5) = Long 423800000, ROUND(CAST(127 AS TINYINT), -1) = 130, ROUND(CAST(2147483647 AS INT), -1) = 2147483650, overflow refused on a BIGINT index; ES 8.18.3 70/0 and ES 6.8.23 58/0 on the round's specs
Quoted numbers as DuckDB casts them unit census plus the area legs; DuckDB 1.5.5 is the oracle
Area legs ES 8.18.3: 489 passed, 0 failed. ES 6.8.23: 364 passed, 0 failed (4 version-gated cancels: cached-mapping reload, Painless regex, enrich policy), on the second fix round's bytes
After the last round bridges es8 354/0, es6 361/0, es9 354/0; core 1420/0; sql 2196 passed plus the one replaced row green; client units es6 jest 58/0, es6 rest 37/0, es7 rest 33/0, es8 java 40/0; sbt "+ compile" OK
No statement answered on main is refused, other than the four rows above area legs on the branch; processor census against main: every moved byte is a negative-scale ROUND, a quoted number, a wrapped day term, or GREATEST / LEAST (release notes)
CI lint line (headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck) green on the committed bytes

DuckDB 1.5.5 is the type oracle. It keeps the integer type for ROUND and wraps silently on overflow; core declares BIGINT for a narrower operand at a negative scale instead, and fails a BIGINT that overflows.

Cost (warm JIT, setup split from the row loop, main and branch runs alternated on the same machine, load 3.8 to 7.5):

Measure main this PR Ratio
Computed-only scroll, 20,000 documents of about 2 KB: total 139.3 ms 72.3 ms 0.52x
Same, first page 18.0 ms 11.2 ms 0.62x
Computed-only window statement, total 153.1 ms 94.0 ms 0.61x
Planning per statement (row search) 151.6 us 154.8 us +2.1 %
Planning per statement (window) 229.0 us 232.3 us +1.4 %

The window base-request rewrite adds 2.4 us per statement. The planning figures predate the last two fix rounds, which add render-time memo objects only (nothing per row).

Recorded, not fixed

Measured the same on main and on this branch:

  • GREATEST / COALESCE over DATEDIFF, ROUND, SQRT and ABS of an integer fail to compile in Elasticsearch (the arm's NULL-guarded ternary is inlined without parentheses); ABS(n) + 1 and ABS(n) + '1' fail the same way.
  • COALESCE(f, c) over those functions answers NULL where f's operand is NULL; DuckDB answers c.
  • Arithmetic over a function that now declares DOUBLE or BIGINT keeps the integral type: SQRT(n) * n, POW(n, 2) + n, DATEDIFF(d, d2) * n, ROUND(n, -1) + n are declared INT and can hold a Double or a Long.
  • A BIGINT near 2^63 rounded to a negative scale is computed in double: ROUND(9223372036854775000, -3) answers 9223372036854774784 (DuckDB: 9223372036854775000).
  • DATE_TRUNC cells that fail in Painless: HOUR over a LocalDate receiver (CURRENT_DATE, CAST(ts AS DATE), DATE_PARSE, LAST_DAY), QUARTER over CURRENT_TIMESTAMP / NOW and over DATE_PARSE, and DATE_ADD at DAY.
  • A window PARTITION BY a dotted path (obj.a) or a multi-field sub-path (name.keyword) reads NULL on every row.
  • A computed-only global window (SELECT UPPER(id) AS u, COUNT(*) OVER ()) answers one row with u NULL.
  • INSERT ... SELECT writes script fields as one-element arrays (k: [4]), and a computed key gives _id {0=r1}.
  • INSERT ... SELECT into a PARTITION BY table answers 404 on main, so the PARTITION BY half of the new refusal cannot be reached for a freshly created partitioned table.
  • A literal on the left of IN / BETWEEN keeps its raw text render ('1.5' BETWEEN n AND big fails the shard; the IN form is a parse error).

Test plan

  • sql/test (2196 passed; the one failing row replaced and green, NumericWidthReportedTypeSpec 21/0), core/test 1420/0, bridges es8 354/0, es6 361/0, es9 354/0, sbt "+ compile" (2.12 + 2.13)
  • Docker-free client specs: es6 jest 58/0, es6 rest 37/0, es7 rest 33/0, es8 java 40/0
  • ES 8.18.3: area leg 489/0, then 70/0 on the last round's specs
  • ES 6.8.23: area leg 364/0, then 58/0 on the last round's specs
  • CI lint line (headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck) on the committed bytes
  • ES 7.17 and ES 9 integration legs, es6 jest leg (CI)

🤖 Generated with Claude Code

fupelaqu and others added 3 commits November 2, 2026 23:03
… widths (SCHEMA-5)

Story SCHEMA-5. Reviewed: r0:must-fix, r1:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…res BIGINT (SCHEMA-5)

Story SCHEMA-5, fix round 3 (lead ruling R-67): the rounded value always fits a BIGINT, so it no longer overflows; a BIGINT operand keeps the overflow failure. Reviewed: r0:must-fix, r1:clean, r2:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SCHEMA-5)

ROUND takes its operand's type, so ROUND(r, 1) over a REAL is a REAL
column read as java.lang.Float: the same float DuckDB answers, where the
test compared the text of the float widened to a Double. The test now
asserts the Float class apart (== accepts the widened Double) and the
float values; the WHERE assertion keeps proving the float rounding.

Story SCHEMA-5, CI fix. Reviewed: r3:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A computed column comes back as a one-element list, and a projection of computed columns only returns every source field: SELECT 2 AS k FROM t

1 participant