Repository navigation
Conversation
Step 1 of the PPL vector search sub-task (opensearch-project#5876, RFC opensearch-project#5875) on the Calcite engine. It adds the type, the canonical plan and the rules that the knn() function and the knn command will lower to; the grammar and the k-NN query pushdown come in the next steps. SQL and the V2 engine keep their current behaviour. Vector type - knn_vector fields map to the new ExprCoreType.VECTOR and, on Calcite, to ExprVectorType (REAL or TINYINT elements) carrying the dimension, data type and space type from the mapping. PPL with Calcite reports the type as vector; SQL and V2 type names, values, schemas and error texts are unchanged. - Values are read from _source and converted at the Calcite boundary, in scans and in pushed-down scripts, including vectors inside object and nested columns, aliases, and base64-encoded vectors (k-NN 3.8+), decoded with the data type of the index each hit comes from. - Multi-index mappings merge as before; indices that disagree on dimension, data type or the space type k-NN resolves (including k-NN 2.x remote indices) are recorded on the type. - Vectors are not comparable: comparisons, sorting, grouping, dedup keys, keyless joins on a shared vector, graphlookup keys and similar uses fail with a 400. Plan and k - KNN_SCORE operator and a shared plan builder producing Sort(fetch=k, _score DESC) <- Project(..., _score := KNN_SCORE(...)) <- [Filter(pre-filter)] <- Scan, with field, dimension, placement and pre-filter checks. - k is explicit (k=) or derived from the first following head through fields, eval, rename and sort - _score; it is resolved once per statement on the AST and is stable when a statement is lowered twice. - In a k-NN result, and in rows combined with it, _score is the k-NN score: no command may replace it, rename another column to _score0, or bring a second _score; inputs without k-NN lose only the scan's metadata _score. - Errors are 400s that never fall back to V2; a statement with a k-NN step skips the V2 attempt. k-NN steps on the analytics-engine route are rejected in the planner, and the SQL transpiler refuses KNN_SCORE. Aggregation responses - first, last, earliest, latest and dedup pushdowns read vectors whole from _source, and dedup rows give every column the value and type of the field it reads (aliases, copies, swaps, script columns, multi- valued fields, object fields). These also fix pre-existing defects for non-vector fields; the same fixes are proposed separately. - Projections of objects that hold vectors name the vector paths in the _source includes, so derived-source indices return them. Tests: unit tests for every component, including a test over every PPL command after a k-NN step, and opt-in integration tests against the k-NN plugin (-Pknn.integ=true). Signed-off-by: Lantao Jin <ltjin@amazon.com>
Contributor
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 09974dd. 'Diff too large, requires skip by maintainers after manual review' Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
LantaoJin
added
skip-diff-analyzer
Codecov Report❌ Patch coverage is ❌ Your project check has failed because the head coverage (66.56%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #5893 +/- ##
============================================
+ Coverage 63.87% 66.56% +2.68%
- Complexity 9091 10045 +954
============================================
Files 962 977 +15
Lines 41033 42799 +1766
Branches 4668 5029 +361
============================================
+ Hits 26211 28489 +2278
+ Misses 13944 13171 -773
- Partials 878 1139 +261
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
RelOptUtil.toString ends each line with System.lineSeparator(), so on Windows the plans of KnnLoweringTest and KnnPlanBuilderTest ended lines with \r\n against expected plans written with \n, and 19 tests failed. KnnTestSchema.explain now gives the plan with \n line ends, as Calcite's Matchers.hasTree compares plans for the other Calcite tests. CalcitePPLKnnTest lowers each query twice in fresh contexts and expects the same plan. timewrap writes the query start time, in seconds, into the plan, so when the two contexts were created in different seconds the plans differed (seen once on Windows with JDK 25). Both contexts are now created in the same second. Signed-off-by: Lantao Jin <ltjin@amazon.com>
…VersionUID The V2 scripts in these plans are Java-serialized and embed the serialVersionUID of OpenSearchDataType, which declares none. It changed from 7128bdc1452f35d3 to 3b91ac532260aa52 because the class gained public methods (describeTypeName and asAliasTarget). Only those eight bytes of the script blobs change. Signed-off-by: Lantao Jin <ltjin@amazon.com>
LantaoJin
marked this pull request as ready for review
October 16, 2026 06:47
LantaoJin
requested review from
RyanL1997,
Swiddis,
acarbonetto,
ahkcs,
anirudha,
dai-chen,
joshuali925,
mengweieric,
noCharger,
penghuo,
ps48,
qianheng-aws,
songkant-aws,
vamsimanohar,
ykmr1224 and
yuancu
as code owners
October 16, 2026 06:47
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
First of three pull requests for task 1 of the PPL vector search RFC (#5875), the sub-task tracked in #5876. It targets the feature branch
feature/ppl-vector-search(to be created frommain), which one final pull request merges intomain, as the SQLvectorSearch()feature was delivered (#5394). This step adds the type, the canonical plan and the rules that theknn()function (step 2, with the k-NN query pushdown) and theknncommand (step 3) will lower to. It adds no grammar, so no user query can reach the k-NN plan yet; the tests build the AST directly.Vector type (Calcite engine; SQL and the V2 engine keep their current type names, values, schemas and error texts)
knn_vectorfields map to a newExprCoreType.VECTORand, on Calcite, toExprVectorType(REALorTINYINTelements) carrying the dimension, data type and space type from the mapping; PPL with Calcite reports the type asvector(describeincluded)._sourceand converted at the Calcite boundary, in scans and in pushed-down scripts: vectors inside objects and nested fields, aliases, and base64-encoded vectors (k-NN 3.8+), decoded with the data type of the index each hit comes from.Plan and k
KNN_SCOREoperator and one plan builder producingSort(fetch=k, _score DESC) ← Project(…, _score := KNN_SCORE(field, vector, options)) ← [Filter(pre-filter)] ← Scan, with field, dimension, placement and pre-filter checks.k=) or derived from the first followingheadthroughfields,eval,renameandsort - _score; it is resolved once per statement on the AST and stays the same when a statement is lowered twice.join,lookup,append,appendcol,appendpipe,multisearchorunion,_scoreis the k-NN score: no command may replace it, rename another column to_score0, or bring a second_score.KNN_SCORE.Aggregation responses and projections.
first,last,earliest,latestanddeduppushdowns read vectors whole from_source, dedup rows give every column the value and type of the field it reads, and projections of objects that hold vectors name the vector paths in the_sourceincludes (needed on k-NN derived-source indices). These also fix pre-existing defects for non-vector fields; the same fixes are proposed tomainas three separate pull requests (to be linked), and this branch will be rebased onto them once they merge.Related Issues
Resolves #5876
Check List
--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.