Skip to content

fix(server): clarify condition resolution semantics for label queries - #2994

Open
contrueCT wants to merge 40 commits into
apache:masterfrom
contrueCT:task/improve-condition-query-semantics
Open

fix(server): clarify condition resolution semantics for label queries#2994
contrueCT wants to merge 40 commits into
apache:masterfrom
contrueCT:task/improve-condition-query-semantics

Conversation

@contrueCT

@contrueCT contrueCT commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

Purpose of the PR

ConditionQuery.condition() historically combines several meanings in one API:

  • no matching condition
  • EQ/IN conditions whose intersection is empty
  • one resolved value
  • the raw list from a sole IN relation
  • an exception when several relations still resolve to multiple values

This PR preserves that legacy behavior, adds explicit condition-resolution APIs, and
migrates the high-risk LABEL call sites to semantics that match each caller.

Visual overview

Condition resolution semantics and label-query optimization for PR #2994

The diagram contrasts strict and tolerant single-value resolution and shows why
negative or ambiguous label predicates stay local to preserve complete results.

Main Changes

Make condition resolution explicit

  • containsCondition(key) reports any top-level relation for the key.
  • containsConditionValues(key) reports whether a top-level EQ/IN relation exists,
    including an empty IN relation.
  • conditionValues(key) returns the resolved EQ/IN intersection. Pair it with
    containsConditionValues(key) when absence and an empty intersection must differ.
  • conditionValue(key) returns null for an empty result, returns the value for a
    singleton, and rejects a multi-value result.
  • singleConditionValueOrNull(key) returns the value only for a singleton and returns
    null for both empty and multi-value results.
  • condition(key) remains backward-compatible, including returning the raw list for a
    sole IN relation.

Migrate label-sensitive callers

  • Use strict conditionValue() semantics where serializers and sort-key paths require
    one resolved label.
  • Use singleConditionValueOrNull() where an optimization is valid only for exactly one
    resolved label.
  • Resolve label intersections when collecting matched indexes, including multi-label and
    conflicting-label queries.
  • Apply the same semantics to graph/index transactions, traversers, the in-memory table,
    and HStore.

Preserve correctness for negative-label predicates

A downstream unsafe label predicate, including one after a range, side effect, or
inside a child traversal, keeps candidate-filtering predicates local. This avoids
losing matches from labels without equivalent index coverage.

The fallback handles query controls and SEARCH predicates separately:

  • ~page is consumed as query metadata. The backend page is bounded while local
    filters and range steps keep their order. A filtered page can be empty while its
    cursor still points to more data; callers must follow the cursor to exhaustion.
  • Text.contains() uses the same analyzer and term matcher as SEARCH indexes,
    including (word), (word1|word2), and analyzed text. For example, searching
    body = "alpha" with Text.contains("(alpha)") still matches when a negative label
    follows limit().
  • HugeGraph.searchPredicate(text) creates this matcher without exposing graph
    configuration; the authorization proxy verifies graph access before delegation.

This is deliberately conservative and can reduce property-index pushdown for those
chains. A nearby FIXME records the follow-up optimization: restore selective pushdown
only after proving compatible index coverage for every candidate schema label.

Verifying these changes

Regression coverage includes:

  • absent, empty, singleton, conflicting, and multi-value condition resolution
  • a sole raw IN relation and non-EQ/IN label predicates
  • single-label and multi-label edge sort-key queries
  • negative-label queries next to indexed properties, across barriers, and with mixed
    connectives or multiple label containers
  • matched-index collection for joint labels with indexed properties
  • paging before downstream negative labels, including continuation through empty
    filtered pages without missing or duplicate IDs
  • SEARCH positive controls, explicit terms, analyzed text, and matching unindexed labels
  • real offset/limit ordering, aggregate contents, and indexed interference labels
  • singleton/duplicate IN compatibility, serializer and RamTable contracts, and
    non-admin access to the SEARCH matcher

Targeted verification:

mvn -q test -pl hugegraph-server/hugegraph-test -am \
  -P unit-test -DfailIfNoTests=false \
  -Dtest='QueryTest,TraversalUtilOptimizeTest' \
  -Djacoco.skip=true -Dcheckstyle.skip=true

mvn -q test -pl hugegraph-server/hugegraph-test -am \
  -P core-test,memory -DfailIfNoTests=false \
  -Dtest='VertexCoreTest#testQueryByNonEqLabelAndIndexedProperty+testQueryByIndexedPropertyAndNonEqLabel+testQueryByNonEqLabelAndIndexedPropertyAcrossBarrier+testQueryByNonEqLabel+testQueryByMixedConnectiveLabel+testQueryByMultipleNegativeLabelContainers+testCollectMatchedIndexesByJointLabelsWithIndexedProperties,EdgeCoreTest#testQueryOutEdgesBySingleResolvedLabelAndSortKey+testQueryOutEdgesByMultiLabelsAndSortKey+testQueryEdgesByNonEqLabel+testQueryEdgesByNonEqLabelAndIndexedPropertyAcrossBarrier+testQueryEdgesByMixedConnectiveLabel+testQueryEdgesByMultipleNegativeLabelContainers' \
  -Djacoco.skip=true -Dcheckstyle.skip=true

mvn test -pl hugegraph-server/hugegraph-test -am \
  -P core-test,rocksdb -Dsurefire.failIfNoSpecifiedTests=false \
  -Dtest='VertexCoreTest#testSearchBeforeDownstreamNegativeLabel+testPageBeforeDownstreamNegativeLabel+testNegativeLabelPreservesRangeAndAggregate+testSelectedLabelsExcludeIndexedInterference'

mvn clean compile -Dmaven.javadoc.skip=true
  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as shown above.

Does this PR potentially affect the following parts?

The public Java API of ConditionQuery gains explicit resolution methods, and
HugeGraph.searchPredicate(text) exposes the existing SEARCH matching semantics for
local traversal filters. No REST API, configuration, or dependency changes are included.

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

The API semantics are documented in Javadocs and exercised by regression tests.

@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 13, 2026
@contrueCT contrueCT changed the title improve(query): clarify condition resolution semantics for label queries fix(query): clarify condition resolution semantics for label queries Apr 19, 2026
@codecov

codecov Bot commented May 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 32.22222% with 183 lines in your changes missing coverage. Please review.
✅ Project coverage is 37.76%. Comparing base (3681148) to head (15e052f).

Files with missing lines Patch % Lines
...he/hugegraph/traversal/optimize/TraversalUtil.java 20.95% 118 Missing and 14 partials ⚠️
...apache/hugegraph/backend/query/ConditionQuery.java 59.01% 15 Missing and 10 partials ⚠️
...he/hugegraph/backend/tx/GraphIndexTransaction.java 28.57% 10 Missing and 5 partials ⚠️
...he/hugegraph/backend/store/hstore/HstoreStore.java 0.00% 3 Missing ⚠️
.../org/apache/hugegraph/auth/HugeGraphAuthProxy.java 0.00% 2 Missing ⚠️
...g/apache/hugegraph/backend/store/ram/RamTable.java 0.00% 2 Missing ⚠️
...n/java/org/apache/hugegraph/StandardHugeGraph.java 0.00% 1 Missing ⚠️
...hugegraph/backend/serializer/BinarySerializer.java 50.00% 1 Missing ⚠️
...e/hugegraph/backend/serializer/TextSerializer.java 50.00% 1 Missing ⚠️
...e/hugegraph/traversal/algorithm/HugeTraverser.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2994      +/-   ##
============================================
- Coverage     37.77%   37.76%   -0.01%     
- Complexity     6560     6593      +33     
============================================
  Files           800      800              
  Lines         68960    69168     +208     
  Branches       9166     9222      +56     
============================================
+ Hits          26052    26124      +72     
- Misses        39841    39953     +112     
- Partials       3067     3091      +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@contrueCT
contrueCT force-pushed the task/improve-condition-query-semantics branch 2 times, most recently from 4c42786 to cc9af24 Compare May 26, 2026 12:29

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one correctness issue in the latest revision. The CI failures were posted separately as a PR-level reminder.

CI/status checks are failing on the latest head (cc9af24929e42af1c90e1f55f3e60adc351e0318). Could you check the failed jobs before the next review round?

Failed checks include:

@contrueCT
contrueCT force-pushed the task/improve-condition-query-semantics branch from cc9af24 to 2e82f83 Compare May 30, 2026 10:20

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a clear blocking correctness issue in the latest head, and the previous LABEL-resolution comments look addressed. One remaining merge risk is that the latest checks are still red: hstore failed in VertexCoreTest#testQueryByDateProperty.

Since this PR also touches HstoreStore, could you rerun or clarify whether the hstore failure is an existing flaky/environment issue?

contrueCT added 5 commits June 5, 2026 12:59
Add explicit condition resolution APIs to ConditionQuery while preserving the legacy condition() behavior. Introduce containsCondition(Object), conditionValues(Object), and conditionValue(Object) so callers can distinguish missing, empty, unique, and multi-value results without overloading null semantics.

Migrate LABEL-specific consumers in graph/index transactions, serializers, traversers, and stores to use the new APIs for unique-label resolution and conservative fallback behavior. Extend QueryTest and VertexCoreTest to cover absent, conflicting, and multi-value label conditions as well as collectMatchedIndexes() behavior for multi-label and conflicting label queries.
@contrueCT
contrueCT force-pushed the task/improve-condition-query-semantics branch from 94408b7 to b10e3c2 Compare June 5, 2026 05:10
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jun 5, 2026
@contrueCT
contrueCT force-pushed the task/improve-condition-query-semantics branch from 801923a to ebc31c8 Compare June 5, 2026 18:06
@contrueCT

contrueCT commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for your patience. The hstore CI failure exposed an existing latent issue in hstore's range-index query path. For range-index scans with limit/paging, the upper layer assumed that backend scan results were globally ordered by the range-index key and that the returned page state could be reused as a HugeGraph range cursor. In hstore, multi-node/tablet scans can return entries in backend iterator order, and the page state is an internal storage cursor, so those assumptions may lead to unstable ordering or skipped results. This PR keeps the fix intentionally scoped: hstore range-index queries whose visible result depends on limit/offset/paging are sorted and sliced in the index layer, while unbounded scans still use the original streaming path to avoid disturbing count, joint-index, and cleanup paths. I think this is enough for the current PR, but the underlying hstore scan/page-state contract should be handled in a dedicated follow-up, ideally by defining whether range scans must be globally ordered and fixing the hstore iterator/page-state semantics at the storage-client layer.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: HStore range-index offset queries can skip too many sorted results. Evidence: static review of GraphIndexTransaction/query offset handling.

@contrueCT

Copy link
Copy Markdown
Contributor Author

Thanks. I fixed this by resetting scanQuery.offset(0L) before the full sorted range-index scan, so the fallback now reads the complete matched range first and lets the original query apply offset/limit only once after sorting. I also added range-offset coverage to VertexCoreTest#testQueryByDateProperty to guard the double-skip case. Local checks passed with git diff --check, hugegraph-core compile, and VertexCoreTest#testQueryByDateProperty under the rocksdb core-test profile.

@dosubot dosubot Bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Aug 6, 2026
@contrueCT contrueCT changed the title fix(server): clarify condition resolution and ensure negative-label index completeness fix(server): clarify condition resolution semantics for label queries Aug 6, 2026
@contrueCT

Copy link
Copy Markdown
Contributor Author

I will first create a separate PR to fix #3053 and merge it into this branch before continuing with this PR.

@contrueCT
contrueCT marked this pull request as draft August 9, 2026 11:09
@contrueCT
contrueCT marked this pull request as ready for review August 31, 2026 05:56
@contrueCT
contrueCT requested a review from imbajin August 31, 2026 05:56

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The condition() refactor is behaviour-preserving and the migrated label call sites each pick a defensible semantic (several latent multi-label ClassCastExceptions are removed), but the new hasUnsafeLabelInChain() guard scans a narrower step chain than the extraction loop it protects, and the new empty-label-intersection branch in collectMatchedIndexes() reaches callers as NoIndexException instead of an empty result. Evidence: static review of the exact head 7337d403b10e43ca0c9c64812e7e34ac949fa586 read via gh api -X GET repos/apache/hugegraph/pulls/2994/files and gh api -X GET repos/apache/hugegraph/contents/<path>?ref=7337d403 -H 'Accept: application/vnd.github.raw' (TraversalUtil.java:177,282-285,667; GraphIndexTransaction.java:487-491,779; ConditionQueryFlatten.java optimizeRelations/mergeRelations; Condition.java:143); all 24 check-runs at this head are green, and no local build or focused Maven run backs these findings because the PR head could not be fetched into the checkout in this session.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The three findings from head 7337d40 are fixed, but the new hasUnsafeLabelInChain() pre-scan walks only HasStep/NoOpBarrierStep, so a RangeGlobalStep from limit()/range()/skip() between an indexed property filter and an unsafe label filter reverts to partial pushdown and silently drops matching elements whose label lacks the index. Evidence: end-to-end memory-backend run at this head on the fixture used by testQueryByNonEqLabelAndIndexedPropertyAcrossBarrier - the .barrier() form returns 4 vertices, the .skip(0) and .limit(1000) forms return 3.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: A mixed-key logical label filter can still drop matching elements when property-index coverage differs by schema label. Evidence: exact-head static trace through TraversalUtil.java:677-699 and GraphIndexTransaction.collectMatchedIndexes(); no runtime regression test covers this path.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: the ConditionQuery resolution API and the LABEL call-site migration read consistently, but hasUnsafeLabelInChain() still lets a negative-label predicate sit behind a step that TinkerPop will not hoist a has() across, while the earlier indexed property is pushed down anyway, so results go silently incomplete. One nit on a duplicated serializer helper. Evidence: built this head and ran a probe method on the memory backend (mvn -o test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest='VertexCoreTest#<probe>'). On the testQueryByIndexedPropertyAndNonEqLabel fixture, g.V().has("city","Beijing").aggregate("x").has(T.label, P.neq("author")) returns 3 of the 4 expected names, and the coin(1.0) variant behaves the same, while the mixed-key or(), order() and dedup() shapes all return 4. Non-memory backends were not exercised. Step lists are in the inline comment.

@contrueCT
contrueCT marked this pull request as draft September 3, 2026 04:42
@contrueCT

Copy link
Copy Markdown
Contributor Author

CI follow-up: the Store retry passed completely (11m40s), confirming the earlier failure was the pre-checkout Zulu JDK download timeout. The HStore lane then failed twice, including repository-side rerun attempt 2, at the exact failure already tracked in #3180: VertexCoreTest#testQueryByJointIndexesWithSearchAndTwoRangeIndexesAndWithin, expected:<3> but was:<1>. The same PR code passed the full HStore lane in run 33718055523, and #3180 documents the identical intermittent failure on multiple master heads. All other visible GitHub Actions checks on current head are green. I do not think skipping/changing the assertion or folding the separate cross-partition range/search bug into this PR would be appropriate. Could a maintainer please rerun only the HStore job once more? That should also allow its coverage report to upload so Codecov can produce the final aggregate status.

@SebastianGruza

Copy link
Copy Markdown

Cross-backend results for this branch (head 5cb9a51), from the same run reported in #3090: 35 label-semantics queries (neq / without / within / conflicting labels / or / and / not, negative labels across barrier() and sideEffect(), incoming edges from multi-label sources, vertices with label + range index; edge labels with sort keys) executed through REST/Gremlin against HStore (PD + 3 stores) and RocksDB on the same server build, compared as sets of element ids.

Before — master 98477f0, both backends:

g.V().has('age',gte(30)).hasLabel(without('person','robot'))
  -> Can't do index query with [LABEL != 5, LABEL != 6] and [12 >= 30]
g.V().hasLabel(without('person')).barrier().has('age',gte(60))
  -> Don't accept query based on properties [age] that are not indexed in any label, may not match range/not-equal

Both throw on RocksDB as well, so this is server-side condition resolution, not a backend issue. The other 33 cases are identical sets on both backends already on master.

After — master + this branch (+ #3184 and the since-merged #3182): 35/35 identical sets; the two queries above return 0 and 100 vertices respectively, identically on both backends.

Version axis (RocksDB master vs RocksDB with the branch, backend held constant): 168 of 174 queries across all four sections (sort-key pushdown, range-index paging, label semantics, within × search × range) return identical id sets; the only differences are the two without() cases above (the remaining 4 are REST string-predicate rejections present on both versions). So on this corpus the branch changes visible results exactly where it intends to and nowhere else.

Caveat: the "after" run was the combined branch, not #2994 alone; #3184 only touches the HStore pushdown path and #3182 is now in master, so attribution of the two fixes to this PR is by elimination. I can run #2994 alone on request.

Raw reports (ids included) and the case list: https://github.com/SebastianGruza/hugegraph-oracle-suiteresults/compare_master.txt, results/compare_master-oracle_vs_combined-oracle.txt, finding F9 in docs/findings.md. Happy to re-run on the next head — ping me.

@contrueCT
contrueCT marked this pull request as ready for review September 5, 2026 04:35

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The condition-resolution split reads well and all 35 earlier review threads are resolved at this head; I have one error-handling regression in the new local-filter path, one gap in the unsafe-label gate that the PR does not close, and one minor redundancy. Evidence: mvn -o -q compile -pl hugegraph-server/hugegraph-core -am passed; all 24 GitHub checks are green at 0ce61d0; findings below come from static review of git diff 98477f0f..0ce61d0 and the surrounding TraversalUtil / GraphIndexTransaction code. Not verified by a local test run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improve]: clarify ConditionQuery.condition() semantics for missing, conflicting, and multi-value conditions

5 participants