Repository navigation
fix(search): long queries no longer throw TooManyNestedClauses - #135
Merged
Merged
Conversation
A query of many words (a whole sentence, e.g. a question) can fail with IndexSearcher.TooManyNestedClauses: each token brings up to 32 dictionary forms, listed in the presence filter and in each of the four phrase queries, and the synonym boosts add up to 256 forms of each kind, past Lucene's default limit of 1024 clauses. - The forms per token now share a budget across the query's tokens (unchanged up to five tokens, fewer beyond, at least four). - The engine raises Lucene's clause limit to 4096 for the boosts. Checked on 555 question-length queries (Hebrew questions of 6 to 20 words): none fails now, 230 ms on average; before, they overflowed. Signed-off-by: Elie Gambache <elyahou.hadass@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A lexical query of many words — a whole sentence, such as a question typed in free language — can throw
IndexSearcher.TooManyNestedClauses(maxClauseCount is set to 1024):MAX_SYNONYM_TERMS_PER_TOKEN(32) dictionary forms, listed in the presence filter and in each of the four phrase queries (synonym phrase + the three slop phrases);MAX_SYNONYM_BOOST_TERMS(256) forms of each kind.With ten or so words the query passes Lucene's default 1024 clauses and the search fails instead of returning results.
Fix
SYNONYM_TERMS_BUDGET= 160): unchanged up to five tokens (32 each), fewer beyond, at least four. Short queries behave exactly as before.LuceneSearchEngineraises Lucene's clause limit to 4096 (only if lower), which leaves room for the boosts.Verification
555 question-length Hebrew queries (6 to 20 words) through
openSession(...).nextPage(30): none fails, 230 ms on average. Before the fix, such queries overflowed the limit.