Conversation
upload.sh built one Meilisearch document per page and cut its cleaned text at 6000 characters with GNU cut -c, which counts bytes. Chinese pages were cut at about 2000 characters, often inside a UTF-8 sequence, and the long tail of most long pages never reached the index. scripts/build_index_docs.py now builds the documents: each page (frontmatter description + body) is packed into chunks of whole headings and paragraphs up to 3000 UTF-8 bytes, so a chunk plus its title fits the embedder's 4000-byte document template. A chunk starts with a 'Page > Section > Subsection' breadcrumb. Tags are stripped outside code fences only, keeping title attributes, ALL_CAPS placeholders and code samples. Document ids are '<md5 of page path>-<chunk index>'; the existing reconcile step deletes ids a build no longer produces, including the old one-per-page ids.
Contributor
Author
|
Superseded by #420, which indexes each page in full as one document instead of splitting it into heading-aligned chunks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
scripts/upload.shbuilt one Meilisearch document per page and cut the cleaned text withcut -c1-6000. GNUcut -ccounts bytes, so Chinese pages were cut at about 2000 characters, often inside a UTF-8 sequence (the stored text then ends in U+FFFD). Re-running the old cleaning over the current tree:The longest pages (changelog, CLI reference, OpenAPI catalog) kept 8–12% of their text. The embedder also sees at most 4000 bytes of each document (
documentTemplateMaxBytes), so a single embedding per page covers only its opening.Change
scripts/build_index_docs.py(new) builds the index documents. Each page (frontmatterdescriptionplus body) is packed into chunks of whole headings and paragraphs of up to 3000 UTF-8 bytes. A chunk plus its title fits the 4000-byte embedder template.Page > Section > Subsectionbreadcrumb, so it is understandable on its own.title="…"attribute text, ALL_CAPS placeholders such as<YOUR_APP_KEY>, and code samples are kept.import/exportlines are dropped.<md5 of page path>-<chunk index>; fields are unchanged (id/title/content/locale/url).scripts/upload.shuploads the builder's output in batches of 20. The reconcile step still deletes ids the build no longer produces. On the first run that removes the old one-per-page ids, after the new chunks are uploaded.Full tree: 560 pages → about 2160 chunks, median 2.7 KB, max 3000 bytes, no U+FFFD.
Test
python3 -m unittest tests.test_build_index_docsruns 6 tests: long Chinese page fully indexed within the byte limit, oversized line split on character boundaries, tag stripping keeps titles/placeholders/code, breadcrumb, ids/locale/urls, link-only page indexed by its description.upload.shagainst a scratch index: every chunk was embedded, with 0 failed tasks.Search callers that dedupe hits by
urlshould now merge same-page chunks instead of dropping them.