Search reads each index once for names, folders, words and meaning

docs/HELP.md

@@ -49,14 +49,14 @@ The folder tree beside a viewer. Markdown, code, images, PDFs, office documents,
 
 ## Search
 
-One box searches three ways, and each result says which one matched: **name** (any part of a file or folder name), **words** (whole words inside files: PDFs, office documents, notes, code, and text in images), and **meaning** (passages that say the same thing in other words, so a plain question finds a report that never uses its words). The line under the box says which libraries are searched by meaning; **How search works** explains each kind.
+One box searches four ways, and each result says how it was found: **name** (any part of a file's name), **folder** (a folder it is in has the search in its name), **words** (whole words inside files: PDFs, office documents, notes, code, and text in images), and **meaning** (passages that say the same thing in other words, so a plain question finds a report that never uses its words). Hover the label to see every way a result was found. The line under the box says which libraries are searched by meaning; **How search works** explains each kind.
 
 What a search can find in a library depends on its **index depth**, a scale of five stops, each adding to the one before and taking longer to index: **Names**; **Text** (the words inside files); **Text + OCR** (text in images and scanned pages); **Meaning on documents** (search by meaning over the first few passages of PDF, Office, text, Markdown and image files); and **Meaning on all files** (every file with text, about three times the indexing time of Meaning on documents). The main folder starts at Meaning on documents and other libraries at Text + OCR; change them in Settings, Libraries, in the Depth column. For the main folder and a library built on this computer, a change is applied in the background at once, over the files already indexed: only what the new depth adds runs (the text, OCR, or search by meaning), and a lower depth runs nothing. With the background sync paused it waits for **sync now**. A library built on a remote system takes a new depth at its next **Rebuild**. A search with quotes or operators, one short word, or a code such as ABC-12 matches words only.
 
 When a search leaves a library out, or searches it only in part, a line above the results names it and says why: it is being built, its text has not been read yet, its index is missing, it did not answer within 20 seconds, it could not be reached over ssh, or macOS has not allowed {appName} to read its folder. Each name opens the library's row in Settings, Libraries. Searching all libraries, choose a library that did not answer in time alone in **Searching** to wait for it. Query shows the same line, and the assistant's search says the same in its answer.
 
 - Use "quotes" for a phrase, OR between alternatives, -word to leave a word out, and word* for words that start the same way.
-- Word matches come best first: a file that uses the words more, in less text, ranks higher. Name matches come with the exact name first, then names that start with the search.
+- Results found more than one way come first, then the best of each kind: a file that uses the words more, in less text; the exact name, then names that start with the search; the passage nearest in meaning. Each library's index is read once per search for all of them.
 - Label chips under the box narrow the results.
 - Tick results to label them together.
 - **GUFI commands**, under the results, shows the commands that found them, ready to copy and run in a terminal.

docs/REFERENCE.md

@@ -59,9 +59,14 @@ The details behind the user guide ([`HELP.md`](HELP.md)), by part of the app. Th
 - In the parallel default each library has 20 seconds to answer (`STUDIO_SEARCH_DEADLINE_MS`). One that has not is left out and `GET /api/search` names it in `slow`; choosing that library alone waits for it.
 - A remote host whose login runs only `gufi_query` (`server/src/remoteExec.ts`): when the session loop is refused there (no bash, a restricted shell, a forced command), the site is marked direct and each command is its own ssh call over the shared connection, the program name bare and each other argument a POSIX shell word; only `gufi_query` and `gufi_sqlite3` are sent, and anything else answers 409 `<host> runs only gufi_query for this login (no shell), so this cannot run there.` The connection tests (`POST /api/user-libraries/ssh/test`, `.../pw/test`) answer `shell: false` there, the library is saved with `shell: false` and `gufiBin` `''` when `gufi_query` runs from the login's PATH, and a build check answers its ssh step with "This host runs only gufi_query: connect existing indexes read only; building an index there is not available." A connected index whose top levels have a `words` table is saved with `fullText`, so search reads its words. On macOS the run loop passes on a `KRB5CCNAME` the login shell sets, for ssh with Kerberos. Details in [`LIBRARIES.md`](LIBRARIES.md).
 - No library drops out silently (`server/src/omissions.ts`). `GET /api/search` and `POST /api/query` carry `omitted` when any library in scope was left out, or searched only in part: one entry per library, `{ id, label, reason, partial }`, `partial` false when it added nothing. The reasons: `being built; searchable once its first pass is done`; `its build failed: ...`; `index not found`; `words not searched yet: its text is being read`, or `words not searched: its index has no text yet` (a library built at a depth that reads text whose index has no `words` table yet); `not searched by meaning: its index has no vectors yet`, for a search by meaning; `did not answer within 20 s`; `not reachable over ssh: ...`; `search failed: ...`; on macOS `macOS has not allowed ACTIVATE Studio to read <folder>; allow it in System Settings, Privacy & Security, Files and Folders (or Full Disk Access)` (elsewhere `permission denied reading <folder>`), with `; results are from its last pass` when its index still answers; and `a folder could not be read (...)` or `N folders could not be read (...)` for folders inside it. Query names a library with no extracted text when the query reads the `words` table (`its index has no extracted text, so text conditions match nothing there`). A library with nothing to answer from is not asked: across libraries it is in `omitted` and in Query's `failed`; asked alone, Query answers 409 `<label> cannot be queried: <reason>`. The Search and Query pages show one line above the results, each name opening its row in Settings, Libraries, and `search_kb` ends its answer with the same list. An index connected as it is, without text, is searched by name and metadata and not named.
-- Name and full-text search of a library run in one walk of its index: one aggregate `gufi_query` (`-a 1`) whose per-directory step (`-E`) looks up the names and then the words while that directory's database is open, where separate queries walked the tree twice (and, for a library on another system, ran two remote commands). Results by meaning run apart, through `gufi_sqlite3`. On a 102,000-file test index this took a search from about 150 ms to about 115 ms, and one across two such libraries from about 290 ms to about 205 ms.
-- Each kind keeps its best rows where the index is, so a common word in a large index sends back those rows and not every folder's matches. Full-text matches are ranked by SQLite's `bm25` over the directory's fts5 `words` table: each thread keeps its best rows after every directory, and the merge (`-J`, `-G`) returns the best of the whole tree, not the first found. `bm25` is computed within each directory's table, so scores from two directories compare roughly. Name matches are ranked by how closely they match: the query itself as the name, with or without an extension, first; then names that start with its first word; then names that contain a term; then by path. Across libraries, each kind is merged in the same order. `GET /api/search` hits carry `rank`, lower first within its kind.
-- Several index paths can go to one `gufi_query`, and it walks them in parallel in one thread pool; on the test index two paths in one run took as long as two runs side by side, so each library keeps a run of its own, with its own deadline.
+- A search runs one walk of a library's index for every kind of match it can find there (`walkSearch` in `server/src/gufi.ts`), the one-pass pattern GUFI's authors recommend: one aggregate `gufi_query` (`-a 1`) whose per-directory step (`-E`) adds, while that directory's database is open, the name matches, the files below a folder whose path matches, the Studio's word matches (its fts5 `words` table) and its nearest passages by meaning (its vec0 `gvec`, with the query's embedding passed in `-I` as a blob literal, one copy per thread), each into one per-thread table that the merge (`-J`) gathers and the final step (`-G`) ranks. The statements run in that order, names and paths first, so a directory without a `words` or vector table answers what it can: the step that needs the missing table fails there and only the steps after it are skipped. No empty tables are written to make every directory alike (an empty fts5 table adds about 20 KB to a directory's 48 KB database, and empty vector tables 40 KB more, as measured). `--no-print-sql-on-err` keeps each such failure to one short line, and only the last 16 KB of `gufi_query`'s error output is kept. gufi_query compiles every statement again in every directory, so there are four at most: one trim that keeps each kind's best, then one insert per kind of table read.
+- On a synthetic index of 24,000 directories and 100,000 files, with words in a quarter of the directories and vectors in a tenth, a search took about 2.9 s before (the name and word walk, about 0.8 s, beside the search by meaning, which walked the tree for databases with vectors and then attached each one, about 2.9 s) and about 1.0 s in one walk. Without vectors in the index the walk takes about 0.85 s; names and words alone, as before, about 0.7 s, and the folder-path matches add about 0.15 s. Timings on a shared host, median of five.
+- Results are fused with reciprocal rank fusion (`fuseRows`): each kind is ranked on its own (words by `bm25`, vectors by distance, names by how closely they match, folder-path matches by how near the matching folder is), each file once per kind, and a file's score is the sum over the kinds that found it of 1 / (60 + its rank in that kind). The final step of each walk computes the same order for its own rows; across libraries and walks the Studio fuses them again the same way. Ties go to the more exact kind (words, then name, then folder path, then meaning), then the path, so the order is the same on every run. A hit is labelled by its most exact kind (`mode`: `fts`, `name`, `path` or `vector`) and lists all of them (`kinds`); `score` is its fused score. `bm25` is computed within each directory's table, so word scores from two directories compare roughly; vec0 distances for unit vectors compare exactly.
+- Folder-path matches compare the path below the library's root, never the index's own folders, with the same rules as names (a term of three letters or fewer must start a word, here also after a slash). A file whose name matches is a name match only.
+- Meaning comes from the same walk for directories embedded with the ONNX model (`gvecmodel` `minilm-l6-mean-int8`, with or without a `-t<N>` truncation). A library whose walk returns no passages by meaning while its index has vectors (embedded with the earlier GGUF model, say) is searched by meaning the earlier way, through `gufi_sqlite3`.
+- External databases in the shared text and vector schema are read in the same walk with `--external-copy`. In each directory, each file other than `db.db` whose name matches `STUDIO_EXTERNAL_DB_PATTERN` (default `\.db$`; empty turns it off) is attached, and its best text (`ext_bm25_by_file`, an fts5 table, by `bm25`) and, where `vector_model_name` names a model whose query vector the Studio has (all-MiniLM-L6-v2, which the ONNX model quantizes), its nearest chunks (`ext_vectors_by_file`, by cosine distance, put on the scale of vec0's distance for unit vectors) join the same table. Rows are matched to the directory's entries by `binode` and `bmtime`, so a file changed since its text was extracted is not matched. So an index carrying those databases is searched by text, and by meaning for that model, with no Studio enrichment, and a library's "words not searched" note is withdrawn when its external databases answered. A GUFI that does not know `--external-copy` or `--no-print-sql-on-err` (from before 2026) is asked again without them.
+- On a rolled-up index, the walk does not visit the directories rolled into their parents, so their own tables (the Studio's words and vectors, external databases) are not read there; names and paths are. The Studio does not roll up the indexes it builds.
+- Libraries on the same remote host and account share one walk: their index roots go to one `gufi_query` in one remote command, so a search across them is one round trip, and each row goes back to its library by the root it is under. Before, each library on another system ran one remote command for names and words, and none for meaning. A walk that does not answer in its 20 seconds names every library in it. Several local index paths in one `gufi_query` measured no faster than a run each at once (about 0.84 s either way for four libraries of 6,000 directories), so libraries on this computer keep a walk each with its own deadline; `STUDIO_STACK_LOCAL=1` stacks them too.
 - Local `gufi_query` runs use one thread per core up to 16; a remote system's use its own processor count up to 16, or `STUDIO_REMOTE_THREADS` ([`LIBRARIES.md`](LIBRARIES.md)).
 
 ## Workflows

server/src/chat/tools.ts

@@ -6,7 +6,6 @@ import fs from 'node:fs/promises'
 import path from 'node:path'
 import { gufiAvailable, isMissingCli, KB_ROOT, MAX_PREVIEW_BYTES, NO_CLI_MESSAGE, PROJECT_ROOT, PW_CLI } from '../config.js'
 import { listDir, readFileContent, KbError } from '../kb.js'
-import { blendHits, searchFts, searchNames, searchVector } from '../gufi.js'
 import { getLibrary } from '../libraries.js'
 import { annotateHits } from '../tags.js'
 import { effectiveSettings } from '../settings.js'
@@ -1414,7 +1413,6 @@ async function executeToolImpl(name: string, argsJson: string, ctx?: { labelScop
               : 'No matches.') + lateNote
             return { result, summary: `${hits.length} hits in ${libs.length} librar${libs.length === 1 ? 'y' : 'ies'}` }
           }
-          const fts = scoped.filter(h => h.mode === 'fts'), names = scoped.filter(h => h.mode === 'name'), vec = scoped.filter(h => h.mode === 'vector')
           // A conversation-level label scope is enforced here regardless of
           // what the model asked for; model-requested tags narrow further
           // only when they fall inside the scope.
@@ -1425,7 +1423,8 @@ async function executeToolImpl(name: string, argsJson: string, ctx?: { labelScop
             const within = requested?.filter(t => scope.includes(t))
             filter = within?.length ? within : scope
           }
-          const hits = await annotateHits(blendHits(fts, names, vec, limit + 5), filter)
+          // searchScope's order already fuses every kind of match.
+          const hits = await annotateHits(scoped.slice(0, limit + 5), filter)
           const result = (hits.length
             ? hits.map(h => `${h.path}${h.tags?.length ? ` [labels: ${h.tags.join(', ')}]` : ''}\n  ${h.snippet || '(filename match)'}`).join('\n')
             : 'No matches.') + lateNote