diff --git a/docs/HELP.md b/docs/HELP.md
index cd85927..d59abcb 100644
--- a/docs/HELP.md
+++ b/docs/HELP.md
@@ -49,14 +49,14 @@ The folder tree beside a viewer. Markdown, code, images, PDFs, office documents,
## Search
-One box searches three ways, and each result says which one matched: **name** (any part of a file or folder name), **words** (whole words inside files: PDFs, office documents, notes, code, and text in images), and **meaning** (passages that say the same thing in other words, so a plain question finds a report that never uses its words). The line under the box says which libraries are searched by meaning; **How search works** explains each kind.
+One box searches four ways, and each result says how it was found: **name** (any part of a file's name), **folder** (a folder it is in has the search in its name), **words** (whole words inside files: PDFs, office documents, notes, code, and text in images), and **meaning** (passages that say the same thing in other words, so a plain question finds a report that never uses its words). Hover the label to see every way a result was found. The line under the box says which libraries are searched by meaning; **How search works** explains each kind.
What a search can find in a library depends on its **index depth**, a scale of five stops, each adding to the one before and taking longer to index: **Names**; **Text** (the words inside files); **Text + OCR** (text in images and scanned pages); **Meaning on documents** (search by meaning over the first few passages of PDF, Office, text, Markdown and image files); and **Meaning on all files** (every file with text, about three times the indexing time of Meaning on documents). The main folder starts at Meaning on documents and other libraries at Text + OCR; change them in Settings, Libraries, in the Depth column. For the main folder and a library built on this computer, a change is applied in the background at once, over the files already indexed: only what the new depth adds runs (the text, OCR, or search by meaning), and a lower depth runs nothing. With the background sync paused it waits for **sync now**. A library built on a remote system takes a new depth at its next **Rebuild**. A search with quotes or operators, one short word, or a code such as ABC-12 matches words only.
When a search leaves a library out, or searches it only in part, a line above the results names it and says why: it is being built, its text has not been read yet, its index is missing, it did not answer within 20 seconds, it could not be reached over ssh, or macOS has not allowed {appName} to read its folder. Each name opens the library's row in Settings, Libraries. Searching all libraries, choose a library that did not answer in time alone in **Searching** to wait for it. Query shows the same line, and the assistant's search says the same in its answer.
- Use "quotes" for a phrase, OR between alternatives, -word to leave a word out, and word* for words that start the same way.
-- Word matches come best first: a file that uses the words more, in less text, ranks higher. Name matches come with the exact name first, then names that start with the search.
+- Results found more than one way come first, then the best of each kind: a file that uses the words more, in less text; the exact name, then names that start with the search; the passage nearest in meaning. Each library's index is read once per search for all of them.
- Label chips under the box narrow the results.
- Tick results to label them together.
- **GUFI commands**, under the results, shows the commands that found them, ready to copy and run in a terminal.
diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md
index 18b9c16..409199f 100644
--- a/docs/REFERENCE.md
+++ b/docs/REFERENCE.md
@@ -59,9 +59,14 @@ The details behind the user guide ([`HELP.md`](HELP.md)), by part of the app. Th
- In the parallel default each library has 20 seconds to answer (`STUDIO_SEARCH_DEADLINE_MS`). One that has not is left out and `GET /api/search` names it in `slow`; choosing that library alone waits for it.
- A remote host whose login runs only `gufi_query` (`server/src/remoteExec.ts`): when the session loop is refused there (no bash, a restricted shell, a forced command), the site is marked direct and each command is its own ssh call over the shared connection, the program name bare and each other argument a POSIX shell word; only `gufi_query` and `gufi_sqlite3` are sent, and anything else answers 409 ` runs only gufi_query for this login (no shell), so this cannot run there.` The connection tests (`POST /api/user-libraries/ssh/test`, `.../pw/test`) answer `shell: false` there, the library is saved with `shell: false` and `gufiBin` `''` when `gufi_query` runs from the login's PATH, and a build check answers its ssh step with "This host runs only gufi_query: connect existing indexes read only; building an index there is not available." A connected index whose top levels have a `words` table is saved with `fullText`, so search reads its words. On macOS the run loop passes on a `KRB5CCNAME` the login shell sets, for ssh with Kerberos. Details in [`LIBRARIES.md`](LIBRARIES.md).
- No library drops out silently (`server/src/omissions.ts`). `GET /api/search` and `POST /api/query` carry `omitted` when any library in scope was left out, or searched only in part: one entry per library, `{ id, label, reason, partial }`, `partial` false when it added nothing. The reasons: `being built; searchable once its first pass is done`; `its build failed: ...`; `index not found`; `words not searched yet: its text is being read`, or `words not searched: its index has no text yet` (a library built at a depth that reads text whose index has no `words` table yet); `not searched by meaning: its index has no vectors yet`, for a search by meaning; `did not answer within 20 s`; `not reachable over ssh: ...`; `search failed: ...`; on macOS `macOS has not allowed ACTIVATE Studio to read ; allow it in System Settings, Privacy & Security, Files and Folders (or Full Disk Access)` (elsewhere `permission denied reading `), with `; results are from its last pass` when its index still answers; and `a folder could not be read (...)` or `N folders could not be read (...)` for folders inside it. Query names a library with no extracted text when the query reads the `words` table (`its index has no extracted text, so text conditions match nothing there`). A library with nothing to answer from is not asked: across libraries it is in `omitted` and in Query's `failed`; asked alone, Query answers 409 ` cannot be queried: `. The Search and Query pages show one line above the results, each name opening its row in Settings, Libraries, and `search_kb` ends its answer with the same list. An index connected as it is, without text, is searched by name and metadata and not named.
-- Name and full-text search of a library run in one walk of its index: one aggregate `gufi_query` (`-a 1`) whose per-directory step (`-E`) looks up the names and then the words while that directory's database is open, where separate queries walked the tree twice (and, for a library on another system, ran two remote commands). Results by meaning run apart, through `gufi_sqlite3`. On a 102,000-file test index this took a search from about 150 ms to about 115 ms, and one across two such libraries from about 290 ms to about 205 ms.
-- Each kind keeps its best rows where the index is, so a common word in a large index sends back those rows and not every folder's matches. Full-text matches are ranked by SQLite's `bm25` over the directory's fts5 `words` table: each thread keeps its best rows after every directory, and the merge (`-J`, `-G`) returns the best of the whole tree, not the first found. `bm25` is computed within each directory's table, so scores from two directories compare roughly. Name matches are ranked by how closely they match: the query itself as the name, with or without an extension, first; then names that start with its first word; then names that contain a term; then by path. Across libraries, each kind is merged in the same order. `GET /api/search` hits carry `rank`, lower first within its kind.
-- Several index paths can go to one `gufi_query`, and it walks them in parallel in one thread pool; on the test index two paths in one run took as long as two runs side by side, so each library keeps a run of its own, with its own deadline.
+- A search runs one walk of a library's index for every kind of match it can find there (`walkSearch` in `server/src/gufi.ts`), the one-pass pattern GUFI's authors recommend: one aggregate `gufi_query` (`-a 1`) whose per-directory step (`-E`) adds, while that directory's database is open, the name matches, the files below a folder whose path matches, the Studio's word matches (its fts5 `words` table) and its nearest passages by meaning (its vec0 `gvec`, with the query's embedding passed in `-I` as a blob literal, one copy per thread), each into one per-thread table that the merge (`-J`) gathers and the final step (`-G`) ranks. The statements run in that order, names and paths first, so a directory without a `words` or vector table answers what it can: the step that needs the missing table fails there and only the steps after it are skipped. No empty tables are written to make every directory alike (an empty fts5 table adds about 20 KB to a directory's 48 KB database, and empty vector tables 40 KB more, as measured). `--no-print-sql-on-err` keeps each such failure to one short line, and only the last 16 KB of `gufi_query`'s error output is kept. gufi_query compiles every statement again in every directory, so there are four at most: one trim that keeps each kind's best, then one insert per kind of table read.
+- On a synthetic index of 24,000 directories and 100,000 files, with words in a quarter of the directories and vectors in a tenth, a search took about 2.9 s before (the name and word walk, about 0.8 s, beside the search by meaning, which walked the tree for databases with vectors and then attached each one, about 2.9 s) and about 1.0 s in one walk. Without vectors in the index the walk takes about 0.85 s; names and words alone, as before, about 0.7 s, and the folder-path matches add about 0.15 s. Timings on a shared host, median of five.
+- Results are fused with reciprocal rank fusion (`fuseRows`): each kind is ranked on its own (words by `bm25`, vectors by distance, names by how closely they match, folder-path matches by how near the matching folder is), each file once per kind, and a file's score is the sum over the kinds that found it of 1 / (60 + its rank in that kind). The final step of each walk computes the same order for its own rows; across libraries and walks the Studio fuses them again the same way. Ties go to the more exact kind (words, then name, then folder path, then meaning), then the path, so the order is the same on every run. A hit is labelled by its most exact kind (`mode`: `fts`, `name`, `path` or `vector`) and lists all of them (`kinds`); `score` is its fused score. `bm25` is computed within each directory's table, so word scores from two directories compare roughly; vec0 distances for unit vectors compare exactly.
+- Folder-path matches compare the path below the library's root, never the index's own folders, with the same rules as names (a term of three letters or fewer must start a word, here also after a slash). A file whose name matches is a name match only.
+- Meaning comes from the same walk for directories embedded with the ONNX model (`gvecmodel` `minilm-l6-mean-int8`, with or without a `-t` truncation). A library whose walk returns no passages by meaning while its index has vectors (embedded with the earlier GGUF model, say) is searched by meaning the earlier way, through `gufi_sqlite3`.
+- External databases in the shared text and vector schema are read in the same walk with `--external-copy`. In each directory, each file other than `db.db` whose name matches `STUDIO_EXTERNAL_DB_PATTERN` (default `\.db$`; empty turns it off) is attached, and its best text (`ext_bm25_by_file`, an fts5 table, by `bm25`) and, where `vector_model_name` names a model whose query vector the Studio has (all-MiniLM-L6-v2, which the ONNX model quantizes), its nearest chunks (`ext_vectors_by_file`, by cosine distance, put on the scale of vec0's distance for unit vectors) join the same table. Rows are matched to the directory's entries by `binode` and `bmtime`, so a file changed since its text was extracted is not matched. So an index carrying those databases is searched by text, and by meaning for that model, with no Studio enrichment, and a library's "words not searched" note is withdrawn when its external databases answered. A GUFI that does not know `--external-copy` or `--no-print-sql-on-err` (from before 2026) is asked again without them.
+- On a rolled-up index, the walk does not visit the directories rolled into their parents, so their own tables (the Studio's words and vectors, external databases) are not read there; names and paths are. The Studio does not roll up the indexes it builds.
+- Libraries on the same remote host and account share one walk: their index roots go to one `gufi_query` in one remote command, so a search across them is one round trip, and each row goes back to its library by the root it is under. Before, each library on another system ran one remote command for names and words, and none for meaning. A walk that does not answer in its 20 seconds names every library in it. Several local index paths in one `gufi_query` measured no faster than a run each at once (about 0.84 s either way for four libraries of 6,000 directories), so libraries on this computer keep a walk each with its own deadline; `STUDIO_STACK_LOCAL=1` stacks them too.
- Local `gufi_query` runs use one thread per core up to 16; a remote system's use its own processor count up to 16, or `STUDIO_REMOTE_THREADS` ([`LIBRARIES.md`](LIBRARIES.md)).
## Workflows
diff --git a/server/src/chat/tools.ts b/server/src/chat/tools.ts
index 17b6320..b40f6e3 100644
--- a/server/src/chat/tools.ts
+++ b/server/src/chat/tools.ts
@@ -6,7 +6,6 @@ import fs from 'node:fs/promises'
import path from 'node:path'
import { gufiAvailable, isMissingCli, KB_ROOT, MAX_PREVIEW_BYTES, NO_CLI_MESSAGE, PROJECT_ROOT, PW_CLI } from '../config.js'
import { listDir, readFileContent, KbError } from '../kb.js'
-import { blendHits, searchFts, searchNames, searchVector } from '../gufi.js'
import { getLibrary } from '../libraries.js'
import { annotateHits } from '../tags.js'
import { effectiveSettings } from '../settings.js'
@@ -1414,7 +1413,6 @@ async function executeToolImpl(name: string, argsJson: string, ctx?: { labelScop
: 'No matches.') + lateNote
return { result, summary: `${hits.length} hits in ${libs.length} librar${libs.length === 1 ? 'y' : 'ies'}` }
}
- const fts = scoped.filter(h => h.mode === 'fts'), names = scoped.filter(h => h.mode === 'name'), vec = scoped.filter(h => h.mode === 'vector')
// A conversation-level label scope is enforced here regardless of
// what the model asked for; model-requested tags narrow further
// only when they fall inside the scope.
@@ -1425,7 +1423,8 @@ async function executeToolImpl(name: string, argsJson: string, ctx?: { labelScop
const within = requested?.filter(t => scope.includes(t))
filter = within?.length ? within : scope
}
- const hits = await annotateHits(blendHits(fts, names, vec, limit + 5), filter)
+ // searchScope's order already fuses every kind of match.
+ const hits = await annotateHits(scoped.slice(0, limit + 5), filter)
const result = (hits.length
? hits.map(h => `${h.path}${h.tags?.length ? ` [labels: ${h.tags.join(', ')}]` : ''}\n ${h.snippet || '(filename match)'}`).join('\n')
: 'No matches.') + lateNote
diff --git a/server/src/gufi.ts b/server/src/gufi.ts
index 9c7876a..5957d7c 100644
--- a/server/src/gufi.ts
+++ b/server/src/gufi.ts
@@ -1,14 +1,18 @@
-import { execFile } from 'node:child_process'
+import { execFile, spawn } from 'node:child_process'
import fs from 'node:fs'
import os from 'node:os'
import path from 'node:path'
import { GUFI_BIN, GUFI_INDEX, INDEX_BASE, gufiAvailable, gufiEnv } from './config.js'
-import { REMOTE_PREFIX, remoteIndexRoot, remoteQuery, remoteTarget } from './userLibraries.js'
+import { REMOTE_PREFIX, remoteIndexRoot, remoteQuery, remoteQueryMany, remoteTarget } from './userLibraries.js'
import { noteGufi } from './gufiTrace.js'
-import { embedQueries, isOnnxModel, onnxModelPresent } from './embedWorker.js'
+import { ONNX_MODEL_ID, embedQueries, isOnnxModel, onnxModelPresent } from './embedWorker.js'
const DELIM = '\x1e' // record separator; never appears in KB text
+/** How a hit was found: its words (fts), its name, the path of a folder it
+ * is in, or its meaning (vector). */
+export type HitMode = 'fts' | 'name' | 'path' | 'vector'
+
export interface SearchHit {
path: string
name: string
@@ -16,7 +20,11 @@ export interface SearchHit {
mtime: number
snippet: string
score: number
- mode: 'fts' | 'name' | 'vector'
+ /** The most exact way it was found: words, then name, then folder path,
+ * then meaning. */
+ mode: HitMode
+ /** Every way it was found, most exact first (searchScope's fused hits). */
+ kinds?: HitMode[]
tags?: string[]
/** Order within its kind, lower first: a word match's bm25, a name
* match's 0 (the query is the name), 1 (the name starts with it) or 2. */
@@ -201,21 +209,31 @@ export const nameHit = (r: string[], rel: string): SearchHit => ({ path: rel, na
* term is required to start a word: "tin" must not surface routine.md.
*/
export function nameClause(term: string): string {
+ return likeClause('name', term)
+}
+
+/** The same clause over any text column or expression; a folder path also
+ * starts a word after a slash. */
+function likeClause(col: string, term: string): string {
const t = term.replace(/[%_]/g, '')
if (!t) return '0'
- if (t.length > 3) return `name LIKE ${q('%' + t + '%')}`
- const starts = ['', ' ', '-', '_', '.', '(', '[', ',']
- return '(' + starts.map(sep => `name LIKE ${q((sep ? '%' + sep : '') + t + '%')}`).join(' OR ') + ')'
+ if (t.length > 3) return `${col} LIKE ${q('%' + t + '%')}`
+ const starts = ['', ' ', '-', '_', '.', '(', '[', ',', ...(col === 'name' ? [] : ['/'])]
+ return '(' + starts.map(sep => `${col} LIKE ${q((sep ? '%' + sep : '') + t + '%')}`).join(' OR ') + ')'
}
-export function nameWhere(parsed: ParsedQuery): string {
+function whereOver(col: string, parsed: ParsedQuery): string {
const groups = parsed.groups
- .map(g => g.items.map(i => nameClause(i.text)).filter(c => c !== '0'))
+ .map(g => g.items.map(i => likeClause(col, i.text)).filter(c => c !== '0'))
.filter(cs => cs.length)
.map(cs => cs.length > 1 ? `(${cs.join(' AND ')})` : cs[0])
return groups.length > 1 ? groups.join(' OR ') : (groups[0] ?? '0')
}
+export function nameWhere(parsed: ParsedQuery): string {
+ return whereOver('name', parsed)
+}
+
/**
* Whether semantic (vector) hits belong in the blend. They do for a
* question or a topic; they are noise for an identifier, an acronym, a
@@ -343,6 +361,284 @@ export async function searchNames(queryText: string, limit = 20, indexRoot?: str
return (await searchWordsAndNames(queryText, { ftsLimit: 0, nameLimit: limit }, indexRoot)).names
}
+// ---- one walk for names, folder paths, words and meaning ----
+
+/** What one walk looks for, and how many of each kind it keeps. 0 leaves a
+ * kind out. Per directory, a kind keeps at most its own limit, words at
+ * most `wordsPerDir` and meaning `vectorsPerDir`; overall, each its total.
+ * `studioText` and `studioVectors` say whether the index carries the
+ * Studio's own words and vector tables; `external`, a basename pattern for
+ * external databases in the shared text and vector schema, which are read
+ * wherever they are (null: not looked for). */
+export interface WalkLimits {
+ names: number; paths: number; words: number; wordsPerDir: number; vectors: number; vectorsPerDir: number
+ studioText: boolean; studioVectors: boolean; external: string | null
+}
+
+/** The basename pattern for external databases in the shared text and
+ * vector schema (ext_bm25_by_file, ext_vectors_by_file): any file in an
+ * index directory ending in .db, which gufi_query matches against every
+ * file there but db.db. STUDIO_EXTERNAL_DB_PATTERN changes it; empty turns
+ * it off. */
+export const EXTERNAL_DB_PATTERN = process.env.STUDIO_EXTERNAL_DB_PATTERN ?? '\\.db$'
+
+/** A query embedding for the vectors of the models a LIKE pattern names
+ * (gvecmodel), as hex for a blob literal. */
+export interface QueryVector { pattern: string; hex: string }
+
+/** One row of a walk: its kind, its score within the kind (lower first:
+ * nameRank, folder depth, bm25 or distance), and the file. */
+export interface WalkRow { kind: HitMode; r: number; path: string; name: string; size: number; mtime: number; snippet: string; root: string }
+
+const CODE_KIND: Record = { n: 'name', p: 'path', f: 'fts', v: 'vector' }
+/** Reciprocal rank fusion's constant: a hit's score is the sum over the
+ * kinds that found it of 1 / (60 + its rank in that kind). */
+export const RRF_K = 60
+
+/** A directory's path below the root it was reached from, in SQL: rpath()
+ * prints the index path given on the command line, so each root's own
+ * length is cut off. */
+function relDirSql(roots: string[]): string {
+ const rp = 'rpath(sname, sroll)'
+ // SQLite's substr counts characters; so does a string's code points.
+ const len = (r: string) => [...r].length
+ if (roots.length === 1) return `substr(${rp}, ${len(roots[0]) + 2})`
+ return `CASE ${roots.map(r => `WHEN substr(${rp}, 1, ${len(r)}) = ${q(r)} THEN substr(${rp}, ${len(r) + 2})`).join(' ')} ELSE ${rp} END`
+}
+
+/**
+ * The one-pass pattern GUFI's authors recommend, for search: one gufi_query
+ * walk whose per-directory step (-E) adds every kind of match while that
+ * directory's tables are open, each into the same per-thread table (-I),
+ * trimmed to its best as it goes, merged per thread (-J), and ranked once
+ * at the end (-G), where reciprocal rank fusion combines the kinds into
+ * one order. A tree walked once for all of them is the win, and the larger
+ * the more directories it has.
+ *
+ * The statements run in a fixed order, names and folder paths first, then
+ * words, then meaning: in a directory without a words table (or vectors),
+ * the statement that needs it fails and those after it are skipped there,
+ * while the rows before it stand. That lets one query serve an index whose
+ * directories have text in some places and none in others, without writing
+ * empty tables into every directory (20 KB for an empty fts5 table and 40 KB
+ * for empty vector tables, per directory, as measured).
+ *
+ * The query's embedding goes in -I as a blob literal, one copy per thread,
+ * so this needs nothing beyond what every GUFI here has (--global-db would
+ * do the same on newer GUFI).
+ *
+ * External databases in the shared schema are read in the same walk, with
+ * --external-copy: in each directory, every file matching lim.external is
+ * attached and its best text (ext_bm25_by_file, by bm25) and, for a model
+ * whose query vector is here, its nearest chunks (ext_vectors_by_file, by
+ * cosine distance on the scale of vec0's distance for unit vectors) join
+ * the same table, matched to the directory's entries by inode and mtime, so
+ * a file changed since its text was extracted is not matched. That runs
+ * before -E, which trims what it added first.
+ *
+ * Null when the query names nothing.
+ */
+export function onePassStages(queryText: string, lim: WalkLimits, roots: string[], vectors: QueryVector[] = []): string[] | null {
+ const parsed = parseSearchQuery(queryText)
+ const nWhere = lim.names > 0 ? nameWhere(parsed) : '0'
+ const relDir = relDirSql(roots)
+ const pWhere = lim.paths > 0 ? whereOver(relDir, parsed) : '0'
+ const expr = lim.words > 0 ? ftsExprFor(parsed) : ''
+ const vecs = lim.vectors > 0 ? vectors : []
+ const studioWords = lim.studioText && !!expr
+ const studioVecs = lim.studioVectors && vecs.length > 0
+ const external = lim.external && (expr || vecs.length) ? lim.external : null
+ if (nWhere === '0' && pWhere === '0' && !studioWords && !studioVecs && !external) return null
+ const cols = 'k TEXT, r REAL, p TEXT, n TEXT, s INT64, m INT64, x TEXT'
+ const rp = `rpath(sname, sroll)||'/'||name`
+ // What a thread keeps of each kind; the final step keeps vectors' own limit.
+ const held = `CASE k WHEN 'n' THEN ${lim.names} WHEN 'p' THEN ${lim.paths} WHEN 'f' THEN ${lim.words} ELSE ${lim.vectors * 3} END`
+ // gufi_query compiles every statement again in every directory, which on a
+ // tree of many directories costs more than the rows do, so there are few:
+ // one trim, first, keeping each kind's best (it also bounds what an
+ // external database and the previous directory added, whichever of their
+ // steps failed), then one insert per kind of table read.
+ const E: string[] = [`DELETE FROM t WHERE rowid IN (SELECT rowid FROM (SELECT rowid, k, row_number() OVER (PARTITION BY k ORDER BY r, p) AS rn FROM t) WHERE rn > ${held});`]
+ const names = nWhere !== '0'
+ ? `SELECT * FROM (SELECT 'n', ${nameRank(parsed)}, ${rp}, name, size, mtime, '' FROM vrpentries WHERE ${nWhere} ORDER BY 2, 3 LIMIT ${lim.names})` : ''
+ // Files below a folder whose path matches; a name that matches is a name
+ // match already. Nearer the matching folder first.
+ const paths = pWhere !== '0'
+ ? `SELECT * FROM (SELECT 'p', length(${relDir}), ${rp}, name, size, mtime, '' FROM vrpentries WHERE (${pWhere})${nWhere !== '0' ? ` AND NOT (${nWhere})` : ''} ORDER BY 2, 3 LIMIT ${lim.paths})` : ''
+ if (names || paths) E.push(`INSERT INTO t ${[names, paths].filter(Boolean).join(' UNION ALL ')};`)
+ if (studioWords) {
+ E.push(`INSERT INTO t SELECT 'f', bm25(words), ${FTS_COLUMNS} ${FTS_FROM(expr)} ORDER BY 2, 3 LIMIT ${lim.wordsPerDir};`)
+ }
+ const ext: string[] = []
+ if (external && expr) {
+ ext.push(
+ `INSERT INTO t SELECT 'f', bm25(ext_bm25_by_file), rpath(e.sname, e.sroll)||'/'||e.name, e.name, e.size, e.mtime, ` +
+ `replace(replace(snippet(ext_bm25_by_file, -1, '[', ']', ' … ', 12), char(10), ' '), char(13), ' ') ` +
+ `FROM ext_bm25_by_file JOIN vrpentries e ON CAST(binode AS INT64) = e.inode AND CAST(bmtime AS INT64) = e.mtime ` +
+ `WHERE ext_bm25_by_file MATCH ${q(expr)} ORDER BY 2, 3 LIMIT ${lim.wordsPerDir};`)
+ }
+ if (external && vecs.length) {
+ ext.push(
+ `INSERT INTO t SELECT 'v', sqrt(2.0 * vec_distance_cosine(substr(x.embedding, 1, 1536), qv.qe)), rpath(e.sname, e.sroll)||'/'||e.name, e.name, e.size, e.mtime, ` +
+ `replace(replace(substr(x.chunk_content, 1, 200), char(10), ' '), char(13), ' ') ` +
+ `FROM ext_vectors_by_file x JOIN qv ON x.vector_model_name LIKE qv.pat ` +
+ `JOIN vrpentries e ON CAST(x.vinode AS INT64) = e.inode AND CAST(x.vmtime AS INT64) = e.mtime ` +
+ `WHERE length(x.embedding) >= 1536 ORDER BY 2, 3 LIMIT ${lim.vectorsPerDir};`)
+ }
+ if (studioVecs) {
+ // The directory's own model picks the query vector; a directory whose
+ // model has none here fails this step alone, as the last one.
+ E.push(
+ `INSERT INTO t SELECT 'v', k.distance, rpath(e.sname, e.sroll)||'/'||e.name, e.name, e.size, e.mtime, ` +
+ `replace(replace(substr(g.ctext, 1, 200), char(10), ' '), char(13), ' ') ` +
+ `FROM (SELECT cid, distance FROM gvec WHERE fp384 MATCH (SELECT qe FROM qv WHERE (SELECT model FROM gvecmodel LIMIT 1) LIKE pat LIMIT 1) ORDER BY distance LIMIT ${lim.vectorsPerDir}) k ` +
+ `JOIN gchunks g ON g.cid = k.cid JOIN vrpentries e ON e.inode = g.tinode;`,
+ )
+ }
+ const cap = `CASE k WHEN 'n' THEN ${lim.names} WHEN 'p' THEN ${lim.paths} WHEN 'f' THEN ${lim.words} ELSE ${lim.vectors} END`
+ const I = [`CREATE TABLE t (${cols});`]
+ if (vecs.length) I.push('CREATE TABLE qv (pat TEXT, qe BLOB);', `INSERT INTO qv VALUES ${vecs.map(v => `(${q(v.pattern)}, X'${v.hex}')`).join(', ')};`)
+ if (!E.length) E.push('SELECT 1 WHERE 0;')
+ return [
+ '-I', I.join(' '),
+ '-K', `CREATE TABLE agg (${cols});`,
+ ...(ext.length ? ['--external-copy', external!, ext.join(' ')] : []),
+ '-E', E.join(' '),
+ '-J', 'INSERT INTO agg SELECT * FROM t;',
+ // Each file once per kind (its best row: several chunks of one file can
+ // match by meaning), ranked within its kind, kept to the kind's limit,
+ // and fused: the order a search across these roots alone would show.
+ '-G', `WITH best AS (SELECT k, min(r) AS r, p, n, s, m, x FROM agg GROUP BY k, p), ` +
+ `ranked AS (SELECT *, row_number() OVER (PARTITION BY k ORDER BY r, p) AS rn FROM best), ` +
+ `kept AS (SELECT * FROM ranked WHERE rn <= ${cap}), ` +
+ `fused AS (SELECT p AS fp, sum(1.0 / (${RRF_K}.0 + rn)) AS score FROM kept GROUP BY p) ` +
+ `SELECT k, r, p, n, s, m, x FROM kept JOIN fused ON p = fp ORDER BY score DESC, p, k;`,
+ ]
+}
+
+/** The query embeddings a walk can use: the ONNX model's, for every
+ * truncation of it, when the model is here. A directory embedded with
+ * another model is left to searchVector. */
+export async function queryVectors(queryText: string): Promise {
+ if (!onnxModelPresent()) return []
+ const vec = await embedQueries([queryText.slice(0, 500)]).then(v => v[0], e => {
+ console.warn(`search by meaning: ${(e as Error).message}`)
+ return null
+ })
+ if (!vec || vec.length !== 384 * 4) return []
+ const hex = vec.toString('hex')
+ // The ONNX model is all-MiniLM-L6-v2 with mean pooling, quantized, so its
+ // query vector also searches vectors an external database names by that
+ // model (by cosine distance, which does not depend on their scale).
+ return [{ pattern: ONNX_MODEL_ID, hex }, { pattern: `${ONNX_MODEL_ID}-t%`, hex }, { pattern: '%all-MiniLM-L6-v2', hex }]
+}
+
+/** gufi_query, keeping all of its output but only the end of its error
+ * output: a walk prints a line for every directory where a step failed (a
+ * directory without a words table, say), which on a large tree is more
+ * than a buffer should hold. */
+function runQuery(args: string[], timeoutMs = 60_000): Promise<{ stdout: string; stderr: string }> {
+ const bin = path.join(GUFI_BIN, 'gufi_query')
+ const t0 = Date.now()
+ return new Promise((resolve, reject) => {
+ const child = spawn(bin, args, { env: gufiEnv(), stdio: ['ignore', 'pipe', 'pipe'] })
+ const out: Buffer[] = []
+ let err = ''
+ child.stdout.on('data', (d: Buffer) => out.push(d))
+ child.stderr.on('data', (d: Buffer) => { err = (err + d.toString()).slice(-16_384) })
+ const timer = setTimeout(() => child.kill('SIGKILL'), timeoutMs)
+ child.on('error', e => { clearTimeout(timer); reject(e) })
+ child.on('close', (code, signal) => {
+ clearTimeout(timer)
+ noteGufi([bin, ...args], { ms: Date.now() - t0 })
+ const stdout = Buffer.concat(out).toString()
+ // An unknown option prints the usage on stdout, so it is told by its error.
+ if ((code !== 0 || signal) && (!stdout || unknownOption(err))) return reject(new Error(`gufi_query failed: ${(err.trim().split('\n').pop() || signal || `exit ${code}`)}`))
+ resolve({ stdout, stderr: err })
+ })
+ })
+}
+
+/** Whether a failure is gufi_query not knowing an option: an older GUFI. */
+const unknownOption = (e: unknown) => /unrecognized option|invalid option|unknown option/i.test(String((e as Error)?.message ?? e))
+
+/**
+ * One walk over one or more index roots on one machine: local roots in one
+ * gufi_query, or the roots of one remote site in one remote command. The
+ * rows come back tagged with the root each path is under.
+ */
+export async function walkSearch(queryText: string, lim: WalkLimits, indexRoots: string[], vectors: QueryVector[] = []): Promise {
+ if (!indexRoots.length) return []
+ const remote = indexRoots[0].startsWith(REMOTE_PREFIX)
+ const printed = remote ? indexRoots.map(r => remoteIndexRoot(r)) : indexRoots.map(r => path.resolve(r))
+ const stages = onePassStages(queryText, lim, printed, vectors)
+ if (!stages) return []
+ const exec = async (current: boolean) => {
+ // An older GUFI is asked again without what it may not know:
+ // --no-print-sql-on-err (which keeps a failed step's error to one short
+ // line) and --external-copy.
+ let s = stages
+ if (!current) {
+ const i = s.indexOf('--external-copy')
+ if (i >= 0) s = [...s.slice(0, i), ...s.slice(i + 3)]
+ }
+ const args = ['-n', String(LOCAL_THREADS), '-d', DELIM, '-a', '1', ...(current ? ['--no-print-sql-on-err'] : []), ...s]
+ return remote ? remoteQueryMany(indexRoots, args) : (await runQuery([...args, ...printed])).stdout
+ }
+ const stdout = await exec(true).catch(e => (unknownOption(e) ? exec(false) : Promise.reject(e)))
+ // Longest root first, so a root inside another claims its own paths.
+ const order = printed.map((p, i) => ({ p, root: indexRoots[i] })).sort((a, b) => b.p.length - a.p.length)
+ const rows: WalkRow[] = []
+ for (const r of parseRows(stdout, 7)) {
+ const kind = CODE_KIND[r[0]]
+ if (!kind) continue
+ const at = order.find(o => r[2] === o.p || r[2].startsWith(o.p + '/'))
+ if (!at) continue
+ rows.push({ kind, r: Number(r[1]), path: r[2] === at.p ? '' : r[2].slice(at.p.length + 1), name: r[3], size: Number(r[4]) || 0, mtime: Number(r[5]) || 0, snippet: r[6] ?? '', root: at.root })
+ }
+ return rows
+}
+
+const KIND_ORDER: HitMode[] = ['fts', 'name', 'path', 'vector']
+
+/**
+ * Reciprocal rank fusion over the rows of one or more walks, each row's
+ * path already unique per library: every kind is ranked on its own (by its
+ * score, then path, so the order is the same on every run), kept to its
+ * limit, and a file's score is the sum over the kinds that found it of
+ * 1 / (60 + its rank). Ties go to the more exact kind, then the path. A hit
+ * is labelled by the most exact way it was found, and carries all of them.
+ */
+export function fuseRows(rows: WalkRow[], caps: Partial>, limit: number): SearchHit[] {
+ const best = new Map()
+ for (const row of rows) {
+ const key = `${row.kind}\0${row.path}`
+ const was = best.get(key)
+ if (!was || row.r < was.r) best.set(key, row)
+ }
+ const byKind = new Map()
+ for (const row of best.values()) byKind.set(row.kind, [...(byKind.get(row.kind) ?? []), row])
+ const files = new Map> }>()
+ for (const kind of KIND_ORDER) {
+ const list = (byKind.get(kind) ?? []).sort((a, b) => a.r - b.r || (a.path < b.path ? -1 : a.path > b.path ? 1 : 0))
+ list.slice(0, caps[kind] ?? list.length).forEach((row, i) => {
+ const f = files.get(row.path) ?? { score: 0, kinds: [], rows: {} }
+ f.score += 1 / (RRF_K + i + 1)
+ f.kinds.push(kind)
+ f.rows[kind] = row
+ files.set(row.path, f)
+ })
+ }
+ const prio = (m: HitMode) => KIND_ORDER.indexOf(m)
+ return [...files].map(([p, f]) => {
+ const mode = f.kinds[0]
+ const main = f.rows[mode]!
+ const sized = KIND_ORDER.map(k => f.rows[k]).find(r => r && (r.size || r.mtime)) ?? main
+ const snippet = f.rows.fts?.snippet || f.rows.vector?.snippet || ''
+ return { path: p, name: main.name, size: sized.size, mtime: sized.mtime, snippet, score: f.score, mode, kinds: f.kinds, rank: main.r }
+ }).sort((a, b) => b.score - a.score || prio(a.mode) - prio(b.mode) || (a.path < b.path ? -1 : a.path > b.path ? 1 : 0)).slice(0, limit)
+}
+
/** gufi_query prints paths rooted at the index tree; strip the index prefix. */
function toKbRel(p: string, indexRoot: string = GUFI_INDEX): string {
if (indexRoot.startsWith(REMOTE_PREFIX)) {
diff --git a/server/src/libraryScope.ts b/server/src/libraryScope.ts
index 42894d0..e35e511 100644
--- a/server/src/libraryScope.ts
+++ b/server/src/libraryScope.ts
@@ -1,8 +1,9 @@
import { getLibrary, listLibraries, type Library } from './libraries.js'
-import { blendHits, ftsExprBare, ftsHit, ftsSql, nameHit, namesSql, searchVector, searchWordsAndNames, type SearchHit } from './gufi.js'
+import { EXTERNAL_DB_PATTERN, blendHits, ftsExprBare, ftsHit, ftsSql, fuseRows, nameHit, namesSql, queryVectors, searchVector, searchWordsAndNames, walkSearch, type HitMode, type SearchHit, type WalkLimits, type WalkRow } from './gufi.js'
import { gufiVtAvailable, runAcross } from './gufiVt.js'
import { effectiveSettings } from './settings.js'
import { addOmission, failureReason, libraryGap, type Omission } from './omissions.js'
+import { REMOTE_PREFIX, remoteHostKey } from './userLibraries.js'
/**
* Which libraries a request means: one, by id (the knowledge base when
@@ -33,13 +34,6 @@ export function scopeWithOmitted(param: string | null | undefined, viewerId: str
export interface ScopedHit extends SearchHit { library: { id: string; label: string } }
-/**
- * Search over one or several libraries. Each library runs what it
- * supports (full text and semantic where its index has them, names
- * always); one that fails adds nothing and the search goes on.
- * The hits are blended across libraries as within one, each tagged with
- * its library, so the same path in two libraries stays two hits.
- */
/** Whether a search looks in a library by meaning: one the Studio builds
* when its level is not off, and an index it only reads when it has vectors. */
export function meaningSearched(l: Library): boolean {
@@ -58,11 +52,120 @@ export function wordsSearched(l: Library): boolean {
* index never holds up the others. STUDIO_SEARCH_DEADLINE_MS changes it. */
export const SEARCH_DEADLINE_MS = Number(process.env.STUDIO_SEARCH_DEADLINE_MS) || 20_000
-export async function searchScope(libs: Library[], q: string, limit: number,
- opts: { semantic: boolean; nameLimit?: number; deadlineMs?: number; slow?: string[]; omitted?: Omission[] }): Promise {
- // What each library's state leaves out, said before anything runs; one
- // with nothing to answer from is not searched at all.
+/** Whether libraries on this machine share one walk. Several index paths in
+ * one gufi_query measured no faster here than a gufi_query each running at
+ * once, and a walk each keeps a library's own deadline, so they do not;
+ * libraries on one remote host do, which saves a round trip each.
+ * STUDIO_STACK_LOCAL=1 stacks them too. */
+export const STACK_LOCAL = process.env.STUDIO_STACK_LOCAL === '1'
+
+type ScopeOpts = { semantic: boolean; nameLimit?: number; deadlineMs?: number; slow?: string[]; omitted?: Omission[] }
+
+/**
+ * Search over one or several libraries, each in one walk of its index for
+ * every kind of match it supports: names and folder paths always, words
+ * where its index has text, meaning where it has vectors (walkSearch). The
+ * libraries of one remote host share one walk and one round trip. The kinds
+ * are fused across libraries with reciprocal rank fusion (fuseRows), each
+ * hit tagged with its library, so the same path in two libraries stays two
+ * hits. A library that fails, or does not answer in time, is named in
+ * opts.omitted and the search goes on. With Settings, General, "Search
+ * across libraries" set to gufi_vt, the earlier form runs (searchScopeVt).
+ */
+export async function searchScope(libs: Library[], q: string, limit: number, opts: ScopeOpts): Promise {
const omitted = opts.omitted ?? []
+ libs = withoutGaps(libs, opts, omitted)
+ if (libs.length > 1 && effectiveSettings().searchEngine === 'gufi_vt' && await gufiVtAvailable()) return searchScopeVt(libs, q, limit, opts, omitted)
+ const SEP = '\u0001'
+ const nameLimit = opts.nameLimit ?? Math.min(Math.max(10, Math.floor(limit / 5)), 50)
+ const vecLimit = Math.min(limit, 10)
+ const caps: Partial> = { name: nameLimit, path: nameLimit, fts: limit * 3, vector: vecLimit }
+ const failed = (l: Library, e: unknown, partial: boolean, what = '') => {
+ addOmission(omitted, { id: l.id, label: l.label, reason: `${what}${failureReason(l, e)}`, partial })
+ }
+ const wantVec = (l: Library) => opts.semantic && meaningSearched(l)
+ const external = EXTERNAL_DB_PATTERN || null
+ // The query is embedded for any search by meaning: external databases can
+ // carry vectors where the Studio's own tables have none.
+ const vectors = opts.semantic && (libs.some(wantVec) || external) ? await queryVectors(q) : []
+ const limitsFor = (words: boolean, vec: boolean): WalkLimits => ({
+ names: nameLimit, paths: nameLimit,
+ words: limit * 3, wordsPerDir: limit,
+ vectors: vectors.length ? vecLimit : 0, vectorsPerDir: 3,
+ studioText: words, studioVectors: vec, external,
+ })
+ // Libraries that share a walk: the same machine, and the same kinds, so
+ // one query fits them all. A remote host's are stacked; this machine's
+ // only with STACK_LOCAL.
+ const groups = new Map()
+ for (const l of libs) {
+ const words = wordsSearched(l), vec = wantVec(l) && vectors.length > 0
+ const remote = l.indexRoot.startsWith(REMOTE_PREFIX)
+ let host: string | null = null
+ try { host = remote ? remoteHostKey(l.indexRoot) : 'local' } catch { host = null }
+ const shared = remote ? host !== null : STACK_LOCAL
+ const key = shared ? `${host}\0${words}\0${vec}` : `one\0${l.id}`
+ groups.set(key, [...(groups.get(key) ?? []), l])
+ }
+ // One library alone is waited for; among several, each walk gets the deadline.
+ const deadline = libs.length > 1 ? (opts.deadlineMs ?? SEARCH_DEADLINE_MS) : 0
+ const late = (ls: Library[]) => {
+ for (const l of ls) {
+ opts.slow?.push(l.label)
+ addOmission(omitted, { id: l.id, label: l.label, reason: `did not answer within ${deadline >= 1000 ? `${Math.round(deadline / 1000)} s` : `${deadline} ms`}`, partial: false })
+ }
+ }
+ const results = await Promise.all([...groups.values()].map(async members => {
+ const words = wordsSearched(members[0]), vec = wantVec(members[0]) && vectors.length > 0
+ const byRoot = new Map()
+ for (const l of members) byRoot.set(l.indexRoot, [...(byRoot.get(l.indexRoot) ?? []), l])
+ const work = walkSearch(q, limitsFor(words, vec), [...byRoot.keys()], vec ? vectors : [])
+ .then(rows => rows.flatMap(r => (byRoot.get(r.root) ?? []).map(l => ({ ...r, path: `${l.id}${SEP}${r.path}` }))))
+ .catch(e => { for (const l of members) failed(l, e, false); return [] as WalkRow[] })
+ // Meaning for a library whose walk could not carry it: one embedded
+ // with another model (an index from before the ONNX model), or one
+ // read here without that model.
+ const extra = work.then(async rows => {
+ const out: WalkRow[] = []
+ for (const l of members) {
+ if (!wantVec(l) || l.indexRoot.startsWith(REMOTE_PREFIX)) continue
+ if (vec && rows.some(r => r.kind === 'vector' && r.path.startsWith(`${l.id}${SEP}`))) continue
+ const hits = await searchVector(q, vecLimit, l.indexRoot).catch(e => { failed(l, e, true, 'not searched by meaning: '); return [] as SearchHit[] })
+ for (const h of hits) out.push({ kind: 'vector', r: h.score, path: `${l.id}${SEP}${h.path}`, name: h.name, size: h.size, mtime: h.mtime, snippet: h.snippet, root: l.indexRoot })
+ }
+ return [...rows, ...out]
+ })
+ if (!deadline) return extra
+ let timer: ReturnType | undefined
+ const timeout = new Promise(resolve => { timer = setTimeout(() => resolve(null), deadline) })
+ const first = await Promise.race([extra, timeout])
+ clearTimeout(timer)
+ if (first) return first
+ late(members)
+ return [] as WalkRow[]
+ }))
+ const rows = results.flat()
+ // A library whose words or meaning came from external databases was
+ // searched there after all: the note that said it could not be goes.
+ const found = (kind: HitMode, id: string) => rows.some(r => r.kind === kind && r.path.startsWith(`${id}${SEP}`))
+ for (let i = omitted.length - 1; i >= 0; i--) {
+ const o = omitted[i]
+ if (!o.partial) continue
+ const left = o.reason.split('; ').filter(r => !((r.startsWith('words not searched') && found('fts', o.id)) || (r.startsWith('not searched by meaning') && found('vector', o.id))))
+ if (!left.length) omitted.splice(i, 1)
+ else omitted[i] = { ...o, reason: left.join('; ') }
+ }
+ const byId = new Map(libs.map(l => [l.id, l]))
+ return fuseRows(rows, caps, limit).map(h => {
+ const i = h.path.indexOf(SEP)
+ const l = byId.get(h.path.slice(0, i))!
+ return { ...h, path: h.path.slice(i + 1), library: { id: l.id, label: l.label } }
+ })
+}
+
+/** What each library's state leaves out, said before anything runs; one
+ * with nothing to answer from is not searched at all. */
+function withoutGaps(libs: Library[], opts: ScopeOpts, omitted: Omission[]): Library[] {
const gone = new Set()
for (const l of libs) {
const gap = libraryGap(l, { words: true, meaning: opts.semantic })
@@ -70,7 +173,13 @@ export async function searchScope(libs: Library[], q: string, limit: number,
addOmission(omitted, gap)
if (!gap.partial) gone.add(l.id)
}
- if (gone.size) libs = libs.filter(l => !gone.has(l.id))
+ return gone.size ? libs.filter(l => !gone.has(l.id)) : libs
+}
+
+/** The search as it ran before the one walk, for the gufi_vt setting: full
+ * text and names across libraries as one gufi_vt statement each, meaning
+ * per library, blended by blendHits. */
+async function searchScopeVt(libs: Library[], q: string, limit: number, opts: ScopeOpts, omitted: Omission[]): Promise {
const failed = (l: Library, e: unknown, partial: boolean, what = '') => {
addOmission(omitted, { id: l.id, label: l.label, reason: `${what}${failureReason(l, e)}`, partial })
}
diff --git a/server/src/userLibraries.ts b/server/src/userLibraries.ts
index 0452942..c1a7b38 100644
--- a/server/src/userLibraries.ts
+++ b/server/src/userLibraries.ts
@@ -211,6 +211,34 @@ export async function remoteQuery(root: string, args: string[], subdir?: string)
return r.stdout
}
+/** One gufi_query over several of a site's indexes in one remote command:
+ * the index roots of libraries on the same host and account, stacked as
+ * gufi_query's paths, so a search across them is one round trip. */
+export async function remoteQueryMany(roots: string[], args: string[]): Promise {
+ const targets = roots.map(r => remoteTarget(r)!)
+ const t = targets[0]
+ if (targets.some(x => x.site !== t.site || x.key !== t.key || x.gufiBin !== t.gufiBin)) throw new KbError(400, 'indexes on different hosts cannot share a query')
+ const n = args.indexOf('-n')
+ if (n >= 0 && n + 1 < args.length) args = [...args.slice(0, n + 1), String(await remoteThreads(t.site, t.key)), ...args.slice(n + 2)]
+ const t0 = Date.now()
+ // As remoteQuery: gufi_query alone, which a host whose login shell
+ // runs nothing else accepts.
+ const cmd = [gufiTool(t.gufiBin), ...args, ...targets.map(x => x.indexRoot)]
+ const r = await remoteRun(t.site, t.key, cmd)
+ noteGufi(cmd, { site: siteName(t.site), ms: Date.now() - t0 })
+ // An option an older GUFI does not know prints the usage on stdout; the
+ // caller asks again without it.
+ if (r.code !== 0 && (!r.stdout || /unrecognized option|invalid option|unknown option/i.test(r.stderr))) throw new KbError(502, `gufi_query on ${siteName(t.site)} failed: ${r.stderr.trim().split('\n').pop() ?? r.code}`)
+ return r.stdout
+}
+
+/** The site a library on another system is reached through, and the
+ * account and GUFI there: libraries that share it can share a query. */
+export function remoteHostKey(root: string): string | null {
+ const t = remoteTarget(root)
+ return t ? JSON.stringify([t.site, t.key, t.gufiBin]) : null
+}
+
/** The remote index's own root, to strip from the paths gufi prints. */
export function remoteIndexRoot(root: string): string { return remoteTarget(root)!.indexRoot }
diff --git a/server/test/libraryScope.test.mjs b/server/test/libraryScope.test.mjs
index 224e4e4..e9899db 100644
--- a/server/test/libraryScope.test.mjs
+++ b/server/test/libraryScope.test.mjs
@@ -38,7 +38,9 @@ case "$args" in
else row "$last/large.dat" 80 "2026-01-03 00:00:00"; row "$last/small.dat" 10 "2026-01-04 00:00:00"; fi ;;
*"GROUP BY ext"*)
if [ "$lib" = a ]; then row pdf 2 300; row txt 1 10; else row pdf 1 100; fi ;;
- *"INSERT INTO tn"*) row n 2 "$last/notes/$lib-report.md" "$lib-report.md" 7 1700000000 "" ;;
+ *"SELECT 'n', "*)
+ # The one-pass search: a name row for every index root it was given.
+ for a in "$@"; do case "$a" in "$HOME"/idx/*) b=$(basename "$a"); row n 2 "$a/notes/$b-report.md" "$b-report.md" 7 1700000000 "" ;; esac; done ;;
esac
`, { mode: 0o755 })
diff --git a/server/test/searchOnePass.test.mjs b/server/test/searchOnePass.test.mjs
new file mode 100644
index 0000000..7b270c1
--- /dev/null
+++ b/server/test/searchOnePass.test.mjs
@@ -0,0 +1,231 @@
+// One walk of an index for every kind of match: names, folder paths, the
+// Studio's words and vectors, and external databases in the shared text and
+// vector schema, fused by reciprocal rank fusion. A directory missing a
+// table skips only the steps that need it; rows from several roots go back
+// to their own. Needs GUFI with sqlite-vec (GUFI_BIN, else /opt/gufi/bin)
+// and a Python with fts5 for the real-index tests; the rest run anywhere.
+import test from 'node:test'
+import assert from 'node:assert/strict'
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
+import { execFileSync, spawnSync } from 'node:child_process'
+
+const realGufi = process.env.GUFI_BIN || '/opt/gufi/bin'
+const haveGufi = fs.existsSync(path.join(realGufi, 'gufi_query')) && fs.existsSync(path.join(realGufi, 'gufi_dir2index'))
+const haveFts = spawnSync('python3', ['-c', "import sqlite3; sqlite3.connect(':memory:').execute('CREATE VIRTUAL TABLE t USING fts5(x)')"]).status === 0
+const haveVec = haveGufi && spawnSync(path.join(realGufi, 'gufi_sqlite3'), [':memory:', 'SELECT vec_version();']).status === 0
+const skip = (!haveGufi && 'GUFI not installed') || (!haveFts && 'no Python with fts5')
+
+const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'onepass-'))
+test.after(() => fs.rmSync(dir, { recursive: true, force: true }))
+const gufiBin = path.join(dir, 'gufi')
+const gufiLog = path.join(dir, 'gufi.log')
+fs.mkdirSync(gufiBin)
+if (haveGufi) {
+ for (const f of fs.readdirSync(realGufi)) if (f !== 'gufi_query') fs.symlinkSync(path.join(realGufi, f), path.join(gufiBin, f))
+ fs.writeFileSync(path.join(gufiBin, 'gufi_query'), `#!/bin/sh\nprintf '%s\\n' "$*" >> ${gufiLog}\nexec ${realGufi}/gufi_query "$@"\n`, { mode: 0o755 })
+}
+process.env.GUFI_BIN = gufiBin
+process.env.KB_ROOT = path.join(dir, 'kb')
+process.env.INDEX_BASE = path.join(dir, 'ix')
+fs.mkdirSync(process.env.KB_ROOT)
+
+const G = await import('../dist/gufi.js')
+
+const LIMITS = { names: 10, paths: 10, words: 30, wordsPerDir: 10, vectors: 10, vectorsPerDir: 3, studioText: true, studioVectors: true, external: null }
+
+/** A unit vector leaning toward one axis, as hex for a blob: two vectors
+ * leaning the same way are near each other. */
+function unit(axis, lean = 1) {
+ const v = new Float32Array(384).fill(0.01)
+ v[axis] = lean
+ const n = Math.hypot(...v)
+ return Buffer.from(new Float32Array(v.map(x => x / n)).buffer)
+}
+
+/**
+ * A folder tree indexed by gufi_dir2index. `text` gives the Studio's words
+ * for some files (a words table only in their directories), `vectors` a
+ * vector for some files (gchunks, gvec and gvecmodel in their directories),
+ * `external` the shared schema's text and vectors (in a file ext.db in their
+ * directories). Returns the index root.
+ */
+function library(name, files, { text = {}, vectors = {}, external = {} } = {}) {
+ const src = path.join(dir, name)
+ for (const f of files) { fs.mkdirSync(path.dirname(path.join(src, f)), { recursive: true }); fs.writeFileSync(path.join(src, f), 'x') }
+ const ix = path.join(dir, `${name}-ix`)
+ fs.mkdirSync(ix)
+ execFileSync(path.join(realGufi, 'gufi_dir2index'), ['-x', src, ix], { stdio: 'ignore' })
+ const root = path.join(ix, name)
+ const vec = Object.fromEntries(Object.entries(vectors).map(([k, b]) => [k, b.toString('hex')]))
+ const ext = Object.fromEntries(Object.entries(external).map(([k, e]) => [k, { text: e.text, model: e.model, vector: e.vector?.toString('hex'), stale: !!e.stale }]))
+ const script = execFileSync('python3', ['-c', `
+import json, os, sqlite3, sys
+root, text, vec, ext = sys.argv[1], json.loads(sys.argv[2]), json.loads(sys.argv[3]), json.loads(sys.argv[4])
+out = []
+for d, _, names in os.walk(root):
+ if 'db.db' not in names: continue
+ rel = os.path.relpath(d, root)
+ db = sqlite3.connect(os.path.join(d, 'db.db'))
+ rows = db.execute('SELECT name, inode, mtime FROM entries').fetchall()
+ key = lambda n: n if rel == '.' else rel + '/' + n
+ if any(key(n) in text for n, _, _ in rows):
+ db.execute('CREATE VIRTUAL TABLE words USING fts5(tinode UNINDEXED, fname UNINDEXED, wordf)')
+ for n, inode, _ in rows:
+ if key(n) in text: db.execute('INSERT INTO words VALUES (?, ?, ?)', (str(inode), n, text[key(n)]))
+ mine = [(n, inode) for n, inode, _ in rows if key(n) in vec]
+ if mine:
+ db.execute('CREATE TABLE gchunks (cid INTEGER PRIMARY KEY, tinode TEXT, fname TEXT, seq INTEGER, ctext TEXT, h TEXT)')
+ for i, (n, inode) in enumerate(mine):
+ db.execute('INSERT INTO gchunks VALUES (?, ?, ?, 0, ?, ?)', (i + 1, str(inode), n, 'chunk of ' + n, 'h'))
+ out.append("ATTACH '%s' AS d;" % os.path.join(d, 'db.db').replace("'", "''"))
+ out.append('CREATE VIRTUAL TABLE d.gvec USING vec0(cid INTEGER PRIMARY KEY, fp384 float[384]);')
+ for i, (n, _) in enumerate(mine):
+ out.append("INSERT INTO d.gvec(cid, fp384) VALUES (%d, X'%s');" % (i + 1, vec[key(n)]))
+ out.append("CREATE TABLE d.gvecmodel (model TEXT); INSERT INTO d.gvecmodel VALUES ('minilm-l6-mean-int8');")
+ out.append('DETACH d;')
+ db.commit()
+ theirs = [(n, inode, mtime) for n, inode, mtime in rows if key(n) in ext]
+ if theirs:
+ x = sqlite3.connect(os.path.join(d, 'ext.db'))
+ x.execute("CREATE VIRTUAL TABLE ext_bm25_by_file USING fts5(bfsid UNINDEXED, binode UNINDEXED, bmtime UNINDEXED, path_segments_json_file UNINDEXED, component_asset_id UNINDEXED, component_type UNINDEXED, path_segments_json_component UNINDEXED, chunk_sequence UNINDEXED, start_position UNINDEXED, end_position UNINDEXED, page_start UNINDEXED, page_end UNINDEXED, content, tokenize = 'porter unicode61')")
+ x.execute('CREATE TABLE ext_vectors_by_file (vfsid TEXT, vinode TEXT, vmtime TEXT, page_number INT64, chunk_content TEXT, vector_mechanism TEXT, vector_model_name TEXT, vector_native_dimensions INT64, embedding BLOB)')
+ for n, inode, mtime in theirs:
+ e = ext[key(n)]
+ m = mtime - 1 if e['stale'] else mtime
+ if e.get('text'): x.execute('INSERT INTO ext_bm25_by_file VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)', ('fs', str(inode), str(m), '[]', 'a', 'body', '[]', 0, 0, 1, 1, 1, e['text']))
+ if e.get('vector'): x.execute('INSERT INTO ext_vectors_by_file VALUES (?,?,?,?,?,?,?,?,?)', ('fs', str(inode), str(m), 1, 'their chunk of ' + n, 'semantic', e['model'], 384, bytes.fromhex(e['vector'])))
+ x.commit()
+print('\\n'.join(out))
+`, root, JSON.stringify(text), JSON.stringify(vec), JSON.stringify(ext)]).toString()
+ if (script.trim()) execFileSync(path.join(realGufi, 'gufi_sqlite3'), [], { input: script })
+ return root
+}
+
+const queryVec = axis => [{ pattern: 'minilm-l6-mean-int8', hex: unit(axis).toString('hex') }, { pattern: '%all-MiniLM-L6-v2', hex: unit(axis).toString('hex') }]
+const calls = () => (fs.existsSync(gufiLog) ? fs.readFileSync(gufiLog, 'utf8').split('\n').filter(Boolean).length : 0)
+
+test('fusion: each kind ranked on its own, summed as 1/(60 + rank), ties to the more exact kind, then the path', () => {
+ const row = (kind, r, p) => ({ kind, r, path: p, name: path.basename(p), size: 1, mtime: 1, snippet: kind === 'fts' ? `w:${p}` : kind === 'vector' ? `v:${p}` : '', root: '/' })
+ const rows = [
+ row('fts', -3, 'both.md'), row('fts', -1, 'words.md'),
+ row('name', 2, 'notes/name.md'), row('name', 2, 'both.md'),
+ row('vector', 0.2, 'meaning.md'), row('vector', 0.5, 'both.md'), row('vector', 0.1, 'meaning.md'),
+ row('path', 5, 'proj/inside.md'),
+ ]
+ const out = G.fuseRows(rows, { name: 10, path: 10, fts: 30, vector: 10 }, 10)
+ // both.md: first by words, first by name (before notes/name.md on an
+ // equal score), second by meaning. meaning.md and proj/inside.md: first in
+ // their kinds, so equal, the folder match first. words.md and
+ // notes/name.md: second in theirs, the word match first.
+ assert.deepEqual(out.map(h => [h.path, h.mode]), [
+ ['both.md', 'fts'], ['proj/inside.md', 'path'], ['meaning.md', 'vector'], ['words.md', 'fts'], ['notes/name.md', 'name'],
+ ])
+ assert.deepEqual(out[0].kinds, ['fts', 'name', 'vector'])
+ assert.equal(out[0].snippet, 'w:both.md', 'a word match lends its snippet')
+ assert.ok(Math.abs(out[0].score - (2 / 61 + 1 / 62)) < 1e-12)
+ assert.equal(out[2].snippet, 'v:meaning.md', 'its best chunk')
+ // A kind past its limit adds nothing.
+ assert.deepEqual(G.fuseRows(rows, { name: 1, path: 0, fts: 1, vector: 1 }, 10).map(h => h.path), ['both.md', 'meaning.md'])
+ // The same rows in another order fuse the same way.
+ assert.deepEqual(G.fuseRows([...rows].reverse(), { name: 10, path: 10, fts: 30, vector: 10 }, 10), out)
+})
+
+test('the walk runs names and folder paths first, then words, then meaning, external databases before -E', () => {
+ const s = G.onePassStages('report', { ...LIMITS, external: '\\.db$' }, ['/ix/lib'], queryVec(1))
+ const E = s[s.indexOf('-E') + 1]
+ const at = k => E.indexOf(`SELECT '${k}'`)
+ assert.ok(at('n') < at('p') && at('p') < at('f') && at('f') < at('v'), E)
+ assert.ok(E.startsWith('DELETE FROM t WHERE rowid IN'), 'one trim first, which also bounds what external databases added')
+ assert.equal((E.match(/INSERT INTO t /g) ?? []).length, 3, 'names and paths in one insert, then words, then meaning')
+ const x = s.indexOf('--external-copy')
+ assert.ok(x >= 0 && x < s.indexOf('-E'))
+ assert.equal(s[x + 1], '\\.db$')
+ assert.match(s[x + 2], /FROM ext_bm25_by_file JOIN vrpentries e ON CAST\(binode AS INT64\) = e\.inode AND CAST\(bmtime AS INT64\) = e\.mtime/)
+ assert.match(s[s.indexOf('-I') + 1], /INSERT INTO qv VALUES \('minilm-l6-mean-int8', X'/)
+ assert.match(s[s.indexOf('-G') + 1], /sum\(1\.0 \/ \(60\.0 \+ rn\)\)/)
+ // A names-only index: no words, no vectors, nothing external.
+ const bare = G.onePassStages('report', { ...LIMITS, studioText: false, studioVectors: false }, ['/ix/lib'], [])
+ assert.doesNotMatch(bare.join(' '), /words|gvec|external-copy|qv/)
+ assert.equal(G.onePassStages('-only', LIMITS, ['/ix/lib'], []), null, 'nothing to look for')
+})
+
+test('a tree with words in some directories: those without answer their names, those with their words, in one walk', { skip }, async () => {
+ const root = library('mixed', ['a/report-2024.md', 'a/notes.txt', 'b/plain.md', 'b/c/deep.md'], { text: { 'a/notes.txt': 'the quarterly report is late', 'b/c/deep.md': 'report report report' } })
+ const before = calls()
+ const rows = await G.walkSearch('report', LIMITS, [root])
+ assert.equal(calls() - before, 1, 'one gufi_query')
+ const got = (k) => rows.filter(r => r.kind === k).map(r => r.path).sort()
+ assert.deepEqual(got('name'), ['a/report-2024.md'])
+ assert.deepEqual(got('fts'), ['a/notes.txt', 'b/c/deep.md'])
+ // b has no words table: its step failed there, and nothing else was lost.
+ const hits = G.fuseRows(rows, { name: 10, path: 10, fts: 30, vector: 10 }, 10)
+ assert.equal(hits.length, 3)
+ // The same files the separate name and word search find.
+ const old = await G.searchWordsAndNames('report', { ftsLimit: 50, nameLimit: 50 }, root)
+ assert.deepEqual([...old.fts.map(h => h.path), ...old.names.map(h => h.path)].sort(), hits.map(h => h.path).sort())
+})
+
+test('a folder whose path matches finds the files in it, after the names that match', { skip }, async () => {
+ const root = library('paths', ['harbor/one.md', 'harbor/sub/two.md', 'elsewhere/harbor-map.md', 'other/three.md'])
+ const rows = await G.walkSearch('harbor', LIMITS, [root])
+ assert.deepEqual(rows.filter(r => r.kind === 'name').map(r => r.path), ['elsewhere/harbor-map.md'])
+ assert.deepEqual(rows.filter(r => r.kind === 'path').map(r => r.path).sort(), ['harbor/one.md', 'harbor/sub/two.md'])
+ // The index's own folder name never counts: only the path below the root.
+ const none = await G.walkSearch(path.basename(path.dirname(root)).slice(0, 8), { ...LIMITS, names: 0 }, [root])
+ assert.equal(none.filter(r => r.kind === 'path').length, 0)
+})
+
+test('folder paths are matched below the root when its path has spaces, quotes and characters outside the basic plane', { skip }, async () => {
+ const root = library("lib it's 📁 two", ['harbor/one.md', 'other/two.md'])
+ const rows = await G.walkSearch('harbor', { ...LIMITS, studioText: false, studioVectors: false }, [root])
+ assert.deepEqual(rows.filter(r => r.kind === 'path').map(r => r.path), ['harbor/one.md'])
+ assert.equal((await G.walkSearch('two', { ...LIMITS, names: 0, studioText: false, studioVectors: false }, [root])).filter(r => r.kind === 'path').length, 0, 'the root\'s own name is not a folder below it')
+})
+
+test('meaning comes from the same walk, nearest first, each file once', { skip: skip || (!haveVec && 'no sqlite-vec in GUFI') }, async () => {
+ // As embed.py makes them: vectors only where the words they came from are.
+ const root = library('meaning', ['x/near.md', 'x/far.md', 'y/other.md', 'z/plain.md'], {
+ text: { 'x/near.md': 'one', 'x/far.md': 'two', 'y/other.md': 'three' },
+ vectors: { 'x/near.md': unit(5), 'x/far.md': unit(200), 'y/other.md': unit(5, 0.5) },
+ })
+ const before = calls()
+ const rows = await G.walkSearch('anything', LIMITS, [root], queryVec(5))
+ assert.equal(calls() - before, 1)
+ const vec = rows.filter(r => r.kind === 'vector').sort((a, b) => a.r - b.r).map(r => r.path)
+ assert.equal(vec[0], 'x/near.md')
+ assert.equal(vec[vec.length - 1], 'x/far.md')
+ // A model with no query vector here fails that step alone.
+ const other = await G.walkSearch('near', LIMITS, [root], [{ pattern: 'some-other-model', hex: unit(5).toString('hex') }])
+ assert.deepEqual(other.filter(r => r.kind === 'name').map(r => r.path), ['x/near.md'])
+ assert.equal(other.filter(r => r.kind === 'vector').length, 0)
+})
+
+test('external databases in the shared schema are searched by text and by a model we embed, matched by inode and mtime', { skip: skip || (!haveVec && 'no sqlite-vec in GUFI') }, async () => {
+ const root = library('shared', ['d/theirs.md', 'd/stale.md', 'd/plain.md', 'e/vec.md', 'e/othermodel.md'], {
+ external: {
+ 'd/theirs.md': { text: 'notes on quillwort and quillwort' },
+ 'd/stale.md': { text: 'quillwort again', stale: true },
+ 'e/vec.md': { vector: unit(9), model: 'sentence-transformers/all-MiniLM-L6-v2' },
+ 'e/othermodel.md': { vector: unit(9), model: 'another/model-768' },
+ },
+ })
+ const lim = { ...LIMITS, studioText: false, studioVectors: false, external: '\\.db$' }
+ const rows = await G.walkSearch('quillwort', lim, [root], queryVec(9))
+ assert.deepEqual(rows.filter(r => r.kind === 'fts').map(r => r.path), ['d/theirs.md'], 'a file changed since its text was extracted is not matched')
+ assert.match(rows.find(r => r.kind === 'fts').snippet, /\[quillwort\]/)
+ assert.deepEqual(rows.filter(r => r.kind === 'vector').map(r => r.path), ['e/vec.md'], 'only a model whose query vector is here')
+ // Without the pattern, none of it.
+ const off = await G.walkSearch('quillwort', { ...lim, external: null }, [root], queryVec(9))
+ assert.equal(off.filter(r => r.kind === 'fts' || r.kind === 'vector').length, 0)
+})
+
+test('several roots in one walk: each row goes back to its own root', { skip }, async () => {
+ const a = library('one', ['shared-name.md', 'x/only-one.md'])
+ const b = library('two', ['shared-name.md', 'y/only-two.md'])
+ const before = calls()
+ const rows = await G.walkSearch('shared', { ...LIMITS, studioText: false, studioVectors: false }, [a, b])
+ assert.equal(calls() - before, 1)
+ assert.deepEqual(rows.filter(r => r.kind === 'name').map(r => [path.basename(r.root), r.path]).sort(), [['one', 'shared-name.md'], ['two', 'shared-name.md']])
+})
diff --git a/server/test/searchStackedRemote.test.mjs b/server/test/searchStackedRemote.test.mjs
new file mode 100644
index 0000000..3add28c
--- /dev/null
+++ b/server/test/searchStackedRemote.test.mjs
@@ -0,0 +1,69 @@
+// Libraries on the same remote host are searched in one remote gufi_query
+// with their index roots stacked, so a search across them is one round trip;
+// a library on another host gets its own. Each hit goes back to its own
+// library, and one host failing names its libraries and leaves the rest.
+// Fake cluster libraries stand in, through the remote session with a local
+// bash and a fake gufi_query that logs each run.
+import test from 'node:test'
+import assert from 'node:assert/strict'
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
+import { createHash } from 'node:crypto'
+import { spawn } from 'node:child_process'
+
+const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'stack-'))
+const IX = fs.mkdtempSync(path.join(os.tmpdir(), 'stack-ix-'))
+process.env.KB_ROOT = fs.mkdtempSync(path.join(os.tmpdir(), 'stack-kb-'))
+process.env.INDEX_BASE = IX
+process.env.PW_GATEWAY_URL = 'https://platform.test/api/openai/v1'
+process.env.PW_API_KEY = 'deployment-key'
+process.env.GUFI_BIN = path.join(dir, 'no-local-gufi')
+test.after(() => fs.rmSync(dir, { recursive: true, force: true }))
+
+const home = path.join(dir, 'home')
+const bin = path.join(home, 'gufi', 'bin')
+const log = path.join(dir, 'runs.log')
+fs.mkdirSync(bin, { recursive: true })
+for (const n of ['a', 'b', 'c']) fs.mkdirSync(path.join(home, 'idx', n), { recursive: true })
+// One line per run with its index roots; a name row for each root given,
+// except on the broken host, whose gufi_query fails.
+fs.writeFileSync(path.join(bin, 'gufi_query'), `#!/bin/sh
+R=$(printf '\\036')
+row() { printf '%s' "$1"; shift; for c in "$@"; do printf '%s%s' "$R" "$c"; done; printf '\\n'; }
+roots=""
+for a in "$@"; do case "$a" in "$HOME"/idx/*) roots="$roots $(basename "$a")" ;; esac; done
+echo "$roots" >> ${log}
+case "$roots" in *c*) echo "Error: cannot open database" >&2; exit 1 ;; esac
+case "$*" in
+ *"SELECT 'n', "*)
+ for a in "$@"; do case "$a" in "$HOME"/idx/*) b=$(basename "$a"); row n 2 "$a/notes/$b-plan.md" "$b-plan.md" 7 1700000000 "" ;; esac; done ;;
+esac
+`, { mode: 0o755 })
+
+const owner = 'single'
+const hash = createHash('sha256').update(owner).digest('hex').slice(0, 16)
+fs.mkdirSync(path.join(IX, 'users', hash), { recursive: true })
+fs.writeFileSync(path.join(IX, 'users', hash, 'owner'), owner)
+const def = (id, label, site, n) => ({ id, label, site, resource: `pw://u/${site}`, sourceDir: '~', sourceRoot: path.join(home, 'src', n), indexRoot: path.join(home, 'idx', n), gufiBin: bin, state: 'ready', run: null, error: null, builtAt: null, directories: 1, files: 1, scheduler: null })
+// a and b share a host; c is on another, which fails.
+fs.writeFileSync(path.join(IX, 'users', hash, 'libraries.json'), JSON.stringify([def('u-a', 'Library A', 'cluster-1', 'a'), def('u-b', 'Library B', 'cluster-1', 'b'), def('u-c', 'Library C', 'cluster-2', 'c')]))
+
+const R = await import('../dist/remoteExec.js')
+R.setRemoteSpawn((site, key, command) => spawn('bash', ['-c', command], { stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, HOME: home } }))
+const S = await import('../dist/libraryScope.js')
+test.after(() => R.closeRemoteChannels())
+
+test('libraries on one host share one remote gufi_query; another host has its own; hits keep their library', async () => {
+ fs.rmSync(log, { force: true })
+ const libs = S.scopeLibraries('all', null, 'kb').filter(l => l.site)
+ assert.deepEqual(libs.map(l => l.id), ['u-a', 'u-b', 'u-c'])
+ const omitted = []
+ const hits = await S.searchScope(libs, 'plan', 10, { semantic: false, omitted })
+ const runs = fs.readFileSync(log, 'utf8').split('\n').filter(Boolean).map(l => l.trim()).sort()
+ // The search's one run per host (the remote session may first ask its
+ // thread count, which is not gufi_query).
+ assert.deepEqual(runs, ['a b', 'c'])
+ assert.deepEqual(hits.map(h => [h.library.id, h.path]).sort(), [['u-a', 'notes/a-plan.md'], ['u-b', 'notes/b-plan.md']])
+ assert.deepEqual(omitted.map(o => o.id), ['u-c'], 'the failing host names its library')
+})
diff --git a/web/src/api.ts b/web/src/api.ts
index a62d51e..d1432ff 100644
--- a/web/src/api.ts
+++ b/web/src/api.ts
@@ -226,7 +226,9 @@ export interface SearchHit {
size: number
mtime: number
snippet: string
- mode: 'fts' | 'name' | 'vector'
+ mode: 'fts' | 'name' | 'path' | 'vector'
+ /** Every way it was found, most exact first. */
+ kinds?: ('fts' | 'name' | 'path' | 'vector')[]
tags?: string[]
/** The library the hit is in, for a search over several. */
library?: { id: string; label: string }
diff --git a/web/src/views/SearchView.tsx b/web/src/views/SearchView.tsx
index 51fd89e..c9fd56d 100644
--- a/web/src/views/SearchView.tsx
+++ b/web/src/views/SearchView.tsx
@@ -14,12 +14,23 @@ const MODE_LABEL: Record = {
fts: 'words',
vector: 'meaning',
name: 'name',
+ path: 'folder',
}
// The badge tones have no purple; matches by meaning keep theirs from .pill-vector.
const MODE_TONE: Record = {
fts: 'info',
vector: 'default',
name: 'muted',
+ path: 'muted',
+}
+
+/** Every way a hit was found, for its badge's tooltip: the badge names the
+ * most exact one. */
+export function foundBy(h: Pick): string {
+ const what: Record = { fts: 'its words', name: 'its name', path: 'a folder it is in', vector: 'its meaning' }
+ const kinds = h.kinds?.length ? h.kinds : [h.mode]
+ const parts = kinds.map(k => what[k])
+ return `Found by ${parts.length > 1 ? `${parts.slice(0, -1).join(', ')} and ${parts[parts.length - 1]}` : parts[0]}`
}
export function SearchView({ onOpen }: { onOpen: (path: string, q?: string) => void }) {
@@ -148,7 +159,7 @@ export function SearchView({ onOpen }: { onOpen: (path: string, q?: string) => v
{h.library.label}
)}
{(h.tags ?? []).map(t => {t} )}
- {MODE_LABEL[h.mode]}
+ {MODE_LABEL[h.mode]}
{h.snippet && {h.snippet}
}
@@ -203,7 +214,8 @@ export function SearchHelp({ libraries }: { libraries: PublicLibrary[] }) {
{open && (
-
By name: any part of a file or folder name.
+
By name: any part of a file's name.
+
By folder: a folder the file is in has the search in its name.
By words: whole words in the text of documents, notes and code. "exact phrase", OR,{' '}
-exclude and prefix* narrow it.
@@ -217,7 +229,10 @@ export function SearchHelp({ libraries }: { libraries: PublicLibrary[] }) {
{i ? ', ' : ''}{l.label}: {meaningOn(l) ? (l.meaning === 'full' ? 'all files' : l.meaning === 'documents' ? 'documents' : 'its own vectors') : 'off'}
))}.
-
Each result says how it matched: name, words, or meaning.
+
+ Each result says how it matched: name, folder, words, or meaning; hover it for every way it did. Files found more than one way
+ come first.
+
)}
diff --git a/web/test/searchHelp.test.tsx b/web/test/searchHelp.test.tsx
index f5fa86d..15de55e 100644
--- a/web/test/searchHelp.test.tsx
+++ b/web/test/searchHelp.test.tsx
@@ -18,7 +18,8 @@ describe('the Search page says what search matches', () => {
expect(screen.getByText(/passages that say the same thing in other words/)).toBeInTheDocument()
expect(screen.getByText('Notes: documents')).toBeInTheDocument()
expect(screen.getByRole('link', { name: 'Settings, Libraries' })).toHaveAttribute('href', '#view=settings:libraries')
- expect(screen.getByText(/Each result says how it matched: name, words, or meaning/)).toBeInTheDocument()
+ expect(screen.getByText(/Each result says how it matched: name, folder, words, or meaning/)).toBeInTheDocument()
+ expect(screen.getByText(/a folder the file is in has the search in its name/)).toBeInTheDocument()
})
it('names and words only where it is off', () => {
@@ -114,3 +115,12 @@ describe('what the search box and the page say a search can match', () => {
expect(SEARCH_NOTE.words).toBe('Names and words, including Office and PDF content')
})
})
+
+describe('how a result was found', () => {
+ it('names every way, the most exact first, and a folder match has its own label', async () => {
+ const { foundBy } = await import('../src/views/SearchView')
+ expect(foundBy({ mode: 'fts', kinds: ['fts', 'name', 'vector'] })).toBe('Found by its words, its name and its meaning')
+ expect(foundBy({ mode: 'path', kinds: ['path'] })).toBe('Found by a folder it is in')
+ expect(foundBy({ mode: 'vector' })).toBe('Found by its meaning')
+ })
+})