Skip to content

Latest commit

 

History

History
103 lines (68 loc) · 11 KB

API.md

File metadata and controls

103 lines (68 loc) · 11 KB

Library API

Import only from the package root. Deep imports into src/ are not supported and may change without notice.

import {
  buildIndex, updateIndex, saveIndex, loadIndex,
  search, evaluate,
  defaultConfig, validateConfig,
  SUPPORTED_EXTENSIONS, createAdapter,
} from 'joons-search-engine';

The package is an ES module (ESM) and requires Node.js 20 or newer. Use import. require('joons-search-engine') works only where the runtime can require ES modules (Node.js 22.12 and later); on Node.js 20 it fails, so use import() there. All errors thrown by the engine are instances of an internal error class whose kind is one of input_unreadable, invalid_config, invalid_index, limit_exceeded, usage_error, output_error, internal_error. Error messages never contain document content or full paths.

Mistakes by the library caller surface as follows (checked against the code; the kind is not always the most specific one):

  • buildIndex and updateIndex throw internal_error for a document path that is absolute (/x.md, C:/x.md), contains .. or a backslash, is empty, missing or not a string; for two documents with the same path in buildIndex; for documents (or remove) that is not an array or an element that is not an object; and, for buildIndex only, for no options object at all (updateIndex(index) and updateIndex(index, {}) return normally); and for roots that are not strings or that name a network or device path (for example //server/share, the same written with backslashes, or a //?/ or //./ device prefix). A configuration that fails validation throws invalid_config. Exceeding a corpus cap throws limit_exceeded.
  • updateIndex merges documents with the same path (the last one wins) instead of rejecting them.
  • search throws usage_error for a query that is not a string, an unknown mode or a query that breaks the query limits, and limit_exceeded for a limit outside 1 to 1000. Passing something that is not an index throws internal_error.
  • createAdapter(...).index and .search are asynchronous: they reject (they do not throw synchronously) with usage_error for documents that is not an array and for search before index, and with the buildIndex errors above for bad documents.
  • loadIndex of a missing folder throws input_unreadable; saveIndex into a path that cannot be written, or into a foreign non-empty folder, throws output_error.

End-to-end example

The library does not walk directories or read files: you build the documents yourself (from files, a database or anything else) and hand them over. Only path is required; title, body and meta default to empty strings.

import { buildIndex, search, defaultConfig, saveIndex, loadIndex } from 'joons-search-engine';

const documents = [
  { path: 'notes/release.md', title: 'Release checklist', body: 'Tag the build, update the changelog, announce it.' },
  { path: 'notes/backup.md', title: 'Backup policy', body: 'Nightly snapshots are kept for thirty days.' },
];

const index = buildIndex({ documents, config: defaultConfig() });
const result = search(index, 'backup', { limit: 5 });
for (const hit of result.results) console.log(hit.rank, hit.path, hit.score);

// Optional: keep the index on disk and load it again later.
saveIndex(index, './my-index');
const again = loadIndex('./my-index');
console.log(search(again, 'release checklist').total);

buildIndex(options) -> Index

Builds an in-memory index from documents.

  • options.documents: array of documents { path, title?, body?, meta?, format?, bytes? }. path must be a relative path with forward slashes and no ... The duplicate-collapsing hash is always computed by the engine from the body; a supplied hash is ignored.
  • options.config: a configuration (see validateConfig). It is validated; an invalid one throws invalid_config.
  • options.roots: the input roots, stored as absolute paths.

The corpus caps apply: at most 50000 documents, maxTotalTokens term occurrences and maxPostings postings in total (see docs/CONFIG.md). Exceeding one throws limit_exceeded.

updateIndex(index, options) -> Index

Returns a new index. options.documents is upserted by path into the existing index, and options.remove is a list of paths to drop. Without any of the options documents, remove or replaceAll, the call only rebuilds the index from the documents it already holds. With options.replaceAll: true, options.documents is treated as the complete new set (this is what the update command uses after reloading every stored root). options.config and options.roots replace the stored values when given. The result is byte-identical to a fresh buildIndex over the same final documents.

saveIndex(index, dir, options)

Writes index.json and then the zero-byte marker .joons-search-engine-index into dir. The files are written to a temporary name, flushed and renamed. A non-empty directory that does not contain the marker is refused unless options.force is true. force never deletes anything and never replaces a foreign index.json: only an index.json in a directory that already holds our marker is overwritten. An existing marker is replaced only if it is a regular file with a single link; any other marker is refused before the directory is changed. A directory that has an index.json but no marker is also accepted when that file starts with this tool's format header and its checksum matches (an interrupted earlier write); the marker is then written.

loadIndex(dir) -> Index

Reads and validates an index. The file is treated as untrusted: the size is checked before reading (cap 128 MiB), the checksum is verified over the raw bytes before parsing, a structural pre-scan bounds the number of elements, and then exact shape and type checks, bounds on all counts and offsets, the stored configuration and the stored roots (absolute paths) are validated. Any failure throws invalid_index.

search(index, query, options) -> SearchResult

query is a string using the query syntax in the README. Options: limit (1 to 1000, default 10), mode (all or any, default all), snippet (boolean, default true) and config (an object that may hold synonyms, pathRules and metaRules, overriding the stored values for this call). Every result always carries explain; the --explain flag only controls whether the command line prints it.

{
  query, mode, total, truncated,
  results: [{ rank, path, title, score, snippet, explain, duplicates }]
}
  • snippet is a string of at most 240 characters, or null. Control characters (U+0000 to U+001F, U+007F to U+009F, U+2028, U+2029) become a space, so a, a control character and b read as a b; runs of white space collapse to one space and the ends are trimmed. The zero-width and direction characters U+200B to U+200F, the bidirectional controls U+202A to U+202E and U+2066 to U+2069, and U+FEFF are removed, so text that was separated only by one of them appears joined. The word joiner U+2060, the Arabic letter mark U+061C and the tag characters U+E0000 to U+E007F are kept in the library result and in JSON output. title and path are returned as stored. When the command line prints text or markdown, it replaces control characters, U+2028, U+2029, U+200B to U+200F, U+061C, U+202A to U+202E, U+2060, U+2066 to U+2069, U+FEFF and tag characters with ? in titles, paths and snippets alike (in a snippet only the characters that survived the cleaning, that is U+2060, U+061C and tag characters, can still be replaced); JSON output is left as it is.
  • explain lists matched terms per field, match types, numeric score components and applied rules. explain.bm25 has one entry per lookup term that matched the document, not per query word: a compound word contributes one entry for each of its parts plus one for the whole token, a quoted phrase one for each word slot, and each synonym alternative its own. When a lookup term matches several index terms (the word itself, longer words with the same prefix, words containing it), only the best-scoring one counts and alternatives tells how many others matched. Each lookup term is scored on its own, with one document frequency for that word, even when two words reach the same index term, so scores and ranking do not depend on the order of the words in the query. explain.fields lists the matched index terms with the query word (queryTerm) that reached them, so one index term can appear once for each word that matched it. The components always sum to the score.
  • duplicates lists the paths of documents with an identical normalized body that were collapsed into this result. The best-scoring member of such a group represents it; the lowest path breaks ties.
  • truncated is true when a bounded step may have left something out. The causes are: the dictionary scan cap (the term dictionary, all fields together, has more than 200000 terms, so substring matching is skipped); the cap of 1000 prefix and substring expansions per lookup term and field; the cap of 1024 prefix and substring hits kept per document; the work budget for path and metadata rules; and the budget of 4000000 posting entries shared by all prefix and substring expansions of one search. A spent budget leaves expansions out, so in mode all it can return zero results with truncated: true although documents that match exist. The flag can be conservative: a lookup term of three or more characters whose exact hit and expansions together reach exactly 1000 is flagged even if nothing was cut (the exact hit counts toward the 1000), a word of two characters has no substring step and is flagged only when more than 1000 prefix expansions exist, and a dictionary above 200000 terms flags every query that contains an unquoted word of three or more characters (a quoted phrase is not flagged for this cause). truncated: false means that no cap or budget was hit.
  • Ordering is deterministic: score rounded to nine decimals (descending), then path (ascending by UTF-16 code unit), then document id.

evaluate(options) -> EvalReport

options: datasetDir, k, repeat. Builds a temporary index, runs the queries and returns metrics for the engine, the engine without synonyms and two baselines. The report carries a fixed notes list. See docs/EVALUATION.md.

createAdapter(options) -> Adapter

Returns an object with index(documents) and search(query, options) for host applications (see docs/ADAPTER.md). options.config is validated once; options.roots is recorded in the index. Calling search before index rejects with usage_error (both methods return promises).

defaultConfig() -> Config

Returns a fresh default configuration object.

validateConfig(obj) -> Config

Validates a configuration object, fills defaults and returns a new object. Throws invalid_config for unknown keys, wrong types, out-of-range numbers, prototype keys, and path patterns containing .. or absolute paths. See docs/CONFIG.md.

SUPPORTED_EXTENSIONS

Frozen array of the file extensions that are read: .md, .markdown, .txt, .json, .csv.