diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a5fd61..160a9fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ All notable changes to this project are documented in this file. ### Breaking Changes - **The experimental agent pipeline, GEPA prompt optimization, and the distillation surfaces are removed.** The `tablassert agent` command (supervisor loop, PMC fetching, config derivation, judge/reflexion improvement, checkpoint state, and the `TABLASSERT_AGENT_*` env vars), the `--optimize` flag family and its `[optimize]` extra (`dspy`), and the `tablassert distill-export` / `tablassert distill-weigh` commands with their `[distill]` extra (`datasets`) are gone, along with the modules behind them (`agent.py`, `distill.py`, `distill_reward.py`, `graph_target.py`), the agent-only error codes `llm-transient` and `reward-config-invalid`, the `extras.require_module` lazy-import helper, `docs/agent.md`, and the `examples/` tree; the `[agent]` extra is removed from the extras registry too. The stable core pipeline (`build-kg`, `validate`, `validate-kgx`, `validate-infores`, `build-fullmap`, `quick-map`) is untouched. ([#215](https://github.com/SkyeAv/Tablassert/pull/215)) **Migration:** drop `tablassert[agent]` / `[optimize]` / `[distill]` installs and any `agent`, `--optimize`, or `distill-*` invocations; scripts that still need those surfaces must pin `tablassert<20`. -- **The QC embedding stage now runs the Babel-fine-tuned SapBERT.** Stage 4's checkpoint swaps from `cambridgeltl/SapBERT-from-PubMedBERT-fulltext` to `RenaissanceComputing/SapBERT-PubMedBERT-BabelSynonyms-v2025-01-23`, RENCI's fine-tune of the same PubMedBERT base on synonym pairs sampled from NCATS Translator Babel identifier cliques (release 2025-01-23), so the fine-tuning signal matches the stage's audit task (whether two names denote the same concept) on exactly the equivalence data Translator normalizes against. The checkpoint is a drop-in for `sentence-transformers` (same architecture, 768-dim embeddings), and the cache dir moves `.tablassert/sapbert/` -> `.tablassert/sapbert-babel/` so installs holding the base checkpoint re-download instead of silently serving it; stale `.tablassert/sapbert/` directories can be deleted. `qc.SIMILARITY_THRESHOLD` stays at `0.5` un-tuned, so reject rates may shift with the new score distribution. ([#214](https://github.com/SkyeAv/Tablassert/pull/214)) +- **The QC embedding stage now runs the Babel-fine-tuned SapBERT.** Stage 4's checkpoint swaps from `cambridgeltl/SapBERT-from-PubMedBERT-fulltext` to `RenaissanceComputing/SapBERT-PubMedBERT-BabelSynonyms-v2025-01-23`, RENCI's fine-tune of the same PubMedBERT base on synonym pairs sampled from NCATS Translator Babel identifier cliques (release 2025-01-23), so the fine-tuning signal matches the stage's audit task (whether two names denote the same concept) on exactly the equivalence data Translator normalizes against. The checkpoint is a drop-in for `sentence-transformers` (same architecture, 768-dim embeddings), and the cache dir moves `.tablassert/sapbert/` to `.tablassert/sapbert-babel/` so installs holding the base checkpoint re-download instead of silently serving it; stale `.tablassert/sapbert/` directories can be deleted. `qc.SIMILARITY_THRESHOLD` stays at `0.5` un-tuned, so reject rates may shift with the new score distribution. ([#214](https://github.com/SkyeAv/Tablassert/pull/214)) ### Added - **`build-kg` gains `--linkouts`/`-lo` for EdgeLinkouts edge deep links.** The opt-in flag appends DAKP's EdgeLinkouts record URL template (`https://linkouts.skyelanegoetz.com/edges/{edge_id}`) to the graph's own infores entry in every emitted edge's `sources`, so linkouts.skyelanegoetz.com can deep-link each edge without hand-authoring `provenance.override.sources`; `{edge_id}` rides the op list unresolved and is substituted per-edge by the existing post-dedup sweep, sections declaring a `provenance.override` are never touched, the default-off build is unchanged, and linkouts builds cache under their own `.linkouts` store suffix so a cached plain build can never quick-exit one (or vice versa). ([#216](https://github.com/SkyeAv/Tablassert/pull/216)) @@ -33,7 +33,7 @@ All notable changes to this project are documented in this file. - **`biolink-model` is bumped to 4.4.6 and the grant it absorbed is dropped.** Unlike the 4.4.5 wheel, whose bundled schema still carried the 4.4.4 version string, the 4.4.6 wheel's schema registers as 4.4.6, so `BIOLINK_VERSION` now reports the pinned version accurately. The release also attached `population_context_qualifier` to `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation`, so the `CLASS_FIELD_OVERRIDES` tripwire removed that grant from both classes (the removed entries stay in the source as commented examples for future grants); the remaining deliberate grants are `temporal_context_qualifier` on both classes plus `sex_qualifier` on `EntityToDiseaseAssociation`, and `prune_to_class` now keeps `population_context_qualifier` natively on the whole disease/phenotype family, including previously ungranted classes. ([#213](https://github.com/SkyeAv/Tablassert/pull/213)) ### Added -- **`validate-infores` checks every emitted knowledge-source identifier against the Translator registry.** The new command classifies each `infores:` CURIE on the edges (`primary_knowledge_source`, `sources[].resource_id`, `sources[].upstream_resource_ids`), the nodes (`provided_by`), and optionally the RIG document against `biolink/information-resource-registry`'s `infores_catalog.yaml` -- the authoritative catalog translator-ingests' SOP points at but never checks programmatically. The registry ships as a bundled offline snapshot (vendored from upstream commit `a72dbc3b55d5`), `--registry refresh` fetches the live catalog through the bounded-retry network layer, and `--registry off` reports structure only. The default posture is advisory (exit 0 with warnings) because Tablassert legitimately mints graph-local CURIEs; repeatable `--allow-infores` exempts them and `--strict` makes unregistered CURIEs fail in CI. Values without the `infores:` prefix are malformed, not unregistered. This closes the validation gap that only the CURIE prefix was ever checked. ([#212](https://github.com/SkyeAv/Tablassert/pull/212)) +- **`validate-infores` checks every emitted knowledge-source identifier against the Translator registry.** The new command classifies each `infores:` CURIE on the edges (`primary_knowledge_source`, `sources[].resource_id`, `sources[].upstream_resource_ids`), the nodes (`provided_by`), and optionally the RIG document against `biolink/information-resource-registry`'s `infores_catalog.yaml`, the authoritative catalog translator-ingests' SOP points at but never checks programmatically. The registry ships as a bundled offline snapshot (vendored from upstream commit `a72dbc3b55d5`), `--registry refresh` fetches the live catalog through the bounded-retry network layer, and `--registry off` reports structure only. The default posture is advisory (exit 0 with warnings) because Tablassert legitimately mints graph-local CURIEs; repeatable `--allow-infores` exempts them and `--strict` makes unregistered CURIEs fail in CI. Values without the `infores:` prefix are malformed, not unregistered. This closes the validation gap that only the CURIE prefix was ever checked. ([#212](https://github.com/SkyeAv/Tablassert/pull/212)) ## 19.6.0 - 2026-10-05 @@ -106,7 +106,7 @@ All notable changes to this project are documented in this file. ### Added - **`tablassert quick-map` inspects fullmap entity resolution for one or more terms before a build.** Each term runs the exact build op chain (level-one/level-two normalization, probe-key extraction, redb fetch, filter_and_rank) as one batched redb round trip, and one rich table per term renders in input order with the normalized probe keys in the title so a miss is diagnosable. Flags mirror the NodeEncoding config fields one-to-one (`--taxon` defaulting to 9606, `--prioritize`, `--avoid`, `--exclude-prefixes`, `--exclude-regex`); category values are validated at run time against the live vocabulary, with invalid input exiting 2 and naming the nearest valid names. A miss prints "no matches" and exits 0; only usage errors and an unreadable fullmap exit 2. The core lives in `fullmap.quick_map()`, importable without the `[cli]` extra. ([#191](https://github.com/SkyeAv/Tablassert/pull/191)) -- **`Predicates` now carries the DAKP prevention family ahead of the pinned model via `PREDICATE_OVERRIDES`.** `prevents`, `applied_to_prevent`, and `contraindicated_in_the_prevention_of` are deliberate local extensions of the model-derived predicate vocabulary: each emits as its ordinary `biolink:` CURIE and validates as a config `statement.predicate` today, while remaining absent from the pinned biolink-model 4.4.4 `related to` hierarchy (DAKP's assertion pipeline already defers the first two for exactly this reason). Association classes with open-`str` predicate slots -- including the `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` classes DAKP pins via `category_override` -- accept the predicates natively; constrained predicate vocabularies (e.g. `ChemicalEntityToBiologicalProcessAssociation`'s exhaustive `Literal`, the `GeneToDiseasePredicateEnum` family) reject them, and `validate_kgx`'s pending score (`valid_excluding_pending`) now forgives exactly those `predicate: literal_error` / `predicate: enum` failures for records whose own predicate is an override member, while strict `valid` stays strict and unlisted predicates fail both scores. A tripwire test asserts every member is still absent from installed `related to` slots: the moment a biolink-model release adopts one, the suite fails and the stale name is removed, restoring fully model-derived vocabulary for that predicate. No DAKP config changes; downstream can author the new predicates through the existing `statement.predicate` field. ([#193](https://github.com/SkyeAv/Tablassert/pull/193)) +- **`Predicates` now carries the DAKP prevention family ahead of the pinned model via `PREDICATE_OVERRIDES`.** `prevents`, `applied_to_prevent`, and `contraindicated_in_the_prevention_of` are deliberate local extensions of the model-derived predicate vocabulary: each emits as its ordinary `biolink:` CURIE and validates as a config `statement.predicate` today, while remaining absent from the pinned biolink-model 4.4.4 `related to` hierarchy (DAKP's assertion pipeline already defers the first two for exactly this reason). Association classes with open-`str` predicate slots, including the `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` classes DAKP pins via `category_override`, accept the predicates natively; constrained predicate vocabularies (e.g. `ChemicalEntityToBiologicalProcessAssociation`'s exhaustive `Literal`, the `GeneToDiseasePredicateEnum` family) reject them, and `validate_kgx`'s pending score (`valid_excluding_pending`) now forgives exactly those `predicate: literal_error` / `predicate: enum` failures for records whose own predicate is an override member, while strict `valid` stays strict and unlisted predicates fail both scores. A tripwire test asserts every member is still absent from installed `related to` slots: the moment a biolink-model release adopts one, the suite fails and the stale name is removed, restoring fully model-derived vocabulary for that predicate. No DAKP config changes; downstream can author the new predicates through the existing `statement.predicate` field. ([#193](https://github.com/SkyeAv/Tablassert/pull/193)) ### Changed - **The README PyPI downloads badge now serves from pepy instead of shields.** pepy.tech serves pre-rendered SVGs from its own CDN and does not rate-limit like the shields PyPI endpoint, which had been intermittently failing. ([#190](https://github.com/SkyeAv/Tablassert/pull/190), [#192](https://github.com/SkyeAv/Tablassert/pull/192)) @@ -125,23 +125,23 @@ All notable changes to this project are documented in this file. - **Level-one normalization now defines fullmap keys by cleaned Unicode-lowercase, whitespace-tokenized, Porter2-normalized terms, with duplicate removal and byte-wise token ordering.** Existing fullmap databases are schema-v5 and are rejected by the current resolver; rebuild fullmaps before use. Preferred-name ranking now compares normalized forms under the same level-one semantics. ([#171](https://github.com/SkyeAv/Tablassert/pull/171), [#172](https://github.com/SkyeAv/Tablassert/pull/172), [#174](https://github.com/SkyeAv/Tablassert/pull/174), [#175](https://github.com/SkyeAv/Tablassert/pull/175), [#176](https://github.com/SkyeAv/Tablassert/pull/176)) **Migration:** rebuild every fullmap database with `tablassert build-fullmap` (or re-download the prebuilt archive). A schema-v5 file is rejected rather than silently misread, so no build can run against a stale database. -- **`build-fullmap --aria2c` / `-a` is removed; the downloader is now chosen automatically.** Whenever the optional `[aria2]` extra is installed, both the prebuilt archive download and the BABEL file downloads use the bundled aria2c binary; without the extra (including on macOS, where it ships no wheels), Tablassert's Python downloader is used. The choice is resolved once per run, after the reuse short-circuit, and one info-level log line states it either way, echoed on stderr so the choice is also visible in a normal terminal run (loguru's console sink exists only inside a pipeline stage, and file logging needs the `[log]` extra) -- the old exit-2 missing-extra preflight is gone, so a missing extra is no longer an error. Failure semantics are unchanged: a failed prebuilt download still falls back to a from-scratch BABEL build with a warning (`PrebuiltFullmapUnavailable`), and a failing aria2c download still fails loud (`BabelDownloadError`) instead of being silently rescued by the Python downloader. ([#178](https://github.com/SkyeAv/Tablassert/pull/178)) +- **`build-fullmap --aria2c` / `-a` is removed; the downloader is now chosen automatically.** Whenever the optional `[aria2]` extra is installed, both the prebuilt archive download and the BABEL file downloads use the bundled aria2c binary; without the extra (including on macOS, where it ships no wheels), Tablassert's Python downloader is used. The choice is resolved once per run, after the reuse short-circuit, and one info-level log line states it either way, echoed on stderr so the choice is also visible in a normal terminal run (loguru's console sink exists only inside a pipeline stage, and file logging needs the `[log]` extra); the old exit-2 missing-extra preflight is gone, so a missing extra is no longer an error. Failure semantics are unchanged: a failed prebuilt download still falls back to a from-scratch BABEL build with a warning (`PrebuiltFullmapUnavailable`), and a failing aria2c download still fails loud (`BabelDownloadError`) instead of being silently rescued by the Python downloader. ([#178](https://github.com/SkyeAv/Tablassert/pull/178)) - **Migration:** drop `--aria2c` / `-a` from every invocation -- parsing now fails with an unknown-option error. To keep using aria2c, install the `[aria2]` extra; to stop using it, uninstall the extra. + **Migration:** drop `--aria2c` / `-a` from every invocation; parsing now fails with an unknown-option error. To keep using aria2c, install the `[aria2]` extra; to stop using it, uninstall the extra. - **`build-fullmap` always applies the built-in top-100 experimental-taxon allowlist, and `--taxon-allowlist` is removed.** The checked-in `src/tablassert/data/experimental_taxa.yaml` filter is no longer opt-in: every source build passes those 100 NCBI taxon IDs to Rust, and the resulting database records the same deterministic `META.taxon_allowlist` identity introduced in 18.1.0. The identity now also guards both reuse paths, so an unfiltered database can never satisfy an allowlisted `build-fullmap` invocation. A prebuilt archive whose recorded identity is absent or different fails extraction validation before anything is renamed, surfacing as the existing `PrebuiltFullmapUnavailable` fallback to a filtered BABEL build; an existing file at `--output` is reused only when its identity matches, and is otherwise rebuilt with a warning. `build_fullmap_pipeline`, `fetch_prebuilt_fullmap`, and `_extract_prebuilt_fullmap` accept the IDs through their existing/extended `taxon_allowlist` parameter, and the Rust extension exposes `taxon_allowlist_identity` plus `fullmap_taxon_allowlist_identity` so Python can compare identities without duplicating the encoding. ([#153](https://github.com/SkyeAv/Tablassert/pull/153)) - **Migration:** drop `--taxon-allowlist` / `--no-taxon-allowlist` -- parsing now fails with an unknown-option error. Unfiltered databases built by earlier releases no longer satisfy a reuse check and are rebuilt with a warning; budget for one filtered rebuild. + **Migration:** drop `--taxon-allowlist` / `--no-taxon-allowlist`; parsing now fails with an unknown-option error. Unfiltered databases built by earlier releases no longer satisfy a reuse check and are rebuilt with a warning; budget for one filtered rebuild. - **Logging now requires the optional `log` extra and is completely disabled without it.** The stdlib-backed loguru fallback, the startup missing-extra warning, and the base-install creation of `.tablassert/log/` are removed. Installs without loguru emit no file, progress, or warning logs. ([#152](https://github.com/SkyeAv/Tablassert/pull/152)) **Migration:** install `pip install "tablassert[log]"` anywhere logs are relied on; a base install is now silent rather than degraded. ### Added -- **DAKP sparse qualifier stack now survives `prune_to_class` on the pinned classes.** `CLASS_FIELD_OVERRIDES` grants `anatomical_context_qualifier`, `sex_qualifier`, `population_context_qualifier`, `frequency_qualifier`, and `temporal_context_qualifier` to `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` alongside the existing `disease_context_qualifier` / `regulatory_approvals` grants, so DAKP assertion rows carrying those qualifiers keep them on the edge instead of having them nulled and rescued into the pruned column. Deliberately ungranted: `species_context_qualifier` (disabled -- never emittable) plus `temporal_interval_qualifier` and `severity_qualifier` (unsatisfiable -- attached to no class, so a grant could never validate). Record validation already strips granted fields from its in-memory copy, so no new `extra_forbidden` surface appears; the existing tripwire test fails the suite the moment a biolink-model release attaches a granted slot, until the stale grant is removed. ([#188](https://github.com/SkyeAv/Tablassert/pull/188)) -- **`tablassert.net` is the new stdlib-only network-resilience seam shared by the agent and the BABEL downloader.** It classifies transient versus permanent failures through one ordered table (DNS/connection/timeout failures, HTTP 408/425/429 and every 5xx, and rate-limit/quota messages are transient; other 4xx and local `OSError`s such as `ENOSPC` fail fast), retries transient failures with bounded jittered backoff -- 4 attempts, 1.0 s base doubling to 20.0 s, 60 s total -- and honors `Retry-After` headers and `(reset after ...)` quota hints (which set the wait, capped at 60 s). The agent's article fetches (`_http_get_text` / `_http_get_bytes`) route through it, so a resolver blip no longer terminal-skips an article: exhausted retries raise a coded `network-transient` error instead. ([#163](https://github.com/SkyeAv/Tablassert/pull/163), [#164](https://github.com/SkyeAv/Tablassert/pull/164)) +- **DAKP sparse qualifier stack now survives `prune_to_class` on the pinned classes.** `CLASS_FIELD_OVERRIDES` grants `anatomical_context_qualifier`, `sex_qualifier`, `population_context_qualifier`, `frequency_qualifier`, and `temporal_context_qualifier` to `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` alongside the existing `disease_context_qualifier` / `regulatory_approvals` grants, so DAKP assertion rows carrying those qualifiers keep them on the edge instead of having them nulled and rescued into the pruned column. Deliberately ungranted: `species_context_qualifier` (disabled, never emittable) plus `temporal_interval_qualifier` and `severity_qualifier` (unsatisfiable, attached to no class, so a grant could never validate). Record validation already strips granted fields from its in-memory copy, so no new `extra_forbidden` surface appears; the existing tripwire test fails the suite the moment a biolink-model release attaches a granted slot, until the stale grant is removed. ([#188](https://github.com/SkyeAv/Tablassert/pull/188)) +- **`tablassert.net` is the new stdlib-only network-resilience seam shared by the agent and the BABEL downloader.** It classifies transient versus permanent failures through one ordered table (DNS/connection/timeout failures, HTTP 408/425/429 and every 5xx, and rate-limit/quota messages are transient; other 4xx and local `OSError`s such as `ENOSPC` fail fast), retries transient failures with bounded jittered backoff (4 attempts, 1.0 s base doubling to 20.0 s, 60 s total) and honors `Retry-After` headers and `(reset after ...)` quota hints (which set the wait, capped at 60 s). The agent's article fetches (`_http_get_text` / `_http_get_bytes`) route through it, so a resolver blip no longer terminal-skips an article: exhausted retries raise a coded `network-transient` error instead. ([#163](https://github.com/SkyeAv/Tablassert/pull/163), [#164](https://github.com/SkyeAv/Tablassert/pull/164)) - **Agent LLM calls now use one bounded, jittered retry layer across the inner agent, reflexion, and judge.** Smolagents' long rate-limit retryer and the OpenAI client's transport retries are disabled, while the shared seam retries DNS, timeouts, 429, and 5xx failures with Retry-After and a 45-second per-call budget; LiteLLM's backend-internal retry remains outside its constructor controls. The worst-case 29 logical calls per article add at most 21.8 minutes, and exhausted calls persist `llm-transient` without exposing configured credentials in logs or state. ([#167](https://github.com/SkyeAv/Tablassert/pull/167)) - **Agent checkpoint records now expose machine-readable `error_code` values for skipped articles.** `network-transient` and `llm-transient` identify outcomes that a fleet consumer should requeue; deterministic gates and pre-field state files remain `null`. The supervisor also emits one visible `ERROR` log line for every catch-all skip while preserving the existing `SKIPPED` status and notes prefix for mixed-version consumers. ([#166](https://github.com/SkyeAv/Tablassert/pull/166)) - **`build-kg` now uses an automatic ephemeral shared-prefix TCode cache when guarded expensive prefixes are shared by at least two sections.** Content-addressed XXH64 op digests key each prefix, worth-caching guards (`OP_COST` plus checkpoint boundaries) keep trivial or unshared prefixes out of the cache, the planner reorders section execution so shared prefixes are computed once, and the parquet snapshots live in a temp dir that is cleaned up after the build, including on failure. The final write reuses the producer's snapshot rather than recollecting it. ([#155](https://github.com/SkyeAv/Tablassert/pull/155), [#156](https://github.com/SkyeAv/Tablassert/pull/156), [#157](https://github.com/SkyeAv/Tablassert/pull/157), [#158](https://github.com/SkyeAv/Tablassert/pull/158), [#159](https://github.com/SkyeAv/Tablassert/pull/159), [#161](https://github.com/SkyeAv/Tablassert/pull/161)) -- **The `--distill` corpus is now schema-uniform, carries each run's outcome, and can be turned into a weighted SFT dataset.** Every recorded line emits the full canonical key set with explicit nulls (v2 adds `record_type`, `schema_version`, and a `run_id` join key), because `datasets.load_dataset("json")` infers features from the first block of the first file and raises `CastError` when a later block of an append-only corpus introduces a new column. A sibling `outcomes.ndjson` receives one 35-key outcome line per supervisor run -- captured at every terminal exit including the two validate-gate skips, the derive-mode exit, and the crash path, so failed runs persist as negative examples -- assembled by the new pure `src/tablassert/distill_reward.py` from data the supervisor already holds, with no extra build, audit, or KGX read. That module also scores each run with a deterministic reward clamped to [0, 1] (0.40 coverage, 0.28 Biolink validity, 0.17 specificity, 0.07 tool-call cleanliness, 0.08 breadth, with multiplicative hard gates and farming penalties) whose coefficients are tunable through a YAML/JSON `RewardConfig` that fails loud with the coded `reward-config-invalid`. The new `tablassert distill-weigh` composes join, reward, and selection (`threshold` / `best-of-n` / `replication`) into one row per input record plus a deterministic manifest, and is zero-dependency (stdlib plus PyYAML, no `[distill]` extra). This is data selection for supervised LoRA/QLoRA fine-tuning and explicitly not RLHF: no reward model, no PPO/GRPO/DPO/KTO, and no trainer import, enforced by an executable source-scan test. ([#180](https://github.com/SkyeAv/Tablassert/pull/180), [#181](https://github.com/SkyeAv/Tablassert/pull/181), [#182](https://github.com/SkyeAv/Tablassert/pull/182), [#183](https://github.com/SkyeAv/Tablassert/pull/183)) +- **The `--distill` corpus is now schema-uniform, carries each run's outcome, and can be turned into a weighted SFT dataset.** Every recorded line emits the full canonical key set with explicit nulls (v2 adds `record_type`, `schema_version`, and a `run_id` join key), because `datasets.load_dataset("json")` infers features from the first block of the first file and raises `CastError` when a later block of an append-only corpus introduces a new column. A sibling `outcomes.ndjson` receives one 35-key outcome line per supervisor run, captured at every terminal exit including the two validate-gate skips, the derive-mode exit, and the crash path, so failed runs persist as negative examples, assembled by the new pure `src/tablassert/distill_reward.py` from data the supervisor already holds, with no extra build, audit, or KGX read. That module also scores each run with a deterministic reward clamped to [0, 1] (0.40 coverage, 0.28 Biolink validity, 0.17 specificity, 0.07 tool-call cleanliness, 0.08 breadth, with multiplicative hard gates and farming penalties) whose coefficients are tunable through a YAML/JSON `RewardConfig` that fails loud with the coded `reward-config-invalid`. The new `tablassert distill-weigh` composes join, reward, and selection (`threshold` / `best-of-n` / `replication`) into one row per input record plus a deterministic manifest, and is zero-dependency (stdlib plus PyYAML, no `[distill]` extra). This is data selection for supervised LoRA/QLoRA fine-tuning and explicitly not RLHF: no reward model, no PPO/GRPO/DPO/KTO, and no trainer import, enforced by an executable source-scan test. ([#180](https://github.com/SkyeAv/Tablassert/pull/180), [#181](https://github.com/SkyeAv/Tablassert/pull/181), [#182](https://github.com/SkyeAv/Tablassert/pull/182), [#183](https://github.com/SkyeAv/Tablassert/pull/183)) ### Fixed - **`distill-export` no longer breaks on an append-only corpus whose files disagree in shape.** `datasets.load_dataset("json")` infers its column schema from the first block of the first file only and casts every later table to it, so a directory holding both v1 and v2 records either raised `CastError` or silently JSON-encoded a varying-type column into a string, and outcome files sitting alongside records made a record-shaped corpus unexportable. Export now partitions the directory by content (`is_outcome_file` reads the first non-blank line's `record_type`, so a renamed file still partitions correctly), unions the schema across every record file into one uniform temporary NDJSON written outside `--distill-dir` and always removed, and rejects genuine type conflicts with exit 2 naming the key and types before the `[distill]` preflight and the datasets import. Record files containing zero rows now fail loud instead of surfacing a bare `StopIteration` from inside the datasets JSON reader. Flags, defaults, exit codes, and preflight ordering are unchanged. ([#184](https://github.com/SkyeAv/Tablassert/pull/184)) @@ -149,8 +149,8 @@ All notable changes to this project are documented in this file. - **BABEL download retry warnings now survive the loguru sink.** The retry-warning log bound the raw exception object, which the `enqueue=True` sink cannot pickle (`HTTPError.__init__` requires five positional arguments), so every BABEL retry warning line was silently dropped with an internal `--- End of logging error ---`. It now binds the stringified error, and the same hazard was fixed on the new `NET` retry logging at its seam. ([#163](https://github.com/SkyeAv/Tablassert/pull/163)) ### Performance -- **PMC article downloads are idempotent, atomic, and bounded-parallel.** A file that already exists non-empty is skipped without any request (a torn zero-length file re-downloads), each download writes a `.part` sibling that is atomically `os.replace`d into place on completion — and removed even on `KeyboardInterrupt`, so no half-written file survives — and up to 8 files fetch concurrently by default with results collected in submission order. `concurrency < 1` now fails before any network call is made. Previously every file downloaded sequentially, single-attempt, on every rerun. ([#165](https://github.com/SkyeAv/Tablassert/pull/165)) -- **Reflexion and judge prompts are hard-bounded to fit a context window.** The reflexion prompt caps its current-config, coverage-report, and article-context blocks at 8,000 / 8,000 / 40,000 chars and the judge prompt caps its serialized report at 8,000 chars; every `unresolved` term list keeps 20 entries plus a visible `+N more` marker and an `unresolved_count` naming the original length. The worst-case prompt is ≈ 58,000 chars ≈ 15,000 tokens — the fleet distill telemetry that motivated this recorded 1.6M-character / 689,241-input-token reflexion prompts (a ≥ 27× reduction). Public coverage and audit reports are unchanged; compaction happens only at prompt serialization. ([#168](https://github.com/SkyeAv/Tablassert/pull/168)) +- **PMC article downloads are idempotent, atomic, and bounded-parallel.** A file that already exists non-empty is skipped without any request (a torn zero-length file re-downloads), each download writes a `.part` sibling that is atomically `os.replace`d into place on completion (and removed even on `KeyboardInterrupt`, so no half-written file survives) and up to 8 files fetch concurrently by default with results collected in submission order. `concurrency < 1` now fails before any network call is made. Previously every file downloaded sequentially, single-attempt, on every rerun. ([#165](https://github.com/SkyeAv/Tablassert/pull/165)) +- **Reflexion and judge prompts are hard-bounded to fit a context window.** The reflexion prompt caps its current-config, coverage-report, and article-context blocks at 8,000 / 8,000 / 40,000 chars and the judge prompt caps its serialized report at 8,000 chars; every `unresolved` term list keeps 20 entries plus a visible `+N more` marker and an `unresolved_count` naming the original length. The worst-case prompt is ~ 58,000 chars ~ 15,000 tokens, the fleet distill telemetry that motivated this recorded 1.6M-character / 689,241-input-token reflexion prompts (a >= 27x reduction). Public coverage and audit reports are unchanged; compaction happens only at prompt serialization. ([#168](https://github.com/SkyeAv/Tablassert/pull/168)) ### Documentation - **`docs/agent.md` now documents the network-resilience layer, `error_code`, and the prompt bounds.** A new `## Network resilience` section covers the transient/permanent classification table, the HTTP and LLM retry budgets, `Retry-After` honoring with its 60 s cap, idempotent atomic parallel downloads, the worst-case wall-clock arithmetic against the 90-minute per-article budget, and the single-retry-layer rule; the checkpoint prose documents `error_code` values and requeue semantics; the reflexion/judge prose documents the prompt-bound constants and the fleet evidence behind them. `llms.txt` indexes the new `net.py` module. ([#169](https://github.com/SkyeAv/Tablassert/pull/169)) @@ -164,12 +164,12 @@ All notable changes to this project are documented in this file. ## 18.1.0 - 2026-09-14 ### Added -- **`build-fullmap --taxon-allowlist` builds an opt-in, source-derived fullmap restricted to the top 100 experimental taxa, and InChIKey terms now survive ingestion.** The flag reads the checked-in `src/tablassert/data/experimental_taxa.yaml` allowlist (100 NCBI taxon IDs ranked by observed frequency across the `MultiomicsNext` and `MultiomicsHarness` corpora) and filters non-`OrganismTaxon` synonym rows carrying valid taxon metadata in `rust/src/fullmap.rs` before interning, yielding a smaller database while retaining taxonless rows (including `NCBITaxon:0` and malformed taxon values), rows with any allowlisted taxon, and every row whose categories include `OrganismTaxon`; the first taxon is still the one stored, so hydration and lookup behavior are unchanged. Filtered builds always come from BABEL sources, never reuse the unfiltered prebuilt archive, and record a deterministic `META.taxon_allowlist` identity so a filtered database cannot be mistaken for a full one. Separately, Rust token QC no longer rejects `inchikey` -- newline, tab, hypothetical, uncharacterized, and dead-token filtering are all preserved -- so InChIKey terms are indexed and resolvable in both raw normalized (`abc-def`) and CURIE (`inchikey:abc-def`) form. Default fullmap builds, the prebuilt download path, and existing Python callers are untouched. ([#149](https://github.com/SkyeAv/Tablassert/pull/149)) +- **`build-fullmap --taxon-allowlist` builds an opt-in, source-derived fullmap restricted to the top 100 experimental taxa, and InChIKey terms now survive ingestion.** The flag reads the checked-in `src/tablassert/data/experimental_taxa.yaml` allowlist (100 NCBI taxon IDs ranked by observed frequency across the `MultiomicsNext` and `MultiomicsHarness` corpora) and filters non-`OrganismTaxon` synonym rows carrying valid taxon metadata in `rust/src/fullmap.rs` before interning, yielding a smaller database while retaining taxonless rows (including `NCBITaxon:0` and malformed taxon values), rows with any allowlisted taxon, and every row whose categories include `OrganismTaxon`; the first taxon is still the one stored, so hydration and lookup behavior are unchanged. Filtered builds always come from BABEL sources, never reuse the unfiltered prebuilt archive, and record a deterministic `META.taxon_allowlist` identity so a filtered database cannot be mistaken for a full one. Separately, Rust token QC no longer rejects `inchikey` (newline, tab, hypothetical, uncharacterized, and dead-token filtering are all preserved) so InChIKey terms are indexed and resolvable in both raw normalized (`abc-def`) and CURIE (`inchikey:abc-def`) form. Default fullmap builds, the prebuilt download path, and existing Python callers are untouched. ([#149](https://github.com/SkyeAv/Tablassert/pull/149)) ### Fixed -- **The `Categories` config vocabulary now names fullmap-emitted Biolink mixins via `CATEGORY_OVERRIDES`.** Biolink declares some node categories as mixins, and the pydantic model generates a mixin as a plain `ConfiguredBaseModel` that is not an `Entity` subclass -- invisible to the scan that builds `Categories`. The fullmap's BABEL vocabulary assigns one such mixin (`GenomicEntity`) to real terms, so an `avoid` allow-list could never name it: the category sailed through the guard and the edge fell back to bare `biolink:Association`, whose `prune_to_class` then nulled every class-specific slot into the inlined `has_supporting_studies` junk drawer (DAKP shipped 35 such edges). `GenomicEntity` is now a first-class member -- valid in `avoid`/`prioritize` and as a `category_override` key -- and a tripwire test drops the override the moment the installed model makes the mixin an `Entity` subclass (tests in `tests/test_biolink.py` and `tests/test_models.py`). ([#151](https://github.com/SkyeAv/Tablassert/pull/151)) +- **The `Categories` config vocabulary now names fullmap-emitted Biolink mixins via `CATEGORY_OVERRIDES`.** Biolink declares some node categories as mixins, and the pydantic model generates a mixin as a plain `ConfiguredBaseModel` that is not an `Entity` subclass, invisible to the scan that builds `Categories`. The fullmap's BABEL vocabulary assigns one such mixin (`GenomicEntity`) to real terms, so an `avoid` allow-list could never name it: the category sailed through the guard and the edge fell back to bare `biolink:Association`, whose `prune_to_class` then nulled every class-specific slot into the inlined `has_supporting_studies` junk drawer (DAKP shipped 35 such edges). `GenomicEntity` is now a first-class member, valid in `avoid`/`prioritize` and as a `category_override` key, and a tripwire test drops the override the moment the installed model makes the mixin an `Entity` subclass (tests in `tests/test_biolink.py` and `tests/test_models.py`). ([#151](https://github.com/SkyeAv/Tablassert/pull/151)) - **`filter_and_rank` drops categories the `Categories` enum cannot name when `avoid` is set.** An `avoid` list is an allow-list by complement, and a complement can only name what the enum knows: a fullmap `CATEGORY_NAME` outside the enum could never appear in it, so without this guard the hard filter had a silent hole for any future vocabulary drift. Degradation is now to lost rows (surfaced in the resolution report) rather than mis-classed edges (regression tests in `tests/test_fullmap.py`). ([#151](https://github.com/SkyeAv/Tablassert/pull/151)) -- **The `--qc` study stage now fails the build on edges demoted to bare `biolink:Association` when any section declares a `category_override`.** A pinned section promises every row lands on a pinned class, so a demotion means a row escaped the pin -- typically an object category the config vocabulary could not name -- and shipped without its class-specific evidence slots. Without a pin, bare `Association` remains the author's accepted default, so the assertion is gated. `study_kgx` gained the `category_override_declared` keyword; `study_final_ndjson` forwards it from the built sections (regression tests in `tests/test_study.py`). ([#151](https://github.com/SkyeAv/Tablassert/pull/151)) +- **The `--qc` study stage now fails the build on edges demoted to bare `biolink:Association` when any section declares a `category_override`.** A pinned section promises every row lands on a pinned class, so a demotion means a row escaped the pin, typically an object category the config vocabulary could not name, and shipped without its class-specific evidence slots. Without a pin, bare `Association` remains the author's accepted default, so the assertion is gated. `study_kgx` gained the `category_override_declared` keyword; `study_final_ndjson` forwards it from the built sections (regression tests in `tests/test_study.py`). ([#151](https://github.com/SkyeAv/Tablassert/pull/151)) ## 18.0.0 - 2026-09-10 @@ -193,17 +193,17 @@ All notable changes to this project are documented in this file. ## 17.0.1 - 2026-09-09 ### Fixed -- **The fullmap resolver now retains every CURIE tied on a term's best ranking tier instead of keeping one arbitrary winner.** `distinct` sorted by `term` / `PR` / `NLP_LEVEL` (plus the `column_context` category-`FREQUENCY` tiebreaker when set) and kept only the first row per term, so distinct CURIEs tied across the entire ranking tuple were silently discarded before `resolve` ever saw them — a term matching `HGNC:1` and `HGNC:2` equally well shipped only whichever the sort happened to surface. The op now computes each term's best ranking tier, retains every row in it, and collapses duplicates only by `(term, CURIE)`; tied CURIEs propagate through `resolve`, `resolve_batch`, and `join_matches`, so a source row may expand to one output row per tied best CURIE (level-one still beats level-two, and `column_context` still prefers the more frequent category). Regression tests pin the tier retention, duplicate `(term, CURIE)` collapse, and lower-tier exclusion (`test_filter_and_rank_collapses_duplicate_curies_and_keeps_equal_best_curies`), plus end-to-end row expansion through `resolve` and `join_matches`. ([#145](https://github.com/SkyeAv/Tablassert/pull/145)) +- **The fullmap resolver now retains every CURIE tied on a term's best ranking tier instead of keeping one arbitrary winner.** `distinct` sorted by `term` / `PR` / `NLP_LEVEL` (plus the `column_context` category-`FREQUENCY` tiebreaker when set) and kept only the first row per term, so distinct CURIEs tied across the entire ranking tuple were silently discarded before `resolve` ever saw them; a term matching `HGNC:1` and `HGNC:2` equally well shipped only whichever the sort happened to surface. The op now computes each term's best ranking tier, retains every row in it, and collapses duplicates only by `(term, CURIE)`; tied CURIEs propagate through `resolve`, `resolve_batch`, and `join_matches`, so a source row may expand to one output row per tied best CURIE (level-one still beats level-two, and `column_context` still prefers the more frequent category). Regression tests pin the tier retention, duplicate `(term, CURIE)` collapse, and lower-tier exclusion (`test_filter_and_rank_collapses_duplicate_curies_and_keeps_equal_best_curies`), plus end-to-end row expansion through `resolve` and `join_matches`. ([#145](https://github.com/SkyeAv/Tablassert/pull/145)) ## 17.0.0 - 2026-09-08 ### Breaking Changes - **The `--threads` / `-t` CLI options (on `build-kg` and `build-fullmap`), the `threads` parameter on the Python API (`fullmap.resolve`, etc.), and the agent command's `--gepa-threads` option (`dspy.GEPA` evaluation pool) are gone.** All parallelism is now automatic: every parallel stage uses all available CPU threads, with the fullmap build still capped on Linux by available memory (~2 GB per thread, read from `/proc/meminfo`) to avoid OOMs, and entity-resolution lookups fanning out across the fullmap shards automatically only for batches of 1024+ terms (smaller batches stay serial). GEPA now uses its library default parallelism. Results are unchanged. ([#142](https://github.com/SkyeAv/Tablassert/pull/142)) - **Migration:** drop the `--threads` / `-t` and `--gepa-threads` flags from invocations and the `threads` / `num_threads` arguments from Python and Rust calls — they were optional, and the automatic behavior is exactly the previous default, so builds, lookups, and results need no other change; GEPA evaluation now runs at the library's default parallelism. + **Migration:** drop the `--threads` / `-t` and `--gepa-threads` flags from invocations and the `threads` / `num_threads` arguments from Python and Rust calls; they were optional, and the automatic behavior is exactly the previous default, so builds, lookups, and results need no other change; GEPA evaluation now runs at the library's default parallelism. ### Performance -- **The `uuid_on_collision: merge` edge pass is now near-linear in divergent rows per edge instead of quadratic, and its canonical serializer no longer clones per record.** The merge fold re-canonicalized and re-sorted every list item already stored on an edge on each incoming row — O(n²) in rows × `supporting_case_ids` — and `canonical_json_bytes` deep-cloned into a sorted-key `Value` tree before writing, a fixed cost paid once per record on every pass. Array fields now carry their items' canonical bytes, computed once when an item arrives, so each fold is O(1) per incoming item (hash-set membership, one deferred stable sort at write-out, and only for lists a real union touched), while the sorted-key form is written directly into the output buffer with scalars delegated to `serde_json`'s own writers. Measured on release builds (`uv run maturin develop --release`) over the seeded corpora (seed 42) of the committed harness — `TABLASSERT_BENCH=1 uv run pytest tests/bench_merge_bench.py -s -n 0 -q` — on an AMD Ryzen AI 9 HX 370 with Python 3.13: **37.45 s → 1.311 s (28.6x)** for 100,000 lines of 1k triples × 100 divergent rows at 50 case ids per row, **20.69 s → 2.155 s (9.6x)** for 100,000 lines of 5k triples × 20 rows at 100 case ids per row, **6.71 s → 5.184 s (1.29x)** for 600,000 lines of 200k triples × 3 rows at 10–20 case ids per row, and **0.59 s → 0.495 s (1.19x)** for 100,000 byte-identical rows, so the no-fold path does not pay for the new bookkeeping either. Merged-record and conflict counters are unchanged on every scenario (400,000 / 1,199,899; 95,000 / 284,979; 99,000 / 296,969), and the emitted bytes are identical to the previous fold — pinned by a seeded fuzz against a frozen verbatim copy of the old implementation (`cargo test merge_fold_matches_reference_on_fuzz`), an end-to-end pure-Python reference (`tests/test_rs.py::test_dedup_edges_merge_matches_python_reference`), and an in-process guard that fails below a 5x bound against that frozen quadratic reference (`cargo test fold_speedup`). The buffered merge state does carry per-list-item bookkeeping memory — one canonical-bytes copy per distinct list item, a single allocation shared between the membership set and the ordered write-out list — which is the price of the O(1) membership check and one more reason the mode stays opt-in: sharing that allocation rather than storing two copies drops peak RSS on the harness's 1k triples × 100 rows corpus (~5M stored list items) from 1.93 GB to 1.26 GB. ([#143](https://github.com/SkyeAv/Tablassert/pull/143)) +- **The `uuid_on_collision: merge` edge pass is now near-linear in divergent rows per edge instead of quadratic, and its canonical serializer no longer clones per record.** The merge fold re-canonicalized and re-sorted every list item already stored on an edge on each incoming row, O(n^2) in rows x `supporting_case_ids`, and `canonical_json_bytes` deep-cloned into a sorted-key `Value` tree before writing, a fixed cost paid once per record on every pass. Array fields now carry their items' canonical bytes, computed once when an item arrives, so each fold is O(1) per incoming item (hash-set membership, one deferred stable sort at write-out, and only for lists a real union touched), while the sorted-key form is written directly into the output buffer with scalars delegated to `serde_json`'s own writers. Measured on release builds (`uv run maturin develop --release`) over the seeded corpora (seed 42) of the committed harness, `TABLASSERT_BENCH=1 uv run pytest tests/bench_merge_bench.py -s -n 0 -q`, on an AMD Ryzen AI 9 HX 370 with Python 3.13: **37.45 s to 1.311 s (28.6x)** for 100,000 lines of 1k triples x 100 divergent rows at 50 case ids per row, **20.69 s to 2.155 s (9.6x)** for 100,000 lines of 5k triples x 20 rows at 100 case ids per row, **6.71 s to 5.184 s (1.29x)** for 600,000 lines of 200k triples x 3 rows at 10-20 case ids per row, and **0.59 s to 0.495 s (1.19x)** for 100,000 byte-identical rows, so the no-fold path does not pay for the new bookkeeping either. Merged-record and conflict counters are unchanged on every scenario (400,000 / 1,199,899; 95,000 / 284,979; 99,000 / 296,969), and the emitted bytes are identical to the previous fold, pinned by a seeded fuzz against a frozen verbatim copy of the old implementation (`cargo test merge_fold_matches_reference_on_fuzz`), an end-to-end pure-Python reference (`tests/test_rs.py::test_dedup_edges_merge_matches_python_reference`), and an in-process guard that fails below a 5x bound against that frozen quadratic reference (`cargo test fold_speedup`). The buffered merge state does carry per-list-item bookkeeping memory, one canonical-bytes copy per distinct list item, a single allocation shared between the membership set and the ordered write-out list, which is the price of the O(1) membership check and one more reason the mode stays opt-in: sharing that allocation rather than storing two copies drops peak RSS on the harness's 1k triples x 100 rows corpus (~5M stored list items) from 1.93 GB to 1.26 GB. ([#143](https://github.com/SkyeAv/Tablassert/pull/143)) ## 16.7.0 - 2026-09-04 @@ -213,43 +213,43 @@ All notable changes to this project are documented in this file. ## 16.6.2 - 2026-09-04 ### Fixed -- **`number_of_cases` is now emitted as a JSON integer instead of source text.** The slot is typed `int` on biolink-model's `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation`, but it was absent from `lib.numeric_columns`' exact set — since 15.1's `STUDY_SIZE_EXEMPT_PATTERN` (#119) stopped it being *renamed* onto `study_size`, nothing cast its *values*, so the raw TSV cell (`"1"`) shipped on the edge NDJSON. Pydantic's lax validation coerced the string back to int inside `validate_kgx`, so the record still validated, and the Rust `uuid_on_collision: merge` recompute already wrote a real int — leaving the shipped graph type-inconsistent edge to edge. ([#139](https://github.com/SkyeAv/Tablassert/pull/139)) `number_of_cases` now rides the same `clean_numeric` / `format_numeric` machinery as `study_size`: `numeric_slot_kind` reads the `int` typing off the installed model, fractional / negative / non-numeric counts become null, and the release-mode `drop_low_number_of_cases` filter is untouched (it already cast inline). -- **`--release` builds no longer reuse section parquets cached by other build modes.** The section-parquet cache key was `STORE / f"{mkhash(s)}.parquet"` with `release` injected into `Tcode` outside the hashed dict, and `Tcode.collect` quick-exits on any existing store file — so a parquet cached by a non-release build (or a pre-16.6.0 build, where the case-count filter silently no-op'd) was reused verbatim in `--release` builds, skipping the release filters and shipping `applied_to_treat` edges with `number_of_cases < 25`. A new `_section_store_path` helper generalizes the existing `.head.parquet` idiom: `--head`, `--release`, and `--qc` each get their own suffix, composing when combined (`.head.release.qc.parquet`). `--qc` belongs because `fullmap_audit` drops rejects from the cached parquet too — the same cache-poisoning class. ([#140](https://github.com/SkyeAv/Tablassert/pull/140)) +- **`number_of_cases` is now emitted as a JSON integer instead of source text.** The slot is typed `int` on biolink-model's `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation`, but it was absent from `lib.numeric_columns`' exact set; since 15.1's `STUDY_SIZE_EXEMPT_PATTERN` (#119) stopped it being *renamed* onto `study_size`, nothing cast its *values*, so the raw TSV cell (`"1"`) shipped on the edge NDJSON. Pydantic's lax validation coerced the string back to int inside `validate_kgx`, so the record still validated, and the Rust `uuid_on_collision: merge` recompute already wrote a real int, leaving the shipped graph type-inconsistent edge to edge. ([#139](https://github.com/SkyeAv/Tablassert/pull/139)) `number_of_cases` now rides the same `clean_numeric` / `format_numeric` machinery as `study_size`: `numeric_slot_kind` reads the `int` typing off the installed model, fractional / negative / non-numeric counts become null, and the release-mode `drop_low_number_of_cases` filter is untouched (it already cast inline). +- **`--release` builds no longer reuse section parquets cached by other build modes.** The section-parquet cache key was `STORE / f"{mkhash(s)}.parquet"` with `release` injected into `Tcode` outside the hashed dict, and `Tcode.collect` quick-exits on any existing store file, so a parquet cached by a non-release build (or a pre-16.6.0 build, where the case-count filter silently no-op'd) was reused verbatim in `--release` builds, skipping the release filters and shipping `applied_to_treat` edges with `number_of_cases < 25`. A new `_section_store_path` helper generalizes the existing `.head.parquet` idiom: `--head`, `--release`, and `--qc` each get their own suffix, composing when combined (`.head.release.qc.parquet`). `--qc` belongs because `fullmap_audit` drops rejects from the cached parquet too, the same cache-poisoning class. ([#140](https://github.com/SkyeAv/Tablassert/pull/140)) - **The release-mode case-count and effect-size filters tolerate non-numeric cells instead of crashing.** `drop_low_number_of_cases` and `drop_zero_effect_size` cast with strict `Float64`, but the `csv` op reads sources with `has_header=False`, so a TSV's header row flows through as data and the strict cast raised `InvalidOperationError` on the non-numeric header cell. Both ops now cast with `strict=False` (the `clean_numeric` idiom); non-numeric cells become null and are kept, dying later at entity resolution as before. ([#140](https://github.com/SkyeAv/Tablassert/pull/140)) ## 16.6.1 - 2026-09-04 ### Performance -- **The agent's LLM workflow now carries a far smaller tool surface, tighter context, and smaller persisted configs.** Full mode registers exactly four tools — `read_table`, `pmc_article_context`, `derive_config`, `build_and_audit` — because coverage improvement left the LLM's hands entirely: `map_coverage` and `propose_config_edit` are pure helpers the deterministic supervisor calls in its own improve loop, and the opt-in `--reflexion` path is the only place an LLM call is still spent on improvement. The task text now injects a bounded per-column `column_digest` for every previewed table/worksheet (separator fractions over the first 500 data rows, plus non-null/distinct counts, max cell length, and sample values), so the canonical path is a fixed derive → build → answer of three or fewer steps with zero inspection tool calls, and the agent rebuilds only on a coded build error — fixing exactly the field the error names, at most twice. The LLM sees a compact `build_and_audit` observation (twelve high-signal keys, with `unresolved` capped at 20 entries and a visible `+N more` marker) while the supervisor still receives the full report, and the accepted best config is **compacted deterministically** before persistence: `compact_config` strips keys equal to their Pydantic model defaults while preserving semantics — the pinned accuracy-invariance test proves the compacted config builds the identical KGX and scores the identical `quality_score`, and any compaction failure falls back to the exact original config. `state.json` records `config_chars` so the shrink is auditable per article. Builds, scores, and persisted configs remain behaviorally equivalent. ([#138](https://github.com/SkyeAv/Tablassert/pull/138)) +- **The agent's LLM workflow now carries a far smaller tool surface, tighter context, and smaller persisted configs.** Full mode registers exactly four tools (`read_table`, `pmc_article_context`, `derive_config`, `build_and_audit`) because coverage improvement left the LLM's hands entirely: `map_coverage` and `propose_config_edit` are pure helpers the deterministic supervisor calls in its own improve loop, and the opt-in `--reflexion` path is the only place an LLM call is still spent on improvement. The task text now injects a bounded per-column `column_digest` for every previewed table/worksheet (separator fractions over the first 500 data rows, plus non-null/distinct counts, max cell length, and sample values), so the canonical path is a fixed derive, then build, then answer of three or fewer steps with zero inspection tool calls, and the agent rebuilds only on a coded build error, fixing exactly the field the error names, at most twice. The LLM sees a compact `build_and_audit` observation (twelve high-signal keys, with `unresolved` capped at 20 entries and a visible `+N more` marker) while the supervisor still receives the full report, and the accepted best config is **compacted deterministically** before persistence: `compact_config` strips keys equal to their Pydantic model defaults while preserving semantics; the pinned accuracy-invariance test proves the compacted config builds the identical KGX and scores the identical `quality_score`, and any compaction failure falls back to the exact original config. `state.json` records `config_chars` so the shrink is auditable per article. Builds, scores, and persisted configs remain behaviorally equivalent. ([#138](https://github.com/SkyeAv/Tablassert/pull/138)) ## 16.6.0 - 2026-09-04 ### Added -- **The agent can now record every LLM call as a ChatML fine-tuning dataset.** `tablassert agent --distill` (`-d` / `-dt`) appends one ChatML record per line to `/distill/records.ndjson` for every LLM call of the run — inner agent, judge, and reflexion alike. The file is append-only across invocations, so a corpus accumulates run over run and loads directly in Unsloth Studio or via `datasets.load_dataset("json")`, and every record is tagged with its `pmc_id` so `state.json` filtering can slice per article. Recording lives in the zero-dependency `tablassert.distill` (`DistillRecorder` plus ChatML serializers) and never raises into a batch; `make_distilling_model` wraps the model's `generate()` — the single capture seam — and `tablassert distill-export` (new `[distill]` extra) converts the NDJSON to an on-disk Hugging Face dataset. ([#136](https://github.com/SkyeAv/Tablassert/pull/136)) -- **`uuid_on_collision: merge` now recomputes `number_of_cases` as the exact union of a new build-internal `supporting_case_ids` edge field.** Folding two divergent same-id edges used to keep the first record's `number_of_cases` (under-reporting the merged evidence) and count the divergence as a scalar conflict; summing the two counts would instead double-count every shared case. Edge frames may now carry a `supporting_case_ids` `list[str]` column — allow-listed via `TABLASERT_EDGE_EXTRAS` so it is never folded into `supporting_text`, and declared by no association class so `prune_to_class` leaves it untouched without a `CLASS_FIELD_OVERRIDES` grant. When a merged record carries the list and either side carried a count, the Rust merge sets `number_of_cases` to the union length (shared case IDs count once) and reports no scalar conflict; merges with no carrier keep first-wins exactly. The carrier is stripped from every edge record before write in both the merge and default streaming dedup paths, so it never ships in the final NDJSON and QC/`validate_kgx` never see it. +- **The agent can now record every LLM call as a ChatML fine-tuning dataset.** `tablassert agent --distill` (`-d` / `-dt`) appends one ChatML record per line to `/distill/records.ndjson` for every LLM call of the run, inner agent, judge, and reflexion alike. The file is append-only across invocations, so a corpus accumulates run over run and loads directly in Unsloth Studio or via `datasets.load_dataset("json")`, and every record is tagged with its `pmc_id` so `state.json` filtering can slice per article. Recording lives in the zero-dependency `tablassert.distill` (`DistillRecorder` plus ChatML serializers) and never raises into a batch; `make_distilling_model` wraps the model's `generate()`, the single capture seam, and `tablassert distill-export` (new `[distill]` extra) converts the NDJSON to an on-disk Hugging Face dataset. ([#136](https://github.com/SkyeAv/Tablassert/pull/136)) +- **`uuid_on_collision: merge` now recomputes `number_of_cases` as the exact union of a new build-internal `supporting_case_ids` edge field.** Folding two divergent same-id edges used to keep the first record's `number_of_cases` (under-reporting the merged evidence) and count the divergence as a scalar conflict; summing the two counts would instead double-count every shared case. Edge frames may now carry a `supporting_case_ids` `list[str]` column, allow-listed via `TABLASERT_EDGE_EXTRAS` so it is never folded into `supporting_text`, and declared by no association class so `prune_to_class` leaves it untouched without a `CLASS_FIELD_OVERRIDES` grant. When a merged record carries the list and either side carried a count, the Rust merge sets `number_of_cases` to the union length (shared case IDs count once) and reports no scalar conflict; merges with no carrier keep first-wins exactly. The carrier is stripped from every edge record before write in both the merge and default streaming dedup paths, so it never ships in the final NDJSON and QC/`validate_kgx` never see it. ### Fixed -- **The release-mode case-count filter for `applied_to_treat` edges now reads the column the pipeline actually emits.** The filter shipped in 16.1.0 read a `case_count` column that never exists by the `significance` phase: a literal `case_count` source header is coerced onto `study_size` (unit `case` + count noun `count`, alias dropped), and the case counts that do survive travel under the exempt Biolink slot `number_of_cases` — so the filter was a silent no-op in every release build since, and small-cohort treatment edges still shipped. The op is renamed `drop_low_number_of_cases` and now reads `number_of_cases`; gating (`--release` *and* `applied_to_treat`), the 25-case threshold, null handling (nulls kept), and `significance`-phase placement are unchanged — but release builds now actually drop `applied_to_treat` edges backed by fewer than 25 cases, as documented since 16.1.0. ([#137](https://github.com/SkyeAv/Tablassert/pull/137)) +- **The release-mode case-count filter for `applied_to_treat` edges now reads the column the pipeline actually emits.** The filter shipped in 16.1.0 read a `case_count` column that never exists by the `significance` phase: a literal `case_count` source header is coerced onto `study_size` (unit `case` + count noun `count`, alias dropped), and the case counts that do survive travel under the exempt Biolink slot `number_of_cases`, so the filter was a silent no-op in every release build since, and small-cohort treatment edges still shipped. The op is renamed `drop_low_number_of_cases` and now reads `number_of_cases`; gating (`--release` *and* `applied_to_treat`), the 25-case threshold, null handling (nulls kept), and `significance`-phase placement are unchanged, but release builds now actually drop `applied_to_treat` edges backed by fewer than 25 cases, as documented since 16.1.0. ([#137](https://github.com/SkyeAv/Tablassert/pull/137)) ## 16.5.0 - 2026-09-03 ### Added -- **The agent now authors detail-first configs, backed by actionable audit feedback and deterministic `explode_by`/predicate fixes.** Generated configs used to ship shallow — generic `associated_with` predicates, missed `explode_by` on delimited cells, absent qualifiers, fragile regexes — so goal ordering is now breadth → coverage → Biolink/QC → efficiency and `INSTRUCTIONS` gained a regex cookbook (Rust-regex semantics, YAML single-quoting), positive qualifier guidance, and a most-specific-legal-predicate rule, with exemplars moved off generic `associated_with` (including a new rich exemplar combining `explode_by`, a qualifier, a regex, and a statistical pair) and the template converted to a raw string, fixing a real `\1` octal-escape corruption. `build_and_audit` now returns actionable signals — `predicate_advice` (legal predicates per demoted category pair), `multivalued_suspects` (missed-`explode_by` detection), and a `head` fidelity flag — that the deterministic proposer consumes to add `explode_by` from separator-carrying unresolved terms and fix demoted predicates, also correcting a pre-existing misfire where joined cells like `brca1;tp53` were misread as taxonomic. Tier-2 LLM reflexion can now touch `explode_by`/`split_by`/qualifiers/predicate, while `_is_improvement` rejects any edit that shrinks the full-build edge count by more than 25%; `docs/agent.md` documents the new behavior. ([#135](https://github.com/SkyeAv/Tablassert/pull/135)) +- **The agent now authors detail-first configs, backed by actionable audit feedback and deterministic `explode_by`/predicate fixes.** Generated configs used to ship shallow: generic `associated_with` predicates, missed `explode_by` on delimited cells, absent qualifiers, fragile regexes, so goal ordering is now breadth, then coverage, then Biolink/QC, then efficiency and `INSTRUCTIONS` gained a regex cookbook (Rust-regex semantics, YAML single-quoting), positive qualifier guidance, and a most-specific-legal-predicate rule, with exemplars moved off generic `associated_with` (including a new rich exemplar combining `explode_by`, a qualifier, a regex, and a statistical pair) and the template converted to a raw string, fixing a real `\1` octal-escape corruption. `build_and_audit` now returns actionable signals: `predicate_advice` (legal predicates per demoted category pair), `multivalued_suspects` (missed-`explode_by` detection), and a `head` fidelity flag that the deterministic proposer consumes to add `explode_by` from separator-carrying unresolved terms and fix demoted predicates, also correcting a pre-existing misfire where joined cells like `brca1;tp53` were misread as taxonomic. Tier-2 LLM reflexion can now touch `explode_by`/`split_by`/qualifiers/predicate, while `_is_improvement` rejects any edit that shrinks the full-build edge count by more than 25%; `docs/agent.md` documents the new behavior. ([#135](https://github.com/SkyeAv/Tablassert/pull/135)) ### Fixed -- **Underscore-glued adjusted P value headers no longer ship as raw `p_value`.** `CONTEXTUAL_ADJUSTED_PATTERN` anchored on `\b`, but `_` is a word character, so the boundary never fired between `adjusted` and `_p_value` — the exact hazard `BARE_P_TOKEN_PATTERN` already documents and works around with alphanumeric lookarounds, a workaround that was never mirrored onto the adjustment pattern. Every underscore-glued spelling (`adjusted_p_value`, `adj_p_value`, `corrected_p_value`, `p_adjusted`) classified as a **raw** p-value, so a column literally named `adjusted_p_value` was renamed to `p_value` and `sig` banded an FDR-adjusted number as a raw one. The spaced and hyphenated spellings were always correct, which is why it went unnoticed: `test_sig_prefers_exact_p_value_column` supplies both columns, so canonical-wins masked the misclassification. The pattern now uses the same lookarounds as the bare-P token, and the adjustment vocabulary gained `adjust` / `adjustment` so the real DESeq2-family header `p_adjust` is recognized too. Adjusted values in existing graphs were being emitted in the raw `p_value` slot; rebuild to correct them. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) +- **Underscore-glued adjusted P value headers no longer ship as raw `p_value`.** `CONTEXTUAL_ADJUSTED_PATTERN` anchored on `\b`, but `_` is a word character, so the boundary never fired between `adjusted` and `_p_value`, the exact hazard `BARE_P_TOKEN_PATTERN` already documents and works around with alphanumeric lookarounds, a workaround that was never mirrored onto the adjustment pattern. Every underscore-glued spelling (`adjusted_p_value`, `adj_p_value`, `corrected_p_value`, `p_adjusted`) classified as a **raw** p-value, so a column literally named `adjusted_p_value` was renamed to `p_value` and `sig` banded an FDR-adjusted number as a raw one. The spaced and hyphenated spellings were always correct, which is why it went unnoticed: `test_sig_prefers_exact_p_value_column` supplies both columns, so canonical-wins masked the misclassification. The pattern now uses the same lookarounds as the bare-P token, and the adjustment vocabulary gained `adjust` / `adjustment` so the real DESeq2-family header `p_adjust` is recognized too. Adjusted values in existing graphs were being emitted in the raw `p_value` slot; rebuild to correct them. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) ### Changed -- **Every coercion now drops its losing alias columns, not just the study-size and study-metadata ones.** The five coercions disagreed: `study_size` and `study_metadata` dropped their losers, while `p_value`, `effect_size` and `effect_type` left theirs on the frame for `fold_unknown_to_supporting_text` to pick up. A table with `effect size` and `odds ratio` therefore emitted `effect_size: 0.85` on the edge **and** `"odds ratio: 0.85"` inside `supporting_text` — one measurement twice, which is the duplication `coerce_study_size_columns` had always dropped aliases to prevent. The `-log10` case was worse than duplication: an un-logged `p_value` of 1e-8 rode beside a `"negative log10 p value: 8.0"` entry that reads as a contradiction. The policy is now uniform and lives as one `drop_aliases` field on each rule rather than in five divergent code paths. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) +- **Every coercion now drops its losing alias columns, not just the study-size and study-metadata ones.** The five coercions disagreed: `study_size` and `study_metadata` dropped their losers, while `p_value`, `effect_size` and `effect_type` left theirs on the frame for `fold_unknown_to_supporting_text` to pick up. A table with `effect size` and `odds ratio` therefore emitted `effect_size: 0.85` on the edge **and** `"odds ratio: 0.85"` inside `supporting_text`, one measurement twice, which is the duplication `coerce_study_size_columns` had always dropped aliases to prevent. The `-log10` case was worse than duplication: an un-logged `p_value` of 1e-8 rode beside a `"negative log10 p value: 8.0"` entry that reads as a contradiction. The policy is now uniform and lives as one `drop_aliases` field on each rule rather than in five divergent code paths. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) - **Migration:** if a table's headers are genuinely *different* statistics that all recognize onto one canonical slot — `beta` beside `odds ratio`, or a two-analysis header set like `p_value_analysis1` / `p_value_analysis2` — only the winner now survives; the others used to reach `supporting_text` and no longer do. Rename such columns to spellings the classifiers do not claim when every one of them must reach the edge. `docs/configuration/table.md` states the rule and this cost up front. + **Migration:** if a table's headers are genuinely *different* statistics that all recognize onto one canonical slot: `beta` beside `odds ratio`, or a two-analysis header set like `p_value_analysis1` / `p_value_analysis2`, only the winner now survives; the others used to reach `supporting_text` and no longer do. Rename such columns to spellings the classifiers do not claim when every one of them must reach the edge. `docs/configuration/table.md` states the rule and this cost up front. - **The five clean-phase coercion ops are one op, `coerce_columns`.** All five already carried the same `"clean"` phase label, so `compile_subgraph` emitted a single phase transition either way and the progress UI is unchanged. Fusing them means one schema resolution, one rename node and one `with_columns` batch per section, and lets the `effect_type` Biolink class rule read the renamed `effect_size` from the plan instead of re-resolving the frame's schema. The five per-slot functions remain exported and are thin slices of the same code path, so they stay the unit of testing and of documentation. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) -- **Column coercion is a registry plus four pure helpers rather than five hand-written pipelines.** "An existing canonical column wins, otherwise the best `rapidfuzz` match" was copy-pasted four times, `sig` re-derived the p-value selection `coerce_pvalue_columns` had already computed, and `coerced_target` was a hand-maintained `or` chain whose docstring asked the reader to keep it in sync with the op order in `Tcode._source_ops`. Selection, planning, and application now exist once (`_select` / `_plan` / `_value_exprs` / `_apply`), each coercion is a `_Rule` record naming only what distinguishes it, and both `coerced_target` and `coerce_columns` read the same `_RULES` tuple — so the classification a config-time validator reports and the rename the build performs can no longer drift apart. A drift-guard test pins the classifier disjointness the single-pass planner depends on, and an equivalence test pins the fused op against the five applied in sequence. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) +- **Column coercion is a registry plus four pure helpers rather than five hand-written pipelines.** "An existing canonical column wins, otherwise the best `rapidfuzz` match" was copy-pasted four times, `sig` re-derived the p-value selection `coerce_pvalue_columns` had already computed, and `coerced_target` was a hand-maintained `or` chain whose docstring asked the reader to keep it in sync with the op order in `Tcode._source_ops`. Selection, planning, and application now exist once (`_select` / `_plan` / `_value_exprs` / `_apply`), each coercion is a `_Rule` record naming only what distinguishes it, and both `coerced_target` and `coerce_columns` read the same `_RULES` tuple; so the classification a config-time validator reports and the rename the build performs can no longer drift apart. A drift-guard test pins the classifier disjointness the single-pass planner depends on, and an equivalence test pins the fused op against the five applied in sequence. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) ### Performance -- **`effect_type` value coercion is bounded by distinct values instead of by row count.** Each cell went through a `map_elements` Python UDF behind a process-global `lru_cache(maxsize=4096)`; past 4096 distinct values the cache thrashed and stopped helping entirely. The column is now resolved once over its distinct values and rewritten by a single native `replace_strict`. Measured on 200k rows: **56.8 ms → 9.7 ms** at 10 distinct values, **455.6 ms → 16.8 ms** at 5 000, and **1106.0 ms → 62.2 ms** at 20 000 — faster at every cardinality, with output verified identical over 4 037 probe values. The fuzzy fallback uses `rapidfuzz.process.extractOne` rather than `process.cdist`: `cdist` returns a numpy matrix, and numpy is not a Tablassert dependency, so it would raise inside the polars UDF. The global cache is gone with it — arbitrary source-cell values are not state this module should own. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) +- **`effect_type` value coercion is bounded by distinct values instead of by row count.** Each cell went through a `map_elements` Python UDF behind a process-global `lru_cache(maxsize=4096)`; past 4096 distinct values the cache thrashed and stopped helping entirely. The column is now resolved once over its distinct values and rewritten by a single native `replace_strict`. Measured on 200k rows: **56.8 ms to 9.7 ms** at 10 distinct values, **455.6 ms to 16.8 ms** at 5 000, and **1106.0 ms to 62.2 ms** at 20 000, faster at every cardinality, with output verified identical over 4 037 probe values. The fuzzy fallback uses `rapidfuzz.process.extractOne` rather than `process.cdist`: `cdist` returns a numpy matrix, and numpy is not a Tablassert dependency, so it would raise inside the polars UDF. The global cache is gone with it; arbitrary source-cell values are not state this module should own. ([#134](https://github.com/SkyeAv/Tablassert/pull/134)) ## 16.4.0 - 2026-09-03 ### Added @@ -292,7 +292,7 @@ All notable changes to this project are documented in this file. ## 16.2.0 - 2026-09-03 ### Added -- **`uuid_on_collision: merge` folds divergent same-id edges into one instead of aborting the build.** `uuid_fields` (16.0.0) narrows what feeds the derived edge id; when the declared set is the *resolved statement* — subject, predicate, object, qualifiers — two rows with different raw mention spellings that resolve to the same CURIE derive one id, and the build failed with `uuid-fields-not-a-key` even though the correct output is plainly a single edge with the combined evidence. `Graph` gained an optional `uuid_on_collision: Literal["error", "merge"]` (default `error`, behavior unchanged in every respect). Under `merge`, the Rust deduper buffers one full record per unique id, folds each divergent same-id record into the first — list fields (`publications`, `sources`, `source_record_urls`, `supporting_text`, `category`, `upstream_resource_ids`, `has_supporting_studies`, …) unioned, deduplicated by canonical content (object entries like `sources[]` that differ only in key order collapse), and **sorted**, so merged output is identical regardless of row order; scalar fields keep the first record's value, with each conflict counted into a build-log summary — and writes the merged edges in first-seen order at end-of-stream. That buffering is the memory trade-off the default streaming path avoids, which is why the mode is opt-in, and why it is rejected without `uuid_fields` (under the whole-record hash no two different records can share an id, so there would be nothing to merge) as `uuid-merge-without-fields`. ([#126](https://github.com/SkyeAv/Tablassert/pull/126)) +- **`uuid_on_collision: merge` folds divergent same-id edges into one instead of aborting the build.** `uuid_fields` (16.0.0) narrows what feeds the derived edge id; when the declared set is the *resolved statement*, subject, predicate, object, qualifiers, two rows with different raw mention spellings that resolve to the same CURIE derive one id, and the build failed with `uuid-fields-not-a-key` even though the correct output is plainly a single edge with the combined evidence. `Graph` gained an optional `uuid_on_collision: Literal["error", "merge"]` (default `error`, behavior unchanged in every respect). Under `merge`, the Rust deduper buffers one full record per unique id, folds each divergent same-id record into the first: list fields (`publications`, `sources`, `source_record_urls`, `supporting_text`, `category`, `upstream_resource_ids`, `has_supporting_studies`, ...) unioned, deduplicated by canonical content (object entries like `sources[]` that differ only in key order collapse), and **sorted**, so merged output is identical regardless of row order; scalar fields keep the first record's value, with each conflict counted into a build-log summary, and writes the merged edges in first-seen order at end-of-stream. That buffering is the memory trade-off the default streaming path avoids, which is why the mode is opt-in, and why it is rejected without `uuid_fields` (under the whole-record hash no two different records can share an id, so there would be nothing to merge) as `uuid-merge-without-fields`. ([#126](https://github.com/SkyeAv/Tablassert/pull/126)) ## 16.1.1 - 2026-09-03 @@ -308,34 +308,34 @@ All notable changes to this project are documented in this file. ## 16.0.0 - 2026-08-25 ### Breaking Changes -- **Edge ids are canonicalized before hashing, so every existing edge id changes once.** The derivation sorted only *top-level* keys: nested values reached the digest through `Value::to_string`, which under serde_json's `preserve_order` emits insertion order. Every edge carries a `sources` array, so a polars struct-field reordering — or a Biolink release that shifts a nested slot — silently re-minted ids across a whole graph while nothing about the assertion had changed. Canonicalization now recurses (object keys sorted at every depth; array order preserved, because it is semantic), which makes ids robust to that churn at the cost of one migration. Relatedly, `false` is now hashed as `"false"` instead of being dropped along with its key: `strip_nulls` deliberately keeps `false` (`negated: false` is a meaningful Biolink value), so dropping it made `{subject, negated: false}` and `{subject}` — two distinct records — derive the *same* id. No shipped graph is affected by that second fix (the 1.27M-edge MultiomicsKG 3.0.0 build contains zero `false` values); the canonicalization fix moves every id. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) +- **Edge ids are canonicalized before hashing, so every existing edge id changes once.** The derivation sorted only *top-level* keys: nested values reached the digest through `Value::to_string`, which under serde_json's `preserve_order` emits insertion order. Every edge carries a `sources` array, so a polars struct-field reordering, or a Biolink release that shifts a nested slot, silently re-minted ids across a whole graph while nothing about the assertion had changed. Canonicalization now recurses (object keys sorted at every depth; array order preserved, because it is semantic), which makes ids robust to that churn at the cost of one migration. Relatedly, `false` is now hashed as `"false"` instead of being dropped along with its key: `strip_nulls` deliberately keeps `false` (`negated: false` is a meaningful Biolink value), so dropping it made `{subject, negated: false}` and `{subject}`, two distinct records, derive the *same* id. No shipped graph is affected by that second fix (the 1.27M-edge MultiomicsKG 3.0.0 build contains zero `false` values); the canonicalization fix moves every id. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) - **Migration:** readers pinning Tablassert edge ids must re-key against the rebuilt graph. There is no mapping from old id to new — the old value was a hash of a serialization detail. This is a one-time event: adopt `uuid_fields` (below) in the same rebuild and ids stop moving for attribute changes thereafter. + **Migration:** readers pinning Tablassert edge ids must re-key against the rebuilt graph. There is no mapping from old id to new; the old value was a hash of a serialization detail. This is a one-time event: adopt `uuid_fields` (below) in the same rebuild and ids stop moving for attribute changes thereafter. ### Added -- **`uuid_fields`: a graph config declares which edge fields constitute edge identity.** Edge ids were UUIDv3 hashes of the *entire* emitted record, so every field was an identity field — a corrected `p_value`, a bumped `subject_nlp_level` inside `supporting_text`, a changed `sources[].source_record_urls`, or a reordered source row (the `row:` inside `has_supporting_studies`) each minted a brand-new id, and downstream Translator consumers saw a new edge rather than the same edge with updated attributes. `Graph` gained an optional `uuid_fields` list; when set, only those top-level edge keys feed the hash, and everything else is free to change. Entries canonicalize onto their allow-listed spelling exactly as annotations do, and a list that cannot be a key — empty, repeating, containing `id`, or naming a field no edge emits — is rejected at config time as `uuid-bad-fields`. Leaving `uuid_fields` unset preserves the whole-record hash, so the feature is strictly opt-in. Measured on MultiomicsKG 3.0.0 (1,265,355 edges): re-analysing `p_value` and `effect_size` on every row moved **930,081 ids (73%)** under the whole-record hash and **none at all** under a declared `uuid_fields`. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) +- **`uuid_fields`: a graph config declares which edge fields constitute edge identity.** Edge ids were UUIDv3 hashes of the *entire* emitted record, so every field was an identity field: a corrected `p_value`, a bumped `subject_nlp_level` inside `supporting_text`, a changed `sources[].source_record_urls`, or a reordered source row (the `row:` inside `has_supporting_studies`) each minted a brand-new id, and downstream Translator consumers saw a new edge rather than the same edge with updated attributes. `Graph` gained an optional `uuid_fields` list; when set, only those top-level edge keys feed the hash, and everything else is free to change. Entries canonicalize onto their allow-listed spelling exactly as annotations do, and a list that cannot be a key: empty, repeating, containing `id`, or naming a field no edge emits, is rejected at config time as `uuid-bad-fields`. Leaving `uuid_fields` unset preserves the whole-record hash, so the feature is strictly opt-in. Measured on MultiomicsKG 3.0.0 (1,265,355 edges): re-analysing `p_value` and `effect_size` on every row moved **930,081 ids (73%)** under the whole-record hash and **none at all** under a declared `uuid_fields`. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) -- **The UUID namespace derives from the graph's infores when `uuid_fields` is declared.** Full-record hashing gave cross-graph uniqueness by accident: two graphs asserting the same triple from the same publication were separated by their differing `sources` and `supporting_text`. A narrow field set removes that accident, so the domain moves from the `TABLASSERT` constant onto `rig.source_info.infores_id`, making the separation structural instead. A new optional `uuid_domain` overrides it for the opposite case — graphs that must deliberately *share* an id space, such as a KG compiled in shards or renamed across versions while keeping its published ids. Both default to the historic constant when `uuid_fields` is unset. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) +- **The UUID namespace derives from the graph's infores when `uuid_fields` is declared.** Full-record hashing gave cross-graph uniqueness by accident: two graphs asserting the same triple from the same publication were separated by their differing `sources` and `supporting_text`. A narrow field set removes that accident, so the domain moves from the `TABLASSERT` constant onto `rig.source_info.infores_id`, making the separation structural instead. A new optional `uuid_domain` overrides it for the opposite case: graphs that must deliberately *share* an id space, such as a KG compiled in shards or renamed across versions while keeping its published ids. Both default to the historic constant when `uuid_fields` is unset. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) - **A `duplicate-edge-ids` assertion in `--qc` study output.** `study_kgx` checked `duplicate-node-ids` but had no edge equivalent, even though KGX requires edge ids to be unique. The symmetric check now streams the final edges file and reports duplicates independently of the writer, catching an id collision from any source. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) - **RIG `source_files` come from graph config, and RIG YAML output is indented.** `target_info.edge_type_info.source_files` was previously scraped from edge `source_record_urls`; an optional `rig.source_files` graph-config key now supplies the list, applied to every edge type, while edges keep emitting `source_record_urls` unchanged. Generated `.RIG.yaml` files are written with block sequences indented under their parent key, unicode allowed, and a 120-column wrap so long prose fields stay readable. ([#121](https://github.com/SkyeAv/Tablassert/pull/121)) ### Fixed -- **Two edges deriving one id abort the build instead of shipping a duplicate.** Dedup keyed on the full canonical record bytes, which was safe only because the id was a pure function of those bytes. That equivalence breaks the moment the hash covers a subset, and it was already imperfect: `stable_json_bytes` never sorted despite its name, so two logically identical records arriving with different key order derived one id, produced different bytes, and *both* shipped. Edges now dedup on the derived id itself — an exact repeat collapses as before, while two genuinely different edges claiming one id raise `uuid-fields-not-a-key` with the id, the fields that differ (recovered by re-reading the partial output, on the failure path only), and the declared field list. The nodes path is unchanged: node ids are CURIEs, not derived hashes. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) +- **Two edges deriving one id abort the build instead of shipping a duplicate.** Dedup keyed on the full canonical record bytes, which was safe only because the id was a pure function of those bytes. That equivalence breaks the moment the hash covers a subset, and it was already imperfect: `stable_json_bytes` never sorted despite its name, so two logically identical records arriving with different key order derived one id, produced different bytes, and *both* shipped. Edges now dedup on the derived id itself: an exact repeat collapses as before, while two genuinely different edges claiming one id raise `uuid-fields-not-a-key` with the id, the fields that differ (recovered by re-reading the partial output, on the failure path only), and the declared field list. The nodes path is unchanged: node ids are CURIEs, not derived hashes. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) ### Performance -- **Edge dedup holds 24 bytes per edge instead of a full copy of every record: 2469 MB -> 100 MB peak on a 1.27M-edge graph.** `record_if_new` retained the complete bytes of every unique record to make suppression byte-exact — roughly 800 bytes per edge, and ~2.4 GB resident on MultiomicsKG 3.0.0 purely for the dedup set. The edge path now keys a `FxHashMap<[u8; 16], u64>` on the raw UUID bytes with an xxh64 of the id-free record as the value: a measured **25x** reduction that also removes the per-record heap allocation, and on the same graph took the dedup pass from 477s to 37s (the old path's allocation churn dominated; the margin will be smaller on machines under less memory pressure). The content hash is computed *before* the id is inserted, so no record is cloned to strip it back out, and a declared `uuid_fields` list additionally cuts the number of keys the hasher visits per edge. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) +- **Edge dedup holds 24 bytes per edge instead of a full copy of every record: 2469 MB to 100 MB peak on a 1.27M-edge graph.** `record_if_new` retained the complete bytes of every unique record to make suppression byte-exact: roughly 800 bytes per edge, and ~2.4 GB resident on MultiomicsKG 3.0.0 purely for the dedup set. The edge path now keys a `FxHashMap<[u8; 16], u64>` on the raw UUID bytes with an xxh64 of the id-free record as the value: a measured **25x** reduction that also removes the per-record heap allocation, and on the same graph took the dedup pass from 477s to 37s (the old path's allocation churn dominated; the margin will be smaller on machines under less memory pressure). The content hash is computed *before* the id is inserted, so no record is cloned to strip it back out, and a declared `uuid_fields` list additionally cuts the number of keys the hasher visits per edge. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) ### Changed -- **`docs/api/utils.md` now describes the derivation that actually runs.** The page documented a `"\t".join(values)` encoding removed some releases ago (the join has been length-prefixed for collision-safety) and claimed edge ids covered only "subject, predicate, object, qualifiers, and publication" — which was never true, and is precisely what `uuid_fields` now makes achievable. Both are corrected, and the canonicalization, namespacing, and uniqueness rules are documented alongside. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) +- **`docs/api/utils.md` now describes the derivation that actually runs.** The page documented a `"\t".join(values)` encoding removed some releases ago (the join has been length-prefixed for collision-safety) and claimed edge ids covered only "subject, predicate, object, qualifiers, and publication", which was never true, and is precisely what `uuid_fields` now makes achievable. Both are corrected, and the canonicalization, namespacing, and uniqueness rules are documented alongside. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) - **Golden UUID vectors pin the derivation.** No test asserted a concrete UUID value, so any change to the encoding could land silently. Three vectors now fix the full-record, declared-field, and raw-parts derivations; if one moves, edge ids in every published graph moved with it. ([#122](https://github.com/SkyeAv/Tablassert/pull/122)) ## 15.1.0 - 2026-08-25 ### Added -- **`disease_context_qualifier` is granted to `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` edges as a class-scoped policy override.** Biolink declares the slot only on the `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation` lineage, while `FDA_regulatory_approvals` lives only on the `EntityToDisease` / `EntityToPhenotypicFeature` classes DAKP pins via `category_override` — so a contraindication edge could natively carry one slot or the other, never both. The new `CLASS_FIELD_OVERRIDES` table in `biolink.py` grants the qualifier to the pinned classes: `prune_to_class` keeps the value on those rows instead of nulling and rescuing it, and record validation strips the granted field from its in-memory copy so the deliberate gap is not reported as `extra_forbidden`. The grant is deliberately ahead of the pinned model (pending an upstream Biolink widening); a tripwire test asserts every granted field stays absent from its class, so a future biolink-model release that attaches the slot fails the suite until the stale grant is removed ([#120](https://github.com/SkyeAv/Tablassert/pull/120)). +- **`disease_context_qualifier` is granted to `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` edges as a class-scoped policy override.** Biolink declares the slot only on the `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation` lineage, while `FDA_regulatory_approvals` lives only on the `EntityToDisease` / `EntityToPhenotypicFeature` classes DAKP pins via `category_override`; so a contraindication edge could natively carry one slot or the other, never both. The new `CLASS_FIELD_OVERRIDES` table in `biolink.py` grants the qualifier to the pinned classes: `prune_to_class` keeps the value on those rows instead of nulling and rescuing it, and record validation strips the granted field from its in-memory copy so the deliberate gap is not reported as `extra_forbidden`. The grant is deliberately ahead of the pinned model (pending an upstream Biolink widening); a tripwire test asserts every granted field stays absent from its class, so a future biolink-model release that attaches the slot fails the suite until the stale grant is removed ([#120](https://github.com/SkyeAv/Tablassert/pull/120)). ### Fixed - **`number_of_cases` annotations are no longer coerced into `study_size`.** `number_of_cases` is a legitimate Biolink `Association` slot (cases carrying the phenotype/disease), but `STUDY_SIZE_PREFIX_PATTERN` matched the column via "number of cases" and `coerce_study_size_columns` renamed it to `study_size`, destroying the edge field. A new `STUDY_SIZE_EXEMPT_PATTERN` makes `study_size_target` return `None` for the exact slot, so the column reaches the edge as `number_of_cases` ([#119](https://github.com/SkyeAv/Tablassert/pull/119)). @@ -345,10 +345,10 @@ All notable changes to this project are documented in this file. ### Breaking Changes - **The `approval_ids` biolink override is removed; the column is no longer a curated edge field.** `TABLASERT_EDGE_EXTRAS` no longer carries `approval_ids`, so the FDA-application-number pass-through no longer bypasses Biolink validation: `ALLOWED_EDGE_FIELDS` and `KNOWN_PENDING_EDGE_FIELDS` derive without it, an `approval_ids` annotation now emits a relocation warning and folds into `supporting_text` like any unknown edge field, per-record class pruning drops it, and strict KGX validation counts edges carrying it as real `extra_forbidden` failures (the pending exemption no longer forgives them). The DAKP real-world fixtures drop their `approval_ids` annotations accordingly ([#117](https://github.com/SkyeAv/Tablassert/pull/117)). - **Migration:** annotate `FDA_regulatory_approvals` instead — it is a real Biolink slot declared on `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation`, so no curated override is needed. It is multivalued, so declare `split_by` for joined cells (e.g. `011111|022222`), and pin the subclass with the new `category_override` (below) when the section's derived pair class is `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation`, which does not declare the slot and would prune it. Any casing of the name works; the annotation canonicalizes onto the exact `FDA_regulatory_approvals` spelling. Readers consuming a curated `approval_ids` edge field should read `FDA_regulatory_approvals`; strict `validate-kgx` consumers lose the pending exemption for the old name. + **Migration:** annotate `FDA_regulatory_approvals` instead; it is a real Biolink slot declared on `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation`, so no curated override is needed. It is multivalued, so declare `split_by` for joined cells (e.g. `011111|022222`), and pin the subclass with the new `category_override` (below) when the section's derived pair class is `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation`, which does not declare the slot and would prune it. Any casing of the name works; the annotation canonicalizes onto the exact `FDA_regulatory_approvals` spelling. Readers consuming a curated `approval_ids` edge field should read `FDA_regulatory_approvals`; strict `validate-kgx` consumers lose the pending exemption for the old name. ### Added -- **A per-section association-category override keyed on the resolved object category.** `Statement` gained `category_override`: a `{object category: association category}` map (validated against the `Categories` and `EdgeCategories` enums) that the edge-category derivation applies per row in place of the `(subject, object)` pair lookup, keyed on the raw object category *before* the role rollup that merges `Disease` and `PhenotypicFeature` into one pair key. Rows whose object category is absent from the map derive as before, and pinned classes still pass through `resolve_association_class` reconciliation against the section predicate — a pinned class whose `predicate` slot rejects it is walked up the association hierarchy, with a `BiolinkRelocationWarning` at config time. This is the DAKP split: drug→disease and drug→phenotype rows can now pin `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` respectively, so the subclass-only slots those classes declare (`FDA_regulatory_approvals`, `number_of_cases`) survive `prune_to_class` on the edge instead of being nulled and rescued into the pruned column ([#118](https://github.com/SkyeAv/Tablassert/pull/118)). +- **A per-section association-category override keyed on the resolved object category.** `Statement` gained `category_override`: a `{object category: association category}` map (validated against the `Categories` and `EdgeCategories` enums) that the edge-category derivation applies per row in place of the `(subject, object)` pair lookup, keyed on the raw object category *before* the role rollup that merges `Disease` and `PhenotypicFeature` into one pair key. Rows whose object category is absent from the map derive as before, and pinned classes still pass through `resolve_association_class` reconciliation against the section predicate: a pinned class whose `predicate` slot rejects it is walked up the association hierarchy, with a `BiolinkRelocationWarning` at config time. This is the DAKP split: drug-to-disease and drug-to-phenotype rows can now pin `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` respectively, so the subclass-only slots those classes declare (`FDA_regulatory_approvals`, `number_of_cases`) survive `prune_to_class` on the edge instead of being nulled and rescued into the pruned column ([#118](https://github.com/SkyeAv/Tablassert/pull/118)). - **Annotation casing now canonicalizes onto allow-listed spellings, preserving mixed-case slots like `FDA_regulatory_approvals`.** The `Annotation` validator no longer blind-lowercases: a declared name whose lowercase form matches an edge allow-list entry canonicalizes onto that entry's exact spelling. Model-slot annotations are unchanged (`P_Value` still becomes `p_value`), but Biolink's mixed-case `FDA_regulatory_approvals` slot (on `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation`) can now be authored in any casing and reaches the final edge verbatim as `FDA_regulatory_approvals` instead of being lowercased into an unknown name and folded into `supporting_text` ([#117](https://github.com/SkyeAv/Tablassert/pull/117)). ## 14.0.0 - 2026-08-24 @@ -362,76 +362,76 @@ All notable changes to this project are documented in this file. - **An explicit retrieval-`sources` template, with `{edge_id}` record-URL interpolation.** `ManualProvenance` gained `sources`: an ordered list of Biolink `RetrievalSource` entry templates that replaces the derived "graph infores as primary, one `supporting_data_source` per upstream" emission entirely. Each entry declares `resource_id`, `resource_role` (validated against the three `ResourceRoleEnum` members), and optional `upstream_resource_ids` / `source_record_urls`; the entries are emitted verbatim, in the order given. The template is mutually exclusive with `upstream_resource_ids` and `upstream_source_record_urls` (it subsumes both), must be non-empty with unique `resource_id` values, and must carry at least one `primary_knowledge_source` or `aggregator_knowledge_source` entry; every violation reports as `override-bad-sources`. A `source_record_urls` value may embed the literal `{edge_id}` placeholder to build per-edge drill-down URLs: the edge `id` is a content hash assigned at the final dedup stage, *after* subgraphs are written, so the literal placeholder is what the subgraph carries and what the hash covers (edge ids stay deterministic), and a post-dedup sweep of the final `*.edges.ndjson` substitutes each record's own `id`. Files carrying no marker are left byte-identical. This lets an ingest such as DAKP reproduce legacy KG retrieval provenance declaratively from the table config (an aggregator primary entry with a per-edge viewer URL, the real primary knowledge source as a sibling entry) instead of post-processing the built edges ([#116](https://github.com/SkyeAv/Tablassert/pull/116)). ### Fixed -- **Negative-log p-value columns are un-logged instead of shipped verbatim.** `pvalue_target` already routed spellings like `negative log p value` / `-log10(p)` onto `p_value` / `adjusted_p_value`, but the score rode through untouched: a −log10(p)=8 enrichment column (the `mokg-v12` HOYER1 fixture ships exactly this shape) emitted `p_value` 8.0 — wildly out of range — and `sig` banded it `not_significant`, the exact inverse of the truth. The coercion now recognizes an explicit negation marker (`negative` / `negated` / `neg` / `-`) before a `log`/`log10` p-or-q token (`is_neglog10_column`) — requiring a complete p/q-value token (or a bare delimited `P`), so a word merely starting with p or q (`negative log protein`) no longer matches — and converts the score (`p = 10**-x`) when it lands on the numeric slot, in both `coerce_pvalue_columns` and `sig`. A raw p/q candidate now always beats a −log10 alias of the same statistic before fuzzy ranking runs, so a short raw name (`P`, `FDR`) can no longer lose to a long alias it would then be un-logged over. Float64 underflow floors extreme scores at `0.0` (indistinguishable from `p ~ 0`, and the band is identical); nulls stay null. A plain `log10` spelling without a negation marker is deliberately left verbatim — the sign convention is ambiguous there — and when a raw p-value column coexists with a −log10 alias the raw column still wins and the alias is left untouched ([#114](https://github.com/SkyeAv/Tablassert/pull/114)). +- **Negative-log p-value columns are un-logged instead of shipped verbatim.** `pvalue_target` already routed spellings like `negative log p value` / `-log10(p)` onto `p_value` / `adjusted_p_value`, but the score rode through untouched: a -log10(p)=8 enrichment column (the `mokg-v12` HOYER1 fixture ships exactly this shape) emitted `p_value` 8.0, wildly out of range, and `sig` banded it `not_significant`, the exact inverse of the truth. The coercion now recognizes an explicit negation marker (`negative` / `negated` / `neg` / `-`) before a `log`/`log10` p-or-q token (`is_neglog10_column`), requiring a complete p/q-value token (or a bare delimited `P`), so a word merely starting with p or q (`negative log protein`) no longer matches, and converts the score (`p = 10**-x`) when it lands on the numeric slot, in both `coerce_pvalue_columns` and `sig`. A raw p/q candidate now always beats a -log10 alias of the same statistic before fuzzy ranking runs, so a short raw name (`P`, `FDR`) can no longer lose to a long alias it would then be un-logged over. Float64 underflow floors extreme scores at `0.0` (indistinguishable from `p ~ 0`, and the band is identical); nulls stay null. A plain `log10` spelling without a negation marker is deliberately left verbatim; the sign convention is ambiguous there, and when a raw p-value column coexists with a -log10 alias the raw column still wins and the alias is left untouched ([#114](https://github.com/SkyeAv/Tablassert/pull/114)). ## 13.0.0 - 2026-08-24 ### Breaking Changes -- **The inlined supporting study now carries the current Biolink Study metadata, with disjoint ids and names and no `#` composition.** Supporting studies were verbose and self-duplicating: a published Excel section keyed its `Study` by `"PMC11708054#all correlations"`, repeated the sheet as the `name`, and fabricated a per-edge `StudyResult` (`"…#row42"` / `"all correlations row 42"`) even when it carried no evidence; an unpublished section keyed the study by its own config filename (`my_table.yaml`). The contract is now: `Study.id` is a real identifier — the section's publication CURIE, or the config stem (YAML filename minus its extension) as the rescue fallback — and `Study.name` a *disjoint* human label (worksheet for spreadsheet sources, source filename otherwise; omitted when it would equal the id). Study-level metadata declared as annotations (`study_size`, `study_cohort`, `study_context`, `study_date_range`, `study_method_description`, `study_method_types` — the `Study` node properties that [biolink-model#1770](https://github.com/biolink/biolink-model/pull/1770) replaced the deprecated `supporting_study_*` association slots with) lands as real typed fields **on the Study**, never on the edge and never inside a description string. The `StudyResult` is identified only by the scoped CURIE `row:` — its source row — and carries no name; its `description` is reserved for values with no structured home (routed unsatisfiable columns and class-pruned qualifiers). No `#` delimiter appears anywhere in the output. A published section always keeps the `{id, name}` wrapper; an unpublished one emits the struct only when it has metadata, routed unsatisfiable values, or pruned values to carry (the contentless-skip behavior is unchanged). +- **The inlined supporting study now carries the current Biolink Study metadata, with disjoint ids and names and no `#` composition.** Supporting studies were verbose and self-duplicating: a published Excel section keyed its `Study` by `"PMC11708054#all correlations"`, repeated the sheet as the `name`, and fabricated a per-edge `StudyResult` (`"...#row42"` / `"all correlations row 42"`) even when it carried no evidence; an unpublished section keyed the study by its own config filename (`my_table.yaml`). The contract is now: `Study.id` is a real identifier: the section's publication CURIE, or the config stem (YAML filename minus its extension) as the rescue fallback, and `Study.name` a *disjoint* human label (worksheet for spreadsheet sources, source filename otherwise; omitted when it would equal the id). Study-level metadata declared as annotations (`study_size`, `study_cohort`, `study_context`, `study_date_range`, `study_method_description`, `study_method_types`, the `Study` node properties that [biolink-model#1770](https://github.com/biolink/biolink-model/pull/1770) replaced the deprecated `supporting_study_*` association slots with) lands as real typed fields **on the Study**, never on the edge and never inside a description string. The `StudyResult` is identified only by the scoped CURIE `row:`, its source row, and carries no name; its `description` is reserved for values with no structured home (routed unsatisfiable columns and class-pruned qualifiers). No `#` delimiter appears anywhere in the output. A published section always keeps the `{id, name}` wrapper; an unpublished one emits the struct only when it has metadata, routed unsatisfiable values, or pruned values to carry (the contentless-skip behavior is unchanged). - **Migration:** readers keying `has_supporting_studies` by the composed `publication#sheet` string should key by the publication CURIE; consumers scraping `"row N"` result names or `supporting_study_size=…` description entries should read `StudyResult.id` (`row:`) and the typed `Study.study_*` fields instead. Declaring the deprecated `supporting_study_size` / `sample_size` / `supporting_study_*` annotation names still builds — the coercion phase renames them onto the canonical `study_*` properties — but emits a `BiolinkRelocationWarning` naming the destination. If multiple study-size synonyms coexist, the canonical/winning value is retained and the losing synonyms are dropped as duplicate representations of the same Study property. + **Migration:** readers keying `has_supporting_studies` by the composed `publication#sheet` string should key by the publication CURIE; consumers scraping `"row N"` result names or `supporting_study_size=...` description entries should read `StudyResult.id` (`row:`) and the typed `Study.study_*` fields instead. Declaring the deprecated `supporting_study_size` / `sample_size` / `supporting_study_*` annotation names still builds; the coercion phase renames them onto the canonical `study_*` properties, but emits a `BiolinkRelocationWarning` naming the destination. If multiple study-size synonyms coexist, the canonical/winning value is retained and the losing synonyms are dropped as duplicate representations of the same Study property. -- **biolink-model 4.4.4: the statistical edge slots the model now declares are emitted as real model-typed values.** The dependency moved from 4.4.3 to 4.4.4 and every model-derived set follows it with no code change: `effect_size` (`float`) and `effect_type` (`EffectTypeEnum`) are real `Association` slots ([#1774](https://github.com/biolink/biolink-model/pull/1774)) and leave the pending-exemption set, so `effect_size` ships as a real JSON number instead of a `{:.4g}` string; `statistical_significance_qualifier` is attached to every association class and rides the edge as a bare `StatisticalSignificanceQualifierEnum` token (`"strongly_significant"`, no `biolink:` prefix) instead of being relocated into the study description; `EffectTypes` is now derived from the installed model's enum rather than a local tuple. `relationship_strength` is deliberately no longer listed among unsatisfiable study-routed names — it is a legacy alias the coercion renames to the edge slot `effect_size`. +- **biolink-model 4.4.4: the statistical edge slots the model now declares are emitted as real model-typed values.** The dependency moved from 4.4.3 to 4.4.4 and every model-derived set follows it with no code change: `effect_size` (`float`) and `effect_type` (`EffectTypeEnum`) are real `Association` slots ([#1774](https://github.com/biolink/biolink-model/pull/1774)) and leave the pending-exemption set, so `effect_size` ships as a real JSON number instead of a `{:.4g}` string; `statistical_significance_qualifier` is attached to every association class and rides the edge as a bare `StatisticalSignificanceQualifierEnum` token (`"strongly_significant"`, no `biolink:` prefix) instead of being relocated into the study description; `EffectTypes` is now derived from the installed model's enum rather than a local tuple. `relationship_strength` is deliberately no longer listed among unsatisfiable study-routed names; it is a legacy alias the coercion renames to the edge slot `effect_size`. - **Migration:** consumers parsing `effect_size` as a `{:.4g}` string must read a real JSON number now (`-0.85`, not `"-0.85"`), and `statistical_significance_qualifier` moved out of the study description onto the edge as a bare enum token. Configs naming a class 4.4.4 renamed no longer validate: `AffinityMeasurement` — which appears in generated `avoid:` / `prioritize:` lists, since those are emitted as the complement of the prioritized set — is now `ProteinLigandAssayResult`; rename it in place, or regenerate the config against the current model. P-value columns are deliberately unaffected: they stay controlled scientific-notation strings regardless of the model's `float` typing (12.1.0). + **Migration:** consumers parsing `effect_size` as a `{:.4g}` string must read a real JSON number now (`-0.85`, not `"-0.85"`), and `statistical_significance_qualifier` moved out of the study description onto the edge as a bare enum token. Configs naming a class 4.4.4 renamed no longer validate: `AffinityMeasurement`, which appears in generated `avoid:` / `prioritize:` lists, since those are emitted as the complement of the prioritized set, is now `ProteinLigandAssayResult`; rename it in place, or regenerate the config against the current model. P-value columns are deliberately unaffected: they stay controlled scientific-notation strings regardless of the model's `float` typing (12.1.0). -- **An unpaired `effect_size` / `effect_type` annotation is dropped with a warning instead of failing the section.** 11.0.0 made the pair mandatory ([#92](https://github.com/SkyeAv/Tablassert/pull/92)): a section declaring one half without the other raised `annotation-effect-size-without-type` / `annotation-effect-type-without-size` and took the rest of the table down with it. `Section.drop_unpaired_effect_annotations` replaces that hard fail — the unpaired half is dropped from the section with an `UnpairedEffectAnnotationWarning` naming what was dropped and from where, and the section and its edges are kept. The value was going to be discarded by the build anyway (`coerce_effect_type_columns` nulls an effect type that has no effect size), so failing the section traded a real table for a column the pipeline would never have emitted. Names are judged by their coerced target rather than their raw spelling, so the aliases the clean phase renames (`odds ratio`, the legacy `relationship_strength`) are seen exactly as the build sees them; the validator runs after `ingests.to_sections` expands `template`/`sections`, so an unpaired half declared once on a template is dropped from every expanded section. Declaring both together is still the recommendation — neither half carries evidence alone ([#105](https://github.com/SkyeAv/Tablassert/pull/105)). +- **An unpaired `effect_size` / `effect_type` annotation is dropped with a warning instead of failing the section.** 11.0.0 made the pair mandatory ([#92](https://github.com/SkyeAv/Tablassert/pull/92)): a section declaring one half without the other raised `annotation-effect-size-without-type` / `annotation-effect-type-without-size` and took the rest of the table down with it. `Section.drop_unpaired_effect_annotations` replaces that hard fail: the unpaired half is dropped from the section with an `UnpairedEffectAnnotationWarning` naming what was dropped and from where, and the section and its edges are kept. The value was going to be discarded by the build anyway (`coerce_effect_type_columns` nulls an effect type that has no effect size), so failing the section traded a real table for a column the pipeline would never have emitted. Names are judged by their coerced target rather than their raw spelling, so the aliases the clean phase renames (`odds ratio`, the legacy `relationship_strength`) are seen exactly as the build sees them; the validator runs after `ingests.to_sections` expands `template`/`sections`, so an unpaired half declared once on a template is dropped from every expanded section. Declaring both together is still the recommendation; neither half carries evidence alone ([#105](https://github.com/SkyeAv/Tablassert/pull/105)). **Migration:** the two error codes are retired; callers asserting on `annotation-effect-size-without-type` / `annotation-effect-type-without-size` should assert on `UnpairedEffectAnnotationWarning` instead. Configs that failed validation under 11.x and 12.x now build, minus the unpaired column. ### Added -- **The final graph QC now asserts no field in the emitted NDJSON is null or empty, and that every node carries a name.** `build-kg --qc`'s stage-7 study pass (in the spirit of `studyKGtsvs.pl`) gained two assertions. `empty-or-null-values`: any field in the nodes or edges file whose value is JSON `null`, a string that strips to empty, or an empty container — checked recursively, so a null or blank nested inside an `attributes` list counts — fails the build. The check is deliberately stricter than the NDJSON writer's `strip_nulls` (`rust/src/json.rs`), which scrubs dict entries at every depth but passes array scalars (`["x", ""]`) and emptied nested objects (`[{}]`) through verbatim; the study stage now asserts the stronger contract — no null or empty value anywhere — so the first such shape to reach an emitted file fails the build loudly instead of shipping silently. Null-like *strings* (`NA`/`NaN`/`null`/`none`) are also dropped by the writer but are neither null nor empty, and are deliberately not flagged; the `original_*` whitespace exemption does not extend to emptiness. `unnamed-nodes`: a node record whose `name` key is missing, `null`, or strips to empty fails the build — the missing-key case is what pipeline output surfaces, since `strip_nulls` deletes empty and null-like names before the file is written. Both report offenders per field or per node id, capped at 10 examples like the other assertions. -- **The final graph QC now asserts every node has an `id` and every edge has `subject`, `predicate`, and `object`.** Two more stage-7 study assertions join the `unnamed-nodes` check (which already requires a non-empty node `name`). `unidentified-nodes`: a node record whose `id` key is missing, `null`, or strips to empty fails the build; since the id is exactly what is absent, examples key on the node's `name` (or `` when it has none). `incomplete-edges`: an edge record missing any of the three core slots — missing key, `null`, or strips-to-empty — fails, counted per slot (e.g. `predicate (2)`). The writer's `strip_nulls` deletes a null slot outright rather than emitting it, so on pipeline output a hit means the slot was null upstream and the record shipped broken — exactly the condition these assertions exist to catch loudly. Non-string, non-null ids and slots pass, mirroring the name convention (no writer emits them). +- **The final graph QC now asserts no field in the emitted NDJSON is null or empty, and that every node carries a name.** `build-kg --qc`'s stage-7 study pass (in the spirit of `studyKGtsvs.pl`) gained two assertions. `empty-or-null-values`: any field in the nodes or edges file whose value is JSON `null`, a string that strips to empty, or an empty container, checked recursively, so a null or blank nested inside an `attributes` list counts, fails the build. The check is deliberately stricter than the NDJSON writer's `strip_nulls` (`rust/src/json.rs`), which scrubs dict entries at every depth but passes array scalars (`["x", ""]`) and emptied nested objects (`[{}]`) through verbatim; the study stage now asserts the stronger contract: no null or empty value anywhere, so the first such shape to reach an emitted file fails the build loudly instead of shipping silently. Null-like *strings* (`NA`/`NaN`/`null`/`none`) are also dropped by the writer but are neither null nor empty, and are deliberately not flagged; the `original_*` whitespace exemption does not extend to emptiness. `unnamed-nodes`: a node record whose `name` key is missing, `null`, or strips to empty fails the build; the missing-key case is what pipeline output surfaces, since `strip_nulls` deletes empty and null-like names before the file is written. Both report offenders per field or per node id, capped at 10 examples like the other assertions. +- **The final graph QC now asserts every node has an `id` and every edge has `subject`, `predicate`, and `object`.** Two more stage-7 study assertions join the `unnamed-nodes` check (which already requires a non-empty node `name`). `unidentified-nodes`: a node record whose `id` key is missing, `null`, or strips to empty fails the build; since the id is exactly what is absent, examples key on the node's `name` (or `` when it has none). `incomplete-edges`: an edge record missing any of the three core slots: missing key, `null`, or strips-to-empty, fails, counted per slot (e.g. `predicate (2)`). The writer's `strip_nulls` deletes a null slot outright rather than emitting it, so on pipeline output a hit means the slot was null upstream and the record shipped broken, exactly the condition these assertions exist to catch loudly. Non-string, non-null ids and slots pass, mirroring the name convention (no writer emits them). ### Changed - **`openpyxl` is an explicit test dependency.** It was only ever transitive via `linkml` 1.10; biolink-model 4.4.4's `linkml` 1.11 bump dropped it, and `tests/test_cover_lib.py` imports it directly. ### Performance -- **Entity resolution now materializes each section once instead of once per node column.** `resolve_batch` used to re-execute the entire upstream lazy plan — the source scan, every encoding/regex op, and both NLP normalization levels — once per resolved column: `distinct(...).collect()` per column, `join_matches`' collect per column, and `log_unmatched`'s anti-join collect when `--log` is on, so a section resolving subject + object + two qualifiers ran that pipeline roughly five times (up to nine with logging). The batch now collects the frame a single time and runs term extraction, unmatched logging, and the level-one/level-two join-backs against the in-memory frame; `join_matches` keeps its LazyFrame-in/LazyFrame-out contract over a new eager core, and output rows are unchanged. On a 50k-row synthetic section with four resolved columns the resolve phase runs about 1.75× faster, with the gain growing as the upstream plan gets heavier. Separately, `trim` — the op that drops the spent raw `column_` columns — moved from the finalize ops to just before `resolve_batch`, so the materialized frame and every downstream join no longer carry columns nothing reads; its progress phase label falls back to `transform` instead of flashing `finalize` early. -- **The agent pre-renders every deterministic inspection payload into the task, stops re-planning, caps in-agent improve rounds, and memoizes identical builds.** `pmc_article_context` and `read_table` are pure functions of files the supervisor has already downloaded, yet each call cost a whole LLM step: fleet logs showed roughly 2,100 article-context and 2,500 `read_table` emissions, with 68% of articles exhausting the 20-step budget largely on that inspection overhead. `render_task_context` now ships the article summary and a head preview of every candidate table — and every Excel worksheet, capped at 10 with the remainder noted — inside the task text (truncated at 60k characters), so the agent can author a config with zero inspection tool calls; both tools stay registered as fallbacks for rows beyond a preview, and a table that cannot be previewed renders a visible in-band note rather than aborting the run. `build_agent`'s `planning_interval` default changed from `3` to `None` — each smolagents planning turn is an extra round trip carrying the full prompt, and the task already prescribes a fixed short workflow (derive → build → optional edit → answer). The instructions cap in-agent improve rounds at two, after which the supervisor's deterministic improve loop continues, and `build_and_audit` memoizes per tool instance so a model re-running an unchanged config no longer pays a full validate + build + coverage pass ([#113](https://github.com/SkyeAv/Tablassert/pull/113)). +- **Entity resolution now materializes each section once instead of once per node column.** `resolve_batch` used to re-execute the entire upstream lazy plan, the source scan, every encoding/regex op, and both NLP normalization levels, once per resolved column: `distinct(...).collect()` per column, `join_matches`' collect per column, and `log_unmatched`'s anti-join collect when `--log` is on, so a section resolving subject + object + two qualifiers ran that pipeline roughly five times (up to nine with logging). The batch now collects the frame a single time and runs term extraction, unmatched logging, and the level-one/level-two join-backs against the in-memory frame; `join_matches` keeps its LazyFrame-in/LazyFrame-out contract over a new eager core, and output rows are unchanged. On a 50k-row synthetic section with four resolved columns the resolve phase runs about 1.75x faster, with the gain growing as the upstream plan gets heavier. Separately, `trim`, the op that drops the spent raw `column_` columns, moved from the finalize ops to just before `resolve_batch`, so the materialized frame and every downstream join no longer carry columns nothing reads; its progress phase label falls back to `transform` instead of flashing `finalize` early. +- **The agent pre-renders every deterministic inspection payload into the task, stops re-planning, caps in-agent improve rounds, and memoizes identical builds.** `pmc_article_context` and `read_table` are pure functions of files the supervisor has already downloaded, yet each call cost a whole LLM step: fleet logs showed roughly 2,100 article-context and 2,500 `read_table` emissions, with 68% of articles exhausting the 20-step budget largely on that inspection overhead. `render_task_context` now ships the article summary and a head preview of every candidate table, and every Excel worksheet, capped at 10 with the remainder noted, inside the task text (truncated at 60k characters), so the agent can author a config with zero inspection tool calls; both tools stay registered as fallbacks for rows beyond a preview, and a table that cannot be previewed renders a visible in-band note rather than aborting the run. `build_agent`'s `planning_interval` default changed from `3` to `None`; each smolagents planning turn is an extra round trip carrying the full prompt, and the task already prescribes a fixed short workflow (derive, build, optional edit, answer). The instructions cap in-agent improve rounds at two, after which the supervisor's deterministic improve loop continues, and `build_and_audit` memoizes per tool instance so a model re-running an unchanged config no longer pays a full validate + build + coverage pass ([#113](https://github.com/SkyeAv/Tablassert/pull/113)). ## 12.1.0 - 2026-08-18 ### Added -- **Per-upstream `source_record_urls` via a provenance override.** `ManualProvenance` gained `upstream_source_record_urls`: a mapping of upstream infores CURIE to source record URLs. When set, the section's `source.url` values serve the RIG only and are no longer emitted on the primary `sources` entry; each listed upstream `supporting_data_source` entry carries its own `source_record_urls` instead. Keys must be infores CURIEs declared in `upstream_resource_ids` (`override-bad-upstream-urls`). This lets transforming resources (e.g. DAKP's `infores:multiomics-drugapprovals`) keep their primary entry bare while the dataset download URLs live on the `infores:dailymed` / `infores:faers` entries they were actually retrieved from. Legacy placement — URLs on the primary entry — is unchanged when the mapping is absent (#104). +- **Per-upstream `source_record_urls` via a provenance override.** `ManualProvenance` gained `upstream_source_record_urls`: a mapping of upstream infores CURIE to source record URLs. When set, the section's `source.url` values serve the RIG only and are no longer emitted on the primary `sources` entry; each listed upstream `supporting_data_source` entry carries its own `source_record_urls` instead. Keys must be infores CURIEs declared in `upstream_resource_ids` (`override-bad-upstream-urls`). This lets transforming resources (e.g. DAKP's `infores:multiomics-drugapprovals`) keep their primary entry bare while the dataset download URLs live on the `infores:dailymed` / `infores:faers` entries they were actually retrieved from. Legacy placement, URLs on the primary entry, is unchanged when the mapping is absent (#104). ### Fixed -- **P-value columns are emitted in scientific notation again.** The KGX-compliance rework (#71) made `format_numeric()` emit `p_value` / `adjusted_p_value` as real JSON numbers because the `numeric_slot_kind` lookup reports the `float`-typed Biolink slots — which silently retired the `{:.4e}` scientific-notation branch for exactly the columns it existed for, so edges shipped shortest-repr numbers (`"p_value":0.0001`, `"p_value":0.05`) instead of the controlled notation the tutorial documents (`"p_value":"1.0000e-03"`). P-value-like columns are now always formatted as scientific-notation strings regardless of the slot's model type: Biolink validation here and downstream runs in Pydantic's lax mode, which coerces the numeric string back, so `validate-kgx` stays green (verified against the pinned `biolink-model` classes). Non-p-value numeric columns keep the model-typed behavior: a real JSON number once a future biolink model types the slot (`biolink/biolink-model#1770` / `#1774`), controlled `{:.4g}` strings until then. +- **P-value columns are emitted in scientific notation again.** The KGX-compliance rework (#71) made `format_numeric()` emit `p_value` / `adjusted_p_value` as real JSON numbers because the `numeric_slot_kind` lookup reports the `float`-typed Biolink slots, which silently retired the `{:.4e}` scientific-notation branch for exactly the columns it existed for, so edges shipped shortest-repr numbers (`"p_value":0.0001`, `"p_value":0.05`) instead of the controlled notation the tutorial documents (`"p_value":"1.0000e-03"`). P-value-like columns are now always formatted as scientific-notation strings regardless of the slot's model type: Biolink validation here and downstream runs in Pydantic's lax mode, which coerces the numeric string back, so `validate-kgx` stays green (verified against the pinned `biolink-model` classes). Non-p-value numeric columns keep the model-typed behavior: a real JSON number once a future biolink model types the slot (`biolink/biolink-model#1770` / `#1774`), controlled `{:.4g}` strings until then. ## 12.0.0 - 2026-08-14 ### Breaking Changes -- **`tablassert agent` now builds into a caller-owned graph config; the internal graph registry is gone.** Every successful agent build used to self-register into an agent-maintained aggregate at `/graph.yaml` — a dedicated registry module fabricated its metadata (hardcoded name `tablassert-agent`, version `1`, a synthetic PMC-flavored `rig:` block, first-wins `fullmap`), quarantined a corrupt registry to `graph.yaml.corrupt-`, and `tablassert rebuild-agent-graph` reconstructed it from `state.json`. All of that is gone. The agent now takes a required `--configuration-file`/`-f` naming a graph YAML *you* own: the graph's real `name`, `version`, `fullmap`, and `rig:` drive every per-article audit build — nothing is synthesized — and each successful result mutates that file in place, replacing any `tables` entry for the same PMC id and appending the new config's absolute path under an exclusive sidecar flock with an atomic rename. The target is validated against the `Graph` model up front, before any model is built or article fetched; a malformed target raises `graph-validation-failed` instead of being quarantined and silently rebuilt. +- **`tablassert agent` now builds into a caller-owned graph config; the internal graph registry is gone.** Every successful agent build used to self-register into an agent-maintained aggregate at `/graph.yaml`: a dedicated registry module fabricated its metadata (hardcoded name `tablassert-agent`, version `1`, a synthetic PMC-flavored `rig:` block, first-wins `fullmap`), quarantined a corrupt registry to `graph.yaml.corrupt-`, and `tablassert rebuild-agent-graph` reconstructed it from `state.json`. All of that is gone. The agent now takes a required `--configuration-file`/`-f` naming a graph YAML *you* own: the graph's real `name`, `version`, `fullmap`, and `rig:` drive every per-article audit build, nothing is synthesized, and each successful result mutates that file in place, replacing any `tables` entry for the same PMC id and appending the new config's absolute path under an exclusive sidecar flock with an atomic rename. The target is validated against the `Graph` model up front, before any model is built or article fetched; a malformed target raises `graph-validation-failed` instead of being quarantined and silently rebuilt. - **Migration:** replace `--fullmap ` with `--configuration-file ` (same `-f` short flag, new meaning) and build the aggregate afterward with `tablassert build-kg -f ./graph.yaml`. Remove any `tablassert rebuild-agent-graph` invocations — the subcommand is gone, and an existing `/graph.yaml` registry is orphaned: carry old results forward by adding their `configs/.yaml` paths to your graph's `tables:` list, or rerun the PMC ids (terminal records are now deliberately reprocessed every invocation, so a rerun replaces its target-graph entry). Per-article artifacts move from `builds//agent_0.0.1.*` to `builds//artifacts/_.{nodes,edges}.ndjson`. Concurrent agents may now share one target graph (flock-serialized appends, last-writer-wins per PMC) but need separate `--state-dir`s — the inverse of the old guidance. The `tablassert.graph_registry` module is deleted; direct API callers using `fullmap=` keep working, and `run_supervisor` / `build_and_audit` / `make_tools` now also accept a `graph=` (with `graph_path=` required on `run_supervisor`). -- **The QC embedding stage migrated from BioBERT to SapBERT, and the model cache moved with it.** The cascade's final stage now scores residual pairs with `cambridgeltl/SapBERT-from-PubMedBERT-fulltext` — PubMedBERT self-aligned on UMLS synonym pairs (NAACL 2021) — rather than `pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb`: SapBERT's training objective is exactly the relation the stage audits (whether two names denote the same concept), whereas the old model was trained for sentence entailment. Consequences of the swap: the cosine pass threshold stays at `0.5` — now exposed as `qc.SIMILARITY_THRESHOLD` — carried over from the BioBERT gate and *not* re-tuned for SapBERT's score distribution, so reject rates may shift until it is tuned; the `on_phase` sub-phase label renamed `qc:bert` → `qc:sapbert`; rejection logs now report the score as `sapbert=` (was `bert=`); and the model cache moved from `.tablassert/biobert/` to `.tablassert/sapbert/` (`qc.MODEL`) — existing `.tablassert/biobert/` directories are stale and can be deleted, and SapBERT (~440 MB) is downloaded and cached on the first QC run that reaches the embedding stage. No new dependencies: the same `[qc]` extra (`sentence-transformers`, `scikit-learn`) serves both models. + **Migration:** replace `--fullmap ` with `--configuration-file ` (same `-f` short flag, new meaning) and build the aggregate afterward with `tablassert build-kg -f ./graph.yaml`. Remove any `tablassert rebuild-agent-graph` invocations; the subcommand is gone, and an existing `/graph.yaml` registry is orphaned: carry old results forward by adding their `configs/.yaml` paths to your graph's `tables:` list, or rerun the PMC ids (terminal records are now deliberately reprocessed every invocation, so a rerun replaces its target-graph entry). Per-article artifacts move from `builds//agent_0.0.1.*` to `builds//artifacts/_.{nodes,edges}.ndjson`. Concurrent agents may now share one target graph (flock-serialized appends, last-writer-wins per PMC) but need separate `--state-dir`s, the inverse of the old guidance. The `tablassert.graph_registry` module is deleted; direct API callers using `fullmap=` keep working, and `run_supervisor` / `build_and_audit` / `make_tools` now also accept a `graph=` (with `graph_path=` required on `run_supervisor`). +- **The QC embedding stage migrated from BioBERT to SapBERT, and the model cache moved with it.** The cascade's final stage now scores residual pairs with `cambridgeltl/SapBERT-from-PubMedBERT-fulltext`, PubMedBERT self-aligned on UMLS synonym pairs (NAACL 2021), rather than `pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb`: SapBERT's training objective is exactly the relation the stage audits (whether two names denote the same concept), whereas the old model was trained for sentence entailment. Consequences of the swap: the cosine pass threshold stays at `0.5`, now exposed as `qc.SIMILARITY_THRESHOLD`, carried over from the BioBERT gate and *not* re-tuned for SapBERT's score distribution, so reject rates may shift until it is tuned; the `on_phase` sub-phase label renamed `qc:bert` to `qc:sapbert`; rejection logs now report the score as `sapbert=` (was `bert=`); and the model cache moved from `.tablassert/biobert/` to `.tablassert/sapbert/` (`qc.MODEL`); existing `.tablassert/biobert/` directories are stale and can be deleted, and SapBERT (~440 MB) is downloaded and cached on the first QC run that reaches the embedding stage. No new dependencies: the same `[qc]` extra (`sentence-transformers`, `scikit-learn`) serves both models. ### Added -- **A QC abbreviation-expansion stage between fuzzy matching and the embedding stage.** Pairs that fail RapidFuzz now go through a Schwartz-Hearst (2003) matcher — the source text abbreviates the resolved preferred name, or vice versa — before any model runs, and pass deterministically at zero inference cost. This rescues the class both fuzzy matching and embedding cosine miss: `AML` ↔ `acute myeloid leukemia`, `EGFR` ↔ `epidermal growth factor receptor`. Matching is case-insensitive, requires a short form of length ≥ 2 that is strictly shorter than the long form, and demands the abbreviation's first character land on a long-form word boundary (so `na` is not an abbreviation of `banana`). The audit is now a four-stage cascade (exact → fuzzy → abbreviation → SapBERT), and a new `qc:abbrev` sub-phase fires through `on_phase` when the stage runs. -- **`approval_ids` is now a curated pass-through edge field.** FDA application numbers declared as an `approval_ids` annotation (the DAKP translator-ingest precedent) now reach the final KGX edges as their own top-level field instead of being folded into `supporting_text`, notwithstanding that no association class declares the slot: the allow-list keeps the column out of the fold sweep, annotation validation emits no `BiolinkRelocationWarning`, and strict KGX validation counts it as *pending* Biolink support exactly like `effect_size` / `effect_type`. The representation follows the ingest: a pipe-joined scalar (e.g. `011111|022222`) is emitted verbatim as a scalar string — Tablassert does not split it into a JSON array. -- **`build-kg` gained `--threads` (`-t`) to control parallel fullmap reads, and those reads now scale past the 16 record shards.** Entity resolution always looked fullmap terms up with the Rust default: batches of ≥ 1024 terms fan out one reader per RECORDS shard file (16), smaller batches stay serial, and nothing above the Rust layer ever supplied an explicit worker count. The flag overrides that count — pin or limit readers on a shared host (`--threads 4`), force the serial path for reproducibility (`--threads 1`), or force parallel reads for small batches. And because redb readers hold a SHARED lock and each reader opens its own read transaction, readers never contend with each other — the one-reader-per-shard ceiling was an implementation choice, not a redb limit. The shard fan-out now splits the busiest shards' term buckets across additional concurrent readers of the SAME shard file whenever `--threads` exceeds the non-empty shard count, and results stay content- and order-identical at any worker count. Unset keeps the prior auto behavior. +- **A QC abbreviation-expansion stage between fuzzy matching and the embedding stage.** Pairs that fail RapidFuzz now go through a Schwartz-Hearst (2003) matcher: the source text abbreviates the resolved preferred name, or vice versa, before any model runs, and pass deterministically at zero inference cost. This rescues the class both fuzzy matching and embedding cosine miss: `AML` or `acute myeloid leukemia`, `EGFR` or `epidermal growth factor receptor`. Matching is case-insensitive, requires a short form of length >= 2 that is strictly shorter than the long form, and demands the abbreviation's first character land on a long-form word boundary (so `na` is not an abbreviation of `banana`). The audit is now a four-stage cascade (exact, fuzzy, abbreviation, SapBERT), and a new `qc:abbrev` sub-phase fires through `on_phase` when the stage runs. +- **`approval_ids` is now a curated pass-through edge field.** FDA application numbers declared as an `approval_ids` annotation (the DAKP translator-ingest precedent) now reach the final KGX edges as their own top-level field instead of being folded into `supporting_text`, notwithstanding that no association class declares the slot: the allow-list keeps the column out of the fold sweep, annotation validation emits no `BiolinkRelocationWarning`, and strict KGX validation counts it as *pending* Biolink support exactly like `effect_size` / `effect_type`. The representation follows the ingest: a pipe-joined scalar (e.g. `011111|022222`) is emitted verbatim as a scalar string; Tablassert does not split it into a JSON array. +- **`build-kg` gained `--threads` (`-t`) to control parallel fullmap reads, and those reads now scale past the 16 record shards.** Entity resolution always looked fullmap terms up with the Rust default: batches of >= 1024 terms fan out one reader per RECORDS shard file (16), smaller batches stay serial, and nothing above the Rust layer ever supplied an explicit worker count. The flag overrides that count: pin or limit readers on a shared host (`--threads 4`), force the serial path for reproducibility (`--threads 1`), or force parallel reads for small batches. And because redb readers hold a SHARED lock and each reader opens its own read transaction, readers never contend with each other; the one-reader-per-shard ceiling was an implementation choice, not a redb limit. The shard fan-out now splits the busiest shards' term buckets across additional concurrent readers of the SAME shard file whenever `--threads` exceeds the non-empty shard count, and results stay content- and order-identical at any worker count. Unset keeps the prior auto behavior. - **`build-kg --release` now drops edges whose `effect_size` is exactly zero.** When an `effect_size` column is present, release-mode builds filter out rows with a non-null zero effect size before fullmap resolution, matching the existing release-mode drop of `biolink:not_significant` edges. Rows with a null effect size are kept. ### Changed -- **`build-fullmap`'s prebuilt extraction now runs in the Rust extension, and extracted archives are validated against the force-build contract before install.** The prebuilt download (and its `sha256sum.txt` check) is unchanged, but the extraction is no longer Python-side (`tarfile` zstd with an installed-`zstd`-binary fallback): the extension streams the archive through zstd → tar — the multi-GB decompressed tar is never materialized on disk — with the GIL released, extracts into a temp directory on the output's filesystem, and validates the bundle BEFORE renaming anything into place: the primary's `meta` schema tag must be exactly `tablassert.fullmap.v5` (older schemas are rejected), a `build_id` must be present, the shard files must be exactly the set the primary advertises — no gaps, no extras — and every shard's `build_id` must equal the primary's. Only a passing bundle is atomically renamed beside `--output` (primary → `--output`, shard `i` → `.s.redb`); unsupported entry types (symlinks, devices) and PAX-sparse archives are rejected loudly, and path traversal is blocked. Any failure raises, so the command falls back to the from-scratch BABEL build exactly as a failed download does (the archive is kept for that retry). For end users the visible effect is faster extraction and the guarantee that an invalid prebuilt can never land a broken database at `--output`; the progress callback keeps firing its step-detail strings (`opening archive`, `extracting `, `validating`). The private Python helpers this replaces — `_extract_zst_tar` / `_stream_tar` — are gone; they were never public API, so the removal breaks only code that reached into those internals, and the external `zstd` executable (or Python 3.14 native zstd) is no longer needed at runtime. +- **`build-fullmap`'s prebuilt extraction now runs in the Rust extension, and extracted archives are validated against the force-build contract before install.** The prebuilt download (and its `sha256sum.txt` check) is unchanged, but the extraction is no longer Python-side (`tarfile` zstd with an installed-`zstd`-binary fallback): the extension streams the archive through zstd, then tar, the multi-GB decompressed tar is never materialized on disk, with the GIL released, extracts into a temp directory on the output's filesystem, and validates the bundle BEFORE renaming anything into place: the primary's `meta` schema tag must be exactly `tablassert.fullmap.v5` (older schemas are rejected), a `build_id` must be present, the shard files must be exactly the set the primary advertises, no gaps, no extras, and every shard's `build_id` must equal the primary's. Only a passing bundle is atomically renamed beside `--output` (primary to `--output`, shard `i` to `.s.redb`); unsupported entry types (symlinks, devices) and PAX-sparse archives are rejected loudly, and path traversal is blocked. Any failure raises, so the command falls back to the from-scratch BABEL build exactly as a failed download does (the archive is kept for that retry). For end users the visible effect is faster extraction and the guarantee that an invalid prebuilt can never land a broken database at `--output`; the progress callback keeps firing its step-detail strings (`opening archive`, `extracting `, `validating`). The private Python helpers this replaces, `_extract_zst_tar` / `_stream_tar`, are gone; they were never public API, so the removal breaks only code that reached into those internals, and the external `zstd` executable (or Python 3.14 native zstd) is no longer needed at runtime. ### Fixed - **`species_context_qualifier` is no longer auto-derived or relocated into supporting studies.** The field was derived from resolved node taxon metadata, then stored as a `StudyResult.description` key/value when the selected association class could not accept it. It is now disabled for both qualifiers and annotations; node taxon metadata remains available on node records. -- **A section with no publication and nothing to preserve no longer fabricates a supporting study.** `inline_supporting_study` emitted `has_supporting_studies` unconditionally, so a table declaring no `publications` got a `Study` keyed by its own config filename (`my_table.yaml`) whose single `StudyResult` was a row index into a file the pipeline regenerates. Biolink defines `has supporting studies` as "studies that produced information used as evidence", and nothing in `NCATSTranslator/translator-ingests` models evidence that way — there a `Study` is a real cohort (ICEES), dataset (COHD), trial (CTKP), or text-mining group (SemMedDB/TMKP) with typed `StudyResult` slots. The struct is now emitted only when it carries something: a section with a real publication keeps it unchanged (`PMID:123#Table_S7 row 12` is genuine provenance), as does any section with routed `UNSATISFIABLE_EDGE_FIELDS` statistics or class-pruned values to preserve. The sheet-name and row-number columns are consumed either way — they are never meant to reach the edge. +- **A section with no publication and nothing to preserve no longer fabricates a supporting study.** `inline_supporting_study` emitted `has_supporting_studies` unconditionally, so a table declaring no `publications` got a `Study` keyed by its own config filename (`my_table.yaml`) whose single `StudyResult` was a row index into a file the pipeline regenerates. Biolink defines `has supporting studies` as "studies that produced information used as evidence", and nothing in `NCATSTranslator/translator-ingests` models evidence that way; there a `Study` is a real cohort (ICEES), dataset (COHD), trial (CTKP), or text-mining group (SemMedDB/TMKP) with typed `StudyResult` slots. The struct is now emitted only when it carries something: a section with a real publication keeps it unchanged (`PMID:123#Table_S7 row 12` is genuine provenance), as does any section with routed `UNSATISFIABLE_EDGE_FIELDS` statistics or class-pruned values to preserve. The sheet-name and row-number columns are consumed either way; they are never meant to reach the edge. ## 11.0.0 - 2026-08-13 ### Breaking Changes -- **Edges no longer carry a flat `primary_knowledge_source` scalar; retrieval provenance lives only in the nested `sources` list.** Each edge emitted the graph infores three times — as the flat `primary_knowledge_source` column, as `sources[0].id`, and as `sources[0].resource_id`. The flat scalar is now gone, matching current `NCATSTranslator/translator-ingests` practice (no ingest sets it anymore; their KGX validation plugin reads only `sources`) and the Biolink Model's direction. The `sources` structure is unchanged: the primary entry (`resource_role: primary_knowledge_source`) carries `upstream_resource_ids` and `source_record_urls`, with one `supporting_data_source` entry per upstream. RIG generation is unaffected — it already read the nested `sources` first and still accepts the flat columns from legacy parquet inputs. Downstream consumers reading the flat column should read the primary `sources` entry's `resource_id` instead. +- **Edges no longer carry a flat `primary_knowledge_source` scalar; retrieval provenance lives only in the nested `sources` list.** Each edge emitted the graph infores three times: as the flat `primary_knowledge_source` column, as `sources[0].id`, and as `sources[0].resource_id`. The flat scalar is now gone, matching current `NCATSTranslator/translator-ingests` practice (no ingest sets it anymore; their KGX validation plugin reads only `sources`) and the Biolink Model's direction. The `sources` structure is unchanged: the primary entry (`resource_role: primary_knowledge_source`) carries `upstream_resource_ids` and `source_record_urls`, with one `supporting_data_source` entry per upstream. RIG generation is unaffected; it already read the nested `sources` first and still accepts the flat columns from legacy parquet inputs. Downstream consumers reading the flat column should read the primary `sources` entry's `resource_id` instead. `sources[].id` still mirrors `resource_id`: `RetrievalSource` inherits `id` from `entity`, and the LinkML-generated Biolink Pydantic classes (PyPI 4.4.3 and GitHub master 4.4.4 alike) require it, so omitting it fails KGX validation. The mirror disappears once biolink-model [#1706](https://github.com/biolink/biolink-model/issues/1706) / [#1731](https://github.com/biolink/biolink-model/pull/1731) land. -- **Graph configs now require a `rig:` section, and every generated RIG is audited before anything is written.** The generated Resource Ingest Guides were structurally weaker than upstream Translator RIGs — output KGX filenames stood in for source access locations, `relevant_files`/`included_content` were unstructured strings, and edge summaries read raw intermediate parquets — and nothing validated the document before it shipped. Every human-authored RIG fact now lives in a required `rig:` section mirroring the released `resource-ingest-guide-schema`: `source_info` (explicit `infores_id`, non-empty terms-of-use assessment, URL-bearing `data_access_locations`, enum `source_status`), `ingest_info` (required utility/scope, structured relevant/included/filtered content), `provenance_info` (required contributions), optional supporting-data/target-level extras, and `artifact_base_url`/`artifact_base_path`. Legacy top-level `description`/`contributions`/`ui_explanation`/`infores` keys are rejected with a `rig-legacy-keys` migration pointer. `compile_graph` writes artifacts under `rig.artifact_base_path`, composes `relevant_files`/`included_content` for the exact generated `.nodes.ndjson`/`.edges.ndjson` names (with observed record counts and fields), cross-checks configured upstream `relevant_files` against the table sources (stale entries fail the build), and streams edge/node type summaries from the FINAL deduplicated KGX — per-predicate edge types with resolved SPO categories, list-valued KL/AT, qualifier shapes, observed `edge_properties`, and role-separated knowledge sources. `compile_rig` audits the assembled document in memory and raises `rig-validation-failed` without writing anything on violation, so every emitted RIG is schema-shaped and internally consistent by construction. +- **Graph configs now require a `rig:` section, and every generated RIG is audited before anything is written.** The generated Resource Ingest Guides were structurally weaker than upstream Translator RIGs: output KGX filenames stood in for source access locations, `relevant_files`/`included_content` were unstructured strings, and edge summaries read raw intermediate parquets, and nothing validated the document before it shipped. Every human-authored RIG fact now lives in a required `rig:` section mirroring the released `resource-ingest-guide-schema`: `source_info` (explicit `infores_id`, non-empty terms-of-use assessment, URL-bearing `data_access_locations`, enum `source_status`), `ingest_info` (required utility/scope, structured relevant/included/filtered content), `provenance_info` (required contributions), optional supporting-data/target-level extras, and `artifact_base_url`/`artifact_base_path`. Legacy top-level `description`/`contributions`/`ui_explanation`/`infores` keys are rejected with a `rig-legacy-keys` migration pointer. `compile_graph` writes artifacts under `rig.artifact_base_path`, composes `relevant_files`/`included_content` for the exact generated `.nodes.ndjson`/`.edges.ndjson` names (with observed record counts and fields), cross-checks configured upstream `relevant_files` against the table sources (stale entries fail the build), and streams edge/node type summaries from the FINAL deduplicated KGX: per-predicate edge types with resolved SPO categories, list-valued KL/AT, qualifier shapes, observed `edge_properties`, and role-separated knowledge sources. `compile_rig` audits the assembled document in memory and raises `rig-validation-failed` without writing anything on violation, so every emitted RIG is schema-shaped and internally consistent by construction. - **Migration:** move your graph's human-authored RIG facts into a `rig:` section; the built-in graph registry and the agent's measurement configs emit honest built-in `rig` blocks to copy. Edge `primary_knowledge_source` now always derives from the explicit `rig.source_info.infores_id` — it is never implicitly derived from the graph name. + **Migration:** move your graph's human-authored RIG facts into a `rig:` section; the built-in graph registry and the agent's measurement configs emit honest built-in `rig` blocks to copy. Edge `primary_knowledge_source` now always derives from the explicit `rig.source_info.infores_id`; it is never implicitly derived from the graph name. -- **`effect_size` and `effect_type` are now mandatory as a pair.** A section declaring one without the other fails validation with `annotation-effect-size-without-type` / `annotation-effect-type-without-size`. Only half the rule existed before, and only as silent data nulling at build time: `coerce_effect_type_columns` nulls an `effect_type` on every row where `effect_size` is null, but a bare `effect_size` validated clean, warned nothing, and shipped on the edge. That is uninterpretable evidence — `"effect_size": "0.85"` says nothing without knowing whether 0.85 is an odds ratio, a Spearman rho, or a log2 fold change — and the only guard against it was prose advice in the agent prompt, which nothing enforced. +- **`effect_size` and `effect_type` are now mandatory as a pair.** A section declaring one without the other fails validation with `annotation-effect-size-without-type` / `annotation-effect-type-without-size`. Only half the rule existed before, and only as silent data nulling at build time: `coerce_effect_type_columns` nulls an `effect_type` on every row where `effect_size` is null, but a bare `effect_size` validated clean, warned nothing, and shipped on the edge. That is uninterpretable evidence: `"effect_size": "0.85"` says nothing without knowing whether 0.85 is an odds ratio, a Spearman rho, or a log2 fold change, and the only guard against it was prose advice in the agent prompt, which nothing enforced. - The check lives on `Section`, so it covers `tablassert validate`, `build-kg`, and the agent's `validate_section` final-answer gate alike, and it runs after `template`/`sections` expansion — annotation lists are concatenated, so a constant `effect_type` declared once on the template pairs with each section's own `effect_size` column: + The check lives on `Section`, so it covers `tablassert validate`, `build-kg`, and the agent's `validate_section` final-answer gate alike, and it runs after `template`/`sections` expansion; annotation lists are concatenated, so a constant `effect_type` declared once on the template pairs with each section's own `effect_size` column: ```yaml template: @@ -441,50 +441,50 @@ All notable changes to this project are documented in this file. - annotations: [{annotation: effect_size, method: column, encoding: B}] ``` - Names are judged by their **coerced** target, not their raw spelling, so alias forms are caught the same way the build sees them: `odds ratio` and the legacy `relationship_strength` both coerce to `effect_size` and both now require a sibling `effect_type`. The runtime coercion behavior is unchanged — this adds a config-time gate in front of it, nothing more. + Names are judged by their **coerced** target, not their raw spelling, so alias forms are caught the same way the build sees them: `odds ratio` and the legacy `relationship_strength` both coerce to `effect_size` and both now require a sibling `effect_type`. The runtime coercion behavior is unchanged; this adds a config-time gate in front of it, nothing more. - **Migration:** add an `effect_type` to any config declaring an effect size (`method: value` when every row shares one statistic, `method: column` when the table provides it); its value is coerced to the permissible `EffectTypes` set as before. Drop a lone `effect_type` that had no effect size — the build was already discarding it. + **Migration:** add an `effect_type` to any config declaring an effect size (`method: value` when every row shares one statistic, `method: column` when the table provides it); its value is coerced to the permissible `EffectTypes` set as before. Drop a lone `effect_type` that had no effect size; the build was already discarding it. ### Changed - **The `build-kg --qc` study no longer flags `original_*` fields for leading/trailing whitespace.** Those slots are verbatim copies of the source-table cell (written by `Tcode.encoding` under an `original_` prefix before any regex/normalization runs), so retaining the cell's whitespace is faithful to the source, not a defect. The whitespace assertion now skips any key prefixed `original_`, while every other field is still checked exactly as before. ### Fixed -- **Duplicate qualifier declarations now fail at config time instead of crashing mid-build with a raw polars error.** A table config that declared the same qualifier key twice died deep into the build with `polars.exceptions.ColumnNotFoundError: unable to find column "_two"` — both declarations resolved the same column, and the fullmap join drops the `_two` working column right after the first resolve pass, so the second pass hit an already-dropped column. The statement validator now rejects duplicate qualifier keys at config time with the `qualifier-duplicated` code, and `resolve_batch` gained a defense-in-depth guard: it validates its specs schema-only before any term extraction or database access, raising `resolve-bad-specs` on duplicate spec columns or a spec missing its `` / `+tag` normalization column, so the same failure class can never resurface as a raw polars crash from another call path. +- **Duplicate qualifier declarations now fail at config time instead of crashing mid-build with a raw polars error.** A table config that declared the same qualifier key twice died deep into the build with `polars.exceptions.ColumnNotFoundError: unable to find column "_two"`: both declarations resolved the same column, and the fullmap join drops the `_two` working column right after the first resolve pass, so the second pass hit an already-dropped column. The statement validator now rejects duplicate qualifier keys at config time with the `qualifier-duplicated` code, and `resolve_batch` gained a defense-in-depth guard: it validates its specs schema-only before any term extraction or database access, raising `resolve-bad-specs` on duplicate spec columns or a spec missing its `` / `+tag` normalization column, so the same failure class can never resurface as a raw polars crash from another call path. ## 10.1.0 - 2026-08-13 ### Added -- **`build-kg --qc` now studies the final KGX NDJSON and fails the build on violations.** After the graph is compiled, a seventh stage (only with `--qc`) streams the emitted `{name}_{version}.nodes.ndjson` / `.edges.ndjson` and asserts, in the spirit of the legacy `studyKGtsvs.pl` QC script: no duplicate node ids (Rust dedup only removes byte-identical lines, so same-id/different-content nodes are caught here), no nodes referenced by edges but never declared, no declared nodes participating in no edge, no empty/malformed JSON lines, and no string values with leading/trailing whitespace. Violations print a one-line-per-assertion summary with examples to stderr and exit non-zero, so a build can be gated in CI; a clean study logs and continues. The stage is stdlib-only — it rides the existing `--qc` flag but needs nothing from the `[qc]` extra itself. A missing NDJSON file is itself a violation, so a typo'd path can never read as a pass. +- **`build-kg --qc` now studies the final KGX NDJSON and fails the build on violations.** After the graph is compiled, a seventh stage (only with `--qc`) streams the emitted `{name}_{version}.nodes.ndjson` / `.edges.ndjson` and asserts, in the spirit of the legacy `studyKGtsvs.pl` QC script: no duplicate node ids (Rust dedup only removes byte-identical lines, so same-id/different-content nodes are caught here), no nodes referenced by edges but never declared, no declared nodes participating in no edge, no empty/malformed JSON lines, and no string values with leading/trailing whitespace. Violations print a one-line-per-assertion summary with examples to stderr and exit non-zero, so a build can be gated in CI; a clean study logs and continues. The stage is stdlib-only; it rides the existing `--qc` flag but needs nothing from the `[qc]` extra itself. A missing NDJSON file is itself a violation, so a typo'd path can never read as a pass. ### Fixed -- **`prune_to_class` no longer crashes a subgraph build on a scalar column bound to a multivalued qualifier slot.** Any section carrying a scalar column for a slot that Biolink declares multivalued *somewhere* (`anatomical_context_qualifier`, declared multivalued by 14 association classes, was the case in the wild) died at collect with `polars.exceptions.InvalidOperationError: cannot cast List type (inner: 'String', to: 'String')` — surfacing inside `format_numeric`, so the traceback pointed nowhere near the cause. The multivalued wrap was built per row (`when(class-is-multivalued).then(concat_list(...)).otherwise(scalar)`), which asks one column for two dtypes and makes Polars insert an impossible `strict_cast`. Worse, the scan that decided "is this slot multivalued?" counted classes that do not declare the slot at all (`is_multivalued` is `False` there), so every qualifier slot looked mixed. The scan now covers only *declaring* classes and the wrap applies unconditionally when all of them type the slot multivalued — uniform for every qualifier slot in the installed biolink-model — with nulls preserved (`concat_list` maps null → `[null]`). A hypothetically mixed slot keeps its scalar instead of crashing. Refused values are still nulled per row and rescued into the inlined `StudyResult` description, unchanged. -- **The "folded into `supporting_text`" annotation warning no longer fires on aliases that do reach the edge.** `Annotation.warn_when_the_slot_cannot_reach_the_edge` judged the raw alias against the edge allow-list, but the clean-phase coercions rename statistical aliases to their canonical slot before any relocation runs. `adjusted p value` — which reaches the edge as the real Biolink `float` slot `adjusted_p_value` — was reported as folded away, and so were `odds ratio` and `q_value`. The validator now resolves the name through the same classifiers the pipeline uses and judges the *coerced* target; aliases of slots that genuinely cannot reach the edge name that target instead of the alias (e.g. `sample size` → `supporting_study_size`). +- **`prune_to_class` no longer crashes a subgraph build on a scalar column bound to a multivalued qualifier slot.** Any section carrying a scalar column for a slot that Biolink declares multivalued *somewhere* (`anatomical_context_qualifier`, declared multivalued by 14 association classes, was the case in the wild) died at collect with `polars.exceptions.InvalidOperationError: cannot cast List type (inner: 'String', to: 'String')`, surfacing inside `format_numeric`, so the traceback pointed nowhere near the cause. The multivalued wrap was built per row (`when(class-is-multivalued).then(concat_list(...)).otherwise(scalar)`), which asks one column for two dtypes and makes Polars insert an impossible `strict_cast`. Worse, the scan that decided "is this slot multivalued?" counted classes that do not declare the slot at all (`is_multivalued` is `False` there), so every qualifier slot looked mixed. The scan now covers only *declaring* classes and the wrap applies unconditionally when all of them type the slot multivalued, uniform for every qualifier slot in the installed biolink-model, with nulls preserved (`concat_list` maps null to `[null]`). A hypothetically mixed slot keeps its scalar instead of crashing. Refused values are still nulled per row and rescued into the inlined `StudyResult` description, unchanged. +- **The "folded into `supporting_text`" annotation warning no longer fires on aliases that do reach the edge.** `Annotation.warn_when_the_slot_cannot_reach_the_edge` judged the raw alias against the edge allow-list, but the clean-phase coercions rename statistical aliases to their canonical slot before any relocation runs. `adjusted p value`, which reaches the edge as the real Biolink `float` slot `adjusted_p_value`, was reported as folded away, and so were `odds ratio` and `q_value`. The validator now resolves the name through the same classifiers the pipeline uses and judges the *coerced* target; aliases of slots that genuinely cannot reach the edge name that target instead of the alias (e.g. `sample size` to `supporting_study_size`). ## 10.0.0 - 2026-08-13 A major bump for one removal: `method: list` is gone, and a table config that still declares it no longer validates. A `~=9.1` pin cannot cross that silently. The other change in the release is purely additive. ### Breaking Changes -- **`method: list` is removed; use `split_by` instead.** The literal-list encoding — `method: list` with a list `encoding`, added in 9.0.0 as the successor of `Annotation.delimiter` — is gone. Every edge carried the *same* array (the list is fixed at config time), which is exactly the one shape `split_by` subsumes in practice: a multivalued annotation is declared on a column whose cells hold delimited text, and each row splits into its own JSON array. Multivalued annotations now have exactly one encoding: +- **`method: list` is removed; use `split_by` instead.** The literal-list encoding, `method: list` with a list `encoding`, added in 9.0.0 as the successor of `Annotation.delimiter`, is gone. Every edge carried the *same* array (the list is fixed at config time), which is exactly the one shape `split_by` subsumes in practice: a multivalued annotation is declared on a column whose cells hold delimited text, and each row splits into its own JSON array. Multivalued annotations now have exactly one encoding: ```yaml annotations: - {annotation: has_evidence, method: column, encoding: D, split_by: "|"} ``` - Configs still declaring `method: list` fail at validation with the new `encoding-list-method-removed` code and a pointer to `split_by`, instead of a bare enum error — including under `tablassert agent`, whose error-recovery loop reads coded errors verbatim. The `encoding` field is scalar-only now (`str | int | float`); a list value is rejected by the schema. + Configs still declaring `method: list` fail at validation with the new `encoding-list-method-removed` code and a pointer to `split_by`, instead of a bare enum error, including under `tablassert agent`, whose error-recovery loop reads coded errors verbatim. The `encoding` field is scalar-only now (`str | int | float`); a list value is rejected by the schema. - **Accepted caveat:** an array known at config time (identical on every edge) has no literal form anymore — materialize it as a source column (the same delimited value per row) and split it with `split_by`, or record it once as graph-level metadata. This trades a rare literal shape for one multivalued encoding instead of two that overlapped. + **Accepted caveat:** an array known at config time (identical on every edge) has no literal form anymore; materialize it as a source column (the same delimited value per row) and split it with `split_by`, or record it once as graph-level metadata. This trades a rare literal shape for one multivalued encoding instead of two that overlapped. ### Added -- **`nullable` qualifiers — optional per-edge qualifiers without edge loss.** A `method: column` qualifier may now declare `nullable: true` so that a blank or unresolvable cell **keeps the edge and omits the qualifier for that row**, instead of dropping the edge: +- **`nullable` qualifiers, optional per-edge qualifiers without edge loss.** A `method: column` qualifier may now declare `nullable: true` so that a blank or unresolvable cell **keeps the edge and omits the qualifier for that row**, instead of dropping the edge: ```yaml qualifiers: - {qualifier: disease_context_qualifier, method: column, encoding: F, nullable: true} ``` - A qualifier is a node encoding resolved through the fullmap alongside subject/object, and `join_matches` drops any row whose resolved node column is null — the right default for subject/object (an edge with a missing node is meaningless) but a hard constraint on qualifiers: a declared qualifier had to resolve on **every** row or the edge was lost. That forbade per-edge *optional* qualifiers, which downstream builders worked around by only declaring qualifiers backed by dense columns and dropping otherwise-usable context. + A qualifier is a node encoding resolved through the fullmap alongside subject/object, and `join_matches` drops any row whose resolved node column is null, the right default for subject/object (an edge with a missing node is meaningless) but a hard constraint on qualifiers: a declared qualifier had to resolve on **every** row or the edge was lost. That forbade per-edge *optional* qualifiers, which downstream builders worked around by only declaring qualifiers backed by dense columns and dropping otherwise-usable context. `nullable: true` threads a `drop_unresolved=False` flag through `join_matches`/`resolve_batch` for that column alone: the row survives with a null qualifier that the existing null-stripper omits from the edge, subject/object stay strict, and the miss is still reported by `log_unmatched`. QC `fullmap_audit` skips nullable qualifier columns (their nulls are expected, not resolution errors to delete). `nullable` on a literal qualifier (`method: value`) is rejected at config time with `qualifier-nullable-literal`, since a config-time constant can never be null. @@ -493,11 +493,11 @@ A major bump for one removal: `method: list` is gone, and a table config that st ## 9.1.0 - 2026-08-12 ### Added -- **`tablassert build-fullmap` now downloads a prebuilt database by default, with `--force` / `-f` to rebuild from scratch.** Without `--force`, the command first fetches the prebuilt `fullmap.tar.zst` published for the INSTALLED Tablassert version at `https://stars.renci.org/var/babel_outputs//fullmap//` (the version directory is derived from installed-package metadata, never hardcoded), verifies it against the co-published `sha256sum.txt`, and stream-extracts it beside `--output` — far faster than building from BABEL. If no prebuilt exists for this version (or the download/extract fails), it falls back to the existing from-scratch BABEL build and logs a warning; `--force` / `-f` skips the prebuilt attempt entirely. A database already present at `--output` is reused. Extraction uses Python 3.14+ native `tarfile` zstd, falling back to the installed `zstd` binary on older interpreters. The `--aria2c` / `-a` flag accelerates the prebuilt download through the same shared downloader the BABEL build uses (so it benefits from the optional bundled `[aria2]` extra when that is installed). -- **Missing optional extras now name themselves and the command that installs them.** Reaching a feature whose extra was never installed used to surface as whatever the import happened to throw — most often a bare `ModuleNotFoundError: No module named 'sklearn'`, which does not tell anyone that `tablassert[qc]` is the fix, or that `sklearn` is installed as `scikit-learn`. Every one of those paths now reports both: +- **`tablassert build-fullmap` now downloads a prebuilt database by default, with `--force` / `-f` to rebuild from scratch.** Without `--force`, the command first fetches the prebuilt `fullmap.tar.zst` published for the INSTALLED Tablassert version at `https://stars.renci.org/var/babel_outputs//fullmap//` (the version directory is derived from installed-package metadata, never hardcoded), verifies it against the co-published `sha256sum.txt`, and stream-extracts it beside `--output`, far faster than building from BABEL. If no prebuilt exists for this version (or the download/extract fails), it falls back to the existing from-scratch BABEL build and logs a warning; `--force` / `-f` skips the prebuilt attempt entirely. A database already present at `--output` is reused. Extraction uses Python 3.14+ native `tarfile` zstd, falling back to the installed `zstd` binary on older interpreters. The `--aria2c` / `-a` flag accelerates the prebuilt download through the same shared downloader the BABEL build uses (so it benefits from the optional bundled `[aria2]` extra when that is installed). +- **Missing optional extras now name themselves and the command that installs them.** Reaching a feature whose extra was never installed used to surface as whatever the import happened to throw, most often a bare `ModuleNotFoundError: No module named 'sklearn'`, which does not tell anyone that `tablassert[qc]` is the fix, or that `sklearn` is installed as `scikit-learn`. Every one of those paths now reports both: ```text - Missing optional dependencies 'scikit-learn', 'sentence-transformers' — required by the QC audit. + Missing optional dependencies 'scikit-learn', 'sentence-transformers', required by the QC audit. Install the [qc] extra: pip install "tablassert[qc]" (uv: uv tool install "tablassert[qc]") ``` @@ -510,26 +510,26 @@ A major bump for one removal: `method: list` is gone, and a table config that st | `tablassert agent --optimize` | `[agent]` + `[optimize]` | Same point; both reported at once | | `build-fullmap --aria2c` | `[aria2]` | Before the first download, rather than on it (BABEL URL discovery already hit the network by then). Keeps the platform-aware message: on macOS, where the `aria2` distribution publishes no wheels, it still says to drop the flag rather than to install a dead end | - `build-kg --qc` is the one that mattered most. The QC audit is the pipeline's LAST stage, so a missing extra was discovered only after entity resolution had already finished — the whole build spent, then a traceback. The check now costs one `importlib.util.find_spec` probe (no import, so nothing is paid for having the extra) and happens before stage 1. + `build-kg --qc` is the one that mattered most. The QC audit is the pipeline's LAST stage, so a missing extra was discovered only after entity resolution had already finished, the whole build spent, then a traceback. The check now costs one `importlib.util.find_spec` probe (no import, so nothing is paid for having the extra) and happens before stage 1. A half-installed extra reports every package it is still missing rather than one per attempt. That was a real failure mode in `[qc]`: `scikit-learn` is imported at the start of the audit and `sentence-transformers` only if Stage 3 is reached, so installing the first would have run the audit again just to fail on the second. - Flag and secret validation still comes first — `tablassert agent` reports a missing `--model-id` before a missing extra, since fixing an install only to be told the model id was never set is the worse loop. + Flag and secret validation still comes first; `tablassert agent` reports a missing `--model-id` before a missing extra, since fixing an install only to be told the model id was never set is the worse loop. - New `tablassert.extras` module holds the package → extra mapping as the single source of truth, and a test asserts it against `pyproject.toml`, so an extra cannot be added or a dependency moved without the hints following. + New `tablassert.extras` module holds the package-to-extra mapping as the single source of truth, and a test asserts it against `pyproject.toml`, so an extra cannot be added or a dependency moved without the hints following. -- **`split_by` on annotations — per-row multivalued slots.** A `method: column` annotation may declare a separator that splits each cell's own delimited text into a real JSON array: +- **`split_by` on annotations, per-row multivalued slots.** A `method: column` annotation may declare a separator that splits each cell's own delimited text into a real JSON array: ```yaml annotations: - {annotation: has_evidence, method: column, encoding: D, split_by: "|"} ``` - This closes the gap 8.2.1 opened. Removing `Annotation.delimiter` left `method: list` as the only multivalued encoding, but a list `encoding` is a literal — it emits the same array on every row — so a column whose cells hold aggregated values had no migration at all. `split_by` is the per-row counterpart: `method: list` for an array known at config time, `split_by` for one that differs per row. + This closes the gap 8.2.1 opened. Removing `Annotation.delimiter` left `method: list` as the only multivalued encoding, but a list `encoding` is a literal; it emits the same array on every row, so a column whose cells hold aggregated values had no migration at all. `split_by` is the per-row counterpart: `method: list` for an array known at config time, `split_by` for one that differs per row. - The failure it prevents is silent rather than loud. `mask_illegal_edge_fields` wraps a scalar bound for a multivalued slot into a one-element list, so a joined cell emits `has_evidence: ["EFO:0001|EFO:0002"]` — structurally valid Biolink that passes `validate-kgx` while handing consumers one unusable blob instead of two ids. + The failure it prevents is silent rather than loud. `mask_illegal_edge_fields` wraps a scalar bound for a multivalued slot into a one-element list, so a joined cell emits `has_evidence: ["EFO:0001|EFO:0002"]`, structurally valid Biolink that passes `validate-kgx` while handing consumers one unusable blob instead of two ids. - `split_by` requires `method: column` (`annotation-split-by-requires-column`) and rejects an empty separator (`annotation-split-by-empty`), which would split into individual characters. It is unrelated to the `source.delimiter` CSV/TSV field separator, and it is annotation-only — subject/object/qualifier nodes are single entities. + `split_by` requires `method: column` (`annotation-split-by-requires-column`) and rejects an empty separator (`annotation-split-by-empty`), which would split into individual characters. It is unrelated to the `source.delimiter` CSV/TSV field separator, and it is annotation-only: subject/object/qualifier nodes are single entities. Purely additive: existing configs are unaffected, and upgrading from 8.2.x with column-based `delimiter` configs is now a rename to `split_by`. @@ -538,35 +538,35 @@ A major bump for one removal: `method: list` is gone, and a table config that st | | Destination | Use for | | --- | --- | --- | - | `explode_by` | one **row** per item | node encodings — each item is its own entity, its own edge | - | `split_by` | one **array** on the row | annotations — the items are one multivalued slot on a single edge | + | `explode_by` | one **row** per item | node encodings: each item is its own entity, its own edge | + | `split_by` | one **array** on the row | annotations: the items are one multivalued slot on a single edge | - This tightens `explode_by`: items are now trimmed and blanks dropped, so a trailing or doubled separator (`"P1;P2;"`, `"P1;;P2"` — routine in hand-maintained spreadsheets) no longer fans out a row carrying `""`. Those rows only ever failed entity resolution and were discarded downstream, so no edge changes; the work is simply not done. Trimming is likewise not a behavior change for nodes — `level_one` already strips before resolution — but it is load-bearing for annotations, which are never resolved and previously would have carried `" b"` straight onto the edge. + This tightens `explode_by`: items are now trimmed and blanks dropped, so a trailing or doubled separator (`"P1;P2;"`, `"P1;;P2"`, routine in hand-maintained spreadsheets) no longer fans out a row carrying `""`. Those rows only ever failed entity resolution and were discarded downstream, so no edge changes; the work is not done. Trimming is likewise not a behavior change for nodes (`level_one` already strips before resolution) but it is load-bearing for annotations, which are never resolved and previously would have carried `" b"` straight onto the edge. - **`tablassert build-fullmap --aria2c` / `-a` now uses the optional `[aria2]` PyPI extra instead of a system `aria2c` install.** Install with `pip install "tablassert[aria2]"` to get the bundled static aria2c binary from the `aria2` package (`aria2==0.0.1b0`, imported as `aria2c`). The extra has Linux/Windows wheels only; on macOS, `--aria2c` fails loud and the default Python downloader remains available. `aria2` is a separate optional GPL-2.0 runtime dependency; Tablassert remains Apache-2.0, but redistributors who ship the optional extra should review GPL-2.0 obligations. ### Fixed -- **Suggested install commands now quote the extra: `pip install "tablassert[qc]"`.** Unquoted brackets are a glob pattern in zsh — the default shell on macOS — so every `pip install tablassert[agent]` this project printed or documented failed with `zsh: no matches found` before pip was ever reached. A suggestion the user's own shell rejects is worse than none. `agent.AGENT_EXTRA` and `agent.OPTIMIZE_EXTRA` carry the quoted form and are now derived from the registry rather than written out by hand. -- **The Excel error messages no longer recommend an extra that has never contained an Excel engine.** An unreadable workbook told users to `install tablassert[agent] or tablassert[rt]`. Neither extra ships an Excel engine, and the calamine engine (`fastexcel`) became a core dependency, so the advice both misdirected and described an install the user already had. The message now says what is actually true — calamine ships with the base install, so a failure is usually the workbook itself — and points at `pip install openpyxl` for the pure-Python fallback engine. `docs/installation.md` carried the same drift (`pip install python-calamine`) and is corrected. +- **Suggested install commands now quote the extra: `pip install "tablassert[qc]"`.** Unquoted brackets are a glob pattern in zsh, the default shell on macOS, so every `pip install tablassert[agent]` this project printed or documented failed with `zsh: no matches found` before pip was ever reached. A suggestion the user's own shell rejects is worse than none. `agent.AGENT_EXTRA` and `agent.OPTIMIZE_EXTRA` carry the quoted form and are now derived from the registry rather than written out by hand. +- **The Excel error messages no longer recommend an extra that has never contained an Excel engine.** An unreadable workbook told users to `install tablassert[agent] or tablassert[rt]`. Neither extra ships an Excel engine, and the calamine engine (`fastexcel`) became a core dependency, so the advice both misdirected and described an install the user already had. The message now says what is actually true (calamine ships with the base install, so a failure is usually the workbook itself) and points at `pip install openpyxl` for the pure-Python fallback engine. `docs/installation.md` carried the same drift (`pip install python-calamine`) and is corrected. - **`agent --backend litellm` names the `[agent]` extra when `litellm` is absent.** The path only required `smolagents`, leaving smolagents' own import error to explain a Tablassert extra. -- **A `polars` import failure now points at the `[rt]` extra.** polars is a core dependency, so it is never merely absent — the realistic failure is a wheel whose instruction set the CPU does not support, which is exactly what `polars[rtcompat]` (the `[rt]` extra) exists to fix. The extra cannot be detected by inspection (it imports as plain `polars`), so this hint is the only place a user learns it exists. +- **A `polars` import failure now points at the `[rt]` extra.** polars is a core dependency, so it is never merely absent: the realistic failure is a wheel whose instruction set the CPU does not support, which is exactly what `polars[rtcompat]` (the `[rt]` extra) exists to fix. The extra cannot be detected by inspection (it imports as plain `polars`), so this hint is the only place a user learns it exists. ### Documentation -- **Issue and pull-request templates.** `.github/ISSUE_TEMPLATE/bug_report.md` asks for the parts of a Tablassert report that are otherwise missing on the first round-trip — the exact command, expected versus actual behavior, and the OS/Python/Rust environment — and `feature_request.md` asks a proposal to state its problem and success metrics before its implementation. `.github/pull_request_template.md` asks for the validation commands actually run and their results, rather than an unqualified "tests pass". +- **Issue and pull-request templates.** `.github/ISSUE_TEMPLATE/bug_report.md` asks for the parts of a Tablassert report that are otherwise missing on the first round-trip (the exact command, expected versus actual behavior, and the OS/Python/Rust environment) and `feature_request.md` asks a proposal to state its problem and success metrics before its implementation. `.github/pull_request_template.md` asks for the validation commands actually run and their results, rather than an unqualified "tests pass". - `llms.txt` listed the optional extras as `rt` and `qc` only, predating `aria2`, `agent`, and `optimize`; all five are now named in both places the file lists them, and the new `tablassert.extras` registry is on the implementation map. - The `--version` example in `docs/cli.md` had been left at `tablassert 8.2.0`. ## 9.0.0 - 2026-08-11 -**Identical in content to 8.2.1 — re-released under a major version to correct the version signal.** +**Identical in content to 8.2.1, re-released under a major version to correct the version signal.** 8.2.1 shipped as a patch release but carried two breaking configuration changes: `source.url` became a list (`url: list[HttpUrl]`, no longer accepting the scalar form) and `Annotation.delimiter` was removed in favor of `method: list`. A patch bump advertises a drop-in upgrade, so a downstream pinned to `~=8.2.0` or `>=8.2,<8.3` would have picked 8.2.1 up automatically and broken on its existing table configs. -9.0.0 contains no code, test, or documentation changes over 8.2.1 — only the version bump. It exists so the breaking changes are announced by the version number itself, and so a compatible-release pin cannot cross them silently. 8.2.1 remains on PyPI and is unaffected; upgrade from any 8.x directly to 9.0.0 and apply the migrations described under 8.2.1's **Breaking Changes** below. +9.0.0 contains no code, test, or documentation changes over 8.2.1, only the version bump. It exists so the breaking changes are announced by the version number itself, and so a compatible-release pin cannot cross them silently. 8.2.1 remains on PyPI and is unaffected; upgrade from any 8.x directly to 9.0.0 and apply the migrations described under 8.2.1's **Breaking Changes** below. ## 8.2.1 - 2026-08-11 ### Breaking Changes -- **`source.url` is now a list of URLs (`url: list[HttpUrl]`).** A table-config section may declare one or more remote source URLs, all recorded as provenance (emitted in the edge `source_record_urls` list and the RIG). The legacy scalar form `url: https://example.com/x.tsv` is no longer accepted — wrap it in a list. Update existing configs from `url: https://...` to a sequence: +- **`source.url` is now a list of URLs (`url: list[HttpUrl]`).** A table-config section may declare one or more remote source URLs, all recorded as provenance (emitted in the edge `source_record_urls` list and the RIG). The legacy scalar form `url: https://example.com/x.tsv` is no longer accepted; wrap it in a list. Update existing configs from `url: https://...` to a sequence: ```yaml source: @@ -574,36 +574,36 @@ A major bump for one removal: `method: list` is gone, and a table config that st - https://example.com/data.tsv ``` -- **`Annotation.delimiter` is removed; use `method: list` instead.** Multivalued annotations no longer split an encoded scalar on a separator — declare the list directly with the new `method: list` (see Added). Update `annotations: [{annotation: has_evidence, method: value, encoding: "a|b", delimiter: "|"}]` to `annotations: [{annotation: has_evidence, method: list, encoding: ["a", "b"]}]`. The `source.delimiter` CSV/TSV separator is unrelated and unchanged. +- **`Annotation.delimiter` is removed; use `method: list` instead.** Multivalued annotations no longer split an encoded scalar on a separator; declare the list directly with the new `method: list` (see Added). Update `annotations: [{annotation: has_evidence, method: value, encoding: "a|b", delimiter: "|"}]` to `annotations: [{annotation: has_evidence, method: list, encoding: ["a", "b"]}]`. The `source.delimiter` CSV/TSV separator is unrelated and unchanged. **`method: list` only replaces the literal (`method: value`) form of `delimiter`.** Because a list `encoding` is a literal, the same array is emitted on every row, so a *column-based* annotation that relied on `delimiter` to split each cell's own value (`{annotation: has_evidence, method: column, encoding: D, delimiter: "|"}`) has no direct migration. `explode_by` is not a substitute: it fans one row out into many rows rather than emitting a per-row JSON array. Such configs must either pre-split the column upstream (emit one already-delimited source column per value, or reshape the source so each row carries a single value) or stay on 8.2.x until per-column list support ships. ### Added - **`tablassert build-fullmap --aria2c` / `-a`** opt-in downloader acceleration. When requested, the BABEL download stage uses the installed `aria2c` executable with segmented HTTP downloads plus resume/retry flags (`--continue=true`, `--max-tries`, `--retry-wait`) while keeping the existing Python downloader as the default. Missing or failing `aria2c` fails loud instead of silently falling back, and aria2 `.aria2` control files are preserved so interrupted downloads can resume on rerun. -- **`method: list` annotation encoding.** A new encoding method — the multivalued counterpart of `method: value` — lets an annotation carry a manually-defined list of values, emitted verbatim as a real JSON array for multivalued Biolink slots such as `has_evidence`. It replaces the removed `Annotation.delimiter` (which split an encoded scalar). `method: list` is annotation-only: subject/object/qualifier nodes are single entities and reject it at config time, and it is incompatible with the scalar string ops (`regex`/`remove`/`prefix`/`suffix`/`transformations`/`fill`/`explode_by`). -- **The autonomous agent now measures and optimizes the Biolink validity of its own output.** 8.2.0 rebuilt the emit path so KGX validates against the Biolink Model, but the agent — the component that authors configs unsupervised — was never retrofitted: `build_and_audit` wrote its NDJSON, counted the lines, and returned, so a run could converge on, persist, and register a config whose output validated at 0%. It now constructs every emitted record as its own Biolink class (the same check `validate-kgx` runs) and reports four new fields: - - `biolink_valid_pct` — the pass rate excluding known-pending fields; the number the objective function scores. - - `biolink_valid_pct_strict` — the unexempted rate, so the pending gap stays visible. - - `biolink_problems` — the top `"field: error-type"` failures with counts, so the model can self-correct. - - `demoted_edge_pct` — the fraction of edges that fell back to bare `biolink:Association`. This is the **predicate** signal: a predicate its association class forbids is never an error, it silently discards the class and every qualifier and evidence slot the class declared, and nothing else surfaces it. +- **`method: list` annotation encoding.** A new encoding method, the multivalued counterpart of `method: value`, lets an annotation carry a manually-defined list of values, emitted verbatim as a real JSON array for multivalued Biolink slots such as `has_evidence`. It replaces the removed `Annotation.delimiter` (which split an encoded scalar). `method: list` is annotation-only: subject/object/qualifier nodes are single entities and reject it at config time, and it is incompatible with the scalar string ops (`regex`/`remove`/`prefix`/`suffix`/`transformations`/`fill`/`explode_by`). +- **The autonomous agent now measures and optimizes the Biolink validity of its own output.** 8.2.0 rebuilt the emit path so KGX validates against the Biolink Model, but the agent, the component that authors configs unsupervised, was never retrofitted: `build_and_audit` wrote its NDJSON, counted the lines, and returned, so a run could converge on, persist, and register a config whose output validated at 0%. It now constructs every emitted record as its own Biolink class (the same check `validate-kgx` runs) and reports four new fields: + - `biolink_valid_pct`: the pass rate excluding known-pending fields; the number the objective function scores. + - `biolink_valid_pct_strict`: the unexempted rate, so the pending gap stays visible. + - `biolink_problems`: the top `"field: error-type"` failures with counts, so the model can self-correct. + - `demoted_edge_pct`: the fraction of edges that fell back to bare `biolink:Association`. This is the **predicate** signal: a predicate its association class forbids is never an error, it silently discards the class and every qualifier and evidence slot the class declared, and nothing else surfaces it. - **`tablassert agent --biolink-threshold`** gates `MAPPED` on that pass rate. Defaults to `0.0` (report only), so terminal statuses are unchanged unless you opt in; `biolink_valid_pct` / `demoted_edge_pct` are recorded on every `state.json` record either way. -- **A generated legal-predicate cheat-sheet in the agent prompt.** The agent's only vocabulary channel was the ~30 KB `Section.model_json_schema()` enum dump — 247 predicates and 159 categories with nothing tying the two together. The prompt now carries a compact predicate↔class table for the pairs the agent meets in practice, rendered at import from the installed `biolink-model`, so it cannot drift from the model the build validates against. -- **`biolink.legal_predicates()` and `lib.predicate_options()`** — the missing authoring-time helpers. `predicate_options("Gene", "Disease")` returns `{affects, associated_with, contributes_to}`; there was previously no way to ask which predicates a subject/object pair may carry without composing three private functions. -- **`biolink.KNOWN_PENDING_EDGE_FIELDS`** — the curated extras Tablassert emits deliberately that the pinned model does not declare (`effect_size` / `effect_type` pending [biolink-model#1774](https://github.com/biolink/biolink-model/pull/1774), plus the KGX denormalized carryovers). `validate_kgx` now reports `valid_excluding_pending` / `ok_excluding_pending` alongside the strict counts, so a deliberate gap is not scored as a modelling error. Derived from the installed package, so it empties itself as the model catches up. +- **A generated legal-predicate cheat-sheet in the agent prompt.** The agent's only vocabulary channel was the ~30 KB `Section.model_json_schema()` enum dump: 247 predicates and 159 categories with nothing tying the two together. The prompt now carries a compact predicate-to-class table for the pairs the agent meets in practice, rendered at import from the installed `biolink-model`, so it cannot drift from the model the build validates against. +- **`biolink.legal_predicates()` and `lib.predicate_options()`**, the missing authoring-time helpers. `predicate_options("Gene", "Disease")` returns `{affects, associated_with, contributes_to}`; there was previously no way to ask which predicates a subject/object pair may carry without composing three private functions. +- **`biolink.KNOWN_PENDING_EDGE_FIELDS`**, the curated extras Tablassert emits deliberately that the pinned model does not declare (`effect_size` / `effect_type` pending [biolink-model#1774](https://github.com/biolink/biolink-model/pull/1774), plus the KGX denormalized carryovers). `validate_kgx` now reports `valid_excluding_pending` / `ok_excluding_pending` alongside the strict counts, so a deliberate gap is not scored as a modelling error. Derived from the installed package, so it empties itself as the model catches up. - **A `BiolinkRelocationWarning` on annotations whose values cannot reach the edge.** An annotation named `supporting_study_size` or `sample_size` is routed onto the inlined `StudyResult`; one like `q_value` is folded into `supporting_text`. Both were silent. This is a warning, not an error: nothing is lost and every existing config keeps building. - New regression tests: three `sig()` rigor tests (raw-`p_value`-over-adjusted preference, canonical-column-wins-over-alias, and `pvalue_target`-based exclusion of a look-alike substring column) plus an end-to-end smoke proving the real pipeline normalizes raw statistical annotation names (`p value`, `sample size`, `odds ratio`, `effect type`) to canonical Biolink edge fields and routes `supporting_study_size` / `statistical_significance_qualifier` into the inlined Study. ### Fixed -- **`map_coverage` no longer resolves enum-ranged qualifiers the build deliberately skips.** `lib.Tcode._node_ops` excludes them (their vocabulary wants the token `increased`, not a CURIE), but the agent's coverage measurement sent every qualifier through the fullmap — counting terms the build never looks up, depressing `overall` for a column working exactly as designed, and potentially flipping a good config to `SKIPPED`. -- **The final-answer gates no longer swallow the coded error text.** `validate_section` / `validate_table_config` keep their boolean contract, but the reason is now available via the new `section_error()` / `table_config_error()`, and `derive_config` returns it to the agent instead of forwarding an invalid config — so the model finally sees the actionable messages (`qualifier-unsatisfiable`: use a concrete subtype; `qualifier-bad-value`: here is the permitted vocabulary) those errors were written to carry. +- **`map_coverage` no longer resolves enum-ranged qualifiers the build deliberately skips.** `lib.Tcode._node_ops` excludes them (their vocabulary wants the token `increased`, not a CURIE), but the agent's coverage measurement sent every qualifier through the fullmap, counting terms the build never looks up, depressing `overall` for a column working exactly as designed, and potentially flipping a good config to `SKIPPED`. +- **The final-answer gates no longer swallow the coded error text.** `validate_section` / `validate_table_config` keep their boolean contract, but the reason is now available via the new `section_error()` / `table_config_error()`, and `derive_config` returns it to the agent instead of forwarding an invalid config, so the model finally sees the actionable messages (`qualifier-unsatisfiable`: use a concrete subtype; `qualifier-bad-value`: here is the permitted vocabulary) those errors were written to carry. - **The improve loop no longer trades Biolink validity for coverage.** A candidate is accepted only when it regresses on neither axis and improves on at least one. Still monotonic. -- **`validate-kgx` no longer passes on a file it never read.** A missing or misspelled path yielded `total=0, valid=0`, kept `ok` true, and exited 0 reporting "KGX output is Biolink-compliant" — a false pass in CI. +- **`validate-kgx` no longer passes on a file it never read.** A missing or misspelled path yielded `total=0, valid=0`, kept `ok` true, and exited 0 reporting "KGX output is Biolink-compliant", a false pass in CI. - **`examples/agent/optimized_instructions.yaml` no longer recommends `gene_associated_with_condition`** for gene~disease tables. `GeneToDiseaseAssociation` forbids it, which is exactly the 723,595-edge failure 8.2.0's Biolink fix measured; post-fix it no longer errors, it silently demotes. The stats-annotation guidance was likewise half-updated and listed `sample_size` and other names that fold into `supporting_text`. The next `--optimize` run reseeds from the corrected built-in `INSTRUCTIONS`. -- **`examples/agent/qc/qc_report.py` no longer flags the correct predicate as wrong.** Its `GENERIC_PREDICATES` set marked `associated_with` a "generic fallback" — but that is one of only three predicates `GeneToDiseaseAssociation` permits. It now asks the model whether the predicate demotes the edge instead of matching on spelling. +- **`examples/agent/qc/qc_report.py` no longer flags the correct predicate as wrong.** Its `GENERIC_PREDICATES` set marked `associated_with` a "generic fallback", but that is one of only three predicates `GeneToDiseaseAssociation` permits. It now asks the model whether the predicate demotes the edge instead of matching on spelling. ### Changed - **`quality_score` reweighted** to coverage 0.40, Biolink validity 0.25, node/edge F1 0.15, QC 0.10, schema validity 0.10 (still a hard gate). Most of the new weight came out of `w_qc`, which scores `build_and_audit`'s structurally-constant `qc_pass_rate`. GEPA's feedback string now carries `biolink_problems` and `demoted_edge_pct`, and the judge rubric gained a `biolink_validity` dimension. -- **`sig()` now applies the same fuzzy-matching rigor as the column coercions.** The `statistical_significance_qualifier` is derived from a p-value column chosen by `pvalue_target()` — the same delimiter-anchored classifier `coerce_pvalue_columns` uses — rather than a naive substring. A raw `p_value` column is preferred over `adjusted_p_value`, and an existing canonical column always wins over a higher-scoring spaced alias, mirroring the selection rule every other `coerce_*` step already uses. The five-band cascade and the Biolink class rule (qualifier omitted when no p-value column is present) are unchanged. No realistic build is affected: `coerce_pvalue_columns` canonicalizes every p-value column before `sig` runs, so the old and new selectors pick the same column; the lone divergence is a contrived column that merely contains a `p_value` substring but is not a real p-value column, which now correctly omits the qualifier instead of deriving a bogus one. +- **`sig()` now applies the same fuzzy-matching rigor as the column coercions.** The `statistical_significance_qualifier` is derived from a p-value column chosen by `pvalue_target()`, the same delimiter-anchored classifier `coerce_pvalue_columns` uses, rather than a naive substring. A raw `p_value` column is preferred over `adjusted_p_value`, and an existing canonical column always wins over a higher-scoring spaced alias, mirroring the selection rule every other `coerce_*` step already uses. The five-band cascade and the Biolink class rule (qualifier omitted when no p-value column is present) are unchanged. No realistic build is affected: `coerce_pvalue_columns` canonicalizes every p-value column before `sig` runs, so the old and new selectors pick the same column; the lone divergence is a contrived column that merely contains a `p_value` substring but is not a real p-value column, which now correctly omits the qualifier instead of deriving a bogus one. ### Documentation - `docs/agent.md` gains a **Biolink validity** section; its "NCATS Translator-compliant KGX" claim is now verified by the loop rather than asserted. @@ -614,53 +614,53 @@ A major bump for one removal: `method: list` is gone, and a table config that st ### Breaking Changes - **`dspy` moved out of the `[agent]` extra into a new `[optimize]` extra.** `dspy` is used ONLY by the GEPA prompt-optimization path (`agent --optimize`); ordinary agent runs never import it. Installs that use `--optimize` must now install `pip install "tablassert[agent,optimize]"` (or add `tablassert[optimize]`); the missing-package error now points at `tablassert[optimize]` accordingly. Installs that never run `--optimize` get a lighter `[agent]` install (no `dspy`). -- **Fullmap databases built by older releases must be rebuilt.** The Rust extension upgraded its embedded database engine from redb 2.6 to redb 4.1, and redb ≥ 3 dropped the old v2 file format. Existing `fullmap.redb` (and sibling `fullmap.s*.redb`) files fail to open with `fullmap DB is outdated or needs repair; rebuild with 'tablassert build-fullmap'`. Run `tablassert build-fullmap` once after upgrading. BABEL downloads stay cached, but the command rebuilds the fullmap files. The on-disk fullmap schema is now `tablassert.fullmap.v5` (the table layout is unchanged; the bump makes the redb-4 rebuild explicit and lets an older extension reject new files loudly). +- **Fullmap databases built by older releases must be rebuilt.** The Rust extension upgraded its embedded database engine from redb 2.6 to redb 4.1, and redb >= 3 dropped the old v2 file format. Existing `fullmap.redb` (and sibling `fullmap.s*.redb`) files fail to open with `fullmap DB is outdated or needs repair; rebuild with 'tablassert build-fullmap'`. Run `tablassert build-fullmap` once after upgrading. BABEL downloads stay cached, but the command rebuilds the fullmap files. The on-disk fullmap schema is now `tablassert.fullmap.v5` (the table layout is unchanged; the bump makes the redb-4 rebuild explicit and lets an older extension reject new files loudly). ### Changed -- **Fullmap reads no longer serialize across processes.** The lookup path (`lookup_fullmap_terms` and the `hydrate_*` helpers) now opens the fullmap redb files READ-ONLY with a SHARED file lock (redb ≥ 3 `ReadOnlyDatabase`) instead of an exclusive lock: concurrent readers — the agent supervisor, its code-executor subprocesses, and parallel `agent run` processes — no longer contend on the fullmap lock ("Database already open"); only a running `build-fullmap` rebuild can briefly block readers. Read-only opens also never touch the file mtime, making the mtime-keyed Python lookup caches fully stable. The redb 4.1 upgrade additionally speeds up multi-threaded shard reads (~15% on upstream benchmarks) and the fullmap build's redb write phase (~1.5x on upstream write benchmarks). +- **Fullmap reads no longer serialize across processes.** The lookup path (`lookup_fullmap_terms` and the `hydrate_*` helpers) now opens the fullmap redb files READ-ONLY with a SHARED file lock (redb >= 3 `ReadOnlyDatabase`) instead of an exclusive lock: concurrent readers, the agent supervisor, its code-executor subprocesses, and parallel `agent run` processes, no longer contend on the fullmap lock ("Database already open"); only a running `build-fullmap` rebuild can briefly block readers. Read-only opens also never touch the file mtime, making the mtime-keyed Python lookup caches fully stable. The redb 4.1 upgrade additionally speeds up multi-threaded shard reads (~15% on upstream benchmarks) and the fullmap build's redb write phase (~1.5x on upstream write benchmarks). ### Performance - **The fullmap build's redb engine advanced to the 4.2-to-be (pinned `cberner/redb` master, rev `a35e7cc`) and its Phase-4 RECORDS write now appends through an end-of-table cursor.** Two independent wins, both measured on this repo's build path: - - **Ascending-key insert page optimization** (automatic in the new engine): an insert past a table's last key now starts a new leaf page instead of splitting the full one and leaving dead free space, so a key-order-loaded table occupies about half as many pages. Measured effect: the 16 RECORDS shard files shrink **~50%** on identical input (a 539 MB shard set rebuilds to 270 MB), which also makes cold lookups faster (half the bytes to page in) and halves the DB's disk footprint. The file format is unchanged (still redb v3), so this is a drop-in engine swap — existing `tablassert.fullmap.v5` databases keep opening and no rebuild is required; a redb-4.1 extension also reads files the new engine writes (and vice versa). - - **Experimental cursor bulk inserts** (`experimental_cursor`): `write_shard_records` now opens one `upper_bound_mut(Unbounded)` cursor per shard batch and appends the hash-sorted merged groups via `insert_before`, which redb documents as ~3× faster than per-key `insert()` for ascending data. An internally-controlled microbenchmark (2 M ascending fixed-size records, both paths in one process) measured the cursor path at **~4× the insert throughput** of plain `insert()` (0.22 s vs 0.89 s) at identical file size. The cursor requires strictly ascending keys and never overwrites, so the rare duplicate xxh64 hash (two distinct terms colliding) falls back to a plain `insert()` for that one record — preserving the historical overwrite semantics — and reopens the cursor; covered by a dedicated regression test. When redb 4.2.0 publishes on crates.io the git pin swaps back to `redb = "4.2"`. -- **BABEL gzip decompression switched from flate2's default miniz_oxide backend to `zlib-rs`** (`flate2 = { features = ["zlib-rs"], default-features = false }`). The build's producer threads decompress ~30–40 GB of BABEL `.gz`; on a real 46 MB BABEL synonym stream zlib-rs decompressed at ~4.6 GB/s vs miniz_oxide's ~3.0 GB/s (**~1.5× faster**), and it is pure Rust with runtime SIMD multiversioning, so shipped wheels need no C toolchain. + - **Ascending-key insert page optimization** (automatic in the new engine): an insert past a table's last key now starts a new leaf page instead of splitting the full one and leaving dead free space, so a key-order-loaded table occupies about half as many pages. Measured effect: the 16 RECORDS shard files shrink **~50%** on identical input (a 539 MB shard set rebuilds to 270 MB), which also makes cold lookups faster (half the bytes to page in) and halves the DB's disk footprint. The file format is unchanged (still redb v3), so this is a drop-in engine swap: existing `tablassert.fullmap.v5` databases keep opening and no rebuild is required; a redb-4.1 extension also reads files the new engine writes (and vice versa). + - **Experimental cursor bulk inserts** (`experimental_cursor`): `write_shard_records` now opens one `upper_bound_mut(Unbounded)` cursor per shard batch and appends the hash-sorted merged groups via `insert_before`, which redb documents as ~3x faster than per-key `insert()` for ascending data. An internally-controlled microbenchmark (2 M ascending fixed-size records, both paths in one process) measured the cursor path at **~4x the insert throughput** of plain `insert()` (0.22 s vs 0.89 s) at identical file size. The cursor requires strictly ascending keys and never overwrites, so the rare duplicate xxh64 hash (two distinct terms colliding) falls back to a plain `insert()` for that one record, preserving the historical overwrite semantics, and reopens the cursor; covered by a dedicated regression test. When redb 4.2.0 publishes on crates.io the git pin swaps back to `redb = "4.2"`. +- **BABEL gzip decompression switched from flate2's default miniz_oxide backend to `zlib-rs`** (`flate2 = { features = ["zlib-rs"], default-features = false }`). The build's producer threads decompress ~30-40 GB of BABEL `.gz`; on a real 46 MB BABEL synonym stream zlib-rs decompressed at ~4.6 GB/s vs miniz_oxide's ~3.0 GB/s (**~1.5x faster**), and it is pure Rust with runtime SIMD multiversioning, so shipped wheels need no C toolchain. ## 8.1.0 - 2026-08-03 ### Breaking Changes -- **Removed the `TABLASSERT_FULLMAP_SHARDS` environment variable.** The number of on-disk RECORDS shard files is now fixed at the compile-time cap (`16`) and can no longer be overridden at build time; the variable is silently ignored if set. Default builds are unaffected — the previous default was already `16`. The read path still honors the shard count recorded in an existing database's `meta` table (`shards`), so databases built with fewer shards under the old variable continue to open and resolve correctly. See `docs/fullmap.md`. -- **Removed the `build-kg --table-config`/`-tc` flag** (and the `build-kg --fullmap`/`-fm` flag, which existed solely to feed it). `build-kg` now always takes a graph YAML; the throwaway `TEMP_KG` wrapper for building a bare table (Section) config is gone. To build a single table config, wrap it in a graph config — `tablassert agent` already writes a ready `graph.yaml` alongside each build under `builds//`. (`validate --schema table` still checks a bare table config on its own.) +- **Removed the `TABLASSERT_FULLMAP_SHARDS` environment variable.** The number of on-disk RECORDS shard files is now fixed at the compile-time cap (`16`) and can no longer be overridden at build time; the variable is silently ignored if set. Default builds are unaffected: the previous default was already `16`. The read path still honors the shard count recorded in an existing database's `meta` table (`shards`), so databases built with fewer shards under the old variable continue to open and resolve correctly. See `docs/fullmap.md`. +- **Removed the `build-kg --table-config`/`-tc` flag** (and the `build-kg --fullmap`/`-fm` flag, which existed solely to feed it). `build-kg` now always takes a graph YAML; the throwaway `TEMP_KG` wrapper for building a bare table (Section) config is gone. To build a single table config, wrap it in a graph config: `tablassert agent` already writes a ready `graph.yaml` alongside each build under `builds//`. (`validate --schema table` still checks a bare table config on its own.) - **Renamed `build-kg`'s configuration-file parameter** from `configuration_file` to `graph_configuration_file` to reflect that it is always a graph config. The CLI flags are unchanged (`--configuration-file`/`-f` and positional); only the positional metavar (`GRAPH-CONFIGURATION-FILE`) and the Python / bound-argument name change. -- **`validate` no longer auto-detects the config kind.** The YAML-sniffing heuristic (a top-level `tables` key ⇒ graph, otherwise table) is removed; `validate` now requires an explicit `--schema {graph,table}`/`-s` flag selecting which schema to validate against. `tablassert validate foo.yaml` now fails without `--schema`; use `--schema graph` (validates the `Graph` model and every referenced table) or `--schema table` (validates section syntax only). +- **`validate` no longer auto-detects the config kind.** The YAML-sniffing heuristic (a top-level `tables` key means graph, otherwise table) is removed; `validate` now requires an explicit `--schema {graph,table}`/`-s` flag selecting which schema to validate against. `tablassert validate foo.yaml` now fails without `--schema`; use `--schema graph` (validates the `Graph` model and every referenced table) or `--schema table` (validates section syntax only). ### Added - **The autonomous `tablassert agent` pipeline grew from one-section-per-paper to a richer, still-deterministic supervisor:** multi-section configs, local (non-open-access) payloads, first-class GEPA prompt optimization, and opt-in reflexion/judge gates. - - **Multi-section configs:** a derived config is now `{template, sections}` — `template` carries only shared provenance (`repo` + `publication`, no `source`); each entry in `sections` owns its own `source` (`local` path and `source.url`, plus `sheet`/`row_slice`/`delimiter`) and `statement`, so one paper maps every supplementary table/worksheet. `validate_table_config` (the final-answer gate) validates **every** section; `map_coverage` measures each section and reports an aggregate (`overall` mean, `min` weakest, `measured`) plus a per-section breakdown; `propose_config_edit` edits each section from its own coverage entry. + - **Multi-section configs:** a derived config is now `{template, sections}`: `template` carries only shared provenance (`repo` + `publication`, no `source`); each entry in `sections` owns its own `source` (`local` path and `source.url`, plus `sheet`/`row_slice`/`delimiter`) and `statement`, so one paper maps every supplementary table/worksheet. `validate_table_config` (the final-answer gate) validates **every** section; `map_coverage` measures each section and reports an aggregate (`overall` mean, `min` weakest, `measured`) plus a per-section breakdown; `propose_config_edit` edits each section from its own coverage entry. - **Local payloads (`--local`/`-l`):** runs the same derive/build/improve pipeline on an article held locally (e.g. a non-open-access paper) and skips the PMC-AWS fetch. Accepts one `DIR` applied to every id or per-article `PMCid=DIR` mappings; a missing directory fails loud (exit 2). - **GEPA prompt optimization (`--optimize`/`-o`):** a first-class path that runs `dspy.GEPA` with a real reflection LM and **persists** the optimized instructions (`--instructions-out`, default `/optimized_instructions.yaml`; reload via `--instructions-file`) instead of running the supervisor. Budget via `--max-metric-calls` (default `8`); example datasets via `--dataset` (YAML/JSON `{table_summary, coverage_feedback}`). - **Reflexion improver (`--reflexion`):** an opt-in tier-2 LLM reflexion improver for edits that may change predicate/source when the deterministic proposer stalls. - **Semantic judge (`--judge-model`/`--judge-threshold`):** an opt-in judge scores the output; when set, `MAPPED` additionally requires the normalized score to clear `--judge-threshold` (default `0.5`). Without it, the coverage gate alone decides. - **PDF context:** `pdfminer.six` (new `[agent]` dependency) extracts a `.pdf` main text into data-fenced context, so PDF-only articles still give the agent main-text context. - - **`BUILT_UNMEASURED`:** a new terminal **non-failure** outcome for a config that builds but whose fullmap coverage can't be measured (unreproducible source frame) — neither `MAPPED` nor `SKIPPED`; the best config is still written and reusable. + - **`BUILT_UNMEASURED`:** a new terminal **non-failure** outcome for a config that builds but whose fullmap coverage can't be measured (unreproducible source frame), neither `MAPPED` nor `SKIPPED`; the best config is still written and reusable. - **GEPA optimization now works end-to-end against real dspy and scores the genuine objective.** `gepa_metric` satisfies `dspy.GEPA`'s 5-arg metric contract (`gold, pred, trace, pred_name, pred_trace`) while keeping the legacy single-bundle call; when a dataset example carries a `fullmap` (+ optional `workdir`/`head`), the metric scores each proposed config with **real fullmap coverage** (a `build_and_audit` head-sample) instead of a validity-only proxy. New knobs: `--task-model` splits a fast task LM from the strong reflection LM, and `--gepa-threads` parallelizes LM forward passes (builds stay serialized on an internal build lock). `make_dspy_lm` adds reasoning-model-safe defaults (`max_tokens=16000`, `temperature=1.0`) and a 600 s request timeout so a truncated/stalled call can't hang the optimizer. -- **`derive_mode` (`full` | `derive_only` | `derive_coverage`) on `make_tools`/`run_supervisor`** for scalable config derivation — derive-and-validate, or derive-plus-coverage-measurement, without running the full loop. +- **`derive_mode` (`full` | `derive_only` | `derive_coverage`) on `make_tools`/`run_supervisor`** for scalable config derivation: derive-and-validate, or derive-plus-coverage-measurement, without running the full loop. - **Automated LLM-QC loop for derived agent configs** (example tooling under `examples/agent/qc/`): `qc_report.py` builds a deterministic per-PMC QC report (derived config + predicate/encodings/provenance + KG node/edge counts + sample edges + aggregate metrics), and `qc_reviewer.py` runs an LLM-as-judge that critiques each derived config + table + KG sample across `predicate_appropriateness` / `encoding_correctness` / `provenance` / `coverage` / `other_mistakes` and proposes one prompt improvement per PMC. Reports and reviews (`QC_REPORT.md`, `QC_REVIEW.md`) are regenerated together from the same state dir and each entry carries a shared config sha256 so report/review/config drift is detectable. Two rounds of this loop hardened the shipped derivation prompt (see Changed). -- **The test suite now runs parallel by default:** `pytest-xdist` joined the dev group and `-n auto` is in `addopts` (~7× faster; disable for one serial run with `pytest -n 0`). Coverage is identical. -- New regression tests: multi-section agent configs (`tests/test_agent_multisection.py`), example-QC guards (`tests/test_example_qc.py` — downloads-dir allowlist, column derivation, judge-JSON schema validation, path redaction, committed-artifact yaml-fence parseability + shared sha256), CLI parser locks proving the removed `build-kg --table-config`/`--fullmap` options stay gone and `validate` requires `--schema`, GEPA regression tests (5-arg contract, real-coverage metric, task-LM split, `num_threads`, `make_dspy_lm` defaults, no model built before `--gepa-threads` validation), and a public-build test proving `TABLASSERT_FULLMAP_SHARDS` is ignored end-to-end. +- **The test suite now runs parallel by default:** `pytest-xdist` joined the dev group and `-n auto` is in `addopts` (~7x faster; disable for one serial run with `pytest -n 0`). Coverage is identical. +- New regression tests: multi-section agent configs (`tests/test_agent_multisection.py`), example-QC guards (`tests/test_example_qc.py`, downloads-dir allowlist, column derivation, judge-JSON schema validation, path redaction, committed-artifact yaml-fence parseability + shared sha256), CLI parser locks proving the removed `build-kg --table-config`/`--fullmap` options stay gone and `validate` requires `--schema`, GEPA regression tests (5-arg contract, real-coverage metric, task-LM split, `num_threads`, `make_dspy_lm` defaults, no model built before `--gepa-threads` validation), and a public-build test proving `TABLASSERT_FULLMAP_SHARDS` is ignored end-to-end. ### Changed -- **The agent's derivation prompt was hardened through the iterative LLM-QC loop.** Added a Quality-principles section addressing the recurring mistakes the reviewer surfaced (over-interpretation, hard-coded objects, dropped statistical annotations, wrong object columns, generic predicates): pick the most-specific valid biolink predicate (`gene_associated_with_condition`, `correlated_with`, `expressed_in`, `biomarker_for`, `has_sequence_variant`, `affects`, …), falling back to `associated_with`/`related_to` only when nothing specific fits; add `prioritize` only when confident, otherwise omit it (a wrong prioritize is worse than none); verify each table/worksheet via `read_table` before authoring a section; use exact case/space-sensitive worksheet names and only candidate-table absolute paths; parse `build_and_audit`/`map_coverage` JSON output with `yaml.safe_load`, never `import json`. QC assay: 10/10 diverse PMCs `MAPPED` first-attempt (mean best coverage 0.974, 8/10 specific predicates); the two weakest PMCs improved poor→acceptable in round 2. +- **The agent's derivation prompt was hardened through the iterative LLM-QC loop.** Added a Quality-principles section addressing the recurring mistakes the reviewer surfaced (over-interpretation, hard-coded objects, dropped statistical annotations, wrong object columns, generic predicates): pick the most-specific valid biolink predicate (`gene_associated_with_condition`, `correlated_with`, `expressed_in`, `biomarker_for`, `has_sequence_variant`, `affects`, ...), falling back to `associated_with`/`related_to` only when nothing specific fits; add `prioritize` only when confident, otherwise omit it (a wrong prioritize is worse than none); verify each table/worksheet via `read_table` before authoring a section; use exact case/space-sensitive worksheet names and only candidate-table absolute paths; parse `build_and_audit`/`map_coverage` JSON output with `yaml.safe_load`, never `import json`. QC assay: 10/10 diverse PMCs `MAPPED` first-attempt (mean best coverage 0.974, 8/10 specific predicates); the two weakest PMCs improved from poor to acceptable in round 2. - **Pre-commit hooks trimmed to the fast, auto-fixing ones** (`ruff --fix`, `ruff-format`, `cargo-fmt`). The slow gates that used to run on every commit (pyright, the full pytest suite with a Rust-extension rebuild, cargo-clippy, cargo-test) are owned by CI on every PR; `pre-commit run --all-files` drops from ~2.5+ min to ~0.3 s. -- **QC reviewer/report hardening:** the judge's JSON is schema-validated (dimension mappings, score 0–3, quality enum) before rendering; failed entries carry the config sha256; the table summary derives `max_cols` from the configured subject/object/annotation columns and accumulates all sections; every review string is sanitized via `redact_paths` (generalized to any absolute local path, URL/ratio-safe) before JSON+markdown; `--rerender` rebuilds markdown from `qc_review.json` without re-querying the judge; the config fence cap rose 3000 → 8000 so no YAML block truncates mid-mapping; the reviewer runs with an untrusted-data system message, a downloads-dir allowlist for `source.local`, and a bounded litellm timeout. +- **QC reviewer/report hardening:** the judge's JSON is schema-validated (dimension mappings, score 0-3, quality enum) before rendering; failed entries carry the config sha256; the table summary derives `max_cols` from the configured subject/object/annotation columns and accumulates all sections; every review string is sanitized via `redact_paths` (generalized to any absolute local path, URL/ratio-safe) before JSON+markdown; `--rerender` rebuilds markdown from `qc_review.json` without re-querying the judge; the config fence cap rose 3000 to 8000 so no YAML block truncates mid-mapping; the reviewer runs with an untrusted-data system message, a downloads-dir allowlist for `source.local`, and a bounded litellm timeout. - CI now caches uv dependencies on every `setup-uv` step, runs the test step with explicit `-n auto`, and publishes docs with `peaceiris/actions-gh-pages` v4 (Node 20; v3 ran on the deprecated Node 16 runtime). ### Fixed - **Corrected the BioBERT Hugging Face repo id.** `pritamdeka/BioBERT-mnli-snli-scitail-mednli-stsb` no longer exists (HF API 401); the real sentence-transformers repo is `pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb`. With the old id, `get_biobert()`'s download path could never succeed for a fresh `.tablassert/biobert/` cache. -- **Agent KG builds are now reliable against real filesystems.** `build_and_audit` resolves its workdir to an absolute path (a relative workdir made `build_pipeline`'s `.tablassert/store` parquets resolve against the wrong base once the build chdir'd → build failure → false 0.0 coverage → every article `SKIPPED`); the supervisor presents absolute table paths so `source.local` resolves in the build workdir; a relative `source.local` is resolved against the build workdir as well as the cwd before a config is declared unmeasurable; the local executor timeout rose 30 s → 600 s so a large-table `build_and_audit` is not killed mid-build (which also stranded the fullmap redb lock); `build_and_audit` retries its coverage measurement on transient failure; fullmap lookups retry on transient redb lock contention (`Database already open`) without re-retrying exhausted contention. +- **Agent KG builds are now reliable against real filesystems.** `build_and_audit` resolves its workdir to an absolute path (a relative workdir made `build_pipeline`'s `.tablassert/store` parquets resolve against the wrong base once the build chdir'd: build failure, then false 0.0 coverage, then every article `SKIPPED`); the supervisor presents absolute table paths so `source.local` resolves in the build workdir; a relative `source.local` is resolved against the build workdir as well as the cwd before a config is declared unmeasurable; the local executor timeout rose 30 s to 600 s so a large-table `build_and_audit` is not killed mid-build (which also stranded the fullmap redb lock); `build_and_audit` retries its coverage measurement on transient failure; fullmap lookups retry on transient redb lock contention (`Database already open`) without re-retrying exhausted contention. - `agent --gepa-threads < 1` now fails loud (exit 2) **before** any model is built, and `--instructions-out` is resolved to an absolute path (GEPA's parallel builds chdir the process cwd). ### Performance -- **The test suite is ~7× faster:** 622 passed / 27 skipped at 90% coverage in ~22 s with `-n auto`, down from ~153 s serial (identical coverage; serial is one flag away via `pytest -n 0`). CI test runs reuse the uv package cache across runs. +- **The test suite is ~7x faster:** 622 passed / 27 skipped at 90% coverage in ~22 s with `-n auto`, down from ~153 s serial (identical coverage; serial is one flag away via `pytest -n 0`). CI test runs reuse the uv package cache across runs. ### Documentation - `docs/cli.md`: completed the SSOT `agent` flag table with all new flags (enforced by the `tests/test_docs_cli_coverage.py` guardrails) and replaced the shell-unsafe `--schema {graph,table}` brace expansion in the `validate` usage block with concrete commands. @@ -672,50 +672,50 @@ A major bump for one removal: `method: list` is gone, and a table config that st ## 8.0.1 - 2026-07-30 ### Breaking Changes -- **Slimmed the `tablassert agent` CLI surface.** Removed the `--qc-threshold`/`-qt` flag (it was echoed into the run metrics but never gated any decision — the accept/`MAPPED` outcome is driven solely by `--map-threshold` coverage), the `--executor`/`-e` flag and its Docker sandbox option (model-written code now always runs on the in-process `local` executor), and the `--no-fetch`/`-nf` flag (the supervisor always fetches the PMC payload). The `--map-threshold` default was lowered from `0.8` to `0.25`, and the agent workspace default moved from the standalone `.tablassert-agent/` directory to `.tablassert/agent/` (under the shared, already-git-ignored `.tablassert/` parent). -- Renamed CLI commands to a consistent verb-noun scheme and retired the legacy `datassert` naming throughout: `build` → `build-kg` (`validate` and `build-fullmap` are unchanged — they already fit the scheme). The `Graph` config field `datassert` is renamed to `fullmap` (also renaming the `resolve_many()` keyword argument from `datassert` to `fullmap`); existing graph YAML configs must rename their `datassert:` key to `fullmap:`. This is a pure naming migration. `build-fullmap`'s default `--output`/`--cache` directories moved from `./datassert/` to `./fullmap/`. (Note: the embedded fullmap database format was independently changed — see the redb schema item below — so existing `fullmap.redb` files **do** need to be rebuilt with `tablassert build-fullmap`.) +- **Slimmed the `tablassert agent` CLI surface.** Removed the `--qc-threshold`/`-qt` flag (it was echoed into the run metrics but never gated any decision, the accept/`MAPPED` outcome is driven solely by `--map-threshold` coverage), the `--executor`/`-e` flag and its Docker sandbox option (model-written code now always runs on the in-process `local` executor), and the `--no-fetch`/`-nf` flag (the supervisor always fetches the PMC payload). The `--map-threshold` default was lowered from `0.8` to `0.25`, and the agent workspace default moved from the standalone `.tablassert-agent/` directory to `.tablassert/agent/` (under the shared, already-git-ignored `.tablassert/` parent). +- Renamed CLI commands to a consistent verb-noun scheme and retired the legacy `datassert` naming throughout: `build` to `build-kg` (`validate` and `build-fullmap` are unchanged; they already fit the scheme). The `Graph` config field `datassert` is renamed to `fullmap` (also renaming the `resolve_many()` keyword argument from `datassert` to `fullmap`); existing graph YAML configs must rename their `datassert:` key to `fullmap:`. This is a pure naming migration. `build-fullmap`'s default `--output`/`--cache` directories moved from `./datassert/` to `./fullmap/`. (Note: the embedded fullmap database format was independently changed; see the redb schema item below, so existing `fullmap.redb` files **do** need to be rebuilt with `tablassert build-fullmap`.) - Removed the `syntax` key entirely from both configuration formats. Graph configs no longer accept `syntax: GC2`/`GC3` and table configs no longer accept `syntax: TC4`; any `syntax:` field is now rejected (`extra = "forbid"`). - Removed the `status` template-metadata field from table configurations. - Removed the `provenance.contributors` block from table configurations. `Provenance` now carries only `repo`, `publication`, `knowledge_level`, and `agent_type`; curation-style attribution moves to the graph-level RIG `contributions` field (below). - Removed `qc` and `log` from the graph configuration. Both are now `build-kg` CLI flags (`--qc/-q` and `--log/-l`); graph configs that set `qc:` or `log:` are rejected. - Removed `pubmed_db` and `pmc_db` from graph configuration and removed the downstream PubMed metadata and PMC caption enrichment steps from graph builds. -- **Entity resolution was rewritten onto an embedded [redb](https://github.com/cberner/redb) database.** The DuckDB-shard store built by the external `datassert` Go CLI is replaced by an embedded redb database — a primary file (`fullmap.redb`) plus sibling RECORDS shard files (`fullmap.s*.redb`) that must remain co-located for lookup — built in-process by Tablassert's own Rust extension (`tablassert.rs`); no external tool or install step is required. The on-disk fullmap schema is now `tablassert.fullmap.v4` (a primary file plus hash-sharded RECORDS files — see the sharded-fullmap entry under Changed below); v1/v2/v3 databases are rejected at lookup time and must be rebuilt with `tablassert build-fullmap`. +- **Entity resolution was rewritten onto an embedded [redb](https://github.com/cberner/redb) database.** The DuckDB-shard store built by the external `datassert` Go CLI is replaced by an embedded redb database, a primary file (`fullmap.redb`) plus sibling RECORDS shard files (`fullmap.s*.redb`) that must remain co-located for lookup, built in-process by Tablassert's own Rust extension (`tablassert.rs`); no external tool or install step is required. The on-disk fullmap schema is now `tablassert.fullmap.v4` (a primary file plus hash-sharded RECORDS files, see the sharded-fullmap entry under Changed below); v1/v2/v3 databases are rejected at lookup time and must be rebuilt with `tablassert build-fullmap`. - **The QC runtime migrated from ONNX Runtime to PyTorch / sentence-transformers (BioBERT).** The `qc` extra now installs `torch`, `sentence-transformers`, `rapidfuzz`, `scikit-learn`, and `numpy`. The `qc-cuda` extra and all ONNX/CUDA provider selection were removed; `fullmap_audit()` no longer takes a `provider` argument and runs on the sentence-transformers backend. QC thresholds were tightened: the fuzzy stage now passes on `fuzz.ratio >= 70 OR partial_token_sort_ratio >= 80` (previously 20/30), and the BioBERT cosine-similarity stage on `>= 0.5` (previously 0.2). The cached model moved from `.tablassert/onnx/` to `.tablassert/biobert/`. -- **Removed automatic source-file downloading.** A table's `source.url` is no longer fetched into `source.local`; the input file must already exist at `source.local`. `source.url` is retained as provenance metadata — it is emitted as the edge `source_record_urls` column and recorded in the generated RIG. The parse-time URL-reachability check (`httpx.head`) and its `.cachassert/` disk cache were removed. +- **Removed automatic source-file downloading.** A table's `source.url` is no longer fetched into `source.local`; the input file must already exist at `source.local`. `source.url` is retained as provenance metadata: it is emitted as the edge `source_record_urls` column and recorded in the generated RIG. The parse-time URL-reachability check (`httpx.head`) and its `.cachassert/` disk cache were removed. - Consolidated the four scattered hidden working directories (`.storassert/`, `.logassert/`, `.onnxassert/`, and the previously-documented-but-removed `.cachassert/`) into a single parent `.tablassert/` with three intuitively-named subdirectories: `.tablassert/store/` (intermediate parquet, `utils.STORE`), `.tablassert/log/` (loguru sink, `log.LOGASSERT`), and `.tablassert/biobert/` (cached BioBERT model, `qc.MODEL`). All three paths are now derived from a single `utils.BASE = Path("./.tablassert")` constant. The loguru sink file was renamed from `logassert.log` to `tablassert.log`. Any external tooling or log tailers pointed at the old paths need to be updated; the on-disk content is auto-regenerated on the next run (parquet is reproducible, the BioBERT model re-downloads if `.tablassert/biobert/` is not pre-populated). - Removed runtime dependencies that are no longer used: `onnxruntime`, `diskcache`, `httpx`, `pyexcel`, `orjson`, `duckdb`, and `polars_hash`. -- **Knowledge-graph edge IDs are now canonicalized.** Edge IDs (assigned during Rust NDJSON deduplication/labeling) are derived from a key-order-independent, injective encoding: object keys are sorted before hashing and each key/value is fed as a separate length-prefixed part, so the same logical edge always yields the same UUID regardless of JSON field order (field order previously leaked into the ID) and distinct edges cannot collide on the encoding. This changes generated edge IDs relative to earlier builds — regenerate any stored/published graphs; downstream consumers that pin edge IDs must update. +- **Knowledge-graph edge IDs are now canonicalized.** Edge IDs (assigned during Rust NDJSON deduplication/labeling) are derived from a key-order-independent, injective encoding: object keys are sorted before hashing and each key/value is fed as a separate length-prefixed part, so the same logical edge always yields the same UUID regardless of JSON field order (field order previously leaked into the ID) and distinct edges cannot collide on the encoding. This changes generated edge IDs relative to earlier builds; regenerate any stored/published graphs; downstream consumers that pin edge IDs must update. ### Added -- **Rust/PyO3 extension (`tablassert.rs`)** replacing the external `datassert` tool. Public functions include `build_fullmap_db`, `lookup_fullmap_terms`, `hydrate_prefixes`/`hydrate_categories`/`hydrate_sources`/`hydrate_curies`, `dedup_ndjson`, `namespace_uuid`, and `fullmap_source_version`. Published as maturin wheels (CPython 3.11–3.14, Linux + macOS) through the PyPI workflow. +- **Rust/PyO3 extension (`tablassert.rs`)** replacing the external `datassert` tool. Public functions include `build_fullmap_db`, `lookup_fullmap_terms`, `hydrate_prefixes`/`hydrate_categories`/`hydrate_sources`/`hydrate_curies`, `dedup_ndjson`, `namespace_uuid`, and `fullmap_source_version`. Published as maturin wheels (CPython 3.11-3.14, Linux + macOS) through the PyPI workflow. - **`tablassert build-fullmap`** now downloads NCATS Translator BABEL class/synonym files from RENCI (resumable, range-request downloads) and builds the embedded fullmap redb database in-process, with a live progress display. - **Resource Ingest Guide (RIG) generation.** `build-kg` now also emits `_.RIG.yaml` alongside the nodes/edges NDJSON. A new required graph field `description` and optional `contributions` and `ui_explanation` fields populate the RIG (with sensible defaults for `contributions`/`ui_explanation`). - **`build-kg` flags:** `--release/-r` (emit a slim, significant-only graph by dropping `biolink:not_significant` edges before resolution), `--qc/-q` (run the QC audit stage), and `--log/-l` (verbose per-section logging). -- **`build-fullmap` shorthand flags:** every `build-fullmap` option now has a short alias — `--output/-o`, `--cache/-c`, `--version/-v`, and `--threads/-t` — matching the shorthand style already used by `build-kg`. +- **`build-fullmap` shorthand flags:** every `build-fullmap` option now has a short alias: `--output/-o`, `--cache/-c`, `--version/-v`, and `--threads/-t`, matching the shorthand style already used by `build-kg`. - **`build-kg` configuration-file flag:** `build-kg`'s configuration file now also accepts `--configuration-file`/`-f` (positional usage is unchanged), matching `validate`. To free `-f` for the configuration file, `build-kg`'s `--fullmap` short alias moved from `-f` to `-fm` (the `--fullmap` long form is unchanged). - **Build-time CURIE prefix exclusion:** setting `TABLASSERT_FULLMAP_EXCLUDE_PREFIXES` (comma-separated prefixes, e.g. `INCHIKEY,Publication`) drops matching CURIE rows during the fullmap build. Excluding prefixes you never resolve can cut build time, peak memory, and database size several-fold. The build also skips terms the lookup path can never query (the `distinct()` dead-term filter: purely numeric or generic labels), and exposes memory/speed tunables (`TABLASSERT_FULLMAP_{LOCAL_SPILL,CURIE_SPILL,EQUIV_SPILL}_ENTRIES`, `_INSERT_BATCH`, `_REDB_CACHE_BYTES`, `_SPILL_DIR`); see `docs/fullmap.md`. -- **`TABLASSERT_FULLMAP_SHARDS`** environment variable (default `16`): the number of on-disk RECORDS shard files the fullmap term index is hash-partitioned across (and the number of concurrent redb writers in the write phase). Non-powers-of-two round down to the nearest power of two (6→4, 3→2) and the value is clamped to the compile-time cap (16). The resolved shard count is recorded in the primary file's `meta` table (`shards`) and the read path opens exactly that many shard files; see `docs/fullmap.md`. +- **`TABLASSERT_FULLMAP_SHARDS`** environment variable (default `16`): the number of on-disk RECORDS shard files the fullmap term index is hash-partitioned across (and the number of concurrent redb writers in the write phase). Non-powers-of-two round down to the nearest power of two (6 to 4, 3 to 2) and the value is clamped to the compile-time cap (16). The resolved shard count is recorded in the primary file's `meta` table (`shards`) and the read path opens exactly that many shard files; see `docs/fullmap.md`. - **Structured, coded errors** (`tablassert.errors`): `TablassertError`, `TablassertValidationError`, `GraphValidationError`, `SectionValidationError`, `QcRuntimeMissingError`, and `BabelDownloadError`, each carrying a stable error code that resolves to a documentation URL. - Experimental Biolink feature support: per-section knowledge-level / agent-type provenance (`provenance.knowledge_level`, `provenance.agent_type`), auto-derived Biolink association edge categories (`EdgeCategories`), and a Biolink-compliant edge-column allow-list. Annotation columns not on the allow-list are folded into the edge `supporting_text` list (as `"name: value"` entries) rather than dropped. - New edge provenance columns: `upstream_resource_ids` (from the Biolink information-resource mapping) and `source_record_urls` (list column). - Google-style docstrings on all functions; `pytest-cov` coverage configuration; and an expanded end-to-end build-pipeline test suite. ### Changed -- **The fullmap database is now hash-sharded on disk.** The single redb file is split into a small primary file (`fullmap.redb` — dimensions, `curies`, and `meta`) plus 16 RECORDS shard files by default (`fullmap.s0.redb` … `fullmap.s15.redb`), with each normalized term routed to a shard by `xxh64(term) & 15`. The on-disk schema is now `tablassert.fullmap.v4`; existing `v3` databases are rejected at lookup time and must be rebuilt with `tablassert build-fullmap`. Two phases were parallelized as a result: the RECORDS write now runs `shard_count` independent k-way merges in parallel — the term spill runs are partitioned per-shard at write time (each `run_s{shard}_{id}.bin` holds only terms hashing to that shard), so one thread per shard merges only its own runs and inserts the merged groups inline into that shard's redb file (one database per shard — redb is single-writer per file) with no shared producer or per-shard channel, parallelizing both the merge and the B-tree insert (~N× write throughput); and fullmap lookup now releases the GIL and fans the query terms out across the shards in parallel (one reader per non-empty shard, re-merged into input order). The shard files are discovered as siblings of the primary path and must remain alongside it; see `docs/fullmap.md`. -- The fullmap build is now parallelized with rayon: class/synonym files are processed in parallel through an in-memory sharded pipeline (equivalents map → dimension/CURIE-id assignment → term aggregation → a single write transaction), replacing the old per-row database lookups and the stage-database/temp-file copy. `--threads` now defaults to ~90% of available CPUs when unset (previously effectively single-threaded). The redb stores six tables: `records`, `prefixes`, `categories`, `sources`, `curies`, and `meta`. -- **The fullmap build is now memory-bounded.** The two structures that previously grew with the number of unique CURIEs and drove the synonym phase into swap/OOM on a full BABEL build are no longer held in RAM: the CURIE dedup map is keyed by a 128-bit hash (`xxh3_128(curie) → id`) instead of the CURIE string, and the per-CURIE rows are spilled to bounded on-disk run files (streamed into the `curies` table at write time) instead of a global in-memory vector. Peak RSS now stays flat regardless of input size (measured synonym-phase RSS −45% on an 8 M-CURIE subset, scaling to ~70 GB avoided at full build), and the build is ~18–25% faster (the in-memory CURIE-row sort was removed and the redb write batch enlarged). This was a build-time-only change that did not itself bump the on-disk schema (the schema has since moved to `tablassert.fullmap.v4`; see the sharded-fullmap entry below). -- **The fullmap synonym phase now uses intra-file parallelism.** Previously each synonym file was processed by a single thread, so the few very large BABEL files (protein/smallmolecule/gene/drugchemicalconflated, ~290 GB of the ~300 GB input) ran single-threaded while other cores idled. The phase is now a producer–consumer pipeline: a small pool of producer threads decompresses/reads the files into byte-bounded line-chunks fed through a bounded channel, and all worker threads pull chunks and process rows in parallel — so a giant file is processed by every worker. This is intra-file parallelism only (at the time the output was still a single redb file; the sharded-fullmap entry below later split the RECORDS table across shard files). Measured ~19–25% faster builds on 1–6 GB subsets with identical output (table counts and lookup-equivalence unchanged). The crate also now uses the mimalloc global allocator, which keeps resident memory flat under the heavy multi-threaded allocation (glibc malloc's per-thread arenas otherwise retain freed memory and inflate peak RSS several-fold). New tunables: `TABLASSERT_FULLMAP_CHUNK_BYTES` (default 8 MiB) and `TABLASSERT_FULLMAP_PRODUCERS`. +- **The fullmap database is now hash-sharded on disk.** The single redb file is split into a small primary file (`fullmap.redb`, dimensions, `curies`, and `meta`) plus 16 RECORDS shard files by default (`fullmap.s0.redb` ... `fullmap.s15.redb`), with each normalized term routed to a shard by `xxh64(term) & 15`. The on-disk schema is now `tablassert.fullmap.v4`; existing `v3` databases are rejected at lookup time and must be rebuilt with `tablassert build-fullmap`. Two phases were parallelized as a result: the RECORDS write now runs `shard_count` independent k-way merges in parallel: the term spill runs are partitioned per-shard at write time (each `run_s{shard}_{id}.bin` holds only terms hashing to that shard), so one thread per shard merges only its own runs and inserts the merged groups inline into that shard's redb file (one database per shard, redb is single-writer per file) with no shared producer or per-shard channel, parallelizing both the merge and the B-tree insert (~Nx write throughput); and fullmap lookup now releases the GIL and fans the query terms out across the shards in parallel (one reader per non-empty shard, re-merged into input order). The shard files are discovered as siblings of the primary path and must remain alongside it; see `docs/fullmap.md`. +- The fullmap build is now parallelized with rayon: class/synonym files are processed in parallel through an in-memory sharded pipeline (equivalents map, then dimension/CURIE-id assignment, then term aggregation, then a single write transaction), replacing the old per-row database lookups and the stage-database/temp-file copy. `--threads` now defaults to ~90% of available CPUs when unset (previously effectively single-threaded). The redb stores six tables: `records`, `prefixes`, `categories`, `sources`, `curies`, and `meta`. +- **The fullmap build is now memory-bounded.** The two structures that previously grew with the number of unique CURIEs and drove the synonym phase into swap/OOM on a full BABEL build are no longer held in RAM: the CURIE dedup map is keyed by a 128-bit hash (`xxh3_128(curie) to id`) instead of the CURIE string, and the per-CURIE rows are spilled to bounded on-disk run files (streamed into the `curies` table at write time) instead of a global in-memory vector. Peak RSS now stays flat regardless of input size (measured synonym-phase RSS -45% on an 8 M-CURIE subset, scaling to ~70 GB avoided at full build), and the build is ~18-25% faster (the in-memory CURIE-row sort was removed and the redb write batch enlarged). This was a build-time-only change that did not itself bump the on-disk schema (the schema has since moved to `tablassert.fullmap.v4`; see the sharded-fullmap entry below). +- **The fullmap synonym phase now uses intra-file parallelism.** Previously each synonym file was processed by a single thread, so the few very large BABEL files (protein/smallmolecule/gene/drugchemicalconflated, ~290 GB of the ~300 GB input) ran single-threaded while other cores idled. The phase is now a producer-consumer pipeline: a small pool of producer threads decompresses/reads the files into byte-bounded line-chunks fed through a bounded channel, and all worker threads pull chunks and process rows in parallel, so a giant file is processed by every worker. This is intra-file parallelism only (at the time the output was still a single redb file; the sharded-fullmap entry below later split the RECORDS table across shard files). Measured ~19-25% faster builds on 1-6 GB subsets with identical output (table counts and lookup-equivalence unchanged). The crate also now uses the mimalloc global allocator, which keeps resident memory flat under the heavy multi-threaded allocation (glibc malloc's per-thread arenas otherwise retain freed memory and inflate peak RSS several-fold). New tunables: `TABLASSERT_FULLMAP_CHUNK_BYTES` (default 8 MiB) and `TABLASSERT_FULLMAP_PRODUCERS`. - TCode operation ordering was optimized to accelerate graph build times. - The CLI now uses a Rich-based progress interface across all pipeline stages. - **Section content hashes widened from XXH32 to XXH64.** `utils.mkhash` now calls the extension's new `xxh64` primitive (16 hex chars) instead of `xxh32` (8) for the content-addressed section parquet store (`.tablassert/store/`) and section labels, removing the 32-bit birthday-collision risk at high section counts. User-facing labels still truncate to 8 chars. The first build after upgrading rebuilds the section cache; orphaned 8-char store files are safe to delete. ### Performance -- **The Phase-4 RECORDS write now runs per-shard independent k-way merges in parallel.** The term spill runs are partitioned per-shard *at write time* (one `run_s{shard}_{id}.bin` per non-empty shard, each holding only terms hashing to that shard), so the write phase runs `shard_count` independent k-way merges concurrently — one thread per shard, merging and inserting inline — instead of the prior design's single k-way-merge producer routing merged groups through bounded per-shard channels. This eliminates the single-threaded merge producer that previously capped the build: both the merge and the B-tree insert now scale across shards. Measured Phase-4 write wall on a slow disk (24 worker threads, mean of two runs): the medium subset dropped to ~`8.9s` at `TABLASSERT_FULLMAP_SHARDS=4` from ~`55.9s` at `=1` (~6.3×), and the stress subset to ~`20.0s` from ~`112.6s` (~5.6×). The `=1` single-merge control reproduces the old single-producer cap (~62s medium / ~104s stress), and the `SHARDS=4` wall is ~7× (medium) / ~5× (stress) below that baseline. No schema change (still `tablassert.fullmap.v4`) and no on-disk layout change (primary file plus `fullmap.s0..s.redb`); databases built by the prior sharded design remain compatible. +- **The Phase-4 RECORDS write now runs per-shard independent k-way merges in parallel.** The term spill runs are partitioned per-shard *at write time* (one `run_s{shard}_{id}.bin` per non-empty shard, each holding only terms hashing to that shard), so the write phase runs `shard_count` independent k-way merges concurrently, one thread per shard, merging and inserting inline, instead of the prior design's single k-way-merge producer routing merged groups through bounded per-shard channels. This eliminates the single-threaded merge producer that previously capped the build: both the merge and the B-tree insert now scale across shards. Measured Phase-4 write wall on a slow disk (24 worker threads, mean of two runs): the medium subset dropped to ~`8.9s` at `TABLASSERT_FULLMAP_SHARDS=4` from ~`55.9s` at `=1` (~6.3x), and the stress subset to ~`20.0s` from ~`112.6s` (~5.6x). The `=1` single-merge control reproduces the old single-producer cap (~62s medium / ~104s stress), and the `SHARDS=4` wall is ~7x (medium) / ~5x (stress) below that baseline. No schema change (still `tablassert.fullmap.v4`) and no on-disk layout change (primary file plus `fullmap.s0..s.redb`); databases built by the prior sharded design remain compatible. ### Fixed - **Config YAML is parsed with `yaml.CSafeLoader` (safe construction).** Config loading previously used `yaml.CLoader`, which honors arbitrary construction tags (e.g. `!!python/object/apply`) and could execute code from an untrusted table/graph config; it now uses the safe loader with no speed loss (still libyaml-backed). - `build-kg` now opens Polars NDJSON output streams with explicit UTF-8 encoding, avoiding `InvalidOperationError: file encoding is not UTF-8` during graph compilation on non-UTF-8 locales. -- **Base installs no longer crash at the `build-kg` clean phase with `ModuleNotFoundError: No module named 'rapidfuzz'`.** `rapidfuzz` was previously declared only in the optional `qc` extra, but it is imported by three operations that run on every build regardless of `--qc` — `coerce_pvalue_columns` and `coerce_study_size_columns` (clean phase) and `sig` (significance phase). `rapidfuzz` is now a core dependency. The `qc` extra now contains only `scikit-learn` and `sentence-transformers` (`torch`/`numpy` arrive transitively); installs that already use `tablassert[qc]` are unaffected. +- **Base installs no longer crash at the `build-kg` clean phase with `ModuleNotFoundError: No module named 'rapidfuzz'`.** `rapidfuzz` was previously declared only in the optional `qc` extra, but it is imported by three operations that run on every build regardless of `--qc`, `coerce_pvalue_columns` and `coerce_study_size_columns` (clean phase) and `sig` (significance phase). `rapidfuzz` is now a core dependency. The `qc` extra now contains only `scikit-learn` and `sentence-transformers` (`torch`/`numpy` arrive transitively); installs that already use `tablassert[qc]` are unaffected. ### Removed - The external `datassert` Go CLI dependency and the sharded DuckDB entity-resolution store. @@ -725,7 +725,7 @@ A major bump for one removal: `method: list` is gone, and a table config that st - The `.onnxassert/` and `.cachassert/` working directories. ### Documentation -- Comprehensive 8.0.0 documentation overhaul reconciling every surface (README, `llms.txt`, CONTRIBUTING, CITATION, and the full MkDocs site) against the codebase: removed the retired `syntax`/`status`/`contributors` config keys and the ONNX/CUDA QC documentation, documented the redb v4 schema and the new `build-kg` flags and RIG output, and corrected every shipped YAML example so it validates against the current schema. +- 8.0.0 documentation overhaul reconciling every surface (README, `llms.txt`, CONTRIBUTING, CITATION, and the full MkDocs site) against the codebase: removed the retired `syntax`/`status`/`contributors` config keys and the ONNX/CUDA QC documentation, documented the redb v4 schema and the new `build-kg` flags and RIG output, and corrected every shipped YAML example so it validates against the current schema. - Upgraded the documentation site from the plain `readthedocs` theme to Material for MkDocs. - Removed the Docker documentation (the project no longer publishes a container image). - Added a CI test that validates the shipped example configurations against the Pydantic models and guards against reintroducing removed 8.0.0 config keys. @@ -733,7 +733,7 @@ A major bump for one removal: `method: list` is gone, and a table config that st ## 7.5.2 - 2026-07-01 ### Changed -- `sig()` in `lib.py` now emits a third significance label, `"INCONCLUSIVE"`, for p-values that fall between the existing significance `cutoff` (default `0.05`, inclusive) and a new `threshold` parameter (default `0.10`, exclusive). A p-value `p` is now mapped as: null → `"UNSURE"`; `p <= cutoff` → `"YES"`; `cutoff < p < threshold` → `"INCONCLUSIVE"`; `p >= threshold` → `"NO"`. The upper bound is exclusive so `0.10` (and the `0.1` `NO` cases in the existing tests) continue to map to `"NO"`. `threshold` is added to `sig()`'s signature alongside `cutoff`; the function remains wired into `Tcode.collect()` at its default arguments, so builds are unaffected unless a caller overrides the new bound. +- `sig()` in `lib.py` now emits a third significance label, `"INCONCLUSIVE"`, for p-values that fall between the existing significance `cutoff` (default `0.05`, inclusive) and a new `threshold` parameter (default `0.10`, exclusive). A p-value `p` is now mapped as: null to `"UNSURE"`; `p <= cutoff` to `"YES"`; `cutoff < p < threshold` to `"INCONCLUSIVE"`; `p >= threshold` to `"NO"`. The upper bound is exclusive so `0.10` (and the `0.1` `NO` cases in the existing tests) continue to map to `"NO"`. `threshold` is added to `sig()`'s signature alongside `cutoff`; the function remains wired into `Tcode.collect()` at its default arguments, so builds are unaffected unless a caller overrides the new bound. ### Added - One regression test in `test_lib.py` (`test_sig_marks_inconclusive_band`) asserting all four bands in a single frame: a value at/below cutoff (`YES`), a value in the inconclusive range (`INCONCLUSIVE`), and a value at the exclusive threshold (`NO`). @@ -741,9 +741,9 @@ A major bump for one removal: `method: list` is gone, and a table config that st ## 7.5.1 - 2026-07-01 ### Changed -- Numeric annotation columns are now coerced and emitted as controlled-notation strings in NDJSON output instead of raw values. Two new pipeline steps wired into `Tcode.collect()` (`lib.py`): `clean_numeric()` lazily casts matching columns to `Float64` with `strict=False` (non-numeric entries drop to null), and `format_numeric()` renders them as strings — p-value columns (any name containing `"p value"`, case-insensitive) in scientific notation (`{:.4e}`), and `relationship strength` / `sample size` in decimal general format (`{:.4g}`, ≥4 significant figures). Non-matching columns are left untouched, and nulls are subsequently dropped by `strip_nulls()`. `math_op()` now also casts with `strict=False` so it tolerates residual junk in numeric annotation columns. `format_numeric()` formats via numpy-backed batch conversion rather than `map_elements` for throughput. +- Numeric annotation columns are now coerced and emitted as controlled-notation strings in NDJSON output instead of raw values. Two new pipeline steps wired into `Tcode.collect()` (`lib.py`): `clean_numeric()` lazily casts matching columns to `Float64` with `strict=False` (non-numeric entries drop to null), and `format_numeric()` renders them as strings: p-value columns (any name containing `"p value"`, case-insensitive) in scientific notation (`{:.4e}`), and `relationship strength` / `sample size` in decimal general format (`{:.4g}`, >=4 significant figures). Non-matching columns are left untouched, and nulls are subsequently dropped by `strip_nulls()`. `math_op()` now also casts with `strict=False` so it tolerates residual junk in numeric annotation columns. `format_numeric()` formats via numpy-backed batch conversion rather than `map_elements` for throughput. - Removed dead `pl.Config(set_fmt_float=...)` and `pl.Config(float_precision=...)` context managers from `compile_graph()` (`lib.py`); they were no-ops for NDJSON serialization (`write_ndjson` emits raw f64 via serde shortest-repr and ignores float display options), and the `fmt`/`precision` parameters of `compile_graph()` were removed alongside them. -- Fixed an off-by-one in the build/validate progress bar so each section loop now shows the configuration currently being processed instead of the last-completed one. `PipelineProgress.section_loop()` (`progress.py`) previously returned a single `advance(info)` callback that set the description and ticked the completed counter together, called after each item's work — so while section *K* ran the bar still displayed section *K−1*. It now returns a `(start, advance)` pair: `start(info)` updates the description to the in-flight item without incrementing, and `advance()` ticks the counter afterwards (so the counter never claims an in-flight item is complete). All five call sites in `cli.py` (TCode build, Collect, Subgraph, Graph, Validate) were updated to `start(...)` before the work and `advance()` after; the long-running Collect and Subgraph stages continue to show the full `format_section_oneline()` summary (including the `CONFIG` name) of the in-flight section. +- Fixed an off-by-one in the build/validate progress bar so each section loop now shows the configuration currently being processed instead of the last-completed one. `PipelineProgress.section_loop()` (`progress.py`) previously returned a single `advance(info)` callback that set the description and ticked the completed counter together, called after each item's work; so while section *K* ran the bar still displayed section *K-1*. It now returns a `(start, advance)` pair: `start(info)` updates the description to the in-flight item without incrementing, and `advance()` ticks the counter afterwards (so the counter never claims an in-flight item is complete). All five call sites in `cli.py` (TCode build, Collect, Subgraph, Graph, Validate) were updated to `start(...)` before the work and `advance()` after; the long-running Collect and Subgraph stages continue to show the full `format_section_oneline()` summary (including the `CONFIG` name) of the in-flight section. ### Added - Fifteen regression tests in `test_lib.py` covering `numeric_columns()` detection (p-value substring, exact-name match, case-insensitivity), `clean_numeric()` (parse/coerce numeric and scientific notation, null out non-numeric junk, leave non-matching columns untouched, noop, idempotent on Float64), `format_numeric()` (scientific notation for p-value, decimal general format for relationship strength/sample size, null preservation, floating-point-noise cleaning, noop), null-stripped NDJSON rows, `compile_graph()` NDJSON emission, and `sig()` operating over a cleaned Float64 p-value column. @@ -759,7 +759,7 @@ A major bump for one removal: `method: list` is gone, and a table config that st - Four regression tests in `test_lib.py`: `publication_curie()` for PMC and PMID namespaces, and two `Tcode` tcode-inspection tests covering presence/ordering of the table-literal column for column encodings and its absence for value encodings. ### Documentation -- Comprehensive accuracy pass across the API, configuration, and Docker documentation, reconciling every page against the current codebase. Highlights: corrected invalid examples that would not load (`syntax: TC2`; `publication` integers and missing `PMC` prefixes; a non-existent `Qualifiers` member; `reindex` placed at section level; a subject missing `method: column`), fixed wrong field types (`rows`/`row_slice`/`taxon` → `PositiveInt`, `remove` → regex patterns), corrected the QC fuzzy thresholds (`fuzz.ratio >= 20 OR partial_token_sort_ratio >= 30`), removed a non-existent `uuid:` prefix from `utils.md` return examples, fixed the `resolve_many()` parameter order and added the original-column-capture and optional QC-audit pipeline steps, corrected graph-config path resolution (CWD, not config-relative) and processing-flow ordering, documented the strict QC GPU no-fallback behavior and the `.cachassert/` working directory, and aligned `Categories` enum member names (`GENE`/`PROTEIN`) and Docker CI triggers with the source. +- Accuracy pass across the API, configuration, and Docker documentation, reconciling every page against the current codebase. Highlights: corrected invalid examples that would not load (`syntax: TC2`; `publication` integers and missing `PMC` prefixes; a non-existent `Qualifiers` member; `reindex` placed at section level; a subject missing `method: column`), fixed wrong field types (`rows`/`row_slice`/`taxon` to `PositiveInt`, `remove` to regex patterns), corrected the QC fuzzy thresholds (`fuzz.ratio >= 20 OR partial_token_sort_ratio >= 30`), removed a non-existent `uuid:` prefix from `utils.md` return examples, fixed the `resolve_many()` parameter order and added the original-column-capture and optional QC-audit pipeline steps, corrected graph-config path resolution (CWD, not config-relative) and processing-flow ordering, documented the strict QC GPU no-fallback behavior and the `.cachassert/` working directory, and aligned `Categories` enum member names (`GENE`/`PROTEIN`) and Docker CI triggers with the source. ## 7.4.14 - 2026-06-30 @@ -786,7 +786,7 @@ A major bump for one removal: `method: list` is gone, and a table config that st ## 7.4.9 - 2026-05-26 ### Bug Fixes -- Fixed `OSError: Too many open files` during subgraph build at large scales (700+ sections). `with_mesh()` and `with_captions()` in `lib.py` opened a `sqlite_utils.Database` per section but never closed the underlying SQLite connection, leaving FD release to GC. In the tight sequential `compile_subgraph` loop the leaked FDs accumulated past the OS soft limit, causing the next `to_store()` → `df.write_parquet()` (which polars 1.39 routes through `sink_parquet`) to fail opening its target `.storassert/*.parquet`. Both functions now wrap their query bodies in `try:` / `finally: db.conn.close()`. +- Fixed `OSError: Too many open files` during subgraph build at large scales (700+ sections). `with_mesh()` and `with_captions()` in `lib.py` opened a `sqlite_utils.Database` per section but never closed the underlying SQLite connection, leaving FD release to GC. In the tight sequential `compile_subgraph` loop the leaked FDs accumulated past the OS soft limit, causing the next `to_store()`, then `df.write_parquet()` (which polars 1.39 routes through `sink_parquet`) to fail opening its target `.storassert/*.parquet`. Both functions now wrap their query bodies in `try:` / `finally: db.conn.close()`. ## 7.4.8 - 2026-05-12 @@ -825,18 +825,18 @@ A major bump for one removal: `method: list` is gone, and a table config that st ## 7.4.2 - 2026-05-07 ### Changes -- Added Pydantic field and model validators to `models.py` that enforce configuration correctness at parse time: `url` fields are verified reachable via `httpx.head()`, `rows` and `row_slice` are mutually exclusive, `Reindex.comparator` type must match its `comparison` operator (`eq`/`ne` require `str`, numeric operators require `int`/`float`), `encoding` values under `method: column` must be Excel-style letters (`A`–`ZZZ`), `Regex` pattern/replacement strings are validated against the Polars regex engine, `remove` entries are validated as Polars-compatible regex, and `annotation` names have underscores replaced with spaces. +- Added Pydantic field and model validators to `models.py` that enforce configuration correctness at parse time: `url` fields are verified reachable via `httpx.head()`, `rows` and `row_slice` are mutually exclusive, `Reindex.comparator` type must match its `comparison` operator (`eq`/`ne` require `str`, numeric operators require `int`/`float`), `encoding` values under `method: column` must be Excel-style letters (`A`-`ZZZ`), `Regex` pattern/replacement strings are validated against the Polars regex engine, `remove` entries are validated as Polars-compatible regex, and `annotation` names have underscores replaced with spaces. - Changed `rows` and `row_slice` element type from `NonNegativeInt` to `PositiveInt` in `BaseSource`. ## 7.4.1 - 2026-05-05 ### Bug Fixes -- Fixed `AttributeError: 'str' object has no attribute 'value'` raised by `format_section_oneline()` in `progress.py` during the BUILDING TCODE stage. The `Section` model sets `use_enum_values=True`, so `Tcode.status` is already a plain string — removed the stale `.value` access. +- Fixed `AttributeError: 'str' object has no attribute 'value'` raised by `format_section_oneline()` in `progress.py` during the BUILDING TCODE stage. The `Section` model sets `use_enum_values=True`, so `Tcode.status` is already a plain string; removed the stale `.value` access. ## 7.4.0 - 2026-05-05 ### Changes -- Renamed CLI commands for brevity: `build-knowledge-graph` → `build`, `verify-table-configuration-syntax` → `validate`. Version display moved from `tablassert version` subcommand to `tablassert --version` flag. +- Renamed CLI commands for brevity: `build-knowledge-graph` to `build`, `verify-table-configuration-syntax` to `validate`. Version display moved from `tablassert version` subcommand to `tablassert --version` flag. - Added `qc` parameter to `resolve_many()` for optional QC auditing during standalone batch resolution. ONNX Runtime provider is auto-detected via `get_qc_provider()`. - Added `has_qc_runtime()` helper to `qc.py` for ONNX Runtime detection. - Added `empty_matches()` helper to `fullmap.py` for empty result fallback. @@ -851,8 +851,8 @@ A major bump for one removal: `method: list` is gone, and a table config that st - Fixed tutorial and example YAML configurations to use Excel column letter references (`A`, `B`, `C`, `D`) for `method: column` encodings instead of header names, matching the headerless source reading behavior. - Fixed `encoding` values in `docs/examples/` gallery configurations. - Updated `resolve_many()` API reference with new `qc` parameter and auto-detected QC provider. -- Fixed CITATION.cff version (7.2.2 → 7.4.0). -- Fixed CONTRIBUTING.md lazy-loaded package list (`typer` → `cyclopts`, added missing packages). +- Fixed CITATION.cff version (7.2.2 to 7.4.0). +- Fixed CONTRIBUTING.md lazy-loaded package list (`typer` to `cyclopts`, added missing packages). ## 7.3.6 - 2026-04-29 @@ -870,7 +870,7 @@ A major bump for one removal: `method: list` is gone, and a table config that st - Fixed `downloader.from_url()` failing on URLs that trigger an immediate download. The Playwright session now opens a browser context with `accept_downloads=True`, wraps `page.goto()` inside `page.expect_download()`, and tolerates the expected `net::ERR_ABORTED` navigation error that fires when the response is a download rather than a page. ### Documentation -- Documented `miscellaneous notes` as a freetext catch-all annotation in the table configuration and advanced-example pages — used for assay caveats, non-standard units, and qualitative observations that don't map cleanly to a structured field. Supports both `method: value` (constant) and `method: column` (per-row). +- Documented `miscellaneous notes` as a freetext catch-all annotation in the table configuration and advanced-example pages, used for assay caveats, non-standard units, and qualitative observations that don't map cleanly to a structured field. Supports both `method: value` (constant) and `method: column` (per-row). - Documented Polars regex constraints for the `regex` and `remove` transforms: patterns are passed to Polars `str.replace_all()` (Rust `regex` crate), so capturing groups (`(...)` / `\1`) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(? check loop +## Daily edit/check loop ```bash # after changing Rust code, or when the extension may be stale @@ -102,7 +102,7 @@ What the gates cover: - **Ruff linting and formatting.** The current tree enforces core pycodestyle/pyflakes safety checks plus stale-suppression detection. It also enforces an expanded rule set covering common bug patterns (bugbear), simplifications, Python-version upgrades, pytest style, import order, and comprehensions. Treat `uv run ruff check .` and `uv run ruff format --check .` as the stable interface rather than relying on individual rule codes. - **Pyright.** Type checking runs through `uv run pyright`; the project is tightening this as a strict-inference ratchet over time. -- **Python tests.** The suite is offline and runs in parallel by default via [pytest-xdist](https://pypi.org/project/pytest-xdist/) (`-n auto` in `pyproject.toml`'s addopts, which also enable `--cov` so coverage is reported inline); expect roughly 20-30 seconds for a full local run. Disable parallelism for a single serial run with `pytest -n 0`. CI installs the `ci` dependency group with the `cli`, `qc`, and `log` extras (`--no-default-groups --group ci --extra cli --extra qc --extra log`) and runs the suite as a single job rather than sharding it across a matrix of runners: most of a run is fixed overhead (interpreter start, imports, xdist worker spin-up, coverage init) rather than test execution, so splitting the suite would cost more in per-runner setup than it recovers. +- **Python tests.** The suite is offline and runs in parallel by default via [pytest-xdist](https://pypi.org/project/pytest-xdist/) (`-n auto` in `pyproject.toml`'s addopts, which also enable `--cov` so coverage is reported inline); Disable parallelism for a single serial run with `pytest -n 0`. CI installs the `ci` dependency group with the `cli`, `qc`, and `log` extras (`--no-default-groups --group ci --extra cli --extra qc --extra log`) and runs the suite as a single job rather than sharding it across a matrix of runners: most of a run is fixed overhead (interpreter start, imports, xdist worker spin-up, coverage init) rather than test execution, so splitting the suite would cost more in per-runner setup than it recovers. - **Rust tests.** `cargo test --manifest-path rust/Cargo.toml` runs the extension's Rust unit tests. - **Rust style and lints.** `cargo fmt --check` enforces formatting; clippy runs all targets with warnings denied. @@ -130,7 +130,7 @@ On every **push**, the whole-repo gates: The ruff hooks cover the **whole tree**, matching CI's `ruff check .`. They used to be scoped to `src/` and `tests/`, which meant `examples/` could only ever fail in CI. -The full pytest suite and `cargo test` are deliberately in neither stage: they rebuild the Rust extension, which is too slow and too stateful for a commit/push hook, and CI already runs them on every pull request -- the Python suite as a single job, deliberately not sharded across a matrix. Use `make check` when you want everything locally. +The full pytest suite and `cargo test` are deliberately in neither stage: they rebuild the Rust extension, which is too slow and too stateful for a commit/push hook, and CI already runs them on every pull request (the Python suite as a single job, deliberately not sharded across a matrix). Use `make check` when you want everything locally. ## Running subsets @@ -166,6 +166,7 @@ If your OS hard limit is lower, raise the system/user NOFILE limit first, then o - Keep PRs focused on one concern. - Include tests or explain why none are needed. - Run `make check` and include the result in the PR description. +- For documentation changes, also run the docs guards (`uv run pytest tests/test_docs_source_of_truth.py tests/test_docs_cli_coverage.py tests/test_docs_examples.py`) and `uv run mkdocs build --strict`; doc pages are bound to live models, CLI flags, and output by those tests. - PRs are squash-merged to `main`; write commits and PR titles so the squashed history stays clear. ## Getting help diff --git a/README.md b/README.md index a70c9b4..444ea20 100644 --- a/README.md +++ b/README.md @@ -109,15 +109,15 @@ Build the knowledge graph: tablassert build-kg graph.yaml ``` -Output is one JSON object per line: nodes with Biolink categories, edges with annotations. +Output is one JSON object per line: nodes with Biolink categories, edges with annotations. The edge sample below is abridged; real records also carry provenance fields such as `knowledge_level`, `agent_type`, `sources`, and `original_subject`/`original_object`. ```json -{"id":"HGNC:11998","name":"TP53","category":["biolink:Gene"],"taxon":"NCBITaxon:9606"} -{"id":"MONDO:0008903","name":"lung cancer","category":["biolink:Disease"]} +{"id":"HGNC:11998","name":"TP53","category":["biolink:Gene"],"in_taxon":["NCBITaxon:9606"],"provided_by":["infores:my-kg"]} +{"id":"MONDO:0008903","name":"lung cancer","category":["biolink:Disease"],"in_taxon":["NCBITaxon:9606"],"provided_by":["infores:my-kg"]} ``` ```json -{"id":"2cfea591-0f8f-33af-a7df-03da531d3359","subject":"HGNC:11998","predicate":"biolink:associated_with","object":"MONDO:0008903","p_value":"1.0000e-03","statistical_significance_qualifier":"strongly_significant","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"gene-disease.csv","study_size":450,"has_study_results":[{"id":"row:2"}]}},"publications":["PMID:12345678"]} +{"id":"8f41c2d7-9b3e-35c1-a4e8-6d2f9a0c5b71","subject":"HGNC:11998","predicate":"biolink:associated_with","object":"MONDO:0008903","p_value":"1.0000e-03","statistical_significance_qualifier":"very_strongly_significant","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"gene-disease.yaml SECTION 1","study_size":450,"has_study_results":[{"id":"row:2"}]}},"publications":["PMID:12345678"]} ``` See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full walkthrough. @@ -140,8 +140,10 @@ See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full w pip install tablassert ``` -Or with uv: `uv tool install "tablassert[cli]"`. The base install provides the Python API; `[cli]` -provides the `tablassert` command. Other extras are opt-in: +To use the Python API, install Tablassert into your project's environment with `pip install +"tablassert"` (or `uv add tablassert`). To get the `tablassert` command as a standalone tool, use +`uv tool install "tablassert[cli]"`; note that a tool install is isolated and does not expose the +Python API to your project. Other extras are opt-in: | Extra | Adds | Install | | ----- | ---- | ------- | diff --git a/docs/api/fullmap.md b/docs/api/fullmap.md index d113ed1..7dc8695 100644 --- a/docs/api/fullmap.md +++ b/docs/api/fullmap.md @@ -42,9 +42,11 @@ Path to the fullmap redb file (already resolved; see `fullmap_db_path()` and [Fu **`taxon: Optional[str]`** -Optional NCBI Taxon ID for filtering results. +Optional NCBI Taxon ID for filtering results. The constraint applies to matches that carry taxon +metadata; matches with `TAXON_ID` 0 (no taxon recorded) are always retained. -Example: `"9606"` filters to human-specific entities. +Example: `"9606"` keeps human entries and entries with no taxon recorded, and drops entries recorded to +another taxon. **`prioritize: Optional[list[Categories]]`** @@ -142,7 +144,12 @@ db = Path("/path/to/fullmap/data/fullmap.redb") # LazyFrame with data to resolve lf = pl.scan_parquet("data.parquet") -# Resolve gene symbols to CURIEs +# Resolve gene symbols to CURIEs. level_one and level_two must already have been +# applied, so the frame carries both "gene_symbol" and "gene_symbol_two". +from tablassert.nlp import level_one, level_two + +lf = level_one(lf, "gene_symbol") +lf = level_two(lf, "gene_symbol") result = resolve( lf=lf, col="gene_symbol", @@ -206,7 +213,10 @@ Rows without a valid CURIE are filtered from the returned frame. ### Provenance Tracking -Every resolved entity carries its source database, source version (snapshot date), and the matched synonym that triggered the match, enabling auditing and quality control. Case is handled by the NLP levels above: `level_one` matches any case variant, `level_two` further strips punctuation for hyphenated or slash-delimited names. +Every resolved entity carries its source database (`SOURCE_NAME`), source version (`SOURCE_VERSION`, the +snapshot date), and the normalized term that produced the match, enabling auditing and quality control. +Case is handled by the NLP levels above: `level_one` matches any case variant, `level_two` further +strips punctuation for hyphenated or slash-delimited names. ## Integration with QC @@ -297,8 +307,9 @@ for term, matches in quick_map(["TP53", "BRCA1", "nonsense"], db).items(): - `column_context` is fixed `False`: the frequency tiebreaker counts category occurrences within one resolved column of a real table, and a handful of probe terms is not that population. -- A term whose keys were all dropped by the junk-term filter (purely numeric or sentinel values) - resolves to the empty frame, because a build would never probe it either. +- A term whose keys were all dropped by the junk-term filter (an empty string, an all-digit string, or + one of the generic labels such as `none`, `nan`, `na`, `null`, `unknown`, `p_value`, `symbol`) resolves + to the empty frame, because a build would never probe it either. ## Next Steps diff --git a/docs/api/lib.md b/docs/api/lib.md index 68b1bef..11c239f 100644 --- a/docs/api/lib.md +++ b/docs/api/lib.md @@ -43,7 +43,9 @@ Filesystem path to the fullmap redb file, or a base directory containing it (res Optional NCBI Taxon ID for filtering results to a specific organism. -Example: `"9606"` restricts **gene** matches to human-specific entries; non-gene categories (e.g., diseases, chemicals) are returned regardless of taxon. When `None`, no taxon filtering is applied and matches from all organisms are returned. +Example: `"9606"` keeps matches recorded to human and matches carrying no taxon (`TAXON_ID` 0), which is +why disease and chemical rows usually survive it, and drops matches recorded to any other taxon. When +`None`, no taxon filtering is applied and matches from all organisms are returned. **`prioritize: Optional[list[Categories]]` (default: `None`)** @@ -59,7 +61,7 @@ Example: `[Categories.GENE]` prevents gene mappings from appearing in the output **`column_context: bool` (default: `True`)** -Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the deduplication stage adds a category-frequency score (computed in Polars after the SQL query) and prefers the category that appears most frequently across all matched terms in the batch. When `False`, frequency-based tie-breaking is disabled. +Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the deduplication stage adds a `FREQUENCY` column scoring how often each category appears across all matched terms in the batch, computed in Polars over the fetched rows, and prefers the most frequent category. When `False`, frequency-based tie-breaking is disabled and the column is absent. This is useful when resolving a column of related entities (e.g., all genes): the shared context helps disambiguate terms that map to multiple categories. @@ -164,7 +166,7 @@ The resolver requires a schema-v6 fullmap. Existing schema-v5 databases are reje - If the `fullmap` path does not resolve to a valid redb file, or the file's schema tag doesn't match the expected version, the Rust extension raises a `RuntimeError`. - If `entities` is empty, the function returns `[]`. -- Unresolved entities are silently filtered from the output (logged at INFO level by default via `resolve()`). +- Unresolved entities are filtered from the output and reported through `resolve()` logging. ## Integration diff --git a/docs/api/qc.md b/docs/api/qc.md index 25d1ac8..891910d 100644 --- a/docs/api/qc.md +++ b/docs/api/qc.md @@ -4,7 +4,7 @@ The `qc` module validates entity-resolution mappings through a four-stage pipeli QC runtime support is optional. Install `tablassert[qc]` to enable it: the extra pulls `scikit-learn` and `sentence-transformers` (`torch` and `numpy` arrive transitively); `rapidfuzz` is a core dependency and is always available. -`fullmap_audit()` checks the whole extra before it does any work and raises `QcRuntimeMissingError` naming every absent package and the install command. Checking up front matters because the two packages are needed at different stages (`scikit-learn` from the start, `sentence-transformers` only if Stage 4 is reached), so a half-installed extra would otherwise fail after the audit had already run. `build-kg --qc` performs the same check before the build begins, since the audit does not run until the very end of the build. +`fullmap_audit()` checks the whole extra before it does any work and raises `QcRuntimeMissingError` naming every absent package and the install command. Checking up front matters because the two packages are needed at different stages (`scikit-learn` from the start, `sentence-transformers` only if Stage 4 is reached), so a half-installed extra would otherwise fail after the audit had already run. `build-kg --qc` performs the same check before the build begins, so a missing extra fails immediately instead of after resolution has already cost time. ## fullmap_audit() @@ -81,33 +81,35 @@ The function applies four validation stages in sequence. Each stage progressivel **Fast path for high-confidence mappings.** A row passes if any of these hold: ```python -original == preferred_name +_qc_alias(original, aliases) == preferred_name ``` -- The pre-resolution text exactly equals the resolved preferred name. +- The alias-resolved pre-resolution text equals the resolved preferred name. `aliases` maps a casefolded + source text to its preferred spelling, so a configured alias can pass on exact match. - The resolved CURIE matches an exempt prefix (`CHEBI`, `PR`, `UniProtKB`, `NCBIGene`, `UMLS`, `UNII`, `PUBCHEM`, `MONDO`). - The original text contains `:` (already looks like a CURIE). - The preferred name matches an exception prefix (`^LOC` or `^si:`). -**Performance:** O(1) string comparison per row. - #### Stage 2: Fuzzy Matching **Medium confidence using RapidFuzz (batched via `rapidfuzz.process.cpdist`).** -Two fuzzy matching algorithms: -1. **Ratio:** Overall string similarity -2. **Partial token sort ratio:** Combined token/subsequence matching +Three scorers run over each unique pending pair, batched through `rapidfuzz.process.cpdist`: +1. **Ratio:** `fuzz.ratio`, overall string similarity +2. **Partial token sort ratio:** `fuzz.partial_token_sort_ratio`, subsequence matching +3. **Token set ratio:** `fuzz.token_set_ratio`, duplicate-tolerant token overlap -**Thresholds:** `fuzz.ratio` >= 70 OR `fuzz.partial_token_sort_ratio` >= 80 +A pair passes if any scorer clears its threshold. `fuzz.ratio` is compared against +`fuzzy_ratio_threshold` (default 70.0); the two token scorers share `fuzzy_partial_threshold` +(default 80.0). Both thresholds are graph-configurable through `qc_fuzzy_ratio_threshold` and +`qc_fuzzy_partial_threshold`. ```python -fuzz.ratio(original, preferred) >= 70 -or fuzz.partial_token_sort_ratio(original, preferred) >= 80 +fuzz.ratio(original, preferred) >= fuzzy_ratio_threshold +or fuzz.partial_token_sort_ratio(original, preferred) >= fuzzy_partial_threshold +or fuzz.token_set_ratio(original, preferred) >= fuzzy_partial_threshold ``` -**Performance:** O(n) string operations, batched. - #### Stage 3: Abbreviation Expansion **Deterministic pass for abbreviation/expansion pairs (Schwartz-Hearst).** A row passes when the pre-resolution text abbreviates the preferred name, or vice versa: @@ -118,8 +120,6 @@ _is_abbrev(original, preferred_name) or _is_abbrev(preferred_name, original) The matcher scans the short form right-to-left against the long form (case-insensitively); the first character of the short form must land on a word boundary of the long form. This rescues the class both fuzzy matching and embedding similarity can miss, the `AML` / `acute myeloid leukemia` pair. -**Performance:** O(n) character scans per row, no model inference. - #### Stage 4: SapBERT Semantic Similarity **High confidence using SapBERT embeddings.** @@ -134,7 +134,8 @@ similarity = cosine_similarity(embeddings[:n], embeddings[n:]).diagonal() return similarity >= qc.SIMILARITY_THRESHOLD # 0.5 ``` -**Performance:** Expensive (transformer inference); the model is loaded once and cached. +The model is loaded once and cached for the lifetime of the process, so transformer inference is the +costly stage of the cascade. ### SapBERT Model @@ -178,15 +179,18 @@ validated = fullmap_audit( ### Pipeline Flow +The cascade runs on unique `(original, preferred)` pairs, deduplicated before scoring, so the counts +below are pairs rather than source rows. Illustrative counts: + ``` -Input: 1000 rows with entity mappings +Input: 1000 unique pairs with entity mappings Stage 1 (Exact): 700 pass, 300 pending Stage 2 (Fuzzy): 250 pass, 50 pending Stage 3 (Abbreviation): 10 pass, 40 pending Stage 4 (SapBERT): 30 pass, 10 rejected -Output: 990 rows (700 + 250 + 10 + 30) +Output: 990 pairs pass (700 + 250 + 10 + 30) ``` ### Confidence Levels diff --git a/docs/api/utils.md b/docs/api/utils.md index 60903ca..bab68ed 100644 --- a/docs/api/utils.md +++ b/docs/api/utils.md @@ -12,7 +12,7 @@ The single parent working directory. All runtime artifacts live beneath it. Intermediate parquet storage for compiled subgraphs (`.tablassert/store/`). Created on import. -The log directory (`log.LOGASSERT`, `.tablassert/log/`) holding the loguru sink file `.tablassert/log/tablassert.log` when the `log` extra is installed, and the cached SapBERT model (`.tablassert/sapbert/`, `qc.MODEL`) are likewise derived from `BASE`. +The log directory (`log.LOGASSERT`, `.tablassert/log/`) holding the loguru sink file `.tablassert/log/tablassert.log` when the `log` extra is installed, and the cached SapBERT model (`qc.MODEL`, `.tablassert/sapbert-babel/`) are likewise derived from `BASE`. ## mkhash() @@ -64,11 +64,11 @@ def section_store_key( For a section with a local source, the key combines the section configuration digest with the source content digest and hashes the result into one 16-character xxh64 value. Callers may provide a precomputed `content_digest` to avoid rereading the file; otherwise `local` is passed to `file_content_hash`. If neither is provided, the function returns the section's plain `mkhash` value. The build pipeline uses the content-aware form for local sources, so changing one source invalidates the cached parquet for each section that reads it without invalidating sections backed by unchanged sources. -The key formula is `mkhash(f"{mkhash(section)}:{content_digest}")`. Build-mode flags add their own filename suffixes (`.head`, `.release`, or `.qc`) after the key; they do not change the digest formula. +The key formula is `mkhash(f"{mkhash(section)}:{content_digest}")`. Build-mode flags add their own filename suffixes (`.head`, `.release`, `.qc`, or `.linkouts`, concatenated when several are active) before the `.parquet` extension; they do not change the digest formula. --- -## namespace_uuid() *(Rust extension: `tablassert.rs`)* +## namespace_uuid() *(Rust extension: `tablassert.rs`)* Generates deterministic UUIDs for KGX edge identifiers using UUID v3 (MD5-based namespacing). Provided by the Rust extension, not `tablassert.utils`. @@ -86,7 +86,10 @@ Domain string used to create the namespace UUID. The default domain used interna **`values: list[str]`** -The values to incorporate into the UUID. Empty entries are dropped; each surviving value is length-prefixed as `:` and concatenated, then hashed within the domain namespace. +The values to incorporate into the UUID. Every entry is used, empty strings included, and order is +significant: each value is length-prefixed as `:` and the parts are concatenated in +the order given, then hashed within the domain namespace. Two different value lists therefore almost +never derive the same id, and dropping an empty entry changes the result. ### Return Value @@ -164,7 +167,8 @@ A graph that *expects* such collisions (e.g. one whose `uuid_fields` are the res so two rows with different raw mention spellings resolve to the same CURIE) can instead set `uuid_on_collision: merge` (requires `uuid_fields`). Divergent same-id records are then folded into one edge: list fields are unioned, deduplicated by content, and sorted (so the merged edge is -identical regardless of row order), conflicting scalars keep the first record's value, and a +identical regardless of row order), conflicting scalars keep the lexicographically smallest canonical +value, and a build-log summary reports how many records merged and how many scalar conflicts were arbitrated. Merge mode buffers one full record per unique id until end-of-stream (the memory cost the default streaming path avoids), which is why it is opt-in. See diff --git a/docs/cli.md b/docs/cli.md index 3291adc..06d04e9 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -22,7 +22,7 @@ These are flags on the root `tablassert` command, **not** subcommands. | Flag | Description | | --- | --- | -| `--version` | Print the installed package version as `tablassert ` (e.g. `tablassert 10.1.0`) and exit | +| `--version` | Print the installed package version as `tablassert ` (e.g. `tablassert 21.0.0`) and exit | | `--help`, `-h` | Show help for the app or a subcommand | !!! warning "Two different versions" @@ -57,7 +57,7 @@ tablassert build-fullmap [ARGS] tablassert build-fullmap --output /data/fullmap/fullmap.redb # Force a from-scratch rebuild from BABEL outputs (e.g. after a BABEL snapshot bump) tablassert build-fullmap --force --output /data/fullmap/fullmap.redb -# Install the [aria2] extra and the bundled aria2c is used automatically for either download (the multi-GB prebuilt is the ideal aria2 use case) +# With the [aria2] extra installed, the bundled aria2c is used automatically for either download pip install "tablassert[aria2]" tablassert build-fullmap --output /data/fullmap/fullmap.redb ``` @@ -68,8 +68,9 @@ Every one of those invocations produces a database filtered by the checked-in By default `build-fullmap` looks for a prebuilt `fullmap.tar.zst` at `https://stars.renci.org/var/babel_outputs//fullmap//` (the version -directory is the **installed Tablassert package version**, never hardcoded), verifies it against the -published `sha256sum.txt`, and extracts it beside `--output` in the Rust extension, streaming the +directory is the **installed Tablassert package version**, never hardcoded). It verifies the archive +against the published `sha256sum.txt` when that file is present and parseable, and otherwise warns and +proceeds unverified, then extracts it beside `--output` in the Rust extension, streaming the archive through zstd decompression and untarring it with the GIL released (the decompressed tar never touches disk), then validating the extracted primary + shards against the force-build contract (exact `v6` schema, a recorded `build_id`, the exact @@ -91,9 +92,10 @@ A database already present at `--output` is reused only when it carries that sam an unfiltered (or differently filtered) leftover from an older Tablassert is rebuilt, with a warning. Pass `--force` to rebuild unconditionally. -A from-scratch build parallelizes automatically across all available CPU threads: on Linux the -worker count is capped by available memory (~2 GB per thread, read from `/proc/meminfo`) to avoid -OOMs. There is no flag to tune. +A from-scratch build parallelizes automatically. On Linux the worker count is +`min(available_CPUs, MemAvailable_GB / 2)`, read from `/proc/meminfo`, so each worker is budgeted 2 GB +of addressable memory and a 16 GB machine caps at 8 workers; elsewhere it falls back to 90 percent of +the CPU count. There is no flag to tune. See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config usage. @@ -101,13 +103,13 @@ See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config ## quick-map -Use this to see what fullmap entity resolution will do with one or more terms (the rows a -`build-kg` run would emit for a cell holding that term under the filters shown. Each term runs -through the exact build path (level-one/level-two normalization, probe-key extraction, one redb -fetch, then filter, rank, and dedup), one `rich` table per term in input order, with the term's normalized -probe keys in the title so a miss is diagnosable. The whole input is one batched redb round trip, -never one per term. A term with no matches prints a `no matches` line and still exits 0: a miss is -a finding, not a failure; only usage errors and an unreadable fullmap exit 2. +Use this to see what fullmap entity resolution will do with one or more terms: the rows a `build-kg` +run would emit for a cell holding that term under the filters shown. Each term runs through the exact +build path (level-one/level-two normalization, probe-key extraction, one redb fetch, then filter, rank, +and dedup). Matched terms print one `rich` table each, in input order, with the term's normalized probe +keys in the title so a miss is diagnosable. The whole input is one batched redb round trip, never one +per term. A term with no matches prints a `no matches` line and still exits 0. Only a bad flag or a +missing fullmap exits 2; a fullmap that fails to open surfaces its own error. ```bash tablassert quick-map TERMS... --fullmap ./fullmap/data/fullmap.redb [OPTIONS] @@ -167,17 +169,17 @@ The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is | `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `not_significant` edges before resolution) | | `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact match, fuzzy match, abbreviation expansion, then SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies), and, when any section declares a `category_override`, no edge demoted to bare `biolink:Association` (a row escaped its pin and shipped without the class-specific slots `prune_to_class` nulled into `has_supporting_studies`; the assertion is graph-wide, so one pinned section also gates edges from unpinned sections) and fails the build (non-zero exit) on any violation | | `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging; requires the `[log]` extra, and without it Tablassert produces no logs | -| `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: <=5 random rows/section, cached to `.head.parquet`, never clobbers a full build | -| `--linkouts`, `-lo` | Flag | No | `False` | Append the EdgeLinkouts per-edge record URL (`https://linkouts.skyelanegoetz.com/edges/{edge_id}`, resolved to the final edge id after dedup) to the graph infores entry (`rig.source_info.infores_id`) in every emitted edge's `sources` list, so linkouts.skyelanegoetz.com can deep-link each edge; sections declaring `provenance.override` are untouched; cached to `.linkouts.parquet`, never clobbers a full build | +| `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: <=5 random rows/section, cached under a `.head` store suffix, never clobbers a full build | +| `--linkouts`, `-lo` | Flag | No | `False` | Append the EdgeLinkouts per-edge record URL (`https://linkouts.skyelanegoetz.com/edges/{edge_id}`, resolved to the final edge id after dedup) to the graph infores entry (`rig.source_info.infores_id`) in every emitted edge's `sources` list, so linkouts.skyelanegoetz.com can deep-link each edge; sections declaring `provenance.override` are untouched; cached under a `.linkouts` store suffix, never clobbers a full build | ```bash tablassert build-kg graph.yaml --qc --log ``` The parallel fullmap reads behind entity resolution are automatic: large lookup batches (>= 1024 -terms) fan out across the record-shard files on all available CPU threads (redb readers share-lock, -so they never contend), while smaller batches stay serial. There is no flag to tune, and results are -identical at any worker count. +terms) fan out across the 16 record-shard files, one thread per shard (redb readers share-lock, so they +never contend), while smaller batches stay serial. There is no flag to tune, and results are identical +at any worker count. Output is written to `rig.artifact_base_path` (created when missing) as `{name}_{version}.nodes.ndjson`, `{name}_{version}.edges.ndjson`, and `{name}_{version}.RIG.yaml`; intermediate parquet lands in @@ -189,13 +191,14 @@ incomplete RIG fails the build with `[rig-validation-failed]` and nothing is emi [Graph Configuration](configuration/graph.md). ??? info "Build progress & stages" - The build runs six parallel stages (Loading Tables, Extracting Sections, Building TCode, + The build runs six stages in sequence (Loading Tables, Extracting Sections, Building TCode, Collecting Instructions, Building Subgraphs, Compiling Graph) plus a seventh, Studying Graph, - only when `--qc` is passed. They run under a three-row live progress block (stage header; - section bar with count/elapsed/ETA; in-flight item detail). Each completed - stage prints a green `✓ Stage N · NAME · elapsed` line above the live block. During Building - Subgraphs the detail line also shows the per-section phase (`load`, `filter`, `clean`, `encode`, - `resolve`, `qc`, `edge`, `provenance`, `significance`, `finalize`, `write`). + only when `--qc` is passed. They render under a three-row live progress block (stage header; + section bar with count/elapsed/ETA; in-flight item detail). Each completed stage prints a green + confirmation line above the block, naming the stage number, its uppercased name, and its elapsed + time. During Building Subgraphs the detail line also shows the per-section phase (`load`, + `filter`, `clean`, `encode`, `resolve`, `qc`, `edge`, `provenance`, `significance`, `finalize`, + `write`, plus `cache` for a shared run-scoped prefix and `transform` for an unlabeled op). ### Automatic run-scoped TCode cache @@ -214,8 +217,8 @@ section; other section-local operations retain their own section labels. ## validate -Use this to validate a configuration against a schema without running the build, ideal for CI and -pre-commit hooks. The required `--schema` flag selects which schema to validate against (the kind is +Use this to validate a configuration against a schema without running the build; it is what CI and +pre-commit hooks call. The required `--schema` flag selects which schema to validate against (the kind is no longer sniffed from the YAML). ```bash @@ -260,11 +263,11 @@ tablassert validate-kgx --nodes MY_KG_1.0.0.nodes.ndjson --edges MY_KG_1.0.0.edg | `--prune`, `-p` | Flag | No | `False` | Rewrite the edges file in place, keeping only edges that validate strictly (pending carryovers are dropped too); nodes are never touched, and a clean file is not rewritten | | `--rig` | Path | No | n/a | RIG yaml to include in the identity scan (its `source_info.infores_id` and target `primary_knowledge_sources` are checked) | | `--allow-infores` | list[str] | No | n/a | Registry-exempt CURIE, repeatable; use for the locally-minted graph identifier and other deliberate locals | -| `--registry` | snapshot\\|refresh\\|off | No | `snapshot` | `snapshot` validates offline against the bundled copy; `refresh` fetches the live upstream catalog (network) and caches it under the user cache dir; `off` skips the identity half entirely | +| `--registry` | snapshot\|refresh\|off | No | `snapshot` | `snapshot` validates offline against the bundled copy; `refresh` fetches the live upstream catalog (network) and caches it under the user cache dir; `off` skips the identity half entirely | | `--strict` | Flag | No | `False` | Exit non-zero while any unregistered CURIE remains; without it the identity half is advisory | Failures are grouped by field and error type, so a systematic modelling problem shows up as one line -rather than a million: +per problem rather than one per record: ```text biolink-model 4.4.4 @@ -316,14 +319,14 @@ non-zero in every mode: a file that was never read must never count as a pass. ### Repairing a graph with `--prune` -`--prune` (`-p`) repairs an already-emitted graph instead of just scoring it: the edges file is +`--prune` (`-p`) repairs an already-emitted graph instead of only scoring it: the edges file is rewritten in place and every edge that does not validate strictly is removed. That includes real defects (non-coercible values, extras the model will never declare, rejected predicates), the deliberate pending carryovers (`synonym`, `xref`, `relation`, `provided_by`, ...), and lines that -are not JSON at all (counted separately as malformed). On a real graph the carryovers are most of -the edges -- a DAKP-style build prunes roughly 800085 of 2000085 -- so `--prune` trades them for a -strictly-compliant graph. Nodes are validated and reported, never pruned: dropping one would -strand its references. +are not JSON at all (counted separately as malformed). On a real graph the carryovers dominate the +failure list: a DAKP-style build prunes 800085 of 2000085 records (about 40 percent), so `--prune` +trades them for a strictly-compliant graph. Nodes are validated and reported, never pruned: dropping +one would strand its references. ```bash tablassert validate-kgx -n MY_KG_1.0.0.nodes.ndjson -e MY_KG_1.0.0.edges.ndjson --prune @@ -338,7 +341,7 @@ only what ships is judged. ```text edges: pruned 12/424159 non-compliant records - 9 p_value: float_parsing + 9 p_value: float_parsing e.g. DISEASE_EDGE_000017: p_value: float_parsing edges: 424147/424159 valid (0 failures) KGX output is Biolink-compliant. diff --git a/docs/configuration/advanced-example.md b/docs/configuration/advanced-example.md index 5bde5ff..114ed69 100644 --- a/docs/configuration/advanced-example.md +++ b/docs/configuration/advanced-example.md @@ -82,7 +82,8 @@ whole table or `method: column` to pull per-row notes from the source (see - **Mixed annotations**: `method: value` for constants (same every row), `method: column` for per-row values. - **Subject-predicate-object**: subject varies per row (column), predicate `correlated_with` is - fixed, and object `CHEBI:41774` is fixed, yielding `Lactobacillus rhamnosus --[correlated_with]--> 13C-tamoxifen`. + fixed, and object `CHEBI:41774` is fixed, yielding a `correlated_with` edge from `Lactobacillus + rhamnosus` to `13C-tamoxifen`. ??? note "Regex dialect constraint" Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Backreferences diff --git a/docs/configuration/graph.md b/docs/configuration/graph.md index 8ca3a30..9bb2f73 100644 --- a/docs/configuration/graph.md +++ b/docs/configuration/graph.md @@ -34,15 +34,15 @@ The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation | `uuid_fields` | List[str] | Edge fields that constitute edge identity. Only these feed the derived edge `id` (see [Stable edge ids](#stable-edge-ids)) | | `uuid_domain` | String | Explicit UUID namespace. Defaults to `rig.source_info.infores_id` when `uuid_fields` is set, `TABLASSERT` otherwise | | `uuid_on_collision` | `error` \| `merge` | What to do when two different edges derive one id. `error` (default) aborts; `merge` folds them into one edge. Requires `uuid_fields` (see [Merging collisions instead](#merging-collisions-instead)) | -| `qc_similarity_threshold` | Float 0-1 | SapBERT cosine threshold used by QC | -| `qc_fuzzy_ratio_threshold` | Float 0-100 | RapidFuzz ratio threshold used by QC | -| `qc_fuzzy_partial_threshold` | Float 0-100 | RapidFuzz partial/token-set threshold used by QC | -| `qc_aliases` | Mapping[String, String] | Source-text aliases applied before QC similarity stages | +| `qc_similarity_threshold` | Float 0-1 | SapBERT cosine threshold used by QC (default 0.5) | +| `qc_fuzzy_ratio_threshold` | Float 0-100 | RapidFuzz ratio threshold used by QC (default 70) | +| `qc_fuzzy_partial_threshold` | Float 0-100 | RapidFuzz partial/token-set threshold used by QC (default 80) | +| `qc_aliases` | Mapping[String, String] | Source-text aliases applied before QC similarity stages (default empty) | ## Stable edge ids Every edge gets a deterministic `id`: a UUID v3 derived from the edge itself. By default it is -derived from the **whole record**, which makes it maximally brittle: a corrected `p_value`, a new +derived from the **whole record**, which makes it change on any field edit: a corrected `p_value`, a new `supporting_text` entry, a reordered source row, or a Biolink release that renames a slot all mint a brand-new id. Downstream Translator consumers then see a new edge where they should see the same edge with updated attributes. @@ -75,7 +75,7 @@ Leave out anything that is an observation *about* the edge rather than the edge' **Then build, and let the failure tell you what is missing.** That list is a starting point, not an answer: whether it is a key depends on your data, and the only way to find out is to run it. On a -real 1.27M-edge graph the set above left 2,476 collisions (0.2% of edges): pairs whose subject, +real 1.27M-edge MultiomicsKG 3.0.0 build, the set above left 2,476 collisions (0.2% of edges): pairs whose subject, predicate, object, publication and row were identical, differing only in the NLP level recorded in `supporting_text` because two raw strings had resolved onto the same CURIE. Adding `supporting_text` made it a key. @@ -85,7 +85,7 @@ offending edge, so each round is mechanical. ### What it buys -On that same 1.27M-edge graph, re-analysing the statistics (new `p_value` and `effect_size` on +On that same MultiomicsKG 3.0.0 build, re-analysing the statistics (new `p_value` and `effect_size` on every row) and rebuilding: | | edge ids that changed | @@ -127,18 +127,18 @@ uuid_on_collision: merge Under `merge`, a divergent same-id record is folded into the first record that claimed the id: -- **list fields** (`publications`, `sources`, `source_record_urls`, `supporting_text`, `category`, - `upstream_resource_ids`, `has_supporting_studies`, ...) are unioned, deduplicated, and **sorted**, +- **list fields** (`publications`, `sources`, `supporting_text`, `category`, + `has_supporting_studies`, ...) are unioned, deduplicated, and **sorted**, so the merged edge is identical regardless of which row arrived first. Object entries such as `sources[]` dedup by content; key order alone never keeps two copies. - **scalar fields** keep the lexicographically smallest canonical value; each conflict is counted and reported in a build-log summary. -- fields only the later record carries are copied over: first-wins arbitrates *conflicts*, not - additions. -- **one scalar is exempt from first-wins:** when the merged record carries the build-internal +- fields only the later record carries are copied over: the lexicographic-minimum rule arbitrates + *conflicts*, not additions. +- **one scalar is exempt from that rule:** when the merged record carries the build-internal `supporting_case_ids` list (the case IDs behind a `number_of_cases` count), the merged count becomes the length of the unioned list, so a case ID shared by both records counts once where - first-wins would under-report and summing would double-count. The superseded divergence is not + the minimum rule would under-report and summing would double-count. The superseded divergence is not reported as a scalar conflict, and the carrier list is stripped from every edge before write in both modes, so it never ships in the final NDJSON. @@ -171,20 +171,20 @@ tell your consumers. ### The `rig:` section -The `rig:` section carries every human-authored RIG fact. Its shape mirrors the released [RIG schema](https://github.com/biolink/resource-ingest-guide-schema), so the generated `.RIG.yaml` is always schema-shaped. The generator derives only mechanical facts from the build (generated artifact file entries, observed edge/node type summaries) and **validates the complete document before writing anything**, so a build never leaves behind an invalid or incomplete RIG. +The `rig:` section carries every human-authored RIG fact. Its shape follows the released [RIG schema](https://github.com/biolink/resource-ingest-guide-schema), with two deliberate differences: the `artifact_base_url` and `artifact_base_path` keys below are Tablassert-only config consumed by the generator (never written to the `.RIG.yaml`), and a few fields differ in requiredness from the upstream schema and template, which disagree with each other on `source_status`; Tablassert requires `provenance_info`, `source_status`, and `ingest_info.scope`, and treats `ingest_info.relevant_files` as optional because the generator pre-fills the generated-artifact entries. The generator derives only mechanical facts from the build (generated artifact file entries, observed edge/node type summaries) and **validates the complete document before writing anything**, so a build never leaves behind an invalid or incomplete RIG. | Field | Type | Required | Description | |-------|------|----------|-------------| | `rig.name` | String | No | RIG display name; defaults to ` v Resource Ingest Guide` | | `rig.source_info` | Object | Yes | Information about the source being ingested (see below) | | `rig.ingest_info` | Object | Yes | Rationale and scope of the ingest (see below) | -| `rig.target_info` | Object | No | Target-level `future_considerations` and `additional_notes` (edge/node type summaries are always generated) | +| `rig.target_info` | Object | No | Target-level `future_considerations` (each with a `category`: `spoq_pattern`, `predicates`, `qualifiers`, `edge_properties`, `node_properties`, or `other`) and `additional_notes`; edge/node type summaries are always generated | | `rig.ui_explanation` | String | No | Per-edge-type UI explanation **prefix**; the built-in Tablassert explanation is always appended after it | -| `rig.source_files` | List[String] | No | Upstream source file names/URLs listed verbatim as `source_files` on every generated `edge_type_info` entry; never scraped from edge `source_record_urls` | +| `rig.source_files` | List[String] | No | Upstream source file names/URLs, sorted and deduplicated, written as `source_files` on every generated `edge_type_info` entry; never scraped from edge `source_record_urls` | | `rig.provenance_info` | Object | Yes | Contributor statements and provenance artifacts | | `rig.supporting_data_source_info` | List[Object] | No | Upstream data sources for data-derived graphs (each needs `infores_id`, `terms_of_use_info`, and `relevant_files`) | -| `rig.artifact_base_url` | String | Yes | Public URL prefix for the generated KGX artifacts; each `.nodes.ndjson`/`.edges.ndjson` name is appended to build RIG `relevant_files` locations | -| `rig.artifact_base_path` | Path | Yes | Output directory the generated artifacts (and the RIG) are written into; created when missing | +| `rig.artifact_base_url` | String | Yes | Public URL prefix for the generated KGX artifacts; each `.nodes.ndjson`/`.edges.ndjson` name is appended to build RIG `relevant_files` locations. Tablassert-only config key, not a RIG schema slot, and never written to the `.RIG.yaml` | +| `rig.artifact_base_path` | Path | Yes | Output directory the generated artifacts (and the RIG) are written into; created when missing. Tablassert-only config key, not a RIG schema slot, and never written to the `.RIG.yaml` | #### `rig.source_info` @@ -195,13 +195,15 @@ The `rig:` section carries every human-authored RIG fact. Its shape mirrors the | `description` | String | No | What the source contains and how its knowledge is produced | | `citations` | List[String] | No | PMIDs, DOIs, URLs, or free-text citations | | `terms_of_use_info` | Object | Yes | At least one of `terms_of_use_url`, `terms_of_use_description`, `license_name`, `license_url` must carry a real assessment | -| `data_access_locations` | List[String] | Yes | Where the upstream source data is accessed; each entry must contain an http(s) or file URL | +| `data_access_locations` | List[String] | Yes | Where the upstream source data is accessed; at least one entry required, each containing an http(s) or file URL | | `data_provision_mechanisms` | List[Enum] | No | `file_download`, `api_endpoint`, `database_dump`, `other` | | `data_formats` | List[Enum] | No | `tsv`, `csv`, `xml`, `json`, `yaml`, `obo`, `protobuff`, `kgx`, `mysql`, `postgresql`, `sqlite`, `other` | | `data_versioning_and_releases` | String | No | How the source versions/releases its data | | `source_status` | Enum | Yes | `maintained_regular_updates`, `maintained_as_needed_updates`, `not_maintained`, `unknown` | | `additional_notes` | List[String] | No | Anything not captured by dedicated fields | +`source_status` is required here because Tablassert follows the translator-ingests template, which marks it required; the released LinkML schema leaves it optional. + #### `rig.ingest_info` | Field | Type | Required | Description | @@ -241,7 +243,7 @@ Before the `.RIG.yaml` is written, the build audits the assembled document and f - edge/node summaries disagree with the observed graph (categories, predicates, KL/AT values, infores CURIEs); - a configured upstream `relevant_files` entry matches no table source file or URL. -This is what makes every emitted RIG PR-worthy by construction: placeholders and silent gaps fail the build instead of shipping. +Every emitted RIG passes this audit before it is written: placeholders and silent gaps fail the build instead of shipping. ## Minimal Example @@ -328,6 +330,7 @@ When you run `tablassert build-kg graph.yaml`: - Validate with the QC audit when `build-kg --qc` is passed 5. **Build subgraphs** - Compile each section's resolved data into a parquet file 6. **Compile graph** - Aggregate all subgraph parquets, export `{name}_{version}.nodes.ndjson` / `.edges.ndjson` into `rig.artifact_base_path`, summarize the final graph, audit the RIG document, and write `{name}_{version}.RIG.yaml` +7. **Study graph** (`--qc` only) - Run the QC audit over the compiled graph ## Output Files diff --git a/docs/configuration/table.md b/docs/configuration/table.md index cc0f2f6..2515c34 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -92,8 +92,8 @@ Defines the data file location and format. | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. | -| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. | -| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | +| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | +| `rows` | List[NonNegativeInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | | `reindex` | List[Reindex] | No | Conditional row filtering | **Example:** @@ -117,8 +117,8 @@ source: | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | | `delimiter` | String | No | Field delimiter. Defaults to `","`. | -| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. | -| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | +| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | +| `rows` | List[NonNegativeInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | | `reindex` | List[Reindex] | No | Conditional filtering | **Example:** @@ -147,8 +147,8 @@ as-is. | `kind` | String | No | Source kind. Model default is `"parquet"`, but specify it explicitly in configs. | | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. | | `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. | -| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. | -| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | +| `row_slice` | List[NonNegativeInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a non-negative integer or `"auto"`. Mutually exclusive with `rows`. | +| `rows` | List[NonNegativeInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. | | `reindex` | List[Reindex] | No | Conditional row filtering | **Example:** @@ -187,7 +187,7 @@ Defines subject-predicate-object relationships. | `predicate` | String | No | Biolink predicate. Defaults to `"related_to"`. Three DAKP prevention predicates are deliberate local extensions ahead of Biolink support (see `biolink.PREDICATE_OVERRIDES`): `prevents`, `applied_to_prevent`, `contraindicated_in_the_prevention_of`. | | `object` | NodeEncoding | Yes | Object entity configuration | | `qualifiers` | List[Qualifier] | No | Edge qualifiers (context) | -| `category_override` | Map[Categories, EdgeCategories] | No | Pin the association class per resolved object category (bare names, no `biolink:` prefix), replacing the derived (subject, object) pair lookup for those rows. Rows whose object category is absent from the map derive as before. Pinned classes are still reconciled against the section predicate (a class whose `predicate` slot rejects it is walked up the association hierarchy, with a warning at config time). Use it when one section mixes object categories the pair lookup merges: e.g. `Disease` and `PhenotypicFeature` both derive to `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation`, but only `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` declare slots like `regulatory_approvals` and `number_of_cases`. Rows pinned to either of those classes may additionally carry DAKP's sparse qualifier stack ahead of the model under deliberate Tablassert policy grants: `population_context_qualifier` and `temporal_context_qualifier` on both classes, plus `sex_qualifier` on the disease side. Deliberately ungranted: `species_context_qualifier` (disabled -- never emittable) plus `temporal_interval_qualifier` and `severity_qualifier` (unsatisfiable -- attached to no class, so a grant could never validate). | +| `category_override` | Map[Categories, EdgeCategories] | No | Pin the association class per resolved object category (bare names, no `biolink:` prefix), replacing the derived (subject, object) pair lookup for those rows. Rows whose object category is absent from the map derive as before. Pinned classes are still reconciled against the section predicate (a class whose `predicate` slot rejects it is walked up the association hierarchy, with a warning at config time). Use it when one section mixes object categories the pair lookup merges: e.g. `Disease` and `PhenotypicFeature` both derive to `ChemicalEntityToDiseaseOrPhenotypicFeatureAssociation`, but only `EntityToDiseaseAssociation` / `EntityToPhenotypicFeatureAssociation` declare slots like `regulatory_approvals` and `number_of_cases`. Rows pinned to either of those classes may additionally carry DAKP's sparse qualifier stack ahead of the model under deliberate Tablassert policy grants: `temporal_context_qualifier` on both classes, plus `sex_qualifier` on the disease side (`population_context_qualifier` needed a grant only until biolink-model 4.4.6 declared it natively on both pinned classes). Deliberately ungranted: `species_context_qualifier` (disabled, never emittable) plus `temporal_interval_qualifier` and `severity_qualifier` (unsatisfiable: attached to no class, so a grant could never validate). | **Example:** ```yaml @@ -227,7 +227,7 @@ Defines how to extract and resolve entities. |-------|------|----------|-------------| | `method` | String | No | `"value"` (literal) or `"column"` (source column letters). Defaults to `"value"`. | | `encoding` | String\|Int\|Float | Yes | Literal value or source column letters, depending on `method` | -| `taxon` | PositiveInt | No | NCBI Taxon ID for filtering (e.g., `9606` for human) | +| `taxon` | PositiveInt | No | NCBI Taxon ID for filtering (e.g., `9606` for human); defaults to `9606` (Homo sapiens), set null to disable | | `prioritize` | List[String] | No | Preferred Biolink categories (must be valid `Categories` enum values such as `Gene`, `Protein`) | | `avoid` | List[String] | No | Excluded Biolink categories (must be valid `Categories` enum values). Setting `avoid` makes the column an allow-list by complement: categories the `Categories` enum cannot name (fullmap-emitted Biolink mixins; see `biolink.CATEGORY_OVERRIDES`) are dropped as well, so the guard has no silent hole | | `exclude_prefixes` | List[String] | No | CURIE namespace prefixes excluded during entity resolution: the prefix is the text before the first `:` of a resolved CURIE, and every candidate whose prefix is listed is dropped. Matching is exact and case-sensitive (`OMIM` drops `OMIM:100100` but not `OMIMPS:100` or `omim:100100`). Optional; defaults to null (no prefix filtering). See [Resolution Filters](#resolution-filters). | @@ -605,7 +605,7 @@ Any other annotation name is treated as **supporting context**. At the end of `c This means nothing in your source data is silently dropped: context that doesn't map to a structured Biolink slot travels along inside `supporting_text` instead. -In addition to user-declared annotations, every edge automatically carries `extracted_from_row_number`, a 1-based index into the original source table (matching Excel-style row numbering). It is not declared as an annotation. Tablassert emits it internally so each edge always carries its source-row provenance, and the inlined supporting study consumes it: the `StudyResult` that anchors the study's row is identified by the scoped CURIE `row:`. The row and sheet columns are never folded into `supporting_text`. +In addition to user-declared annotations, Tablassert tracks each edge's source row internally as `extracted_from_row_number`, a 1-based index into the original source table (matching Excel-style row numbering). It is never emitted as an edge field: the inlined supporting study consumes it to anchor each `StudyResult` by the scoped CURIE `row:`, and the column is then dropped, along with `sheet_name`, rather than folded into `supporting_text`. The supporting study's identity is disjoint by construction: `Study.id` is the section's publication CURIE (`PMID:123`, `PMCID:PMC11708054`); `Study.name` is a human label naming the originating table config file and the section's 1-based position within it (e.g. `KANG12.yaml SECTION 2`). It is never a composition of the two. A section declaring **no `publications`** falls back to the config stem (the YAML filename without its extension) as the id, and the struct is then emitted **only when it carries something to preserve**, such as study metadata, routed unsatisfiable values, or class-pruned values. Biolink defines `has supporting studies` as "studies that produced information used as evidence", so a contentless unpublished section emits no `has_supporting_studies` at all; a section with a real publication always keeps the wrapper: diff --git a/docs/fullmap.md b/docs/fullmap.md index abd9f85..beb684a 100644 --- a/docs/fullmap.md +++ b/docs/fullmap.md @@ -1,8 +1,8 @@ # Fullmap Build a fullmap once and every `resolve()` / `resolve_many()` call maps free text to the right -biological CURIE; the completeness and freshness of this database directly sets how many of your -entities resolve correctly and how trustworthy the resulting graph is. +biological CURIE. How complete and how fresh this database is sets how many of your entities resolve +and how trustworthy the resulting graph is. Fullmap is Tablassert's embedded entity-resolution database: a small set of [redb](https://github.com/cberner/redb) files containing biological synonyms, CURIEs, Biolink categories, taxon IDs, and source provenance, @@ -10,7 +10,10 @@ built from NCATS Translator BABEL export files. It comprises a primary file hold CURIEs, and schema metadata, plus hash-sharded RECORDS files (`fullmap.s0.redb` ... `fullmap.s15.redb` by default) holding the term-to-postings index. -Fullmap is built entirely in-process by Tablassert's own Rust extension: no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). When the optional `[aria2]` extra is installed (`pip install "tablassert[aria2]"`; Linux/Windows wheels only), both download stages (the prebuilt `fullmap.tar.zst` archive and, for a from-scratch build, the BABEL class and synonym files) use the bundled aria2c binary automatically; without it, Tablassert's Python downloader is used. +Fullmap is built entirely in-process by Tablassert's own Rust extension: no external tool and no install +step. When the optional `[aria2]` extra is installed (`pip install "tablassert[aria2]"`; Linux/Windows +wheels only), both download stages use the bundled aria2c binary automatically; without it, Tablassert's +Python downloader is used. ## Build Command @@ -21,7 +24,7 @@ tablassert build-fullmap # Force a from-scratch build from BABEL outputs (skips the prebuilt download) tablassert build-fullmap --force -# Optional: with `pip install "tablassert[aria2]"`, the same command downloads through the bundled aria2c automatically (the multi-GB prebuilt is the ideal aria2 use case) +# Optional: with `pip install "tablassert[aria2]"`, the same command downloads through the bundled aria2c automatically tablassert build-fullmap ``` @@ -34,9 +37,10 @@ records its allowlist identity in `META.taxon_allowlist`, and that identity gate - A prebuilt archive whose recorded identity does not match the current allowlist fails validation during extraction, so it is never installed; the command falls back to a filtered source build. - Note the operational consequence: until a filtered archive is published for a Tablassert version, - the default invocation downloads that version's unfiltered archive once, fails the identity check, - and then performs the full BABEL download + build. + The operational consequence depends on what RENCI has published: if an archive for the requested + version and allowlist exists it is used directly, and otherwise the default invocation downloads the + closest published archive, fails the identity check, and then performs the full BABEL download and + build. - A database already present at `--output` is reused only when its recorded identity matches; otherwise it is rebuilt. @@ -46,7 +50,8 @@ more examples. By default, `build-fullmap` first downloads a **prebuilt** database published for this Tablassert version, a `fullmap.tar.zst` under `.../fullmap//` (the version directory is the -installed package version, never hardcoded), verified against a co-published `sha256sum.txt` and +installed package version, never hardcoded), verified against a co-published `sha256sum.txt` when that +file is present and parseable (otherwise the download proceeds with a warning) and extracted beside `--output` entirely in the Rust extension: it streams the archive through zstd decompression and untars it (the multi-GB decompressed tar is never materialized on disk) with the GIL released, extracts into a @@ -59,7 +64,7 @@ passes is atomically renamed into place (the primary to `--output`, shards besid and the command falls back to the from-scratch build below. If no prebuilt is published for this version it falls back the same way; `--force` / `-f` skips the prebuilt attempt and always builds. When the optional `[aria2]` extra is installed, the bundled aria2c from that extra accelerates -**either** download automatically (the multi-GB prebuilt archive is the ideal aria2 use case), and +**either** download automatically, and the Python downloader is used otherwise. Two facts matter most when planning a build: @@ -92,9 +97,9 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens multi-threaded allocation from bloating resident memory. - **Download**: files come from `https://stars.renci.org/var/babel_outputs` via resumable, - range-request downloads; cached files are reused. When the `[aria2]` extra is installed, only - this stage of the from-scratch pipeline uses the bundled `aria2c` binary from that extra (the - prebuilt archive download above uses it too), with aria2's segmented HTTP downloads and + range-request downloads; cached files are reused. When the `[aria2]` extra is installed, both + this stage and the prebuilt archive download use the bundled `aria2c` binary, with aria2's + segmented HTTP downloads and retry/resume control files while suppressing aria2's own progress UI so Tablassert's progress bar stays clean. The progress detail remains file-level (`aria2c downloading`) rather than byte-level in that mode. @@ -109,7 +114,7 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens assigns compact integer CURIE IDs via a dedup map keyed on `xxh3_128(curie)`, and accumulates postings mapping each normalized term to its (CURIE, source) pairs. Each worker drains its per-CURIE rows and term postings to bounded on-disk spill runs once its buffer fills, the term postings partitioned - per-shard at spill time (each `run_s{shard}_{id}.bin` holds only terms with + per-shard at spill time (each `run_s{shard}_{run_id:08}.bin` holds only terms with `xxh64(term) & (shards-1)` matching that shard), which keeps peak RAM flat regardless of input size and lets the write phase merge each shard independently. Dead terms (purely numeric, or generic labels like `none`/`nan`/`null`) are skipped, since they can never be queried. @@ -121,11 +126,11 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens runs and inserting the merged term groups inline into that shard's redb file (one database per shard, since redb allows a single writer per file) in hash-sorted batches for near-sequential B-tree appends (each batch is appended through redb's end-of-table cursor API, the faster ascending - bulk-load path; the engine's ascending-key page optimization also lets a key-order-loaded shard - occupy about half as many pages, so current builds write ~50% smaller shard files at the same + bulk-load path; loading each shard in ascending key order also lets the engine pack its pages + densely, so current builds write smaller shard files at the same schema). Because every term's postings already live in its own shard's runs, each merge groups a - term completely with no cross-shard coordination; as the merge+insert is the bottleneck, - `shard_count` writers deliver ~Nx single-threaded write throughput. + term completely with no cross-shard coordination, and one writer per shard keeps the merge+insert + bottleneck off the critical path. ### Build Tunables (environment) @@ -134,7 +139,7 @@ override only when targeting an unusual machine. | Variable | Default | Description | |----------|---------|-------------| -| `TABLASSERT_FULLMAP_EXCLUDE_PREFIXES` | *(empty)* | Comma-separated CURIE prefixes to drop at build time (e.g. `Publication`). Excluding prefixes you never resolve dramatically cuts build time, peak memory, and database size. InChIKey terms are retained unless excluded explicitly. | +| `TABLASSERT_FULLMAP_EXCLUDE_PREFIXES` | *(empty)* | Comma-separated CURIE prefixes to drop at build time (e.g. `InChIKey`, `CAID`). Dropping prefixes you never resolve cuts build time, peak memory, and database size. InChIKey terms are retained unless excluded explicitly. | | `TABLASSERT_FULLMAP_CHUNK_BYTES` | `8388608` (8 MiB) | Byte budget per producer-to-worker line-chunk. Bounded by bytes (not line count) so chunk memory is fixed even for large synonym records. | | `TABLASSERT_FULLMAP_PRODUCERS` | `clamp(workers/4, 4, #files)` | Number of producer (decompressor) threads. Decompression far outpaces parallel processing, so a handful keeps all workers fed. | | `TABLASSERT_FULLMAP_LOCAL_SPILL_ENTRIES` | `1000000` | Per-worker term-posting buffer size before spilling a sorted run to disk. Lower values use less RAM but produce more run files. | @@ -172,7 +177,7 @@ missing tag raises rather than silently reading incompatible data. Databases bui (the v3 to v4 move to sharded files, the v4 to v5 move to the redb 4 engine, and the v5 to v6 move to normalized level-one keys) requires rebuilding via `tablassert build-fullmap`. -Readers open every fullmap file READ-ONLY with a SHARED file lock (redb >= 3 `ReadOnlyDatabase`), so any +Readers open every fullmap file READ-ONLY with a SHARED file lock (redb 4 `ReadOnlyDatabase`), so any number of processes can run lookups against the same fullmap concurrently; only a `build-fullmap` rebuild (an exclusive-lock writer) briefly blocks readers. Each lookup pins one primary-plus-shards file generation, cached handles are validated against the file's `(dev, ino)` on every use, so a reader @@ -221,7 +226,7 @@ Pass the fullmap path (file or base directory) as the `fullmap` argument to `res ## Inspecting a Fullmap `tablassert quick-map TERMS... -f ` resolves one or more terms against a built database -through the exact op chain a build runs per node column, printing one `rich` table per term with +through the exact op chain a build runs per node column, printing one `rich` table per matched term with its ranked matches (`CURIE`, `PREFERRED_NAME`, `CATEGORY_NAME`, `TAXON_ID`, `SOURCE_NAME`, `NLP_LEVEL`, `PR`) and the normalized probe keys the term became. Its flags mirror the `NodeEncoding` config fields (`--taxon`, `--prioritize`, `--avoid`, `--exclude-prefixes`, diff --git a/docs/index.md b/docs/index.md index 7191712..114ea20 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,8 +3,8 @@ Tablassert turns biomedical tables (Excel, CSV, TSV) into NCATS Translator-compliant KGX knowledge graphs, declaratively, with entity resolution built in and optional quality control. -**Installing?** See [Installation](installation.md). **First build?** Follow the -[Tutorial](tutorial.md). **Automating?** Use the [CLI](cli.md). +New to Tablassert? Start with [Installation](installation.md), work through the +[Tutorial](tutorial.md), or go straight to the [CLI](cli.md) reference. ## Who this is for diff --git a/docs/installation.md b/docs/installation.md index f1fa93f..690b4eb 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -23,7 +23,7 @@ pip install uv ### Method 1: Install from PyPI -Recommended for most users. The base install exposes Tablassert's Python API; add `[cli]` to install the `tablassert` command. QC and other extras are opt-in. +Recommended for most users. Install into your project's environment to use the Python API (`pip install "tablassert"` or `uv add tablassert`), or install the standalone command-line tool with `uv tool install "tablassert[cli]"`. QC and other extras are opt-in. ```bash uv tool install "tablassert[cli]" # or: pip install "tablassert[cli]" @@ -44,22 +44,22 @@ Install any extra the same way: `uv tool install "tablassert[]"` or Linux and macOS; elsewhere pip builds from the sdist, which needs a Rust toolchain. !!! note "`[aria2]` platform and license notes" - The `[aria2]` extra depends on the PyPI `aria2` package, which imports as `aria2c` and bundles a static aria2c binary. Its wheels are available for Linux and Windows only; the extra ships no macOS wheels, so a normal macOS install resolves to Tablassert's Python downloader. + The `[aria2]` extra depends on the PyPI `aria2` package, which imports as `aria2c` and bundles a static aria2c binary. Its wheels are available for Linux and Windows only; the extra ships no macOS wheels, and the sdist's build hook fails on macOS, so do not install `[aria2]` there: `build-fullmap` falls back to Tablassert's Python downloader. The bundled aria2c dependency is GPL-2.0. Tablassert remains Apache-2.0 and does not vendor aria2c, but redistributors who ship the optional extra should review GPL-2.0 obligations. Excel (`.xlsx`) input is read through Polars' `calamine` engine, which ships with the base install -(`fastexcel`). A handful of workbooks calamine rejects are readable by the pure-Python fallback -engine: `pip install openpyxl`. +(`fastexcel`). Workbooks that calamine cannot parse are rejected with its error; there is no fallback +engine. #### When an extra is missing Reaching a feature whose extra was never installed is a normal, recoverable mistake, so Tablassert -never lets it surface as a bare `ModuleNotFoundError`. Every one of these paths reports the absent -distribution **and** the command that fixes it: +never lets it surface as a bare `ModuleNotFoundError`. Every one of these paths names the absent +distribution **and** the command that fixes it, in this shape (the first sentence lists the missing +distributions; the second is the install command): ```text -Missing optional dependencies 'scikit-learn', 'sentence-transformers', required by the QC audit. Install the [qc] extra: pip install "tablassert[qc]" (uv: uv tool install "tablassert[qc]") ``` diff --git a/docs/tutorial.md b/docs/tutorial.md index c10a74d..40f707c 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -2,8 +2,7 @@ **By the end of this tutorial you will have built a KGX-compliant knowledge graph from a CSV of gene-disease associations**: nodes and edges with standardized CURIEs, biolink categories, provenance, -and statistical annotations, ready for NCATS Translator. You'll learn the complete workflow: creating -configurations, running Tablassert, and examining the output. +and statistical annotations, ready for NCATS Translator. The input files ship under `docs/examples/` and are validated against the live schema by the test suite; the configurations below reproduce them exactly. @@ -19,7 +18,7 @@ scratch (see [Fullmap](fullmap.md) for sizes, disk requirements, and build tunab - The fullmap entity-resolution database. Build or download it once: ```bash - tablassert build-fullmap --output ./fullmap/data/fullmap.redb + tablassert build-fullmap ``` Then point the graph config's `fullmap:` field at `./fullmap` in Step 3. See @@ -173,9 +172,9 @@ head -n 3 tutorial-output/TUTORIAL_KG_1.0.0.nodes.ndjson Example output: ```json -{"id":"HGNC:11998","name":"TP53","category":["biolink:Gene"],"taxon":"NCBITaxon:9606"} -{"id":"MONDO:0008903","name":"lung cancer","category":["biolink:Disease"]} -{"id":"HGNC:1100","name":"BRCA1","category":["biolink:Gene"],"taxon":"NCBITaxon:9606"} +{"id":"HGNC:11998","name":"TP53","category":["biolink:Gene"],"in_taxon":["NCBITaxon:9606"],"provided_by":["infores:tutorial-kg"]} +{"id":"MONDO:0008903","name":"lung cancer","category":["biolink:Disease"],"in_taxon":["NCBITaxon:9606"],"provided_by":["infores:tutorial-kg"]} +{"id":"HGNC:1100","name":"BRCA1","category":["biolink:Gene"],"in_taxon":["NCBITaxon:9606"],"provided_by":["infores:tutorial-kg"]} ``` **Edges file:** @@ -186,10 +185,11 @@ head -n 2 tutorial-output/TUTORIAL_KG_1.0.0.edges.ndjson Example output (p-values are emitted as controlled scientific-notation strings and the derived significance band as a bare enum token; the study size is `Study.study_size` metadata on the -inlined supporting study, anchored to the source row by the `row:` StudyResult): +inlined supporting study, anchored to the source row by the `row:` StudyResult; records are +abridged here, omitting provenance fields such as `knowledge_level`, `agent_type`, and `sources`): ```json -{"id":"2cfea591-0f8f-33af-a7df-03da531d3359","subject":"HGNC:11998","predicate":"biolink:associated_with","object":"MONDO:0008903","p_value":"1.0000e-03","statistical_significance_qualifier":"strongly_significant","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"tutorial-data.csv","study_size":450,"has_study_results":[{"id":"row:2"}]}},"publications":["PMID:12345678"]} -{"id":"7b1c9d02-5e8a-4f3b-9c1d-2a6e8f0b4d7c","subject":"HGNC:1100","predicate":"biolink:associated_with","object":"MONDO:0005041","p_value":"1.0000e-04","statistical_significance_qualifier":"very_strongly_significant","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"tutorial-data.csv","study_size":1200,"has_study_results":[{"id":"row:3"}]}},"publications":["PMID:12345678"]} +{"id":"2cfea591-0f8f-33af-a7df-03da531d3359","subject":"HGNC:11998","predicate":"biolink:associated_with","object":"MONDO:0008903","p_value":"1.0000e-03","statistical_significance_qualifier":"very_strongly_significant","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"tutorial-table.yaml SECTION 1","study_size":450,"has_study_results":[{"id":"row:2"}]}},"publications":["PMID:12345678"]} +{"id":"7b1c9d02-5e8a-3f3b-9c1d-2a6e8f0b4d7c","subject":"HGNC:1100","predicate":"biolink:associated_with","object":"MONDO:0005041","p_value":"1.0000e-04","statistical_significance_qualifier":"very_strongly_significant","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"tutorial-table.yaml SECTION 1","study_size":1200,"has_study_results":[{"id":"row:3"}]}},"publications":["PMID:12345678"]} ``` **RIG file:** diff --git a/llms.txt b/llms.txt index 1141769..a397f5a 100644 --- a/llms.txt +++ b/llms.txt @@ -24,7 +24,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu ## CLI and Runtime - [Fullmap Guide](docs/fullmap.md): entity-resolution database build pipeline and redb schema. - [CLI Entry Point](src/tablassert/cli_entry.py): console-script launcher (`main()`) and the `[cli]` runtime preflight. -- [CLI Module](src/tablassert/cli.py): the six command implementations and the build pipeline stages. +- [CLI Module](src/tablassert/cli.py): the five command implementations and the build pipeline stages. ## API Reference - [Entity Resolution](docs/api/fullmap.md): the lower-level `resolve()` and `quick_map()` API. @@ -46,7 +46,7 @@ Current CLI behavior queries an embedded redb database whose primary file is `fu - [Coercion Helpers](src/tablassert/coerce.py): annotation and value coercion helpers. - [Enums Catalog](src/tablassert/enums.py): Tablassert-owned configuration vocabularies (`Tokens`, `Repositories`, `InformationResources`, `Contributions`, `Comparisons`, `Functions`, `Files`, `EncodingMethods`, `FillMethods`, `SourceStatuses`, `ProvisionMechanisms`, `DataFormats`, `IngestCategories`, `ContentCategories`, `ModelingCategories`). - [Error Types](src/tablassert/errors.py): coded exceptions and validation warnings. -- [Extras Registry](src/tablassert/extras.py): the package -> extra mapping behind the missing-dependency errors and their install hints. +- [Extras Registry](src/tablassert/extras.py): the package-to-extra mapping behind the missing-dependency errors and their install hints. - [Fullmap Module](src/tablassert/fullmap.py): embedded redb synonym resolution and ranking logic. - [YAML Ingestion](src/tablassert/ingests.py): `from_yaml()`, `to_sections()`, template/section merge behavior, YAML loading, and section expansion. - [InfoRes Registry](src/tablassert/infores_registry.py): bundled infores registry snapshot loader, KGX/RIG CURIE collection, and three-way registered/allowed/unregistered classification. diff --git a/tests/test_docs_style.py b/tests/test_docs_style.py new file mode 100644 index 0000000..e4b0333 --- /dev/null +++ b/tests/test_docs_style.py @@ -0,0 +1,75 @@ +"""Typography style gate for user-facing documentation. + +Guards the docs-overhaul fixes so they cannot regress: non-ASCII characters, +prose arrows, em-dash substitutes, and mid-prose double spaces are rejected in +user-facing prose. Inline code spans and fenced code blocks are exempt because +they carry real CLI flags, error text, and syntax where such characters are +legitimate. + +To permit a new exception, add the exact character to ALLOWED_NON_ASCII with a +comment naming its source. +""" + +import re +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent + +DOC_FILES = [ + REPO_ROOT / "README.md", + REPO_ROOT / "CHANGELOG.md", + REPO_ROOT / "CONTRIBUTING.md", + REPO_ROOT / "CODE_OF_CONDUCT.md", + REPO_ROOT / "llms.txt", + *sorted((REPO_ROOT / "docs").rglob("*.md")), +] + +# Exact non-ASCII characters permitted in docs prose, mapped to their source. +ALLOWED_NON_ASCII = {"ê": "Gwênlyn Glusman, author name in CITATION.cff"} + +_CODE_SPAN = re.compile(r"`[^`]*`") +_ARROW = re.compile(r"->|=>|<-|→|←|⟶|⟵|↔|⇒") +_EMDASH_SUBSTITUTE = re.compile(r"\S -- \S|\S --$|^-- \S") +_DOUBLE_SPACE = re.compile(r"\S +\S") + + +def _prose_lines(text: str) -> list[tuple[int, str]]: + """Yield (line number, line) for non-fence lines, with inline code spans blanked.""" + prose: list[tuple[int, str]] = [] + in_fence = False + for n, line in enumerate(text.splitlines(), 1): + if line.strip().startswith("```"): + in_fence = not in_fence + continue + if in_fence: + continue + prose.append((n, _CODE_SPAN.sub("\ue000", line))) + return prose + + +def test_no_unlisted_non_ascii() -> None: + for path in DOC_FILES: + text = path.read_text(encoding="utf-8") + offenders = sorted({c for c in text if ord(c) > 127 and c not in ALLOWED_NON_ASCII}) + assert not offenders, f"{path.name}: unlisted non-ASCII characters {offenders!r}" + + +def test_no_prose_arrows() -> None: + for path in DOC_FILES: + for n, line in _prose_lines(path.read_text(encoding="utf-8")): + hit = _ARROW.search(line) + assert hit is None, f"{path.name}:{n}: prose arrow {hit.group(0)!r}: {line.strip()[:120]}" + + +def test_no_em_dash_substitutes() -> None: + for path in DOC_FILES: + for n, line in _prose_lines(path.read_text(encoding="utf-8")): + assert not _EMDASH_SUBSTITUTE.search(line), f"{path.name}:{n}: space-flanked -- in prose: {line.strip()[:120]}" + + +def test_no_mid_prose_double_spaces() -> None: + for path in DOC_FILES: + for n, line in _prose_lines(path.read_text(encoding="utf-8")): + if line.startswith((" ", "|")): # list continuations and table rows + continue + assert not _DOUBLE_SPACE.search(line), f"{path.name}:{n}: double space in prose: {line.strip()[:120]}"