feat!: re-home source record URLs onto the upstream entry and name inlined Study after config section

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 All notable changes to this project are documented in this file.
 
+## Unreleased
+
+### Breaking Changes
+- **Derived-path source record URLs now ride the upstream repository's supporting `sources` entry, not the primary.** Sections without a `provenance.override` emit their `source.url` values as `source_record_urls` on the supporting entry matching the publication repository (`infores:pubmed-central` for PMC sections, `infores:pubmed` for PMID sections), because the primary knowledge source is the transforming resource, not the downloadable record; the primary entry keeps `upstream_resource_ids` and, with `build-kg --linkouts`, the per-edge linkout template (`{edge_id}` still resolves in the post-dedup sweep). Override sections are byte-identical: `override.upstream_source_record_urls` re-homes exactly as specified, other overrides keep the legacy primary-entry emission, and `--linkouts` still never injects into override sections. Downstream consumers keying off the primary entry's `source_record_urls` must read the supporting entry instead. ([#220](https://github.com/SkyeAv/Tablassert/pull/220))
+  **Migration:** read record URLs from the `supporting_data_source` entry whose `resource_id` matches the publication repository; no config change is required.
+- **The inlined supporting Study is named after the table config file and section index.** `Study.name` reads `<table-config.yaml> SECTION <n>` (1-based position within that file, stamped by `ingests.to_sections`) instead of the worksheet name or source data filename, so edge provenance points back to the exact configuration; `Study.id` is unchanged (publication CURIE, or the config stem when unidentified). Section cache parquets content-address the new stamp, so pre-upgrade caches miss once and rebuild. ([#220](https://github.com/SkyeAv/Tablassert/pull/220))
+
 ## 20.0.0 - 2026-10-14
 
 ### Breaking Changes

docs/configuration/advanced-example.md

@@ -103,7 +103,7 @@ whole table or `method: column` to pull per-row notes from the source (see
 top-level edge fields (`p_value` in controlled scientific-notation string form, `effect_size` as a real
 JSON number since biolink-model 4.4.4). Study metadata (`study_size`, here declared as a constant)
 lands on the edge's inlined supporting `Study`; the `Study.id` is the publication CURIE, `Study.name`
-the worksheet, and the single `StudyResult` anchors the source row as its `row:<N>` id. Any
+the table config file and 1-based section index (e.g. `KANG12.yaml SECTION 2`), and the single `StudyResult` anchors the source row as its `row:<N>` id. Any
 non-Biolink-slot name such as `assertion_method`, `multiple_testing_correction_method`, and
 `miscellaneous_notes` folds into the edge's `supporting_text` list as `"name: value"` entries
 (sorted alphabetically):

docs/configuration/table.md

@@ -90,7 +90,7 @@ Defines the data file location and format.
 |-------|------|----------|-------------|
 | `kind` | String | No | Source kind. Model default is `"excel"`, but specify it explicitly in configs. |
 | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
-| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
+| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
 | `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. |
 | `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
 | `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
@@ -115,7 +115,7 @@ source:
 |-------|------|----------|-------------|
 | `kind` | String | No | Source kind. Model default is `"text"`, but specify it explicitly in configs. |
 | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
-| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
+| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
 | `delimiter` | String | No | Field delimiter. Defaults to `","`. |
 | `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
 | `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
@@ -146,7 +146,7 @@ as-is.
 |-------|------|----------|-------------|
 | `kind` | String | No | Source kind. Model default is `"parquet"`, but specify it explicitly in configs. |
 | `local` | Path | Yes | Local file path the source is read from. The file must already exist here; Tablassert does not download it. |
-| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the primary `sources` entry's `source_record_urls` list and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
+| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (derived sections emit them as the upstream repository supporting entry's `source_record_urls` list, infores:pubmed-central for PMC sections, and in the RIG; when `provenance.override.upstream_source_record_urls` is set, RIG only; the per-upstream mapping determines edge placement instead). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
 | `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
 | `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
 | `reindex` | List[Reindex] | No | Conditional row filtering |
@@ -522,7 +522,7 @@ Override fields:
 | `knowledge_level` | String | No | Override-specific KL value. Defaults to `statistical_association`. |
 | `agent_type` | String | No | Override-specific AT value. Defaults to `data_analysis_pipeline`. |
 
-Tablassert emits the graph-level infores (or `infores:<graph-name>` when unset) as the primary entry of the Biolink `sources` list on each edge, `{resource_id: "infores:multiomics-kg", resource_role: "primary_knowledge_source", upstream_resource_ids: [...], source_record_urls: [...]}`, with one additional `supporting_data_source` entry per upstream. When `override.upstream_source_record_urls` is set, the primary entry emits no `source_record_urls` and each mapped supporting entry carries its own instead. When `override.sources` is set, that derivation is skipped entirely and the template entries are emitted verbatim, in order. No flat `primary_knowledge_source` scalar is emitted: current translator-ingests practice carries retrieval provenance only in `sources`, and the Biolink `RetrievalSource` class is where `resource_id` / `upstream_resource_ids` / `source_record_urls` are defined. Each retrieval-source entry uses `resource_id` as its sole identifier. Without `override.sources`, the override cannot set a per-section primary source; manual infores CURIEs belong in `upstream_resource_ids`. Older flat `resource_id` / `primary_knowledge_source` output has been removed so generated KGX matches the Biolink edge contract.
+Tablassert emits the graph-level infores (or `infores:<graph-name>` when unset) as the primary entry of the Biolink `sources` list on each edge, `{resource_id: "infores:multiomics-kg", resource_role: "primary_knowledge_source", upstream_resource_ids: [...]}`, with one additional `supporting_data_source` entry per upstream. On the derived path the primary entry emits no section record URLs: the section's `source.url` values ride on the upstream repository's supporting entry (infores:pubmed-central for PMC sections), because that is where the downloadable record lives, and `build-kg --linkouts` puts only the per-edge linkout template on the primary entry. When `override.upstream_source_record_urls` is set, the primary entry emits no `source_record_urls` and each mapped supporting entry carries its own instead; sections with any other `provenance.override` keep the legacy primary-entry emission. When `override.sources` is set, that derivation is skipped entirely and the template entries are emitted verbatim, in order. No flat `primary_knowledge_source` scalar is emitted: current translator-ingests practice carries retrieval provenance only in `sources`, and the Biolink `RetrievalSource` class is where `resource_id` / `upstream_resource_ids` / `source_record_urls` are defined. Each retrieval-source entry uses `resource_id` as its sole identifier. Without `override.sources`, the override cannot set a per-section primary source; manual infores CURIEs belong in `upstream_resource_ids`. Older flat `resource_id` / `primary_knowledge_source` output has been removed so generated KGX matches the Biolink edge contract.
 
 #### Explicit sources template
 
@@ -607,7 +607,7 @@ This means nothing in your source data is silently dropped: context that doesn't
 
 In addition to user-declared annotations, every edge automatically carries `extracted_from_row_number`, a 1-based index into the original source table (matching Excel-style row numbering). It is not declared as an annotation. Tablassert emits it internally so each edge always carries its source-row provenance, and the inlined supporting study consumes it: the `StudyResult` that anchors the study's row is identified by the scoped CURIE `row:<N>`. The row and sheet columns are never folded into `supporting_text`.
 
-The supporting study's identity is disjoint by construction: `Study.id` is the section's publication CURIE (`PMID:123`, `PMCID:PMC11708054`); `Study.name` is a human label, the worksheet name for spreadsheet sources and the source filename otherwise. It is never a composition of the two. A section declaring **no `publications`** falls back to the config stem (the YAML filename without its extension) as the id, and the struct is then emitted **only when it carries something to preserve**, such as study metadata, routed unsatisfiable values, or class-pruned values. Biolink defines `has supporting studies` as "studies that produced information used as evidence", so a contentless unpublished section emits no `has_supporting_studies` at all; a section with a real publication always keeps the wrapper:
+The supporting study's identity is disjoint by construction: `Study.id` is the section's publication CURIE (`PMID:123`, `PMCID:PMC11708054`); `Study.name` is a human label naming the originating table config file and the section's 1-based position within it (e.g. `KANG12.yaml SECTION 2`). It is never a composition of the two. A section declaring **no `publications`** falls back to the config stem (the YAML filename without its extension) as the id, and the struct is then emitted **only when it carries something to preserve**, such as study metadata, routed unsatisfiable values, or class-pruned values. Biolink defines `has supporting studies` as "studies that produced information used as evidence", so a contentless unpublished section emits no `has_supporting_studies` at all; a section with a real publication always keeps the wrapper:
 
 ```json
 "has_supporting_studies": {

src/tablassert/cli.py

@@ -420,12 +420,12 @@ def build_graph_pipeline(
                 advance()
         with RunCache() as cache:
             for planned in plan.execution_order:
-                section_index, x, op = pending[planned]
+                flat_index, x, op = pending[planned]
                 entry: PlanEntry = plan.entries[planned]
                 start(format_section_compact(x))
                 # on_phase drives the per-op sub-step indicator (cache → resolve → write ...);
                 # resume/snapshot are the producer/consumer seams the plan assigned this section.
-                results[section_index] = compile_subgraph(
+                results[flat_index] = compile_subgraph(
                     op, on_phase=sub_step, resume=_run_cache_resume(cache, entry), snapshot=_run_cache_snapshot(cache, entry)
                 )
                 advance()

src/tablassert/ingests.py

@@ -82,12 +82,14 @@ def to_yaml(p: Path, data: object) -> None:
         yaml.dump(data, f, Dumper=_IndentedSafeDumper, sort_keys=False, default_flow_style=False, allow_unicode=True, width=120)
 
 
-def to_sections(instructions: dict[str, Any], table: Path) -> list[list[dict[str, Any]]]:
+def to_sections(instructions: dict[str, Any], table: Path) -> list[dict[str, Any]]:
     """Expand a table config dict into a list of section dicts.
 
     Each section is the deep merge of the table-level ``template`` over a
     per-section entry. The originating ``table`` path is stamped into each
-    section as ``config``.
+    section as ``config``, and the section's 1-based position in the file
+    (document order) as ``section_index`` so provenance can name the inlined
+    Study after its origin ("<table config> SECTION <n>").
 
     Args:
         instructions: Parsed YAML config with optional ``template`` and
@@ -100,4 +102,7 @@ def to_sections(instructions: dict[str, Any], table: Path) -> list[list[dict[str
     template: dict[str, Any] = instructions.get("template", {})
     template["config"] = table
     sections: list[dict[str, Any]] = instructions.get("sections", [{}])
-    return [fastmerge(deepcopy(template), x) for x in sections]
+    merged: list[dict[str, Any]] = [fastmerge(deepcopy(template), x) for x in sections]
+    for index, section in enumerate(merged, start=1):
+        section["section_index"] = index
+    return merged