Skip to content

fix(chip): state the CHIP detection limit from real per-base depth, not the average - #449

Open
geiresx wants to merge 454 commits into
GeiserX:mainfrom
geiresx:pgp/i411-chip-read-support-limit
Open

geiresx wants to merge 454 commits into
GeiserX:mainfrom
geiresx:pgp/i411-chip-read-support-limit

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #411: claim trait-59, with owner decision D10 (a coding base is "callable" at 10 reads or more).

The problem

Step 49 gave each CHIP gene one detection limit, worked out from the gene's mean depth. A gene with a fifth of its coding bases unread then read covered, with almost the same limit as a gene read evenly. On main, DNMT3A with 8 exons at 33x and 2 exons at 0x gives 26.4 0.157 covered, and 27x everywhere gives 27.0 0.152 covered. The step line and both reports then said a clone above that VAF "is found 8 times in 10", from the limit at the median depth. A dark exon stayed hidden, and the wording overstated a read count as the caller's sensitivity.

The fix

  • og chip report keeps each gene's per-base depths (bases with no pileup line at 0). The limit is the smallest VAF at which a clone shows 3 reads with the change 8 times in 10, averaged over the gene's coding bases, each at its own depth. With one depth everywhere it equals the old limit.
  • A gene where no VAF reaches that power has no limit: partly_covered when some bases have 3 reads, too_shallow when none do. A gene with a fifth of its bases or more under 3 reads cannot reach 8 in 10.
  • The stated limit of a run is the median of the covered genes' own limits. The step line, og summary collect's median_limit and so both reports use it. Before, it was the limit at the median depth.
  • chip_depth.tsv gets two columns at the end: callable_fraction (share of coding bases with 10 reads or more, D10) and assessable_fraction (share of coding bases where a clone at the stated limit shows its reads). The first seven columns keep their names and order. summary.json's chip genes carry the two columns when the table has them.
  • The step line now reads "Read-support limit ... shows at least 3 reads with the change 8 times in 10, counted base by base. Mutect2 and its filters call fewer than that." It names each gene with no limit, with its share of bases read deep enough.
  • detection_limit_vaf is written rounded up at the third decimal, so a clone at the written VAF reaches 8 in 10. At 30x it reads 0.138; rounded to the nearest it read 0.137, where the power falls just short. The stated limit inside og chip report is the median of these written limits, the same figure the step line and og summary collect read back, so assessable_fraction is counted at the figure the reports show.
  • A coding base that two genes of the BED share counts for the limit and both shares of each gene. Before, it counted only for the last gene in the BED, so the earlier gene read it as depth 0 and could drop to partly_covered. The CHIP genes' exons do not overlap, so this matters only for a BED with overlapping or padded rows.

Output changes, as 1.0.0 allows: two columns added to chip_depth.tsv, a new partly_covered status, new detection_limit_vaf and median_limit values (rounded up at the third decimal), and new wording in the step line. Nothing is renamed or removed.

Decisions taken in this lane:

  • The stated limit is the median of the per-gene limits, not a limit pooled over all bases. og summary collect only sees chip_depth.tsv, so this keeps the step line, summary.json and the reports on one figure.
  • assessable_fraction uses the run's stated limit, so it is empty when no gene is covered.
  • A shared base still counts in the mean depth of only the last gene in the BED, as before. The mean column keeps its old values; only the per-base limit and the shares count the base for every gene.
  • The report and live-page wording stay with the display lanes (i411-display-traits, i411-display-live). They read the same median_limit, now the honest figure.

Files outside the package list, each needed for CI to stay true: tests/parity/golden/chip/report-{calls,table,overlap,empty}/ (deliberate-fix goldens), tests/test_chip_report.py (3 checks pinned the old mean-depth behaviour), and modules/local/chip/main.nf (the stub header gets the two columns).

How it was checked

  • Red first: rust/crates/og-chip/tests/i411_chip_read_support_limit.rs on the test-only commit c28e011 fails all 4 tests in CI: https://github.com/GeiserX/OwnGenome/actions/runs/43744441741/job/132904521191 (main writes covered where partly_covered is due, 0.137 where 0.159 is due, and no share columns). The same tests pass on the fix.
  • Red first for shared bases: red_first_a_base_two_genes_share_is_read_for_both in the same file, run against the report code of bab3a34 (before the shared-base fix), fails with left: "partly_covered", right: "covered" for DNMT3A. It passes on 2c0715e. The report-overlap parity golden changes with it: DNMT3A now reads covered with all its coding bases callable.
  • New unit tests in og_chip::binom: one depth everywhere equals the old limit to 1e-12, an unread fifth leaves no limit (the power at a VAF near 1 rounds to exactly 1, so the share of deep bases is tested on its own), and the readable-at-a-VAF rule.
  • Parity, ORACLE_RUNNER=local scripts/ci/oracle-diff.sh chip summary-collect on the head after the main merge: all 352 items in 89 cases agree, with goldens for the deliberate fixes. The whole-file cases all and all-history get goldens for the new chip genes keys. The c generator copies the full folder with cp -Rp and gives each table the float cases rewrite the fixture's time, so those goldens do not depend on the clock (a first CI run caught two dates of the clock: https://github.com/GeiserX/OwnGenome/actions/runs/43747267494).
  • tests/test_chip_report.py, test_live_results.py, test_live_widgets.py and test_render_report.py pass with the new og.
  • chip-clonal-1 e2e: GATK's image is amd64-only and cannot run on this box, so I ran the depth half on public data. I took GIAB HG002 60x reads over the MANE coding exons of the six CHIP genes on the fixture contigs (pileup at MQ 20, BQ 10, duplicates out) and ran og chip report. DNMT3A: mean 52.7x, limit 0.081 (covered, under the case's 10%), callable_fraction 1.000, assessable_fraction 0.293. The case's new checks pass on that output and fail on main's output (no share columns, old "Detection limit" line). The full case runs in the e2e train.
  • Speed and memory, /usr/bin/time -l: on the HG002 input (21,277 depth lines) both old and new take 0.01 to 0.06 s and 3.2 MB. On 2 Mb of synthetic depth up to 5,000x, both take 0.50 s; peak memory goes from 5.0 MB to 5.3 MB.
  • Fast checks on the head 2c0715e: Guard with ci-ok, Rust, Lint, Nextflow, Docs, Container Test and Personal data are green. Guard: https://github.com/GeiserX/OwnGenome/actions/runs/43848791630 Rust: https://github.com/GeiserX/OwnGenome/actions/runs/43848791286
  • Red first for the written limit: uniform_30x_states_the_binomial_limit_and_says_what_it_is now pins 0.138 and the power on both sides of it. Against the report code of 2c0715e it fails with left: "0.137", right: "0.138", and the two other tests that pin a written limit fail the same way (0.125 where 0.126 is due). All pass on 65baf9e. The parity goldens of og chip report do not change: their limits already sat just under a whole thousandth.
  • The full chip-clonal-1 e2e case has not yet run green in CI. It waits for the e2e train.
  • cargo fmt --check, cargo clippy --workspace --all-targets -D warnings and cargo doc -D warnings are clean.

Beads: trait-59, D10.

Summary by CodeRabbit

  • New Features
    • CHIP reports now calculate read-support limits using depth across each coding base, including bases with no reads.
    • Reports show callable and assessable fractions and distinguish genes with complete, partial, insufficient, or absent coverage.
    • The run-wide read-support limit is now based on the median of covered genes’ limits. Reports also identify genes without a usable limit.
  • Documentation
    • Updated CHIP guidance explains coverage statuses, reported fractions, and how read-support limits are calculated and interpreted.

Red-first tests for decision D10: a call below GQ 20, or one the caller
filtered, at APOE, HFE, SERPINA1 or a traits-table row reads low_gq, and
the summary counts low_gq rows. They fail on main, which reads such calls
as called.

Bead: trait-01
Bead: trait-67
Bead: D10
… refuse a licence outside the allow list

Red first: against the scaffold stub all four tests fail (the verb says
"not ported yet").

Bead: lic-02
CPIC sets no heteroplasmy level at which aminoglycosides are safe, so a
named MT-RNR1 allele at AF 0.05 or more is now the call and the log gives
its level. Below 0.05 it is held back as below the detection floor, and
no Reference is passed.

Bead: pgx-04
Bead: D10
…rong

A filtered 0/0, a 0/0 at depth 1, a 0/0 without GQ, a haploid 1, a
haploid 0 and a ./. record each get a test of the words they must read.
All six fail on the current reader. og clinvar deep-intronic also takes
--min-dp, the step's depth floor (default 10), unused until the fix.

Bead: clin-14
…ost's

Docker Desktop and colima run the containers in a VM that is usually
smaller than the host, but run-all.sh gave Nextflow the host's CPUs and
RAM, so Nextflow booked tasks the VM could not hold and they died with
exit 137. When docker info reports less, run-all.sh now passes the
engine's figures as the default --max_cpus and --max_memory and writes
them to <sample>/nextflow/executor.config, passed with -c before the
user's options. On native Linux nothing changes. validate-setup.sh adds
an INFO line in that case. The launcher e2e case checks it with a docker
shim that reports 8 GB and 2 CPUs, with a negative control.

Bead: plat-18
…wn page

docs/interpreting-results.md held every chapter in one 883-line file. Each
chapter now has a page under docs/results/, moved word for word. The
overview keeps the Before You Look bullets, the Before You Panic table, the
example outputs and the general advice, and lists the chapter pages. Every
old #anchor on the overview lands on a link to the section's new place.

The root .gitignore drops every results/ folder (Nextflow output), so the
seven pages are added with -f; once tracked, git follows their edits.

The repeat table moved with its chapter, so tests/test_str_ranges.py reads
it from docs/results/structural-and-repeats.md.
…sites

One state table decides each site's words. Only FILTER PASS, '.' and
RefCall leave a reference call plain; any other FILTER makes it
"ref (low confidence)" and names the FILTER. A reference call with no GQ
is low confidence too, since nothing checked it against the floor. A
depth below the step's depth floor reads "ref (shallow)". On chrX and
chrY outside the pseudoautosomal regions a one-copy 1 is "hemizygous"
and a one-copy 0 a reference call; elsewhere a one-copy GT is no call,
with why. A no-call record says what it holds (no genotype, not a
reference call) and never claims a PASS check or that no record exists.

Bead: clin-09
Bead: clin-10
Bead: clin-11
Bead: clin-12
Bead: clin-13
Bead: clin-14
…ep 03a runs without BQSR

Bead: plat-55
Bead: plat-71
og-live's ops module gives each run its hours left, memory and disk
headroom, last activity and five alerts, and writes the Prometheus text,
from og-runlog's eta and headroom instead of the Python run_eta and
run_headroom modules. og-runlog gains a fault-inject feature: named
planted mistakes the red controls turn on per thread; without the
feature none is compiled in.

Bead: D1
…ader, and count REF mismatches

Both mirrors of step 06 query each site's FILTER as a ninth column and
pass the step's depth floor (CLINVAR_MIN_DP, ext.min_dp) to
og clinvar deep-intronic. The sample's bcftools norm -c w keeps a record
whose REF is not the reference's and only warns REF_MISMATCH; the step
now counts those records in its summary and prints a notice when there
are any, since such a record cannot match a ClinVar allele.

Bead: clin-09
Bead: clin-10
Bead: clin-13
…s for the new calls

The parity goldens record the deliberate changes against the old Python:
AF 0.4 is now the call, --min-af 0.3 logs it as heteroplasmic, and a nan
AF is held back as having no allele fraction. The unit test of step 07
expects the call down to AF 0.05.

Bead: pgx-32
…peat map

The fake-docker cases fail on main: step 04c runs Sniffles2 on a
short-read BAM and exits 0, never passes --tandem-repeats, and its
closing hint sends long-read SVs to duphold for a GQ.

Bead: sv-18
Bead: sv-17
…te query in both mirrors

Step 06 runs to its summary against a fake bcftools that warns
REF_MISMATCH twice, then not at all: the count line reads 2 and then 0,
and the notice shows only for 2. The always-report call gets the depth
floor, and the query_format of the script and of the module, run with
the Groovy escapes undone, both end with %FILTER. Red on main: the count
line is missing, and the module's format has no FILTER.

Bead: clin-13
… og links

It walks cargo metadata per target from og along normal edges, checks
each crate's licence against an allow list, and writes each licence text
once, including the C library texts of crates that set links (htslib and
htscodecs from hts-sys).

Bead: lic-02
rust/og.sh build writes it in OG_BUILDER with the new og; release.yml
copies it to dist, adds it to SHA256SUMS and the attestation, and checks
the upload with check-release.sh dist.

Bead: lic-02
… warning

Step 07, the PHARMCAT module comment, the step 07 and step 20 docs and the
CHANGELOG now describe the AF 0.05 floor. The pharmcat-mito e2e case expects
the outside call and the drug recommendation at AF 0.40, and adds a sample
at AF 0.03 that stays No Result with a below-floor line.

Bead: pgx-04
Bead: pgx-32
Bead: D10
…epeat map

Step 04c ran Sniffles2 on any BAM. On short reads Sniffles2 finds no SV
and still exits 0, so the step reported success with an empty VCF. It
now reads the mean length of the first 10,000 primary reads with the
pinned samtools image and stops below 1,000 bp, naming the long-read
BAM to use.

setup.sh installs GIAB's GRCh38 tandem-repeat stratification (v3.6,
NIST data, pinned by sha256) and keeps it unpacked; step 04c passes it
as --tandem-repeats when present and says so when not. Sniffles' own
file comes from PacBio's pbsv under a licence limited to PacBio data,
so it could not serve ONT reads.

The closing hint no longer sends long-read SVs to duphold for a GQ:
Sniffles2 writes its own, and step 15 reads Manta's calls only.

Bead: sv-18
Bead: sv-17
Bead: D14
…lanted faults

Every check of tests/test_run_eta.py and tests/test_run_headroom.py now
runs on og-runlog in og-live/tests/ops_eta.rs and ops_headroom.rs, and
each of their source mutants is a named fault of the fault-inject feature
whose planted_* test must fail.

Bead: D1
… on purpose

The three clinvar-deep-intronic cases differ from the awk oracle where
the reader now says more: a ./. gVCF block has no genotype, a block
without a GQ is a low-confidence reference call, a gVCF variant record
is not a reference call (no PASS claim), and a VCF ./. without a gVCF
names the VCF record. The summary test's sites-mode check takes the new
no-genotype text. The fake-docker case gets shellcheck notes on the
module text it greps for.

Bead: clin-13
Bead: clin-14
…e REF_MISMATCH count

docs/06 lists each call word of the always-report list with when it
applies, says a no call is not a negative result, and says the summary
counts REF_MISMATCH records. The CHANGELOG names the output change.

Bead: clin-09
Bead: clin-10
Bead: clin-11
Bead: clin-12
Bead: clin-13
Bead: clin-14
…puts

append-n, rfc1, str-extra, tas2r38, mito-sam, hba-tiles, cut-haplotype
and contains write the bytes the inline Python of the e2e cases wrote,
pinned by FNV-1a hashes of the Python output.

Bead: D1
… car gets it before its PR can run

A stacked train car conflicts with main until the car before it lands, so its
pull request gets no pull_request runs until then. The landing train now runs
the fast checks on the car's own tree on the fork, while its E2E runs. Every
required check can be run that way except pull-request, which ran only on a
pull_request event.

A manual run with the new pr_base input (full shas, each an ancestor of the
ref) now runs the pull-request job too: the changed files and the added URLs
are taken against each base, and the job checks the union. Pull request runs
do the same as before. keepalive skips a pr_base run.
…ity against the Python plugin

og-live/tests/ops.rs runs every check of tests/test_owngenome_live_ops.py
on the Rust card, and compares the card (by meaning) and the metrics text
(byte for byte) with bin/owngenome_live_ops.py on the ops fixture and on
the run_eta and run_headroom fixtures. The plugin's eight source mutants
are fault-inject faults; three more plant drift the parity cases must see.

Bead: D1
GitHub reported a conflict, so no check could start. Both commits are
already in the landing train below this branch; every conflict keeps the
train's side, and the merged tree equals the branch's own tree.
The engines that replace ADMIXTURE and RFMix in step 26b need one fixed
interface to build against in parallel. og-admix now holds:

- panel: a phased panel VCF read into bit-packed haplotypes for chosen
  people at chosen sites, biallelic SNVs only; an unphased, missing or
  haploid genotype of a chosen person stops the read.
- map: the PLINK genetic map reader that chroms.rs had, plus the inverse
  (position by cM). A non-finite cM or a cM that goes down is refused.
- geno: model/ref.geno, 2-bit genotypes with a magic, version, J and N
  header, its writer and reader.
- rng: xoshiro256** seeded by SplitMix64, checked against the authors'
  reference code.
- simulate: the admixed-genome simulator with a truth BED.

og ancestry-fine chroms reads its maps with og-admix's reader. The
feature planted-red feeds every red control its broken input.

Bead: D5
Bead: D14
Bead: D1
…d its truth

The verb copies held-out panel people's haplotypes into a diploid mosaic:
ancestry tracts at T events per Morgan drawn from mu, copying switches
inside a tract. It writes an unphased VCF (BGZF when the name ends in .gz)
and a truth BED, the same bytes for the same seed. The crate docs and the
changelog name it.

Bead: D5
Bead: D14
Bead: D1
Turns the planted-red feature on by default, so Guard's cargo test runs
each control on its broken input and must show these tests fail:
rng reference vectors, map interpolation (shuffled cM), ref.geno round
trip (a flipped byte), the panel reader against bcftools (people in
another order), the tract-length law (T doubled), the truth cover (a row
dropped) and same seed same bytes (a second seed). The next commit
reverts it.

Bead: D5
… at c.803 keep its B copy

The e2e case builds a chr9 reference over ABO and runs step 41 on the
same deletion written two ways, and on a multi-allelic C>A,G 1/2. og on
the right-shifted VCF as written stays in the case as the control: it
reads A1/A1, which is what step 41 gave before it left-aligned.

Bead: trait-30
…::Options now carries

The always-report sites reader gained a depth floor (min_dp, min_dp_value) in the car before, so the struct literal in this
test no longer compiled. It now passes the CLI default, 10.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants