Repository navigation
Conversation
Red-first tests for decision D10: a call below GQ 20, or one the caller filtered, at APOE, HFE, SERPINA1 or a traits-table row reads low_gq, and the summary counts low_gq rows. They fail on main, which reads such calls as called. Bead: trait-01 Bead: trait-67 Bead: D10
… refuse a licence outside the allow list Red first: against the scaffold stub all four tests fail (the verb says "not ported yet"). Bead: lic-02
CPIC sets no heteroplasmy level at which aminoglycosides are safe, so a named MT-RNR1 allele at AF 0.05 or more is now the call and the log gives its level. Below 0.05 it is held back as below the detection floor, and no Reference is passed. Bead: pgx-04 Bead: D10
…rong A filtered 0/0, a 0/0 at depth 1, a 0/0 without GQ, a haploid 1, a haploid 0 and a ./. record each get a test of the words they must read. All six fail on the current reader. og clinvar deep-intronic also takes --min-dp, the step's depth floor (default 10), unused until the fix. Bead: clin-14
…ost's Docker Desktop and colima run the containers in a VM that is usually smaller than the host, but run-all.sh gave Nextflow the host's CPUs and RAM, so Nextflow booked tasks the VM could not hold and they died with exit 137. When docker info reports less, run-all.sh now passes the engine's figures as the default --max_cpus and --max_memory and writes them to <sample>/nextflow/executor.config, passed with -c before the user's options. On native Linux nothing changes. validate-setup.sh adds an INFO line in that case. The launcher e2e case checks it with a docker shim that reports 8 GB and 2 CPUs, with a negative control. Bead: plat-18
…lair3 is compared Bead: plat-71
…wn page docs/interpreting-results.md held every chapter in one 883-line file. Each chapter now has a page under docs/results/, moved word for word. The overview keeps the Before You Look bullets, the Before You Panic table, the example outputs and the general advice, and lists the chapter pages. Every old #anchor on the overview lands on a link to the section's new place. The root .gitignore drops every results/ folder (Nextflow output), so the seven pages are added with -f; once tracked, git follows their edits. The repeat table moved with its chapter, so tests/test_str_ranges.py reads it from docs/results/structural-and-repeats.md.
…sites One state table decides each site's words. Only FILTER PASS, '.' and RefCall leave a reference call plain; any other FILTER makes it "ref (low confidence)" and names the FILTER. A reference call with no GQ is low confidence too, since nothing checked it against the floor. A depth below the step's depth floor reads "ref (shallow)". On chrX and chrY outside the pseudoautosomal regions a one-copy 1 is "hemizygous" and a one-copy 0 a reference call; elsewhere a one-copy GT is no call, with why. A no-call record says what it holds (no genotype, not a reference call) and never claims a PASS check or that no record exists. Bead: clin-09 Bead: clin-10 Bead: clin-11 Bead: clin-12 Bead: clin-13 Bead: clin-14
…ep 03a runs without BQSR Bead: plat-55 Bead: plat-71
og-live's ops module gives each run its hours left, memory and disk headroom, last activity and five alerts, and writes the Prometheus text, from og-runlog's eta and headroom instead of the Python run_eta and run_headroom modules. og-runlog gains a fault-inject feature: named planted mistakes the red controls turn on per thread; without the feature none is compiled in. Bead: D1
…ader, and count REF mismatches Both mirrors of step 06 query each site's FILTER as a ninth column and pass the step's depth floor (CLINVAR_MIN_DP, ext.min_dp) to og clinvar deep-intronic. The sample's bcftools norm -c w keeps a record whose REF is not the reference's and only warns REF_MISMATCH; the step now counts those records in its summary and prints a notice when there are any, since such a record cannot match a ClinVar allele. Bead: clin-09 Bead: clin-10 Bead: clin-13
…s for the new calls The parity goldens record the deliberate changes against the old Python: AF 0.4 is now the call, --min-af 0.3 logs it as heteroplasmic, and a nan AF is held back as having no allele fraction. The unit test of step 07 expects the call down to AF 0.05. Bead: pgx-32
…peat map The fake-docker cases fail on main: step 04c runs Sniffles2 on a short-read BAM and exits 0, never passes --tandem-repeats, and its closing hint sends long-read SVs to duphold for a GQ. Bead: sv-18 Bead: sv-17
…te query in both mirrors Step 06 runs to its summary against a fake bcftools that warns REF_MISMATCH twice, then not at all: the count line reads 2 and then 0, and the notice shows only for 2. The always-report call gets the depth floor, and the query_format of the script and of the module, run with the Groovy escapes undone, both end with %FILTER. Red on main: the count line is missing, and the module's format has no FILTER. Bead: clin-13
… og links It walks cargo metadata per target from og along normal edges, checks each crate's licence against an allow list, and writes each licence text once, including the C library texts of crates that set links (htslib and htscodecs from hts-sys). Bead: lic-02
rust/og.sh build writes it in OG_BUILDER with the new og; release.yml copies it to dist, adds it to SHA256SUMS and the attestation, and checks the upload with check-release.sh dist. Bead: lic-02
… warning Step 07, the PHARMCAT module comment, the step 07 and step 20 docs and the CHANGELOG now describe the AF 0.05 floor. The pharmcat-mito e2e case expects the outside call and the drug recommendation at AF 0.40, and adds a sample at AF 0.03 that stays No Result with a below-floor line. Bead: pgx-04 Bead: pgx-32 Bead: D10
…epeat map Step 04c ran Sniffles2 on any BAM. On short reads Sniffles2 finds no SV and still exits 0, so the step reported success with an empty VCF. It now reads the mean length of the first 10,000 primary reads with the pinned samtools image and stops below 1,000 bp, naming the long-read BAM to use. setup.sh installs GIAB's GRCh38 tandem-repeat stratification (v3.6, NIST data, pinned by sha256) and keeps it unpacked; step 04c passes it as --tandem-repeats when present and says so when not. Sniffles' own file comes from PacBio's pbsv under a licence limited to PacBio data, so it could not serve ONT reads. The closing hint no longer sends long-read SVs to duphold for a GQ: Sniffles2 writes its own, and step 15 reads Manta's calls only. Bead: sv-18 Bead: sv-17 Bead: D14
…lanted faults Every check of tests/test_run_eta.py and tests/test_run_headroom.py now runs on og-runlog in og-live/tests/ops_eta.rs and ops_headroom.rs, and each of their source mutants is a named fault of the fault-inject feature whose planted_* test must fail. Bead: D1
… on purpose The three clinvar-deep-intronic cases differ from the awk oracle where the reader now says more: a ./. gVCF block has no genotype, a block without a GQ is a low-confidence reference call, a gVCF variant record is not a reference call (no PASS claim), and a VCF ./. without a gVCF names the VCF record. The summary test's sites-mode check takes the new no-genotype text. The fake-docker case gets shellcheck notes on the module text it greps for. Bead: clin-13 Bead: clin-14
…e REF_MISMATCH count docs/06 lists each call word of the always-report list with when it applies, says a no call is not a negative result, and says the summary counts REF_MISMATCH records. The CHANGELOG names the output change. Bead: clin-09 Bead: clin-10 Bead: clin-11 Bead: clin-12 Bead: clin-13 Bead: clin-14
…puts append-n, rfc1, str-extra, tas2r38, mito-sam, hba-tiles, cut-haplotype and contains write the bytes the inline Python of the e2e cases wrote, pinned by FNV-1a hashes of the Python output. Bead: D1
… car gets it before its PR can run A stacked train car conflicts with main until the car before it lands, so its pull request gets no pull_request runs until then. The landing train now runs the fast checks on the car's own tree on the fork, while its E2E runs. Every required check can be run that way except pull-request, which ran only on a pull_request event. A manual run with the new pr_base input (full shas, each an ancestor of the ref) now runs the pull-request job too: the changed files and the added URLs are taken against each base, and the job checks the union. Pull request runs do the same as before. keepalive skips a pr_base run.
…ity against the Python plugin og-live/tests/ops.rs runs every check of tests/test_owngenome_live_ops.py on the Rust card, and compares the card (by meaning) and the metrics text (byte for byte) with bin/owngenome_live_ops.py on the ops fixture and on the run_eta and run_headroom fixtures. The plugin's eight source mutants are fault-inject faults; three more plant drift the parity cases must see. Bead: D1
GitHub reported a conflict, so no check could start. Both commits are already in the landing train below this branch; every conflict keeps the train's side, and the merged tree equals the branch's own tree.
The engines that replace ADMIXTURE and RFMix in step 26b need one fixed interface to build against in parallel. og-admix now holds: - panel: a phased panel VCF read into bit-packed haplotypes for chosen people at chosen sites, biallelic SNVs only; an unphased, missing or haploid genotype of a chosen person stops the read. - map: the PLINK genetic map reader that chroms.rs had, plus the inverse (position by cM). A non-finite cM or a cM that goes down is refused. - geno: model/ref.geno, 2-bit genotypes with a magic, version, J and N header, its writer and reader. - rng: xoshiro256** seeded by SplitMix64, checked against the authors' reference code. - simulate: the admixed-genome simulator with a truth BED. og ancestry-fine chroms reads its maps with og-admix's reader. The feature planted-red feeds every red control its broken input. Bead: D5 Bead: D14 Bead: D1
…d its truth The verb copies held-out panel people's haplotypes into a diploid mosaic: ancestry tracts at T events per Morgan drawn from mu, copying switches inside a tract. It writes an unphased VCF (BGZF when the name ends in .gz) and a truth BED, the same bytes for the same seed. The crate docs and the changelog name it. Bead: D5 Bead: D14 Bead: D1
Turns the planted-red feature on by default, so Guard's cargo test runs each control on its broken input and must show these tests fail: rng reference vectors, map interpolation (shuffled cM), ref.geno round trip (a flipped byte), the panel reader against bcftools (people in another order), the tract-length law (T doubled), the truth cover (a row dropped) and same seed same bytes (a second seed). The next commit reverts it. Bead: D5
… at c.803 keep its B copy The e2e case builds a chr9 reference over ABO and runs step 41 on the same deletion written two ways, and on a multi-allelic C>A,G 1/2. og on the right-shifted VCF as written stays in the case as the control: it reads A1/A1, which is what step 41 gave before it left-aligned. Bead: trait-30
… the landing train
… the landing train
…the landing train
…for the landing train
…::Options now carries The always-report sites reader gained a depth floor (min_dp, min_dp_value) in the car before, so the struct literal in this test no longer compiled. It now passes the CLI default, 10.
…the landing train
…for the landing train
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #411: claim trait-59, with owner decision D10 (a coding base is "callable" at 10 reads or more).
The problem
Step 49 gave each CHIP gene one detection limit, worked out from the gene's mean depth. A gene with a fifth of its coding bases unread then read
covered, with almost the same limit as a gene read evenly. On main, DNMT3A with 8 exons at 33x and 2 exons at 0x gives26.4 0.157 covered, and 27x everywhere gives27.0 0.152 covered. The step line and both reports then said a clone above that VAF "is found 8 times in 10", from the limit at the median depth. A dark exon stayed hidden, and the wording overstated a read count as the caller's sensitivity.The fix
og chip reportkeeps each gene's per-base depths (bases with no pileup line at 0). The limit is the smallest VAF at which a clone shows 3 reads with the change 8 times in 10, averaged over the gene's coding bases, each at its own depth. With one depth everywhere it equals the old limit.partly_coveredwhen some bases have 3 reads,too_shallowwhen none do. A gene with a fifth of its bases or more under 3 reads cannot reach 8 in 10.og summary collect'smedian_limitand so both reports use it. Before, it was the limit at the median depth.chip_depth.tsvgets two columns at the end:callable_fraction(share of coding bases with 10 reads or more, D10) andassessable_fraction(share of coding bases where a clone at the stated limit shows its reads). The first seven columns keep their names and order. summary.json's chipgenescarry the two columns when the table has them.detection_limit_vafis written rounded up at the third decimal, so a clone at the written VAF reaches 8 in 10. At 30x it reads 0.138; rounded to the nearest it read 0.137, where the power falls just short. The stated limit insideog chip reportis the median of these written limits, the same figure the step line andog summary collectread back, soassessable_fractionis counted at the figure the reports show.partly_covered. The CHIP genes' exons do not overlap, so this matters only for a BED with overlapping or padded rows.Output changes, as 1.0.0 allows: two columns added to
chip_depth.tsv, a newpartly_coveredstatus, newdetection_limit_vafandmedian_limitvalues (rounded up at the third decimal), and new wording in the step line. Nothing is renamed or removed.Decisions taken in this lane:
og summary collectonly seeschip_depth.tsv, so this keeps the step line, summary.json and the reports on one figure.assessable_fractionuses the run's stated limit, so it is empty when no gene is covered.median_limit, now the honest figure.Files outside the package list, each needed for CI to stay true:
tests/parity/golden/chip/report-{calls,table,overlap,empty}/(deliberate-fix goldens),tests/test_chip_report.py(3 checks pinned the old mean-depth behaviour), andmodules/local/chip/main.nf(the stub header gets the two columns).How it was checked
rust/crates/og-chip/tests/i411_chip_read_support_limit.rson the test-only commit c28e011 fails all 4 tests in CI: https://github.com/GeiserX/OwnGenome/actions/runs/43744441741/job/132904521191 (main writescoveredwherepartly_coveredis due, 0.137 where 0.159 is due, and no share columns). The same tests pass on the fix.red_first_a_base_two_genes_share_is_read_for_bothin the same file, run against the report code of bab3a34 (before the shared-base fix), fails withleft: "partly_covered",right: "covered"for DNMT3A. It passes on 2c0715e. Thereport-overlapparity golden changes with it: DNMT3A now readscoveredwith all its coding bases callable.og_chip::binom: one depth everywhere equals the old limit to 1e-12, an unread fifth leaves no limit (the power at a VAF near 1 rounds to exactly 1, so the share of deep bases is tested on its own), and the readable-at-a-VAF rule.ORACLE_RUNNER=local scripts/ci/oracle-diff.sh chip summary-collecton the head after the main merge: all 352 items in 89 cases agree, with goldens for the deliberate fixes. The whole-file casesallandall-historyget goldens for the new chipgeneskeys. The c generator copies the full folder withcp -Rpand gives each table the float cases rewrite the fixture's time, so those goldens do not depend on the clock (a first CI run caught two dates of the clock: https://github.com/GeiserX/OwnGenome/actions/runs/43747267494).tests/test_chip_report.py,test_live_results.py,test_live_widgets.pyandtest_render_report.pypass with the newog.chip-clonal-1e2e: GATK's image is amd64-only and cannot run on this box, so I ran the depth half on public data. I took GIAB HG002 60x reads over the MANE coding exons of the six CHIP genes on the fixture contigs (pileup at MQ 20, BQ 10, duplicates out) and ranog chip report. DNMT3A: mean 52.7x, limit 0.081 (covered, under the case's 10%),callable_fraction1.000,assessable_fraction0.293. The case's new checks pass on that output and fail on main's output (no share columns, old "Detection limit" line). The full case runs in the e2e train./usr/bin/time -l: on the HG002 input (21,277 depth lines) both old and new take 0.01 to 0.06 s and 3.2 MB. On 2 Mb of synthetic depth up to 5,000x, both take 0.50 s; peak memory goes from 5.0 MB to 5.3 MB.uniform_30x_states_the_binomial_limit_and_says_what_it_isnow pins 0.138 and the power on both sides of it. Against the report code of 2c0715e it fails withleft: "0.137",right: "0.138", and the two other tests that pin a written limit fail the same way (0.125 where 0.126 is due). All pass on 65baf9e. The parity goldens ofog chip reportdo not change: their limits already sat just under a whole thousandth.chip-clonal-1e2e case has not yet run green in CI. It waits for the e2e train.cargo fmt --check,cargo clippy --workspace --all-targets -D warningsandcargo doc -D warningsare clean.Beads: trait-59, D10.
Summary by CodeRabbit