Repository navigation
Conversation
…peat map The fake-docker cases fail on main: step 04c runs Sniffles2 on a short-read BAM and exits 0, never passes --tandem-repeats, and its closing hint sends long-read SVs to duphold for a GQ. Bead: sv-18 Bead: sv-17
…te query in both mirrors Step 06 runs to its summary against a fake bcftools that warns REF_MISMATCH twice, then not at all: the count line reads 2 and then 0, and the notice shows only for 2. The always-report call gets the depth floor, and the query_format of the script and of the module, run with the Groovy escapes undone, both end with %FILTER. Red on main: the count line is missing, and the module's format has no FILTER. Bead: clin-13
… og links It walks cargo metadata per target from og along normal edges, checks each crate's licence against an allow list, and writes each licence text once, including the C library texts of crates that set links (htslib and htscodecs from hts-sys). Bead: lic-02
rust/og.sh build writes it in OG_BUILDER with the new og; release.yml copies it to dist, adds it to SHA256SUMS and the attestation, and checks the upload with check-release.sh dist. Bead: lic-02
… warning Step 07, the PHARMCAT module comment, the step 07 and step 20 docs and the CHANGELOG now describe the AF 0.05 floor. The pharmcat-mito e2e case expects the outside call and the drug recommendation at AF 0.40, and adds a sample at AF 0.03 that stays No Result with a below-floor line. Bead: pgx-04 Bead: pgx-32 Bead: D10
…epeat map Step 04c ran Sniffles2 on any BAM. On short reads Sniffles2 finds no SV and still exits 0, so the step reported success with an empty VCF. It now reads the mean length of the first 10,000 primary reads with the pinned samtools image and stops below 1,000 bp, naming the long-read BAM to use. setup.sh installs GIAB's GRCh38 tandem-repeat stratification (v3.6, NIST data, pinned by sha256) and keeps it unpacked; step 04c passes it as --tandem-repeats when present and says so when not. Sniffles' own file comes from PacBio's pbsv under a licence limited to PacBio data, so it could not serve ONT reads. The closing hint no longer sends long-read SVs to duphold for a GQ: Sniffles2 writes its own, and step 15 reads Manta's calls only. Bead: sv-18 Bead: sv-17 Bead: D14
…lanted faults Every check of tests/test_run_eta.py and tests/test_run_headroom.py now runs on og-runlog in og-live/tests/ops_eta.rs and ops_headroom.rs, and each of their source mutants is a named fault of the fault-inject feature whose planted_* test must fail. Bead: D1
… on purpose The three clinvar-deep-intronic cases differ from the awk oracle where the reader now says more: a ./. gVCF block has no genotype, a block without a GQ is a low-confidence reference call, a gVCF variant record is not a reference call (no PASS claim), and a VCF ./. without a gVCF names the VCF record. The summary test's sites-mode check takes the new no-genotype text. The fake-docker case gets shellcheck notes on the module text it greps for. Bead: clin-13 Bead: clin-14
…e REF_MISMATCH count docs/06 lists each call word of the always-report list with when it applies, says a no call is not a negative result, and says the summary counts REF_MISMATCH records. The CHANGELOG names the output change. Bead: clin-09 Bead: clin-10 Bead: clin-11 Bead: clin-12 Bead: clin-13 Bead: clin-14
…puts append-n, rfc1, str-extra, tas2r38, mito-sam, hba-tiles, cut-haplotype and contains write the bytes the inline Python of the e2e cases wrote, pinned by FNV-1a hashes of the Python output. Bead: D1
… car gets it before its PR can run A stacked train car conflicts with main until the car before it lands, so its pull request gets no pull_request runs until then. The landing train now runs the fast checks on the car's own tree on the fork, while its E2E runs. Every required check can be run that way except pull-request, which ran only on a pull_request event. A manual run with the new pr_base input (full shas, each an ancestor of the ref) now runs the pull-request job too: the changed files and the added URLs are taken against each base, and the job checks the union. Pull request runs do the same as before. keepalive skips a pr_base run.
…ity against the Python plugin og-live/tests/ops.rs runs every check of tests/test_owngenome_live_ops.py on the Rust card, and compares the card (by meaning) and the metrics text (byte for byte) with bin/owngenome_live_ops.py on the ops fixture and on the run_eta and run_headroom fixtures. The plugin's eight source mutants are fault-inject faults; three more plant drift the parity cases must see. Bead: D1
GitHub reported a conflict, so no check could start. Both commits are already in the landing train below this branch; every conflict keeps the train's side, and the merged tree equals the branch's own tree.
The engines that replace ADMIXTURE and RFMix in step 26b need one fixed interface to build against in parallel. og-admix now holds: - panel: a phased panel VCF read into bit-packed haplotypes for chosen people at chosen sites, biallelic SNVs only; an unphased, missing or haploid genotype of a chosen person stops the read. - map: the PLINK genetic map reader that chroms.rs had, plus the inverse (position by cM). A non-finite cM or a cM that goes down is refused. - geno: model/ref.geno, 2-bit genotypes with a magic, version, J and N header, its writer and reader. - rng: xoshiro256** seeded by SplitMix64, checked against the authors' reference code. - simulate: the admixed-genome simulator with a truth BED. og ancestry-fine chroms reads its maps with og-admix's reader. The feature planted-red feeds every red control its broken input. Bead: D5 Bead: D14 Bead: D1
…d its truth The verb copies held-out panel people's haplotypes into a diploid mosaic: ancestry tracts at T events per Morgan drawn from mu, copying switches inside a tract. It writes an unphased VCF (BGZF when the name ends in .gz) and a truth BED, the same bytes for the same seed. The crate docs and the changelog name it. Bead: D5 Bead: D14 Bead: D1
Turns the planted-red feature on by default, so Guard's cargo test runs each control on its broken input and must show these tests fail: rng reference vectors, map interpolation (shuffled cM), ref.geno round trip (a flipped byte), the panel reader against bcftools (people in another order), the tract-length law (T doubled), the truth cover (a row dropped) and same seed same bytes (a second seed). The next commit reverts it. Bead: D5
… at c.803 keep its B copy The e2e case builds a chr9 reference over ABO and runs step 41 on the same deletion written two ways, and on a multi-allelic C>A,G 1/2. og on the right-shifted VCF as written stays in the case as the control: it reads A1/A1, which is what step 41 gave before it left-aligned. Bead: trait-30
…g them og bloodtype report matches a marker by position, REF and ALT, and the table writes every indel left-aligned. A caller can write the same indel further right in its repeat, and then the marker missed it: with reads there, the site read as the reference allele at confidence high (ABO c.1061delC as GG>G gave A1/A1, a right-shifted Kell-null deletion gave k/k). Step 41 and the BLOODTYPE process now take the VCF's records within 100 bp of each marker site and left-align them with bcftools norm -c w -f against the reference, keeping multi-allelic records whole. Records on a contig the reference lacks go through unchanged. BLOODTYPE now runs in BCFTOOLS_IMAGE (og is a static binary from bin/), and the step needs the reference and its .fai. The fake-docker case plays the new bcftools call as a copy. Bead: trait-30
… cargo ones The three Python test files go: their checks run in og-live/tests/ops*.rs. Guard's rust job runs every planted_* test with the fault-inject feature and requires each to fail and none to pass. The alerts guide, the example rules, the fixture note and the two module comments name the cargo tests. Bead: D1
…rkers The input contract said DeepVariant writes left-aligned records. It does not always do so without --normalize_reads, and a VCF from another caller is accepted too. The page now says how the step makes the records left-aligned itself, what stays whole and what goes through unchanged. Bead: trait-30
Step 42 kept each site's GQ but never compared it, so a GQ 0 call at APOE, HFE, SERPINA1 or any traits-table row gave a confident reading, and a call the caller filtered (LowQual) was read too. A call below GQ 20, or with a FILTER other than PASS, . or RefCall, now reads low_gq with no reading and names the GQ or the filter; such a site is not phased. The genotypes table gains the record's FILTER as a seventh column, and the summary's traits section counts low_gq. Bead: trait-01 Bead: trait-67 Bead: D10
… LowQual call Bead: trait-01 Bead: trait-67
The float cases now keep their fixture file times, so a golden's file_date does not depend on the minute the case ran. Bead: trait-01
The engine-size section now also checks an engine with less memory only (THREADS=3 and a user -c, which must come after executor.config) and one with fewer CPUs only (an explicit --max_memory stays the only one). Bead: plat-18
…l on chrX The SV rows passed on any record within 500 bp of the planted deletion, of any type, size or filter, and nothing caught false calls elsewhere. Sniffles2, Manta, Delly and TIDDIT now need one PASS deletion of 2.7 to 3.3 kb there; Sniffles2 also needs no PASS record elsewhere, and a second row runs it with --tandem-repeats as step 04c does. GRIDSS keeps the loose check: it writes breakends, not DEL records. A new DELLY_IMAGE row runs delly sr --sex male and female on mini.bam moved onto a chrX of GRCh38's length past PAR1 (input chrx): the male genotype must be haploid, the female 1/1, and an unknown value must fail. Delly ignores --sex without a word on a chrX of another length; the row goes red when the chrX is one base longer. The self-test plants records for each rule of the new checks. Bead: sv-43 Bead: sv-41
The long-read guide said duphold re-genotypes SVs and that Sniffles2's QUAL is reliable, and listed SV counts and repeat accuracy with no source. It now says what duphold really reads, describes the read-length check and the tandem-repeat map, and gives the run we measured on a public HG002 HiFi slice next to the Sniffles2 paper. testing.md says what the smoke rows and fake-docker cases cover. Bead: sv-17 Bead: sv-43 Bead: sv-41
…ft-align # Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # docs/23-clinical-filter.md # docs/rust.md # tests/test_ancestry_fine_report.py
A field spelled password_set read as a hard-coded password to a secret scanner; the flag says whether the site asks for a login. Bead: D1
# Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # docs/23-clinical-filter.md # docs/rust.md # tests/test_ancestry_fine_report.py
…an unreadable licence file Neither is a cargo package, so the notices left them out. og dev licences notices takes --extra NAME=FILE (HTML notices become text); og.sh passes the toolchain's COPYRIGHT-library.html and musl's COPYRIGHT from the builder image, and check-release.sh dist wants both. A licence file that exists but cannot be read is now an error instead of a silent gap. Bead: lic-02
… the landing train
… the landing train
…the landing train
…for the landing train
…::Options now carries The always-report sites reader gained a depth floor (min_dp, min_dp_value) in the car before, so the struct literal in this test no longer compiled. It now passes the CLI default, 10.
…the landing train
…for the landing train
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #411. Settles D1 (no Python in 1.0.0) for four test files, and keeps the blood-group, HPA and HNA rows of D3 under test.
Problem
Four Python files tested
og cn,og bloodtypeandog traitsby running the binary as a subprocess:tests/test_hba_copy_number.py,tests/test_bloodtype_report.py,tests/test_bloodtype_hpa_hna.pyandtests/test_traits_table.py, about 1,050 lines. 1.0.0 ships without Python, so these checks had to move.test_traits_table.pywas also a tool: the e2e casetraits-panel-1and the "Adding a row" steps of the traits page ran its--fastaand--onlinemodes.Fix
Every check moves into
cargo test, through each tool's own command line, in process. The four Python files are deleted.test_hba_copy_number.py, all 21 checks: 4, 3, 2 and 5 genes, an estimate between counts, low or disagreeing flanks, the MAPQ 0 guard (none, 1%, 3%), no gene at all, the command line's refusals, the demo's HBA tablerust/crates/og-cn/tests/port_tests_cn_traits.rs, 17 teststest_bloodtype_report.pysections 1 to 14: markers against the rules, nothing planted, ABO, RhD from depth, Duffy and Kell, the VCF edge cases, a BAM without some contigs, null-row and HPA orientation, every null and rare HPA row positive and negative, gVCF records and blocks, null alleles, common variants, another ALT at a null row, two null copiesrust/crates/og-bloodtype/tests/port_tests_cn_traits.rs,s01tos14test_bloodtype_hpa_hna.pysections 1 to 4: HPA and HNA sites at their GRCh38 positions, two-allele antigens at 0/0, 0/1 and 1/1, the meaning of a missing antigen, HNA-1s15tos18test_traits_table.pyschema part: at least 30 rows,og traits checkpasses, a planted bad row failsrust/crates/og-steps/tests/traits_table.rs, 3 teststest_traits_table.py --fastaand--onlineog dev traits-sources(rust/crates/og-dev/src/traits_sources.rs), with unit tests on a synthetic FASTA and canned PubMed and dbSNP answersThe bloodtype test reads
assets/bloodtype_markers.tsvwith its own parser, not the crate's, so a reader bug cannot hide a table bug.Files outside the three test paths, each needed by the switch or the deletion:
rust/crates/og-dev/src/traits_sources.rs,rust/crates/og-dev/src/lib.rs,rust/og/src/main.rs: the newog dev traits-sourcesverb and its usage line.tests/e2e/traits-panel-1.sh: runsog dev traits-sourceswhere it ran the Python file. Same checks, same controls.docs/42-traits.md: the "Adding a row" steps nameog traits checkandog dev traits-sources.docs/35-paralogs.md,docs/cn-loci.md,tests/demo/make_demo_sample.py,assets/traits.tsv(a header comment): they named the deleted files.CHANGELOG.md: one line.No pipeline output, option or file name changes.
Decision:
traits-panel-1now runsog dev traits-sources, but its scope row does not namerust/crates/og-dev/src/traits_sources.rs. Arust/path in a row must also be in the e2e scope list (SCOPE_FILESinscripts/ci/e2e-scope.sh) and in the push paths of.github/workflows/e2e.yml, asog-dev/src/fixture.rsis. Both files are outside this package, so the row stays as it is and the gap is a follow-up. The verb's own unit tests run incargo teston every change to it. Python'stests/test_traits_table.pyhad the same gap.traits-panel-1also runsog traits checkon the table before the source checks, as the deleted Python did.How it was checked
og dev traits-sources. Guard'scargo test --workspace --no-fail-fastwent red on each, and on nothing else: https://github.com/GeiserX/OwnGenome/actions/runs/43745565116/job/132907983428 (step "cargo test"). The six red targets: og-bloodtypeport_tests_cn_traits(s08), og-cnport_tests_cn_traits(3 tests) and its lib (hba_counts_and_guards), og-dev lib (3traits_sourcestests), og-stepstraits_table(the_table_passes_og_traits_check) and its lib (4traitstests that read the shipped table). The next commit reverts them; the tree is the same as before the plant.s08now compares an insertion's added bases on the gene strand, not only its length; the reference-block check ins10uses fewer reads than the depth floor at the block sites, so it fails if the blocks are ignored. Both shown red before the fix on the build host (cargo test -p og-bloodtype --test port_tests_cn_traits --no-fail-fast: the two wrong-base insertion controls returned true, and without the blocks HPA-21 read "not determined"), green after.cargo fmt --check,cargo clippy --workspace --all-targets --all-features -D warningsandcargo doc -D warningspass.traits-panel-1ran on the fixture reference with the new verb: 25 rows checked against the FASTA, 38 PubMed ids and 36 rsIDs looked up, all pass; the planted copy fails and names both the wrong base and the missing PubMed id.ogmany times. The cargo tests run in about 2.7 s (og-bloodtype 2.5 s, og-cn under 0.1 s, og-steps and og-dev under 0.1 s).Beads: D1, D3.