Skip to content

test(cn,bloodtype,traits): check copy number, blood groups and the traits table in cargo test - #452

Open
geiresx wants to merge 467 commits into
GeiserX:mainfrom
geiresx:pgp/port-tests-cn-traits
Open

geiresx wants to merge 467 commits into
GeiserX:mainfrom
geiresx:pgp/port-tests-cn-traits

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #411. Settles D1 (no Python in 1.0.0) for four test files, and keeps the blood-group, HPA and HNA rows of D3 under test.

Problem

Four Python files tested og cn, og bloodtype and og traits by running the binary as a subprocess: tests/test_hba_copy_number.py, tests/test_bloodtype_report.py, tests/test_bloodtype_hpa_hna.py and tests/test_traits_table.py, about 1,050 lines. 1.0.0 ships without Python, so these checks had to move. test_traits_table.py was also a tool: the e2e case traits-panel-1 and the "Adding a row" steps of the traits page ran its --fasta and --online modes.

Fix

Every check moves into cargo test, through each tool's own command line, in process. The four Python files are deleted.

Python check (on main) Now
test_hba_copy_number.py, all 21 checks: 4, 3, 2 and 5 genes, an estimate between counts, low or disagreeing flanks, the MAPQ 0 guard (none, 1%, 3%), no gene at all, the command line's refusals, the demo's HBA table rust/crates/og-cn/tests/port_tests_cn_traits.rs, 17 tests
test_bloodtype_report.py sections 1 to 14: markers against the rules, nothing planted, ABO, RhD from depth, Duffy and Kell, the VCF edge cases, a BAM without some contigs, null-row and HPA orientation, every null and rare HPA row positive and negative, gVCF records and blocks, null alleles, common variants, another ALT at a null row, two null copies rust/crates/og-bloodtype/tests/port_tests_cn_traits.rs, s01 to s14
test_bloodtype_hpa_hna.py sections 1 to 4: HPA and HNA sites at their GRCh38 positions, two-allele antigens at 0/0, 0/1 and 1/1, the meaning of a missing antigen, HNA-1 same file, s15 to s18
test_traits_table.py schema part: at least 30 rows, og traits check passes, a planted bad row fails rust/crates/og-steps/tests/traits_table.rs, 3 tests
test_traits_table.py --fasta and --online a new verb, og dev traits-sources (rust/crates/og-dev/src/traits_sources.rs), with unit tests on a synthetic FASTA and canned PubMed and dbSNP answers

The bloodtype test reads assets/bloodtype_markers.tsv with its own parser, not the crate's, so a reader bug cannot hide a table bug.

Files outside the three test paths, each needed by the switch or the deletion:

  • rust/crates/og-dev/src/traits_sources.rs, rust/crates/og-dev/src/lib.rs, rust/og/src/main.rs: the new og dev traits-sources verb and its usage line.
  • tests/e2e/traits-panel-1.sh: runs og dev traits-sources where it ran the Python file. Same checks, same controls.
  • docs/42-traits.md: the "Adding a row" steps name og traits check and og dev traits-sources.
  • docs/35-paralogs.md, docs/cn-loci.md, tests/demo/make_demo_sample.py, assets/traits.tsv (a header comment): they named the deleted files.
  • CHANGELOG.md: one line.

No pipeline output, option or file name changes.

Decision: traits-panel-1 now runs og dev traits-sources, but its scope row does not name rust/crates/og-dev/src/traits_sources.rs. A rust/ path in a row must also be in the e2e scope list (SCOPE_FILES in scripts/ci/e2e-scope.sh) and in the push paths of .github/workflows/e2e.yml, as og-dev/src/fixture.rs is. Both files are outside this package, so the row stays as it is and the gap is a follow-up. The verb's own unit tests run in cargo test on every change to it. Python's tests/test_traits_table.py had the same gap.

traits-panel-1 also runs og traits check on the table before the source checks, as the deleted Python did.

How it was checked

Beads: D1, D3.

…peat map

The fake-docker cases fail on main: step 04c runs Sniffles2 on a
short-read BAM and exits 0, never passes --tandem-repeats, and its
closing hint sends long-read SVs to duphold for a GQ.

Bead: sv-18
Bead: sv-17
…te query in both mirrors

Step 06 runs to its summary against a fake bcftools that warns
REF_MISMATCH twice, then not at all: the count line reads 2 and then 0,
and the notice shows only for 2. The always-report call gets the depth
floor, and the query_format of the script and of the module, run with
the Groovy escapes undone, both end with %FILTER. Red on main: the count
line is missing, and the module's format has no FILTER.

Bead: clin-13
… og links

It walks cargo metadata per target from og along normal edges, checks
each crate's licence against an allow list, and writes each licence text
once, including the C library texts of crates that set links (htslib and
htscodecs from hts-sys).

Bead: lic-02
rust/og.sh build writes it in OG_BUILDER with the new og; release.yml
copies it to dist, adds it to SHA256SUMS and the attestation, and checks
the upload with check-release.sh dist.

Bead: lic-02
… warning

Step 07, the PHARMCAT module comment, the step 07 and step 20 docs and the
CHANGELOG now describe the AF 0.05 floor. The pharmcat-mito e2e case expects
the outside call and the drug recommendation at AF 0.40, and adds a sample
at AF 0.03 that stays No Result with a below-floor line.

Bead: pgx-04
Bead: pgx-32
Bead: D10
…epeat map

Step 04c ran Sniffles2 on any BAM. On short reads Sniffles2 finds no SV
and still exits 0, so the step reported success with an empty VCF. It
now reads the mean length of the first 10,000 primary reads with the
pinned samtools image and stops below 1,000 bp, naming the long-read
BAM to use.

setup.sh installs GIAB's GRCh38 tandem-repeat stratification (v3.6,
NIST data, pinned by sha256) and keeps it unpacked; step 04c passes it
as --tandem-repeats when present and says so when not. Sniffles' own
file comes from PacBio's pbsv under a licence limited to PacBio data,
so it could not serve ONT reads.

The closing hint no longer sends long-read SVs to duphold for a GQ:
Sniffles2 writes its own, and step 15 reads Manta's calls only.

Bead: sv-18
Bead: sv-17
Bead: D14
…lanted faults

Every check of tests/test_run_eta.py and tests/test_run_headroom.py now
runs on og-runlog in og-live/tests/ops_eta.rs and ops_headroom.rs, and
each of their source mutants is a named fault of the fault-inject feature
whose planted_* test must fail.

Bead: D1
… on purpose

The three clinvar-deep-intronic cases differ from the awk oracle where
the reader now says more: a ./. gVCF block has no genotype, a block
without a GQ is a low-confidence reference call, a gVCF variant record
is not a reference call (no PASS claim), and a VCF ./. without a gVCF
names the VCF record. The summary test's sites-mode check takes the new
no-genotype text. The fake-docker case gets shellcheck notes on the
module text it greps for.

Bead: clin-13
Bead: clin-14
…e REF_MISMATCH count

docs/06 lists each call word of the always-report list with when it
applies, says a no call is not a negative result, and says the summary
counts REF_MISMATCH records. The CHANGELOG names the output change.

Bead: clin-09
Bead: clin-10
Bead: clin-11
Bead: clin-12
Bead: clin-13
Bead: clin-14
…puts

append-n, rfc1, str-extra, tas2r38, mito-sam, hba-tiles, cut-haplotype
and contains write the bytes the inline Python of the e2e cases wrote,
pinned by FNV-1a hashes of the Python output.

Bead: D1
… car gets it before its PR can run

A stacked train car conflicts with main until the car before it lands, so its
pull request gets no pull_request runs until then. The landing train now runs
the fast checks on the car's own tree on the fork, while its E2E runs. Every
required check can be run that way except pull-request, which ran only on a
pull_request event.

A manual run with the new pr_base input (full shas, each an ancestor of the
ref) now runs the pull-request job too: the changed files and the added URLs
are taken against each base, and the job checks the union. Pull request runs
do the same as before. keepalive skips a pr_base run.
…ity against the Python plugin

og-live/tests/ops.rs runs every check of tests/test_owngenome_live_ops.py
on the Rust card, and compares the card (by meaning) and the metrics text
(byte for byte) with bin/owngenome_live_ops.py on the ops fixture and on
the run_eta and run_headroom fixtures. The plugin's eight source mutants
are fault-inject faults; three more plant drift the parity cases must see.

Bead: D1
GitHub reported a conflict, so no check could start. Both commits are
already in the landing train below this branch; every conflict keeps the
train's side, and the merged tree equals the branch's own tree.
The engines that replace ADMIXTURE and RFMix in step 26b need one fixed
interface to build against in parallel. og-admix now holds:

- panel: a phased panel VCF read into bit-packed haplotypes for chosen
  people at chosen sites, biallelic SNVs only; an unphased, missing or
  haploid genotype of a chosen person stops the read.
- map: the PLINK genetic map reader that chroms.rs had, plus the inverse
  (position by cM). A non-finite cM or a cM that goes down is refused.
- geno: model/ref.geno, 2-bit genotypes with a magic, version, J and N
  header, its writer and reader.
- rng: xoshiro256** seeded by SplitMix64, checked against the authors'
  reference code.
- simulate: the admixed-genome simulator with a truth BED.

og ancestry-fine chroms reads its maps with og-admix's reader. The
feature planted-red feeds every red control its broken input.

Bead: D5
Bead: D14
Bead: D1
…d its truth

The verb copies held-out panel people's haplotypes into a diploid mosaic:
ancestry tracts at T events per Morgan drawn from mu, copying switches
inside a tract. It writes an unphased VCF (BGZF when the name ends in .gz)
and a truth BED, the same bytes for the same seed. The crate docs and the
changelog name it.

Bead: D5
Bead: D14
Bead: D1
Turns the planted-red feature on by default, so Guard's cargo test runs
each control on its broken input and must show these tests fail:
rng reference vectors, map interpolation (shuffled cM), ref.geno round
trip (a flipped byte), the panel reader against bcftools (people in
another order), the tract-length law (T doubled), the truth cover (a row
dropped) and same seed same bytes (a second seed). The next commit
reverts it.

Bead: D5
… at c.803 keep its B copy

The e2e case builds a chr9 reference over ABO and runs step 41 on the
same deletion written two ways, and on a multi-allelic C>A,G 1/2. og on
the right-shifted VCF as written stays in the case as the control: it
reads A1/A1, which is what step 41 gave before it left-aligned.

Bead: trait-30
…g them

og bloodtype report matches a marker by position, REF and ALT, and the
table writes every indel left-aligned. A caller can write the same indel
further right in its repeat, and then the marker missed it: with reads
there, the site read as the reference allele at confidence high (ABO
c.1061delC as GG>G gave A1/A1, a right-shifted Kell-null deletion gave
k/k).

Step 41 and the BLOODTYPE process now take the VCF's records within
100 bp of each marker site and left-align them with bcftools norm -c w
-f against the reference, keeping multi-allelic records whole. Records
on a contig the reference lacks go through unchanged. BLOODTYPE now runs
in BCFTOOLS_IMAGE (og is a static binary from bin/), and the step needs
the reference and its .fai. The fake-docker case plays the new bcftools
call as a copy.

Bead: trait-30
… cargo ones

The three Python test files go: their checks run in og-live/tests/ops*.rs.
Guard's rust job runs every planted_* test with the fault-inject feature
and requires each to fail and none to pass. The alerts guide, the example
rules, the fixture note and the two module comments name the cargo tests.

Bead: D1
…rkers

The input contract said DeepVariant writes left-aligned records. It does
not always do so without --normalize_reads, and a VCF from another caller
is accepted too. The page now says how the step makes the records
left-aligned itself, what stays whole and what goes through unchanged.

Bead: trait-30
Step 42 kept each site's GQ but never compared it, so a GQ 0 call at
APOE, HFE, SERPINA1 or any traits-table row gave a confident reading, and
a call the caller filtered (LowQual) was read too. A call below GQ 20, or
with a FILTER other than PASS, . or RefCall, now reads low_gq with no
reading and names the GQ or the filter; such a site is not phased. The
genotypes table gains the record's FILTER as a seventh column, and the
summary's traits section counts low_gq.

Bead: trait-01
Bead: trait-67
Bead: D10
The float cases now keep their fixture file times, so a golden's
file_date does not depend on the minute the case ran.

Bead: trait-01
The engine-size section now also checks an engine with less memory only
(THREADS=3 and a user -c, which must come after executor.config) and one
with fewer CPUs only (an explicit --max_memory stays the only one).

Bead: plat-18
…l on chrX

The SV rows passed on any record within 500 bp of the planted deletion,
of any type, size or filter, and nothing caught false calls elsewhere.
Sniffles2, Manta, Delly and TIDDIT now need one PASS deletion of 2.7 to
3.3 kb there; Sniffles2 also needs no PASS record elsewhere, and a
second row runs it with --tandem-repeats as step 04c does. GRIDSS keeps
the loose check: it writes breakends, not DEL records.

A new DELLY_IMAGE row runs delly sr --sex male and female on mini.bam
moved onto a chrX of GRCh38's length past PAR1 (input chrx): the male
genotype must be haploid, the female 1/1, and an unknown value must
fail. Delly ignores --sex without a word on a chrX of another length;
the row goes red when the chrX is one base longer.

The self-test plants records for each rule of the new checks.

Bead: sv-43
Bead: sv-41
The long-read guide said duphold re-genotypes SVs and that Sniffles2's
QUAL is reliable, and listed SV counts and repeat accuracy with no
source. It now says what duphold really reads, describes the read-length
check and the tandem-repeat map, and gives the run we measured on a
public HG002 HiFi slice next to the Sniffles2 paper. testing.md says
what the smoke rows and fake-docker cases cover.

Bead: sv-17
Bead: sv-43
Bead: sv-41
…ft-align

# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	docs/23-clinical-filter.md
#	docs/rust.md
#	tests/test_ancestry_fine_report.py
A field spelled password_set read as a hard-coded password to a secret
scanner; the flag says whether the site asks for a login.

Bead: D1
# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	docs/23-clinical-filter.md
#	docs/rust.md
#	tests/test_ancestry_fine_report.py
…an unreadable licence file

Neither is a cargo package, so the notices left them out. og dev licences
notices takes --extra NAME=FILE (HTML notices become text); og.sh passes
the toolchain's COPYRIGHT-library.html and musl's COPYRIGHT from the
builder image, and check-release.sh dist wants both. A licence file that
exists but cannot be read is now an error instead of a silent gap.

Bead: lic-02
…::Options now carries

The always-report sites reader gained a depth floor (min_dp, min_dp_value) in the car before, so the struct literal in this
test no longer compiled. It now passes the CLI default, 10.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants