Skip to content

fix(couple): write the couple screen only when every couple is computed - #457

Open
geiresx wants to merge 501 commits into
GeiserX:mainfrom
geiresx:pgp/port-couple
Open

geiresx wants to merge 501 commits into
GeiserX:mainfrom
geiresx:pgp/port-couple

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Step 39 still ran bin/couple_report.py. The Python wrote each couple's files as it went. So a bad input in the second couple left the first couple's files new and the second's old or missing. The screen also gave wrong answers a couple could act on:

  • It told the carrier of a BRCA1, MLH1 or LDLR change that a child "can be a carrier, not affected". These genes also cause a dominant condition, so a child who inherits the change can be affected. When that partner carries the change on both copies, every child inherits it.
  • It gave MECP2 and PCDH19 the recessive wording "every daughter is a carrier". In these genes a daughter who inherits the change can be affected.
  • Without a PAR file, it read a male's call in PAR1 as hemizygous. A shared CSF2RA change then read 1 in 2 instead of 1 in 4.
  • It read a haploid call on an autosome (GT 1) as homozygous, so the risk doubled.
  • It counted a GQ 1, DP 1 call like any other, with no mark.

The fix

Five commits.

  1. 0194f98 ports the screen to og couple screen and og couple genes (crate og-couple). scripts/39-couple.sh and the Nextflow COUPLE_CARRIER process now call og. bin/couple_report.py is deleted, together with the two modules only it still imported, bin/cnvpytor_to_vcf.py and bin/paraphase_report.py, and their three Python tests. The screen reuses the depth rule of og-steps sv, the SMN1 rule of og-cn and the pedigree reader of og-pedigree, so no rule exists twice. It computes every couple first and writes the files only when all are done.
  2. 9d4e9e0 fixes the five answers above:
    • assets/carrier_genes.tsv gets a dominant column. AD or SD marks the 43 genes the ACMG secondary findings list gives as dominant. A one-partner row in such a gene reads "a child who inherits it can be affected, 1 in 2 per child". XLD marks the 28 X-linked genes that also have an X-linked dominant disorder. Their rows add "females can show signs".
    • The inheritance column gets the value XLD for 28 genes, under rule (y): a gene is X-linked dominant only when its main disorder is dominant. MECP2, CDKL5, IKBKG, OFD1 and PCDH19 are XLD. GJB1, GLA, PLP1 and FMR1 stay XL, with "females can show signs".
    • Without --par-bed, the screen uses GRCh38's PARs, the rows of assets/par_grch38.bed.
    • A haploid call on an autosome is one copy.
    • A call below step 06's floor is marked "low-confidence call (GQ .., DP ..): check the reads". It still counts, and its row says to confirm it first.
    • Each ClinVar finding names the condition ClinVar gives for the allele (CLNDN).
  3. c206e13 fixes the dominant one-partner row when that partner carries the change on both copies (GT 1/1). The row read "each child 1 in 2", but every child inherits one of the two copies. It now reads every child, dominant, and the explanation says every child inherits the change and can be affected.
  4. b72696c lets the personal-data scan skip three invented ExpansionHunter inputs of the parity cases (see below).
  5. eb449b8 fixes what the CodeRabbit review found:
    • When a female and a male partner both carry a change in the same X-linked recessive gene, each daughter gets the father's changed X. offspring_risk now adds daughters 1 in 2, can be affected, or every daughter can be affected when the mother has two copies.
    • Two heterozygous changes in a dominant gene in one partner add a note: when they sit on different copies, every child inherits one.
    • A haploid call in a PAR is one copy, as on an autosome. Before, a male's GT 1 in PAR1 read hemizygous and doubled the risk.
    • Two couples whose file names would clash (ids with _: A with B_C, A_B with C) stop the screen before it writes anything. The Python let the second overwrite the first.
    • The XML reader of og couple genes refuses a file with no root or one that ends before its root closes. A cut-off Orphadata download passed before and dropped the Orphanet rows.

How it was checked

  • Parity: ORACLE_RUNNER=local scripts/ci/oracle-diff.sh couple-screen couple-genes with og built from eb449b8 (macOS arm64) and the Python at d4d3d4b: all 208 items in 37 cases agree. Goldens hold the deliberate changes: all-or-nothing (commit 1), par-and-haploid, the couple-genes rebuild and bad-header (commit 2).
  • The parity job goes red on a planted change in the carrier decision. An og built with the shared rule's zygosity removed (half always 1 in ar_row, so two copies count as one) makes the same run exit 1: "10 of 208 items in 37 cases differ". The cases hom-two and cyp21a2-called turn red, with the risk 2 in the oracle against 4 in og. A wording change made through an og wrapper also turns it red (21 of 198 items in 34 cases).
  • Mutation controls on the moved tests: SV_MIN_SIZE 49, PARAPHASE_REGIONS 164 and VWF renamed in the catalog each turn their test red (length_and_common_deletions, the_region_count_in_the_docs_and_the_step, one_owner_per_locus).
  • cargo test -p og-couple -p og-summary, clippy with -D warnings, cargo fmt --check and rustdoc with -D warnings pass on eb449b8. The Python tests below also pass with og from eb449b8.
  • The Python tests that called couple_report now run og couple screen and pass with this head's og: test_acmg_sf_rules, test_cross_engine_biallelic, test_render_report, test_copy_number, test_collect_summary. The demo builder runs. The fake-docker case run-all-families passes. scripts/ci/e2e-scope.sh check and og dev remnants pass.
  • Speed: the synthetic couple has 200,000 ClinVar hits and 100,000 Delly deletions each, 300,000 benign SV rows and 10 exons per gene. Each side ran twice under /usr/bin/time -l. The Python took 25.5 s and 28.8 s, at 327 MB and 329 MB peak. og took 3.0 s and 3.5 s, at 206 MB. The four output files are the same bytes.
  • og couple genes on the public Orphadata (2026-06-23) and GENCODE v50: 4.9 s and 41 MB, where the Python took 8.7 s and 320 MB. With commit 1's code the rebuild gives the committed list byte for byte. Commit 2's list is that rebuild with the new rule.
  • fdd6af2 and 355f764 merge main (aa8b7af, then 84f0e38 with the Rust report frame) into the branch. CHANGELOG.md, docs/rust.md, tests/parity/cases.tsv and tests/parity/oracle.ref had conflicts; each keeps both sides, and docs/rust.md drops og couple screen|genes from the list of stubs. On 355f764, cargo test -p og-couple -p og-reports -p og-steps -p og-summary -p og, workspace clippy with -D warnings and cargo fmt --check pass. With og built from 355f764, test_acmg_sf_rules, test_cross_engine_biallelic, test_render_report, test_copy_number and test_collect_summary pass.
  • Fast checks on 355f764: Rust https://github.com/GeiserX/OwnGenome/actions/runs/43865799968, Lint https://github.com/GeiserX/OwnGenome/actions/runs/43865800044, Nextflow https://github.com/GeiserX/OwnGenome/actions/runs/43865800100, Docs https://github.com/GeiserX/OwnGenome/actions/runs/43865799845 and Personal data https://github.com/GeiserX/OwnGenome/actions/runs/43865800261 are green. In Guard https://github.com/GeiserX/OwnGenome/actions/runs/43865799812 the rust, og, parity, fake-docker and coupled-versions jobs are green; the unit-test shards, helper-binaries and images, and Container Test https://github.com/GeiserX/OwnGenome/actions/runs/43865799897, were still in the runner queue when this was written. On fdd6af2, every Guard job except images was green, and so was Container Test https://github.com/GeiserX/OwnGenome/actions/runs/43847673997.
  • Guard https://github.com/GeiserX/OwnGenome/actions/runs/43847674386 on fdd6af2 is red in one job, images, and every other Guard job (unit tests, rust, og, parity, fake-docker) is green. The red is not in the couple code. The self-test of scripts/ci/check-images.sh copies the tree with git ls-files -z | xargs -0 tar -cf - | tar -xf -. This PR adds 83 files, and the list of tracked paths grows from 128,903 to about 133,000 bytes. That is past GNU xargs' 128 KiB limit, so xargs runs tar twice. The reading tar stops at the end of the first archive, and the second tar fails with a broken pipe. GNU xargs on Ubuntu 24.04 split the list of eb449b8, the head before the main merge, into 3014 and 45 paths, and main's list stays in one batch of 2991. The fix is one line in that script (git ls-files -z | tar --null -T - -cf -), outside this package's files, so it is left to its own change. Any change that grows main's file list by about 2 kB hits the same red.

Red first

  • 2ce5dc3 geiresx@2ce5dc36 puts the new tests/inheritance_and_confidence.rs on the parity commit. cargo test -p og-couple --test inheritance_and_confidence fails all 7 tests there. The dominant genes read "not affected". The list has no dominant column and no XLD. PAR1 and the haploid call both give 1 in 2. The weak call has no mark. og couple genes refuses the new column. On 9d4e9e0 all 7 pass.
  • a_partner_with_both_copies_of_a_dominant_gene_passes_it_to_every_child (commit 3) fails on 9d4e9e0: offspring_risk is "each child 1 in 2, dominant" where "every child, dominant" is expected. It passes on c206e13 with the other 7 tests of the file.
  • The five tests of eb449b8 fail on b72696c's code (run with cargo test -p og-couple --no-fail-fast): a_truncated_or_rootless_file_is_refused, two_couples_with_one_file_name_are_refused, two_changes_in_a_dominant_gene_say_the_chance_can_be_higher, both_partners_in_one_x_linked_recessive_gene_give_the_daughters_risk and the PAR part of a_haploid_call_on_an_autosome_is_one_copy. All pass on eb449b8.
  • The planted parity controls above show the parity job goes red on a carrier-decision change.

Beads

D1, D14, clin-16, clin-20, clin-21, clin-31. Part of #411.

Decisions

  • All-or-nothing writes. The screen writes no file until every couple is computed. The PR title is this fix. The all-or-nothing golden holds it.
  • Rule (y) as code. An X-linked gene is XLD when every X-linked disorder Orphanet gives it is dominant. A gene with dominant and recessive disorders is XLD only when og couple genes names it: CDKL5, IKBKG, MECP2 and OFD1. Orphanet has no "main disorder" field, so these four genes are named by hand. Every other mixed gene stays XL and gets the "females can show signs" mark. An XLD gene's condition names a dominant disorder first, so IKBKG reads "Incontinentia pigmenti".
  • A column, not new values. The dominant marks are a new dominant column. No column is renamed. AR keeps its meaning, so the sample report's two-hit rule, which reads AR, does not change.
  • The AD and SD marks are kept by hand. og couple genes keeps them like the severity column, and a test holds them equal to assets/acmg_sf.tsv. So og couple genes takes no new option, and its parity cases still run the Python with the same arguments.
  • Built-in PARs, not a required --par-bed. Both callers pass the file, so a pipeline run does not change. A hand run without it no longer reads PAR1 as hemizygous.
  • A weak call still counts. Dropping it would hide a real carrier. Its finding is marked, its row says to confirm it, and the report's table counts the weak calls.
  • The result values stay the same. A one-partner row in a dominant gene keeps one_partner, because the summary reads only shared, x_linked and one_partner. Its offspring_risk reads each child 1 in 2, dominant, or every child, dominant when that partner carries the change on both copies.
  • Two heterozygous changes in a dominant gene keep each child 1 in 2, dominant in the risk cell. The phase is unknown, so the explanation says the chance is higher when they sit on different copies, as the shared rows already do.
  • ClinVar's placeholders are left out. "not provided" and "not specified" are dropped from the condition.
  • og-summary carrier.rs is unchanged. It passes the inheritance value through, so the sample's own carrier list shows XLD with no code change.

Outputs that change

  • couple_<a>_<b>.tsv, .md and the partner copies change in these places: the wording of dominant and XLD rows, the offspring_risk of a dominant one-partner row (each child 1 in 2, dominant, or every child, dominant for two copies), and of an X-linked recessive row where both partners carry a change (it adds the daughters' risk). Findings get a condition and a low-confidence mark. Haploid calls and PAR1 calls without --par-bed read one copy.
  • Two couples with one file name, and a cut-off Orphadata file, now stop the step. No column changes.
  • assets/carrier_genes.tsv: a new dominant column, the XLD value for 28 genes, and header lines that name og.
  • The screen writes nothing when one couple fails.

Files outside the package list, and why

  • assets/paralog_loci.tsv, assets/paraphase_genes.tsv, modules/local/parascopy/main.nf, scripts/35-paralogs.sh, scripts/ci/build-callability.sh: comments and an error message named the deleted Python tests. They now name the cargo tests.
  • tests/e2e/paralog-rows-1.sh: it read the Paraphase region count from tests/test_paraphase_report.py. It now reads const PARAPHASE_REGIONS from rust/crates/og-steps/tests/paraphase_report.rs.
  • rust/crates/og-steps/tests/paralog_loci.rs and paraphase_report.rs: the moved assertions of the two deleted catalog tests.
  • rust/crates/og-steps/src/sv.rs: a doc comment named bin/couple_report.py.
  • bin/family_report.py: die, read_par and in_par lost their last importer, so they go, and tests/parity/remnants.tsv says so.
  • tests/test_acmg_sf_rules.py, tests/test_cross_engine_biallelic.py, tests/test_copy_number.py: they imported couple_report. They now run og couple screen.
  • tests/fake-docker/run-all-families.sh: the expected step 39 command line is og's.
  • docs/35-paralogs.md, docs/cn-loci.md: they named the deleted tests.
  • rust/Cargo.lock: og-couple's dependencies.
  • rust/README.md: the og-couple row no longer says "not ported yet".
  • scripts/ci/check-personal-data.sh: one SKIP row for tests/parity/inputs/couple-screen/*_eh*.vcf. These three invented ExpansionHunter VCFs carry repeat counts beside REPCN, which the scan reads as a genotype. The Personal data job failed on c206e13 for this reason only: https://github.com/GeiserX/OwnGenome/actions/runs/43754924291
  • tests/parity/gen/couple-oracle.sh, tests/parity/inputs/couple-screen/, tests/parity/golden/: the parity cases.

Where each deleted assertion went

Deleted test Section Now
test_couple_report.py 1 the carrier list og-couple couple_report.rs the_carrier_list_holds_the_panels
2 shared, one partner, outside the list a_shared_gene_deletions_over_exons_and_the_report
3 Delly deletions over exons a_shared_gene_deletions_over_exons_and_the_report, read_depth_decides_long_deletions, length_and_common_deletions
4 homozygous, two hets homozygous_and_two_hets
5 X-linked genes and repeats x_linked_genes_and_repeats
6 SMN1 and CYP21A2 not assessed smn1_and_cyp21a2
7 residual risk, partner copies a_shared_gene_deletions_over_exons_and_the_report
8 red controls red_controls
9 same inputs, same bytes a_shared_gene_deletions_over_exons_and_the_report
10 genes rebuild genes_rebuilds_the_list
11 carrier deletion alleles, AZF, shared risk carrier_deletion_alleles_and_azf, read_depth_alleles_add_a_shared_risk
12 a split half call is one copy unit test zygosity_counts_a_missing_allele_in_the_ploidy, a_half_call_is_one_copy
test_paralog_loci.py the model list og-steps paralog_loci.rs the_model_list
the catalog and its planted faults the_catalog_and_its_controls
one owner per locus, callability one_owner_per_locus
the readers (summary, SMN1 rule, couple screen) og-couple paralog_readers.rs step_35s_not_assessed_smn_table_names_its_reason_everywhere
test_paraphase_report.py 1 check-bam og-steps paraphase_report.rs check_bam_reads_the_platform
2 and 3 the copy-number and variant tables table_reads_each_region
4 SMN1 beside Paraphase og-couple paralog_readers.rs smn1_from_parascopy_and_beside_paraphase
5 the region catalog the_region_catalog
6 the region count the_region_count_in_the_docs_and_the_step
7 og's own fixes og_s_own_fixes_in_the_table

Not done here

  • No real-genome run. The e2e cases couple-carrier-1 and couple-common-sv-1 need container mounts this box cannot give, so they were not run here. E2E stays with the landing train.
  • The live family page (assets/live/family-page.js) still writes "Carriers are not affected" under every shared 1 in 4 row, a shared dominant gene included. The fixed wording reaches the couple .md and .tsv only. The page is outside this PR's files and belongs to the live-site port.

With OWNGENOME_MEM_WARN_PERCENT or OWNGENOME_DISK_WARN_PERCENT set in the
shell, the Rust card read them and the Python oracle did not, so four
checks failed. The tests now pass 90 to both sides (or the case's own
disk percent); the default path stays covered in ops_headroom.rs.

Bead: D1
The pin ran on a window with no read over HBA2, so dropping the other
half of them passed. The test now checks the first read goes and the
second stays, as the inline Python did.

Bead: D1
…t, and keep main's views

The person page now reads who is a child from state.json's family plugin, which the server computes from
the family files on every request by one rule (child_of): a child in any family is one, and while a family
folder cannot be read, the people it might list are closed alike on their page, home, the family page and
the finding route, and reopen as soon as the files read again. An adult's page reads only the state through
the gate. A child's results answer is checked on the text the gate holds, so answers reach the views in the
order they arrived. The map's tooltip starts hidden="hidden" again, as on main. On the family page a released
child shows summary.json's count (results.json's saved_attention), adults keep their own member line, and a
gated partner keeps the partner role and the couple line.
…-based child rule and the family page's roles

The live section-address check now asserts the outcomes main's own view files give, for an adult and a
no-act child, with the family first and with the results first, and the map tooltip's hidden="hidden".
New checks: a family file that cannot be read closes the people that family lists on the server, in
state.json, on the finding route, on their page and home tile, before any family.json request, and all
reopen once it reads; a released child shows summary.json's count; an adult's member line is unchanged
when its summary.json cannot be read; a gated partner is not counted as a child and keeps the couple line;
a child in any family stays one when the family where it is a partner sorts first.
…the whole block

Nextflow merges -c files key by key: an executor block of the user's that
sets only cpus keeps the engine's memory from executor.config.

Bead: plat-18
…at run-all queues the step for

run-all.sh queues the benchmark step for any vcf_*/<sample>.vcf.gz, so
skipping vcf_strelka2/<sample>.vcf.gz left a DeepVariant plus Strelka2
sample with one caller and exit 1. It now counts when step 03c's
results/variants/variants.vcf.gz is absent.

Bead: plat-71
Re-measured on the synthetic 1,000-person, 100,000-site panel: 0.9 to
1.0 s and 13 to 14 MB over five runs.

Bead: D5
…oom tests ignore the shell's warn percents

Ports checks 14 and 15 of the deleted Python suite: while bin/owngenome_live.py
lives, it must serve plugins.ops and /metrics, and a hook error must stay on
the card. Two controls run the site on a plugin copy with its hook unregistered
and with its catch-all cut. The ops_headroom tests pass explicit warn percents,
and the environment check requires its child test to have run.

Bead: D1
…n too

The parent's page compared whichever card holds a child's results raced the family against the map's
document, so the tooltip ended hidden="hidden" or hidden="" by who won, as on main. Hold map.json back
half a second as the other live comparisons do, and print where the pages differ.
… way the docs say

cm_at took the first marker at a first position that holds two, while
every other position takes the last one. pos_at returned the last
position of an inner or trailing flat stretch, where its doc promises
the first. Both now do what the docs say; the tests cover both cases.

Bead: D5
…puts it cannot write

A T or switch rate of about 1e16 per Morgan or more left the event's
place in cM unchanged, so the draw never ended; both are now capped at
10,000 per Morgan. og ancestry-fine simulate now refuses a chromosome
named twice in --chroms, an --out or --truth that is a folder, and one
file for both, before it writes anything; a folder as --truth used to
leave the VCF in place without its truth.

Bead: D5
…arts

Under load the read right after the last click could meet the reload that opening starts, and Chrome
answered 'Inspected target navigated or closed'. That read now counts as released and waits for the
opened page, as it already did when the read came after the reload.
…the landing train

# Conflicts:
#	CHANGELOG.md
…, and check them

The contributor guides still sent output notes to the overview page, and
the report's repeat note named the overview for the thresholds table that
now lives in docs/results/structural-and-repeats.md. The lint link loop
and the T1K/Parascopy version check globbed docs/*.md only, so the new
chapter pages were outside both. A planted wrong T1K version on a chapter
page now fails the version check; before, it passed.
…nding train

# Conflicts:
#	CHANGELOG.md
#	docs/rust.md
…ts .tbi, must still read A2

Both are read once from the start. Before the fix step 41 stopped on each:
bcftools writes no record on an undeclared contig, and bcftools loads the
stale .csi before the current .tbi.

Bead: trait-30
…tigs before streaming

The freshness check now looks at the .csi when there is one, as htslib
loads it first. A VCF read once from the start gets a ##contig line for
each marker window contig its header lacks, so bcftools can write its
records.

Bead: trait-30
…er engine

When THREADS, --max_cpus or --max_memory was above the engine's figure,
executor.config still held the engine's CPUs and memory, and Nextflow
refused every task capped above the pool. The pool now rises to a cap
the user set, and leaves out a line whose cap it cannot read.
…::Options now carries

The always-report sites reader gained a depth floor (min_dp, min_dp_value) in the car before, so the struct literal in this
test no longer compiled. It now passes the CLI default, 10.
…rain

# Conflicts:
#	CHANGELOG.md
#	docs/rust.md
#	rust/README.md
#	scripts/ci/e2e-case-paths.tsv
#	tests/e2e/paralog-rows-1.sh
@geiresx
geiresx requested a review from GeiserX as a code owner November 16, 2026 20:00
GeiserX and others added 11 commits November 16, 2026 22:12
… each other forever

The run-all-launcher case's small-engine shim forwarded every call but
'docker info' to the next docker on PATH, skipping only its own folder.
The e2e job puts tests/e2e/bin/docker on PATH, which also skips only its
own folder, so write_manifest.sh's 'docker image inspect' bounced between
the two with no output until the case's 3600 s timeout. The shim now drops
its folder from PATH before it hands the call on.
… run-all-launcher looping with the e2e docker shim
…rX#462 into pgp/port-couple: land them with the batch train
…AT line

Step 41 now streams the records near the markers through bcftools as BCF
before og reads them. BCF refuses a record that uses a tag the header does
not declare, so a VCF with GT records but no ##FORMAT=<ID=GT> line (as the
platelet-antigens-1 case writes, and as a hand-made or converted VCF can be)
stopped the step with exit 255. Before the left-align change og read such a
VCF directly and accepted it.

The picked records now go out as VCF text, and a header line is added for
each FILTER, INFO and FORMAT tag they use and the header lacks. A VCF that
declares every tag passes through unchanged, so its table and evidence stay
the same. The BLOODTYPE process gets the same change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants