Repository navigation
Conversation
…acks a chromosome or is unsorted in its tail
… document the gVCF stops
… for the landing train # Conflicts: # CHANGELOG.md
…thout one With the archaic table installed, a BAM+VCF row with no gVCF beside the VCF, and a VCF-only row, list and record step 43 as skipped (no gVCF: ...) and nextflow gets no --archaic_data; the same row with its gVCF runs step 43.
… landing train # Conflicts: # CHANGELOG.md
Red on main: og pgx mito-merge holds back every named allele below AF 0.95. Bead: pgx-32
…Clair3 and any vcf_*/ caller Bead: plat-71
…ts of a combined hit A synthetic sample with a split T>A,C call where only T>A is a step 6 hit, and a combined hit whose left-aligned position matches neither part. Red on main: the tier holds T>C and misses both parts. Bead: clin-22 Bead: clin-23
…ified Step 23 picked VEP records by the hits' CHROM and POS, so another allele at a hit's position entered the ClinVar tier, and a combined hit, whose joined allele VEP never saw, dropped out. The tier now keys on CHROM, POS, REF and ALT, and a combined hit gives its PGP_JOINED_FROM parts. Bead: clin-22 Bead: clin-23
Red-first tests for decision D10: a call below GQ 20, or one the caller filtered, at APOE, HFE, SERPINA1 or a traits-table row reads low_gq, and the summary counts low_gq rows. They fail on main, which reads such calls as called. Bead: trait-01 Bead: trait-67 Bead: D10
… refuse a licence outside the allow list Red first: against the scaffold stub all four tests fail (the verb says "not ported yet"). Bead: lic-02
CPIC sets no heteroplasmy level at which aminoglycosides are safe, so a named MT-RNR1 allele at AF 0.05 or more is now the call and the log gives its level. Below 0.05 it is held back as below the detection floor, and no Reference is passed. Bead: pgx-04 Bead: D10
…rong A filtered 0/0, a 0/0 at depth 1, a 0/0 without GQ, a haploid 1, a haploid 0 and a ./. record each get a test of the words they must read. All six fail on the current reader. og clinvar deep-intronic also takes --min-dp, the step's depth floor (default 10), unused until the fix. Bead: clin-14
…ost's Docker Desktop and colima run the containers in a VM that is usually smaller than the host, but run-all.sh gave Nextflow the host's CPUs and RAM, so Nextflow booked tasks the VM could not hold and they died with exit 137. When docker info reports less, run-all.sh now passes the engine's figures as the default --max_cpus and --max_memory and writes them to <sample>/nextflow/executor.config, passed with -c before the user's options. On native Linux nothing changes. validate-setup.sh adds an INFO line in that case. The launcher e2e case checks it with a docker shim that reports 8 GB and 2 CPUs, with a negative control. Bead: plat-18
…lair3 is compared Bead: plat-71
…wn page docs/interpreting-results.md held every chapter in one 883-line file. Each chapter now has a page under docs/results/, moved word for word. The overview keeps the Before You Look bullets, the Before You Panic table, the example outputs and the general advice, and lists the chapter pages. Every old #anchor on the overview lands on a link to the section's new place. The root .gitignore drops every results/ folder (Nextflow output), so the seven pages are added with -f; once tracked, git follows their edits. The repeat table moved with its chapter, so tests/test_str_ranges.py reads it from docs/results/structural-and-repeats.md.
…sites One state table decides each site's words. Only FILTER PASS, '.' and RefCall leave a reference call plain; any other FILTER makes it "ref (low confidence)" and names the FILTER. A reference call with no GQ is low confidence too, since nothing checked it against the floor. A depth below the step's depth floor reads "ref (shallow)". On chrX and chrY outside the pseudoautosomal regions a one-copy 1 is "hemizygous" and a one-copy 0 a reference call; elsewhere a one-copy GT is no call, with why. A no-call record says what it holds (no genotype, not a reference call) and never claims a PASS check or that no record exists. Bead: clin-09 Bead: clin-10 Bead: clin-11 Bead: clin-12 Bead: clin-13 Bead: clin-14
…ep 03a runs without BQSR Bead: plat-55 Bead: plat-71
og-live's ops module gives each run its hours left, memory and disk headroom, last activity and five alerts, and writes the Prometheus text, from og-runlog's eta and headroom instead of the Python run_eta and run_headroom modules. og-runlog gains a fault-inject feature: named planted mistakes the red controls turn on per thread; without the feature none is compiled in. Bead: D1
…ader, and count REF mismatches Both mirrors of step 06 query each site's FILTER as a ninth column and pass the step's depth floor (CLINVAR_MIN_DP, ext.min_dp) to og clinvar deep-intronic. The sample's bcftools norm -c w keeps a record whose REF is not the reference's and only warns REF_MISMATCH; the step now counts those records in its summary and prints a notice when there are any, since such a record cannot match a ClinVar allele. Bead: clin-09 Bead: clin-10 Bead: clin-13
…s for the new calls The parity goldens record the deliberate changes against the old Python: AF 0.4 is now the call, --min-af 0.3 logs it as heteroplasmic, and a nan AF is held back as having no allele fraction. The unit test of step 07 expects the call down to AF 0.05. Bead: pgx-32
…peat map The fake-docker cases fail on main: step 04c runs Sniffles2 on a short-read BAM and exits 0, never passes --tandem-repeats, and its closing hint sends long-read SVs to duphold for a GQ. Bead: sv-18 Bead: sv-17
…te query in both mirrors Step 06 runs to its summary against a fake bcftools that warns REF_MISMATCH twice, then not at all: the count line reads 2 and then 0, and the notice shows only for 2. The always-report call gets the depth floor, and the query_format of the script and of the module, run with the Groovy escapes undone, both end with %FILTER. Red on main: the count line is missing, and the module's format has no FILTER. Bead: clin-13
… og links It walks cargo metadata per target from og along normal edges, checks each crate's licence against an allow list, and writes each licence text once, including the C library texts of crates that set links (htslib and htscodecs from hts-sys). Bead: lic-02
rust/og.sh build writes it in OG_BUILDER with the new og; release.yml copies it to dist, adds it to SHA256SUMS and the attestation, and checks the upload with check-release.sh dist. Bead: lic-02
… warning Step 07, the PHARMCAT module comment, the step 07 and step 20 docs and the CHANGELOG now describe the AF 0.05 floor. The pharmcat-mito e2e case expects the outside call and the drug recommendation at AF 0.40, and adds a sample at AF 0.03 that stays No Result with a below-floor line. Bead: pgx-04 Bead: pgx-32 Bead: D10
The self-test copied the tree with git ls-files -z | xargs -0 tar -cf -. GNU xargs splits a list over 128 KiB into two tar runs, the reading tar stops after the first archive, and Guard's images job fails on a broken pipe. One tar now reads the NUL-separated list itself.
… the landing train
… the landing train
…the landing train
…::Options now carries The always-report sites reader gained a depth floor (min_dp, min_dp_value) in the car before, so the struct literal in this test no longer compiled. It now passes the CLI default, 10.
…the landing train
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #411. Settles owner decision D1 (Python goes away for 1.0.0) for the PGx tool tests.
Problem
Six Python files (about 1,900 lines) tested the
og pgxverbs by running a builtogbinary on fixtures. The code is Rust now. The checks lived far from it, needed Python and a built binary, and the unit-tests job skipped most of them because it has noog. A change to the PGx code could passcargo testand only fail later in a separate Guard leg.Fix
Every check moves into
cargo test, in the crate that holds the code, on the same fixtures:og-pgx:cpic-reportandstep36-outside(tests/port_tests_pgx_cpic.rs), and what step 27 makes of step 36's table (tests/port_tests_pgx_outside.rs).og-steps:outside-calls,hla-concordanceandpypgx-summary(tests/pgx_outside_calls.rs,tests/pgx_hla_second_caller.rs).og-filters:mito-mergeandrefblock-fill(tests/port_tests_pgx_mito.rs,tests/port_tests_pgx_refblock.rs).og-bam: what the CYP2D6 depth goldens must mean (tests/cyp2d6_depth.rs).The six Python files are deleted. No output, option or behaviour of
ogchanges, so there is no CHANGELOG entry.Each Rust assertion carries the old check's text as its message, so a search for an old check finds its new test.
Decisions
PATHrun in a child copy of the test binary (a hiddenog_childtest that returns at once in a normal run). So they are checked as theogbinary runs them, with no new crate dependency and noCargo.tomlchange.og-pgxandog-stepsdo not depend on each other. So the step 27 cases handcpic-reportthe consensus tables as literal text, andog-stepshas a test (consensus_tables_the_cpic_report_tests_read) that holdsoutside-callsto exactly those tables.tests/fixtures/pgxandtests/fixtures/pharmcat, unchanged. The og-bam parity generator, the parity harness and e2e cases read them there, so moving them would change paths and buy nothing.tests/parity/cases.tsv, a doc comment inog-summary) still name the old Python test. They say what tested the rule before; they stay true.Files outside the package list
.github/workflows/guard.yml: one new step, "PGx test controls" (below).CLAUDE.md,docs/27-cpic-lookup.md,docs/testing.md,tests/fixtures/pharmcat/README.md: they named the deleted tests; they now name the cargo tests.modules/local/cpic_lookup/main.nf: a comment line, same reason.assets/pgx_inhibitors.tsv,assets/pgx_contraindications.tsv: one header comment line each says which test checks the rows. No row changes.rust/crates/og-pgx/tests/fixtures.rs: its doc comment named the deleted test.guard.ymlanddocs/testing.mdare also edited by other lanes of this wave, in other places. A conflict there is a mechanical merge. Main now adds its own control step at the same place inguard.yml(the F_ROH fix); keeping both steps resolves it.Old check to new test
test_cyp2d6_depth.py: status, message, the eleven rows and the shares of the five depth tables, with and without the measure fileog-bamcyp2d6_depth::each_depth_table_gives_its_verdict_and_the_sharestest_cyp2d6_depth.py: clean_no_alt MAPQ >= 1 share 0.91 and the four depthscyp2d6_depth::clean_no_alt_depths_and_sharetest_cyp2d6_depth.py:cyp2d6-bedprints CYP2D6 between two flankscyp2d6_depth::cyp2d6_bed_prints_the_gene_between_two_flankstest_cyp2d6_depth.py: six bad inputs exit 2 and write nothingcyp2d6_depth::bad_inputs_exit_2_and_write_nothingtest_cyp2d6_depth.py: the three measure checks on synthetic readscyp2d6_depth::measure_shares_on_synthetic_readstest_hla_second_caller.pycases 1 to 8 (agree, disagree, homozygous, no call, asked but missing, T1K alone, quality 0, concordance table)og-stepspgx_hla_second_caller::case1_*tocase8_*test_hla_second_caller.pycase 9, the control without OptiTypepgx_hla_second_caller::case9_control_without_the_agreement_rule_case2_failstest_pgx_outside_calls.pycases 1 to 7 (both files, byte for byte)og-stepspgx_outside_calls::case1_*tocase7_*test_pgx_outside_calls.pycases 8, 9, 10 (step 27 on the consensus table, with their controls)og-pgxport_tests_pgx_outside::case8_cyp2d6_held_back,case9_hla_passed_on,case10_a_stale_reporttest_pgx_outside_calls.pycase 13pgx_outside_calls::case13_pypgx_alone_passes_nothingtest_pgx_outside_calls.pycase 14: fixture VCF, the summary row by row and its log, tables beside the pypgx on PATH, no tablespgx_outside_calls::case14_*(four tests)test_pgx_outside_calls.pycase 15, both files or neitherpgx_outside_calls::case15_both_files_or_neithertest_pgx_mito_merge.pychecks 1 to 7og-filtersport_tests_pgx_mito::case1_*tocase7_*test_pgx_refblock_fill.pychecks 1 to 3 (five filled, four left, replaced in place, sorted, header, summary line)og-filtersport_tests_pgx_refblock::the_fixture_fills_five_and_leaves_fourtest_pgx_refblock_fill.pycheck 4, the GQ thresholdport_tests_pgx_refblock::the_gq_thresholdtest_pgx_refblock_fill.pycheck 5, edges and refused inputsport_tests_pgx_refblock::edges_that_fill_nothing_and_inputs_that_are_refusedtest_cpic_parser.py1: the real HG002 reports of 3.2.0 and 3.4.0og-pgxport_tests_pgx_cpic::real_hg002_reports_parse_and_ambiguous_genes_stay_ambiguoustest_cpic_parser.py1: no gene, no gene sectionport_tests_pgx_cpic::a_report_with_no_gene_fails_loudtest_cpic_parser.py2: PharmCAT's example report, VKORC1 -1639 GA, codeine for CYP2D6 *1/*3port_tests_pgx_cpic::the_example_report_has_three_non_normal_and_seven_unclassified_genestest_cpic_parser.py2: drug-list fallbacksport_tests_pgx_cpic::drug_list_fallbackstest_cpic_parser.py2: the called diplotype's guidance, the exact phenotype's guidanceport_tests_pgx_cpic::the_recommendation_is_the_called_diplotypestest_cpic_parser.py3: layoutsport_tests_pgx_cpic::flat_nested_list_and_recommendation_layouts_parsetest_cpic_parser.py4: pypgx call where PharmCAT has noneport_tests_pgx_cpic::pypgx_call_where_pharmcat_has_no_resulttest_cpic_parser.py4: an ambiguous gene in the comparisonport_tests_pgx_cpic::an_ambiguous_gene_is_compared_as_ambiguoustest_cpic_parser.py4: an unreadable pypgx summaryport_tests_pgx_cpic::an_unreadable_pypgx_summary_is_reported_not_fataltest_cpic_parser.py5: Indeterminate at gene level, one possible diplotypeport_tests_pgx_cpic::an_indeterminate_gene_takes_one_drug_level_phenotypetest_cpic_parser.py5: Indeterminate at gene level, two possible diplotypesport_tests_pgx_cpic::an_indeterminate_gene_with_two_possible_diplotypestest_cpic_parser.py6: single-position labels and the comparison headerport_tests_pgx_cpic::single_position_labels_compare_by_meaningtest_cpic_parser.py6: a long recommendation prints wholeport_tests_pgx_cpic::a_long_recommendation_prints_wholetest_cpic_parser.py6: CYP4F2 and VKORC1 single-position fallbackport_tests_pgx_cpic::cyp4f2_and_vkorc1_single_position_fallbacktest_cpic_parser.py7: MT-RNR1 outside call, its control,step36-outsideport_tests_pgx_cpic::an_mt_rnr1_outside_call_is_not_checked_against_step36test_cpic_parser.py8a: the inhibitor table's shapeport_tests_pgx_cpic::inhibitor_table_has_the_shape_the_reader_readstest_cpic_parser.py8b: strong CYP2D6, CYP2C19 note, unknown name, a leading separatorport_tests_pgx_cpic::a_strong_cyp2d6_inhibitor_phenoconverts_and_cyp2c19_gets_a_notetest_cpic_parser.py8c: moderate inhibitor, no activity scoreport_tests_pgx_cpic::a_moderate_inhibitor_halves_the_activity_scoretest_cpic_parser.py8d: a list that names no drugport_tests_pgx_cpic::a_list_that_names_no_drug_changes_nothingtest_cpic_parser.py8e: control, factor set to 1port_tests_pgx_cpic::control_a_factor_of_1_converts_nothingtest_cpic_parser.py8f: NaN score, broken inhibitor rows,--medicationswithout--inhibitorsport_tests_pgx_cpic::a_nan_score_or_a_broken_inhibitor_row_converts_nothingtest_cpic_parser.py8g: CYP2C9, CYP3A5, a boosted combinationport_tests_pgx_cpic::cyp2c9_and_cyp3a5_inhibitors_give_a_notetest_cpic_parser.py9a: the contraindication table's shapeport_tests_pgx_cpic::contraindication_table_has_the_shape_the_reader_readstest_cpic_parser.py9b: pathogenic POLG and the hits that add nothingport_tests_pgx_cpic::a_pathogenic_polg_hit_adds_the_valproate_warningtest_cpic_parser.py9c: no or an unreadable hits table,--clinvar-hitsaloneport_tests_pgx_cpic::no_or_an_unreadable_hits_tabletest_cpic_parser.py9d: control, no POLG rowport_tests_pgx_cpic::control_without_the_polg_row_no_valproate_lineHow it was checked
cargo fmt --all --check,cargo clippy -p og-pgx -p og-filters -p og-steps -p og-bam --all-targets --all-features -- -D warningsandcargo test --locked --no-fail-fast -p og-pgx -p og-filters -p og-steps -p og-bam: clean and green on this head. The new targets: cpic 27 tests, outside 3, outside-calls 15, HLA 9, mito 8, refblock 4, CYP2D6 depth 5. Each target runs in under 0.2 s.og-steps), the moderate-inhibitor activity score left unscaled and the stale outside-call rule switched off (og-pgx),MIN_AF0.94 andMIN_GQ19 (og-filters), and the multi-mapped golden's status set to ok (og-bam). The step fails if any one fault changes nothing or if any of the seven named tests passes:case2_optitype_disagrees_at_hla_b,case3_both_agree_and_the_depth_check_passed,a_moderate_inhibitor_halves_the_activity_score,case10_a_stale_report,case2_the_af_bound,the_gq_thresholdandeach_depth_table_gives_its_verdict_and_the_shares. Run before push with the same script: all seven FAILED. Run again with only the two new faults, then with only the five old ones: each named test failed only under its own fault, so no fault hides another. CI on head 8e13464: the step passed and its log shows all seven named tests FAILED, thencargo testof the whole workspace passed with the sources restored: https://github.com/GeiserX/OwnGenome/actions/runs/43767665815/job/132988129110mito-mergeandrefblock-filltests require exit 0 and a written output before they check that nothing was called or filled. Shown red on the box: an invalid--min-afin the AF 0.949 run and an invalid--min-gqboth fail their test with the exit-0 message. The no-tables summary case runs with an empty folder asPATH, so an installed pypgx cannot reach it.Bead: D1
Summary by CodeRabbit