From 360dc797dd466565d030fc424ea909ad30d45049 Mon Sep 17 00:00:00 2001 From: Ian Driver Date: Tue, 17 Nov 2026 09:28:49 -0500 Subject: [PATCH] CellReads.stats values as STARsolo writes them --soloCellReadStats takes STAR's value, Standard (CB stays as a deprecated alias), and --genomeChrSetMitochondrial defaults to STAR's chrM M MT. Each feature keeps its own table; region columns come from the feature's own overlap type; every read is counted under cbMatch; one-mismatch and 1MM_multi barcodes are resolved with the run's exact counts, as the matrix resolves them; featureM, countedM, nUMImulti and nGenesMulti are filled. Rows are folded in input order by the in-order consumer. On CellRanger's tiny 10x data every value matches STARsolo 2.7.11b except on the 3 barcodes of 3 reads rustar-aligner maps and STAR leaves unmapped. Row order still differs from STAR's (which was the same at 1 and 8 threads); DIVERGENCE.md 3.2 now says what was measured. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 19 + DIVERGENCE.md | 12 +- .../content/docs/reference/cli-parameters.md | 2 + src/lib.rs | 11 + src/params/mod.rs | 32 +- src/solo/cell_reads.rs | 645 ++++++++++++------ src/solo/count.rs | 79 ++- src/solo/mod.rs | 165 +++-- tests/solo_cell_read_stats.rs | 334 +++++++++ 9 files changed, 993 insertions(+), 306 deletions(-) create mode 100644 tests/solo_cell_read_stats.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ab8f0e6..af3c498d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -232,6 +232,25 @@ Sections commonly used: Features, Bug fixes, Other changes. ### Bug fixes +- **`CellReads.stats` as STARsolo writes it.** `--soloCellReadStats` now + takes STAR's value, `Standard`; `CB`, which rustar-aligner took before, + is kept as a deprecated alias. `--genomeChrSetMitochondrial` defaults to + STAR's `chrM M MT` instead of naming nothing. The file's values now follow + STAR's (`SoloReadFeature_record.cpp`, `SoloReadFeature_inputRecords.cpp`): + each feature keeps its own table, so `Gene` and `GeneFull` no longer print + the same rows, and the region columns come from the feature's own overlap + type (`exonic` for a `Gene` read with a gene, nothing for `GeneFull`) + instead of CellRanger's positional regions. Every read is counted under + `cbMatch`, including those whose barcode or UMI was rejected; one-mismatch + corrections and 1MM_multi barcodes are resolved with the run's exact counts, + as the matrix resolves them, so `cbMMunique`/`cbMMmultiple` and the rows + match; `featureM` and `countedM` are filled; and `nUMImulti`/`nGenesMulti` + come from the first `--soloMultiMappers` matrix. Rows are folded in input + order by the in-order consumer, so their order no longer depends on thread + timing; it is still not STAR's order (DIVERGENCE.md 3.2). On CellRanger's tiny 10x data every value matches STARsolo 2.7.11b + except on the 3 barcodes of 3 reads rustar-aligner maps and STAR leaves + unmapped. + - **STARsolo's stats files as STAR writes them.** `Solo.out/Barcodes.stats`, `/Features.stats` and `/UMIperCellSorted.txt` were not written, SJ and Velocyto had no `Summary.csv`, and the Gene/GeneFull diff --git a/DIVERGENCE.md b/DIVERGENCE.md index f0890fbd..166ceb06 100644 --- a/DIVERGENCE.md +++ b/DIVERGENCE.md @@ -225,17 +225,15 @@ There are none at present. The four yeast PE reads once listed here now match ST ### 3.2 `CellReads.stats` row order -**What STAR does.** `--soloCellReadStats CB` emits its rows by iterating a libc++ `std::unordered_map`, so the order is a hash-table walk rather than a sort. At the map sizes this produces, libc++ chains new entries at the head of their bucket and walks buckets in order, which comes out as the reverse of each barcode's first appearance in read order. +**What STAR does.** `--soloCellReadStats Standard` writes its rows by iterating `readFlagCounts.flagCounts`, a `std::unordered_map` keyed by whitelist index and reserved at `nCB*3/2` buckets (`SoloCommon.h`, `SoloFeature_countCBgeneUMI.cpp`, `SoloFeature_statsOutput.cpp`), so the order is a hash-table walk rather than a sort. -**What rustar-aligner does.** Emits that same reverse-first-appearance order, including across threads: the per-read accumulator merges in read order, so a threaded run writes the same file as a serial one. +**What rustar-aligner does.** Writes the rows in reverse order of each barcode's first appearance in the reads. Each read's entry is folded by the in-order consumer of the alignment loop, so a threaded run writes the same file as a serial one. A read whose barcode is resolved only at the end (a one-mismatch correction under `oneExact`, or a 1MM_multi barcode) counts as appearing where it was read. -**Why.** Reproducing the order where it is reproducible costs nothing and keeps a byte-comparison against STAR usable on the sizes where it can work at all. +**Measured.** On CellRanger's tiny 10x data (6,982 barcodes), STAR 2.7.11b (macOS build) writes the same order at 1 and at 8 threads, and rustar-aligner's order matches it only on the first rows. A model of a libc++ walk (identity hash, the bucket a key modulo a prime, a node for an empty bucket at the front of the list), fed rustar-aligner's first-appearance order, does not reproduce STAR's order for any bucket count up to 12,000 or any of three barcode keys (sorted-whitelist rank, file position, packed sequence). What sets STAR's order is not settled, and it may differ between standard libraries. -**Impact.** Past libc++'s load factor the map rehashes, and the order then depends on the bucket count, which depends on how many distinct barcodes were seen; beyond that size the order diverges. The **values never do** — only which line they appear on. Reading the file by barcode rather than by position is unaffected either way. +**Why.** The values are what the file reports; the row order is an artefact of STAR's container. -**Source.** `src/solo/cell_reads.rs`, locked by `rows_are_emitted_in_reverse_first_appearance_order`. STAR: `SoloFeature_statsOutput.cpp`. - ---- +**Impact.** Every value matches (on tiny 10x, all but the 3 barcodes of 3 reads rustar-aligner maps and STAR leaves unmapped); only the line each barcode is on differs. Read the file by barcode, not by position. A byte comparison against STAR's file does not work; sort both by barcode first. ### 3.3 SAM/BAM header program lines diff --git a/docs/src/content/docs/reference/cli-parameters.md b/docs/src/content/docs/reference/cli-parameters.md index ed87d8c6..906b69d6 100644 --- a/docs/src/content/docs/reference/cli-parameters.md +++ b/docs/src/content/docs/reference/cli-parameters.md @@ -24,6 +24,7 @@ Run `rustar-aligner --help` for the full machine-generated listing. | `--genomeSAindexNbases` | `14` | Length of the SA pre-indexing string (log2). Lower for small genomes. | | `--genomeChrBinNbits` | `18` | Log2 of chromosome bin size. | | `--genomeSAsparseD` | `1` | SA sparsity (higher = less RAM, slower mapping). | +| `--genomeChrSetMitochondrial` | `chrM M MT` | Chromosomes counted as mitochondrial, for the `mito` column of `CellReads.stats`. `-` names none. | ## Read input @@ -162,6 +163,7 @@ Run `rustar-aligner --help` for the full machine-generated listing. | `--quantGeneCoverage` | `No` | rustar-aligner extension. `Yes` writes per-gene exonic coverage to `GeneCoverage.out.bin` for DegNorm. See [DegNorm](/rustar-aligner/guides/degnorm/). | | `--degNormSampleId` | basename of `--outFileNamePrefix` | rustar-aligner extension. Sample name stored in `GeneCoverage.out.bin`. | | `--degNormCoverageFiles` | — | rustar-aligner extension. Two or more `GeneCoverage.out.bin` files: runs the DegNorm merge (no reads, no `--runMode`) and writes `DegNorm.out/`. Tuning: `--degNormIter`, `--degNormNmfIter`, `--degNormDownsampleRate`, `--degNormMinimaxCoverage`, `--degNormSkipBaselineSelection`, `--degNormBins`, `--degNormMinHighCoverage` (see the guide). | +| `--soloCellReadStats` | `None` | `Standard` writes `Solo.out//CellReads.stats` as STARsolo does: per cell barcode, how its reads' barcodes matched (`cbMatch`, `cbPerfect`, `cbMMunique`, `cbMMmultiple`), whether they mapped uniquely (`genomeU`/`genomeM`), reached one or several genes of the feature (`featureU`/`featureM`), their region (`exonic` for `Gene`; `GeneFull` sets none), `mito`, whether they reached the unique or multi-gene matrix (`countedU`/`countedM`), and the cell's UMI and gene totals. Reads whose barcode or UMI did not pass are summed in the `CBnotInPasslist` row. Rows come in reverse first-appearance order, which is not STAR's order (see the divergence notes); the values match, so read it by barcode. `CB` is a deprecated alias of `Standard`; STAR rejects it. | | `--soloCellRangerSummary` | `No` | rustar-aligner extension. `Yes` writes `Solo.out//CellRanger.summary.csv` for `Gene` and `GeneFull` (both needed): the fractions of all reads that map uniquely in exonic, intronic or intergenic regions, or antisense to a gene, under CellRanger's metric names. STARsolo's `Summary.csv` is unchanged. | ## Two-pass mode diff --git a/src/lib.rs b/src/lib.rs index 1d0c9932..3fb6a01f 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -2522,6 +2522,7 @@ fn align_reads_solo( per_feature: Vec, sj: crate::solo::SjRecords, velocyto: Option, + cell_read: Option, } info!("STARsolo: aligning cDNA reads and quantifying barcodes..."); @@ -2575,6 +2576,7 @@ fn align_reads_solo( (0..n_feat).map(|_| Vec::new()).collect(); let mut no_feature_multi: Vec> = (0..n_feat).map(|_| Vec::new()).collect(); + let mut cell_reads: Vec = Vec::new(); for result in products { let product = result?; writer.write_batch(&product.sam_records.records)?; @@ -2597,7 +2599,9 @@ fn align_reads_solo( } sj_batch.append(product.sj); velo_batch.extend(product.velocyto); + cell_reads.extend(product.cell_read); } + solo.fold_cell_reads(cell_reads); for (fi, recorder) in solo.recorders.iter().enumerate() { recorder.extend( std::mem::take(&mut feat_records[fi]), @@ -2710,6 +2714,7 @@ fn align_reads_solo( per_feature: outcome.per_feature, sj: outcome.sj, velocyto: outcome.velocyto, + cell_read: outcome.cell_read, }); } @@ -2873,6 +2878,7 @@ fn align_reads_solo( per_feature: outcome.per_feature, sj: outcome.sj, velocyto: outcome.velocyto, + cell_read: outcome.cell_read, }) }) .collect(); @@ -2971,6 +2977,7 @@ fn align_reads_solo_pe( per_feature: Vec, sj: crate::solo::SjRecords, velocyto: Option, + cell_read: Option, } info!("STARsolo: aligning 5' paired-end reads and quantifying barcodes..."); @@ -3012,6 +3019,7 @@ fn align_reads_solo_pe( (0..n_feat).map(|_| Vec::new()).collect(); let mut no_feature_multi: Vec> = (0..n_feat).map(|_| Vec::new()).collect(); + let mut cell_reads: Vec = Vec::new(); for result in products { let product = result?; writer.write_batch(&product.sam_records.records)?; @@ -3034,7 +3042,9 @@ fn align_reads_solo_pe( } sj_batch.append(product.sj); velo_batch.extend(product.velocyto); + cell_reads.extend(product.cell_read); } + solo.fold_cell_reads(cell_reads); for (fi, recorder) in solo.recorders.iter().enumerate() { recorder.extend( std::mem::take(&mut feat_records[fi]), @@ -3398,6 +3408,7 @@ fn align_reads_solo_pe( per_feature: outcome.per_feature, sj: outcome.sj, velocyto: outcome.velocyto, + cell_read: outcome.cell_read, }) }) .collect(); diff --git a/src/params/mod.rs b/src/params/mod.rs index 47e4ecf3..825c0eef 100644 --- a/src/params/mod.rs +++ b/src/params/mod.rs @@ -1414,15 +1414,20 @@ pub struct Parameters { #[arg(long = "soloCBmatchWLtype", default_value = "1MM_multi")] pub solo_cb_match_wl_type: String, - /// `CB`: write `Solo.out//CellReads.stats`, a per-cell-barcode - /// summary of what happened to the reads carrying it. `None` (the default) - /// writes nothing. + /// `Standard`: write `Solo.out//CellReads.stats`, a + /// per-cell-barcode summary of what happened to the reads carrying it. + /// `None` (the default) writes nothing. `CB` is a deprecated alias of + /// `Standard` (STAR rejects it). #[arg(long = "soloCellReadStats", default_value = "None")] pub solo_cell_read_stats: String, /// Chromosome names treated as mitochondrial, for the `mito` column of - /// `CellReads.stats`. `-` (the default) names none. - #[arg(long = "genomeChrSetMitochondrial", num_args = 1.., default_values_t = vec!["-".to_string()])] + /// `CellReads.stats`. STAR's default is `chrM M MT`; `-` names none. + #[arg( + long = "genomeChrSetMitochondrial", + num_args = 1.., + default_values_t = ["chrM", "M", "MT"].map(String::from) + )] pub genome_chr_set_mitochondrial: Vec, /// Two-column `CB cluster` file assigning cells to clusters, for /// `--soloFeatures Transcript3p`. @@ -1537,6 +1542,12 @@ impl Parameters { } } + /// `--soloCellReadStats Standard` (or its deprecated alias `CB`): write + /// `CellReads.stats`. + pub fn solo_cell_read_stats_yes(&self) -> bool { + matches!(self.solo_cell_read_stats.as_str(), "Standard" | "CB") + } + /// Whether the run has to track STAR's readInfo, the per-read (cell, /// corrected UMI) pair that UMI collapsing fills and the sorted-BAM writer /// turns into `CB`/`UB`. `CB_samTagOut` corrects the barcode inline instead, @@ -2601,12 +2612,17 @@ impl Parameters { SOLUTION: rerun with --soloUMIfiltering MultiGeneUMI_CR --soloUMIdedup 1MM_CR", )); } - // --soloCellReadStats: `CB` is the only value STAR defines. - if !matches!(params.solo_cell_read_stats.as_str(), "CB" | "None") { + // --soloCellReadStats: STAR defines `Standard` and `None` + // (`ParametersSolo.cpp:441-451`); `CB`, which rustar-aligner took + // before, stays as a deprecated alias of `Standard`. + if !matches!( + params.solo_cell_read_stats.as_str(), + "Standard" | "None" | "CB" + ) { return Err(command.error( ErrorKind::InvalidValue, format!( - "unknown --soloCellReadStats '{}'; expected CB or None", + "unknown --soloCellReadStats '{}'; expected Standard or None", params.solo_cell_read_stats ), )); diff --git a/src/solo/cell_reads.rs b/src/solo/cell_reads.rs index 5d140a51..2b8aa942 100644 --- a/src/solo/cell_reads.rs +++ b/src/solo/cell_reads.rs @@ -1,162 +1,290 @@ -//! `--soloCellReadStats CB`: the per-cell-barcode read summary STARsolo writes -//! as `Solo.out//CellReads.stats`. +//! `--soloCellReadStats Standard`: the per-cell-barcode read summary STARsolo +//! writes as `Solo.out//CellReads.stats`. //! //! One row per cell barcode, fifteen counters describing what happened to the -//! reads carrying it — how the barcode matched, whether the read mapped -//! uniquely, whether it landed on a feature, where in the gene, and whether it -//! reached the matrix — plus the per-cell UMI and gene totals. The reads whose -//! barcode never resolved are not dropped; they are summed into a single -//! `CBnotInPasslist` row, so the columns account for every read rather than -//! only the ones that succeeded. +//! reads carrying it (how the barcode matched, whether the read mapped +//! uniquely, whether it landed on a feature, and whether it reached the +//! matrix) plus the per-cell UMI and gene totals. Reads whose barcode never +//! resolved are summed into a single `CBnotInPasslist` row, so `cbMatch` +//! accounts for every input read. //! -//! # D24: row order +//! # What STAR counts +//! +//! Each feature keeps its own table (`SoloReadFeature::readFlag`), because the +//! flags depend on the feature's annotation of the read: +//! +//! - At the read stage (`SoloReadFeature_record.cpp:25-62`) every read gets +//! `genomeU`/`genomeM` from its number of loci (0 for an unmapped or +//! too-many-loci read), `mito` if any alignment is on a +//! `--genomeChrSetMitochondrial` chromosome, a region bit from the feature's +//! `ovType`, and `featureU`/`featureM` from the size of the feature's gene +//! set. Only `Gene` sets an `ovType` (`exonic`, whenever it found a gene, +//! `Transcriptome_classifyAlign.cpp:262-263`); `GeneFull` sets none, so its +//! region columns stay zero, and no feature rustar quantifies ever sets +//! `intronic` or the antisense columns. A read whose barcode did not match +//! the whitelist, or whose UMI was rejected, goes to `CBnotInPasslist` here. +//! - At the counting stage (`SoloReadFeature_inputRecords.cpp:100-150`) the +//! rest are resolved: an exact barcode is `cbPerfect`; a single one-mismatch +//! correction is `cbMMunique`, unless `oneExact` drops it because no read +//! matched that barcode exactly; several one-mismatch candidates are +//! `cbMMmultiple` when the posterior picks one. The dropped ones also go to +//! `CBnotInPasslist`. A resolved read that reached the unique matrix is +//! `countedU`; one that reached the multi-gene distribution +//! (`--soloMultiMappers`) is `countedM`. //! -//! STAR iterates a libc++ `std::unordered_map` to emit these rows, so the order -//! is a hash-table walk, not a sort. For the small maps this produces, libc++ -//! chains new entries at the head of their bucket and walks buckets in order, -//! which comes out as the reverse of first appearance in read order. That is -//! what this reproduces. +//! Corrections that wait on the exact counts are held in [`CellReadStats`] +//! until [`CellReadStats::resolve`] is given them. +//! +//! # D24: row order //! -//! It is not reproducible in general: past the load factor libc++ rehashes, and -//! the order after a rehash depends on the bucket count, which depends on how -//! many distinct barcodes were seen. At that size the order diverges. The -//! **values never do** — only which line they appear on. A consumer that reads -//! this file by barcode rather than by position is unaffected either way, and -//! sorting by barcode is the only stable thing to do with it. +//! STAR emits these rows by iterating a `std::unordered_map` keyed by +//! whitelist index, so the order is a hash-table walk, not a sort. This writes +//! them in reverse order of first appearance in the reads, which agrees with +//! STAR only on the first few rows of CellRanger's tiny 10x data; what sets +//! STAR's order is not settled (DIVERGENCE.md 3.2). The **values** match, only +//! which line they appear on differs. Read this file by barcode, not by +//! position; sorting by barcode is the only stable thing to do with it. //! -//! Reads are folded in under a mutex held by `SoloContext`, so the order is the -//! order reads were processed in regardless of thread count. +//! Reads are folded in by the in-order consumer of the alignment loop, so +//! "first appearance" is input order regardless of thread count. A barcode +//! first reached through a deferred correction appears where that read was. -use std::collections::{BTreeMap, HashSet}; +use std::collections::HashMap; use std::fmt::Write as _; -/// What happened to one read, as the fourteen optional flags STAR tracks. -/// -/// `cbMatch` is not here: every read that reaches the accumulator matched a -/// barcode well enough to be attributed somewhere, so it is always set. -/// -/// Fourteen bools rather than a bitfield because they are written once per read -/// and read once per fold, and the names are the column names of the file. -#[allow(clippy::struct_excessive_bools)] -#[derive(Debug, Clone, Copy, Default)] -pub struct CellReadFlag { - /// The barcode was an exact whitelist hit. - pub cb_perfect: bool, - /// Corrected via a single one-mismatch neighbour. - pub cb_mm_unique: bool, - /// Corrected via several one-mismatch neighbours, resolved by the posterior. - pub cb_mm_multiple: bool, - /// The read mapped to exactly one genomic locus. - pub genome_u: bool, - /// The read mapped to several genomic loci. - pub genome_m: bool, - /// It fell on exactly one feature (gene). - pub feature_u: bool, - /// It fell on several features. - pub feature_m: bool, - /// Exonic, on the annotated strand. - pub exonic: bool, - /// Intronic, on the annotated strand. - pub intronic: bool, - /// Exonic, antisense to the annotation. - pub exonic_as: bool, - /// Intronic, antisense to the annotation. - pub intronic_as: bool, - /// On a chromosome named by `--genomeChrSetMitochondrial`. - pub mito: bool, - /// Counted into the unique-gene matrix. - pub counted_u: bool, - /// Counted through the multi-gene distribution. - pub counted_m: bool, +use crate::solo::whitelist::CbCandidate; + +/// STAR's `SoloReadFlagClass` bits, in its column order. +pub mod bit { + pub const CB_MATCH: u16 = 1 << 0; + pub const CB_PERFECT: u16 = 1 << 1; + pub const CB_MM_UNIQUE: u16 = 1 << 2; + pub const CB_MM_MULTIPLE: u16 = 1 << 3; + pub const GENOME_U: u16 = 1 << 4; + pub const GENOME_M: u16 = 1 << 5; + pub const FEATURE_U: u16 = 1 << 6; + pub const FEATURE_M: u16 = 1 << 7; + pub const EXONIC: u16 = 1 << 8; + pub const INTRONIC: u16 = 1 << 9; + pub const EXONIC_AS: u16 = 1 << 10; + pub const INTRONIC_AS: u16 = 1 << 11; + pub const MITO: u16 = 1 << 12; + pub const COUNTED_U: u16 = 1 << 13; + pub const COUNTED_M: u16 = 1 << 14; + /// Number of bits, i.e. counter columns. + pub const N: usize = 15; +} + +/// The column names, in bit order (STAR's `statNames`). +const STAT_NAMES: [&str; bit::N] = [ + "cbMatch", + "cbPerfect", + "cbMMunique", + "cbMMmultiple", + "genomeU", + "genomeM", + "featureU", + "featureM", + "exonic", + "intronic", + "exonicAS", + "intronicAS", + "mito", + "countedU", + "countedM", +]; + +/// How one read's cell barcode matched, as far as the read stage knows. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ReadCb { + /// No whitelist match, a rejected UMI, or no barcode at all (`cbMatch < 0`). + None, + /// Exact whitelist hit (`cbMatch` 0). + Exact(u32), + /// A single one-mismatch correction (`cbMatch` 1). + Corrected(u32), + /// Several one-mismatch candidates (`cbMatch` > 1), for the posterior. + Multi(Vec), +} + +/// One read's contribution: its barcode and, per feature, the read-stage flag +/// bits (`genomeU`/`genomeM`, `featureU`/`featureM`, region, `mito`). +#[derive(Debug, Clone)] +pub struct CellReadEntry { + pub cb: ReadCb, + pub flags: Vec, +} + +/// One barcode's counters and the input position of its first read (D24). +#[derive(Debug, Clone)] +struct CellRow { + first_seq: u64, + counts: [u64; bit::N], } -/// The accumulator behind `CellReads.stats`. #[derive(Debug, Default, Clone)] +struct FeatureTable { + cells: HashMap, + no_cb: [u64; bit::N], +} + +/// A read whose barcode resolution waits on the whole run's exact counts. +#[derive(Debug, Clone)] +struct Pending { + seq: u64, + cb: ReadCb, + flags: Vec, +} + +/// The accumulator behind `CellReads.stats`: one table per quantified feature. +#[derive(Debug, Clone)] pub struct CellReadStats { - /// Whitelist index to its fifteen counters. - cells: BTreeMap, - /// The single bucket for reads whose barcode did not resolve. - no_cb: [u64; 15], - /// Whitelist indices in first-appearance order; emitted reversed (D24). - order: Vec, - seen: HashSet, + features: Vec, + next_seq: u64, + /// `oneExact`: single corrections are kept only for barcodes some read + /// matched exactly, which is known only at the end. + defer_corrected: bool, + /// `--soloMultiMappers` other than `Unique`: gene-ambiguous reads are + /// counted (`countedM`). + count_multi_gene: bool, + /// With a whitelist, unmatched reads are summed into `CBnotInPasslist`; + /// without one STAR records nothing for them. + whitelist: bool, + pending: Vec, } impl CellReadStats { - pub fn new() -> Self { - Self::default() + pub fn new( + n_features: usize, + defer_corrected: bool, + count_multi_gene: bool, + whitelist: bool, + ) -> Self { + Self { + features: vec![FeatureTable::default(); n_features], + next_seq: 0, + defer_corrected, + count_multi_gene, + whitelist, + pending: Vec::new(), + } + } + + fn add(v: &mut [u64; bit::N], flags: u16) { + for (i, c) in v.iter_mut().enumerate() { + *c += u64::from((flags >> i) & 1); + } } - fn fold(v: &mut [u64; 15], f: &CellReadFlag) { - v[0] += 1; // cbMatch: every read that got this far - for (i, set) in [ - f.cb_perfect, - f.cb_mm_unique, - f.cb_mm_multiple, - f.genome_u, - f.genome_m, - f.feature_u, - f.feature_m, - f.exonic, - f.intronic, - f.exonic_as, - f.intronic_as, - f.mito, - f.counted_u, - f.counted_m, - ] - .into_iter() - .enumerate() - { - if set { - v[i + 1] += 1; + fn add_no_cb(&mut self, flags: &[u16]) { + for (t, &f) in self.features.iter_mut().zip(flags) { + Self::add(&mut t.no_cb, f | bit::CB_MATCH); + } + } + + /// `countsAdd(cb)`: the read resolved to whitelist cell `cb`, matched as + /// `cb_bit`. A resolved read with a feature was counted. + fn add_cell(&mut self, cb: u32, seq: u64, cb_bit: u16, flags: &[u16]) { + for (t, &f) in self.features.iter_mut().zip(flags) { + let mut f = f | bit::CB_MATCH | cb_bit; + if f & bit::FEATURE_U != 0 { + f |= bit::COUNTED_U; + } + if self.count_multi_gene && f & bit::FEATURE_M != 0 { + f |= bit::COUNTED_M; } + let row = t.cells.entry(cb).or_insert(CellRow { + first_seq: seq, + counts: [0; bit::N], + }); + row.first_seq = row.first_seq.min(seq); + Self::add(&mut row.counts, f); } } - /// Record a read that resolved to whitelist cell `cb`. - pub fn add_cell(&mut self, cb: u32, flag: &CellReadFlag) { - if self.seen.insert(cb) { - self.order.push(cb); + /// Fold in one read, in input order. + pub fn fold(&mut self, entry: CellReadEntry) { + debug_assert_eq!(entry.flags.len(), self.features.len()); + let seq = self.next_seq; + self.next_seq += 1; + match entry.cb { + ReadCb::None => { + if self.whitelist { + self.add_no_cb(&entry.flags); + } + } + ReadCb::Exact(cb) => self.add_cell(cb, seq, bit::CB_PERFECT, &entry.flags), + ReadCb::Corrected(cb) if !self.defer_corrected => { + self.add_cell(cb, seq, bit::CB_MM_UNIQUE, &entry.flags); + } + cb @ (ReadCb::Corrected(_) | ReadCb::Multi(_)) => self.pending.push(Pending { + seq, + cb, + flags: entry.flags, + }), } - Self::fold(self.cells.entry(cb).or_insert([0; 15]), flag); } - /// Record a read whose barcode did not resolve, or whose UMI was rejected. - pub fn add_no_cb(&mut self, flag: &CellReadFlag) { - Self::fold(&mut self.no_cb, flag); + /// Resolve the held corrections once the exact counts are known: + /// `has_exact(cb)` is `oneExact`'s test, `posterior(candidates)` the + /// 1MM_multi choice. Reads neither accepts go to `CBnotInPasslist`. + pub fn resolve( + &mut self, + has_exact: impl Fn(u32) -> bool, + posterior: impl Fn(&[CbCandidate]) -> Option, + ) { + for p in std::mem::take(&mut self.pending) { + let resolved = match &p.cb { + ReadCb::Corrected(cb) => has_exact(*cb).then_some((*cb, bit::CB_MM_UNIQUE)), + ReadCb::Multi(cands) => posterior(cands).map(|cb| (cb, bit::CB_MM_MULTIPLE)), + ReadCb::None | ReadCb::Exact(_) => unreachable!("never held"), + }; + match resolved { + Some((cb, cb_bit)) => self.add_cell(cb, p.seq, cb_bit, &p.flags), + None => self.add_no_cb(&p.flags), + } + } } - /// Render the file. `umi_gene` gives each cell its final - /// `(nUMIunique, nGenesUnique)`; a cell absent from it prints zeros. - /// `barcode_of` renders a whitelist index as its barcode string. - pub fn render( + /// Render feature `fi`'s file. `unique` gives a cell's + /// `(nUMIunique, nGenesUnique)` and `multi` its `(nUMImulti, nGenesMulti)`; + /// a cell absent from them prints zeros. `barcode_of` renders a whitelist + /// index as its barcode. + pub fn render( &self, + fi: usize, barcode_of: impl Fn(u32) -> String, - umi_gene: &BTreeMap, + unique: &HashMap, + multi: &HashMap, ) -> String { - let mut s = String::from( - "CB\tcbMatch\tcbPerfect\tcbMMunique\tcbMMmultiple\tgenomeU\tgenomeM\tfeatureU\t\ - featureM\texonic\tintronic\texonicAS\tintronicAS\tmito\tcountedU\tcountedM\t\ - nUMIunique\tnGenesUnique\tnUMImulti\tnGenesMulti\n", - ); - s.push_str("CBnotInPasslist"); - for v in &self.no_cb { + let table = &self.features[fi]; + let mut s = String::from("CB"); + for name in STAT_NAMES { s.push('\t'); - s.push_str(&v.to_string()); + s.push_str(name); + } + s.push_str("\tnUMIunique\tnGenesUnique\tnUMImulti\tnGenesMulti\n"); + s.push_str("CBnotInPasslist"); + for v in &table.no_cb { + let _ = write!(s, "\t{v}"); } s.push_str("\t0\t0\t0\t0\n"); - // Reverse first-appearance order — see D24 in the module docs. - for &cb in self.order.iter().rev() { + // Reverse first-appearance order (D24). + let mut rows: Vec<(&u32, &CellRow)> = table.cells.iter().collect(); + rows.sort_unstable_by_key(|(_, r)| std::cmp::Reverse(r.first_seq)); + for (&cb, row) in rows { s.push_str(&barcode_of(cb)); - for v in &self.cells[&cb] { - s.push('\t'); - s.push_str(&v.to_string()); + for v in &row.counts { + let _ = write!(s, "\t{v}"); } - let (n_umi, n_gene) = umi_gene.get(&cb).copied().unwrap_or((0, 0)); - // nUMImulti and nGenesMulti stay zero: multi-gene UMIs are not - // collapsed into per-cell totals here. - let _ = writeln!(s, "\t{n_umi}\t{n_gene}\t0\t0"); + let (n_umi, n_gene) = unique.get(&cb).copied().unwrap_or((0, 0)); + let (m_umi, m_gene) = multi.get(&cb).copied().unwrap_or((0.0, 0)); + // `nUMIperCBmulti` is a double (`ostream << double`). + let _ = writeln!( + s, + "\t{n_umi}\t{n_gene}\t{}\t{m_gene}", + crate::solo::count::fmt_double(m_umi) + ); } s } @@ -166,123 +294,224 @@ impl CellReadStats { mod tests { use super::*; - fn flag_perfect_counted() -> CellReadFlag { - CellReadFlag { - cb_perfect: true, - genome_u: true, - feature_u: true, - exonic: true, - counted_u: true, - ..Default::default() + const ONE: [u16; 1] = [bit::GENOME_U | bit::FEATURE_U | bit::EXONIC]; + + fn entry(cb: ReadCb, flags: &[u16]) -> CellReadEntry { + CellReadEntry { + cb, + flags: flags.to_vec(), } } - fn header_and_rows(text: &str) -> (Vec<&str>, Vec<&str>) { - let mut lines = text.lines(); - let header: Vec<&str> = lines.next().unwrap().split('\t').collect(); - (header, lines.collect()) + fn stats(n_features: usize) -> CellReadStats { + CellReadStats::new(n_features, false, false, true) + } + + fn render(st: &CellReadStats, fi: usize) -> String { + st.render(fi, |i| format!("CB{i}"), &HashMap::new(), &HashMap::new()) + } + + fn rows(text: &str) -> Vec> { + text.lines() + .skip(1) + .map(|l| l.split('\t').collect()) + .collect() + } + + /// The column of `name` in a split row. + fn col(name: &str) -> usize { + 1 + STAT_NAMES.iter().position(|&n| n == name).unwrap() } /// Every column has a name, and every row has a value under each of them. #[test] fn each_row_fills_the_header() { - let mut st = CellReadStats::new(); - st.add_cell(0, &flag_perfect_counted()); - st.add_no_cb(&CellReadFlag::default()); - let out = st.render(|i| format!("CB{i}"), &BTreeMap::new()); - let (header, rows) = header_and_rows(&out); + let mut st = stats(1); + st.fold(entry(ReadCb::Exact(0), &ONE)); + st.fold(entry(ReadCb::None, &[0])); + let out = render(&st, 0); + let header: Vec<&str> = out.lines().next().unwrap().split('\t').collect(); assert_eq!(header.len(), 20); + assert_eq!(header[1], "cbMatch"); + assert_eq!(header[15], "countedM"); + let rows = rows(&out); assert_eq!(rows.len(), 2, "the passlist-miss row plus one cell"); for row in rows { - assert_eq!(row.split('\t').count(), header.len()); + assert_eq!(row.len(), header.len()); } } - /// A read is counted once under `cbMatch` and once under each flag it sets, - /// so the flag columns can exceed neither `cbMatch` nor each other's logic. + /// A read is counted once under `cbMatch` and once under each flag it sets. #[test] fn flags_accumulate_per_read() { - let mut st = CellReadStats::new(); + let mut st = stats(1); for _ in 0..3 { - st.add_cell(7, &flag_perfect_counted()); + st.fold(entry(ReadCb::Exact(7), &ONE)); } - st.add_cell( - 7, - &CellReadFlag { - cb_mm_unique: true, - genome_m: true, - ..Default::default() - }, - ); - let out = st.render(|i| format!("CB{i}"), &BTreeMap::new()); - let row: Vec<&str> = out.lines().nth(2).unwrap().split('\t').collect(); + st.fold(entry(ReadCb::Corrected(7), &[bit::GENOME_M])); + let out = render(&st, 0); + let row = &rows(&out)[1]; assert_eq!(row[0], "CB7"); - assert_eq!(row[1], "4", "cbMatch counts every read"); - assert_eq!(row[2], "3", "cbPerfect"); - assert_eq!(row[3], "1", "cbMMunique"); - assert_eq!(row[5], "3", "genomeU"); - assert_eq!(row[6], "1", "genomeM"); - assert_eq!(row[14], "3", "countedU"); + assert_eq!(row[col("cbMatch")], "4"); + assert_eq!(row[col("cbPerfect")], "3"); + assert_eq!(row[col("cbMMunique")], "1"); + assert_eq!(row[col("genomeU")], "3"); + assert_eq!(row[col("genomeM")], "1"); + assert_eq!(row[col("countedU")], "3", "a resolved featureU read counts"); + } + + /// Each feature keeps its own table: the same read can be featureU for + /// one and featureM for the other, as STAR's per-feature `readFlag` does. + #[test] + fn features_are_counted_separately() { + let mut st = stats(2); + st.fold(entry( + ReadCb::Exact(1), + &[ + bit::GENOME_U | bit::FEATURE_U | bit::EXONIC, + bit::GENOME_U | bit::FEATURE_M, + ], + )); + let gene = render(&st, 0); + let full = render(&st, 1); + let (g, f) = (&rows(&gene)[1], &rows(&full)[1]); + assert_eq!( + (g[col("featureU")], g[col("featureM")], g[col("exonic")]), + ("1", "0", "1") + ); + assert_eq!( + (f[col("featureU")], f[col("featureM")], f[col("exonic")]), + ("0", "1", "0") + ); + assert_eq!(g[col("countedU")], "1"); + assert_eq!(f[col("countedU")], "0"); + assert_eq!( + f[col("countedM")], + "0", + "multi-gene reads count only with --soloMultiMappers" + ); } - /// Reads whose barcode never resolved are summed rather than dropped, so - /// the file accounts for the whole input. + /// With `--soloMultiMappers`, a resolved gene-ambiguous read is countedM. + #[test] + fn multi_gene_reads_count_with_multimappers() { + let mut st = CellReadStats::new(1, false, true, true); + st.fold(entry(ReadCb::Exact(1), &[bit::GENOME_M | bit::FEATURE_M])); + st.fold(entry(ReadCb::None, &[bit::GENOME_M | bit::FEATURE_M])); + let out = render(&st, 0); + let rows = rows(&out); + assert_eq!(rows[1][col("countedM")], "1"); + assert_eq!(rows[0][col("featureM")], "1"); + assert_eq!( + rows[0][col("countedM")], + "0", + "an unresolved read is never counted" + ); + } + + /// Reads whose barcode never resolved are summed rather than dropped, and + /// keep their mapping and feature flags. #[test] fn unresolved_reads_land_in_the_passlist_miss_row() { - let mut st = CellReadStats::new(); - st.add_cell(0, &flag_perfect_counted()); + let mut st = stats(1); + st.fold(entry(ReadCb::Exact(0), &ONE)); for _ in 0..5 { - st.add_no_cb(&CellReadFlag { - genome_u: true, - ..Default::default() - }); + st.fold(entry(ReadCb::None, &ONE)); } - let out = st.render(|i| format!("CB{i}"), &BTreeMap::new()); - let row: Vec<&str> = out.lines().nth(1).unwrap().split('\t').collect(); + let out = render(&st, 0); + let row = &rows(&out)[0]; assert_eq!(row[0], "CBnotInPasslist"); - assert_eq!(row[1], "5"); - assert_eq!(row[5], "5", "genomeU"); + assert_eq!(row[col("cbMatch")], "5"); + assert_eq!(row[col("cbPerfect")], "0"); + assert_eq!(row[col("featureU")], "5"); + assert_eq!(row[col("countedU")], "0"); assert_eq!(&row[16..], ["0", "0", "0", "0"], "no UMI columns for it"); } - /// D24: rows come out in reverse first-appearance order, which is what - /// STAR's libc++ hash-map walk produces at these sizes. + /// Without a whitelist STAR records nothing for unmatched reads. + #[test] + fn no_whitelist_records_no_passlist_misses() { + let mut st = CellReadStats::new(1, false, false, false); + st.fold(entry(ReadCb::None, &ONE)); + assert_eq!(rows(&render(&st, 0))[0][col("cbMatch")], "0"); + } + + /// `oneExact`: a correction is held until the exact counts are known, then + /// kept only for a barcode some read matched exactly. + #[test] + fn one_exact_corrections_wait_for_the_exact_counts() { + let mut st = CellReadStats::new(1, true, false, true); + st.fold(entry(ReadCb::Corrected(3), &ONE)); + st.fold(entry(ReadCb::Corrected(4), &ONE)); + assert_eq!(rows(&render(&st, 0)).len(), 1, "nothing resolved yet"); + st.resolve(|cb| cb == 3, |_| None); + let out = render(&st, 0); + let rows = rows(&out); + assert_eq!(rows.len(), 2); + assert_eq!( + rows[0][col("cbMatch")], + "1", + "the drop goes to CBnotInPasslist" + ); + assert_eq!(rows[0][col("featureU")], "1"); + assert_eq!(rows[1][0], "CB3"); + assert_eq!(rows[1][col("cbMMunique")], "1"); + assert_eq!(rows[1][col("countedU")], "1"); + } + + /// A 1MM_multi barcode is `cbMMmultiple` when the posterior picks a cell, + /// `CBnotInPasslist` when it does not. + #[test] + fn multi_match_barcodes_follow_the_posterior() { + let cand = |wl_index| CbCandidate { + wl_index, + mismatch_pos: 0, + mismatch_qual: b'I', + }; + let mut st = stats(1); + st.fold(entry(ReadCb::Multi(vec![cand(5), cand(6)]), &ONE)); + st.fold(entry(ReadCb::Multi(vec![cand(8), cand(9)]), &ONE)); + st.resolve(|_| true, |c| (c[0].wl_index == 5).then_some(6)); + let out = render(&st, 0); + let rows = rows(&out); + assert_eq!(rows.len(), 2); + assert_eq!(rows[0][col("cbMatch")], "1"); + assert_eq!(rows[1][0], "CB6"); + assert_eq!(rows[1][col("cbMMmultiple")], "1"); + assert_eq!(rows[1][col("countedU")], "1"); + } + + /// D24: rows come out in reverse first-appearance order, a held read + /// placed where it was read rather than where it was resolved. #[test] fn rows_are_emitted_in_reverse_first_appearance_order() { - let mut st = CellReadStats::new(); - for cb in [4u32, 1, 9] { - st.add_cell(cb, &flag_perfect_counted()); - } - st.add_cell(1, &flag_perfect_counted()); // seen again: order unchanged - let out = st.render(|i| format!("CB{i}"), &BTreeMap::new()); - let cbs: Vec<&str> = out - .lines() - .skip(2) - .map(|l| l.split('\t').next().unwrap()) - .collect(); - assert_eq!(cbs, ["CB9", "CB1", "CB4"]); + let mut st = CellReadStats::new(1, true, false, true); + st.fold(entry(ReadCb::Exact(4), &ONE)); + st.fold(entry(ReadCb::Corrected(2), &ONE)); + st.fold(entry(ReadCb::Exact(1), &ONE)); + st.fold(entry(ReadCb::Exact(9), &ONE)); + st.fold(entry(ReadCb::Exact(1), &ONE)); // seen again: order unchanged + st.resolve(|_| true, |_| None); + let out = render(&st, 0); + let cbs: Vec<&str> = rows(&out).iter().skip(1).map(|r| r[0]).collect(); + assert_eq!(cbs, ["CB9", "CB1", "CB2", "CB4"]); } - /// The UMI and gene totals come from the final matrix, not from the read - /// counters, so a cell missing from that map prints zeros rather than - /// inheriting a read count. + /// The UMI and gene totals come from the matrices, not from the read + /// counters, so a cell missing from them prints zeros; `nUMImulti` prints + /// as the double it is. #[test] - fn umi_and_gene_totals_come_from_the_matrix() { - let mut st = CellReadStats::new(); - st.add_cell(2, &flag_perfect_counted()); - st.add_cell(3, &flag_perfect_counted()); - let mut umi_gene = BTreeMap::new(); - umi_gene.insert(2u32, (17u32, 5u32)); - let out = st.render(|i| format!("CB{i}"), &umi_gene); - let rows: Vec> = out - .lines() - .skip(2) - .map(|l| l.split('\t').collect()) - .collect(); - // Reverse order: CB3 first, then CB2. - assert_eq!(rows[0][0], "CB3"); - assert_eq!(&rows[0][16..18], ["0", "0"]); - assert_eq!(rows[1][0], "CB2"); - assert_eq!(&rows[1][16..18], ["17", "5"]); + fn umi_and_gene_totals_come_from_the_matrices() { + let mut st = stats(1); + st.fold(entry(ReadCb::Exact(2), &ONE)); + st.fold(entry(ReadCb::Exact(3), &ONE)); + let unique = HashMap::from([(2u32, (17u64, 5u32))]); + let multi = HashMap::from([(2u32, (1.5f64, 1u32))]); + let out = st.render(0, |i| format!("CB{i}"), &unique, &multi); + let rows = rows(&out); + assert_eq!(rows[1][0], "CB3"); + assert_eq!(&rows[1][16..], ["0", "0", "0", "0"]); + assert_eq!(rows[2][0], "CB2"); + assert_eq!(&rows[2][16..], ["17", "5", "1.5", "1"]); } } diff --git a/src/solo/count.rs b/src/solo/count.rs index 2acfb8ab..aa410f47 100644 --- a/src/solo/count.rs +++ b/src/solo/count.rs @@ -924,6 +924,11 @@ fn fmt_real(v: f64) -> String { /// counts, cb-ascending) and merges each cell with its gene-ambiguous molecules /// (deduplicated by UMI, gene set = union). Cells present only in multi records /// (no unique gene) are skipped. +/// +/// Returns, per cell, what the first method added to it: STAR's +/// `nUMIperCBmulti` (the multi-gene UMIs it distributed) and `nGenePerCBmulti` +/// (the genes they reached that had no unique count), for `CellReads.stats` +/// (`SoloFeature_outputResults.cpp:134-221`). #[allow(clippy::too_many_arguments)] fn build_multi_matrices( raw_body: &tempfile::NamedTempFile, @@ -934,9 +939,10 @@ fn build_multi_matrices( n_features: usize, n_barcodes: usize, gzip: bool, -) -> Result<(), Error> { +) -> Result, Error> { + let mut per_cell: HashMap = HashMap::default(); if methods.is_empty() { - return Ok(()); + return Ok(per_cell); } use rayon::prelude::*; let mut multi: Vec<&crate::solo::MultiGeneRecord> = multi_records.iter().collect(); @@ -991,6 +997,16 @@ fn build_multi_matrices( let mols = cell_molecules(cb, mptr); for (k, &m) in methods.iter().enumerate() { let counts = distribute_multi(m, u, &mols); + if k == 0 && !mols.is_empty() { + let genes: std::collections::BTreeSet = + mols.iter().flatten().copied().collect(); + let entry = per_cell.entry(cb).or_insert((0.0, 0)); + for g in genes { + let unique = u.get(&g).copied(); + entry.0 += counts.get(&g).copied().unwrap_or(0.0) - unique.unwrap_or(0.0); + entry.1 += u32::from(unique.is_none()); + } + } let mut entries: Vec<(u32, f64)> = counts.into_iter().filter(|&(_, v)| v > 1e-9).collect(); entries.sort_unstable_by_key(|&(g, _)| g); @@ -1051,7 +1067,7 @@ fn build_multi_matrices( })?; } let _ = matrix_name; // UniqueAndMult uses a fixed name scheme - Ok(()) + Ok(per_cell) } /// CellRanger's multi-gene UMI resolution, as STAR implements it for @@ -1601,6 +1617,15 @@ pub fn write_gene_matrix( // The Gene feature's called cells, which Velocyto's filtered matrices use. let mut gene_called: Option> = None; + // `CellReads.stats`: the corrections that waited on the exact counts are + // resolved as `inputRecords` resolves them, the same for every feature. + if let Some(cell_stats) = &ctx.cell_read_stats { + cell_stats.lock().unwrap().resolve( + |cb| cb_passes_one_exact(true, &exact_counts, cb), + |cands| resolve_multi_cb(cands, &exact_counts, pseudocount), + ); + } + // One {prefix}{soloOutFileNames[0]}/{raw,filtered}/ per feature. for (fi, (feature, recorder)) in ctx.features.iter().zip(&ctx.recorders).enumerate() { let feature_dir = params.output_path(&format!("{solo_dir}{}/", feature.dir_name())); @@ -1652,27 +1677,6 @@ pub fn write_gene_matrix( if gzip { " [gzip]" } else { "" }, ); - // `--soloCellReadStats CB`: the per-cell read summary, alongside the - // raw matrix because its UMI and gene columns are the raw totals. - if let Some(cell_stats) = &ctx.cell_read_stats { - let umi_gene: std::collections::BTreeMap = mstats - .cells - .iter() - .map(|c| (c.cb, (c.n_umis as u32, c.n_genes))) - .collect(); - let path = feature_dir.join("CellReads.stats"); - let text = cell_stats.lock().unwrap().render( - |cb| { - ctx.whitelist - .barcode_string(cb) - .unwrap_or_else(|| cb.to_string()) - }, - &umi_gene, - ); - std::fs::write(&path, text).map_err(|e| Error::io(e, &path))?; - log::info!("STARsolo: wrote {}/CellReads.stats", feature.dir_name()); - } - // Filtered (cell-called) matrix per --soloCellFilter. EmptyDrops_CR runs // the Monte-Carlo rescue (needs the per-cell profiles in the body). let called = if params @@ -1730,13 +1734,14 @@ pub fn write_gene_matrix( // --soloMultiMappers: UniqueAndMult-.mtx alongside raw. let (mut multi_counted, mut multi_dropped) = (0u64, 0u64); + let mut multi_per_cell: HashMap = HashMap::default(); if !multi_methods.is_empty() { let mut mg = recorder.multi_gene.lock().unwrap(); let before = mg.len(); mg.retain(|r| cb_passes_one_exact(one_exact, &exact_counts, r.cb)); multi_counted = mg.len() as u64; multi_dropped = (before - mg.len()) as u64; - build_multi_matrices( + multi_per_cell = build_multi_matrices( &body, &mg, &multi_methods, @@ -1754,6 +1759,30 @@ pub fn write_gene_matrix( ); } + // `--soloCellReadStats Standard`: the per-cell read summary. Its UMI + // and gene columns are the raw matrices' totals, so it follows them + // (`SoloFeature_statsOutput.cpp:88-123`). + if let Some(cell_stats) = &ctx.cell_read_stats { + let unique: HashMap = mstats + .cells + .iter() + .map(|c| (c.cb, (c.n_umis, c.n_genes))) + .collect(); + let path = feature_dir.join("CellReads.stats"); + let text = cell_stats.lock().unwrap().render( + fi, + |cb| { + ctx.whitelist + .barcode_string(cb) + .unwrap_or_else(|| cb.to_string()) + }, + &unique, + &multi_per_cell, + ); + std::fs::write(&path, text).map_err(|e| Error::io(e, &path))?; + log::info!("STARsolo: wrote {}/CellReads.stats", feature.dir_name()); + } + // Features.stats: the counting-stage figures (`inputRecords`, // `collapseUMIall`) join the per-read ones from `process_read`. { diff --git a/src/solo/mod.rs b/src/solo/mod.rs index b91d1fbc..1ce428b7 100644 --- a/src/solo/mod.rs +++ b/src/solo/mod.rs @@ -32,6 +32,7 @@ use crate::error::Error; use crate::io::fastq::{EncodedRead, FastqReader, decode_base}; use crate::params::{Parameters, SoloType}; use crate::quant::GeneAnnotation; +use crate::solo::cell_reads::{CellReadEntry, ReadCb}; use std::path::PathBuf; use std::sync::Mutex; use std::sync::atomic::{AtomicU64, Ordering}; @@ -1288,9 +1289,10 @@ pub struct SoloContext { /// `--soloMultiMappers` includes a non-`Unique` method → capture gene- /// ambiguous reads for distribution into `UniqueAndMult-*.mtx`. pub want_multi: bool, - /// `--soloCellReadStats CB`: the per-cell read summary, or `None` when the - /// flag is off. Behind a mutex like the other per-read collections; the - /// lock is only taken when the flag asks for the file. + /// `--soloCellReadStats Standard`: the per-cell read summary, or `None` + /// when the flag is off. Each read's entry rides in its + /// [`SoloReadOutcome`] and is folded by the in-order consumer, one lock + /// per batch ([`SoloContext::fold_cell_reads`]). pub cell_read_stats: Option>, /// Chromosome indices named by `--genomeChrSetMitochondrial`, for the /// `mito` column. @@ -1404,6 +1406,9 @@ pub struct SoloReadOutcome { /// Barcode facts for the per-read SAM tags. `None` when the barcode read was /// too short to extract a CB+UMI at all. pub barcode: Option, + /// This read's `CellReads.stats` entry, when `--soloCellReadStats` asked + /// for the file; the consumer folds it in input order. + pub cell_read: Option, } impl SoloReadOutcome { @@ -1621,6 +1626,13 @@ impl SoloContext { (tags || velocyto).then(|| Mutex::new(Vec::new())) }) .collect(); + let n_features = features.len(); + let has_whitelist = !matches!(whitelist, CbWhitelist::NoWhitelist { .. }); + if params.solo_cell_read_stats == "CB" { + log::warn!( + "--soloCellReadStats CB is deprecated: use Standard, STAR's name for the same file" + ); + } Ok(Self { layout: SoloBarcodeLayout::from_params(params), whitelist, @@ -1641,8 +1653,14 @@ impl SoloContext { velocyto_enabled, velocyto_records: Mutex::new(VelocytoRecords::default()), want_multi, - cell_read_stats: (params.solo_cell_read_stats == "CB") - .then(|| Mutex::new(crate::solo::cell_reads::CellReadStats::new())), + cell_read_stats: params.solo_cell_read_stats_yes().then(|| { + Mutex::new(crate::solo::cell_reads::CellReadStats::new( + n_features, + params.solo_cb_match_type().one_exact(), + want_multi, + has_whitelist, + )) + }), mito_chr: params .genome_chr_set_mitochondrial .iter() @@ -1950,11 +1968,24 @@ impl SoloContext { } } + // `CellReads.stats` counts every read, whatever its barcode did. + let mut cell_flags = self + .cell_read_stats + .is_some() + .then(|| self.cell_read_flags(cdna_transcripts, n_loci, &class)); + let no_cb_entry = |flags: &mut Option>| { + flags.take().map(|flags| CellReadEntry { + cb: ReadCb::None, + flags, + }) + }; + // No barcode → nothing to count. let bc = match barcode { Ok(bc) => bc, Err(miss) => { self.record_barcode_miss(miss); + out.cell_read = no_cb_entry(&mut cell_flags); return out; } }; @@ -1980,11 +2011,16 @@ impl SoloContext { let cb_resolved: Option = match &cb_match { CbMatch::Exact(idx) | CbMatch::Corrected(idx) => Some(*idx), CbMatch::Multi(_) => None, // deferred to collation - CbMatch::NoMatch | CbMatch::NinCb | CbMatch::MultMatchRejected => return out, + CbMatch::NoMatch | CbMatch::NinCb | CbMatch::MultMatchRejected => { + out.cell_read = no_cb_entry(&mut cell_flags); + return out; + } }; - // UMI validity. + // UMI validity. A rejected UMI replaces the read's `cbMatch` with a + // negative code, so `CellReads.stats` has it under `CBnotInPasslist`. let UmiCheck::Ok(umi) = umi_status else { + out.cell_read = no_cb_entry(&mut cell_flags); return out; }; @@ -2136,14 +2172,17 @@ impl SoloContext { // The read's `cbMatch` is >= 0 here: STAR's `SoloReadFeature::record` // counts it for every feature. self.record_feature_stats(n_loci, &class, &out, code == 0, junctions); - self.record_cell_read( - cb_resolved, - &cb_match, - n_loci, - &class, - cdna_transcripts, - &out, - ); + out.cell_read = cell_flags.map(|flags| CellReadEntry { + cb: match cb_match { + CbMatch::Exact(idx) => ReadCb::Exact(idx), + CbMatch::Corrected(idx) => ReadCb::Corrected(idx), + CbMatch::Multi(cands) => ReadCb::Multi(cands), + CbMatch::NoMatch | CbMatch::NinCb | CbMatch::MultMatchRejected => { + unreachable!("returned above") + } + }, + flags, + }); // Transcript3p: every transcript this read is concordant with, and how // far its 3' end sits from each transcript's. Uniquely-mapped reads // only — a read at several genomic loci says nothing about isoforms. @@ -2165,52 +2204,62 @@ impl SoloContext { out } - /// Fold one read into `CellReads.stats`, when `--soloCellReadStats CB` - /// asked for it. - /// - /// Called at the end of read processing so the counted flags reflect what - /// the read actually produced, rather than what it looked eligible for. - #[allow(clippy::too_many_arguments)] - fn record_cell_read( + /// One read's read-stage `CellReads.stats` flags, per feature + /// (`SoloReadFeature_record.cpp:25-60`): `genomeU`/`genomeM` from its + /// number of loci, `mito` if any alignment is on a mitochondrial + /// chromosome, `featureU`/`featureM` from the size of the feature's gene + /// set, and the region bit from the feature's `ovType`. Only `Gene` has + /// one: `exonic` whenever it found a gene + /// (`Transcriptome_classifyAlign.cpp:262-263`); `GeneFull` sets none. + fn cell_read_flags( &self, - cb_resolved: Option, - cb_match: &CbMatch, - n_loci: usize, - class: &crate::solo::gene::ReadClass, transcripts: &[Transcript], - out: &SoloReadOutcome, - ) { - let Some(stats) = &self.cell_read_stats else { - return; - }; - let counted_u = out.per_feature.iter().any(|f| f.record.is_some()); - let counted_m = out.per_feature.iter().any(|f| f.multi_gene.is_some()); - let feature_u = counted_u || out.per_feature.iter().any(|f| f.multi.is_some()); - let flag = crate::solo::cell_reads::CellReadFlag { - cb_perfect: matches!(cb_match, CbMatch::Exact(_)), - cb_mm_unique: matches!(cb_match, CbMatch::Corrected(_)), - cb_mm_multiple: matches!(cb_match, CbMatch::Multi(_)), - genome_u: n_loci == 1, - genome_m: n_loci > 1, - feature_u, - feature_m: counted_m, - // The region columns split by strand: STAR reports an antisense - // read under `exonicAS`/`intronicAS`, not under `exonic`/`intronic`. - exonic: !class.antisense && class.region == Some(Region::Exonic), - intronic: !class.antisense && class.region == Some(Region::Intronic), - exonic_as: class.antisense && class.region == Some(Region::Exonic), - intronic_as: class.antisense && class.region == Some(Region::Intronic), - mito: !self.mito_chr.is_empty() - && transcripts - .iter() - .any(|t| self.mito_chr.contains(&t.chr_idx)), - counted_u, - counted_m, + n_loci: usize, + class: &ReadClass, + ) -> Vec { + use crate::solo::cell_reads::bit; + let mut common = match n_loci { + 0 => 0, + 1 => bit::GENOME_U, + _ => bit::GENOME_M, }; - let mut stats = stats.lock().unwrap(); - match cb_resolved { - Some(cb) => stats.add_cell(cb, &flag), - None => stats.add_no_cb(&flag), + if transcripts + .iter() + .any(|t| self.mito_chr.contains(&t.chr_idx)) + { + common |= bit::MITO; + } + self.features + .iter() + .map(|&feature| { + let assignment = match feature { + SoloFeature::Gene => class.gene, + SoloFeature::GeneFull => class.gene_full, + }; + let found = match assignment { + GeneAssignment::Gene(_) => bit::FEATURE_U, + GeneAssignment::Ambiguous => bit::FEATURE_M, + GeneAssignment::NoFeature | GeneAssignment::Unmapped => 0, + }; + let region = if feature == SoloFeature::Gene && found != 0 { + bit::EXONIC + } else { + 0 + }; + common | found | region + }) + .collect() + } + + /// Fold a batch of reads' `CellReads.stats` entries, in input order. + pub fn fold_cell_reads(&self, entries: Vec) { + if let Some(stats) = &self.cell_read_stats + && !entries.is_empty() + { + let mut stats = stats.lock().unwrap(); + for e in entries { + stats.fold(e); + } } } diff --git a/tests/solo_cell_read_stats.rs b/tests/solo_cell_read_stats.rs new file mode 100644 index 00000000..3bb55f4b --- /dev/null +++ b/tests/solo_cell_read_stats.rs @@ -0,0 +1,334 @@ +//! `--soloCellReadStats Standard` writes STARsolo's `CellReads.stats`, one table +//! per feature. +//! +//! The fixture is built so that the features disagree about a read: an +//! intronic read is a gene for `GeneFull` and nothing for `Gene`, which a single +//! table shared by both features cannot show. It also carries the reads STAR +//! sums into `CBnotInPasslist` (a barcode off the whitelist, a rejected UMI, a +//! correction `oneExact` drops) and a read on `chrM`, which STAR's default +//! `--genomeChrSetMitochondrial` (`chrM M MT`) counts as mitochondrial. + +use assert_cmd::cargo::cargo_bin_cmd; +use std::collections::HashMap; +use std::fs; +use std::io::Write; +use std::path::{Path, PathBuf}; +use tempfile::TempDir; + +const CB: &str = "AAAACCCCGGGGTTTT"; +const DECOY_CB: &str = "TTTTGGGGCCCCAAAA"; +/// Two exons of G1 with an intron between them, and a one-exon G2. +const G1_EXON1: (usize, usize) = (2_000, 2_600); +const G1_EXON2: (usize, usize) = (3_400, 4_000); +const G1_INTRON: usize = 2_900; +const G2: (usize, usize) = (8_000, 8_600); +const INTERGENIC: usize = 12_000; +const READ_LEN: usize = 60; + +fn lcg_seq(seed: u32, length: usize) -> Vec { + let bases: [u8; 4] = *b"ACGT"; + let mut state = seed; + let mut seq = Vec::with_capacity(length); + for _ in 0..length { + state = state.wrapping_mul(1_103_515_245).wrapping_add(12345); + seq.push(bases[((state >> 16) & 3) as usize]); + } + seq +} + +fn chr1() -> Vec { + lcg_seq(88888, 20_000) +} + +fn chr_m() -> Vec { + lcg_seq(4242, 2_000) +} + +fn write_fixture(dir: &Path) -> (PathBuf, PathBuf) { + let fasta = dir.join("genome.fa"); + let mut f = fs::File::create(&fasta).unwrap(); + for (name, seq) in [("chr1", chr1()), ("chrM", chr_m())] { + writeln!(f, ">{name}").unwrap(); + f.write_all(&seq).unwrap(); + writeln!(f).unwrap(); + } + + let gtf = dir.join("genes.gtf"); + let mut g = fs::File::create(>f).unwrap(); + for (name, (s, e)) in [("G1", G1_EXON1), ("G1", G1_EXON2), ("G2", G2)] { + writeln!( + g, + "chr1\tsyn\texon\t{}\t{}\t.\t+\t.\tgene_id \"{name}\"; transcript_id \"{name}_T1\";", + s + 1, + e + ) + .unwrap(); + } + (fasta, gtf) +} + +fn build_index(fasta: &Path, gtf: &Path, genome_dir: &Path) { + fs::create_dir_all(genome_dir).unwrap(); + cargo_bin_cmd!("rustar-aligner") + .args([ + "--runMode", + "genomeGenerate", + "--genomeDir", + genome_dir.to_str().unwrap(), + "--genomeFastaFiles", + fasta.to_str().unwrap(), + "--genomeSAindexNbases", + "7", + "--sjdbGTFfile", + gtf.to_str().unwrap(), + "--sjdbOverhang", + "59", + ]) + .assert() + .success(); +} + +/// Reads: `(contig, start, barcode, UMI, count)`. +fn write_reads(dir: &Path, plan: &[(&[u8], usize, &str, &str, usize)]) { + let mut cf = fs::File::create(dir.join("cdna.fq")).unwrap(); + let mut bf = fs::File::create(dir.join("barcode.fq")).unwrap(); + let mut i = 0usize; + for (contig, start, cb, umi, n) in plan { + for k in 0..*n { + // Stagger the start so the reads are not identical duplicates. + let s = start + (k % 5) * 7; + writeln!(cf, "@read{i}").unwrap(); + cf.write_all(&contig[s..s + READ_LEN]).unwrap(); + writeln!(cf, "\n+\n{}", "I".repeat(READ_LEN)).unwrap(); + writeln!( + bf, + "@read{i}\n{cb}{umi}\n+\n{}", + "I".repeat(cb.len() + umi.len()) + ) + .unwrap(); + i += 1; + } + } + let mut wf = fs::File::create(dir.join("whitelist.txt")).unwrap(); + for bc in [CB, DECOY_CB] { + writeln!(wf, "{bc}").unwrap(); + } +} + +/// One feature's file: barcode -> column -> value, plus the header. +type Table = HashMap>; + +fn run(dir: &Path, genome_dir: &Path, tag: &str, extra: &[&str]) -> (Vec, Table, Table) { + let out = dir.join(format!("out_{tag}")); + fs::create_dir_all(&out).unwrap(); + let prefix = format!("{}/", out.display()); + cargo_bin_cmd!("rustar-aligner") + .args([ + "--runMode", + "alignReads", + "--genomeDir", + genome_dir.to_str().unwrap(), + "--readFilesIn", + dir.join("cdna.fq").to_str().unwrap(), + dir.join("barcode.fq").to_str().unwrap(), + "--soloType", + "CB_UMI_Simple", + "--soloCBwhitelist", + dir.join("whitelist.txt").to_str().unwrap(), + "--soloCBstart", + "1", + "--soloCBlen", + "16", + "--soloUMIstart", + "17", + "--soloUMIlen", + "10", + "--soloFeatures", + "Gene", + "GeneFull", + "--outFileNamePrefix", + &prefix, + ]) + .args(extra) + .assert() + .success(); + let load = |feature: &str| -> (Vec, Table) { + let text = + fs::read_to_string(out.join(format!("Solo.out/{feature}/CellReads.stats"))).unwrap(); + let mut lines = text.lines(); + let header: Vec = lines + .next() + .unwrap() + .split('\t') + .map(String::from) + .collect(); + let rows = lines + .map(|l| { + let v: Vec<&str> = l.split('\t').collect(); + assert_eq!(v.len(), header.len(), "short row {l}"); + let row = header[1..] + .iter() + .cloned() + .zip(v[1..].iter().map(|s| (*s).to_string())) + .collect(); + (v[0].to_string(), row) + }) + .collect(); + (header, rows) + }; + let (header, gene) = load("Gene"); + let (header_full, full) = load("GeneFull"); + assert_eq!(header, header_full); + (header, gene, full) +} + +fn fixture() -> (TempDir, PathBuf) { + let dir = TempDir::new().unwrap(); + let root = dir.path(); + let (fasta, gtf) = write_fixture(root); + let genome_dir = root.join("genome"); + build_index(&fasta, >f, &genome_dir); + let (c1, cm) = (chr1(), chr_m()); + // One substitution from DECOY_CB, a barcode no read matches exactly. + let near_decoy = format!("A{}", &DECOY_CB[1..]); + write_reads( + root, + &[ + (&c1, G1_EXON1.0 + 20, CB, "ACGTACGTAC", 3), + (&c1, G1_INTRON, CB, "TGCATGCATG", 2), + (&c1, INTERGENIC, CB, "CCCCAAAAGT", 1), + (&cm, 500, CB, "GATTACAGAT", 1), + (&c1, G2.0 + 20, &near_decoy, "AACCGGTTAA", 2), + (&c1, G2.0 + 20, "CCCCAAAAGGGGTTTT", "TTGGCCAATT", 1), + (&c1, G2.0 + 20, CB, "ACGTNCGTAC", 1), + ], + ); + (dir, genome_dir) +} + +fn get<'a>(t: &'a Table, cb: &str, col: &str) -> &'a str { + &t[cb][col] +} + +#[test] +fn each_feature_counts_its_own_flags() { + let (dir, genome_dir) = fixture(); + let (header, gene, full) = run( + dir.path(), + &genome_dir, + "std", + &["--soloCellReadStats", "Standard"], + ); + assert_eq!(header.len(), 20); + assert_eq!(header[0], "CB"); + assert_eq!(header[1], "cbMatch"); + + // STAR's rows: the passlist misses plus the one cell that resolved. + for t in [&gene, &full] { + let mut cbs: Vec<&str> = t.keys().map(String::as_str).collect(); + cbs.sort_unstable(); + assert_eq!(cbs, [CB, "CBnotInPasslist"]); + // Every read lands in one row or the other. + let total: u64 = t + .values() + .map(|r| r["cbMatch"].parse::().unwrap()) + .sum(); + assert_eq!(total, 11); + assert_eq!(get(t, CB, "cbMatch"), "7"); + assert_eq!(get(t, CB, "cbPerfect"), "7"); + assert_eq!(get(t, CB, "genomeU"), "7"); + assert_eq!(get(t, CB, "mito"), "1", "chrM is mitochondrial by default"); + assert_eq!( + get(t, CB, "intronic"), + "0", + "no feature sets an intronic ovType" + ); + } + + // Gene: only the exonic reads are a feature, and `exonic` follows them. + assert_eq!(get(&gene, CB, "featureU"), "3"); + assert_eq!(get(&gene, CB, "exonic"), "3"); + assert_eq!(get(&gene, CB, "countedU"), "3"); + assert_eq!(get(&gene, CB, "nUMIunique"), "1"); + // GeneFull: the intronic reads are G1's too, and GeneFull has no ovType. + assert_eq!(get(&full, CB, "featureU"), "5"); + assert_eq!(get(&full, CB, "exonic"), "0"); + assert_eq!(get(&full, CB, "countedU"), "5"); + assert_eq!(get(&full, CB, "nUMIunique"), "2"); + + // CBnotInPasslist: the two corrections oneExact drops, the barcode off the + // whitelist, and the rejected UMI, all on G2, none counted. + for t in [&gene, &full] { + let miss = "CBnotInPasslist"; + assert_eq!(get(t, miss, "cbMatch"), "4"); + assert_eq!(get(t, miss, "cbPerfect"), "0"); + assert_eq!(get(t, miss, "cbMMunique"), "0"); + assert_eq!(get(t, miss, "genomeU"), "4"); + assert_eq!(get(t, miss, "featureU"), "4"); + assert_eq!(get(t, miss, "countedU"), "0"); + } +} + +/// STAR's value is `Standard`; the `CB` rustar-aligner took before is kept as +/// an alias of it, and anything else is refused. +#[test] +fn standard_is_the_value_and_cb_an_alias() { + let (dir, genome_dir) = fixture(); + let std = run( + dir.path(), + &genome_dir, + "std", + &["--soloCellReadStats", "Standard"], + ); + let cb = run( + dir.path(), + &genome_dir, + "cb", + &["--soloCellReadStats", "CB"], + ); + assert_eq!(std, cb); + + let out = dir.path().join("out_bad"); + fs::create_dir_all(&out).unwrap(); + cargo_bin_cmd!("rustar-aligner") + .args([ + "--runMode", + "alignReads", + "--genomeDir", + genome_dir.to_str().unwrap(), + "--readFilesIn", + dir.path().join("cdna.fq").to_str().unwrap(), + dir.path().join("barcode.fq").to_str().unwrap(), + "--soloType", + "CB_UMI_Simple", + "--soloCBwhitelist", + dir.path().join("whitelist.txt").to_str().unwrap(), + "--soloUMIlen", + "10", + "--soloCellReadStats", + "Yes", + "--outFileNamePrefix", + &format!("{}/", out.display()), + ]) + .assert() + .failure() + .stderr(predicates::str::contains("expected Standard or None")); +} + +/// `--genomeChrSetMitochondrial -` names no mitochondrial chromosome. +#[test] +fn mito_follows_the_chromosome_set() { + let (dir, genome_dir) = fixture(); + let (_, gene, _) = run( + dir.path(), + &genome_dir, + "nomito", + &[ + "--soloCellReadStats", + "Standard", + "--genomeChrSetMitochondrial", + "-", + ], + ); + assert_eq!(get(&gene, CB, "mito"), "0"); +}