Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
9882129
Let sub_matrix subset cupy arrays and keep float32 when moving data t…
CarlKCarlK Nov 15, 2026
1988d19
Detect cupy arrays directly in sub_matrix and reject out-of-range cup…
CarlKCarlK Nov 15, 2026
4d3cbe1
Pass the cupy module to _sub_matrix_cupy instead of importing it again
CarlKCarlK Nov 15, 2026
2f65034
Avoid a second GPU copy in _sub_matrix_cupy
CarlKCarlK Nov 15, 2026
5a45e81
Keep dtype when moving PstData val between numpy and cupy, in both di…
CarlKCarlK Nov 15, 2026
2640a5e
Match the Rust standardizer on cupy: raise for an all-missing SNP, an…
CarlKCarlK Nov 15, 2026
481bde4
Restructure PstData val conversion into one helper with a case per in…
CarlKCarlK Nov 15, 2026
d06c81b
Require sub_matrix's precision warning on both paths in test_sub_matr…
CarlKCarlK Nov 15, 2026
d647dbc
Check that a cupy SnpData subset stays on the GPU with its dtype
CarlKCarlK Nov 15, 2026
89d259b
Document GPU arrays and preview explicit reader backend APIs
CarlKCarlK Nov 16, 2026
366d6bb
Add explicit reader backends and fix GPU standardization
CarlKCarlK Nov 17, 2026
73cce29
Save detailed GPU workflow notebook before simplification
CarlKCarlK Nov 17, 2026
1a60cfc
Preserve CubeCL GPU allocations and simplify the GPU workflow
CarlKCarlK Nov 17, 2026
e99633f
Update BGEN tests for sample paths returned by bgen-reader
CarlKCarlK Nov 17, 2026
cab4781
Honor ARRAY_DTYPE across reads, kernels, and GPU workflows
CarlKCarlK Nov 17, 2026
5ff6a0e
Prepare PySnpTools 1.1.0a1 with published GPU dependencies
CarlKCarlK Nov 18, 2026
8006a8a
Use published alpha CPU wheels in PySnpTools qualification
CarlKCarlK Nov 18, 2026
f739d82
Record native Linux NVIDIA alpha validation
CarlKCarlK Nov 18, 2026
75652ce
Honor num_threads across MKL and OpenBLAS local runners
CarlKCarlK Nov 18, 2026
110ef95
Rely on bed-reader's GPU memory handling and use CuPy for kernels
CarlKCarlK Nov 19, 2026
238aae1
Make runner thread tests respect platform BLAS visibility
CarlKCarlK Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ jobs:
env:
# cbgen's bundled zlib needs POSIX declarations on Linux.
CFLAGS: ${{ runner.os == 'Linux' && '-D_POSIX_C_SOURCE=200809L' || '' }}
run: uv sync --frozen --all-extras
run: uv sync --frozen --extra dev --extra hdf5 --extra bgen

- name: Run the optional BGEN suite
run: uv run --frozen --no-sync pytest pysnptools/distreader/test_bgen2.py
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
env:
# cbgen's bundled zlib uses POSIX fileno in strict C mode.
CFLAGS: -D_POSIX_C_SOURCE=200809L
run: uv sync --frozen --all-extras
run: uv sync --frozen --extra dev --extra hdf5 --extra bgen

- name: Run the canonical core suite
run: uv run --frozen --no-sync python tests/test.py
Expand Down
55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,61 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [1.1.0a1] - 2026-11-17

* Add the `cupy` installation extra, using the published bed-reader 1.3.0a1
alpha for GPU reads and native allocations. CPU execution remains the
default. Initial GPU support targets Python 3.11 through 3.14 on NVIDIA
Windows/Linux x86-64, including WSL2, following CuPy's CUDA prerequisites.

* Add `ARRAY_DTYPE` defaults for reads, kernels, submatrix copies, and untyped
containers, with explicit dtype taking precedence. Optional dtype signatures
now default to `None`; the unconfigured float64 defaults are unchanged.
Existing floating arrays and standardizers keep their precision.
* Keep Pheno, Dense, Dat, and Ped text parsing on the CPU until a read chooses
its backend and precision, avoiding GPU allocation during metadata access.

* Keep newly copied supported GPU subsets in CubeCL-backed storage so fitting
Unit/Beta can use native standardization, including FaST-LMM test-SNP blocks
from preloaded `SnpData`. Preserve caller-owned arrays and shared views.

* Prefer bed-reader's CubeCL allocator for new supported GPU matrices and kernel
outputs, retaining caller-supplied CuPy storage. Fit Unit/Beta natively only
for complete registered matrices; use CuPy for other arrays/views and applying
trained statistics, without CPU fallback.
Honor empty-container factories so `val=None` retains the intended floating
dtype and backend for empty `SnpData` and `KernelData`.

* Reject zero, negative, and nonfinite `DiagKtoN` normalization denominators with
`ValueError` before changing values on NumPy or CuPy, including empty inputs.
GPU SNP normalization now propagates NaNs into the denominator rather than
silently ignoring them.

* Reject invalid kernel block sizes, including negative sizes that previously
could silently produce an all-zero kernel.

* Apply Unit/Beta trained statistics across float32/float64 CPU arrays in either
memory order, fixing a native type error for Fortran-order targets.

* Fix blocked kernel construction with `BetaTrained`: validate the complete
SNP ID order, reuse the matching fitted moments in each block, and preserve
the original trained standardizer when returning training information.

* Fix Python CPU Beta standardization of constant SNPs with singular endpoint
densities (for example, `Beta(0.5, 0.5)`): return zeros without invalid
multiplications during fitting or application of trained statistics.

* Validate `sub_matrix` indices before conversion on both NumPy and CuPy.
Reject floating-point indices, Boolean masks, invalid shapes, and out-of-range
values with `IndexError` instead of truncating, wrapping, or reaching a native panic.
* Preserve float32/float64 when moving numerical values between NumPy and CuPy.
* Support CuPy arrays in `sub_matrix`, preserving the input device, requested
layout, and dtype; reject negative or out-of-range CuPy indices and warn when
narrowing float64 to float32.
* Support Beta standardization on CuPy and reject wholly missing SNPs when
fitting Python/CuPy Unit or Beta standardizers, matching the native path.
* Add a GPU guide with examples and current backend and reader limitations.

## [1.0.1] - 2026-11-14

* Make `Hashdown` downloads (used by `example_file`) more robust: retry
Expand Down
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ Main Features:
New features include multi-threaded BED reading, cluster-ready BED data, on-the-fly SNP generation,
and larger in-memory data.

* **Experimental GPU arrays:** Read, standardize, and construct kernels with CuPy.
`ARRAY_MODULE` and `ARRAY_DTYPE` supply optional backend/precision environment defaults.
See the [GPU guide](https://github.com/fastlmm/PySnpTools/blob/cupy-sub-matrix/doc/source/gpu.rst) and [workflow notebook](https://github.com/fastlmm/PySnpTools/blob/cupy-sub-matrix/doc/ipynb/gpu_workflow.ipynb).
The `cupy` extra uses CUDA-enabled bed-reader wheels for native GPU BED reads.

* [DistReader](https://fastlmm.github.io/PySnpTools/#module-pysnptools.distreader): Efficiently work with
unphased BGEN format and other diploid, biallelic distribution data.
Also, efficiently read parts of files. See [Distribution IPython Notebook](https://nbviewer.jupyter.org/github/fastlmm/PySnpTools/blob/main/doc/ipynb/Dist.ipynb).
Expand All @@ -30,6 +35,13 @@ Install

Requires Python 3.11.5 or later. Python 3.15 qualification is in progress.

For the experimental GPU alpha:

pip install "pysnptools[cupy]==1.1.0a1"

GPU use requires compatible NVIDIA hardware, a driver, and CuPy's CUDA setup.
Python 3.15 is currently CPU-only; see the GPU guide for platform limits.

HDF5 support is optional. To read or write HDF5 files, install:

pip install pysnptools[hdf5]
Expand Down
301 changes: 301 additions & 0 deletions doc/ipynb/gpu_workflow.ipynb

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions doc/source/gpu-api-preview.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
GPU reader API
==============

The reader API is documented in :doc:`gpu`, with complete executable examples
for explicit backend selection, standardization, native BED reads, and kernels.
The API is part of the current development implementation; this page retains
its earlier URL for existing links.
Loading
Loading