Fetch only what you need. Store each unique chunk once.
SparseIO is infrastructure and an extensible Rust library for coordinating sparse, out-of-order ranged fetches to materialize large-object content-addressable storage (CAS).
Large objects are often consumed a few ranges at a time: a tensor from a model, a row group from a dataset, several blocks from a backup, or a segment from a media file. Fetching the entire object before serving the first useful byte wastes time, bandwidth, and storage. SparseIO materializes an object incrementally instead. A requested range is fetched from its upstream source, split into stable chunks, and stored by content hash so future reads can reuse it.
| Whole-object caching | SparseIO |
|---|---|
| Downloads every byte on the first miss | Fetches only the ranges callers request |
| Stores repeated data once per object | Deduplicates equal chunks by content hash |
| Can duplicate work during concurrent misses | Coalesces in-flight requests for the same chunk |
| Couples the cache to a source or runtime | Uses pluggable, executor-neutral backend traits |
This is especially useful for:
- AI/ML models and datasets where only selected tensors or shards are needed.
- Database backups, VM images, and archives explored without a full restore.
- Columnar data, logs, and scientific data read non-sequentially.
- Media and other large remote objects served through byte-range requests.
The 64 KiB loopback API workload applies deterministic network latency, jitter, bandwidth, and a 16-request upstream concurrency limit. In a 1,000-request local run:
| p99 result | Regional profile | WAN profile |
|---|---|---|
| SparseIO cold range versus direct OpenDAL | 30.83 ms, 1.4% higher | 83.95 ms, 0.8% higher |
| SparseIO warm range versus direct source | 184.45 us, 99.4% lower | 237.95 us, 99.7% lower |
| SparseIO singleflight versus 32-way direct fan-out | 32.16 ms, 47.6% lower | 83.89 ms, 48.6% lower |
Across 32 fan-out batches, direct readers made 1,024 observed upstream range requests while SparseIO made 32. These are reproducible workload results, not public internet claims or portable hardware guarantees. See Benchmarking for the schedules, profiles, full metrics, and reproduction commands.
For each range read, SparseIO:
- Normalizes the requested range into fixed-size chunks.
- Looks up each chunk in the metadata index and local cache.
- Fetches missing chunks from the registered upstream
Reader. - Coalesces concurrent misses so only one upstream fetch does the work.
- Hashes and writes new chunks into the CAS through the configured
Writer. - Returns the requested bytes while the object becomes incrementally available.
Because chunks are addressed by their content, identical regions can be shared across objects and versions. A fine-tuned model, incremental database backup, or revised disk image only needs storage for the chunks that actually changed.
Compose storage systems directly in Rust using small, object-safe
Reader,
Writer, and
Metadata traits. The core remains independent
of Tokio or any other specific async executor.
The backend contracts intentionally stay narrow:
| Component | Responsibility | Example implementations |
|---|---|---|
Reader |
Fetch byte ranges from an upstream object | HTTP, S3, Hugging Face, local files |
Writer |
Store and retrieve content-addressed chunks | Local disk, object storage, distributed caches |
Metadata |
Track object coverage and chunk lifecycle | Redis, another key-value store, embedded state |
ReaderRegistry |
Route canonical object paths to readers | Application-defined source schemes |
Bring the systems that fit your workload; SparseIO coordinates the read path, sparse coverage, in-flight work, and CAS materialization.
- Sparse by default: requesting one range never requires materializing the whole object.
- Backend agnostic: sources, chunk storage, and metadata are replaceable.
- Runtime neutral: the library remains usable from Tokio, smol, async-std, and other executors.
- Safe under concurrency: overlapping requests share work instead of multiplying upstream traffic.
- Cache, not custody: missing or expired cached chunks fall back to the source of truth.
SparseIO is under active pre-1.0 development. The initial sparse read coordinator, cache lifecycle, reference backends, validators, and feature-gated test support are implemented. The public API may still evolve before production stability is declared.