Repository navigation
Conversation
# Conflicts: # src/sf_merger/cut.h # src/sf_merger/main.cpp # unittests/sf_merger/mask.cpp
Partition merger output from input index topology alone, so production reads each supplier once and never downscales. Define recovery fingerprints, require matching output codecs, add an optional compression setting, and report per-category output statistics that are checkpointed for recovery. Prepare resampled suppliers with the halo reader.
Add rf-merger, which merges two published RF snapshots by attribution priority. The output partition follows both input indices, coarse suppliers are upscaled with the shared halo reader and paired scaler, and unchanged tiles are hard-linked. Checkpoints record per-category statistics for recovery and the final report. Move TilePool to raster_store so the RF builder and merger share it, and run the merger suite in CI.
Error records a stack trace where it is made and gains Error::Exception, raise, throwing_unwrap and asserting_unwrap. The code style now separates bugs (assertions), recoverable failures (Expected, including all of terrainlib) and unrecoverable failures in tools (exceptions). rf-merger throws for invalid input and production failures, asserts bug-only results, and keeps Expected in its reusable functions. Preflight rejects sRGB on scalar payloads and zoom gaps above 30 levels. A statistics guard keeps a failed merge recoverable, and cancellation saves active tiles before throwing Cancelled. The priority list is parsed with CPLJSON.
Add rf-merger with topology partitioning, statistics and recovery, share TilePool between the RF builder and merger, and add Error exceptions, stack traces and unwrap helpers.
Envelopes get a new magic and a Header with a binary hash of the
uncompressed payload, verified on read. Hash and compression enumerators
are renumbered from zero: hash::Algorithm { None, Xxh3_64 } and
CompressionAlgorithm { None, ZstdBestCompression, ZstdDefaultCompression }.
CRC-32C and the Zstandard frame-checksum variants are removed, and Xxh3_64
with ZstdDefaultCompression is the default for every envelope writer.
io/hash.h provides hash::data over byte spans; only io/hash.cpp includes
the vendored xxHash v0.8.3. io/compression.h only compresses and
decompresses. read_header decodes a header from a file prefix of at most
222 bytes without reading the payload; class names are limited to 128
bytes and hashes to 64. Invalid enumerators from files are Unsupported,
and invalid ones passed by callers PANIC.
The RF merger fingerprints its inputs with the header hashes of their
metadata and index files and has no checksum option. Byte buffers of file
I/O, image encoding and decoding and the RF builder HTTP client are
std::byte. ADR 0005 records the decision and the measured 18.3 GB/s.
Hash envelope payloads with XXH3-64, expose envelope headers without reading the payload, fingerprint RF merger inputs with the header hashes, and switch byte buffers to std::byte.
…wn bespoke reader - merger doesn't need downscaling, - tb builder needs reading the already the generated level from below (tb) and the rf source (if reading ancestors/same level)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds persistent raster storage and tools to import, merge, and inspect attributed raster snapshots, while moving existing SF and DAG applications onto a shared storage foundation.
rf-builderfor GDAL datasets and online tiles, with attribution and validity handling, NoData filling, parallel preparation, reusable caches, checkpoints, cancellation, and progress reporting.rf-tile2image.rf-mergerwith attribution priorities, topology-based output partitioning, halo-aware upscaling, hard-link reuse, recovery, and per-category statistics.std::byte.Compatibility: the new envelope magic and renumbered hash/compression fields make envelopes written before this change unreadable; existing envelope-backed data must be regenerated.
Validation: repository status documents record Linux GCC Debug builds and all CI suites passing for the envelope-hash implementation, plus focused RF merger and sanitizer checks. The five GitHub Actions jobs for the current head are still running. No new build was run while preparing this PR; the latest history cleanup changed only commit messages, with every rewritten commit's file tree and merge structure verified unchanged.