Repository navigation
database_api: read block headers without unpacking the whole block - #286
Merged
Merged
Conversation
get_block_header() went through fetch_block_by_number(), which deserializes the whole signed_block (all transactions) and then returns only its base. Every unique block therefore cost ~129 KB of heap that glibc does not return to the OS, and a client enumerating blocks turned that into gigabytes of RSS. Measured 19.11.2026 on three stands, 2000 fresh blocks each, batch form: +245...249 MiB (+128...130 KB per block) — identical with thread pool 8 and 256 and with prediction_market_api disabled. Control: the same batch of 2000 get_accounts calls grew the process by 1.6 MiB, so the cost is the block payload, not batching or logging. On the live node a minute with 86 401 get_block_header calls (a 24 h block enumeration) stepped anon from 9.43 to 16.67 GB and the node was OOM-killed shortly after. - block_log::read_block_header_by_num() unpacks the reflected block_header from the same mmap position and stops there; the stored block is a signed_block whose wire order puts the header fields first (FC_REFLECT_DERIVED), and a header that names another height is rejected the same way read_block_by_num rejects a mismatched block - database::fetch_block_header_by_number() keeps the source order of fetch_block_by_number() (fork_db, block_summary disambiguation, main-branch walk) and uses the header-only read only for the block log; the DLT rolling log keeps the full-block path, it has no header-only read and is bounded - database_api.get_block_header() now calls it Verification in this commit: single-TU -fsyntax-only for block_log.cpp, database.cpp and plugins/database_api/api.cpp (rc=0), plus a new test that pins the premise the change rests on — a stored block unpacks as block_header, every header field matches, and the parser consumes the same number of bytes whether the tail has 0 or 50 transactions. The negative control for the test harness (a deliberately broken TU) fails as expected, so the green compile is real. Not verified here: end-to-end on a node image (no local build by project rule).
On1x
marked this pull request as ready for review
November 19, 2026 19:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
get_block_header()goes throughfetch_block_by_number(), which deserializes the wholesigned_block(all transactions) and then returns only its base. This change adds a header-only read path so that a header request never parses transaction payload.block_log::read_block_header_by_num()— unpacks the reflectedblock_headerfrom the same mmap position and stops there. The stored block is asigned_blockwhose wire order puts the header fields first (FC_REFLECT_DERIVED). A header naming another height is rejected exactly likeread_block_by_numrejects a mismatched block.database::fetch_block_header_by_number()— keeps the source order offetch_block_by_number()(fork_db → block_summary disambiguation → main-branch walk) and uses the header-only read only for the block log. The DLT rolling log keeps the full-block path: it has no header-only read and is bounded by_dlt_block_log_max_blocks.database_api.get_block_header()calls the new path.tests/chain/block_header_unpack_test.cpppins the premise: a stored block unpacks asblock_header, every header field matches, and the parser consumes the same number of bytes whether the tail holds 0 or 50 transactions.What this does NOT do (measured, not assumed)
It does not fix a memory problem, and an earlier version of this description claimed it did. Measurements on a stand with images built from
masterand from this branch:get_accounts(a method that reads no blocks) costs +2458 MiBConclusion: the large memory cost is a per-thread heap-arena inflation caused by processing very large JSON-RPC batches, independent of the method and of block deserialization. A public node's answer to that is client-side batch sizing (our own probes moved to 200-item batches: 16 clients × 200 → +48 MiB measured), not this patch. This patch remains a CPU/allocation-efficiency improvement on the header path.
Verification
-fsyntax-onlyfor the three touched TUs plus a negative control on the harness.viz/tools/rpc-block-cost.phpand a concurrent-batch harness; the block range never repeats, so the webserver response cache cannot stand in for the work.Behaviour is unchanged for callers: same header values, same "empty when unknown" semantics; only the deserialization depth changes.
Opened as draft: direction and merge are the owner's by project policy.