Skip to content

database_api: read block headers without unpacking the whole block - #286

Merged
On1x merged 1 commit into
masterfrom
fix/header-only-read
Nov 19, 2026
Merged

On1x merged 1 commit into
masterfrom
fix/header-only-read

Conversation

@On1x

@On1x On1x commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

What this changes

get_block_header() goes through fetch_block_by_number(), which deserializes the whole signed_block (all transactions) and then returns only its base. This change adds a header-only read path so that a header request never parses transaction payload.

  • block_log::read_block_header_by_num() — unpacks the reflected block_header from the same mmap position and stops there. The stored block is a signed_block whose wire order puts the header fields first (FC_REFLECT_DERIVED). A header naming another height is rejected exactly like read_block_by_num rejects a mismatched block.
  • database::fetch_block_header_by_number() — keeps the source order of fetch_block_by_number() (fork_db → block_summary disambiguation → main-branch walk) and uses the header-only read only for the block log. The DLT rolling log keeps the full-block path: it has no header-only read and is bounded by _dlt_block_log_max_blocks.
  • database_api.get_block_header() calls the new path.
  • New test tests/chain/block_header_unpack_test.cpp pins the premise: a stored block unpacks as block_header, every header field matches, and the parser consumes the same number of bytes whether the tail holds 0 or 50 transactions.

What this does NOT do (measured, not assumed)

It does not fix a memory problem, and an earlier version of this description claimed it did. Measurements on a stand with images built from master and from this branch:

  • sequential client, one 2000-block batch: +221 MiB (before) vs +193.6 MiB (after); a second batch on the warmed heap costs +0.7 MiB
  • 16 concurrent clients, 2000-block batches: +3427 MiB (before) vs +3272 MiB (after) — i.e. no difference
  • the same concurrent shape with get_accounts (a method that reads no blocks) costs +2458 MiB

Conclusion: the large memory cost is a per-thread heap-arena inflation caused by processing very large JSON-RPC batches, independent of the method and of block deserialization. A public node's answer to that is client-side batch sizing (our own probes moved to 200-item batches: 16 clients × 200 → +48 MiB measured), not this patch. This patch remains a CPU/allocation-efficiency improvement on the header path.

Verification

  • CI on this branch: Build docker image — success; Tests — 4/4 success (consensus_sim + pm unit tests, testnet 1/2/4 validators). Range-floor and arm64 are skipped by their own conditions, not green.
  • single-TU -fsyntax-only for the three touched TUs plus a negative control on the harness.
  • Stand A/B numbers above come from viz/tools/rpc-block-cost.php and a concurrent-batch harness; the block range never repeats, so the webserver response cache cannot stand in for the work.

Behaviour is unchanged for callers: same header values, same "empty when unknown" semantics; only the deserialization depth changes.

Opened as draft: direction and merge are the owner's by project policy.

get_block_header() went through fetch_block_by_number(), which deserializes the
whole signed_block (all transactions) and then returns only its base. Every
unique block therefore cost ~129 KB of heap that glibc does not return to the
OS, and a client enumerating blocks turned that into gigabytes of RSS.

Measured 19.11.2026 on three stands, 2000 fresh blocks each, batch form:
+245...249 MiB (+128...130 KB per block) — identical with thread pool 8 and
256 and with prediction_market_api disabled. Control: the same batch of 2000
get_accounts calls grew the process by 1.6 MiB, so the cost is the block
payload, not batching or logging. On the live node a minute with 86 401
get_block_header calls (a 24 h block enumeration) stepped anon from 9.43 to
16.67 GB and the node was OOM-killed shortly after.

- block_log::read_block_header_by_num() unpacks the reflected block_header from
  the same mmap position and stops there; the stored block is a signed_block
  whose wire order puts the header fields first (FC_REFLECT_DERIVED), and a
  header that names another height is rejected the same way read_block_by_num
  rejects a mismatched block
- database::fetch_block_header_by_number() keeps the source order of
  fetch_block_by_number() (fork_db, block_summary disambiguation, main-branch
  walk) and uses the header-only read only for the block log; the DLT rolling
  log keeps the full-block path, it has no header-only read and is bounded
- database_api.get_block_header() now calls it

Verification in this commit: single-TU -fsyntax-only for block_log.cpp,
database.cpp and plugins/database_api/api.cpp (rc=0), plus a new test that pins
the premise the change rests on — a stored block unpacks as block_header, every
header field matches, and the parser consumes the same number of bytes whether
the tail has 0 or 50 transactions. The negative control for the test harness
(a deliberately broken TU) fails as expected, so the green compile is real.
Not verified here: end-to-end on a node image (no local build by project rule).
@On1x
On1x marked this pull request as ready for review November 19, 2026 19:41
@On1x
On1x merged commit 6f6f35c into master Nov 19, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant