Skip to content

perf(cpp): share a writer-owned schema cache across tree and table writes - #945

Closed
ColinLeeo wants to merge 1 commit into
apache:developfrom
ColinLeeo:perf/cpp-writer-owned-schema-cache
Closed

perf(cpp): share a writer-owned schema cache across tree and table writes#945
ColinLeeo wants to merge 1 commit into
apache:developfrom
ColinLeeo:perf/cpp-writer-owned-schema-cache

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Repeated writes currently resolve each measurement name through a per-device schema map. This PR reuses those lookups through one bounded cache owned by TsFileWriter, covering tree records, tree tablets, and table FIELD columns.

Related to #885 and the earlier tree-only implementation in #934. This is an alternative implementation based on the current develop branch; the initial cache regression cases build on #934.

Changes

  • Keep an LRU of at most 64 device entries per writer, with a fast path for the most recent device. Entries hold non-owning schema pointers and preserve the input column order; positional names are checked on every call, and unresolved columns are looked up again after registration.
  • Share the cache across aligned/non-aligned tree writes and table FIELD lookup. Clear it before schema destruction; flush preserves the schema objects. Eviction does not invalidate the chunk writers already saved in a table write context.
  • Keep omitted aligned fields synchronized with time rows, pages, and chunks. Cache the omitted-field list so repeated full-schema writes avoid rescanning all registered fields. Fields registered before the first flush receive the preceding empty pages and null positions; adding a new aligned field after the device's first flush returns E_NOT_SUPPORT rather than producing incomplete historical alignment.
  • Read the existing one-varint empty aligned-page representation and reset the corresponding reader state. Preserve a single empty aligned value page during crash recovery so later valid chunks are not truncated.

The entry-count bound is 64 devices, not a byte limit: each entry retains vectors sized for its device's schema. This does not add support for concurrent public calls on one writer; internal parallel encoding continues to use the existing thread pool.

Validation

  • macOS arm64, Apple Clang 21; Release build with bundled compression dependencies and LZMA2 enabled.
  • Full C++ test binary: 957 passed, 3 skipped because their external fixture environment variables were unset; 10 existing disabled tests were not run.
  • Explicitly instrumented Debug ASan/UBSan run: all 92 targeted tests completed successfully with no ASan findings (detect_leaks=0). UBSan reported existing null-pointer arithmetic at common/allocator/page_arena.h:92; a standalone PageArena::alloc(2048) followed by alloc(8) with a 1024-byte page reproduces it without any writer or cache. The allocator is unchanged from the base, and this is not a clean UBSan run.
  • All ten changed C++ files pass clang-format 17.0.6 verification and git diff --check.

The regression suite checks row values and timestamps across repeated writes, changed names/order/width, late registration, multiple devices and writers, destroy/re-init, eviction beyond 64 devices, table TAG/FIELD layouts, sparse aligned fields, serial/parallel page boundaries, and recovery preserving an empty-page backfill and the following valid chunks.

@ColinLeeo ColinLeeo closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant