Skip to content

SNOMED CT: hold each loaded edition once, not twice (use cache sections in place) - #329

Merged
grahamegrieve merged 1 commit into
HealthIntersections:mainfrom
jmandel:snomed-sections-in-place
Nov 1, 2026
Merged

grahamegrieve merged 1 commit into
HealthIntersections:mainfrom
jmandel:snomed-sections-in-place

Conversation

@jmandel

@jmandel jmandel commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Each loaded SNOMED CT edition currently takes twice its file size in RAM. This change makes it take its size once, with no behaviour change.

Cause

In tx/sct/structures.js:

  1. SnomedFileReader.load() reads the whole .cache file into one Buffer.
  2. Every structure constructor (SnomedStrings, SnomedConceptList, SnomedDescriptions, SnomedRelationshipList, …) copies its section with Buffer.from(buffer).
  3. The factory's shared data keeps the reader's results, which are views into the file Buffer, so the file Buffer is never freed.

So each edition is held once as the file Buffer and once as the per-structure copies.

Measured

process.memoryUsage().arrayBuffers, loading the SNOMED CT editions from tx.fhir.org's source list:

cache files on disk ArrayBuffers before after
International 20250201 822 MB 1.64 GB 0.82 GB
9 editions (the production set) 6.29 GB 12.0 GiB 6.0 GiB

The ratio is exactly 2× for 1, 3 and 9 editions, and a heap snapshot shows no other SNOMED CT retainer of any size. Loading is a little faster too, since nothing is copied.

For context, the IG Publisher transcripts on build.fhir.org (qa-tx.html) show tx.fhir.org aborting 1,583 calls on 29 Oct with "server memory usage (23.6 GB) exceeds safe threshold (98% of 24.0 GB limit)". In our replays of that traffic, SNOMED CT was about 12 GB of main's ~15 GB anonymous RSS.

Change

A WeakSet records the ArrayBuffers that SnomedFileReader loaded. Each structure constructor now calls adopt(buffer):

  • If the buffer is a section of a reader-loaded file, the structure uses it in place.
  • Any other buffer is still copied, so a structure always owns its bytes when it isn't backed by a loaded file.

Nothing writes to a loaded structure; the builders allocate their own buffers. 1 file, +22 / −10.

Testing

  • Unit tests: tests/cs/cs-snomed.test.js (41 tests) passes. So do the ECL, search, subsumption, MRCM and expression-grammar suites (206 passed; 25 skipped because they need data not present here).
  • Real traffic: the full tx.fhir.org source list, replaying real IG Publisher traffic, gives the same answers before and after.

A separate, optional follow-up could memory-map the cache files (about 15 GB → 3 GB anonymous RSS, same warm latency). It needs an optional native dependency, so it's not in this PR.

🤖 Generated with Claude Code

…opying them

Each structure copied its section of the cache file (Buffer.from), while the
factory's shared data kept the reader's views of the file, so the whole file
buffer stayed alive too: every loaded edition took twice its file size
(sct_intl_20250201: 1.64 GB of ArrayBuffers for an 822 MB file). Sections of
a file loaded by SnomedFileReader are now used in place. Nothing writes to a
loaded structure; buffers from anywhere else are still copied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@grahamegrieve
grahamegrieve merged commit 2b0a29c into HealthIntersections:main Nov 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants