[jmandel]

Decisions for you

#	What we chose	Alternative	Why
1	VSAC value sets are read on demand from the SQLite database, with a small cache of recently used value sets (64 MB of stored JSON, least recently used dropped first); there is no in-memory map of them all	Keep the map; or read on demand with no cache	The map is about 1 GB, and rebuilding it at the end of a sync crashed the process with Node's default heap limit. With no cache, a value set used over and over was parsed every time: a 1,000-entry $batch-validate-code took 5.3 s instead of 1.15 s; with the cache it takes 1.08 s
2	A new provider method, scanAllValueSets(), for the usage-tracker scan and /problems.html; its default is the old loop	Keep the callers looping over fetchValueSet	On demand the loop takes 10 s; VSAC's version reads in batches and takes 1.9 s
3	Search reads only the matches on the page it returns	Parse every match, then page	status=active would parse about 40,000 definitions to return 200
4	A raw VSAC id passed as a url no longer finds the value set	Keep finding it	It only worked by accident, because the map was keyed by id too; the id-space rules in resource-ids.js say it shouldn't
5	A sync's changes become visible as each value set is written	Visible all at once when the sync finishes	No second copy is built, so there is nothing to swap in at the end

Why

• About 1 GB less memory per process, with the same answers. Today the VSAC provider parses all ~40,000 VSAC value sets into memory at startup: 585 MB of heap, about 1.1 GB of process memory. They are already stored and indexed in vsac-valuesets.db; this reads each one when a request asks for it.
• More processes fit on one machine. Memory is what limits running several FHIRsmith processes on a 16 GB machine. On a 16 GB, 4-core t2d-standard-4 running 4 processes at 400 requests/s, the processes used 6.1 GB of their own memory with this change and 10.0 GB without, and handled the same peak load. (Measured on our development branch, which carries the same change.)
• A VSAC sync no longer needs a second copy. At the end of every sync the old code builds a new 1 GB map while the old one is still in use. With Node's default heap limit, that sync crashed the process ("JavaScript heap out of memory"). Now the heap grows by 65 MB.
• The cost is a read and parse the first time a request needs a very large value set: $validate-code against the 57,017-code Medical Cohort goes from 10 to 53 ms, and against a 331-member grouping value set from 20 to 100 ms. A typical VSAC value set costs 0.1 ms. Value sets used again are kept parsed (see "Cache for repeated use"), so repeated use costs what it did with the map. Over 13,718 replayed IG Publisher calls the responses were identical and the total time was the same.

It applies cleanly on main and on #331. 7 commits, 37 new tests.

Cause

In tx/vs/vs-vsac.js, _reloadMap() calls database.loadAllValueSets(). That parses all ~40,000 rows of vsac-valuesets.db (about 400 MB of JSON) into a Map keyed by id, url, and url|version.

• At startup the map takes 585 MB of heap, or about 1.1 GB RssAnon.
• During a sync _reloadMap() runs again at the end of every sync and builds the new map while the old one is still in use. The heap goes up by another ~1 GB until the old map is collected.

Every value set is already stored in that database, indexed by url/version and by id. The map is used for three things: picking the version a url refers to, map.get(id) after the search SQL has run, and the sync's "have we seen this url|version / has its hash changed" test.

Measured

The test server ran the production source list with two SNOMED CT editions, plus internal:vsac with a pre-synced snapshot of 40,267 value sets. Each process was capped at 6 GB. The replay was the IG Publisher calls from build.fhir.org's qa-tx.html transcripts, one call at a time.

	before	after
anonymous RSS after startup	3.6–3.8 GB	2.7–2.9 GB
heap after startup	1,421 MB	848 MB
heap growth during a sync	+1.0 GB (to 2.45 GB)	+65 MB
sync with Node's default heap limit in a 6 GB container (2.2 GB)	process died: JavaScript heap out of memory	completes, heap peaks at 944 MB
13,718 IG calls (6,029 that name VSAC value sets)		same responses, same total time (within run-to-run noise)
typical ValueSet?url= for a VSAC value set	1.6–1.9 ms	1.9–2.2 ms
$validate-code against the largest list (Medical Cohort, 57,017 codes)	10 ms	53 ms
$validate-code against a 331-member grouping value set	20 ms	100 ms
ValueSet?status=active (no _elements)	100–116 ms	123 ms, no extra memory
all 4,933 IG ValueSet searches, 2 passes	7.6–11.5 s	8.2–8.9 s

These were measured before the cache was added. Without it, the biggest lists pay a read and parse on every request: 38 ms for the 6 MB Medical Cohort, 11–16 ms for the 2.5–4.6 MB lists, and 0.1 ms for a typical value set. Grouping value sets pay about 0.25 ms for each child value set. With the cache, they pay this only when they are not among the recently used ones.

Cache for repeated use

Read on demand with no cache, every fetchValueSet re-read and parsed the value set: 15 ms for Medication Brand Name (33,341 codes), 3.6 ms for a 9,500-code list. $batch-validate-code makes a new worker for each entry, so it parsed the value set once per entry.

The provider now keeps the value sets it has read, parsed and prepared, by row id (the oid-version id), up to 64 MB of their stored JSON. When it is over that, it drops the least recently used. The size is valueSetCacheMB in the VSAC config, next to refreshIntervalHours; 0 turns the cache off.

• Which version a url gets is unchanged. It is still worked out from the rows for that url (listVersions, about 0.2 ms) before the cache is looked at.
• The cache is kept in step with the sync. The sync already compares each value set's hash with the stored one; whenever it writes a row (new, updated, or a hash backfill) it drops that row from the cache. An operator resync drops that url's rows. A full sync, a new database and close() empty it. A read that was under way when anything was dropped is not kept. Nothing checks the hash on each request: reading content_hash from a 2.5 MB row costs 0.54 ms.
• Search doesn't fill it. A search reads the matches on its page from the cache when they are there, but doesn't add them, so a broad search doesn't push out the value sets in use.
• Cached value sets are shared between requests, as the map's were. Nothing writes to them: $expand and the R4 conversion work on copies. A server run with every cached value set deep-frozen, covering $validate-code, $batch-validate-code, $expand, read, search, and HTML and XML output, gave the same results with no errors.
• Small value sets are cached too. A 3 KB value set uses 3 KB of the 64 MB, and caching it still pays: a 1,000-entry batch against one took 0.39 s cached and 0.70 s uncached.

Measured on the same snapshot and machine as above, one request at a time. Median of 200 calls, or the best of 3 runs for batches:

	main (map)	on demand, no cache	on demand, with cache
$validate-code, Medication Brand Name (33,341 codes), same code 200 times	3.7 ms	18.3 ms	4.1 ms
the same, through a value set that includes it	4.5 ms	18.4 ms	4.0 ms
$validate-code, a 20-code value set	0.9 ms	1.1 ms	0.8 ms
$batch-validate-code, 200 entries, Medication Brand Name	5.4 s	8.2 s	4.9 s
$batch-validate-code, 1,000 entries, a 9,500-code list	1.15 s	5.28 s	1.08 s
$batch-validate-code, 1,000 entries, a 20-code value set	0.53 s	0.70 s	0.39 s
$validate-code round robin over 20 large lists (15 MB of JSON), 10 rounds	2.0 ms	5.4 ms	2.1 ms
the same over 20 small value sets	1.2 ms	1.4 ms	1.3 ms
peak process memory over the whole run	4.3 GB	3.7 GB	3.6 GB

When full, 64 MB of stored JSON is about 90 MB of heap if it is all large lists, and up to about 190 MB if it is all small value sets.

Change

• fetchValueSet(url, version) reads only the ids and versions stored for that url. It picks one with the same addToMap / isNewerVersion rules as before: exact version, then major.minor, then latest. Then it reads and parses that one row and applies the same changes _reloadMap() applied: it strips compose.include/exclude[].version, prefixes the id, and attaches sourcePackage and contentHash. A row that fails new ValueSet() is logged and skipped, as it was when the map was loaded, and the next candidate version is used.
• fetchValueSetById reads by id.
• Search reads only the page it returns. The search worker asks every provider for all its matches, sorts them, then returns one page (200 by default). Read on demand, that would have meant parsing every match: status=active read about 40,000 definitions to return 200, taking 680 ms and +200 MB. Now:
  • VSAC runs the existing _buildSearchQuery SQL and returns matches carrying the fields a search sorts by (id, url, version, date, name), each with a non-enumerable load().
  • SearchWorker.loadPage reads only the matches on the page, in parallel.
  • Totals, _summary, _total, _sort, order across providers, _elements and paging links are unchanged.
  • If a match on the page can't be read, every match is read, so the page and the total are what they would have been. The one gap: an unreadable row that is on no page asked for is still counted. The snapshot has no such rows.
  • The indexed-column _elements fast path is untouched, and providers that hold their resources in memory work as before.
• Sync compares against content_hash in SQLite and no longer builds a map. After a sync, only the counts are refreshed.
• The usage-tracker startup scan and /problems.html used to call fetchValueSet for every row of listAllValueSets(). They now go through a provider method, scanAllValueSets(), whose default is that same loop. VSAC's version reads the latest version of each url once, in batches, so the scan takes 1.9 s instead of 10 s on demand (0.9 s with the map). The tracker's counts are identical.
• The recently used value sets are cached, up to 64 MB of their stored JSON, and dropped as the sync writes them (see "Cache for repeated use").

Behaviour differences:

• fetchValueSet() with a url that is really a raw VSAC id used to find that value set by accident, because the map was keyed by id as well as by url. Now it finds nothing, which is what the id-space rules in resource-ids.js require.
• Changes written by a sync can be seen as soon as each value set is written, instead of when the sync finishes.

Files: tx/vs/vs-vsac.js, tx/vs/vs-database.js, tx/vs/vs-api.js, tx/usage-tracker.js, tx/problems.js, tx/workers/search.js, plus two new test files. 7 commits.

Testing

• Unit tests: tests/vs and tests/tx have 5,693 tests before and 5,730 after. The VSAC suites pass, and so does everything else apart from 69 SNOMED, LOINC and language-pack cases in test-cases.test.js that need terminology data my latest checkout lacks; they fail the same way without the cache. The new tests are 21 in tests/vs/vs-vsac-on-demand.test.js and 16 in tests/tx/search-deferred.test.js. The search tests compare the worker's bundles for providers returning full and deferred matches across 14 query shapes, and check that only the page is read. The provider tests run on a real ValueSetDatabase file with no network and cover version choice, normalisation, ids, rejected rows, search, counts, the hash comparison in sync, the sync's "already known" test, the scan, and the startup sync of a new database. The cache tests check that repeated fetches parse once, that a row the sync writes is dropped and the next fetch sees the new content, that a read under way when a row is dropped isn't kept, that the size limit drops the least recently used, that search doesn't fill the cache, that a full sync, an operator resync, a new database and close() empty it, that cached value sets can be frozen without anything failing, and that the version rules (exact, then major.minor, then latest) give the same answers whether or not the value set is cached. vs-vsac-resync, vs-vsac-next-link and vs-package pass unchanged.
• Provider level: I compared the old provider against the new one on the same database. Every url (with no version, with a semver version, and with a version that doesn't exist), every url|version and every id gave the same result in 175,331 of 175,332 fetches. The one difference is the raw-id case above. 24,311 searches (every ?url=, every 7th ?url=&version=, and name/title/publisher/status/system/identifier/description/date/version searches, each with and without _elements) were identical, including order.
• Search: 14 broad and paged searches gave identical bundles before and after, as did all 4,933 IG ValueSet searches, run as 2 passes × 2 run pairs. The broad set includes status=active with _count/_offset, _sort=url, _summary=count, _summary=true, _elements, and title/name/publisher/?url=.
• Real traffic: 13,718 distinct IG Publisher calls replayed against before and after gave identical responses, ignoring timestamps and diagnostics. That set is every call that names a VSAC value set, every other ValueSet search, and samples of $validate-code, $expand and $batch-validate-code. The VSAC entries of a separate expansion catalog (30 requests) also gave identical responses.
• Sync: a forced sync against a mock VSAC server, built from the same snapshot with 10 value sets changed, left identical databases and the same events.

A follow-up could do the same for npm-package value sets (PackageValueSetProvider), which hold about 230 MB of heap, 196 MB of it PHIN VADS. That needs its own version rule, THO precedence, and a decision about the isCached filter memoisation, so it isn't in this PR.

🤖 Generated with Claude Code

[grahamegrieve]

so I chose to preload the vsac value sets because of the parsing time. And it does take time, but the numbers here are mostly persuasive that this is not a good trade-off.

However there's one case in which it is a very good trade-off, which is repeated $validate-code hammering on a value set. I'd like to keep a small MRU cache of loaded VSAC value sets for this, particularly for a batch-validate-code where the vsac code is an import etc.

[jmandel]

Thanks, you were right that repeated use needed it. It's done (e62c9f2): the VSAC provider keeps the value sets it has read, parsed, up to 64 MB of their stored JSON (valueSetCacheMB; 0 turns it off). The least recently used is dropped first.

Measured on main / on demand without the cache / with the cache:

	main	no cache	cache
$validate-code, Medication Brand Name (33k codes)	3.7 ms	18.3 ms	4.1 ms
the same, through a value set that includes it	4.5 ms	18.4 ms	4.0 ms
$batch-validate-code, 1,000 entries, a 9,500-code set	1.15 s	5.28 s	1.08 s
round robin over 20 large sets	2.0 ms	5.4 ms	2.1 ms

How it stays correct:

• The version a url gets is still worked out from the database rows first, so the version rules are unchanged; the cache is keyed only by row id.
• Whenever the sync writes a row (it already compares hashes), that row is dropped. An operator resync drops that url, and a full sync empties the cache. A read under way when something is dropped isn't cached.
• Searches read through the cache but don't fill it, so they can't push out the sets in use.
• Cached value sets are shared between requests, as the map's were. To check nothing writes to them, I ran the server with them all frozen: $expand, read, search and $validate-code, including the HTML views, gave the same results with no errors.

[grahamegrieve]

should the size by configurable?

[jmandel]

Yes, it is: valueSetCacheMB in the VSAC config, next to refreshIntervalHours (the vsacCfg block in config.json). It caps the stored JSON of the cached value sets, default 64 MB, and 0 turns the cache off.

For scale: Medication Brand Name, one of the largest, is about 3.6 MB in the heap once cached. 64 MB covered the 20-largest-set round robin in the measurements. Only about 175 of the 18,176 VSAC urls are over 100 KB, so the default holds every large set in use plus plenty of small ones.