[jmandel]

What it does

This follows the model in nginx.md. One master runs everything. N tx-workers run only
the terminology module and spread the tx load. Each process puts its instance code on the
cache-ids it issues, and a front door sends each request to the right process.

node server.js --role=master    --instance-code=m0  --port=3001
node server.js --role=tx-worker --instance-code=tx1 --port=3002
node server.js --role=tx-worker --instance-code=tx2 --port=3003
# or, for all of it behind one port:
node cluster.js [--workers=N]

• The master runs every configured module, every background job (VSAC sync, package
  crawler, XIG download, registry crawl, daily report, npm audit, timers), $closure and the
  syndication feed. It loads the terminologies, since closure needs subsumption. By default it
  also serves tx requests (server.cluster.masterServesTx).

• A tx-worker runs the tx module only. It has no background jobs, no statistics database,
  and no closure or syndication store. Its $cache-control caches stay in its own memory, in
  the existing memory store. It picks up the master's VSAC syncs through a generation counter,
  checked when a request uses the value sets (at most once a second). It reads nothing else the
  master writes: XIG, the registry, packages and the rest aren't loaded in a worker. The only
  tx-to-xig import is a pure string-escaping function.

• Routing, the same in cluster.js and in the new nginx example:

  1. X-Cache-Id goes to the process named by the prefix. If that process is down, or the
     prefix is unknown, the request goes to any tx process, which answers cache-id-unknown.
  2. $closure in any endpoint goes to the master, and so do /syndication… and /info/….
     The VSAC resync form on /info/vsac writes the VSAC database.
  3. Every non-tx path goes to the master.
  4. Any other tx request goes to the ready tx process with the fewest outstanding requests.

• cluster.js is an optional launcher. It starts the master first and waits for /health,
  so the workers never build the package caches or the VSAC database at the same time as it
  (Cluster mode: several FHIRsmith processes on one port #336's register F2). It then starts the workers and proxies on one port with node http and
  keep-alive agents. It appends the client address to X-Forwarded-For. It restarts a process
  that dies, after 1 s, 2 s, 4 s and so on, up to a minute. On SIGTERM it drains: it stops
  accepting connections, closes idle keep-alive connections, waits for the requests in
  progress, then sends SIGTERM to each process.

• Memory. With a role, the memory guard measures the cgroup's private memory
  (memory.stat anon), since the kernel's limit applies to all the processes together. Every
  process refuses new requests above 85%. Above 92% only the process with the largest RSS in
  the cgroup aborts its operations.

• Local file:/folder: sources. Each library load wipes and rebuilds its cache folder.
  With a role, every process uses the master's folder (folder-source-master-<hash>): the
  master builds it in a temporary sibling and renames it into place; a worker reads it
  read-only (waiting for it like a package cache) and never prunes.

• Two options, both off by default (server.cluster, read only with a role; see
  cluster.md "Options"):

  • sharedCache: true: the $cache-control caches go in one SQLite file
    (databases/resource-cache.db) that every process shares, so any process answers any
    cache-id and caches survive restarts. The launcher then spreads cached requests by load;
    cacheAffinity: true keeps the prefix routing. sharedCacheMaxMB caps the file. The
    store is Cluster mode: several FHIRsmith processes on one port #336's, cut to what tx needs, and pre-fills each resource's expansion-key hash
    from its stored row (P6, cherry-picked from Small fixes: 422s instead of 500s, a request that never got an answer, LOINC data, and two robustness fixes #337).
  • sharedClosure: true: every tx process answers $closure on databases/closure.db, and
    the launcher sends $closure to the least-loaded tx process. Correctness across processes
    comes from optimistic, version-checked writes (no lease): an add is worked out against
    table version n and written in one BEGIN IMMEDIATE transaction that first checks the
    table is still at n; if not, it is worked out again from the fresh table (bounded
    retries, jitter, then the existing 429 CLOSURE_TABLE_BUSY). Creating a table and the
    table limit are checked in the insert's transaction.

Docs: cluster.md replaces #336's. nginx.md gains "A master and tx workers on one machine".

What changes for a single process: nothing

node server.js with no --role and no FHIRSMITH_ROLE behaves exactly as before.

How that's guaranteed. parseRole returns null when no role is given. The only changes
to server.js are if (role) applyRole(config, role), if (role) log… and
const PORT = (role && role.port) || process.env.PORT || …. So with no role the configuration
object is the one read from disk, untouched. Every new behaviour in a shared file is behind a
value that only applyRole sets:

File	Guard	Unset without a role
server.js	role	null
tx/tx.js (closure store, syndication store, local cache tag)	config.clusterRole	not in config
tx/library.js (cache folder name, build-and-rename, pruning)	this.localSourceTag, this.packageCachesReadOnly	null, false
tx/vs/vs-vsac.js (follower start-up, catch-up, generation bumps, resync form)	this.cluster (from vsacCfg.cluster)	null
tx/vs/vs-database.js (WAL, busy timeout)	this.shared	false
library/memory-guard.js (group measure, largest-aborts, GC share)	state.group (from server.memory.group)	null
tx/tx.js (shared resource cache)	config.sharedCache (set by applyRole only when server.cluster.sharedCache is true)	not in config
tx/tx.js (closure store in workers, shared mode)	config.sharedClosure (likewise)	not in config
tx/closure/closure-store.js (busy timeout, IMMEDIATE schema, checked create and add, bounded resync, one-statement sweep)	this.shared (from the shared option)	false
tx/workers/closure.js (checked create, retries, bounded resync)	this.store.shared	false
cluster.js (cache-id and $closure routing)	settings.routing	{cacheAffinity: true, sharedClosure: false}: the old rules

Three changes run without a role, and are equivalent to what they replace:
ClosureWorker.add is split into workOutAdd and addResult around the same recordAdd
call (same reads, writes and errors, in the same order); loadSupplements reads
cs.supplementsUrl, a new getter returning cs.jsonObj.supplements; and P6 (cherry-picked
from #337 as its own commit) remembers each wrapper's expansion-key hash, giving
byte-identical keys.

The other additions are new functions that nothing calls without a role: getGeneration,
bumpGeneration, catchUp, largestInGroup, ownCgroupDir, readCgroupAnon and
readCgroupProcessRss.

Checked three ways:

1. By diff. Every hunk in the files above, comments left out, is one of the guarded lines
   listed (git diff -U0 origin/main -- server.js tx/ library/memory-guard.js).
2. By running it. node server.js was run with no flags on main (9059fb9) and on this
   branch. After the options were added, the same was done at 9ee5293 (before them) and
   6c66ebc (with them; later commits are tests and docs), with a config of tx with closure
   (maxTables 3) and vcl. 29 requests: /health, metadata, a $cache-control session
   used by $expand (twice), $validate-code and $lookup, then checked, ended and used
   again, unknown cache-ids with another instance's prefix and with its own, and a $closure
   table through init, re-init (409), adds (one unknown code), resyncs, reset with an add,
   dispose, use after dispose (404) and the table limit (422). The responses were identical
   (UUIDs and timestamps masked), the data directories held the same files, and the logs
   differed only in paths, PIDs, UUIDs and stack-trace line numbers (one frame is now
   workOutAdd). The first run, before the options, was with the same config (tx with closure, and vcl) on a fresh data directory. /health,
   /r4/metadata, $cache-control?mode=start, $closure and $lookup gave identical
   responses (timestamps and UUIDs masked). The server logs were identical apart from the
   checkout path. The data directory held the same files.
3. By the HL7 tx-ecosystem suite, run on a plain single process (m331/run_eco.sh). At
   8b31d29 (port 20040) and again at 6c66ebc, with both options in (port 20090,
   runs/cluster-lite-options), it failed 13 of 5061 tests, and they are the same 13 tests
   that fail on main (runs/fixes1103-main, 9059fb9): cs-batch-validate* and
   translate-reverse-r5+*.

tests/library/role.test.js also checks that a config passed through server.js's
parse-and-maybe-apply step, with no role, serialises byte for byte as it was.

Decisions for you

Decision	Chosen	Alternative	Why
Does the master serve tx requests?	Yes by default; server.cluster.masterServesTx: false (or --master-serves-tx=false) turns it off. In nginx, leave it out of the pool.	Master never serves tx	The master has the terminologies loaded anyway, for closure, so one more tx process costs nothing. Turn it off if the admin work (crawls, VSAC sync) should never slow tx answers.
Where $cache-control caches live	By default in each process's memory (the existing store), with routing by cache-id prefix. server.cluster.sharedCache: true puts them in a shared SQLite file instead, with least-load routing (cacheAffinity: true keeps the prefix routing)	Shared store always, as in #336; or no shared store at all	The default adds no store and no new failure modes, and is what nginx.md already describes; its cost is that a restarted process loses its caches and its clients start new ones. The option removes that cost and lets the load spread freely, for a file read per use and the store's failure modes (a disk file, write retries, a 503 when a write fails twice). Pending measurements: the benchmark agent on GCP is comparing the two; if the shared store wins on uneven load, its default could flip.
Front door	Both: an nginx example in nginx.md, and the optional cluster.js	nginx only	The public servers already run nginx, and the example is all they need. cluster.js (about 300 lines of code) gives a one-command setup for those without nginx, plus the start ordering, restart and drain that a set of nginx upstreams doesn't do. Dropping it would leave operators to order the start-up themselves (systemd units, a /health wait).
Memory guard with a role	cgroup anon for the whole group; every process refuses; only the largest aborts	#336's full guard: RSS reports over IPC, a "processes" measure for macOS and Windows, drain-and-recycle of the largest worker	The minimum that is correct for several processes under one limit. Without a cgroup of its own, each process falls back to guarding its own RSS. If the kernel kills a process anyway, the launcher restarts it.
Master-only tx paths	$closure (unless sharedClosure), /syndication…, /info/…	$closure only	The syndication store is written as the library loads, and the VSAC resync form writes the database. A worker opens neither store, and its resync form is off.
Role configuration	Flags and environment (--role, --instance-code, --port, FHIRSMITH_ROLE, FHIRSMITH_INSTANCE_CODE, PORT). The master's code can also come from modules.tx.instanceCode.	A role in config.json	All the processes share one data directory, and so one config.json. A role in it would apply to every process.
$closure correctness across processes (sharedClosure)	Optimistic: compute against table version n, write in one BEGIN IMMEDIATE transaction that checks the table is still n (same id), else recompute from the fresh table; at most closure.writeRetries (10) retries, 0-50 ms jitter, then 429 CLOSURE_TABLE_BUSY. Code lookups and subsumption tests are remembered across retries. The per-process queue stays	A lease per table (#336's f4b19f5)	No lease to expire, renew or lose to a crash; a stalled process can't hold a table. The cost is repeated work under contention: closuregen's shared scenario (8 writers on one table, 3 processes) gave 172 retries in 277 adds, almost all a single retry, and no 429.
Reset with concepts, shared	The reset and the add are two atomic writes; another process's add can land between them, and the add then retries on top of it	One transaction for both	Every outcome is one some order of the calls gives; keeping them apart needs no second code path
P6 in this branch	Cherry-picked from #337 (d3aa24e) as its own commit, ahead of the shared store	Port #336's cacheContentHash change to computeKey	P6 is already proposed and makes keys byte-identical; the store only calls setExpansionKeyHash(row.hash) and answers expansionKeyHash() from the row
Unknown cache-id with the shared store	CACHE_ID_UNKNOWN_SHARED (#336's 1b1c632 wording) whatever the id's prefix	Keep CACHE_ID_OTHER_INSTANCE for another member's prefix	With every process seeing every cache, "routed to the wrong instance" can't be the cause
Where the options are set	server.cluster in config.json only	Flags or environment too	Every process must agree, and they all read one config.json; a flag on the launcher would have to be passed on to each process

What was kept from #336, and what was dropped

Kept, cut down:

• the VSAC generation counter (cf38dde), for the master/worker split only. The master bumps
  the counter; workers check it on use, at most once a second. Both use WAL and a 5 s busy
  timeout.
• the cgroup anon measure and the largest-process-aborts rule (9466ddb). The "largest"
  comes from cgroup.procs and /proc/<pid>/status, not from IPC.
• the group-sample throttle: the in-operation check reuses a group measure up to 100 ms old.
  This plays the part of ce7d0ae0 and ecee02a. The single-process check is unchanged.
• start the master first, then the workers (F2).
• as the sharedCache option: the SQLite resource cache (4656acae, dea114c4, 80d2e834,
  76f37f51, 1b1c632), with WAL, busy timeout, BEGIN IMMEDIATE, the idle sweep, the
  maxBytes cap, the throttled last_used and lazy loading. Not its memory-store cap,
  resourceCache.store config, FHIRSMITH_CLUSTER_SIZE switch or Retry-After middleware.

Dropped: shared logins and sessions (cluster-state.db), the rate-limit store, validator
port juggling, leases (and the closure lease), the background-jobs lock, crawler, XIG and
registry catch-up, the "processes" memory measure and its IPC, and
memory-driven drain and recycle. None of them is needed when the non-tx modules run only on the
master.

Size

	Code	Tests	Docs
This branch, roles only (at 9ee5293)	+789 / -18 (582 lines without comments or blanks; cluster.js is 299 of them)	+815	+234
P6 (cherry-picked from #337)	+63 / -10	+369
The two options	+1134 / -29 (819 without comments or blanks; the SQLite store is 613 of them)	+1503 / -22	+123 / -6
#336	+3028 / -146	+2309 / -16	+305

Code files: cluster.js (new), library/role.js (new), library/cgroup-memory.js,
library/memory-guard.js, server.js, tx/tx.js, tx/library.js, tx/vs/vs-vsac.js and
tx/vs/vs-database.js. #336 changed 56 files; this branch changes 19.

Tests

• tests/library/role.test.js: flags, environment and errors; no role means the config is
  untouched; what each role turns off.

• tests/server/cluster-launcher.test.js: the routing rules (cache-id prefix, $closure in
  each FHIR version and encoded, syndication and info, non-tx paths), the least-loaded choice,
  settings, restart after a crash, and the drain on stop. The drain test checks that no new
  connections are taken, that the request in progress gets 200 with Connection: close, and
  that the processes stop only afterwards. All of these run against fake upstreams.

• tests/server/cluster-integration.test.js runs real processes: node cluster.js with a
  master and 2 workers on a temporary data directory, with a fake VSAC.

  • 6 $cache-control sessions are started through the launcher and used by 48 later
    requests, with no cache-id-unknown. With cache-id routing switched off, the same test
    fails.
  • $closure and the non-tx paths reach the master.
  • A VSAC sync on the master is seen by both workers.
  • On SIGTERM the requests in progress finish, every port closes and the launcher exits 0.

• tests/tx/vsac-generation.test.js runs two real processes on one VSAC database. The worker
  sees a sync within about a second and reloads once. With nothing changing, many uses cost at
  most two generation reads. A resync reaches the worker. A worker started before the master
  doesn't create the database.

• tests/library/memory-guard-group.test.js: the group measure refuses requests while this
  process is small; only the largest process aborts; the fallback without a cgroup; and the
  describe() text is unchanged without a role.

• The options:

  • tests/tx/resource-cache.test.js runs against both stores; plus the shared store's own
    cases (lazy loading, touch throttle, write retry then 503, the cap across endpoints,
    restart, schema check) and P6's pre-fill: the same expansion key as the memory store's
    resources, hashing only the key and a plain resource, whether the resources are loaded
    or not.
  • tests/tx/resource-cache-multiprocess.test.js: two forked processes on one file (use,
    end, idle sweep from either, restart, 100 concurrent writes, the size cap across
    processes).
  • tests/server/cluster-shared-cache.test.js: cluster.js with sharedCache. A session
    started on each of the three processes is used on all three (18 of 18, no
    cache-id-unknown); 60 concurrent cached requests through the launcher; an unknown id
    in the shared wording; a worker SIGKILLed and restarted still has every session. With
    sharedCache: false the same test fails (12 cache-id-unknown).
  • tests/tx/closure-shared.test.js: a second store on the same file writes between the
    module's read and its write. Forced conflicts retry and end as a single writer would;
    exhaustion gives 429 with nothing written; a table dropped meanwhile is 404; bounded
    resync; create and the limit checked in the write.
  • tests/server/cluster-shared-closure.test.js: 3 processes x 4 clients adding at once to
    one table and to four shared-in-threes tables, checked against a single-writer reference
    (concepts, entries, no loss or duplicates), versions 1..n with no gaps, each write's
    answer equal to what is stored at its version, and "changes since v" from every process
    for every v. 12 concurrent creates of one name make one; creates racing for the limit
    stop at it. With the version check removed, it fails.
  • closuregen (closure-model/closuregen.py) against a launcher port (20070, master and 2
    workers, both options on, SNOMED CT and LOINC): shared for 60 s (277 adds, 99
    resyncs) and mixed for 60 s (1041 adds, 23 tables); every request 200, and its oracle
    check passed on every table.

All the touched suites were run under systemd-run --user --scope -p MemoryMax=6G: role, memory-guard, memory-guard-group, cgroup-memory, vs-vsac, vs-vsac-next-link, vsac-generation, library, library-local-source-cache, closure, closure-enabled, syndication-routes, syndication-store, cache-control, cache-id, stats, cluster-launcher and cluster-integration. All 18 suites passed: 232 tests. With the options: those 18 plus closure-shared,
cache-id-params, cluster-shared-cache, cluster-shared-closure, resource-cache,
resource-cache-multiprocess, expansion-key-hash, expansion-cache, supplement-filters, worker
and provider-tho-precedence, at f81442a: all 29 suites passed: 469 tests (4 skipped:
the size-cap cases, for the memory store, which has no cap).

🤖 Generated with Claude Code

[jmandel]

Measured on GCP e2-highmem-4 (4 vCPU = 2 cores, 32 GB, production's machine type), with a separate load generator. The workload replays 35 recorded IG Publisher builds (11,920 calls with their $cache-control sessions) at C concurrent builds, one run per point.

What the numbers say:

• Use the defaults: per-process session caches, $closure on the master, 2 tx workers, SNOMED CT mmap on.

  • The default was fastest at the median in every comparison.
  • Two workers already fill 2 cores: mid-pass throughput is 223 / 263 / 247 / 228 req/s at N = 1 / 2 / 3 / 4, measured at C = 8 with light closure traffic.

• Against one main process, at C = 8 on the same machine:

  • one main process: 87 req/s mid-pass, 80 req/s over the whole pass;
  • this PR on main plus our open fixes, with mmap: 263 and 154 req/s.

  The whole-pass figure includes a tail where only a few long builds are still running.

• Same answers: 280,399 IG answers were identical to single-process main, with 0 cache-id-unknown.

• The front door costs about nothing. With the same main processes behind it, nginx and cluster.js give the same throughput up to C = 16.

• Memory needs mmap. Without it, each main process holds its own SNOMED CT: about 6.3 GB, so 4 processes need about 25 GB. With mmap, each holds about 2 GB private and N = 4 fits in 9 GB.

• Shared session cache (sharedCache, off by default):

  • Cost: each lookup takes 0.7 ms from SQLite against 0.015 ms in memory. Requests are +1.4 to +2.6 ms at the median, throughput is 4–14% lower, and memory is a few hundred MB higher.
  • Gain: caches survive a worker restart, and routing doesn't need the header.
  • Size: the cache's live content peaked at about 12 MB over the whole replay. Builds end their caches, so the file stayed at 10–20 MB.

• Shared closure (sharedClosure, off by default): closure itself gets faster (mixed-load 99th percentile 9.7 s → 2.8 s). But the work moves onto the processes serving IG traffic: under a nightly bulk rebuild, IG throughput halves (113 against 222 req/s). Closure on the master keeps IG traffic isolated.

• The memory guard's /proc reads were 6–7% of the main thread; they're now sampled (merged via SNOMED CT: optional memory-mapped cache files, chosen per edition in config.json #330).

Main-based setups (before our fixes, no mmap). Left: throughput against concurrency. Right: memory.

This PR, with the four cache/closure combinations. Throughput against the number of workers:

Memory against workers, mmap on:

IG traffic under closure stress (N = 2):

Found along the way:

• On main: CodeSystem/$validate-code never answers an unknown cache-id, whether it comes in the header or the body. Fixed by Small fixes: 422s instead of 500s, a request that never got an answer, LOINC data, and two robustness fixes #337 (G1).
• nginx keep-alive race. nginx's 60 s upstream keepalive against Node's 5 s timeout causes resets, which mark the cache's owner down, so cached requests go to the backup. Fixed by Small fixes: 422s instead of 500s, a request that never got an answer, LOINC data, and two robustness fixes #337; nginx.md now says so.
• In this branch: a worker crash-loop on a fresh package cache, and a wait for per-worker file: source caches. Both are fixed here, and a fresh-cache start now has 0 restarts.

[grahamegrieve]

Thanks - a big piece of work, and I'm happy with the defaults (master serves tx, per-process
caches with prefix routing, sharedCache and sharedClosure off). Before merging:

1. Rebase onto main: it conflicts in library/memory-guard.js (SNOMED CT: optional memory-mapped cache files, chosen per edition in config.json #330), tx/tx.js and
   tx/vs/vs-vsac.js. On VSAC, VSAC: read value sets from the database when asked for, instead of holding them all in memory #333 replaced the in-memory map with on-demand reads and a
   recently-used cache, so a follower seeing the generation move should clear that cache, not
   reload a map. Every worker holding the whole map would also have multiplied the memory.

2. The graphs on the home page and the dashboard should include the workers, not just the
   master - and so do the request counts that go in the stats database and the daily report.
   A small shared SQLite file in the data folder is a possible implementation: each worker writes its per-minute sample and request counts, and the master
   merges them (requests summed; memory and event-loop delay per process). Alternatively, the master could just get the stats from the workers directly

3. Caches should not be resurrected after a server restart. A client can't be sure the cache
   is intact across operations it tried while the server was down, and the server can't either.
   With sharedCache, one worker restarting while the others carry on is fine, but when the
   master starts it should move every live cache to a tombstone ("lost to a server restart")
   instead of carrying them on; CACHE_ID_UNKNOWN_SHARED and the store's header comment need to
   change to match. It would be useful, though, to tell "never issued" from "lost to a
   restart". With the memory store, that could be done without persisting anything: put the
   process start time in the cache-id (..), so an id with this
   instance's code and an earlier start can be reported as issued before the restart.

4. tests/utils/cluster-harness.js uses ss, which is Linux-only. On macOS (and here) the
   cluster-shared-cache test fails with ENOENT; please use lsof, or skip off Linux.

5. Where commits stand alone (P6, which is also in Small fixes: 422s instead of 500s, a request that never got an answer, LOINC data, and two robustness fixes #337; the mime types temp file; the
   cluster.js idle-connection timing), separate PRs would make this easier to review.