[jmandel]

Cluster mode: several FHIRsmith processes on one port

One FHIRsmith process runs all its JavaScript on one main thread: request handling, expansion and validation, SNOMED CT (held in memory), and building responses. Some work runs on other threads: queries made through the async sqlite3 package (LOINC, RxNorm, NDC and others) run on libuv's thread pool, and V8 does part of its garbage collection on helper threads. Replaying recorded IG Publisher traffic against main on a 12-core desktop, one process at saturation used about 1.4 cores. That was about 0.9 of a core on the main thread, about 0.4 across the four pool threads (each about 10% busy), and about 0.1 in V8 helpers. Throughput stopped rising at about 230 requests/s once the main thread was about 90% busy, and an 8-thread pool made no difference. So a process can use more than one core, but its throughput is capped by the main thread; to use more cores, run more processes. This PR adds node cluster.js (npm run start:cluster): several complete server processes behind one port, with Node handing each connection to any of them. Everything that has to agree across the processes is kept in SQLite files in the data directory. Run as today (node server.js), a single process behaves exactly as before.

What changes when there are several processes

• $cache-control caches are kept in resource-cache.db, so a cache-id issued by one process works on every other, and caches survive a restart. A request reads only the resources it uses.
• Logins, generated session secrets and rate-limit counts (publisher, IPS, testing, token, registry, and all 13 rate limiters) are kept in cluster-state.db. A login made through one process holds on the others, and a limit of 20 attempts means 20 for the service.
• Background jobs (VSAC sync, package crawler, XIG download, registry crawl, the daily report) run in one process at a time, chosen by a lock file. The other processes pick up what a job wrote when a request next uses it. VSAC uses a generation counter in its database; the others use the file the job replaces.
• $closure locks its table across processes.
• The memory guard measures the whole service against one limit, refuses together, aborts only in the largest process, and restarts a process that stays too big. Its in-flight requests finish first.
• SHL and IPS FHIR validators: the first process keeps the configured port, and the others take free ports.

What stays per process: statistics and the dashboard, the expansion cache, and the npm projector's manual /_reload. docs/cluster.md covers the setup, the memory model, and Linux against other platforms.

Measured

• Same answers. Replaying 35 recorded IG Publisher builds (11,920 calls, with their $cache-control sessions) against 4 processes: 0 cache-id-unknown, and all 11,810 compared calls give the same answer as one process. The 110 metadata calls weren't compared. The calls were spread evenly, 2,920–3,024 per process.
• Faster. On one development machine, an earlier run of the same replay took 67.0 s with one process, 47.7 s with 2 and 44.4 s with 4.
• Memory under pressure. 3 processes under a 2 GB systemd limit, with one process driven to 96% of the service's memory:
  • all three refused new requests;
  • a heavy operation was aborted only in that process;
  • the primary drained and restarted it 15 s later;
  • a 10-second request already running on it was answered before it exited;
  • the service then fell back to 503 MB, with no OOM kills.
• HL7 tx-ecosystem suite (validator 7.0.3): the same 13 failures as main (cs-batch-validate*, translate-reverse-r5+*).

Decisions for you

Each of these is a choice this branch makes that you could accept or change. For each: what we chose, the main alternative, and why.

1. Shared state lives in SQLite files in the data directory

Chosen: everything the worker processes must agree on is in SQLite files next to the other data:

• resource-cache.db holds the $cache-control caches.
• cluster-state.db holds logins, generated session secrets, rate-limit counts, and the locks that cover work which waits on something.
• background-jobs.lock says which worker runs the timed jobs.

Each file is opened the same way: WAL mode, a 5-second busy timeout, and every change in one BEGIN IMMEDIATE transaction.

Alternative: a separate store such as Redis, or keeping the state in the primary process and having workers ask it over IPC.

Why: FHIRsmith already depends on SQLite and already keeps its data directory on local disk. A shared file adds no new service to install, survives restarts, and holds up if a worker crashes. Asking the primary would put a round trip on every request and lose everything when the primary restarts. The cost is that these files must be on a local disk, never NFS or SMB; cluster.md says so.

2. In cluster mode the memory resource cache becomes the SQLite one

Chosen: if a configuration sets resourceCache.store to "memory", or leaves it unset, and runs with more than one worker, the workers use the SQLite store. The log says so. One process keeps whatever is configured, so the default there is still memory.

Alternative: refuse to start, which is what the branch did before. Or require operators to set "sqlite" themselves.

Why: with the memory store, a cache-id only works on the worker that issued it, so cluster mode cannot work with it at all. Refusing a config that has only one sensible meaning just makes operators edit a file. Switching, and logging it, gets the same result with less friction.

3. The memory guard works on the whole service in cluster mode

Chosen: with more than one worker, the limit and the measure cover all the workers together.

• The limit: server.memory.limit if set, now meaning the whole service. Otherwise the cgroup limit. Otherwise the machine's RAM.
• The measure on Linux: used when the service has its own cgroup with a limit (MemoryMax on the unit or scope, or a container limit). It is that cgroup's private memory: memory.stat anon, or total_rss on cgroup v1. The page cache is left out because the kernel can take it back.
• The measure elsewhere: used on macOS, on Windows, and on Linux without a limit of the service's own. Every worker sends its RSS to the primary once a second. The primary adds them up and sends each worker the total and the per-worker table.
• Refusing: above 85% of the limit, every worker refuses new requests with 503.
• Aborting: above 92%, operations are aborted with 422 only in the largest worker, or in a worker holding more than 1.5 times its fair share of the abort threshold. The others carry on.
• Recycling: if the service stays above 85% for recycleAfterSeconds (default 60), the primary drains the largest worker and restarts it. Draining means it takes no new connections, finishes the requests it has for up to drainSeconds (default 30), then exits. Only one worker is recycled at a time, never while another is starting. Each further recycle in the same spell waits twice as long as the last, up to 15 minutes.
• One process: behaves exactly as before.

Alternatives:

• Keep a per-worker limit, as the earlier text in cluster.md told operators to set.
• Abort in every worker when the service is over.
• Never recycle, and leave restarts to the kernel's OOM killer or systemd.

Why:

• The workers share one cgroup, so the kernel enforces one limit on all of them together. A per-worker share either wastes memory or does nothing when one worker is heavy and the rest are light.
• Aborting everywhere means one heavy request cancels everyone's work. The largest worker most likely holds the operation that pushed the service over, and aborting there frees the most.
• The 1.5-times-share clause lets two heavy workers both abort in the same round.
• Recycling covers the case aborting cannot fix: a worker that has grown and stays big. The backoff stops a loop of restarts when the workers are simply too big for the limit.

The thresholds are the existing ones. The 1.5 factor, the 60-second wait and the 30-second drain are judgement calls.

4. Workers catch up with VSAC syncs by a generation counter

Chosen:

• The VSAC database keeps a generation number. The worker that syncs adds one to it in the same transaction that records the end of the sync. A failed run and an operator resync of one value set also add one.
• Every worker remembers the generation its value set list was loaded at.
• When a request uses VSAC value sets, the worker reads the generation, at most once a second, with one read by primary key. It reloads only if the number changed.
• The reload runs in the background. The request that noticed the change uses the old list.
• XIG, the registry crawl and the package crawler status follow the same pattern. Each job replaces a whole file, so the file's identity (inode, size, modification time) serves as the version.

Alternative: reload on a timer. That is what the branch did before, at the VSAC refresh interval, 24 hours by default. Or watch files for changes.

Why: a timer either serves a stale list for up to a whole interval or reloads for nothing. A counter that changes only when the data changes costs one indexed read a second while nothing happens, and shows a sync on every worker about a second after it finishes.

5. New wording for an unknown cache-id with the SQLite store

Chosen: a new message, CACHE_ID_UNKNOWN_SHARED, used when the store is SQLite: "The cache '{0}' was never issued by this server for this endpoint … This server keeps its caches on disk, where they survive a restart". It drops the memory store's "or this server has restarted since it was issued". The memory store keeps CACHE_ID_UNKNOWN unchanged. Only the English messages file has the CACHE_ID_* keys, so the new key is English only, as the others are.

Alternative: keep one message for both stores.

Why: caches in the SQLite store survive restarts. The old wording would send someone looking for a restart that cannot be the cause.

6. Each worker runs its own FHIR validator (SHL, IPS)

Chosen:

• In cluster mode each worker starts its own validator.
• The first worker uses the configured port. The other workers take free ports from the operating system.
• Each worker writes its own tx log, named <name>-<worker>.log.
• The validator jar is downloaded by one process at a time, under a lock, because the download writes the jar in place.
• A single process keeps the configured port.

Alternative: one validator for the whole service, either owned by one worker or started by the primary, with the other workers calling it.

Why: the validator wrapper assumes the process that started the validator owns it. Sharing one would mean a validator that disappears whenever its owning worker restarts, or teaching the primary about validators. The cost is N Java processes, which count against the memory limit; cluster.md says so.

7. One process can opt in to the shared store: server.sharedState

Chosen: off by default, so one process behaves as main does: sessions in memory, and a random session secret for a module that has none configured. With "server": {"sharedState": true}, one process keeps sessions, generated secrets and rate-limit counts in cluster-state.db, so logins survive a restart. Cluster mode, and the background-jobs lock, turn it on regardless of the setting.

Alternative: always use the shared store, even for one process. Or have no way to opt in.

Why: it leaves the default single-server behaviour exactly as it is today, while giving a one-process deployment that wants logins to survive restarts a single switch.

8. $closure tables are locked across workers with a lease

Chosen:

• Within a worker, calls on a table still queue as they do today.
• The call at the head of a worker's queue also takes a lease on that table in cluster-state.db, within the same lockTimeout. The holder renews the lease while it works. If the holder dies, the lease lapses after 30 seconds.
• maxQueue counts the calls waiting in each worker.
• maxTables is now checked inside the transaction that creates the table.

Alternative: optimistic concurrency, where a call computes its result, then writes only if the table's version is unchanged, and retries otherwise.

Why: the lease keeps the existing "one call per table at a time" behaviour and its error responses (429 when busy). An optimistic retry would repeat the expensive closure computation under contention and change what clients see.

🤖 Generated with Claude Code

[grahamegrieve]

This is the kind of thing you should talk to me about before doing all this work.

It shouldn't apply to some of the modules - it's dumb to try to apply it to all the existing ones but the terminology server module. I think.

And have you looked at the nginx documentation?

But what's the performance hit for moving shared state out of memory and into a database? it can't be small?

[grahamegrieve]

I'm also concerned about the state and size of some of these databases. Particularly caching on behalf of the client - what's to stop it growing and growing?

How can this not make any changes in the singleton case? That's what is says, but it's evidently not true

[jmandel]

Thanks, fair points all round. Closing this in favour of #342, which follows the model in nginx.md:

• One master runs every module, all background jobs (VSAC sync, crawlers, XIG) and $closure.
• N tx-only workers each put their instance code on the cache-ids they issue, and requests are routed to the owner by that prefix.
• Routing is either nginx (a new section in nginx.md) or an optional small launcher, cluster.js.
• Scope: only the terminology module is spread across processes. The other modules run on the master alone, so there are no shared logins, rate-limit stores or validator ports, and no shared database by default.
• Single process unchanged: with no --role, a single process runs exactly the same code paths as main, and the PR shows how that's guaranteed.
• About a quarter of the size of this one.

On your questions:

• Shared state in a database, and cache growth. A shared session cache and a shared $closure are now opt-in options in A master and terminology workers on one machine (replaces #336) #342, both off by default. We're measuring their cost against per-process caches on an e2-highmem-4 (production's machine type) and will post the numbers there. The shared cache has a size cap and the idle sweep.
• nginx. Yes. Two things turned up in testing that nginx.md should mention:
  • Node's 5 s keep-alive timeout races nginx's 60 s upstream keepalive. The reset marks the owner down, and cached requests then go to the backup; Small fixes: 422s instead of 500s, a request that never got an answer, LOINC data, and two robustness fixes #337 has the fix.
  • Older publishers send the cache-id in the body, which nginx can't route on.