big bug hunt #2 - various issues per Josh's report

CHANGELOG.md

@@ -24,6 +24,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - SNOMED CT importer: effectiveTimes were stored a day early when a cache was built in a time zone ahead of UTC (the day count to the 1899 epoch used local dates), so `$lookup` reported e.g. 2002-01-30 for 20020131. Caches built in such a zone need rebuilding
 - `CodeSystem/$batch-validate-code`: a top-level `url` (and `version`) is the code system for every entry, as for `CodeSystem/$validate-code`, and every entry is validated against a code system - an entry with its own `url` was validated as a value set
+- `regex` filters match the whole value, for every code system: the pattern is `^(?:pattern)$`. Code systems from CodeSystem resources, ISO 3166 and filters on supplement properties anchored it as `^pattern$`, so an alternation went wrong (`code2|code3` matched every code starting with code2); LOINC matched anywhere in the value (a CLASS regex of `CYTO` matched PANEL.HL7.CYTOGEN). LOINC regex filters may now select fewer codes
+- ValueSet `$validate-code` with `activeOnly=true`: an inactive code was only rejected when the value set includes all of its code system; when the value set lists the code, or selects it with a filter, it was valid (with an inactive warning). It is now not in the value set whatever the include, for every code system
+- Only one version of a supplement applies: when the server has one version and the request passes another in `tx-resource` (or a value set names one and the request another), the version the request asks for by version (`useSupplement url|version`, or the value set's `valueset-supplement`) is used - else the request's own, else the latest - and `used-supplement` reports that exact version. Both were applied before, and the server's version was the one reported (and whose displays won)
+- ValueSet `$validate-code`: a code found through a value set filter didn't report a status the code has while active (`deprecated`: no `status` parameter and no `DEPRECATED_CONCEPT_FOUND` warning), as it does when the value set lists the code
+- `$expand` text filters (`filter`) now search one language: the first language the request asks for (`displayLanguage`, else `Accept-Language`) that the code system has text in, else the code system's own language - the same rule as the SNOMED CT display filter. All the displays and designations in that language take part, including those from supplements in use; designations in other languages no longer match. LOINC searches its names in other languages (`filter=Volumen&displayLanguage=de` finds Glucose [Masse/Volumen] in Serum oder Plasma) and supplement designations (Uzbek); in English it still searches the displays. SNOMED CT builds a text index per language on editions with more than one (the first search in a language takes a second or two), and a national edition with no language asked for searches its default language
+- A value set filter on a property that a supplement in use defines (`catalog-group = renal-diabetes` on LOINC or SNOMED CT, with a supplement giving codes a `catalog-group`) failed (500 for LOINC and SNOMED CT, 'not understood' for code systems from CodeSystem resources). Such filters are now evaluated from the supplement, for any code system, in `$expand` (includes and excludes) and `$validate-code`, alone or with the code system's own filters (=, in, regex and exists = true can select codes on their own; not-in and exists = false only narrow other filters). A filter the code system does itself still goes to the code system
+- Supplements pinned to a version of their code system (`supplements` = `http://loinc.org|2.80`) were applied to whichever version a request without a version resolved to (LOINC 2.83), and reported as used; a value set requiring one didn't fail. Which supplements apply is now decided by the version actually used: one pinned to another version is not applied (so requiring it fails with `VALUESET_SUPPLEMENT_MISSING`), and one from the server's own content pinned to that version now applies when the request doesn't name a version
+- `$expand` returned no properties for codes from LOINC, SNOMED CT, RxNorm, UCUM, OMOP, CPT or NDC, whether asked for with `property`, `ValueSet.compose.property` or `*` (`$lookup` returned them): these code systems now report the requested properties on each code, as `$lookup` does, and only compute them when some are asked for. UCUM's `canonical` property is returned when asked for (`$lookup` never returned it). An R4 expansion declared a property with no uri with an empty uri extension. OMOP codes from a filter were JSON numbers in an expansion
 
 ## [0.18.2] - 2026-10-29
 

tests/cs/cs-cpt.test.js

@@ -207,6 +207,19 @@ describe('CPT Provider', () => {
   });
 
   describe('Extended Lookup', () => {
+    // $expand reports properties from properties(), which is built from extendLookup - but not
+    // the copyright, which is about the code system
+    test('properties() for $expand: only the named ones, never the copyright', async () => {
+      for (const [code, expectedProps] of Object.entries(testData.properties)) {
+        expect(await provider.properties(code, [])).toEqual([]);
+        const names = Object.keys(expectedProps);
+        const got = await provider.properties(code, names);
+        expect(got.length).toBeGreaterThan(0);
+        expect(got.every(p => names.includes(p.code))).toBe(true);
+        expect((await provider.properties(code, ['*'])).some(p => p.code === 'copyright')).toBe(false);
+      }
+    });
+
     test('should extend lookup with properties and designations', async () => {
       for (const code of testData.allCodes) {
         const paramSet = [];

tests/cs/cs-cs.test.js

@@ -1548,8 +1548,30 @@ describe('FHIR CodeSystem Provider', () => {
           });
 
           test('should match a search whose words all start words of one designation', async () => {
-            expect(codes(await ptProvider.searchFilter(filterContext, new SearchFilterText('myocardial acute'), true))).toEqual(['I21']);
-            expect(codes(await ptProvider.searchFilter(filterContext, new SearchFilterText('acute miocardio'), true))).toEqual([]);
+            // the English designation is searched when English is asked for (a text filter
+            // searches one language: the request's, else the code system's - Portuguese here)
+            const english = new Languages();
+            english.add(new Language('en'));
+            const enContext = await ptProvider.getPrepContext(false, english);
+            expect(codes(await ptProvider.searchFilter(enContext, new SearchFilterText('myocardial acute'), true))).toEqual(['I21']);
+            expect(codes(await ptProvider.searchFilter(enContext, new SearchFilterText('acute miocardio'), true))).toEqual([]);
+            expect(codes(await ptProvider.searchFilter(filterContext, new SearchFilterText('myocardial acute'), true))).toEqual([]);
+          });
+
+          test('should search a supplement\'s designations in the language searched', async () => {
+            const supplement = new CodeSystem({
+              resourceType: 'CodeSystem', url: 'http://example.org/icd-10-cm-pt-de', status: 'active',
+              content: 'supplement', supplements: 'http://example.org/icd-10-cm-pt',
+              concept: [{code: 'A00', designation: [{language: 'de', value: 'Cholera'}]}]
+            });
+            const withSupplement = factory.build(opContext, [supplement], ptProvider.codeSystem);
+            const german = new Languages();
+            german.add(new Language('de'));
+            const deContext = await withSupplement.getPrepContext(false, german);
+            expect(codes(await withSupplement.searchFilter(deContext, new SearchFilterText('cholera'), true))).toEqual(['A00']);
+            // with no language asked for, the code system's own (Portuguese) is searched
+            expect(codes(await withSupplement.searchFilter(filterContext, new SearchFilterText('cholera'), true))).toEqual([]);
+            expect(codes(await withSupplement.searchFilter(filterContext, new SearchFilterText('colera'), true))).toEqual(['A00']);
           });
 
           test('should not match a word that is only inside another word', async () => {
@@ -1651,10 +1673,18 @@ describe('FHIR CodeSystem Provider', () => {
           expect(results.findConceptByCode('code1')).toBeNull();
         });
 
+        test('a regex matches the whole code, alternation included', async () => {
+          const results = await simpleProvider.filter(filterContext, true, 'code', 'regex', 'code2|code3');
+          expect(results.findConceptByCode('code2')).toBeDefined();
+          expect(results.findConceptByCode('code3')).toBeDefined();
+          expect(results.findConceptByCode('code2a')).toBeNull(); // ^code2|code3$ would have matched it
+          expect(results.size()).toBe(2);
+        });
+
         test('should handle invalid regex gracefully', async () => {
           await expect(
             simpleProvider.filter(filterContext, true, 'code', 'regex', '[invalid')
-          ).rejects.toThrow('The regex \'[invalid\' is not valid: error parsing regexp: missing closing ]: `[invalid$`');
+          ).rejects.toThrow('The regex \'[invalid\' is not valid: error parsing regexp: missing closing ]: `[invalid)$`');
         });
       });
 

tests/cs/cs-loinc.test.js

@@ -1083,6 +1083,20 @@ describe('LOINC Provider', () => {
   });
 
   describe('Extended Lookup', () => {
+    // $expand reports properties from properties(), which is built from extendLookup
+    test('properties() for $expand: only the named ones, none when none are named', async () => {
+      expect(await provider.properties('100007-4', [])).toEqual([]);
+      const got = await provider.properties('100007-4', ['COMPONENT', 'SCALE_TYP']);
+      expect(got.sort((a, b) => a.code.localeCompare(b.code))).toEqual([
+        { code: 'COMPONENT', valueCode: 'LP430726-2' },
+        { code: 'SCALE_TYP', valueCode: 'LP7751-3' }
+      ]);
+      // Coding-valued (a mapping), without the equivalence subproperty
+      expect(await provider.properties('LP14355-9', ['sct'])).toEqual([
+        { code: 'sct', valueCoding: { system: 'http://snomed.info/sct', version: 'http://snomed.info/sct/900000000000207008/version/20260101', code: '15373003', display: 'Creatinine (substance)' } }
+      ]);
+    });
+
     test('should extend lookup with properties and relationships', async () => {
       const testCode = expectedResults.basic.knownCodes[0];
       const paramSet = [];

tests/cs/cs-ndc.test.js

@@ -498,6 +498,14 @@ describe('NDC Provider', () => {
   });
 
   describe('Extended Lookup', () => {
+    // $expand reports properties from properties(), which is built from extendLookup
+    test('properties() for $expand', async () => {
+      expect(await provider.properties('0002-0152', [])).toEqual([]);
+      const got = await provider.properties('0002-0152', ['code-type', 'active']);
+      expect(got).toContainEqual({ code: 'code-type', valueString: 'product' });
+      expect(got.map(p => p.code)).toContain('active');
+    });
+
     test('should extend lookup for product with properties', async () => {
       const paramSet = [];
       await provider.extendLookup('0002-0152', [], paramSet);

tests/cs/cs-omop.test.js

@@ -233,6 +233,16 @@ describe('OMOP Provider', () => {
   });
 
   describe('Extended Lookup', () => {
+    // $expand reports properties from properties(), which is built from extendLookup
+    test('properties() for $expand: only the named ones, none when none are named', async () => {
+      if (testData.sampleConcepts.length > 0) {
+        const conceptId = testData.sampleConcepts[0];
+        expect(await provider.properties(conceptId, [])).toEqual([]);
+        const got = await provider.properties(conceptId, ['domain-id', 'vocabulary-id']);
+        expect(got.map(p => p.code).sort()).toEqual(['domain-id', 'vocabulary-id']);
+      }
+    });
+
     test('should extend lookup with basic properties', async () => {
       if (testData.sampleConcepts.length > 0) {
         const conceptId = testData.sampleConcepts[0];

tests/cs/cs-rxnorm-filters.test.js

@@ -250,6 +250,15 @@ describe('RxNorm filters, totals, status and lookup', () => {
       expect(params.map(p => p.part[0].valueCode).sort()).toEqual(['STY', 'has_dose_form']);
     });
 
+    // $expand reports properties from properties(), which is built from extendLookup
+    test('properties() for $expand: only the named ones, none when none are named', async () => {
+      expect(await provider.properties('197589', [])).toEqual([]);
+      expect(await provider.properties('197589', ['has_dose_form', 'STY'])).toEqual([
+        { code: 'STY', valueCode: 'T200' },
+        { code: 'has_dose_form', valueCode: '317541' }
+      ]);
+    });
+
     test('a relationship property value finds the code again as a filter', async () => {
       // 197589 has_dose_form 317541 (as a lookup property) <=> has_dose_form = CUI:317541 (as a filter)
       expect(await codes(['has_dose_form', '=', 'CUI:317541'])).toContain('197589');

tests/cs/cs-snomed.test.js

@@ -1970,4 +1970,19 @@ describe('SNOMED CT $lookup properties', () => {
     const expr = await props('367430006:{405813007=(85562004:272741003=24028007)}', ['normalFormTerse']);
     expect(expr.normalFormTerse).toEqual(['387713003:{260686004=257903006,405813007=(118632007:{272741003=182353008})},{405813007=(85562004:{272741003=24028007})}']);
   });
+
+  // $expand reports properties from properties(), which is built from extendLookup
+  test('properties() for $expand: only the named ones, none when none are named', async () => {
+    const { context } = await provider.locate('367430006');
+    expect(await provider.properties(context, [])).toEqual([]);
+    const got = await provider.properties(context, ['moduleId', 'sufficientlyDefined', '260686004']);
+    expect(got).toEqual([
+      { code: 'moduleId', valueCode: '900000000000207008' },
+      { code: 'sufficientlyDefined', valueBoolean: true },
+      { code: '260686004', valueCode: '257903006' }
+    ]);
+    // the same values $lookup returns, without the lookup-only parts (description, code-display)
+    const lookup = await props('367430006', ['moduleId', 'sufficientlyDefined', '260686004']);
+    expect(got.map(p => p.code).sort()).toEqual(Object.keys(lookup).sort());
+  });
 });

tests/cs/cs-ucum.test.js

@@ -357,6 +357,27 @@ describe('UCUM Provider Integration Tests', () => {
       }
     });
   });
+
+  describe('Properties', () => {
+    test('$lookup returns canonical when asked for it, not by default', async () => {
+      const { context } = await provider.locate('mg/dL');
+      let params = [];
+      await provider.extendLookup(context, [], params);
+      expect(params.filter(p => p.name === 'property')).toEqual([]);
+      params = [];
+      await provider.extendLookup(context, ['canonical'], params);
+      expect(params).toEqual([{ name: 'property', part: [
+        { name: 'code', valueCode: 'canonical' }, { name: 'value', valueString: ucumService.getCanonicalUnits('mg/dL') }] }]);
+    });
+
+    // $expand reports properties from properties(), which is built from extendLookup
+    test('properties() for $expand', async () => {
+      const { context } = await provider.locate('mg/dL');
+      expect(await provider.properties(context, [])).toEqual([]);
+      expect(await provider.properties(context, ['canonical'])).toEqual([
+        { code: 'canonical', valueString: ucumService.getCanonicalUnits('mg/dL') }]);
+    });
+  });
 });
 
 // Usage example for external integration

tests/tx/supplement-filters.test.js

@@ -0,0 +1,55 @@
+// Filters on properties that a supplement defines (catalog X2): evaluated from the supplement,
+// whatever the code system
+const { SupplementPropertyFilter, splitFilters } = require('../../tx/workers/supplement-filters');
+
+const supplement = {
+  url: 'http://example.org/supp',
+  jsonObj: {
+    property: [{ code: 'group', type: 'code' }],
+    concept: [
+      { code: 'a', property: [{ code: 'group', valueCode: 'red' }] },
+      { code: 'b', property: [{ code: 'group', valueCode: 'blue' }, { code: 'group', valueCode: 'red' }] },
+      { code: 'c', property: [{ code: 'other', valueString: 'x' }], concept: [
+        { code: 'c1', property: [{ code: 'group', valueCoding: { system: 'http://x', code: 'green' } }] }
+      ] }
+    ]
+  }
+};
+
+const f = (op, value) => new SupplementPropertyFilter('group', op, value, [supplement]);
+
+describe('SupplementPropertyFilter', () => {
+  test('= matches any of a code\'s values, including nested concepts and Codings', () => {
+    expect(f('=', 'red').codes()).toEqual(['a', 'b']);
+    expect(f('=', 'green').codes()).toEqual(['c1']);
+    expect(f('=', 'red').matches('c')).toBe(false);
+  });
+
+  test('in, regex and exists', () => {
+    expect(f('in', 'blue,green').codes()).toEqual(['b', 'c1']);
+    expect(f('regex', 'r.*').codes()).toEqual(['a', 'b']);
+    expect(f('exists', 'true').codes()).toEqual(['a', 'b', 'c1']);
+  });
+
+  test('not-in and exists = false select codes the supplement says nothing about, so cannot drive', () => {
+    expect(f('not-in', 'red').enumerable).toBe(false);
+    expect(f('not-in', 'red').matches('zzz')).toBe(true);
+    expect(f('not-in', 'red').matches('b')).toBe(false);
+    expect(f('exists', 'false').enumerable).toBe(false);
+    expect(f('exists', 'false').matches('c')).toBe(true);
+  });
+
+  test('splitFilters: the code system\'s own filters stay with it, even on a property a supplement also defines', async () => {
+    const cs = { supplements: [supplement], doesFilter: async (prop) => prop === 'concept' };
+    const split = await splitFilters(cs, [
+      { property: 'concept', op: 'is-a', value: 'x' },
+      { property: 'group', op: '=', value: 'red' },
+      { property: 'unknown', op: '=', value: 'y' }
+    ]);
+    expect(split.base.map(fc => fc.property)).toEqual(['concept', 'unknown']);
+    expect(split.supplement.map(sf => sf.toString())).toEqual(['group = red']);
+
+    const owns = { supplements: [supplement], doesFilter: async () => true };
+    expect((await splitFilters(owns, [{ property: 'group', op: '=', value: 'red' }])).supplement).toEqual([]);
+  });
+});