feat(conformance): mark provider-supplied leaves in expected responses

conformance/README.md

@@ -14,7 +14,7 @@ runner/                Conformance runner that replays fixtures against any prov
 
 * **Fixtures** (`fixtures/v1/`): one JSON file per scenario. Each scenario defines a session with request/response steps, optional CLI flags, and the expected exit code. Responses are compared after JSON parsing so key order does not matter. The directory contains LPP 1.0, LPP 1.1, LPP 1.2, LPP 1.3, LPP 1.4, and LPP 1.5 scenarios. `fixtures/v1/sessions/` holds the standard initialize/shutdown handshake once per protocol version; a scenario whose steps don't themselves exercise the handshake sets `"session": "<version>"` and the runner splices those steps around its own.
 * **Mock provider** (`mock-provider/`): a small Rust binary implementing the full LPP v1 surface for a demonstration language distinct from OPY and OSTW. It runs over stdio so clients (like the Wright LPP client in wrightkit/wright#142) can test against it directly.
-* **Runner** (`runner/`): spawns a fresh provider process per scenario, feeds requests over stdin, validates stdout responses against expectations, and checks the process exit code.
+* **Runner** (`runner/`): spawns a fresh provider process per scenario, feeds requests over stdin, validates stdout responses against expectations, and checks the process exit code. Expected responses compare verbatim except at leaves a fixture marks provider-supplied (see the scenario format below).
 
 ## Running the suite
 
@@ -42,7 +42,8 @@ LPP has no wire dependency on Rust. Any provider that reads newline-delimited JS
 1. Build your provider as a stdio binary.
 2. Run the protocol-scope scenarios:
    `lpp-conformance-runner --provider <your-provider> --scope protocol`
-3. The semantic scenarios use `x-demo-lang` (an equation-puzzle DSL). To test a provider for another language, substitute your own language source texts and artifact payloads while keeping the protocol envelope and message sequence.
+   Scenarios that do not exercise `x-demo-lang` semantics pass out of the box: provider identity (`serverInfo`, `languages`), negotiated capability values, `supportedProtocolVersions`, and provider prose are asserted by contract shape, not by reference-adapter values.
+3. The remaining scenarios are adapter-specific by design: `semantics` scenarios exercise `x-demo-lang` content, and `providerArgs` scenarios configure the reference mock's CLI. To cover those, substitute your own language source texts, artifact payloads, and adapter configuration while keeping the protocol envelope and message sequence.
 
 Passing this suite verifies wire protocol conformance. It does not check Workshop engine correctness or runtime performance.
 
@@ -74,7 +75,11 @@ Passing this suite verifies wire protocol conformance. It does not check Worksho
   own steps call `lpp/initialize` or `lpp/shutdown`; session-behavior tests
   (mismatch, double-init, notifications) keep the handshake explicit.
 * `providerArgs`: optional extra command-line arguments for the provider
-  binary (used to exercise capability negotiation).
+  binary (used to exercise capability negotiation). These name the reference
+  mock's own CLI flags; a scenario carrying `providerArgs` is adapter-specific
+  and third-party providers are not expected to run it verbatim. Its
+  `expectResponse` keeps the negotiated capability map verbatim — the
+  `false` leaves are the assertion the scenario exists for.
 * `projectFiles`: optional relative path/content pairs that the runner writes
   to an isolated temporary project for entry-based project-loading scenarios.
   `${PROJECT_URI}` in requests and expected responses is replaced with that
@@ -86,6 +91,26 @@ Passing this suite verifies wire protocol conformance. It does not check Worksho
   sides, and an `expectResponse` without `id` expects the request's id (null
   for `rawLine` steps). Scenarios that exercise a non-`"2.0"` or unusual id
   keep the fields explicit.
+* Inside `expectResponse`, a leaf of the form `{ "$provider": "<kind>" }`
+  marks a value the spec leaves provider-supplied and asserts its contract
+  shape instead of a literal:
+
+  | Marker | Asserted shape |
+  | --- | --- |
+  | `nonEmptyString` | any non-empty string |
+  | `boolean` | any boolean |
+  | `languageList` | non-empty array of `{ "id", "extensions" }` language entries, extensions lowercase without a leading dot |
+  | `protocolVersions` | non-empty array of `MAJOR.MINOR` strings |
+
+  Marker positions are a whitelist — the provider-supplied positions named
+  in [§20.1](../docs/spec/conformance.md#201-provider-supplied-leaves), and
+  fixture validation enforces it, including the sibling `data.lpp.kind` and
+  negotiated `protocolVersion` a position can depend on. A marked
+  `capabilities` object must mark exactly the capability ids declared for
+  the negotiated version: a provider may advertise any subset (absent means
+  not offered), but an undeclared capability id still fails the closed-set
+  check. Every other leaf keeps verbatim JSON equality; markers are not
+  valid in `request`.
 * `expectExitCode`: the provider's exit status after stdin is closed (default
   0).
 

conformance/fixtures/v1/01-initialize-success.json

@@ -3,7 +3,7 @@
   "description": "A full handshake: initialize with client info, then a clean shutdown.",
   "scope": "protocol",
   "steps": [
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0","clientInfo":{"name":"wright","version":"0.2.0"}}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0","clientInfo":{"name":"wright","version":"0.2.0"}}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
     {"request":{"id":2,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/02-initialize-protocol-mismatch.json

@@ -3,6 +3,6 @@
   "description": "The provider must reject an unsupported protocol version and list the versions it supports. The session ends with EOF on stdin.",
   "scope": "protocol",
   "steps": [
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"0.9"}},"expectResponse":{"error":{"code":-32000,"message":"unsupported protocol version 0.9","data":{"lpp":{"kind":"protocolVersionMismatch","details":{"supportedProtocolVersions":["1.0"]}}}}}}
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"0.9"}},"expectResponse":{"error":{"code":-32000,"message":{"$provider":"nonEmptyString"},"data":{"lpp":{"kind":"protocolVersionMismatch","details":{"supportedProtocolVersions":{"$provider":"protocolVersions"}}}}}}}
   ]
 }

conformance/fixtures/v1/03-request-before-initialize.json

@@ -3,8 +3,8 @@
   "description": "No method except lpp/initialize may be processed before the handshake.",
   "scope": "protocol",
   "steps": [
-    {"request":{"id":1,"method":"lpp/check","params":{"documents":{}}},"expectResponse":{"error":{"code":-32000,"message":"invalid request: session not initialized","data":{"lpp":{"kind":"invalidRequest","details":{"reason":"notInitialized"}}}}}},
-    {"request":{"id":2,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"request":{"id":1,"method":"lpp/check","params":{"documents":{}}},"expectResponse":{"error":{"code":-32000,"message":{"$provider":"nonEmptyString"},"data":{"lpp":{"kind":"invalidRequest","details":{"reason":"notInitialized"}}}}}},
+    {"request":{"id":2,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
     {"request":{"id":3,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/04-initialize-twice.json

@@ -3,8 +3,8 @@
   "description": "A second lpp/initialize after a successful one is a session violation.",
   "scope": "protocol",
   "steps": [
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
-    {"request":{"id":2,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"error":{"code":-32000,"message":"invalid request: already initialized","data":{"lpp":{"kind":"invalidRequest","details":{"reason":"alreadyInitialized"}}}}}},
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
+    {"request":{"id":2,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"error":{"code":-32000,"message":{"$provider":"nonEmptyString"},"data":{"lpp":{"kind":"invalidRequest","details":{"reason":"alreadyInitialized"}}}}}},
     {"request":{"id":3,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/05-unknown-method.json

@@ -4,6 +4,6 @@
   "scope": "protocol",
   "session": "1.0",
   "steps": [
-    {"request":{"id":2,"method":"lpp/frobnicate","params":{}},"expectResponse":{"error":{"code":-32601,"message":"Method not found"}}}
+    {"request":{"id":2,"method":"lpp/frobnicate","params":{}},"expectResponse":{"error":{"code":-32601,"message":{"$provider":"nonEmptyString"}}}}
   ]
 }

conformance/fixtures/v1/06-malformed-json.json

@@ -3,8 +3,8 @@
   "description": "A line that is not valid JSON is answered with the standard JSON-RPC -32700 parse error (id null), and the session continues.",
   "scope": "protocol",
   "steps": [
-    {"rawLine":"this is not json","expectResponse":{"error":{"code":-32700,"message":"Parse error"}}},
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"rawLine":"this is not json","expectResponse":{"error":{"code":-32700,"message":{"$provider":"nonEmptyString"}}}},
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
     {"request":{"id":2,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/07-invalid-jsonrpc-version.json

@@ -3,8 +3,8 @@
   "description": "A message with a non-2.0 jsonrpc version is answered with the standard JSON-RPC -32600 invalid request error.",
   "scope": "protocol",
   "steps": [
-    {"rawLine":"{\"jsonrpc\":\"1.0\",\"id\":1,\"method\":\"lpp/initialize\",\"params\":{\"protocolVersion\":\"1.0\"}}","expectResponse":{"error":{"code":-32600,"message":"Invalid Request"}}},
-    {"request":{"id":2,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"rawLine":"{\"jsonrpc\":\"1.0\",\"id\":1,\"method\":\"lpp/initialize\",\"params\":{\"protocolVersion\":\"1.0\"}}","expectResponse":{"error":{"code":-32600,"message":{"$provider":"nonEmptyString"}}}},
+    {"request":{"id":2,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
     {"request":{"id":3,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/08-notification-not-supported.json

@@ -3,8 +3,8 @@
   "description": "LPP v1 defines no notifications: a message without an id is a protocol violation answered with an LPP invalidRequest error (id null in the response).",
   "scope": "protocol",
   "steps": [
-    {"request":{"method":"lpp/shutdown"},"expectResponse":{"error":{"code":-32000,"message":"invalid request: LPP v1 defines no notifications","data":{"lpp":{"kind":"invalidRequest","details":{"reason":"notificationNotSupported"}}}}}},
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"request":{"method":"lpp/shutdown"},"expectResponse":{"error":{"code":-32000,"message":{"$provider":"nonEmptyString"},"data":{"lpp":{"kind":"invalidRequest","details":{"reason":"notificationNotSupported"}}}}}},
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
     {"request":{"id":2,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/09-batch-not-supported.json

@@ -3,8 +3,8 @@
   "description": "Batch messages are not supported in LPP v1: an array message is answered with the standard JSON-RPC -32600 error (id null).",
   "scope": "protocol",
   "steps": [
-    {"request":[{"jsonrpc":"2.0","id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}}],"expectResponse":{"error":{"code":-32600,"message":"Invalid Request"}}},
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":true,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"request":[{"jsonrpc":"2.0","id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}}],"expectResponse":{"error":{"code":-32600,"message":{"$provider":"nonEmptyString"}}}},
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":{"$provider":"boolean"},"compile":{"$provider":"boolean"},"reconstruct":{"$provider":"boolean"},"symbols":{"$provider":"boolean"},"definition":{"$provider":"boolean"},"references":{"$provider":"boolean"},"rename":{"$provider":"boolean"},"editValidation":{"$provider":"boolean"}}}}},
     {"request":{"id":2,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]
 }

conformance/fixtures/v1/10-capability-negotiation.json

@@ -4,8 +4,8 @@
   "scope": "protocol",
   "providerArgs": ["--without", "reconstruct"],
   "steps": [
-    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":"lpp-mock-provider","version":"0.1.0"},"languages":[{"id":"x-demo-lang","extensions":["xdl"]}],"capabilities":{"check":true,"compile":true,"reconstruct":false,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
-    {"request":{"id":2,"method":"lpp/reconstruct","params":{"artifact":{"format":"x-demo/puzzle-eval-v1","content":"{}"}}},"expectResponse":{"error":{"code":-32000,"message":"capability 'reconstruct' is not available","data":{"lpp":{"kind":"capabilityUnavailable","details":{"capability":"reconstruct","method":"lpp/reconstruct"}}}}}},
+    {"request":{"id":1,"method":"lpp/initialize","params":{"protocolVersion":"1.0"}},"expectResponse":{"result":{"protocolVersion":"1.0","serverInfo":{"name":{"$provider":"nonEmptyString"},"version":{"$provider":"nonEmptyString"}},"languages":{"$provider":"languageList"},"capabilities":{"check":true,"compile":true,"reconstruct":false,"symbols":true,"definition":true,"references":true,"rename":true,"editValidation":true}}}},
+    {"request":{"id":2,"method":"lpp/reconstruct","params":{"artifact":{"format":"x-demo/puzzle-eval-v1","content":"{}"}}},"expectResponse":{"error":{"code":-32000,"message":{"$provider":"nonEmptyString"},"data":{"lpp":{"kind":"capabilityUnavailable","details":{"capability":"reconstruct","method":"lpp/reconstruct"}}}}}},
     {"request":{"id":3,"method":"lpp/check","params":{"documents":{"file:///project/puzzle.xdl":{"uri":"file:///project/puzzle.xdl","languageId":"x-demo-lang","version":3,"text":"puzzle clean {\n  target = 40\n  start = 10\n  ops {\n    double: x => x * 2\n    plus1: x => x + 1\n  }\n  solution = [ double, double ]\n}"}}}},"expectResponse":{"result":{"documents":[{"uri":"file:///project/puzzle.xdl","version":3,"diagnostics":[]}]}}},
     {"request":{"id":4,"method":"lpp/shutdown","params":{}},"expectResponse":{"result":null}}
   ]