Skip to content

feat(conformance): mark provider-supplied leaves in expected responses - #53

Merged
Teakowa merged 3 commits into
mainfrom
feat/provider-owned-leaves
Oct 6, 2026
Merged

Teakowa merged 3 commits into
mainfrom
feat/provider-owned-leaves

Conversation

@e54-bot

@e54-bot e54-bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #49. Rebased onto main after #51 (refactor/conformance-tooling-seams) merged.

Expected responses used verbatim JSON equality, so the six session templates and inline handshake scenarios pinned reference-mock identity (serverInfo, languages, capability values, supportedProtocolVersions) and provider prose (error.message, refusalCode, provider-defined details.reason). This change introduces a provider-leaf marker:

  • A leaf written { "$provider": "<kind>" } in expectResponse asserts the leaf's contract shape instead of a literal. Four kinds cover every spec-declared provider-supplied position: nonEmptyString (serverInfo fields, error prose, refusalCode, unenumerated reason), boolean (capability values), languageList (the languages array), protocolVersions (supportedProtocolVersions as MAJOR.MINOR strings).
  • matcher.rs owns the recursive compare with path-aware mismatch errors. Fixture validation enforces the marker positions as a whitelist — a marker on a contract-owned leaf (protocolVersion, error.code, an enumerated invalidRequest reason, an undeclared capability id) is a fixture error, with sibling context (data.lpp.kind, negotiated protocolVersion) deciding positions that are provider-supplied only under some error kinds.
  • capabilities semantics follow §7.3: a provider may advertise any subset (absent == not offered), so missing keys are exempt at that position while undeclared ids still fail the closed-set check. A marked capabilities object must cover exactly the ids declared for the negotiated version. §7.2's "requires the eight fields" wording is corrected to match — the object is required, its keys are any declared subset.
  • Every unmarked leaf keeps verbatim JSON equality, including object key sets. Enumerated invalidRequest reasons (notInitialized, alreadyInitialized, notificationNotSupported) stay pinned — they are spec values, not provider prose.
  • providerArgs remains an adapter-owned surface: scenarios carrying it keep their negotiated capability map verbatim (the false leaves are the assertion) and are not expected to run on third-party providers.
  • The serverInfo.version coupling to workspace CARGO_PKG_VERSION is gone: no fixture pins lpp-mock-provider, 0.1.0, or the x-demo-lang language entry anymore.

Documented in docs/spec/conformance.md §20.1 and conformance/README.md.

Test plan

  • cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, cargo test --workspace (incl. 6 new matcher unit tests)
  • --validate-only: 61/61 pass
  • Full suite vs lpp-mock-provider: 61/61 pass — no behavioral change for the reference adapter
  • Independent acceptance: a scratch Python stub provider (different language id, name, version, prose) passes all 10 protocol-scope scenarios that exercise neither x-demo-lang semantics nor providerArgs — fixture content unchanged
  • git diff --check

@Teakowa Teakowa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three contract-correctness findings block merge. CI is green, but the current matcher can both weaken normative fixtures and reject/accept provider results inconsistently with the v1 spec. Please resolve these against the owning spec before retargeting/merging.

Comment thread conformance/runner/src/matcher.rs
Comment thread conformance/runner/src/matcher.rs
Comment thread conformance/runner/src/matcher.rs Outdated
Base automatically changed from refactor/conformance-tooling-seams to main October 6, 2026 17:51
e54-bot pushed a commit that referenced this pull request Oct 6, 2026
…pability set

Review on #53 found three contract gaps: markers validated kind but not position, so a fixture could weaken a contract-owned leaf; exact key matching at result.capabilities required every declared id despite §7.3 permitting any advertised subset (absent == not offered), which also contradicted §7.2's required-fields wording; and languageList accepted malformed extensions. Marker validation now enforces the §20.1 position whitelist with sibling context (data.lpp.kind, negotiated protocolVersion), a marked capabilities object must cover exactly the version's declared ids while a provider may advertise any subset, and extensions must be lowercase without a leading dot. §7.2 now states the object is required while keys are any subset of the declared ids, matching §7.3.
@e54-bot
e54-bot requested a review from Teakowa October 6, 2026 17:57
Expected responses previously asserted verbatim JSON equality, so the six session templates and inline handshake scenarios pinned reference-mock identity (serverInfo, languages, capability values) and provider prose. A leaf written { "$provider": "<kind>" } now asserts the leaf's contract shape — boolean, languageList, nonEmptyString, or protocolVersions — while every unmarked leaf keeps verbatim equality, letting providers other than lpp-mock-provider satisfy protocol-scope scenarios. Scenarios carrying providerArgs keep their negotiated capability map verbatim and remain adapter-specific. Verified with a foreign-language stub: all ten protocol scenarios that exercise neither x-demo-lang semantics nor providerArgs pass unchanged. Fixes #49
…pability set

Review on #53 found three contract gaps: markers validated kind but not position, so a fixture could weaken a contract-owned leaf; exact key matching at result.capabilities required every declared id despite §7.3 permitting any advertised subset (absent == not offered), which also contradicted §7.2's required-fields wording; and languageList accepted malformed extensions. Marker validation now enforces the §20.1 position whitelist with sibling context (data.lpp.kind, negotiated protocolVersion), a marked capabilities object must cover exactly the version's declared ids while a provider may advertise any subset, and extensions must be lowercase without a leading dot. §7.2 now states the object is required while keys are any subset of the declared ids, matching §7.3.
@e54-bot
e54-bot force-pushed the feat/provider-owned-leaves branch from 67d894e to ce33751 Compare October 6, 2026 17:59

@Teakowa Teakowa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three previous findings are resolved. One regression remains in the new capability-subset matching path.

Comment thread conformance/runner/src/matcher.rs Outdated
Review on #53: the §7.3 missing-key exemption applied to every result.capabilities object, so a providerArgs scenario asserting an explicit negotiated map — for example reconstruct: false — would pass when the provider omitted the id entirely. The exemption now applies only when the expected capabilities object is fully marker-checked; verbatim maps keep exact key matching. Adds a regression test for the omitted-false-key case.
@e54-bot
e54-bot requested a review from Teakowa October 6, 2026 18:11

@Teakowa Teakowa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Teakowa
Teakowa merged commit f149a4d into main Oct 6, 2026
2 checks passed
@Teakowa
Teakowa deleted the feat/provider-owned-leaves branch October 6, 2026 18:13
@e54-bot e54-bot mentioned this pull request Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the conformance suite's protocol scope pass for providers other than the mock

2 participants