Skip to content

Fix twelve Client API spec divergences and add a conformance guard - #39

Merged
barkz merged 3 commits into
mainfrom
fix/client-api-spec-conformance
Sep 28, 2026
Merged

barkz merged 3 commits into
mainfrom
fix/client-api-spec-conformance

Conversation

@barkz

@barkz barkz commented Sep 28, 2026

Copy link
Copy Markdown
Owner

What

Fixes the twelve Client API calls that diverged from Glean's published OpenAPI
spec, and adds the guard that stops it happening again.

Spec diffed against: 0.9.0, source commit 0fef9ef.

Why they were invisible

_mock_response is keyed on the same path strings the live client posts to. A
wrong path was wrong consistently, so nothing disagreed and all 1,151 tests
passed while twelve commands would have failed against a real tenant.

The fixes

Paths

Was Now
/getdocumentpermissions + {documentSpec:{id}} /getdocpermissions + {documentId}
/announcements/create /createannouncement
/announcements/delete /deleteannouncement
/createpin /pin

Methods and parameter location — /tools/list is a GET with an optional
toolNames query param, not a POST. listverifications takes count in the
query string. _post gained optional method and query arguments for these;
mock and local dispatch stay keyed on the bare path, so offline behaviour is
unchanged.

Bodies — arguments→parameters, email→emailIds[],
documentSpec→documentSpecs[], agentId→agent_id, query→name, and
/feedback now sends {event, trackingTokens[]} with THUMBS_UP/DOWN mapping to
the spec's UPVOTE/DOWNVOTE, free text moving to manualFeedbackInfo.comments,
and category kept as the separate enum it is.

Two product decisions

  • /announcements.list is removed. Glean documents only create, update and
    delete — nothing backs a list. A command that can only 404 is worse than no
    command.
  • /pins.create keeps --url by resolving it to a document id through
    /getdocuments before pinning, since the API pins by id against a list of
    queries. --doc-id was added to skip the round trip.

The guard

tests/test_spec_conformance.py reads client.py with ast and asserts every
path and method it calls exists in the spec. It handles f-string paths
(f"/debug/{datasource}/status"), agent_run passing a variable, and
{group} vs {groupName}.

The suite must stay offline, so it compares against a committed manifest
(tests/spec_manifest.json, ~7 KB) generated by
tools/refresh_spec_manifest.py — the only file here that touches the network.
It needs PyYAML, which is tooling only: neither the package nor the tests
import it, so zero runtime dependencies still holds and the zipapp is unchanged
(verified: 150 KB, nothing from tools/ or tests/ leaks in).

The new scheduled spec-drift workflow runs the refresher with --check
weekly. It is separate from release.yml on purpose — a Glean spec update is an
external event and shouldn't fail an unrelated PR.

Verified by regression injection. Reintroducing each defect class makes the
guard fail with an actionable message:

wrong path    -> client: POST /getdocumentpermissions
wrong method  -> client: POST /tools/list (spec allows GET)
invented path -> client: POST /announcements/list

It found a thirteenth item

/indexemployeelist — flagged as uncertain in the August audit. The spec
declares it with no operation, so it is neither present nor absent. The manifest
now records declared-but-undocumented paths separately, and a self-cleaning test
fails if Glean documents it again, which is the signal to drop the exception.

Notes

  • One pre-existing test was passing for the wrong reason: the /people mock test
    sent {"email": ...}, the mock read nothing, and person(None) happened to
    return the person it asserted on. It now asserts the response echoes the
    address requested.
  • Eleven existing assertions encoded the old, wrong shapes and now encode the
    contract.

Tests

1,151 → 1,173, including the first coverage of _post's live path — the
method and query-string handling had none, which is exactly how those two
divergences survived. Confirmed with a socket guard that all 1,173 make zero
network calls.

🤖 Generated with Claude Code

barkz and others added 3 commits September 27, 2026 20:37
Diffed every Client API call against Glean's OpenAPI spec (0.9.0, source
commit 0fef9ef) and corrected the ten defects that could be fixed without a
product decision. The mock dispatcher was keyed on the same wrong paths and
field names, so the suite passed while these calls would have failed against a
live tenant.

Paths:
- /getdocumentpermissions -> /getdocpermissions, body {documentSpec:{id}} ->
  {documentId}
- /announcements/create   -> /createannouncement
- /announcements/delete   -> /deleteannouncement

Methods and parameter location:
- /tools/list is a GET with an optional toolNames query parameter, not a POST
- /listverifications takes count as a query parameter, not a body field

_post now accepts optional method and query arguments for these two cases.
Mock and local dispatch stay keyed on the bare path, so offline behaviour is
unchanged.

Request bodies:
- /tools/call        arguments   -> parameters
- /people            email       -> emailIds[]
- /summarize         documentSpec -> documentSpecs[]
- /agents/runs/*     agentId     -> agent_id
- /agents/search     query       -> name (and the mock now filters by name,
                                   so offline and live agree)
- /feedback          {trackingToken, category} -> {event, trackingTokens[]};
  THUMBS_UP/THUMBS_DOWN map to the spec's UPVOTE/DOWNVOTE, free text moves to
  manualFeedbackInfo.comments, and category stays a separate optional enum

Tests: 13 added, covering the wire-level method/URL/body of the GET and
query-parameter endpoints (the live path had no coverage at all) and the
corrected request bodies. Eleven existing assertions encoded the old, wrong
shapes and were updated to the contract.

One of those was passing for the wrong reason: the /people mock test sent
{"email": ...}, the mock read nothing, and person(None) happened to return the
person the test asserted on. It now asserts the response echoes the requested
address.

Two defects remain and need a product decision: /announcements.list has no
endpoint in the API at all, and /createpin -> /pin also changes the pin model
from {url, query} to {documentId, queries[]}.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…list

Both needed a product decision rather than a rename.

/pins.create — the API pins by document id against a list of queries, while the
CLI has always taken a URL. The URL is now resolved through /getdocuments
first, so the existing flag keeps working, and --doc-id was added to skip that
round trip. An unresolvable URL raises an error naming --doc-id rather than
failing obscurely. The mock branch echoes documentId and queries so offline
output shows what was actually pinned.

/announcements.list — removed. Glean documents only create, update and delete
for announcements; nothing backs a list. Keeping a command that can only 404
against a real tenant is worse than not having it. Gone from HANDLERS, the
client, the mock dispatcher, DOCS, the /help grouping, COMMANDS.md,
COMMAND_INDEX.md and REST_PATHS.md.

Every Client API path and method in client.py now conforms to spec 0.9.0
(source commit 0fef9ef), verified by diffing all 37 calls against the parsed
spec.

incompat_report.md keeps the original findings as the record of what was wrong,
with a status header noting the resolution and the two decisions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The twelve Client API defects fixed in the previous two commits were invisible
to 1,151 tests because _mock_response is keyed on the same path strings the live
client posts to: a wrong path was wrong consistently, so nothing disagreed. This
adds the check that disagrees.

tests/test_spec_conformance.py reads client.py with `ast` and asserts every REST
path and HTTP method it calls exists in Glean's published spec. It handles the
awkward call shapes — f-string paths like f"/debug/{datasource}/status", and
agent_run passing a variable chosen between wait and stream — and normalises
path parameters so {group} matches the spec's {groupName}.

Because the suite must stay offline, it compares against a committed manifest
rather than the live spec. tools/refresh_spec_manifest.py generates that
manifest (82 client, 38 indexing, 3 metadata paths, ~7 KB) and is the only file
in the repo that touches the network. It needs PyYAML, which is tooling-only:
neither the package nor the tests import it, so the zero-dependency guarantee
holds and the zipapp is unchanged.

The scheduled spec-drift workflow runs the refresher with --check weekly. It is
deliberately separate from the release workflow — a Glean spec update is an
external event and should not fail an unrelated pull request.

Verified by regression injection. Reintroducing each original defect class makes
the guard fail with an actionable message:
  wrong path      -> client: POST /getdocumentpermissions
  wrong method    -> client: POST /tools/list (spec allows GET)
  invented path   -> client: POST /announcements/list

The guard also found a thirteenth item the audit had flagged as uncertain:
/indexemployeelist, which the spec declares with no operation. The manifest now
records declared-but-undocumented paths separately from absent ones, so this is
an acknowledged risk rather than a hard failure — and the test fails if Glean
documents it again, which is the signal to drop the exception.

Two deliberate limits, both documented in docs/TESTING.md: it does not require
the client to implement every documented endpoint, and it does not check request
body fields (TestSpecConformantBodies covers those).

Tests: 1,165 -> 1,173. Suite confirmed to make zero network calls.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@barkz
barkz merged commit 04baebd into main Sep 28, 2026
7 checks passed
@barkz
barkz deleted the fix/client-api-spec-conformance branch September 28, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant