Repository navigation
Fix twelve Client API spec divergences and add a conformance guard - #39
Merged
Merged
Conversation
Diffed every Client API call against Glean's OpenAPI spec (0.9.0, source
commit 0fef9ef) and corrected the ten defects that could be fixed without a
product decision. The mock dispatcher was keyed on the same wrong paths and
field names, so the suite passed while these calls would have failed against a
live tenant.
Paths:
- /getdocumentpermissions -> /getdocpermissions, body {documentSpec:{id}} ->
{documentId}
- /announcements/create -> /createannouncement
- /announcements/delete -> /deleteannouncement
Methods and parameter location:
- /tools/list is a GET with an optional toolNames query parameter, not a POST
- /listverifications takes count as a query parameter, not a body field
_post now accepts optional method and query arguments for these two cases.
Mock and local dispatch stay keyed on the bare path, so offline behaviour is
unchanged.
Request bodies:
- /tools/call arguments -> parameters
- /people email -> emailIds[]
- /summarize documentSpec -> documentSpecs[]
- /agents/runs/* agentId -> agent_id
- /agents/search query -> name (and the mock now filters by name,
so offline and live agree)
- /feedback {trackingToken, category} -> {event, trackingTokens[]};
THUMBS_UP/THUMBS_DOWN map to the spec's UPVOTE/DOWNVOTE, free text moves to
manualFeedbackInfo.comments, and category stays a separate optional enum
Tests: 13 added, covering the wire-level method/URL/body of the GET and
query-parameter endpoints (the live path had no coverage at all) and the
corrected request bodies. Eleven existing assertions encoded the old, wrong
shapes and were updated to the contract.
One of those was passing for the wrong reason: the /people mock test sent
{"email": ...}, the mock read nothing, and person(None) happened to return the
person the test asserted on. It now asserts the response echoes the requested
address.
Two defects remain and need a product decision: /announcements.list has no
endpoint in the API at all, and /createpin -> /pin also changes the pin model
from {url, query} to {documentId, queries[]}.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…list Both needed a product decision rather than a rename. /pins.create — the API pins by document id against a list of queries, while the CLI has always taken a URL. The URL is now resolved through /getdocuments first, so the existing flag keeps working, and --doc-id was added to skip that round trip. An unresolvable URL raises an error naming --doc-id rather than failing obscurely. The mock branch echoes documentId and queries so offline output shows what was actually pinned. /announcements.list — removed. Glean documents only create, update and delete for announcements; nothing backs a list. Keeping a command that can only 404 against a real tenant is worse than not having it. Gone from HANDLERS, the client, the mock dispatcher, DOCS, the /help grouping, COMMANDS.md, COMMAND_INDEX.md and REST_PATHS.md. Every Client API path and method in client.py now conforms to spec 0.9.0 (source commit 0fef9ef), verified by diffing all 37 calls against the parsed spec. incompat_report.md keeps the original findings as the record of what was wrong, with a status header noting the resolution and the two decisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The twelve Client API defects fixed in the previous two commits were invisible
to 1,151 tests because _mock_response is keyed on the same path strings the live
client posts to: a wrong path was wrong consistently, so nothing disagreed. This
adds the check that disagrees.
tests/test_spec_conformance.py reads client.py with `ast` and asserts every REST
path and HTTP method it calls exists in Glean's published spec. It handles the
awkward call shapes — f-string paths like f"/debug/{datasource}/status", and
agent_run passing a variable chosen between wait and stream — and normalises
path parameters so {group} matches the spec's {groupName}.
Because the suite must stay offline, it compares against a committed manifest
rather than the live spec. tools/refresh_spec_manifest.py generates that
manifest (82 client, 38 indexing, 3 metadata paths, ~7 KB) and is the only file
in the repo that touches the network. It needs PyYAML, which is tooling-only:
neither the package nor the tests import it, so the zero-dependency guarantee
holds and the zipapp is unchanged.
The scheduled spec-drift workflow runs the refresher with --check weekly. It is
deliberately separate from the release workflow — a Glean spec update is an
external event and should not fail an unrelated pull request.
Verified by regression injection. Reintroducing each original defect class makes
the guard fail with an actionable message:
wrong path -> client: POST /getdocumentpermissions
wrong method -> client: POST /tools/list (spec allows GET)
invented path -> client: POST /announcements/list
The guard also found a thirteenth item the audit had flagged as uncertain:
/indexemployeelist, which the spec declares with no operation. The manifest now
records declared-but-undocumented paths separately from absent ones, so this is
an acknowledged risk rather than a hard failure — and the test fails if Glean
documents it again, which is the signal to drop the exception.
Two deliberate limits, both documented in docs/TESTING.md: it does not require
the client to implement every documented endpoint, and it does not check request
body fields (TestSpecConformantBodies covers those).
Tests: 1,165 -> 1,173. Suite confirmed to make zero network calls.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the twelve Client API calls that diverged from Glean's published OpenAPI
spec, and adds the guard that stops it happening again.
Spec diffed against: 0.9.0, source commit
0fef9ef.Why they were invisible
_mock_responseis keyed on the same path strings the live client posts to. Awrong path was wrong consistently, so nothing disagreed and all 1,151 tests
passed while twelve commands would have failed against a real tenant.
The fixes
Paths
/getdocumentpermissions+{documentSpec:{id}}/getdocpermissions+{documentId}/announcements/create/createannouncement/announcements/delete/deleteannouncement/createpin/pinMethods and parameter location —
/tools/listis a GET with an optionaltoolNamesquery param, not a POST.listverificationstakescountin thequery string.
_postgained optionalmethodandqueryarguments for these;mock and local dispatch stay keyed on the bare path, so offline behaviour is
unchanged.
Bodies —
arguments→parameters,email→emailIds[],documentSpec→documentSpecs[],agentId→agent_id,query→name, and/feedbacknow sends{event, trackingTokens[]}with THUMBS_UP/DOWN mapping tothe spec's UPVOTE/DOWNVOTE, free text moving to
manualFeedbackInfo.comments,and
categorykept as the separate enum it is.Two product decisions
/announcements.listis removed. Glean documents only create, update anddelete — nothing backs a list. A command that can only 404 is worse than no
command.
/pins.createkeeps--urlby resolving it to a document id through/getdocumentsbefore pinning, since the API pins by id against a list ofqueries.
--doc-idwas added to skip the round trip.The guard
tests/test_spec_conformance.pyreadsclient.pywithastand asserts everypath and method it calls exists in the spec. It handles f-string paths
(
f"/debug/{datasource}/status"),agent_runpassing a variable, and{group}vs{groupName}.The suite must stay offline, so it compares against a committed manifest
(
tests/spec_manifest.json, ~7 KB) generated bytools/refresh_spec_manifest.py— the only file here that touches the network.It needs PyYAML, which is tooling only: neither the package nor the tests
import it, so zero runtime dependencies still holds and the zipapp is unchanged
(verified: 150 KB, nothing from
tools/ortests/leaks in).The new scheduled
spec-driftworkflow runs the refresher with--checkweekly. It is separate from
release.ymlon purpose — a Glean spec update is anexternal event and shouldn't fail an unrelated PR.
Verified by regression injection. Reintroducing each defect class makes the
guard fail with an actionable message:
It found a thirteenth item
/indexemployeelist— flagged as uncertain in the August audit. The specdeclares it with no operation, so it is neither present nor absent. The manifest
now records declared-but-undocumented paths separately, and a self-cleaning test
fails if Glean documents it again, which is the signal to drop the exception.
Notes
/peoplemock testsent
{"email": ...}, the mock read nothing, andperson(None)happened toreturn the person it asserted on. It now asserts the response echoes the
address requested.
contract.
Tests
1,151 → 1,173, including the first coverage of
_post's live path — themethod and query-string handling had none, which is exactly how those two
divergences survived. Confirmed with a socket guard that all 1,173 make zero
network calls.
🤖 Generated with Claude Code