Skip to content

docs(matrix): audit the feature matrix against what actually ships - #1196

Merged
ako merged 1 commit into
mainfrom
docs/capability-matrix-audit
Sep 25, 2026
Merged

ako merged 1 commit into
mainfrom
docs/capability-matrix-audit

Conversation

@ako

@ako ako commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

What

An audit of docs/01-project/MDL_FEATURE_MATRIX.md against what actually ships, plus the guard extension that would have caught the drift. Every cell changed here was measured against the repo, not inferred.

Two shipped document types were listed as unavailable

This is the failure the matrix's own "Keeping this honest" note warns about — a row claiming a gap that has since been filled is worse than no table, because it sends people to Studio Pro for work mxcli can do.

Row Claimed Actually
Microflow rules "Not Yet Implemented" — no MDL surface at all CREATE [OR MODIFY] RULE ships, with a skill, rules.mdl, a microflow.rule syntax topic, a CATALOG.RULES view and rule REFS edges
Message definitions "Not Yet Implemented" SHOW / DESCRIBE / CREATE OR MODIFY / DROP / ALTER all ship, with tests and 40-message-definition-examples.mdl

Both now have rows in Core Document Types.

Layouts were recorded as read-only in four places

The row had CREATE / OR MODIFY / DROP / ALTER / Examples / Tests / Skills / Syntax all N, and three gap lists repeated "Layouts — Read-only". All eight are Y.

OR MODIFY was verified at exec, not parse — parsing proves nothing about upsert semantics:

create or replace layout … → Created layout …      (re-run → Unchanged)
create or modify  layout … → Unchanged layout …    (over the stored one)

Sixteen Examples cells, two of them pointing at the wrong file

Corrected against the actual mdl-examples/doctype-tests/ listing. Task Queues and Scheduled Events both cited 21, which is 21-import-export-mapping-examples.mdl; the mapping rows cited 06, which is the REST client.

Rule counts: measured, not remembered

19 built-in (lint --list-rules with no project rules dir) and 31 Starlark (.claude/lint-rules/*.star). Four different wrong pairs were in circulation across five live docs — one claiming 41 built-in rules.

CHANGELOG.md and the dated eval proposal are left alone: they are records of a past state, not claims about the present.

The guard could not have caught any of this

TestCapabilityDocsDoNotClaimShippedFeaturesAreMissing reads only the "Not Yet Implemented" section, and checks a hand-maintained list of seven capabilities naming neither rules, message definitions nor layouts. So:

  • the list is hoisted to shippedCapabilities and extended with all three;
  • a second test asserts "Missing Syntax Topics" may not list a capability whose mxcli syntax topic resolves — a flat contradiction, which is what makes it mechanically assertable.

Deliberately not extended to the Skills and Examples gap lists: an entry there can be true at the same time as a syntax topic exists (Regular Expressions has a topic and genuinely has no skill), so that check would false-positive.

Verified by reinstating all three false claims — two tests fail, naming each one. A guard only ever run against green docs has not been shown to detect anything.

Testing

make test exit 0, zero failures. make lint-go clean. make check-wiki-pages clean.

One trap worth recording for the next person: mxcli syntax <unknown-topic> exits 0 while printing Unknown topic, so an exit-code check reports every topic as present. Read the output. That false positive is what nearly made me record dataset and xml-schema as shipping.

🤖 Generated with Claude Code

The matrix claimed two shipped document types were unavailable, which is the
failure its own "Keeping this honest" note warns about -- a row claiming a gap
that has since been filled sends people to Studio Pro for work mxcli can do.

  - Microflow rules sat under "Not Yet Implemented" ("no MDL surface at all")
    while CREATE [OR MODIFY] RULE ships with a skill, rules.mdl, a
    `microflow.rule` syntax topic, a CATALOG.RULES view and rule REFS edges.
  - Message definitions likewise: SHOW / DESCRIBE / CREATE OR MODIFY / DROP /
    ALTER all ship, with tests and examples.

Both now have rows in Core Document Types instead.

Layouts were recorded as read-only in four places -- the row's CREATE, OR
MODIFY, DROP, ALTER, Examples, Tests, Skills and Syntax cells were all N, and
three gap lists repeated "Read-only". All eight are Y. OR MODIFY was verified
at exec rather than parse: `create or replace` reports Created then Unchanged,
and `create or modify` over the stored layout reports Unchanged.

Sixteen Examples cells corrected against the doctype-tests listing, including
two that cited the wrong file -- Task Queues and Scheduled Events both pointed
at 21 (import/export mappings) and the mapping rows pointed at 06 (the REST
client).

Rule counts are measured, not remembered: 19 built-in (`lint --list-rules`
with no project rules dir) and 31 Starlark (`.claude/lint-rules/*.star`). Four
different wrong pairs were in circulation across five live docs, one claiming
41 built-in rules. CHANGELOG and the dated eval proposal are left alone: they
are records of a past state, not claims about the present.

The guard could not have caught any of it. TestCapabilityDocsDoNotClaim...
reads only the "Not Yet Implemented" section and checks a hand-maintained list
of seven capabilities naming neither rules, message definitions nor layouts.
So the list is hoisted to `shippedCapabilities`, extended with all three, and a
second test asserts that "Missing Syntax Topics" may not list a capability
whose `mxcli syntax` topic resolves -- a flat contradiction, which is what
makes it mechanically assertable. It is deliberately not extended to the
Skills and Examples gap lists, where an entry can be true at the same time as
a syntax topic exists (Regular Expressions has a topic and no skill).

Checked by reinstating all three false claims: two tests fail, naming each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

AI Code Review

Summary

This PR audits and corrects the MDL feature matrix documentation against what actually ships in the codebase, plus adds guard tests to prevent future documentation drift. It fixes multiple instances where the matrix incorrectly claimed features were unavailable or had limited support.

Critical Issues

None found.

Moderate Issues

None found.

Minor Issues

  • None - the changes are focused and correct

What Looks Good

  • Documentation accuracy: Corrected multiple false claims in MDL_FEATURE_MATRIX.md:
    • Microflow rules and Message definitions were marked as "Not Yet Implemented" but actually ship
    • Layouts were incorrectly marked as read-only but support full CRUD operations
    • Fixed incorrect example file references for Task Queues, Scheduled Events, and others
    • Corrected lint rule counts from 14+27 to 19+31 built-in+Starlark rules across multiple docs
  • Preventive guards: Added meaningful tests in capability_docs_drift_test.go:
    • Extended existing test to use a centralized shippedCapabilities list
    • Added new test TestMissingSyntaxTopicsAreActuallyMissing that mechanically verifies the "Missing Syntax Topics" section doesn't list features that actually have syntax topics
    • The guard uses a direct contradiction approach (topic exists vs doc claims it doesn't) which is highly reliable
  • Consistency: All lint rule count references updated consistently across docs-site and project docs
  • Verification: The author verified by reinstating false claims to confirm tests fail appropriately

Recommendation

Approve - This PR successfully addresses documentation drift with both immediate corrections and preventive measures. The changes are minimal, focused, and improve the reliability of the documentation. The added tests provide lasting value by preventing similar drift in the future. No changes are needed.


Automated review via OpenRouter (Nemotron Super 120B) — workflow source

@ako
ako merged commit adc2928 into main Sep 25, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant