ast-grep matching-semantics differential: verify rules match the same nodes across an upgrade

[thecodedrift]

What this is

#182 upgrades the vendored ast-grep from 0.41.0 to 0.45.2 and verifies that the binary's interface did not move: JSON stream shape, exit codes, severity vocabulary, sg test output, $$$ arity, language: handling, .tests/ invisibility. It also verifies that no rule shipped in this repo uses the one breaking schema construct (a rewriter without fix).

It does not verify that a valid rule still matches the same nodes. That is a different question, and nothing local answers it today. A rule can keep parsing, keep exiting 0, and quietly match a different set of nodes than it did at 0.41.0. Four upstream changes can do exactly that, with no error at any point:

Change	Effect
#2677	metavariable bindings no longer leak across nthChild siblings
#2676	nor out of a negated not
#2868	root metavariables now match comments
#2803	ignore files are no longer read outside rule_dirs

The first three are the silent-wrong-answer class. The fourth changes which files are scanned rather than which nodes match, and it intersects the --no-ignore hidden and --globs work from #153.

Why this is not a vendor-contract test

The existing suite pins the interface by asking the binary a question and comparing the answer to a constant. A differential cannot work that way: the expected value is the other binary's output, so both 0.41.0 and 0.45.2 have to be resolvable in the same run. That is the piece of infrastructure this needs and the vendor-contract suite does not have.

It is also explicitly not a Vale-shaped corpus problem. ast-grep publishes a schema, so validity is already covered. The open question here is regression, not vocabulary.

Shape

• Both binaries resolvable at once. The one hard prerequisite.
• Compare findings as sets of (file, range, ruleId), never as counts. A count is equal in the case that matters least and hides a swap where one match appears as another disappears.
• A corpus exercising the changed constructs: nthChild with sibling metavariables, metavariables bound inside a negated not, root metavariables adjacent to comments.
• A fixture tree with hidden directories and nested ignore files, arranged so #2803 is isolated from fix(cli): let ast-grep rules see inside hidden directories #153's --no-ignore hidden / --globs behavior. Without that isolation a difference cannot be attributed to either change.
• Four to eight cases is the target. This is a regression probe, not a corpus sweep.

Also unverified

Whether #2803 altered the --no-ignore hidden behavior from #153. #182 did not exercise it, and that is precisely what this harness is for.

Note on scope

This is deliberately out of scope for #182 and is not a blocker for it. #182's own body records the same reasoning, so this issue is the tracking counterpart to that note.

Refs #162
Refs #153

[thecodedrift]

Decision: defer until the next engine upgrade, after 0.11.

The harness has no signal to produce until a version actually moves. Its whole output is a comparison between two binaries, so building it now means building it against 0.45.2 and 0.45.2 — the fixtures would be written and verified against a diff that does not exist yet, which tests the harness rather than the question.

Building it at the next upgrade also means the corpus is chosen against the changes that upgrade actually carries, rather than against the four this issue happens to name.

Nothing is lost by waiting: the shape, the prerequisites (both binaries resolvable at once, findings compared as sets of (file, range, ruleId), a fixture tree isolating the ignore-file change from #153's flags) are recorded here and do not decay.

What is deliberately still unverified in the meantime: whether the 0.41.0 → 0.45.2 move changed what any existing rule matches. #182 verified the interface did not move and that no shipped rule uses the one breaking schema construct. It did not verify node-for-node equivalence, and nothing since has. That is a known, accepted gap, not an oversight.

[thecodedrift]

Closing. The question this issue existed to answer is answered, and building it showed the method it prescribes is the wrong one.

The motivating question, answered

Did the 0.41.0 to 0.45.2 upgrade change what this repository's rules match? No, provably.

Not by sampling. None of the seven rules under .taskless/rules/sg/ uses any affected construct: no nthChild, no ofRule, and no metavariable bound inside a not. The two behaviour changes cannot reach them.

That is a stronger answer than a differential would have produced. An early run of the real rules against both binaries returned zero findings on each and looked like confirmation; it was vacuous, since the repository is clean. An empty result comparing equal to an empty result proves nothing, which is the same trap as an empty scan reporting success.

Three reasons the prescribed harness is the wrong tool

The comparison specified here would have missed one of the two real bugs. This issue says to compare findings as sets of (file, range, ruleId). The negated-not binding leak (ast-grep/ast-grep#2676) produces an identical finding across the upgrade: same file, same range, same rule. Only the rendered metavariable differs. A harness built exactly to spec reports "no change" and closes the question wrongly.

A guessed corpus fails silently, and convincingly. Rules written from the changelog over nthChild, negated not and root metavariables produced non-empty, identical results across five constructs on both binaries. That reads as evidence the upgrade was safe. Neither bug fires unless a metavariable is bound in the exact position the fix changed, so the corpus tested nothing while appearing to test everything.

The cheap method works better. Reading the upstream pull requests produced reproducing cases in minutes, and both showed a real difference immediately:

change	0.41.0	0.45.2
#2677 nthChild + binding ofRule	matched nothing	matches b();
#2676 $A across a negated not	rendered foo, leaked	unbound

What is in place instead

#199 pins both against the vendored binary, plus the fact that nthChild counts from 1 rather than 0, which the vendored schema's own minimum: 0 contradicts. These need no second binary, no network, and no skip condition. A skipping test is one nobody notices has stopped running, which is a failure mode this repository hit twice in one day.

#198 turns each change into an instruction in the 0.11.x ledger, so a user whose rules DO use these constructs is told what to do: a nthChild rule was inert and now reports findings nobody has seen, and a not-bound metavariable in a fix has been writing leaked text into files.

#199 also documents the two ways a relational rule verifies clean and reports nothing forever, both of which cost real time to diagnose here.

The method, for the next upgrade

Read the upstream pull requests for the release, reproduce their cases against both binaries, and pin the ones that differ in the vendor contract. That is what worked, it is cheaper than a harness, and it does not depend on anyone guessing which shapes matter.

What is genuinely not covered

A new, unanticipated change in a future release that nobody reads the notes for. A two-binary harness would not catch that either unless its corpus already exercised the shape, which is precisely the trap above.

Also still unreproduced: #2868, root metavariables against comments. No shape tried showed a difference, including the TSX case that PR names. Recorded rather than asserted.

Refs #199
Refs #198
Refs #162