verification-before-completion: a distilled set of instrument-failure classes the gate doesn't yet name (positive controls, reported≠verified, coverage by surface…) — offer to contribute a table

[scos-lab]

The verification-before-completion gate covers the main failure — claiming without running the command. Over ~14 months of logged autonomous runs we kept hitting a second layer of failures that pass that gate: the agent did run a verification command, read the output, and was still wrong, because the instrument itself was the problem. We distilled them into eight short clauses, each traced to an actual incident, and published them as part of a verdict vocabulary (CC BY 4.0):

https://github.com/scos-lab/semantic-tension-language/blob/main/spec/eval-verdict-vocabulary.md (§6, V1–V8)

The ones that seem most directly useful to this skill:

• V2 Positive control — before acting on a zero/negative reading ("0 matches", "no change", "not found"), prove the instrument can produce a hit on a different input. Most of our false "clean" results were a grep that couldn't have matched anything. The inverse holds too: an alarming reading is more likely a broken reading — validate the instrument before fixing what it accuses.
• V4 Reported is not verified — a tool's return value is a claim by the tool; promoting it to "verified" requires an independent read of the changed state through a different path (an accessibility action that returned True and did nothing; a type command that reported success with zero keystrokes). It's the general form of the "agent said success → check the diff" row the skill already has.
• V6 Coverage counts surfaces, not checks — two passing checks on the same surface prove one surface; our bug was in the CLI argument parsing nobody exercised. The closing move we use: run the exact real command, read the number it prints itself.
• V7 Silence is not success — a dead alarm and a healthy system emit the same signal. Monitors die silently; a checker that crashed reports no problems.
• V8 Absence needs a second instrument — "everything I produced exists in the source" is blind to deletion; sandboxed or filtered instruments are blind to what they can't see.

(V1 asymmetric evidence, V3 independent criteria, V5 records decay round it out.)

Each clause carries an "executable form" line — a yes/no check an agent can run — which is roughly the shape the gate table already uses. #614 ("require tool verification before asserting facts about external systems") is V4/V5 in different words, which is part of why we think these would slot in.

If useful, we're happy to PR a compact "Instrument failures" table into the skill in its existing style (Claim / Requires / Not sufficient, ~8 rows, no prose) — or just leave this here as a reference. Either is fine; the text is self-contained and CC BY, so there's no dependency to take on.

[dajiaohuang]

Related maintainer guidance on #2145 says deliberately tuned skill wording lands maintainer-authored with eval gating, and #614 already covers part of the reported-versus-verified territory. Should this proposal follow that same maintainer-authored route, or would an external before/after-eval PR be welcome for verification-before-completion? I will not prepare a competing skill-text patch until that ownership and evaluation route is clear.

[scos-lab]

Maintainer-authored is the right route as far as we're concerned — we're not preparing a skill-text patch, and this issue is best read as reference material for whoever does (thanks for the pointer to #2145; that policy makes sense for text this tuned, and #614 is indeed the same territory as V4/V5).

For the evidence side, since #2145 says the maintainers act on underlying sessions rather than proposals, here is what exists publicly per clause, so nobody has to take the list on faith:

• Each clause in the vocabulary (§6) carries an executable form — the yes/no check an agent runs — which is the part that would map onto the gate table.
• The incidents behind the clauses are written up, anonymised, by failure class in "Seven ways an autonomous agent walks into a wall" (instrument fallacies → V1/V2, silent deaths → V7, report≠effect → V4, coverage illusions → V6, absence/deletion blindness → V8, record rot → V5).
• Two full, unedited clean-room sessions in which agents given only the spec text ran the loop and produced the verdict records — transcripts and artifacts in exhibit-records/ (run1 Claude, run2 Codex), walked through at https://stl-lang.org/exhibit — plus the DelegationBench pilot report, whose "agents backgrounded their watchdog and ended the session" finding is V7 caught in a harness rather than in the field.

If a before/after eval is the gate and a maintainer wants one designed for these specific failure shapes (a grep that can't match, a tool that returns success with no effect, a check that exercises the wrong surface), we're glad to help draft it or to run it against our harness — but we'd rather that be at a maintainer's request than a competing artifact. Happy to leave it here otherwise.