Skip to content

chore(ci): count review denials from every source, and census the run - #197

Merged
thecodedrift merged 1 commit into
mainfrom
chore/review-guard-denials
Aug 27, 2026
Merged

chore(ci): count review denials from every source, and census the run#197
thecodedrift merged 1 commit into
mainfrom
chore/review-guard-denials

Conversation

@thecodedrift

Copy link
Copy Markdown
Member

The #185 guard caught the zero-turn review on #196, which is what it exists for. Then it reported the wrong reason.

What went wrong

The guard printed permission_denials=0. The streamed job log for the same run carried permission_denials_count: 6.

That zero was then used as evidence that permissions were not the cause, which sent the investigation the wrong way. Reporting the wrong number was worse than reporting nothing, because it looked like an answer.

Measured on run 33039882331: the saved execution file's result record omits permission_denials_count, while the streamed stdout carries it. Every other field matched exactly, num_turns, is_error, and total_cost_usd to sixteen digits, so the guard found the right record. The file just lacks that one field.

The naming loop was already there. It sat behind if denials:, so a missing count made it unreachable:

denials = result.get("permission_denials_count") or 0   # 0 when the file omits it
...
if denials:                                             # never true
    records = result.get("permission_denials")
    if not isinstance(records, list):
        records = [m for m in messages if "denial" in ...]   # the scan that would have worked

What changes

Denials are counted from every source. The maximum of the count field and the number of denial records found in the file, and the naming loop reads those records directly instead of re-deriving them inside a gate that could not open.

The run now describes itself. A structural census of the execution file: message types and counts. Two zero-turn runs have billed real inference, #182 at $1.08 and #196 at $2.20, and neither left anything to diagnose from afterwards.

Types and counts only. No message content, no tool inputs, nothing the model produced while reading an untrusted diff, which is why show_full_output stays off on a public repo. None of that is needed to say what shape a run had.

Verification

Three fixtures, covering both observed shapes:

fixture before after
#196 shape: count omitted, denial records present 0 denials, nothing named 2 denials, names TodoWrite, Task
#182 shape: count present 4 denials 4 denials, unchanged
healthy run exit 0 exit 0, unchanged

All three also emit the census line.

What this still does not tell us

Why the run takes zero turns. Six denials on #196 and four on #182 make the allowlist the strongest hypothesis again, but a denial does not have to be fatal, and nothing here explains why inference is billed before any turn completes. The census is what should narrow that next time: the shape of a zero-turn run is currently unknown, and one run of it will now say so in the log.

Refs #185

The guard added in #185 caught the zero-turn run on #196, which is what it was
for. It then reported `permission_denials=0` while the streamed job log carried
`permission_denials_count: 6`, so it skipped naming the denied tools, and that
zero was read as evidence that permissions were not the problem. Reporting the
wrong number was worse than reporting nothing.

MEASURED on run 33039882331: the saved execution file's result record omits
`permission_denials_count`, while the streamed stdout carries it. Every other
field matched exactly, including `total_cost_usd` to sixteen digits, so the
guard found the right record and the file simply lacks that one field.

Denials are now the maximum of the count field and the number of denial
records found in the file, and the naming loop reads those records rather than
re-deriving them behind a gate that could never open. The old code already had
the scan; it sat inside `if denials:`, so a missing count made it unreachable.

Also added: a structural census of the execution file, message types and
counts only. Two zero-turn runs have now billed real inference, #182 at $1.08
and #196 at $2.20, and neither left anything to diagnose from afterwards. No
message content, no tool inputs, nothing the model produced while reading an
untrusted diff; those are the reason `show_full_output` stays off on a public
repo, and none of it is needed to say what shape a run had.

Verified against three fixtures: the #196 shape with the count omitted and
denial records present now reports 2 denials and names them, where the old
guard reported 0 and named nothing; the #182 shape with the count present is
unchanged; a healthy run still exits 0.
@thecodedrift
thecodedrift merged commit 45e9997 into main Aug 27, 2026
6 checks passed
@thecodedrift
thecodedrift deleted the chore/review-guard-denials branch August 27, 2026 05:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant