chore(ci): count review denials from every source, and census the run - #197
Merged
Conversation
The guard added in #185 caught the zero-turn run on #196, which is what it was for. It then reported `permission_denials=0` while the streamed job log carried `permission_denials_count: 6`, so it skipped naming the denied tools, and that zero was read as evidence that permissions were not the problem. Reporting the wrong number was worse than reporting nothing. MEASURED on run 33039882331: the saved execution file's result record omits `permission_denials_count`, while the streamed stdout carries it. Every other field matched exactly, including `total_cost_usd` to sixteen digits, so the guard found the right record and the file simply lacks that one field. Denials are now the maximum of the count field and the number of denial records found in the file, and the naming loop reads those records rather than re-deriving them behind a gate that could never open. The old code already had the scan; it sat inside `if denials:`, so a missing count made it unreachable. Also added: a structural census of the execution file, message types and counts only. Two zero-turn runs have now billed real inference, #182 at $1.08 and #196 at $2.20, and neither left anything to diagnose from afterwards. No message content, no tool inputs, nothing the model produced while reading an untrusted diff; those are the reason `show_full_output` stays off on a public repo, and none of it is needed to say what shape a run had. Verified against three fixtures: the #196 shape with the count omitted and denial records present now reports 2 denials and names them, where the old guard reported 0 and named nothing; the #182 shape with the count present is unchanged; a healthy run still exits 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The #185 guard caught the zero-turn review on #196, which is what it exists for. Then it reported the wrong reason.
What went wrong
The guard printed
permission_denials=0. The streamed job log for the same run carriedpermission_denials_count: 6.That zero was then used as evidence that permissions were not the cause, which sent the investigation the wrong way. Reporting the wrong number was worse than reporting nothing, because it looked like an answer.
Measured on run 33039882331: the saved execution file's result record omits
permission_denials_count, while the streamed stdout carries it. Every other field matched exactly,num_turns,is_error, andtotal_cost_usdto sixteen digits, so the guard found the right record. The file just lacks that one field.The naming loop was already there. It sat behind
if denials:, so a missing count made it unreachable:What changes
Denials are counted from every source. The maximum of the count field and the number of denial records found in the file, and the naming loop reads those records directly instead of re-deriving them inside a gate that could not open.
The run now describes itself. A structural census of the execution file: message types and counts. Two zero-turn runs have billed real inference, #182 at $1.08 and #196 at $2.20, and neither left anything to diagnose from afterwards.
Types and counts only. No message content, no tool inputs, nothing the model produced while reading an untrusted diff, which is why
show_full_outputstays off on a public repo. None of that is needed to say what shape a run had.Verification
Three fixtures, covering both observed shapes:
0 denials, nothing named2 denials, namesTodoWrite, Task4 denials4 denials, unchangedAll three also emit the census line.
What this still does not tell us
Why the run takes zero turns. Six denials on #196 and four on #182 make the allowlist the strongest hypothesis again, but a denial does not have to be fatal, and nothing here explains why inference is billed before any turn completes. The census is what should narrow that next time: the shape of a zero-turn run is currently unknown, and one run of it will now say so in the log.
Refs #185