feat(seidroid-review): widen the trigger to every comment event, and let a caller name it - #103
Conversation
PR SummaryMedium Risk Overview Restores Reaction ack/answer/withdraw steps now use a new Adds a Reviewed by Cursor Bugbot for commit a31efa6. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
One risk this change adds, for a reviewer to weigh: Also: PR #100 edits |
There was a problem hiding this comment.
Widening the seidroid-review guard to pull_request_review_comment / pull_request_review, restoring allowed-bots and trigger-phrase, and refusing unhandled events all look correct: the per-event payload reads are consistently comment.* || review.*, the reactions endpoint is now routed per event, and the new run-guard.sh harness covers the admission paths thoroughly. No blockers; a few suggestions around a stale comment, the double duty comment_id now serves, and two gaps in how the new harness is wired into CI.
Findings: 0 blocking | 8 non-blocking | 4 posted inline
Blockers
- None at the file/PR level.
Non-blocking
contains(fromJSON(inputs.allowed-bots), ...)sits in the guard's jobif, and the input docs promise that malformed JSON "refuses it rather than denying quietly". That promise depends on the PR's own stated-but-unmeasured assumption that GitHub evaluates both operands of||. If GitHub short-circuits||(it is documented as returning the first truthy operand), a request from a human MEMBER would satisfy the person branch first, thefromJSONwould never be reached, and the run would proceed to be denied byAdmit the requestinstead — behaviourally still fail-closed, but not what the description or harness group 2 (a list that is not JSON->error) asserts. Worth one real run with a deliberately brokenallowed-botsto settle it, since gha.py cannot.- Every previously-ignored event now hard-fails the guard rather than skipping it. That is the intent of PLT-1161, but any existing caller that wires e.g.
workflow_dispatchormerge_groupat the same call site will flip from a silent skip to a red run on upgrade. Worth a line in the release notes for this file. - 4 suggestion(s)/nit(s) flagged inline on specific lines.
- 2 non-blocking pre-existing issue(s) listed below under pre-existing issues.
Pre-existing issues
- [suggestion] The guard's
issue_commentbranch has nogithub.event.actionfilter, unlike the two new events which are pinned tocreated/submitted. A caller that wireson: issue_commentwithouttypes: [created]can start a review by editing an old comment to add the phrase, and adeletedevent still carries the body, so a review starts against a comment that no longer exists (the acknowledgement then reacts to a missing object). Present on the base branch; the PR only makes the asymmetry visible by constraining the other two events. - [suggestion]
ai-assistant.yml'sreplyjob reserves only the exact body@seidroid review(github.event.comment.body != format('{0} review', inputs.trigger-phrase)), so@seidroid review closeand any body carrying the command amid prose are answered by both the assistant and the reviewer. This is pre-existing onissue_comment; note that it now applies on the diff-thread and review-body paths as well.
| # | ||
| # Emitted with the endpoint that reaches it, and held back when there is | ||
| # none: the two are read together by every step that reacts. | ||
| if [ -n "$comment_api" ]; then |
There was a problem hiding this comment.
[suggestion] comment_id now serves two consumers with different needs, and this change only reasons about one of them.
It is the reactable-object id for Acknowledge the trigger / Answer the request — correct to blank it when there is no reactions endpoint. But it is also --trigger-id on the driver (if [ -n "${TRIGGER_ID:-}" ]; then args+=(--trigger-id "$TRIGGER_ID"); fi in Drive session + collect verdict). Two consequences:
- A review-body trigger silently stops passing
--trigger-idto the driver, even thoughreview.idis known and was read intoCOMMENT_IDtwo steps up. - A diff-thread trigger now passes a pull request review comment id where the driver previously always received an issue comment id. Those are separate id spaces, so a value that used to identify a conversation comment can now collide with an unrelated one.
If --trigger-id is only a log discriminator both are harmless, but that is not stated anywhere here. Consider emitting the reactable id as its own output (e.g. reaction_id, paired with comment_api) and keeping comment_id as the always-populated trigger id.
There was a problem hiding this comment.
Taken, in the smaller form you suggested. The guard emits trigger_id — always populated, on every comment event — beside comment_id, and only Drive session + collect verdict moved to it. No step condition changed, so conditions.py's context model did not have to learn a new key.
On the two consequences: the first was real and is fixed — a review-body dispatch had silently stopped passing --trigger-id. The second is bounded by the fact that --trigger-id only labels the dispatch in the log, which the file does state, three lines below the code you flagged. A new group asserts which of the four outputs each consumer reads, so the driver taking the reactable id again fails an assertion.
| - name: Install the YAML reader | ||
| run: python3 -m pip install --quiet pyyaml | ||
| - name: Run the guard harness | ||
| run: test/seidroid-review/run-guard.sh |
There was a problem hiding this comment.
[suggestion] Group 16 of run-guard.sh reads .github/workflows/ai-assistant.yml and asserts a cross-workflow invariant (its trigger-phrase default equals this one's, and which of the two answers a given body). But this workflow's paths: filters only list seidroid-review.yml, workflow-test-self.yml and test/seidroid-review/** — so a PR that changes ai-assistant.yml's trigger phrase or its reply condition, which is exactly the drift group 16 exists to catch, never runs this job. Add .github/workflows/ai-assistant.yml to both paths: lists.
| # reasoning holds only while the two defaults agree. | ||
| check "the assistant takes the same phrase" '@seidroid' "$(input_of_file "$ASSISTANT" trigger-phrase default)" | ||
| claims() { # body -- true when the assistant's reply job would run | ||
| jq -nc --arg b "$1" '{github: {event_name: "issue_comment", |
There was a problem hiding this comment.
[suggestion] claims() hardcodes event_name: "issue_comment" and only populates event.comment, so group 16 measures the ai-assistant overlap on the one event that already had it. ai-assistant.yml's reply condition has parallel branches for pull_request_review_comment and pull_request_review with the same contains(body, phrase) && body != '<phrase> review' reservation — so this PR newly widens the double-answer cases (@seidroid review close, a command amid prose) onto the two events it adds, and none of that is covered. Parameterising claims() by event and adding the two review-side contexts would make the widened overlap measured rather than inferred, which is the stated point of the group.
There was a problem hiding this comment.
Taken. claims() is parameterised by event, every body runs on all three, and an empty review body is added — the case ai-assistant.yml names and this workflow reads as no command.
The overlap turns out identical on all three events, which is now measured rather than inferred. Group 16 went from 11 assertions to 33.
| # API call. An issue_comment payload carries no head repository, so the API | ||
| # answers there. Repository ids, not names, so a rename does not read as a | ||
| # fork. A null head repository reads as a fork, which is the safe reading. | ||
| # API call. No comment event carries a head repository, so the API answers on |
There was a problem hiding this comment.
[nit] "No comment event carries a head repository" is not accurate for two of the three. pull_request_review_comment and pull_request_review payloads both carry a full pull_request object, including pull_request.head.repo.id and pull_request.base.repo.id — only issue_comment does not. The code is still correct because the payload branch is gated on EVENT_NAME = pull_request, but as written the comment justifies an API call that two of the three comment paths do not actually need, and the HEAD_REPO_ID / BASE_REPO_ID env comment above ("Empty on every other event") is wrong for the same two events.
5c62183 to
42ffc3d
Compare
… a step that cannot post (#100) A run cancelled by a newer `@seidroid review` now clears the reactions it left on its own trigger comment, from a step that cannot post one. It left the 👀 there for good. Carries **PLT-1166**. **PLT-1159 comes out ruled out**, on the evidence below. Nobody can do it as written. The permission comment records the reason beside the scope the ticket asked to drop. ## PLT-1166 — the defect `Answer the request` took `!cancelled()`. Two `@seidroid review` comments in quick succession put both runs in one concurrency group under `cancel-in-progress`, so the newer one cancels the older. By then the older comment already wears the 👀 — the acknowledgement is the first step of the job. The newer run answers its **own** comment id, so nothing ever reads the older one again. The result: a comment that asked for a review, wears eyes, and never gets an answer. That is the defect PLT-1144 fixed on the no-verdict path, reached by the one path that fix does not cover. ## What ships `Answer the request` keeps `!cancelled()` — unchanged from the base. The withdrawal is a new step, last in the job: ```yaml - name: Withdraw the reactions on a cancelled run if: ${{ inputs.mode == 'review' && cancelled() && needs.guard.outputs.comment_id != '' && steps.verdict.outputs.posted != 'true' }} ``` `Post the verdict` gains `id: verdict` and a `posted` output so that last term can read it, and `Answer the request` gains `id: answer`. Nothing else in the workflow changes. ### Why not `always()` on `Answer the request` That was the first shape here and it was wrong. A step output persists once its step completes, so a cancellation landing any time after `drive` finishes leaves `check_path` and `verdict_produced` populated. `Answer the request` would read a real conclusion and post a thumb, while every publisher skips on `!cancelled()`. A thumb reads as an answer. That is worse than the stale eyes this PR set out to remove. ### Why the new step cannot state an outcome **It contains no POST.** The script lists this bot's reactions and deletes the three this workflow posts. No code path in it adds one. Nothing the step receives can therefore make it state an outcome. That property holds whatever its inputs are, which is what makes it structural rather than a matter of what a cancellation happens to look like. It reads no check file, no `verdict_produced` and no conclusion. ### What the `steps.verdict.outputs.posted` term is, and why it does not break that A cancellation can arrive once the verdict is already on the pull request — during thread resolution, say — and the thumb `Answer the request` posted answers it correctly. Withdrawing it there leaves a published review with no reaction on the request that asked for it, which reads as never answered. That is this step's own defect, one window later. The posting step's **`posted` output** separates the two. It is one boolean about another step, written from the comment POST's own result. No conclusion is in it. It says whether an answer already stands, never which answer it would be, so reading it gives the step nothing to state. Handing it `verdict_produced` instead would have restored the conditional reasoning above: that flag is true whenever the driver reached a verdict, including when nothing published. **Its outcome will not do, and that took a second pass to see.** `Post the verdict` runs under `continue-on-error` and tolerates a refused comment POST. Its failure path ends on a call whose failure it swallows. The step therefore exits 0, and its outcome reads `success` whether the verdict landed or not. The first version of this gate read that outcome. A refused POST followed by a cancellation then kept a thumb standing for a review nobody can see. That step already tracked the POST's result in a shell variable. It now writes it as an output. Anything but a posted verdict withdraws. A value the step cannot read therefore clears. It does not leave a thumb standing for a verdict that may not be on the pull request. ### The cross-run half of the same problem A re-run replays the trigger comment id. The comment can therefore already carry a thumb from an **earlier** run whose verdict is on the pull request. A re-run cancelled before it answered took that thumb along with its own eyes. The comment then ended bare while the verdict it asked for still stood. `Answer the request` gains `id: answer`, and the withdrawal reads its outcome to decide what this run may take: | `steps.answer.outcome` | what it means | withdrawn | |---|---|---| | `skipped` | this run never touched the comment, so a thumb there is an earlier run's | `eyes` only | | `success` | this run withdrew the stale thumb and posted its own, and published nothing | `+1 -1 eyes` | | `failure`, `cancelled`, unreadable | the step ran partway and most likely took the earlier thumb already | `+1 -1 eyes` | The structural property is untouched: still no POST, and an outcome is still four words about another step with no conclusion among them. **One case survives, and the comment states it rather than claiming it away.** A run answers, which withdraws an earlier thumb and posts its own. A cancellation then arrives before publishing, and the comment ends bare. The answer step already took the earlier thumb, so nothing at the end of the job can put it back. Knowing it happened would need a read of the pull request this step deliberately does not make. A cancellation lands during the driver far more often than in that gap. The step's comment records the limit instead of asserting the invariant outright. **One tension with the stated acceptance criterion, deliberately.** "Given a run cancelled by a newer request, its trigger comment carries no reaction from this bot" now fails on one path. A cancelled re-run leaves an earlier run's thumb. The criterion's intent holds. The comment does not wear 👀 with no answer coming, because the answer is on the pull request. Satisfying the literal wording would restore the defect above. I flag it rather than read the criterion loosely. ### Why last in the job, and what holds it there The runner evaluates a step's condition when it reaches the step. **Any step after the withdrawal is a step during which a cancellation leaves the eyes standing.** The runner already evaluated the withdrawal and skipped it by then. Placed last it also reads `steps.verdict.outputs.posted` after that step has reported. `conditions.py` checks the position rather than any one ordering, which covers a step appended later. Four mutations fail it. Move the withdrawal ahead of `Post the verdict`, ahead of the resolve step, or ahead of the no-verdict report. Or append a step after it. The id-ordering check caught only the first. In the other three `Post the verdict` still ran earlier. ### Why not the fix as named Gating the conclusion read inside the script needs the job status in the shell, and GitHub does not offer it there. `cancelled()` is readable only in a step or job `if`. `PipelineTemplateEvaluator.EvaluateStepEnvironment` calls `CreateContext(contextData, expressionFunctions)` with no `expressionState`, where `EvaluateStepIf` passes `step.ExecutionContext.ToExpressionState()`. And `CancelledFunction.EvaluateCore` reads `templateContext.State[nameof(IExecutionContext)]` and `ArgUtil.NotNull`s it. `StepsRunner` turns that throw into `CompleteStep(step, TaskResult.Failed)`, so `env: CANCELLED: ${{ cancelled() }}` fails the step on every run, before the runner evaluates its condition. actionlint refuses it too: `calling function "cancelled" is not allowed here. "cancelled" is only available in "jobs.<job_id>.if", "jobs.<job_id>.steps.if"`. The same holds for `run:`. ## The cancellation shapes, and which this covers | when the cancellation lands | what runs | outcome | covered | |---|---|---|---| | while queued, job never starts | nothing | no eyes were ever posted | n/a | | before `drive` completes | withdrawal | all three withdrawn, no thumb | yes | | **after `drive` completes** | withdrawal | **it cannot read the populated outputs** | yes | | while `Answer the request` runs | its `!cancelled()` re-test fires, the runner kills it, then the withdrawal | the withdrawal takes whatever it left | yes | | after the thumb, before the verdict published | answer, then withdrawal | the withdrawal takes the thumb: it would stand for nothing | yes | | **after the verdict published** | answer only | **thumb survives beside the published verdict** | yes | | **the verdict POST refused, then cancelled** | answer, then withdrawal | **thumb withdrawn: its outcome still reads success** | yes | | during the withdrawal step | withdrawal | its own condition is `cancelled()`, so the re-test keeps it alive | yes | | once the runner reached every step | answer only | a thumb this run earned stays | correct | | a re-run, cancelled before it answered | withdrawal | an earlier run's thumb stays, its eyes go | yes | | a re-run, answered then cancelled before publishing | answer, then withdrawal | bare comment, earlier verdict stands | **no** | | **runner process shutdown** (`RunnerShutdownToken`) | nothing | `StepsRunner` skips condition evaluation outright | **no** | Two rows are gaps. In the answered-then-cancelled row the answer step has already taken the thumb, so no later step can restore it. A hard kill of the runner leaves the eyes on the comment, and nothing inside a workflow closes that. ## The reaction table Each case declares two job states: the one the runner reached `Answer the request` in, and the one it reached the withdrawal step in. `success>cancelled` is a cancellation that arrived after the answer, so the answer step posts its own thumb and the fixture places nothing by hand. | case | states | verdict outcome | ran | left on the comment | |---|---|---|---|---| | success | `success>success` | success | answer | `bot:+1` | | failure | `success>success` | success | answer | `bot:-1` | | no verdict | `success>success` | skipped | answer | *none* | | neutral | `success>success` | skipped | answer | *none* | | **cancelled after `drive`, outputs populated** | `cancelled>cancelled` | skipped | withdraw | ***none*** | | cancelled before `drive` finished | `cancelled>cancelled` | skipped | withdraw | *none* | | cancelled mid-publish | `success>cancelled` | cancelled | answer+withdraw | *none* | | the verdict failed to post | `success>cancelled` | failure | answer+withdraw | *none* | | the outcome went unreported | `success>cancelled` | *empty* | answer+withdraw | *none* | | **cancelled after the verdict published** | `success>cancelled` | success | answer | ***`bot:+1`*** | | the same, beside a human's | `success>cancelled` | success | answer | `brandon:-1`, `bot:+1` | | success, human `+1 -1 eyes` | `success>success` | success | answer | human ×3, `bot:+1` | | failure, human ×3 | `success>success` | success | answer | human ×3, `bot:-1` | | no verdict, human ×3 | `success>success` | skipped | answer | human ×3 | | cancelled, human ×3 | `cancelled>cancelled` | skipped | withdraw | human ×3 | | stale `bot:-1` + `human:+1` | `success>success` | success | answer | `human:+1`, `bot:+1` | | the same, cancelled | `cancelled>cancelled` | skipped | withdraw | `human:+1` | | stale `bot:+1`, no verdict | `success>success` | skipped | answer | *none* | | `bot:rocket` from another workflow | `success>success` | success | answer | `human:+1`, `bot:+1`, `bot:rocket` | | the same, cancelled | `cancelled>cancelled` | skipped | withdraw | `human:+1`, `bot:rocket` | | the list call refused | `success>success` | success | answer | `human:+1`, `bot:+1`, `bot:eyes` + warning | | a delete refused | `success>success` | success | answer | `bot:+1`, `bot:eyes` + warning | | the add refused | `success>success` | success | answer | *none* + warning | | the list refused, cancelled | `cancelled>cancelled` | skipped | withdraw | `bot:eyes` + warning | | a delete refused, cancelled | `cancelled>cancelled` | skipped | withdraw | `bot:eyes` + warning | | the acknowledgement refused | `success>success` | success | answer | `bot:+1` | A human's reaction survives every path. A `bot:rocket` some other workflow left survives too, because each step deletes only the contents this workflow posts. And a thumb that answers a published verdict survives a later cancellation. ## PLT-1159 — ruled out, with the evidence The ticket's premise is that `ai-review.yml` reaches the same reactions through GraphQL `addReaction` under `pull-requests: write`. **It does not, and the workflow file does not decide the question.** **1. `ai-review.yml` does call both mutations, and its `permissions:` blocks do lack `issues`.** `preflight` is `contents: read` + `pull-requests: write` and calls `addReaction(content: EYES)`; `complete_review_reaction` is `pull-requests: write` alone and calls `addReaction(THUMBS_UP)` then `removeReaction(EYES)`. **2. But neither call uses `GITHUB_TOKEN`.** Both steps pass `github-token: ${{ steps.app-token.outputs.token || github.token }}`, and in production the App token wins. I checked live comments. Every reaction on an `@seidroid review` trigger in `sei-chain` belongs to `seidroid[bot]`, not to `github-actions[bot]`: ``` comment 5536974191 +1 by seidroid[bot] sei-chain#4088 comment 5544795940 +1 by seidroid[bot] sei-chain#4101 comment 5531689281 +1 by seidroid[bot] sei-chain#4095 comment 5493838638 +1 by seidroid[bot] sei-chain#4063 ``` The workflow's `permissions:` block does not bound an App installation token. Its own installation grant governs, and that App holds Issues: write — `ai-assistant.yml` posts `POST /repos/{o}/{r}/issues/comments/{id}/reactions` with the same token. ai-review is therefore **no evidence at all** about what `pull-requests: write` alone can do. It is the same class of wrong premise as the `enable-cursor: false` one. **3. GitHub documents no permission for any GraphQL mutation.** Not a gap in my reading — checked at the data source. In `github/docs`, `src/graphql/data/fpt/schema-reactions.json` gives `addReaction` and `removeReaction` the keys `name, id, href, description, isDeprecated, inputFields, returnFields, category` and no permission field. The public SDL (`docs.github.com/public/fpt/schema.docs.graphql`) carries only `@docsCategory(name: "reactions")`. The GraphQL guide's whole statement on the subject is that the API returns an error naming the permission it wanted. One route therefore remains to the requirement: make the call. **4. GitHub documents what REST requires, and the alias stops at the reaction.** `github/docs`, `src/github-apps/data/fpt-2026-03-10/server-to-server-permissions.json`: | endpoint | permission | |---|---| | `GET/PATCH/DELETE /repos/{o}/{r}/issues/comments/{id}` | listed under **both** `issues` and `pull_requests` | | `POST /repos/{o}/{r}/issues/comments/{id}/reactions` | `issues: write` **only** | | `DELETE /repos/{o}/{r}/issues/comments/{id}/reactions/{rid}` | `issues: write` **only** | | `POST /repos/{o}/{r}/pulls/comments/{id}/reactions` | `pull_requests: write` (a *review* comment — a different resource) | That file expresses "either permission" by listing an endpoint twice. A single listing on the reactions endpoints is therefore a distinction, not an omission. It confirms the claim the guard job already makes in prose. **Verdict.** The premise is void and the documentation says nothing. Nothing here can mint a fine-grained token scoped to `pull-requests` to test it. Shipping the drop blind would regress the defect this PR fixes. `continue-on-error` and a `::warning::` swallow a 403 on the reaction, so the eyes would stay on every comment and no run would fail. Ruled out. ### Two things worth keeping for whoever re-files it **`removeReaction` beats the REST loop, whatever the scope turns out to be.** The ticket assumed it takes a reaction node id. It does not. `RemoveReactionInput` is `{content: ReactionContent!, subjectId: ID!}`, and the subject is the *comment*: `IssueComment` sits in its `@possibleTypes`. It takes no actor input, so it can only ever remove the viewer's own reaction. That makes the human-scoping property structural rather than a `select(.user.login == $me)` filter. It also retires the hardcoded `me="github-actions[bot]"` login and the paginated list whose miss leaves eyes behind. **A third shape the ticket does not name looks likelier than either.** This workflow already mints an App token (`steps.identity.outputs.token`) and already computes `REVIEWER_LOGIN` from `app-slug`. Reacting under that identity needs no caller scope at all, and it is how production already posts these reactions. Two obstacles stand in the way. The acknowledgement runs before the mint, on purpose. And the mint is optional, so a `GITHUB_TOKEN` fallback keeps the scope required — unless a caller without App credentials may lose the reaction. ## The cost this change carries The list-and-delete block is now duplicated between `Answer the request` and the withdrawal step. `me="github-actions[bot]"` and the set of contents each step may delete are two copies, and a reader has to keep them in step by hand. Edit one and not the other and a reaction stays behind on whichever path lost the edit. That is the price of the separate step, and it buys the structural property: the withdrawing step has no POST. Sharing the block would mean one step doing both jobs, which is the shape that produced this PR's blocker. GitHub Actions offers no way to share a script between two steps without a checkout, and YAML anchors are not supported. Both harnesses cover both copies, so a drift fails rather than ships. The GraphQL `removeReaction` above is what would remove the duplication outright. It needs no login and no listing, so the whole block collapses to one mutation per content. ## Every reaction site Three steps, six calls, all in the `review` job, all on the ISSUE comments endpoint: | line | step | call | |---|---|---| | 1036 | `Acknowledge the trigger` | `POST .../issues/comments/{id}/reactions` (`eyes`) | | 2494 | `Answer the request` | `GET .../reactions --paginate` | | 2504 | `Answer the request` | `DELETE .../reactions/{rid}` | | 2519 | `Answer the request` | `POST .../reactions` (`+1` / `-1`) | | 3292 | `Withdraw the reactions on a cancelled run` | `GET .../reactions --paginate` | | 3304 | `Withdraw the reactions on a cancelled run` | `DELETE .../reactions/{rid}` | The withdrawal step has no `POST`, and that is the fix. `guard` reacts nowhere. `ai-assistant.yml` and `ai-review.yml` have their own sites; neither is in this workflow. ## Verification **Committed, not kept locally.** An uncommitted harness is how the first blocker survived a reading and seven mutations. Two additions to `test/seidroid-review/`, wired into `workflow-test-self.yml` as their own job so the placement check keeps its name. **`reactions.sh` — 62 assertions over 32 cases.** It runs the reaction steps under `bash`, extracted from the workflow on every run. No case names the step it runs. `conditions.py --select` names it, from the job state and the posting step's outcome, so the two layers cannot drift. The `gh` stub keeps the reaction list a comment carries and serves it through the step's own `--jq`. It honours idempotence per (user, content). It can refuse the list, a delete or the add. The acknowledgement's calls go to a separate log, so every count belongs to the step under test. The stub reports any call it cannot serve. **`conditions.py` — 76 assertions.** A step condition decides which reaction step runs in which job state, and a shell harness cannot see it. The model applies the runner's own rule: a condition naming none of `always`/`cancelled`/`failure`/`success` becomes `success() && (...)`. It treats a term it cannot decide as unknown rather than false. Two checks read the file rather than a table, so they cover a step added later: Both walk **every job's raw steps list** and search the **whole step**: - No step that can run on a cancelled job may reach `check_path` or `verdict_produced`. - Every `steps.<id>` a step reads must be a real id on an earlier step. - The withdrawal is the last step of the review job. Keyed off a name they dropped an unnamed step. `- uses: ...` with no `name:` is the usual shape, so the step most likely to arrive later was the one they could not see. The guard job already carries one. The id check also reached only `if`, which left the withdrawal's new `env` read unchecked. **Each fixture, mutation-tested.** A fixture that cannot fail the invariant is not a test of it: | mutation | which case fails | |---|---| | **the gate reads `steps.verdict.outcome` again** | **`THUMB GOES` on a refused POST, and the matching condition row** | | **the `skipped` arm takes all three** | **`EARLIER THUMB SURVIVES`, plus 3 more** | | **`id: answer` deleted** | **`reads steps.answer, which is no step's id`** | | **an unnamed `always()` step reading `verdict_produced`** | **`step 17 runs on a cancelled run and reads verdict_produced`** | | the `posted` term dropped | `THUMB SURVIVES` + 3 more + the condition row | | the same term inverted | 16 script cases, 5 condition cases | | `id: verdict` deleted | `reads steps.verdict, which is no step's id` | | the withdrawal moved ahead of `Post the verdict` | the position check, and `reads steps.verdict, which runs later` | | **the withdrawal moved ahead of the resolve step** | **the position check alone — the id check passes it** | | **the withdrawal moved ahead of the no-verdict report** | **the position check alone** | | **any step appended after the withdrawal** | **the position check** | | `Answer the request` back to `always()` | its cancelled row, and the sweep | | `check_path` interpolated inline into `run:` | the sweep — an `env`-only sweep passes it | | the withdrawal step deleted | `no step named 'Withdraw the reactions on a cancelled run'` | | the withdrawal not scoped to this bot | the human and `rocket` cancelled cases | | the withdrawal ignoring which contents it may take | `foreign-cancelled` loses a `rocket` | | the missing-conclusion arm clearing only the eyes | cancelled-with-stale-thumb, no-verdict-with-stale-thumb | | the answer's list not scoped to this bot | every human case | | the empty-reaction guard removed | every clear-only case posts a blank reaction | | the `VERDICT_PRODUCED` gate removed | no verdict thumbs the requester down | | the success arm no longer naming the eyes | eyes survive a green review | Only `conditions.py` catches `id: answer` deleted. `reactions.sh` derives that outcome itself, which is exactly why the id check has to exist. `conditions.py` also models `steps.verdict.outcome` beside `posted`, derived rather than passed. It reads `success` whenever that step reported at all, which is what the runner sees. A gate that regresses to the outcome therefore fails an assertion instead of crashing the model. **Five checks ran narrower than their own claim.** Mutating the thing each claimed to cover is what found them. - The haystack read `env` only, so an inline interpolation passed. - The model took an outcome as an argument and never checked the id existed. - The sweep keyed off a step name, so an unnamed step was invisible. - `run_case` exported each per-case knob without clearing it. An `ANSWERED_AS` override leaked forward and disarmed a later case. - The gate read an exit code that cannot express whether the verdict landed. The first three now read the file. `run_case` clears the fourth at the top of every case. The fifth reads an output written from the POST's own result. **actionlint**, base vs branch, both rule sets. `seidroid-review.yml`: 4 × `SC2102:info`, unchanged. Whole `.github/workflows`: 37 findings, identical after line-number normalisation. `workflow-test-self.yml`: 0. `shellcheck -S warning` clean on both new harness files. The incumbent harness still passes, now 271 assertions after #102. **Base checked three times, and no move trusted.** It went to `b1b51f8` (#97), then `2f58b80` (#101), then `98c2619` (#102). #97 rewrote 341 lines of this file and added the harness. #102 rewrote the resolve step and placement, and renamed the harness job. Each time all three harnesses and actionlint ran again on the new history. None of them carried a result from before it. I checked each rebase by reading back four things: the step order, the step ids, the gate, and the withdrawal step's POST count. Not by its exit status. The #102 rebase conflicted once, in the harness README, and I resolved it keeping both sides. `workflow-test-self.yml` merged cleanly: `place-findings` keeps #102's renamed display name `Place findings and resolve threads`, and the reaction job keeps its own. **One handoff for whoever lands second.** #103 replaces the hardcoded `issues/comments` with a `comment_api` output. Its author wrote it against **two** reaction steps. This branch leaves three, carrying six hardcoded paths rather than four: one in `Acknowledge the trigger`, three in `Answer the request`, two in the withdrawal step. Whichever of us rebases second has to reach all three steps. ## Not verified **Nothing here ran on a GitHub runner.** Five things stay unverified. That a real cancelled run reaches the last step of the job. That `steps.verdict.outcome` reads `success` on a run cancelled after that step completed. That `steps.answer.outcome` reads `skipped` rather than empty on a run cancelled before that step. That the runner's condition re-test kills `Answer the request` mid-flight, as its source says. And every permission claim above. The cancellation semantics rest on `actions/runner` source and on actionlint. `conditions.py` models the expression engine; it is not the engine. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Widening the seidroid-review guard to pull_request_review_comment / pull_request_review, restoring allowed-bots and trigger-phrase, and refusing unhandled events all look correct: every payload read is dual-keyed (comment.* or review.*), the reaction collection is routed per event with comment_id held back where no endpoint exists, and the new bin-guard/gha.py harness covers the paths well. No blocking issues; two notes about test fidelity and CI trigger coverage, plus one stale comment.
Findings: 0 blocking | 4 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
.github/workflows/seidroid-review.yml:1204-1206(unchanged line, but now incomplete because of this PR): thepermissions:comment still says a reaction on a PR comment "goes to the ISSUE comments endpoint, which pull-requests: write does not cover". Withcomment_apinow able to bepulls/comments, the diff-thread acknowledgement reachesPOST /repos/{o}/{r}/pulls/comments/{id}/reactions, which needspull-requests: writeand notissues: write. Both are granted so nothing breaks today, but anyone trimming permissions on the strength of that comment would silently break diff-thread acks. Worth extending it to name both collections.test/seidroid-review/gha.pyhardcodescancelled()→ false andsuccess()→ true, so the!cancelled()clause the review job's condition rests on is never exercised in either direction by group 4. Not worth modelling job status generally, but a one-line note in the docstring about what the model cannot answer would keep a later reader from over-trusting group 4.- 2 suggestion(s)/nit(s) flagged inline on specific lines.
| left = self.and_() | ||
| while self.peek()[1] == "||": | ||
| self.take() | ||
| right = self.and_() |
There was a problem hiding this comment.
[suggestion] or_ evaluates the right operand unconditionally (right = self.and_() runs before the truthiness test), and and_ does the same on line 103. GitHub's expression evaluator short-circuits: its Or/And nodes return on the first truthy/falsy operand and never evaluate the rest, so an error in an unreached operand never surfaces.
That divergence is load-bearing for one shipped assertion. run-guard.sh:190 feeds issue_comment / alice / User / MEMBER with allowed-bots: 'not json' and asserts the guard condition yields error. On a real runner the person branch — (...user.type) != 'Bot' && contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), ...author_association) — is already true, so fromJSON(inputs.allowed-bots) is never evaluated and the condition is simply true: the guard runs and admits the member. The malformed value would only fail the expression when the requester is a bot or a non-member, i.e. exactly the requests that were going to be denied anyway.
The production behaviour is still safe (Admit the request denies on unparseable allowed-bots, and run-guard.sh:379 covers that), so this is a test-fidelity issue rather than a guard defect. Suggest either modelling short-circuit evaluation and re-deriving the expected value for that case, or dropping the error expectation and stating in the docstring that eager evaluation is a deliberate over-approximation.
There was a problem hiding this comment.
Correct, and it caught a claim in my own report: I had listed "GitHub evaluates both operands of ||" as read-not-measured, and had it backwards.
gha.py short-circuits now — Or and And walk the unreached operand for its tokens but hold back its function calls, which is where fromJSON would refuse. Exactly the one assertion you predicted flipped, and only that one. It is re-derived per requester: a person yields true (admitted on association before the list is read), a bot yields error (the requester whose admission depends on parsing it), and an automatic review yields true. The docstring and --selftest both state it, 18 readings now.
| - name: Install the YAML reader | ||
| run: python3 -m pip install --quiet pyyaml | ||
| - name: Run the guard harness | ||
| run: test/seidroid-review/run-guard.sh |
There was a problem hiding this comment.
[suggestion] run-guard.sh reads a second workflow that this trigger does not watch: group 16 asserts ai-assistant.yml's trigger-phrase default equals @seidroid (line 523) and evaluates that file's reply job condition (line 529). The paths: filters above only list seidroid-review.yml, workflow-test-self.yml and test/seidroid-review/**, so a PR that changes ai-assistant.yml's default phrase or its reply condition will not run this job — the non-overlap claim the PR describes as "measured rather than argued" would go back to being unmeasured, and the break would only show up on some later unrelated PR. Add .github/workflows/ai-assistant.yml to both paths: lists.
42ffc3d to
6dfc97c
Compare
There was a problem hiding this comment.
Widening the seidroid-review guard to pull_request_review_comment / pull_request_review, restoring allowed-bots and trigger-phrase, and refusing unhandled events all look correct: every payload read is dual-keyed, the reaction collection is resolved per event and threaded through all three reacting steps (which already gate on comment_id), and the new run-guard.sh/gha.py harness covers the admission path well. No blocking issues; three non-blocking notes on caller-input robustness, a test-trigger gap, and two now-stale comments.
Findings: 0 blocking | 5 non-blocking | 3 posted inline
Blockers
- None at the file/PR level.
Non-blocking
reviewjob permissions comment (seidroid-review.yml:1207-1213) is now half the story: it explains only whyissues: writeis needed forissues/comments/{id}/reactions. The diff-thread path reacts onpulls/comments/{id}/reactions, which is covered bypull-requests: writeinstead. Worth a sentence so nobody prunes either scope on the strength of that comment alone.README.mdends without a trailing newline.- 3 suggestion(s)/nit(s) flagged inline on specific lines.
| contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), | ||
| github.event.comment.author_association || github.event.review.author_association)) || | ||
| ((github.event.comment.user.type || github.event.review.user.type) == 'Bot' && | ||
| contains(fromJSON(inputs.allowed-bots), |
There was a problem hiding this comment.
[suggestion] fromJSON(inputs.allowed-bots) sits in the job condition, so a value it cannot parse fails the expression and takes the whole run down rather than denying. The documented intent is "malformed JSON is a caller error", but the most likely accident is an empty value, not malformed JSON: workflow_call defaults apply only when the input is omitted, so a caller wiring allowed-bots: ${{ vars.SEIDROID_ALLOWED_BOTS }} with the variable unset passes '', and fromJSON('') is not [].
That asymmetry is worth closing — Admit the request reads the same value through jq and simply denies, and trigger-phrase/guidelines-file both warn-and-fall-back. A one-token fix keeps the loud failure for genuinely malformed JSON while treating an unset input as the documented default:
contains(fromJSON(inputs.allowed-bots || '[]'),There was a problem hiding this comment.
Taken. The condition reads fromJSON(inputs.allowed-bots || '[]') now, so an unset input takes the documented default and denies every bot, while a non-empty non-JSON value still fails loudly.
The fix holds under either evaluation order, which turned out to matter: the sibling finding on gha.py showed my eager model was wrong, and under the runner's real short-circuiting a bad list only ever reached a bot request. Four cases pin it — empty denies a bot even with MEMBER association, admits a person, and leaves an automatic review alone.
| - name: Check the step conditions | ||
| run: python3 test/seidroid-review/conditions.py .github/workflows/seidroid-review.yml | ||
|
|
||
| guard-admission: |
There was a problem hiding this comment.
[suggestion] run-guard.sh group 16 asserts a cross-file invariant against .github/workflows/ai-assistant.yml — that its trigger-phrase default is still @seidroid, and what its reply condition claims of each body. But ai-assistant.yml is not in either paths: filter above, so editing that file runs none of this. The invariant can break silently and then surface as a red Guard the request check on the next unrelated seidroid-review.yml change, pointing at the wrong PR.
Adding '.github/workflows/ai-assistant.yml' to both paths: lists makes the check fire where the breakage happens.
There was a problem hiding this comment.
Taken — ai-assistant.yml is in both paths: lists.
I audited every file the three harnesses read, as you asked. It was the only one outside the filter: run.sh and reactions.sh read seidroid-review.yml alone, and conditions.py takes its target from the CI command line, which names the same watched file. So no other cross-file assertion has this shape.
| # API call. An issue_comment payload carries no head repository, so the API | ||
| # answers there. Repository ids, not names, so a rename does not read as a | ||
| # fork. A null head repository reads as a fork, which is the safe reading. | ||
| # API call. No comment event carries a head repository, so the API answers on |
There was a problem hiding this comment.
[nit] "No comment event carries a head repository" holds for issue_comment only. The pull_request_review_comment and pull_request_review payloads both carry the full pull_request object, including head.repo and base.repo — which is why HEAD_REPO_ID/BASE_REPO_ID (lines 833-834) are in fact populated on those two events and then deliberately ignored by the EVENT_NAME = pull_request branch below.
No behavioural bug: the API read returns the same answer. But the two new paths spend an avoidable authenticated call per request, and the comment states something about the payload shape that a later reader would be wrong to trust. Either widen the payload branch to "a payload that carries a base repository id" (which would cover all three) or correct the sentence to name issue_comment as the one event that needs the API.
There was a problem hiding this comment.
Corrected the sentence rather than widening the branch, and the reason decided it: the label check twelve lines below reads the same GET /repos/{owner}/{repo}/pulls/{n} endpoint unconditionally on all three comment paths. Reading the payload here would drop one of two identical round trips and neither the failure mode nor the dependency, so the saving is a round trip, not a refused review.
The comment now names issue_comment as the one event that needs the API, says the other two carry head.repo.id and base.repo.id, and records what a payload-keyed branch would have to preserve — so the saving stays available to whoever wants it.
The bigger one is collapsing those two reads of the same endpoint into one, which is available on all four paths rather than two. That belongs in a ticket: it moves the fork check, which is the highest-consequence control in the guard, and I would rather not move it in a change that has rebased three times.
…let a caller name it
Four changes to the guard's admission, shipped together because all four edit
one region of the file.
PLT-1147. The guard admitted `pull_request` and `issue_comment` only, so
`@seidroid review` typed in a diff thread or in a review body did nothing: no
reaction, no run, no log line the requester could find. Both events are admitted
now, and every read in `parse` and `Admit the request` takes the comment key or
the review key, whichever the event populated. The two diff-side events are held
to their creating action, because a `dismissed` review replays the body of the
review it dismisses.
The guard emits three facts about the request, not one. `comment_id` is the
reactable-object id and `comment_api` the collection it lives in; the three
reacting steps read both, so all three reach the object that asked. `trigger_id`
is the id for the log, populated on every comment event including the one that
can be reacted on nowhere, and the driver's --trigger-id takes it. GitHub
publishes no reactions endpoint for a pull request REVIEW, so a command in a
review body starts a review, earns no reaction, and says so in the run log.
PLT-1153. `allowed-bots` returns, a JSON array of exact logins defaulting to
`[]`. It is checked in the guard's job condition and again in `Admit the
request`, exactly and case-insensitively, as ai-review.yml checks it. A listed
bot skips the team read and is held to every other rule. The condition reads
`inputs.allowed-bots || '[]'`, because a workflow_call default applies only to
an input the caller OMITS: a caller passing an unset variable passes the empty
string, and fromJSON('') is not `[]`. Empty takes the documented default and
denies every bot; a non-empty value that is not JSON still fails loudly.
PLT-1161. An event this workflow does not handle is refused by name, before the
identity mint and the secret check. `pull_request_target` is refused apart, with
its reason: it runs with the base repository's secrets and a writable token over
a head this workflow did not check out.
PLT-1164. `trigger-phrase` returns, defaulting to `@seidroid`, and the pattern
is built from it. The optional `@` stays: whole-line anchoring keeps it safe, and
ai-assistant.yml claims a body only in the `@` form, so the bare form reaches
this workflow alone -- measured, on all three events. The phrase reaches a
`grep -E` pattern, so its shape is constrained rather than escaped: an optional
`@` and then letters, digits, `_` and `-`, none of which is an ERE
metacharacter. A phrase outside that shape falls back to the default with a
warning.
test/seidroid-review/ gains a third harness, `run-guard.sh`: 241 assertions over
the five steps it reads out of the workflow, the two job conditions, the
per-event env mappings, which guard output each consumer reads, the declared
input defaults, and what ai-assistant.yml claims of the same body on each of the
three events. It extracts the two reaction steps under names of its own, because
reactions.sh extracts the same two and a shared path lets one overwrite the
other.
`gha.py` models the runner's SHORT-CIRCUIT evaluation. Or and And return on the
first truthy or falsy operand and never evaluate the rest, so a `fromJSON` an
operand nothing reaches never runs. Modelled eagerly, one assertion here stated
the opposite of what a real event does.
reactions.sh sets COMMENT_API in its own defaults -- without it every extracted
step dies on an unset variable under `set -u` -- and gains a group covering the
collection each of the three steps reaches. 62 assertions to 77.
workflow-test-self.yml watches ai-assistant.yml, because run-guard.sh states an
invariant about it: without that path an edit there breaks the invariant and the
break lands on the next unrelated pull request.
PLT-1147 PLT-1153 PLT-1161 PLT-1164
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6dfc97c to
a31efa6
Compare
|
Taken. The permissions comment explains both scopes now, one per collection: Separately, on the measurement discipline: five processes were reported running concurrently. They were my five polling shells, not sweeps — The instruction stands regardless and I followed it: every poller killed, count verified zero, tree confirmed identical to |
There was a problem hiding this comment.
Widens the seidroid-review guard to pull_request_review_comment and pull_request_review, restores allowed-bots and trigger-phrase, and refuses unsupported events — all backed by a new 241-assertion harness that evaluates the shipped conditions and step scripts rather than restating them. The per-event payload reads, the comment_api split (issues/comments vs pulls/comments), the comment_id/trigger_id separation, and the fail-closed bot check all check out; only two non-blocking points.
Findings: 0 blocking | 4 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
gha.pymodelscancelled()/failure()as constantfalseandsuccess()/always()as constanttrue. That's fine for the guard/review admission conditions it is used on, but the module docstring lists them alongside the genuinely modelled operators without saying they are fixed, so a later assertion written against!cancelled()would pass vacuously. A one-line note next toapply_functionwould keep the next author from trusting it further than it goes.- 2 suggestion(s)/nit(s) flagged inline on specific lines.
- 1 non-blocking pre-existing issue(s) listed below under pre-existing issues.
Pre-existing issues
- [suggestion]
.github/workflows/seidroid-review.ymlguard condition: theissue_commentbranch is not held togithub.event.action == 'created', unlike the two branches this PR adds.issue_commentalso fires oneditedanddeleted, and both payloads carry the full comment body — so editing (or deleting) a comment that already contains@seidroid reviewre-runs the review. This is exactly the replay hazard the PR guards against forpull_request_review.dismissed; the workflow's comment notes the asymmetry ("issue_comment is left as it stands") but does not close it. Callers that wiretypes: [created]themselves are unaffected.
| Matched case-sensitively. `@Seidroid review` starts no review. | ||
| required: false | ||
| type: string | ||
| default: '@seidroid' |
There was a problem hiding this comment.
[suggestion] trigger-phrase is now caller-settable and the grep pattern is built from it, but two runtime notices still hardcode the default:
- line 1103:
deny "... comment @seidroid review to ask for one" - line 928:
... An automatic pull_request review and @seidroid review close do not reach this check"
A caller that sets trigger-phrase: '@mybot' gets told to type a phrase that this workflow's own parse will not match — and line 1103 is the primary "how do I ask for a review" guidance, emitted on the once-per-PR deny path. Both steps already have the phrase available (parse reads it into TRIGGER_PHRASE; Admit the request would need it added to env:), so interpolating it there would close the gap. PLT-1164 built the pattern from the input; these two messages are the remaining hardcoded readings of it.
| # it carries the @ form, so the bare form reaches this workflow alone. | ||
| cmdline="$(printf '%s\n' "$cmd" \ | ||
| | grep -m1 -E '^[[:space:]]*@?seidroid[[:space:]]+review([[:space:]]+close)?[[:space:]]*$' || true)" | ||
| | grep -m1 -E "^[[:space:]]*@?${phrase}[[:space:]]+review([[:space:]]+close)?[[:space:]]*$" || true)" |
There was a problem hiding this comment.
[suggestion] Worth reconsidering now that the trigger reaches diff threads and review bodies. The comment above says whole-line anchoring keeps a comment that discusses the command from matching because it "has other words on the line" — but that does not hold for a markdown code block. A fenced line @seidroid review, or a 4-space-indented @seidroid review, matches ^[[:space:]]*@?${phrase}... exactly, and GitHub renders it without notifying anyone, so the author gets no hint they fired a review.
This existed on issue_comment, but the two new events move it to where it is most likely to fire: a diff-thread comment or review body on this very workflow file or on test/seidroid-review/README.md, both of which quote the command verbatim. The cost is a real sandbox and model spend per occurrence; the concurrency group only collapses concurrent ones.
If you want to keep the optional @ (and the PR's argument for it is reasonable), rejecting a line indented by four or more spaces would remove the indented-block case cheaply. Fenced blocks would need the parse to track ``` state, which is a bigger change and may not be worth it.
Four tickets, one region: the guard's job condition, its
parseandAdmit the requeststeps, and the twoworkflow_callinputs they read.PLT-1147 — accept
pull_request_review_commentandpull_request_review.Both events are admitted. Every read in
parseandAdmit the requesttakes thecomment key or the review key, whichever the event populated: a review body names
its author under
review.userand its id underreview.id, and a step readingcomment.*alone saw an empty body there and refused in silence. The twodiff-side events are held to their creating action, because a
dismissedreviewreplays the body of the review it dismisses — a caller wiring that type would
re-review on every dismissal.
PLT-1153 — restore
allowed-bots. A JSON array of exact logins, default[]. Checked in the guard's job condition, so an unlisted bot starts no runner,and again in
Admit the request. Exact and case-insensitive both times, asai-review.yml checks it. A listed bot skips the team read — a bot is not a team
member — and is held to the fork check, the skip label and the command grammar.
The job condition now admits a person on
author_associationand a bot only bylogin, because association does not discriminate a bot: one with write access
carries MEMBER like anyone else.
PLT-1161 — refuse an unsupported event. A first step names the event that
arrived and the four this workflow handles, and exits 1. It runs before the
identity mint and before the secret check, so a mis-wired caller spends no
credential. The guard's condition gained a clause admitting an unsupported event
for exactly that step: without it the job is skipped, every job after it is
skipped, and the run reports success having done nothing.
pull_requestisexcluded from that clause — a
pull_requestclose skips the guard deliberately,and the review job reads that skip as its own trigger.
pull_request_targetis refused apart, with its reason: it runs with the baserepository's secrets and a writable token over a head this workflow did not check
out. Nothing here checks anything out today; the refusal is the control that does
not depend on that staying true.
PLT-1164 — restore
trigger-phrase. Default@seidroid, and the pattern isbuilt from it rather than hardcoded — in the command grammar and in the
repository-target refusal beside it.
The two decisions the tickets asked for
The optional
@stays. A person who types the phrase without the mentionstill means it, and the wider form costs nothing here. Whole-line anchoring is
what makes it safe, and it is intact. The non-overlap with
ai-assistant.ymlisnow measured rather than argued: that workflow's reply condition requires
contains(body, '@seidroid'), so a bareseidroid reviewreaches this workflowalone. Group 16 of the harness evaluates the assistant's own condition beside the
parse for five bodies and records which tool answers each.
Two bodies both tools answer today, and both predate this change:
@seidroid review close, and a body carrying the command on its own line amid prose. Theassistant reserves the exact body only, and neither of those is it. Whole-line
anchoring is what admits the second — and it is also what keeps
Do we need @seidroid review here?from starting a review, so the overlap is the price ofthe property the ticket told me not to lose. The harness asserts the present, so
a later change that closes either overlap fails a case and has to re-read it.
The phrase's shape is constrained, not escaped. After stripping one leading
@, the phrase must be letters, digits,_and-. None of those is an EREmetacharacter, so the pattern carries the phrase verbatim with no escaping.
Anything else falls back to
@seidroidwith a warning, which is whatguidelines-filedoes with a name it cannot trust. Escaping would have to coverevery ERE metacharacter correctly forever; a character class is one thing to
read. Two harness cases show what the constraint buys: with
@my.bot,@myXbot reviewdoes not match; with@a|b, the linea note about the diffdoes not match. Unconstrained, the
|would split the pattern into^[[:space:]]*@?a— which every line starting withamatches.One deliberate step outside the stated region
Three reaction steps build their reactions URL from a new guard output,
comment_api, instead of a hardcodedissues/comments:Acknowledge the trigger(one path),Answer the request(three) and, since #100,Withdraw the reactions on a cancelled run(two). Six paths, threeenv:keys.The endpoint differs per event —
issues/comments/{id}/reactionsfor aconversation comment,
pulls/comments/{id}/reactionsfor a diff-thread one — andwithout this PLT-1147's acknowledgement would post to a path that holds no
object, and the two steps that withdraw it would look for it somewhere else
again. That is the trap the ticket names, and it cannot be fixed from inside the
guard alone. The review job already holds both scopes: GitHub grants the first to
Issues and the second to Pull requests.
Two properties of #100 survive the edit, and both are asserted rather than
argued. The withdrawal step is still the last step of the review job — index
16 of 17, and
conditions.pychecks the position rather than one ordering. Andit still contains zero POSTs:
-X POST,--method POSTand-f contenteach appear 0 times in it,
DELETEis the only verb it names, andreactions.shasserts the POST count. Adding an
env:key changes neither.repos/{owner}/{repo}/issues/comments/{id}in two other steps is untouched:those delete comments this workflow posted on the conversation, not the trigger.
One acceptance criterion that REST cannot meet
GitHub publishes no reactions endpoint for a pull request review. Only the
GraphQL schema makes a review reactable, and PLT-1159 already proposed that route
and was declined. So a command in a review body starts a review,
comment_apiand
comment_idboth go out empty, both reacting steps skip on their existingcondition, and a
::notice::in the run log says the review started and why noreaction landed. The review, the verdict comment and the inline findings all
still arrive. A diff-thread comment gets the full acknowledgement.
That is a read claim, not a measured one — see below.
The review round
Seven findings taken.
An unset
allowed-botsno longer takes the run down. Aworkflow_calldefault applies only to an input the caller OMITS, so
allowed-bots: ${{ vars.SOMETHING }}with that variable unset arrives as'',and
fromJSON('')is not[]. The condition readsfromJSON(inputs.allowed-bots || '[]'), so empty takes the documented defaultand denies every bot, while a non-empty non-JSON value still fails loudly. The
fix holds under either evaluation order, which turned out to matter — see below.
gha.pyshort-circuits, because the runner does. Or and And return on thefirst truthy or falsy operand and never evaluate the rest. My model evaluated
eagerly, and one shipped assertion therefore stated the opposite of what a real
event does: with a malformed list and a human MEMBER, the person branch is
already true, so
fromJSONis never reached and the guard admits. Re-derived perrequester — a person yields
true, a bot yieldserror(the requester whoseadmission depends on parsing the list), an automatic review yields
true. Thiscorrects a claim in my own earlier report, where I had listed eager evaluation as
read-not-measured and had it backwards.
Group 16 now measures the overlap on all three events.
claims()was keyedto
issue_comment, so it measured the division of labour on the one path thatalready had it and inferred the two this branch adds. It takes an event now, and
every body runs on all three plus an empty review body. The overlap is identical
on all three — measured, not reasoned. 11 assertions to 33.
ai-assistant.ymlis inworkflow-test-self.yml'spaths:. Group 16 statesan invariant about that file, so an edit there could break it and surface later as
a red
Guard the requeston an unrelated change. I audited every file the threeharnesses read: it was the only one outside the filter, and
conditions.pytakesits target from the CI command line, which names a watched file. No other
cross-file assertion has this shape.
The log id and the reactable id are two facts.
comment_idis the reactableobject and goes out empty where nothing can react; the driver's
--trigger-idwas reading it, so a review-body dispatch had silently stopped carrying a label.
The guard emits
trigger_idbeside it, always populated, and onlyDrive session + collect verdictmoved to it — no step condition changed, soconditions.py's context model needed nothing. A new group asserts which of thefour outputs each consumer reads.
A comment claimed something false about the payload. "No comment event
carries a head repository" holds for
issue_commentalone;pull_request_review_commentandpull_request_reviewboth carrypull_request.head.repo.idand.base.repo.id. I corrected the sentence ratherthan widening the branch, because the label check twelve lines below reads the
same
GET /repos/{owner}/{repo}/pulls/{n}endpoint unconditionally on all threecomment paths: reading the payload here would drop one of two identical round
trips and neither the failure mode nor the dependency. The comment now names the
one event that needs the API and records what a payload-keyed branch would have
to preserve. The bigger saving is collapsing those two reads of one endpoint into
one, available on all four paths — that belongs in a ticket, because it moves the
fork check.
The permissions comment explains both scopes, one per collection, and names
what pruning either costs: the reaction fails on the path that scope serves, and
all three reacting steps treat a lost reaction as a courtesy and only warn.
Verification
Everything below ran on this machine. Nothing ran on a GitHub runner.
test/seidroid-review/run-guard.shis new, beside the placement harness. Itreads five steps out of the shipped YAML by name or id, runs them under
bashagainst a
ghstub of its own, and evaluates the two job conditions, theper-event
env:mappings and the declared input defaults with a newgha.py.reactions.shneeded the change, not just the extra cases. Itsrun_casedid notexport
COMMENT_API, so every extracted step died on an unset variable underset -uand 25 of its 62 assertions failed with every API count at zero. Thedefault is set there now, and a group varies it: three steps, six paths, and a
pulls/commentscase asserting nothing reachedissues/comments.conditions.pywent from 76 to 77 on its own. Two of its checks walk every job'sraw steps list, so the refusal step this branch adds to the guard job earns one
more assertion without anything being written for it.
gha.pymodels four GitHub expression semantics the conditions rest on:case-insensitive string comparison,
||and&&yielding one operand each,both short-circuiting, and
containsover an array testing membership ratherthan substring.
--selftestchecks all eighteen readings, and group 0 of the runfails if any is wrong. The model is read from GitHub's published semantics; it is
not measured against a runner.
Mutation check. 41 mutations of the shipped workflow, applied one at a time,
each killed at least one assertion. 0 alive, 0 skipped. The sweep runs all
four harnesses per mutation, because one edit spans steps three of them cover —
a mutation only
reactions.shorconditions.pycan see would have survived asweep that ran the guard harness alone. Four of the 41 cover this review round:
dropping the empty-input fallback, holding
trigger_idback with the reactableid, and pointing either the driver or the acknowledgement at the other's id.
Among them: dropping either new event from the condition, dropping the
creating-action gates, reading
allowed-botsas a string rather than JSON,dropping the
pull_request_targetarm, emitting the id where no endpoint reachesit, dropping the phrase's shape check, hardcoding the phrase in either pattern,
dropping the whole-line anchors, requiring the
@, reading only thecomment.*payload keys in either step, matching a listed bot by substring or
case-sensitively, letting the once-per-PR gate reach a comment, and applying the
requester check to a teardown.
Six mutations cover the six reaction paths — one in the acknowledgement, three in
the answer, two in the withdrawal — and each is killed by at least two
assertions. Two more cover #100's properties: appending a step after the
withdrawal is killed by
conditions.py's position check, and adding a POST tothe withdrawal step is killed 21 times by
reactions.sh.actionlint, before and after. Base
2f7efad, all workflows:This branch, all workflows: the identical set, finding for finding — compared as
rule + code, not just as a count, and identical per file too.seidroid-review.yml's own four are the pre-existingSC2102:info. The rest arein
ai-assistant.yml(3),ai-review.yml(4),release-check.yml(22) andrelease-publish.yml(4), all untouched.workflow-test-self.ymllints clean. Both files parse under PyYAML.Rebase note
Written against
3544bf5; rebased three times as the base moved, tob1b51f8(#97),
98c2619(#101, #102) and2f7efad(#100). Head isa31efa6.The third rebase conflicted in four files:
seidroid-review.yml— one hunk, inAcknowledge the trigger: fix(seidroid-review): withdraw the reactions on a cancelled run, from a step that cannot post #100rewrote the comment above the POST while this branch rewrote the URL below it.
Union.
workflow-test-self.yml— a three-way union. Three jobs now, underdistinct names:
Place findings and resolve threads,The reaction steps,Guard the request.README.md— two hunks; one document with a section per harness..gitignore— union of three extractor lists.One collision the conflict markers did not show:
reactions.shandrun-guard.shboth extractAcknowledge the triggerandAnswer the request,and both wrote them to
ack.shandanswer.sh. Running both would have oneoverwrite the other's extraction. This branch is the newcomer, so it moved:
guard-ack.shandguard-answer.sh. The README now says which harness askswhat of those two steps.
Everything was re-run on the rebased history rather than carried forward: all
four harnesses, the full mutation sweep, the actionlint comparison against the
new base, and the group counts, recounted from the shipped file (unchanged this
time — 204 over the same seventeen groups).
The sweep caught itself. Its first pass on this base reported 36 killed and
one SKIP:
M28, which hardcodes the issue collection in the answering step'sread. After this branch routed the withdrawal step through
COMMENT_API, thatstep's read became byte-identical to the answering step's, so
M28's anchormatched twice and stopped applying. A sweep that only counted kills would have
read 36/36 and looked clean.
M28now carries the comment line above the call,which the two steps do not share, and is killed by four assertions across two
harnesses.
What rests on reading rather than measurement
pull_request_review_commentascreatedandpull_request_reviewassubmittedfor a new request, and that a dismissalreplays the dismissed review's body.
fromJSONover a non-JSON input fails the expression rather thanevaluating false, and that GitHub evaluates both operands of
||.gha.pymodels. A case here can only be as right as that model.if:still requires the steps before it to havesucceeded, which is what makes the refusal skip the identity mint. The file's
own comments already rest on this.
Nothing in this branch has been exercised by a real event on a runner.
Case table
Guard the request: 241 assertions — 95 runs of an extracted step script, and 71call sites evaluating a shipped condition,
env:mapping or declared input.Recounted from the shipped file.
gha.pyusesallowed-botsin the job condition, malformed and unsetallowed-botsai-assistant.ymlclaims of the same body, on each eventThe reaction steps: 77 + 77.reactions.shcarries 15 assertions over fourcases for the collection each of the three steps reaches;
conditions.pygains 1for the guard's new refusal step, which its file-wide sweep picks up on its own.
Place findings and resolve threads: 271, untouched.PLT-1147 PLT-1153 PLT-1161 PLT-1164
🤖 Generated with Claude Code