Skip to content

An approval-gated CI run reads as 'no checks reported', which is neither the green nor the red Phases 1 and 4 branch on #129

Description

@dmccoystephenson

Problem

Phase 1's carried-over-PR handling in the generated-skill template branches on
whether the open PR's checks are green:

If the PR is still valid ({{EXTERNAL_SIGNAL_LABEL}} green, no conflicts), first
confirm a Phase 4 self-review was actually posted …

and Phase 4 step 1 anchors the self-review rubric on gh pr checks <number> --watch.
Both treat the anchor as binary: green, or red-and-must-be-fixed. GitHub has a
third state that neither covers.

When a commit is pushed to a PR branch by an account whose workflow runs require
manual approval — a first-time contributor, or an agent account such as
copilot-swe-agent — the run is created with conclusion action_required and
executes nothing (0s duration). The workflow never ran; no job produced a result.
The failure mode is that this does not surface as a red check. It surfaces as an
absence:

  • gh pr checks <number> prints no checks reported on the '<branch>' branch
  • gh pr checks <number> --watch returns that same line immediately rather than
    waiting, so a step written as "wait for the anchor to go green" returns at once
    having established nothing
  • gh run list --branch <branch> still shows a success run — but on an older
    head SHA, so a glance at the run history reads as a healthy branch

Nothing in Phase 1 or Phase 4 distinguishes "the anchor has not been run" from
"the anchor has no jobs configured for this branch". An unattended cycle can
therefore score the Phase 4 {{EXTERNAL_SIGNAL_LABEL}} rubric item off a run that
never executed, or read --watch returning instantly as confirmation.

Observed instance

Stephenson-Software/create-dev-loop PR #119. A merge-conflict resolution was
pushed to the branch by copilot-swe-agent as 26b149f; run 32547957642 on
that SHA completed action_required in 0s. gh pr checks 119 reported no checks
reported, while gh run list showed a success CI run two commits back on
9d7af16. The condition was resolved by pushing a further commit from an
already-permitted account, which retriggered the workflow with no approval gate
and produced a real green on the new head.

Suggested direction

Two changes, both small:

  1. Phase 1 carried-over-PR handling — add a third branch alongside the
    existing green / stale-or-conflicted cases and the TEMP: mid-revert check:
    an anchor run in a not-executed state (action_required, waiting, or
    stale) on the PR head is neither green nor red, and the PR is not ready for
    the Phase 5 jump. Where the loop's own account can push, adding a commit or
    re-running the workflow is enough to clear it; where it cannot, this is a
    hand-off naming the approval gate.

  2. Phase 4 step 1 — state that the anchor must be confirmed against the
    PR head SHA, and that no checks reported is not a pass. This parallels
    the rule already added by PR Require unique scratch filenames and define the structurally-red-anchor path #121 for a build tool that exits 0 having
    executed zero tests: a status that reports success without having run anything
    verifies nothing. The same sentence covers both, and the existing
    "Confirm the anchor actually executed tests" paragraph at Phase 3 is the
    natural place to point at.

A useful check for either: compare gh run list --branch <branch> --json headSha,status,conclusion
against the head SHA from gh pr view <number> --json headRefOid, rather than
trusting the aggregate status line.

Research grounding

RESEARCH.md §1 (Self-critique without an external signal is unreliable,
confidence: high). This is the same failure the finding's PR #121 Implementations
entry already records — an anchor that carries no information being read as one
that does — reached by a third route. The existing entry covers a job that has
never succeeded on the base branch and a build tool that reports success having
executed zero tests; a run that was created but never permitted to execute is a
further instance, and would extend that entry rather than needing a new finding.

This issue body was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).


drafted by Claude on behalf of Daniel Stephenson

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    template-ruleShould be promoted into create-dev-loop.md

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions