Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 19 additions & 15 deletions .github/workflows/pr-risk.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
name: PR Risk Grade (reusable)

# Reusable ADVISORY PR risk grader — the shadow-check rung of the PR risk-grading ladder.
# Grades every PR event into a tier R0 (safest) .. R3 (riskiest) and syncs ONE label
# (`risk:R0` .. `risk:R3`, or `risk:ungraded` when an input was unreadable). That label is
# Grades every PR event into a tier low (safest) .. xhigh (riskiest) and syncs ONE label
# (`risk:low` .. `risk:xhigh`, or `risk:ungraded` when an input was unreadable). That label is
# the entire product: nothing is gated, nothing is blocked, nothing merges, no comment is
# posted. Humans look at the label and agree or disagree; disagreement is recorded by adding
# the `risk-dispute` label (which this workflow never touches) plus a comment saying why.
# Existing maps and label_map inputs may still use R0..R3 as deprecated aliases; the grader
# normalizes them to low..xhigh before grading, and all new output uses the canonical names.
#
# grade = worst(path_floor, provenance, reversibility) — three deterministic axes; the worst
# tier wins, so no axis can move a PR into a safer lane than another axis put it. No LLM, no
Expand Down Expand Up @@ -75,7 +77,7 @@ name: PR Risk Grade (reusable)
# read-only GITHUB_TOKEN that the caller's `permissions:` block cannot elevate, so the label
# write — this workflow's entire product — would 403. On a dispatch the token is writable, so
# the reason evaporates. Fork RISK is unaffected either way, because `external` is derived
# from the API's `isCrossRepository`, never from the actor, and forks grade R3 with no
# from the API's `isCrossRepository`, never from the actor, and forks grade xhigh with no
# exceptions. DEPENDABOT-TRIGGERED RUNS need no such hatch and are graded on BOTH paths: the
# caller pattern below deliberately carries no `github.actor != 'dependabot[bot]'` clause,
# because the caller's `permissions:` block elevates Dependabot's read-only token and this
Expand All @@ -97,7 +99,7 @@ name: PR Risk Grade (reusable)
# it is not in that rollup at all and a settled PR reads its true state (`SUCCESS`, nothing
# pending) on the first poll. It is still not FREE: `0` breaks out after a single read, ahead
# of both the not-yet-registered grace window and the "require a settled reading to repeat"
# confirmation, so a target someone pushed to minutes ago lands the honest R2 floor. `1` costs
# confirmation, so a target someone pushed to minutes ago lands the honest high floor. `1` costs
# one 15s backoff and a second read per PR and keeps the confirmation. Prefer `1`, not `0`.
#
# Batch (`pr_numbers`) grades one target at a time and ONE UNREADABLE PR NEVER ABANDONS THE REST:
Expand All @@ -114,8 +116,9 @@ name: PR Risk Grade (reusable)
# last-writer-wins with still EXACTLY ONE `risk:*` label — possibly the staler tier, which gates
# nothing meanwhile and which the next grade re-syncs (on the LAST push there is no next grade, so
# re-dispatch on `pr_number` if a final grade looks wrong). It can no longer leave two contradictory
# labels on a PR under one `label_map`; remapping `label_map` orphans the old names, which is a
# one-time repo-side cleanup. What the shape does cost is a narrower residual: the PUT is built from
# labels on a PR under one `label_map`; remapping `label_map` orphans custom old names, which is a
# one-time repo-side cleanup (the known former defaults risk:R0..risk:R3 are retired automatically).
# What the shape does cost is a narrower residual: the PUT is built from
# a snapshot read, so a NON-owned label added by someone else in the read→PUT window is dropped
# (`risk-dispute` included) and one removed in it is resurrected. That window opens only on a run
# that actually changes the grade and is roughly one API round-trip — about three on the first
Expand All @@ -139,7 +142,7 @@ name: PR Risk Grade (reusable)
# ENROLL THIS AS ITS OWN WORKFLOW, not as one job inside an existing CI workflow. The grading
# job is part of the check rollup it reads, so it excludes its own RUN from that rollup; a job
# sharing a run with the rest of CI therefore excludes its siblings too and lands on the honest
# R2 floor instead of grading off a full rollup. (It can never grade a red PR green either way
# high floor instead of grading off a full rollup. (It can never grade a red PR green either way
# — a FAILING check is never excluded.)
#
# Caller pattern (consumer repo, .github/workflows/ci-pr-risk.yml):
Expand Down Expand Up @@ -175,7 +178,7 @@ name: PR Risk Grade (reusable)
# # guard, not a risk judgement: a fork's `pull_request` run gets a read-only GITHUB_TOKEN,
# # the `permissions:` block below CANNOT elevate it (the Dependabot carve-out below does
# # not extend to forks), so the label write would 403 and the check would go red. Fork
# # RISK is untouched either way — forks grade R3 from the API's own fork flag, never from
# # RISK is untouched either way — forks grade xhigh from the API's own fork flag, never from
# # the actor — so a fork PR is simply ungraded-by-absence here; grade it by dispatching on
# # `pr_number`, or enroll with `pull_request_target` instead, which is safe by construction
# # for this workflow (see the fork paragraph above). Keep the clause BEHIND the event test:
Expand Down Expand Up @@ -319,7 +322,7 @@ on:
bot with no runbook registry entry grades as human — identity alone
never buys trust. LOAD-BEARING, not a hint: a listed login skips the
first-time-contributor test, moving a non-fork NONE PR from `external`
(R3) to `human` (R1), and nothing validates that the login really is a
(xhigh) to `human` (medium), and nothing validates that the login really is a
machine account. List only accounts you control; prune retired ones.
type: string
required: false
Expand All @@ -328,17 +331,18 @@ on:
description: >-
Rename the five grader-owned labels, as `tier=label` pairs
(comma-separated). Default:
`R0=risk:R0,R1=risk:R1,R2=risk:R2,R3=risk:R3,unknown=risk:ungraded`.
Tier KEYS are fixed; only the label text is yours. Missing labels are
created on first use, color-coded green through red.
`low=risk:low,medium=risk:medium,high=risk:high,xhigh=risk:xhigh,unknown=risk:ungraded`.
Canonical tier keys are low, medium, high, xhigh, and unknown.
Deprecated R0..R3 keys remain accepted. Missing labels are created on
first use, color-coded green through red.
type: string
required: false
default: ''
wait_for_checks_minutes:
description: >-
How long to wait, PER TARGET, for the REST of the check rollup to settle before
labeling (the grading run itself is excluded from the rollup it
reads). 0 labels immediately — expect R2 floors from still-pending
reads). 0 labels immediately — expect high floors from still-pending
checks. CLAMPED to what a 30-minute job can actually spend waiting (25),
so an over-large value degrades to a shorter wait instead of a job
cancelled mid-sleep with the label never applied. On the by-number path a dispatched
Expand Down Expand Up @@ -1052,14 +1056,14 @@ jobs:
st="$(jq -r '.status // ""' <<<"$one")"
note="$(jq -r '.note // ""' <<<"$one")"
# The HEADING tier falls back to the recorded row. `tier` is a step OUTPUT, so it is
# empty on a cancelled step — and heading a row that recorded R1 as "grade: unknown"
# empty on a cancelled step — and heading a row that recorded medium as "grade: unknown"
# contradicts the table printed directly under it.
head_tier="${TIER:-}"
[ -n "$head_tier" ] || head_tier="$(jq -r '.tier // "unknown"' <<<"$one")"
if [ "$st" = failed ]; then
# A FAILED TARGET NEVER RENDERS A GRADE TABLE, even when it HAS a record. A label
# write that 403s leaves the PR carrying the previous push's grade, and heading
# that outcome "PR risk grade: R1" with the full axis table under it reads as "R1
# that outcome "PR risk grade: medium" with the full axis table under it reads as "medium
# was applied" — the one outcome where the label on the PR is not what the summary
# shows. The note says which input or write failed instead; reporting it as "could
# not be read via the API" would name the wrong cause.
Expand Down
4 changes: 2 additions & 2 deletions README.md

Large diffs are not rendered by default.

19 changes: 12 additions & 7 deletions docs/callers/pr-risk.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,20 @@ Read [the shared caller contract](README.md) first.

## What it does

Grades every PR into a tier `R0` (safest) .. `R3` (riskiest) and syncs **one**
label (`risk:R0`..`risk:R3`, or `risk:ungraded` when an input was unreadable).
Grades every PR into a tier `low` (safest) .. `xhigh` (riskiest) and syncs **one**
label (`risk:low`..`risk:xhigh`, or `risk:ungraded` when an input was unreadable).
The label is the entire product: nothing is gated, routed, commented, or
merged — a human looks at the label and agrees or disagrees (recorded with a
`risk-dispute` label this workflow never touches).

`R0`, `R1`, `R2`, and `R3` remain accepted in existing risk maps and
`label_map` keys as deprecated aliases for `low`, `medium`, `high`, and
`xhigh`. New maps and integrations should use only the canonical names; output
records and publish surfaces always do.

Deterministic, no LLM: `grade = worst(path_floor, provenance, reversibility)` —
a path-glob map, what process produced the diff (registered runbooks, forks
always `R3`), and revertability (persistent-state mutation, sensitive
always `xhigh`), and revertability (persistent-state mutation, sensitive
deletions, whether green checks actually covered the changed lines). The
grader and its generic defaults live in
[`scripts/pr-risk/`](../../scripts/pr-risk) and load from **this repo** at the
Expand Down Expand Up @@ -116,9 +121,9 @@ fail the caller's next run at startup.
|---|---|---|
| `workflows_ref` | — (**required**) | Pin to the SAME full commit SHA as `uses:`. No default on purpose: a floating default let a caller SHA-pin `uses:` and still load the grader from HEAD of main. Checked before the tool checkout on two axes: it must be a full 40-hex lowercase SHA, **and** that commit must be an ancestor of `main` of this repo. So a branch, a tag, a `refs/pull/N/head` and any **not-yet-merged** SHA all fail the run — **merge the change here first, then bump the pin.** There is no opt-out. |
| `fleet_logins` | `mattmillerai` | Logins whose PRs grade provenance `agent-supervised` alongside `agent-coded`. Both are read for **human** authors only: an author GitHub types as a `Bot` is a runbook candidate regardless, so listing a bot here (or labelling its PR) buys it nothing — only a registry entry that asserts can promote it. |
| `bot_logins` | `github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow` | Extra logins treated as bots. Needed only for **machine USER accounts** — a real GitHub App is recognized from GitHub's own actor type, no list entry required. A bot with no runbook entry still grades as human — identity alone buys no trust. **This list is load-bearing, not a hint:** a listed login skips the first-time-contributor test, so it moves a non-fork `NONE`/`FIRST_TIME_CONTRIBUTOR` PR from `external` (R3) to `human` (R1). Nothing validates that a listed login is really a machine account, so add one only for an account you control, and remove it when it is retired. |
| `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Tier keys are fixed; only the label text is yours. |
| `wait_for_checks_minutes` | `10` | How long to wait for the rest of the check rollup to settle before labeling (clamped to 25 — what a 30-minute job can spend waiting). `0` labels immediately, expect R2 floors from still-pending checks. |
| `bot_logins` | `github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow` | Extra logins treated as bots. Needed only for **machine USER accounts** — a real GitHub App is recognized from GitHub's own actor type, no list entry required. A bot with no runbook entry still grades as human — identity alone buys no trust. **This list is load-bearing, not a hint:** a listed login skips the first-time-contributor test, so it moves a non-fork `NONE`/`FIRST_TIME_CONTRIBUTOR` PR from `external` (xhigh) to `human` (medium). Nothing validates that a listed login is really a machine account, so add one only for an account you control, and remove it when it is retired. |
| `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Canonical keys are `low`, `medium`, `high`, `xhigh`, and `unknown`; deprecated `R0`–`R3` keys remain accepted. |
| `wait_for_checks_minutes` | `10` | How long to wait for the rest of the check rollup to settle before labeling (clamped to 25 — what a 30-minute job can spend waiting). `0` labels immediately, expect high floors from still-pending checks. |
| `repo_map_path` | `.github/risk.json` | Consumer risk-map override, read from the PR **base ref**. |
| `repo_runbooks_path` | `.github/risk-runbooks.json` | Consumer runbook-registry override, read from the PR **base ref**. |

Expand Down Expand Up @@ -179,7 +184,7 @@ usual one).
**Enroll it as its own workflow, not a job inside an existing CI workflow.**
The grading job excludes its own *run* from the check rollup it reads; a job
sharing a run with the rest of CI excludes its siblings too and lands on the
honest R2 floor instead of grading off the full rollup.
honest high floor instead of grading off the full rollup.

**Pair the caller with a per-PR `cancel-in-progress` concurrency group** (shown
above). The reversibility axis waits for other checks to settle, so a stale run
Expand Down
10 changes: 5 additions & 5 deletions scripts/pr-derisk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The model proposes a **partition**: which files go in which step, in what order,
step is inert. **It never states a tier.** Every floor rendered in the comment is computed by
`grade-pr-risk.sh --stdin` over a synthetic scorecard record built from that step's files — the
same deterministic judge, the same map, the same rules that graded the PR. A model that
hallucinates "this split lands R0" cannot put that number in front of a reviewer: the plan object
hallucinates "this split lands low" cannot put that number in front of a reviewer: the plan object
is rebuilt field by field from a fixed list, so a model-claimed `tier` key does not survive.

That is also why this lives *outside* the grader rather than inside it. pr-risk's grading path
Expand All @@ -47,14 +47,14 @@ either way. A split can land *worse* than its floor; it can never land better.

A single-class monolith — every file already at the PR's **path floor** — has no lane win
available. The verdict line above the fold is computed from the **floors**, not from the model's
prose, so in that case it reads "N smaller single-concern R3s, same lane" and there is no wording
prose, so in that case it reads "N smaller single-concern xhigh tiers, same lane" and there is no wording
available to it that claims a reduction. A prompt can *ask* for that; only the renderer can
guarantee it.

**The comparison axis is the PATH FLOOR, not the headline tier**, and that distinction is the rule
rather than a detail of it. `grade = worst(path_floor, provenance, reversibility)` but a split only
ever moves the path axis, so on a fork PR (provenance R3, path floor R0) or a `/derisk` typed while
checks are still pending (reversibility R2) *every* step sits below the headline while the axis
ever moves the path axis, so on a fork PR (provenance xhigh, path floor low) or a `/derisk` typed while
checks are still pending (reversibility high) *every* step sits below the headline while the axis
that actually set the grade is untouched. Comparing against the headline there would print a
reduction no partition can deliver, on exactly the pull requests this rule exists for. When a
non-path axis holds the grade up the verdict says so in the same breath as the split.
Expand Down Expand Up @@ -112,6 +112,6 @@ which is then rejected unless it covers the changed-file set exactly.

## What is deliberately NOT here

No auto-running on every R3 (on-demand only in beta), no ticket filing from a plan (its own rung,
No auto-running on every xhigh (on-demand only in beta), no ticket filing from a plan (its own rung,
behind its own command), no routing, no check, no label, no auto-merge. The offer threshold and
which repos see it at all are post-merge flips of a repo variable, not code.
8 changes: 4 additions & 4 deletions scripts/pr-derisk/plan-derisk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# is inert. It NEVER states a tier. Every floor rendered in the comment is computed by
# grade-pr-risk.sh --stdin over a synthetic scorecard record built from that step's files — the
# same deterministic judge, the same map, the same rules that graded the PR. So a model that
# hallucinates "this split lands R0" cannot put that number in front of a reviewer: the number
# hallucinates "this split lands low" cannot put that number in front of a reviewer: the number
# comes from the grader or it does not appear.
#
# That is also why this lives OUTSIDE the grader rather than inside it. pr-risk's grading path
Expand Down Expand Up @@ -124,9 +124,9 @@ trap 'rm -rf "$SCRATCH"' EXIT
TIER="$(jq -r '.risk.tier // "unknown"' "$RECORD" 2>/dev/null)"
# THE PATH-AXIS FLOOR, CARRIED SEPARATELY FROM THE HEADLINE, and the distinction is load-bearing.
# `grade = worst(path_floor, provenance, reversibility)`, but a split only ever moves the PATH axis.
# Comparing a step's computed path floor against the HEADLINE would read "below R3" for every step
# of a fork PR (provenance R3, path floor R0) or of a `/derisk` typed while checks are pending
# (reversibility R2) — a lane win the split cannot deliver, claimed on exactly the pull requests
# Comparing a step's computed path floor against the HEADLINE would read "below xhigh" for every step
# of a fork PR (provenance xhigh, path floor low) or of a `/derisk` typed while checks are pending
# (reversibility high) — a lane win the split cannot deliver, claimed on exactly the pull requests
# the no-fake-lane-win rule exists for. The renderer compares against this instead.
PATH_TIER="$(jq -r '.risk.axes.path_floor.tier // ""' "$RECORD" 2>/dev/null)"
STATUS="$(jq -r '.risk.status // "unknown"' "$RECORD" 2>/dev/null)"
Expand Down
Loading
Loading