From cbdc39e79039af558a56766a116fb3e337687359 Mon Sep 17 00:00:00 2001 From: ShihChi Huang Date: Tue, 1 Sep 2026 17:17:31 -0700 Subject: [PATCH 1/3] feat(pr-risk): allow repository label maps --- .github/workflows/pr-risk.yml | 11 +++++++---- docs/callers/pr-risk.md | 7 ++++++- scripts/pr-risk/README.md | 8 +++----- scripts/pr-risk/apply-risk-label.sh | 9 ++++----- scripts/pr-risk/tests/test_apply_risk_label.sh | 10 ++++++++++ 5 files changed, 30 insertions(+), 15 deletions(-) diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index d62bb4d7..f2c3aa3b 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -114,8 +114,10 @@ name: PR Risk Grade (reusable) # last-writer-wins with still EXACTLY ONE `risk:*` label — possibly the staler tier, which gates # nothing meanwhile and which the next grade re-syncs (on the LAST push there is no next grade, so # re-dispatch on `pr_number` if a final grade looks wrong). It can no longer leave two contradictory -# labels on a PR under one `label_map`; remapping `label_map` orphans the old names, which is a -# one-time repo-side cleanup. What the shape does cost is a narrower residual: the PUT is built from +# labels on a PR under one `label_map`. A caller can switch maps without a PR via +# `vars.RISK_LABEL_MAP`; the first remapped grade retires the default `risk:R0`..`risk:R3` labels. +# Custom names from an older map still need one-time repo-side cleanup. What the shape does cost is +# a narrower residual: the PUT is built from # a snapshot read, so a NON-owned label added by someone else in the read→PUT window is dropped # (`risk-dispute` included) and one removed in it is resurrected. That window opens only on a run # that actually changes the grade and is roughly one API round-trip — about three on the first @@ -330,7 +332,8 @@ on: (comma-separated). Default: `R0=risk:R0,R1=risk:R1,R2=risk:R2,R3=risk:R3,unknown=risk:ungraded`. Tier KEYS are fixed; only the label text is yours. Missing labels are - created on first use, color-coded green through red. + created on first use, color-coded green through red. The caller repo's + `RISK_LABEL_MAP` variable overrides this input without a caller PR. type: string required: false default: '' @@ -988,7 +991,7 @@ jobs: RB_PATH: ${{ inputs.repo_runbooks_path }} FLEET_LOGINS: ${{ inputs.fleet_logins }} BOT_LOGINS: ${{ inputs.bot_logins }} - LABEL_MAP: ${{ inputs.label_map }} + LABEL_MAP: ${{ vars.RISK_LABEL_MAP || inputs.label_map }} SELF_CONTEXT: ${{ github.workflow }} SELF_RUN_ID: ${{ github.run_id }} WAIT_MINUTES: ${{ inputs.wait_for_checks_minutes }} diff --git a/docs/callers/pr-risk.md b/docs/callers/pr-risk.md index 4c3c5fdf..967c986c 100644 --- a/docs/callers/pr-risk.md +++ b/docs/callers/pr-risk.md @@ -117,11 +117,16 @@ fail the caller's next run at startup. | `workflows_ref` | — (**required**) | Pin to the SAME full commit SHA as `uses:`. No default on purpose: a floating default let a caller SHA-pin `uses:` and still load the grader from HEAD of main. Checked before the tool checkout on two axes: it must be a full 40-hex lowercase SHA, **and** that commit must be an ancestor of `main` of this repo. So a branch, a tag, a `refs/pull/N/head` and any **not-yet-merged** SHA all fail the run — **merge the change here first, then bump the pin.** There is no opt-out. | | `fleet_logins` | `mattmillerai` | Logins whose PRs grade provenance `agent-supervised` alongside `agent-coded`. Both are read for **human** authors only: an author GitHub types as a `Bot` is a runbook candidate regardless, so listing a bot here (or labelling its PR) buys it nothing — only a registry entry that asserts can promote it. | | `bot_logins` | `github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow` | Extra logins treated as bots. Needed only for **machine USER accounts** — a real GitHub App is recognized from GitHub's own actor type, no list entry required. A bot with no runbook entry still grades as human — identity alone buys no trust. **This list is load-bearing, not a hint:** a listed login skips the first-time-contributor test, so it moves a non-fork `NONE`/`FIRST_TIME_CONTRIBUTOR` PR from `external` (R3) to `human` (R1). Nothing validates that a listed login is really a machine account, so add one only for an account you control, and remove it when it is retired. | -| `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Tier keys are fixed; only the label text is yours. | +| `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Tier keys are fixed; only the label text is yours. The caller repo's `RISK_LABEL_MAP` variable overrides this input without a PR. | | `wait_for_checks_minutes` | `10` | How long to wait for the rest of the check rollup to settle before labeling (clamped to 25 — what a 30-minute job can spend waiting). `0` labels immediately, expect R2 floors from still-pending checks. | | `repo_map_path` | `.github/risk.json` | Consumer risk-map override, read from the PR **base ref**. | | `repo_runbooks_path` | `.github/risk-runbooks.json` | Consumer runbook-registry override, read from the PR **base ref**. | +For `risk:low` through `risk:xhigh`, set `RISK_LABEL_MAP` to +`R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:ungraded`. +The next grade removes a default `risk:R0`..`risk:R3` label and leaves only the +mapped label. Custom names from an older map still need one-time cleanup. + ## Gotchas **Fork PRs need `pull_request_target`, not `pull_request`.** A fork PR under a diff --git a/scripts/pr-risk/README.md b/scripts/pr-risk/README.md index 2a4cbd76..3d317dd5 100644 --- a/scripts/pr-risk/README.md +++ b/scripts/pr-risk/README.md @@ -198,11 +198,9 @@ Operational caveats for a backfill: invisible: GitHub records it on the PR timeline as an `unlabeled` event by the grader token. Dispatch when the queue is quiet, and use `pr_number` when you want the per-PR group to serialize a re-grade against event runs. -- **Remapping `label_map` orphans the old names.** Ownership is defined by the - *current* map, so labels applied under a previous one are no longer owned: - they ride through every future PUT beside the new target and no re-grade will - clear them. Delete the retired label names repo-side once, as part of the - remap. +- **Remapping retires the default `risk:R0`..`risk:R3` labels.** The first + re-grade removes the old default and leaves one mapped label. Custom names + from an older map are unknowable and still need one-time repo-side cleanup. - **The pre-grader reads retry.** Rate limits are global, not per-PR, so the base-ref and override reads — the first hop for every target — retry a transient failure with backoff, as the grader already does. Without it one diff --git a/scripts/pr-risk/apply-risk-label.sh b/scripts/pr-risk/apply-risk-label.sh index b413a6d7..55e98f00 100755 --- a/scripts/pr-risk/apply-risk-label.sh +++ b/scripts/pr-risk/apply-risk-label.sh @@ -36,11 +36,9 @@ # before-snapshot and the after-read, so the diff that would have to catch it is empty by # construction. # -# ONE MORE LIMIT, on a different axis: ownership is defined by the CURRENT LABEL_MAP. Change a -# caller's `label_map` and labels applied under the old map are, by definition, no longer owned — -# they are carried through every future PUT beside the new target, and nothing here will clean them -# up. Remapping is a one-time repo-side cleanup (delete the retired label names), not something a -# re-grade heals. +# ONE MORE LIMIT, on a different axis: ownership is defined by the CURRENT LABEL_MAP plus the four +# default grade labels. A remap retires those defaults on the first re-grade. Custom names from an +# older map are unknowable and still need one-time repo-side cleanup. # # The label is applied with the plain GITHUB_TOKEN on purpose: GITHUB_TOKEN-applied labels do # not fire `labeled` workflow triggers, which makes the shadow check incapable of starting a @@ -107,6 +105,7 @@ for t in R0 R1 R2 R3 unknown; do [ -n "$l" ] || die "LABEL_MAP maps tier '$t' to an empty label" OWNED+=("$l") done +OWNED+=("risk:R0" "risk:R1" "risk:R2" "risk:R3") TARGET="$(label_for "$TIER")" # Colors keyed by TIER (not label text, which callers may remap): green .. red, gray unknown. diff --git a/scripts/pr-risk/tests/test_apply_risk_label.sh b/scripts/pr-risk/tests/test_apply_risk_label.sh index 78ab30fe..fc5293c1 100755 --- a/scripts/pr-risk/tests/test_apply_risk_label.sh +++ b/scripts/pr-risk/tests/test_apply_risk_label.sh @@ -203,6 +203,16 @@ eq "the extra owned label is squashed down to the one target" \ "api -X PUT repos/test/repo/issues/7/labels -f labels[]=keep-me -f labels[]=risk:R2" \ "$putheal" +echo "— a remap retires the default grade label instead of keeping both —" +: > "$GH_LOG"; printf 'risk:R2\nkeep-me\n' > "$CURRENT_LABELS" +SEVERITY_MAP='R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:ungraded' +PATH="$SANDBOX/bin:$PATH" REPO=test/repo PR_NUMBER=7 TIER=R2 LABEL_MAP="$SEVERITY_MAP" \ + bash "$SCRIPT" >/dev/null 2>&1 +putremap="$(grep -- '-X PUT repos/test/repo/issues/7/labels ' "$GH_LOG")" +eq "the old default is replaced by the mapped label" \ + "api -X PUT repos/test/repo/issues/7/labels -f labels[]=keep-me -f labels[]=risk:high" \ + "$putremap" + # Already-correct label: no write at all beyond the read. : > "$GH_LOG"; printf 'risk:R2\nkeep-me\n' > "$CURRENT_LABELS" outsync="$(PATH="$SANDBOX/bin:$PATH" REPO=test/repo PR_NUMBER=7 TIER=R2 bash "$SCRIPT" 2>/dev/null)" From 0310a63e07e876e2da96fc4b0c3e8a8c2831100c Mon Sep 17 00:00:00 2001 From: ShihChi Huang Date: Tue, 1 Sep 2026 17:26:49 -0700 Subject: [PATCH 2/3] docs(pr-risk): use risk unknown label --- docs/callers/pr-risk.md | 2 +- scripts/pr-risk/tests/test_apply_risk_label.sh | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/callers/pr-risk.md b/docs/callers/pr-risk.md index 967c986c..c984df09 100644 --- a/docs/callers/pr-risk.md +++ b/docs/callers/pr-risk.md @@ -123,7 +123,7 @@ fail the caller's next run at startup. | `repo_runbooks_path` | `.github/risk-runbooks.json` | Consumer runbook-registry override, read from the PR **base ref**. | For `risk:low` through `risk:xhigh`, set `RISK_LABEL_MAP` to -`R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:ungraded`. +`R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:unknown`. The next grade removes a default `risk:R0`..`risk:R3` label and leaves only the mapped label. Custom names from an older map still need one-time cleanup. diff --git a/scripts/pr-risk/tests/test_apply_risk_label.sh b/scripts/pr-risk/tests/test_apply_risk_label.sh index fc5293c1..92ffa138 100755 --- a/scripts/pr-risk/tests/test_apply_risk_label.sh +++ b/scripts/pr-risk/tests/test_apply_risk_label.sh @@ -33,6 +33,8 @@ echo "— caller remap (a 1-indexed R1..R4 scheme is one input) —" MAP='R0=risk:R1,R1=risk:R2,R2=risk:R3,R3=risk:R4,unknown=risk:ungraded' eq "R0 remaps to risk:R1" "risk:R1" "$(run R0 "$MAP")" eq "R3 remaps to risk:R4" "risk:R4" "$(run R3 "$MAP")" +NAMED_MAP='R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:unknown' +eq "unknown remaps to risk:unknown" "risk:unknown" "$(run unknown "$NAMED_MAP")" echo "— validation refuses bad input before any write —" run R7 >/dev/null 2>&1; eq "bad tier exits 2" 2 "$?" @@ -205,8 +207,7 @@ eq "the extra owned label is squashed down to the one target" \ echo "— a remap retires the default grade label instead of keeping both —" : > "$GH_LOG"; printf 'risk:R2\nkeep-me\n' > "$CURRENT_LABELS" -SEVERITY_MAP='R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:ungraded' -PATH="$SANDBOX/bin:$PATH" REPO=test/repo PR_NUMBER=7 TIER=R2 LABEL_MAP="$SEVERITY_MAP" \ +PATH="$SANDBOX/bin:$PATH" REPO=test/repo PR_NUMBER=7 TIER=R2 LABEL_MAP="$NAMED_MAP" \ bash "$SCRIPT" >/dev/null 2>&1 putremap="$(grep -- '-X PUT repos/test/repo/issues/7/labels ' "$GH_LOG")" eq "the old default is replaced by the mapped label" \ From e7b8a69e389eb9c4410af720f26d2dfd78af054a Mon Sep 17 00:00:00 2001 From: ShihChi Huang Date: Tue, 1 Sep 2026 18:43:54 -0700 Subject: [PATCH 3/3] refactor(pr-risk): reuse label map input --- .github/workflows/pr-risk.yml | 12 +++++------- docs/callers/pr-risk.md | 9 +++++---- scripts/pr-risk/README.md | 6 +++--- scripts/pr-risk/apply-risk-label.sh | 6 +++--- scripts/pr-risk/tests/test_apply_risk_label.sh | 4 ++-- 5 files changed, 18 insertions(+), 19 deletions(-) diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index f2c3aa3b..7e1fda2c 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -114,10 +114,9 @@ name: PR Risk Grade (reusable) # last-writer-wins with still EXACTLY ONE `risk:*` label — possibly the staler tier, which gates # nothing meanwhile and which the next grade re-syncs (on the LAST push there is no next grade, so # re-dispatch on `pr_number` if a final grade looks wrong). It can no longer leave two contradictory -# labels on a PR under one `label_map`. A caller can switch maps without a PR via -# `vars.RISK_LABEL_MAP`; the first remapped grade retires the default `risk:R0`..`risk:R3` labels. -# Custom names from an older map still need one-time repo-side cleanup. What the shape does cost is -# a narrower residual: the PUT is built from +# labels on a PR under one `label_map`; the first remapped grade retires the default +# `risk:R0`..`risk:R3` and `risk:ungraded` labels. Custom names from an older map still need cleanup. +# What the shape does cost is a narrower residual: the PUT is built from # a snapshot read, so a NON-owned label added by someone else in the read→PUT window is dropped # (`risk-dispute` included) and one removed in it is resurrected. That window opens only on a run # that actually changes the grade and is roughly one API round-trip — about three on the first @@ -332,8 +331,7 @@ on: (comma-separated). Default: `R0=risk:R0,R1=risk:R1,R2=risk:R2,R3=risk:R3,unknown=risk:ungraded`. Tier KEYS are fixed; only the label text is yours. Missing labels are - created on first use, color-coded green through red. The caller repo's - `RISK_LABEL_MAP` variable overrides this input without a caller PR. + created on first use, color-coded green through red. type: string required: false default: '' @@ -991,7 +989,7 @@ jobs: RB_PATH: ${{ inputs.repo_runbooks_path }} FLEET_LOGINS: ${{ inputs.fleet_logins }} BOT_LOGINS: ${{ inputs.bot_logins }} - LABEL_MAP: ${{ vars.RISK_LABEL_MAP || inputs.label_map }} + LABEL_MAP: ${{ inputs.label_map }} SELF_CONTEXT: ${{ github.workflow }} SELF_RUN_ID: ${{ github.run_id }} WAIT_MINUTES: ${{ inputs.wait_for_checks_minutes }} diff --git a/docs/callers/pr-risk.md b/docs/callers/pr-risk.md index c984df09..24fb8551 100644 --- a/docs/callers/pr-risk.md +++ b/docs/callers/pr-risk.md @@ -117,15 +117,16 @@ fail the caller's next run at startup. | `workflows_ref` | — (**required**) | Pin to the SAME full commit SHA as `uses:`. No default on purpose: a floating default let a caller SHA-pin `uses:` and still load the grader from HEAD of main. Checked before the tool checkout on two axes: it must be a full 40-hex lowercase SHA, **and** that commit must be an ancestor of `main` of this repo. So a branch, a tag, a `refs/pull/N/head` and any **not-yet-merged** SHA all fail the run — **merge the change here first, then bump the pin.** There is no opt-out. | | `fleet_logins` | `mattmillerai` | Logins whose PRs grade provenance `agent-supervised` alongside `agent-coded`. Both are read for **human** authors only: an author GitHub types as a `Bot` is a runbook candidate regardless, so listing a bot here (or labelling its PR) buys it nothing — only a registry entry that asserts can promote it. | | `bot_logins` | `github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow` | Extra logins treated as bots. Needed only for **machine USER accounts** — a real GitHub App is recognized from GitHub's own actor type, no list entry required. A bot with no runbook entry still grades as human — identity alone buys no trust. **This list is load-bearing, not a hint:** a listed login skips the first-time-contributor test, so it moves a non-fork `NONE`/`FIRST_TIME_CONTRIBUTOR` PR from `external` (R3) to `human` (R1). Nothing validates that a listed login is really a machine account, so add one only for an account you control, and remove it when it is retired. | -| `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Tier keys are fixed; only the label text is yours. The caller repo's `RISK_LABEL_MAP` variable overrides this input without a PR. | +| `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Tier keys are fixed; only the label text is yours. | | `wait_for_checks_minutes` | `10` | How long to wait for the rest of the check rollup to settle before labeling (clamped to 25 — what a 30-minute job can spend waiting). `0` labels immediately, expect R2 floors from still-pending checks. | | `repo_map_path` | `.github/risk.json` | Consumer risk-map override, read from the PR **base ref**. | | `repo_runbooks_path` | `.github/risk-runbooks.json` | Consumer runbook-registry override, read from the PR **base ref**. | -For `risk:low` through `risk:xhigh`, set `RISK_LABEL_MAP` to +To show `risk:low` through `risk:xhigh`, pass this `label_map` in the caller: `R0=risk:low,R1=risk:medium,R2=risk:high,R3=risk:xhigh,unknown=risk:unknown`. -The next grade removes a default `risk:R0`..`risk:R3` label and leaves only the -mapped label. Custom names from an older map still need one-time cleanup. +The next grade removes a default `risk:R0`..`risk:R3` or `risk:ungraded` label +and leaves only the mapped label. Custom names from an older map still need +one-time cleanup. ## Gotchas diff --git a/scripts/pr-risk/README.md b/scripts/pr-risk/README.md index 3d317dd5..8334a998 100644 --- a/scripts/pr-risk/README.md +++ b/scripts/pr-risk/README.md @@ -198,9 +198,9 @@ Operational caveats for a backfill: invisible: GitHub records it on the PR timeline as an `unlabeled` event by the grader token. Dispatch when the queue is quiet, and use `pr_number` when you want the per-PR group to serialize a re-grade against event runs. -- **Remapping retires the default `risk:R0`..`risk:R3` labels.** The first - re-grade removes the old default and leaves one mapped label. Custom names - from an older map are unknowable and still need one-time repo-side cleanup. +- **Remapping retires the default `risk:R0`..`risk:R3` and `risk:ungraded` + labels.** The first re-grade removes the old default and leaves one mapped + label. Custom names from an older map still need one-time repo-side cleanup. - **The pre-grader reads retry.** Rate limits are global, not per-PR, so the base-ref and override reads — the first hop for every target — retry a transient failure with backoff, as the grader already does. Without it one diff --git a/scripts/pr-risk/apply-risk-label.sh b/scripts/pr-risk/apply-risk-label.sh index 55e98f00..56f0e45f 100755 --- a/scripts/pr-risk/apply-risk-label.sh +++ b/scripts/pr-risk/apply-risk-label.sh @@ -36,8 +36,8 @@ # before-snapshot and the after-read, so the diff that would have to catch it is empty by # construction. # -# ONE MORE LIMIT, on a different axis: ownership is defined by the CURRENT LABEL_MAP plus the four -# default grade labels. A remap retires those defaults on the first re-grade. Custom names from an +# ONE MORE LIMIT, on a different axis: ownership is defined by the CURRENT LABEL_MAP plus the five +# default labels. A remap retires those defaults on the first re-grade. Custom names from an # older map are unknowable and still need one-time repo-side cleanup. # # The label is applied with the plain GITHUB_TOKEN on purpose: GITHUB_TOKEN-applied labels do @@ -105,7 +105,7 @@ for t in R0 R1 R2 R3 unknown; do [ -n "$l" ] || die "LABEL_MAP maps tier '$t' to an empty label" OWNED+=("$l") done -OWNED+=("risk:R0" "risk:R1" "risk:R2" "risk:R3") +OWNED+=("risk:R0" "risk:R1" "risk:R2" "risk:R3" "risk:ungraded") TARGET="$(label_for "$TIER")" # Colors keyed by TIER (not label text, which callers may remap): green .. red, gray unknown. diff --git a/scripts/pr-risk/tests/test_apply_risk_label.sh b/scripts/pr-risk/tests/test_apply_risk_label.sh index 92ffa138..d46010f1 100755 --- a/scripts/pr-risk/tests/test_apply_risk_label.sh +++ b/scripts/pr-risk/tests/test_apply_risk_label.sh @@ -206,11 +206,11 @@ eq "the extra owned label is squashed down to the one target" \ "$putheal" echo "— a remap retires the default grade label instead of keeping both —" -: > "$GH_LOG"; printf 'risk:R2\nkeep-me\n' > "$CURRENT_LABELS" +: > "$GH_LOG"; printf 'risk:R2\nrisk:ungraded\nkeep-me\n' > "$CURRENT_LABELS" PATH="$SANDBOX/bin:$PATH" REPO=test/repo PR_NUMBER=7 TIER=R2 LABEL_MAP="$NAMED_MAP" \ bash "$SCRIPT" >/dev/null 2>&1 putremap="$(grep -- '-X PUT repos/test/repo/issues/7/labels ' "$GH_LOG")" -eq "the old default is replaced by the mapped label" \ +eq "the old defaults are replaced by the mapped label" \ "api -X PUT repos/test/repo/issues/7/labels -f labels[]=keep-me -f labels[]=risk:high" \ "$putremap"