From 1474ab866d100143aca37beb0dadc906c492289c Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Sun, 25 Oct 2026 19:37:20 +0100 Subject: [PATCH 1/6] Show each fork CI job's status on gated pull requests (#75616) A gated pull request showed one aggregate fork CI status, so a maintainer or contributor had to open the fork run to see which jobs were running, which failed and which passed. Reporting every job as its own check, each linking to the job in the fork, puts that on the pull request itself, where project CI shows its jobs. Generated-by: Claude Opus 5 (cherry picked from commit 2f7a7ea4be9c5c48e45d2d45d14595f2e2e594d3) --- .github/fork-ci-gate.toml | 5 + dev/AGENTS.md | 17 + .../08_fork_ci_for_external_contributors.md | 87 ++++- .../src/airflow_breeze/utils/fork_ci_gate.py | 5 + .../airflow_breeze/utils/fork_ci_messages.py | 36 ++ .../airflow_breeze/utils/fork_ci_reconcile.py | 159 +++++++- dev/breeze/tests/test_fork_ci_gate.py | 6 + dev/breeze/tests/test_fork_ci_messages.py | 22 +- dev/breeze/tests/test_fork_ci_reconcile.py | 362 +++++++++++++++++- 9 files changed, 690 insertions(+), 9 deletions(-) diff --git a/.github/fork-ci-gate.toml b/.github/fork-ci-gate.toml index c80593d221c43..6fc7a0de1ce3c 100644 --- a/.github/fork-ci-gate.toml +++ b/.github/fork-ci-gate.toml @@ -29,6 +29,11 @@ enabled_since = 2026-01-01T00:00:00Z # Days without any fork CI activity after which a gated pull request is closed. close_after_days = 7 +# Report each job of a gated pull request's fork run as its own commit status. Set to false when +# the fork CI monitor exceeds GitHub's rate limits (see dev/AGENTS.md); the `Fork CI / Tests (AMD)` +# status is posted either way. +job_statuses = true + # Organisations exempted under AIP-120 §2.2. Keys are display names used in comments, # values are GitHub logins. Changes go through a reviewed pull request. [allowlist] diff --git a/dev/AGENTS.md b/dev/AGENTS.md index 947e0ac9e8e42..313c786018397 100644 --- a/dev/AGENTS.md +++ b/dev/AGENTS.md @@ -7,6 +7,7 @@ - [dev/ directory guidelines](#dev-directory-guidelines) - [Scripts](#scripts) + - [Fork CI job statuses and GitHub's rate limits](#fork-ci-job-statuses-and-githubs-rate-limits) @@ -42,3 +43,19 @@ uv run dev/my_script.py [args...] ``` Document `uv run` (not `python`) as the invocation method in READMEs and instructions. + +## Fork CI job statuses and GitHub's rate limits + +The fork CI monitor (`breeze ci fork-ci-reconcile`, run by `.github/workflows/fork-ci-monitor.yml`) +reports every job of a gated pull request's fork run as its own commit status. Those writes are the +largest share of what the monitor does with the repository's `GITHUB_TOKEN`, whose rate limits every +workflow of `apache/airflow` shares (see "Write budget" in +`dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md`). + +If the budget is exceeded, turn the job statuses off: set `job_statuses = false` in +`.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack note that job +statuses "did not fit in the budget" persisting across runs, rate-limit retries ("GitHub rate limit, +retrying") or "still rate limited" errors in the monitor's run log or its Slack messages, or other +workflows of the repository failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)` +status and `CI passed` are posted either way, so turning job statuses off loses detail, not gating. +Lowering `MAX_JOB_STATUSES_PER_RUN` in `fork_ci_reconcile.py` is the softer alternative. diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index 1b2bd742f90ac..799a233e9ca32 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -661,6 +661,87 @@ Commit statuses were chosen over check runs because a check run created with the token is attached to the creating workflow's check suite and its details link then opens the monitor run rather than `details_url`. A status honours `target_url`. +### Job statuses + +Next to the aggregate status, each job of the fork run is reported as its own commit status, so +the pull request shows which jobs run, failed or passed, each linking to the job in the fork. +The run is the one `Fork CI / Tests (AMD)` reports (its `databaseId` is fetched with the check +suites); its jobs come from `GET /repos//actions/runs//jobs?filter=latest`. + +- The context is `Fork CI job / `, with non-printable characters replaced and the name + shortened to 100 characters. Job names come from the workflow of the contributor's commit, so + the prefix is what keeps a job from posting `Fork CI / Tests (AMD)`, from which the reconciler + reads its decisions back, or `CI passed`, which the merge ruleset requires. Jobs that share a + context, which GitHub compares case-insensitively, are reported as the worst of them. +- A queued or running job is `pending` ("Running in the fork"), a successful one `success`, and a + failed, timed-out, cancelled or unstartable one `failure`. Skipped and neutral jobs, and the jobs + every run has (`Fork CI gate`, `Check if the run is needed`, `Platform: AMD` and `CI passed`), are + not reported. Queued and running share one status so that starting a job costs no write. +- The desired statuses are compared with the head commit's current ones + (`GET /repos/apache/airflow/commits//status`), and only differences are posted, failures + first. GraphQL has no mutation for commit statuses (`createCheckRun` exists, but see above for + why check runs do not fit), so each one is a REST call, and they count towards GitHub's limits on + content-creating requests: a reconciler run posts at most `MAX_JOB_STATUSES_PER_RUN` (60) of + them, each write charged as it is made. What does not fit is posted by the next run, and once + the budget is spent no further pull request's jobs or statuses are read in that run. A run that + defers job statuses adds a note to the Slack message, with fixed wording so it is posted once + while deferrals persist; the run log lists the pull requests. +- Commit statuses cannot be deleted and have no neutral state. When a run has completed, a job + context posted for an earlier run of the same commit that the completed run does not have, or + skipped, is overwritten with `success` and the description "Not in the latest fork run - see + Fork CI / Tests (AMD)", so it no longer shows as running or failed. +- Once every job of a completed run is reported, the run is recorded in the marker as + `jobs: "::"` (`runAttempt` is fetched with the check suites), so + later runs do not read its jobs again. A run in progress is read on every run, and a re-run in + the fork is read again because its attempt changes, even when it ends with the same conclusion. +- Reporting job statuses is caught on its own: when it fails, the failure is listed with the + unknown items for Slack, and the pull request is still classified, so for example a `CI passed` + that no longer holds is still revoked. + +The job statuses are informational. Merging still depends on `CI passed`, and the state machine +on `Fork CI / Tests (AMD)`; a job status never changes either. + +`job_statuses = false` in `.github/fork-ci-gate.toml` turns them off; the reconciler then posts +only `Fork CI / Tests (AMD)` and `CI passed`, as before. Statuses already posted stay on their +commits. Turn them off when the fork CI monitor's runs hit GitHub's rate limits (rate-limit +retries in the run log, or "still rate limited" errors in Slack); `dev/AGENTS.md` records this for +agents. + +#### Write budget + +Measured on 2026-10-25 from `apache/airflow`'s own `Tests (AMD)` pull request runs (the same +workflow runs in the forks; `GET /repos/apache/airflow/actions/runs//jobs?filter=latest`, +jobs whose conclusion is not `skipped`, minus the four jobs that are not reported): + +| Pull request | Run | Started jobs | Reported job statuses | +|---|---|---|---| +| Breeze test fix | [40138784120](https://github.com/apache/airflow/actions/runs/40138784120) | 11 | 7 | +| Documentation fix | [40169750323](https://github.com/apache/airflow/actions/runs/40169750323) | 16 | 12 | +| Google provider change | [40158907296](https://github.com/apache/airflow/actions/runs/40158907296) | 43 | 39 | +| CI change (full tests) | [40207881565](https://github.com/apache/airflow/actions/runs/40207881565) | 131 | 127 | + +`apache/airflow` had 1898 `Tests (AMD)` pull request runs from 2026-10-18 to 2026-10-24, about 270 a +day or 11 an hour. Each reported job costs at most two writes (running, then its result) and one +when it starts and finishes between two reconciler runs; a fork run of 30-60 minutes spans two to +four of them, so most jobs cost two. + +The limits that matter are GitHub's documented ones for the `GITHUB_TOKEN` of `apache/airflow`, +shared by every workflow of the repository: 1000 REST requests an hour, and 80 content-creating +requests a minute and 500 an hour. Assuming about 35 reported jobs for an average push: + +- **Contributor pilot**, a few gated pushes a day: about 70 writes a push, under 20 an hour, under + 5% of the content-creating limit. +- **Gate enabled**, assuming about half of the pull request runs come from gated pull requests + (about 5-6 pushes an hour): about 400 writes an hour without a cap, 80% of the content-creating + limit and more at peaks. The cap of 60 writes a run, four runs an hour, holds job statuses to at + most 240 an hour: 48% of the content-creating limit and 24% of the REST limit, plus two reads per + pull request with a run in progress. Job statuses then lag behind at busy times, the aggregate + status does not. + +These are estimates: the share of gated runs, the jobs per push and the pushes per pull request +are assumptions, not measurements. Watch the monitor's call counts in its run log after enabling +the gate, and turn `job_statuses` off if they approach the limits. + ### Relationship to the triage skill The `pr-management-triage` skill described in `contributing-docs/25_maintainer_pr_triage.md` @@ -760,7 +841,9 @@ Per run, with ~700 gated pull requests: ~28 GraphQL calls for pass 2, ~14 for fo calls for the recent `pull_request` runs, one REST call per base branch for its workflow files, a jobs call and a compare call only per long `ci-amd.yml` run of a gated pull request, one changed-files call per head commit that turns green, plus one mutation per actual state -change. Well inside +change. Job statuses add a jobs call and a status call per pull request with a fork run in +progress or a completed run not yet reported, and up to two status mutations per job, at most +`MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Well inside the `GITHUB_TOKEN` limits at a 15-minute cadence. The reconciler logs its call counts so the budget stays observable. @@ -906,6 +989,8 @@ as a failed check rather than a crash. workflow files on the base branch. - A green fork run is trusted only for pull requests that do not change what fork CI runs; see "Trusting a green fork run". +- Job statuses carry names from the contributor's workflow under their own `Fork CI job / ` + prefix, so they cannot post the reconciler's own contexts or `CI passed`; see "Job statuses". - The reconciler runs on `main` with `pull-requests: write` and never checks out or executes pull request content. Its inputs are API data; its comment templates are static. - Fork runs get the fork's `GITHUB_TOKEN` only. Nothing in project CI trusts a fork run's diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py index 80c3632ce77cb..8ce90a34d97b3 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py @@ -128,6 +128,7 @@ class GateConfig: close_after_days: int = DEFAULT_CLOSE_AFTER_DAYS allowlist: Mapping[str, tuple[str, ...]] = field(default_factory=dict) pilot: Mapping[str, datetime] = field(default_factory=dict) + job_statuses: bool = True def is_allowlisted(self, login: str) -> bool: wanted = login.casefold() @@ -173,12 +174,16 @@ def parse_config(text: str) -> GateConfig: login.casefold(): _require_aware_datetime(joined_at, f"pilot entry {login!r}") for login, joined_at in data.get("pilot", {}).items() } + job_statuses = data.get("job_statuses", True) + if not isinstance(job_statuses, bool): + raise ValueError(f"job_statuses must be true or false, got {job_statuses!r}") return GateConfig( enabled=enabled, enabled_since=enabled_since, close_after_days=close_after_days, allowlist=allowlist, pilot=pilot, + job_statuses=job_statuses, ) diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py index d28e5f54283d2..e00048df167c4 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py @@ -30,6 +30,10 @@ from datetime import datetime STATUS_CONTEXT = "Fork CI / Tests (AMD)" +# Each job of the fork run is reported under this prefix. Job names come from the contributor's +# workflow, so the prefix keeps them from posting STATUS_CONTEXT or CI_PASSED_CONTEXT. +JOB_STATUS_CONTEXT_PREFIX = "Fork CI job / " +JOB_NAME_MAX_LENGTH = 100 # The check the "Require CI on main" ruleset requires before a pull request can (auto-)merge. Project # CI's gate job reports it; for a gated pull request, which never runs project CI, the reconciler # reports it from the fork run. @@ -74,12 +78,16 @@ class Marker: since: datetime # Commits the head commit was behind its base branch when its fork CI turned green. behind: int | None = None + # The completed fork run (``::``) whose jobs are all reported. + jobs: str | None = None def render_marker(marker: Marker) -> str: data: dict[str, object] = {"state": marker.state, "sha": marker.sha, "since": marker.since.isoformat()} if marker.behind is not None: data["behind"] = marker.behind + if marker.jobs is not None: + data["jobs"] = marker.jobs return f"{MARKER_PREFIX}{json.dumps(data, sort_keys=True)} -->" @@ -95,6 +103,7 @@ def parse_marker(body: str) -> Marker | None: sha=data["sha"], since=datetime.fromisoformat(data["since"]), behind=data.get("behind"), + jobs=data.get("jobs"), ) except (json.JSONDecodeError, KeyError, TypeError, ValueError) as e: raise ValueError(f"Malformed fork CI marker: {match.group(0)!r}") from e @@ -217,6 +226,33 @@ def tamper_close_comment(*, run_url: str) -> str: "ungated": "Project CI in use (base branch without fork CI) - see the checks", } +# (commit status state, description) of a fork job, by its status and conclusion. A queued job is +# reported like a running one: each change is one more write, and commit statuses can only be posted +# one at a time (GraphQL has no mutation for them). +JOB_STATUSES = { + "running": ("pending", "Running in the fork - see Details"), + "success": ("success", "Passed in the fork - see Details"), + "failure": ("failure", "Failed in the fork - see Details"), + "timed_out": ("failure", "Timed out in the fork - see Details"), + "startup_failure": ("failure", "Failed to start in the fork - see Details"), + "cancelled": ("failure", "Cancelled in the fork - see Details"), +} + + +# Commit statuses cannot be deleted and have no neutral state. A job reported for an earlier run that +# the latest completed run does not have is overwritten with this, so it no longer shows as running +# or failed; `Fork CI / Tests (AMD)` is what tells the result. +STALE_JOB_STATUS = ("success", "Not in the latest fork run - see Fork CI / Tests (AMD)") + + +def job_status_context(job_name: str) -> str: + # A line break or control character would make an unreadable check name. + name = "".join(char if char.isprintable() else " " for char in job_name).strip() + if len(name) > JOB_NAME_MAX_LENGTH: + name = f"{name[: JOB_NAME_MAX_LENGTH - 1]}…" + return f"{JOB_STATUS_CONTEXT_PREFIX}{name}" + + STATUS_STATES = { "setup_required": "error", "awaiting_push": "pending", diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index 41ffaab6f2620..6b54f6c7fac26 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -57,11 +57,15 @@ CI_PASSED_CONTEXT, CI_PASSED_DESCRIPTION, CI_PASSED_REVOKED_DESCRIPTION, + JOB_STATUS_CONTEXT_PREFIX, + JOB_STATUSES, + STALE_JOB_STATUS, STATUS_CONTEXT, STATUS_STATES, Marker, fork_actions_url, inactivity_close_comment, + job_status_context, nudge_comment, parse_marker, released_comment, @@ -122,6 +126,22 @@ # can change what they run. pull_request_target and issue_comment use the base branch's files. MERGE_REF_EVENTS = ("pull_request", "pull_request_review", "pull_request_review_comment") +# Jobs of every fork run whose outcome the `Fork CI / Tests (AMD)` status already shows. +UNREPORTED_JOB_NAMES = GATED_RUN_JOB_NAMES | {GATE_JOB_NAME, "Platform: AMD"} +RUNNING_JOB_STATUSES = frozenset({"queued", "waiting", "pending", "requested", "in_progress"}) +JOB_STATE_RANKS = {"failure": 0, "pending": 1, "success": 2} +# Commit statuses count towards GitHub's limit of 500 content-creating requests per hour, shared by +# every workflow of the repository. At four runs an hour this keeps job statuses to at most about +# half of it; the rest follow on the next run, failures first. See "Job statuses" in the design doc. +MAX_JOB_STATUSES_PER_RUN = 60 +# Fixed text, so the note is posted to Slack once while deferrals persist; the run log lists the pull +# requests. +JOB_STATUSES_DEFERRED_NOTE = ( + f"job statuses did not fit in the budget of {MAX_JOB_STATUSES_PER_RUN} writes per run and are " + "posted by later runs; if this persists, set `job_statuses = false` in .github/fork-ci-gate.toml " + "(see dev/AGENTS.md)" +) + RUNNING_STATUSES = frozenset({"QUEUED", "IN_PROGRESS", "PENDING", "WAITING", "REQUESTED"}) FAILED_CONCLUSIONS = frozenset({"FAILURE", "CANCELLED", "TIMED_OUT", "STARTUP_FAILURE", "ACTION_REQUIRED"}) SETUP_WORKFLOW_STATES = frozenset({"disabled_fork", "disabled_manually", "missing"}) @@ -212,6 +232,8 @@ class ForkObservation: suite_status: str | None = None suite_conclusion: str | None = None run_url: str | None = None + run_id: int | None = None + run_attempt: int | None = None workflow_state: str | None = None error: str | None = None @@ -429,6 +451,7 @@ def classify( find_bypass_run: Callable[[], str | None], changes_ci_files: Callable[[], bool | None], count_behind: Callable[[], int | None] = lambda: None, + jobs_reported: str | None = None, ) -> Plan: """ Decide the mutations for one gated pull request. @@ -443,6 +466,8 @@ def classify( commit the answer may not belong to. :param count_behind: how many commits the head commit is behind the base branch, None when unknown; called at the same time as ``changes_ci_files``. + :param jobs_reported: the completed fork run whose jobs are all reported as statuses, recorded + in the marker so that later runs do not read its jobs again. """ plan = Plan() labels = set(pr.labels) @@ -598,7 +623,7 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: plan, pr, state_comment( - Marker(state.value, pr.head_sha, since, behind=behind if green else None), + Marker(state.value, pr.head_sha, since, behind=behind if green else None, jobs=jobs_reported), fork=pr.head_repo, run_url=fork.run_url, base_ref=pr.base_ref, @@ -888,7 +913,7 @@ def fetch_fork_observations( fields.append( f"pr{index}: repository(owner: $o{index}, name: $n{index}) {{ object(oid: $s{index}) {{ " "... on Commit { checkSuites(first: 30) { nodes { createdAt status conclusion " - "workflowRun { url event workflow { resourcePath } } } } } } }" + "workflowRun { databaseId runAttempt url event workflow { resourcePath } } } } } } }" ) variables.update({f"o{index}": owner, f"n{index}": name, f"s{index}": pr.head_sha}) query = f"query({', '.join(declarations)}) {{ {' '.join(fields)} }}" @@ -916,10 +941,113 @@ def fetch_fork_observations( suite_status=latest["status"], suite_conclusion=latest["conclusion"], run_url=latest["workflowRun"]["url"], + run_id=latest["workflowRun"].get("databaseId"), + run_attempt=latest["workflowRun"].get("runAttempt"), ) return observations +def build_job_statuses(jobs: Iterable[dict[str, Any]]) -> dict[str, tuple[str, str, str]]: + """The ``(state, target URL, description)`` to report for each queued or started fork job, by context.""" + statuses: dict[str, tuple[str, str, str]] = {} + # GitHub compares contexts case-insensitively, so differently capitalised names are one context. + contexts: dict[str, str] = {} + for job in jobs: + if job["name"] in UNREPORTED_JOB_NAMES: + continue + if job.get("status") in RUNNING_JOB_STATUSES: + outcome = "running" + else: + outcome = job.get("conclusion") or "" + if outcome not in JOB_STATUSES: + # Skipped and neutral jobs did no work worth a check on the pull request. + continue + state, description = JOB_STATUSES[outcome] + context = contexts.setdefault( + job_status_context(job["name"]).casefold(), job_status_context(job["name"]) + ) + # Jobs can share a name, or two names can be the same once shortened; the worst one counts. + reported = statuses.get(context) + if reported is None or JOB_STATE_RANKS[state] < JOB_STATE_RANKS[reported[0]]: + statuses[context] = (state, job["html_url"], description) + return statuses + + +class JobStatusReporter: + """Reports each job of a pull request's fork run as a commit status, within a per-run budget.""" + + def __init__(self, client: GitHubClient, *, repo: str, budget: int = MAX_JOB_STATUSES_PER_RUN): + self._client = client + self._repo = repo + self.remaining = budget + self.deferred: list[int] = [] + + def report(self, pr: PullRequestSnapshot, fork: ForkObservation) -> str | None: + """ + Post the job statuses that changed; return the completed run's key once all are reported. + + The key is recorded in the marker, so a completed run's jobs are read once. A run in + progress, or one whose statuses did not all fit in the budget, is read again next time. The + key holds the run attempt, so a re-run in the fork is read again even when it ends with the + same conclusion. + """ + if fork.run_id is None or not pr.head_repo: + return None + key = ( + f"{fork.run_id}:{fork.run_attempt}:{fork.suite_conclusion}" + if fork.suite_status == "COMPLETED" + else None + ) + recorded = pr.marker.jobs if pr.marker is not None and pr.marker.sha == pr.head_sha else None + if key is not None and key == recorded: + return key + # Without writes left, the reads are spent for nothing: a backlog of hundreds of pull requests + # would otherwise cost two reads each on every run. + if self.remaining == 0: + self.deferred.append(pr.number) + return None + jobs = self._client.rest_get( + f"repos/{pr.head_repo}/actions/runs/{fork.run_id}/jobs?filter=latest&per_page=100", + paginate=True, + items_key="jobs", + ) + # By the case-folded context, as GitHub compares them. + current = { + status["context"].casefold(): ( + status["context"], + (status["state"], status.get("target_url") or "", status.get("description") or ""), + ) + for status in self._client.rest_get( + f"repos/{self._repo}/commits/{pr.head_sha}/status?per_page=100", + paginate=True, + items_key="statuses", + ) + } + desired = build_job_statuses(jobs) + if key is not None: + # Jobs of an earlier run of this commit that the completed run did not run, or skipped. + wanted = {context.casefold() for context in desired} + for folded, (context, (_, target_url, _)) in current.items(): + if context.startswith(JOB_STATUS_CONTEXT_PREFIX) and folded not in wanted: + desired[context] = (STALE_JOB_STATUS[0], target_url, STALE_JOB_STATUS[1]) + changes = sorted( + ( + (context, status) + for context, status in desired.items() + if current.get(context.casefold(), ("", None))[1] != status + ), + key=lambda change: (JOB_STATE_RANKS[change[1][0]], change[0]), + ) + for context, (state, target_url, description) in changes: + if self.remaining == 0: + self.deferred.append(pr.number) + return None + self._client.set_status(pr.number, pr.head_sha, state, target_url, description, context=context) + # Charged per write, so writes made before a later one fails still count. + self.remaining -= 1 + return key + + def fetch_workflow_state(client: GitHubClient, fork: str) -> str: try: return client.rest_get(f"repos/{fork}/actions/workflows/{CI_WORKFLOW_FILE}")["state"] @@ -1135,8 +1263,14 @@ def find_state_comment(client: GitHubClient, repo: str, pr: PullRequestSnapshot) def read_head_status(client: GitHubClient, repo: str, pr: PullRequestSnapshot) -> PullRequestSnapshot: """Read the reconciler's statuses from the head commit, for a snapshot that could not.""" - combined = client.rest_get(f"repos/{repo}/commits/{pr.head_sha}/status?per_page=100") - statuses = {status["context"]: status for status in combined.get("statuses") or []} + # Every page: with job statuses a commit can carry more than 100 contexts, and a missed `CI passed` + # success would not be revoked. + statuses = { + status["context"]: status + for status in client.rest_get( + f"repos/{repo}/commits/{pr.head_sha}/status?per_page=100", paginate=True, items_key="statuses" + ) + } status = statuses.get(STATUS_CONTEXT) ci_passed = statuses.get(CI_PASSED_CONTEXT) return replace( @@ -1359,6 +1493,7 @@ def run_reconcile( bypass_finder = BypassFinder( client, repo=repo, now=now, base_triggers=load_workflow_triggers(workflows_dir) ) + job_reporter = JobStatusReporter(client, repo=repo) for listed in prs: try: @@ -1373,6 +1508,15 @@ def run_reconcile( and needs_workflow_state(pr.marker) ): fork = ForkObservation(workflow_state=fetch_workflow_state(client, pr.head_repo)) + jobs_reported = None + if config.job_statuses: + # Job statuses are informational; failing to post them must not keep the pull request + # from being reconciled, such as a `CI passed` that no longer holds being revoked. + try: + jobs_reported = job_reporter.report(pr, fork) + except Exception as e: + console_print(f"[warning]#{pr.number}: reporting job statuses failed: {e}[/]") + unknown.append((pr.number, f"reporting job statuses failed: {type(e).__name__}: {e}")) plan = classify( pr, @@ -1385,6 +1529,7 @@ def run_reconcile( pull_request_changes_ci_files, client, repo, pr.number, pr.head_sha ), count_behind=functools.partial(count_commits_behind, client, repo, pr), + jobs_reported=jobs_reported, ) draft_change = apply_plan(client, pr, plan) if draft_change is not None: @@ -1402,6 +1547,12 @@ def run_reconcile( unknown.append((listed.number, f"{type(e).__name__}: {e}")) notes = [bypass_finder.incomplete] if bypass_finder.incomplete else [] + if job_reporter.deferred: + console_print( + f"[info]Job statuses of {len(job_reporter.deferred)} pull requests left for the next run: " + f"{job_reporter.deferred}[/]" + ) + notes.append(JOB_STATUSES_DEFERRED_NOTE) # The full list for the run log; Slack may show only part of a very long one. for number, reason in [*unknown, *attention]: console_print(f"[warning]#{number}: {reason}[/]") diff --git a/dev/breeze/tests/test_fork_ci_gate.py b/dev/breeze/tests/test_fork_ci_gate.py index 186dd3956737f..b52a0d29a1c78 100644 --- a/dev/breeze/tests/test_fork_ci_gate.py +++ b/dev/breeze/tests/test_fork_ci_gate.py @@ -320,6 +320,11 @@ def test_parse_config_defaults(): assert config.close_after_days == 7 assert config.allowlist == {} assert config.pilot == {} + assert config.job_statuses is True + + +def test_parse_config_turns_job_statuses_off(): + assert parse_config("job_statuses = false\n").job_statuses is False def test_parse_config_lowercases_pilot_logins(): @@ -334,6 +339,7 @@ def test_parse_config_lowercases_pilot_logins(): ("text", "message"), [ pytest.param("enabled = 1\n", "enabled must be", id="enabled-not-bool"), + pytest.param('job_statuses = "no"\n', "job_statuses must be", id="job-statuses-not-bool"), pytest.param( "enabled_since = 2026-01-01T00:00:00\n", "enabled_since must be", id="enabled-since-naive" ), diff --git a/dev/breeze/tests/test_fork_ci_messages.py b/dev/breeze/tests/test_fork_ci_messages.py index ab21a0c277ea1..659c22d8ff028 100644 --- a/dev/breeze/tests/test_fork_ci_messages.py +++ b/dev/breeze/tests/test_fork_ci_messages.py @@ -28,8 +28,11 @@ CI_PASSED_DESCRIPTION, CI_PASSED_REVOKED_DESCRIPTION, FOOTER, + JOB_NAME_MAX_LENGTH, + JOB_STATUS_CONTEXT_PREFIX, Marker, inactivity_close_comment, + job_status_context, nudge_comment, parse_marker, released_comment, @@ -43,7 +46,11 @@ @pytest.mark.parametrize( "marker", - [pytest.param(MARKER, id="without-behind"), pytest.param(replace(MARKER, behind=12), id="behind")], + [ + pytest.param(MARKER, id="without-behind"), + pytest.param(replace(MARKER, behind=12), id="behind"), + pytest.param(replace(MARKER, jobs="123:SUCCESS"), id="jobs"), + ], ) def test_marker_round_trip(marker): assert parse_marker(f"text before\n{render_marker(marker)}\ntext after") == marker @@ -51,6 +58,7 @@ def test_marker_round_trip(marker): def test_marker_without_behind_keeps_its_rendering(): assert "behind" not in render_marker(MARKER) + assert "jobs" not in render_marker(MARKER) @pytest.mark.parametrize( @@ -123,3 +131,15 @@ def test_setup_texts_give_the_exact_command(text): def test_status_descriptions_fit_github_limit(): descriptions = [*_STATUS_DESCRIPTIONS.values(), CI_PASSED_DESCRIPTION, CI_PASSED_REVOKED_DESCRIPTION] assert all(len(description) < 140 for description in descriptions) + + +@pytest.mark.parametrize( + ("name", "expected"), + [ + ("Static checks", "Static checks"), + ("a\nb\tc", "a b c"), + ("x" * (JOB_NAME_MAX_LENGTH + 5), "x" * (JOB_NAME_MAX_LENGTH - 1) + "…"), + ], +) +def test_job_status_context(name, expected): + assert job_status_context(name) == f"{JOB_STATUS_CONTEXT_PREFIX}{expected}" diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index ec8828ac9b7df..9f3aa61f751d0 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -30,20 +30,25 @@ from airflow_breeze.utils.fork_ci_github import GitHubClient, GitHubError from airflow_breeze.utils.fork_ci_messages import ( CI_PASSED_CONTEXT, + JOB_STATUS_CONTEXT_PREFIX, + STALE_JOB_STATUS, STATUS_CONTEXT, Marker, parse_marker, render_marker, ) from airflow_breeze.utils.fork_ci_reconcile import ( + JOB_STATUSES_DEFERRED_NOTE, BaseGateConfigs, BypassFinder, DraftChange, ForkObservation, ForkState, + JobStatusReporter, Plan, PullRequestSnapshot, apply_plan, + build_job_statuses, build_slack_payload, changes_ci_definition, classify, @@ -1226,15 +1231,26 @@ def test_parse_pull_request_leaves_statuses_of_another_commit_unread(): id="both-statuses", ), pytest.param([], None, None, id="no-statuses"), + pytest.param( + [ + *({"context": f"{JOB_STATUS_CONTEXT_PREFIX}job {i}", "state": "success"} for i in range(150)), + {"context": CI_PASSED_CONTEXT, "state": "success"}, + ], + None, + "success", + id="ci-passed-after-many-job-statuses", + ), ], ) def test_read_head_status(client, statuses, expected_status, expected_ci_passed): - client.rest_get.return_value = {"statuses": statuses} + client.rest_get.return_value = statuses pr = dataclasses.replace(parse_pull_request(_pr_node(1)), head_status_unread=True) read = read_head_status(client, REPO, pr) - client.rest_get.assert_called_once_with(f"repos/{REPO}/commits/sha1/status?per_page=100") + client.rest_get.assert_called_once_with( + f"repos/{REPO}/commits/sha1/status?per_page=100", paginate=True, items_key="statuses" + ) assert read.current_status == expected_status assert read.ci_passed_state == expected_ci_passed assert read.head_status_unread is False @@ -1332,6 +1348,8 @@ def test_fork_observations_use_the_fork_name_and_latest_suite(client): "status": "COMPLETED", "conclusion": "SUCCESS", "workflowRun": { + "databaseId": 77, + "runAttempt": 2, "url": "new", "event": "push", "workflow": {"resourcePath": "/x/ci-amd.yml"}, @@ -1366,7 +1384,7 @@ def test_fork_observations_use_the_fork_name_and_latest_suite(client): observations = fetch_fork_observations(client, [pr]) assert observations[1] == ForkObservation( - suite_status="COMPLETED", suite_conclusion="SUCCESS", run_url="new" + suite_status="COMPLETED", suite_conclusion="SUCCESS", run_url="new", run_id=77, run_attempt=2 ) assert client.graphql.call_args.args[1] == {"o0": "someone", "n0": "my-airflow", "s0": "sha1"} @@ -1717,3 +1735,341 @@ def test_count_commits_behind(client, side_effect, expected): assert count_commits_behind(client, REPO, _pr(base_ref="v3-1-test")) == expected assert client.rest_get.call_args.args[0] == f"repos/{REPO}/compare/v3-1-test...someone:abc123?per_page=1" + + +def _fork_job(name, status="completed", conclusion="success", url=None): + return { + "name": name, + "status": status, + "conclusion": conclusion, + "html_url": url or f"https://job/{name}", + } + + +def test_build_job_statuses_reports_each_started_or_queued_job(): + jobs = [ + _fork_job("Static checks"), + _fork_job("Postgres tests: core / A", conclusion="failure"), + _fork_job("Helm tests", status="in_progress", conclusion=None), + _fork_job("Kubernetes tests", status="queued", conclusion=None), + _fork_job("Special tests", conclusion="cancelled"), + _fork_job("Java SDK tests", conclusion="skipped"), + _fork_job("Fork CI gate"), + _fork_job("CI passed"), + _fork_job("Check if the run is needed"), + ] + + statuses = build_job_statuses(jobs) + + assert { + context.removeprefix(JOB_STATUS_CONTEXT_PREFIX): state for context, (state, _, _) in statuses.items() + } == { + "Static checks": "success", + "Postgres tests: core / A": "failure", + "Helm tests": "pending", + "Kubernetes tests": "pending", + "Special tests": "failure", + } + assert statuses[f"{JOB_STATUS_CONTEXT_PREFIX}Static checks"][1] == "https://job/Static checks" + + +def test_build_job_statuses_reports_the_worst_of_jobs_sharing_a_name(): + jobs = [ + _fork_job("Tests", url="ok"), + _fork_job("Tests", conclusion="failure", url="bad"), + _fork_job("Tests", url="ok2"), + ] + + assert build_job_statuses(jobs)[f"{JOB_STATUS_CONTEXT_PREFIX}Tests"][:2] == ("failure", "bad") + + +@pytest.mark.parametrize("name", ["Tests (AMD)", "CI passed (fork)", "x\nCI passed"]) +def test_build_job_statuses_cannot_post_the_reconciler_contexts(name): + (context,) = build_job_statuses([_fork_job(name)]) + + assert context.startswith(JOB_STATUS_CONTEXT_PREFIX) + assert context not in (STATUS_CONTEXT, CI_PASSED_CONTEXT) + assert "\n" not in context + + +RUNNING_RUN = ForkObservation(suite_status="IN_PROGRESS", run_url=RUN_URL, run_id=1) +COMPLETED_RUN = ForkObservation( + suite_status="COMPLETED", suite_conclusion="FAILURE", run_url=RUN_URL, run_id=1, run_attempt=1 +) + + +def _reporter_client(client, jobs, current=()): + def rest_get(path, paginate=False, items_key=None): + if path.startswith("repos/someone/airflow/actions/runs/1/jobs"): + return jobs + if path.startswith(f"repos/{REPO}/commits/abc123/status"): + return list(current) + raise AssertionError(path) + + client.rest_get.side_effect = rest_get + return client + + +def test_job_status_reporter_posts_changed_statuses_failures_first(client): + already = { + "context": f"{JOB_STATUS_CONTEXT_PREFIX}Static checks", + "state": "success", + "target_url": "https://job/Static checks", + "description": "Passed in the fork - see Details", + } + _reporter_client( + client, + [_fork_job("Static checks"), _fork_job("Helm tests"), _fork_job("Unit", conclusion="failure")], + [already], + ) + + key = JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) + + assert key == "1:1:FAILURE" + assert [(c.args[2], c.kwargs["context"]) for c in client.set_status.call_args_list] == [ + ("failure", f"{JOB_STATUS_CONTEXT_PREFIX}Unit"), + ("success", f"{JOB_STATUS_CONTEXT_PREFIX}Helm tests"), + ] + + +def test_job_status_reporter_leaves_what_does_not_fit_in_the_budget_for_the_next_run(client): + _reporter_client(client, [_fork_job("A", conclusion="failure"), _fork_job("B"), _fork_job("C")]) + reporter = JobStatusReporter(client, repo=REPO, budget=2) + + assert reporter.report(_pr(), COMPLETED_RUN) is None + assert [c.kwargs["context"] for c in client.set_status.call_args_list] == [ + f"{JOB_STATUS_CONTEXT_PREFIX}A", + f"{JOB_STATUS_CONTEXT_PREFIX}B", + ] + assert reporter.remaining == 0 + assert reporter.deferred == [42] + + +def test_job_status_reporter_does_not_record_a_run_in_progress(client): + _reporter_client(client, [_fork_job("A", status="in_progress", conclusion=None)]) + + assert JobStatusReporter(client, repo=REPO).report(_pr(), RUNNING_RUN) is None + client.set_status.assert_called_once() + + +@pytest.mark.parametrize( + ("marker_jobs", "marker_sha", "reads"), + [ + pytest.param("1:1:FAILURE", "abc123", False, id="already-reported"), + pytest.param("1:1:SUCCESS", "abc123", True, id="re-run-with-another-conclusion"), + pytest.param("1:2:FAILURE", "abc123", True, id="another-attempt-with-the-same-conclusion"), + pytest.param("1:1:FAILURE", "other", True, id="marker-of-another-commit"), + ], +) +def test_job_status_reporter_reads_a_completed_run_once(client, marker_jobs, marker_sha, reads): + _reporter_client(client, []) + pr = _pr(marker=dataclasses.replace(_marker("failed", sha=marker_sha), jobs=marker_jobs)) + + assert JobStatusReporter(client, repo=REPO).report(pr, COMPLETED_RUN) == "1:1:FAILURE" + assert client.rest_get.called is reads + + +@pytest.mark.parametrize( + "fork", + [ForkObservation(), ForkObservation(error="fork not readable"), ForkObservation(workflow_state="active")], +) +def test_job_status_reporter_without_a_fork_run(client, fork): + assert JobStatusReporter(client, repo=REPO).report(_pr(), fork) is None + client.rest_get.assert_not_called() + + +def test_classify_records_the_reported_run_in_the_marker(): + plan = classify( + _pr(marker=_marker("running")), + dataclasses.replace(COMPLETED_RUN), + config=CONFIG, + now=NOW, + repo=REPO, + find_bypass_run=mock.Mock(return_value=None), + changes_ci_files=mock.Mock(return_value=False), + jobs_reported="1:1:FAILURE", + ) + + assert parse_marker(plan.marker_comment).jobs == "1:1:FAILURE" + + +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) +def test_run_reconcile_reports_job_statuses_before_classifying( + mock_classify, mock_reporter, client, tmp_path +): + mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_reporter.return_value.report.return_value = "7:SUCCESS" + mock_reporter.return_value.deferred = [] + client.graphql.side_effect = [ + _search_result(), + _search_result(), + _search_result(_pr_node(1, headRepository={"nameWithOwner": "someone/airflow"})), + {"pr0": {"object": None}}, + ] + client.rest_get_pages.return_value = [{"total_count": 0, "workflow_runs": []}] + client.rest_get.return_value = {"state": "active"} + + run_reconcile( + client, + CONFIG, + now=NOW, + run_url="https://run", + output_file=tmp_path / "s.json", + workflows_dir=tmp_path, + ) + + (pr, fork), _ = mock_reporter.return_value.report.call_args + assert pr.number == 1 + assert mock_classify.call_args.kwargs["jobs_reported"] == "7:SUCCESS" + + +def test_build_job_statuses_reports_a_queued_job_like_a_running_one(): + queued = build_job_statuses([_fork_job("A", status="queued", conclusion=None, url="u")]) + running = build_job_statuses([_fork_job("A", status="in_progress", conclusion=None, url="u")]) + + assert queued == running + + +def _stale(state): + return { + "context": f"{JOB_STATUS_CONTEXT_PREFIX}Gone", + "state": state, + "target_url": "https://job/Gone", + "description": "Running in the fork - see Details", + } + + +@pytest.mark.parametrize( + ("fork", "resolved"), + [pytest.param(COMPLETED_RUN, True, id="completed"), pytest.param(RUNNING_RUN, False, id="running")], +) +def test_job_status_reporter_resolves_jobs_missing_from_a_completed_run(client, fork, resolved): + _reporter_client(client, [], [_stale("pending")]) + + JobStatusReporter(client, repo=REPO).report(_pr(), fork) + + expected = [ + mock.call( + 42, + "abc123", + STALE_JOB_STATUS[0], + "https://job/Gone", + STALE_JOB_STATUS[1], + context=f"{JOB_STATUS_CONTEXT_PREFIX}Gone", + ) + ] + assert client.set_status.call_args_list == (expected if resolved else []) + + +def test_job_status_reporter_leaves_other_contexts_alone(client): + other = {"context": STATUS_CONTEXT, "state": "pending", "target_url": "u", "description": "d"} + _reporter_client(client, [], [other]) + + JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) + + client.set_status.assert_not_called() + + +def test_job_status_reporter_charges_each_write_even_when_a_later_one_fails(client): + _reporter_client(client, [_fork_job("A", conclusion="failure"), _fork_job("B"), _fork_job("C")]) + client.set_status.side_effect = [None, GitHubError("HTTP 500")] + reporter = JobStatusReporter(client, repo=REPO, budget=10) + + with pytest.raises(GitHubError): + reporter.report(_pr(), COMPLETED_RUN) + + assert reporter.remaining == 9 + + +def _reconcile_with_reporter(client, tmp_path, config=CONFIG): + client.graphql.side_effect = [ + _search_result(), + _search_result(), + _search_result(_pr_node(1, headRepository={"nameWithOwner": "someone/airflow"})), + {"pr0": {"object": None}}, + ] + client.rest_get_pages.return_value = [{"total_count": 0, "workflow_runs": []}] + client.rest_get.return_value = {"state": "active"} + return run_reconcile( + client, + config, + now=NOW, + run_url="https://run", + output_file=tmp_path / "s.json", + workflows_dir=tmp_path, + ) + + +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) +def test_run_reconcile_classifies_when_reporting_job_statuses_fails( + mock_classify, mock_reporter, client, tmp_path +): + mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_reporter.return_value.report.side_effect = GitHubError("HTTP 502") + mock_reporter.return_value.deferred = [] + + assert _reconcile_with_reporter(client, tmp_path) is True + + assert mock_classify.call_args.kwargs["jobs_reported"] is None + assert "reporting job statuses failed" in (tmp_path / "s.json").read_text() + + +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) +def test_run_reconcile_without_job_statuses(mock_classify, mock_reporter, client, tmp_path): + mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_reporter.return_value.deferred = [] + + _reconcile_with_reporter(client, tmp_path, config=dataclasses.replace(CONFIG, job_statuses=False)) + + mock_reporter.return_value.report.assert_not_called() + assert mock_classify.call_args.kwargs["jobs_reported"] is None + + +def test_job_status_reporter_does_not_read_without_writes_left(client): + reporter = JobStatusReporter(client, repo=REPO, budget=0) + + assert reporter.report(_pr(), COMPLETED_RUN) is None + client.rest_get.assert_not_called() + assert reporter.deferred == [42] + + +@pytest.mark.parametrize(("deferred", "has_alerts"), [([1], True), ([], False)]) +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.JobStatusReporter", autospec=True) +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.classify", autospec=True) +def test_run_reconcile_reports_deferred_job_statuses_to_slack( + mock_classify, mock_reporter, client, tmp_path, deferred, has_alerts +): + mock_classify.return_value = Plan(state=ForkState.RUNNING) + mock_reporter.return_value.report.return_value = None + mock_reporter.return_value.deferred = deferred + + assert _reconcile_with_reporter(client, tmp_path) is has_alerts + + if has_alerts: + assert JOB_STATUSES_DEFERRED_NOTE in (tmp_path / "s.json").read_text() + + +def test_build_job_statuses_treats_contexts_case_insensitively(): + statuses = build_job_statuses( + [_fork_job("Tests", conclusion="failure", url="bad"), _fork_job("tests", url="ok")] + ) + + assert statuses == { + f"{JOB_STATUS_CONTEXT_PREFIX}Tests": ("failure", "bad", "Failed in the fork - see Details") + } + + +def test_job_status_reporter_compares_contexts_case_insensitively(client): + posted = { + "context": f"{JOB_STATUS_CONTEXT_PREFIX}TESTS", + "state": "failure", + "target_url": "https://job/Tests", + "description": "Failed in the fork - see Details", + } + _reporter_client(client, [_fork_job("Tests", conclusion="failure")], [posted]) + + assert JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) == "1:1:FAILURE" + client.set_status.assert_not_called() From e52fced76c45cd9c032a36512171462736c546c3 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Sun, 25 Oct 2026 20:10:11 +0100 Subject: [PATCH 2/6] Cancel the workflow runs of pull requests that have already merged (#75619) Merging a pull request does not stop its pull_request runs, so they keep using the project's GitHub-hosted minutes and self-hosted runners for a result nobody reads once the change is on its target branch. Until now maintainers cancelled such runs by hand, when they noticed them. Generated-by: Claude Opus 5 (cherry picked from commit e5f2c87a3ab06c7dde2b84c64d9fd9a555b3ad12) --- .github/fork-ci-gate.toml | 5 + .github/workflows/fork-ci-monitor.yml | 4 +- dev/breeze/doc/08_ci_tasks.rst | 3 +- .../08_fork_ci_for_external_contributors.md | 33 ++- .../src/airflow_breeze/utils/fork_ci_gate.py | 5 + .../airflow_breeze/utils/fork_ci_github.py | 8 + .../airflow_breeze/utils/fork_ci_reconcile.py | 8 + .../utils/merged_pull_request_runs.py | 211 ++++++++++++++ dev/breeze/tests/test_fork_ci_gate.py | 8 + dev/breeze/tests/test_fork_ci_github.py | 17 ++ dev/breeze/tests/test_fork_ci_reconcile.py | 40 +++ .../tests/test_merged_pull_request_runs.py | 266 ++++++++++++++++++ 12 files changed, 604 insertions(+), 4 deletions(-) create mode 100644 dev/breeze/src/airflow_breeze/utils/merged_pull_request_runs.py create mode 100644 dev/breeze/tests/test_merged_pull_request_runs.py diff --git a/.github/fork-ci-gate.toml b/.github/fork-ci-gate.toml index 6fc7a0de1ce3c..6119f610ef2d8 100644 --- a/.github/fork-ci-gate.toml +++ b/.github/fork-ci-gate.toml @@ -34,6 +34,11 @@ close_after_days = 7 # status is posted either way. job_statuses = true +# Cancel the unfinished `pull_request` runs of pull requests that have already merged, of any +# author, so they stop using GitHub-hosted minutes and self-hosted runners for a result nobody +# reads. The fork CI monitor does it on its schedule. +cancel_merged_runs = true + # Organisations exempted under AIP-120 §2.2. Keys are display names used in comments, # values are GitHub logins. Changes go through a reviewed pull request. [allowlist] diff --git a/.github/workflows/fork-ci-monitor.yml b/.github/workflows/fork-ci-monitor.yml index ec34b9615b028..278787be5af74 100644 --- a/.github/workflows/fork-ci-monitor.yml +++ b/.github/workflows/fork-ci-monitor.yml @@ -41,8 +41,8 @@ jobs: name: "Reconcile pull requests gated to fork CI" if: github.repository == 'apache/airflow' permissions: - # Listing the runs and jobs of the gate tampering check, and deleting the runs that only - # gated a pull request to fork CI. + # Listing the runs and jobs of the gate tampering check, deleting the runs that only gated a + # pull request to fork CI, and cancelling the unfinished runs of merged pull requests. actions: write contents: read # Labels, comments and closing of gated pull requests. diff --git a/dev/breeze/doc/08_ci_tasks.rst b/dev/breeze/doc/08_ci_tasks.rst index ca16b4483febc..b1bf0d9f5fb99 100644 --- a/dev/breeze/doc/08_ci_tasks.rst +++ b/dev/breeze/doc/08_ci_tasks.rst @@ -188,7 +188,8 @@ Pull requests from external contributors can run their CI in the contributor's f gated pull requests ``fork ci required``, keeps them in draft until the fork run is green, maintains one state comment and a ``Fork CI / Tests (AMD)`` commit status linking to the run in the fork, closes pull requests without fork CI activity for a week, closes gated pull requests that made project CI run -by changing workflows, and writes everything it could not classify to a Slack message. With +by changing workflows, and writes everything it could not classify to a Slack message. It also cancels +the unfinished workflow runs of pull requests, of any author, that have already merged. With ``--dry-run`` it prints the changes instead of applying them. See `Fork CI for external contributors `_ for the design. diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index 799a233e9ca32..1f3983739442d 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -787,6 +787,36 @@ name within its time limit, and its run is the evidence. A re-run is kept too: d deletes every attempt, and an earlier attempt may have run project CI. Nothing is deleted for a pull request in the `gate_bypassed` state. A dry run only reports the deletions. +### Cancelling runs of merged pull requests + +Merging a pull request does not stop its `pull_request` runs. They keep using `apache/airflow`'s +capacity, GitHub-hosted minutes from the ASF's allocation and the self-hosted runners, for a +result nobody reads once the change is on its target branch. This is not specific to fork CI; the +monitor does it because it already runs every 15 minutes with `actions: write`. + +Each monitor run lists the `pull_request` runs of `apache/airflow` that are `queued`, +`in_progress`, `waiting`, `pending` or `requested`, and looks up the pull requests of their head +branches in one GraphQL query per 25 branches, paging through every pull request of a branch: a +common branch name such as `main` has pull requests from many forks, and an open one with the +run's head must not be missed. A run is cancelled when: + +- a merged pull request has the run's head repository, branch and commit (runs of pull requests + from forks do not list their pull request, so they are matched this way); +- no open pull request has the same head, since a branch can be reused for a new pull request; +- the run was created before the merge. A workflow triggered by the merge itself, such as one + on the `closed` event, is meant to run. + +It applies to the pull requests of every author, not only gated ones. A run is cancelled with +`POST /repos/apache/airflow/actions/runs//cancel`; one still `in_progress` 30 minutes after the +merge, for example because a step ignores the cancellation, is force-cancelled +(`.../force-cancel`). A run that finished in the meantime (HTTP 409) is not a problem; any other +failure is reported to Slack with the pull request, and a failure of the whole pass is reported +as a note, without affecting the reconciliation of gated pull requests. A dry run only reports +the cancellations. `cancel_merged_runs = false` in `.github/fork-ci-gate.toml` turns it off. + +Only runs of the merged head commit are matched. A run of an earlier commit of the pull request is +normally cancelled by the workflow's concurrency group when the next commit is pushed. + ### Far behind the base branch A fork run tests the pushed commit as it is, while project CI tests it merged with the base @@ -841,7 +871,8 @@ Per run, with ~700 gated pull requests: ~28 GraphQL calls for pass 2, ~14 for fo calls for the recent `pull_request` runs, one REST call per base branch for its workflow files, a jobs call and a compare call only per long `ci-amd.yml` run of a gated pull request, one changed-files call per head commit that turns green, plus one mutation per actual state -change. Job statuses add a jobs call and a status call per pull request with a fork run in +change. Cancelling runs of merged pull requests adds five paginated run listings, one GraphQL +query per 25 head branches with unfinished runs, and one mutation per cancelled run. Job statuses add a jobs call and a status call per pull request with a fork run in progress or a completed run not yet reported, and up to two status mutations per job, at most `MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Well inside the `GITHUB_TOKEN` limits at a 15-minute cadence. The reconciler logs its call counts so the diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py index 8ce90a34d97b3..dc05a3b67a570 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py @@ -129,6 +129,7 @@ class GateConfig: allowlist: Mapping[str, tuple[str, ...]] = field(default_factory=dict) pilot: Mapping[str, datetime] = field(default_factory=dict) job_statuses: bool = True + cancel_merged_runs: bool = True def is_allowlisted(self, login: str) -> bool: wanted = login.casefold() @@ -177,6 +178,9 @@ def parse_config(text: str) -> GateConfig: job_statuses = data.get("job_statuses", True) if not isinstance(job_statuses, bool): raise ValueError(f"job_statuses must be true or false, got {job_statuses!r}") + cancel_merged_runs = data.get("cancel_merged_runs", True) + if not isinstance(cancel_merged_runs, bool): + raise ValueError(f"cancel_merged_runs must be true or false, got {cancel_merged_runs!r}") return GateConfig( enabled=enabled, enabled_since=enabled_since, @@ -184,6 +188,7 @@ def parse_config(text: str) -> GateConfig: allowlist=allowlist, pilot=pilot, job_statuses=job_statuses, + cancel_merged_runs=cancel_merged_runs, ) diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_github.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_github.py index 73a44503acb8e..fe3fdfb651eb4 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_github.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_github.py @@ -172,6 +172,14 @@ def delete_run(self, pr: int, run_id: int) -> None: None, ) + def cancel_run(self, pr: int, run_id: int, *, force: bool = False) -> None: + action = "force-cancel" if force else "cancel" + self._mutate( + f"#{pr}: {action} workflow run {run_id}", + ["api", "-X", "POST", f"repos/{self.repo}/actions/runs/{run_id}/{action}"], + None, + ) + def set_status( self, pr: int, diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index 6b54f6c7fac26..00698b7849499 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -74,6 +74,7 @@ tamper_close_comment, ungated_comment, ) +from airflow_breeze.utils.merged_pull_request_runs import cancel_merged_pull_request_runs CI_WORKFLOW_FILE = "ci-amd.yml" # The files the gate job of ci-amd.yml reads from a pull request's base branch. @@ -1547,6 +1548,13 @@ def run_reconcile( unknown.append((listed.number, f"{type(e).__name__}: {e}")) notes = [bypass_finder.incomplete] if bypass_finder.incomplete else [] + if config.cancel_merged_runs: + # Not tied to gated pull requests; a failure here must not hide the reconciliation's own. + try: + unknown.extend(cancel_merged_pull_request_runs(client, repo=repo, now=now)) + except Exception as e: + console_print(f"[error]Cancelling runs of merged pull requests failed: {e}[/]") + notes.append(f"cancelling runs of merged pull requests failed: {type(e).__name__}: {e}") if job_reporter.deferred: console_print( f"[info]Job statuses of {len(job_reporter.deferred)} pull requests left for the next run: " diff --git a/dev/breeze/src/airflow_breeze/utils/merged_pull_request_runs.py b/dev/breeze/src/airflow_breeze/utils/merged_pull_request_runs.py new file mode 100644 index 0000000000000..7eed315630ba5 --- /dev/null +++ b/dev/breeze/src/airflow_breeze/utils/merged_pull_request_runs.py @@ -0,0 +1,211 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Cancels the workflow runs that are still going for pull requests that have already merged. + +Merging a pull request does not stop its `pull_request` runs, which keep using the project's +GitHub-hosted minutes and self-hosted runners for a result nobody reads. The fork CI monitor calls +this on its schedule. See "Cancelling runs of merged pull requests" in +dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md. +""" + +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import datetime, timedelta +from typing import TYPE_CHECKING, Any + +from airflow_breeze.utils.console import console_print +from airflow_breeze.utils.fork_ci_gate import parse_github_datetime + +if TYPE_CHECKING: + from airflow_breeze.utils.fork_ci_github import GitHubClient + +# The statuses of a run that has not finished. A run waiting for approval is "completed". +UNFINISHED_RUN_STATUSES = ("queued", "in_progress", "waiting", "pending", "requested") +# A cancelled run whose jobs keep running, for example in a step that ignores the signal, is +# force-cancelled once this long has passed since the merge, which leaves a normal cancel at least +# one monitor run to work. +FORCE_CANCEL_AFTER = timedelta(minutes=30) +BRANCH_BATCH_SIZE = 25 + + +@dataclass(frozen=True) +class UnfinishedRun: + id: int + workflow: str + status: str + head_repo: str + head_branch: str + head_sha: str + created_at: datetime + + +@dataclass(frozen=True) +class PullRequestHead: + number: int + state: str + head_repo: str + head_sha: str + merged_at: datetime | None + + +def list_unfinished_runs(client: GitHubClient, repo: str) -> list[UnfinishedRun]: + runs = [] + for status in UNFINISHED_RUN_STATUSES: + for page in client.rest_get_pages( + f"repos/{repo}/actions/runs?event=pull_request&status={status}&per_page=100" + ): + for run in page["workflow_runs"]: + runs.append( + UnfinishedRun( + id=run["id"], + workflow=run.get("name") or "", + status=run["status"], + head_repo=(run.get("head_repository") or {}).get("full_name") or "", + head_branch=run.get("head_branch") or "", + head_sha=run["head_sha"], + created_at=parse_github_datetime(run["created_at"]), # type: ignore[arg-type] + ) + ) + return runs + + +PULL_REQUEST_FIELDS = ( + "pageInfo { hasNextPage endCursor } " + "nodes { number state mergedAt headRefOid headRepository { nameWithOwner } }" +) +BRANCH_PAGE_QUERY = ( + "query($owner: String!, $name: String!, $branch: String!, $cursor: String) " + "{ repository(owner: $owner, name: $name) { pullRequests(headRefName: $branch, first: 100, " + f"after: $cursor) {{ {PULL_REQUEST_FIELDS} }} }} }}" +) + + +def _parse_pull_requests(connection: dict[str, Any] | None) -> list[PullRequestHead]: + return [ + PullRequestHead( + number=node["number"], + state=node["state"], + head_repo=(node.get("headRepository") or {}).get("nameWithOwner") or "", + head_sha=node["headRefOid"], + merged_at=parse_github_datetime(node.get("mergedAt") or ""), + ) + for node in (connection or {}).get("nodes") or [] + ] + + +def find_pull_requests_by_branch( + client: GitHubClient, repo: str, branches: Iterable[str] +) -> dict[str, list[PullRequestHead]]: + """ + Every pull request of each head branch name, in any state and from any head repository. + + All of them are read, not only the latest: a common branch name such as ``main`` has pull + requests from many forks, and an open one with a run's head must not be missed. + """ + owner, name = repo.split("/", 1) + found: dict[str, list[PullRequestHead]] = {} + unique = sorted(set(branches)) + for start in range(0, len(unique), BRANCH_BATCH_SIZE): + batch = unique[start : start + BRANCH_BATCH_SIZE] + declarations = ", ".join(f"$b{index}: String!" for index in range(len(batch))) + fields = " ".join( + f"b{index}: pullRequests(headRefName: $b{index}, first: 100) {{ {PULL_REQUEST_FIELDS} }}" + for index in range(len(batch)) + ) + data = client.graphql( + f"query($owner: String!, $name: String!, {declarations}) " + f"{{ repository(owner: $owner, name: $name) {{ {fields} }} }}", + {"owner": owner, "name": name, **{f"b{index}": branch for index, branch in enumerate(batch)}}, + )["repository"] + for index, branch in enumerate(batch): + connection = data[f"b{index}"] or {} + found[branch] = _parse_pull_requests(connection) + while (connection.get("pageInfo") or {}).get("hasNextPage"): + connection = client.graphql( + BRANCH_PAGE_QUERY, + { + "owner": owner, + "name": name, + "branch": branch, + "cursor": connection["pageInfo"]["endCursor"], + }, + )["repository"]["pullRequests"] + found[branch].extend(_parse_pull_requests(connection)) + return found + + +def find_merged_pull_request( + run: UnfinishedRun, pull_requests: Iterable[PullRequestHead] +) -> PullRequestHead | None: + """ + The merged pull request a run was started for, if no open pull request has the same head. + + A run is matched by head repository, branch and commit, because runs of pull requests from + forks do not list their pull request. Only runs started before the merge count: a workflow + triggered by the merge itself, such as one on `closed`, is meant to run. + """ + same_head = [ + pr + for pr in pull_requests + if pr.head_repo.casefold() == run.head_repo.casefold() and pr.head_sha == run.head_sha + ] + if any(pr.state == "OPEN" for pr in same_head): + return None + return next( + ( + pr + for pr in same_head + if pr.state == "MERGED" and pr.merged_at is not None and run.created_at < pr.merged_at + ), + None, + ) + + +def cancel_merged_pull_request_runs( + client: GitHubClient, *, repo: str, now: datetime +) -> list[tuple[int, str]]: + """Cancel the unfinished runs of merged pull requests; return ``(pull request, problem)`` for Slack.""" + runs = list_unfinished_runs(client, repo) + if not runs: + return [] + pull_requests = find_pull_requests_by_branch(client, repo, (run.head_branch for run in runs)) + problems: list[tuple[int, str]] = [] + cancelled = 0 + for run in runs: + merged = find_merged_pull_request(run, pull_requests.get(run.head_branch, [])) + if merged is None: + continue + force = ( + run.status == "in_progress" + and merged.merged_at is not None + and now - merged.merged_at >= FORCE_CANCEL_AFTER + ) + try: + client.cancel_run(merged.number, run.id, force=force) + cancelled += 1 + except Exception as e: + # A run that finished meanwhile cannot be cancelled; that is the outcome wanted anyway. + # gh reports the status as "(HTTP 409)"; a bare "409" could be part of the run id. + if "(HTTP 409)" in str(e): + continue + problems.append((merged.number, f"cancelling run {run.id} ({run.workflow}) failed: {e}")) + verb = "Would cancel" if client.dry_run else "Cancelled" + console_print(f"[info]{verb} {cancelled} runs of merged pull requests[/]") + return problems diff --git a/dev/breeze/tests/test_fork_ci_gate.py b/dev/breeze/tests/test_fork_ci_gate.py index b52a0d29a1c78..e641e45ca8277 100644 --- a/dev/breeze/tests/test_fork_ci_gate.py +++ b/dev/breeze/tests/test_fork_ci_gate.py @@ -321,12 +321,17 @@ def test_parse_config_defaults(): assert config.allowlist == {} assert config.pilot == {} assert config.job_statuses is True + assert config.cancel_merged_runs is True def test_parse_config_turns_job_statuses_off(): assert parse_config("job_statuses = false\n").job_statuses is False +def test_parse_config_turns_cancelling_merged_runs_off(): + assert parse_config("cancel_merged_runs = false\n").cancel_merged_runs is False + + def test_parse_config_lowercases_pilot_logins(): config = parse_config('[pilot]\n"Some-User" = 2026-10-20T00:00:00Z\n') @@ -340,6 +345,9 @@ def test_parse_config_lowercases_pilot_logins(): [ pytest.param("enabled = 1\n", "enabled must be", id="enabled-not-bool"), pytest.param('job_statuses = "no"\n', "job_statuses must be", id="job-statuses-not-bool"), + pytest.param( + "cancel_merged_runs = 0\n", "cancel_merged_runs must be", id="cancel-merged-runs-not-bool" + ), pytest.param( "enabled_since = 2026-01-01T00:00:00\n", "enabled_since must be", id="enabled-since-naive" ), diff --git a/dev/breeze/tests/test_fork_ci_github.py b/dev/breeze/tests/test_fork_ci_github.py index d993add8bb53c..09257d0697cd7 100644 --- a/dev/breeze/tests/test_fork_ci_github.py +++ b/dev/breeze/tests/test_fork_ci_github.py @@ -128,6 +128,7 @@ def test_dry_run_does_not_mutate(mock_run): client.close(1) client.remove_label(1, "fork ci required") client.delete_run(1, 7) + client.cancel_run(1, 7) mock_run.assert_not_called() @@ -170,3 +171,19 @@ def test_set_status_context(mock_run, kwargs, context): GitHubClient().set_status(1, "abc", "success", "https://x", "desc", **kwargs) assert json.loads(mock_run.call_args.kwargs["input"])["context"] == context + + +@pytest.mark.parametrize(("force", "action"), [(False, "cancel"), (True, "force-cancel")]) +@mock.patch("airflow_breeze.utils.fork_ci_github.subprocess.run", autospec=True) +def test_cancel_run(mock_run, force, action): + mock_run.return_value = _completed() + + GitHubClient().cancel_run(1, 7, force=force) + + assert mock_run.call_args.args[0] == [ + "gh", + "api", + "-X", + "POST", + f"repos/apache/airflow/actions/runs/7/{action}", + ] diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index 9f3aa61f751d0..3df786d4a837a 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -2073,3 +2073,43 @@ def test_job_status_reporter_compares_contexts_case_insensitively(client): assert JobStatusReporter(client, repo=REPO).report(_pr(), COMPLETED_RUN) == "1:1:FAILURE" client.set_status.assert_not_called() + + +@pytest.mark.parametrize("enabled", [True, False]) +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.cancel_merged_pull_request_runs", autospec=True) +def test_run_reconcile_cancels_runs_of_merged_pull_requests(mock_cancel, client, tmp_path, enabled): + mock_cancel.return_value = [(10, "cancelling run 7 failed")] + client.graphql.side_effect = [_search_result(), _search_result(), _search_result()] + client.rest_get_pages.return_value = [{"total_count": 0, "workflow_runs": []}] + + has_alerts = run_reconcile( + client, + dataclasses.replace(CONFIG, cancel_merged_runs=enabled), + now=NOW, + run_url="https://run", + output_file=tmp_path / "s.json", + workflows_dir=tmp_path, + ) + + assert mock_cancel.called is enabled + assert has_alerts is enabled + + +@mock.patch("airflow_breeze.utils.fork_ci_reconcile.cancel_merged_pull_request_runs", autospec=True) +def test_run_reconcile_reports_a_failed_cancellation_pass(mock_cancel, client, tmp_path): + mock_cancel.side_effect = GitHubError("HTTP 502") + client.graphql.side_effect = [_search_result(), _search_result(), _search_result()] + client.rest_get_pages.return_value = [{"total_count": 0, "workflow_runs": []}] + + assert ( + run_reconcile( + client, + CONFIG, + now=NOW, + run_url="https://run", + output_file=tmp_path / "s.json", + workflows_dir=tmp_path, + ) + is True + ) + assert "cancelling runs of merged pull requests failed" in (tmp_path / "s.json").read_text() diff --git a/dev/breeze/tests/test_merged_pull_request_runs.py b/dev/breeze/tests/test_merged_pull_request_runs.py new file mode 100644 index 0000000000000..5c6223963c022 --- /dev/null +++ b/dev/breeze/tests/test_merged_pull_request_runs.py @@ -0,0 +1,266 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from unittest import mock + +import pytest + +from airflow_breeze.utils.fork_ci_github import GitHubClient, GitHubError +from airflow_breeze.utils.merged_pull_request_runs import ( + BRANCH_BATCH_SIZE, + PullRequestHead, + UnfinishedRun, + cancel_merged_pull_request_runs, + find_merged_pull_request, + find_pull_requests_by_branch, +) + +REPO = "apache/airflow" +NOW = datetime(2026, 10, 25, 19, 0, tzinfo=UTC) +MERGED_AT = NOW - timedelta(minutes=10) +RUN = UnfinishedRun( + id=7, + workflow="Tests (AMD)", + status="in_progress", + head_repo="someone/airflow", + head_branch="fix", + head_sha="aaa", + created_at=MERGED_AT - timedelta(minutes=20), +) +MERGED = PullRequestHead( + number=10, state="MERGED", head_repo="someone/airflow", head_sha="aaa", merged_at=MERGED_AT +) + + +@pytest.fixture +def client(): + github = mock.create_autospec(GitHubClient, instance=True) + github.repo = REPO + github.dry_run = False + return github + + +@pytest.mark.parametrize( + ("pull_requests", "expected"), + [ + pytest.param([MERGED], MERGED, id="merged-pull-request"), + pytest.param( + [PullRequestHead(10, "MERGED", "Someone/Airflow", "aaa", MERGED_AT)], + PullRequestHead(10, "MERGED", "Someone/Airflow", "aaa", MERGED_AT), + id="head-repository-in-another-case", + ), + pytest.param( + [PullRequestHead(11, "OPEN", "someone/airflow", "aaa", None), MERGED], + None, + id="open-pr-same-head", + ), + pytest.param( + [PullRequestHead(10, "MERGED", "other/airflow", "aaa", MERGED_AT)], None, id="other-fork" + ), + pytest.param( + [PullRequestHead(10, "MERGED", "someone/airflow", "bbb", MERGED_AT)], None, id="other-commit" + ), + pytest.param( + [PullRequestHead(10, "MERGED", "someone/airflow", "aaa", RUN.created_at - timedelta(minutes=1))], + None, + id="run-started-after-the-merge", + ), + pytest.param( + [PullRequestHead(10, "CLOSED", "someone/airflow", "aaa", None)], None, id="closed-not-merged" + ), + pytest.param([], None, id="no-pull-request"), + ], +) +def test_find_merged_pull_request(pull_requests, expected): + assert find_merged_pull_request(RUN, pull_requests) == expected + + +def _run(run_id, *, status="in_progress", branch="fix", sha="aaa", created_at=RUN.created_at): + return { + "id": run_id, + "name": "Tests (AMD)", + "status": status, + "head_repository": {"full_name": "someone/airflow"}, + "head_branch": branch, + "head_sha": sha, + "created_at": created_at.strftime("%Y-%m-%dT%H:%M:%SZ"), + } + + +def _pr_node(number, state, sha="aaa", merged_at=MERGED_AT): + return { + "number": number, + "state": state, + "mergedAt": merged_at.strftime("%Y-%m-%dT%H:%M:%SZ") if merged_at else None, + "headRefOid": sha, + "headRepository": {"nameWithOwner": "someone/airflow"}, + } + + +def _runs_by_status(runs): + def pages(path): + status = path.split("status=", 1)[1].split("&", 1)[0] + return [{"workflow_runs": [run for run in runs if run["status"] == status]}] + + return pages + + +@pytest.mark.parametrize( + ("merged_minutes_ago", "status", "force"), + [ + pytest.param(10, "in_progress", False, id="recently-merged"), + pytest.param(31, "in_progress", True, id="still-running-long-after-the-merge"), + pytest.param(31, "queued", False, id="queued-long-after-the-merge"), + ], +) +def test_cancel_merged_pull_request_runs(client, merged_minutes_ago, status, force): + merged_at = NOW - timedelta(minutes=merged_minutes_ago) + client.rest_get_pages.side_effect = _runs_by_status( + [_run(7, status=status, created_at=merged_at - timedelta(minutes=5))] + ) + client.graphql.return_value = { + "repository": {"b0": {"nodes": [_pr_node(10, "MERGED", merged_at=merged_at)]}} + } + + assert cancel_merged_pull_request_runs(client, repo=REPO, now=NOW) == [] + + client.cancel_run.assert_called_once_with(10, 7, force=force) + + +def test_cancel_merged_pull_request_runs_leaves_open_pull_requests_alone(client): + client.rest_get_pages.side_effect = _runs_by_status([_run(7)]) + client.graphql.return_value = {"repository": {"b0": {"nodes": [_pr_node(10, "OPEN", merged_at=None)]}}} + + cancel_merged_pull_request_runs(client, repo=REPO, now=NOW) + + client.cancel_run.assert_not_called() + + +def test_cancel_merged_pull_request_runs_without_unfinished_runs(client): + client.rest_get_pages.side_effect = _runs_by_status([]) + + assert cancel_merged_pull_request_runs(client, repo=REPO, now=NOW) == [] + client.graphql.assert_not_called() + + +@pytest.mark.parametrize( + ("error", "problems"), + [ + pytest.param( + GitHubError( + "gh api -X POST repos/apache/airflow/actions/runs/7/cancel failed: gh: done (HTTP 409)" + ), + [], + id="finished-meanwhile", + ), + pytest.param( + GitHubError("gh api repos/apache/airflow/actions/runs/1409/cancel failed: HTTP 500"), + [ + ( + 10, + "cancelling run 7 (Tests (AMD)) failed: gh api repos/apache/airflow/actions/runs/1409/cancel failed: HTTP 500", + ) + ], + id="failed-with-409-in-the-path", + ), + ], +) +def test_cancel_merged_pull_request_runs_reports_failed_cancellations(client, error, problems): + client.rest_get_pages.side_effect = _runs_by_status([_run(7), _run(8)]) + client.graphql.return_value = {"repository": {"b0": {"nodes": [_pr_node(10, "MERGED")]}}} + client.cancel_run.side_effect = [error, None] + + assert cancel_merged_pull_request_runs(client, repo=REPO, now=NOW) == problems + assert client.cancel_run.call_count == 2 + + +def test_find_pull_requests_by_branch_batches_branches(client): + branches = [f"branch-{index:02d}" for index in range(BRANCH_BATCH_SIZE + 1)] + + def graphql(query, variables): + batch = sorted(key for key in variables if key.startswith("b")) + return {"repository": {key: {"nodes": [_pr_node(1, "MERGED")]} for key in batch}} + + client.graphql.side_effect = graphql + + found = find_pull_requests_by_branch(client, REPO, [*branches, branches[0]]) + + assert sorted(found) == branches + assert client.graphql.call_count == 2 + assert client.graphql.call_args_list[0].args[1]["owner"] == "apache" + + +def test_find_pull_requests_by_branch_reads_every_page(client): + first = {"pageInfo": {"hasNextPage": True, "endCursor": "c1"}, "nodes": [_pr_node(1, "MERGED")]} + second = { + "pageInfo": {"hasNextPage": False, "endCursor": None}, + "nodes": [_pr_node(2, "OPEN", merged_at=None)], + } + client.graphql.side_effect = [ + {"repository": {"b0": first}}, + {"repository": {"pullRequests": second}}, + ] + + found = find_pull_requests_by_branch(client, REPO, ["main"]) + + assert [pr.number for pr in found["main"]] == [1, 2] + assert client.graphql.call_args.args[1] == { + "owner": "apache", + "name": "airflow", + "branch": "main", + "cursor": "c1", + } + + +def test_open_pull_request_on_a_later_page_keeps_the_run(client): + client.rest_get_pages.side_effect = _runs_by_status([_run(7)]) + client.graphql.side_effect = [ + { + "repository": { + "b0": { + "pageInfo": {"hasNextPage": True, "endCursor": "c1"}, + "nodes": [_pr_node(10, "MERGED")], + } + } + }, + { + "repository": { + "pullRequests": { + "pageInfo": {"hasNextPage": False}, + "nodes": [_pr_node(11, "OPEN", merged_at=None)], + } + } + }, + ] + + cancel_merged_pull_request_runs(client, repo=REPO, now=NOW) + + client.cancel_run.assert_not_called() + + +@pytest.mark.parametrize(("dry_run", "summary"), [(False, "Cancelled 1 runs"), (True, "Would cancel 1 runs")]) +@mock.patch("airflow_breeze.utils.merged_pull_request_runs.console_print", autospec=True) +def test_cancel_merged_pull_request_runs_summary_tells_a_dry_run(mock_print, client, dry_run, summary): + client.dry_run = dry_run + client.rest_get_pages.side_effect = _runs_by_status([_run(7)]) + client.graphql.return_value = {"repository": {"b0": {"nodes": [_pr_node(10, "MERGED")]}}} + + cancel_merged_pull_request_runs(client, repo=REPO, now=NOW) + + assert summary in mock_print.call_args.args[0] From 0da1a3d564b7df1ee1d048d21e4e3af63d959b17 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Sun, 25 Oct 2026 21:14:06 +0100 Subject: [PATCH 3/6] Auto-approve cheap CI runs of external contributors' PRs (#75455) * Auto-approve cheap CI runs of external contributors' PRs GitHub holds the workflow runs of fork PRs until a maintainer approves them. With self-hosted runners (AIP-118) the approval gate becomes part of the trust model, so it should cover all external contributors, but approving every push by hand does not scale and makes contributors wait. Runs whose change cannot touch the CI definition and whose CI is cheap are safe to start without a maintainer. Expensive or large changes, and anything that changes how CI or runners are chosen, still get a human look first, which also keeps the load on the shared public runners down. The CI-minutes estimate lets maintainers see what a run will cost before approving it. Generated-by: Claude Opus 5 * Add inline script metadata to the CI-minutes calibration script The script uses only the standard library, so with a PEP 723 block it runs with plain `uv run dev/calibrate_ci_minutes_estimates.py` instead of syncing the whole Breeze environment. Document that invocation in the script and in .github/AGENTS.md. * Keep one PR's API or git error from aborting the approval tick Each PR is now processed in its own try block. A requests.RequestException or subprocess.CalledProcessError for one PR no longer stops the remaining PRs or skips the job summary. The failed PR is recorded with failed=True and shown as "failed" in the summary, keeping its CI-time estimate when one was computed and giving a short, single-line form of the error as the reason. The workflow also tries to add the 'need workflow approval' label to it, so it is not lost once its runs leave the query window. Nothing is approved on failure. approve_workflow_runs exits 1 after writing the summary when any PR failed, so a persistent failure turns the scheduled run red instead of passing silently. * Approve the runs of fork CI pull requests unless they change CI A pull request whose CI runs in the contributor's fork (AIP-120) still gets pull_request runs in apache/airflow, and GitHub holds them until a maintainer approves them. Those runs only execute the fork CI gate, so judging them by the cost and size of project CI left cheap gate-only runs waiting for a maintainer for no benefit, and a gated run that never ran could not be cleaned up by the reconciler. What still needs a maintainer is a change to the files that define CI, since it could remove the gate or add a workflow that does run. Using the reconciler's own list keeps the two decisions consistent, and the author is told why their runs wait instead of only seeing a label. Generated-by: Claude Opus 5 * Keep CI-minutes estimates true to the runs they approve A run selects its tests from the labels the PR had when it started, so approving it on the current labels could start a full-tests run whose label was removed while it waited. Calibrating from self-hosted and all-versions runs, and leaving out the jobs that need no CI image, made the estimate describe runs other than those it is used to approve. Generated-by: Claude Opus 5 --------- Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> (cherry picked from commit 2a3ee7ffe25f2751e181638e2964dcdb06e0d6f0) --- .github/AGENTS.md | 38 + .../workflows/auto-approve-workflow-runs.yml | 68 ++ contributing-docs/05_pull_requests.rst | 41 + contributing-docs/25_maintainer_pr_triage.md | 1 + dev/breeze/doc/08_ci_tasks.rst | 20 + dev/breeze/doc/ci/05_workflows.md | 62 + .../08_fork_ci_for_external_contributors.md | 46 + dev/breeze/doc/images/output_ci.svg | 42 +- dev/breeze/doc/images/output_ci.txt | 2 +- .../output_ci_approve-workflow-runs.svg | 142 +++ .../output_ci_approve-workflow-runs.txt | 1 + ...utput_setup_check-all-params-in-groups.svg | 96 +- ...utput_setup_check-all-params-in-groups.txt | 2 +- ...output_setup_regenerate-command-images.svg | 104 +- ...output_setup_regenerate-command-images.txt | 2 +- .../airflow_breeze/commands/ci_commands.py | 54 + .../commands/ci_commands_config.py | 17 + .../utils/ci_minutes_estimate.py | 226 ++++ .../airflow_breeze/utils/fork_ci_messages.py | 27 + .../utils/workflow_run_approval.py | 793 +++++++++++++ .../test_calibrate_ci_minutes_estimates.py | 101 ++ dev/breeze/tests/test_ci_minutes_estimate.py | 229 ++++ dev/breeze/tests/test_fork_ci_messages.py | 18 + .../tests/test_workflow_run_approval.py | 1044 +++++++++++++++++ dev/calibrate_ci_minutes_estimates.py | 270 +++++ 25 files changed, 3332 insertions(+), 114 deletions(-) create mode 100644 .github/AGENTS.md create mode 100644 .github/workflows/auto-approve-workflow-runs.yml create mode 100644 dev/breeze/doc/images/output_ci_approve-workflow-runs.svg create mode 100644 dev/breeze/doc/images/output_ci_approve-workflow-runs.txt create mode 100644 dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py create mode 100644 dev/breeze/src/airflow_breeze/utils/workflow_run_approval.py create mode 100644 dev/breeze/tests/test_calibrate_ci_minutes_estimates.py create mode 100644 dev/breeze/tests/test_ci_minutes_estimate.py create mode 100644 dev/breeze/tests/test_workflow_run_approval.py create mode 100644 dev/calibrate_ci_minutes_estimates.py diff --git a/.github/AGENTS.md b/.github/AGENTS.md new file mode 100644 index 0000000000000..e7689490f687d --- /dev/null +++ b/.github/AGENTS.md @@ -0,0 +1,38 @@ + + + + +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [The `.github` folder — Agent Instructions](#the-github-folder--agent-instructions) + - [Keep the CI-minutes estimates in sync with the workflows](#keep-the-ci-minutes-estimates-in-sync-with-the-workflows) + + + +# The `.github` folder — Agent Instructions + +## Keep the CI-minutes estimates in sync with the workflows + +`breeze ci approve-workflow-runs` (run by `workflows/auto-approve-workflow-runs.yml`) decides whether +to approve an external contributor's CI runs from an estimate of the job-minutes they take. The +numbers live in `dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py` and are medians +measured on recent `Tests (AMD)` PR runs on public runners. + +After you change the CI workflows in a way that adds, removes, splits or resizes jobs (new test +groups, a changed matrix, new composite workflows, a move between runner types), refresh the +estimates once the change has run on enough PRs, and in any case about once a month: + +1. Run `uv run dev/calibrate_ci_minutes_estimates.py --since ` + from the repository root with an authenticated `gh` CLI. Pick a date that covers at least a + week of runs after the workflow change. +2. Copy the medians and fits it prints into the constants of `ci_minutes_estimate.py`, and update + the "Last calibrated" line next to them. +3. When you add a new job group to the workflows, add a matching group to both + `ci_minutes_estimate.py` (with the selective-check output that turns it on) and the + `JOB_GROUPS` patterns of the calibration script. Unmatched jobs fall into the base group. +4. Run `uv run --project dev/breeze pytest dev/breeze/tests/test_ci_minutes_estimate.py + dev/breeze/tests/test_workflow_run_approval.py -xvs`. + +The estimates cover public runners only. Self-hosted (RunsOn) runners need their own set once there +is enough data from them. diff --git a/.github/workflows/auto-approve-workflow-runs.yml b/.github/workflows/auto-approve-workflow-runs.yml new file mode 100644 index 0000000000000..b89f201bb2f67 --- /dev/null +++ b/.github/workflows/auto-approve-workflow-runs.yml @@ -0,0 +1,68 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +# Approves the waiting workflow runs of external contributors' PRs to main when the change is safe to +# run without a maintainer's look, and labels the others "need workflow approval". A PR whose CI runs +# in the contributor's fork (AIP-120) is approved unless it changes CI definition files, and then gets +# a comment saying why its runs wait. The rules are in +# dev/breeze/src/airflow_breeze/utils/workflow_run_approval.py and documented in +# contributing-docs/05_pull_requests.rst. +# +# This workflow runs only trusted code from main. It reads the PR's metadata, changed-file list and +# git objects, and never checks out or runs the PR's code. Approving with GITHUB_TOKEN starts the +# approved runs right away, and workflow_run workflows that follow them still trigger. +name: Auto approve workflow runs +on: # yamllint disable-line rule:truthy + schedule: + - cron: '*/30 * * * *' + workflow_dispatch: + inputs: + report-only: + description: "Only report the decisions and estimates, without approving or labelling" + type: boolean + default: false +permissions: + # Approving workflow runs. + actions: write + # Adding and removing the "need workflow approval" label, and commenting on a fork CI PR whose + # CI change keeps its runs waiting. + pull-requests: write + contents: read +concurrency: + group: auto-approve-workflow-runs + cancel-in-progress: false +jobs: + approve-workflow-runs: + name: "Approve workflow runs of external contributors" + runs-on: ubuntu-latest + if: github.repository == 'apache/airflow' + timeout-minutes: 20 + steps: + - name: "Checkout main" + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: main + - name: "Install Breeze" + uses: ./.github/actions/breeze + - name: "Approve workflow runs" + env: + GITHUB_TOKEN: ${{ github.token }} + GITHUB_REPOSITORY: ${{ github.repository }} + REPORT_ONLY: ${{ inputs.report-only && 'true' || 'false' }} + run: breeze ci approve-workflow-runs diff --git a/contributing-docs/05_pull_requests.rst b/contributing-docs/05_pull_requests.rst index c7fee6d267771..0b178c1332206 100644 --- a/contributing-docs/05_pull_requests.rst +++ b/contributing-docs/05_pull_requests.rst @@ -242,6 +242,47 @@ is the PMC private list — see the ``private@airflow.apache.org``. That document also covers the further steps the project may take when the same behaviour persists across PRs. +CI workflow runs of external contributors +----------------------------------------- + +GitHub does not start the CI workflows of a PR from a fork until a maintainer approves them, +for the contributors that the repository's approval setting covers. To keep you from waiting +for a maintainer on every push, a scheduled workflow checks the waiting runs every 30 minutes +and approves them automatically when all of these hold: + +* The PR targets ``main``. PRs to release branches, such as ``v3-4-test``, always wait for a + maintainer. +* The PR does not change anything under ``.github/``, ``dev/breeze/`` or ``scripts/ci/``. Those + files define the CI workflows and choose the runners they use. +* The change does not need the full set of tests, does not change dependencies (``pyproject.toml``, + lock files, ``package.json``, ``go.mod``, Gradle build files) and does not add a new provider. +* The CI jobs your change selects are estimated to take at most 310 job-minutes, about a quarter of + a full CI run. Changes to documentation, to the UI, to the Helm chart, to the language SDKs, to + one area of Airflow core or to a small provider usually stay under it. Changes to large providers + such as Amazon or Google, to several providers, or to the Task SDK (which runs the tests of all + providers) usually do not. +* The PR changes at most 100 lines of source code. Tests (including end-to-end tests), + documentation, newsfragments, translations and lock files are not counted, so a fix with + thorough tests still qualifies. Larger changes are usually new features, and a maintainer looks + at them first. + +If your PR's CI runs in your fork (it carries the ``fork ci required`` label), the CI run in +``apache/airflow`` only checks that, so none of the rules above apply. Its runs are approved +automatically unless the PR changes files that define CI: anything under ``.github/``, +``dev/breeze/``, ``scripts/ci/``, ``scripts/docker/`` or ``scripts/in_container/``, a ``Dockerfile`` +or ``.pre-commit-config.yaml``. Then the runs wait for a maintainer and a comment on the PR explains +why. CI in your fork runs either way. + +Each new push is checked again, because approval applies to one commit. Only runs started in the +last 24 hours are considered, so older waiting runs stay with maintainers. If your PR does not +qualify, it gets the ``need workflow approval`` label and a maintainer approves the runs after +looking at the change. That can take a while, so please do not ping maintainers about it; it is +not a judgement of your PR. The label is removed when a later commit is approved automatically. + +Maintainers can see each decision, the CI jobs selective checks expect to run and their estimated +time in the summary of the ``Auto approve workflow runs`` workflow. The rules are described in +more detail in `CI workflows <../dev/breeze/doc/ci/05_workflows.md>`__. + Gen-AI Assisted contributions ----------------------------- diff --git a/contributing-docs/25_maintainer_pr_triage.md b/contributing-docs/25_maintainer_pr_triage.md index e5f5313c5b770..e3f6909194c94 100644 --- a/contributing-docs/25_maintainer_pr_triage.md +++ b/contributing-docs/25_maintainer_pr_triage.md @@ -238,6 +238,7 @@ The triage workflow uses the following labels and states: | Label / State | Meaning | |---|---| | `ready for maintainer review` | PR has passed Stage 1 and is queued for human review. Applied only after the skill has verified that no workflow runs are awaiting approval. | +| `need workflow approval` | The PR's CI workflow runs wait for a maintainer, because the `Auto approve workflow runs` workflow did not approve them: a PR to a branch other than `main`, a change to CI definition files, a full-tests or dependency change, a change whose CI jobs are estimated at more than 310 job-minutes, or a change of more than 100 lines of source code (see [Pull Requests](05_pull_requests.rst)). For a PR whose CI runs in the contributor's fork, only a change to CI definition files keeps its runs waiting, and the workflow comments on the PR to say so. The workflow also tries to add the label when processing the PR failed; its run then fails too. Approve the runs after looking at the diff. The workflow removes the label when a later commit is approved automatically. | | `closed because of multiple quality violations` | PR was closed because the author has multiple open PRs with quality issues. | | `suspicious changes detected` | PR (and all open PRs by the same author) was closed because the diff contained suspicious patterns (secret exfiltration, malicious CI modifications, etc.). | | `closed because of open PR limit` | PR was closed in the one-time closure that introduced the [open PR limit](32_open_pull_request_limit.rst). Applied by `dev/close_prs_over_open_pr_limit.py`, never by the triage skill. If the author reopens the PR, triage it like any other. | diff --git a/dev/breeze/doc/08_ci_tasks.rst b/dev/breeze/doc/08_ci_tasks.rst index b1bf0d9f5fb99..591e796eff615 100644 --- a/dev/breeze/doc/08_ci_tasks.rst +++ b/dev/breeze/doc/08_ci_tasks.rst @@ -200,6 +200,26 @@ These are all available flags of ``fork-ci-reconcile`` command: :width: 100% :alt: Breeze ci fork-ci-reconcile +Approving workflow runs of external contributors +------------------------------------------------ + +The ``auto-approve-workflow-runs`` workflow runs ``approve-workflow-runs`` every 30 minutes. It approves +the waiting workflow runs of external contributors' PRs to ``main`` whose CI jobs are estimated at no more +than 310 job-minutes, that change at most 100 lines of source code and that do not touch CI definition files, and labels the other PRs ``need workflow approval``. PRs whose CI runs in +the contributor's fork (AIP-120) only run the fork CI gate in ``apache/airflow``, so their runs are +approved whatever their size unless they change CI definition files; such a PR also gets a comment +explaining that its CI change needs a maintainer. For every PR it also +estimates the CI jobs selective checks will run and how long they take, and writes them to the job +summary. Use ``--report-only`` to see the decisions without approving runs or changing labels. The rules +are described in `CI workflows `_. + +These are all available flags of ``approve-workflow-runs`` command: + +.. image:: ./images/output_ci_approve-workflow-runs.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_approve-workflow-runs.svg + :width: 100% + :alt: Breeze ci approve-workflow-runs + ----- Next step: Follow the `Release management tasks <09_release_management_tasks.rst>`_ guide to learn how diff --git a/dev/breeze/doc/ci/05_workflows.md b/dev/breeze/doc/ci/05_workflows.md index 7ec7871993cb8..c06d4f8a267b2 100644 --- a/dev/breeze/doc/ci/05_workflows.md +++ b/dev/breeze/doc/ci/05_workflows.md @@ -98,6 +98,68 @@ whose CI runs in the contributor's fork (AIP-120), the gate skips `build-info` a everything after it, so the run finishes in seconds; see [Fork CI for external contributors](08_fork_ci_for_external_contributors.md). +### Approval of pull request runs + +GitHub holds the runs of a fork PR in `action_required` until someone with write access approves +them, for the contributors the repository's approval setting covers. The +`auto-approve-workflow-runs.yml` workflow runs every 30 minutes on `main` and calls +`breeze ci approve-workflow-runs`, which approves the waiting runs of a commit when all of these hold: + +- the PR targets `main`; PRs to release branches (`v3-4-test`, `airflow-ctl/v0-1-test`, ...) run those + branches' own workflows and always wait for a maintainer +- the PR changes nothing under `.github/`, `dev/breeze/` or `scripts/ci/`, which define the workflows + and choose the runners +- the PR changes no dependency manifest or lock file (`pyproject.toml`, `uv.lock`, `package.json`, + `pnpm-lock.yaml`, `go.mod`, Gradle build files, ...) and adds no new provider (selective checks + run from `main`'s tree and do not know it) +- selective checks for the PR's merge commit succeed, and do not need the full set of tests, all Python + and database versions, a canary build or upgraded dependencies (the estimate does not model them) +- the jobs selective checks select are estimated at no more than 310 job-minutes on public runners + (about 23% of a full-tests run, just above the cluster of single core areas and the Helm chart) +- the PR changes at most 100 lines of source code; tests (including e2e tests), docs, newsfragments, + translations, lock files and images are not counted + +A PR to `main` whose CI runs in the contributor's fork (AIP-120: the `Fork CI gate` job would skip +project CI for it) is decided by one rule instead, because its `Tests (AMD)` run only executes the +gate: its runs are approved whatever its size, cost or dependency changes, unless it changes a file +the fork CI reconciler counts as CI-defining (`.github/`, `dev/breeze/`, `scripts/ci/`, +`scripts/docker/`, `scripts/in_container/`, `Dockerfile*`, `.pre-commit-config.yaml`). Such a PR gets +the `need workflow approval` label and one comment telling the author that the CI change needs a +maintainer; see +[Approving gated runs](08_fork_ci_for_external_contributors.md#approving-gated-runs). + +Only runs created in the last 24 hours (`--max-run-age-hours`) are considered, so the workflow never +approves a backlog of older waiting runs at once; those stay with maintainers. A commit that is the +head of several open PRs is never approved automatically, because its runs cannot be told apart by +PR, and only runs started after the PR was opened and after its base branch last changed are +approved (retargeting a PR does not start new runs). A run selects its jobs from the labels the PR +had when it started, while the decision uses the current labels, so for PRs that run project CI a +run started before a label that changes which tests run was added or removed +(`full tests needed`, `all versions`, `default versions only`, `latest versions only`, `canary`, +`upgrade to newer dependencies`, `area:e2e-tests`, `area:kubernetes-tests`) is left for a +maintainer, since it still runs the selection it started with; the next push is decided afresh. +For a PR whose CI runs in the fork the selection does not matter, but the gate falls back to the +labels a run started with when it cannot read the current ones, so a run started before +`use fork ci` or `use project ci` was added or removed is left for a maintainer instead. Other PRs get the +`need workflow approval` label and wait for a maintainer. When processing a PR fails (for example on +a GitHub API error), the workflow tries to add the label to it and the workflow run fails, so the +failure stays visible even if the label could not be added. The label is removed when a later commit +of the PR is approved automatically. + +The workflow uses only `main`'s code. It reads the PR's metadata and fetches its merge commit as git +objects to list the changed files; it never checks out or runs the PR's code. The merge commit must +have been built from the head commit whose runs get approved, so a push after the check cannot slip +into an approved run. Approving with the workflow's `GITHUB_TOKEN` starts the runs right away, and +`workflow_run` workflows that follow them still trigger. + +The estimate comes from `dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py`, which turns the +selective-check outputs into job groups with job-minutes and a wall-clock estimate, weighting +providers by the size of their test suites. The job summary lists every waiting PR with its decision, +the expected job groups and the estimated time. The numbers are medians of recent `Tests (AMD)` PR +runs on public runners; refresh them with `dev/calibrate_ci_minutes_estimates.py`, as +`.github/AGENTS.md` describes. Run `breeze ci approve-workflow-runs --report-only` (or dispatch the +workflow with `report-only`) to see the decisions without approving or labelling anything. + ## Fork push run These runs are triggered by a push to any branch of a contributor's fork in which the diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index 1f3983739442d..5f58617cf5656 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -337,6 +337,49 @@ where `*` is any characters except `/`). The `check-ci-workflows-in-sync` prek hook allowlist is extended to permit the `gate` job and the `build-info` dependency to exist only in the AMD workflow. +### Approving gated runs + +GitHub holds the `pull_request` runs of a fork pull request in `action_required` until someone with +write access approves them, for the contributors the repository's approval setting covers. Until +then the gate job does not run either. A gated pull request would stay in the maintainers' approval +queue even though its project CI run only executes the gate, and the reconciler could not delete a +gated run that never ran. + +`breeze ci approve-workflow-runs`, run every 30 minutes by `auto-approve-workflow-runs.yml` (see +[CI workflows](05_workflows.md#approval-of-pull-request-runs)), handles them. For a pull request to +`main` it calls `fork_ci_gate.decide` with the configuration of its own `main` checkout and the +pull request's author, association, creation time, head repository and current labels, the inputs +the gate job of the waiting run uses. When the reason is gated (`external`, `pilot` or +`label_fork_ci`), the rules for pull requests that run project CI (estimated job-minutes, lines of +source code, dependency files, full tests) do not apply, because none of those jobs run. One rule +applies instead: + +- The waiting runs of the head commit are approved when the pull request's merge commit changes no + file that the reconciler counts as CI-defining: `.github/`, `dev/breeze/`, `scripts/ci/`, + `scripts/docker/`, `scripts/in_container/`, `Dockerfile*` and `.pre-commit-config.yaml` (see + "Trusting a green fork run"). The approved `ci-amd.yml` run executes only the gate, and the other + `pull_request` workflows (CodeQL for the changed languages, the newsfragment check, dependency + review) run on GitHub-hosted runners. +- Otherwise the runs wait for a maintainer, the pull request is labelled `need workflow approval`, + and the workflow posts one comment explaining that the change to the listed CI files needs a + maintainer to review it and approve the runs. The comment starts with the marker + `` and is posted once per pull request; only comments by + `github-actions[bot]` with the marker count. A change to `.github/` can remove the gate or add a + workflow that does run, so such a run must not start without a review. + +A run started before `use fork ci` or `use project ci` was last added or removed is not approved +automatically: the gate falls back to the labels of the run's event when it cannot read the current +ones, and those can still send the run to project CI. The next push is decided afresh. + +Both rules use the same list as "Trusting a green fork run", so a gated pull request whose runs are +approved automatically is also one whose green fork run is trusted, and one that changes CI is both +left waiting for approval and labelled `needs project ci` once its fork run is green. The gate +tampering check still applies to every run a maintainer approves by hand. + +The approval reads only API metadata and the git objects of the merge commit; it never checks out or +runs the pull request's code. Pull requests to other branches keep the rules for project CI, which +leave them to a maintainer. + ## Fork mode CI ### Trigger @@ -1022,6 +1065,9 @@ as a failed check rather than a crash. see "Trusting a green fork run". - Job statuses carry names from the contributor's workflow under their own `Fork CI job / ` prefix, so they cannot post the reconciler's own contexts or `CI passed`; see "Job statuses". +- Workflow runs of a gated pull request are approved without a maintainer only when it changes no + CI-defining file, so a pull request that edits the gate away still waits for a maintainer before + any run starts; see "Approving gated runs". - The reconciler runs on `main` with `pull-requests: write` and never checks out or executes pull request content. Its inputs are API data; its comment templates are static. - Fork runs get the fork's `GITHUB_TOKEN` only. Nothing in project CI trusts a fork run's diff --git a/dev/breeze/doc/images/output_ci.svg b/dev/breeze/doc/images/output_ci.svg index 7158eac2e10d3..d175e86af7b1f 100644 --- a/dev/breeze/doc/images/output_ci.svg +++ b/dev/breeze/doc/images/output_ci.svg @@ -1,4 +1,4 @@ - +