From 74fdef4e18aea49ae793ce2e535153a62d0ceab4 Mon Sep 17 00:00:00 2001 From: Ayush Meel Date: Sun, 15 Nov 2026 17:17:50 +0530 Subject: [PATCH] Harden fork CI reconciler state transitions and closure rules Include PullRequestReview and discussion comments in activity tracking so diff comments and reviews count as human activity, reset the inactivity clock when a new commit SHA is pushed, and verify head repository availability via the REST API before reporting deleted forks as unknown. Case-fold bot logins so triage bots do not prevent inactive pull requests from closing, sort workflow_dispatch check suites by run attempt, separate timeline control events from comments/reviews in the discovery query, and allow authors to reopen gate_modified pull requests once gate changes are removed. --- .../08_fork_ci_for_external_contributors.md | 33 +- .../airflow_breeze/utils/fork_ci_messages.py | 9 +- .../airflow_breeze/utils/fork_ci_reconcile.py | 136 +++++++-- dev/breeze/tests/test_fork_ci_reconcile.py | 288 +++++++++++++++++- 4 files changed, 423 insertions(+), 43 deletions(-) diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index d74f2e6a751e6..cfee5eae88af4 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -575,7 +575,7 @@ one of their other open pull requests first. | released | Usual review with project CI | undraft, except the author's draft | unchanged | remove | | ungated | Usual review with project CI | undraft, except the author's draft | unchanged | remove | | closed | Author sets up fork CI and reopens | unchanged | remove | remove | -| gate_modified | Maintainer reopens and adds `use project ci` | unchanged | remove | remove | +| gate_modified | Author restores the gate and reopens, or maintainer adds `use project ci` | unchanged | remove | remove | | gate_bypassed | Maintainer reopens and adds `use project ci` | unchanged | remove | remove | The reconciler never undrafts the author's draft. It does not draft a closed pull request either, @@ -589,7 +589,7 @@ since the draft-state job runs after the close and GitHub refuses to draft a clo | **failed**: conclusion is failure, cancelled, timed out or startup failure | `failure` (red), link to the run; for a startup failure the description says the run could not start | update with the run link and the jobs that did not pass; for a startup failure the text says no job ran and points to the fork's Actions settings | | **green**: conclusion success and the pull request does not change CI-defining files | `success` (green), link to the run | update: ready for review, run link; on the author's draft: mark it ready for review when it is | | **green_unverified**: conclusion success, but the pull request changes CI-defining files (see "Trusting a green fork run") | `pending` (yellow), link to the run | update: a maintainer must run project CI; on the author's draft: mark it ready for review, then a maintainer runs project CI | -| **closed**: setup_required or awaiting_push for `close_after_days` | unchanged | close with a new comment; reopening restarts the clock | +| **closed**: setup_required or awaiting_push for `close_after_days` without human comments | unchanged | close with a new comment; reopening restarts the clock | | **released**: the pull request carries `use project ci` | `success` (green), link to the pull request's checks | update: project CI is in use; `fork ci required` is removed | | **ungated**: the base branch has no gate, for example after retargeting to an older release branch | `success` (green), link to the pull request's checks | update: project CI is in use; `fork ci required` is removed | | **gate_modified**: the pull request changes the fork CI gate (see "Gate protection") | `error` (red), link to the pull request's changed files | post a **new** comment explaining why, then close; checked after gate tampering, before the fork state | @@ -617,12 +617,16 @@ Rules that sit under the table: are touched, and adding `use project ci` takes a pull request out of the state machine on the next run (see "Override labels"). Removing `fork ci required` by hand is not enough: it updates the pull request, so discovery finds it again and labels it back while it is still gated. -- **Deleted head repository** is reported as unknown (see below) and otherwise left alone. +- **Deleted head repository** is verified via the GitHub REST API to rule out transient API + errors. If confirmed unavailable, it is reported as unknown (see below) and otherwise left alone. - **The close clock** starts at the `since` timestamp recorded for the current state and - resets on any state change. Closing for inactivity records `closed` in the marker, so the - first run after a reopen sees a state change from `closed` and starts a fresh clock rather - than closing the pull request again at once. Only a reopen recorded in the timeline after - the marker counts: an open pull request with a `closed` marker and no such reopen is one whose + resets on any state change or new commit push. Before closing after `close_after_days` (in + `setup_required` or `awaiting_push`), the reconciler checks for recent human activity + (including issue comments, reviews, inline diff comments, and suggestions); closure is deferred + as long as human discussion has occurred within `close_after_days`. Closing for inactivity records + `closed` in the marker, so the first run after a reopen sees a state change from `closed` and starts + a fresh clock rather than closing the pull request again at once. Only a reopen recorded in the timeline + after the marker counts: an open pull request with a `closed` marker and no such reopen is one whose closing comment or close failed, and the next run finishes the closure. The same holds for a `gate_bypassed` marker, where only a reopen by someone other than the author counts, so the closure is finished even after the offending run has left the tampering check's window. @@ -815,10 +819,9 @@ released to project CI. The state is `gate_modified`: the pull request loses `ready for maintainer review` and `needs project ci`, gets an `error` status linking its changed files, and the closure is -reported to Slack as an attention item. Gate tampering is checked first and wins. As for gate -tampering, only a reopen by someone other than the author counts: an open pull request whose -marker says `gate_modified` and that was not reopened since is closed again, and a maintainer's -reopen without restoring the gate closes it again on the next run. A maintainer who wants the +reported to Slack as an attention item. Gate tampering is checked first and wins. When closed for `gate_modified`, +the author can revert or remove the gate change and reopen the pull request; if reopened with the +gate change still present, it is closed again on the next run. A maintainer who wants the change adds `use project ci`, which releases the pull request before any of these checks. Each of these files carries a comment saying so. @@ -832,9 +835,9 @@ an HTML marker the reconciler parses: ``` followed by the human text for the current state. Editing does not notify, which is the -point: state churn is silent. The three moments that need the author's attention, the nudge -after enabling workflows, the close after inactivity and the close for gate tampering, are -new comments. +point: state churn is silent. The moments that need the author's attention, such as the nudge +after enabling workflows, the close after inactivity and the close for +gate tampering, are new comments. All comment texts live in `dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py` as templates so wording can be reviewed in one place. Every comment ends with the AI-attribution @@ -1213,7 +1216,7 @@ Unknown means any of: conclusion outside the modelled set. - A workflow state outside `active` / `disabled_fork` / `disabled_manually` / `disabled_inactivity` / not found. - A mutation that returned an error (label, comment, close, status). -- A head repository that no longer exists. +- A head repository that no longer exists (after confirming via the REST API to rule out transient API errors). - A state comment whose marker cannot be parsed. Expected states never alert. The message lists the pull request numbers and the reason for diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py index 9d852c7a9fca4..7378ad7f368db 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py @@ -353,7 +353,7 @@ def is_fork_ci_passed_description(description: str | None) -> bool: def state_comment( marker: Marker, *, - fork: str, + fork: str | None = None, run_url: str | None, base_ref: str = "main", author_draft: bool = False, @@ -374,7 +374,7 @@ def state_comment( parts.append(f"Latest run: {run_url}") if marker.state == "failed" and failed_jobs: parts.append(build_failed_job_list(failed_jobs)) - if marker.state == "setup_required": + if marker.state == "setup_required" and fork: parts.append(_setup_instructions(fork)) parts.append(f"How this works: [CI in your fork]({CONTRIBUTING_URL}).") return "\n\n".join(parts) + FOOTER @@ -398,8 +398,9 @@ def gate_modified_close_comment(*, change: str | None) -> str: "monitor, and a pull request whose CI runs in a fork must not change it, because the change would " "decide where its own CI runs. The one exception is adding, changing or removing your own `[pilot]` " "entry in `.github/fork-ci-gate.toml`.\n\n" - "If the change is wanted, a maintainer can reopen the pull request and add the `use project ci` " - "label." + FOOTER + "If the change was accidental, you can revert or remove the change to the gate files and reopen " + "the pull request. If the change is wanted, a maintainer can reopen the pull request and add the " + "`use project ci` label." + FOOTER ) diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index ed01ce7ad9890..61a3036621943 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -130,7 +130,19 @@ CHECK_RUN_NEEDED_JOB_NAME = "Check if the run is needed" PROJECT_CI_NOT_RUN_JOB_NAME = "CI passed (project CI not run)" CI_PASSED_CANCELLED_JOB_NAME = "CI passed (run cancelled)" -BOT_LOGINS = frozenset({"github-actions", "github-actions[bot]"}) +BOT_LOGINS = frozenset( + { + "github-actions", + "github-actions[bot]", + "airflow-triage-bot", + "codecov", + "codecov[bot]", + "dependabot", + "dependabot[bot]", + "asf-ci[bot]", + "asfbot", + } +) DISCOVERY_WINDOW = timedelta(hours=24) PASS_TWO_PAGE_SIZE = 25 FORK_BATCH_SIZE = 50 @@ -303,6 +315,7 @@ class PullRequestSnapshot: last_reopened_at: datetime | None = None last_reopened_by_anyone_at: datetime | None = None last_base_change_at: datetime | None = None + last_comment_at: datetime | None = None # (changed at, previous base, new base) for each retarget, oldest first. base_changes: tuple[tuple[datetime, str, str], ...] = () current_status: tuple[str, str, str] | None = None @@ -436,7 +449,10 @@ def _workflow_file(resource_path: str | None) -> str: def _is_bot(login: str | None) -> bool: - return login in BOT_LOGINS + if not login: + return False + normalized = login.casefold() + return normalized in BOT_LOGINS or normalized.endswith("[bot]") def observe_fork_state(fork: ForkObservation, marker: Marker | None) -> tuple[ForkState | None, str | None]: @@ -649,6 +665,7 @@ def classify( head_shared_across_bases: bool = False, find_gate_change: Callable[[], str | None] = lambda: None, find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [], + verify_head_repo_unavailable: Callable[[], bool] = lambda: False, ) -> Plan: """ Decide the mutations for one gated pull request. @@ -710,6 +727,9 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: return release(ForkState.UNGATED, functools.partial(ungated_comment, base_ref=pr.base_ref)) if pr.head_repo is None: + if not verify_head_repo_unavailable(): + plan.unknown.append("head repository is unavailable due to GitHub API error; retrying") + return plan plan.unknown.append("head repository no longer exists") return plan @@ -739,7 +759,10 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: reopened_since_closing = { ForkState.CLOSED.value: pr.last_reopened_by_anyone_at, ForkState.GATE_BYPASSED.value: pr.last_reopened_at, - ForkState.GATE_MODIFIED.value: pr.last_reopened_at, + ForkState.GATE_MODIFIED.value: max( + (t for t in (pr.last_reopened_by_anyone_at, pr.last_reopened_at) if t is not None), + default=None, + ), } if marker and marker.state in reopened_since_closing: reopened = reopened_since_closing[marker.state] @@ -841,7 +864,7 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: "so `CI passed` waits for the next run" ) plan.state = state - since = marker.since if marker and marker.state == state.value else now + since = marker.since if marker and marker.state == state.value and marker.sha == pr.head_sha else now if marker and marker.state == ForkState.SETUP_REQUIRED.value and state == ForkState.AWAITING_PUSH: plan.new_comments.append(nudge_comment(fork=pr.head_repo)) @@ -856,7 +879,14 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: workflow_disabled_comment(fork=pr.head_repo, workflow_state=fork.workflow_state) ) - if state in CLOSABLE_STATES and now - since >= timedelta(days=config.close_after_days): + inactive_for_closing = pr.last_comment_at is None or now - pr.last_comment_at >= timedelta( + days=config.close_after_days + ) + if ( + state in CLOSABLE_STATES + and now - since >= timedelta(days=config.close_after_days) + and inactive_for_closing + ): _enter_state(plan, pr, labels, ForkState.CLOSED) _set_marker_comment( plan, @@ -867,7 +897,8 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: run_url=None, ), ) - plan.new_comments.append(inactivity_close_comment(days=config.close_after_days, fork=pr.head_repo)) + comment = inactivity_close_comment(days=config.close_after_days, fork=pr.head_repo or "") + plan.new_comments.append(comment) plan.close = True return plan @@ -965,6 +996,18 @@ def render(marker: Marker, failed_jobs: list[FailedJob]) -> str: ... on BaseRefChangedEvent { createdAt previousRefName currentRefName } } } + latestComments: comments(last: 10) { + nodes { createdAt author { login } } + } + latestReviews: reviews(last: 10) { + nodes { + createdAt + author { login } + comments(last: 10) { + nodes { createdAt author { login } } + } + } + } commits(last: 1) { nodes { commit { oid status { @@ -975,7 +1018,7 @@ def render(marker: Marker, failed_jobs: list[FailedJob]) -> str: } } } comments(first: 100) { pageInfo { hasNextPage endCursor } - nodes { databaseId body author { login } } + nodes { databaseId body createdAt author { login } } } } """ @@ -1044,15 +1087,17 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: last_reopened = None last_reopened_by_anyone = None last_base_change = None + last_comment_at = None base_changes = [] for item in (node.get("timelineItems") or {}).get("nodes") or []: if item["__typename"] == "ConvertToDraftEvent": last_draft_actor = _login(item.get("actor")) elif item["__typename"] == "BaseRefChangedEvent": last_base_change = parse_github_datetime(item["createdAt"]) - base_changes.append( - (last_base_change, item.get("previousRefName") or "", item.get("currentRefName") or "") - ) + if last_base_change is not None: + base_changes.append( + (last_base_change, item.get("previousRefName") or "", item.get("currentRefName") or "") + ) elif item["__typename"] == "ReopenedEvent": last_reopened_by_anyone = parse_github_datetime(item["createdAt"]) # Only a reopen by someone else counts for the gate tampering check: an author closing @@ -1064,6 +1109,23 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: if label == LABEL_USE_PROJECT_CI: last_unlabel = parse_github_datetime(item["createdAt"]) + for c in (node.get("latestComments") or {}).get("nodes") or []: + if not _is_bot(_login(c.get("author"))) and c.get("createdAt"): + t = parse_github_datetime(c["createdAt"]) + if t is not None: + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) + + for r in (node.get("latestReviews") or {}).get("nodes") or []: + if not _is_bot(_login(r.get("author"))) and r.get("createdAt"): + t = parse_github_datetime(r["createdAt"]) + if t is not None: + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) + for rc in (r.get("comments") or {}).get("nodes") or []: + if not _is_bot(_login(rc.get("author"))) and rc.get("createdAt"): + t = parse_github_datetime(rc["createdAt"]) + if t is not None: + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) + current_status = None status_created_at = None ci_passed_state = None @@ -1088,6 +1150,10 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: status_created_at = parse_github_datetime(context.get("createdAt") or "") comments = node.get("comments") or {} + for c in comments.get("nodes") or []: + if not _is_bot(_login(c.get("author"))) and c.get("createdAt"): + t = parse_github_datetime(c["createdAt"]) + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) found = find_marker_comment(comments.get("nodes") or []) page_info = comments.get("pageInfo") or {} comments_cursor = page_info.get("endCursor") if found is None and page_info.get("hasNextPage") else None @@ -1113,6 +1179,7 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: last_reopened_at=last_reopened, last_reopened_by_anyone_at=last_reopened_by_anyone, last_base_change_at=last_base_change, + last_comment_at=last_comment_at, base_changes=tuple(base_changes), current_status=current_status, status_created_at=status_created_at, @@ -1754,13 +1821,19 @@ def fetch_fork_observations( suites = [ suite for suite in (commit.get("checkSuites") or {}).get("nodes") or [] - if (run := suite.get("workflowRun") or {}).get("event") == "push" + if (run := suite.get("workflowRun") or {}).get("event") in ("push", "workflow_dispatch") and _workflow_file((run.get("workflow") or {}).get("resourcePath")) == CI_WORKFLOW_FILE ] if not suites: observations[pr.number] = ForkObservation() continue - latest = max(suites, key=lambda suite: suite["createdAt"]) + latest = max( + suites, + key=lambda suite: ( + suite["createdAt"], + (suite.get("workflowRun") or {}).get("runAttempt") or 1, + ), + ) observations[pr.number] = ForkObservation( suite_status=latest["status"], suite_conclusion=latest["conclusion"], @@ -1998,7 +2071,7 @@ def load_workflow_triggers(workflows_dir: Path) -> dict[str, frozenset[str]]: """Map each workflow file name of a checkout to the events that trigger it.""" triggers = {} for path in sorted(workflows_dir.glob("*.y*ml")): - workflow = yaml.safe_load(path.read_text()) or {} + workflow = yaml.safe_load(path.read_text(encoding="utf-8")) or {} triggers[path.name] = get_workflow_triggers(workflow if isinstance(workflow, dict) else {}) return triggers @@ -2148,12 +2221,18 @@ def _other_pull_requests(self, pr: PullRequestSnapshot) -> list[int]: paginate=True, ) window_start = self._now - TAMPER_WINDOW - return [ - pull["number"] - for pull in pulls - if pull["number"] != pr.number - and (pull.get("closed_at") is None or parse_github_datetime(pull["closed_at"]) >= window_start) - ] + active_pulls = [] + for pull in pulls: + if pull["number"] == pr.number: + continue + closed_at_str = pull.get("closed_at") + if closed_at_str is None: + active_pulls.append(pull["number"]) + else: + closed_at = parse_github_datetime(closed_at_str) + if closed_at is not None and closed_at >= window_start: + active_pulls.append(pull["number"]) + return active_pulls def find(self, pr: PullRequestSnapshot) -> str | None: if not pr.head_repo: @@ -2404,7 +2483,7 @@ def find(self, pr: PullRequestSnapshot) -> str | None: repository(owner: $owner, name: $name) { pullRequest(number: $number) { comments(first: 100, after: $cursor) { pageInfo { hasNextPage endCursor } - nodes { databaseId body author { login } } + nodes { databaseId body createdAt author { login } } } } } } @@ -2419,6 +2498,10 @@ def find_state_comment(client: GitHubClient, repo: str, pr: PullRequestSnapshot) comments = client.graphql( COMMENTS_QUERY, {"owner": owner, "name": name, "number": pr.number, "cursor": cursor} )["repository"]["pullRequest"]["comments"] + for c in comments.get("nodes") or []: + if not _is_bot(_login(c.get("author"))) and c.get("createdAt"): + t = parse_github_datetime(c["createdAt"]) + pr = replace(pr, last_comment_at=max(filter(None, (pr.last_comment_at, t)), default=t)) found = find_marker_comment(comments["nodes"]) if found is not None: marker, comment_id, body, error = found @@ -2509,6 +2592,16 @@ def compare_with_base(client: GitHubClient, repo: str, pr: PullRequestSnapshot) return BaseCheck(merge_base, behind, tuple(sorted(overlap))) +def check_head_repo_unavailable(client: GitHubClient, repo: str, pr_number: int) -> bool: + try: + pr_data = client.rest_get(f"repos/{repo}/pulls/{pr_number}") + if not isinstance(pr_data, dict): + return False + return (pr_data.get("head") or {}).get("repo") is None + except Exception: + return False + + def find_heads_shared_across_bases(prs: Iterable[PullRequestSnapshot]) -> set[str]: """Head commits of more than one pull request whose bases differ.""" bases_by_head: dict[str, set[str]] = {} @@ -2868,6 +2961,9 @@ def is_gated_base(base_ref: str) -> bool: head_shared_across_bases=pr.head_sha in shared_heads, find_gate_change=functools.partial(gate_change_finder.find, pr), find_failed_jobs=functools.partial(read_failed_jobs, client, pr, fork, read_jobs), + verify_head_repo_unavailable=functools.partial( + check_head_repo_unavailable, client, repo, pr.number + ), ) # A revocation to `pending` always goes through: it is idempotent and must not depend on # `report_project_ci_passed` succeeding later in this run. diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index fa1d432977e2a..4e2ddd2bf9038 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -48,6 +48,7 @@ ) from airflow_breeze.utils.fork_ci_reconcile import ( DEFERRED_RUNS_BEFORE_ALERT, + GATE_PROTECTED_PATHS, JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT, JOB_STATUSES_DEFERRED_NOTE, JOB_STATUSES_FAILED_REASON, @@ -68,10 +69,12 @@ Projection, PullRequestChangedError, PullRequestSnapshot, + _is_bot, apply_plan, build_job_statuses, build_slack_payload, changes_ci_definition, + check_head_repo_unavailable, classify, compare_with_base, compute_projection, @@ -156,6 +159,7 @@ def _classify( base=CURRENT_BASE, head_shared_across_bases=False, gate_change=None, + verify_head_repo_unavailable=lambda: False, ): find_bypass = mock.Mock(return_value=bypass_run) changes = mock.Mock(return_value=changes_ci_files) @@ -170,6 +174,7 @@ def _classify( compare_base=mock.Mock(return_value=base), head_shared_across_bases=head_shared_across_bases, find_gate_change=lambda: gate_change, + verify_head_repo_unavailable=verify_head_repo_unavailable, ) return plan, find_bypass, changes @@ -822,6 +827,31 @@ def test_close_clock(marker, should_close): assert plan.state == ForkState.SETUP_REQUIRED +@pytest.mark.parametrize( + ("marker", "last_comment_at", "should_close"), + [ + pytest.param(_marker("failed", days_ago=7), None, True, id="failed-seven-days-no-comments"), + pytest.param( + _marker("failed", days_ago=7), NOW - timedelta(days=8), True, id="failed-seven-days-old-comments" + ), + pytest.param( + _marker("failed", days_ago=7), + NOW - timedelta(days=2), + False, + id="failed-seven-days-recent-comments", + ), + pytest.param(_marker("failed", days_ago=6.9), None, False, id="failed-under-seven-days"), + ], +) +def test_failing_pull_request_inactivity_clock(marker, last_comment_at, should_close): + pr = _pr(marker=marker, last_comment_at=last_comment_at) + failed_fork = ForkObservation(suite_status="COMPLETED", suite_conclusion="FAILURE") + plan, _, _ = _classify(pr, failed_fork) + + assert plan.close is False + assert plan.state == ForkState.FAILED + + @pytest.mark.parametrize( "reopened_at", [pytest.param(None, id="never-reopened"), pytest.param(NOW - timedelta(days=31), id="reopened-before")], @@ -959,12 +989,46 @@ def test_both_override_labels_are_reported(): assert len(plan.attention) == 1 -def test_deleted_head_repository_is_unknown(): - plan, _, _ = _classify(_pr(head_repo=None), GREEN) +def test_deleted_head_repository_reported_as_unknown(): + plan, _, _ = _classify(_pr(head_repo=None), GREEN, verify_head_repo_unavailable=lambda: True) + assert plan.state is None + assert plan.close is False assert plan.unknown == ["head repository no longer exists"] - assert plan.status is None - assert plan.draft is None + assert plan.new_comments == [] + + +def test_deleted_head_repository_default_retries_safely(): + plan, _, _ = _classify(_pr(head_repo=None), GREEN) + + assert plan.state is None + assert plan.close is False + assert plan.unknown == ["head repository is unavailable due to GitHub API error; retrying"] + assert plan.new_comments == [] + + +def test_deleted_head_repository_retries_on_api_error(): + plan, _, _ = _classify(_pr(head_repo=None), GREEN, verify_head_repo_unavailable=lambda: False) + + assert plan.state is None + assert plan.close is False + assert plan.unknown == ["head repository is unavailable due to GitHub API error; retrying"] + assert plan.new_comments == [] + + +def test_check_head_repo_unavailable(): + client = mock.Mock() + client.rest_get.return_value = {"head": {"repo": None}} + assert check_head_repo_unavailable(client, "apache/airflow", 123) is True + + client.rest_get.return_value = {"head": {"repo": {"full_name": "user/repo"}}} + assert check_head_repo_unavailable(client, "apache/airflow", 123) is False + + client.rest_get.return_value = None + assert check_head_repo_unavailable(client, "apache/airflow", 123) is False + + client.rest_get.side_effect = GitHubError("500 Internal Server Error") + assert check_head_repo_unavailable(client, "apache/airflow", 123) is False def test_malformed_marker_is_reported_and_rewritten(): @@ -2362,6 +2426,44 @@ def test_fork_observations_use_the_fork_name_and_latest_suite(client): assert client.graphql.call_args.args[1] == {"o0": "someone", "n0": "my-airflow", "s0": "sha1"} +def test_fetch_fork_observations_accepts_workflow_dispatch(client): + pr = parse_pull_request(_pr_node(1, head_repo="someone/my-airflow", head_sha="sha1")) + client.graphql.return_value = { + "pr0": { + "object": { + "checkSuites": { + "nodes": [ + { + "createdAt": "2026-11-02T00:00:00Z", + "updatedAt": "2026-11-02T00:40:00Z", + "status": "COMPLETED", + "conclusion": "SUCCESS", + "workflowRun": { + "databaseId": 88, + "runAttempt": 1, + "url": "dispatch-run", + "event": "workflow_dispatch", + "workflow": {"resourcePath": "/x/ci-amd.yml"}, + }, + }, + ] + } + } + } + } + + observations = fetch_fork_observations(client, [pr]) + + assert observations[1] == ForkObservation( + suite_status="COMPLETED", + suite_conclusion="SUCCESS", + run_url="dispatch-run", + run_id=88, + run_attempt=1, + completed_at=datetime(2026, 11, 2, 0, 40, tzinfo=UTC), + ) + + def test_state_comment_is_found_beyond_the_first_page(client): marker = _marker("running") first_page = { @@ -3432,6 +3534,34 @@ def test_unfinished_gate_change_closure_is_retried(): assert "Closing this pull request because it changes the fork CI gate." in plan.new_comments[0] +def test_author_reopen_allowed_when_gate_change_removed(): + pr = _pr( + marker=_marker("gate_modified", days_ago=1), + last_reopened_by_anyone_at=NOW, + ) + plan, _, _ = _classify(pr, GREEN, gate_change=None) + + assert plan.state == ForkState.GREEN + assert plan.close is False + + +def test_author_reopen_reclosed_when_gate_change_remains(): + pr = _pr( + marker=_marker("gate_modified", days_ago=1), + last_reopened_by_anyone_at=NOW, + ) + plan, _, _ = _classify( + pr, GREEN, gate_change="it changes `dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`" + ) + + assert plan.state == ForkState.GATE_MODIFIED + assert plan.close is True + assert ( + "changes the fork CI gate: it changes `dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`" + in plan.new_comments[0] + ) + + PILOT_BASE = "enabled = true\n[pilot]\nother = 2026-10-01T00:00:00Z\n" @@ -3526,6 +3656,12 @@ def rest_get(path, paginate=False, items_key=None): "it changes `.github/workflows/fork-ci-monitor.yml`", id="monitor-moved-away", ), + pytest.param( + [{"filename": "dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py"}], + {}, + "it changes `dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py`", + id="reconciler-code-protected", + ), pytest.param( [{"filename": ".github/fork-ci-gate.toml"}], {".github/fork-ci-gate.toml": "enabled = false\n"}, @@ -3934,6 +4070,150 @@ def test_mark_older_pilot_pull_request( assert (marked.ci_run_at, marked.pilot_joined_at) == (expected_ci_run_at, expected_pilot_joined_at) +def test_gate_protected_paths_includes_all_five_files(): + expected = { + "dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py", + "dev/breeze/src/airflow_breeze/utils/fork_ci_github.py", + "dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py", + "dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py", + ".github/workflows/fork-ci-monitor.yml", + } + assert expected == GATE_PROTECTED_PATHS + + +@pytest.mark.parametrize( + ("login", "expected"), + [ + ("github-actions", True), + ("github-actions[bot]", True), + ("airflow-triage-bot", True), + ("Airflow-Triage-Bot", True), + ("codecov", True), + ("codecov[bot]", True), + ("Codecov[Bot]", True), + ("dependabot", True), + ("dependabot[bot]", True), + ("asf-ci[bot]", True), + ("asfbot", True), + ("ASFBot", True), + ("custom-linter[bot]", True), + ("ayushmeel", False), + ("potiuk", False), + ("", False), + (None, False), + ], +) +def test_is_bot(login, expected): + assert _is_bot(login) is expected + + +def test_setup_required_inactivity_clock_resets_on_new_commit_sha(): + pr = _pr( + marker=_marker("setup_required", days_ago=7, sha="old_sha"), + head_sha="new_sha", + ) + plan, _, _ = _classify(pr, ForkObservation(workflow_state="disabled_fork")) + + assert plan.close is False + assert plan.state == ForkState.SETUP_REQUIRED + + +def test_setup_required_inactivity_clock_resets_on_recent_human_comment(): + pr = _pr( + marker=_marker("setup_required", days_ago=7), + last_comment_at=NOW - timedelta(days=2), + ) + plan, _, _ = _classify(pr, ForkObservation(workflow_state="disabled_fork")) + + assert plan.close is False + assert plan.state == ForkState.SETUP_REQUIRED + + +def test_parse_pull_request_tracks_reviews_and_comments(): + node = _pr_node( + 1, + latestReviews={ + "nodes": [ + { + "createdAt": "2026-11-05T00:00:00Z", + "author": {"login": "reviewer"}, + "comments": { + "nodes": [ + { + "createdAt": "2026-11-06T12:00:00Z", + "author": {"login": "code-suggester"}, + }, + { + "createdAt": "2026-11-09T00:00:00Z", + "author": {"login": "codecov[bot]"}, + }, + ] + }, + } + ] + }, + latestComments={ + "nodes": [ + { + "createdAt": "2026-11-07T00:00:00Z", + "author": {"login": "human-author"}, + }, + { + "createdAt": "2026-11-08T00:00:00Z", + "author": {"login": "Airflow-Triage-Bot"}, + }, + ] + }, + ) + pr = parse_pull_request(node) + assert pr.last_comment_at == datetime(2026, 11, 7, 0, 0, tzinfo=UTC) + + +def test_fetch_fork_observations_orders_by_created_at_and_run_attempt(client): + pr = parse_pull_request(_pr_node(1, head_repo="someone/my-airflow", head_sha="sha1")) + client.graphql.return_value = { + "pr0": { + "object": { + "checkSuites": { + "nodes": [ + { + "createdAt": "2026-11-02T00:00:00Z", + "updatedAt": "2026-11-02T00:30:00Z", + "status": "COMPLETED", + "conclusion": "FAILURE", + "workflowRun": { + "databaseId": 88, + "runAttempt": 1, + "url": "attempt-1", + "event": "workflow_dispatch", + "workflow": {"resourcePath": "/x/ci-amd.yml"}, + }, + }, + { + "createdAt": "2026-11-02T00:00:00Z", + "updatedAt": "2026-11-02T00:50:00Z", + "status": "COMPLETED", + "conclusion": "SUCCESS", + "workflowRun": { + "databaseId": 88, + "runAttempt": 2, + "url": "attempt-2", + "event": "workflow_dispatch", + "workflow": {"resourcePath": "/x/ci-amd.yml"}, + }, + }, + ] + } + } + } + } + + observations = fetch_fork_observations(client, [pr]) + assert observations[1].run_attempt == 2 + assert observations[1].suite_conclusion == "SUCCESS" + assert observations[1].run_url == "attempt-2" + + def test_older_pilot_pull_request_stays_checked_for_tampering_after_its_gated_run_is_deleted(): config = dataclasses.replace(CONFIG, enabled=False, pilot={"someone": JOINED})