diff --git a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md index d74f2e6a751e6..cfee5eae88af4 100644 --- a/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md +++ b/dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md @@ -575,7 +575,7 @@ one of their other open pull requests first. | released | Usual review with project CI | undraft, except the author's draft | unchanged | remove | | ungated | Usual review with project CI | undraft, except the author's draft | unchanged | remove | | closed | Author sets up fork CI and reopens | unchanged | remove | remove | -| gate_modified | Maintainer reopens and adds `use project ci` | unchanged | remove | remove | +| gate_modified | Author restores the gate and reopens, or maintainer adds `use project ci` | unchanged | remove | remove | | gate_bypassed | Maintainer reopens and adds `use project ci` | unchanged | remove | remove | The reconciler never undrafts the author's draft. It does not draft a closed pull request either, @@ -589,7 +589,7 @@ since the draft-state job runs after the close and GitHub refuses to draft a clo | **failed**: conclusion is failure, cancelled, timed out or startup failure | `failure` (red), link to the run; for a startup failure the description says the run could not start | update with the run link and the jobs that did not pass; for a startup failure the text says no job ran and points to the fork's Actions settings | | **green**: conclusion success and the pull request does not change CI-defining files | `success` (green), link to the run | update: ready for review, run link; on the author's draft: mark it ready for review when it is | | **green_unverified**: conclusion success, but the pull request changes CI-defining files (see "Trusting a green fork run") | `pending` (yellow), link to the run | update: a maintainer must run project CI; on the author's draft: mark it ready for review, then a maintainer runs project CI | -| **closed**: setup_required or awaiting_push for `close_after_days` | unchanged | close with a new comment; reopening restarts the clock | +| **closed**: setup_required or awaiting_push for `close_after_days` without human comments | unchanged | close with a new comment; reopening restarts the clock | | **released**: the pull request carries `use project ci` | `success` (green), link to the pull request's checks | update: project CI is in use; `fork ci required` is removed | | **ungated**: the base branch has no gate, for example after retargeting to an older release branch | `success` (green), link to the pull request's checks | update: project CI is in use; `fork ci required` is removed | | **gate_modified**: the pull request changes the fork CI gate (see "Gate protection") | `error` (red), link to the pull request's changed files | post a **new** comment explaining why, then close; checked after gate tampering, before the fork state | @@ -617,12 +617,16 @@ Rules that sit under the table: are touched, and adding `use project ci` takes a pull request out of the state machine on the next run (see "Override labels"). Removing `fork ci required` by hand is not enough: it updates the pull request, so discovery finds it again and labels it back while it is still gated. -- **Deleted head repository** is reported as unknown (see below) and otherwise left alone. +- **Deleted head repository** is verified via the GitHub REST API to rule out transient API + errors. If confirmed unavailable, it is reported as unknown (see below) and otherwise left alone. - **The close clock** starts at the `since` timestamp recorded for the current state and - resets on any state change. Closing for inactivity records `closed` in the marker, so the - first run after a reopen sees a state change from `closed` and starts a fresh clock rather - than closing the pull request again at once. Only a reopen recorded in the timeline after - the marker counts: an open pull request with a `closed` marker and no such reopen is one whose + resets on any state change or new commit push. Before closing after `close_after_days` (in + `setup_required` or `awaiting_push`), the reconciler checks for recent human activity + (including issue comments, reviews, inline diff comments, and suggestions); closure is deferred + as long as human discussion has occurred within `close_after_days`. Closing for inactivity records + `closed` in the marker, so the first run after a reopen sees a state change from `closed` and starts + a fresh clock rather than closing the pull request again at once. Only a reopen recorded in the timeline + after the marker counts: an open pull request with a `closed` marker and no such reopen is one whose closing comment or close failed, and the next run finishes the closure. The same holds for a `gate_bypassed` marker, where only a reopen by someone other than the author counts, so the closure is finished even after the offending run has left the tampering check's window. @@ -815,10 +819,9 @@ released to project CI. The state is `gate_modified`: the pull request loses `ready for maintainer review` and `needs project ci`, gets an `error` status linking its changed files, and the closure is -reported to Slack as an attention item. Gate tampering is checked first and wins. As for gate -tampering, only a reopen by someone other than the author counts: an open pull request whose -marker says `gate_modified` and that was not reopened since is closed again, and a maintainer's -reopen without restoring the gate closes it again on the next run. A maintainer who wants the +reported to Slack as an attention item. Gate tampering is checked first and wins. When closed for `gate_modified`, +the author can revert or remove the gate change and reopen the pull request; if reopened with the +gate change still present, it is closed again on the next run. A maintainer who wants the change adds `use project ci`, which releases the pull request before any of these checks. Each of these files carries a comment saying so. @@ -832,9 +835,9 @@ an HTML marker the reconciler parses: ``` followed by the human text for the current state. Editing does not notify, which is the -point: state churn is silent. The three moments that need the author's attention, the nudge -after enabling workflows, the close after inactivity and the close for gate tampering, are -new comments. +point: state churn is silent. The moments that need the author's attention, such as the nudge +after enabling workflows, the close after inactivity and the close for +gate tampering, are new comments. All comment texts live in `dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py` as templates so wording can be reviewed in one place. Every comment ends with the AI-attribution @@ -1213,7 +1216,7 @@ Unknown means any of: conclusion outside the modelled set. - A workflow state outside `active` / `disabled_fork` / `disabled_manually` / `disabled_inactivity` / not found. - A mutation that returned an error (label, comment, close, status). -- A head repository that no longer exists. +- A head repository that no longer exists (after confirming via the REST API to rule out transient API errors). - A state comment whose marker cannot be parsed. Expected states never alert. The message lists the pull request numbers and the reason for diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py index 9d852c7a9fca4..7378ad7f368db 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py @@ -353,7 +353,7 @@ def is_fork_ci_passed_description(description: str | None) -> bool: def state_comment( marker: Marker, *, - fork: str, + fork: str | None = None, run_url: str | None, base_ref: str = "main", author_draft: bool = False, @@ -374,7 +374,7 @@ def state_comment( parts.append(f"Latest run: {run_url}") if marker.state == "failed" and failed_jobs: parts.append(build_failed_job_list(failed_jobs)) - if marker.state == "setup_required": + if marker.state == "setup_required" and fork: parts.append(_setup_instructions(fork)) parts.append(f"How this works: [CI in your fork]({CONTRIBUTING_URL}).") return "\n\n".join(parts) + FOOTER @@ -398,8 +398,9 @@ def gate_modified_close_comment(*, change: str | None) -> str: "monitor, and a pull request whose CI runs in a fork must not change it, because the change would " "decide where its own CI runs. The one exception is adding, changing or removing your own `[pilot]` " "entry in `.github/fork-ci-gate.toml`.\n\n" - "If the change is wanted, a maintainer can reopen the pull request and add the `use project ci` " - "label." + FOOTER + "If the change was accidental, you can revert or remove the change to the gate files and reopen " + "the pull request. If the change is wanted, a maintainer can reopen the pull request and add the " + "`use project ci` label." + FOOTER ) diff --git a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py index ed01ce7ad9890..61a3036621943 100644 --- a/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py +++ b/dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py @@ -130,7 +130,19 @@ CHECK_RUN_NEEDED_JOB_NAME = "Check if the run is needed" PROJECT_CI_NOT_RUN_JOB_NAME = "CI passed (project CI not run)" CI_PASSED_CANCELLED_JOB_NAME = "CI passed (run cancelled)" -BOT_LOGINS = frozenset({"github-actions", "github-actions[bot]"}) +BOT_LOGINS = frozenset( + { + "github-actions", + "github-actions[bot]", + "airflow-triage-bot", + "codecov", + "codecov[bot]", + "dependabot", + "dependabot[bot]", + "asf-ci[bot]", + "asfbot", + } +) DISCOVERY_WINDOW = timedelta(hours=24) PASS_TWO_PAGE_SIZE = 25 FORK_BATCH_SIZE = 50 @@ -303,6 +315,7 @@ class PullRequestSnapshot: last_reopened_at: datetime | None = None last_reopened_by_anyone_at: datetime | None = None last_base_change_at: datetime | None = None + last_comment_at: datetime | None = None # (changed at, previous base, new base) for each retarget, oldest first. base_changes: tuple[tuple[datetime, str, str], ...] = () current_status: tuple[str, str, str] | None = None @@ -436,7 +449,10 @@ def _workflow_file(resource_path: str | None) -> str: def _is_bot(login: str | None) -> bool: - return login in BOT_LOGINS + if not login: + return False + normalized = login.casefold() + return normalized in BOT_LOGINS or normalized.endswith("[bot]") def observe_fork_state(fork: ForkObservation, marker: Marker | None) -> tuple[ForkState | None, str | None]: @@ -649,6 +665,7 @@ def classify( head_shared_across_bases: bool = False, find_gate_change: Callable[[], str | None] = lambda: None, find_failed_jobs: Callable[[], list[FailedJob] | None] = lambda: [], + verify_head_repo_unavailable: Callable[[], bool] = lambda: False, ) -> Plan: """ Decide the mutations for one gated pull request. @@ -710,6 +727,9 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: return release(ForkState.UNGATED, functools.partial(ungated_comment, base_ref=pr.base_ref)) if pr.head_repo is None: + if not verify_head_repo_unavailable(): + plan.unknown.append("head repository is unavailable due to GitHub API error; retrying") + return plan plan.unknown.append("head repository no longer exists") return plan @@ -739,7 +759,10 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: reopened_since_closing = { ForkState.CLOSED.value: pr.last_reopened_by_anyone_at, ForkState.GATE_BYPASSED.value: pr.last_reopened_at, - ForkState.GATE_MODIFIED.value: pr.last_reopened_at, + ForkState.GATE_MODIFIED.value: max( + (t for t in (pr.last_reopened_by_anyone_at, pr.last_reopened_at) if t is not None), + default=None, + ), } if marker and marker.state in reopened_since_closing: reopened = reopened_since_closing[marker.state] @@ -841,7 +864,7 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: "so `CI passed` waits for the next run" ) plan.state = state - since = marker.since if marker and marker.state == state.value else now + since = marker.since if marker and marker.state == state.value and marker.sha == pr.head_sha else now if marker and marker.state == ForkState.SETUP_REQUIRED.value and state == ForkState.AWAITING_PUSH: plan.new_comments.append(nudge_comment(fork=pr.head_repo)) @@ -856,7 +879,14 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: workflow_disabled_comment(fork=pr.head_repo, workflow_state=fork.workflow_state) ) - if state in CLOSABLE_STATES and now - since >= timedelta(days=config.close_after_days): + inactive_for_closing = pr.last_comment_at is None or now - pr.last_comment_at >= timedelta( + days=config.close_after_days + ) + if ( + state in CLOSABLE_STATES + and now - since >= timedelta(days=config.close_after_days) + and inactive_for_closing + ): _enter_state(plan, pr, labels, ForkState.CLOSED) _set_marker_comment( plan, @@ -867,7 +897,8 @@ def release(state: ForkState, comment: Callable[[Marker], str]) -> Plan: run_url=None, ), ) - plan.new_comments.append(inactivity_close_comment(days=config.close_after_days, fork=pr.head_repo)) + comment = inactivity_close_comment(days=config.close_after_days, fork=pr.head_repo or "") + plan.new_comments.append(comment) plan.close = True return plan @@ -965,6 +996,18 @@ def render(marker: Marker, failed_jobs: list[FailedJob]) -> str: ... on BaseRefChangedEvent { createdAt previousRefName currentRefName } } } + latestComments: comments(last: 10) { + nodes { createdAt author { login } } + } + latestReviews: reviews(last: 10) { + nodes { + createdAt + author { login } + comments(last: 10) { + nodes { createdAt author { login } } + } + } + } commits(last: 1) { nodes { commit { oid status { @@ -975,7 +1018,7 @@ def render(marker: Marker, failed_jobs: list[FailedJob]) -> str: } } } comments(first: 100) { pageInfo { hasNextPage endCursor } - nodes { databaseId body author { login } } + nodes { databaseId body createdAt author { login } } } } """ @@ -1044,15 +1087,17 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: last_reopened = None last_reopened_by_anyone = None last_base_change = None + last_comment_at = None base_changes = [] for item in (node.get("timelineItems") or {}).get("nodes") or []: if item["__typename"] == "ConvertToDraftEvent": last_draft_actor = _login(item.get("actor")) elif item["__typename"] == "BaseRefChangedEvent": last_base_change = parse_github_datetime(item["createdAt"]) - base_changes.append( - (last_base_change, item.get("previousRefName") or "", item.get("currentRefName") or "") - ) + if last_base_change is not None: + base_changes.append( + (last_base_change, item.get("previousRefName") or "", item.get("currentRefName") or "") + ) elif item["__typename"] == "ReopenedEvent": last_reopened_by_anyone = parse_github_datetime(item["createdAt"]) # Only a reopen by someone else counts for the gate tampering check: an author closing @@ -1064,6 +1109,23 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: if label == LABEL_USE_PROJECT_CI: last_unlabel = parse_github_datetime(item["createdAt"]) + for c in (node.get("latestComments") or {}).get("nodes") or []: + if not _is_bot(_login(c.get("author"))) and c.get("createdAt"): + t = parse_github_datetime(c["createdAt"]) + if t is not None: + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) + + for r in (node.get("latestReviews") or {}).get("nodes") or []: + if not _is_bot(_login(r.get("author"))) and r.get("createdAt"): + t = parse_github_datetime(r["createdAt"]) + if t is not None: + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) + for rc in (r.get("comments") or {}).get("nodes") or []: + if not _is_bot(_login(rc.get("author"))) and rc.get("createdAt"): + t = parse_github_datetime(rc["createdAt"]) + if t is not None: + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) + current_status = None status_created_at = None ci_passed_state = None @@ -1088,6 +1150,10 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: status_created_at = parse_github_datetime(context.get("createdAt") or "") comments = node.get("comments") or {} + for c in comments.get("nodes") or []: + if not _is_bot(_login(c.get("author"))) and c.get("createdAt"): + t = parse_github_datetime(c["createdAt"]) + last_comment_at = max(filter(None, (last_comment_at, t)), default=t) found = find_marker_comment(comments.get("nodes") or []) page_info = comments.get("pageInfo") or {} comments_cursor = page_info.get("endCursor") if found is None and page_info.get("hasNextPage") else None @@ -1113,6 +1179,7 @@ def parse_pull_request(node: dict[str, Any]) -> PullRequestSnapshot: last_reopened_at=last_reopened, last_reopened_by_anyone_at=last_reopened_by_anyone, last_base_change_at=last_base_change, + last_comment_at=last_comment_at, base_changes=tuple(base_changes), current_status=current_status, status_created_at=status_created_at, @@ -1754,13 +1821,19 @@ def fetch_fork_observations( suites = [ suite for suite in (commit.get("checkSuites") or {}).get("nodes") or [] - if (run := suite.get("workflowRun") or {}).get("event") == "push" + if (run := suite.get("workflowRun") or {}).get("event") in ("push", "workflow_dispatch") and _workflow_file((run.get("workflow") or {}).get("resourcePath")) == CI_WORKFLOW_FILE ] if not suites: observations[pr.number] = ForkObservation() continue - latest = max(suites, key=lambda suite: suite["createdAt"]) + latest = max( + suites, + key=lambda suite: ( + suite["createdAt"], + (suite.get("workflowRun") or {}).get("runAttempt") or 1, + ), + ) observations[pr.number] = ForkObservation( suite_status=latest["status"], suite_conclusion=latest["conclusion"], @@ -1998,7 +2071,7 @@ def load_workflow_triggers(workflows_dir: Path) -> dict[str, frozenset[str]]: """Map each workflow file name of a checkout to the events that trigger it.""" triggers = {} for path in sorted(workflows_dir.glob("*.y*ml")): - workflow = yaml.safe_load(path.read_text()) or {} + workflow = yaml.safe_load(path.read_text(encoding="utf-8")) or {} triggers[path.name] = get_workflow_triggers(workflow if isinstance(workflow, dict) else {}) return triggers @@ -2148,12 +2221,18 @@ def _other_pull_requests(self, pr: PullRequestSnapshot) -> list[int]: paginate=True, ) window_start = self._now - TAMPER_WINDOW - return [ - pull["number"] - for pull in pulls - if pull["number"] != pr.number - and (pull.get("closed_at") is None or parse_github_datetime(pull["closed_at"]) >= window_start) - ] + active_pulls = [] + for pull in pulls: + if pull["number"] == pr.number: + continue + closed_at_str = pull.get("closed_at") + if closed_at_str is None: + active_pulls.append(pull["number"]) + else: + closed_at = parse_github_datetime(closed_at_str) + if closed_at is not None and closed_at >= window_start: + active_pulls.append(pull["number"]) + return active_pulls def find(self, pr: PullRequestSnapshot) -> str | None: if not pr.head_repo: @@ -2404,7 +2483,7 @@ def find(self, pr: PullRequestSnapshot) -> str | None: repository(owner: $owner, name: $name) { pullRequest(number: $number) { comments(first: 100, after: $cursor) { pageInfo { hasNextPage endCursor } - nodes { databaseId body author { login } } + nodes { databaseId body createdAt author { login } } } } } } @@ -2419,6 +2498,10 @@ def find_state_comment(client: GitHubClient, repo: str, pr: PullRequestSnapshot) comments = client.graphql( COMMENTS_QUERY, {"owner": owner, "name": name, "number": pr.number, "cursor": cursor} )["repository"]["pullRequest"]["comments"] + for c in comments.get("nodes") or []: + if not _is_bot(_login(c.get("author"))) and c.get("createdAt"): + t = parse_github_datetime(c["createdAt"]) + pr = replace(pr, last_comment_at=max(filter(None, (pr.last_comment_at, t)), default=t)) found = find_marker_comment(comments["nodes"]) if found is not None: marker, comment_id, body, error = found @@ -2509,6 +2592,16 @@ def compare_with_base(client: GitHubClient, repo: str, pr: PullRequestSnapshot) return BaseCheck(merge_base, behind, tuple(sorted(overlap))) +def check_head_repo_unavailable(client: GitHubClient, repo: str, pr_number: int) -> bool: + try: + pr_data = client.rest_get(f"repos/{repo}/pulls/{pr_number}") + if not isinstance(pr_data, dict): + return False + return (pr_data.get("head") or {}).get("repo") is None + except Exception: + return False + + def find_heads_shared_across_bases(prs: Iterable[PullRequestSnapshot]) -> set[str]: """Head commits of more than one pull request whose bases differ.""" bases_by_head: dict[str, set[str]] = {} @@ -2868,6 +2961,9 @@ def is_gated_base(base_ref: str) -> bool: head_shared_across_bases=pr.head_sha in shared_heads, find_gate_change=functools.partial(gate_change_finder.find, pr), find_failed_jobs=functools.partial(read_failed_jobs, client, pr, fork, read_jobs), + verify_head_repo_unavailable=functools.partial( + check_head_repo_unavailable, client, repo, pr.number + ), ) # A revocation to `pending` always goes through: it is idempotent and must not depend on # `report_project_ci_passed` succeeding later in this run. diff --git a/dev/breeze/tests/test_fork_ci_reconcile.py b/dev/breeze/tests/test_fork_ci_reconcile.py index fa1d432977e2a..4e2ddd2bf9038 100644 --- a/dev/breeze/tests/test_fork_ci_reconcile.py +++ b/dev/breeze/tests/test_fork_ci_reconcile.py @@ -48,6 +48,7 @@ ) from airflow_breeze.utils.fork_ci_reconcile import ( DEFERRED_RUNS_BEFORE_ALERT, + GATE_PROTECTED_PATHS, JOB_STATUS_FAILURE_RUNS_BEFORE_ALERT, JOB_STATUSES_DEFERRED_NOTE, JOB_STATUSES_FAILED_REASON, @@ -68,10 +69,12 @@ Projection, PullRequestChangedError, PullRequestSnapshot, + _is_bot, apply_plan, build_job_statuses, build_slack_payload, changes_ci_definition, + check_head_repo_unavailable, classify, compare_with_base, compute_projection, @@ -156,6 +159,7 @@ def _classify( base=CURRENT_BASE, head_shared_across_bases=False, gate_change=None, + verify_head_repo_unavailable=lambda: False, ): find_bypass = mock.Mock(return_value=bypass_run) changes = mock.Mock(return_value=changes_ci_files) @@ -170,6 +174,7 @@ def _classify( compare_base=mock.Mock(return_value=base), head_shared_across_bases=head_shared_across_bases, find_gate_change=lambda: gate_change, + verify_head_repo_unavailable=verify_head_repo_unavailable, ) return plan, find_bypass, changes @@ -822,6 +827,31 @@ def test_close_clock(marker, should_close): assert plan.state == ForkState.SETUP_REQUIRED +@pytest.mark.parametrize( + ("marker", "last_comment_at", "should_close"), + [ + pytest.param(_marker("failed", days_ago=7), None, True, id="failed-seven-days-no-comments"), + pytest.param( + _marker("failed", days_ago=7), NOW - timedelta(days=8), True, id="failed-seven-days-old-comments" + ), + pytest.param( + _marker("failed", days_ago=7), + NOW - timedelta(days=2), + False, + id="failed-seven-days-recent-comments", + ), + pytest.param(_marker("failed", days_ago=6.9), None, False, id="failed-under-seven-days"), + ], +) +def test_failing_pull_request_inactivity_clock(marker, last_comment_at, should_close): + pr = _pr(marker=marker, last_comment_at=last_comment_at) + failed_fork = ForkObservation(suite_status="COMPLETED", suite_conclusion="FAILURE") + plan, _, _ = _classify(pr, failed_fork) + + assert plan.close is False + assert plan.state == ForkState.FAILED + + @pytest.mark.parametrize( "reopened_at", [pytest.param(None, id="never-reopened"), pytest.param(NOW - timedelta(days=31), id="reopened-before")], @@ -959,12 +989,46 @@ def test_both_override_labels_are_reported(): assert len(plan.attention) == 1 -def test_deleted_head_repository_is_unknown(): - plan, _, _ = _classify(_pr(head_repo=None), GREEN) +def test_deleted_head_repository_reported_as_unknown(): + plan, _, _ = _classify(_pr(head_repo=None), GREEN, verify_head_repo_unavailable=lambda: True) + assert plan.state is None + assert plan.close is False assert plan.unknown == ["head repository no longer exists"] - assert plan.status is None - assert plan.draft is None + assert plan.new_comments == [] + + +def test_deleted_head_repository_default_retries_safely(): + plan, _, _ = _classify(_pr(head_repo=None), GREEN) + + assert plan.state is None + assert plan.close is False + assert plan.unknown == ["head repository is unavailable due to GitHub API error; retrying"] + assert plan.new_comments == [] + + +def test_deleted_head_repository_retries_on_api_error(): + plan, _, _ = _classify(_pr(head_repo=None), GREEN, verify_head_repo_unavailable=lambda: False) + + assert plan.state is None + assert plan.close is False + assert plan.unknown == ["head repository is unavailable due to GitHub API error; retrying"] + assert plan.new_comments == [] + + +def test_check_head_repo_unavailable(): + client = mock.Mock() + client.rest_get.return_value = {"head": {"repo": None}} + assert check_head_repo_unavailable(client, "apache/airflow", 123) is True + + client.rest_get.return_value = {"head": {"repo": {"full_name": "user/repo"}}} + assert check_head_repo_unavailable(client, "apache/airflow", 123) is False + + client.rest_get.return_value = None + assert check_head_repo_unavailable(client, "apache/airflow", 123) is False + + client.rest_get.side_effect = GitHubError("500 Internal Server Error") + assert check_head_repo_unavailable(client, "apache/airflow", 123) is False def test_malformed_marker_is_reported_and_rewritten(): @@ -2362,6 +2426,44 @@ def test_fork_observations_use_the_fork_name_and_latest_suite(client): assert client.graphql.call_args.args[1] == {"o0": "someone", "n0": "my-airflow", "s0": "sha1"} +def test_fetch_fork_observations_accepts_workflow_dispatch(client): + pr = parse_pull_request(_pr_node(1, head_repo="someone/my-airflow", head_sha="sha1")) + client.graphql.return_value = { + "pr0": { + "object": { + "checkSuites": { + "nodes": [ + { + "createdAt": "2026-11-02T00:00:00Z", + "updatedAt": "2026-11-02T00:40:00Z", + "status": "COMPLETED", + "conclusion": "SUCCESS", + "workflowRun": { + "databaseId": 88, + "runAttempt": 1, + "url": "dispatch-run", + "event": "workflow_dispatch", + "workflow": {"resourcePath": "/x/ci-amd.yml"}, + }, + }, + ] + } + } + } + } + + observations = fetch_fork_observations(client, [pr]) + + assert observations[1] == ForkObservation( + suite_status="COMPLETED", + suite_conclusion="SUCCESS", + run_url="dispatch-run", + run_id=88, + run_attempt=1, + completed_at=datetime(2026, 11, 2, 0, 40, tzinfo=UTC), + ) + + def test_state_comment_is_found_beyond_the_first_page(client): marker = _marker("running") first_page = { @@ -3432,6 +3534,34 @@ def test_unfinished_gate_change_closure_is_retried(): assert "Closing this pull request because it changes the fork CI gate." in plan.new_comments[0] +def test_author_reopen_allowed_when_gate_change_removed(): + pr = _pr( + marker=_marker("gate_modified", days_ago=1), + last_reopened_by_anyone_at=NOW, + ) + plan, _, _ = _classify(pr, GREEN, gate_change=None) + + assert plan.state == ForkState.GREEN + assert plan.close is False + + +def test_author_reopen_reclosed_when_gate_change_remains(): + pr = _pr( + marker=_marker("gate_modified", days_ago=1), + last_reopened_by_anyone_at=NOW, + ) + plan, _, _ = _classify( + pr, GREEN, gate_change="it changes `dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`" + ) + + assert plan.state == ForkState.GATE_MODIFIED + assert plan.close is True + assert ( + "changes the fork CI gate: it changes `dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`" + in plan.new_comments[0] + ) + + PILOT_BASE = "enabled = true\n[pilot]\nother = 2026-10-01T00:00:00Z\n" @@ -3526,6 +3656,12 @@ def rest_get(path, paginate=False, items_key=None): "it changes `.github/workflows/fork-ci-monitor.yml`", id="monitor-moved-away", ), + pytest.param( + [{"filename": "dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py"}], + {}, + "it changes `dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py`", + id="reconciler-code-protected", + ), pytest.param( [{"filename": ".github/fork-ci-gate.toml"}], {".github/fork-ci-gate.toml": "enabled = false\n"}, @@ -3934,6 +4070,150 @@ def test_mark_older_pilot_pull_request( assert (marked.ci_run_at, marked.pilot_joined_at) == (expected_ci_run_at, expected_pilot_joined_at) +def test_gate_protected_paths_includes_all_five_files(): + expected = { + "dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py", + "dev/breeze/src/airflow_breeze/utils/fork_ci_github.py", + "dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py", + "dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py", + ".github/workflows/fork-ci-monitor.yml", + } + assert expected == GATE_PROTECTED_PATHS + + +@pytest.mark.parametrize( + ("login", "expected"), + [ + ("github-actions", True), + ("github-actions[bot]", True), + ("airflow-triage-bot", True), + ("Airflow-Triage-Bot", True), + ("codecov", True), + ("codecov[bot]", True), + ("Codecov[Bot]", True), + ("dependabot", True), + ("dependabot[bot]", True), + ("asf-ci[bot]", True), + ("asfbot", True), + ("ASFBot", True), + ("custom-linter[bot]", True), + ("ayushmeel", False), + ("potiuk", False), + ("", False), + (None, False), + ], +) +def test_is_bot(login, expected): + assert _is_bot(login) is expected + + +def test_setup_required_inactivity_clock_resets_on_new_commit_sha(): + pr = _pr( + marker=_marker("setup_required", days_ago=7, sha="old_sha"), + head_sha="new_sha", + ) + plan, _, _ = _classify(pr, ForkObservation(workflow_state="disabled_fork")) + + assert plan.close is False + assert plan.state == ForkState.SETUP_REQUIRED + + +def test_setup_required_inactivity_clock_resets_on_recent_human_comment(): + pr = _pr( + marker=_marker("setup_required", days_ago=7), + last_comment_at=NOW - timedelta(days=2), + ) + plan, _, _ = _classify(pr, ForkObservation(workflow_state="disabled_fork")) + + assert plan.close is False + assert plan.state == ForkState.SETUP_REQUIRED + + +def test_parse_pull_request_tracks_reviews_and_comments(): + node = _pr_node( + 1, + latestReviews={ + "nodes": [ + { + "createdAt": "2026-11-05T00:00:00Z", + "author": {"login": "reviewer"}, + "comments": { + "nodes": [ + { + "createdAt": "2026-11-06T12:00:00Z", + "author": {"login": "code-suggester"}, + }, + { + "createdAt": "2026-11-09T00:00:00Z", + "author": {"login": "codecov[bot]"}, + }, + ] + }, + } + ] + }, + latestComments={ + "nodes": [ + { + "createdAt": "2026-11-07T00:00:00Z", + "author": {"login": "human-author"}, + }, + { + "createdAt": "2026-11-08T00:00:00Z", + "author": {"login": "Airflow-Triage-Bot"}, + }, + ] + }, + ) + pr = parse_pull_request(node) + assert pr.last_comment_at == datetime(2026, 11, 7, 0, 0, tzinfo=UTC) + + +def test_fetch_fork_observations_orders_by_created_at_and_run_attempt(client): + pr = parse_pull_request(_pr_node(1, head_repo="someone/my-airflow", head_sha="sha1")) + client.graphql.return_value = { + "pr0": { + "object": { + "checkSuites": { + "nodes": [ + { + "createdAt": "2026-11-02T00:00:00Z", + "updatedAt": "2026-11-02T00:30:00Z", + "status": "COMPLETED", + "conclusion": "FAILURE", + "workflowRun": { + "databaseId": 88, + "runAttempt": 1, + "url": "attempt-1", + "event": "workflow_dispatch", + "workflow": {"resourcePath": "/x/ci-amd.yml"}, + }, + }, + { + "createdAt": "2026-11-02T00:00:00Z", + "updatedAt": "2026-11-02T00:50:00Z", + "status": "COMPLETED", + "conclusion": "SUCCESS", + "workflowRun": { + "databaseId": 88, + "runAttempt": 2, + "url": "attempt-2", + "event": "workflow_dispatch", + "workflow": {"resourcePath": "/x/ci-amd.yml"}, + }, + }, + ] + } + } + } + } + + observations = fetch_fork_observations(client, [pr]) + assert observations[1].run_attempt == 2 + assert observations[1].suite_conclusion == "SUCCESS" + assert observations[1].run_url == "attempt-2" + + def test_older_pilot_pull_request_stays_checked_for_tampering_after_its_gated_run_is_deleted(): config = dataclasses.replace(CONFIG, enabled=False, pilot={"someone": JOINED})