Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions contributing-docs/18_contribution_workflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,12 @@ does not run for 30 days is closed with an explanation; you can reopen it after
files that decide which CI jobs run and where, such as ``.github/``, the breeze sources or
``scripts/ci/prek/``, a green fork run is not enough: a maintainer reviews the change and runs project CI for it.

Fork CI tests your branch as it is, while project CI tests it merged with the target branch. The
``CI passed`` check that merging requires is therefore reported only when your branch is fewer than
200 commits behind the target branch and the target branch has not changed any file your pull request
changes since your branch's base. Otherwise the pull request comment lists those files and asks you
to rebase on ``upstream/main`` (or the target branch) and push, which runs fork CI again.

**Joining the pilot.** Open a pull request that adds your GitHub login to the ``[pilot]`` table in
``.github/fork-ci-gate.toml``, with a join instant after the expected merge, for example the next day
at 00:00 UTC:
Expand Down
68 changes: 57 additions & 11 deletions dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
Original file line number Diff line number Diff line change
Expand Up @@ -1051,6 +1051,8 @@ run.
pending or failing one would say, and it never competes with project CI's own check.
- `green_unverified` gets no `CI passed`: the fork run used CI files the pull request changed, so
it is not evidence. Once a maintainer adds `use project ci`, project CI reports the check.
- A `green` run whose base is not current gets a `pending` `CI passed` asking to rebase, and one
whose base could not be compared gets none (see "Tested base").
- When the head commit leaves `green`, for example a re-run in the fork that fails, a `success`
posted earlier is overwritten with `pending` so the pull request cannot merge on it.
- A pull request carrying `use project ci` cannot merge on the fork result. GitHub requires both to
Expand Down Expand Up @@ -1164,14 +1166,46 @@ conditions: completed, first attempt, its commit does not change `.github/workfl
did project CI work by the tampering check's rules. Of its jobs only the gate and the jobs that
also run in a gated run started, none failed, and at least one was cancelled.

### Far behind the base branch

A fork run tests the pushed commit as it is, while project CI tests it merged with the base
branch. When a head commit turns green, the reconciler counts the commits it is behind its base
branch (`behind_by` of `GET /repos/apache/airflow/compare/<base>...<fork-owner>:<sha>`, once per
head commit, stored in the marker). From 200 commits behind, about four days of `main`, the
state comment says so and asks the contributor to rebase, since the merged result can still
fail. The count is advisory and does not change the state.
### Tested base

A fork run tests the pushed commit as it is, while project CI's `pull_request` run tests GitHub's
merge commit of the pull request into its base branch. A fork run on an old base could otherwise go
green and merge although the merged result was never tested. So `CI passed` is only reported for a
fork run whose base is close enough to the base branch that the merge changes nothing the pull
request touches.

When a head commit turns green, the reconciler compares it with the base branch, once per head
commit, and stores the result in the marker as `"base": {"sha", "behind", "overlap"}`:

- `GET /repos/apache/airflow/compare/<base>...<fork-owner>:<sha>` gives the merge base, the commit
of the base branch the fork run tested on top of (`sha`), the commits the base branch has since
(`behind`), and the files the pull request changes.
- When `behind` is not zero, `GET /repos/apache/airflow/compare/<merge base>...<base>` gives the
files the base branch changed since. `overlap` is the paths in both lists, a renamed file
counting under both paths.

`CI passed` is reported only when the base is current: fewer than `MAX_BEHIND_COMMITS` (200, about
four days of `main`) commits behind and no overlap. Otherwise the state stays `green` (the pull
request can be reviewed, and its status, labels and draft state are those of `green`), the state
comment lists the overlapping files or says how far behind the base is and gives the rebase
commands, and `CI passed` is posted as `pending` with a description saying to rebase. A rebase and
push starts a new fork run, which is compared again when it turns green. Every green state comment
names the base commit the run tested and how far behind it was, and the `CI passed` description
names it as well.

The compare API lists at most 300 files. A pull request with more is listed with
`GET /repos/apache/airflow/pulls/<n>/files` instead (up to 3000); a base branch that changed 300 or
more files since the merge base cannot be checked and counts as overlapping, which a rebase
resolves. When the compare fails, `CI passed` is not reported, the failure is reported to Slack and
the next run compares again. A marker written before this check, with no `base`, is compared again
on the next run, so a pull request that already had `CI passed` loses it if its base turns out not
to be current. A retarget compares again, as it decides green trust again.

This narrows the gap to project CI without closing it. A base change in other files can still break
the pull request, within at most 200 commits, which matches what project CI leaves open after its
run: a `pull_request` run is not repeated when the base branch moves, so the merged result at merge
time is not the one it tested either. Testing the merge commit in the fork itself was considered and
rejected; see "Known limitations".

### Unknown failures and attention items go to Slack

Expand Down Expand Up @@ -1218,8 +1252,8 @@ Per run, with ~700 gated pull requests: ~28 GraphQL calls for pass 2, ~14 for fo
~1 for pass 1, plus one REST call per pull request in `setup_required`, a few paginated REST
calls for the recent `pull_request` runs, one REST call per base branch for its workflow
files, a jobs call and a compare call only per long `ci-amd.yml` run of a gated pull request,
one changed-files call per head commit that turns green, plus one mutation per actual state
change. Job statuses add a jobs call and a status call per pull request with a fork run in
one changed-files call and one or two compare calls per head commit that turns green, plus one
mutation per actual state change. Job statuses add a jobs call and a status call per pull request with a fork run in
progress or a completed run not yet reported, and up to two status mutations per job, at most
`MAX_JOB_STATUSES_PER_RUN` per run; see "Write budget". Pull requests that pass 1 labelled and
the search does not list yet add one GraphQL call per 25. Well inside
Expand Down Expand Up @@ -1426,6 +1460,17 @@ as a failed check rather than a crash.
- An external pull request that legitimately changes workflow files, was released with `use
project ci` and then had that label removed is closed on the next run, because both
signals hold. Keeping `use project ci` on such a pull request is the intended handling.
- Fork CI tests the head commit, not the pull request merged into its base branch; "Tested base"
only bounds the difference. Testing the merged result in the fork was rejected because:
every job of the fork run checks out the pushed commit (about 60 `actions/checkout` steps across
`ci-amd.yml` and its reusable workflows, plus the image builds and caches keyed by the commit),
and each would have to check out one merge commit fixed for the whole run; the first push of a
branch happens before its pull request exists, so there is no `refs/pull/<n>/merge` to test yet;
GitHub computes that ref asynchronously after each push and replaces it whenever the base branch
moves, so a run would have to wait for it and could find its commit gone mid-run; and the
reconciler would have to trust the run's own report of which merge it tested. Merging the base
branch into the head in every job instead needs the shared history of both, which shallow
checkouts do not have.

## Testing

Expand All @@ -1442,7 +1487,8 @@ as a failed check rather than a crash.
close), a fork with a name other than `airflow`, marker round-trips, status
desired-vs-current comparison, the unknown-failure collection, green trust (a change to
CI-defining files keeps the status pending without drafting the pull request, decided once
per commit) and the gate
per commit), the tested base (`CI passed` only on a current base, the marker and comment naming
the base, compared again for a new commit, a retarget or a marker without a base) and the gate
tampering check: a tampered run on an earlier commit is caught after a clean push, each
signal alone does not close, short runs and gate-only jobs do not count, an in-progress
non-gate job, a removed, duplicated or long-running gate job and an injected workflow with a
Expand Down
122 changes: 103 additions & 19 deletions dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@
# CI's gate job reports it; for a gated pull request, which never runs project CI, the reconciler
# reports it from the fork run.
CI_PASSED_CONTEXT = "CI passed"
CI_PASSED_DESCRIPTION = "Fork CI passed for this commit"
CI_PASSED_REVOKED_DESCRIPTION = "Fork CI result no longer trusted for this commit"
# Replaces a fork `success` on a pull request carrying `use project ci` until project CI has passed.
CI_PASSED_WAITING_PROJECT_CI_DESCRIPTION = "Waiting for project CI (use project ci)"
Expand All @@ -56,9 +55,12 @@
CI_RUN_STARTED_CONTEXT = "CI run started"
CI_RUN_STARTED_DESCRIPTION = "CI runs in the contributor's fork - details open the run"
CI_RUN_STARTED_PROJECT_DESCRIPTION = "Project CI runs in apache/airflow - details open the run"
# A fork run tests the head commit as it is, not merged with the base branch. Further behind than
# this, it says little about the result after merge.
BEHIND_WARNING_COMMITS = 200
# A fork run tests the head commit as it is, not merged with the base branch as project CI does. From
# this many commits behind, `CI passed` is withheld until the branch is rebased, as it is when the
# base branch changed any file the pull request changes.
MAX_BEHIND_COMMITS = 200
# Overlapping files named in the state comment; the marker keeps them all.
OVERLAP_LISTED = 10
MARKER_PREFIX = "<!-- fork-ci-gate: "
MARKER_PATTERN = re.compile(r"<!-- fork-ci-gate: (?P<payload>.*?) -->")
# Texts on pull requests are for their authors, so they link the contributor documentation; it links
Expand Down Expand Up @@ -87,13 +89,34 @@
)


@dataclass(frozen=True)
class BaseCheck:
"""
The base a green fork run tested, compared with the base branch when the run turned green.

:param sha: the merge base of the head commit and the base branch, the base the run tested.
:param behind: commits on the base branch since ``sha``.
:param overlap: paths that both the pull request and the base branch since ``sha`` change;
None when GitHub listed too many changed files on either side to tell.
"""

sha: str
behind: int
overlap: tuple[str, ...] | None = ()

@property
def is_current(self) -> bool:
"""Whether the run tested close enough to the merged result for `CI passed`."""
return self.behind < MAX_BEHIND_COMMITS and self.overlap == ()


@dataclass(frozen=True)
class Marker:
state: str
sha: str
since: datetime
# Commits the head commit was behind its base branch when its fork CI turned green.
behind: int | None = None
# Set when the head commit's fork CI turned green and its base could be compared.
base: BaseCheck | None = None
# The completed fork run (``<run id>:<attempt>:<conclusion>``) whose jobs are all reported.
jobs: str | None = None
# The fork run (``<run id>:<attempt>``) whose failed jobs the comment lists.
Expand All @@ -105,8 +128,12 @@ class Marker:

def render_marker(marker: Marker) -> str:
data: dict[str, object] = {"state": marker.state, "sha": marker.sha, "since": marker.since.isoformat()}
if marker.behind is not None:
data["behind"] = marker.behind
if marker.base is not None:
data["base"] = {
"sha": marker.base.sha,
"behind": marker.base.behind,
"overlap": None if marker.base.overlap is None else list(marker.base.overlap),
}
if marker.jobs is not None:
data["jobs"] = marker.jobs
if marker.failed_run is not None:
Expand All @@ -123,11 +150,19 @@ def parse_marker(body: str) -> Marker | None:
return None
try:
data = json.loads(match.group("payload"))
# A marker written before the base was checked carries only "behind"; its base is checked again.
base = data.get("base")
return Marker(
state=data["state"],
sha=data["sha"],
since=datetime.fromisoformat(data["since"]),
behind=data.get("behind"),
base=None
if base is None
else BaseCheck(
sha=base["sha"],
behind=base["behind"],
overlap=None if base["overlap"] is None else tuple(base["overlap"]),
),
jobs=data.get("jobs"),
failed_run=data.get("failed_run"),
gate_checked=data.get("gate_checked") is True,
Expand Down Expand Up @@ -248,11 +283,64 @@ def strip_failed_job_list(body: str) -> str:
return "\n\n".join(part for part in body.split("\n\n") if not part.startswith(FAILED_JOB_LIST_HEADING))


def behind_warning(behind: int, *, base_ref: str) -> str:
def tested_base_text(base: BaseCheck, *, base_ref: str) -> str:
text = (
f"Fork CI tested this branch on top of `{base_ref}` at "
f"[`{base.sha[:10]}`](https://github.com/apache/airflow/commit/{base.sha}), "
f"{base.behind} commits behind `{base_ref}` when the run turned green. Fork CI tests the branch "
f"as it is, while project CI would test it merged with the current `{base_ref}`."
)
if base.is_current:
return f"{text} No commit since changed a file this pull request changes."
if base.overlap is None:
reason = f"`{base_ref}` changed too many files since to tell whether it changed the same files."
elif base.overlap:
listed = ", ".join(f"`{path}`" for path in base.overlap[:OVERLAP_LISTED])
more = len(base.overlap) - OVERLAP_LISTED
reason = f"`{base_ref}` changed files this pull request changes since: {listed}" + (
f" and {more} more." if more > 0 else "."
)
else:
reason = "That is too far behind to say much about the merged result."
return (
f"**This branch is {behind} commits behind `{base_ref}`.** Fork CI tests the branch as it is, "
f"while the merged result also contains those commits, so it can still fail after the merge. "
f"Rebase on `{base_ref}` and push to test it against the current code."
f"{text}\n\n**Rebase needed.** {reason} The `CI passed` check is not reported until fork CI is "
f"green for a branch rebased on `{base_ref}`:\n\n"
f"```bash\ngit fetch upstream\ngit rebase upstream/{base_ref}\ngit push --force-with-lease\n```"
)


_FORK_CI_PASSED_PREFIX = "Fork CI passed"
_REBASE_NEEDED_PREFIX = "Rebase on "


def ci_passed_description(base: BaseCheck, *, base_ref: str) -> str:
return (
f"{_FORK_CI_PASSED_PREFIX}; base {base_ref}@{base.sha[:10]} was {base.behind} commits behind, "
"no overlap"
)


def rebase_needed_description(base: BaseCheck, *, base_ref: str) -> str:
if base.overlap is None:
why = f"{base_ref} changed too many files since to check"
elif base.overlap:
why = f"{base_ref} changed {len(base.overlap)} of the same files since"
else:
why = f"base is {base.behind} commits behind"
return f"{_REBASE_NEEDED_PREFIX}{base_ref} and push: {why}"


def is_fork_ci_passed_description(description: str | None) -> bool:
"""
Whether a `CI passed` status description is one the reconciler wrote from a fork run.

The prefix also matches "Fork CI passed for this commit", written before the base was checked.
"""
if not description:
return False
return description.startswith((_FORK_CI_PASSED_PREFIX, _REBASE_NEEDED_PREFIX)) or description in (
CI_PASSED_WAITING_PROJECT_CI_DESCRIPTION,
CI_PASSED_REVOKED_DESCRIPTION,
)


Expand All @@ -274,12 +362,8 @@ def state_comment(
if startup_failure:
text = STARTUP_FAILURE_TEXT
parts = [render_marker(marker), text or _STATE_TEXTS[marker.state], WHY_FORK_CI]
if (
marker.state in ("green", "green_unverified")
and marker.behind is not None
and marker.behind >= BEHIND_WARNING_COMMITS
):
parts.append(behind_warning(marker.behind, base_ref=base_ref))
if marker.state in ("green", "green_unverified") and marker.base is not None:
parts.append(tested_base_text(marker.base, base_ref=base_ref))
if run_url:
parts.append(f"Latest run: {run_url}")
if marker.state == "failed" and failed_jobs:
Expand Down
Loading
Loading