Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/fork-ci-gate.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@
# under the License.

# Fork CI gate (AIP-120). See dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
#
# This file is part of the gate. A pull request whose CI runs in the contributor's fork may only
# add, change or remove its author's own [pilot] entry here; the fork CI monitor closes it if it
# changes anything else, because that change would decide where its own CI runs. See
# "Gate protection" in the document above.

# Master switch. When false the gate job lets every pull request run project CI, except
# pull requests labelled `use fork ci` and pull requests of [pilot] participants. Push
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/ci-amd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,9 @@ jobs:
gate:
# Fork CI gate (AIP-120): decides whether project CI runs for this event. Configuration and
# code come from the base branch so a pull request cannot change its own eligibility. See
# dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.
# dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md. A pull request whose CI runs
# in the contributor's fork and changes this job, or how build-info depends on it, is closed
# by the fork CI monitor ("Gate protection" in that document).
name: "Fork CI gate"
runs-on: ["ubuntu-slim"]
timeout-minutes: 5
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/fork-ci-monitor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
# those pull requests, and deletes their project CI runs that only skipped every job. It never checks
# out or runs pull request code. See
# dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.
#
# Part of the fork CI gate: a pull request whose CI runs in the contributor's fork and changes
# this file is closed by this monitor ("Gate protection" in the document above).
name: "Fork CI monitor"
on: # yamllint disable-line rule:truthy
schedule:
Expand Down
5 changes: 4 additions & 1 deletion contributing-docs/18_contribution_workflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,10 @@ at 00:00 UTC:
"your-login" = YYYY-MM-DDT00:00:00Z

Every pull request you open after that instant uses fork CI; your earlier pull requests keep project
CI. To leave the pilot, open a pull request that removes the line. A pilot entry gates your pull
CI. To leave the pilot, open a pull request that removes the line. Your own ``[pilot]`` line is
the only part of the fork CI gate a pull request using fork CI may change: one that changes the
gate's code, the rest of its configuration or the ``gate`` job of ``ci-amd.yml`` is closed with an
explanation. A pilot entry gates your pull
requests whatever your role, so a committer in the pilot also enables the workflow;
``breeze ci audit`` tells you which applies to you. The design is described in
`Fork CI for external contributors <../dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md>`__.
Expand Down
6 changes: 5 additions & 1 deletion contributing-docs/25_maintainer_pr_triage.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ The triage workflow uses the following labels and states:
## Pull requests gated to fork CI

Pull requests labelled `fork ci required` run their CI in the contributor's fork. The scheduled
fork CI reconciler (`breeze ci fork-ci-reconcile`, run by `fork-ci-monitor.yml`) owns four things
fork CI reconciler (`breeze ci fork-ci-reconcile`, run by `fork-ci-monitor.yml`) owns five things
for them, and the triage skill must treat those pull requests as out of scope for each:

- **Draft state.** The reconciler keeps the pull request in draft until the fork run for its
Expand All @@ -261,6 +261,10 @@ for them, and the triage skill must treat those pull requests as out of scope fo
- **`needs project ci`.** The reconciler adds it when the fork run is green but the pull request
changes CI-defining files, and removes it once the state is anything else. These pull requests
stay in draft and wait for a maintainer to review the CI change and add `use project ci`.
- **Closing for changing the gate.** The reconciler closes a pull request that changes the fork
CI gate (its code, configuration other than the author's own `[pilot]` entry, the `gate` job
of `ci-amd.yml` or the monitor), and reports it on Slack. Add `use project ci` to accept such
a change.
- **Closing after inactivity.** The reconciler closes a pull request whose fork CI has not run for
`close_after_days`.

Expand Down
97 changes: 77 additions & 20 deletions dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,7 @@ pull requests without a fork run for their head commit, not by the total.
| **green_unverified**: conclusion success, but the pull request changes CI-defining files (see "Trusting a green fork run") | draft | remove | `pending` (yellow), link to the run | update: a maintainer must run project CI; labelled `needs project ci` |
| setup_required or awaiting_push for `close_after_days` | unchanged | unchanged | unchanged | close with a new comment; reopening restarts the clock |
| **ungated**: the base branch has no gate, for example after retargeting to an older release branch | undraft\* | unchanged | `success` (green), link to the pull request's checks | update: project CI is in use; `fork ci required` is removed |
| **gate_modified**: the pull request changes the fork CI gate (see "Gate protection") | draft | remove | `error` (red), link to the pull request's changed files | post a **new** comment explaining why, then close; checked after gate tampering, before the fork state |
| **gate_bypassed**: a recorded project CI run of the pull request's branch bypassed the gate (see "Gate tampering") | draft | remove | `error` (red), link to the project CI run that should not have happened | post a **new** comment explaining why, then close; takes precedence over every row above |

\* only if the last convert-to-draft event on the pull request was by `github-actions[bot]`;
Expand Down Expand Up @@ -545,6 +546,10 @@ Rules that sit under the table:
closing comment or close failed, and the next run finishes the closure. The same holds for a
`gate_bypassed` marker, where only a reopen by someone other than the author counts, so the
closure is finished even after the offending run has left the tampering check's window.
- **New comments go first.** A nudge, a "fork CI stopped running" notice, a closing explanation
or a tampering notice is posted before the state comment records the new state, so when
posting fails the next run sees the old state and posts it again; a repeat is preferred to a
lost notice.
- **`fork ci required` goes last.** When a run removes the label (on release, or for an
ungated base branch), it does so after every other change, so a failure on the way leaves
the label in place and the next run retries the rest.
Expand Down Expand Up @@ -572,7 +577,10 @@ The decision is read back from the `Fork CI / Tests (AMD)` commit status on the
which only `apache/airflow`'s own tokens can write, so it is made once per commit and cannot be
forged through a comment. The changed files depend on the base branch as well, so a decision
recorded before the pull request was last retargeted (a `BaseRefChangedEvent` in the timeline
newer than the status) is made again, and the status is written again to date it.
newer than the status) is made again, and the status is written again to date it. A commit status
belongs to the commit, so it cannot hold one decision per base: when another gated pull
request with a different base has the same head commit, both are `green_unverified` and wait
for a maintainer, reported to Slack once.

This is a guard against the cheapest way to fake a green run, not a guarantee. Any change can
weaken the tests it ships with (a `conftest.py` that skips tests, for example), exactly as it
Expand Down Expand Up @@ -608,28 +616,36 @@ ignored, because they ran legitimately while the label was on. So are runs creat
most recent reopen of the pull request by someone other than its author, which is how a
maintainer brings back a pull request closed for tampering; reopening one whose gate is still
missing triggers a new run, which is caught again. A reopen by the author does not count, so
closing and reopening one's own pull request cannot hide a tampered run. A run counts as a
bypass when both signals hold:
closing and reopening one's own pull request cannot hide a tampered run. Runs are listed per
fork branch, because GitHub records no pull request on runs from forks. A run made while the
pull request targeted a base branch without the gate, known from its retargets in the
timeline, is project CI by design and ignored; its changes are compared with that base. When
another pull request from the same branch was open within the window, a matching run could be
that one's, so it is reported to Slack as an attention item rather than closing the pull
request. A run counts as a bypass when both signals hold:

1. **The run did project CI work.** Either the default branch's version of the run's workflow
is not triggered by the run's event, which covers an injected workflow and a trigger the
pull request added to an existing one (for example `pull_request` on `ci-arm.yml`, which
has no gate), or it is a `ci-amd.yml` run that has run for more than 5 minutes, finished or
still running, and whose jobs (`GET .../actions/runs/<id>/jobs`) show more than the gate: a
started job that was not skipped, other than `Fork CI gate` and the jobs that run in every
gated run (`Check if the run is needed`, which has no steps for pull requests, and the
`CI passed` job, named `CI passed (project CI not run)` when project CI was skipped and
`CI passed` when a newer push cancelled the run before `Build info` started), a missing or
duplicated gate job, one of those jobs appearing twice (either `CI passed` name counts as
the same job), or one of them running for more than 5 minutes. The last two cases catch
project CI work done in jobs renamed to one of their names. A job counts as started
only when a runner picked it up (a non-zero `runner_id`, or steps): a queued job reports its
creation as `started_at`, and a run cancelled by a newer push marks its never-started jobs
`completed` and `cancelled`, so a gate waiting for a runner would otherwise look like project
CI work. A running run is judged as it goes, so one
tampered to outlast the 6-hour window is still caught. A gate-only run finishes in about a
minute, so jobs are listed only for the rare long runs. The triggers are read from the
monitor's own checkout of `main`, so no API call is needed; a pull request into a release
has no gate), and one of its jobs started and was not skipped (judged as for `ci-amd.yml`
below when it is that workflow, and counted as work when the jobs cannot be listed), so a
fork-only workflow the pull request adds and guards off in `apache/airflow` does not count;
or it is a `ci-amd.yml` run that has run for more than 5 minutes, finished or still running,
and whose jobs (`GET .../actions/runs/<id>/jobs`) show more than the gate: a started job that
was not skipped, other than `Fork CI gate` and the jobs that run in every gated run
(`Check if the run is needed`, which has no steps for pull requests, and the `CI passed` job,
named `CI passed (project CI not run)` when project CI was skipped and, when a newer push
cancelled the run before `Build info` started, `CI passed (run cancelled)` -- or `CI passed`
from a `ci-amd.yml` that does not use that name yet), a missing or duplicated gate job, one
of those jobs appearing twice (every `CI passed` name counts as the same job), or one of them
running for more than 5 minutes. The last two cases catch project CI work done in jobs
renamed to one of their names. A job counts as started only when a runner picked it up (a
non-zero `runner_id`, or steps): a queued job reports its creation as `started_at`, and a run
cancelled by a newer push marks its never-started jobs `completed` and `cancelled`, so a gate
waiting for a runner would otherwise look like project CI work. A running run is judged as it
goes, so one tampered to outlast the 6-hour window is still caught. A gate-only run finishes
in about a minute, so jobs are listed only for the rare long runs. The triggers are read from
the monitor's own checkout of `main`, so no API call is needed; a pull request into a release
branch is judged by `main`'s workflows.
2. **The run's own commit changes workflow files.** The compare of the base branch with that
commit (`GET /repos/apache/airflow/compare/<base>...<fork-owner>:<sha>`, first page only:
Expand Down Expand Up @@ -668,6 +684,46 @@ signal 1 no longer holds and the pull request re-enters the fork state machine.
ci` releases it like any other gated pull request; a maintainer who wants to keep an external
workflow change has to leave that label on.

### Gate protection

The gate decides where a pull request's CI runs, so a pull request whose CI runs in a fork must
not change it: the change would decide where its own CI runs, and a maintainer reviewing it later
would be reviewing a pull request that already escaped the gate. For every pull request the
tampering check applies to (`external` and `pilot`), the reconciler lists the changed files once
per head commit, counting a renamed file under both paths, and closes the pull request
with a comment explaining why when it changes any of:

- `dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py`, the reconciler modules
`fork_ci_reconcile.py`, `fork_ci_github.py` and `fork_ci_messages.py`, or
`.github/workflows/fork-ci-monitor.yml`;
- `.github/fork-ci-gate.toml`, other than adding, changing or removing the author's own
`[pilot]` entry, which is how contributors join and leave the pilot (the configuration is
parsed from the head commit and compared with its version at the pull request's merge base,
so comments and later changes on the base branch do not count);
- the `gate` job of `.github/workflows/ci-amd.yml`, the `needs` and `if` that make
`build-info` depend on it, the set of jobs that do not depend on the gate (directly or through
other jobs), or the workflow-level `on`, `defaults`, `env`, `permissions` and `concurrency` (the head and merge-base versions are parsed and those parts
compared, so other changes to the workflow are left to the green-trust check; a head version
that does not parse as a workflow counts as a change).

A changed-file listing at GitHub's 3000-file cap may leave out a gate file, so such a pull
request is closed as one whose gate change cannot be checked.

The state comment's marker records `"gate_checked": true` with the head commit only when this
check ran for it, so a marker written before the check existed, or while `use fork ci` exempted
the pull request, does not count, and a retarget since the state began makes it check again.
The changed files are listed for the pull request as it is when listed, so the head commit is
read again afterwards: if it moved, nothing is recorded and the next run checks the new one.

The state is `gate_modified`: the pull request is drafted, loses `ready for maintainer review`
and `needs project ci`, gets an `error` status linking its changed files, and the closure is
reported to Slack as an attention item. Gate tampering is checked first and wins. As for gate
tampering, only a reopen by someone other than the author counts: an open pull request whose
marker says `gate_modified` and that was not reopened since is closed again, and a maintainer's
reopen without restoring the gate closes it again on the next run. A maintainer who wants the
change adds `use project ci`, which releases the pull request before any of these checks. Each
of these files carries a comment saying so.

### State comment

One comment per pull request, created on first contact and edited afterwards. It starts with
Expand Down Expand Up @@ -1091,7 +1147,8 @@ as a failed check rather than a crash.
CI-defining files keeps the pull request a draft, decided once per commit) and the gate
tampering check: a tampered run on an earlier commit is caught after a clean push, each
signal alone does not close, short runs and gate-only jobs do not count, an in-progress
non-gate job, a removed, duplicated or long-running gate job and an injected workflow do,
non-gate job, a removed, duplicated or long-running gate job and an injected workflow with a
job that ran do, an added workflow whose jobs all skipped does not,
runs before the last `use project ci` removal or the last reopen are ignored, an
uncheckable 300-file change set is reported, `label_fork_ci` pull requests are exempt and
`pilot` pull requests are checked. Discovery labels a pilot participant's
Expand Down
5 changes: 5 additions & 0 deletions dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@

The gate job in ``ci-amd.yml`` runs this file with the runner's ``python3`` from a sparse
checkout, without installing breeze, so it must only import from the standard library.

This file is part of the fork CI gate. A pull request whose CI runs in the contributor's fork
must not change it, because the change would decide where its own CI runs: the fork CI
monitor closes such a pull request with a comment explaining why. See "Gate protection" in
the document above.
"""

from __future__ import annotations
Expand Down
8 changes: 7 additions & 1 deletion dev/breeze/src/airflow_breeze/utils/fork_ci_github.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,13 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""GitHub access for the fork CI reconciler (AIP-120); the only module of it that calls ``gh``."""
"""
GitHub access for the fork CI reconciler (AIP-120); the only module of it that calls ``gh``.

Part of the fork CI gate: a pull request whose CI runs in the contributor's fork and changes
this file is closed by the fork CI monitor. See "Gate protection" in
dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.
"""

from __future__ import annotations

Expand Down
Loading
Loading