Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [The `.github` folder — Agent Instructions](#the-github-folder--agent-instructions)
- [Keep the CI-minutes estimates in sync with the workflows](#keep-the-ci-minutes-estimates-in-sync-with-the-workflows)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# The `.github` folder — Agent Instructions

## Keep the CI-minutes estimates in sync with the workflows

`breeze ci approve-workflow-runs` (run by `workflows/auto-approve-workflow-runs.yml`) decides whether
to approve an external contributor's CI runs from an estimate of the job-minutes they take. The
numbers live in `dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py` and are medians
measured on recent `Tests (AMD)` PR runs on public runners.

After you change the CI workflows in a way that adds, removes, splits or resizes jobs (new test
groups, a changed matrix, new composite workflows, a move between runner types), refresh the
estimates once the change has run on enough PRs, and in any case about once a month:

1. Run `uv run dev/calibrate_ci_minutes_estimates.py --since <date>`
from the repository root with an authenticated `gh` CLI. Pick a date that covers at least a
week of runs after the workflow change.
2. Copy the medians and fits it prints into the constants of `ci_minutes_estimate.py`, and update
the "Last calibrated" line next to them.
3. When you add a new job group to the workflows, add a matching group to both
`ci_minutes_estimate.py` (with the selective-check output that turns it on) and the
`JOB_GROUPS` patterns of the calibration script. Unmatched jobs fall into the base group.
4. Run `uv run --project dev/breeze pytest dev/breeze/tests/test_ci_minutes_estimate.py
dev/breeze/tests/test_workflow_run_approval.py -xvs`.

The estimates cover public runners only. Self-hosted (RunsOn) runners need their own set once there
is enough data from them.
10 changes: 10 additions & 0 deletions .github/fork-ci-gate.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ enabled_since = 2026-01-01T00:00:00Z
# Days without any fork CI activity after which a gated pull request is closed.
close_after_days = 7

# Report each job of a gated pull request's fork run as its own commit status. Set to false when
# the fork CI monitor exceeds GitHub's rate limits (see dev/AGENTS.md); the `Fork CI / Tests (AMD)`
# status is posted either way.
job_statuses = true

# Cancel the unfinished `pull_request` runs of pull requests that have already merged, of any
# author, so they stop using GitHub-hosted minutes and self-hosted runners for a result nobody
# reads. The fork CI monitor does it on its schedule.
cancel_merged_runs = true

# Organisations exempted under AIP-120 §2.2. Keys are display names used in comments,
# values are GitHub logins. Changes go through a reviewed pull request.
[allowlist]
Expand Down
68 changes: 68 additions & 0 deletions .github/workflows/auto-approve-workflow-runs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
---
# Approves the waiting workflow runs of external contributors' PRs to main when the change is safe to
# run without a maintainer's look, and labels the others "need workflow approval". A PR whose CI runs
# in the contributor's fork (AIP-120) is approved unless it changes CI definition files, and then gets
# a comment saying why its runs wait. The rules are in
# dev/breeze/src/airflow_breeze/utils/workflow_run_approval.py and documented in
# contributing-docs/05_pull_requests.rst.
#
# This workflow runs only trusted code from main. It reads the PR's metadata, changed-file list and
# git objects, and never checks out or runs the PR's code. Approving with GITHUB_TOKEN starts the
# approved runs right away, and workflow_run workflows that follow them still trigger.
name: Auto approve workflow runs
on: # yamllint disable-line rule:truthy
schedule:
- cron: '*/15 * * * *'
workflow_dispatch:
inputs:
report-only:
description: "Only report the decisions and estimates, without approving or labelling"
type: boolean
default: false
permissions:
# Approving workflow runs.
actions: write
# Adding and removing the "need workflow approval" label, and commenting on a fork CI PR whose
# CI change keeps its runs waiting.
pull-requests: write
contents: read
concurrency:
group: auto-approve-workflow-runs
cancel-in-progress: false
jobs:
approve-workflow-runs:
name: "Approve workflow runs of external contributors"
runs-on: ubuntu-latest
if: github.repository == 'apache/airflow'
timeout-minutes: 20
steps:
- name: "Checkout main"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: main
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: "Approve workflow runs"
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
REPORT_ONLY: ${{ inputs.report-only && 'true' || 'false' }}
run: breeze ci approve-workflow-runs
1 change: 0 additions & 1 deletion .github/workflows/ci-amd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,6 @@ jobs:
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
PR_AUTHOR_ASSOCIATION: ${{ github.event.pull_request.author_association }}
PR_CREATED_AT: ${{ github.event.pull_request.created_at }}
PR_EVENT_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
# A base branch with neither gate file predates the gate (the pull request adding it,
# or a release branch cut before it) and keeps running project CI. With only one of
# them present, the missing file fails the job instead of opening the gate.
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/fork-ci-monitor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ jobs:
name: "Reconcile pull requests gated to fork CI"
if: github.repository == 'apache/airflow'
permissions:
# Listing the runs and jobs of the gate tampering check, and deleting the runs that only
# gated a pull request to fork CI.
# Listing the runs and jobs of the gate tampering check, deleting the runs that only gated a
# pull request to fork CI, and cancelling the unfinished runs of merged pull requests.
actions: write
contents: read
# Labels, comments and closing of gated pull requests.
Expand Down
41 changes: 41 additions & 0 deletions contributing-docs/05_pull_requests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,47 @@ is the PMC private list — see the
``private@airflow.apache.org``. That document also covers the further steps the project may
take when the same behaviour persists across PRs.

CI workflow runs of external contributors
-----------------------------------------

GitHub does not start the CI workflows of a PR from a fork until a maintainer approves them,
for the contributors that the repository's approval setting covers. To keep you from waiting
for a maintainer on every push, a scheduled workflow checks the waiting runs every 15 minutes
and approves them automatically when all of these hold:

* The PR targets ``main``. PRs to release branches, such as ``v3-4-test``, always wait for a
maintainer.
* The PR does not change anything under ``.github/``, ``dev/breeze/`` or ``scripts/ci/``. Those
files define the CI workflows and choose the runners they use.
* The change does not need the full set of tests, does not change dependencies (``pyproject.toml``,
lock files, ``package.json``, ``go.mod``, Gradle build files) and does not add a new provider.
* The CI jobs your change selects are estimated to take at most 310 job-minutes, about a quarter of
a full CI run. Changes to documentation, to the UI, to the Helm chart, to the language SDKs, to
one area of Airflow core or to a small provider usually stay under it. Changes to large providers
such as Amazon or Google, to several providers, or to the Task SDK (which runs the tests of all
providers) usually do not.
* The PR changes at most 100 lines of source code. Tests (including end-to-end tests),
documentation, newsfragments, translations and lock files are not counted, so a fix with
thorough tests still qualifies. Larger changes are usually new features, and a maintainer looks
at them first.

If your PR's CI runs in your fork (it carries the ``fork ci required`` label), the CI run in
``apache/airflow`` only checks that, so none of the rules above apply. Its runs are approved
automatically unless the PR changes files that define CI: anything under ``.github/``,
``dev/breeze/``, ``scripts/ci/``, ``scripts/docker/`` or ``scripts/in_container/``, a ``Dockerfile``
or ``.pre-commit-config.yaml``. Then the runs wait for a maintainer and a comment on the PR explains
why. CI in your fork runs either way.

Each new push is checked again, because approval applies to one commit. Only runs started in the
last 24 hours are considered, so older waiting runs stay with maintainers. If your PR does not
qualify, it gets the ``need workflow approval`` label and a maintainer approves the runs after
looking at the change. That can take a while, so please do not ping maintainers about it; it is
not a judgement of your PR. The label is removed when a later commit is approved automatically.

Maintainers can see each decision, the CI jobs selective checks expect to run and their estimated
time in the summary of the ``Auto approve workflow runs`` workflow. The rules are described in
more detail in `CI workflows <../dev/breeze/doc/ci/05_workflows.md>`__.

Gen-AI Assisted contributions
-----------------------------

Expand Down
1 change: 1 addition & 0 deletions contributing-docs/25_maintainer_pr_triage.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ The triage workflow uses the following labels and states:
| Label / State | Meaning |
|---|---|
| `ready for maintainer review` | PR has passed Stage 1 and is queued for human review. Applied only after the skill has verified that no workflow runs are awaiting approval. |
| `need workflow approval` | The PR's CI workflow runs wait for a maintainer, because the `Auto approve workflow runs` workflow did not approve them: a PR to a branch other than `main`, a change to CI definition files, a full-tests or dependency change, a change whose CI jobs are estimated at more than 310 job-minutes, or a change of more than 100 lines of source code (see [Pull Requests](05_pull_requests.rst)). For a PR whose CI runs in the contributor's fork, only a change to CI definition files keeps its runs waiting, and the workflow comments on the PR to say so. The workflow also tries to add the label when processing the PR failed; its run then fails too. Approve the runs after looking at the diff. The workflow removes the label when a later commit is approved automatically. |
| `closed because of multiple quality violations` | PR was closed because the author has multiple open PRs with quality issues. |
| `suspicious changes detected` | PR (and all open PRs by the same author) was closed because the diff contained suspicious patterns (secret exfiltration, malicious CI modifications, etc.). |
| `closed because of open PR limit` | PR was closed in the one-time closure that introduced the [open PR limit](32_open_pull_request_limit.rst). Applied by `dev/close_prs_over_open_pr_limit.py`, never by the triage skill. If the author reopens the PR, triage it like any other. |
Expand Down
17 changes: 17 additions & 0 deletions dev/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

- [dev/ directory guidelines](#dev-directory-guidelines)
- [Scripts](#scripts)
- [Fork CI job statuses and GitHub's rate limits](#fork-ci-job-statuses-and-githubs-rate-limits)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand Down Expand Up @@ -42,3 +43,19 @@ uv run dev/my_script.py [args...]
```

Document `uv run` (not `python`) as the invocation method in READMEs and instructions.

## Fork CI job statuses and GitHub's rate limits

The fork CI monitor (`breeze ci fork-ci-reconcile`, run by `.github/workflows/fork-ci-monitor.yml`)
reports every job of a gated pull request's fork run as its own commit status. Those writes are the
largest share of what the monitor does with the repository's `GITHUB_TOKEN`, whose rate limits every
workflow of `apache/airflow` shares (see "Write budget" in
`dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md`).

If the budget is exceeded, turn the job statuses off: set `job_statuses = false` in
`.github/fork-ci-gate.toml`. Signs that it is exceeded are the monitor's Slack note that job
statuses "did not fit in the budget" persisting across runs, rate-limit retries ("GitHub rate limit,
retrying") or "still rate limited" errors in the monitor's run log or its Slack messages, or other
workflows of the repository failing on secondary rate limits. The aggregate `Fork CI / Tests (AMD)`
status and `CI passed` are posted either way, so turning job statuses off loses detail, not gating.
Lowering `MAX_JOB_STATUSES_PER_RUN` in `fork_ci_reconcile.py` is the softer alternative.
23 changes: 22 additions & 1 deletion dev/breeze/doc/08_ci_tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,8 @@ Pull requests from external contributors can run their CI in the contributor's f
gated pull requests ``fork ci required``, keeps them in draft until the fork run is green, maintains
one state comment and a ``Fork CI / Tests (AMD)`` commit status linking to the run in the fork, closes
pull requests without fork CI activity for a week, closes gated pull requests that made project CI run
by changing workflows, and writes everything it could not classify to a Slack message. With
by changing workflows, and writes everything it could not classify to a Slack message. It also cancels
the unfinished workflow runs of pull requests, of any author, that have already merged. With
``--dry-run`` it prints the changes instead of applying them. See
`Fork CI for external contributors <ci/08_fork_ci_for_external_contributors.md>`_ for the design.

Expand All @@ -199,6 +200,26 @@ These are all available flags of ``fork-ci-reconcile`` command:
:width: 100%
:alt: Breeze ci fork-ci-reconcile

Approving workflow runs of external contributors
------------------------------------------------

The ``auto-approve-workflow-runs`` workflow runs ``approve-workflow-runs`` every 15 minutes. It approves
the waiting workflow runs of external contributors' PRs to ``main`` whose CI jobs are estimated at no more
than 310 job-minutes, that change at most 100 lines of source code and that do not touch CI definition files, and labels the other PRs ``need workflow approval``. PRs whose CI runs in
the contributor's fork (AIP-120) only run the fork CI gate in ``apache/airflow``, so their runs are
approved whatever their size unless they change CI definition files; such a PR also gets a comment
explaining that its CI change needs a maintainer. For every PR it also
estimates the CI jobs selective checks will run and how long they take, and writes them to the job
summary. Use ``--report-only`` to see the decisions without approving runs or changing labels. The rules
are described in `CI workflows <ci/05_workflows.md>`_.

These are all available flags of ``approve-workflow-runs`` command:

.. image:: ./images/output_ci_approve-workflow-runs.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_approve-workflow-runs.svg
:width: 100%
:alt: Breeze ci approve-workflow-runs

-----

Next step: Follow the `Release management tasks <09_release_management_tasks.rst>`_ guide to learn how
Expand Down
Loading
Loading