Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1040,7 +1040,8 @@ repos:
^airflow-core/docs/.*/diagram_[^/]*\.py$|
^contributing-docs/images/diagram_[^/]*\.py$|
^docs/images/.*\.py$|
^airflow-ctl/docs/images/diagrams/.*\.py$
^airflow-ctl/docs/images/diagrams/.*\.py$|
^dev/breeze/doc/ci/images/diagram_[^/]*\.py$
pass_filenames: true
- id: prevent-deprecated-sqlalchemy-usage
name: Prevent deprecated sqlalchemy usage
Expand Down
104 changes: 7 additions & 97 deletions dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,26 +78,7 @@ Non-goals:

## Overview

```
contributor fork apache/airflow
┌──────────────────────────────────┐ ┌───────────────────────────────────────────┐
│ git push feature-branch │ │ pull_request event │
│ └─ ci-amd.yml (push trigger) │ │ └─ ci-amd.yml │
│ ├─ gate job: fork -> run │ │ └─ gate job: eligibility -> skip │
│ ├─ build-info: fork mode │ │ (build-info and everything after │
│ │ changed files from │ │ are skipped for gated PRs) │
│ │ compare API vs upstream │ │ │
│ └─ public runners only │ │ fork-ci-monitor.yml (every 5 minutes) │
└──────────────────────────────────┘ │ └─ breeze ci fork-ci-reconcile │
▲ │ ├─ discover new gated PRs -> label │
│ GraphQL: checkSuites │ ├─ per labelled PR: fork state -> │
│ REST: workflow state │ │ draft/undraft, labels, comment, │
└───────────────────────│ │ commit status with run link, │
│ │ close after 30 days or when the │
│ │ PR tampered with the gate │
│ └─ unknown failures -> Slack │
└───────────────────────────────────────────┘
```
![Fork CI architecture](images/diagram_fork_ci_architecture.svg)

Four pieces, in dependency order:

Expand All @@ -109,52 +90,18 @@ Four pieces, in dependency order:

Plus `breeze ci audit` for the contributor's local setup, and documentation.

The diagrams in this document are SVG files in [`images/`](images/), generated from their
`diagram_fork_ci_*.py` Graphviz sources by the `generate-airflow-diagrams` prek hook whenever a
source changes. The ADIP-12 wiki page embeds the same files. Edit the source, not the SVG.

### Pull request lifecycle

What a gated pull request shows, and who acts next, from the moment it is opened until it is merged
or closed. The [state machine](#state-machine) below is the reconciler's view of the same flow; this
diagram is the contributor's and the maintainer's view. Pull requests of committers, collaborators
and exempted organisations are not gated and follow the usual project CI flow.

```mermaid
%%{init: {'theme': 'neutral'}}%%
stateDiagram-v2
state "Waiting for the author: draft, fork CI not set up or no run for the head commit" as waiting
state "Fork CI running" as running
state "Fork CI failed: out of the review queue" as failed
state "Green: ready for maintainer review, CI passed" as ready
state "Green, author's draft: stays a draft, no review label until the author marks it ready" as author_draft
state "Green, base too old: ready for maintainer review, CI passed pending until a rebase" as rebase
state "Green, CI files changed: needs project ci" as needs_project_ci
state "Project CI: use project ci, runs on the next push or a close and reopen" as project_ci
state "Approved" as approved
state "Merged" as merged
state "Closed after 30 days without fork CI activity" as closed

[*] --> waiting: an external contributor opens a pull request
waiting --> running: the author enables the fork workflow and pushes
running --> waiting: the fork workflow is disabled
running --> failed: a job fails, times out or is cancelled, or the run cannot start
running --> ready: the run passes on a current base
running --> author_draft: the run passes, and the author made the pull request a draft
running --> rebase: the run passes, but the base is 200+ commits behind or changed the same files
running --> needs_project_ci: the run passes, and the pull request changes CI-defining files
author_draft --> ready: the author marks it ready for review
failed --> running: the author pushes a fix
rebase --> running: the author rebases and pushes
ready --> running: the author pushes
needs_project_ci --> running: the author pushes
ready --> approved: a maintainer approves
rebase --> approved: a maintainer approves, the merge waits for the rebase
ready --> project_ci: a maintainer adds use project ci
needs_project_ci --> project_ci: a maintainer reviews the CI change and adds use project ci
project_ci --> approved: project CI passes and a maintainer approves
approved --> running: the author pushes or rebases
approved --> merged: merged, or auto-merged once CI passed is green
waiting --> closed: 30 days without fork CI activity
closed --> waiting: the author reopens it
merged --> [*]
```
![Pull request lifecycle](images/diagram_fork_ci_lifecycle.svg)

- **Draft** is used only while the pull request waits for its author to set up fork CI or push. A
failed run leaves the draft state unchanged and removes `ready for maintainer review`, so a flaky
Expand Down Expand Up @@ -601,44 +548,7 @@ pull requests without a fork run for their head commit, not by the total.

### State machine

```mermaid
%%{init: {'theme': 'neutral'}}%%
stateDiagram-v2
[*] --> gated: discovery adds fork ci required
state gated {
[*] --> setup_required: workflow not enabled in the fork
[*] --> awaiting_push: workflow enabled, no run for the head commit
[*] --> running: fork run in progress
setup_required --> awaiting_push: author enables the workflow
awaiting_push --> running: author pushes
running --> failed: run fails, is cancelled, times out or cannot start
running --> green: run passes, no CI-defining change
running --> green_unverified: run passes, CI-defining change
failed --> running: author pushes
green --> running: author pushes
green_unverified --> running: author pushes
running --> setup_required: workflow disabled
awaiting_push --> setup_required: workflow disabled
state green {
[*] --> green_ready: not a draft, or the bot's draft
[*] --> green_author_draft: the author's draft
green_ready --> green_author_draft: author converts to draft
green_author_draft --> green_ready: author marks it ready
}
state green_unverified {
[*] --> unverified_ready: not a draft, or the bot's draft
[*] --> unverified_author_draft: the author's draft
unverified_ready --> unverified_author_draft: author converts to draft
unverified_author_draft --> unverified_ready: author marks it ready
}
}
gated --> closed: setup_required or awaiting_push for close_after_days (30 days)
gated --> released: maintainer adds use project ci
gated --> ungated: base branch without fork CI, e.g. v3-3-test
gated --> gate_modified: pull request changes the gate
gated --> gate_bypassed: project CI ran without use project ci (gate tampering)
closed --> gated: reopened
```
![Fork CI reconciler states](images/diagram_fork_ci_states.svg)

The draft state and the two review labels depend only on the state and on whether the pull
request is the author's draft: a draft whose last convert-to-draft event was not by
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0680ff8001f6f267b5b900af33ac16e3
80 changes: 80 additions & 0 deletions dev/breeze/doc/ci/images/diagram_fork_ci_architecture.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "rich>=13.6.0",
# "graphviz>=0.20.1",
# ]
# ///
"""Where fork CI runs: the contributor fork and apache/airflow. Used in dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md and on the ADIP-12 wiki page."""

from __future__ import annotations

from pathlib import Path

import graphviz
from rich.console import Console

MY_DIR = Path(__file__).parent
MY_FILENAME = Path(__file__).with_suffix("").name

console = Console(width=400, color_system="standard")

DOT = r"""
digraph arch {
graph [rankdir=LR, fontname="Helvetica", fontsize=11, nodesep=0.4, ranksep=0.7, compound=true, bgcolor="white"];
node [shape=box, style="rounded,filled", fillcolor="#f2f4f7", color="#5b6b80", fontname="Helvetica", fontsize=11];
edge [fontname="Helvetica", fontsize=9, color="#5b6b80", fontcolor="#33404d"];

subgraph cluster_fork {
label="contributor's fork (GitHub-hosted runners, contributor's token)"; style="rounded"; color="#3d8b57"; fontname="Helvetica";
push [label="git push feature-branch"];
ci [label="Tests (AMD) workflow\npush trigger"];
gatef [label="gate job:\nfork -> run"];
bi [label="build-info in fork mode\nchanged files vs upstream\n(compare API)"];
tests [label="tests and checks\non public runners"];
push -> ci -> gatef -> bi -> tests;
}

subgraph cluster_apache {
label="apache/airflow"; style="rounded"; color="#5b6b80"; fontname="Helvetica";
pr [label="pull_request event"];
gatea [label="gate job:\neligibility -> skip\n(build-info and all later jobs skipped)"];
mon [label="fork-ci-monitor.yml\n(scheduled, never runs PR code)"];
rec [label="breeze ci fork-ci-reconcile"];
out [label="per gated PR: draft state, labels,\nstate comment, commit statuses\n(Fork CI / Tests (AMD), CI passed),\nclose after 30 days or on gate tampering"];
slack [label="unknown failures -> Slack", fillcolor="#eceff3"];
pr -> gatea;
mon -> rec -> out;
rec -> slack [style=dashed];
}

tests -> rec [label="GraphQL: check suites\nREST: runs, jobs, workflow state\n(read by the monitor)", dir=back, style=dashed];
}
"""


def generate_diagram() -> None:
image_file = MY_DIR / f"{MY_FILENAME}.svg"
console.print(f"[bright_blue]Generating {image_file}")
graphviz.Source(DOT).render(outfile=image_file, format="svg", cleanup=True)
console.print(f"[green]Generated {image_file}")


if __name__ == "__main__":
generate_diagram()
Loading
Loading