Skip to content

Report fork CI's tested base and withhold CI passed on an old base - #76837

Merged
potiuk merged 1 commit into
mainfrom
fork-ci-test-merge-with-target
Nov 15, 2026
Merged

potiuk merged 1 commit into
mainfrom
fork-ci-test-merge-with-target

Conversation

@potiuk

@potiuk potiuk commented Nov 10, 2026

Copy link
Copy Markdown
Member

Why

One of the points that cancelled the AIP-120 vote: fork CI tests the tip of the contributor's
branch, while project CI tests the PR merged into the target branch, so a gated PR could go
green on a stale base and merge on that result.

What

  • When a fork run turns green, the fork CI monitor compares the head commit with the base
    branch: the merge base the run tested, how many commits the base branch has since, and which
    of the PR's files the base branch changed since.
  • CI passed is reported only when the base is fewer than 200 commits behind and changed none
    of the PR's files. Otherwise it is posted as pending with "Rebase on and push", and the
    state comment lists the overlapping files and the rebase commands.
  • Every green state comment and the CI passed description name the base commit the run was
    tested on.
  • Testing the merge commit in the fork itself was considered and rejected (every checkout of the
    fork run would change, there is no merge ref before the PR exists, GitHub recomputes it
    asynchronously, and the monitor would have to trust the run's own report); the reasons are in
    the design doc's "Known limitations".
  • The check runs once per head commit, like project CI's pull_request run, and no paths are
    exempt from the overlap check; both can be revisited if rebase churn becomes a problem.

related: https://lists.apache.org/thread/5603zwgpj7pgj6j979hbgpr72xsjbbws


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

@potiuk
potiuk force-pushed the fork-ci-test-merge-with-target branch from 302d22d to 28def61 Compare November 10, 2026 22:46
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The re-run of the jobs stopped by EC2 Spot interruptions finished: attempt 2 passed.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tested-base check is sound and well covered by tests. Two small things to look at when you rebase. The rebase is needed anyway: the PR conflicts with main after #76672 and #76681.

1. The large-PR fallback in compare_with_base does not check that the head is still the same (minor, dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py, compare_with_base)

head_files = head_side.get("files") or []
if len(head_files) >= COMPARE_FILES_CAP:
    head_files = client.rest_get(f"repos/{repo}/pulls/{pr.number}/files?per_page=100", paginate=True)

The compare is pinned to pr.head_sha, but pulls/<n>/files lists the PR's current head. Both existing callers of that endpoint re-read the head after listing and give no answer if it moved: pull_request_changes_ci_files returns None, and GateChangeFinder.find raises PullRequestChangedError. This one does not. If a push lands between the two reads, the overlap for pr.head_sha is computed from another commit's files. That can produce a BaseCheck marked current, which then sits in the marker for that SHA. The window is narrow (300+ files, a push mid-run), so the fix is to follow the same pattern: re-read pulls/<n> after the listing and return None (or raise PullRequestChangedError) when head.sha != pr.head_sha. A test for this would add one parametrize case.

2. green_unverified gets a "Rebase needed" paragraph that contradicts its state text (minor, dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py, state_comment / tested_base_text)

state_comment appends tested_base_text for both green and green_unverified. For a non-current base, the paragraph ends with "The CI passed check is not reported until fork CI is green for a branch rebased on <base>". For green_unverified, fork CI never reports CI passed, rebased or not. The state text above it says a maintainer will run project CI, and project CI then tests the merged result anyway. So the contributor is asked for a rebase that does not unblock anything. The overlap case in test_state_comment_says_which_base_was_tested uses green_unverified and asserts this text, so the behaviour looks deliberate, but the wording is wrong for that state. Options: keep only the first sentence (the tested base) for green_unverified, or skip compare_base() for it altogether. The second option also saves the compare calls for a state where the result is never used for CI passed.

Smaller observation, no change needed: when the head commit already has a pending CI passed, for example from the "no longer trusted" revoke after a compare failure on a marker without base, _set_ci_passed does not rewrite the description to "Rebase on … and push". The state comment still says to rebase, so this only matters if you want the check's description to say why.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The findings
below are observations, not blockers; an Apache Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review:
Contributing guide.

Drafted-by: Claude Code (Opus 5.5); reviewed by shahar1 before posting

Fork CI tests the head commit of a gated pull request, while project CI
tests it merged into the target branch, so a pull request could go green
on a stale base and merge on that result.

When a head commit turns green, the fork CI monitor now compares it with
the base branch: the merge base it was tested on, how many commits the
base branch has since, and which of the pull request's files the base
branch changed since. `CI passed` is reported only when the base is
fewer than 200 commits behind and changed none of those files;
otherwise it is posted as pending with a rebase hint, and the state
comment lists the overlapping files. Every green state comment and the
`CI passed` description name the base commit the run tested.

Testing the merge commit in the fork itself was rejected: it would have
to change every checkout of the fork run, there is no merge ref before
the pull request exists, GitHub recomputes it asynchronously, and the
monitor would have to trust the run's own report of what it tested.
The design document lists the reasons.

Generated-by: Claude Opus 5
@potiuk
potiuk force-pushed the fork-ci-test-merge-with-target branch from 28def61 to b52864f Compare November 15, 2026 01:25
@potiuk
potiuk merged commit eb39f15 into main Nov 15, 2026
177 checks passed
@potiuk
potiuk deleted the fork-ci-test-merge-with-target branch November 15, 2026 04:00
potiuk added a commit that referenced this pull request Nov 15, 2026
* [v3-4-test] Sync fork CI changes from main

Several fork CI changes merged to main since the last sync never reached
v3-4-test, and the fork CI monitor reads the gate configuration of a pull
request's base branch, so pull requests to v3-4-test were still closed
after 7 days and missed ten pilot participants. The fork CI modules, their
tests, the design document and its diagrams, the monitor workflow, the
gate configuration checks and the contributor documentation now match
main, taking in #75814, #75696, #76272, #76316, #76327, #76510, #76538,
#76677, #76681, #76835, #76836, #76837, #76839, #76841, #76866, #76267 and
#77100.

The pilot participants new to this branch join at 2026-11-16 00:00 UTC
rather than at their main dates, as the gate configuration check requires
a future instant for an entry that is new on the base branch.

Generated-by: Claude Opus 5

* [v3-4-test] Say the fork CI monitor runs every 5 minutes in the CI tasks doc

The sentence changed with #76836 on main and was missed by the sync.

Generated-by: Claude Opus 5

* [v3-4-test] Bring the fork CI GitHub client tests in line with main

The tests for the smaller job listing pages (#76327) were missed by the
sync; the client code they cover is already in it.

Generated-by: Claude Opus 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants