Skip to content

Harden fork CI reconciler state transitions and closure rules - #76956

Merged
potiuk merged 1 commit into
apache:mainfrom
meel-ayush:fork-ci-reconciler-fixes
Nov 15, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
meel-ayush:fork-ci-reconciler-fixes

Conversation

@meel-ayush

@meel-ayush meel-ayush commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part of AIP-120:

The reconciler had a few edge cases where it closed pull requests that were actually still active:

  • It only checked issue comments on the main conversation tab, ignoring inline review comments and code suggestions on the diff.
  • Pushing a new failing commit kept the old 30-day inactivity timer running because it only checked if the state was still failed.
  • When GitHub returned headRepository: null on transient GraphQL glitches, the reconciler closed the PR immediately instead of checking if the repo was actually gone.
  • Manual workflow_dispatch re-runs with multiple attempts didn't always pick up the latest attempt.
  • Bot filtering was case-sensitive, so accounts like Airflow-Triage-Bot were treated as humans and kept dead PRs open.

This change:

  • Include PullRequestReview and PullRequestReviewComment in activity tracking so diff comments and reviews count as activity.
  • Reset the 30-day inactivity timer to day 0 when a new commit SHA is pushed.
  • Check discussion activity on setup_required and awaiting_push so PRs with ongoing conversations don't get auto-closed.
  • Verify head repository availability via the REST API before closing deleted forks, and retry on API errors.
  • Case-fold bot logins so bots like Airflow-Triage-Bot or codecov don't prevent inactive PRs from closing.
  • Sort workflow_dispatch check suites by (createdAt, runAttempt) so re-runs resolve to the latest attempt.
  • Let authors reopen gate_modified PRs once the gate changes are reverted.
  • Restore gate protection paths for all reconciler modules.

Tests

  • Unit tests added in test_fork_ci_reconcile.py for review comments, commit SHA resets, deleted fork REST checks, bot case-folding, and dispatch attempt ordering.
  • Reopen and status message checks verified in test_fork_ci_messages.py.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

@github-actions github-actions Bot added the fork ci required Gated by AIP-120: CI must pass in the contributor's fork. Managed by the fork CI reconciler. label Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Your CI change needs a maintainer. CI for this pull request runs in your fork, and the workflow runs it starts in apache/airflow are approved automatically while they only run the fork CI gate. This pull request changes files that define what CI runs:

  • dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py
  • dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py

Such a change can make project CI run on the project's runners, so its workflow runs in apache/airflow wait for a maintainer to review the change and approve them. CI in your fork is not affected, and its result stays visible on this pull request. Pull requests from contributors run their CI in the contributor's own fork. GitHub Actions is free and has no minute limit on public forks, while the CI capacity of the Apache Software Foundation is shared by every ASF project and is finite.


This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated.

@github-actions

Copy link
Copy Markdown
Contributor

Closing this pull request because it changes the fork CI gate: it changes dev/breeze/src/airflow_breeze/utils/fork_ci_messages.py.

The gate decides whether a pull request runs its CI in apache/airflow or in the contributor's fork. It is made of the gate job of .github/workflows/ci-amd.yml, dev/breeze/src/airflow_breeze/utils/fork_ci_gate.py, .github/fork-ci-gate.toml and the fork CI monitor, and a pull request whose CI runs in a fork must not change it, because the change would decide where its own CI runs. The one exception is adding, changing or removing your own [pilot] entry in .github/fork-ci-gate.toml.

If the change is wanted, a maintainer can reopen the pull request and add the use project ci label.


This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Project CI in use. A maintainer added the use project ci label, so CI for this pull request runs in apache/airflow. It starts on the next push, or when a maintainer closes and reopens this pull request.


This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated.

@github-actions github-actions Bot closed this Nov 12, 2026
@meel-ayush

Copy link
Copy Markdown
Contributor Author

Pushed commit cf0f0d8198 to the branch with the reconciler hardening updates:

  • Added PullRequestReview and PullRequestReviewComment to activity tracking so diff comments and code reviews count toward activity alongside regular discussion comments.
  • Reset the inactivity clock whenever a new commit SHA is pushed.
  • Added GitHub REST API verification for deleted forks to prevent false closures from transient GraphQL nulls.
  • Case-folded bot detection (_is_bot) and restored full gate protection paths for all reconciler modules.

Since GitHub freezes the pull request commit view while closed, the commit is live on the branch here:
meel-ayush@cf0f0d8

@meel-ayush

Copy link
Copy Markdown
Contributor Author

Updated Docs

Since the bot closed the PR, GitHub freezes the commits view until it gets reopened. The latest changes are live on the branch here: meel-ayush@ecb730f

Could a maintainer please reopen this PR for review?

@Dev-iL Dev-iL added the use project ci Committer override: run project CI for a PR that fork CI would otherwise gate. label Nov 13, 2026
@Dev-iL Dev-iL reopened this Nov 13, 2026
@github-actions github-actions Bot removed the fork ci required Gated by AIP-120: CI must pass in the contributor's fork. Managed by the fork CI reconciler. label Nov 13, 2026
@meel-ayush
meel-ayush force-pushed the fork-ci-reconciler-fixes branch from ecb730f to e366753 Compare November 13, 2026 20:30
@meel-ayush
meel-ayush marked this pull request as ready for review November 14, 2026 00:43

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Counting review comments as activity and picking the latest dispatch attempt are good fixes, but the discovery query no longer validates against GitHub, the shared timeline window can now lose the events the trust decisions depend on, and two closure-policy changes need a decision of their own. The branch also conflicts with main after #76835, #76836, #76837 and #76839 merged, so it needs a rebase.

The branch currently conflicts with main and needs a rebase before this can merge.

Blocking — the discovery query no longer validates (dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py:951)

itemTypes: [
  ...
  PULL_REQUEST_REVIEW,
  PULL_REQUEST_REVIEW_COMMENT   # <- not a PullRequestTimelineItemsItemType
]
...
... on PullRequestReviewComment { createdAt author { login } }

GitHub rejects this query. Running the same selection against apache/airflow returns Argument 'itemTypes' on Field 'timelineItems' has an invalid value (... PULL_REQUEST_REVIEW_COMMENT). Expected type '[PullRequestTimelineItemsItemType!]' and Fragment on PullRequestReviewComment can't be spread inside PullRequestTimelineItems. This selection is used by discovery before any pull request is reconciled, and GitHubClient.graphql raises on errors, so the monitor would stop reconciling every gated pull request. The unit tests mock the GraphQL response, so they pass. Inline review comments are reachable on the timeline as PULL_REQUEST_REVIEW_THREAD (or through each PullRequestReview's comments), not as PULL_REQUEST_REVIEW_COMMENT. Whatever the fix, please run the query once against the real API (gh api graphql -F query=@...) before pushing.

Flagged by both the primary and adversarial reviewers.

comments now share the last-100 timeline window with the control events (dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py:943)

timelineItems(
  last: 100,   # <- comments and reviews now compete with BaseRefChanged / Reopened / ConvertToDraft

The window is still last: 100 and unpaginated, but it now also holds every comment and review. On a pull request with a busy conversation, the BaseRefChangedEvent, ReopenedEvent and ConvertToDraftEvent the reconciler depends on drop out of it. Losing a retarget means last_base_change_at stays unset, so a green decision and the gate check made for the old base are reused for the new one, and an author can force that by retargeting and posting enough comments. Losing a reopen makes the reconciler finish a closure the author already undid; losing the convert-to-draft actor makes a bot draft look like the author's. Read the latest human comment or review with a separate query (or comments(last: 1) / reviews(last: 1) on the pull request), and keep the control-event window as it was.

Flagged by both the primary and adversarial reviewers.

two closure rules change under a hardening title (dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py:253)

CLOSABLE_STATES = frozenset({ForkState.SETUP_REQUIRED, ForkState.AWAITING_PUSH, ForkState.FAILED})

Two of these changes are policy rather than hardening: pull requests whose fork CI failed now close after 30 days, which the description does not mention, and a pull request whose head repository is deleted is closed instead of being reported, which the description words as if the reconciler already closed such pull requests. Both reverse rules the design document and ADIP-12 state (closure applies only while waiting for the author to set up fork CI or push; a deleted head repository is reported and otherwise left alone). Closing failed also covers an approved pull request whose re-run flaked, which the "failed does not draft" rule was written to protect. These need a maintainer decision, and the ADIP-12 page would change with them. Move the two closure changes into a pull request of their own with the reasoning in its description, so they can be discussed and decided on their own; keep the activity and attempt-ordering fixes here.

Smaller observations

  • See inline comments on dev/breeze/src/airflow_breeze/utils/fork_ci_reconcile.py:646.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. After you've
addressed the points above and pushed an update, an Apache Airflow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review:
Contributing guide.

Include PullRequestReview and discussion comments in activity tracking so diff comments and reviews count as human activity, reset the inactivity clock when a new commit SHA is pushed, and verify head repository availability via the REST API before reporting deleted forks as unknown. Case-fold bot logins so triage bots do not prevent inactive pull requests from closing, sort workflow_dispatch check suites by run attempt, separate timeline control events from comments/reviews in the discovery query, and allow authors to reopen gate_modified pull requests once gate changes are removed.
@meel-ayush
meel-ayush force-pushed the fork-ci-reconciler-fixes branch from 3b1c4f5 to 74fdef4 Compare November 15, 2026 11:48
@meel-ayush
meel-ayush requested a review from potiuk November 15, 2026 11:53

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all four points are addressed: the pass-two selection validates against the API (I ran it once against this pull request), comments and reviews are read outside the control-event window, both closure-policy changes are reverted, and nothing closes on the head-repository check any more.

A few optional clean-ups:

  • check_head_repo_unavailable now only chooses between two "unknown" messages, at the cost of a REST call every run for each pull request with a missing head repository. It could go, or at least catch the client's error type rather than Exception.
  • state_comment(fork: str | None = None) and fork=pr.head_repo or "" in the inactivity close are left over from the reverted closure; head_repo can no longer be None there.
  • The comment above reopened_since_closing still says only a maintainer's reopen lifts a gate closure. For gate_modified the author's reopen counts now (safe, since the gate is re-checked on the reopened head). Please also mention this in the PR description, as it changes the documented recovery path.

Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting

@potiuk
potiuk merged commit febd7b4 into apache:main Nov 15, 2026
154 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

potiuk pushed a commit that referenced this pull request Nov 15, 2026
…ules (#76956) (#77124)

Include PullRequestReview and discussion comments in activity tracking so diff comments and reviews count as human activity, reset the inactivity clock when a new commit SHA is pushed, and verify head repository availability via the REST API before reporting deleted forks as unknown. Case-fold bot logins so triage bots do not prevent inactive pull requests from closing, sort workflow_dispatch check suites by run attempt, separate timeline control events from comments/reviews in the discovery query, and allow authors to reopen gate_modified pull requests once gate changes are removed.
(cherry picked from commit febd7b4)

Co-authored-by: Ayush Meel <ayushmeel.dev@gmail.com>
@meel-ayush
meel-ayush deleted the fork-ci-reconciler-fixes branch November 15, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-4-test Backport to v3-4-test use project ci Committer override: run project CI for a PR that fork CI would otherwise gate.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants