You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Created more for visibility to others. CI now takes a long time to start across many PRs, both on push and in the merge queue. This seems to have started sometime between 2026-09-08 07:30 and 08:44 UTC.
The TLDR of my poking around is that some projects have many queued jobs (often from matrix'd runs) that are consuming significant numbers of runners and the DataFusion (and arrow-rs backlogs) are queued.
I'll ask the projects politely if there is some way to reduce the runner usage
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
I suspect it may be the fact that it is far easier than ever to make PRs via ai agents, and each PR often starts large amounts of CI
Note this only includes the github hosted runners (not the RunsOn runners):
My numbers show that airflow has used used 77.2 days of runner time in the last 17h
1000 runs of apache/airflow created between 2026-10-20T22:20:11Z and 2026-10-21T15:10:08Z (16.8 h window)
jobs loaded: 900 runs from cache, 98 from the API, 2 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
success 12235 57.1% 111218 77.2
skipped 7424 34.7% 0 0.0
cancelled 1234 5.8% 5745 4.0
in_progress 274 1.3% 0 0.0
failure 186 0.9% 2304 1.6
queued 65 0.3% 0 0.0
total 21418 119267 82.8
average concurrent jobs over the window: 118
By comparison DataFusion only used 11.7 days of runner time in the last 2 days
1000 runs of apache/arrow-rs created between 2026-10-19T10:20:44Z and 2026-10-21T15:06:27Z (52.8 h window)
jobs loaded: 0 runs from cache, 1000 from the API, 0 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
success 3492 69.7% 16846 11.7
skipped 913 18.2% 0 0.0
queued 327 6.5% 0 0.0
failure 141 2.8% 853 0.6
cancelled 125 2.5% 1946 1.4
in_progress 12 0.2% 0 0.0
total 5010 19645 13.6
average concurrent jobs over the window: 6
arrow-rs used 2.7 days over 17 hours
1000 runs of apache/datafusion created between 2026-10-20T22:07:51Z and 2026-10-21T15:13:03Z (17.1 h window)
warning: jobs for run 39541706327 failed: gh: Server Error (HTTP 502)
jobs loaded: 69 runs from cache, 930 from the API, 1 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
skipped 1757 48.8% 0 0.0
success 1410 39.2% 3943 2.7
queued 212 5.9% 0 0.0
cancelled 184 5.1% 2762 1.9
failure 24 0.7% 140 0.1
neutral 7 0.2% 0 0.0
in_progress 5 0.1% 0 0.0
total 3599 6845 4.8
average concurrent jobs over the window: 7
I did it for the "pull_request_target" replacement (doc-only) and few other actions and it seems that other ASF projects are picking it up - so it might be a good idea to use the opportunity when they need it.
Thanks for the suggestion; I haven't been doing the actual github action work (that is others with agents) so I don't think I have a lot to add as a shared action at this point (and just having my agent vomit a PR into that repo is probably less effectively than someone who knows what they are doing)
It's just a shared repo with actions - nothing special and it has a lot of harrness - it's very friendly for people who know how to use theor agents and would like to help other projects in ASF - with README/ AGENTS.md guiding the agents. And people like me reviewing it - so that shoudl be possible to add it by anyone.
"
I personally would like to have all the ASF maintainers to think in "How can I help infra and other projects" rather than "What I can demand from infra". That's my personal goal to increase a number of people who take care about more than their own garden.
31. I personally would like to have all the ASF maintainers to think in "How can I help infra and other projects" rather than "What I can demand from infra".
I agree
I was just trying to say I didn't think my adding a vibe coded github action to a shared repo is the best way to help other projects (their agents can probably do just as well, if not better)
@potiuk would something like #25591 be a good addition to the asf infra repo? I would be open to make it more generic and contribute towards the shared actions.
@potiuk would something like #25591 be a good addition to the asf infra repo? I would be open to make it more generic and contribute towards the shared actions.
If generalized and made ready to use by others with instructions, readme and examples how to use it- absolutely.
So while I am certainly a biased observer, my conclusion is that the work the Datafusion and Arrow communities have done over the last week have made our projects CI usage an order of magnitude lower than other large Apache projects (e.g. 1.6 runner days vs over 30 runner days)
Good ! hope others will follow - and to be 100% we are not done yet - we
are still testing the "Your PR - Your CI" case, this is not yet turned on
for external contributors, so I expect even more usage decrease from
Airflow.
feels like now the longest wait is happening due to gh hosted runners not runs-on ones... maybe we should switch more workers to runs-on? or maybe we could ask infra to raise a github ticket...
also, now we're running checks twice: one time in merge queue and another time on push to develop (which is the same thing). I think we should not run "push" action if pr was merged through the MQ. That should reduce our usage of runners...
also, now we're running checks twice: one time in merge queue and another time on push to develop (which is the same thing). I think we should not run "push" action if pr was merged through the MQ. That should reduce our usage of runners...
It is true, I also saw duplicated checks on MQ, so push trigger actually repeats the already triggered same task
Activity
blaginin commentedon Sep 8, 2026
I think that is related to other ASF projects too:
alamb commentedon Sep 8, 2026
I hit a bunch of timeouts when trying to merge PRs using the merge queue in arrow-rs-object-store:
alamb commentedon Sep 10, 2026
The root cause appears to be our DataFusion github runner usage is consuming all the github runners:
alamb commentedon Oct 20, 2026
This has been happening again a lot recently (taking hours to start running)
I have been exploring what apache projects are consuming jobs and Claude came up with this one
The current output is
It looks like a few PRs consumed a lot of the runner time (each launches 14 jobs via some matrix)
alamb commentedon Oct 20, 2026
The TLDR of my poking around is that some projects have many queued jobs (often from matrix'd runs) that are consuming significant numbers of runners and the DataFusion (and arrow-rs backlogs) are queued.
I'll ask the projects politely if there is some way to reduce the runner usage
mbutrovich commentedon Oct 20, 2026
Thank you!
alamb commentedon Oct 20, 2026
Current queue across projects
tballison commentedon Oct 21, 2026
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
This is not good.
Thank you for opening this ticket.
alamb commentedon Oct 21, 2026
I suspect it may be the fact that it is far easier than ever to make PRs via ai agents, and each PR often starts large amounts of CI
Also the new github stacks feature was released around Oct 6 (approximately the right timeline): https://github.blog/changelog/2026-10-06-stacked-pull-requests-generally-available/
And I found at least one example where a stack of PRs ended up running the full CI suite for each stacked PR each time the stack got rebased:
I don't have more data to back up that yet, but will keep searching
alamb commentedon Oct 21, 2026
Here is another snapshot
alamb commentedon Oct 21, 2026
I spent some time on some more analytics this morning on how many runner minutes are used by different projects
Here is the data + script I used: Archive.zip
Note this only includes the github hosted runners (not the RunsOn runners):
My numbers show that airflow has used used 77.2 days of runner time in the last 17h
By comparison DataFusion only used 11.7 days of runner time in the last 2 days
arrow-rs used 2.7 days over 17 hours
alamb commentedon Oct 21, 2026
There are also some great discussions in the ASF slack
https://the-asf.slack.com/archives/CBX4TSBQ8/p1792591324182189?thread_ts=1792520326.623569&cid=CBX4TSBQ8
That have come to the same conclusion
33 remaining items
potiuk commentedon Oct 24, 2026
I did it for the "pull_request_target" replacement (doc-only) and few other actions and it seems that other ASF projects are picking it up - so it might be a good idea to use the opportunity when they need it.
Happy to promote it among others.
alamb commentedon Oct 25, 2026
Thanks for the suggestion; I haven't been doing the actual github action work (that is others with agents) so I don't think I have a lot to add as a shared action at this point (and just having my agent vomit a PR into that repo is probably less effectively than someone who knows what they are doing)
potiuk commentedon Oct 25, 2026
It's just a shared repo with actions - nothing special and it has a lot of harrness - it's very friendly for people who know how to use theor agents and would like to help other projects in ASF - with README/ AGENTS.md guiding the agents. And people like me reviewing it - so that shoudl be possible to add it by anyone.
"
I personally would like to have all the ASF maintainers to think in "How can I help infra and other projects" rather than "What I can demand from infra". That's my personal goal to increase a number of people who take care about more than their own garden.
alamb commentedon Oct 25, 2026
I agree
I was just trying to say I didn't think my adding a vibe coded github action to a shared repo is the best way to help other projects (their agents can probably do just as well, if not better)
kumarUjjawal commentedon Oct 25, 2026
@potiuk would something like #25591 be a good addition to the asf infra repo? I would be open to make it more generic and contribute towards the shared actions.
potiuk commentedon Oct 25, 2026
If generalized and made ready to use by others with instructions, readme and examples how to use it- absolutely.
potiuk commentedon Oct 25, 2026
FYI: apache/infrastructure-website#323 - another optimization I added to Airflow and can be implemented by anyone.
Magpie will automatically use it as a proposed optimization when it is merged to infra website.
alamb commentedon Oct 26, 2026
Anecdotally I see PRs from arrow-rs and datafusion running with shorter queue times.
Compared to last week, Airflow usage is down to about 1/3 of where it was (35days vs 100days) -- thank you @potiuk
alamb commentedon Oct 26, 2026
So while I am certainly a biased observer, my conclusion is that the work the Datafusion and Arrow communities have done over the last week have made our projects CI usage an order of magnitude lower than other large Apache projects (e.g. 1.6 runner days vs over 30 runner days)
potiuk commentedon Oct 26, 2026
blaginin commentedon Nov 17, 2026
ubuntu-latestubuntu-slimubuntu-24.04macos-15feels like now the longest wait is happening due to gh hosted runners not runs-on ones... maybe we should switch more workers to runs-on? or maybe we could ask infra to raise a github ticket...
I'll talk to infra
blaginin commentedon Nov 17, 2026
also, now we're running checks twice: one time in merge queue and another time on push to develop (which is the same thing). I think we should not run "push" action if pr was merged through the MQ. That should reduce our usage of runners...
blaginin commentedon Nov 17, 2026
#27533 should make merging 3x faster
comphead commentedon Nov 17, 2026
It is true, I also saw duplicated checks on MQ, so
pushtrigger actually repeats the already triggered same taskci: use small RunsOn runners for merge queue checks (apache#27533)