Stop core DB test jobs timing out when migration tests run - #73231
Merged
Merged
Conversation
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-3-test
rjgoyln
force-pushed
the
fix/ci-migration-tests-job-budget
branch
2 times, most recently
from
September 16, 2026 08:26
e87470b to
53ba703
Compare
The migration tests run inside the core unit test job, and run_unit_tests.sh gives the unit tests whatever is left of the job budget once the steps before them have taken their share. On MySQL that regularly leaves around half an hour of the 65 minutes for tests that need close to it, so the job fails whenever the setup has a slow day. The 65 was sized as the tests' own timeout plus five minutes, with nothing in it for the steps that run before them.
rjgoyln
force-pushed
the
fix/ci-migration-tests-job-budget
branch
from
September 16, 2026 08:35
53ba703 to
1e2027c
Compare
rjgoyln
marked this pull request as ready for review
September 16, 2026 08:44
rjgoyln
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl and
potiuk
as code owners
September 16, 2026 08:44
potiuk
approved these changes
Sep 21, 2026
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 65d4e7d v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migration tests run as part of the core unit test job, and
run_unit_tests.shgives the unit tests whatever time remains after the preceding steps have completed. On MySQL, this can leave only around half an hour of the 65-minute job budget for tests that can take close to that long:The 65-minute budget was originally sized as the tests' own 60-minute timeout plus five minutes of overhead. #50973 set both values in the same commit, so the budget did not account for the migration tests and other steps that run before the unit tests.
The MySQL core callers now use an 80-minute job timeout. Postgres remains at 65 minutes because its migration overhead is small enough that there is no observed need for the additional budget. Sqlite also remains at 65 minutes: most migration rounds are skipped on that backend, and the migration step costs about a minute.
In the last nine canary runs on
main, the tightest MySQL shard that completed successfully finished only 1m11s before its timeout. In addition, bothAPI...CLIMySQL shards in run 34607414749 exceeded their available test timeout. Increasing the job timeout to 80 minutes provides a substantially larger buffer against these observed runtimes, while remaining above the longest core DB job in that window, which ran for 59 minutes.The providers callers also pass
run-migration-tests: "true", but the migration step is gated ontest-group == 'core'and is skipped for provider jobs. They therefore keep the existing 65-minute timeout.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines