Skip to content

Make Execution API secret reads async - #73414

Open
Dev-iL wants to merge 4 commits into
apache:mainfrom
Dev-iL:2609/async_secrets
Open

Dev-iL wants to merge 4 commits into
apache:mainfrom
Dev-iL:2609/async_secrets

Conversation

@Dev-iL

@Dev-iL Dev-iL commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

related: #67799, #72329

Execution API Variable and Connection reads previously depended on synchronous core resolvers, preventing the server from using native async backend I/O. This change adds async server-side resolution while preserving the existing secrets backend configuration, ordering, and behavior:

  • Adds optional aget_variable and aget_connection dispatch.
  • Uses native async SQLAlchemy reads for the metastore backend.
  • Runs unchanged synchronous backends, cache access, masking, and connection processing in bounded worker threads.
  • Preserves custom sync overrides, team scope, cache semantics, access-denial behavior, cancellation, and existing HTTP responses.
  • Migrates the Variable and Connection GET routes to await the new resolver.
  • Disposes the test async engine inside each TestClient event loop, preventing pooled MySQL connections from crossing loop lifetimes.
  • Documents async backend adoption, fallback precedence, resource ownership, cancellation limits, and rollback.

Native cloud-provider clients remain follow-up work, separate from the SDK Variable API work in #72329.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Codex following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Dev-iL added 4 commits October 1, 2026 17:19
Keep async test connections on the loop that opened them

The metastore async test imported `AsyncSession` by name before reconfiguring, so it kept using the previous engine and left pooled connections bound to a closed loop for later tests to trip over.
The Execution API variable route now resolves values through
resolve_variable instead of Variable.get, so the test's injected
RuntimeError never fired and the request fell through to a 404.
Inject the fault at the call the route actually makes so the test
exercises the exception path it was written for.
Slow secret decoding and masking must not stall other requests. Dag parsing
uses its own event loop, so its async connections must stay separate from
the API server's global pool.
@Dev-iL
Dev-iL force-pushed the 2609/async_secrets branch from d86dcce to b464fc0 Compare October 1, 2026 14:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant