Repository navigation
Conversation
DELETE now dispatches QueryJobId-shaped ids to QueryJobService.cancel with the caller's principal; Spark ids keep the existing path. A non-owner node forwards the request to the owner, as GET does. CancelAsyncQueryActionRequest now serializes queryId so the forwarded request keeps it, and both actions share the 404/403 translation that survives forwarding. Cancelling a finished job is a no-op that keeps its result until keep_alive expires, matching QueryJobService's idempotent cancel contract. Signed-off-by: Peng Huo <penghuo@amazon.com>
PPLQueryRunner.cancel only cancelled its future, so a cancelled job kept scanning until it finished. The runner now takes a cancel hook, and the transport action passes one that cancels the submit PPLQueryTask. Index scans keep polling that task after the RUNNING response unregisters it, so the query stops at its next cancellation check. The hook runs at most once and only when cancel wins against completion. Signed-off-by: Peng Huo <penghuo@amazon.com>
With plugins.calcite.fallback.allowed=true, the TaskCancelledException raised by a cancelled Calcite scan reached QueryService's fallback branch, which reran the query on V2. V2 index scans only checked thread interruption, so the rerun ran to completion after DELETE. The fallback decision now rejects any cancellation found in the cause chain (CancellationException, or TaskCancelledException matched by name because opensearch-core is compile-only in core); the walk tolerates cycles. OpenSearchIndexScan captures the request task in open() and stops before each search and in hasNext/next once it is cancelled, so a query already running on V2 also stops. Unsupported-query fallback is unchanged. Signed-off-by: Peng Huo <penghuo@amazon.com>
Align DELETE with the async lifecycle contract in opensearch-project#5765. QueryJobService gains delete(id, caller): it authorizes, removes the resolved instance, then cancels and returns the final snapshot, so exactly one concurrent delete succeeds and later GET or DELETE returns 404. cancel keeps its cancel-only, idempotent contract. REST answers PPL ids with 200 and {"status": ...}; Spark keeps its 204 response. Retention timers hold jobs weakly, so a deleted result is collectible before keep_alive. Forwarding to an owner that left the cluster now returns 404 instead of 400. Tests drive the runner by hand for the cancel, completion, late-completion, authorization, and removal races; the ITs cover only deterministic deletion of completed jobs. Signed-off-by: Peng Huo <penghuo@amazon.com>
The V2 engine is out of scope for async query. Restore OpenSearchIndexScan and its test to the base, and keep cancellation scoped to Calcite. The QueryService guard stays: a cancelled Calcite query must not start a new legacy execution through fallback. Signed-off-by: Peng Huo <penghuo@amazon.com>
penghuo
commented
Oct 2, 2026
| ### Features | ||
|
|
||
| * Add opt-in asynchronous PPL submission using `wait_for_completion_timeout` and `keep_alive`, with result retrieval through `GET /_plugins/_async_query/{id}`. | ||
| * Add opt-in asynchronous PPL submission using `wait_for_completion_timeout` and `keep_alive`, with result retrieval through `GET /_plugins/_async_query/{id}` and deletion through `DELETE /_plugins/_async_query/{id}`. Deleting a running query cancels it; deleting any job removes its retained state. |
Comment on lines
+309
to
+310
| // Index scans keep polling this task even after the RUNNING response unregisters it, so | ||
| // cancelling it stops the async query's execution. |
| * empty when {@code queryId} is not a {@link QueryJobId} or no job service is wired; the caller | ||
| * then cancels through the Spark path with {@link #cancelQuery}. | ||
| */ | ||
| public Optional<QueryJobStatus> deleteQueryJob(String queryId) { |
Owner
Author
There was a problem hiding this comment.
should resuse cancelQuery, right?
Comment on lines
+140
to
+143
| for (int attempt = 0; attempt < 50 && probe.get() != null; attempt++) { | ||
| System.gc(); | ||
| Thread.sleep(10); | ||
| } |
Owner
Author
There was a problem hiding this comment.
This make the UT running too long time!!!
Address review comments on the DELETE path. cancelQuery is again the single DELETE entry point: a QueryJobId is deleted through QueryJobService.delete and acknowledged with the job's final state name, while other ids keep the Spark cancel path and its id. The transport action formats the response by id shape, and the interface javadoc documents removal as well as cancellation. Drop the GC-timed retention test, which slowed the suite; the existing behavioral eviction tests remain. Remove a redundant comment on the cancel hook, and restore the release notes to the base. Signed-off-by: Peng Huo <penghuo@amazon.com>
The existing DELETE ITs only deleted completed jobs, and the async ITs ran on the V2 engine because PPLIntegTestCase disables Calcite. Enable Calcite in the async ITs and add a test-only plugin, installed only in the async test clusters, that captures the PPL query task, holds the query's first search, counts searches on the fixture index and its PITs, and reports SQL pool activity. AsyncPPLCancellationIT runs on the two-node asyncMultiNodeIT cluster with legacy fallback enabled. It holds a real Calcite scan, deletes it on the owner and through the non-owner node, and asserts the task was cancelled, the pools went idle within two searches against a six-batch baseline, the PITs were closed, and later GET and DELETE return 404. It also covers a failed job and concurrent deletes. With the cancel hook disabled the scan reads all 7 batches; with the fallback guard disabled the legacy restart reaches 9 searches; both fail the oracle. AsyncPPLSecurityIT runs on a new two-node secured cluster that integTestWithSecurity depends on. Bob, who has delete permission, gets 404 for an unknown id and 403 for Alice's running job through forwarding; the job stays uncancelled and running until Alice deletes it through the other node and the scan stops. The lifecycle IT adds expired and absent-owner 404 cases. Signed-off-by: Peng Huo <penghuo@amazon.com>
…oads Per Peng's request, drop the test-only plugin and rely on real PPL commands plus existing REST observability. - Remove integ-test/async-query-test-plugin and its settings.gradle include; strip the plugin install and dependsOn wiring from the two async test clusters. - AsyncQueryFixture bulk-loads a dedicated index (10,000 docs, max_result_window=10) and exposes streamstats, eventstats, and a scalar-overflow query that fails at projection time. Each query asserts an aggregate equal to DOCS so the planner cannot prune the window. - Running proof requires sql-worker.active >= 1, an open PIT, and search progress inside [10, DOCS/MRW/2] so a scan that already finished cannot pass vacuously. searchesAtDelete is captured immediately before the DELETE so the stop delta tracks only post-cancel work. - Stop oracle requires every SQL pool's active+queue to reach zero within 5s, a search delta <= 20, and tracked PITs released. Setup disables plugins.sql.complex_worker_pool.enabled so streamstats/eventstats stay on sql-worker; teardown restores it. - AsyncPPLCancellationIT parameterises on streamstats (local DELETE) and eventstats (forwarded DELETE); the overflow query drives FAILED DELETE via the peer node to keep forwarding coverage. The javadoc notes that the legacy engine does not support these commands, so the fallback guard remains covered only by CancelledQueryFallbackTest. - AsyncPPLSecurityIT uses the same workload on the two-node secured cluster and takes searchesAtDelete after the forbidden Bob calls and the Alice RUNNING check, so legitimate work during those steps is excluded from the post-cancel threshold. Signed-off-by: Peng Huo <penghuo@amazon.com>
… action
Per Peng's scope audit, consolidate the two integ-test helpers and shrink
the PR surface:
- AsyncQueryFixture.java removed; its workloads, running proof, and stop
oracle move into AsyncPPLTestHelpers. One localNodeId implementation.
The class is public, but visibility is narrowed to what callers need:
INDEX, STREAMSTATS_QUERY, localNodeId, createIndex, indexSearchCount,
awaitRunning, awaitPoolsIdle, assertStopped, twoNodeClients, and
RunningSnapshot (with pits public, pools/searches private) are public;
DOCS, MAX_RESULT_WINDOW, EVENTSTATS_QUERY, OVERFLOW_QUERY, and
BASELINE_MIN_SEARCHES are package-private; openPits, assertPitsClosed,
poolStats, poolsIdle, and the deadlines are private.
The existing REST helpers, endpoint constants, and pollUntilTerminal
stay package-private; the security IT reaches the API through
authenticated asUser(...) calls.
- TransportGetAsyncQueryResultAction.java is restored to the PR base: GET
keeps its inline QueryJobNotFoundException / QueryJobForbiddenException
translation. DELETE still goes through AsyncQueryOwnerRouting
.toTransportException, and the absent-owner 404 (ResourceNotFoundException)
is unchanged. GET behavior is unchanged.
Cancellation, retention, fallback, and security coverage are unchanged.
Signed-off-by: Peng Huo <penghuo@amazon.com>
Per Peng's scope audit, replace the standalone cancellation IT with its six cases hosted on the existing ITs; test coverage is unchanged. - AsyncPPLQueryLifecycleIT gains three cases on the single-node cluster: baselineStreamstats, baselineEventstats, deleteOnOwnerCancels. Each installs the large fixture on demand, pins an owner client via buildClient(Settings.EMPTY, host), and resets fallback/complex_pool in a nested finally that first drains the owner's SQL pools. The original 19 lifecycle cases keep their init untouched. - AsyncPPLMultiNodeRoutingIT gains deleteOnPeerForwards, concurrentDeletes, and deleteOfFailedJob on the two-node cluster. It now initialises nodeA and nodeB through the shared AsyncPPLTestHelpers.twoNodeClients and localNodeId; the inline probe helpers are gone. closeNodeClients awaits nodeA idle before closing both clients so a failed test does not leak a scan into the next case. - AsyncPPLCancellationIT.java is deleted. The include/exclude lines and the stale "test plugin" comment are dropped from integ-test/build.gradle. - Fixture settings are only touched by the moved cases; the previous cases are byte-identical in behaviour. - AsyncPPLTestHelpers.openPits retries 10 times with 50ms backoff on a transient 500 from /_search/point_in_time/_all, which the engine can briefly return while a PIT is being released; 403/404 are propagated unchanged. awaitRunning only probes the PIT list once sql-worker is active and the search counter is in the running window, so startup can't race PIT creation. Expected inventory: lifecycle 22, routing 8, security 1 = 31 distinct ITs. PR files 29 -> 28 (12 production, 15 test/build, 1 doc). No production, V2, root settings, or release-notes changes. Signed-off-by: Peng Huo <penghuo@amazon.com>
deleteOnPeerForwardsCancellationOfEventstatsAndStopsExecution now runs with default cluster settings (fallback disabled, complex worker pool enabled). eventstats compiles to a Window plan that ScriptDetector matches, so by default it dispatches to sql-complex-worker, and the forwarded DELETE still stops execution there. AsyncPPLTestHelpers.awaitRunning gains a package-private (owner, nodeId, before, expectedPool) overload. The public three-arg form keeps its original behaviour: it delegates with expectedPool="sql-worker" and the complex-worker-idle assertion stays in place for callers that pin to that pool. assertStopped drops the redundant complex-worker-active check that awaitPoolsIdle already covers; its Javadoc and the class-level Javadoc reflect the new coverage. No other callers change. Signed-off-by: Peng Huo <penghuo@amazon.com>
All async cancellation and security ITs now run on default cluster settings. Window workloads dispatch to sql-complex-worker by default, so the ITs assert cancellation stops execution there — no cluster toggles, no local overrides, no finally resets. Strict awaitPoolsIdle cleanup in each moved lifecycle case and in both multi-node After hooks is kept. - AsyncPPLTestHelpers: single awaitRunning that verifies sql-complex-worker is active; the expectedPool overload and sql-worker wrapper are gone. Class Javadoc updated. - AsyncPPLQueryLifecycleIT: three moved cases drop fallback/complex-pool constants, overrides, and resets; setClusterSetting and the SQLIntegTestCase import go with them. Finally still drains the owner. - AsyncPPLMultiNodeRoutingIT: forwarded eventstats, concurrent deletes, and overflow FAILED cases drop their overrides. setClusterSetting and the SQLIntegTestCase import go with them. closeNodeClients still drains nodeA before closing both clients. - AsyncPPLSecurityIT: init and tearDownFixture drop the literal fallback/complex-pool overrides and resets; drain happens in a try / finally so clients always close. All 31 cases (lifecycle 22, routing 8, security 1) still pass; stop, PIT, ownership, concurrency, and FAILED-deletion checks are unchanged. Signed-off-by: Peng Huo <penghuo@amazon.com>
Bump testClusters.integTestWithSecurity to two nodes so the async ownership IT can exercise owner forwarding on the shared secured cluster; delete the dedicated asyncSecurityMultiNodeIT testClusters block, configureSecurityPlugin call, task, and the integTestWithSecurity dependency. Drop the excludeTestsMatching entry so AsyncPPLSecurityIT runs alongside the other security suites. - integTestRemote still excludes AsyncPPLSecurityIT: the external remote cluster isn't guaranteed to have two nodes for forwarding. - Required-plugin and remote-cluster flags on integTestWithSecurity are unchanged. - The IT uses AsyncPPLTestHelpers.twoNodeClients(getClusterHosts(), ...) to pick two distinct node IDs; getClusterHosts() returns the shared secured cluster's HTTP hosts (not the remote), so owner and peer sit in the same local cluster. Verified on the shared cluster: AsyncPPLSecurityIT 1/1 and full integTestWithSecurity 9 suites / 95 tests / 0 failures / 0 skips. Signed-off-by: Peng Huo <penghuo@amazon.com>
Collapse async baselines and the FAILED fixture to a single workload so the fixture stops carrying two near-identical window pipelines. - AsyncPPLTestHelpers: delete EVENTSTATS_QUERY; OVERFLOW_QUERY now reuses STREAMSTATS_QUERY with a trailing `eval bad = ...` that overflows after the full scan completes. Class javadoc describes only streamstats. - AsyncPPLQueryLifecycleIT: drop async_baselineEventstats and the single-caller assertBaselineAggregate; inline it in the streamstats baseline. Lifecycle count becomes 21. - AsyncPPLMultiNodeRoutingIT: forwarded cancellation now runs STREAMSTATS_QUERY and the method is deleteOnPeerForwardsCancellationOfStreamstatsAndStopsExecution. - deleteOfFailedJobReturnsFailedAndRemovesIt guards against a parse or planner failure masquerading as FAILED: capture the search counter pre-submit, assert the delta >= BASELINE_MIN_SEARCHES by the time the job turns FAILED, and assert the error message contains "overflow". All default settings, stop / PIT / ownership / concurrency assertions, and the other 29 cases retained. Totals: lifecycle 21, routing 8, security 1 — 30 cases. Signed-off-by: Peng Huo <penghuo@amazon.com>
Address the owner audit: restore files touched only incidentally, drop IT redundancy, and sharpen oracles. Production (narrow to DELETE): - AsyncQueryOwnerRouting and its test restored to the PR base: GET keeps its original 400 for an absent owner; the shared toTransportException helper is gone. - TransportCancelAsyncQueryRequestAction inlines QueryJobNotFoundException/QueryJobForbiddenException translation (same shape as the base GET action) and wraps the non-owner forwarding listener so AsyncQueryNotFoundException from forwardToOwner becomes ResourceNotFoundException (404) only on the DELETE forwarding path. Spark DELETE and GET behavior are unchanged. - QueryJobService.delete and RetentionPolicy comments corrected: delete removes before cancelling, eviction timer must not retain a deleted job until TTL. Integ-test fixture: - AsyncPPLTestHelpers simplified. awaitRunning returns List<String> (PIT ids); awaitPoolsIdle returns void; RunningSnapshot nested class and BASELINE_MIN_SEARCHES constant removed. DOCS and MAX_RESULT_WINDOW are now private, with RUNNING_MAX_PROGRESS derived directly from them. Fixture index/doc drops the unused g keyword field. - AsyncPPLQueryLifecycleIT drops the streamstats baseline (coverage implicit in the running-cancel case), merges async_deleteExpiredJob into the base async_keepAliveEvictsAfterTtl (asserts GET 404 then DELETE 404), and renames async_queryIdOfAbsentOwner to async_deleteWithAbsentOwner for DELETE-only coverage. - AsyncPPLMultiNodeRoutingIT drops the FAILED case's >= baseline search delta and its trailing awaitPoolsIdle (@after drains); restores the base GET formatter schema comment. - AsyncPPLSecurityIT drops Bob's GET 404 and GET 403 assertions (GET path unchanged); keeps Bob DELETE unknown-id 404 (permission control), Bob DELETE 403, Alice RUNNING GET, Alice DELETE CANCELLED, stop/PIT cleanup, and GET 404. - CancelAsyncQueryActionResponseTest deleted: the response class is byte-identical to the base and is covered end-to-end by the forwarded IT; the request serde test stays. Totals: lifecycle 19 (= 15 base + 4 DELETE), routing 8 (= 3 base + 5 DELETE), security 1. 101 focused unit tests still pass. Signed-off-by: Peng Huo <penghuo@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to opensearch-project#5832 (25 files total: 11 production, 13 test/build, 1 doc). Adds DELETE support for async PPL jobs through the existing
DELETE /_plugins/_async_query/{id}endpoint, following the lifecycle contract in opensearch-project#5765. The base branch is the opensearch-project#5832 head, so this diff contains only the DELETE work.Scope: async query runs on the Calcite engine. The V2 engine is out of scope, and this PR makes no V2 engine changes. GET and Spark DELETE behavior is unchanged.
Contract
200 {"status":"CANCELLED"}; Calcite execution stops404200 {"status":"SUCCEEDED"}or{"status":"FAILED"}404404404(unchanged for GET, which keeps the baseAsyncQueryNotFoundException→400)403; job is not removed or cancelledSpark DELETE is unchanged:
204withDeleted async query with id: ....Production changes (11 files)
QueryJobService.delete(id, caller)is new: look up, authorize, remove the resolved instance, then cancel and return the final snapshot. Removal races against concurrent deletes and retention eviction, so exactly one caller succeeds; afterwardsgetanddeletethrowQueryJobNotFoundException.cancelkeeps its cancel-only, idempotent contract.AsyncQueryExecutorServiceImpl.cancelQueryis the single DELETE entry point:QueryJobIdids go todeleteand return the final state name, other ids keep the Spark path.TransportCancelAsyncQueryRequestActionforwards to the owner node and inlinesQueryJobNotFoundException/QueryJobForbiddenExceptioncatches (same shape as base GET). The forwardingActionListener.wrapconvertsAsyncQueryNotFoundException(owner-not-in-cluster-state, DELETE-only) toResourceNotFoundException(404). REST returns200for PPL ids and204for Spark.CancelAsyncQueryActionRequestserializesqueryId, so forwarding keeps it.AsyncQueryOwnerRoutingand its unit test are byte-identical to the PR base; no shared translator helper.RetentionPolicyeviction timers hold the job through aWeakReferenceso a deleted result is freed beforekeep_alive.PPLQueryRunner.cancel()runs a cancel hook at most once, only if cancel beats completion.TransportPPLQueryActionsupplies a hook that cancels the submitPPLQueryTask, which the Calcite index scan polls.QueryServicerefuses fallback when the cause chain contains a cancellation (CancellationExceptionorTaskCancelledExceptionmatched by name). Walk is cycle-safe. Unsupported-query fallback is unchanged.Integration-test fixture (plugin-free)
AsyncPPLTestHelperscreatesasync_cancel_fixturewith 10 000 docs andindex.max_result_window=10. The streamstats query retains its computed window output to keep the workload active during cancellation; the tests verify running execution and that DELETE stops it. The FAILED workload reuses the streamstats pipeline with a trailingeval bad = 9223372036854775807 + total | fields badso the overflow fires at projection time after the scan drains. All async ITs run with default cluster settings;streamstatscompiles to a Calcite Window, so by default it dispatches tosql-complex-workerwhere cancellation is observed.Observability uses only existing REST APIs:
GET /_plugins/_async_query/{id}for the job status;GET /_nodes/<owner>/stats/thread_poolforactive + queueon sql-worker, sql-complex-worker, and sql_background_io;GET /<index>/_statsfor the per-indexsearch.query_totalcounter;GET /_search/point_in_time/_allfor PIT closure.awaitRunningreturns the list of PIT ids seen while running once sql-complex-worker is active and search progress sits in[10, DOCS/MRW/2].assertStoppedwaits for every SQL pool to drain, asserts the search delta from the pre-DELETE snapshot ≤ 20, and verifies tracked PITs are released.searchesAtDeleteis captured immediately before DELETE, after any legitimate preceding work.Tests on
d4e6f26ab7AsyncPPLQueryLifecycleIT(19 cases = 15 original + 4 DELETE, single-node integTest):async_deleteCompletedJobReturnsStatusAndRemovesItasync_deleteUnknownQueryIdReturns404async_deleteOnOwnerCancelsStreamstatsAndStopsExecution(owner pinned viabuildClient(Settings.EMPTY, host))async_deleteWithAbsentOwnerReturns404async_keepAliveEvictsAfterTtl: after GET returns 404, a follow-up DELETE must also return 404.AsyncPPLMultiNodeRoutingIT(8 cases = 3 original + 5 DELETE, two-nodeasyncMultiNodeIT):delete_forwardsFromNonOwnerNodeToOwnerdeleteUnknownPplIdFromNonOwner_returns404deleteOnPeerForwardsCancellationOfStreamstatsAndStopsExecutionconcurrentDeletesCancelOnceAndRemoveOncedeleteOfFailedJobReturnsFailedAndRemovesIt(overflow error asserted; DELETE forwarded via peer)AsyncPPLSecurityIT(shared two-node securedintegTestWithSecurity, runs with the other security suites): Bob DELETE unknown-id 404 (permission control), Bob DELETE 403 (ownership), Alice GET RUNNING, Alice DELETE CANCELLED via peer, stop/PIT cleanup, Alice GET 404.Negative controls (historical, captured on
b25fa1289b; AsyncPPLCancellationIT existed then and used eventstats baselines in addition to streamstats, both since folded into the two existing ITs — streamstats only)asyncMultiNodeIT :AsyncPPLCancellationITcancelExecution = () -> {})asyncMultiNodeIT :AsyncPPLCancellationITQueryServiceguard disabled:opensearch:test *CancelledQueryFallbackTestEvidence files in
/tmp/pr18-negative-controls/.Final verification
./gradlew spotlessCheck :integ-test:compileTestJavad4e6f26ab7core,opensearchjob + fallback,pplrunner,async-query-corerouting,async-querycancel/get/serde/routing/REST)d4e6f26ab7:integ-test:integTest --tests '*AsyncPPLQueryLifecycleIT'd4e6f26ab7:integ-test:asyncMultiNodeITd4e6f26ab7:integ-test:integTestWithSecurity --tests '*AsyncPPLSecurityIT'd4e6f26ab7:integ-test:integTestWithSecurity(full secured suite, two-node cluster)e96c3bb9a6(historical)Logs:
/tmp/r14-unit.log,/tmp/r14-life.log,/tmp/r14-multi.log,/tmp/r14-sec.log,/tmp/r14-summary.log.Not rerun on this head: the broader focused unit tests outside the 12 listed above. Not run: the doctest.
FGACIndexScanningITuses JUnit Jupiter parameterized cases that the security task's runner does not enumerate individually; this pre-existing discovery limitation is unchanged by this PR, so its parameterized cases are not part of the 94 executed count.Docs
endpoint.mddocuments the DELETE contract.