Repository navigation
Conversation
Release committed activity attempt locks when workflows close. Release external signal locks after callbacks even when their source has closed. Track lock ownership across retry backoff and rejected command batches. Also release the attempt lock on cancellation before activity start. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
Three more paths in the Java test server take a time-skipping lock and do not release it when the workflow that owns the callback has closed or the request is cancelled. While the lock is held the server stops skipping time. Related to #3167, which fixes two other leaks of the same kind (workflow task and child notification locks); the two changes are independent and touch different code.
processScheduleActivityTasklocks for each activity attempt. If the workflow closes first,updaterejects the completion, failure, cancellation, heartbeat or timeout before its release, andtimeoutActivityalso skips the release afterNOT_FOUND. Attempt locks are now tracked inactivityLocks: normal responses and valid timeouts release them, and the finalizer inupdatereleases any that are outstanding when the workflow becomes terminal. Retry backoff holds no attempt lock, so a late response or a stale timeout cannot release someone else's.processSignalExternalWorkflowExecutionlocks until its success or failure callback. If the source workflow has closed,updaterejects both callbacks before the release. Both callbacks now release the committed lock in afinally(tracked inexternalSignalLocks, consumed once).processRequestCancelActivityTaskremoves anINITIATEDactivity without releasing its attempt lock. It now releases it once the cancellation commits.Why?
With a locked clock the server never fires later virtual timers. We hit this running a large Python test suite on one shared time-skipping server; the first stall after terminated workflows was fixed in #3167, and these paths have the same effect when workflows close with activities or signals in flight.
Checklist
Closes: no issue filed. It is a small bug fix in the test server and I found no existing issue for it.
How was this tested:
New
TestWorkflowMutableStateActivityAndSignalLocksTest(14 tests) on the real workflow store, state machines and timer. Each test schedules a future virtual timer, first checks that a real started activity in another workflow keeps time from advancing (guards against an extra unlock), then expects the timer to fire once that activity completes. Cases: queued and started activities at termination, completion, failure, cancellation and run timeout, six late-response variants, late heartbeat and activity timeouts, retry backoff and the next retry attempt, rejected command batches, external signal success and failure after source closure, and cancel before start. Against the unpatched source 10 of 14 fail ("A future timer must fire automatically"); the 4 controls (backoff termination, backoff cancellation, uncommitted activity and signal ownership) pass both ways. With the change all 14 pass.:temporal-test-server:test(112 tests, 0 failures) and the relatedtemporal-sdkselections (activity retry/timeout, heartbeat, external signal, child/parent lifetime, TestEnvironment, time skipping, terminate; 183 tests, 0 failures) pass offline.This change was developed with AI assistance (Claude) and reviewed and tested locally.