Skip to content

Fix test server time-skipping locks for activities and external signals - #3168

Open
thejens wants to merge 1 commit into
temporalio:mainfrom
thejens:fix/test-server-clock-lock-leaks-2
Open

thejens wants to merge 1 commit into
temporalio:mainfrom
thejens:fix/test-server-clock-lock-leaks-2

Conversation

@thejens

@thejens thejens commented Oct 27, 2026

Copy link
Copy Markdown

What was changed

Three more paths in the Java test server take a time-skipping lock and do not release it when the workflow that owns the callback has closed or the request is cancelled. While the lock is held the server stops skipping time. Related to #3167, which fixes two other leaks of the same kind (workflow task and child notification locks); the two changes are independent and touch different code.

  • Pending activities. processScheduleActivityTask locks for each activity attempt. If the workflow closes first, update rejects the completion, failure, cancellation, heartbeat or timeout before its release, and timeoutActivity also skips the release after NOT_FOUND. Attempt locks are now tracked in activityLocks: normal responses and valid timeouts release them, and the finalizer in update releases any that are outstanding when the workflow becomes terminal. Retry backoff holds no attempt lock, so a late response or a stale timeout cannot release someone else's.
  • External signals. processSignalExternalWorkflowExecution locks until its success or failure callback. If the source workflow has closed, update rejects both callbacks before the release. Both callbacks now release the committed lock in a finally (tracked in externalSignalLocks, consumed once).
  • Cancel before the activity starts. processRequestCancelActivityTask removes an INITIATED activity without releasing its attempt lock. It now releases it once the cancellation commits.

Why?

With a locked clock the server never fires later virtual timers. We hit this running a large Python test suite on one shared time-skipping server; the first stall after terminated workflows was fixed in #3167, and these paths have the same effect when workflows close with activities or signals in flight.

Checklist

  1. Closes: no issue filed. It is a small bug fix in the test server and I found no existing issue for it.

  2. How was this tested:

New TestWorkflowMutableStateActivityAndSignalLocksTest (14 tests) on the real workflow store, state machines and timer. Each test schedules a future virtual timer, first checks that a real started activity in another workflow keeps time from advancing (guards against an extra unlock), then expects the timer to fire once that activity completes. Cases: queued and started activities at termination, completion, failure, cancellation and run timeout, six late-response variants, late heartbeat and activity timeouts, retry backoff and the next retry attempt, rejected command batches, external signal success and failure after source closure, and cancel before start. Against the unpatched source 10 of 14 fail ("A future timer must fire automatically"); the 4 controls (backoff termination, backoff cancellation, uncommitted activity and signal ownership) pass both ways. With the change all 14 pass. :temporal-test-server:test (112 tests, 0 failures) and the related temporal-sdk selections (activity retry/timeout, heartbeat, external signal, child/parent lifetime, TestEnvironment, time skipping, terminate; 183 tests, 0 failures) pass offline.

  1. Any docs updates needed? No. No public API or documented behavior changes. Breaking changes: none. Server PR: none (Java test server only).

This change was developed with AI assistance (Claude) and reviewed and tested locally.

Release committed activity attempt locks when workflows close. Release
external signal locks after callbacks even when their source has closed.
Track lock ownership across retry backoff and rejected command batches.
Also release the attempt lock on cancellation before activity start.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant