Repository navigation
getJobs: refuse new dispatch to a worker that has already ended - #844
Merged
tmaeno merged 1 commit intoOct 30, 2026
Merged
Conversation
getJobs attaches a selected job to the requesting worker through updateWorkers, which keeps a recent terminal worker state. A job could therefore be dispatched to a worker already recorded as finished, failed, cancelled or missed, and stay attached to a parent that never runs it. On an OSG queue, 348 active jobs were found attached to cancelled workers after an HTCondor schedd outage. Lock the worker row in the dispatch transaction before any candidate job is changed, and refuse new work when the worker is terminal. Jobs already returned in the response are kept. Requests without a Harvester worker, and workers not yet registered, are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The worker lock can block dispatch past its request deadline, allowing a job to be marked sent after the budget expires.
1 open issue
What changed in this PR
This PR updates getJobs to prevent new jobs from being attached to Harvester workers already recorded as terminal, reducing the risk of jobs remaining assigned to workers that cannot run them.
Changes:
- Locks the worker row before modifying a candidate job.
- Rejects dispatch to finished, failed, cancelled, or missed workers while preserving jobs already collected.
- Adds an explicit type annotation to the returned job list.
| File | Description |
|---|---|
pandaserver/taskbuffer/db_proxy_mods/job_complex_module.py |
Adds the transactional worker-status guard and types the accumulated job list. |
Review effort: Balanced
| if harvester_id is not None and worker_id is not None: | ||
| self.cur.execute( | ||
| "SELECT status FROM ATLAS_PANDA.Harvester_Workers " | ||
| "WHERE harvesterID=:harvesterID AND workerID=:workerID FOR UPDATE" + comment, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
getJobsattaches a selected job to the requesting worker throughupdateWorkers, which keeps a recent terminal worker state. A job can therefore be dispatched to a worker already recorded as finished, failed, cancelled or missed, and stay attached to a parent that never runs it. After an HTCondor schedd outage on an OSG queue, 348 active jobs were found attached to cancelled workers.The worker row is now locked in the dispatch transaction (
SELECT ... FOR UPDATE) before any candidate job is changed, and new work is refused when the worker is terminal. Jobs already returned in the response are kept. Requests without a Harvester worker, and workers not yet registered, are unchanged.Running on the ePIC PanDA server at BNL since 2026-10-25.
🤖 Generated with Claude Code