[chrisburr]

The Matcher sets a job to Matched with JobDB.setJobAttributes(..., update=True), which writes Status and LastUpdateTime in two separate statements. In between, a job that sat in Waiting for longer than MatchedTime looks exactly like a job stuck in Matched. If the StalledJobAgent's _kickStuckJobs selects in that window, it reschedules the job while the pilot goes on to run it.

The pilot then runs the payload successfully, but the job never leaves Checking: the state machine refuses Checking -> Running/Done, so only the minor and application statuses of the wrapper are recorded. When the On Hold: after rescheduling delay expires, JobScheduling's commit fails because the minor status changed underneath it, and the OptimizationMind never reloads it because its minor status is not an optimizer name. In LHCb this strands about one successful job per day.

Timeline of one such job (2026-10-19, UTC):

Time	Source	Event
05:59:51	JobScheduling	Job goes to Waiting, LastUpdateTime = 05:59:51
08:50:35.875	StalledJobAgent	Logs Rescheduling 2 jobs stuck in Matched status
08:50:36	Matcher	Matched / Assigned: Status written, LastUpdateTime still 05:59:51 (2h51m old > MatchedTime)
08:50:36	JobAgent	Matched / Job Received by Agent
08:50:36	(empty)	Received / Job Rescheduled from rescheduleJobs() called without a source
08:50:36	JobAgent	Installing Software, Submitting To CE: the pilot carries on regardless
08:50:38	JobPath, JobSanity, InputData	Job goes through the optimizers again
08:50:38	JobScheduling	Checking / JobScheduling, On Hold: after rescheduling 1
08:50:41 - 11:01:50	JobWrapper, payload	Payload runs and finishes successfully; status stays Checking
11:02:03	JobWrapper	Checking / Execution Complete / Job Finished Successfully, never updated again

This PR:

• Makes setJobsMajorStatus optionally refresh LastUpdateTime in the same statement as the status, and uses that from setJobAttribute(s) when update=True (without myDate), so the new status is never visible with the old time stamp.
• Makes _kickStuckJobs check each job again just before rescheduling it, which narrows the separate window between the selection and the rescheduling of a long list of jobs.
• Passes source="StalledJobAgent" so these reschedules can be attributed in the job logging.

BEGINRELEASENOTES

*WorkloadManagement
FIX: set the job Status and LastUpdateTime atomically so that the StalledJobAgent does not reschedule jobs that have just been matched
FIX: StalledJobAgent re-checks stuck jobs before rescheduling them and records itself as the source

ENDRELEASENOTES