Skip to content

fix(delayed-job): 8h grab_for for PushClassGrades - #58

Merged
xcompass merged 1 commit into
ubcfrom
fix/push-class-grades-grab-for
Sep 28, 2026
Merged

xcompass merged 1 commit into
ubcfrom
fix/push-class-grades-grab-for

Conversation

@xcompass

Copy link
Copy Markdown
Member

Problem

DelayedJob::PushClassGrades inherits the 2-hour grab_for from DelayedJob::Worker. A whole-class push makes one AGS request per student per linked assignment, so on a very large course it can run longer than that. TheSchwartz then treats it as failed, and another worker starts a second copy while the first is still running.

GetClassMembership already overrides grab_for to 8 hours for exactly this reason (per its comment, a 5,100-student course took about 4 hours). UBC's largest course is 6,518 students with 56 linked assignments.

Change

PushClassGrades::grab_for returns $ENV{DELAYED_JOB_CLASS_GRADES_GRAB_FOR} // 8h, mirroring GetClassMembership and DELAYED_JOB_NRPS_GRAB_FOR.

  • Why not per course: TheSchwartz reads grab_for from the worker class when grabbing ($worker_class->grab_for, TheSchwartz.pm:420), so it can't vary per course or job.
  • Trade-off: if a worker genuinely dies mid-push, that class push waits up to 8 h before a retry instead of 2 h. The nightly class push is a catch-up/backstop, not the path new grades normally take. PushUserGrades (per-student, on submission) keeps its 2 h.
  • Context: worker pods on UBC prod now carry karpenter.sh/do-not-disrupt (chart 0.3.17), so node consolidation no longer kills workers mid-job.

Verification

Loaded in the WeBWorK image (webwork-verf worker pod, patched file ahead of the installed one):

PushClassGrades grab_for=28800s (8.0h)
PushUserGrades  grab_for=7200s (2.0h)
GetClassMembership grab_for=28800s
with env override: 14400s

perltidy 20240903 with .perltidyrc: no changes.

A whole-class grade push makes one AGS request per student per linked
assignment, so on a very large course it can outlive the 2 hour
DelayedJob::Worker default grab_for, after which another worker starts a
second copy of a push that is still running. GetClassMembership already
overrides grab_for to 8 hours for the same reason (a 5,100-student course
took ~4 hours); UBC's largest course is 6,518 students with 56 links.

TheSchwartz reads grab_for from the worker class, so this applies to all
class pushes. Overridable with DELAYED_JOB_CLASS_GRADES_GRAB_FOR, like
DELAYED_JOB_NRPS_GRAB_FOR. PushUserGrades keeps 2 hours.
@xcompass
xcompass merged commit c801585 into ubc Sep 28, 2026
11 checks passed
@xcompass
xcompass deleted the fix/push-class-grades-grab-for branch September 28, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant