Repository navigation
fix(dgw): reclaim stale auto-assigned upload tasks - #235
Merged
Merged
Conversation
Auto-assigned uploads that fail or are abandoned can stay in uploading without an episode forever. loadTaskSupplyCounts counts those rows against the plan target, so plans eventually reject new uploads even though the remaining recordings were never ingested. Mark auto-assigned tasks in task metadata, reclaim them after a hard-coded 2h lease, and run the reclaim before allocating a task and when the target is reached. Skip tasks whose logical upload still has fresh STS credentials so a genuinely long upload is not released. When target is blocked by unfinished uploads, return a retryable Aborted status so ego-portal keeps retrying instead of marking the job terminal.
shark0F0497
force-pushed
the
fix/dgw-reclaim-stale-auto-assigned-uploads
branch
from
November 17, 2026 11:19
0a9f88e to
7104add
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reclaim auto-assigned upload tasks that stay in
uploadingwithout an episode after a hard-coded 2-hour lease, and return a retryable status when a plan target is temporarily blocked by unfinished uploads.Motivation
An auto-assigned upload that fails or is abandoned currently leaves its task in
uploadingforever.loadTaskSupplyCountscounts those rows inlocal_reserved, so they permanently consume the plan's target budget. Once enough failures accumulate, the last recordings cannot receive a task and fail with "dc plan target has been reached".The concrete race is compounded by two behaviours:
CreateLogicalUploadmay be retried with a newauto_assign_taskafter the previous logical upload/session is lost, leaving the old task behind.FailedPrecondition: dc plan target has been reachedas a permanent failure, so it never retries after the task becomes stale.Changes
Modified Files
internal/services/dgwcompat/services.gotasks.metadatawithauto_assigned_upload=trueandcapture_id.reclaimStaleAutoAssignedTasks: after 2 hours, mark auto-assigneduploadingtasks without an episode asfailed, freeing the plan budget.EnsureNextTaskreportstarget reached; retry allocation if anything was reclaimed.uploadingtasks, returncodes.Abortedinstead ofcodes.FailedPrecondition. ego-portal treats this as retryable, so it keeps retrying until the lease expires or the uploads complete.FailedPreconditiononly when no unfinished uploads are holding the budget.internal/services/dgwcompat/services_test.goAdded Files
Deleted Files
Type of Change
Impact Analysis
Breaking Changes
None.
Backward Compatibility
Fully backward compatible. No schema migration or configuration change. The new metadata marker is only written on newly auto-assigned tasks; existing rows are left untouched. Existing production
uploadingtasks without the marker require a one-time metadata backfill before the reclaim path can see them.Testing
Test Environment
Local Go test suite (SQLite-backed service tests).
Test Cases
Manual Testing Steps
go test ./internal/services/dgwcompat ./internal/services -count=1 go build ./... gofmt -l internal/services/dgwcompat/services.go internal/services/dgwcompat/services_test.goTest Coverage
Screenshots / Recordings
Not applicable.
Performance Impact
An auto-assign now performs one extra indexed query on
tasks(dc_plan_id, status, deleted_at)and one metadata write. Reclaim may update multiple stale rows sequentially on the first request after the lease expires; the per-plan count is bounded by the plan target.Documentation
Related Issues
Additional Notes
Deployment does not modify existing rows. For already-stale tasks, the recommended post-deploy operation is to backfill the marker rather than directly changing
status:The next auto-assign then reclaims those rows with the live-session protection in place.
Reviewers
@reviewer1
Notes for Reviewers
Abortedis intentional and is classified as retryable by ego-portal.auto_assigned_uploadis stored in the existingtasks.metadataJSON, so no migration is required.