Skip to content

fix(dgw): reclaim stale auto-assigned upload tasks - #235

Merged
shark0F0497 merged 1 commit into
main-v2from
fix/dgw-reclaim-stale-auto-assigned-uploads
Nov 18, 2026
Merged

shark0F0497 merged 1 commit into
main-v2from
fix/dgw-reclaim-stale-auto-assigned-uploads

Conversation

@shark0F0497

Copy link
Copy Markdown
Collaborator

Summary

Reclaim auto-assigned upload tasks that stay in uploading without an episode after a hard-coded 2-hour lease, and return a retryable status when a plan target is temporarily blocked by unfinished uploads.

Motivation

An auto-assigned upload that fails or is abandoned currently leaves its task in uploading forever. loadTaskSupplyCounts counts those rows in local_reserved, so they permanently consume the plan's target budget. Once enough failures accumulate, the last recordings cannot receive a task and fail with "dc plan target has been reached".

The concrete race is compounded by two behaviours:

  • CreateLogicalUpload may be retried with a new auto_assign_task after the previous logical upload/session is lost, leaving the old task behind.
  • ego-portal treats FailedPrecondition: dc plan target has been reached as a permanent failure, so it never retries after the task becomes stale.

Changes

Modified Files

  • internal/services/dgwcompat/services.go
    • Mark auto-assigned tasks in tasks.metadata with auto_assigned_upload=true and capture_id.
    • Add reclaimStaleAutoAssignedTasks: after 2 hours, mark auto-assigned uploading tasks without an episode as failed, freeing the plan budget.
    • Skip reclaiming a task whose logical upload still has fresh STS credentials (with a 10-minute expiry grace), so a genuinely long upload is not released.
    • Run reclaim before allocating a task and again when EnsureNextTask reports target reached; retry allocation if anything was reclaimed.
    • When target is blocked by unfinished uploading tasks, return codes.Aborted instead of codes.FailedPrecondition. ego-portal treats this as retryable, so it keeps retrying until the lease expires or the uploads complete.
    • Return FailedPrecondition only when no unfinished uploads are holding the budget.
  • internal/services/dgwcompat/services_test.go
    • Test stale task reclamation and target recovery.
    • Test that recent tasks are not reclaimed.
    • Test that a live upload session with valid STS is not reclaimed.
    • Test that a real target-reached without uploading tasks remains permanent.

Added Files

  • None.

Deleted Files

  • None.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Impact Analysis

Breaking Changes

None.

Backward Compatibility

Fully backward compatible. No schema migration or configuration change. The new metadata marker is only written on newly auto-assigned tasks; existing rows are left untouched. Existing production uploading tasks without the marker require a one-time metadata backfill before the reclaim path can see them.

Testing

Test Environment

Local Go test suite (SQLite-backed service tests).

Test Cases

  • Unit tests pass locally
  • Integration tests pass locally
  • E2E tests pass (if applicable)
  • Manual testing completed

Manual Testing Steps

go test ./internal/services/dgwcompat ./internal/services -count=1
go build ./...
gofmt -l internal/services/dgwcompat/services.go internal/services/dgwcompat/services_test.go

Test Coverage

  • New tests added
  • Existing tests updated
  • Coverage maintained or improved

Screenshots / Recordings

Not applicable.

Performance Impact

  • Memory usage: No change
  • CPU usage: No change
  • Throughput: No change
  • Lock contention: No change

An auto-assign now performs one extra indexed query on tasks(dc_plan_id, status, deleted_at) and one metadata write. Reclaim may update multiple stale rows sequentially on the first request after the lease expires; the per-plan count is bounded by the plan target.

Documentation

  • No documentation changes needed

Related Issues

  • Fixes #

Additional Notes

Deployment does not modify existing rows. For already-stale tasks, the recommended post-deploy operation is to backfill the marker rather than directly changing status:

UPDATE tasks
SET metadata = JSON_SET(COALESCE(metadata, JSON_OBJECT()), '$.auto_assigned_upload', true)
WHERE status = 'uploading'
  AND episode_id IS NULL
  AND deleted_at IS NULL
  AND started_at IS NULL
  AND updated_at < NOW() - INTERVAL 2 HOUR;

The next auto-assign then reclaims those rows with the live-session protection in place.

Reviewers

@reviewer1

Notes for Reviewers

  • Please focus on the target-reached handling: Aborted is intentional and is classified as retryable by ego-portal.
  • Confirm that skipping tasks with fresh STS credentials is sufficient to protect long-running uploads.
  • auto_assigned_upload is stored in the existing tasks.metadata JSON, so no migration is required.

Auto-assigned uploads that fail or are abandoned can stay in uploading without an episode forever. loadTaskSupplyCounts counts those rows against the plan target, so plans eventually reject new uploads even though the remaining recordings were never ingested.

Mark auto-assigned tasks in task metadata, reclaim them after a hard-coded 2h lease, and run the reclaim before allocating a task and when the target is reached. Skip tasks whose logical upload still has fresh STS credentials so a genuinely long upload is not released. When target is blocked by unfinished uploads, return a retryable Aborted status so ego-portal keeps retrying instead of marking the job terminal.
@shark0F0497
shark0F0497 force-pushed the fix/dgw-reclaim-stale-auto-assigned-uploads branch from 0a9f88e to 7104add Compare November 17, 2026 11:19
@shark0F0497
shark0F0497 merged commit 4b06737 into main-v2 Nov 18, 2026
4 checks passed
@shark0F0497
shark0F0497 deleted the fix/dgw-reclaim-stale-auto-assigned-uploads branch November 18, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant