Secondary: collect_pull_request_review_comments getting stuck on repos with no pull requests/comments

Note

Migrated from augurlabs/augur#3668
Originally opened by @cdolfi on 2026-02-09

I am unsure if this is its own bug or something thats showing up because of other things:

I am noticing that secondary tasks (collect_pull_request_review_comments) are running for a suspicious amount of time when there is no pull requests / no pull request comments and I am not seeing them complete.

Example repos:
https://github.com/fabiospampinato/nsfw-watcher
https://github.com/helium/erlang-tpke (didnt see any comments)
https://github.com/darcy-framework/darcy-framework
https://github.com/apache/logging-flume-http (empty repo, might just delete)
https://github.com/openela-main/abrt-java-connector
https://github.com/redhat-openstack/rdo-ci-tools

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-09

Dug into this. There are a few things going on that together explain why secondary collection gets stuck on repos with no
PRs/comments. Mainly that we don't have task level timeouts for the individual tasks in the Celerey group() in https://github.com/chaoss/augur/blob/main/augur/tasks/start_tasks.py#L122-L127 .

Now I think this was an intentional design choice. But please correct me if I am mistaken.

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-09

At https://github.com/chaoss/augur/blob/main/augur/tasks/github/pull_requests/tasks.py#L244-L247, the task builds a
pr_review_id_mapping from the DB. For repos with no PRs, this mapping is empty. But instead of returning early, the task still proceeds to create a GithubRandomKeyAuth, spin up a KeyClient, and make a GitHub API call at https://github.com/chaoss/augur/blob/main/augur/tasks/github/pull_requests/tasks.py#L263.

Any comments returned would fail the mapping lookup and get skipped anyway — the API call is completely wasted. Compare with collect_pull_request_reviews which already has an early return at https://github.com/chaoss/augur/blob/main/augur/tasks/github/pull_requests/tasks.py#L483-L485 when there are no PRs. collect_pull_request_review_comments should do the same when there are no reviews.

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-09

A likely bigger problem in my opinion is that process_ossf_dependency_metrics can hang indefinitely. The scorecard subprocess call in https://github.com/chaoss/augur/blob/main/augur/tasks/util/worker_util.py#L132 uses timeout=None. The scorecard binary makes its own network calls to GitHub. If it hangs (network issues, rate limits, whatever), the Celery
worker hangs forever.

Since process_ossf_dependency_metrics is in the same group() as the PR review comment task, maybe the entire group never completes and secondary_task_success_util never runs. This is probably why these repos appear stuck? It might not actually be collect_pull_request_review_comments that's blocking, but the scorecard subprocess. Can we check the logs to see this for those blocking repos?

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-10

Made a new issue for the scorecard stuff #202, but given that cali was noticing scorecard issues as well (and has been for a while) maybe that could be related?

@shlokgilda

Note

Migrated comment from original
Originally posted by @cdolfi on 2026-02-10

@shlokgilda I do think you have identified the problem/solution for the collect_pull_request_review_comments issue

@cdolfi

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-12

Since @cdolfi confirmed the analysis, I guess just adding an early return when pr_review_id_mapping is empty, same pattern that collect_pull_request_reviews already uses.

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-12

That said, I don't think the missing early return is what's actually causing these repos to appear stuck. For repos with no PRs, the GitHub API call to /repos/{owner}/{repo}/pulls/comments would just return an empty array and the task would finish quickly.

So the early return fix is still worth doing (saves a wasted API call and key allocation per empty repo), but it probably won't fix
the "stuck" behavior on its own.

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-12

For the listed repos, can we run scorecard directly? If we see that scorecard is not the blocker, we probably should look elsewhere.

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-12

i mean we could clone those repos locally and run scorecard on them, but I don't know if that would be a proper test since you wouldn't be testing in the same "the keys get used up pretty quickly" environment that i suspect our instance is creating

Note

Migrated comment from original
Originally posted by @shlokgilda on 2026-02-13

Fair point on the environment. Can we check the production logs (if we have those?) for those repos to see which task is actually blocking? That'd confirm whether it's scorecard or something else. Either way, I'll open a PR for the early return fix since that's confirmed.

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-13

wonder if related to #183 as well

@MoralCode

Note

Migrated comment from original
Originally posted by @cdolfi on 2026-02-13

@MoralCode I dont think they are related

Note

Migrated comment from original
Originally posted by @MoralCode on 2026-02-16

reopening for cali to verify this fix in the next release since this was marked as "partially fixes" in the associated PR, inadvertently triggering github to link them

hmmm not sure if this is quite the same issue as is being fixed in #393 .

Navigation Menu

Repository navigation

Secondary: collect_pull_request_review_comments getting stuck on repos with no pull requests/comments #200

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Secondary: collect_pull_request_review_comments getting stuck on repos with no pull requests/comments #200

Description

Activity

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

collectoss-issue-migrator commented on Apr 30, 2026

MoralCode commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions