Skip to content

fix: guard set_scale against invalid concurrency_modifier returns - #586

Open
Swigler wants to merge 1 commit into
runpod:mainfrom
Swigler:fix/concurrency-modifier-none-guard
Open

fix: guard set_scale against invalid concurrency_modifier returns#586
Swigler wants to merge 1 commit into
runpod:mainfrom
Swigler:fix/concurrency-modifier-none-guard

Conversation

@Swigler

@Swigler Swigler commented Sep 3, 2026

Copy link
Copy Markdown

Summary

  • Validate concurrency_modifier return value in set_scale() — reject None, non-int, and values < 1 (defaults to 1 with a warning log)
  • Wrap the callback in try/except — a raising modifier keeps the current concurrency instead of crashing the entire worker

Fixes #458

What was wrong

If a user-provided concurrency_modifier callback returns None (or any non-integer), self.current_concurrency is set to None. This later crashes at the job scheduling loop:

TypeError: '<' not supported between instances of 'int' and 'NoneType'
  File "rp_scale.py", line 241, in run_jobs
    while len(tasks) < self.current_concurrency and not self.jobs_queue.empty():

The crash kills the worker, causing 2-3 minutes of downtime while the Docker container restarts.

What changed

rp_scale.pyset_scale() method:

  • Wrap concurrency_modifier() call in try/except — if it raises, log a warning and keep the current value
  • Validate the return: must be int and >= 1, otherwise default to 1 with a warning

test_scale.py — 5 new tests:

  • None return → defaults to 1
  • Negative return → defaults to 1
  • Zero return → defaults to 1
  • Exception in modifier → keeps current concurrency
  • Valid int → applied correctly

Test plan

  • All 9 tests in test_scale.py pass (4 existing + 5 new)
  • All 17 tests across both modified test files pass

…npod#458)

If a user-provided concurrency_modifier callback returns None, a
non-integer, or a value < 1, the worker crashes with:
  TypeError: '<' not supported between instances of 'int' and 'NoneType'

Validate the return value in set_scale(): reject None, non-int, and
values < 1 (default to 1 with a warning). Wrap the callback in
try/except so a raising modifier keeps the current concurrency instead
of crashing the worker.

Closes runpod#458
@Swigler
Swigler force-pushed the fix/concurrency-modifier-none-guard branch from 7ec7b56 to d539dcb Compare September 3, 2026 17:56
@Swigler

Swigler commented Sep 3, 2026

Copy link
Copy Markdown
Author

Ping — if concurrency_modifier returns an invalid value, set_scale crashes the worker. This adds a guard. Ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runpod self.current_concurrency turned into None in some cases.

1 participant