feat(serverless): run fitness checks at startup, add skip env var - #578
Open
justinwlin wants to merge 6 commits into
Open
feat(serverless): run fitness checks at startup, add skip env var#578justinwlin wants to merge 6 commits into
justinwlin wants to merge 6 commits into
Conversation
Built-in GPU/system fitness checks ran in run_worker, which a handler module only reaches after loading its model. Run them when runpod.serverless is imported instead, so a broken environment fails in seconds. User-registered checks still run at start(); checks that already passed are not repeated. Adds RUNPOD_SKIP_FITNESS_CHECKS to disable all checks and RUNPOD_DEFER_FITNESS_CHECKS to restore the previous start()-only timing.
_cuda_init_check and _benchmark_check import torch and allocate on the device. Running them at import would leave a CUDA context in a process the handler may later fork, which CUDA does not support and vLLM/DeepSpeed trip over. Mark them @defer_to_worker_start so only subprocess-based and non-GPU checks run early.
- run startup pass on a dedicated event loop instead of asyncio.run, which resets the loop policy and breaks asyncio.get_event_loop() in handler code on Python 3.10+ - set RUNPOD_FITNESS_CHECKS_DONE after the startup pass so children re-importing this module under multiprocessing 'spawn' skip the checks - latch check auto-registration state only on success, so a malformed RUNPOD_MIN_*/GPU timeout value re-raises loudly in run_worker instead of silently disabling all system checks - compare completed checks by identity, not equality, so distinct registrations that compare equal (bound methods) are not skipped - bound the nvidia-smi call in rp_cuda.is_available with a 5s timeout - accept 1/true/yes/on for RUNPOD_SKIP_GPU_CHECK and RUNPOD_SKIP_AUTO_SYSTEM_CHECKS, matching the new flags - tests: pin the worker.py and import-time wiring, the full defer behavior, the done marker, the real auto-registration path (guard: no torch import), and bound-method re-registration; fix an orphaned coroutine in test_unexpected_error_does_not_propagate - docs: thresholds/skip flags must be set before import runpod, realtime API mode runs only the import-time checks, refresh stale ARCHITECTURE.md execution flow
…touch-ups - regression test: malformed RUNPOD_MIN_* must re-raise in run_worker, never fail open (latch-on-success) - fix dormant called/calls typo in the done-marker test - README: checks run once per check, not once at startup - ARCHITECTURE.md: failure path is os._exit(1), not sys.exit(1) - docs: GPU benchmark default timeout is 2s, not 100ms - rp_gpu_fitness docstring: lazy registration + truthy flag values
The import-time pass consumes RUNPOD_MIN_*/RUNPOD_SKIP_*/RUNPOD_GPU_* at import; values set from the handler afterwards were silently ignored. run_fitness_checks now diffs the current env against the values snapshot at the startup pass and warns with the exact fix (set before import, or RUNPOD_DEFER_FITNESS_CHECKS=true).
justinwlin
marked this pull request as ready for review
September 8, 2026 18:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DR-1409.
Workers currently load their models before checking GPU and system health. This PR moves most built-in checks to
import runpod, so an unhealthy worker can fail before spending minutes loading a model.What changes
gpu_testbinary (with annvidia-smifallback).RUNPOD_WEBHOOK_GET_JOBis set; spawned child processes skip them.Configuration and compatibility
RUNPOD_SKIP_FITNESS_CHECKS=truedisables all checks, including user-registered checks.RUNPOD_DEFER_FITNESS_CHECKS=truerestores the previous worker-start-only timing.runpod. Later configuration changes produce a warning.A failed check still terminates the worker. Unexpected errors setting up the import-time pass are logged without blocking import.
Validation
pytest tests: 654 passed on Python 3.11.