feat(serverless): run fitness checks at startup, add skip env var

ARCHITECTURE.md

@@ -604,7 +604,7 @@ log.error(message, job_id=None)
 **Responsibilities**:
 - Validate worker health at startup before handler initialization
 - Support both synchronous and asynchronous check functions
-- Exit immediately with sys.exit(1) on any check failure
+- Exit immediately with os._exit(1) on any check failure
 - Enable fail-fast deployment validation
 
 **Key Functions**:
@@ -613,11 +613,11 @@ log.error(message, job_id=None)
 - `clear_fitness_checks()`: Clear registry (testing only)
 
 **Execution Flow**:
-1. Called from `worker.py:40` before heartbeat starts: `asyncio.run(run_fitness_checks())`
+1. Runs twice per worker: built-in checks at `import runpod.serverless` via `run_startup_fitness_checks()`, then user-registered and `@defer_to_worker_start` checks from `worker.py:40` before heartbeat starts: `asyncio.run(run_fitness_checks())`; completed checks are not repeated, and `RUNPOD_DEFER_FITNESS_CHECKS=true` collapses both passes into the `worker.py` one
 2. Runs only in production mode (skipped for local testing)
 3. Auto-detects sync vs async using `inspect.iscoroutinefunction()`
 4. Executes checks in registration order (list preserves order)
-5. On failure: log detailed error, call `sys.exit(1)`
+5. On failure: log detailed error, best-effort unhealthy report, force-kill via `os._exit(1)`
 6. On success: log completion, proceed with worker startup
 
 **Performance**: ~0.5ms framework overhead per check, total depends on check logic
@@ -765,7 +765,7 @@ sequenceDiagram
             CHECK->>CHECK: Log success
         else Check fails
             CHECK->>SYS: Log error + traceback
-            CHECK->>SYS: sys.exit(1)
+            CHECK->>SYS: os._exit(1)
         end
     end
 

README.md

@@ -143,7 +143,7 @@ runpod.serverless.start({"handler": handler})
 
 **Key Features:**
 - Supports both synchronous and asynchronous check functions
-- Checks run only once at worker startup (production mode)
+- Each check runs once per worker: built-ins at import, your checks at start (production mode)
 - Runs before handler initialization and job processing begins
 - Any check failure exits with code 1 (worker marked unhealthy)
 

docs/serverless/worker_fitness_checks.md

@@ -41,6 +41,23 @@ if __name__ == "__main__":
     runpod.serverless.start({"handler": handler})
 ```
 
+## When Checks Run
+
+The built-in GPU and system checks run at **import time** — when your handler module runs `import runpod`, before it loads a model — so a broken GPU or full disk fails the worker in seconds instead of after a multi-minute load.
+
+Two checks stay at `runpod.serverless.start()`: the CUDA initialization check and the GPU compute benchmark. Both import `torch` and allocate on the device, which would leave a CUDA context in a process your handler may later fork — unsupported by CUDA, and something vLLM and DeepSpeed trip over. The remaining built-ins (memory, disk, network, CUDA version via `nvidia-smi`, and the native `gpu_test` binary) run at import.
+
+Your own `@register_fitness_check` functions are registered after that import, so they also run at `start()`. Checks that already passed are not repeated.
+
+Note that the memory check now measures a fresh container rather than one with your model loaded, so `RUNPOD_MIN_MEMORY_GB` validates the environment you were given, not the headroom left after loading.
+
+The import-time pass no-ops outside a real worker (no `RUNPOD_WEBHOOK_GET_JOB`), leaving local runs and tests unaffected — note that inside a worker container *any* `import runpod` triggers it. Set `RUNPOD_DEFER_FITNESS_CHECKS=true` to run everything at `start()` instead.
+
+Two details worth knowing:
+
+- Thresholds and skip flags (`RUNPOD_MIN_*`, `RUNPOD_SKIP_*`, `RUNPOD_GPU_*`) are read when the checks first run, so set them **before** `import runpod` — Dockerfile `ENV` recommended; setting them from Python in your handler is too late on the real platform. If any of them changed since the import, `start()` logs a warning naming the ignored variables.
+- Workers serving the realtime API (`--rp_serve_api`) never enter the worker loop, so only the import-time checks apply there; the two deferred CUDA checks do not run in that mode. Child processes created with multiprocessing's `spawn` start method re-import this module but inherit a marker and skip the checks.
+
 ## Async Fitness Checks
 
 Fitness checks support both synchronous and asynchronous functions:
@@ -343,15 +360,15 @@ ERROR  | Fitness check failed: _cuda_init_check | RuntimeError: Failed to initia
 
 Quick matrix multiplication to verify GPU compute functionality and responsiveness. Skips silently on CPU-only workers.
 
-- **Default**: 100ms maximum execution time
-- **Configure**: `RUNPOD_GPU_BENCHMARK_TIMEOUT=2`
+- **Default**: 2 seconds maximum execution time
+- **Configure**: `RUNPOD_GPU_BENCHMARK_TIMEOUT=2` (seconds)
 
 What it tests:
 - GPU compute capability (matrix multiplication)
 - GPU response time
 - Memory bandwidth to GPU
 
-If the operation takes longer than 100ms, the worker exits as the GPU is too slow for reliable job processing.
+If the operation takes longer than the timeout, the worker exits as the GPU is too slow for reliable job processing.
 
 Example log output:
 ```
@@ -371,13 +388,15 @@ ENV RUNPOD_NETWORK_CHECK_TIMEOUT=10
 ENV RUNPOD_GPU_BENCHMARK_TIMEOUT=2
 ```
 
-Or in Python:
+Or in Python, before `import runpod` (on the real platform the checks run at import):
 
 ```python
 import os
 
 os.environ["RUNPOD_MIN_MEMORY_GB"] = "8.0"
 os.environ["RUNPOD_MIN_DISK_PERCENT"] = "15.0"
+
+import runpod
 ```
 
 ### Disabling Built-in Checks
@@ -388,6 +407,8 @@ For testing or specialized deployments, built-in checks can be disabled via envi
 |---|---|
 | `RUNPOD_SKIP_AUTO_SYSTEM_CHECKS=true` | Skips auto-registration of memory, disk, network, CUDA version, CUDA init, and GPU benchmark checks |
 | `RUNPOD_SKIP_GPU_CHECK=true` | Skips auto-registration of the native GPU memory allocation test (`gpu_test` binary) |
+| `RUNPOD_SKIP_FITNESS_CHECKS=true` | Skips every fitness check, built-in **and** user-registered |
+| `RUNPOD_DEFER_FITNESS_CHECKS=true` | Keeps the checks but runs them only at `runpod.serverless.start()`, not at import |
 
 ```python
 import os
@@ -397,15 +418,19 @@ os.environ["RUNPOD_SKIP_AUTO_SYSTEM_CHECKS"] = "true"
 
 # Disable the automatic GPU memory allocation test
 os.environ["RUNPOD_SKIP_GPU_CHECK"] = "true"
+
+import runpod
 ```
 
-User-registered checks via `@register_fitness_check` still run regardless of these flags.
+As with the thresholds, set these before `import runpod` on the real platform.
+
+User-registered checks via `@register_fitness_check` still run regardless of `RUNPOD_SKIP_AUTO_SYSTEM_CHECKS` and `RUNPOD_SKIP_GPU_CHECK`. Only `RUNPOD_SKIP_FITNESS_CHECKS` disables those too.
 
 ## Behavior
 
 ### Execution Timing
 
-- Fitness checks run **only once at worker startup**
+- Each check runs **once per worker**: built-ins at import, your registered checks and the deferred CUDA checks at `start()`; checks that passed are not repeated
 - They run **before the first job is processed**
 - They run **only on the actual Runpod serverless platform**
 - Local development and testing modes skip fitness checks
@@ -555,7 +580,7 @@ async def check_api_with_retry():
 
 ## Testing
 
-When developing locally, fitness checks don't run. To test them, you can manually invoke the runner:
+When developing locally, fitness checks don't run. To test them, you can manually invoke the runner. Note that each check runs once per process: a second `run_fitness_checks()` call skips checks that already passed, so call `clear_fitness_checks()` (as below) between runs:
 
 ```python
 import asyncio

runpod/serverless/__init__.py

@@ -16,7 +16,7 @@
 from . import worker
 from .modules.rp_logger import RunPodLogger
 from .modules.rp_progress import progress_update
-from .modules.rp_fitness import register_fitness_check
+from .modules.rp_fitness import register_fitness_check, run_startup_fitness_checks
 from .utils.rp_volume_cache import VolumeCache
 
 __all__ = [
@@ -29,6 +29,10 @@
 
 log = RunPodLogger()
 
+# Check the environment here rather than in start(), which a handler module
+# only reaches after loading its model. No-op outside a real worker.
+run_startup_fitness_checks()
+
 
 # ---------------------------------------------------------------------------- #
 #                              Run Time Arguments                              #