Evaluation: Improve fast-eval retries #1320

Describe the bug

During an OpenAI outage (run 738, 2026-10-28), fast-eval rows landed in the score trace with the provider error as the answer instead of being retried through the outage:

"llm_answer": "ERROR: OpenAI server error (code: 500): Error code: 500 - {'error': {'message': 'An error occurred while processing your request. You can retry your request, or contact us through our help center at help.openai.com if the error persists. Please include the request ID req_df2c76fe21384dcc8785446d6b97575a in your message.', 'type': 'server_error', 'param': None, 'code': 'server_error'}}. This is usually transient — retry in a few seconds. If issue persists, contact Kaapi."

Trace: s3://ai-platform-documents-production/6d219839-07d9-4dd2-9aad-3162ff1926b4/evaluations/score/738/traces_738.json

Retry exists but the budget is far too small for a real outage. Per row, in crud/evaluations/fast.py::_execute_llm_call_for_question (@retry_llm_call, crud/evaluations/retry.py):

Up to 9 HTTP calls, ~6s of sleep, all exhausted in under ~15s. An OpenAI 500 incident lasts minutes, so every attempt hits the same outage. On exhaustion _last_llm_result returns the last BlockResult (by design: a raise would abort the chunk and the cron healer would re-enqueue it, re-charging the provider), and fast.py::_llm_call_for_item stamps ERROR: ... into generated_output with failed=True. Below EVAL_FAST_FAILURE_THRESHOLD (0.5) the run still completes with those rows unscoreable. There is no failed-row rerun; the only recovery is a full re-trigger that re-pays every row.

Per-row waits of minutes are not an option: rows go through a 4-thread pool (EVAL_FAST_API_CONCURRENCY), so a 50-row chunk sleeping 8 min per row is 100 min against a 300s/600s Celery limit.

Expected behavior

A transient provider outage of a few minutes is waited out and the row gets a real answer. Deterministic failures (bad config, missing question, guardrails auth 401) do not wait.

Steps to reproduce

1. Trigger a fast text evaluation run (v1 or v2) while OpenAI Responses returns 500 for longer than ~15s.
2. Open the run's score_trace_url; affected rows carry "llm_answer": "ERROR: OpenAI server error (code: 500) ..." and are unscoreable.

Proposed fix

Move the waits from per-row to per-chunk, re-running only failed rows in waves (1 min, 2 min, 5 min; 8 min total, shared across the chunk):

• crud/evaluations/retry.py: drop per-row retry_llm_call; add RETRY_WAVE_DELAYS_SECONDS = (60, 120, 300). Keep retry_openai_call for embeddings/judge.
• crud/evaluations/fast.py: after the pool pass in run_response_chunk, _rerun_failed_rows sleeps a wave delay then re-runs rows with failed and retryable; rows failing every wave keep their last ERROR: result. _run_in_pool keeps item order so re-runs map back by index. SDK's own 2 quick retries still apply per call.
• crud/evaluations/fast_results.py: carry BlockResult.retryable onto ResponseResult.retryable so deterministic failures never trigger a wait.
• core/config.py: new EVAL_FAST_CHUNK_TIME_LIMIT_SECONDS = 1080; bump EVAL_FAST_STALL_THRESHOLD_MINUTES 15 → 20 so the healer does not re-enqueue a chunk mid-wait and double-bill.
• celery/tasks/job_execution.py: run_evaluation_fast_chunk uses the new limit for both Celery time_limit and gevent_timeout.
• Wiki: evaluations.md, llm-call.md.

Out of scope, follow-ups: aggregate task (embeddings, judge) still on the old 3-attempt/≤3s tenacity policy; batch runs have no retry; no failed-row rerun endpoint for outages longer than 8 min.

Original issue

in running evaluations we got this
s3://ai-platform-documents-production/6d219839-07d9-4dd2-9aad-3162ff1926b4/evaluations/score/738/traces_738.json
as score_trace_url
we got this as llm_answer as openai was down maybe
"llm_answer": "ERROR: OpenAI server error (code: 500): ... This is usually transient — retry in a few seconds. If issue persists, contact Kaapi.",
this should not be case and there should be some retry mechanism
can you debug and share findings first so we can plan

can we bump RETRY_MAX_ATTEMPTS to 3 attempts with longer sleep like incremental, 1 min again failed wait 2 min so at 3min and then wait 5 min i.e. 8th minute