Observability: Cron enhancements #1304

Problem

/cron/* endpoints are excluded from tracing on purpose, so an idle cron run doesn't create a trace every minute. The downside is that Celery tasks the cron enqueues (send_eval_completion_notification, fast-eval barriers, iteration resumes) have no parent span and each one shows up as its own root trace. You can't see in one place what a cron run actually processed.

There's a second gap: process_all_pending_evaluations catches every exception and returns {"status": "error"}, so the @sentry_sdk.monitor check-in still reports OK when the run failed.

What we'll do

• Trace only when there's real work. Keep /cron excluded. Open a span (evaluation.cron.poll, evaluation.cron.poll_stt / poll_tts, plus fast-barrier and iteration-resume dispatch) only after a poll finds pending runs. Tasks enqueued inside nest under it; idle runs produce no trace.
  • crud/evaluations/processing.py → poll_all_pending_evaluations
  • crud/evaluations/cron_utils.py → poll_all_pending_evaluations_by_type
  • crud/evaluations/cron.py → fast-eval barrier and iteration-resume dispatch
• Report real failures to the cron monitor. When a run ends with status: error, send an error check-in (re-raise, or capture_checkin(..., status=MonitorStatus.ERROR)), so Sentry marks the run failed instead of OK.
• Span attributes worth having: pending run count, processed / failed / still-processing counts, so a cron trace explains itself.
• Consistent timezone across monitor configs (evaluations uses UTC, the others Asia/Kolkata).
• Tests: span exists only when there are pending runs; failed run sends an error check-in.

What Sentry Crons gives us (besides uptime)

• @monitor / with monitor(...) with monitor_config to create or update the monitor from code (already used in api/routes/cron.py).
• monitor_config: schedule (crontab or interval), checkin_margin, max_runtime, failure_issue_threshold, recovery_threshold, timezone.
• Check-in statuses in_progress / ok / error, plus manual capture_checkin(monitor_slug, check_in_id, status) for jobs that can't use the decorator.
• Missed, failed and timed-out runs become Issues, with per-monitor alerting.
• Monitor environments: check-ins are tracked per environment (production / staging).
• CeleryIntegration(monitor_beat_tasks=True, exclude_beat_tasks=[...]) auto-monitors Celery Beat tasks. Not used today because our crons are HTTP-triggered, but an option if they move to Beat.

Out of scope

Uptime monitors, and moving crons to Celery Beat.