diff --git a/automation/jobs.yaml b/automation/jobs.yaml index 77776eba..e085a2bc 100644 --- a/automation/jobs.yaml +++ b/automation/jobs.yaml @@ -20,11 +20,15 @@ profiles: half_hourly: cron: "*/30 * * * *" - description: "Thirty-minute risk monitoring for active Yearn strategies." + description: "Thirty-minute risk monitoring for active Yearn strategies, and timelock alerts." tasks: - name: "yearn-lender-borrower-ltv" script: protocols/yearn/lender_borrower.py args: { checks: ltv } + # Every monitored timelock delays at least 24h, so 30 minutes leaves the + # reaction window nearly intact. Its cursor (TIMELOCK_LAST_TS) stays in the + # default cache-id.txt, as under the hourly profile: the move skips nothing. + - { name: "timelock-alerts", script: protocols/timelock/timelock_alerts.py } six_hourly: cron: "17 */6 * * *" @@ -67,7 +71,6 @@ profiles: # Cache: tks-trigger-cache.json under $CACHE_DIR (check_stuck_triggers.DEFAULT_CACHE_FILE). - { name: "yearn-check-stuck-triggers", script: protocols/yearn/check_stuck_triggers.py, enabled: false } - { name: "maple", script: protocols/maple/main.py, enabled: false } - - { name: "timelock-alerts", script: protocols/timelock/timelock_alerts.py } - { name: "ustb", script: protocols/ustb/main.py } # Paused 2026-09-23 pending Bedrock response on the supply feeder omitting BOB (alert spam). - { name: "unibtc", script: protocols/unibtc/main.py, enabled: false } diff --git a/deploy/runbook.md b/deploy/runbook.md index c616e29f..7cb13de7 100644 --- a/deploy/runbook.md +++ b/deploy/runbook.md @@ -234,7 +234,7 @@ directly. `monitoring-timelock-subscriber` is the continuous timelock event worker from [#438](https://github.com/yearn/monitoring/issues/438). It runs alongside the -hourly `timelock-alerts` task, which still sends every timelock alert. The pilot +`timelock-alerts` task (every 30 minutes), which still sends every timelock alert. The pilot sends no alerts: it records what it observes in one SQLite inbox per stream (`/srv/cache/timelock_subscriber.db`, and `/srv/cache/morpho_subscriber.db` once the `morpho` stream is enabled) and journald. With `TIMELOCK_SUBSCRIBER_NOTIFY=errors` it also posts diagnostic diff --git a/monitoring.yaml b/monitoring.yaml index b5e40662..ad750328 100644 --- a/monitoring.yaml +++ b/monitoring.yaml @@ -415,7 +415,7 @@ protocols: timelock: display_name: "Timelock Alerts" description: "Cross-protocol timelock event monitoring" - cadence: "Hourly" + cadence: "Every 30 minutes (stale operations: daily)" tasks: - protocols/timelock/timelock_alerts.py - protocols/timelock/stale_operations.py diff --git a/protocols/timelock/README.md b/protocols/timelock/README.md index 795c71d0..f553c71c 100644 --- a/protocols/timelock/README.md +++ b/protocols/timelock/README.md @@ -9,11 +9,11 @@ Monitors all timelock contract types (TimelockController, Aave, Compound, Lido, 3. Routes each alert to the correct Telegram channel based on the protocol mapping. 4. Stores the latest processed `blockTimestamp` in the SQLite state store (namespace `cache-id.txt`, key `TIMELOCK_LAST_TS`) to avoid duplicate alerts between runs. -The script runs hourly via the [monitoring runner](../../automation/jobs.yaml). +The script runs every 30 minutes (the `half_hourly` profile) via the [monitoring runner](../../automation/jobs.yaml). Every monitored timelock delays operations by at least 24h (Infinifi's short timelock has been at 24h since 2026-08-16), so detection takes a small share of the reaction window. ## Continuous Subscriber Pilot -`event_subscriber.py` is a continuously running worker that observes the same `TimelockEvent` rows as `timelock_alerts.py` with lower latency ([#438](https://github.com/yearn/monitoring/issues/438), the first step of [#437](https://github.com/yearn/monitoring/issues/437)). It is an observation pilot: **it sends no alerts and changes nothing about current delivery**. The hourly task remains the only sender to protocol groups. Optionally it posts diagnostic notices to the internal errors chat (see [Pilot notices](#pilot-notices)). It runs as the `monitoring-timelock-subscriber` systemd unit; operations are in the [runbook](../../deploy/runbook.md#timelock-event-subscriber-pilot). +`event_subscriber.py` is a continuously running worker that observes the same `TimelockEvent` rows as `timelock_alerts.py` with lower latency ([#438](https://github.com/yearn/monitoring/issues/438), the first step of [#437](https://github.com/yearn/monitoring/issues/437)). It is an observation pilot: **it sends no alerts and changes nothing about current delivery**. The scheduled `timelock-alerts` task remains the only sender to protocol groups. Optionally it posts diagnostic notices to the internal errors chat (see [Pilot notices](#pilot-notices)). It runs as the `monitoring-timelock-subscriber` systemd unit; operations are in the [runbook](../../deploy/runbook.md#timelock-event-subscriber-pilot). The worker follows **named event streams** ([#447](https://github.com/yearn/monitoring/issues/447)). The shared code lives in [`utils/subscriber/`](../../utils/subscriber/); each stream is an `EventStream` that names its Envio entity, the field holding the monitored contract, its field selection, and its own SQLite file and table prefix: diff --git a/protocols/timelock/event_stream.py b/protocols/timelock/event_stream.py index 33b74e70..9f81ea54 100644 --- a/protocols/timelock/event_stream.py +++ b/protocols/timelock/event_stream.py @@ -4,7 +4,7 @@ from utils.subscriber.reconcile import build_scope from utils.subscriber.stream import EventStream, StreamLabel, StreamScope -# The hourly monitor's selection plus logIndex and blockHash. Both transports use +# The scheduled monitor's selection plus logIndex and blockHash. Both transports use # it, so a row fingerprints identically whichever one fetched it, and a reorg that # replaces the block changes the fingerprint through blockHash. TIMELOCK_FIELDS = """ diff --git a/utils/subscriber/inbox.py b/utils/subscriber/inbox.py index 4275e2f5..d01110a7 100644 --- a/utils/subscriber/inbox.py +++ b/utils/subscriber/inbox.py @@ -11,7 +11,7 @@ Each stream has its own SQLite file and table prefix (see ``EventStream``), so streams never contend for one write lock. Per-chain checkpoints live here too, -independent of the hourly monitors' cursors. The inbox writes nothing else: no +independent of the scheduled monitors' cursors. The inbox writes nothing else: no Telegram, no alert store. """