Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions automation/jobs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,15 @@ profiles:

half_hourly:
cron: "*/30 * * * *"
description: "Thirty-minute risk monitoring for active Yearn strategies."
description: "Thirty-minute risk monitoring for active Yearn strategies, and timelock alerts."
tasks:
- name: "yearn-lender-borrower-ltv"
script: protocols/yearn/lender_borrower.py
args: { checks: ltv }
# Every monitored timelock delays at least 24h, so 30 minutes leaves the
# reaction window nearly intact. Its cursor (TIMELOCK_LAST_TS) stays in the
# default cache-id.txt, as under the hourly profile: the move skips nothing.
- { name: "timelock-alerts", script: protocols/timelock/timelock_alerts.py }

six_hourly:
cron: "17 */6 * * *"
Expand Down Expand Up @@ -67,7 +71,6 @@ profiles:
# Cache: tks-trigger-cache.json under $CACHE_DIR (check_stuck_triggers.DEFAULT_CACHE_FILE).
- { name: "yearn-check-stuck-triggers", script: protocols/yearn/check_stuck_triggers.py, enabled: false }
- { name: "maple", script: protocols/maple/main.py, enabled: false }
- { name: "timelock-alerts", script: protocols/timelock/timelock_alerts.py }
- { name: "ustb", script: protocols/ustb/main.py }
# Paused 2026-09-23 pending Bedrock response on the supply feeder omitting BOB (alert spam).
- { name: "unibtc", script: protocols/unibtc/main.py, enabled: false }
Expand Down
2 changes: 1 addition & 1 deletion deploy/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ directly.

`monitoring-timelock-subscriber` is the continuous timelock event worker from
[#438](https://github.com/yearn/monitoring/issues/438). It runs alongside the
hourly `timelock-alerts` task, which still sends every timelock alert. The pilot
`timelock-alerts` task (every 30 minutes), which still sends every timelock alert. The pilot
sends no alerts: it records what it observes in one SQLite inbox per stream
(`/srv/cache/timelock_subscriber.db`, and `/srv/cache/morpho_subscriber.db` once
the `morpho` stream is enabled) and journald. With `TIMELOCK_SUBSCRIBER_NOTIFY=errors` it also posts diagnostic
Expand Down
2 changes: 1 addition & 1 deletion monitoring.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,7 @@ protocols:
timelock:
display_name: "Timelock Alerts"
description: "Cross-protocol timelock event monitoring"
cadence: "Hourly"
cadence: "Every 30 minutes (stale operations: daily)"
tasks:
- protocols/timelock/timelock_alerts.py
- protocols/timelock/stale_operations.py
Expand Down
4 changes: 2 additions & 2 deletions protocols/timelock/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ Monitors all timelock contract types (TimelockController, Aave, Compound, Lido,
3. Routes each alert to the correct Telegram channel based on the protocol mapping.
4. Stores the latest processed `blockTimestamp` in the SQLite state store (namespace `cache-id.txt`, key `TIMELOCK_LAST_TS`) to avoid duplicate alerts between runs.

The script runs hourly via the [monitoring runner](../../automation/jobs.yaml).
The script runs every 30 minutes (the `half_hourly` profile) via the [monitoring runner](../../automation/jobs.yaml). Every monitored timelock delays operations by at least 24h (Infinifi's short timelock has been at 24h since 2026-08-16), so detection takes a small share of the reaction window.

## Continuous Subscriber Pilot

`event_subscriber.py` is a continuously running worker that observes the same `TimelockEvent` rows as `timelock_alerts.py` with lower latency ([#438](https://github.com/yearn/monitoring/issues/438), the first step of [#437](https://github.com/yearn/monitoring/issues/437)). It is an observation pilot: **it sends no alerts and changes nothing about current delivery**. The hourly task remains the only sender to protocol groups. Optionally it posts diagnostic notices to the internal errors chat (see [Pilot notices](#pilot-notices)). It runs as the `monitoring-timelock-subscriber` systemd unit; operations are in the [runbook](../../deploy/runbook.md#timelock-event-subscriber-pilot).
`event_subscriber.py` is a continuously running worker that observes the same `TimelockEvent` rows as `timelock_alerts.py` with lower latency ([#438](https://github.com/yearn/monitoring/issues/438), the first step of [#437](https://github.com/yearn/monitoring/issues/437)). It is an observation pilot: **it sends no alerts and changes nothing about current delivery**. The scheduled `timelock-alerts` task remains the only sender to protocol groups. Optionally it posts diagnostic notices to the internal errors chat (see [Pilot notices](#pilot-notices)). It runs as the `monitoring-timelock-subscriber` systemd unit; operations are in the [runbook](../../deploy/runbook.md#timelock-event-subscriber-pilot).

The worker follows **named event streams** ([#447](https://github.com/yearn/monitoring/issues/447)). The shared code lives in [`utils/subscriber/`](../../utils/subscriber/); each stream is an `EventStream` that names its Envio entity, the field holding the monitored contract, its field selection, and its own SQLite file and table prefix:

Expand Down
2 changes: 1 addition & 1 deletion protocols/timelock/event_stream.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from utils.subscriber.reconcile import build_scope
from utils.subscriber.stream import EventStream, StreamLabel, StreamScope

# The hourly monitor's selection plus logIndex and blockHash. Both transports use
# The scheduled monitor's selection plus logIndex and blockHash. Both transports use
# it, so a row fingerprints identically whichever one fetched it, and a reorg that
# replaces the block changes the fingerprint through blockHash.
TIMELOCK_FIELDS = """
Expand Down
2 changes: 1 addition & 1 deletion utils/subscriber/inbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

Each stream has its own SQLite file and table prefix (see ``EventStream``), so
streams never contend for one write lock. Per-chain checkpoints live here too,
independent of the hourly monitors' cursors. The inbox writes nothing else: no
independent of the scheduled monitors' cursors. The inbox writes nothing else: no
Telegram, no alert store.
"""

Expand Down
Loading