[spalen0]

Closes #448. Part of #437. Builds on the named event streams from #452.

What

For every newly indexed MorphoGovernanceEvent, the morpho stream now decides what the daily governance jobs would alert, records it durably, and delivers it according to MORPHO_LIVE_DELIVERY:

Mode	What happens
shadow (default)	Posts the would-be alert to the internal errors chat as [morpho-live-shadow]. Without an errors chat it only records (observe), so it never reaches the Morpho group.
observe	Records only; sends nothing.
production	Sends to the Morpho channel and commits the daily jobs' cache (see below). Meant for the #450 cutover, after shadow output matches the daily alerts.

In shadow and observe, the daily governance.py / governance_v2.py jobs keep alerting the Morpho group exactly as before.

Why on-chain confirmation

The indexer (yearn-envio#71) has no V1 revoke/accept/set events and no V2 Revoke, so events alone can't tell a revoked operation from a pending one. Each event is therefore a trigger. live_rules reads the vault's current state on-chain, at a block at or after the event; if the RPC node is behind, it retries. It then runs the daily jobs' own check functions on that state:

Event	Check
V1 SubmitCap, SubmitMarketRemoval	that market's pending cap and removal checks, including the potential-loss alert
V1 SubmitTimelock	the timelock (and guardian) check
V2 Submit	executableAt(data) must still be set, then diff_pending_config (loss alerts) and alert_pending_new (grouping)
V2 SetOwner, SetCurator	diff_single_role
V2 AddAdapter, RemoveAdapter	diff_set("adapters")
V2 Accept	none: executed/revoked notes stay with the daily job

So content, severity, per-vault grouping and loss classification match the daily jobs. An operation that was revoked or executed before the read isn't pending anymore, so it isn't alerted. The checks read the daily cache (an operation the daily job already alerted isn't alerted again) but never write it. V2 state comes from chain reads (owner, curator, adapters, realAssets, totalAssets, the asset's metadata), because the Morpho API lags behind the chain.

Durability and delivery

• Store: live_store adds morpho_live_decisions (one row per event revision) and morpho_live_alerts (rendered messages with mode and status) in morpho_subscriber.db. A restart resumes undecided events. A reorg that rewrites an event bumps its revision, so the event is evaluated again.
• No historical alerts: blocks before the consumer's first run on the inbox (live_baseline_at) are recorded as historical and never evaluated. Events indexed while the worker was down are still evaluated.
• Off the ingestion path: live_governance runs as a worker StreamConsumer that is woken after each ingest, with a 5s polling fallback. RPC reads and Telegram calls never delay ingestion, and a consumer that dies withholds the watchdog ping.
• Retries: a failing vault retries with capped backoff and doesn't block other vaults. In shadow mode, a failed send stays pending and is retried in order.
• Shadow channel: shadow requires the internal errors chat and falls back to observe without it, so its output never reaches the Morpho group.
• Observability: report --streams morpho adds a decision summary (outcomes, block-to-decision p50/p95, request statuses). health fails when events stay undecided, or shadow alerts stay undelivered, for more than 10 minutes.
• Refactors used by both paths: render_vault_alerts is split out of send_vault_alerts, and diff_pending_config is extracted from _diff_pending. Helpers the live path reuses are now public: vault_header, diff_single_role, diff_set, alert_pending_new, load_market_governance_states. The daily behavior is unchanged.

Production mode and the shared lock

production follows the daily jobs' contract. Per vault, it holds governance_lock (a flock file in CACHE_DIR, morpho_governance_<vault>.lock) while it evaluates, sends every message, and only then commits the daily jobs' cache writes. The daily V1/V2 jobs now take the same lock around their check → send → commit, so the two processes can't both treat one change as new. Each change is alerted once, by whichever path sees it first, and the daily jobs keep catching whatever the live path missed. A failed send commits nothing; the events are re-evaluated and re-sent on retry, so you might get a duplicate but never a missed alert.

Review fixes

1. Cache namespace: the daily profile runs with MORPHO_FILENAME=cache-id-daily.txt, but the subscriber read cache-id.txt. daily_governance_cache.py now applies the daily profile's value from jobs.yaml before utils.cache loads, and the consumer refuses to start if a different namespace was bound.
2. Adapter valuation: a failed realAssets() read now fails the evaluation and is retried. Before, it read as an empty adapter, which could turn a potential-loss HIGH into a plain MEDIUM.
3. Starvation: backed-off vaults are excluded before the batch limit applies, so one failing vault can't starve the others.
4. Reorgs: each request stores its source events' revisions. A pending shadow request whose events were removed or rewritten is marked invalidated instead of sent.
5. Message size: the shadow header lists at most 5 event IDs. The final text is split to Telegram's limit and sent unescaped as plain text, straight to the errors channel, so nothing is truncated.
6. Delivery confirmation: send_telegram_message and send_alert now return whether Telegram accepted the message. Shadow and production treat a skipped send (missing credentials) as not delivered and retry it. Existing callers ignore the return value, so nothing else changes.
7. Backoff: the exponent is capped via backoff_delay, so a long outage can't raise OverflowError.

Fail closed on malformed Envio responses (#437 comment)

A response with no data, with data: null, or with a missing or null entity list used to read as an empty page (or {} / or []). Reconciliation then marked every active event in the range removed and moved the checkpoint past it. Once this PR's consumer runs, that would drop undecided Morpho events from evaluation and invalidate their queued alerts. The daily jobs would still catch them, but late.

entity_rows now requires data to be an object and the entity (or chain_metadata) to be a list of row objects, and raises EnvioQueryError otherwise. The cycle fails with event status and the checkpoint untouched. An explicit [] is still a valid empty range. WebSocket snapshots use the same check and reconnect instead of ingesting nothing.

New tests replay the reviewer's scenario (active event at block 12, checkpoint 15, overlap 5, head 20) with {}, {"data": null}, {"data": {}}, {"data": {"TimelockEvent": null}}, a non-list, a non-object row, and a non-object response. Each leaves the event active and the checkpoint at 15. Against the old lenient parsing, all 10 malformed-response tests fail. A real [] still completes the range.

Testing

• ruff and ty pass. Full suite: 2081 passed, 5 failed. The failures also happen on main and come from the local setup: 4 TestSafeApiQuota tests fail with a real SAFE_API_KEY in .env, and test_sd_notify_sends_datagram fails with AF_UNIX path too long on macOS.
• tests/test_morpho_live_governance.py (33 tests; also covers production send → commit ordering under the lock, failure without commit, the daily V2 job taking the lock, the file lock itself, and a regression test for each review fix, including a subprocess check that the real entrypoint binds cache-id-daily.txt):
  • V1/V2 parity: severity and content, standalone loss alerts, grouped submissions.
  • Not alerting when an operation is no longer pending or was already alerted.
  • One check per market, with no cache writes.
  • The RPC-behind retry, owner seeding, and Accept left to the daily job.
  • On-chain snapshot reads.
  • Consumer: historical baseline, per-vault grouping in chain order, observe mode sends nothing, backoff that doesn't block other vaults, restart resume, reorg re-evaluation.
  • Shadow delivery order with retry after a failure, plus the errors-chat requirement.
  • Health and summary.
  • Worker wake-up after a WebSocket ingest, and a dead consumer making the worker unhealthy.
• Live, read-only, on all 5 chains: synthetic events for the first configured V1/V2 vault per chain. Every RPC read works, and adapter realAssets sums match totalAssets. No operation is pending today, so the result is correctly "nothing new".
• Mainnet anvil fork: I impersonated the Yearn USDC V2 curator and submitted removeAdapter for its funded adapter, then ran the real consumer end to end in shadow mode, with sending captured instead of posted. It produced the daily job's exact 🚨 POTENTIAL VAULT LOSS HIGH message (amount at risk, 100% of vault, executable in 7 days, revoke hint) and marked it sent. After revoke, executableAt reads 0, which the rules treat as not pending.

🤖 Generated with Claude Code