Skip to content

Persist emergency deliveries and bypass HIGH cooldown for CRITICAL - #457

Open
spalen0 wants to merge 1 commit into
mainfrom
feat/vaults-mommy-29-delivery
Open

spalen0 wants to merge 1 commit into
mainfrom
feat/vaults-mommy-29-delivery

Conversation

@spalen0

@spalen0 spalen0 commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

A successful HIGH currently suppresses CRITICAL for an hour, and a failed webhook is forgotten once the alert hook returns. Persist destination-specific deliveries before HTTP, let CRITICAL bypass HIGH's cooldown, and drain retries every minute without requiring another alert.

Preserves the existing liquidity-monitoring endpoint, credentials and payload. Optional vaults-mommy delivery has its own secret, endpoint, cooldown and retry state; its signed payload adds a stable event ID. Exact payload bytes and IDs survive restarts and uncertain acknowledgements. SQLite FULL transactions and expiring leases keep concurrent monitor processes coordinated without holding database locks during HTTP.

Timeouts/408/429/5xx retry with bounded backoff. Other HTTP failures remain terminal for explicit correction; redirects are not followed. Operator retry preserves ID/body, can explicitly adopt a corrected endpoint, and never happens automatically after endpoint changes. DEBUG retains queued work without sending. Secrets are not persisted.

Validation: 2,087 passed, 4 skipped, 108 subtests passed in the full monitoring suite; Ruff lint/format, ty check, and all repository prek hooks pass. Tests cover HIGH at t=0 and CRITICAL at t=20m, both destinations, unavailable destination/restart, unchanged liquidity envelope, exact-byte HMAC, uncertain acknowledgement, lease races, concurrency, terminal errors and missing credentials.

Sender portion of vaults-mommy #29, following epic #27. Receiver and real cross-repository integration follow the shared bootstrap PR; this PR alone does not qualify or enable vault exits. No live dispatch, signing or production activation was performed. Independent review/activation approvals remain in vaults-mommy #35.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant