Skip to content

Reconcile bounded scheduler operations after transient RPC failures - #523

Open
lilydoar wants to merge 2 commits into
lilydoar/retry-probe-timingfrom
lilydoar/scheduler-operation-recovery
Open

lilydoar wants to merge 2 commits into
lilydoar/retry-probe-timingfrom
lilydoar/scheduler-operation-recovery

Conversation

@lilydoar

@lilydoar lilydoar commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Add one default-off recover-schedule-operations option for Create, Describe, Update and Delete. Recovery retains the schedule's identity and original window, observes uncertain effects, and fails when the outcome cannot be established.

Review: As a single unit in stack order.

Why

Retrying a whole iteration creates new schedule IDs and repeats unrelated work. A read-only retry cannot recover SDK Update's internal Describe or distinguish a committed write with a lost response from a write that never took effect.

Details worth reviewing

Merge after #527, including #518's SDK prerequisite. Each operation has a shared four-logical-call/five-second budget, including its initial call and probes; SDK retries are additional physical calls within that same context. Create/read/update also retain the original EndAt. These opt-in limits are candidate policy, not measured FI qualification thresholds.

A fresh State.Note witness establishes an uncertain update's effect; ambiguous updates are never replayed. Ownership assumes Omes' unique IDs are not reused by another writer; Delete has no atomic replacement fence. Existing NotFound compatibility is preserved without claiming an acknowledged mutation. This replaces the unpublished read-only retry experiment.

@lilydoar lilydoar changed the title lilydoar/scheduler operation recovery Retry scheduler reads on RPC deadlines when enabled Oct 22, 2026
@lilydoar
lilydoar marked this pull request as ready for review October 26, 2026 20:56
@lilydoar
lilydoar force-pushed the lilydoar/scheduler-operation-recovery branch from c55c9f1 to b375bd7 Compare October 26, 2026 20:56
@lilydoar
lilydoar requested review from a team as code owners October 26, 2026 20:56
@lilydoar
lilydoar changed the base branch from lilydoar/healthy-heartbeat-activities to lilydoar/retry-probe-timing October 26, 2026 20:57
@lilydoar
lilydoar added this pull request to stack #529 October 26, 2026 20:57
@lilydoar lilydoar changed the title Retry scheduler reads on RPC deadlines when enabled Reconcile bounded scheduler operations after transient RPC failures Oct 26, 2026
@lilydoar
lilydoar force-pushed the lilydoar/scheduler-operation-recovery branch 2 times, most recently from 5f58433 to d938a46 Compare October 26, 2026 21:16
@lilydoar
lilydoar force-pushed the lilydoar/scheduler-operation-recovery branch from d938a46 to 5642b27 Compare October 27, 2026 20:55
@lilydoar
lilydoar force-pushed the lilydoar/scheduler-operation-recovery branch from 5642b27 to c5f2825 Compare October 27, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant