Skip to content

feat: persistent cluster state via etcd - #109

Open
halfcyan wants to merge 26 commits into
mainfrom
cypress/etcd
Open

halfcyan wants to merge 26 commits into
mainfrom
cypress/etcd

Conversation

@halfcyan

@halfcyan halfcyan commented Aug 1, 2026

Copy link
Copy Markdown
Member

@halfcyan

halfcyan commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

this might require you to update your rust version to 1.95 if you're not already on that btw

@halfcyan
halfcyan marked this pull request as ready for review August 18, 2026 00:20
@halfcyan
halfcyan requested review from TheHeroBrine422 and korewaChino and removed request for TheHeroBrine422 August 18, 2026 00:20
@halfcyan halfcyan linked an issue Aug 19, 2026 that may be closed by this pull request
9 tasks
@halfcyan
halfcyan enabled auto-merge (squash) September 8, 2026 05:01
…ecords on failed create, confirm VM deletion before removing durable state, agent no longer independently deletes manifests, add regression tests for pre-dispatch state
Comment thread .devcontainer/Dockerfile

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

protobuf-compiler was missing here; i don't remember why i changed this as part of this PR. but it needed to happen anyways

Comment thread .zed/settings.json

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-enable rust-analyzer macro diagnostics

Comment thread Containerfile

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install alpine's protoc package for cross-target container builds

Comment thread docs/cluster-state.md

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation on etcd key layout, versioned records, config fields, failure behavior, and why i decided to fail closed instead of using a non-durable local state on failure

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

introduce versioned durable state abstraction with etcd, atomic create/delete of VM, placement, health checks, retries, optional TLS, and tests

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update cache-cleanup expectations for durable intent and supply an in-memory state store to scheduler tests

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recover a given node's assigned VM manifests from durable state at startup,restore resource accounting, reattach to or recreate VM actors, validate actor startup, and wait for delete confirmation before deleting durable records

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add CH heartbeat checks, stop actors after repeated VMM failures, avoid destroying an already destroyed instance, return deletion failures to callers

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

detect an already running CH socket so a restarted agent can reattach instead of restarting a duplicate VMM. VMM deletion also requires confirmation before durable state gets removed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make VM deletion and heartbeat replies carry optional errors so callers can distinguish actor reachability from CH/VMM health

@halfcyan
halfcyan requested review from TheHeroBrine422 and removed request for TheHeroBrine422 September 28, 2026 17:11
Comment thread odorobo/src/actors/agent_actor.rs Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing code for durable VM shutdowns.

Comment thread odorobo/src/actors/scheduler_actor/handlers.rs Outdated
Comment thread odorobo/src/actors/scheduler_actor/handlers.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add etcd-backed durable cluster state

2 participants