Skip to content

fix(consensus): open the store only after the execution engine is reachable - #467

Merged
ZhiyuCircle merged 1 commit into
circlefin:mainfrom
ZanCorDX:dx/avoid-db-repair-on-startup-failure
Sep 25, 2026
Merged

ZhiyuCircle merged 1 commit into
circlefin:mainfrom
ZanCorDX:dx/avoid-db-repair-on-startup-failure

Conversation

@ZanCorDX

Copy link
Copy Markdown
Contributor

Fixes #466

App::start() opens the redb store before connecting to the execution engine.
If the CL is terminated while it waits for the EL (up to 30 s of retries), no
SIGTERM handler is installed yet, so the process dies with the store open and
without the quick-repair commit that redb performs on Drop. The next start then
runs a full repair: 1h04 on a 103 GB testnet store.

This moves open_store to just before State::builder, its first use, so the
store is not open during the EL wait. No behavior change on the success path.

Reproduce (v0.8.0, EL stopped):

  1. systemctl start arc-malachite; wait for Database opened and the
    Failed to connect to Ethereum node retries.
  2. systemctl stop arc-malachite during the retries.
  3. Start the EL, then systemctl start arc-malachite:
    Database repair in progress: 0.00% instead of Database opened.

With this patch, step 1 logs no Opening database before the EL check, and
step 3 opens the store immediately and completes the handshake.

This is a minimal patch. The underlying issue is that a clean close depends on
the shutdown path: store.savepoint() is called explicitly on every exit path
(7 call sites across node.rs, app.rs, state.rs and coord_upgrade.rs) and
the SIGTERM handler is only installed after start() succeeds. A proper fix
would install the signal handler before the store is opened and funnel every
exit through a single teardown that stops the actors and closes the store, so
no path has to remember the savepoint.

@github-actions

Copy link
Copy Markdown
Contributor

Hi @ZanCorDX,

Thank you for your interest in contributing to Arc Node.

This PR has been automatically closed because you are not assigned to issue #466. We require contributors to be explicitly assigned to an issue before submitting a PR.

To contribute properly:

  1. Comment on issue CL: SIGTERM during startup while waiting for the EL forces a full redb repair on next start #466 requesting assignment
  2. Wait for maintainer approval
  3. Only submit a PR after you have been assigned

Please see our CONTRIBUTING.md for more details.

@github-actions github-actions Bot closed this Sep 24, 2026
@ZhiyuCircle ZhiyuCircle reopened this Sep 25, 2026
@ZhiyuCircle ZhiyuCircle added the pending-import Merged PR awaiting reverse-sync to upstream label Sep 25, 2026
@ZhiyuCircle
ZhiyuCircle merged commit 6e76402 into circlefin:main Sep 25, 2026
41 checks passed
@circle-github-action-bot circle-github-action-bot added import-conflict Import failed 3-way merge; manual resolution required and removed pending-import Merged PR awaiting reverse-sync to upstream labels Sep 26, 2026
@sergio-mena sergio-mena removed the import-conflict Import failed 3-way merge; manual resolution required label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CL: SIGTERM during startup while waiting for the EL forces a full redb repair on next start

4 participants