Skip to content

test: fix timing races from most sleep-based async tests - #3632

Open
Alenar wants to merge 1 commit into
mainfrom
djo/test/fix-flaky-sleep-based-tests
Open

Alenar wants to merge 1 commit into
mainfrom
djo/test/fix-flaky-sleep-based-tests

Conversation

@Alenar

@Alenar Alenar commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Content

This PR includes fixes to tests that used real-clock sleeps to order tasks or check delays, which made them flaky on loaded CI machines.

This problem was identified for one the mithril-client feedback test by the following failing hydra run: https://ci.iog.io/build/15142234/nixlog/1. I then searched other potential offender in the code base.

Details:

  • client: the feedback tests now take turns through Notify handshakes instead of sleeps. They are renamed to *_task(s) since they use tasks, not threads.
  • relay: the repeater delay tests use a paused clock and explicit time::advance calls to check the message is not repeated before the delay ends and is repeated once it does.
    The other repeater tests use a shorter delay so they run faster.
  • aggregator: the signature processor stops once a registration is notified, with a timeout so the test can't hang, and the retry-delay test advances a paused clock to just before the delay ends.
    Both tests now fail if the processor stops on its own instead of passing silently.
  • aggregator: rename the wait_delay_on_error_in_seconds parameter to wait_delay_on_error since it is a Duration.

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • No new TODOs introduced

@Alenar Alenar self-assigned this Oct 15, 2026
@Alenar Alenar added the testing 🔁 Something related to tests label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Test Results

     5 files  ±0     226 suites  ±0   1h 0m 41s ⏱️ - 4m 4s
 3 998 tests ±0   3 998 ✅ ±0  0 💤 ±0  0 ❌ ±0 
12 635 runs  ±0  12 635 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 663b6a9. ± Comparison against base commit 4eb474c.

This pull request removes 3 and adds 3 tests. Note that renamed tests count towards both.
mithril-client ‑ feedback::tests::send_event_in_one_thread_and_receive_in_another_thread
mithril-client ‑ feedback::tests::send_event_multiple_thread
mithril-client ‑ feedback::tests::send_event_same_thread
mithril-client ‑ feedback::tests::send_event_in_one_task_and_receive_in_another_task
mithril-client ‑ feedback::tests::send_event_multiple_tasks
mithril-client ‑ feedback::tests::send_event_same_task

♻️ This comment has been updated with latest results.

@Alenar
Alenar deployed to testing-preview October 15, 2026 11:26 — with GitHub Actions Active
@Alenar
Alenar deployed to testing-2-preview October 15, 2026 11:26 — with GitHub Actions Active
@jpraynaud
jpraynaud requested a lite review from Copilot October 15, 2026 11:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A repeater test can pass its pre-delay assertion before the spawned task has run.

1 open finding
What changed in this PR

This PR makes async tests less dependent on real-clock sleeps, reducing timing-related failures on loaded CI machines.

Changes:

  • Replaces client feedback sleeps with task handshakes.
  • Uses paused clocks to test relay and aggregator delays, and shortens other relay test delays.
  • Renames an aggregator Duration parameter for clarity.
File Description
mithril-relay/​src/​repeater.rs Updates repeater timing tests.
mithril-relay/​Cargo.toml Enables Tokio test utilities.
mithril-client/​src/​feedback.rs Coordinates feedback tests with notifications.
mithril-aggregator/​src/​services/​signature_processor.rs Updates processor timing tests and parameter naming.

🧠 Review effort: Lite


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread mithril-relay/src/repeater.rs
Several tests used real-clock sleeps to order tasks or check delays,
which made them flaky on loaded CI machines.

- client: the feedback tests now take turns through `Notify` handshakes
  instead of sleeps. They are renamed to `*_task(s)` since they use
  tasks, not threads.
- relay: the repeater delay tests use a paused clock and explicit
  `time::advance` calls to check the message is not repeated before the
  delay ends and is repeated once it does. The other repeater tests use
  a shorter delay so they run faster.
- aggregator: the signature processor stops once a registration is
  notified, with a timeout so the test can't hang, and the retry-delay
  test advances a paused clock to just before the delay ends. Both tests
  now fail if the processor stops on its own instead of passing
  silently.
- aggregator: rename the `wait_delay_on_error_in_seconds` parameter to
  `wait_delay_on_error` since it is a `Duration`.
@Alenar
Alenar force-pushed the djo/test/fix-flaky-sleep-based-tests branch from 88e8982 to 663b6a9 Compare October 15, 2026 13:02
@Alenar
Alenar deployed to testing-2-preview October 15, 2026 13:19 — with GitHub Actions Active
@Alenar
Alenar deployed to testing-preview October 15, 2026 13:19 — with GitHub Actions Active

@turmelclem turmelclem left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@jpraynaud jpraynaud left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

This branch was successfully deployed

2 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing 🔁 Something related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants