Skip to content

ci: streamline Rocky dependencies and improve build failure diagnostics - #6961

Open
3for wants to merge 8 commits into
tronprotocol:release_v4.8.3from
3for:fix/ci-gradle-only
Open

3for wants to merge 8 commits into
tronprotocol:release_v4.8.3from
3for:fix/ci-gradle-only

Conversation

@3for

@3for 3for commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

  • Reduce Rocky Linux build dependencies, use C.utf8, and configure JAVA_HOME explicitly.
  • Increase DNF download concurrency and disable weak dependency installation.
  • Capture Gradle console output while preserving build failure exit codes.
  • Upload **/logs/tron-test.log and ci-logs/*.log only on failure, with 7-day retention.
  • Disable automatic test retries while retaining the configuration as comments.
  • Remove the deleted multinode workflow from PR cancellation handling.

Why are these changes required?

Unnecessary packages increase CI setup time. Missing console logs make test failures harder to diagnose, while automatic retries can hide intermittent failures. Workflow cancellation should also reflect the workflows that still exist.

This PR has been tested by:

  • Unit Tests
  • Manual Testing

private BackupManager backupManager;

private Channel channel;
private volatile Channel channel;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question] What's the motivation for making channel volatile here? Reading the surrounding code, this appears to fix a close-vs-bind race: close() can set shutdown while bind() is still in flight and observe a null channel, and the post-bind guard at L79–82 then closes the late-bound channel. Two questions:

  1. Is this race reachable in production, or only in tests? The new BackupServerLifecycleTest#testCloseDuringBind suggests the failing scenario came from the test side. If this production change is primarily to make unit tests pass, is that the right trade-off — or would fixing the test lifecycle be cleaner, or should this be split into a separate fix: PR with its own justification? It also isn't mentioned in the PR title/body, which currently only describes CI changes.

  2. If the cross-thread visibility fix is warranted, should executor (~L35) be volatile as well? It follows the same pattern: written by initServer(), read unsynchronized by close(). A cross-thread close() can still observe a stale-null executor and skip shutdownAndAwaitTermination, leaking the non-daemon worker thread even though the channel is now closed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@warku123 Thanks for pointing this out. The two changes address different parts of shutdown: volatile channel publishes the bound channel to the closing thread, while the post-bind check handles shutdown occurring before that publication.

This was identified while investigating test failures, but the lifecycle is also used in production: TronNetDelegate.processBlock() calls initServer(), which submits binding asynchronously, and Spring invokes close() during bean destruction. The test makes that ordering deterministic; I do not have evidence of an observed production incident. Making the test wait for startup would avoid exercising the race without fixing the shutdown behavior.

Your concern about executor is valid. Its publication is not explicitly synchronized within this class, and the current test does not cover that because its latches and task submission provide synchronization. Making it volatile would address visibility, but would still allow initialization to create an executor after close() has already returned.

I propose moving this production change and its regression test into a separate fix: PR, with coordinated initialization/shutdown and coverage for close-before-init and concurrent init/close. That would also make the scope and justification explicit instead of leaving a lifecycle fix inside a CI-focused PR.

ScheduledExecutorService executor = Mockito.mock(ScheduledExecutorService.class);
Mockito.when(executor.submit(Mockito.any(Runnable.class)))
.thenReturn(CompletableFuture.completedFuture(null));
ReflectUtils.setFieldValue(service, "executor", executor);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should] Replacing the bean's executor with a mock here drops the only reference to the real ScheduledExecutorService created by EffectiveCheckService.init(), and it is never restored or shut down. The real non-daemon pool keeps running after this test, and when the Spring context is destroyed, EffectiveCheckService.close() ends up calling shutdownAndAwaitTermination on the mock instead of the real executor — so the actual pool leaks for the rest of the JVM.

Consider capturing the original executor before overwriting the field and restoring it in @After closeP2p() (or explicitly shutting the original down), so the bean lifecycle still manages the real pool.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@warku123 Thanks for flagging this. Addressed in 04905f4: the test now captures the original executor before installing the mock and restores it in a finally block, including when an assertion fails. This preserves the original bean state and leaves executor ownership with the service lifecycle.

For completeness, the original executor is null in the current fixture because EffectiveCheckService.init() is not invoked, so this addresses missing state restoration rather than a confirmed running-pool leak.

@halibobo1205

Copy link
Copy Markdown
Collaborator

This PR currently combines CI changes, test improvements, and a production code fix, while the title and description focus on CI. Could you please split it into three focused PRs?

  1. CI/build configuration: workflow changes, Rocky Linux dependencies, failure diagnostics, and Gradle test-retry configuration.
  2. Unit test improvements: test isolation, resource cleanup, flaky test fixes, and test utilities. 3. Production code fix: the BackupServer shutdown/bind race fix, together with its BackupServerLifecycleTest regression test.

Please keep regression tests with the production fix they validate. This split will make each change easier to review, validate, and revert independently. If the PRs depend on each other, please document the dependencies and intended merge order.

@3for

3for commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

This PR currently combines CI changes, test improvements, and a production code fix, while the title and description focus on CI. Could you please split it into three focused PRs?

  1. CI/build configuration: workflow changes, Rocky Linux dependencies, failure diagnostics, and Gradle test-retry configuration.
  2. Unit test improvements: test isolation, resource cleanup, flaky test fixes, and test utilities. 3. Production code fix: the BackupServer shutdown/bind race fix, together with its BackupServerLifecycleTest regression test.

Please keep regression tests with the production fix they validate. This split will make each change easier to review, validate, and revert independently. If the PRs depend on each other, please document the dependencies and intended merge order.

@halibobo1205 Thanks for the review. The original intent was to improve the CI setup and disable automatic test retries. While investigating CI failures on this PR, I added test isolation and resource cleanup improvements, along with the BackupServer shutdown/bind fix. That expanded the scope beyond the original title and description.

Although the BackupServer change came out of test stabilization, it changes runtime shutdown behavior. I agree that it should stay together with its regression test in a focused PR.

I’ll split the changes as suggested:

  1. CI/build configuration, diagnostics, and test-retry changes.
  2. General unit test isolation, cleanup, and stability improvements.
  3. The BackupServer fix together with BackupServerLifecycleTest.

I’ll update each PR’s title and description and document any dependencies and the intended merge order. The relevant test and production fixes should land before disabling retries.

- Remove Development Tools and unnecessary packages
- Install only JDK 8, git-core and zstd with weak dependencies disabled
- Use C.utf8 to avoid installing glibc-langpack-en
- Resolve and validate JAVA_HOME so Gradle no longer requires which
- Collect tron-test.log, rotated logs and JUnit XML across five PR build jobs
- Run artifact uploads even when preceding steps fail
- Use distinct artifact names per job and matrix configuration
- Retain artifacts for 7 days and warn when no files are found
Remove the stale integration-test-multinode.yml reference to avoid
unnecessary API requests when cancelling workflows for closed PRs.
- Save build, RocksDB test and coverage output with tee and plain console mode
- Use Bash pipefail to preserve failures when capturing stdout and stderr
- Include console logs and HTML test reports in diagnostic artifacts
- Preserve existing test retry and base coverage failure policies
Remove the test-retry plugin and retry configuration shared by test and
testWithRocksDb so test failures fail the task without retrying.
- Limit diagnostic artifacts to **/logs/tron-test.log and ci-logs/*.log
- Upload logs only when a preceding step fails
- Include base test failures tolerated by continue-on-error
- Keep the existing 7-day retention period
- Set DNF max_parallel_downloads to 10 for Rocky dependency installation
- Restore the test-retry plugin and configuration as comments, keeping retries disabled
@3for
3for force-pushed the fix/ci-gradle-only branch from 04905f4 to 1a1ea8c Compare September 18, 2026 09:02
@3for
3for changed the base branch from develop to release_v4.8.3 September 18, 2026 09:03
@3for

3for commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author
  1. CI/build configuration: workflow changes, Rocky Linux dependencies, failure diagnostics, and Gradle test-retry configuration.

@halibobo1205 To keep the scope of this PR focused, the flaky test issues are being addressed separately in #6974. This PR will remain focused on CI/build configuration: workflow changes, Rocky Linux dependencies, failure diagnostics, and Gradle test-retry configuration.

Since this PR removes automatic test retries, the current CI test failures are expected while the flaky tests are still being addressed. Once #6974 resolves those issues, we can rerun the CI for this PR.

Set DNF minrate=256k and timeout=30 to abort persistently slow
connections and allow fallback to another mirror.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants