[#1161] Keep bc-fips from seeding its DRBG from RDSEED in the test JVMs

[vharseko]

Fixes #1161

Problem

On the Linux build-maven legs the embedded test server sometimes fails to start: bc-fips 2.1.3 seeds its DRBG from the CPU's RDSEED instruction through its native libraries, and on a busy CI host RDSEED runs dry while the server generates its self-signed certificates (RDSEED persistently failed to produce entropy). The class that starts the server fails in @BeforeClass and the rest of it is skipped.

Change

Both test argLine values in the root pom.xml (the default one and the jdk17.options one) now carry -Dorg.bouncycastle.native.cpu_variant=java. With it bc-fips does not load its native libraries, and FipsDRBG.getDefaultEntropySourceProvider() falls back to SecureRandom.getInstanceStrong() from the JDK. The opendj-server-legacy failsafe configuration picks it up through @{argLine}.

The argLine does not cross a process boundary. AdsTrustStoreInstallTestCase and QuickSetupTestCase start the packaged setup, and ServerControllerTest starts start-ds through ServerController, which keeps the environment of the fork and drops only OPENDJ_JAVA_ARGS and CLASSPATH. Those servers generate their certificates through bc-fips too, so the opendj-server-legacy failsafe execution also sets JAVA_TOOL_OPTIONS to the same flag in <environmentVariables>, and every JVM started from the fork reads it.

Two pins:

• BcFipsNativeLibrariesOffTestCase (opendj-core) fails when either root argLine loses the flag. LDAPServer generates its key pairs with bc-fips in that module, and the module takes the root argLine as it is: the default one below JDK 17, the jdk17.options one from JDK 17 on.
• BcFipsNativeLibrariesOffTest (opendj-server-legacy) fails when the failsafe fork loses JAVA_TOOL_OPTIONS.

Only the test JVMs and the JVMs they start change. The product (start scripts, setup) keeps loading the native libraries, and the "Test on Unix FIPS" step in build.yml, which runs the packaged server outside Maven, still exercises that path.

Verification

• mvn help:evaluate -Dexpression=argLine shows the property with jdk17.options active, with it disabled, and in opendj-server-legacy.

• A small program that makes the same KeyPairGenerator call as Platform.newKeyPair(), run against bc-fips 2.1.3 in an eclipse-temurin:17 linux/amd64 container on a CPU with rdseed:

  	without the property	cpu_variant=java
  native status / variant	READY / avx	UNSUPPORTED / none
  native DRBG / NRBG	true / true	false / false
  entropy source	FipsDRBG$1 (NativeEntropySource, RDSEED)	BasicEntropySourceProvider (JDK)
  RSA 2048 key pair	OK	OK

• Reactor run (JDK 26): BcFipsNativeLibrariesOffTestCase 1/1, AdsTrustStoreInstallTestCase 6/6, ServerControllerTest 2/2, BcFipsNativeLibrariesOffTest 2/2. The JVM's Picked up JAVA_TOOL_OPTIONS line breaks nothing.

• The built package's setup and start-ds, run with the fork's environment: setup, start-ds and the server's logs/server.out all print Picked up JAVA_TOOL_OPTIONS: -Dorg.bouncycastle.native.cpu_variant=java, and the server generates ads-certificate.

• Mutants: flag removed from the default argLine → opendj-core pin red on JDK 11; removed from the jdk17.options argLine → red on JDK 26 (both green at the head); JAVA_TOOL_OPTIONS missing → BcFipsNativeLibrariesOffTest red.

The RDSEED failure itself cannot be reproduced on demand, so the CI legs are the end-to-end check.

Follow-up, not in this PR

bc-fips 2.1.4 (a release candidate, see bcgit/bc-java#2434, not yet on Maven Central) adds org.bouncycastle.native.rand=NONE, which turns off only the hardware RNG and keeps the native AES/SHA acceleration, and raises the RDSEED retry limit to 1500. That is the better fit for the product side (start scripts, setup) once it is released.

[maximthomas]

praise: The flag goes where the failing JVM is, and it does not slow the suite.

• Both test argLines carry it (pom.xml:68 and the jdk17.options one at pom.xml:744), and the comment at pom.xml:62-67 tells the next editor to keep them in step. opendj-server-legacy/pom.xml:1285 <argLine>@{argLine}</argLine> is the only override, so the failsafe fork gets it on JDK 11 and on JDK 17+.
• org.bouncycastle.native.cpu_variant / java match the constants and the "java support only" branch in bc-fips 2.1.3 NativeLoader.
• The Linux build-maven cells took 2h19m-2h40m at this head, against 2h11m-2h34m at the base: no measurable cost from dropping the native AES/SHA code.

---

question (non-blocking): Should the setup and server JVMs that AdsTrustStoreInstallTestCase starts also run without the bc-fips native libraries?

opendj-server-legacy/src/test/java/org/opends/quicksetup/AdsTrustStoreInstallTestCase.java:385, :276, :315; opendj-server-legacy/src/main/java/org/opends/quicksetup/util/ServerController.java:373

runSetup starts the packaged setup with new ProcessBuilder(args). The argLine property does not cross a process boundary, and nothing sets OPENDJ_JAVA_ARGS or JAVA_TOOL_OPTIONS for the child. ServerController.startServerViaAnotherProcess removes OPENDJ_JAVA_ARGS before start-ds anyway. testInstalledServerStartsOnTheProvisionedTrustStore needs that server to generate ads-certificate (asserted at :315) through BCFIPS with the native libraries loaded, which is the road of the #1161 occurrence. So that class can still go red with RDSEED persistently failed to produce entropy after "Fixes #1161" has closed the issue. The exposure is a handful of key generations per run, against hundreds in the fork. If the description's "Only the test JVMs change" was meant to leave these JVMs on the native path, saying that #1161 is fixed for the in-JVM road only would be enough. If not, JAVA_TOOL_OPTIONS survives ServerController's env edits and reaches both JVMs. The output asserts in this class use contains(), so the JVM's "Picked up JAVA_TOOL_OPTIONS" line does not break them:

    final ProcessBuilder builder = new ProcessBuilder(args).redirectErrorStream(true).redirectOutput(output);
    // ServerController drops OPENDJ_JAVA_ARGS before start-ds; JAVA_TOOL_OPTIONS reaches setup and the server it starts.
    builder.environment().put("JAVA_TOOL_OPTIONS", "-Dorg.bouncycastle.native.cpu_variant=java");
    final Process process = builder.start();

---

suggestion (non-blocking): Nothing fails if the flag is dropped from either argLine.

pom.xml:68, :744

The two argLines are kept in step by hand, and jdk17.options replaces the default one on JDK 17+. If a later edit loses the flag from one of them, every cell stays green until the rare RDSEED event comes back on that JDK. That mutant is the base, 4870ff3, which ran green on every Linux cell.

package org.opends.server;

import static org.testng.Assert.assertEquals;

import org.testng.annotations.Test;

/** Pins the test argLine flag that keeps bc-fips off RDSEED, see issue #1161. */
@SuppressWarnings("javadoc")
public class BcFipsTestJvmTestCase extends DirectoryServerTestCase
{
  @Test
  public void testJvmRunsBcFipsWithoutNativeLibraries()
  {
    assertEquals(System.getProperty("org.bouncycastle.native.cpu_variant"), "java",
        "the test argLine lost the bc-fips cpu_variant flag, see #1161");
  }
}

Pin: this case fails on the JDK 11 cell when the flag leaves pom.xml:68, and on the JDK 17+ cells when it leaves pom.xml:744.

[vharseko]

Thanks, both points are taken in 8607401 (the branch is also rebased onto the current master).

question (child JVMs): you are right, and it is wider than AdsTrustStoreInstallTestCase. TestUtilities.setupServer() (via QuickSetupTestCase) also starts the packaged setup through its own ProcessBuilder, and ServerControllerTest starts start-ds through the product ServerController.startServerViaAnotherProcess(). That ProcessBuilder is not in test code, so the snippet in the test class would not reach it. The server it starts generates ads-certificate on its first start as well.

So the flag now goes into the environment of the fork instead: the opendj-server-legacy failsafe execution sets JAVA_TOOL_OPTIONS=-Dorg.bouncycastle.native.cpu_variant=java in <environmentVariables>. ServerController keeps it (it drops only OPENDJ_JAVA_ARGS and CLASSPATH), nothing in the repository touches JAVA_TOOL_OPTIONS, and a test that starts a JVM later is covered without further edits. Checked:

• AdsTrustStoreInstallTestCase 6/6 and ServerControllerTest 2/2 with the variable set; the "Picked up JAVA_TOOL_OPTIONS" line breaks nothing (the _script-util.sh JVM checks send their output to /dev/null, StartReader only logs stderr lines);
• the built package's setup and start-ds, run with the fork's environment: setup, start-ds and the server's logs/server.out all print Picked up JAVA_TOOL_OPTIONS: -Dorg.bouncycastle.native.cpu_variant=java, and the server generates ads-certificate.

The description now says "the test JVMs and the JVMs they start".

suggestion (pin): taken, in two pins rather than one. With JAVA_TOOL_OPTIONS in the fork, the system property in opendj-server-legacy arrives through either road, so a property pin there stays green when an argLine loses the flag.

• BcFipsNativeLibrariesOffTestCase in opendj-core pins the argLines: LDAPServer generates its key pairs with bc-fips there, and the module takes the root argLine as it is. Mutants: the flag removed from pom.xml:68 → red on JDK 11; removed from the jdk17.options argLine → red on JDK 26; green at the head on both.
• BcFipsNativeLibrariesOffTest in opendj-server-legacy pins JAVA_TOOL_OPTIONS (and the property of the fork itself). Without the variable → red with "the failsafe fork lost JAVA_TOOL_OPTIONS".

[maximthomas]

praise: The second commit closes the road the argLine cannot reach, and both halves of the fix are pinned.

• opendj-server-legacy/pom.xml:1291-1293 hands the flag to setup and start-ds through JAVA_TOOL_OPTIONS. It is the one road that survives ServerController's env edits, which remove only OPENDJ_JAVA_ARGS and CLASSPATH.
• Both root argLines (pom.xml:69, :745) carry the flag, and BcFipsNativeLibrariesOffTestCase pins each one on its JDK range.