perf(docker): reuse runtime layers and scope Maven builds

[lokidundun]

Purpose of the PR

• close perf(docker): track follow-up image build optimizations #3179

Main Changes

• Move runtime package installation before COPY --from=build in all four Dockerfiles, allowing dependency layers to survive application source changes.
• Keep configuration edits that depend on copied files after the corresponding COPY.
• Build the Server, PD, and Store distribution modules with -pl and -am, reducing the reactor from 38 to 27 modules while retaining required dependencies.
• Preserve existing runtime packages, configurable Maven arguments, and identical shared Maven build stages across all four Dockerfiles.

The existing single-job Bake flow and QEMU-based ARM64 build remain in place.

Verifying these changes

• Trivial rework / code cleanup without any test coverage. (No Need)
• Already covered by existing checks: multi-platform image inspection, Compose integration, Gremlin CRUD, and Standalone smoke tests.
• Additional verification: before/after benchmarks and distribution/image content comparisons.

Runtime dependency layer benchmark

Both variants used the same GitHub-hosted Ubuntu runner class and separate GHCR registry caches. After seeding each cache, identical configuration comments were added to trigger distribution changes and a full Maven rebuild.

Scenario / metric	Before	After	Reduction
Initial cache seed: image build	552 s	510 s	42 s (7.6%)
Initial cache seed: complete job	704 s	669 s	35 s (5.0%)
Source change: image build	601 s	461 s	140 s (23.3%)
Source change: complete job	755 s	602 s	153 s (20.3%)

Before the change, the four ARM64 runtime package installation steps ran under QEMU and took 150.6–160.5 seconds each. After the change, all eight runtime dependency layers across amd64 and arm64 were restored from registry cache in 2.0–9.3 seconds per layer, without rerunning package installation.

Cache export was measured separately:

Cache export vertex	Before	After
Shared build cache	205.2 s	165.7 s
PD	87.8 s	5.5 s
Store	56.1 s	4.9 s
HStore Server	83.3 s	6.6 s
Standalone Server	40.3 s	8.1 s

These BuildKit vertices overlap; their durations must not be added together or treated as independent end-to-end savings.

Maven reactor benchmark

This separate experiment compared the full reactor with the scoped reactor using the same source and existing runtime layer optimization.

Metric	Full reactor	Scoped reactor	Reduction
Reactor modules	38	27	11 modules
Maven-reported build time	270 s	176 s	94 s (34.8%)
Distribution build, cache and local export	424 s	269 s	155 s (36.6%)
Complete distribution job, including artifact upload	492 s	351 s	141 s (28.7%)

The distribution-job timings above are not complete image publication timings.

Correctness checks

• Compared 391 distribution files, including permissions and normalized nested JAR contents.
• Compared all four images on both amd64 and arm64: 780 records per architecture covering application files, installed packages, and image configuration, with no missing, added, or changed records.
• JAR comparisons normalized ZIP timestamps, compression, entry order, and specified build-time metadata; this verifies effective content equivalence rather than byte-identical archives.
• The runtime-layer CI built and published all four images with both architectures, passing local platform inspection, Compose, Gremlin CRUD, Standalone smoke tests, and published manifest checks. Functional checks in this flow ran on amd64.
• A separate native ARM experiment passed functional checks on both architectures and final multi-platform publication. That experimental workflow is not included in this PR.

Successful CI runs

Validation	Result	CI run
Runtime layer benchmark, multi-platform build and publication	All 4 jobs passed	33950351952
Full/scoped distribution benchmark, content comparison, and native architecture validation	All 6 jobs passed	33953324289
Full/scoped image content comparison on amd64 and arm64	Both jobs passed	33953049148

These are fork validation runs for the implementation approach. The Maven experiments enabled the same module selection through MAVEN_ARGS; this PR places that selection directly in the Dockerfiles. These runs do not replace CI on the final PR commit.

Measurements are single-run observations using GHCR, not the official Docker Hub publication environment. Savings from separate experiments are not additive.

Commands for local verification

Run from the repository root using Bash.

Build the selected distributions:

mvn install \
  -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \
  -am -e -B -ntp \
  -Dmaven.test.skip=true \
  -Dmaven.javadoc.skip=true

Inspect the shared Bake configuration:

docker buildx bake -f docker/bake.hcl --print

Build all four images through the shared Bake flow:

docker buildx bake -f docker/bake.hcl --progress=plain

The default Bake targets include amd64 and arm64. Local multi-platform loading requires a compatible builder and the containerd image store; building ARM64 on an amd64 host also requires emulation.

Check a direct Dockerfile build:

docker buildx build \
  --platform linux/amd64 \
  -f hugegraph-server/Dockerfile \
  -t hugegraph-standalone:pr-check \
  --load .

The commands above are build checks; the linked CI runs provide the integration, content-comparison, and publication validation.

Does this PR potentially affect the following parts?

• Dependencies
• Modify configurations
• The public API
• Other affects: Docker build layer caching and Maven reactor selection.
• Nope

Documentation Status

• Doc - TODO
• Doc - Done
• Doc - No Need

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.78%. Comparing base (3681148) to head (304bea1).

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3194   +/-   ##
=========================================
  Coverage     37.77%   37.78%           
  Complexity     6560     6560           
=========================================
  Files           800      800           
  Lines         68960    68960           
  Branches       9166     9166           
=========================================
+ Hits          26052    26054    +2     
  Misses        39841    39841           
+ Partials       3067     3065    -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

Blocking: no. Summary: Both mechanisms hold up. The four build stages stay byte-identical, so the shared Bake cache and the existing docker-bake-check guard still work, and the -pl ... -am scoping is content-safe because all three assembly descriptors are dependency-driven rather than module-driven. Four non-blocking points: the runtime apt layer now refreshes only when the base image digest changes, the reactor scope is no longer settable through MAVEN_ARGS, .dockerignore was not narrowed to match the new scope, and Dockerfile-hstore gained an avoidable layer. Evidence: git diff origin/master...304bea1; lines 20-33 of all four Dockerfiles hash identically; the server, pd and store assembly descriptors contain no <moduleSet>, so -am builds exactly the closure they consume; .github/workflows/cluster-test-ci.yml still runs an unfiltered full-reactor mvn clean package on every pull request, so compile coverage of the 11 dropped modules is retained.

[bitflicker64]

⚠️ With the apt layer now ahead of this COPY, its cache key no longer includes the application source, so the layer is reused until the eclipse-temurin:11-jre-jammy digest changes. What is lost is the per-source-change reinstall: rebuilding the same commit already hit cache before, but any source change used to force a fresh apt-get install, and now nothing short of a base image update does.

This only bites on a registry-cache flow, and that flow is out of tree: docker/bake.hcl gates every cache-to behind EXPORT_CACHE, which defaults to false, and no workflow here runs bake to build or push. The PR's own numbers show the effect, with eight runtime layers restored in 2.0-9.3 s "without rerunning package installation".

Could you add a documented way to force a reinstall, for example ARG RUNTIME_DEPS_EPOCH=1 immediately before the apt block in all four files, bumped when packages need refreshing? --no-cache-filter is not an option as things stand, since the runtime stage has no AS name. A short note on the cache behaviour in docker/README.md would help whoever operates the publish flow.

[bitflicker64]

🧹 The module list is hardcoded ahead of $MAVEN_ARGS, which is the only Maven knob docker/bake.hcl exposes. Maven accumulates repeated -pl values rather than letting a later one replace an earlier one, so MAVEN_ARGS can still add modules or drop them with a ! prefix, but it can no longer set the scope outright. The PR description notes that the fork experiments "enabled the same module selection through MAVEN_ARGS"; that route closes here.

Drift between the four copies is already covered by the docker-bake-check job in .github/workflows/docker-build-ci.yml, so this is only about overridability.

Suggested change: hoist the list into a build arg beside the existing ARG MAVEN_ARGS and pass it through _common.args in docker/bake.hcl.

ARG MAVEN_PROJECTS="hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist"

with the command becoming mvn install -pl "$MAVEN_PROJECTS" -am $MAVEN_ARGS .... All four files stay byte-identical, so the CI identity check still passes.

[bitflicker64]

🧹 The reactor is scoped but the build context is not, so the caching win is smaller than the benchmark suggests. .dockerignore at head excludes only build output, archives, IDE and OS files, .git, .github, **/*.md and the compose files, so hugegraph-test, hg-pd-test, hg-store-test, hugegraph-cluster-test/**, hugegraph-example, hg-pd-cli, hg-store-cli and install-dist all still land in the context and still feed the COPY . . cache key on line 25. Editing any of them therefore invalidates this shared build stage and pays for a full scoped Maven run, for modules the build no longer compiles.

A follow-up rather than a change here, but worth recording. One caveat for whoever picks it up: these directories cannot be ignored wholesale, because the root pom lists them in <modules> and Maven fails when a listed module's pom.xml is absent, so only their src/ subtrees can be excluded.

[bitflicker64]

🧹 Both of these RUNs edit only the tree copied on line 62, so they are invalidated together and the split just costs an image layer. This is the one file where that layer is avoidable: the plain server Dockerfile also gained a standalone sed layer, but it has a single edit with nothing to fold into.

Suggested change

	RUN cd /hugegraph-server/conf/graphs \
	&& rm hugegraph.properties && mv hstore.properties.template hugegraph.properties
	RUN sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties
	RUN cd /hugegraph-server/conf/graphs \
	&& rm hugegraph.properties && mv hstore.properties.template hugegraph.properties \
	&& cd /hugegraph-server \
	&& sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties

The comment on line 63 then covers only part of what the merged RUN does, so it is worth widening at the same time.