feat(rocksdb): add optional ToplingDB integration - #3134
Conversation
…th the HG default configuration
[pull] master from apache:master
[pull] master from apache:master
Co-authored-by: Hervé Boutemy <herve.boutemy@free.fr>
* refactor: Delete the package existing in java8 * chore(format): remove custom line breaks
dosubot
Bot
added
the
size:XL
- package component-local helpers for Server, PD, and Store - separate standard and Topling distribution contents - fail fast on invalid or incomplete Topling runtime selection - validate clean artifacts and real provider loading in CI
- add a discoverable ToplingDB documentation entry point - document standalone and HStore configuration paths - describe build, test, CI, and native runtime workflows - define volume isolation, monitoring, and rollback safety
dosubot
Bot
added
size:XXL
- close graph transactions after OLTP algorithm execution - keep failures visible while releasing traversal resources - cover success and failure paths in the unit suite
- build standard and amd64 Topling targets with provenance labels - isolate provider data paths with fail-atomic marker checks - add standalone and distributed Compose deployment paths - validate runtime packaging, lifecycle, and provider selection in CI
- forward TERM to the Server process - stop waiting after child becomes an unreaped zombie - preserve nonzero status for unexpected Server exits - cover process-group SIGTERM in the entrypoint regression
- destroy thread-local transactions after commit and rollback - clean Gremlin worker transactions before resetting context - keep multi-graph cleanup isolated and concurrency-safe - make RocksDB opened checks free of session side effects - cover cleanup failures and session state regressions
- preserve column families while truncating Topling data - balance temporary RocksDB session ownership - handle SIGTERM safely throughout container startup - keep schema transactions usable after intermediate commits
- preserve existing Ubuntu 24 required runtime checks - exercise Server, PD, and Store Topling runtimes on native Ubuntu 26 x64 - keep preview-runner or dependency failures non-blocking
- resolve Docker README and entrypoint test conflicts - retain Topling provider lifecycle and runtime guards - align standalone, HStore, dev, and HA Compose overlays - add standalone Topling overlay and render contracts - keep master Docker topology and authentication changes
- bind search filters to the original logical query - cover committed HStore IN and search regressions - keep sub-query input ordering behavior unchanged
- inject provider images, roots, and isolated volumes through env - reuse standalone, HStore, dev, and HA topology files - remove duplicated Topling-only Compose overlays - update render contracts, CI, and user/developer commands
- match the dev override image selected by the generic HStore topology - keep the Topling PD and Store parameter checks unchanged - preserve standalone and HA production image contracts
- replace obsolete overlay wording with environment injection - distinguish standalone and HStore provider ownership - keep the standard HStore Server guidance explicit
- point the Store data-path example at application-pd.yml - keep the PD application.yml example unchanged - align the quickstart prose with the component table
- document generic Compose rocksdb defaults - require provider, root, marker, and volume together - prevent image-only switches from masking data safety
- assert provider mismatch leaves the existing marker intact - assert unmarked data and files survive fail-fast validation - cover rollback safety at the entrypoint boundary
Document native Linux and macOS container targets. Explain linux/amd64 platform selection for Docker engines. Exclude native macOS execution from Topling acceptance scope. Keep standard and Topling topology guidance unchanged.
• align image defaults with standalone and HStore files • document the fixed Hubble image and pull policy • keep generic Topling parameter guidance accurate
- state that sample monitors are disabled by default\n- document explicit loopback-only enablement\n- align operations guidance with bundled YAML
- use numeric JDK version comparison in PD and Store\n- quote the PD process-count command substitution\n- avoid checksum wildcard matching in the helper
- exclude lib/topling from standard Server classpath scans\n- keep explicit preload as the only Topling classpath path\n- add fake-JVM regression for provider fallback isolation\n- document the standard-provider fallback contract
- exclude optional Topling jars from init-store scans - preserve explicit Topling preload classpath selection - extend fake-JVM regression coverage to init-store - cover all launcher jar-scan categories
- resolve library paths before excluding the Topling subtree - share canonical filtering across Server and init-store - cover direct-file and directory symlink aliases - document standard-provider classpath isolation
- retain jar-only init-store classpath scanning - stop following library directory symlink aliases - emit canonical regular-file classpath entries - cover non-jar and symlink fixtures in selector tests
- preserve classpath paths containing spaces - quote extension and plugin library scans - exercise launchers from a spaced fixture path
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The ToplingDB switch itself is small - one rocksdb.provider option and a truncate() branch - but it ships wrapped in ~400 lines of unrelated transaction and search-index bug fixes, a 9.5 MB timestamped SNAPSHOT JAR committed into the source tree, three byte-identical 162-line YAML files, and a second copy of download_and_verify; the single biggest simplification is deleting the vendored JAR in favour of the Maven repository this PR already adds to .github/configs/settings.xml. Evidence: net diff for head 18635f6 via gh api repos/apache/hugegraph/pulls/3134 (108 files, +6577/-207); diff of rocksdb_pd.yaml, rocksdb_store.yaml and toplingdb.yaml reports one differing line each (listening_ports); grep -n 'download_and_verify' hugegraph-store/hg-store-dist/src/assembly/static/bin/util.sh -> line 310, with the jemalloc arch-select already at start-hugegraph-store.sh:42-64; gh api repos/apache/hugegraph/contents/.../lib/topling/rocksdbjni-8.10.2-20260725.141011-1.jar?ref=18635f6 -> size: 9522363; grep -rn verify-rocksdb-provider.sh shows callers only in the three docker-entrypoint.sh files.
| <directory>${basedir}/src/assembly/static</directory> | ||
| <filtering>true</filtering> | ||
| <excludes> | ||
| <exclude>lib/topling/**</exclude> |
There was a problem hiding this comment.
src/assembly/static/lib/topling/rocksdbjni-8.10.2-20260725.141011-1.jar is 9.5 MB (size: 9522363 from the contents API) and it is a timestamped SNAPSHOT: unreproducible, and it lands in the source release.
You already added the repository it comes from, three files over in .github/configs/settings.xml:
<url>https://maven.pkg.github.com/hugegraph/toplingdb</url>Declare it as an optional/profile-scoped dependency, or curl it in prepare-topling.sh from that same URL. Then the JAR, this <excludes>, the matching <exclude>lib/topling/**</exclude> in assembly.xml, the 202-line LICENSE-rocksdbjni-8.10.2-SNAPSHOT.txt, and the ls -1 "$lib_dir"/rocksdbjni*.jar | sort -V | tail -n1 pick in common-topling.sh all go away in one move.
| private ConditionQuery constructSearchQuery(ConditionQuery query, MatchedIndex index) { | ||
| ConditionQuery newQuery = query; | ||
| ConditionQuery filterQuery = query; | ||
| Query rootQuery = query.rootOriginQuery(); |
There was a problem hiding this comment.
IN condition against the root query instead of the flattened EQ sub-query. It has nothing to do with rocksdb.provider.
Same for the transaction-leak set that rides along with it: HugeFactory.closeCurrentThreadTransactions, StandardHugeGraph.closeCurrentThreadTransaction, OltpTraverser.close(), the cleanup hook on HugeGraphAuthProxy.ContextTask, the two new HugeFactoryAuthProxy reflection registrations, and the new HugeFactoryTest / ContextTaskTest / OltpTraverserTest. That is roughly 400 lines, applied unconditionally to every provider, inside a PR whose description only claims an opt-in storage engine and states "No public HugeGraph API changes".
Split them into their own PR - they will get a review that is actually about them, and this one shrinks to the ToplingDB change. If any of it is a ToplingDB prerequisite, name which and why in the description; right now nothing in the PR explains it.
| http: | ||
| # normally parent path of db path | ||
| document_root: ./library/rocksdb_resource | ||
| listening_ports: '127.0.0.1:2012' |
There was a problem hiding this comment.
hugegraph-store/.../conf/rocksdb_store.yaml and hugegraph-server/.../conf/toplingdb.yaml are byte-identical apart from this one line:
$ diff rocksdb_pd.yaml toplingdb.yaml
22c22
< listening_ports: '127.0.0.1:2012'
---
> listening_ports: '127.0.0.1:2011'
(rocksdb_store.yaml differs only by 2013.) That is 486 new lines to vary a port three ways. The description says "component-specific configurations for Server, PD, and Store, including separate HTTP ports and JRaft column-family settings" - the column-family settings are identical too.
Ship one toplingdb.yaml. preload-topling.sh already computes the component and exports TOPLINGDB_EASY_MIGRATE_CONF; have it export the port alongside, or template the single file through the resource filtering the assembly already applies to static/. 324 lines deleted.
While you are in there: lines 24-30 are ToplingDB's upstream sample placeholders, shipped verbatim as production config in all three copies -
setenv:
StrSimpleEnvNameNotOverwrite: StringValue
OverwriteThisEnv:
value: force overwrite this env by overwrite true| fi | ||
| } | ||
|
|
||
| function download_and_verify() { |
There was a problem hiding this comment.
download_and_verify already exists at hugegraph-store/hg-store-dist/src/assembly/static/bin/util.sh:310, and the new <file> entry in hg-store-dist/src/assembly/descriptor/server-assembly.xml drops common-topling.sh into that same bin/. So on Store this definition silently overrides the one start-hugegraph-store.sh sourced ~30 lines earlier, and the two disagree on whether argument 3 is an MD5 or a SHA-256.
The copy is also the weaker of the two. The existing one downloads to mktemp and mv -f, with a comment saying why:
# mktemp, not $$: a PID is shared by concurrent background subshells.This one does rm -f "$filepath" and then curl -o "$filepath" - the exact non-atomic shape that comment was added to remove.
Source util.sh and teach the existing function a sha256 mode, instead of shipping a second implementation next to it. Same for download_and_setup_jemalloc below: start-hugegraph-store.sh:42-64 already does the arch-select, the download and the LD_PRELOAD, and your version short-circuits on LD_PRELOAD anyway, so on Store it is 40 lines that never run.
| PROVIDER="$2" | ||
| DATA_PATHS="$3" | ||
| ENFORCE="${4:-false}" | ||
| MARKER_NAME=".hugegraph-rocksdb-provider" |
There was a problem hiding this comment.
flock plus /proc/self/fd pinning plus two stat -Lc '%d:%i' TOCTOU re-checks defends against someone racing the data directory. That is not the failure being prevented; the failure is an operator flipping one line in a config file. And it buys Linux-plus-GNU-coreutils-only for a feature that is already Linux x86_64 only in a different way.
More to the point, grep -rn verify-rocksdb-provider.sh finds callers in exactly three places: the Server, PD and Store docker-entrypoint.sh. A tarball install - bin/start-hugegraph-pd.sh, init-store.sh, the test harness - never runs it, so the guard is absent precisely where hand-edited config files live.
RocksDBStore.open() already reads RocksDBOptions.PROVIDER and is the one place the DB is actually opened. Write and compare the marker file under this.dataPath right there, next to the existing PROVIDER read. A few lines, every launch path covered, and this script plus the <file> entries added to both server-assembly.xml descriptors go away.
6 participants
Purpose of the PR
This PR introduces ToplingDB as an optional RocksDB-compatible storage engine
for HugeGraph.
ToplingDB is explicitly enabled through configuration. Standard RocksDB remains
the default, so existing deployments and data paths are unaffected unless
rocksdb.provider=toplingis configured.Main Changes
rocksdb.provider.standard RocksDB Java API.
separate HTTP ports and JRaft column-family settings.
LD_PRELOAD,LD_LIBRARY_PATH, andTOPLINGDB_EASY_MIGRATE_CONF.libaiocompatibility.documentation, troubleshooting guides, and security notes.
Compatibility
rocksdb.provider=toplingis configured.Verifying these changes
mvn install -pl hugegraph-struct -am -DskipTests mvn test -pl hugegraph-pd/hg-pd-test -ammvn test -pl hugegraph-store/hg-store-test -amToplingDB mode.
Does this PR potentially affect the following parts?
Documentation Status
Doc - TODODoc - DoneDoc - No Need