Skip to content

doc(toolchain): sync spark-connector with master - #474

Open
bitflicker64 wants to merge 1 commit into
apache:masterfrom
bitflicker64:doc-sync/toolchain-spark-connector
Open

doc(toolchain): sync spark-connector with master#474
bitflicker64 wants to merge 1 commit into
apache:masterfrom
bitflicker64:doc-sync/toolchain-spark-connector

Conversation

@bitflicker64

Copy link
Copy Markdown
Contributor

Purpose of the PR

Sync the hugegraph-spark-connector docs (en + cn) with hugegraph-toolchain master (3b385c3d).

Every row below applies to both content/en/docs/quickstart/toolchain/hugegraph-spark-connector.md and content/cn/docs/quickstart/toolchain/hugegraph-spark-connector.md; the two pages are kept in sync.

Page What was wrong What changed Source on master
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md Overview did not say that reading is unsupported or which id strategies work Added a note: the table only implements SupportsWrite; CUSTOMIZE and PRIMARY_KEY supported, AUTOMATIC rejected hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/HGTable.scala:30, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/builder/VertexBuilder.java:79
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md Environment listed a vague "Spark 3.x" and "Scala 2.12" Pinned to Spark 3.2.2 (provided scope) and Scala 2.12.11 hugegraph-spark-connector/pom.xml:43, hugegraph-spark-connector/pom.xml:44, hugegraph-spark-connector/pom.xml:48
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md Build section never said what artifact the build produces Documented the *-jar-with-dependencies.jar output for spark-submit --jars hugegraph-spark-connector/pom.xml:207, hugegraph-spark-connector/pom.xml:210
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md Usage did not state that format must be the full class name, nor how to pass auth Added a sentence on the full class name (no DataSourceRegister service file under src/main/resources) and on username / token options hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:59, hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:82
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md No example for a PRIMARY_KEY vertex label or for ignored-fields Added section 4.4 (software vertex, no id, ignored-fields=ISBN) hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:96
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md No example of an edge whose source and target use different id strategies Added section 4.5 (created edge, customize source + primary key target) hugegraph-spark-connector/src/test/scala/org/apache/hugegraph/spark/connector/SinkExampleTest.scala:157
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md Nothing said what SaveMode.Overwrite does Added a note: overwrite returns a plain writer and does not delete existing data hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/writer/HGWriterBuilder.scala:35
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md Required options and key matching rules were not stated Added an intro to section 5: keys lowercased and trimmed; data-type, label required; source-name, target-name required for edges; marked those rows "Required" hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:73, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:84, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:89, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:93, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:97
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md host row did not mention that a scheme prefix is accepted Noted that host may carry an http:// or https:// prefix hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:47
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md username row omitted the fallback Noted that the graph name is used when username is unset hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:54
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md trust-store-file and trust-store-token rows omitted the https fallbacks Documented ${connector.home.path}/conf/hugegraph.truststore (system property required) and the hugegraph default password hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:65, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGClientHolder.java:72, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:32, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:52
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md id row did not say AUTOMATIC is unsupported Added that the AUTOMATIC id policy is not supported hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/builder/VertexBuilder.java:79
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md batch-size row did not say the batch is per Spark task Explained per-partition buffering and the flush at commit hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/writer/HGVertexWriter.scala:56, hugegraph-spark-connector/src/main/scala/org/apache/hugegraph/spark/connector/writer/HGEdgeWriter.scala:56
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md delimiter row claimed it also splits source-name and target-name; the code always splits those on , Limited delimiter to selected-fields and ignored-fields; noted , for source/target hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:228, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:240, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:248, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/options/HGOptions.java:260
content/{en,cn}/docs/quickstart/toolchain/hugegraph-spark-connector.md No notes on runtime behaviour and limits Added section 6 "Notes and Limitations": per-task HugeClient and LOADING mode, 128 byte vertex id limit, SET/LIST cardinality unsupported, date format yyyy-MM-dd HH:mm:ss in GMT+8, accepted boolean strings, empty vs null id handling; License moved to section 7 hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGLoadContext.java:66, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/client/HGLoadContext.java:81, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/builder/ElementBuilder.java:211, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/utils/DataTypeUtils.java:54, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:48, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/constant/Constants.java:50, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/utils/DataTypeUtils.java:35, hugegraph-spark-connector/src/main/java/org/apache/hugegraph/spark/connector/utils/DataTypeUtils.java:39

Pin Spark 3.2.2 / Scala 2.12.11, correct the delimiter scope, document the username and trust store fallbacks, mark required options, and add PRIMARY_KEY and mixed id strategy examples plus a notes and limitations section, in both en and cn.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant