Skip to content

perf(cluster): call graspologic_native.leiden() directly, skip graspologic's import chain - #3104

Open
Mohammad-Palla wants to merge 1 commit into
Graphify-Labs:v8from
Mohammad-Palla:perf/native-leiden
Open

perf(cluster): call graspologic_native.leiden() directly, skip graspologic's import chain#3104
Mohammad-Palla wants to merge 1 commit into
Graphify-Labs:v8from
Mohammad-Palla:perf/native-leiden

Conversation

@Mohammad-Palla

Copy link
Copy Markdown

Summary

  • _partition() currently reaches graspologic.partition.leiden() through the full graspologic package import, which eagerly pulls in graspologic.layoutsumappynndescent → numba JIT compilation — 7-19s per process, for a layout module the clustering path never calls.
  • graspologic_native (the PyO3/Rust binding graspologic.partition.leiden() itself wraps) imports in ~0.5ms. This adds a _native_leiden() adapter that calls it directly, falling back to the existing graspologic.partition.leiden path (then NetworkX Louvain) if graspologic_native isn't installed or the graph shape isn't supported.
  • Graphify's node IDs are guaranteed-unique strings by construction (extractors/resolution.py's _disambiguate_colliding_node_ids runs before the graph is built), so the string-collision case graspologic's _IdentityMapper guards against can't occur here; the adapter keeps a defensive check anyway.

Measured (Bun corpus, 161k nodes / 336k edges, golden-cache-restored before each run)

before after
cluster stage 27.3-28.6s 17.9s (~36-37% faster)
total build 155.4-159.8s 146.7s (~6-8% faster)

Correctness verified, not assumed

  • Feeding the identical graph to both code paths in the same process produces a byte-identical partition (exact canonical-form match, Adjusted Rand Index = 1.0).
  • Community counts, modularity, and singleton counts track within <0.3% across runs.
  • Full test suite: 4,735 passed, 233 skipped, 12 pre-existing unrelated failures (same set as before this change). tests/test_cluster.py: 11/11.
  • Adversarially tested degenerate inputs (empty graph, single node, resolution ≤ 0, self-loops, negative/zero/NaN edge weights, disconnected components) against both code paths side by side — all diverging behavior traced back to graspologic_native itself being called identically by both, not something this change introduces. Confirmed separately that Graphify's own extraction pipeline never produces a negative/zero/NaN edge weight (every extractor hardcodes weight: 1.0), so that pre-existing panic risk isn't reachable through normal use.

Note on run-to-run variance

Separate full-pipeline invocations show community-label drift between runs (Adjusted Rand Index ≈ 0.73-0.75), but this is pre-existing and identical on both code paths — traced to the already-filed #2817 / #1667 (hash-seed-sensitive iteration in cluster()'s post-processing splitting pass, downstream of _partition()), not introduced or worsened by this change.

Test plan

  • pytest tests/test_cluster.py — 11/11 passed
  • Full suite — no new failures vs. baseline
  • Same-input, same-process partition equivalence check (adapter vs. wrapper)
  • End-to-end timing on Bun corpus, golden-cache-restored, matched load
  • Adversarial edge-case testing (degenerate graphs, invalid resolution, bad weights)

…logic's import chain

graspologic's own package import (graspologic.layouts -> umap ->
pynndescent -> numba JIT) costs 7-19s per process and has nothing to do
with the leiden() call _partition() actually uses; graspologic_native
(the PyO3 binding it wraps) imports in ~0.5ms.

Measured on the Bun corpus (161k nodes / 336k edges), matched load:
cluster stage 27.3-28.6s -> 17.9s (~36-37% faster), total build
155.4-159.8s -> 146.7s (~6-8% faster). Verified correctness by feeding
the identical graph to both code paths in-process (exact partition
match, Adjusted Rand Index 1.0) and by confirming a full test-suite
pass with no regressions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant