feat(algo): add Leiden community detection with multi-table support - #43
feat(algo): add Leiden community detection with multi-table support#43chiangchenghsin-hash wants to merge 11 commits into
Conversation
…LE community detection"
…community detection"
…table support Full 3-phase algorithm (Local Moving + Refinement + Aggregation, Traag 2019) with heterogeneous graph support via flat index mapping (nodeOffsetBase). Includes BFS connectivity guarantee and singleton merge in Refinement phase.
|
Please benchmark against ParallelLeidenView in icebug. Also see: #30 |
|
Thanks for the reference — I wasn't aware of the icebug/NetworKit integration direction. A few questions on the benchmark request:
Happy to do the benchmark — just want to make sure I'm comparing apples to apples. |
|
While waiting on the icebug setup questions, here's what I can share about our implementation's characteristics: Algorithm: Traag, Waltman & van Eck (2019) — 3-phase: Local Moving + Refinement + Aggregation. The Refinement phase (the Leiden-unique part) uses BFS connectivity checks to guarantee well-connected communities, which Louvain does not. Key differentiator from #30: PR #30 notes "Single-node-table graphs for now (multi-table is a small follow-up)" for the icebug bridge. Our implementation already handles multi-table heterogeneous graphs natively — Performance characteristics (100 nodes, ~500 edges, Windows MSVC Release):
Architecture: Uses LadybugDB's existing What I can't do locally: Benchmark against icebug's ParallelLeidenView because:
What I can do: If you point me to a specific dataset (SNAP, etc.) and a Linux build environment with Arrow+OpenMP, I can run both implementations side by side and report wall-clock + modularity scores. Alternatively, if the plan is to eventually replace all hand-rolled algos with icebug bridges (per #30's roadmap), the benchmark question becomes "is our Leiden good enough to ship now, then get replaced by GDS_LEIDEN later?" — same coexistence pattern as PAGE_RANK vs GDS_PAGE_RANK. |
|
icebug repo: https://github.com/Ladybug-Memory/icebug |
Summary
Implements the Leiden community detection algorithm (Traag, Waltman & van Eck 2019) as a 3-phase algorithm: Local Moving + Refinement + Aggregation.
What's new
leiden()/le()function — Full Leiden algorithm with BFS connectivity guarantee in the Refinement phase and singleton merge to prevent oversplittingPROJECT_GRAPH('hg', ['T', 'M'], ['E']))Algorithm details
Phase 1 (Local Moving): Inherited from Louvain's parallel local moving — nodes move to neighboring communities to maximize modularity.
Phase 2 (Refinement — Leiden-unique): Each node starts in its own refined community. Constrained local moving within parent communities from Phase 1, with BFS connectivity check to reject moves that would disconnect the source community. Singletons merge into a random neighbor's refined community.
Phase 3 (Aggregation): Same as Louvain — renumber communities, build aggregated graph, repeat.
Multi-table fix
The original
tableFuncassumedgraph->getNodeTableIDs().size() == 1. For heterogeneous graphs,origNumNodeswas only the first table's row count, but neighbor offsets from the second table exceeded the array bounds. Fixed by:nodeOffsetBase[tableID] = cumulative offsettotalNodesacross all tablesflatIdx = nodeOffsetBase[neighbor.tableID] + neighbor.offsetTests
leiden.LeidenBasicleiden.LeidenAliasLEaliasleiden.LeidenDisconnectedleiden_heterogeneous.LeidenHeterogeneousAll 11 tests pass (Leiden 4 + upstream Louvain 6 + PageRank 1) on Windows 11 x64 MSVC.