Graph Clustering System
Problem
Millions of data records needed clustering. Standard algorithms — Louvain, Leiden, K-core, maximal cliques — produced poor business results and ran as opaque black boxes. If the process crashed midway, everything was lost.
Solution: Single-Sample K-Hop → Loop → Dedup
Complex Task → Simplify to Single-Sample → Loop → Dedup
1. Start from one sample, K-hop expand to find all connected nodes
2. Post-process the small subgraph (manageable, debuggable)
3. Extend to multi-sample by looping Step 1 + deduplicating overlaps
4. To classify all: run every sample through the loop
Why This Beats the Standard Approach
| Dimension | Standard (All-at-once) | Loop + Dedup |
|---|---|---|
| Progress | Black box | Sample N of M visible |
| Crash recovery | Restart from zero | Resume from last completed sample |
| Debugging | Can't inspect mid-process | Single-sample subgraphs are inspectable |
| Results | Poor business alignment | Edge rules are directly controllable |
Unsolved Challenge
This is fundamentally unsupervised learning — we have no ground truth labels. Validation relies on manual review of output reports. Without labeled data, GPUs, and training time, proper ML validation metrics (silhouette score, Davies-Bouldin index) remained out of reach. This is something I aim to address during my Master's.
Connection to LLM Analysis System
Both projects share the same design philosophy: don't solve the whole problem at once. Break it into the smallest processable unit, loop it, and merge the results. LLM Analysis: Leaf Agent × 100 records → recursive merge. Graph Clustering: Single-sample K-hop → loop + dedup. Same pattern, different domain.