LLM Data Analysis System

2026.04 — 2026.05 1 Person

LLM Multi-Agent Python Concurrency

                  ┌── Root Agent（Final Report）
                  │
            ┌─────┴─────┐
            │           │
       Middle Agent  Middle Agent  ...
            │           │
       ┌────┴────┐  ┌──┴───┐
      Leaf    Leaf  Leaf  Leaf  ...
    （~100 records each）

Problem

How do you feed millions of data records to an LLM with a limited context window? The naive approach — cram everything in — fails immediately. But slicing data into chunks loses cross-slice relationships.

Solution: Tree-Based Multi-Agent Architecture

Leaf Agents each process ~100 records — context stays small and manageable.
Middle Agents recursively compress Leaf outputs, preserving cross-slice connections.
Root Agent receives a distilled summary that fits in a single context window.

Key Innovation: Dual Pipeline

Heavy compression risks losing critical facts. The solution: split into two independent pipelines.

Pipeline A: Semantic Expert

Compresses and summarizes meaning. Routes through Middle → Root.

Pipeline B: Extraction Expert

Extracts key facts verbatim — no compression. Injected directly into the final report, bypassing Middle layers.

Hallucination Control: Regex + LLM Scoring

LLMs cannot guarantee verbatim extraction. The solution splits responsibilities:

Raw Data → Broad Regex (catch all candidates) → LLM scores each → High-score retained

Result: Zero information loss (regex guarantees recall). 90%+ noise reduction (LLM handles precision).

Concurrency: Three Iterations to Stability

Attempt A: Free-for-All Tokens (Failed)

100 tokens, Leaf and Middle compete freely → sustained high concurrency → upstream LLM server crashed.

Attempt B: Weighted Tokens (Partial)

Leaf = 1 token, Middle = 10 tokens. Better, but upstream instability remained.

Attempt C: Dynamic Token + Timeout Backoff (Adopted)

On timeout: stop dispatching → wait for running tasks → retry the timed-out task alone with full resources → resume. Timeouts were rare (caused by difficult data, not concurrency), so the pause was negligible.

Key Insights

Strict role separation: LLM compresses & scores. Scripts extract precisely. Humans remove noise. Never let LLM do precision work.: LLM compresses & scores. Scripts extract precisely. Humans remove noise. Never let LLM do precision work.
Single responsibility per agent: One agent, one job. Like OOP classes — split them and they won't get confused.: One agent, one job. Like OOP classes — split them and they won't get confused.
AI coding made fast iteration possible: Tested 5-6 concurrency strategies in one afternoon. Without AI, this would have taken weeks.: Tested 5-6 concurrency strategies in one afternoon. Without AI, this would have taken weeks.