When a construction technology client came to us with a simple-sounding problem - "we need to audit documents faster" - we quickly realized the underlying complexity was substantial. Their team of domain experts was spending 3 to 5 days per document batch, manually reviewing compliance clauses, flagging inconsistencies, and producing audit reports. The bottleneck wasn't knowledge. It was throughput.
This post walks through how we designed and built a production-grade multi-agent document intelligence system that brought that 3–5 day process down to under 20 minutes.
Why multi-agent, and why LangGraph?
The first design decision was whether this was a retrieval problem or an orchestration problem.
A naive RAG (Retrieval-Augmented Generation) approach - embed documents, retrieve relevant chunks, ask an LLM - would handle simple lookups. But auditing a construction document isn't a single lookup. It's a pipeline of specialized checks: clause extraction, compliance verification against regulatory standards, cross-document consistency checks, and risk flagging. Each of these is a different cognitive task, and forcing one LLM to do all of them in a single prompt produces mediocre results across the board.
The right architecture was a supervisor-agent model: one orchestrating agent that understands the audit workflow, delegating to specialized sub-agents that each do one thing well.
We chose LangGraph for orchestration because it gives you explicit control over agent state transitions. Unlike simple chain-based frameworks, LangGraph lets you model the audit workflow as a directed graph - with branching logic, parallel execution paths, and explicit checkpointing. When an agent fails or returns a low-confidence result, the supervisor can retry, escalate, or route to a fallback - behaviors that are awkward to express in linear chains.
The async execution layer: Celery + Redis
LangGraph handles coordination. But running 10 specialized agents concurrently across potentially hundreds of documents requires a distributed task execution layer.
We used Celery with Redis as the message broker. Each document submitted to the system spawns a Celery task group - a set of parallel tasks that fan out to the specialized agents. Celery's chord primitive is particularly useful here: it lets us define a callback that fires only when all parallel agents in a group have completed, collecting their results for the supervisor to synthesize.
Redis serves a second role beyond message brokering: it's the result cache. As each agent completes its analysis, results are pushed to Redis and immediately available for Server-Sent Events (SSE) streaming to the client. This is how we achieve real-time result delivery - users start seeing findings within 30 seconds of submission, while the full batch continues processing in the background.
AWS S3 document streaming
One of the less obvious bottlenecks we encountered early was document download time. Naively pulling a 200-page PDF from S3 to a local worker before processing it added 15–30 seconds of dead latency per document.
The solution was streaming: we use the boto3 streaming body to pipe document bytes directly into the parser without writing to disk. Combined with chunked embedding generation, this eliminated the download bottleneck almost entirely - reducing average document processing time by roughly 60%.
Vector DB integration
Each document chunk is embedded and stored in a vector database (we used Pinecone for this engagement, though the architecture supports any compatible vector store). The embeddings serve two purposes:
- Semantic retrieval: each specialized agent queries the vector store with task-specific prompts, retrieving the most relevant chunks for its analysis domain.
- Cross-document consistency: the compliance-checking agent embeds each extracted clause and queries across all documents in the batch to surface contradictions.
Results and lessons
The system went to production after 8 weeks of development and immediately cut document audit time from 3–5 days to under 20 minutes per batch. It currently processes up to 500 documents concurrently across distributed Celery workers.
A few lessons worth documenting:
Determinism matters in production. LLM outputs are probabilistic, but audit findings need to be reproducible. We implemented structured output schemas with Pydantic validation for every agent, and used temperature=0 for extraction tasks. Non-deterministic behavior in an audit system is a liability.
Failure handling is not optional. Every agent has explicit error handling, retry logic with exponential backoff, and a dead-letter queue for documents that fail after max retries. Silent failures in a document audit system would be worse than no system at all.
Streaming is a UX multiplier. The difference between "results appear progressively as agents complete" and "wait 20 minutes for a results page" was significant in user adoption. SSE streaming added about 3 days of development effort and meaningfully changed how the client's team experienced the system.
If you're evaluating a similar architecture for your document processing needs, we're happy to talk through the trade-offs. The approach generalizes well beyond construction documents to any domain where complex, multi-step document analysis is a bottleneck.