RAG Pipeline Architecture for Research Agents

RAG pipelines used to be simple: grab a chunk, stuff it in a prompt, generate an answer. That model still works fine for narrow lookups against a stable, indexed corpus. It falls apart the moment a research agent has to synthesize across sources, deal with an ambiguous question, or pull from the live web instead of a tidy pre-built index.

A 2025 survey on agentic deep research found something worth sitting with: standard LLMs using basic keyword search scored below 10% on complex multi-hop research benchmarks. Not 40%. Not 25%. Below 10%. That gap is the whole reason this article exists.

The failure modes are predictable once you see them. Retrieval precision collapses when a query is even slightly ambiguous. Single-pass retrieval misses evidence that's scattered across multiple documents, because it only gets one shot to guess what's relevant. And static vector stores go stale, quietly, with zero built-in way to notice or fix it. A research agent has to plan, retrieve, check its own work, and revise, well beyond what a simple search-driven chatbot can manage. That's a different job, and it needs a different architecture.

The architectural layers every research agent pipeline needs

Production RAG pipelines that actually hold up under real research questions tend to run through five stages: query planning, parallel retrieval across vector and keyword search, reranking and fusion, structured data ingestion, and answer synthesis with self-critique.

The split that matters most is agentic versus non-agentic. Non-agentic RAG is a straight line: user asks, system retrieves once, model answers. Done. Agentic RAG is a loop. The model can go back and retrieve again mid-generation, based on what it just learned. That loop is the whole insight, honestly. Retrieval feeds reasoning, and reasoning reshapes the next retrieval query. Round and round it goes.

Every layer in that loop is a place things can quietly break. None of them are checkboxes. Each one is an engineering decision with a real tradeoff attached, and the rest of this piece treats them that way.

Query planning: turning a research prompt into a retrieval strategy

The lazy approach is to take the user's raw question and fire it straight at a search index. This falls apart fast for research tasks, because most real questions are compound, vague, or phrased nothing like the documents that contain the answer.

Better systems do query transformation instead. A few patterns show up again and again in production:

• Generating multiple reformulated versions of the same query (usually 3 to 5), so you're casting several nets shaped differently instead of one
• HyDE, or Hypothetical Document Embeddings: the model writes a fake answer first, embeds that, and searches with it. Turns out a made-up answer looks a lot more like the real answer, semantically, than the original question does.
• Sub-question decomposition, where a compound question gets broken into small, atomic pieces that can each be retrieved on their own

On top of that, prompting patterns like ReAct, Self-Ask, and Search-o1 let the model interleave generation with retrieval. It writes a bit, notices a gap, fires off a targeted sub-query, then keeps going.

None of this is free. Each rewriting step tacks on roughly 200 to 400 milliseconds. Worth knowing before you commit to five rewrites per query and wonder why your agent feels sluggish. The fix is that these rewrites are highly cacheable. If your users ask similar research questions repeatedly (and they do), cache the query plans. What comes out of this layer is a structured set of parallel retrieval intents, rather than a single search string.

Hybrid retrieval and reranking: why neither BM25 nor dense embeddings alone are enough

BM25 and dense embeddings solve different problems, and pretending one covers both is how retrieval quietly underperforms without anyone noticing why.

BM25 is old-school keyword matching, sparse and exact. It's great at rare terms, product codes, and named entities, the stuff dense embeddings tend to smooth over into "close enough." Dense embeddings go the other way: they catch synonymy and paraphrase, so a query about "reducing headcount" can still find a document that says "layoffs." Neither one wins outright.

Across BEIR, MTEB, and Anthropic's Contextual Retrieval benchmarks, the pattern holds up consistently: fuse BM25 and dense embeddings with Reciprocal Rank Fusion, and you beat either method running solo. Stack a cross-encoder reranker on top of that fused list, and hard evaluation sets show another 5 to 15 points of MRR. That's a real payoff for one extra inference step.

The cost is latency, roughly 50 to 150 milliseconds for the reranking pass. Managed rerankers from companies like Voyage and Cohere take the GPU overhead off your plate if you don't want to run that infrastructure yourself. For the index itself, HNSW is the go-to for low-latency approximate search, and IVF-PQ earns its keep when you need memory efficiency at large scale. The practical rule: hybrid plus RRF is your default, full stop. Add the cross-encoder when ambiguous or hard queries are actually the bottleneck, not before.

Fetching live web data: what happens when the answer isn't in the vector store

Research agents constantly hit questions where the answer just isn't sitting in any internal corpus. Pricing changes, regulatory updates, last week's news, what a competitor just announced. None of that lives in a vector store that was indexed three months ago.

The fix is architectural: treat the live web as a retrieval source the agent can call, same as it would call the vector store. That means the pipeline needs a search API returning structured, rankable results (not a pile of raw HTML), a fetching layer that can pull and clean arbitrary URLs on demand, and output formatted so the LLM can actually reason over it, meaning clean Markdown or structured JSON.

Tools in this space split into three camps. Output-first tools like Olostep, Firecrawl, and ScrapeGraphAI hand back clean Markdown or JSON directly, often with MCP integrations built in for agentic workflows. That's the right fit when your pipeline owns the reasoning layer and just needs clean input. Access-focused tools like ScraperAPI, ZenRows, and Scrapingdog handle the messy stuff, JavaScript rendering, anti-bot bypass, and assume you'll write your own parsing logic. Scale-focused tools handle millions of requests against heavily protected targets with dedicated SLAs, built for enterprise data operations running at real volume.

Infrastructure choice here isn't cosmetic. Proxyway's 2025 independent test found only four of eleven scraping APIs held success rates above 80% against well-protected targets. And the traffic itself is exploding: HUMAN Security's 2026 State of AI Traffic report clocked AI scraper traffic growing 597% from January to December 2025. The web is already being consumed at machine scale, and the sites on the other end are hardening in response.

Olostep fits into this picture by collapsing the toolkit into one API: search, scrape, crawl, map, batch, and monitor, all through a single endpoint instead of stitching together four separate vendors. Output comes back as clean Markdown or JSON, ready for an LLM to chew on without extra cleanup.

Structuring ingested web content for LLM reasoning

Raw HTML is basically hostile territory for an LLM. Nav bars, ad slots, inline scripts, boilerplate footers, all of it inflates your token count while diluting the actual signal you wanted.

Clean ingestion fixes this two ways. Markdown keeps the document's real structure (headings, lists, tables) without dragging along the HTML noise, and it's the format most models reason over best. Structured JSON works better for content with a predictable schema, things like product listings, job postings, or financial filings, letting you filter before the LLM ever sees the content.

A 2025 benchmark called NEXT-EVAL found LLMs hitting F1 scores above 0.95 on structured web extraction, but only when the input was formatted properly first. Read that carefully: the extraction layer feeding the model, not the model itself, has become the real bottleneck.

Chunking decisions stack on top of format choice. Chunk size and overlap shape both recall and how coherent the context ends up looking to the model. Semantic chunking, splitting at natural topic boundaries instead of blindly at token N, consistently beats fixed-token chunking on multi-hop tasks. And don't skip metadata: source URL, fetch timestamp, page title, and section heading all need to survive ingestion, because that's what lets you cite sources and filter by freshness later.

RAGFlow's 2025 review put it plainly: a solid, scalable ingestion pipeline is now treated as an indispensable part of a modern RAG engine, bolted on at the end far too often for comfort.

Graph-enhanced retrieval for multi-hop research questions

Some questions can't be answered by finding one chunk of text. They need you to chain facts together, find fact A, use it to figure out what to search for next, find fact B, and only then land on the answer.

Chunk-based retrieval struggles here because the bridge entity connecting two facts and the final answer almost never sit in the same document. The connection lives in the relationship between documents, not inside any single one of them.

That's the problem GraphRAG (from Microsoft Research, Edge et al., 2024, released under MIT license) was built to solve. It extracts entities and relationships out of documents and builds a knowledge graph, treating the corpus as a connected structure rather than a set of isolated documents. It runs the Leiden algorithm to detect hierarchical communities across that graph, then generates community summaries that act as retrieval units for big, cross-document questions. Graph neural network approaches like GNN-Ret and HopRAG push in a similar direction, and they've shown roughly 10% accuracy gains on 2WikiMQA, a benchmark built specifically for multi-hop reasoning.

None of this is free lunch. Building the graph adds real upfront cost and complexity before you get any payoff. So don't reach for GraphRAG by default. Save it for domains where entities are genuinely dense and relational, think financial research, legal discovery, biomedical literature, or competitive intelligence. Everywhere else, you're paying construction costs for a problem you don't have.

Closing the loop: self-critique, corrective retrieval, and hallucination control

Self-RAG is a pattern where the model writes a draft answer, then checks its own draft against the retrieved evidence, flags any claim that isn't actually supported, and kicks off another retrieval round if the answer doesn't hold up. It's the model grading its own homework, and actually failing itself when it deserves it.

CRAG, short for Corrective RAG, adds a step before generation even happens: a retrieval evaluator scores each document for relevance before it reaches the generator. Documents that score poorly get tossed, and instead of quietly polluting the context window, they trigger a corrective web search.

The payoff shows up in hallucination numbers. A June 2026 Carnegie Mellon preprint found hallucinations dropping from 14.1% to 4.9% on a 9,000-question financial-compliance dataset once agentic correction patterns were applied. That's a two-thirds cut, not a rounding error. It costs roughly 220 milliseconds of extra latency per corrective cycle, which is a small, knowable price for that kind of improvement. Separately, a May 2026 MLOps Community benchmark reported roughly 62% fewer hallucinations across 47 production deployments when agentic pipelines were paired with knowledge graphs, compared to naive setups.

A couple of implementation notes worth stealing directly: build explicit confidence thresholds into the critique step, so re-retrieval only fires when confidence actually drops below a set line (otherwise you'll get infinite loops that never converge). And log every query that triggers a corrective cycle. That log is the single most useful signal you'll have for improving the pipeline later.

Anthropic's multi-agent research system beat single-agent approaches by 90.2%, a gap wide enough to mark the ceiling between a system that checks its own work and one that doesn't.

Orchestration frameworks and how to choose among them

LangChain and LangGraph both hit version 1.0 in October 2025. LangChain is built for assembling agents fast. LangGraph is built for running them durably, with explicit state management, which matters a lot more once you're in production and things fail at 2am.

LlamaIndex earns its keep when retrieval quality is the actual bottleneck. It's got deep integration with vector stores, ingestion pipelines, and hybrid retrieval patterns baked in. A common production setup uses LlamaIndex for ingestion and retrieval, then LangGraph for agent control flow and state. The two compose well together, which is more than you can say for a lot of framework pairings.

Multi-agent orchestration is moving from research paper to production default. Specialized agents, one for query planning, one for retrieval, one for critique, one for synthesis, consistently outperform a single generalist agent trying to do everything at once. CrewAI and Microsoft AutoGen offer different tradeoffs here, mostly around how much role-definition verbosity you're willing to write versus how much flexibility you want at runtime.

The decision rule is fairly clean. Reach for LangGraph when you need durable state and your retrieval loop has to survive a failure and pick back up. Reach for LlamaIndex when ingestion and retrieval quality are your real engineering problem. Combine them when both matter, which, honestly, is most of the time. And on the web-retrieval side, tools like Olostep and Firecrawl that expose MCP servers let agents call live web retrieval as a native tool inside these frameworks, which cuts down the integration work considerably.

What production research agents actually look like end to end

Walk through a real research task and you can see every layer above snap into place.

Someone asks: "How has Company X's regulatory exposure changed since their last earnings call?" The query planner breaks that into sub-questions, generates a few reformulated versions, maybe fires off a HyDE-style hypothetical answer to widen the net. Parallel retrieval kicks in: vector search hits the internal corpus, BM25 catches exact regulatory terms and filing numbers that embeddings might blur past, and a live web call pulls anything published after the last earnings call, since that's obviously not in any static index.

Results get fused with RRF, reranked with a cross-encoder to push the sharpest documents to the top, and the messy stuff, HTML scraped fresh off a regulatory site, gets cleaned into Markdown with source URL and fetch timestamp attached. If the question involves tracing a relationship (a subsidiary tied to a new rule, say), a graph layer might supply the bridge entity that chunk-based search would've missed entirely.

The model drafts an answer, then critiques itself against the evidence it actually retrieved. Unsupported claims get flagged. If confidence drops below the threshold, it loops back for one more corrective retrieval before answering. All of it runs inside LangGraph for durable state, or a comparable orchestration layer, so if a step fails, the whole pipeline doesn't collapse, it just resumes.

A search-driven chatbot guesses once and hopes. A real research agent plans, checks its own work, and knows when to go back for more, and that difference is what all these layers add up to.