RAG Pipeline Architecture for Research Agents

Research agents have a built-in expiration date, and nobody tells them when it hits. Every model is trained on a snapshot of the internet up to a certain point, and after that point, it knows nothing. Ask it about yesterday's regulatory filing, last week's product launch, or a paper published this morning: it either goes quiet or makes something up that sounds right.

Why research agents cannot rely on parametric memory

The gap isn't a bug to patch or a weakness to train away. Models are built by freezing a huge amount of text into a set of weights at one point in time, and research agents are asked to answer questions about a world that keeps moving long after that freeze happens. Those are two different jobs, and no amount of additional training closes the distance permanently, because the clock never stops for the second one. A model trained last spring will always be a model trained last spring. The world it's asked about won't hold still and wait for the next release.

This matters more for research agents than for almost any other kind of AI product, because research is, by definition, about finding out what's true right now. A chatbot that writes birthday poems doesn't need to know this morning's headlines. An agent tasked with summarizing a regulatory change or checking a competitor's current pricing does, and if it can't get that information, it either refuses to answer or quietly invents one. Retrieval exists to solve exactly that problem: pull in documents at the moment someone asks a question, instead of depending only on what got baked into the model months earlier.

Static RAG and the limitation it creates

Retrieval-augmented generation, or RAG, was the first real answer to the stale training data that leaves a model unable to speak to events after its cutoff. The classic setup takes a pile of documents, breaks them into chunks, turns each chunk into a vector, and stores all of that in a database. When someone asks a question, the system finds the chunks that look most similar to the question and hands those to the model as grounding material, rather than relying on whatever the model happened to memorize during training.

For a lot of use cases, that setup works well. HR policies, internal product docs, compliance archives: these don't change every day, so an index built last month is probably still accurate today. The gap between index and subject matter widens once the subject matter starts moving faster than the index does, and it breaks in three distinct ways.

First, there's staleness. The index is a photograph of the world taken at crawl time, and the agent has no way to know the photograph is old. It will cite last Tuesday's competitor pricing with total confidence, even if that price changed yesterday, because nothing in the system flags the context as outdated. Second, there's the overhead of just keeping the thing current: someone has to build pipelines that monitor sources, re-crawl pages, re-chunk the new text, re-embed it, and push updates into the store, and that infrastructure is expensive enough that most teams fall behind on it, which only widens the gap between what the index says and what's actually true. Third, there are plain coverage gaps. The database only knows what someone already found and indexed, so a question about a regulatory announcement from this morning or a price change from an hour ago comes back empty, and the model is left to either guess or admit defeat.

The cost of maintaining vector database infrastructure confirms the architecture's limits. A recent breakdown of vector database pricing found managed instances starting around $70 a month and climbing into the thousands, and plenty of teams end up spending more engineering hours maintaining retrieval infrastructure than improving the actual generation logic. Static RAG moves the knowledge ceiling from training time up to crawl time, which is real progress, but for a research agent answering questions about a world that never stops moving, any fixed ceiling at all disqualifies the approach.

What live web retrieval changes about the architecture

Live web retrieval removes the ceiling instead of just raising it. Instead of querying a stored vector database, the retrieval step queries the open web directly at the moment the question gets asked. The system stops asking "what did we store" and starts asking "what exists right now," and that's a difference in kind, not just degree.

A purpose-built search API is doing something fairly different from a plain scraper here. A scraper hands back the full mess of a web page (navigation bars, ad units, sidebar links, all the boilerplate that competes for space in the model's context window) while a search API built for this purpose returns clean, structured content: ranked excerpts, publication dates, and source links, already in the shape a model can use. That distinction is what makes live retrieval practical rather than just theoretically possible.

There's research backing up the idea that this kind of lighter-weight search holds up against the heavier vector approach. Studies on agentic keyword search have found it reaches over 90% of vector-RAG's performance without needing a standing vector database at all, and live web retrieval takes that same logic and applies it to the entire public internet instead of one curated corpus. The rest of this piece walks through how that gets built in practice: a search step, an extraction step, and a step that assembles everything into context the model can actually use.

The three-layer pipeline: search, extract, and context assembly

A live web RAG pipeline runs on three layers: search, extract, and context assembly, and each one replaces a piece of the old vector pipeline with something simpler and permanently current.

Layer one, search or source discovery, is where most pipelines quietly fall apart. A model can't ground an answer in a page it never found, so this layer sets the ceiling for everything that happens after it. This layer turns the intent behind a question into a set of candidate URLs, usually through a search API rather than by guessing which domains might have the answer. Good semantic search APIs take a plain-language objective and handle the query construction, result ranking, and excerpt pulling on their own, so nobody has to hand-craft keyword strings. Location matters here too: pricing, local regulations, and regional product availability all differ by geography, and a search tool without geotargeting will return results nobody in that location would actually see. Doing some query cleanup before this step (stripping filler words, expanding abbreviations, catching signals that someone wants recent information, pulling out named entities) measurably improves the set of URLs that come back.

Layer two is extraction: fetching the page and cleaning it up. Raw HTML is filled with token-heavy noise: navigation menus, footer links, inline SVGs, and DOM structure, none of which answers the question, and feeding it straight into an embedding model wastes context window space and compute budget on that structural boilerplate. Converting HTML into Markdown is the fix that matters most here. It strips out the non-essential elements before chunking even starts and cuts token use significantly, and Cloudflare built a feature called "Markdown for Agents" specifically to do this kind of stripping before content reaches an AI system. GPT-based table extraction tests found Markdown outperforming raw HTML tables, and one 2026 analysis found RAG pipelines seeing up to a 35% accuracy improvement when fed Markdown instead of raw HTML, a measurable gain and not just a tidier-looking one. Extraction also has to deal with modern websites that lean on JavaScript frameworks or bot detection, so the layer needs to render pages properly and pull the fully loaded content rather than an empty shell. Some search APIs now fold the search and extract steps into one call, returning ranked URLs and the cleaned page content together as Markdown or structured JSON, which removes a separate scraping step most pipelines would otherwise need to build and maintain. Once the content is in Markdown, chunking gets smarter too: splitting at Markdown headers keeps each chunk as one complete idea instead of slicing a sentence in half, and semantic chunking goes a step further, comparing sentence embeddings and splitting wherever the topic actually shifts, so the chunk a model receives still carries the context it needs.

Layer three is context assembly. This is where the system takes the top-ranked chunks and builds the actual prompt: the user's question, the retrieved excerpts, and instructions telling the model to answer only from what's provided and to cite where it came from. Citation isn't a nice-to-have for a research agent. Linking every claim back to a source URL is the one thing that separates a verifiable answer from a hallucination that happens to sound confident. Token count matters at this stage too: a typical article or product page, once converted from HTML to Markdown, can drop a great deal in token size, which lowers inference cost and leaves more room in the context window for material that actually matters.

Freshness TTL and the caching logic that separates live RAG from just-always-fetch

Deciding, for any given question, whether to fetch a fresh page or reuse a chunk that's already sitting in cache was always the hard part of live web RAG, and that decision runs on a per-topic freshness TTL, short for time-to-live.

One global timer for every type of content doesn't work. Fast-moving data like stock prices, sports scores, or breaking news needs a TTL measured in seconds or minutes, while stable reference material can sit for hours or weeks without going stale. Applying one setting across the board means the system either burns fetch budget re-checking pages that never change, or serves old information on pages that change constantly. The query-understanding work from layer one earns its keep again here: if the system detects that someone is asking about something time-sensitive, it should shorten or skip the TTL entirely and go fetch fresh content even when a cached version is sitting right there. On top of the timer, event-driven invalidation handles the cases a clock can't catch: a correction gets published, a link goes dead, a known fast-changing source updates, and the system should drop the relevant cached chunks the moment that happens rather than waiting out the TTL. Think of the TTL as the routine checkup and event-driven invalidation as the emergency room.

This caching layer is really the seventh stage bolted onto the three-layer pipeline described above: not a separate system, but the control logic that keeps the whole thing from re-fetching the entire web on every single question.

Routing between a vector database and live retrieval

None of this makes the vector database obsolete. The choice between static and live retrieval comes down to the kind of query, who owns the data, and how fast the answer needs to come back, and most production systems end up needing both approaches running side by side.

Static RAG still wins in three cases. Anything that needs retrieval under 100 milliseconds on a known, stable set of documents does better with a cached index, since a live fetch simply adds latency a stored vector lookup doesn't have. Private internal documents are another clear case, since the open web cannot serve content that was never published on it. And compliance work that depends on a fixed, auditable set of documents needs retrieval to stay deterministic, which a constantly-changing live web search can't guarantee. Live web retrieval, on the other hand, wins whenever the task needs real-time information, coverage beyond anything that was pre-indexed, or research that wanders into topics nobody thought to index in advance.

The mature approach routes between the two architectures based on signals in the query itself: words that suggest the person wants current information, confidence scores that come back low from a vector search, or a general sense of which topic area the question falls into. That routing logic doesn't need to be an elaborate classifier either. Those few signals are usually enough to make the call correctly most of the time.

How model routing controls retrieval costs in production

Running a live web RAG pipeline costs money in three places: calls to search and extraction APIs, the inference cost of processing whatever gets retrieved, and, for any proprietary documents still being indexed, the cost of keeping that index current. Of those three, the biggest lever teams actually control is which model handles which part of the job.

Re-indexing cost is a good place to see this play out. Batch re-indexing charges for the entire corpus every time it runs, whether or not anything in it actually changed, while streaming re-indexing only touches documents that changed, so cost tracks the rate of change instead of the size of the whole library. For a large corpus where only a small slice updates on any given day, that difference in ongoing embedding costs adds up fast. Model routing works on the same principle. Cheap, fast models are plenty capable of handling raw content cleanup and summarization, and the more expensive, higher-reasoning models should be saved for synthesis, the step where answer quality actually depends on how well the model reasons. Sending every task through the expensive model balloons costs for no gain in quality on the easy parts of the job.

The snippet-versus-full-page decision deserves the same kind of scrutiny. Snippets are cheap to pull, but they don't carry enough context for a research-grade answer. Full-page extraction costs more per call, but it skips the separate scraping step and avoids wasting tokens on raw HTML, so the fair comparison is snippet price plus a second scraping step against full-page price alone, and teams that skip that math often assume the cheaper-looking option is actually cheaper.

Tooling that supports the live retrieval pattern natively

Plenty of teams still build this with a SERP API and a separate extraction step bolted on: the SERP API hands back titles, URLs, snippets, and ranking info, and fetching the actual page content is a second job entirely, meaning someone has to crawl each URL, render any JavaScript, strip out the markup, and only then pass clean text to the model.

That two-step version works, but it carries costs that are easy to undercount going in. There are two API contracts to manage instead of one, two separate rate limits to watch, and two distinct sets of failure modes to debug when something breaks.

The newer pattern, where search and extraction are handled in a single call that returns clean, ready-to-use content directly, trades that complexity for something closer to one moving part instead of two. For a research agent that needs to stay current without a team dedicated to maintaining retrieval infrastructure, that consolidation is often the deciding factor between a system that scales and one that quietly accumulates maintenance debt.