Multi-Step Query Planning in Research Agents

There is a class of question that breaks most retrieval systems quietly. Not with an error, not with a refusal. Just a confident, subtly incomplete answer assembled from whatever evidence happened to land in the top retrieved chunks. I have watched this pattern recur across production deployments: a system performs beautifully on evaluation sets built around lookup questions, then degrades the moment users ask something that requires reasoning across time, across documents, or across causal chains. The failure is architectural, and you cannot prompt-engineer your way out of it.

The standard retrieval-augmented generation pipeline, commonly referred to as a RAG pipeline, is elegant in its simplicity: embed the query, pull the top-k chunks, inject them into a prompt, generate. For questions with a single, locatable answer, this works. But a single query vector cannot carry the load of a multidimensional information need. It compresses that need into a point in embedding space and retrieves whatever is geometrically closest, which is the fundamental constraint of vector similarity search. When the right chunk is not in the top-k, there is no recovery mechanism. The system does not know it failed.

On MuSiQue, a benchmark constructed to require two to four reasoning hops while explicitly blocking shortcut strategies, single-hop retrieval achieves 44.6% recall. Less than half the evidence needed to answer the question even reaches the model. That number is not a tuning gap; it is a ceiling built into the architecture.

Query decomposition is the most commonly cited response, and the instinct is right, though the term undersells what is actually required. Breaking a complex question into simpler sub-questions, then reasoning across their combined outputs, is necessary but insufficient without sequencing. Planning also involves determining which sub-questions must be resolved before others can even be meaningfully posed, a step that Chain-of-Thought prompting alone does not enforce.

The systems that handle this well tend to organize around four interacting capabilities: query understanding and planning, which interprets what the user actually needs and selects a solving strategy before retrieval begins; tool and retrieval orchestration, which decides when and how to retrieve, and from which source type; multi-hop reasoning and decomposition, which breaks the task into sub-problems, executes them, and recombines results coherently; and self-reflection, the system's capacity to critique its own outputs, detect gaps in retrieved evidence, and trigger reformulation rather than proceeding on insufficient information. Self-reflection is the most consequential of the four in practice, and the most commonly underdesigned. That last point is not a minor observation; it is where most production failures I have encountered actually live.

The distinction between a naive pipeline and a planning agent is, at its core, about when control flow is determined. A fixed pipeline executes a predetermined sequence regardless of what it finds, while agentic RAG determines its control flow dynamically from intermediate results. A planning agent determines its control flow at runtime, using intermediate outputs to shape what comes next. That raises an important question: if the planning layer makes a wrong assumption early, what catches it? Self-reflection, but only if it is designed into the pipeline from the start, not bolted on after the fact.

Two decomposition modes recur in well-designed systems. Serial decomposition, the basis of iterative retrieval, applies when sub-questions are logically dependent: the answer to the first determines what the second question even is. Parallel decomposition applies when sub-questions are independent, allowing simultaneous retrieval and subsequent merging. Routing, deciding which source to query for a given sub-question, belongs to the planning layer as well. Whether a sub-question is best answered by a vector store, a structured database, a knowledge graph, or a live web search is a planning decision with real consequences, not a retrieval detail.

The main planning strategies and when each applies

Several planning frameworks have been proposed and studied, and the literature's tendency to present them as a ranked set of alternatives obscures the more useful observation: they solve different problems.

ReAct, which interleaves natural-language reasoning traces with tool calls and observations, is the most widely deployed. Its appeal is transparency: the system explicitly articulates its plan before each action, then revises based on what it observes. Its limitation is directional commitment. Once a reasoning path is underway, ReAct cannot backtrack; it tends toward locally reasonable but globally suboptimal solutions. It fits tasks where the next step can be determined from current state without requiring lookahead. That is a narrower domain than it sounds.

Tree of Thoughts organizes reasoning into a branching structure, explores multiple paths simultaneously, and prunes based on intermediate evaluations. The performance differential on tasks requiring lookahead can be striking: GPT-4 using Tree of Thoughts solved 74% of Game of 24 mathematical tasks, compared to 4% with Chain-of-Thought alone (Yao et al., 2023). That gap illustrates what structured exploration buys when early decisions heavily constrain later possibilities. The tradeoff is cost. Enumerating and evaluating branches is computationally expensive, and on tasks where early decisions are recoverable, that cost is difficult to justify.

Reflexion operates on a different axis entirely. Rather than branching within a single attempt, it stores textual reflections on prior failures in an episodic memory buffer and uses those reflections to adjust the next attempt. Improvement is trial-to-trial, not intra-chain, which makes it well-suited to tasks where failure modes are recoverable and learnable within a session. The risk is less obvious: if the agent's self-diagnosis is wrong, the next attempt compounds the error rather than correcting it. The mechanism depends entirely on the quality of the introspective step, which is not guaranteed and not inspectable in every case.

Plan-and-Execute lays out the full task structure upfront and then executes step by step. For long workflows with predictable structure, such as report generation or sequential data pipelines, this is efficient. Its weakness is rigidity. When intermediate results reveal that the original plan was wrong, the system has no native mechanism to revise rather than proceed. It will complete the plan it made.

Language Agent Tree Search synthesizes several of the above, combining Tree of Thoughts' branching structure with ReAct's tool-use loop and enabling structured backtracking rather than commitment to a single chain. On HotpotQA, LATS achieves the highest exact match among acting-based approaches, outperforming both Tree of Thoughts and ReAct-prompted baselines. The computational overhead limits its use in latency-sensitive deployments. That is not a minor caveat; most production environments have latency constraints that disqualify it.

Production systems rarely deploy any of these in isolation. Common pairings include ReAct with Reflexion for adaptive reasoning tasks where the path forward is uncertain, and Plan-and-Execute with Tree of Thoughts for long-horizon tasks where early decisions benefit from explicit lookahead evaluation. The choice reflects what the actual task distribution demands, and teams that select a framework before understanding that distribution tend to regret it, usually after the system is already running.

How multi-step planning changes retrieval performance on hard questions

Three benchmarks have become standard stress tests for multi-hop retrieval. HotpotQA requires reasoning across multiple paragraphs drawn from a large document index. 2WikiMultiHopQA requires cross-document reasoning across two distinct Wikipedia articles. MuSiQue, the most demanding of the three, requires two to four reasoning hops and is explicitly constructed to block the shortcut strategies that inflate performance on easier multi-hop benchmarks.

Agentic retrieval results from the PRISM system (2024) illustrate what planning actually buys. On HotpotQA, PRISM achieves 90.9% recall, against 61.5% for single-hop retrieval and 72.8% for IRCoT. On 2WikiMultiHopQA, the figure reaches 91.1%. On MuSiQue: 83.2% recall against that 44.6% single-hop baseline.

The gap is largest exactly where it should be, on the benchmark designed to make shortcuts unavailable. That correspondence is not a coincidence. The conditions that widen the gap most are also the conditions that define hard research questions: no shortcut paths, no single-document answers, no retrieval that works by geometric proximity alone.

HotpotQA, though, cannot tell us everything. Wikipedia coverage constraints, contamination risk from models trained on web data, and the absence of temporal or step-wise reasoning chain evaluation all limit what it can say about real-world performance. Newer benchmarks like APB (2025) and ScienceAgentBench are pushing toward more realistic complexity. Whether current benchmark construction is keeping pace with current system capability is, at minimum, an open question, and one I think the field has not adequately reckoned with.

How production research agents implement these patterns

Anthropic's Research feature offers an instructive case. An orchestrating agent plans the research process from the user's query, then spawns parallel sub-agents that search simultaneously. The structural instinct is right for independent sub-questions. A failure mode surfaced in practice: sub-agents without effective coordination duplicated work, multiple agents independently investigating the same supply-chain question rather than dividing the problem space. The fix was embedded in the prompting layer, instructing agents to calibrate effort to the complexity of their assigned sub-task. This is a stop-condition design problem as much as a retrieval architecture problem, and it is exactly the kind of thing that does not appear in architecture diagrams but causes real production pain.

FlowSearch (2025) approaches this through an execution graph model. A planner decomposes the research query into interconnected subtasks organized into three node types: search, solve, and answer. Where dependencies between subtasks are weak, parallel execution can be used fully; the architecture is particularly effective for report generation tasks where sections are largely independent.

LangChain's Self-Reflective RAG implementation, built in LangGraph, makes the correction cycle explicit: retrieve, grade, rewrite, re-retrieve, generate. The grading step, evaluating whether retrieved chunks are actually relevant before proceeding, is what prevents the system from building an answer on insufficient evidence. Self-reflection as a first-class system component behaves very differently from self-reflection retrofitted at the end. The operational difference is larger than it looks on paper, and I would argue this is the single most underappreciated design decision in current production RAG systems.

The spectrum of deep research tools deployed in 2025 makes the latency-quality tradeoff concrete. OpenAI's Deep Research, built on o3, integrates web search, MCP servers, and internal vector stores; it scores 26.6% on Humanity's Last Exam and completes in five to thirty minutes. Perplexity's Deep Research, using a custom DeepSeek R1 with test-time compute expansion, scores 21.1% on the same benchmark and completes in under three minutes. Gemini Deep Research combines public web with private workspace data and adds interactive report output. Doubao Deep Research from ByteDance integrates Chain-of-Thought deeply with search engines for dynamic multi-turn reasoning. The benchmark delta between OpenAI and Perplexity reflects a deliberate tradeoff between planning depth and response latency, not a model quality gap in isolation. These are different products optimized for different constraints.

Where multi-step planning still fails in practice

Planning frameworks amplify what the base model can do. They cannot manufacture reasoning the model cannot perform.

LLM reasoning on complex logical deduction, causal analysis, counterfactual reasoning, and multi-step mathematical problems remains substantially limited. On classical PDDL planning benchmarks, GPT-4 achieves roughly 12.3% plan executability, with performance degrading further as planning horizons lengthen (Valmeekam et al., 2024). Fine-tuned models reach substantially higher in-domain validity, then collapse on cross-domain evaluation. That collapse pattern is the part I keep returning to. It suggests the model has learned domain structure rather than transferable planning skill, which means strong benchmark performance in a narrow task distribution can look like real capability until it isn't. But how do you distinguish these two things from the outside? That is harder than it should be, and the field does not yet have a reliable answer.

Context management degrades at scale. As retrieval chains grow longer, maintaining coherent state across many steps pushes against context window limits. Agents have to decide what to carry forward and what to drop, and that prioritization is not yet reliably solved. The failure mode is subtle: the system does not crash; it quietly loses track of something that mattered three steps back.

Cost and latency spirals are a documented failure mode, not an edge case. Without explicit iteration budgets and confidence thresholds, agentic loops can run indefinitely. This failure is particularly expensive because it consumes both budget and user trust simultaneously, and the trust is harder to recover. Stop-condition design is as consequential to a production system as retrieval strategy and receives a fraction of the attention in the literature.

Redundant sub-agent work, as the Anthropic example illustrates, is a coordination failure specific to parallel architectures. Parallel decomposition is efficient only when sub-agents actually divide labor. Without explicit coordination mechanisms, they converge on the same sub-problems, producing the worst of both worlds: the cost of parallelism with the coverage of a single agent.

It is also worth considering a more fundamental issue with evaluation. Current benchmarks emphasize complex instruction-following over deliberative planning. They do not rigorously test plan verification, backtracking under resource constraints, or performance on tasks with genuinely novel structure. Benchmark scores may, as a result, overstate real-world robustness in ways that are difficult to detect until a system is in production. I do not think the field has a good answer to this yet.

One more thing worth stating plainly: single-hop retrieval is not the wrong default in every case. For data-centric tasks with stable semantics and locatable answers, multi-step planning adds latency and cost without proportional accuracy gain. The architecture should match the task distribution, not the other way around.

What it takes to build a research agent that handles hard questions well

The first discipline is matching the planning strategy to the task structure, not to a framework preference or implementation convenience. Serial decomposition belongs on logically dependent sub-questions. Parallel decomposition belongs on independent sub-questions, and only when coordination overhead is explicitly managed. Plan-and-Execute fits long, predictable workflows. ReAct with Reflexion fits tasks where intermediate results are likely to redirect the plan. Getting this wrong is recoverable; it is also avoidable. The cost of mismatching architecture to task compounds across every query.

Routing is not a secondary concern. Directing each sub-query to the right source type, structured data versus document corpus versus live web, is part of the planning layer. Getting it wrong means retrieving from a source that cannot answer the question, regardless of how well everything else is designed.

Self-reflection and grading must be built into the pipeline from the start. The retrieve-grade-rewrite-re-retrieve cycle is what separates a system that recovers from retrieval gaps from one that confabulates with misplaced confidence, a failure mode commonly described as hallucination. Teams that retrofit this loop after initial deployment spend considerably more time debugging mysterious answer degradation than teams that build it in from the beginning.

Stop conditions and iteration budgets need explicit design. An agent without a principled stopping criterion is not a research agent; it is an expensive loop. This decision deserves the same architectural attention as retrieval strategy.

Effort scaling for sub-agents should be encoded explicitly. Without guidance on effort calibration, parallel agents will over-invest in trivial sub-tasks or duplicate each other's work across branches. The Anthropic example is instructive: the solution was a prompting convention, not a system redesign. Sometimes the fix is that simple. The mistake is not having thought about it before deployment.

The research community is converging on two related directions. The first is learnable planning: agents that refine their planning heuristics from deployment experience without requiring human intervention at each iteration. The second is simulation-before-acting, where the agent mentally rolls out candidate trajectories before committing, reducing irreversible planning errors. DAG-based asynchronous planning and hierarchical reinforcement learning for multi-agent optimization are the near-term architectural bets most likely to close the gap between controlled benchmark performance and real-world performance on hard questions. Whether those bets pay out the way current results suggest is uncertain, and I think the field would be better served by treating them that way.