Concurrency Models for High-Volume Agent Scrapers

Most teams build agent scrapers and put the money in the wrong place. They shop for the cheapest price per gigabyte, they argue about which language model writes the best extraction prompt, and then they wonder why the pipeline crawls at a fraction of the speed they expected. The actual bottleneck is almost always concurrency headroom, meaning how many requests can run at the same time. Bandwidth and concurrency sound like the same knob, but they aren't: bandwidth is the total amount of data moved, while concurrency is the number of simultaneous requests or sessions a system allows. A pipeline can have a fat data plan and still choke the moment it hits a cap on concurrent connections. If you want to double scrape speed, you need to double the parallel requests running at once, not buy a bigger data allowance. The concurrent-connection limit sets the real ceiling, and the data plan has very little to do with it.

AI agent fleets made this impossible to ignore. Each agent drives its own browser session, so if hundreds or thousands of agents run at once, you need that many concurrent clean IPs at the same time. That demand is fundamentally different from what a single scraper making requests in sequence needs. "Max concurrent sessions" used to be a footnote in a pricing page. It's now a headline spec, the first question serious buyers ask.

The architectural shift from selectors to protocol-level calls

The way AI reads a web page changed shape entirely. Instead of writing CSS selectors or XPath expressions tuned to one site's HTML, you can call a protocol directly, the Model Context Protocol (MCP), paired with an API that hands back clean Markdown instead of a pile of raw tags. So every agent runs as its own independent concurrent worker, not as one thread competing with others inside a single scraping job.

Three forces pushed this change at once: MCP's rapid adoption as shared industry infrastructure, the rise of agent fleets that treat each browser session as its own concurrent worker, and anti-bot systems getting sharp enough to punish sloppy concurrency patterns. MCP's own growth curve tells the story efficiently. Anthropic introduced it in November 2024 as an open standard, and by December 2025 it had been handed off to the Agentic AI Foundation under the Linux Foundation, co-founded by Anthropic, Block, and OpenAI, with Google, Microsoft, AWS, Cloudflare, and Bloomberg backing it as supporting members. Public SDK download tracking put MCP SDK downloads at 97 million per month by March 2026. A protocol that one company built stopped being that company's property and turned into infrastructure the whole industry runs on.

That shift matters for concurrency because the protocol's growth into shared infrastructure changed what actually limits concurrency: not thread count, but the standard everyone now builds against. Running more threads against one target site used to be the whole concurrency story.

The four concurrency models available to agent scraper architects

High-volume agent scrapers need one of four concurrency models, and each one solves a different problem rather than just running at a different speed.

Thread-based concurrency assigns each request its own thread. It's easy to reason about and supported in nearly every language, but it runs into a hard ceiling on how many threads a server can spawn, and each thread drags a small memory cost that piles up fast at scale. In Python specifically, the Global Interpreter Lock largely forced threads to run sequentially before Python 3.13, so this model costs more in the language most scraping code is written in.

Async I/O runs everything inside a single thread, with an event loop switching between tasks while they wait on network calls, tools like Python's asyncio paired with aiohttp, or Rust's Tokio runtime. Because it isn't threading, it sidesteps the thread-count ceiling. A direct benchmark on a moderate-size list of URLs found Scrapy and asyncio plus aiohttp finishing in essentially the same time, which says Scrapy's overhead barely registers until volume climbs much higher. Scrapy also ships with an AutoThrottle extension that adjusts request rates per domain based on how fast that domain responds, and once a crawl spans many domains at once, rebuilding that behavior by hand in custom asyncio code is genuinely hard. Tokio, Rust's async runtime, handles millions of concurrent tasks on a small pool of threads. You face a steep learning curve up front, but if the crawling work is CPU-and-IO-bound, you get substantial throughput per dollar of compute spent.

With process-based or browser-based concurrency, like the Playwright model with isolated browser contexts, each session runs as something closer to a full browser instance. Playwright supports async operations natively, keeps contexts isolated from each other, and auto-waits for page elements instead of relying on brittle timing hacks. Running hundreds of parallel Chromium contexts needs roughly one to two gigabytes of RAM per instance, so the infrastructure bill scales in a straight line with concurrency. The model best suited to JavaScript-heavy sites is also the priciest one per concurrent session.

Distributed task queues, Celery with async workers or Scrapy paired with Redis, represent the production-grade pattern. This is proxy rotation, rate limiting, retry logic, and a distributed queue working together, commonly Scrapy with a Redis backend and a rotating proxy pool. As a rough map of thresholds: asyncio plus aiohttp handles lower-volume jobs well, Celery with async workers covers the mid-tier, and full distributed infrastructure takes over once volume outgrows both.

Tradeoffs Under Agent Fleets, Anti-Bot Pressure, and API Rate Limits

None of these four models has a fixed ranking. Which one wins depends on which pressure is squeezing the pipeline hardest: agent-fleet parallelism, anti-bot detection, or API rate limits, and real deployments rarely face just one of these at a time.

When fleet parallelism is the binding constraint, the math is unforgiving. Each agent needs its own clean IP running concurrently with every other agent's, so a thin IP pool throttles the fleet just as a hard connection cap would. Async I/O handles this well: there's no per-session thread overhead, and the event loop scales across many sessions without RAM climbing in step. Playwright can also run fleet workloads, but RAM cost grows linearly with each instance, turning concurrency into a direct and growing infrastructure bill.

When anti-bot pressure dominates, the danger flips direction. Agent fleets can generate thousands of requests a minute without meaning to, and that looks a lot like a denial-of-service attack to a smaller target site, so you can no longer skip concurrency limits inside the agent's own networking logic. An API that can handle massive concurrent load doesn't mean the site on the other end can take it. High concurrency pulled from a small IP pool is also an obvious tell to anti-bot systems. If you spread sessions across a large pool, varied by geography and network provider, no single segment lights up as a suspicious burst. Rather than one scraper trying to out-clever advanced detection alone, the work can split across specialized peers running concurrently, CAPTCHA solvers, behavioral simulators, DOM diff analyzers, session handlers, each with its own identity.

When API rate limits become the binding constraint, you have to ask different questions. Scrapy's AutoThrottle, adjusting per-domain request rates automatically based on latency, is the built-in answer for crawls spanning many domains at once. Async frameworks need that logic written by hand, per domain. Distributed task queues offer the tightest control over rate-limit budgets across multiple API targets, but that control comes bundled with the asynchronous overhead that breaks synchronous agent loops.

A fleet running at scale against anti-bot-protected sites, hitting several rate-limited APIs along the way, needs an architecture built for all three colliding at once, which is exactly the setup that exposes the next unresolved problem.

The unresolved tension between synchronous agent loops and asynchronous scraping infrastructure

The property that makes distributed async infrastructure scale, submit a job, poll for its status, fetch the result later, is the same property that makes it a poor fit for an agent reasoning in real time. An agent running a ReAct loop (Reasoning and Acting) needs the web to answer right now. If it decides mid-reasoning that it needs a page, it can't pause and wait for a batch job somewhere in a queue to finish. It needs rendered content back in seconds, with boilerplate stripped out and shaped to fit inside its context window.

Latency adds up in a way that's easy to underestimate. A multi-step agent workflow pays its per-call latency cost once per step, and those costs stack across every hop in the chain. A handful of calls at a few seconds each adds up fast inside a single run, and at a few hundred sessions an hour, that delay turns into a real cost line on someone's infrastructure budget.

Synchronous APIs answer faster per call, but if a fleet runs at scale, they buckle under rate limits. Async APIs scale cleanly but add latency that stacks across every reasoning step, and the field hasn't settled this trade-off yet. An async job model, submit the work, keep reasoning on something else, check back once the result lands, is the right shape for agent pipelines built to do other things while they wait. Most current ReAct implementations don't work that way. They block on outside calls instead of yielding control while they wait. Anyone building agent scrapers today is building on top of that open design gap rather than around it.

Output format as a concurrency multiplier

Format sits further downstream than concurrency architecture, but it feeds straight back into the same throughput ceiling. If you hand an agent raw HTML, you burn part of its concurrency budget on tokens the model never needed. Clean Markdown or structured JSON shrinks the payload, which lets the same agent process more pages inside one context window.

If you didn't ask for HTML specifically, a scraper that returns raw HTML by default got it wrong. The right design parses the page, pulls out the primary content, and converts it into semantic Markdown or a strict JSON schema. That conversion step alone can cut token payloads by as much as 90%. An agent can read through several pages where it used to manage barely one. For reading and reasoning tasks, Markdown and plain text keep the headings and lists that help a model follow a page's structure, and they stay light on tokens. For structured field extraction, JSON is the better format. Raw HTML is the worst option for anything an LLM has to read, since it burns tokens on markup the model has no use for. If token cost matters to you, a cleaned, fitted Markdown output is close to the default right answer.

The throughput connection is direct: a smaller payload per request means the same concurrency budget stretches across more pages. An agent fed a dramatically lighter payload can reason over more sources from a single context window, without needing a single extra parallel session to do it. APIs that keep a warm pool of headless browsers ready, so they can hand back rendered, cleaned content at low latency, close the gap between the instant response an agent needs and the kind of scale that async infrastructure is built to handle.

Concurrency at production scale: monitoring, distributed queues, and push-model event delivery

At real production volume, the concurrency model an organization picks does more than just set throughput. It shapes the whole operational picture: how changes on a target site get noticed, how agents get triggered to act, and how cost scales as volume grows.

The distributed queue pattern remains the standard at high volume: Scrapy, a Redis queue, rotating proxies, rate limiting, and retry logic working together, fit for workloads that can tolerate the latency that comes with async jobs. An alternative is the push model for monitoring, where instead of an agent polling a target repeatedly to check for changes, the system pushes an event the moment a change happens, cutting out the polling loop. So the gap between a web change and a system's response drops from hours to minutes, and agents act the moment the event arrives instead of waiting for the next poll cycle.

Pricing shapes the economics of all of this. A per-connection pricing model taxes concurrency directly: the more parallel sessions a fleet runs, the more it charges. So the pricing structure of the infrastructure layer shapes which concurrency model makes financial sense at scale.

Coordination overhead is its own tax on concurrency. Stitching together separate tools for search, scraping, crawling, mapping, batch jobs, and monitoring means every connection point between those tools becomes a possible rate-limit bottleneck and a fresh source of latency across a fleet of agents. A single API covering all of those functions, with batch processing for large sets of URLs, webhook-based event delivery, and MCP support built in, removes that coordination tax instead of adding another seam for it to hide in.