Concurrency Models for High-Volume Agent Scrapers

One fact shapes every comparison here: scraping is I/O-bound. Your process spends the overwhelming majority of its time waiting on network responses, not computing. Roughly 95% waiting, 5% working. That ratio determines which tools actually help. Think of it like a restaurant where the kitchen is nearly idle but every waiter is stuck at the table waiting for customers to decide — adding more cooks solves nothing.

Async I/O (asyncio)

Cooperative multitasking inside a single thread. Coroutines yield at every await, and the event loop picks up the next ready task. Because coroutines are lightweight objects, you can run tens of thousands of concurrent tasks without proportional memory growth.

The trap is any synchronous operation inside an async function. It blocks every other coroutine until it returns. One slow, blocking call holds up the whole line. The fix is asyncio.runinexecutor(), which offloads blocking code to a thread pool without you having to exit the async context entirely.

Thread pools

Each thread is an OS-level object with real memory overhead. Your practical ceiling is hundreds to low thousands of threads before scheduler overhead becomes visible. Python's GIL sounds scary here, but it's largely irrelevant for scraping. Threads are idle most of the time waiting on the network. The GIL only bites when threads are actively computing, which almost never happens in a scraping workload.

Thread pools are a useful fallback for code that can't be converted to async, or for wrapping synchronous third-party libraries.

Multiprocessing

Separate interpreter per process. Bypasses the GIL entirely. Also carries the heaviest memory footprint of all three options.

This is the right tool for CPU-bound stages: HTML parsing at volume, content deduplication, embedding generation, complex data transforms. It is the wrong tool for the HTTP request layer itself. Spawning processes just to sit and wait on the network is wasteful.

Distributed worker queues

Work items live in a shared queue (Redis, RabbitMQ, Kafka). Multiple worker processes or machines pull from it. This decouples the producer (the agent deciding what to fetch) from the consumer (the workers doing the fetching).

Scrapy Cluster's Redis priority queue is a concrete example: all requests for a spider type and domain flow through a shared queue, enabling coordination across machines that single-process throttling simply cannot achieve.

The cost is operational complexity. Queue management, worker health monitoring, deduplication of in-flight URLs. It's real overhead. Whether it's worth it depends entirely on your scale, which we'll get to.

Async I/O as the default starting point and where it hits its ceiling

For a scraper that needs thousands of concurrent HTTP connections from a single process, asyncio is the current industry standard starting point. The mechanics above explain why.

Library choice

When you're pushing toward extreme concurrency (300 to 5,000-plus simultaneous requests), aiohttp outperforms httpx by a meaningful margin in community benchmarks. Lower tail latency, higher throughput. At moderate concurrency, the difference is negligible and honestly not worth the debate.

httpx has a cleaner API and easier async support. If you're not at the extreme end, it's fine. If you are, aiohttp is the practical choice.

The semaphore problem

Firing 10,000 requests simultaneously exhausts connection pools, triggers IP rate-limiting, and crashes your scraper. asyncio.Semaphore is the standard control mechanism. Tuning that limit isn't just about your own system, though. It requires knowing your proxy pool capacity and your target site's behavior. More on that shortly.

Where async hits its ceiling: the browser

Playwright with async_api works within asyncio. But each browser context is a full resource-heavy instance. You're looking at a practical ceiling of 5 to 20 browser contexts per process. Compare that to hundreds of simultaneous HTTP requests. Why did the browser context go to therapy? It had too many tabs open and couldn't handle the load.

Agent scrapers regularly need to interact before they can extract. Login walls. "Load More" buttons. JavaScript-rendered content. Filter dropdowns. These aren't edge cases anymore.

Playwright is the better choice over Selenium for reliability and async support. But the ceiling has nothing to do with library choice. It's a hardware and memory constraint, and no amount of code optimization moves it much.

That ceiling is architectural. When you hit it, the workload has outgrown single-process concurrency. That's a signal, not a bug.

How browser-dependent agent patterns force a hybrid concurrency design

The shift to JavaScript-heavy rendering accelerated significantly during 2024 and 2025. HTML-only scrapers increasingly failed as more sites adopted single-page application architectures and dynamic rendering. Organizations that had to retrofit browser automation saw infrastructure costs increase and processing slow considerably. It wasn't a gentle transition for anyone.

Modern scraping APIs expose interaction primitives (click, type, navigate) precisely because static HTTP requests can't reach this content anymore.

The hybrid design principle

Each tool maps to the workload it was actually built for:

• Async I/O handles high-volume HTTP requests.
• Browser contexts handle interaction-gated pages.
• Multiprocessing absorbs CPU-heavy stages (parsing, embedding, deduplication).

This is a separation of concerns, not some clever trick. Async-plus-multiprocessing combinations have shown order-of-magnitude speed improvements versus single-model approaches. That result makes sense when you think about it. You're matching each tool to its actual strength rather than forcing one tool to cover everything.

TLS fingerprinting

Traditional scrapers didn't have to think about this. TLS fingerprinting has emerged as a more reliable anti-bot signal than IP address alone, which means rotating IPs is no longer sufficient on its own. The concurrency layer also needs to vary request fingerprints. That adds state management overhead at the async layer. Not enormous, but real, and it wasn't in the design ten years ago.

The serial constraint that parallelism can't fix

The agent's reasoning loop has a sequential dependency baked into it. The agent receives a result, processes it, then decides what to fetch next. Concurrency fills the wait time within a step productively. But the step boundary itself is a hard serial constraint. No concurrency model eliminates it, and understanding this saves you from over-engineering in entirely the wrong direction.

When a distributed worker queue becomes necessary rather than optional

A single async process, even well-tuned, is bounded by one machine's file descriptors, memory, and network interface. At some point you've just hit the wall of what one machine can do, and no amount of tuning gets you past it.

The rough threshold where distribution becomes necessary is around 1 million pages per month. Below that, a well-tuned single-process scraper is usually adequate. Above it, you need distributed scheduling, concurrent connections across machines, and coordinated proxy management. Single-machine architectures simply can't sustain it.

The Scrapy coordination problem

Scrapy's built-in AutoThrottle works for a single spider process. It cannot coordinate throttling across multiple processes on different machines. Scrapy Cluster's Redis-backed priority queue solves this: all requests for a spider type and domain flow through a shared queue, enabling cross-machine rate coordination.

Scrapy 2.14 (early 2026) also introduced AsyncCrawlerProcess and AsyncCrawlerRunner as coroutine-based replacements for the older Twisted-era runners. Worth knowing if your team is integrating Scrapy into async agent pipelines.

Agent-specific queue design

Agents need a different approach to queuing than batch scrapers do:

• Priority queuing matters more. A URL the agent needs to continue its current reasoning chain should preempt background crawl jobs. Not every URL is equally urgent.
• Deduplication must happen at the queue layer. Handle it centrally, not per-worker. Agents with overlapping search paths will flood the same URLs simultaneously if you don't enforce this centrally.
• Queue depth is a leading indicator. A growing queue means workers can't keep up with the agent's request generation rate. Watch it.

The operational costs of distribution are real. Queue management, worker health monitoring, and at 1M-plus pages per month you're looking at dedicated monitoring infrastructure. Dashboards, alerting, on-call rotations. Before committing to that overhead, weigh it honestly against a managed scraping API at that scale. Sometimes the managed service just wins on total cost.

Proxy pool capacity as a hard constraint on every concurrency decision

The relationship people most often miss: concurrency settings and proxy pool capacity are coupled. Setting CONCURRENT_REQUESTS too high concentrates requests on individual proxies and accelerates blocking. Your effective concurrency ceiling is partly a function of pool size and rotation strategy.

From a 2026 benchmark using a 30-million-request dataset (AIMultiple), at 5,000 parallel requests, major providers including Bright Data, Oxylabs, and Decodo all performed reliably. At 100,000 parallel requests, all experienced some degradation. Bright Data, Oxylabs, and Decodo showed the least impact, with minimal changes in success rate or response times.

2026 residential proxy pricing

• Bright Data: ~$8/GB. Enterprise pick, includes Web Unlocker and SERP API.
• Oxylabs: from $6/GB, 175M-plus IP pool. SLA-grade with scraper APIs.
• Decodo: from $3.75/GB, sticky sessions up to 24 hours. Balanced mid-market option.

The proxy tier effectively sets a soft ceiling on sustainable request rate. Pushing concurrency above what the proxy pool can absorb generates retries. Retries amplify load. You create a feedback loop that collapses throughput at exactly the moment pressure is highest. It's like flooring the accelerator in a traffic jam — you burn more fuel and go nowhere faster.

Exponential backoff on rate-limit responses is the standard mitigation. The goal isn't politeness toward the target server — retry storms will hurt you more than the target.

One more thing about browser automation: residential proxies per browser context get expensive fast, even at modest concurrency levels. Cost modeling must account for this before you decide to use browser automation at high volume. Run the numbers before you commit to the architecture.

What agent scraper latency requirements reveal about model fit

Throughput and latency are separate optimization targets. Batch scrapers optimize for throughput. Agent scrapers optimize for both, and that distinction matters more than people usually give it credit for.

Because the agent's reasoning loop stalls while waiting, latency compounds across steps in a way that batch pipelines simply don't experience. Benchmarked scraping APIs had a median response time of 5.05 seconds per request (Proxyway Web Scraping API Report, 2025). For an agent making sequential decisions, five seconds per hop adds up fast across a multi-step research task.

What this means for model choice

• Async I/O is the right model for parallelizing independent fetches within a single agent step. It fills wait time productively.
• The step boundary itself is serial. No concurrency model changes that.
• Distributed queues add queue-insertion and worker-pickup latency on top of network latency. That's acceptable for batch workloads. For a latency-sensitive agent loop, it can be a real problem.

For interactive or low-latency agent use cases, a simpler async architecture close to the request (minimal queue hops) may outperform a theoretically higher-throughput distributed system that adds coordination overhead. The right choice depends on whether you're optimizing for sustained throughput at scale or for response time per agent step. Both are valid goals. They don't always point to the same architecture.

Structuring scraped output so it enters LLM context without a cleanup step

Raw HTML handed to an LLM is a token budget problem that hits you on every single call. Not once in a while. Every call.

A typical documentation page runs 8,000-plus tokens as raw HTML. As clean Markdown, that same page is roughly 1,200 tokens. That's a 60 to 80 percent reduction. Across a high-concurrency pipeline processing thousands of pages, this is a direct cost reduction per LLM call that compounds quickly.

The Markdown default isn't settled science

A 2025 paper from Renmin University (accepted at the WWW conference) found that pruned HTML actually outperforms plain text and Markdown on QA benchmarks. Their HtmlRAG approach compressed 20 retrieved documents from 1.6 million tokens to roughly 4,000 tokens of cleaned HTML while maintaining retrieval quality.

The numbers from that paper: on Natural Questions with Llama-3.1-70B running a 128K context, cleaned HTML hit 42.25% exact match versus plain text's 41.00% and Markdown's 39.00%. Not a massive gap, but consistent, and in the direction opposite to what most people assume.

Clean Markdown is the right default for most pipelines. It's simple, readable, and dramatically cheaper than raw HTML. But if you're building retrieval systems where QA quality really matters, cleaned HTML is worth evaluating. The answer depends on what you're asking the LLM to do with the content.

What's beyond debate: raw HTML should never cross the handoff boundary into LLM context. Whatever format you choose, the extraction layer owns that transformation. Pushing the cleanup step downstream to the LLM is just burning tokens on a problem you already had the tools to solve upstream.