Large-Scale Web Crawling Architecture Patterns

Most crawlers die the same death. They run fine on a laptop against a few hundred pages, then someone points them at a few hundred thousand and everything comes apart at once. Connections hang and time out, IP addresses get blocked mid-run, memory climbs until the process folds, and a script that looked perfectly reasonable at small scale turns out to need roughly 28 hours to finish a large job. None of that is a tuning problem: it's what happens when a design built for one scale gets asked to do a different job.

Scale isn't one dial, either: it's four of them, moving independently: how many pages you're pulling, how many different sites you're pulling them from, how fresh the data has to stay, and how much anti-bot resistance you're up against. A setup that handles heavy volume against a handful of cooperative sites can still fall apart the moment it meets a thousand different sites with a thousand different defenses. Each of those four dimensions has to be designed for on its own terms. Nobody gets to solve volume and call the problem closed.

There's a second force pushing on all of this, and it has nothing to do with raw page counts. AI agents and retrieval pipelines want data that's current, structured, and available on demand, unlike a dataset that's a day or a week stale. Most crawling pipelines were built on the opposite assumption, that running on a schedule every few hours or once a day is good enough because web pages don't change that fast. That assumption holds right up until it doesn't, and once freshness requirements tighten, it breaks completely. The rest of this piece walks through the patterns that hold up once both of those pressures, scale and freshness, hit a system that wasn't built to absorb them.

Async concurrency as the first architectural lever

The fastest fix available to a crawler that's falling over is also the simplest to describe: stop waiting on one request before sending the next one. Web crawling spends almost all its time waiting on network responses, not doing computation, so an engine that can hold many requests open at once turns dead waiting time into useful work. In practice that means an event-driven, non-blocking setup, asyncio paired with aiohttp in plain Python, or Scrapy's Twisted-based engine if you're working inside that framework, firing off requests without pausing for each one to come back.

Turning concurrency on takes little judgment; deciding how much of it to allow at once is what actually takes engineering judgment. Firing off a few thousand connections simultaneously looks fast on paper and gets a crawler banned within minutes. Cap concurrent requests with a semaphore set somewhere between 10 and 50 as a starting point, tuning that number upward based on how much a given target site can tolerate. Think of the semaphore as a bouncer at a door that only lets a fixed number of people in before anyone new can enter, which keeps a crawler running for hours instead of getting it noticed and shut out in minutes.

Timeouts matter just as much as the semaphore does, and they're easy to skip because nothing breaks until something does. A single hung connection, left without a timeout, holds its semaphore slot forever and quietly stalls every other request waiting behind it. A crawler that forgets to set timeouts has built a trap door that opens exactly when traffic gets heavy.

Rate limiting belongs in the same design conversation as concurrency, not added after the first round of bans. A token-bucket or leaky-bucket limiter, applied per domain and capping requests per second, is what keeps the speed that async concurrency buys from turning straight into IP bans. Scrapy's 2.14 release switched the framework's default scheduler to DownloaderAwarePriorityQueue, which spreads requests more evenly across domains and reduces per-domain tuning burden, and the new async def start() method makes async requests a first-class citizen rather than a workaround, as GroupBWT's 2026 production architecture guide documents. Concurrency, timeouts, and rate limits together determine how hard a crawler is allowed to push.

The URL frontier: the component that determines whether a crawler reasons about what it fetches

Speed answers one question: how fast can a crawler fetch. It says nothing about what to fetch, when, or in what order, and that's where the URL frontier comes in. The frontier encodes the crawler's actual policy, deciding what gets fetched next, how often it gets revisited, and in what sequence; getting that policy wrong raises duplicate scrapes, URLs that never get visited, or a fleet of workers sitting idle while a backlog piles up somewhere upstream.

A working frontier has to juggle four jobs at the same time. It has to prioritize, so a page that changes daily gets visited ahead of one that hasn't changed in years, since a first-in-first-out queue burns worker time on stale content while high-value pages wait their turn. It has to shard by domain, using consistent hashing to spread URLs across workers so no single site can hog downloader capacity, and so each domain can be held to its own politeness limits. It has to deduplicate at the moment a URL gets added to the queue, not after a worker has already fetched it, because catching a duplicate after the fact has already cost a worker slot, a proxy IP, and bandwidth that can't be refunded. And it has to manage recrawl scheduling, recognizing that a pricing page and a static documentation page don't belong on the same refresh clock. These four jobs aren't separable, either.

Distributed worker architecture and the infrastructure tiers that match actual volume requirements

Once the frontier has decided what to fetch and when, something has to actually go get it, and that's the worker layer. A production crawler built to hold up under real load needs six distinct layers working together, and a failure in any single one has consequences for everything downstream of it. Which tier of distributed architecture actually matches the volume in front of you is the real question, because over-building costs money and under-building falls over.

At the low end, a single machine with a basic set of proxies covers a modest daily page volume without any need for a fleet. At a moderate daily volume, a small distributed setup using RabbitMQ as the job queue starts to earn its keep. At high daily volumes, the architecture needs to go fully distributed, running on Kafka with a residential proxy layer behind it. Past that, at the very highest volumes, most teams end up either handing the job to a managed service or building dedicated infrastructure sized specifically for that load.

That tier structure is also what settles the argument between asyncio and Celery, because it isn't really an argument about which tool is better. Asyncio earns its keep when the whole job runs on one machine. Celery, or something like it, becomes necessary once the work has to spread across multiple machines and needs automatic retries and ongoing monitoring baked in. One trades operational overhead for simplicity, the other trades simplicity for resilience, and the right call depends entirely on which tier you're actually operating in.

Holding a worker fleet together in practice usually comes down to a dispatcher pattern: URLs get chunked into fixed batches, commonly groups of 1,000, each batch's results get written to disk the moment it finishes, and nothing is left sitting in memory waiting to pile up and eventually crash the process. On the open-source side, Scrapy paired with scrapy-redis has become the standard way to run the distributed tier, since it swaps Scrapy's default in-memory scheduler for Redis and lets multiple spider instances run across separate machines against one shared frontier. On the managed-infrastructure side, a common cloud-native pattern runs a Python-based crawler through AWS Batch for orchestration, writes output to Amazon S3, and lets downstream applications pull directly from that S3 bucket, which is a clean illustration of how managed infrastructure takes on the operational weight once volume climbs high enough. And none of this requires touching spider logic to handle anti-bot defenses, because that work belongs in middleware: proxy rotation, user-agent rotation, cookie and session handling, retry logic with configurable backoff, and filtering out bad responses, all of which Scrapy covers at a baseline level through its built-in RetryMiddleware, HttpProxyMiddleware, and UserAgentMiddleware.

Storage architecture at crawl scale favors units other than raw HTML

Fetching the data is only half the job. Where it lands, and in what shape, decides whether anyone downstream can actually use it. At real crawl scale, storage is an architectural layer that has to be designed around how the data gets used later, and the dominant use case now is feeding AI pipelines that can't do anything useful with raw HTML in the first place.

Production systems tend to converge on a four-tier storage model. Raw HTML goes into compressed object storage, S3 or something equivalent, kept around for reprocessing and audit purposes rather than everyday use. Parsed content gets its own document store built for search and retrieval. Metadata, meaning crawl state, freshness tracking, and URL management, lives in a NoSQL database suited to that kind of fast, flexible lookup. And the link graph, the map of how pages connect to each other, belongs in a graph database or an adjacency-list structure, because frontier decisions and site-structure analysis both depend on having that map available, not reconstructed after the fact.

The cost argument for getting this right isn't abstract. At a large monthly crawl volume, even with pages stored in compressed form, raw HTML storage alone adds up to an enormous total. Decisions about what to store, how compressed to keep it, and in what structure are cost decisions with real weight behind them. The link graph in particular tends to get left out of first-draft architectures and then bolted on later at real expense, when it would have cost far less to design in from the start, since frontier logic, deduplication, and site-structure understanding all lean on it. The dispatcher pattern from the worker section, writing each batch to disk as soon as it's done rather than letting results pile up in memory, is as much a storage decision as it is a memory-management one. Storage and fetching are one design problem, and the next section picks up exactly where storage decisions start shaping what comes out the other end.

Output format designed for model consumption into the crawl pipeline, not added afterward

Everything a crawler fetches eventually has to leave the pipeline in some format, and that choice of format carries real consequences, for how accurately a retrieval system finds the right information, for how many tokens a model burns reading it, and for how well the model actually performs on top of it. Raw HTML isn't a safe default to fall back on while sorting out the rest of the architecture. Feeding it to an AI system does active harm, cluttering the input with scripts, navigation menus, and ad markup that the model has to wade through to find anything useful.

Markdown has become the practitioner's default answer for RAG pipelines for a straightforward set of reasons. It keeps headings, lists, links, and tables intact, it takes up a fraction of the space raw HTML does, and it strips out the scripts, navigation, and ad clutter that would otherwise dilute what the model actually reads, and that smaller footprint translates directly into lower token costs and less noise for the model to sort through. JSON plays a different role, sitting alongside Markdown rather than replacing it, since its value is in preserving the relationships between fields in a machine-readable way, which makes it the right choice for entity extraction and schema learning rather than for free-text retrieval.

However the output gets formatted, chunking it well isn't optional. Cutting text at arbitrary points, mid-sentence or mid-section, strips away the context that made a retrieved chunk relevant in the first place, and that loss lowers retrieval accuracy in RAG systems. ScrapeGraphAI, for one, bundles LLM extraction straight into the crawl loop, trading a higher per-page cost for the removal of a separate processing step. That's a legitimate option for a team that values simplicity over per-page economics, not a universal best answer, and the right call depends on the same kind of tradeoff that decided asyncio versus Celery earlier: speed and simplicity on one side, cost and control on the other. What all of this adds up to is that the seam between crawling and formatting is where complexity quietly accumulates in most pipelines, and collapsing that seam into a single designed step, rather than treating formatting as cleanup work done after the real job is finished, is what keeps that complexity from piling up downstream.

The freshness gap that batch crawl architectures cannot close

Competitive monitoring is the use case that exposes all of this most clearly, because it has zero tolerance for the batch assumption baked into most crawl pipelines. A competitor changes a price, updates a product page, or posts something new, and a monitoring system is only as good as the gap between that change happening and the system noticing it. A batch crawl run once a day leaves a gap that is, at best, a day wide. Running it every few hours shrinks the gap, but it never closes. No batch schedule, run often enough, actually gets to zero, because the architecture itself is built around the idea of checking in periodically rather than knowing the instant something changes. Freshness is a property that has to be designed into the architecture from the start, the same way concurrency, the frontier, storage, and output format all had to be, and a pipeline built around periodic batches will keep hitting that same gap no matter how short the period gets.