Distributed Crawling with Cloud Infrastructure

Distributed crawling breaks down the moment a single machine can't keep up, and that ceiling arrives faster than most teams plan for. One crawler on one box tops out somewhere between 60 and 500 pages a minute under good conditions. At the low end, a few million URLs turns into a multi-week slog, and that's before anything actually breaks.

What distributed crawling actually means: the two setups that matter and where they split

Distributed crawling means splitting three jobs, the frontier (the list of URLs still waiting to be visited), the fetching, and the storage of results, across multiple machines so no single one owns all three. Researchers Shkapenyuk and Suel described two setups that still hold up as the basic menu, and picking wrong here costs teams months later.

The small crawler setup uses one central DNS resolver and central queues per website, with downloading spread across machines. Easier to build. Easier to reason about too. But that central queue becomes the thing everyone lines up behind once traffic gets heavy enough, and that's a certainty, not a maybe.

The large crawler setup spreads the DNS resolver and the queues themselves across machines. That kills the central chokepoint, but it trades one hard problem for another: keeping everything consistent when there's no single source of truth left to check.

Underneath both sits a more basic question, the one worth getting right first. How do you decide which worker gets which URL? Dynamic assignment uses a central scheduler handing out URLs in real time. It's flexible, you can add or pull workers on the fly, but the scheduler itself now needs to be built tough enough that it doesn't turn into the new bottleneck. Static assignment skips that fight entirely: run URLs (or domain names) through a hash function once, at the start, and lock each one to a worker for the whole crawl. Simpler to coordinate. When a link crosses from one domain's worker to another's, though, something has to batch-process the handoff, and pre-loading workers with URLs that got cited a lot in a previous crawl cuts down on that traffic.

Static assignment is the right default once the crawl grows large enough to stress a central scheduler, and this isn't a close call. Boring is what you want when the alternative is a scheduler falling over at 2am. The real reason teams make this jump at all is consolidating shared crawling logic, standardizing output for downstream processing, and supporting varied targets without rebuilding infrastructure every time. The choice mainly concerns something other than raw speed. It's about where the complexity lives, in one scheduler or spread across the whole system, and that decision shapes everything downstream.

Queue design: the piece most teams underestimate and most often get wrong

The queue is where distributed crawling's coordination problem actually lives, and it's also where most teams get burned. Workers need to grab URLs without stepping on each other, track what's already been fetched, and re-queue anything that failed, all without some central process turning into a traffic jam. None of that comes free. It takes real scheduling logic to keep task assignment from falling apart under load.

Politeness rules make it worse. Crawlers have to respect per-domain crawl delays, so the queue needs to track the last time each domain was hit and hold workers back even when plenty of other work is sitting right there. That opens two separate failure modes.

Queue starvation happens when a big share of workers sit idle because the only URLs left in front of them belong to domains under a politeness timeout. Queue explosion happens when the frontier keeps growing because duplicates get filtered after they've already been fetched instead of before, which is far too late. Deduplication belongs at the queue layer, not bolted on after the fact. By the time a duplicate URL lands on a worker's desk, the bandwidth and compute are already spent. Gone. No refunds.

Static assignment pays off again here. Hashing URLs to specific workers naturally splits the dedup state across those same workers, so there's less need for one shared dedup store that everyone checks in with. Teams that drop in a commodity message broker like SQS or Pub/Sub and call the queue "done" are making a mistake that shows up weeks later, when politeness logic and dedup logic turn out to have nowhere to live. Somebody still has to build it, and that somebody is usually whoever's on call the night the frontier fills up with unfiltered URLs.

Execution separation: why fetching, rendering, and parsing need separate workers

One worker that does everything, fetches the page, renders the JavaScript, parses the content, writes the output, sounds efficient on a whiteboard. It isn't, in practice. A slow render job blocks a fast fetch sitting right behind it in line, and a single parsing crash takes down requests that were mid-flight and had nothing to do with the failure.

Splitting the work fixes this, and it's not a close call. Fetcher workers handle HTTP requests, proxy rotation, and connection pooling, and they should stay stateless and easy to scale sideways. Renderer workers run headless browsers (Chromium through Puppeteer or Playwright) for pages that need JavaScript executed before there's anything worth grabbing, and they eat memory and CPU fast enough that they need their own sizing, kept separate from fetchers entirely. Parser workers take the raw HTML or the rendered DOM and pull out the structured data, scaling up or down based purely on how gnarly the extraction logic gets.

One enterprise team pulling competitor pricing data from 14 different marketplaces split its single crawler into 8 job-specific micro-extractors. Job failures dropped sharply, even while the target sites kept changing layouts through the quarter, because one extractor breaking didn't take the other seven down with it.

Rendering also forces an infrastructure call most teams don't see coming until it's too late. Headless browsers require persistent state that serverless functions aren't designed to maintain, so any crawler needing ongoing browser sessions has to run on long-lived instances, not pure serverless functions. Full stop, no workaround. Fetching without rendering runs around 4 seconds a page on average, putting a single worker at roughly 60 to 120 pages a minute. Rendering adds real time on top of that, which is exactly why renderer capacity needs its own dial, tuned apart from the fetchers. Whatever comes out the other end should already be structured, clean Markdown or JSON, not raw HTML dumped downstream. Passing raw HTML through the rest of the pipeline just multiplies storage and processing costs without adding one useful byte of information.

Cloud infrastructure options and what actually decides cost at scale

Three ways to run this, three very different cost curves, and picking the wrong one doesn't show up on the bill until month three. Serverless functions (AWS Lambda, Azure Functions) skip idle costs entirely and scale to zero when there's nothing to crawl, which suits stateless fetch workers fine. They can't hold a persistent browser session, so anything involving rendering needs a different home, and egress fees pile up fast when large HTML payloads or media are moving out of the cloud.

Spot instances (or preemptible instances, depending on the cloud) are the cheapest way to run sustained, high-volume crawling, full stop. The catch: the cloud provider can yank the instance back at any moment, so the system needs checkpointing and graceful eviction handling so in-flight work doesn't just vanish. That makes them a solid fit for renderer and parser workers specifically, since those restart clean without losing anything sitting in the queue.

Dedicated bare metal, Hetzner is the common example, charges fixed bandwidth, which pays off once data volumes get big enough that cloud egress pricing would otherwise eat the budget alive. Less elastic, sure. But the unit economics stay predictable at real scale, and predictable beats flexible once the bill is already five figures a month either way.

Storage splits into two tiers. Object stores like S3 or Google Cloud Storage hold the raw crawled content and scale about as far as anyone needs. Structured metadata, crawl state, and dedup records go into NoSQL (MongoDB, Cassandra) or a SQL database instead. For pipelines feeding AI systems, separating raw crawled content from processed, ready-to-use data with distinct access controls keeps untouched content walled off from whatever's actually ready to feed a model.

Build-versus-buy comes down to where the cost shows up. Build in-house and the cost is ongoing engineering time, forever. Use a managed platform and the cost turns into a subscription line, but architectural control and clear per-request pricing go out the window with it. Egress is the cost that sneaks up late: teams routinely undercount how much data moves between cloud regions, or from cloud storage back to an on-prem system, and that gap can turn a cheap-looking serverless design into something pricier than bare metal once volume climbs.

Anti-detection complexity: the cost that scales with the crawler, not just the target

Roughly 90% of teams building distributed crawlers at scale underestimate how much work anti-detection turns into. It stops being a side task fast and starts competing directly with the actual goal, which was extracting data, not outsmarting a bot detector for a living.

Modern anti-bot platforms, Cloudflare, DataDome, Kasada, HUMAN Security (formerly PerimeterX), work at the browser fingerprint level. They spot a headless browser, flag a data-center IP range, and catch a behavioral anomaly within milliseconds of the connection landing. Rotating IPs alone doesn't get past any of that anymore, and teams still relying on IP rotation as their whole strategy are already behind.

The fingerprint surface is bigger than most people expect walking in. There's the IP type itself (data-center versus residential versus mobile), the browser fingerprint (operating system, installed plugins, Canvas API output, WebRTC behavior), and behavioral signals like mouse movement, keystroke timing, and scroll patterns. The Crawling-Infrastructure project on GitHub, built by developer NikolaiT, calls this a "never-ending fight between the cat and the mouse." Evasion techniques need constant updates because detection keeps evolving too, so the engineering work here never settles down, and anyone who claims they've "solved" bot detection is selling something.

At the architecture level, the fix is keeping proxy management separate from fetcher logic entirely. Proxy pools, residential, mobile, rotating, should plug in as a swappable dependency, not something hardcoded into the fetcher, so a provider swap doesn't mean redeploying every worker. CAPTCHA solving needs to run asynchronously too, so a challenge doesn't freeze a worker thread while it waits. The queue should park that URL and let it come back around once the challenge clears. Teams that would rather spend engineering hours on extraction quality than evasion infrastructure increasingly look at managed scraping APIs that absorb this whole layer. That's a real trade, cost and control against capability, not a free lunch either way.

Managed cloud scraping platforms: what each one covers and where each one stops

The useful test for any platform: does it actually handle queue management, JavaScript rendering, proxy rotation, and structured output, or does it just hand those problems right back with a nicer dashboard?

Scrapy Cloud, from Zyte, hosts and runs Scrapy projects without making anyone manage the underlying servers, and it plugs into the Zyte API. It assumes familiarity with the Scrapy framework, though, there's no point-and-click version here, and pricing climbs into higher tiers fast once a crawl gets large. Billing runs pay-as-you-go, per request.

ScrapeHero Cloud goes the opposite direction: pre-built crawlers for specific sites like Amazon, Google Maps, and Walmart, no code required, with scheduling on an hourly, daily, or weekly basis. ScrapeHero handles the maintenance when a target site changes its layout, which matters more than it sounds like on paper. The trade is being limited to whatever sites they've already built. Plans start at $5 a month, with 400 free credits to test the water, and exports land in CSV, JSON, or XML, with a Dropbox integration on top.

ScraperAPI handles proxy management and CAPTCHA solving automatically, starting around $49 a month for a set request volume. It fits teams that want to write their own extraction logic and hand off the anti-detection headache to somebody else, which is the right call when extraction is where the actual value sits.

Bright Data's Web Scraper runs on a large proxy network with custom, usage-based pricing. Not cheap at low volume, the savings only kick in once monthly spend clears a certain committed threshold, but it is broadly noted for landing high success rates against some of the most heavily defended sites out there.

Octoparse leans no-code: a visual point-and-click builder, cloud scraping, residential proxies, automatic CAPTCHA solving, and scheduled runs. Exports go to CSV, Excel, or JSON, though Google Sheets export needs the Professional plan. The Standard plan runs $69 a month billed annually, and it suits non-engineers or moderate-scale jobs well.

Apify holds the lead in pre-built scrapers, with a large library of maintained "Actors" targeting specific sites: Amazon, LinkedIn, TikTok, Instagram, Twitter/X, Reddit, and plenty more. Pricing is compute-based, harder to budget for than a flat monthly fee, but the upside is that most common targets already have a maintained scraper sitting there. Rarely a need to build one from zero.

Cloud Scraper, from Web Scraper, runs scrapers built through the Webscraper.io Chrome extension and supports JavaScript-heavy pages, with flexible scheduling and API access built in. It's not built for huge scrapes, though. The Chrome extension architecture tends to break down somewhere past a few thousand pages. Pricing starts at $50 a month.

ParseHub offers a point-and-click interface that handles JavaScript-rendered pages, exports to CSV and JSON, and connects with Google Sheets and Tableau. Vendor lock-in is the noted downside. It suits moderately complex targets well for teams that don't want to write code.

For anyone building an AI pipeline specifically, output format matters as much as crawl speed does. A platform that hands back clean Markdown or JSON natively saves a whole processing step. One that dumps raw HTML means building an extraction layer before that content is even usable by a model. A single web data API, one endpoint covering search, crawling, scraping, and structured delivery, cuts out the work of stitching several platforms together, and that's increasingly the shape that actually serves AI agents without turning into a pile of tools someone has to babysit forever.

Deduplication at scale: why URL-level hashing catches the easy stuff and misses the rest

Hashing the URL string catches the obvious duplicates and misses almost everything else. Session-parameterized URLs, trailing slashes, query strings in a different order, tracking parameters tacked onto an otherwise identical link, all of it slips past a plain URL hash while pointing at the exact same page.

There are really two separate dedup jobs, and treating them as one is the mistake most teams make. Frontier deduplication runs before a URL ever enters the queue, stopping the same link from getting fetched twice, and it needs to run fast enough that it doesn't become the new bottleneck sitting in front of the scheduler. Content deduplication runs after the fetch, catching pages with different URLs but nearly identical content, which shows up constantly in paginated listings, mirrored pages, and A/B test variants.

At high volume, a single shared hash set can't keep up as a dedup mechanism. It turns into exactly the coordination bottleneck the whole distributed setup was built to avoid in the first place. A distributed bloom filter or a consistent hashing scheme fixes that by spreading the dedup state across machines instead of funneling every check through one lookup table. Static URL assignment helps here too, since hashing URLs straight to workers naturally spreads that dedup burden out as a side effect of how the work gets split, rather than something bolted on afterward because someone forgot to plan for it.