Distributed Crawling with Cloud Infrastructure

This piece is about the architecture behind crawling the web at scale for AI, and the central claim is simple: the hard part was never writing the scraper. It's building a system that survives contact with the real internet, billions of requests in, without quietly lying to you about whether it worked.

Distributed crawling as an infrastructure problem, not a scripting problem

47Billion's 7Seers team ran into a problem that had nothing to do with their model. They had the prompt engineering dialed in and the model selection sorted, and none of it mattered because the pipeline couldn't fetch a current government policy page. A perfect engine bolted to an empty fuel tank still isn't going anywhere.

AI systems need live web data because training cutoffs leave models stale within months to years, and a pipeline that can't deliver today's page produces wrong answers no matter how good the model behind it is. A single process running sequentially cannot meet that freshness and volume demand. Scraping for a search index just had to answer "did the data arrive?" Scraping for an LLM has to answer whether a model can reason coherently over what landed. That's a different bar, and the web doesn't make it easy to clear: JavaScript-rendered pages return empty shells to naive HTTP requests, rate limits and CAPTCHAs sit in wait, and IP detection blocks any fixed address once request volume gets real. None of that is solvable by writing a better script. It's solvable by building better infrastructure, which is what the rest of this piece is about.

Raw HTML as the wrong unit of work for a distributed crawling pipeline

When a crawling system hands an AI pipeline raw HTML, it hasn't finished the job, it's just relocated it. Raw HTML is dense with navigation menus, cookie banners, and JavaScript loaders that have nothing to do with the content a model actually needs, and 47Billion's pipeline work found that a typical page runs roughly 60-70% noise relative to its real content. Feed that to a model and the output comes back incoherent, and the model isn't the broken part. The input was.

The sharper danger is the failure that doesn't announce itself. A naive HTTP scraper returned documents that were structurally valid and completely empty, with no exceptions raised anywhere, so those empty documents turned into slides with the right formatting and nothing inside them. Nobody caught it until someone audited the output by hand. That's the kind of failure that costs real time, because it looks exactly like success until a human notices the emperor has no slides.

The fix rebuilds the pipeline stages rather than patching the surface. Content cleaning and structure extraction have to be built as first-class pipeline stages from the start, and the output contract for a crawling worker should be clean Markdown or structured JSON, something an LLM can actually use, rather than a raw DOM dump. Natural language extraction prompts ("pull every product with name, price, and availability") hold up when a site's layout changes in a way that would snap a CSS selector clean in half. And the definition of success has to move with it: a worker that returns an empty structured document should be flagged as a retriable error, not logged as a win, because the pass/fail line belongs at the content level, the HTTP status code aside.

Compute Strategy and the Ceiling of a Distributed Crawling System

Once the output format is settled, the next question is how the thing actually runs, and that turns out to be a moving target rather than a one-time decision. Static HTTP fetching and full browser automation cost wildly different amounts, and that gap is enough to decide whether the whole system pencils out at scale.

A headless Chromium instance eats 200-400 MB of RAM just sitting there. Spinning up a fresh instance per request at any real volume drives the bill into horror-movie territory. Session pooling, instance reuse, and routing requests between lightweight and heavyweight paths aren't nice-to-haves bolted on later, they're the baseline requirement for running this in production. 47Billion's team brought in Playwright to handle JavaScript rendering, which solved the rendering of dynamic content, but target sites then began fingerprinting the automation itself and blocking it at the IP level.

Provider choice is cost strategy in disguise. Hetzner's flat-rate pricing on cloud and dedicated servers, with high traffic volume included, suits persistent, high-density scraping clusters because there's no egress bill lurking around the corner. AWS Lambda and GCP Cloud Functions cut idle-time costs to zero but charge meaningfully for egress when large volumes of HTML move through them. Serverless looks appealing for workloads that spike and vanish, but cold starts, execution time ceilings, and egress charges make it a poor fit for sustained, high-volume crawling. It works fine for infrequent triggered jobs and buckles under the kind of constant concurrency a production pipeline actually runs. Which setup wins depends on whether the workload is bursty or sustained and how much data moves through it, not on which provider has the better logo.

How proxy and network architecture determines actual success rate, beyond theoretical throughput

Real throughput is success rate multiplied by request volume, and success rate gets decided almost entirely by what happens at the network layer before any other logic in the system even wakes up. So proxy architecture carries as much load as compute architecture, not a configuration afterthought.

The numbers make the gap hard to ignore: residential proxies land 85-99% pass rates against Cloudflare-protected sites, while datacenter IPs fall substantially short of that. That spread is wide enough to swing the economics of an entire crawl one way or the other. And the cost structure compounds the problem rather than softening it. Residential proxies bill by the gigabyte of bandwidth consumed, and even a failed request still burns bandwidth. So if your success rate is poor, the cost per gigabyte of actual, usable data climbs well past the sticker price. Failure rate here is a line item, not just a reliability number.

Geography adds a second layer on top of plain detection-avoidance. NielsenIQ's spiders run precisely geolocated, a detail its CTO of scale-ups confirmed at OxyCon 2025. Pricing, availability, and localized content change by region, so a request routed through the nearest convenient exit node can return data that's technically successful and substantively wrong. The architectural consequence: proxy rotation, session handling, and geolocation need to be configurable per job, not locked in as a single global setting, because different targets demand different network profiles.

Job orchestration patterns that keep a distributed crawl from becoming a maintenance burden

Fault tolerance in a system like this is an orchestration decision made upfront, not a bug-fixing exercise after the fact. A crawler running without a durable job queue, without retry logic, without any state tracking will quietly accumulate failures that stay invisible until an AI pipeline starts handing back wrong answers downstream. The orchestration layer is where that invisibility either gets caught or gets buried.

Engineering teams are moving away from scheduled cron jobs toward event-driven, cloud-native architectures, where a message queue or event bus handles URL ingestion, worker dispatch, retries, and result delivery, and persists that state independently of whatever individual worker happens to be running at the time. That persistence matters because workers die, restart, and get killed by aggressive bot detection constantly, and the job state needs to survive all of that without anyone noticing a gap. Dead-letter queues are part of the same discipline: URLs that keep failing need to surface for a human to look at rather than vanish silently, and the system needs to distinguish a transient hiccup worth an immediate retry, a rate-limit response worth a backoff, and a permanent block worth escalating or skipping.

Isolation matters just as much as retry logic. Kubernetes handles workloads generally by running each one in its own pod with defined resource limits, and this pattern means one target that trips aggressive bot detection doesn't starve every other job running alongside it. And rather than polling endlessly to check whether a job finished, a well-built orchestration layer emits events when a batch completes or a worker fails, so downstream pipeline stages can start moving the moment results are ready instead of waiting around for an entire run to wrap up. None of these patterns are exotic. They separate a crawl that works once in a demo from a pipeline that still works six months later without anyone babysitting it.

Designing fault tolerance and observability in from the start

At billions of requests, failure is no longer an edge case; it is the operating condition. If a crawling system has no built-in observability, it will misreport its own health, delivering incomplete data to an AI pipeline while every dashboard says everything's fine.

The 7Seers empty-document failure is the clearest illustration available of what this actually looks like in practice. The scraper returned structurally valid output, no errors thrown anywhere, the pipeline accepted it without complaint, and the LLM produced confident, well-formatted nonsense on top of it. Nobody caught the problem until a person read the AI-generated output and found no substance behind the formatting. Observability exists to close that gap between "looks fine" and "is fine.

Closing it means tracking the right things: success rate per target domain, content yield per page (did the structured output contain meaningful data, not just well-formed tags), retry rates, proxy failure rates, and queue depth, not just HTTP status codes. Monitoring tooling that's now standard in Kubernetes environments makes it possible to set anomaly thresholds, alerting when content yield for a given target drops below its usual baseline; this catches the failure before it quietly propagates into a report someone trusts. Pulumi's cloud trends analysis identifies AIOps tooling as maturing into a standard part of cloud operations, capable of automating failure detection and triggering remediation on its own, which matters a great deal once a crawling system is running across thousands of targets at the same time. Infrastructure as Code practices, policy-driven deployment, pre-deployment validation, auto-remediation, carry that monitoring configuration along with the infrastructure itself, so teams that take IaC seriously don't end up with worker pools nobody's watching. Retrofitting observability after a silent failure has already shipped bad data downstream is expensive and unreliable by comparison. Building it in from the start is the cheaper version of the same insurance policy.

Choosing between building the crawling layer and routing through a managed API

Whether you build the full distributed crawling stack or route through a managed web data API hinges on something other than cost. It's a question of whether crawling is a core competency for the team involved, and most teams underestimate how much ongoing maintenance the build path actually demands.

Building gives full control over proxy selection, rendering strategy, output format, and retry logic, and that control earns its keep when the crawling infrastructure itself is a genuine product differentiator, the way geolocated, product-level data is the core business asset for a company like NielsenIQ. For most teams, crawling is a means to an end rather than the product itself, and that's where the managed path earns its keep instead: proxy rotation, browser rendering, CAPTCHA handling, and IP reputation management all get handled by the provider, freeing engineering time to go toward the pipeline logic that actually produces business value. CommercePundit's AI infrastructure guide draws a useful line between components where managed services are always the right call (object storage, basic networking, standard monitoring, pre-trained foundation models) and components worth building when the capability is genuinely differentiating. A web data API sits firmly in the first category for most teams.

The honest objection to going managed is vendor dependency: a provider that changes pricing, deprecates an API, or has an outage puts real risk on any pipeline that's core to a product's intelligence layer. Evaluate providers the way any critical vendor should be evaluated, on reliability track record, SLA terms, and whether the API contract has actually stayed stable over time.

Structuring a distributed crawling system to serve AI agents, not just batch pipelines

Everything built up through this piece, the output contracts, the compute routing, the proxy layer, the orchestration, the observability, gets tested differently once the consumer on the other end is an autonomous agent instead of a scheduled batch job. A batch pipeline runs on a schedule and tolerates some latency, but an agent asks a question mid-reasoning and needs an answer now, so the crawling layer underneath it has to behave less like a nightly job and more like a dependency the agent can call on demand, with freshness and reliability guarantees baked into the response rather than promised in a status page somewhere else.

That shift doesn't rewrite the architecture described above so much as raise the stakes on every piece of it. A silent empty-document failure that feeds a quarterly report is a bad day. The same failure feeding an agent that's making decisions in real time compounds immediately, because there's no human in the loop checking the slides before they go out. The structural requirements this piece has walked through, clean output contracts, smart compute routing, proxy strategy matched to the target, orchestration that retries and isolates failures, observability that catches the quiet failures before they spread, are the same requirements for an agent-facing system, just with less room for anything to slip through unnoticed.