Incremental and Freshness-Based Crawling

Crawling used to mean one thing: sweep the whole web (or the whole site), rebuild the index, wait, repeat. That worked fine when the web was smaller and pages sat still for weeks at a time. It doesn't work anymore, and the reason is simple math: the web changed faster than the old crawling model ever accounted for.

That model rests on two assumptions. First, that pages change slowly and roughly at the same pace as each other. Second, that re-fetching a page you already have, unchanged, is a cost worth eating. Neither one holds up in 2026.

Bot traffic tells the story on its own. Cloudflare found that AI and search crawler traffic climbed 18% on a fixed-customer basis between May 2024 and May 2025. Break that down and it gets sharper: GPTBot alone grew 305% in that window, and Googlebot, already a mature crawler with a huge footprint, still grew 96%. That's not incremental growth, that's crawlers multiplying their own request volume year over year. Any system still re-crawling everything at a fixed cadence runs straight into infrastructure limits and politeness limits (the informal and formal rules against hammering a server) at the same time.

And here's the twist nobody expects going in: crawling more often doesn't make your data fresher. It mostly just means fetching the same unchanged pages more times. Freshness and frequency are related, but they're not the same lever.

For AI pipelines specifically, this shows up as two separate failure modes. One is stale data: a retrieval system built on a recrawl from three days ago answers a question about something that happened yesterday, confidently, and wrong. The other is noisy data: raw HTML, whether it's a week old or five minutes old, drags along navigation bars, ad markup, and tracking scripts that need scrubbing before any language model can use it. Solving freshness without solving format gets you half the problem. So the real question this piece is built to answer: how do you keep a big dataset genuinely current without wasting fetches on pages that haven't moved?

What incremental crawling actually means and how it differs from a scraper

A scraper answers one question: what's on this page right now? It's a snapshot. No memory, no history, no sense of whether the page looked different yesterday.

An incremental crawler asks a different question: has this page changed since the last time it looked, and if so, what changed? That requires keeping state, page by page, fetch by fetch. It's the difference between a photograph and a security camera.

An incremental crawler has to do two jobs at once, and they pull in slightly different directions. It has to keep the stored collection current, meaning it detects changes and pulls in the new content. And it has to pick a smart revisit policy, meaning it decides which pages are worth checking today and which ones can wait. Get the first part right without the second and you're back to full sweeps. Get the second right without the first and you're just guessing.

Freshness, in this context, has a specific meaning: it's the gap between when a page actually changes on the live web and when that change shows up in your dataset. The entire job of an incremental crawler is shrinking that gap, page by page, without blowing up the fetch budget.

This is different from batch refresh in a concrete way. Batch refresh waits for the next scheduled sweep, no matter what. Incremental crawling doesn't wait. It's continuously applying updates to whichever slice of the dataset looks most likely to have moved, right now, this hour.

For AI pipelines, this distinction carries more weight than it does for a traditional search index. A stale search result is annoying. A stale answer from a language model is worse, because the model doesn't say "this might be outdated." It states it as fact, grounded in content that no longer describes reality. Confidently wrong beats visibly outdated for sheer damage per incident.

One more wrinkle: site structure varies wildly, and the crawler has to handle that variation before it can even think about incrementality. Single-page apps render content with JavaScript after the initial load. Traditional server-rendered pages hand over the full HTML upfront. Some sites are best crawled through a sitemap, others only make sense by following links page to page. The incremental layer has to sit on top of a discovery layer flexible enough to handle all of that, not a single fixed sweep pattern bolted onto everything.

The counterintuitive relationship between change frequency and optimal revisit rate

The obvious instinct: check frequently-changing pages often, check rarely-changing pages rarely. Match your fetch rate to the page's own rhythm. Sounds right. Isn't quite right.

Google Research's Kevin S. McCurley laid out the edge case that breaks the intuition: picture a page that changes every single time it's accessed. No matter how often you fetch that page, hourly, every minute, constantly, you will never hold a copy that matches the live version. The fetch itself can't outrun the change. Frequency alone hits a ceiling.

There's a practical wall here too. Crawl too aggressively and you trip politeness conventions, the informal (and sometimes formal, via robots.txt crawl-delay directives) rules against overwhelming a server with requests. Do that and you get rate-limited or blocked outright, which torches the whole point of checking often in the first place.

So raw change frequency is one input into scheduling, not the whole formula. What the crawler actually wants to optimize is freshness gained per fetch, not frequency for its own sake. Research into smarter scheduling approaches suggests the payoff can be substantial: algorithms that weight pages by expected change yield meaningfully higher crawl-hit rates with lower overhead than the baseline approach. That's the payoff of scheduling smart instead of scheduling often.

The bigger idea underneath all this: the goal was never maximum crawl frequency. It's maximum freshness for each unit of crawl cost spent. And this isn't just a search-engine problem dressed up in academic language. Prefetching proxies, alert and notification systems, mirroring and archival tools, competitive intelligence platforms, they all run into the exact same math. Anyone polling a remote resource for change is solving a version of this.

How modern adaptive scheduling systems estimate change probability

Prediction is the engine here. A modern crawler keeps a running model, per URL, estimating how likely that specific page is to have new content right now. That model is what makes scheduling adaptive instead of a flat rule like "check every 24 hours."

A handful of signals feed that model. Historical change patterns for that exact URL matter most: has it flipped weekly, daily, monthly? Content type matters too, since a news article and a corporate "about" page live on completely different clocks. HTTP headers help cheaply: Last-Modified, ETag, and Cache-Control often tell you plenty without a full fetch. Some analysis of crawl scheduling also points to link graph centrality as a contributing signal, though Google's own public documentation doesn't confirm these as Googlebot's specific ML inputs.

What comes out the other end is a next-check timestamp per URL, based on expected freshness gain rather than a fixed interval applied to everything. In practice, that means a news homepage might get checked hourly while a company's "about us" page gets checked once a month, both running on the same infrastructure, just scheduled differently because their content behaves differently.

Indexing has to keep pace on the back end too. Instead of rebuilding a full index after each sweep, updates apply continuously: new documents slot into index segments, deleted documents get flagged in deletion bitmaps, and periodic compaction jobs merge everything back down. That keeps the lag short between a page changing and that change becoming queryable.

The payoff for a developer building on top of this kind of system: the pages that matter most, the time-sensitive ones, stay fresh without dragging the crawl cost up across the entire corpus. You spend your budget where it earns freshness, not evenly across everything regardless of whether it needs it.

Reference architecture for running incremental crawls at scale

Synchronous crawling, fetch a page, wait, process it, fetch the next one, simply can't hold up freshness across thousands of URLs. It's one thread doing one thing at a time. Asynchronous execution breaks fetching and processing apart, and in doing so solves rate-limiting, concurrency, and retry handling all at once, because none of those steps are blocking each other anymore.

A representative cloud-native pattern, using AWS services as an example, looks something like this. EventBridge Scheduler kicks off crawl jobs on a schedule, rate-based or cron-based, which the developer can configure to reflect the crawl cadence they've determined for their use case. AWS Batch or ECS Fargate runs the actual fetchers and parsers in containers, scaling up elastically when volume spikes without needing to keep idle capacity around the rest of the time. Object storage holds the crawled content itself. A metadata store tracks per-URL state: when it was last fetched and when it's expected to change next. A message queue passes work between components. Step Functions handle the workflow logic, including retries and failure paths, so one broken fetch doesn't take down the pipeline.

A monitoring layer surfaces the metrics that actually matter for a crawler — pages fetched per second, deduplication rates — so someone gets alerted the moment freshness starts slipping instead of finding out a week later.

Deduplication deserves its own callout here. Without it, the system just fetches faster while replicating the exact waste of full recrawls, only now at higher speed and higher cost. Every new fetch has to get compared against stored state to confirm something actually changed before it's treated as new.

A few reliability patterns are worth building in early rather than bolting on after an outage: distributed caching, automated failover, adaptive load balancing. These catch the quiet failures, a proxy rotation silently breaking, a headless browser pool running out of capacity, that don't throw an obvious error but slowly corrupt freshness in the background.

Managed crawling APIs change the shape of this whole picture for teams that don't want to own proxy infrastructure, retry logic, and browser rendering themselves. The scheduling logic and the downstream pipeline still belong to the team building it, but the fetch layer itself gets handed off. The trade-off is straightforward: self-hosting gives full control over scheduling policy down to the URL level, while managed APIs cut operational overhead but may limit exactly how granularly the revisit cadence can be tuned.

How freshness-based crawling feeds AI pipelines and RAG systems specifically

A retrieval pipeline built on a stale index has one job and it fails at it quietly: it answers today's question using last month's data. The model stays fluent, the sentences read fine, but it's grounding its answer in something that's already changed underneath it.

The fix is live web access, either through a continuously updated incremental index or a fetch triggered right at query time. Either way, the answer gets grounded in a document that's citable and current, instead of leaning on whatever the model happened to memorize during training.

That grounding matters for more than just freshness. When a model answers from a document it can point back to, it has less room to fabricate. Retrieval is the most direct lever available against hallucination, and hallucination and factual consistency remain widely documented challenges in enterprise RAG deployments. So retrieval quality and freshness aren't separate concerns, they're the same concern viewed from two angles.

Classic RAG runs one retrieve-then-generate pass per user turn, and freshness in that setup depends entirely on when the index was last touched. Agentic RAG works differently: the agent decides when to retrieve, rewrites its own query between steps, switches between tools like vector search, web search, or SQL as needed, checks whether its draft answer is actually supported by what it found, and re-retrieves if a claim isn't backed up. Freshness stops being a fixed schedule and becomes something the system checks for on the fly. That costs more tokens and adds latency per turn, and it's a trade worth naming plainly rather than glossing over, but it buys higher faithfulness in the final answer.

The demand for this is already mainstream rather than experimental. Enterprise adoption of retrieval-augmented generation has moved well beyond the experimental stage, with a large share of organizations deploying generative AI incorporating retrieval frameworks. Freshness-based crawling isn't a niche optimization sitting off to the side, it's infrastructure underneath the dominant pattern enterprises are already running.

Where this is heading: retrieval itself is turning into a learned policy, where the agent picks not just which retriever to call but chunk size and re-ranker depth per individual query. Early systems built this way show meaningful gains over static, one-size-fits-all retrieval configurations.

Why data format is as important as crawl freshness for AI consumption

Freshness alone doesn't finish the job. A page fetched five seconds ago, handed over as raw HTML, still needs cleanup before a language model can do anything useful with it. Navigation menus, ad markup, tracking scripts, layout tags, all of that eats tokens and muddies retrieval, whether the fetch happened five seconds ago or five days ago.

Markdown has become the preferred format for feeding pages to a language model, for reasons that are fairly mechanical. It's readable by a person and parsable by a machine at the same time, and a model can follow document structure through headings, lists, and tables instead of wading through a flat wall of tags. It also uses meaningfully fewer tokens than the equivalent page in raw HTML, which is a real cost saving, not a stylistic preference. Headings, lists, and source metadata survive the conversion, so a RAG pipeline can chunk the content cleanly and still carry citations along with it. And because the format is uniform across every source, the chunker doesn't need a custom rule set for each site it touches. Predictability, here, is an engineering property, not a nice-to-have.

JSON has its place too, specifically when a downstream system needs typed, queryable fields rather than flowing document text.

There's a newer standard worth watching: llms.txt, a plain Markdown file sitting at a site's root that gives AI tools a structured, low-noise map of the site's most important content, essentially a sitemap built for AI rather than for search engines. In May 2026, Google added llms.txt to Chrome Lighthouse's new "Agentic Browsing" audit category, which signals real institutional interest. That said, as of early 2026, broad, confirmed adoption of llms.txt by major AI systems has not been publicly established. And there's an implementation trap worth flagging directly: generating individual indexable Markdown copies of every page on a site raises questions about how those copies should be managed relative to the originals.

None of this is cosmetic. Data quality consistently ranks among the top operational priorities cited by enterprise leaders. Format and cleanliness aren't afterthoughts tacked on after the crawl, they're where AI output quality actually gets decided, upstream of anything the model itself does.

Choosing between crawling approaches and tools for freshness-driven pipelines

Four questions separate a crawler built for freshness from one that just downloads pages and calls it done.

Does it return clean Markdown or typed JSON, or does it hand back raw HTML and leave the cleanup to whoever's building the pipeline? Does it support scheduled re-crawls on a cadence set by the developer, or is it strictly a one-off fetch tool? Does it absorb proxy rotation, headless browser rendering, and retry logic on its own, or does the team have to own all of those failure modes directly? And does it expose REST or MCP (Model Context Protocol) endpoints, so an AI agent can call it as a tool without someone writing glue code to connect the pieces?

Open-source frameworks give full control over crawl strategy. Crawl4AI, in its current 0.8.x line, supports breadth-first search, depth-first search, and best-first deep crawling, along with an AdaptiveCrawler mode that stops on its own once it's gathered enough relevant content. That last piece is a real, working example of the "freshness per fetch" principle covered earlier in this piece, not just a theory. The cost of that control is that the team owns the infrastructure and the anti-bot handling that comes with it.

Managed APIs move that infrastructure burden onto the provider, so the team spends its time writing pipeline logic instead of debugging a browser pool that quietly ran out of memory at 2am. The real question when evaluating one isn't whether it's managed, it's whether its scheduling granularity actually matches the freshness requirements of the pipeline being built on top of it.

The demand side backs up why this decision matters right now rather than in some hypothetical future. A Censuswide survey of 506 web scraping professionals across the US and UK found 74% reported their businesses had seen increased demand for web data over the prior 12 months. Choosing a crawling approach isn't an academic exercise at this point, it's a capacity planning decision.

For teams that want scheduling, structured Markdown or JSON output, and MCP support without standing up and babysitting their own crawling infrastructure, a unified API covering all three is the shape the market is converging on, built for exactly the kind of freshness-driven pipeline this entire piece has been describing.