Duplicate URL and Content Deduplication in Crawlers

Web crawlers waste a shocking amount of effort. They re-fetch pages they've already seen, or fetch new pages that just repeat something already sitting in the index. Duplication sneaks in at two levels, the URL and the content, through different mechanisms entirely, and catching it takes a stack of defenses, not one clever trick. Skip a layer, and the crawler either burns bandwidth re-indexing old pages or floods its dataset with near-copies that quietly wreck whatever gets built on top of it.

The scale here isn't small. Research on an untreated corpus of 20 billion webpages found roughly one in four records was a duplicate. A Tokyo Institute of Technology study on Japanese web corpora (arXiv:2404.17733) found Common Crawl snapshots from around 2020 had a non-duplicate rate under 20 percent, meaning four out of five pages collected were redundant. Pages from before February 2023 fared even worse, dropping below 40 percent non-duplicate. The pattern holds steady, since the older and bigger the archive, the worse the duplication gets, and it never seems to correct itself on its own.

Three sources feed this, and they stack rather than cancel out. The same URL gets crawled again and again over time. The same content sits at several different URLs at once. And near-identical content gets churned out by templates, pagination, and boilerplate that changes just enough to slip past simple checks. Left alone, this costs real money and real quality: wasted fetch budget, bloated indexes that grow without adding anything new, search relevance diluted when a document competes against its own clone, and for anyone training language models, datasets that cost more to process and bake memorization quirks into the model. Duplication attacks from three directions at once, so no single filter catches it all. That's the case for layering, not for picking one favorite tool and calling it done.

How duplication actually presents in the wild: URL-level versus content-level

Two different problems hide under the word "duplicate," and people who reach for one tool to fix both are setting themselves up to fail.

URL-level duplication happens when two different URL strings point at the same or equivalent content. It's catchable before a single byte gets fetched, just by looking at the string. Content-level duplication is sneakier: two different URLs get fetched, and only after the crawler has the actual page in hand does it become clear the two are identical, or close enough to count.

Agarwal and colleagues laid out three concrete ways this shows up. A regular page and a redirect page can share identical content under different URLs. Two separate redirect pages can point at the same target URL while having different source URLs. And a regular page's URL can itself be the target of some other redirect, so two seemingly unrelated pages turn out to be the same document wearing different clothes.

On the URL side, the usual suspects are familiar to anyone who's stared at server logs too long: session IDs and cookie parameters baked into the URL (Agarwal et al. point to YouTube's ytsession parameter as one example), http versus https, www versus no www, trailing slashes, default ports, inconsistent percent-encoding, and the whole family of tracking junk like utm_source, gclid, and fbclid. Pagination suffixes and query strings listed in a different order cause the same headache.

Content-level near-duplicates are the harder nut to crack. A minor edit, a reordered paragraph, or a templated intro sentence is enough to defeat an exact-match hash, even though the page underneath is functionally the same document. Shared boilerplate (headers, footers, nav menus) makes it worse, since it inflates similarity scores across pages that otherwise share nothing.

The part worth remembering: URL-level dedup happens online, before the fetch. Content-level dedup happens offline, after. Catch a duplicate at the URL stage and bandwidth gets saved directly. Catch it at the content stage and the bandwidth is already spent, because the fetch already happened. That ordering is why URL normalization comes first in any sane pipeline, not as an afterthought.

URL normalization: collapsing equivalent URLs before anything is fetched

Convert every URL to one canonical form before it enters the crawl frontier, not after the page comes back. That's the whole idea, and it's not complicated to state even though it's easy to get wrong.

Here's how fast this gets messy. http://example.com/page, https://example.com/page/, https://www.example.com/page, and https://example.com/page?utm_source=email can all serve the exact same content. Without normalization, a crawler treats them as four separate targets, fetches all four, and stores four copies of one page.

Some normalization moves are safe, guaranteed by RFC 3986 not to change what the URL means: lowercasing the hostname, normalizing percent-encoding, dropping default port numbers, resolving dot segments in the path. Other moves are optional and riskier, because they can change what the server actually hands back. Stripping tracking parameters falls into that bucket (one 2025 Ruby library, NormalizeURL 2025, strips UTM parameters and ad click IDs by default, though it's a setting that can be turned off). Sorting query parameters is another risky one, since some servers genuinely treat parameter order as meaningful, and sorting breaks those sites. Stripping trailing slashes is a third.

Then there's a trickier category: the DUST problem (Different URLs with Similar Text). These are URL strings that look nothing alike on the surface but resolve to similar pages, and no simple normalization rule catches them. Finding these means learning substring substitution rules by mining actual crawl logs. Agarwal and colleagues built on this with a machine-learning approach where rules get mined from logs, generalized with a decision tree, and the resulting model gets applied online as the crawl runs. That handles site-specific quirks no universal rulebook could anticipate, because every CMS and every dev team invents its own URL conventions from scratch.

Over-normalization is the mistake worth watching for, and it's more common than people expect. Strip ?page=2 and suddenly page 2 and page 1 look identical to the crawler, and real content disappears. Strip ?category=shoes in the wrong context and two genuinely different product listings collapse into one. Rules need tuning per site, sometimes per CMS, because what's safe to strip on one domain destroys content on another. So the practical rule is this: optional normalizations should be a switch someone can flip, never a blanket policy stamped across every domain the crawler touches.

Normalization also has a hard ceiling. Two URLs that were never structurally similar to begin with, no shared pattern, no shared substring logic, simply won't get caught here. That's a job for content-level checks further down the pipeline, and pretending otherwise just wastes engineering time.

Tracking seen URLs at scale: Bloom filters, hash tables, and distributed sets

Every link a crawler pulls off a page needs a check against the set of URLs already seen. Skip that step and the crawler loops forever, chasing its own tail while the frontier fills up with junk it's already processed.

At small scale this is barely a problem. A hash set in memory handles a few million URLs without breaking a sweat. At billions of pages, that stops working entirely. Neither the RAM nor the disk seeks a naive per-URL lookup demands are realistic anymore, so the data structure itself has to change.

Bloom filters are the classic fix. Burton Bloom described the structure back in 1970: a bit array that answers "have I seen this before?" with zero false negatives and a tunable false-positive rate, using well under twenty bits per element no matter how large the original key was. The failure runs only one direction. A false positive means the crawler wrongly believes it's already seen a URL and skips it, silently dropping something new. It never forgets a URL it actually inserted, so it never re-crawls a page it already has. At billion-URL scale, occasionally dropping a new URL by mistake is a cost worth eating. The memory savings aren't optional at that point, they're the entire reason the crawl runs at all.

IRLbot is the reference case worth knowing. The IRLbot crawl ran for 41 days and pulled down 6,380,051,942 unique HTML pages. That scale required progressively more sophisticated data structures as the page count climbed into the billions. That progression is basically a live demo of how the seen-URL data structure has to evolve as the page count climbs into the billions.

For crawls spread across multiple machines, hash-based assignment solves a different problem: which machine owns which URL. Compute DocID = hash(URL), then Bucket Index = DocID mod n, and every URL lands on exactly one machine, so no two nodes waste effort crawling the same thing twice. Consistent hashing extends this by spreading the URL set evenly across machines and letting servers get added or removed without reshuffling the whole assignment scheme.

Redis Sets show up as a practical middle ground in production distributed crawlers, offering exact membership tracking at the cost of higher memory usage. The tradeoff runs opposite to Bloom filters: Redis gives perfect accuracy, zero false positives, but it eats real memory at scale to get there. Neither wins outright, and anyone claiming otherwise hasn't run both at scale. The right pick depends on crawl size, memory budget, whether a small false-positive rate is tolerable, and whether the whole operation runs on one machine or a fleet.

Exact content hashing: the first content-level filter after fetching

Once a page gets fetched, the first content-level check is usually the simplest one available: hash the content (MD5, SHA-1, whichever), compare that fingerprint against a stored set of fingerprints already seen, and throw the page out on a match.

CCNet, the pipeline behind CC-100 and a lot of the corpora used to train language models, runs this at the paragraph level rather than the whole page. Text gets lowercased, numbers get swapped for a placeholder, punctuation gets stripped, and the document gets split by newline into paragraph-sized chunks. Each chunk gets a SHA-1 hash, and those hashes get compared across the whole corpus to catch exact duplicate paragraphs, not just exact duplicate pages.

Dolma takes a related approach, deduplicating paragraphs by exact matching using a Bloom filter, pairing the precision of exact matching with the memory savings Bloom filters are known for. OSCAR's line-level exact deduplication reportedly cut about 55 percent of its content (per the RefinedWeb/Falcon LLM paper, arXiv:2306.01116), which says something blunt about how much exact duplication survives even in web data that's already been through a filtering pass.

The ceiling here is hard and unforgiving, and it's worth being blunt about what exact hashing simply cannot do. Change one character, reorder a sentence, swap a byline, and the hash comes out completely different, even though a human reader would call the two pages basically the same thing. Exact hashing is the right tool for re-served pages, mirror sites, scraped republications, and unchanged recrawls. It does nothing for templated pages with slightly different variable content, paginated variants of the same listing, or a syndicated article that got lightly edited before it ran somewhere else.

Near-duplicate detection: SimHash and MinHash for content that almost matches

Near-duplicate detection isn't optional at web scale. The kind of variation that defeats exact hashing (minor edits, reordered paragraphs, templated intros) is everywhere in web data, not a rare edge case worth ignoring.

SimHash handles this by building a compact, fixed-size bit fingerprint of a page's content, rather than a cryptographic hash. Similar pages produce similar fingerprints, so the Hamming distance between two fingerprints (how many bits differ) stands in for how similar the actual content is. Comparing bit patterns is cheap, so SimHash scales well to large pairwise comparisons and has been adopted in production deduplication pipelines operating at web scale.

MinHash, developed by Broder in 1997, comes at the same problem from a different angle: estimating Jaccard similarity through sketches. For each document, a sketch gets built from the minimum hash values across shingles (small overlapping chunks) of the text. Comparing the fraction of matching values between two sketches estimates how similar the documents are, without comparing the full text directly. The WanJuan-CC corpus applies 128 hash functions across 5-grams of each document to build its MinHash signatures for dedup.

Running MinHash on every pair of documents directly is still too slow at scale, since pairwise comparison grows quadratically. MinHash-LSH (Locality-Sensitive Hashing) fixes that by grouping documents into buckets so only documents in the same bucket ever get compared. A common setup uses r buckets, each built from a concatenation of b MinHash values. The Japanese web corpus study cited earlier (arXiv:2404.17733) used b = 20 and r = 40 in production, and at that setting, two documents with a Jaccard similarity of 0.9 got flagged as duplicates with roughly 92.5 percent probability.

RefinedWeb, the pipeline behind Falcon LLM, combined exact substring dedup with MinHash fuzzy dedup and removed roughly half its content overall. For comparison, MinHash alone removed only around 10 percent in GPT-3's pipeline and around 26 percent in The Pile's (numbers per arXiv:2306.01116). SimHash is the right call where speed matters most, real-time or near-real-time checks during an active crawl. MinHash-LSH is the right call for offline pipelines built to clean LLM training data, where getting the Jaccard estimate right matters more than getting an answer fast. Treating these as interchangeable is the mistake to avoid; they're built for different clocks.

Where MinHash-LSH breaks under load, and how LSHBloom addresses it

MinHash-LSH solves the quadratic comparison problem. It does not solve the memory problem, and that gap is exactly where deployments start to fall over in production.

Every document's MinHash signature still has to sit in memory, or at least stay fast to reach, for bucketing and comparison to work. Fine at moderate scale. Not fine once the dataset outgrows what a single machine can comfortably hold. One documented case tried LSH-based dedup on a 600 GB dataset using an AWS spot instance with 30 GB of memory, and the job died with out-of-memory errors. The algorithm wasn't wrong. It hit a wall before it ever got the chance to do the job it was built for.

The time LSH saves over brute-force pairwise comparison comes at a direct cost in memory footprint, and that tradeoff is exactly what collapses once dataset size outruns a single machine's memory budget. LSHBloom closes that gap by applying Bloom filter techniques to the LSH bucketing step itself, so the memory curve stops scaling in lockstep with dataset size. It's the same lesson URL-seen tracking already learned with Bloom filters, showing up one layer higher in the stack: past a certain scale, exact bookkeeping stops being affordable, and the tools that survive are the ones willing to trade a small, controlled error rate for a memory footprint that doesn't buckle under its own weight.