Duplicate Content Detection During Crawling

Duplicate content detection during crawling comes down to plumbing rather than philosophy. Two pages that differ only in a session ID or a rotating ad banner still look brand-new to a crawler that isn't watching for it, and that mistake gets expensive fast: wasted bandwidth, bloated storage, and every downstream job redoing work on content wearing a different mask.

The cost shows up three times. Bandwidth burns re-fetching what's already sitting in storage. Storage fills with rows that add nothing. And every stage after that, parsing, indexing, embedding, training, pays the tax again on the same duplicate. For teams building training corpora, this actively hurts the output: duplicate-heavy data pushes a model toward memorizing boilerplate and stray personal info instead of learning from varied examples. URL canonicalization and basic change detection, done right, cut wasted scrapes by a real margin. What matters is the right tool at the right stage, and knowing why that tool works there and nowhere else.

URL canonicalization as the first and cheapest deduplication gate

Before a crawler fetches anything, it should ask a cheaper question first: has this URL, dressed up some other way, already been seen? Canonicalization strips the parts of a URL that don't change the content: UTM tracking, session IDs, http versus https, a trailing slash, a domain alias pointing at the same site. Normalize the URL, check it against the set already seen, skip the fetch if it matches. That's the whole trick, and it costs nothing, because no network request happens at all.

It doesn't catch everything, and it never will. Some traps are baked into the structure of the site. Event calendars that spit out a fresh URL for every single day forever are the classic case: /events/2025/01/01, /events/2025/01/02, on into infinity, each page nearly identical to the last. Session-based URLs mint a new unique string for the same page every visit. Content gets syndicated across sites under a slightly different path. And localization creates duplicates on purpose: the same article sitting at three URLs for three regions.

The fix that holds up is keeping canonical URL state in a feature store, keyed by source, canonical URL, and crawl date. That makes the check idempotent: re-crawl the same site next month and the gate behaves the same way, no surprises. Once canonicalization has done what it can, the only path left is looking at the actual content. That's where hashing takes over.

Exact-match hashing for content already fetched

Once a page is fetched, the next cheap move is a plain fingerprint: run MD5 or SHA-1 over the content, store the hash, compare it against hashes already on file. Match found, duplicate confirmed, done.

The catch: cryptographic hashes are built to be touchy, brutally so. Change one word in a million-word document and the hash comes out completely different, with zero relationship to the original. That unpredictability is the whole point of a cryptographic hash, security depends on it, but it means exact-match hashing only catches perfect copies. A page with a different ad banner or a live timestamp in the footer sails right past as "new."

The fix is normalizing content before hashing: strip whitespace, drop boilerplate, then hash what's left. That catches near-exact duplicates differing only in formatting, sitting in the middle ground between strict exact-match and true near-duplicate detection. At real scale, billions of URLs, memory becomes the wall. A bucketing approach groups similar signatures to avoid holding every signature in RAM at once. In the layered view of a crawl pipeline, this sits at the URL frontier and the post-parse content store, catching what canonicalization missed and handing off the harder cases to the next layer.

SimHash: the production standard for near-duplicate detection at scale

Here's where the real engineering starts. SimHash builds a fingerprint where similar documents produce similar fingerprints, a deliberate contrast with how cryptographic hashes behave. Two pages sharing most of their content end up with 64-bit fingerprints differing in only a couple of bits. Pages with genuinely different content differ in a lot more. Distance between fingerprints becomes a stand-in for distance between meanings.

Google's own research backs a specific setup: for a repository of 8 billion webpages, 64-bit SimHash fingerprints with a maximum Hamming distance of 3 bits gave a workable balance of precision and recall (per Google Research's published findings). That's a threshold tested in production, at web scale.

The harder problem is lookup. Finding every fingerprint within 3 bits of a target, across a billion-item store, isn't something brute-force comparison handles without burning absurd amounts of compute. Probabilistic lookup trades a sliver of recall, landing around 95% against exhaustive search, for a big win in speed (4 to 14 times faster) and memory (2 to 10 times less RAM), tested on a 70-million-page collection. Storage footprint is worth considering when choosing between fingerprinting approaches, and it matters when storage is the line item under scrutiny. Production SimHash deployments typically combine 64-bit fingerprints, a 3-bit Hamming threshold, and a partitioning scheme that cuts down how many pairwise comparisons the system has to run. Pick SimHash when fingerprint size and lookup speed matter more than catching every subtle rewording.

MinHash and shingling for set-overlap similarity

MinHash asks a different question than SimHash. Instead of counting bit differences, it measures Jaccard similarity, the overlap between the sets of word chunks two documents share. That makes it far more sensitive to paraphrasing and rearranged sentences, exactly the cases where SimHash's bit-difference approach can miss the connection entirely.

The building block is shingling: chop a document into overlapping word or character sequences (n-grams), and treat that set of chunks as the document's fingerprint. Comparing full sets directly doesn't scale, so MinHash approximates it: compute a fixed-length signature, a small vector of minimum hash values, that preserves the Jaccard similarity in expectation. Comparison becomes comparing short vectors instead of enormous sets.

The FineWeb pipeline runs this at scale, and it's worth walking through exactly how. Each document gets tokenized into overlapping word chunks, then a fixed number of MinHash signatures per document, split into bands of rows apiece. Documents that collide in at least one band get clustered as near-duplicates, and one representative survives from each cluster. Crucially, this dedup runs separately on each Common Crawl snapshot rather than across all of them combined, to avoid flattening out topical and time-based variety (more on that below). FineWeb was distilled from multiple Common Crawl snapshots, making it one of the biggest public proof points for MinHash dedup at scale. Reach for MinHash over SimHash when the worry is subtle rewrites and rearranged text, not just raw byte-level similarity.

Bloom filters as memory-efficient URL and content dedup at the frontier

A Bloom filter answers one question, cheaply: has this URL been seen before? It says "definitely not" with total certainty, and "probably yes" with a tunable error rate. It never says "definitely not" when the answer is actually yes. That one-directional error is the entire design, and it's a feature, not a flaw.

For a crawler, that lopsidedness is exactly the right shape of risk. A false positive means skipping a URL that hasn't actually been crawled yet, a minor miss, tunable down to a low rate. A false negative, re-crawling something already processed, simply can't happen. The tradeoff sits entirely in the crawler's favor.

The memory savings are substantial. A Bloom filter tracking a billion URLs runs around 1.2 GB, versus more than 12 GB for the same set stored in Redis, something close to a 90% cut in memory footprint. Parameters get tuned ahead of the crawl, sized to expected corpus volume and a target false-positive rate, and the filter's bitmap splits across nodes in a cluster, each one owning a range, so the whole setup scales past what one machine's RAM could ever hold.

Redis Sets still win on precision, zero false positives, so the real question is whether occasional skipped URLs are worth the memory saved. One hard constraint either way: Bloom filters don't support deletion. Once a bit flips on, it's on for good, which matters for any crawler that needs to revisit and re-check content on a schedule.

Where in the pipeline each technique belongs

Layer these tools. Don't pick one and call it done. Layer 1 sits at the URL frontier: canonicalization plus a Bloom filter check, skipping duplicate URLs before any fetch happens. Layer 2 sits right after fetch: a cryptographic hash of normalized content, catching exact duplicates that snuck in through a different URL path. Layer 3 is near-duplicate clustering, SimHash or MinHash run against a signature index, catching whatever slipped past the first two gates.

Cost climbs at each layer, and that's the design, not an accident. URL checks cost nothing, no fetch required. Content hashing costs one fetch and light computation. Near-duplicate clustering is the expensive stage, so it should only ever see content that already survived the cheaper filters first.

What gets hashed matters as much as how it gets hashed. Raw HTML full of navigation menus, ad slots, and cookie banners produces garbage similarity signals; pulling out the main article content before fingerprinting is what makes Layers 2 and 3 actually work. Full-page hashing favors precision, fewer false matches. Main-content extraction favors recall, catching more real duplicates. Which one a pipeline leans toward depends on what the downstream use case actually needs.

One rule holds across all three layers: re-running a stage can't corrupt the layers below it. Dedup state gets stored and reused, not recomputed in a way that shifts the outcome each time it runs. In practice that means raw HTML in object storage, extracted content in a document store built for queries, and derived signatures in a feature store keyed by source, canonical URL, and crawl date. And the earlier dedup happens in that chain, the better: every duplicate that makes it into storage is dead weight, costing money to hold and time to filter out later.

The global versus local deduplication tradeoff in large corpora

Global dedup compares every document in a corpus against every other document, across every crawl snapshot ever collected. It's the maximalist option and it strips out the most redundancy, full stop. Local dedup works snapshot by snapshot, cleaning duplicates within each crawl run but letting the same high-quality page reappear across different snapshots.

Here's the part most people get backwards: more dedup isn't automatically better data. In practice, training on globally deduplicated corpora has been observed to perform worse than training on locally deduplicated ones. Global dedup disproportionately strips out high-quality documents simply because they got republished widely, punishing popularity as if it were noise.

So dedup scope isn't a dial you crank to maximum and walk away from. It depends on the goal. A small, unique corpus for model training wants one answer; a corpus meant to track change over time (watching a competitor's site, tracking prices) wants the opposite, since global dedup would erase the very signal that kind of monitoring exists to capture. There's a second reason to favor local dedup too: comparing every document against every other document at web scale costs a lot more compute than cleaning up one snapshot at a time.

Semantic deduplication and what lexical methods miss

Every technique above is lexical. It looks at words and characters on the page. None of them catch two documents saying the exact same thing in entirely different words: a paraphrase, a translation, an AI-generated rewrite of the same source article. That gap shows up hardest in social media data (noisy, heavily paraphrased), AI-generated web content built from the same underlying facts, and anything crossing language boundaries.

SemDeDup handles this as a follow-up step. In semantic dedup, each document is turned into a semantic vector, documents get clustered by vector similarity, and one representative survives per cluster. Some production pipelines run keyword extraction alongside both embedding vectors and SimHash vectors, then compare against a stored database and toss anything over a similarity threshold.

Semantic comparison is the expensive stage. Pairwise vector comparison at scale isn't cheap, so it belongs at the very end, after cheaper lexical filters have already thinned the pool. For anything feeding a vector store, dedup before embedding isn't optional: duplicate vectors waste vector-store capacity and can skew retrieval toward over-represented content. The full sequence, in order: crawl, extract main content, canonicalize URLs, exact hash, MinHash or SimHash, semantic dedup where warranted, then embed into the vector store.

Tooling and implementation choices for teams building crawl pipelines today

Most of this can be assembled from existing tools rather than built from scratch. DataTrove is built specifically for large-scale web corpus prep and ties this exact set of stages together. Google's deduplicate-text-datasets project uses suffix-array structures to handle exact and near-duplicate detection efficiently. Other open-source toolkits rounds out the open-source options with a production track record behind it.

The decision that actually matters isn't which library to install. It's sequencing: canonicalize before fetching, hash before clustering, cluster before spending money on semantic comparison. Skip a cheap gate early and the expensive gates downstream end up doing that gate's job anyway, at far greater cost.