Sitemap Parsing and URL Discovery Strategies

Sitemaps are the fastest way to pull a URL list off any website. One request, and you've got a seed list. But if you stop there, you're going to miss a chunk of the site, and depending on how the thing's built, maybe a big chunk. Getting full coverage means stacking robots.txt parsing, recursive link extraction, and some amount of JavaScript rendering on top of the sitemap, in that order, for reasons that'll make sense in a minute. Two flavors of sitemap exist, and it's worth knowing both. XML sitemaps are built for bots: structured, machine-parseable, with fields like lastmod, change frequency, and priority sitting right there waiting to be read. HTML sitemaps are built for people, a clickable directory page that happens to help crawlers too, almost by accident. Hit the XML version with a parser and you get the full URL list plus timestamps, and on bigger sites, typed sub-feeds too: image sitemaps, video sitemaps, news sitemaps. Each one surfaces URLs a plain link crawler would just walk past. Quick word on lastmod, because I've watched engineers lean on it way more than they should. It's a crawl-efficiency hint, not a promise. Googlebot looks at it, weighs it against other signals, and recrawls on its own schedule regardless of what the timestamp says. Trust it. Don't marry it. Robots.txt is the conventional front door. It usually names the sitemap location, so one request to /robots.txt unlocks the whole inventory before you've touched a single page of real content. Cheap as data collection gets, honestly. Sitemaps also have hard ceilings. Every sitemap file, XML or plain text, tops out at 50,000 URLs and 50 MB uncompressed. Hit that and the site needs a Sitemap Index instead, a directory file pointing to a stack of child sitemaps rather than cramming everything into one document. Once that ceiling clicks in your head, the whole architecture of large-site sitemaps stops being weird and starts making sense. ## The real gaps sitemaps leave (and why about 15% of sites skip the sitemap entirely) Recent data puts it at roughly 15% of websites with no XML sitemap at all. Not "incomplete," not "outdated." Missing. No cold start, no seed list, nothing to parse. For those sites, step one of your plan evaporates before you've written a line of code. Even when a sitemap exists, it lies sometimes. Rarely on purpose, usually just neglect. lastmod is self-reported, and site owners forget to update it, mis-set it, or leave it frozen at the original publish date even after the page gets rewritten top to bottom. A crawler that takes lastmod at face value skips real changes and files it under "efficiency." Then there's what gets left out deliberately. Staging URLs, parameter variants, paginated archives, faceted navigation filters, anything sitting behind a login or paywall. Some of that's reasonable. Some of it's a site owner hiding content a crawler still has a legitimate reason to find, and telling the difference is more art than science. The structural problem is the bigger one, though. Ahrefs has put the number of orphaned or poorly linked pages on the average site at around 23%, meaning no sitemap lists them and nothing else links to them either. That's a gap no single method closes, sitemap or otherwise. It's basically the whole reason this piece exists past paragraph one. Some sites generate their sitemap dynamically behind a JavaScript layer, so a plain HTTP fetch returns something close to an empty shell, a taunting little XML skeleton with none of the actual URLs inside. Sitemaps are a fine starting point. They're just not the finish line, and everything below is about what you stack on top, and in what order. ## Robots.txt as a discovery document, not a permission slip Most engineers read robots.txt once, note the Disallow paths, and move on with their lives. That's treating a map like a warning label and throwing it away after the warning registers. Robots.txt does tell you what's off-limits. It also tells you a lot about how the site's built, if you read past the Disallow lines instead of skimming them. Start with the obvious payoff: explicit Sitemap: directives, sometimes several, pointing to different sitemap index files for different jobs. Separate declarations for Googlebot, generic crawlers, and AI-specific bots aren't unusual at all. That's the site owner telling you, in writing, who they expect to show up and what they want each visitor looking at. The Disallow lines reward a second look too. A rule like Disallow: /search?q= isn't just a "keep out" sign, it's confirmation a search endpoint exists with that exact query structure. Disallowed paths are basically a negative map of the site's architecture. They point at patterns worth chasing down some other way, even if this particular door stays locked. Parsing order isn't complicated: fetch robots.txt first, pull every Sitemap: line out of it, then walk from sitemap index down to child sitemaps. That sequence banks the maximum structural knowledge before a single HTML page gets touched. Watch the edge cases, they trip people up more than they should. Some sites declare sitemaps at a URL nowhere near the standard /sitemap.xml path, for no reason anyone can explain. Sites with multiple subdomains sometimes run a separate robots.txt per subdomain, each with its own sitemap declarations, like five different departments that never sync their calendars. None of this is exotic. It's just inconsistent, and inconsistency is the normal condition of the open web. Robots.txt stays quiet on anything the owner never bothered to declare, though. Nothing about link topology, nothing about orphaned pages, nothing about how content actually connects to other content. For that, you have to start crawling. ## Recursive link extraction as the fallback that does the real work The mechanism's old and it still works fine: start from a seed URL, fetch the HTML, pull every `` out of it, filter down to same-domain links, throw the unseen ones in a queue, repeat. Breadth-first or depth-first depends on whether you care more about coverage or depth in one section, but the core loop never changes. This is how you catch what sitemaps miss. Editorial links buried in article bodies, footer nav, breadcrumbs, related-content widgets, none of which any sitemap author thought to list. A page that exists purely because another page happens to link to it, with zero deliberate submission anywhere, only turns up through this method. None of it's free. Politeness matters: respect crawl-delay, throttle your request rate, don't hammer a server that's politely asking you to slow down. Deduplication matters just as much, since the same page shows up under five different URL strings once you factor in trailing slashes, query parameters, and fragment identifiers. Normalize before comparing, or you'll re-crawl the same page a dozen times and call it progress when it's really just spinning wheels. Crawl traps are the other hazard. Infinite pagination. Calendar widgets generating a URL for every day since the dawn of time. Session IDs spawning a technically infinite URL space. Scope control isn't optional here, keep the crawl locked to the target domain or subdomain, because without a hard boundary it'll wander off and start indexing somebody else's website entirely, which is a fun bug report to write up. Run link extraction alongside sitemap parsing, not as a replacement for it. Sitemaps hand you the canonical list fast and cheap. Link extraction hands you the long tail nobody ever submitted anywhere. Union the two, dedupe, and coverage improves in a way neither delivers alone. The real signal, the useful one, sits in the mismatch: URLs that show up in the crawl but never appear in the sitemap are either an oversight or something deliberately left off, and both cases deserve a flag. ## JavaScript rendering and SPA route discovery, where static crawling goes blind Here's where a static crawler quietly breaks and nobody notices for weeks. Fetch a React, Next.js, or Angular page with a plain HTTP request, and you often get a nearly empty shell. A handful of script tags, almost no `` elements to extract. All the real navigation gets injected by JavaScript after the page loads, so the static crawler reads a blank page and reports back, cheerfully, that there's nothing there. Headless rendering fixes that. It catches client-side route transitions that never generate a real HTTP request, lazy-loaded content sitting behind infinite scroll or tab switches, and navigation assembled from data the page fetches at runtime. None of that shows up any other way, full stop. The cost isn't small, though, and this is where teams get burned. Headless browsers pull the full page payload down: HTML, JavaScript bundles, images, tracking scripts, often landing around 3 to 5 MB per request. Scale that to a million pages a month through an unoptimized setup, and you're at 3 to 5 terabytes of proxy traffic. Residential proxy pricing runs roughly $2.00 to $8.50 per gigabyte these days, which puts the bandwidth bill somewhere between $15,000 and $25,000 a month, before you've spent a dollar on anything else in the pipeline. Memory's its own headache. Fifty concurrent browser sessions need something like 25 GB of RAM just for the browser processes, and Chromium has a well-earned reputation for leaking memory and leaving orphan processes lying around if nobody's cleaning up. Save this tool for when static extraction comes back clearly thin, and nowhere else. The discipline that actually works: run static extraction first, every time, and escalate to headless rendering only for the URLs or domains that come back sparse. While a page renders anyway, intercept the XHR and fetch calls happening in the background; that traffic often reveals API endpoints and URL patterns without rendering every derived page by hand. For single-page apps, checking the JavaScript bundle for router config (React Router, Vue Router, whatever's running) can hand you the entire route tree without rendering a single page. ## Sitemap Index architecture for sites with hundreds of thousands of URLs Big sites slam into that 50,000 URL ceiling constantly, which is the whole reason the Sitemap Index format exists. It sits on top, points down to a set of child sitemaps, and each child stays under the cap. Large e-commerce portals or news publishers routinely run hundreds of child sitemaps just to hold their inventory. The naming convention on those child files is a signal in its own right, and a free one at that. sitemap_products_1.xml, sitemap_blog.xml, sitemap_news.xml, that tells you the content taxonomy before you've fetched a single URL out of any of them. A crawler that reads the names first can prioritize, pulling product pages ahead of blog posts, say, without parsing every child file's full contents up front. Sitemap hygiene at this scale varies wildly, and knowing the failure modes saves a lot of wasted requests. A listed URL that 301-redirects elsewhere costs you an extra hop for nothing. A soft-404 (a page returning HTTP 200 that's really just an error page in a costume) wastes everything downstream that tries to treat it as real content. And sometimes a sitemap lists a URL that robots.txt disallows in the same breath, which tells you plainly that two different people built these files and never once talked to each other. Traversing a nested index is just recursive descent: fetch the index, grab the child sitemap URLs, fetch each child, grab the page URLs. Track depth as you go, because a malformed index referencing itself will loop forever if you let it, and you will, eventually, forget to check for that. Once traversal finishes, you're holding a typed, segmented URL inventory with lastmod data attached, about as strong a seed as any pipeline gets. ## Putting the layers together: a practical discovery sequence Order matters more than any single technique here. Start with robots.txt: pull the Sitemap directives, note the Disallow patterns as architectural hints. Then parse the sitemap or sitemap index, walking the full tree for a typed URL inventory with metadata attached, at close to zero HTML-fetching cost. Steps one and two, both cheap. Step three is recursive static link extraction, seeded from whatever the sitemap already gave you. This is where undeclared and orphaned pages surface, and where you flag anything found in the crawl but missing from the sitemap for later review. Step four is selective JavaScript rendering, triggered only for domains or sections coming back thin from static extraction, run with the cost discipline from the section above firmly in mind. Step five is dedup: merge sitemap URLs, crawl-discovered URLs, and rendered URLs into one canonical set, normalized so the same page doesn't show up three times under three different query strings. The decision rule for step four is rough but it works: if the static crawl is pulling in noticeably fewer links per page than the sitemap implies should exist, that's usually the tell that the site renders client-side. Time to escalate. No need to guess site by site once that pattern shows up, it's basically a fingerprint. The gap between sitemap and crawl coverage is worth tracking as a quality signal on its own, not just a one-off diagnostic. Sitemap listing far more URLs than the crawler ever finds points to high orphan content. Crawl finding far more than the sitemap lists means the sitemap's stale, or was never complete in the first place. Both directions are actionable. Both deserve logging over time instead of getting treated as a surprise each time they show up. Pagination gets its own callout: don't hardcode a page count, ever. Detect the pattern from the first page or two, infer the URL template, enumerate programmatically from there. Sites change their archive depth all the time, and a hardcoded ceiling breaks the second they do. What comes out the far end of this sequence is a URL inventory that's about as complete as the site's ever going to let you get. Typed by content segment where the data supports it, with quality flags (redirect chains, soft-404s, orphan status) already attached before extraction even begins. ## What to do with discovered URLs: feeding them into an AI pipeline Discovery is stage one of four: discovery, extraction, validation, export. Everything above is the input to the next three stages. It's not the finished product, and treating it like one is how teams end up debugging a bad RAG index three layers away from the actual bug. Raw HTML is an expensive format to hand an LLM, and the token math makes the case without any help from me. Scripts, stylesheets, nav markup, boilerplate, all of it inflates the token count without adding a shred of reasoning value. A typical documentation page runs something like 8,000-plus tokens as raw HTML and drops to roughly 1,200 tokens once converted to clean Markdown. That's a 60 to 80% reduction, and it shows up directly on the embedding and inference bill. Markdown also keeps the structural cues (headings, lists, code blocks) that chunking and retrieval need downstream. The pipeline past discovery looks like this: crawl the discovered URLs, scrape and convert each to clean Markdown, chunk by heading structure, embed with something like text-embedding-3-small, store the vectors in Pinecone, Weaviate, or Chroma, then retrieve by similarity at query time. Loading a whole documentation site, a Docusaurus or GitBook instance, say, into a vector database this way is probably the single most common real-world use of this entire pattern. On tooling: Firecrawl converts websites into structured Markdown that preserves heading hierarchy, and it shows up constantly in RAG pipeline tutorials for exactly that reason. Crawl4AI is an open-source, async Python library that renders JavaScript through Playwright and hands back cleaned HTML, Markdown, structured extraction output, links, and metadata together, with chunk-splitting built in for vector database ingestion. Managed web data APIs exist too, built specifically so a team doesn't have to run the rendering and cleaning infrastructure themselves. Pick based on how much rendering you actually need and how much infrastructure you want to own and maintain at 2am. Whatever tool ends up in the pipeline, the sequence upstream of it, robots.txt, sitemap, recursive crawl, selective rendering, decides whether the URLs going in were ever complete to begin with. Garbage discovery in, garbage RAG index out. That part doesn't change no matter which vector database logo ends up on the slide deck.