Converting Raw Scraped HTML to LLM-Ready Text

When Cloudflare tested its own documentation page, raw HTML produced far more tokens than cleaned Markdown did, an 82% reduction on a single page. That number produces the whole article in miniature: it shows that most of what an LLM reads on a typical web page isn't content at all, and figuring out what to strip, what to keep, and how to package it is the real engineering problem behind every RAG pipeline and agent that touches the live web.

Why Raw HTML Is the Wrong Input for an LLM

Tags, attributes, inline styles, tracking scripts, SVG assets, and navigation chrome make up most of what a raw HTML page contains by token count, and none of it carries meaning a reasoning model can use. The Cloudflare documentation test isn't an outlier. When Cloudflare blogged about the same feature, it found a similarly large gap between the HTML and Markdown token counts. Two separate measurements, same company, same rough ratio. That's not a coincidence: it's a pattern baked into how modern websites get built.

The cost scales in a way that stings quickly. An agent fetching a few dozen pages a day piles up tens of millions of tokens a month if it's handed raw HTML the whole time, while the Markdown version of the same pages runs a small fraction of that. Think of it like paying for a moving truck by the pound and shipping the packing peanuts along with the furniture. You're not wrong that the furniture arrived. You're just paying to move styrofoam across the country.

And the damage isn't only financial. Noise in the context window raises hallucination rates and drags down retrieval quality, because the model has to spend its attention parsing markup instead of reasoning about evidence. A model buried in <div class="sidebar-widget-trending-topics-container-v3"> isn't reading your content; it's doing archaeology. Every pipeline decision that follows in this article exists to dig the model out of that hole before it starts working.

What a Conversion Pipeline Must Do, and in What Order

Turning raw HTML into something an LLM should actually read takes several steps in sequence: fetching the page, rendering any JavaScript it depends on, pulling out the main content, and converting what's left into Markdown. Skipping or shortcutting any one of those steps means every stage after it inherits the damage.

Rendering comes first, because dynamic pages built with React, Vue, or infinite-scroll patterns don't produce a usable DOM until client-side JavaScript runs, and a pipeline that skips this step returns empty or skeletal content for a large chunk of the modern web. Main-content extraction comes next, because even a fully rendered page is still packed with navigation bars, cookie banners, repeated card templates, and footer boilerplate, and an extractor has to isolate the real text before conversion or the converter will faithfully copy the noise along with it. Conversion is the last step: the extracted content gets serialized into a format that keeps structural signals like headings, lists, tables, and code blocks while dropping the tag overhead, and Markdown has become the standard choice for that job.

That order matters as much as the checklist itself, because each answer depends on the one before it. Three sections follow, one for each stage.

When JavaScript Rendering Is Necessary Before Extraction

Skipping JavaScript rendering is only safe on genuinely static pages. On the large and growing share of the web built as single-page applications, skipping it doesn't just leave gaps, it produces content that's flat-out wrong, because dynamic content, deeply nested structures, and inconsistent HTML are common causes of conversion failures.

React apps, Vue apps, and infinite-scroll feeds don't render anything meaningful on the server. A scraper that grabs the raw response gets the application shell, the equivalent of being handed an empty picture frame and told the painting is coming. Headless browsers solve this by running the full JavaScript runtime and handing back the post-render DOM. Playwright is the standard tool for this, with Puppeteer and Selenium as alternatives, but every one of those requests takes several seconds and burns real compute. That's why rendering has to be a deliberate decision rather than something a pipeline does by default on every page it touches.

A well-built reader API treats this as a toggle rather than a default: try standard extraction first, switch on browser rendering only when the returned content turns up incomplete for that specific target, and set the post-load wait to something tested rather than a long default picked out of habit. Cloudflare's Markdown for Agents feature plays out a version of this at the CDN layer. A proxied site that gets a request with Accept: text/markdown converts the page at the edge before it ever leaves Cloudflare's network, falls back to Workers AI's toMarkdown() for sites not on Cloudflare or without the feature enabled, and only then reaches for a hosted headless browser. This only covers sites sitting on Cloudflare's own network.

The decision rule for everyone else holds up well in practice: if an agent just needs to read a page, a rendering-capable reader API beats running a full browser, and only genuinely interactive tasks, like filling out a form, clicking through navigation, or working inside a session, actually require one. Running a browser to read a static blog post is like hiring a moving company to carry in your mail.

How main-content extraction separates signal from layout noise

Rendering gets you the full DOM. Extraction decides how much of that DOM is worth keeping, and a weak extractor passes navigation bars and repeated templates through as if they were knowledge. A fully rendered page is still loaded with global navigation, cookie consent layers, sidebar widgets, footer links, and repeated card templates, every one of which turns into tokens in the final output with zero retrieval value and a real capacity to hurt precision. Retrieval quality takes the hit directly: if a retrieved answer's best evidence turns out to be a cookie banner or a site-wide nav block, that's worse than useless, it actively misleads.

The better approach, used in production reader APIs, scores page regions by content density and semantic coherence instead of relying on CSS selectors that snap the moment a site gets redesigned. CSS-selector extraction is brittle in a specific, predictable way: it works great until a designer changes a class name, and then it's silently broken with no error message to show for it.

The clearest demonstration of what bad extraction costs is visible far from a chatbot's answer box, in a 2025 NVIDIA research project called Nemotron-CC-Math. The team needed to pull math and code content out of huge amounts of web HTML for training data, so they picked lynx, a text-based browser that executes actual HTML layout rules, because the DOM-based parsers used in earlier projects kept stripping or corrupting mathematical notation and code formatting. Lynx produces output that mirrors how a human actually perceives the page, recovering MathJax, KaTeX, and MathML that DOM-based parsers omitted or mangled, with a subsequent LLM cleaning stage handling final normalization. That's a pretraining-scale consequence: if extraction corrupts an equation, that corruption doesn't just cost one wrong answer, it gets baked into whatever the model learns from that data for good.

You shouldn't judge extraction quality by whether the tool returned something labeled Markdown. It's whether each extracted chunk still makes sense to an LLM once the surrounding page has been stripped away, since a technically valid Markdown response can still be a poor piece of retrieval input.

Markdown as the Right Serialisation Format for LLM Input

Markdown keeps the structural signals that chunking and retrieval depend on, and it cuts the tag overhead that inflates token counts, and no other format in wide use manages both at once. Converting scraped HTML into clean Markdown before it reaches an LLM cuts token usage substantially. Internal benchmarks across article, product, and documentation pages showed roughly 60 to 80 percent token reduction, which lowers cost and improves retrieval quality in RAG pipelines at the same time.

The syntax itself is deliberately thin. A heading is #, a table cell boundary is |, a list item is -, *, or +. Structure gets encoded without wrapping every single element in opening and closing tags that the model then has to parse and throw away. The bigger argument isn't the token savings. The bigger one is what that structure makes possible downstream.

Headings written in Markdown let a pipeline do header-aware chunking: splitting text at ## or ### boundaries means each chunk carries its own topic label instead of being an arbitrary slice of text that might start mid-sentence and mid-idea.

Tables show the same principle even more sharply. HTML tables confuse text embedding models, and flattening a table into plain text wipes out the relationship between rows and columns. Markdown tables keep a rigid, predictable structure that holds that relational data together. A price list that survives as a Markdown table can still answer "how much does the enterprise plan cost." The same price list flattened into a paragraph often can't.

For teams converting pages locally and needing full control over the cleaning step, Python's markdownify library and JavaScript's turndown are the standard tools, and the conversion itself adds almost no latency, typically a few milliseconds per page, against significant token savings downstream.

Where the Markdown-first consensus is genuinely contested

Not every researcher agrees that converting HTML to Markdown is a net win, and the disagreement matters before the next section builds on top of it. The HtmlRAG paper, published by Tan and colleagues in 2024 and still actively cited through 2026, argues that headings and table structures encoded directly in HTML carry information that gets lost in plain-text conversion, and that modern LLMs have enough capacity to read HTML directly without that loss. Under that argument, the conversion step isn't a cleanup, it's a leak.

It's a fair point on paper. HTML does encode relationships, like which cell belongs to which row, that Markdown approximates rather than preserves exactly. But empirical evidence from reader benchmarks tips the other way for most production RAG workloads: the token savings, the chunking benefits, and the retrieval accuracy gains from clean Markdown outweigh what's lost in translation for the vast majority of real pipelines.

Multiple teams have settled on a practical split rather than picking one side. HTML gets stored as the archival source of truth, kept around for fidelity and for reprocessing later if extraction logic gets better. Markdown becomes the serving layer, the thing actually placed in front of the model. HTML can always be re-rendered and re-extracted down the line. Markdown is what the LLM reads today. So a pipeline can bank the efficiency gains now without permanently throwing away the raw material HtmlRAG is worried about losing.

How RAG Chunking Strategy Depends on Conversion Quality

Chunking strategy can only work with the structure that survived conversion. Arbitrary text splits applied to poorly converted content wreck the retrieval accuracy a RAG pipeline exists to deliver, no matter how clever the chunking logic itself is.

Header-aware chunking, splitting at ## or ### boundaries, only works if the extractor correctly identified headings as headings in the first place and the converter wrote them out as # markers instead of bold text or plain strings. Get that upstream step wrong, and the chunking strategy has nothing to anchor to.

Arbitrary character or token-window splits applied to Markdown that already lost its heading structure produce chunks that start mid-argument, mid-table, or mid-code-block. Those chunks still match on keyword overlap during retrieval, so they get pulled up as relevant, but they fail to answer the actual question because the surrounding context is gone. That's the retrieval equivalent of handing someone the middle three pages of a mystery novel and asking them who did it.

Chunks also need to respect document boundaries. A chunk that spans the tail end of one article and the start of another, which happens when document-level metadata isn't preserved, ends up retrieving for both topics and answering neither one well. Source identity has to be tracked at the chunk level too: the requested URL, the extraction timestamp, the API result code, and a content fingerprint. Without that, a retrieved chunk can't be traced back to where it came from, and there's no way to audit it later if the source page changes.

Good pipelines build in a regression test habit for exactly this reason: keep a small set of source URLs on hand and rerun extraction against them whenever a setting or SDK version changes, because a change that improves extraction on one page type can quietly break it on another. Chunking strategy gets a lot of attention in RAG discussions. Conversion quality is the thing actually deciding whether that strategy has anything solid to work with.

Making incomplete extraction visible rather than silent

An extraction API can return a clean HTTP 200 while handing back a result that's completely useless for retrieval, and a pipeline that doesn't catch that will quietly poison its own index over time. Status code 200 means "the request succeeded." It says nothing about whether what came back is actually worth indexing.

You should never let a blank or partial response slip into the index unexamined. A pipeline needs to reject or flag for review anything that comes back as empty Markdown after whitespace gets stripped, anything dominated by navigation text instead of real content, or anything whose length is implausibly short for what the page should contain. None of that requires exotic tooling, just a check that runs before the content gets anywhere near the vector store.

Skipping that check doesn't blow up the pipeline on day one. It fails slowly, one bad page at a time, until the index is quietly full of cookie banners and empty shells that answer questions nobody asked. Catching that failure at the door is cheaper than untangling it later, and it's the one step standing between a pipeline that degrades silently and one that actually knows when something went wrong.