Converting Raw Web Content to LLM-Ready Input

Before you write a single function, it helps to see the whole thing laid out. Four stages, each with a distinct job:

• Extraction. Fetch the page and pull out the content that actually matters.
• Cleaning. Strip the noise, normalize encoding, eliminate duplicates.
• Structuring. Preserve or impose the document's hierarchy so downstream tools have something to work with.
• Formatting. Produce an output the LLM can actually consume.

The order matters as much as the steps. Bad extraction passes noise into cleaning. Bad cleaning passes noise into chunking. And bad chunking produces embeddings that are just slightly, invisibly wrong. Think of the pipeline like a water filtration system: each stage removes a different kind of contaminant, and if you skip one, everything downstream tastes wrong. By the time garbage reaches the model, nobody's flagging it. The model doesn't know it's reasoning over garbage. It just does it anyway, confidently.

One thing worth saying upfront: this is not a one-size-fits-all pipeline. Static HTML, JavaScript-rendered single-page apps, PDFs, and live data feeds all need different handling at extraction. They converge on the same downstream format requirements eventually, but you have to account for the differences early. At production scale, what starts as a parsing problem quietly becomes a data infrastructure problem. Reliability, idempotency, freshness management.

This piece covers the transformation from fetch to LLM-ready format. Model fine-tuning and prompt engineering live downstream of all of this. Get the pipeline wrong and neither of those will save you.

Extraction: getting the content that matters off the page

Most developers conflate two separate problems here, and it causes real pain later.

The first problem is fetching the page: handling the network request, JavaScript rendering, and whatever anti-bot friction stands between you and the HTML. The second problem is isolating the content: distinguishing the article, product detail, or data table from the nav bar, footer, cookie banner, sidebar, and related-content carousel that surround it.

Different problems. Different tools.

The JavaScript rendering problem

A significant share of modern pages require JavaScript execution before the DOM contains anything meaningful. Send a plain HTTP request to one of these pages and you get back an empty shell. The content loads after the browser runs the scripts, and a static fetcher never sees it.

The fix is a headless browser or a rendering layer that executes JavaScript before handing you the DOM. The tradeoff is real: rendering adds latency and cost, and at scale those numbers compound faster than you'd expect. Worth internalizing before you design a pipeline that needs to process tens of thousands of URLs.

Isolating content from page furniture

Once you have the rendered HTML, you still have to find the signal inside it. A typical HTML document carries somewhere between two and five times more tokens than the semantic content it actually contains. CSS classes, inline styles, data attributes, script tags, boilerplate navigation. Those extra tokens eat context window space without contributing anything retrievable.

Two approaches exist:

• Heuristic approaches like readability algorithms and boilerplate detection work well on editorial content. They break on unusual or non-standard layouts.
• ML-based approaches like Diffbot's computer vision classification automatically detect page type and extract accordingly. More robust, but more opaque about why they made a given call.

Neither is universally better. The right choice depends on how uniform your source content is.

Content types that will quietly break a naive pipeline

• Tables. The structure must be preserved. Flatten a table to prose and you destroy the row-column relationships the data depends on.
• Code blocks. Whitespace is semantically meaningful. Collapsing it breaks the content.
• Mathematical notation. Raw HTML encoding is ambiguous. LLMs trained on LaTeX handle equations more reliably than HTML entity soup.
• PDFs. Not an HTTP fetch problem. They need a separate parsing path entirely.

What good extraction hands off is the semantic content of the page, hierarchy intact, ready to be cleaned.

Cleaning: the unglamorous work that determines data quality

Nobody wants to build the cleaning stage. Nobody wants to maintain it. And everybody eventually regrets skimping on it. Think of it less like a feature and more like plumbing. You only notice it when something goes wrong, and when something goes wrong, it is very wrong.

It is not a single operation. It is a sequence of decisions, each with a right answer that depends on the content type.

Tag stripping (selectively)

The instinct is to strip all tags. Resist it. Strip <script>, <style>, <nav>, <footer>, inline event handlers, tracking pixels. Preserve structural tags that map to meaningful document hierarchy: headings, lists, tables, blockquotes, code blocks. These become your structuring signals downstream.

Encoding normalization

HTML entities, Unicode variants, smart quotes, non-breaking spaces. All of these create token-level noise. An LLM tokenizer treats & and & as different tokens. Normalization needs to happen before tokenization, not after. This is one of those things that seems minor until you're debugging retrieval failures at 2am and eventually trace it back here.

Boilerplate removal

Navigation text, cookie consent strings, footer disclaimers, repeated site-wide content. In a bulk crawl this gets genuinely dangerous. Boilerplate appearing across hundreds of pages inflates term frequency and can end up dominating your embeddings. You end up with a vector space that's primarily a representation of your sources' footer disclaimers. That's not a retrieval system. That's an expensive filing cabinet full of legal copy.

Deduplication

This is a pipeline-level concern, not just a page-level one. Documentation sites, shared drives, and syndicated content mean the same document can appear at multiple URLs with minor variations. Duplicate content in a vector store degrades retrieval in a specific, insidious way: the same chunk wins multiple top-k slots and crowds out the diversity you need for accurate answers.

Minimum requirement: stable document IDs tied to canonical URLs. Updates should replace existing records, not append new ones.

Freshness as a cleaning concern

Timestamps must be captured at ingestion. Without them, you have no mechanism to expire or re-fetch stale content, and a model without live grounding falls back to its training data. Confident, outdated answers. The gap between what the model knows and what's actually true widens continuously unless you actively manage it.

Structuring: why Markdown became the interchange format for LLM pipelines

After cleaning, you have noise-free content. But noise-free is not the same as usable. Plain text solves the noise problem and immediately creates a different one: no headings, no tables, no lists, just a continuous character stream with nothing to tell downstream tools what matters or where one idea ends and another begins.

Markdown became the standard interchange format for LLM pipelines, and it earned that position for practical reasons, not aesthetic ones.

Why Markdown won

• Explicit hierarchy via heading levels, without the token overhead of HTML tags.
• Tables in pipe syntax. Unambiguous row-column relationships.
• Code blocks fenced with backticks. Whitespace and syntax intact.
• Lists, blockquotes, emphasis. All representable with minimal extra tokens.

Converting raw HTML to Markdown cuts token usage significantly compared to passing raw HTML through. That reduction directly lowers API cost and extends your effective context window. Not a rounding error.

What Markdown gives chunking algorithms

Plain text forces chunking algorithms to fall back on arbitrary character-count splits that cut through sentences mid-thought. Markdown gives them natural split points at heading boundaries. Chunks correspond to actual topics rather than arbitrary lengths. LangChain's MarkdownHeaderTextSplitter and LlamaIndex's MarkdownNodeParser both operate on this principle. They're not doing anything particularly clever. They're just taking advantage of structure that Markdown makes explicit.

Structured metadata extraction

Metadata extraction runs in parallel with Markdown conversion: author, publication date, primary entities, canonical URL. These get stored as JSON payload alongside the vector embedding, which is what enables hybrid search. You can filter by recency, source domain, or entity before computing vector similarity. Querying only documents published in the last 30 days limits retrieval to fresh content without re-ranking your entire corpus.

When Markdown is not enough

JSONL (newline-delimited JSON) is the standard format for fine-tuning datasets and bulk embedding pipelines. One document per line, machine-parseable at scale. JSON-LD is useful when schema.org semantics are already present in the source page and you're doing entity extraction.

There's also an emerging format worth knowing about. Some newer APIs return visual captures alongside text, enabling agents that can reason about layout, charts, or UI elements. Multimodal pipelines are still early, but the direction is clear.

Chunking: the decision that most directly shapes retrieval quality

A chunk is the unit of embedding and retrieval. Its boundaries determine what the model actually sees when a question is asked. Get chunking wrong and it doesn't matter how clean your Markdown is. The model still gets the wrong context.

The research here is genuinely contradictory, which I'll be honest is a little frustrating when you're trying to make a concrete engineering decision. What do you call a chunking strategy that works on every dataset? A myth.

What the research actually says

Semantic chunking can improve recall compared to simpler methods, but it requires embedding every sentence at chunking time. That's expensive. Some evaluations have found the computational cost isn't consistently justified, with fixed-size chunks matching or beating semantic chunking across retrieval and answer-generation tasks. Other evaluations, particularly in specialized domains like clinical decision support, have found adaptive chunking aligned to logical topic boundaries dramatically outperforms fixed-size baselines. And other benchmarks have placed recursive token-based splitting ahead of semantic approaches entirely.

No strategy dominates universally. Domain, document type, and query distribution all shift the outcome. The studies that seem most definitive are usually the ones that tested on a narrow corpus. Test on your own data. There's no shortcut here.

The practical default for web content

For most web content, structure-based chunking is the right starting point. Split along Markdown heading boundaries. Each chunk represents a self-contained topic, and questions about specific subsections map cleanly during retrieval. This requires that your structuring stage produced clean Markdown headings, which is exactly why the earlier stages matter.

Parameters that affect chunk quality

• Overlap. Chunk overlap preserves context across boundaries. Without it, sentences split at chunk edges lose their meaning.
• Size. Smaller chunks improve precision. Larger chunks preserve more surrounding context. Neither is inherently right.
• Metadata tagging. Attach source URL, heading path, and document timestamp to every chunk. This enables filtered retrieval and makes the pipeline auditable when something goes wrong.

RAG adoption reached 51% among enterprises in 2024, up from 31% in 2023 according to Menlo Ventures. Chunking strategy is no longer an academic question. It is an engineering decision running in production at significant scale.

Embedding and vector storage: turning structured text into retrievable representations

Embedding converts a chunk into a dense numerical vector that captures semantic meaning. Similar chunks cluster together in vector space regardless of exact wording. It's how a retrieval system finds a chunk about "container orchestration" when someone asks about "managing Docker deployments." Same concept, different words.

What the embedding model receives determines what it can represent. A clean Markdown chunk with a descriptive heading gives the model a coherent semantic unit. A raw HTML fragment with tag noise gives it a mixed signal. The semantic neighborhood in vector space becomes less precise, and you get retrieval that is close but not quite right. In some ways that's worse than retrieval that obviously fails, because at least obvious failures are easy to diagnose.

Vector database considerations

Not all vector databases are equal for production pipelines. Things worth evaluating:

• Metadata filtering capability. Can the store filter by timestamp, source domain, or entity before ranking by vector similarity? This matters for freshness and scoping.
• Upsert support. Stable document IDs from your cleaning stage must map to deterministic vector IDs. Updates replace records, not duplicate them.
• Scale and query latency. Know your production volume before you pick a store.

Hybrid search as the production pattern

Pure vector similarity is powerful but has a known weakness: it ignores keyword precision. A hybrid of vector and BM25 sparse retrieval improves recall on exact terms. Product codes, proper nouns, technical identifiers. These are things vector similarity alone will miss.

Metadata pre-filtering reduces the search space before vector comparison even runs, improving both accuracy and latency. You're not asking the system to search everything. You're asking it to search the relevant subset.

Build three things into your ingestion contract from the start: stable document IDs, source URIs for audit and human review, and timestamps for freshness management and selective re-ingestion. Skip them and you will rebuild from scratch the first time you need to update anything.

Keeping the pipeline current: freshness as an ongoing operational requirement

A pipeline built once and left static is not a pipeline. It is a snapshot that ages.

The hallucination pathway from stale data is straightforward. Without live grounding, a model falls back to its training data and produces confident answers that were accurate at some point, maybe. The gap between what the model knows and the current state of the world grows continuously unless you actively close it.

Three freshness strategies

Scheduled re-crawl. Re-fetch all sources on a fixed cadence. Simple and predictable. Also wasteful for content that rarely changes, because you're paying to re-process things that haven't moved.

Change detection and monitoring. Watch for diffs on specific URLs and trigger re-ingestion only when something actually changes. More efficient, more complex to maintain.

Event-driven ingestion. Trigger pipeline runs from an upstream signal: a webhook, a publish event, an RSS update. Lowest latency, highest engineering overhead.

The right choice depends on how time-sensitive your content is. A pipeline serving financial data has different freshness requirements than one serving product documentation. Build for your actual use case.

What freshness management requires of the earlier stages

Freshness management requires stable document IDs from the cleaning stage. It requires timestamps captured at ingestion. It requires upsert support from your vector store.

Every decision made in the earlier stages either enables or prevents you from keeping the pipeline current. Nothing about the later stages is independent. The pipeline is not a one-time build. Treat it like a production system from day one, or you'll eventually have a very fast, very confident, and very wrong AI application on your hands.