Feeding Web Pages Directly Into an LLM Without a Preprocessing Step

LLM-ready structured data extraction from web pages requires preprocessing that fetches the actual content, removes boilerplate, preserves useful structure, and validates the output. Feeding raw pages directly into an LLM degrades quality because the input may be noisy, poorly organized, or missing the content entirely. Markdown suits reading and retrieval, while typed JSON suits record extraction.

Why raw HTML is the wrong input unit for a language model

A web page is built for a browser. Its HTML, JavaScript, and CSS tell that browser how to assemble and display content, but a language model receives those instructions as more text to process. The model uses up its token budget on instructions for rendering the page as well as on what the page says. It is like ordering lunch and being served the restaurant’s wiring diagram.

That mismatch makes raw HTML an expensive unit of input. Tags, scripts, and styles consume tokens at a much higher rate per unit of useful information than prose or Markdown. Some markup describes meaningful document structure, while much of it supports the page’s rendering. The distinction is between content and its packaging. Preprocessing separates them so that a model can spend its context on the information the task actually needs.

How the lost-in-the-middle effect turns context bloat into an accuracy problem

Wasted tokens are only the first cost. Transformer attention is not uniform across a context, so information does not receive equal treatment merely because it fits within the available space. The lost-in-the-middle effect means that useful content buried between other material can receive less attention than content near the edges. A page can therefore fit inside a model’s context and still be a poor input.

Raw HTML makes this weakness harder to avoid. Headers and navigation come before the main content, while footers and other page furniture follow it. Those blocks push the useful text toward the middle, where attention is weakest, but add little that helps with the task. The model gets more to read and a worse arrangement of what it needs.

This is an attention problem. Treating it as a temporary vendor bug leaves the data pipeline dependent on the model overcoming an unfavorable input layout. Retrieval-augmented generation, chunking, and contextual compression can help manage context, but all three depend on clean input. Applied directly to raw HTML, they inherit its noise.

What JavaScript-rendered pages send the model when no browser runs first

On a page that depends on JavaScript to display its content, a raw HTTP fetch can return empty containers and the scripts meant to populate them. Without a render step, the model receives the unrendered shell of the page a visitor sees. Cleaning out noise cannot bring back text that was never fetched.

This matters even more for pages built with frameworks, though some of those pages work without a browser. Some return meaningful content from the server, and others populate it in the client. The fetched response needs to be checked for the information the task needs. A request can succeed even when that information is missing.

Apify’s data collection guide shows how this can go wrong with a City Parks Foundation calendar. The raw HTTP client did not run JavaScript, and the guide recommended switching crawler types if the site required JavaScript to display content. An event crawler in that position may capture the calendar shell but miss the listings and cancellation notices. Nothing has to crash for the extraction to fail.

That silence makes the problem dangerous in production. The pipeline fetches a page, passes the response onward, and gets an answer from the model, leaving an apparently successful chain of operations based on empty or near-empty content. The answer’s existence is not evidence that the source was available. A model cannot extract an event listing from a container that merely reserves space for one.

How input format determines extraction accuracy, independent of model capability

Input format affects how well extraction works, even before models are compared. The choice is not simply between more tokens and fewer tokens, because structure also determines how clearly the input presents records and their relationships. Format is an engineering decision.

Wordbricks’ NEXT-EVAL benchmark, published in May 2025, found that Flat JSON enabled LLMs to achieve an F1 score of 0.9567 for web data record extraction, with minimal hallucination. It outperformed both Slimmed HTML and Hierarchical JSON. The benchmark tested ways to preprocess the input that kept DOM semantics while changing the form of the input. The numbers show how well JSON works for structured record extraction, but leave open how it fares on other tasks.

Raw HTML can nevertheless score well on read accuracy. That is a fair objection, but reading a page and reliably extracting records from it are different requirements. A model may identify information in the source without producing the consistent fields a downstream system needs. Reading well is not the whole job.

The distinction becomes sharper when the output supports action. A prose answer can be useful to a reader while still being unsuitable as a typed record that another system must consume. For that task, correct boundaries and field structure belong in the evaluation alongside whether the model understood the page. The benchmark makes preprocessing part of the accuracy discussion.

Why boilerplate duplication at scale collapses retrieval quality in RAG systems

An event recommendation can be wrong even when the listing was captured correctly. Apify’s guide shows a second failure on the same calendar, where a Mozilla Readability extraction setting kept event listings but left out a weather-cancellation notice. Changing the HTML transformer setting brought the notice back. If the incomplete record enters an index, the model can confidently recommend a canceled event. The listing survived, but its meaning did not.

This exposes a limit of “main content” extraction. Cleaning must preserve the details that change what a record means. A cancellation notice is not disposable decoration just because an extractor leaves it outside the selected content. Less text is only better when the right text remains.

Across many pages, repeated boilerplate introduces another retrieval failure. The same header, footer, and navigation block can make up so much of what goes into the embedding layer that different pages look too much alike. The embedding space collapses toward near-identical vectors, so retrieval loses the distinctions needed to return content relevant to a query. Every page starts wearing the same disguise.

The fix belongs before the index. Clean, deduplicated content preserves the discriminative signal that retrieval depends on. A pre-LM pipeline can screen bad responses, extract the main content, strip boilerplate, detect language, remove duplicates, and produce schema-guided typed JSON before records reach the index. This guards against both repeated noise and degraded records. Once either is embedded, a fluent final answer can conceal how poor the retrieved evidence was.

How do you get web content into a format your LLM can actually use?

LLM-ready data is more than HTML with fewer tags. It is clean, structured, and consistent, with each property serving a different purpose. Clean content excludes ads, navigation, scripts, and boilerplate. Structured content preserves headings, lists, and tables so that the model can follow the document’s organization. Consistent content gives each source the same output shape, so fewer chunking rules are needed for each site and schema drift is prevented.

The downstream job determines the right format. Markdown is the default for content-heavy prose, reading, and retrieval because it preserves useful document structure without the bulk of browser markup. Headings keep sections distinct, while lists and tables retain relationships that a flat wall of text would obscure. The aim is to keep the structure that helps a model understand the content and discard the structure intended only for rendering.

JSON and JSONL fit tasks built around typed, schema-enforced fields, including product records and fine-tuning pairs. NEXT-EVAL’s result supports Flat JSON for structured web record extraction in particular. That evidence does not displace Markdown for prose retrieval, because those formats serve different jobs.

Screenshots and raw HTML still have a place when the task needs layout, visual detail, or full DOM control. The format should fit what the next stage needs to understand or work with.

Building the preprocessing pipeline that keeps clean data flowing at production scale

A production pipeline needs a fetch-clean-extract-validate sequence, with fetching costs rising only when the content requires it. A direct HTTP fetch with a lightweight parser is the lowest-cost starting point for static pages and many single-page applications. The important condition is that the response contains the needed content. A framework name alone does not settle that.

Internal JSON API detection provides another route. Many modern web apps populate their pages from an internal JSON endpoint, and calling that endpoint directly is faster and cheaper than rendering a headless browser. A light parser with JavaScript disabled can also handle pages that send useful content server-side along with client-side noise. When the content depends on JavaScript, it needs a browser render. The fetch ladder avoids paying for that step when a simpler route already supplies the data.

After fetching, the processing stages need a fixed order. Bad-response detection comes first, followed by main-content extraction, boilerplate stripping, language detection, deduplication, and schema-guided extraction into typed JSON. Each stage gets the input ready for the next, so a final prompt has less to fix at once. The content needs validation at each stage, before it reaches the model or index. A successful fetch is only the first checkpoint.

For record extraction, the operational rule is to validate every output against a typed JSON schema before embedding it. A malformed record that slips through can corrupt the index and create retrieval failures that are difficult to trace back to their source. Schema validation is not a substitute for content completeness, however. A well-shaped event record can still omit the cancellation notice that changes whether anyone should attend.

A managed API can occupy this preprocessing layer for teams that do not want to build the cleaning path themselves. The relevant selection criteria are whether it can obtain the actual page content, preserve useful structure, return consistent Markdown or schema-guided JSON, and support validation before indexing. Outsourcing that work changes who runs the pipeline. It does not remove the need for it.

The production test is whether the output is ready for its next use. That means readable structure for retrieval and complete, typed records for extraction. Preprocessing establishes those conditions before fluent generation can hide their absence. Without it, the model is being asked to compensate for failures that happened before it received the input.