LLM-Powered Data Extraction From Unstructured Web Pages

This article is about how LLM-powered extraction turns messy, unstructured web pages into clean structured data, and why that only works when developers treat it as a pipeline problem rather than a prompting trick. Start with the thing it's replacing. CSS selectors and XPath have run web scraping for years, and they share one fatal trait: they care about where data sits in a page, not what that data means. A selector finds .price-tag because someone told it to look in that exact spot. Rename the class to .pdp-cost-display during a redesign, and the selector finds nothing. No error message, no warning, just silence where a price used to be.

That wouldn't matter much if sites held still. They do not hold still. One tracked team logged nine separate site-structure changes across multiple marketplaces in a single quarter. Redesigns, A/B tests, framework migrations, each one a small earthquake under a scraper's feet. Maintaining selectors at that pace is no longer a task you finish but a job you staff for.

LLMs sidestep the whole problem by reading a page the way a person does. Feed a model a product listing and it can pick out the price, the title, and whether the item is in stock, based on what those things mean in context, not what div they happen to live in. The class name could be anything. The model doesn't care, because it never learned to look for .price-tag in the first place, it learned to recognize a price. Selectors encode geography, models encode meaning, and geography keeps changing while meaning mostly doesn't.

What LLM extraction actually changes, and what it does not

Swapping selectors for a model doesn't make the tradeoffs disappear. It trades one set of problems (fragile syntax, constant maintenance) for a different set (inference cost, latency, and a new kind of error selectors never produced). Anyone pitching LLM extraction as a free upgrade is skipping the parts that matter.

What genuinely gets better: a model can survive a total DOM overhaul and still find a number sitting next to the word "Price". Setup shrinks from hours of DOM inspection and selector testing down to writing a prompt and a JSON schema, replacing the rule-by-rule work selectors required.

What doesn't get better, and deserves a straight answer rather than a shrug: cost and speed. Per-page inference costs money and takes time, and for high-volume, price-sensitive pipelines, plain selectors still win on economics. Hallucination is the other honest problem. A model can return a confident, plausible, completely wrong number, especially for prices, dates, or anything numerical, and there's no equivalent failure mode in selector-based extraction, because a selector either finds the real text or finds nothing. A broken selector fails loud. A hallucinating model fails quiet, in a format that looks exactly like success.

That difference sets the rule the rest of this piece builds on. LLM extraction fits variable layouts, one-off jobs, and rapid prototyping, where the cost of writing and babysitting selectors would exceed the cost of a few inference calls. Anything that runs daily, feeds a database, or touches a customer-facing decision needs more than a model's word for it. The working split practitioners have settled on: reach for LLM extraction where the task genuinely requires understanding meaning, reach for tested extraction code where a pipeline needs to run unattended for months, and keep the two jobs architecturally separate rather than blurring them together.

Stage one: getting a clean, model-ready page before any model sees it

Before any prompt gets written, the page itself needs work. Converting raw HTML to Markdown is the single highest-leverage step in the entire pipeline, because it strips out navigation bars, cookie banners, ads, and boilerplate that burn tokens without adding anything useful, while keeping the headings, tables, lists, and code blocks that actually carry meaning. If this step is skipped, the model spends its attention wading through footer links instead of reading the product description.

A clean result keeps the structure that matters and nothing else. If the output is dominated by template junk, or missing the section the pipeline was built to pull from, it should get rejected before a model ever sees it, because feeding garbage in just produces confident garbage out. A purpose-built cleaning tool claims roughly 67% fewer tokens than raw HTML, a 2026 analysis found, and that reduction maps directly onto lower inference cost per page.

Rendering is a separate question from cleaning: a lot of modern pages build their content with JavaScript after the initial load, so a standard HTTP request returns an empty shell with no real text in it. Browser rendering fixes that, but it's a tool for getting content onto the page, not a way around access restrictions. The sensible order is to try standard retrieval first, add headless rendering only when the page comes back empty, and compare the two outputs before deciding which one the pipeline needs. Rendering costs more time and money than a plain fetch, so it earns its place only on pages that actually require it, not by default.

Anti-bot defenses sit in front of all of this, and they have to be solved before cleaning or rendering even start. Cloudflare Turnstile, HUMAN Security (the company formerly known as PerimeterX), DataDome, and similar vendors now run TLS fingerprinting, JavaScript environment checks, and behavioral analysis to decide whether a request looks human. A page blocked at this layer never reaches the model, full stop, there's no pipeline sophistication that compensates for a page that never loaded. Fetch success is the whole game at this stage: no page, no data, no matter how good the prompt downstream is. Because this problem is deep, specialized, and constantly shifting, the sensible move is to hand proxy management, CAPTCHA handling, and reliable retrieval to dedicated infrastructure built for that fight, so the rest of the pipeline can assume a page will always show up.

Stage two: defining the output schema before writing a prompt

A clean page gets the pipeline halfway there. The schema sets the shape the model's answer should take, before a single word of the prompt gets written. A JSON schema is the actual interface between what a developer wants and what a model produces: it lays out field names, data types, and which fields are required versus optional, giving the model a form to fill out instead of a blank page to improvise on.

Consistency appears immediately: ask a model to "extract product information" and it'll invent different field names on every run. Given a schema that specifies name (string), price (number), sku (string), it returns the same structure every single time. Something like that schema, sketched out before any prompt gets drafted, is worth having on hand as a working example:

{
 "name": "string",
 "price": "number",
 "sku": "string",
 "in_stock": "boolean"
}

That consistency isn't a nicety, it's load-bearing. Databases, APIs, and analytics pipelines downstream all depend on field names and types staying fixed, and enforcing the schema at the extraction layer stops silent breakage downstream.

A few design choices shape how reliable that schema turns out to be. Keep fields atomic: a plain numeric price field asks less of the model than a combined price_with_currency string field, because there's less for the model to infer and less room for it to get creative. Mark optional fields as optional, explicitly, so the model has permission to return null instead of inventing a plausible-sounding value just to avoid leaving a field empty. And expect accuracy to vary by field type: straightforward factual fields extract far more reliably than complex or subjective ones, where the model has to make a judgment call rather than just reading a number off the page.

The Model Context Protocol lets an AI agent discover what a tool can do at runtime, read its typed parameter schema directly, and call it, all through a standard the agent already speaks. That folds the schema into the tool definition itself rather than leaving it as a separate document someone has to wire together by hand, which matters more as extraction work gets built into larger agent-driven workflows rather than run as standalone scripts.

Stage three: choosing where the model fits in the extraction flow

Production teams don't reach for a model first. The working hierarchy treats LLM reasoning as one tool among several, used to complement the others rather than replace them, and where the model sits in that order changes both the cost and the reliability of the whole pipeline. The first thing to check is whether the target site exposes a structured data endpoint of its own, an official API or an XHR JSON response, because using it costs nothing in inference, carries the highest reliability, and removes hallucination risk entirely. Many pages that look unstructured on the surface are quietly handing over structured JSON in a <script type="application/ld+json"> block, and parsing that block is deterministic, cheap, and doesn't involve a model at all. Only after both of those options come up empty does browser automation enter the picture, for pages that need a login, pagination clicks, or an infinite scroll before the real content even appears. The model comes last, reserved for content that's genuinely unstructured, variable across layouts, or small enough in volume that building and maintaining a selector set costs more engineering time than it's worth.

That's also where LLM extraction earns its keep rather than just filling in as a fallback. It's the right tool for pages with so many structural variants that a single prompt covers dozens of them at once, for one-off or low-frequency jobs where writing a selector set isn't worth the investment, and for tasks that need actual reasoning, summarizing a page, classifying it into a category, or normalizing messy, inconsistent terminology, work no selector was ever built to do.

The architectural pattern that holds all of this together separates what the agent reasons about from what the extraction code actually runs. Agents are well suited to replacing one-off scraping scripts, but a job that runs daily, touches customers, or feeds a database needs more than an agent improvising fresh each time, it needs the agent used for discovery and then tested extraction code shipped to handle the actual runs. The stronger pattern built on top of that idea uses agents to write and maintain the extraction scripts themselves, rather than calling a model fresh for every single extraction, which gets the adaptability of AI without paying AI inference costs on every page at scale.

Vision-language models are a specialist tool in the toolkit, not a default. When text-based extraction fails outright, on heavily JavaScript-rendered pages, image-based content, or pages where anti-bot defenses have scrambled the DOM, taking a screenshot and handing it to a vision-language model sidesteps the DOM problem entirely by reading the page the way a person looking at a screen would.

Validating LLM output before it touches downstream systems

A model's JSON output is not trusted input, and treating it that way is the most common mistake in production pipelines. Everything the model returns needs to pass through schema validation before it touches a database, an API response, or anything else downstream. Pydantic, in Python, is the standard tool for this job: it checks that a returned object matches the declared schema, right field names, right types, required fields actually present, and throws a structured error the moment it doesn't, stopping bad data before it spreads any further.

That validation step catches a specific, predictable set of failures. A product object missing its required sku field gets caught. A model inventing a field the schema never asked for, say a discount_percentage nobody defined, gets caught too, before some downstream system reads it as real and acts on it.

A separate problem, harder to catch, is staleness that produces no error. A value can be completely correct the day the schema and prompt were built, then quietly wrong months later because the source page changed its structure in some way the extraction logic never noticed. Nothing fails. No error fires. The data just drifts out of date while every field still technically validates. Catching that requires a different kind of check: regular spot-checks against known ground-truth values, paired with content fingerprinting and a refresh schedule, catch drift before it corrupts a pricing decision or a sales report. Recording a content fingerprint right after extraction, then flagging the page for re-extraction and human review the moment that fingerprint changes, is a concrete way to build this in, rather than assuming an old value is still good just because nothing complained.

Running a model more than once on the same page adds another signal worth using. Low agreement across multiple runs on the same field often points to hallucination, while high agreement paired with low real-world accuracy suggests something systematically off in either the source page or the prompt itself, and a practical move in production is to trigger a second model pass specifically on fields where confidence runs low. Some teams skip building the fetch and cleaning layer from scratch entirely and route requests through a dedicated web-data API that handles browser rendering, proxy rotation, and anti-bot defenses, handing back clean Markdown or structured JSON that's already ready for a model to work on.

Scaling the pipeline: separating discovery, extraction, and monitoring

Discovery, the job of finding which pages exist and which ones have changed, works best as its own process, distinct from extraction, the job of actually pulling structured data out of a page once it's found. Monitoring sits as a third, ongoing layer on top of both, watching for the drift and staleness described above rather than assuming a pipeline that worked last month still works today.

Keeping these three jobs separate is what lets a pipeline scale without each piece dragging the others down. Discovery can run on a fast, cheap schedule, checking for new or changed pages far more often than the expensive extraction step needs to run. Extraction, the stage where selectors, embedded JSON, and models each do their part according to the hierarchy already laid out, runs only when discovery actually flags something worth pulling. Monitoring closes the loop, catching the silent failures, the stale values, and the structural changes that neither discovery nor extraction would notice on their own, and feeding that back so the pipeline corrects itself rather than drifting quietly out of date.