HTML to Structured Data Extraction Fundamentals

When a browser loads a page, it parses the HTML and builds the Document Object Model: a tree of nodes held in memory, representing every element, attribute, and text fragment. Every extraction tool that operates on HTML is navigating this tree. The DOM is the map, and everything else follows from that.

The vocabulary is worth getting precise about. Nodes come in several types: element nodes (the tags themselves), text nodes (the content between tags), attribute nodes (the key-value pairs that modify elements). These stand in structural relationships. An element containing another is its parent; the contained element is its child; elements sharing a parent are siblings. Selectors and queries work by traversing these relationships to reach a target node.

For static pages, this is straightforward. The DOM is fully present in the HTML response, and any parser can read it directly. Dynamic pages are a different matter entirely, and this is where I've watched more extraction pipelines quietly fail than anywhere else. JavaScript frameworks build or modify the DOM after the initial load, meaning the raw HTML response is often incomplete or nearly empty. A parser working only on that response misses the content. Per Browserbase (2025), over 70% of modern websites use JavaScript frameworks. A tool that cannot render JavaScript, typically via a headless browser, is effectively blind to the majority of the web.

That static-versus-dynamic divide is the first genuine decision point in any extraction project, and it has to be settled before any other consideration enters the picture.

CSS selectors: pattern-matching the DOM by tag, class, and relationship

CSS selectors were designed to tell browsers how to style elements. The targeting logic they encode, however, transfers directly to extraction. Identifying an element for styling and identifying it for data retrieval are, at the structural level, the same operation.

The core patterns cover most practical cases. You can target by tag (h1, td, a), by class (.product-title), by ID (#price), by structural relationship (div span for any descendant, ul > li for direct children, h2 + p for an adjacent sibling), or by attribute value (a[href^="https"] for links beginning with a specific scheme). These patterns compose, respecting CSS specificity rules, and they execute quickly against the DOM.

Two limitations are worth naming explicitly rather than discovering mid-project. CSS selectors traverse only downward from the document root; they cannot walk up to a parent element or select a preceding sibling. They also cannot select text nodes directly or apply functions to content values. What a selector returns is an element, not an arbitrary substring of what's inside it.

In practice, neither limitation causes problems most of the time. The vast majority of routine extraction tasks involve content reliably identified by class, ID, or structural position: product listings, article bodies, navigation links. CSS selectors handle these cases cleanly, with readable syntax and broad support across every major parsing library. Their ceiling is where XPath picks up.

XPath: querying document structure with a full expression language

XPath is not a selector syntax. It is a query language. Where CSS selectors express patterns, XPath expresses computations over a document tree. It can traverse in any direction, evaluate conditions, and apply functions to content values.

The XPath axes CSS selectors lack are where XPath earns its complexity budget. The parent:: axis selects the element containing a target. The preceding-sibling:: axis selects elements that come before it under the same parent. The ancestor:: axis walks up the tree arbitrarily far. These matter when the element you want is not directly addressable but is reliably positioned relative to something that is, which is a surprisingly common situation in real-world markup.

XPath can also target text nodes directly using /text(), retrieving content without the surrounding element. Predicates add conditional logic inline: //div[@class='price' and @data-currency='USD'] filters by multiple attributes simultaneously. Functions like contains(), starts-with(), and normalize-space() handle the inconsistencies that actual markup produces with aggravating regularity.

The syntax is verbose. Complex queries against large documents carry measurable computational cost. The heuristic that holds up in practice: if the target is identified by class or ID, use a CSS selector. If the task requires positional logic, upward traversal, or text-node extraction, XPath is the right tool, and the friction of learning its syntax pays off faster than you'd expect.

Regex and string parsing: why the lowest-level approach has a narrow viable range

Before parsers, there was just the string. Regular expressions and basic string operations can pull content from HTML without any library, any DOM model, or any structural understanding whatsoever. The appeal is real.

So is the brittleness, and I mean that in a specific technical sense. HTML is not a regular language. Nested tags, optional attributes, whitespace variation, entity encoding, and comment blocks all produce character sequences that a pattern written against one specific rendering of a page cannot anticipate. Any change to the markup, an added attribute, a reordered attribute, a shifted whitespace character, silently breaks the match. Regex operates on the character stream with no awareness of what the document structure means.

Practitioner consensus on this point has been consistent for decades: regex is appropriate only for the most constrained extraction tasks, typically machine-generated output with no structural variation. For anything else, a parser is necessary. The existence of CSS selectors and XPath is, in a sense, the industry's collective response to having tried character-level matching and found it wanting.

Regex is not, then, a general extraction technique. It is occasionally useful for post-processing clean text after a parser has already isolated the relevant content, and it is a useful illustration of exactly why structured approaches became necessary.

What breaks rule-based extraction and why

CSS selectors, XPath, and regex share a failure mode that is architectural. Each encodes the structure of a specific page at a specific moment in time. Change the structure, and the rule breaks. This is not a bug in any particular implementation; it is the nature of the approach.

Structural drift is the most common manifestation. A site redesign, an A/B test, or a CMS update changes class names, nesting depth, or element ordering. Every selector pointing at the old structure fails silently, a class of silent failure the pipeline has no mechanism to surface, returning nothing, or worse, returning the wrong thing. There is no warning. The pipeline keeps running.

Cross-site generalization does not exist. A selector tuned to one e-commerce site's product page has zero transferability to a structurally different site's product page. Each new source requires new rules. At any meaningful scale, this becomes an enormous and relentless maintenance burden.

Dynamic rendering compounds everything. JavaScript-rendered pages do not expose their data in the initial HTML response. A static parser operating on that raw response sees incomplete or empty content, and no refinement of the selector logic changes that.

Anti-bot measures add another layer still. Modern sites fingerprint HTTP clients, rotate content based on request headers, and require browser-like interaction before surfacing data. Static requests receive different content than a real browser would, or nothing at all.

Per Browserbase (2025), 68% of successful enterprise scraping projects combine multiple tools, using lightweight parsers for static content and browser automation frameworks for dynamic sites. That figure reflects accumulated practical experience: no single rule-based method covers the full range of real-world pages. The fragility of hand-written rules is what motivated the search for methods that generalize.

Wrapper induction and classical ML: learning extraction rules from examples

Wrapper induction approached the fragility problem from a machine learning angle. Rather than writing selectors by hand, a system learns extraction procedures from a small set of labeled examples. Annotate a handful of pages from a source, let the algorithm generalize a pattern, apply it to new pages from the same source. The goal was to replace the human authoring rules with a process that could infer them.

Extensions pushed the approach further. Boosted wrapper induction combined simple patterns for better coverage. Larger-scale variants were designed to tolerate noisy annotations and gradual template drift. For regular site structures with clean labeling and stable templates, these systems worked reasonably well, essentially treating sites that behave like databases as databases.

The fundamental limitation persisted regardless of variant. Learned wrappers still encode structure, just statistically rather than manually. Change the template, and the wrapper breaks, exactly as hand-written selectors break. Cross-domain generalization remained poor; a wrapper trained on one site's HTML had little predictive power over another site's structurally different HTML. The brittleness moved from the rule-writing process to the training process. It did not disappear.

The deeper issue is that rule-based methods, whether hand-written or learned, understand where content sits in a document. What they cannot understand is what the content means.

LLM-based extraction: finding fields by meaning rather than position

The premise shift is fairly simple to state, even if the implementation is not. Instead of specifying where data lives in a document, you specify what data you want, framing extraction as an information extraction task rather than a structural query. The model finds it based on semantic meaning. Layout becomes irrelevant; the field name and a description of its type are sufficient.

Two practical patterns have emerged. Schema-based extraction defines a JSON schema with field names and types; the model populates it from whatever HTML or text it receives, regardless of how the page is structured. Prompt-based extraction uses natural language instructions with no schema required, and works for exploratory or one-off tasks. Both survive site redesigns in a way no selector can, because they read meaning rather than structure.

What the model actually receives turns out to matter as much as the model itself, which took me longer to fully appreciate than it should have. The 2025 NEXT-EVAL benchmark found that feeding LLMs Flat JSON yields an extraction F1 score of 0.9567, substantially better than passing raw or minimally cleaned HTML. A meaningful portion of LLM extraction error comes from noisy input rather than model capability. The AXE paper (Cairo University, February 2026) extended this further, showing that pruning the DOM before sending it to an LLM reduced input tokens by 97.9% while maintaining an F1 score of 88.1%, even using a model of only 0.6 billion parameters. The model does not need the full document. Related work on fine-tuned small models, published on arXiv in early 2026, has shown that specialized extractors can reach accuracy competitive with much larger general-purpose models, pointing toward a future where LLM-based extraction is both capable and economical at scale.

The real cost of LLM extraction and what the accuracy numbers mean in practice

A single product page can contain tens or hundreds of thousands of tokens of HTML. Multiply that by a million pages, and the API cost of naive LLM-based extraction becomes difficult to justify at any reasonable margin. I have seen teams learn this the hard way, running enthusiastic proof-of-concept benchmarks and then quietly shelving the approach when the production cost estimates came in.

An arXiv benchmark from January 2025 reported approximately 92% accuracy for LLM-based extraction, at 10 to 50 times the cost of traditional methods, concluding it is best suited for small-scale, high-variety tasks where adaptability matters more than throughput. That conclusion deserves more weight than it typically receives.

The 92% figure is high enough to be genuinely useful. It is also low enough to matter depending on what you're building. At small volumes, an 8% error rate may be acceptable. At millions of records, those errors compound into downstream problems that are expensive to diagnose and even more expensive to correct retroactively. Rule-based extraction on well-maintained selectors, for the pages it actually covers, runs close to 100% accuracy. The question worth sitting with is not which method is more accurate in the abstract, but at what scale does the error rate stop being acceptable, and does the answer change your architecture.

Throughput introduces a separate constraint. LLM inference is slower than a local parser. Latency and rate limits become real bottlenecks at scale in ways that CSS selectors simply do not produce.

Where LLM extraction earns its cost is specific: high-variety sources where maintaining selectors for each would require constant engineering attention; unstructured text within pages, review bodies and free-form descriptions, where no selector can isolate a semantically meaningful field; low-volume, high-value extraction where the per-record cost is acceptable. DOM pruning and Markdown or Flat JSON preprocessing are the main levers for reducing cost and managing the token budget without sacrificing much accuracy, and they operate before the LLM ever sees the content.

Structured data already embedded in HTML: Schema.org, JSON-LD, and microdata

There is a dimension of HTML-to-structured-data extraction that is easy to overlook, and I will admit I underweighted it for longer than was sensible: some sites publish their own structured data, machine-readable markup embedded directly in the page and intended for search engines. When it is present, it is the fastest and most stable extraction target available, and checking for it first costs almost nothing.

Schema.org is the shared vocabulary underpinning this ecosystem. Developed collaboratively by Google, Microsoft, Yahoo, and Yandex, it defines standardized types, including Product, Article, Event, and Review, each with named fields and expected value types. Sites implementing Schema.org semantic markup are, in effect, publishing a structured version of their own content alongside the visual presentation.

Three formats are in use. JSON-LD embeds a JSON object inside a script tag with the type attribute set to "application/ld+json." Extracting it requires only parsing a JSON string from a known tag, with no DOM traversal. Microdata uses itemscope and itemprop attributes added directly to HTML elements, requiring a walk of the DOM to collect property values. RDFa is a similar attribute-based approach, more expressive in principle but considerably less common in practice.

When JSON-LD is present, it should be the first extraction target. It is faster than any selector-based approach, more stable than anything that depends on visual layout, and requires almost no maintenance as long as the site continues publishing it. The limitation is coverage: structured markup is common on large commercial sites with strong SEO incentives, but far from universal. Any pipeline relying on it exclusively will miss a substantial portion of the web.

The practical sequencing is to check for embedded structured data first, then fall back to selector-based or LLM-based extraction when it is absent. That captures the easiest cases at near-zero cost and reserves the more expensive methods for content that actually requires them.