Scrape Info

Validating Extracted Data Against a Target Schema

Parsing proves nothing about meaning, so check structure and semantics.

· 9 min read
Cover illustration for “Validating Extracted Data Against a Target Schema”
Data Extraction · November 22, 2026 · 9 min read · 1,915 words

LLM-ready structured data extraction from web pages starts with a target schema. The reliable pattern is to define the contract, fetch rendered content, extract against that contract, and validate both structure and meaning before storing anything. Validation is how a pipeline turns plausible mistakes into visible, recoverable errors.

Why the order of operations in a schema-first pipeline is not obvious but determines whether failures are visible

The model returned JSON, the code parsed it, and the pipeline looked healthy. Yet the price belonged to a related product. Nothing crashed because nothing had checked that distinction. A successful parse proves only that the response is readable by software.

The mistake starts when the schema is defined at the end. Without a target contract in the extraction call, the model generates its best guess at what the output should contain. A validator added afterward can inspect that guess, but it did not constrain the operation that produced it. The steps run from raw input to schema definition, then to the LLM extraction call and validation. Each stage has a separate job.

For live-web systems, planning for the input starts even earlier, with the contract defined before the fetch. That distinction becomes important when a site changes. A field that returned the right value last month may return a different value that looks right this month, while parsing keeps working. At high page volumes, the result is a growing correction bill downstream. That bill does not appear in per-token pricing.

Diagram: Schema-First Pipeline: Four Ordered Stages. Visualizes: Show the ordered sequence of four stages in a schema-first extraction pipeline: (1) Define schema/contract, (2) Fetch rendered content, (3) Extract against contract (LLM call with…

What the schema actually is in this pipeline: a typed contract the LLM receives before it ever reads the page

A Pydantic schema defines field names, types, and which values are required or optional. Its role goes beyond documentation. The definition also supplies the target shape that the model receives through its function-calling definition or structured-output configuration. Conceptually, even a small contract distinguishes a required numeric price from an optional field that may be absent.

Provider-level enforcement can constrain generation at decoding time. OpenAI supports response_format with the type json_schema and constrained decoding. Google Gemini supports JSON Schema features including enums, formats, numeric bounds, and array bounds. AWS Bedrock added structured outputs through its Converse API in early 2026, while Mistral supports schema-constrained JSON across all models except codestral-mamba.

However, provider support is not interchangeable. Claude's strict mode rejects recursive schemas, minimum and maximum constraints, string-length constraints, and external $ref references that OpenAI accepts. Builders therefore need to move unsupported checks into Pydantic post-validation rather than assume the provider enforces the entire contract. The contract can remain strict even when the provider-facing schema must be narrower.

Structural validation: what provider-level enforcement and Pydantic actually check, and what they cannot

Structural validation checks whether the output has the shape, types, and required fields it should have. It catches missing required values, nulls where nulls are forbidden, invalid enum values, and type violations under the configured rules. A price arriving as text when the schema requires a float belongs in this layer. Production code using Pydantic 2.x should use UserData.model_validate_json(raw_response), the replacement for the older parse_raw() form. This is the checkpoint for whether a record fits the contract.

That checkpoint has a firm limit. A numeric price from the wrong product can pass, as can a valid date lifted from a footnote when the document header is the required source. If a layout change redirects extraction toward the wrong DOM node, the same mistake can pass repeatedly. The validator is doing its job, just not a different job that nobody assigned it.

The Guardian Parser Pack paper separates schema-validation outcomes from extraction completeness. In its evaluated run, all LLM outputs passed initial schema validation, so validator-guided repair remained a safeguard with no role in the observed gains. That distinction prevents a misleading conclusion. Passing validation established that the output met the structural rules, leaving completeness and correctness to be evaluated separately.

Semantic validation: confirming that a structurally valid value came from the right place on the page

Semantic validation asks whether a value belongs to the intended field and source element. When a page includes JSON-LD, microdata, or schema.org markup, builders can extract those values deterministically and compare them with the LLM output. A mismatch should flag the record for review. The disagreement is evidence to examine.

Field-specific checks add another useful filter. Email addresses need a regex or a dedicated validator such as email-validator. Dates need parsing and range checks, while prices need plausibility checks against a known floor and upper bound for the product category. Company names and job titles can be checked against a known list or secondary lookup when accuracy is important. These checks narrow the error space without rerunning the LLM for every record.

Plausibility alone is still not enough protection. Cloudflare's AI Labyrinth returns fake data that looks real in place of a normal block page, so a neatly structured response can contain no real content. An independent signal is essential in that situation. The Guardian paper illustrates the broader pattern through deterministic and LLM-assisted paths harmonized to the same schema. The deterministic path provides a separate signal against which the LLM path can be checked, while separate completeness and validation measurements preserve auditability.

Schema drift: how a correct extraction today becomes silent corruption next month without any code change

Schema drift is the growing gap between the pipeline's expected contract and what the source actually returns. It can enter through a changed DOM, a provider model update that alters extraction behavior, or an internal schema revision without a coordinated migration. All of these can happen while the extraction code itself still works. A record can still look fine on its own while the way the pipeline behaves changes across a run.

Teams can watch for this by tracking field fill rates per run. A price field dropping from near-complete coverage to substantially lower coverage provides an aggregate warning that isolated record checks may miss. Fill rate will not expose every wrong-but-present value, so it complements semantic checks. Automated DOM-change detection can also give an early warning. Some production teams use it to create a repair window before the scraper fails.

If an extractor cannot produce a valid critical field twice in a row, a visible stop-and-error is preferable to continued accumulation of suspect data. Regulated workflows raise the stakes further. SEC Rule 17a-4 requires electronic records to be kept in a non-rewriteable, non-erasable format or through an audit-trail alternative, and FINRA Rule 4511 requires them to be kept in a format that complies. Undetected drift in those workflows creates audit exposure as well as bad records.

Failure handling at scale: retry logic, dead-letter queues, and deciding when bad data is worse than no data

Failure handling starts with what downstream systems will do with the result. In most production pipelines, wrong data is worse than absent data, but that choice needs to be explicit. A financial record, CRM write, or compliance workflow calls for strict validation that raises an error on a field violation. Systems that will act on a value cannot treat “looks reasonable” as an acceptance rule. A confident mistake is still a mistake.

Graceful degradation fits a different downstream contract. When consumers can handle nulls and throughput takes priority over completeness, missing optional fields can return None while the rest of the record proceeds. The downstream code must spell out how to handle that missing value. Optionality belongs in the schema and the consumer's behavior.

For recoverable failures, a heal cycle can re-fetch, re-extract, and re-validate before committing the record. Retries need a configured limit. Exhausted failures should go to a persistent dead-letter sink, such as a database table or file, containing the source URL, raw LLM output, and specific validation error. That evidence enables replay after the schema or prompt is repaired. Console output is not a recovery plan.

Agentic pipelines need the same checks before the agent accepts tool output as ground truth. Validation after the agent has acted is too late to prevent the action. Downstream models can amplify upstream noise, making the tool boundary the useful place to interrupt the chain.

How schema-first validation works inside orchestration frameworks: Mastra, n8n, and MCP tool boundaries

Orchestration frameworks put this contract in different places, but the same rule holds across them. The definition used at runtime should match what the model receives. Otherwise, a manually translated schema can drift from the validator it supposedly represents. Tool definitions offer a place to close that gap before extraction results become inputs to another action.

Mastra uses Zod schemas for tool input validation and generates the LLM's JSON Schema function-calling definition from the same schema. This removes a manual translation layer where mismatches can accumulate. Its January 2026 release gives a date for the version of the framework discussed here. Mastra also implements both client and server sides of MCP, allowing schema-first extraction to compose across tools and runtimes. Input validation still has a different job from validating extracted results before use.

In n8n, the enforcement step belongs before the database or CRM write. A documented B2B lead enrichment pipeline uses a FastAPI service to call an LLM with function-calling, convert scraped content into structured JSON, and validate it against strict Pydantic schemas with retry-on-failure. Unvalidated output never reaches the database. Agent-compatible enrichment also calls for a synchronous REST API, deterministic entity resolution, and structured JSON without HTML or Markdown. An MCP server can make enrichment available as a first-class tool.

MCP tool boundaries carry the typed contract from one call to the next. JSON Schema gives the pipeline a definition it can enforce when an extraction request would otherwise be open-ended. The decisive distinction is placement. A boundary check prevents an agent from acting on invalid output, while an exit check may only discover the problem after the action.

How do you get live web content into an LLM-ready, schema-first pipeline?

A production pipeline begins by defining one JSON Schema for the target record type and reusing it across sites in the same category. The fetch layer then supplies rendered Markdown from the live DOM rather than raw HTML, giving the LLM more consistent input despite JavaScript rendering. Bot-protection responses, including fake-data responses, first arrive at this layer and must be identified as failed page retrievals.

The extraction call receives the schema as its function-calling definition, with provider-level constrained decoding where available. Each record that comes back then goes through structural validation and the semantic checks that apply to it. Failures go through a retry path with a set limit, and attempts that use up their retries are kept in the dead-letter sink. Records that pass keep their source URL for provenance, while fill-rate monitoring checks each run for drift. Monitoring continues after deployment because the web does too.

At scale, this design does not require LLM inference for every extraction. A hybrid approach uses an LLM to generate deterministic scraper code, then runs that code while AI agents monitor scripts and regenerate them when sites change. The common schema keeps both paths aligned. This makes source type a routing decision rather than a reason to maintain incompatible output contracts.

The Guardian paper demonstrates the trade-off on a manually aligned subset of complex, heterogeneous sources. Its LLM-assisted path achieved F1 = 0.8664 versus 0.2578 for the deterministic comparator, while the deterministic path was much faster, with mean 0.03 s/record versus 3.95 s/record. The production goal is usable data with enough checks and evidence to know when not to use it.

Diagram: LLM Path vs. Deterministic Path: Speed–Accuracy Trade-off. Visualizes: Show a two-path comparison between the LLM-assisted path and the deterministic path on two dimensions: accuracy (F1 score) and speed (seconds per record).

Sources

  1. LLM-based Schema-Guided Extraction and Validation of Missing-Person Intelligence from Heterogeneous Data Sources
Filed underData Extraction

More in Data Extraction