Schema Design for Web Extracted Data

Picture a developer who just scraped ten thousand company profile pages and got back ten thousand slightly different JSON objects. Some have company_name, some have name, some have companyName. Some store funding as a number, some as a string with a dollar sign still attached. Nothing downstream can read that cleanly: whether web-extracted data is usable later is decided by a single choice made, or skipped, before a single page gets scraped.

Why schema belongs at the start of extraction

The scenario above isn't a bug in some junior developer's script. For years, that's roughly how it worked: one group built the scrapers, another group built the pipelines that consumed the scraped data, and the handoff point between them was the schema, treated as the pipeline team's problem to clean up after the fact rather than the extraction team's constraint from the start, because web extraction and AI pipeline design used to be separate disciplines. Whoever did the extraction treated schema as somebody else's cleanup job. Whoever built the pipeline treated it as something they'd untangle once the data landed on their desk.

Neither side owned it, which meant nobody designed it on purpose.

The cost of that gap doesn't stay small. A messy extractor doesn't hand off a tidy dataset with a few typos in it. It hands off a sprawling mess that gets messier at every step after: the CRM sync chokes on inconsistent field names, the data warehouse ingestion job needs a translation layer nobody budgeted for, and the LLM at the end of the pipeline either hallucinates values to fill gaps or just gets the question wrong. Schema debt compounds the same way technical debt does: quietly, then all at once, usually during a demo.

The fix is deciding the shape of the data before the first page loads, not writing a smarter cleanup script after the fact.

What "AI-ready" requires from a schema

"AI-ready" gets thrown around as a vague compliment, like calling a restaurant "cozy" when what you mean is "small." The actual requirement is specific. An AI-ready schema names consistent fields, assigns each one an explicit type, and sets nullability rules that apply the same way to every record, no matter which source page the record came from.

A schema that holds for records from one site but falls apart for records from another is a guess that worked once, since consistent fields, types, and nullability rules are what let the schema be applied the same way regardless of source.

Output format is part of this same decision, not a separate styling choice made later. Markdown keeps headings and list structure, which helps a model understand how a page is organized, and it does this while staying token-efficient, which makes it the right pick for reasoning tasks. JSON hands over typed fields that slot straight into a database with nothing left to transform. Raw HTML, by contrast, is the format to avoid in an AI pipeline: it runs five to ten times heavier in tokens than the same content converted to clean markdown, and all that extra weight is navigation bars, cookie banners, and ad markup, none of which adds a single fact worth extracting. Choosing a format is choosing a cost structure and a quality ceiling at the same time.

The rule that does the most quiet work in any schema is a nullability rule: when a field is missing, the extractor returns "none found." It does not skip the field, and it does not guess. That one rule is what keeps invented values out of the dataset and keeps a record that's labeled "CRM-ready" actually ready for a CRM, instead of ready for an awkward phone call to sales ops.

Data-Mania's ICP extraction workflow shows what this discipline looks like applied to a real use case. Before any extraction logic gets written, each ideal-customer-profile attribute (industry code, employee band, open roles, tech stack signals, buyer title keywords) gets mapped to one specific structured field and one specific source. The schema is written as the spec for the work, not reverse-engineered from whatever the scraper happened to produce.

Don't flatten everything into one record. Funding rounds, for instance, belong in a separate linked table rather than jammed into the main company record, because that keeps the primary record clean and makes it possible to filter by "most recent round" without wading through every round the company ever raised.

Schema enforcement at the point of collection

LLM-powered extraction moves schema enforcement from a cleanup job done after collection to a guarantee built into the collection step itself. The model gets handed a schema, reads the page's markdown, and returns only the fields that match the schema. No CSS selectors to maintain, no brittle XPath that snaps the moment a site redesigns its product page.

That resilience is the whole argument for using a model instead of a selector. The 2026 scraping landscape analysis puts it simply: a model adapts to a new DOM structure as long as the semantic content is still there, and that is why schema-constrained LLM extraction holds up better than CSS or XPath scraping across sources that are inconsistent or get redesigned often.

The ScrapeGraphAI-100k research dataset backs this up with evidence rather than intuition. It pairs markdown-converted page content with prompts, schemas, and the LLM responses those prompts produced across real production pipelines, and failure rates jump sharply once schema complexity passes a certain threshold. A schema has a complexity budget, and blowing past it breaks results rather than degrading them gracefully.

The obvious objection here is cost: LLM extraction is slower and pricier than firing off an HTTP request and parsing the response with a selector. That objection is fair, and it's also not a reason to avoid LLM extraction, just a reason to sequence it correctly. The practitioner consensus is to start new projects with LLM extraction while the schema is still being worked out, then migrate stable, high-volume targets to selector-based scraping once the schema has settled and throughput starts to matter more than flexibility.

None of this works without a precise objective. "Extract company name, founding year, CEO name, total funding raised, and headquarters city" produces consistent output because it names what the schema requires.

The cost reality of schema-first extraction at scale

Schema decisions are cost decisions, and the cost driver is what kind of rendering each target page needs, not how complicated your schema is.

A workload analysis documented in the sources sorts real pages into three tiers. Most pages resolve with a plain HTTP request, no browser, no proxy, nothing fancy. A small remaining slice needs full anti-bot bypass, the pages that actively fight back against being read. Charging a flat rate per page regardless of which tier it falls into means overpaying for the easy majority to subsidize the hard minority. A tiered pricing approach that charges based on what each page actually required delivers the same data quality for a fraction of the spend.

Scale changes the math further. At high monthly page volumes, the cost gap between AI-only and static-only extraction is extreme in opposite directions: pure AI-powered extraction is prohibitively expensive, while a pure static framework is cheap but fails on a significant share of modern sites. Neither extreme is the answer.

The hybrid approach is: run static parsing on the majority of pages that parse cleanly without help, and reserve LLM extraction for the pages that actually need it. That split keeps the schema consistent across every record in the dataset, because the schema doesn't care which engine produced the data, while keeping the per-page bill from spiraling.

Proxy spend, not the extraction engine, usually ends up the biggest recurring cost for hostile targets, so it deserves attention at schema design time, not after the invoice arrives. Residential proxy pricing runs per gigabyte, cheaper on annual commitments than on pay-as-you-go, and that cost scales directly with how many anti-bot-protected sources sit in the schema's source list. Mapping each source to its likely rendering tier before extraction starts tells you whether the budget you wrote down is realistic or fiction.

What happens to the schema when sources change

A schema built before extraction starts is a snapshot of how sources looked on the day it was written. The web doesn't hold still, and the gap between the snapshot and the current state of a site is where otherwise well-built pipelines start quietly producing bad data without anyone noticing right away.

Schema drift shows up in two different flavors, structural and semantic, and they call for two different fixes. The first is structural: a site redesigns its layout, renames a field, or moves a data point from one element to another. LLM-based extraction shrugs off a lot of this automatically, because it's reading for meaning rather than hunting for a specific tag in a specific spot. The second is semantic, and it's sneakier: a field keeps its name but changes what it means, like a "pricing" page that used to list dollar amounts and now just lists plan tiers with a "contact sales" button. That kind of drift doesn't trip any error message. It requires watching the schema itself and not just the selectors that used to point at it.

Domo's Magic ETL updates, discussed between March and April 2026, describe a response built for exactly this: machine learning that auto-detects schema changes, maps fields against past patterns, and recommends transformations without anyone writing new code by hand. That cuts the setup and recovery time traditional ETL work demands every time a source changes shape.

There's a newer wrinkle specific to agent-based extraction: an AI agent can fail on a site that looks perfectly healthy by every normal measure, because the breakage happens somewhere in the agent's reasoning or tool-discovery path rather than in a server error code.

The design takeaway: flag the fields most likely to drift (pricing, plans, leadership, open roles) at schema design time, and give those fields shorter refresh intervals and tighter monitoring than the fields that barely ever change, like a company's founding year.

Site-side schema standards that extraction pipelines should consume

Schema design isn't only an extraction-side concern anymore. A growing number of publishers are putting out their own machine-readable structure, and extraction pipelines that ignore it are leaving reliability on the table.

Schema.org, expressed as JSON-LD sitting in a page's <head>, is the format worth building around. JSON-LD gets parsed more reliably by LLM crawlers than inline microdata, and it lives in its own script tag instead of bleeding into the visible markup. Treating schema.org as a full semantic layer describing the page, rather than just a trick for getting a rich snippet in search results, is the standard as of 2026.

llms.txt is the newer entry. Jeremy Howard of Answer.AI proposed it in September 2024 as a Markdown file sitting at a site's root, giving AI models a clean, curated map of the site's most important pages without forcing them to chew through heavy HTML to find anything. A common pattern by 2026 pairs a short llms.txt index with a longer llms-full.txt that inlines the actual page content, so a model can go as shallow or as deep as it needs.

When these standards are present on a source, extraction pipelines can lean on them as a higher-confidence field source than whatever gets scraped off the rendered page, lowering the odds of hallucinated values and improving how complete the resulting records are. When they're absent, which is still the more common situation, the pipeline has to infer structure from the page content alone, which makes the nullability rules and field-mapping discipline covered earlier more important, not less.

What the agentic protocol stack demands of a schema

The newest wrinkle in schema design is about permission, not fields. The 2026 agentic protocol stack asks a schema to express something it never had to before: not just what an agent can read off a page, but what the site actually lets an agent do.

WebMCP is the clearest example. Built jointly by Google and Microsoft through the W3C Web Machine Learning Community Group, with an early preview that shipped in Chrome Canary in February 2026, it effectively turns a website into an MCP server that any visiting agent can discover and use. That flips the old relationship on its head. Extraction used to mean scraping by force, pulling data off a page whether the site wanted you to or not. WebMCP points toward capability by consent, where the site declares what an agent is allowed to do and the agent works within that declaration.

Sitting above individual site permissions is the Agent2Agent protocol, A2A, version 1.0, released March 12, 2026 (the protocol itself was first announced back on April 9, 2025). It runs under Linux Foundation governance, backed by more than 150 supporting organizations including AWS, Cisco, Google, IBM, Microsoft, Salesforce, SAP, and ServiceNow. A2A governs how agents talk to each other, a layer of coordination that sits above whatever any single site permits on its own.

Research tracking the MCP ecosystem found something schema designers need to plan for directly: the share of "action" tools, meaning tools that change something out in the world rather than just reading it, grew quickly between November 2024 and February 2026. Agents are shifting from "look but don't touch" to "look and then do something about it," and a schema built only to capture what was read on a page has no field for what an agent went on to do with it.

One tension nobody has resolved yet: WebMCP can describe what a site permits, and llms.txt can describe what a site is offering up for free, but neither one describes what the site gets back in return for granting that access. The economics of agent-friendly access, and whether it's sustainable for the publisher granting it, remains an open question across the industry.

Agent identity makes the whole problem harder to pin down. Most agents move through the web blending into ordinary traffic, and only a small minority identify themselves with a stable, recognizable signature. A schema built to log who (or what) accessed a page, and under what permission, runs into a simple problem: most of the time, there's no reliable way to know who showed up in the first place.