Scraping API Output Formats Ranked by LLM Ingestion Compatibility
Markdown wins for RAG, JSON for databases, and HTML comes last.

For LLM-ready structured data extraction from web pages, schema-conforming JSON is the right output when fields must feed a database or tool. For RAG and reading, boilerplate-stripped Markdown ranks first because it preserves useful structure without the page’s clutter. Plain text wins when structure adds nothing, while raw HTML requires parsing before LLM ingestion.
Why output format is the first architectural decision in an LLM data pipeline
When a webpage reaches a model without cleaning, navigation menus, cookie banners, footer links, ad scripts, and social widgets arrive with the useful content. The model tokenizes that material too. A cookie banner has invited itself to sit in while the model reasons.
That is the basic problem with sending raw HTML straight into an LLM. The scraping layer’s output determines how much useful content fits in each call and how much noise surrounds it. Format therefore sets constraints before retrieval or reasoning begins.
Those limits also shape how much can be batched in one call. Far fewer raw pages than cleaned documents fit in a call, and compact extracted fields can take up even less space. A pipeline that needs complete passages has a different budget from one that needs only product names and prices. Choosing the format first makes that distinction explicit, rather than leaving a downstream component to discover that the inputs are too large or too vague.
What the four formats actually deliver and how they differ mechanically
The four formats differ along two axes, namely how much structure they preserve and how many tokens they use per unit of useful content. Raw HTML has the full DOM, including parts outside the content. Every tag, attribute, and script block becomes text for the model to tokenize. Plain text takes out the markup and leaves the prose. It also removes heading hierarchy, list structure, and table relationships.
Markdown retains lightweight signals for headings, lists, code blocks, and tables while remaining readable as prose. Models encounter those signals in training data. Schema-conforming JSON instead represents content as named, typed fields that validate against a contract. Its organizing unit is the field, and database-bound data needs no second prose-extraction pass.
The size difference can be substantial. Cleaning raw HTML into Markdown or JSON can reduce token usage by up to 60%. That reduction requires removing material. These formats deliver different representations, so a useful comparison must ask what survives the conversion as well as what disappears.
Why Markdown is the default winner for RAG and reading tasks
Boilerplate-stripped Markdown ranks first when a model must read, summarize, or reason over web content. Its structure is familiar to LLMs from training data, including heading syntax, fenced code blocks, and pipe-delimited tables. Those signals help distinguish sections, instructions, code, and relationships within a document. The model gets signposts without the scaffolding.
A chunker can split Markdown at heading boundaries rather than treating a long document as an undifferentiated string. The resulting chunks can then be embedded and retrieved for generation. Plain text gives the chunker fewer explicit boundaries, forcing it to infer structure that Markdown preserves.
However, Markdown is not automatically clean content. A conversion can faithfully reproduce navigation, ads, related-content blocks, and footers in lightweight syntax. Boilerplate-stripped variants such as markdown-llm address that problem at the API layer.
The qualifier “boilerplate-stripped” is therefore part of the recommendation. An API that returns unprocessed Markdown has completed the conversion but has not necessarily completed the preparation.
Where structured JSON outranks Markdown for typed pipelines and database-bound extraction
A database does not need a readable product description when its next operation requires a named price field. It needs an addressable value under an agreed contract. Schema-conforming JSON ranks above Markdown for that job, including product catalogs, pricing feeds, and structured datasets.
An agent may read Markdown to reason about a page, but a tool expecting a price argument needs typed data. JSON serves that handoff directly. Its advantage is that the receiving system can use the fields without another LLM extraction pass.
Schema drift is the production trap. An extraction model might return salePrice for one site, discountedPrice for another, or omit the field entirely. Valid-looking JSON is not necessarily schema-conforming JSON.
A strict JSON schema must be enforced at the API or extraction layer, with validation before data moves downstream. Waiting until the loader makes consistency a late-stage problem. For high-volume, standardized extraction, dedicated APIs returning structured JSON are more cost-effective than retrieving Markdown and paying for another LLM pass to recover the fields.
Plain text’s specific role is maximum compression when structure is irrelevant
Plain text earns its place when prose is the whole signal and document structure adds no retrieval or reasoning value. That is the case for sentiment analysis over large corpora and for topic classification. It also holds for fine-tuning datasets when wording alone supplies what the model needs to learn.
The loss is still real elsewhere. Plain text cannot preserve heading boundaries, table relationships, or the distinction between code and prose in the way Markdown can. Compression does harm when the next step needs those signals.
The practical test is what the pipeline would remove anyway. If all Markdown formatting would be stripped before embedding, plain text avoids that extra processing step. If headings would define chunk boundaries, Markdown retains information the pipeline needs. Plain text is a narrower tool than Markdown.
Why raw HTML ranks last and when it is nonetheless the right choice
Raw HTML ranks last for direct LLM ingestion. A typical product page is overwhelmingly boilerplate by token count, including navigation, scripts, ads, and structural tags. That material dilutes the useful content and consumes context that could hold evidence for the task. A closing div tag seldom has the answer to the customer’s question.
Its legitimate uses begin with a different next consumer. BeautifulSoup, lxml, and browser-automation-based extractors can sit downstream when code owns parsing and custom extraction requires the full DOM. Developers may also need the raw response to debug what was returned. In those cases, preserving HTML serves an explicit operation rather than asking the model to clean up first.
Preprocessed HTML offers a narrow middle ground. An html-llm variant strips boilerplate while retaining structure for pipelines that need CSS selectors or element hierarchy. It is not raw DOM output, so its usefulness does not overturn raw HTML’s position in the ranking.
The dividing line is straightforward. If a human or LLM is the next reader, raw HTML is the wrong output. If an HTML parser owns the next step, it may be exactly the input required.
How the ranking shifts by pipeline task across RAG, agentic workflows, fine-tuning, and structured extraction
For RAG, the default workflow starts with boilerplate-stripped Markdown, chunked by heading boundary and stored with source metadata. That metadata includes the URL, extraction time, and content fingerprint. The ingestion contract should spell out what the extraction must deliver before tools are compared. Otherwise, a format label can stand in for requirements it does not actually satisfy.
Agentic browsing introduces two distinct needs. Clean Markdown helps an agent reason over many pages because the token costs for each page add up across dozens of fetches. Plain text is a viable second choice when structure adds nothing to that reasoning. But an agent that collects specified fields is better served by crawl APIs with extraction built in, which return structured results without a second parsing pass.
Access is a separate prerequisite. Agent traffic from datacenter IPs can encounter 403s or CAPTCHAs before any useful content arrives. For that constraint, residential or mobile carrier IP routing comes before format optimization.
For fine-tuning, the choice depends on what the model should learn and what must be kept for it to do so. Plain text fits prose-only learning, while clean Markdown preserves headings and code blocks when those are part of the objective. Structured JSON fits datasets built around field-level patterns. For product catalogs, pricing, and entity extraction, schema-conforming JSON remains the right deliverable, with enforcement at extraction time.
Many production pipelines therefore use both Markdown and JSON. Markdown serves instructions, long-form content, and retrieval, while JSON serves extracted fields and downstream results. Storage follows the same rule. RAG needs Markdown plus metadata, analytics and BI may need tabular JSON or CSV, and search indexing needs JSON with normalized text fields. One pipeline can have several consumers without forcing them all to eat the same meal.
The token-cost and accuracy tradeoff that the format ranking cannot resolve alone
Removing headings or table relationships can save tokens while depriving the model of useful context. Keeping structure that the task never uses also adds cost without helping the task. Efficiency has to be measured against useful work.
A compact JSON record can be much smaller than a full Markdown page because it contains only selected fields. Token-cost comparisons between compact JSON and long prose for reading involve different deliverables.
Volume makes the tradeoff harder to ignore. LLM-driven extraction adds up across thousands of pages per day, and at millions of pages inference costs can become unmanageable depending on the format and model tier. The recommendation is to retain richer structure where reasoning requires it and use plain text or compact JSON where compression is the limiting constraint.
The scraping layer belongs in that calculation too. Specialized APIs that handle JavaScript rendering, anti-bot measures, and normalization can reduce preparation time compared with building the cleaning pipeline in-house. Saving tokens while taking ownership of an elaborate cleaning system is not automatically a bargain.
How do developers choose scraping output their LLM can use without cleaning it first?
First, the format must suit the output’s next recipient. For a human or LLM reader, the candidates are clean Markdown and plain text. For named fields bound for a database or typed tool, the answer is schema-conforming JSON. For downstream code that owns HTML extraction, raw HTML or html-llm may fit.
Second, the format must preserve the structure the task uses. Markdown works well for chunking by heading and moving between sections, while plain text works well when only the prose needs to be processed. Field identity favors JSON.
Third, the chosen format and model tier must be costed at the intended volume. A design that works for hundreds of pages may be uneconomical at millions. Validate that cost model before committing to the architecture. The relevant comparison includes both the initial fetch and any second extraction pass.
Fourth, responsibility for boilerplate removal must be assigned. APIs returning pre-stripped variants such as markdown-llm or text-llm include that cleaning step, while raw output leaves its complexity with the developer. The practical workflow is to fetch, clean, extract, and validate, with those responsibilities assigned explicitly even when an API combines them. For structured extraction, validation belongs before the downstream handoff.
Together, these questions define what “LLM-ready” must mean for the purchase. The right API delivers the representation the next consumer needs, with cleaning and schema enforcement performed at the appropriate stage under a contract chosen before the pipeline is built.


