Extracting Structured Data From JavaScript-Rendered Pages

Getting structured data off a JavaScript-rendered page is a two-step job: first you get the fully loaded page into your hands, then you turn what's on it into clean, usable fields. If you skip step one, step two never had a shot.

This is where a lot of scraping projects quietly fall apart. The HTML comes back valid. Looks fine on paper. Open it up, though, and there's nothing there worth reading, just an empty div waiting on a script that never ran.

That should worry you more than a crash would. A pipeline that errors out gets noticed fast. A pipeline that runs clean on empty content doesn't get noticed at all, and that's the more dangerous failure. Feed an LLM that hollowed-out page and it won't complain. It'll just write something vague and confident-sounding that says nothing. Nobody catches it until a customer, or worse a client, points out that the "product description" your AI generated reads like fortune-cookie filler. By then it's already shipped. Most modern sites, e-commerce stores, single-page apps, content platforms, build their visible content with JavaScript frameworks like React, Vue, Angular, and Next.js.

The two-layer architecture: rendering first, extraction second

Splitting the problem into two layers (first getting the fully rendered DOM, then mapping that content into a clean, queryable schema) stops it from being a mystery. Layer one is about getting the complete page: run the JavaScript, wait for it to finish loading, capture the network calls, or fetch a pre-rendered version from a service that does that work for you. Different methods, same goal: a fully populated page before you touch a single data point.

Most teams treat this as one blurry problem, and that's how you end up spending weeks debugging the wrong layer. Garbage goes in, garbage comes out, and it happens quietly. The tooling for each layer is genuinely different, too. Layer one is about browsers, networking, and infrastructure. Layer two is about parsing logic and schema design. Cost and slowdowns appear in different spots depending on which layer you're staring at, and that matters once you're the one paying the cloud bill for either.

At layer two, there are two real paths. Rule-based extraction, CSS or XPath selectors, is cheap, fast, and easy to debug when the page markup holds still. LLM-based extraction costs more per page but shrugs off redesigns and cleans up messy data as it goes. Both get a full walkthrough further down.

Layer 1 options: five approaches to getting a fully rendered DOM

Five well-worn ways exist to get from empty shell to fully rendered page, and picking the wrong one is how scraping budgets quietly balloon.

It works almost everywhere. It's the fallback everyone reaches for first, even when it's overkill.

XHR and API interception skips the browser part entirely. Instead of rendering the page, you watch the network requests the page makes on its own and read the JSON straight off the wire. When the backing API is reachable and nobody's blocking you, this is faster and a lot cheaper than spinning up a whole browser just to throw away the visual layer you never needed.

Pre-rendering and SSR services hand the rendering job to a third party, and you get back already-rendered HTML. Useful if running a fleet of browsers isn't something you want on your plate.

Browser automation frameworks, meaning Playwright and Puppeteer, give full programmatic control over a real browser session. Built for something else entirely (get to that in a second), they've become the workhorse tools for scraping anyway, a fit that isn't always good.

Managed cloud APIs push the entire mess onto a vendor, browsers, proxies, retries, anti-bot defenses, all of it, leaving you to just call one endpoint. That tradeoff gets its own section later.

Before reaching for any of these, check the page's source for a <script type="application/ld+json"> block. Plenty of sites drop their schema.org structured data right there in plain JSON. If the fields you need are already sitting in that block, skip rendering entirely and just parse the JSON. It's the scraping equivalent of finding a shortcut nobody bothered to lock.

Some pages won't give up their data no matter what, not until you interact with them: clicking a "load more" button, scrolling to trigger a lazy load, filling out a form. That needs a real browser session that can act like a person clicking around.

Self-hosting any of this comes with a real infrastructure tax, and this is the part the tutorials skip. Each headless browser instance eats somewhere between 200 and 500 megabytes of RAM, and that cost scales linearly as you add concurrent instances https://zackproser.com/blog/how-to-scrape-javascript-sites. This is usually the moment someone on the team asks, out loud, if building this in-house was a good idea.

Headless browsers in practice: Playwright, Puppeteer, and their operating costs

Playwright, built by Microsoft, drives real headless Chromium, Firefox, and WebKit browsers. It runs the page's JavaScript, waits for network requests to settle, and hands back the fully rendered DOM, with support across Chromium, Firefox, and WebKit.

Playwright wasn't built for scraping. It was built for end-to-end testing, checking that a checkout button actually works before you ship. Using it to scrape means bending testing infrastructure toward a job it was never designed for, and that mismatch is visible in its defaults and its abstractions. It's a bit like using a stethoscope to check whether your walls are load-bearing. Works, technically. You can still feel the mismatch.

Puppeteer sits in the same category as Playwright: a browser automation framework offering full programmatic control, built for testing, repurposed for scraping.

Either way, once you're running your own headless browsers, you own a list of problems the tutorial never mentions. Proxy management and IP rotation don't come bundled with either tool, so that's a separate system you now have to build and maintain. Anti-bot defenses, CAPTCHA solving, retry logic: all of that falls on you too. Wait too short and you scrape half a page. Wait too long and you're burning compute for nothing. Puppeteer, being Chrome/Chromium-focused, familiar to developers, and tightly integrated with DevTools, is the older and narrower of the two. Memory usage is in that same 200-500MB band per instance. And the timing gets worse with dynamic imports and deferred JS chunks: extracting after full hydration means writing explicit, fragile waits, and getting the timing wrong either way costs you.

Layer 2 decisions: when CSS selectors versus LLM extraction earns its place

Once the page is fully rendered, the question is how you pull the data out. CSS and XPath selectors are cheap, fast, and predictable, and they're the right default whenever the markup holds still and you're processing a lot of pages.

But selectors carry a hidden cost that becomes visible only months later. Sites redesign. Class names get renamed during a framework upgrade. Every one of those events can quietly break a selector without ever throwing an error: it just starts returning nothing, or the wrong thing. String selectors together across dozens of sites and you've built yourself a maintenance job with no end date, one redesign at a time.

LLM-based extraction sidesteps that fragility because of how it actually works. The model reasons about what information sits on the page, not where it happens to live in the markup. A price is a price whether it's in a <span>, a <div class="pricing-block">, or some JS-rendered component nobody documented. The model doesn't care about the wrapper. It cares about the number that looks like a price.

That's not just a tidy theory. Research published in Scientific Reports found AI-driven extraction beats rule-based crawlers by 35% on accuracy and 40% on processing efficiency across benchmark datasets https://www.nature.com/articles/s41598-025-25616-x. At scale, on messy or changing sources, the LLM approach wins on both fronts, and if you're building for the long haul rather than a one-off scrape, that's the one to build around.

AI-ready output formats: why raw HTML is the wrong input for LLMs

Feeding raw HTML to an LLM is a bit like mailing someone the entire phone book when they asked for one number. Navigation menus, footer links, cookie banners, ad markup: none of it carries any signal, and all of it costs tokens. Raw HTML runs 5 to 10 times heavier in tokens than the same content converted to clean Markdown. It's a meaningful hit to the API bill, not a rounding error. That's what makes a pipeline scale while another quietly doesn't.

A well-built extraction pipeline should hand back data in four different shapes, depending on what's downstream. Markdown is the sweet spot for feeding an LLM: light like plain text, but still structured enough to keep headings and lists intact. JSON skips straight into a database or warehouse with no extra cleanup, which makes it the right call whenever the output has to slot into a system with a fixed schema. Raw HTML still earns its place when whatever's downstream needs to re-parse or re-render the page, though it comes at that same steep token cost.

The stakes get bigger than any single pipeline when you zoom out. By IDC's forecast, 78% of all stored data is unstructured https://www.olostep.com/blog/web-data-extraction-platform. Turning that mess into something a machine can reason over is the entire job description of an extraction pipeline. So when you're sizing up a scraping tool, don't just ask how often it succeeds. Ask what shape it hands you when it does. A tool that dumps raw HTML on you still leaves you with a full parsing job before any LLM can touch it, success rate notwithstanding. Screenshot and PDF output belong to visual verification and OCR pipelines, not to LLM reasoning.

The managed-API and AI-native tool landscape for JS-rendered extraction

Playwright versus Puppeteer doesn't cover the landscape anymore, and treating it as the whole conversation is dated thinking. By 2026, a full ecosystem of managed APIs and AI-native frameworks handles both layers end to end, each with its own tradeoffs on control, cost, and how much babysitting it needs.

On the managed-API side, the philosophies spread wide. Some vendors run massive proxy networks and lean on raw success-rate numbers: one independent 11-provider benchmark put a leading proxy-network vendor at a 98.44% success rate, backed by a commercial proxy footprint spanning 195 countries https://brightdata.com/blog/web-data/best-web-scraping-apis.

Others price per URL with a lot of transparency. Pass a URL and a plain-English goal, get back clean Markdown, at roughly $1 per 1,000 URLs, with a separate structured-output Task API tier starting around $5 per 1,000 runs and SOC 2 Type 2 certification with zero data retention https://parallel.ai/articles/ai-data-extraction-how-to-extract-structured-data-from-websites-at-scale. Others still focus tightly on rendering itself, handling waits, clicks, and infinite scroll as part of a JavaScript rendering API, or expose a query language built specifically for browser automation, paired with stealth routing and in-session CAPTCHA solving for tougher targets, running on a free tier before usage-based pricing kicks in.

On the open-source side, the tools increasingly assume an LLM sits downstream from the start. One framework was built from day one to feed retrieval pipelines, converting pages straight to clean Markdown with multiple extraction strategies on offer, and it's picked up over 80,000 GitHub stars. Another positions itself as an LLM-driven browser agent, with 78,000-plus stars and an 89.1% success rate on the WebVoyager benchmark, the most-starred AI browser agent on GitHub right now https://www.firecrawl.dev/blog/best-browser-agents. A third scores 85.8% on that same benchmark with around 22,000 stars, and does particularly well on repetitive form-filling at scale: insurance quotes, government paperwork, job applications.

One framework extends browser crawling with natural-language primitives, act, extract, observe, agent, and its latest release folds AI directly into the crawling pipeline to deal with the exact selector-breakage problem covered above, alongside support for async iterators.

None of these tools solve the problem the same way, and that's the point. Pick based on what's actually being optimized for: raw success rate, price predictability, output format, or how much infrastructure a team is willing to own outright, because the two layers this piece started with, fetching the page and extracting structured data from it, haven't gone anywhere. There are just a lot more ways to solve it now. Web Data API, the company behind this blog, runs full JavaScript execution by default on every request rather than as a paid add-on, bundles premium residential IPs and proxy rotation into every request so proxies aren't a separate line item, offers one unified API across search, scrape, crawl, batch, and monitoring, handles a batch of 100,000 pages in roughly 5 to 7 minutes and 1 million requests in roughly 15 minutes, and produces Markdown output that strips boilerplate automatically with native LangChain and LlamaIndex integrations https://www.olostep.com/blog/web-data-extraction-platform. Vercel Agent Browser (vercel-labs/agent-browser) is an open-source headless-browser automation CLI with a Rust core and Node fallback, returning a compact accessibility tree with deterministic element references (like @e1) and JSON output built for LLM parsing. It had roughly 39,800 GitHub stars as of August 2026, and it's the right tool for wiring browser control into AI coding assistants like Claude Code, Cursor, and Codex. Managed cloud APIs. Firecrawl is designed for entire-domain extraction, converting pages to clean Markdown or structured JSON for LLM workflows, achieving 96% coverage on its 1,000-URL benchmark with a P95 latency of 3,387ms on the same benchmark, offering an /interact endpoint that lets agents click, fill forms, and navigate before extracting, and providing a free tier at 1,000 credits/month with paid plans starting around $16–$49/month. Firecrawl gives 1,000 free credits per month with no credit card required https://www.firecrawl.dev/blog/ai-powered-web-scraping-solutions.