Headless Browser Scraping with Playwright and Puppeteer

Most modern websites build their pages with JavaScript after the initial load, so a plain HTTP request gets back a blank shell or a spinning loading icon instead of the data anyone actually wants. Picture a product listing page on a typical online store. The prices load after React finishes its work, the product grid fills in through a separate AJAX call, and scrolling down swaps out pagination for infinite scroll that fetches more items behind the scenes. That content is generated after the initial load and is absent from the raw HTML a server first sends over. A tool like BeautifulSoup or Cheerio, built to read static HTML, doesn't throw an error on a page like this. It just comes back empty, because there's nothing in that first HTML response to parse. Most websites now need their JavaScript to actually run before the data a scraper wants even exists. A headless browser fixes this by running a real browser engine that executes JavaScript the same way a human visitor's browser would, so by the time a script goes looking for content, that content is already sitting there, rendered and ready.

How Playwright and Puppeteer differ under the hood

Playwright and Puppeteer are built to solve that exact problem, but they get there through different engineering choices, and those choices shape which jobs each tool handles best. Puppeteer came first, released by Google in 2017 as a Node.js library for driving headless Chrome through the Chrome DevTools Protocol, known as CDP. That was a clear departure from the older WebDriver protocol that Selenium relies on, and Puppeteer quickly became the standard way developers automated a browser. Playwright came later, built by Microsoft, with a team that included some of the original people behind Puppeteer. It builds on the same CDP foundation for Chromium, but for Firefox and WebKit it runs patched versions of those browsers with their own low-level protocols, skipping WebDriver entirely.

Browser coverage is the biggest practical difference. Playwright runs Chromium, Firefox, and WebKit out of the box, while Puppeteer supports Chromium and Firefox (with Firefox support production-ready via WebDriver BiDi, but no WebKit support). Playwright also builds in what's called actionability checks: before it clicks or reads an element, it confirms that element is attached to the DOM, visible, stable and not mid-animation, able to receive events, and enabled. Scrapers using Playwright don't need the old sleep(3000) hack. Playwright also reaches into the Shadow DOM by default, so data tucked inside custom web components is reachable without any extra setup. On the language side, Playwright gives developers both a synchronous and an asynchronous client in Python, plus support for JavaScript, TypeScript, Java, and.NET, while Puppeteer sticks to JavaScript and TypeScript, async only. One detail with consequences down the road: Puppeteer talks to the browser over CDP and, in its default setup, fires off a specific command called Runtime.enable inside the main browser context. That single command turns out to matter a lot once anti-bot systems enter the picture, covered later in this guide.

Choosing a scraping tool in 2026

For a brand-new scraping project, Playwright is the clear default choice, with Puppeteer reserved for a short list of specific situations. The multi-browser support described above isn't just a technical footnote. Switching between Chromium, Firefox, and WebKit makes a scraper harder to fingerprint, which is a real operational advantage once a target site starts watching for bots. Playwright's selector system also holds up better over time: text selectors, role selectors, and chained locators track what's actually on the page more reliably than the querySelector approach Puppeteer leans on, so a scraper breaks less often when a site tweaks its layout. Playwright also separates watching network traffic from controlling it. The page.route() method can intercept, change, or block any request, while page.on() can only watch requests go by without touching them, a distinction that matters a lot once request-filtering logic enters the picture.

Puppeteer still earns its keep in a few specific spots. A team maintaining a legacy codebase that already runs reliably has little reason to tear it apart just to switch tools. Simple Chrome-only jobs, like taking screenshots or generating PDFs, don't benefit from multi-browser support at all, so Puppeteer handles them just fine. And if an existing Node.js project already works and switching would cost real engineering time, there's no reason to rush a migration. None of this is urgent. An existing Puppeteer codebase that works doesn't need to change overnight. Treat a move to Playwright as something to plan for later rather than a fire to put out today.

Core Playwright patterns for scraping dynamic pages

A handful of Playwright patterns cover most of what a scraper runs into, and knowing when to reach for which pattern affects how much of the API a developer actually needs to memorize. This walkthrough targets Playwright 1.62.0, released July 24, 2026, and needs Python 3.10 or newer.

A basic scraper follows the same shape every time: launch a browser, open a context inside it, then open a page inside that context. The context is where settings like viewport size, locale, and saved login state live. A minimal version in Python looks like this, using the async client, a realistic browser identity, and an explicit wait instead of a blind pause:

from playwright.async_api import async_playwright

async def scrape(url):
 async with async_playwright() as p:
 browser = await p.chromium.launch()
 context = await browser.new_context(
 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
 )
 page = await context.new_page()
 await page.goto(url, wait_until="domcontentloaded")
 await page.wait_for_selector(".product-price")
 prices = await page.locator(".product-price").all_text_contents()
 await browser.close()
 return prices

A few habits make patterns like this hold up in practice. Always set a realistic user_agent string. A headless browser running with Playwright's default identity is simple to spot and simple to block. Wait for a specific selector to show up rather than pausing for a fixed number of seconds. wait_for_selector() adjusts to how the page is actually loading, while a fixed timeout is really only good for debugging. When a page loads its content directly into the DOM rather than fetching it afterward, setting wait_until="domcontentloaded" finishes faster than waiting for full network idle and still catches everything needed.

Some sites need more than a straightforward page load. Infinite scroll pages can be handled by running window.scrollTo(0, document.body.scrollHeight) in a loop, with a short delay between each pass so new content has time to load. For sites that pull in data through background API calls, page.expect_response() can grab the JSON a page fetches behind the scenes, which is often more stable than trying to parse the rendered HTML after the fact. A lot of sites with no public API still quietly make internal REST calls to populate their own pages, and catching those calls directly skips the HTML-parsing step entirely. One useful habit while building any of this out: test selectors live, using the synchronous client inside an interactive Python session, since the browser window stays open between commands and lets a developer try a selector, see what comes back, and adjust on the spot.

A few common errors trip people up early on. ModuleNotFoundError: No module named 'playwright' means the package isn't installed where the script is actually running. ModuleNotFoundError: No module named 'playwright.sync_api' usually means a file named playwright.py is sitting in the working folder and blocking the real package, which gets fixed by renaming that file. BrowserType.launch: Executable doesn't exist means the browser binary itself hasn't been downloaded yet, fixed by running playwright install chromium.

Scaling headless browsers without running out of memory or money

Running a headless browser costs real compute per page, and most of that cost can be designed away before anyone needs to reach for more servers. Before writing a single line of Playwright code, check whether a headless browser is even necessary. Opening DevTools' Network tab while a page loads often reveals XHR or Fetch requests quietly returning JSON, since a lot of sites that look dynamic are really just React front ends sitting on top of a plain REST API that can be called directly, no browser required. Practitioners in 2026 generally follow a pipeline in order: try the official API or raw XHR JSON first, then check for embedded JSON in the page, then bring in Playwright only once JavaScript execution, login, or real interaction is unavoidable, and reserve LLM extraction for small, manually reviewed cleanup jobs at the end.

Once a browser is genuinely needed, reusing it efficiently cuts compute cost more than adding more servers does. One browser instance can host many separate contexts, each one an isolated session with its own cookies and cache, running on top of the same process, which spreads out the cost of starting up a browser across many jobs. At larger scale, that turns into a browser pool: a set of browser instances that workers check out, use for a job, and return, which keeps memory use predictable instead of letting it spike unpredictably. Blocking unnecessary resources like images, fonts, stylesheets, and video cuts page load time by a lot without touching the actual data being extracted.

The bigger decision is whether to run this infrastructure in-house or hand it to someone else. At high volume, self-hosted headless browsers tend to keep compute costs roughly flat, while paying per request through a managed service adds up fast as volume grows, which can make self-hosting the cheaper option for workloads that are large and predictable. Managed or cloud-based browser services, on the other hand, take the invisible maintenance work off a team's plate: proxy setup, keeping browser versions current, and scaling logic under load raise engineering time even when they don't appear on an invoice. A lot of real-world operations end up running a mix of both: local headless browsers for routine jobs, and managed infrastructure reserved for the small number of heavily protected targets that would otherwise eat up a disproportionate amount of in-house engineering effort to keep working. Whichever path a team picks, proxies tend to be the single biggest cost on the bill either way. Residential proxies charge meaningfully per gigabyte, and that cost is billed whether a team self-hosts Playwright for free or pays for a managed browser on top of it.

Detecting headless browsers

Anti-bot detection has moved well past checking the User-Agent header, and now looks at protocol-level signals and behavior patterns instead. That's why patching the surface with a stealth plugin only goes so far. The first signal is a protocol-level one. Both Playwright and Puppeteer talk to the browser over CDP, and in their default setup that specific Runtime.enable command fires inside the main browser context. Systems built by companies like Cloudflare and DataDome specifically watch for that command as a sign of automation. The second signal sits right in the browser itself: headless Chromium sets a property called navigator.webdriver to true, and that single flag is probably the most commonly checked bot indicator across the entire web.

A small ecosystem of stealth tools tries to patch these signals. puppeteer-extra-plugin-stealth exists for Puppeteer, and Patchright and Camoufox serve the same role for Playwright. None of them beat CDP-level detection from a system built by a company operating at Cloudflare's scale. They clean up shallow JavaScript leaks but don't touch the deeper architectural signals these systems are built to catch. That's a ceiling to know about going in: not a reason to panic, but a reason to calibrate which sites are realistic to scrape with self-managed tools alone.

What actually moves the needle looks less like a clever patch and more like basic operational hygiene. Residential proxies matter a lot, since datacenter IP addresses get flagged by default on an increasing number of sites, while residential IPs look like ordinary people browsing the internet. Behavior that looks human helps too: random delays between actions, generally somewhere between two and five seconds or more between page loads, along with natural scrolling and mouse movement that doesn't look robotic. Rotating browser profiles (varying viewport size, timezone, language, and other settings between sessions) also helps, since a flood of requests all sharing identical configuration is itself a red flag. And switching between Chromium, Firefox, and WebKit, which Playwright supports natively, makes fingerprint-based detection meaningfully harder, one of the more concrete payoffs of Playwright's multi-browser design.

A short list of things reliably no longer work. Changing only the User-Agent string fools almost nobody anymore. Simple IP rotation using datacenter proxies gets caught quickly. And running plain, unpatched headless Chrome is often enough on its own to get blocked, since that navigator.webdriver flag alone triggers defenses on a lot of sites.

Playwright as the browser layer for AI agents

Scraping and AI agents are converging on the same piece of software for the same reason: an agent that's supposed to act like a person on the internet needs a real browser underneath it instead of a parser working off static HTML. Everything covered above (rendering JavaScript, waiting for real content instead of a fixed timer, checking that an element is actually visible and clickable before touching it, reaching into Shadow DOM components) is what an autonomous agent needs when its job is to click buttons, fill out forms, and read whatever loads back, instead of lifting a predictable product price off a page it already knows the shape of.

Playwright's actionability checks matter more in this setting, not less. A scraper built for one specific page can be tuned by hand until it works. An agent navigating a page it has never seen before has to rely on the browser itself to confirm that a button is visible and ready to be clicked, which is the job Playwright's built-in checks already do. The same goes for network interception: an agent that can watch or redirect requests through page.route() gets a cleaner view of what a page is actually doing behind the scenes, rather than only seeing what ends up rendered on screen.

Anti-bot systems don't know or care whether the thing clicking through a page is a human-written scraper or a language model deciding its next move. The detection surface, from Runtime.enable to the navigator.webdriver flag, treats both exactly the same. That makes everything in this guide, from picking the right tool to understanding where stealth plugins stop working, just as relevant to a team building an agent as it is to a team building a straightforward scraper.