ETL Pipeline Design for Web Data Extraction

Web data breaks the rules that ETL was built around, and that's the whole story here. This piece walks through what actually changes when your source isn't a clean database table but a website that redesigns itself on a Tuesday without telling anyone. The goal: get a developer from "brittle script that dies every time a site changes" to something closer to production-grade.

Traditional ETL grew up in a world of fixed schemas, batch jobs that run at 2 a.m., and APIs that publish a changelog before they break anything. Web data doesn't play along. HTML structure shifts by site, by page, sometimes by which A/B test bucket a visitor lands in. Content hides behind JavaScript renders, paywalls, and anti-bot systems that would rather serve you a puzzle challenge than your data. And schema drift isn't a rare event to plan around, it's the default state of things. One class name change on a product page can quietly break every downstream field mapping, and nobody notices until a report looks wrong three weeks later.

The stakes here are bigger than they used to be, too. One market report estimates that unstructured data (the HTML, PDFs, and scraped text that don't fit neatly into rows and columns) will make up the large majority of enterprise data by 2026. The bottleneck for most companies is no longer storage or compute. Getting web data in reliably is the bottleneck for most companies now.

The three-stage model applied to web data: what changes at each layer

Extract, transform, load. Same three letters everyone learned in a database class. But apply them to the open web and each stage means something different than it does for a relational database export.

Extraction used to mean "connect to the API, pull the rows." For web data, extraction means negotiating HTTP access, sometimes rendering a full browser session, and getting past whatever anti-bot wall stands between the crawler and the content. The output isn't a clean row, it's raw HTML, a JSON blob, or sometimes a binary file you still have to figure out how to open.

Transformation stops being simple type-casting. Turning a string into a date is the easy part. The hard part is figuring out the structure: identifying which div holds the price, which span holds the date, and confirming that's still true after the site's last redesign. That's structural inference, not conversion.

Loading changes too, because the destination isn't always a data warehouse anymore. It might be a vector store feeding a retrieval system, or a context window for an LLM agent that needs clean text, not a wall of HTML tags. The format requirements at the end of the pipeline reach all the way back and dictate decisions at the very start. More on that later.

Building the extraction layer: access, rendering, and anti-bot realities

Extraction is the hardest part of this whole exercise, full stop. A parser is useless if the thing it receives is a Cloudflare challenge page instead of the product listing. No amount of clever regex fixes a blocked request.

Developers generally reach for a handful of tools, and each comes with a real trade-off:

BeautifulSoup handles HTML and XML parsing well and lightly. It doesn't render JavaScript and doesn't output structured data for an LLM to consume directly. It's also the most-used tool in the space, with 43.5% developer adoption, mostly because it's simple and does one job cleanly. Scrapy scales up. It's built for large, async crawls, handles robots.txt on its own, and gives you storage options out of the box. Adoption sits at 13.0% per the Apify State of Web Scraping 2026, notably lower than BeautifulSoup, and it shares the same blind spot: no native JavaScript rendering. Selenium and other headless browsers solve the problem of pages that require JavaScript rendering by running an actual browser. That fixes rendering but costs latency and compute. Running a full headless browser instance per page is not free, and at scale it adds up fast. Managed scraping APIs remove proxy rotation, rendering, and anti-bot negotiation from your plate. These make sense when reliability affects uptime and success rate more than shaving cents off a per-request cost does.

Anti-bot systems are the elephant in every one of these decisions. Cloudflare, Akamai, PerimeterX, and Datadome sit in front of most high-value targets now, and they're good at their job. The Browser Use Stealth Benchmark, run across 71 websites, found Browser Use hitting an 81% success rate at evading detection, compared to Browserbase's 42%. Two tools built for the same purpose can perform wildly differently in practice, and picking blind means picking a coin flip.

Proxy strategy deserves its own line item, not a footnote. Rotating residential proxies work for distributed collection where evasion reduces detection risk more than speed reduces collection time. Datacenter proxies make sense when throughput and cost outweigh the need to blend in. The design mistake to avoid: hard-coding proxy assignments into the crawler logic itself. Separate that layer out, assign proxy type by workload, and the routing strategy can change without a rebuild every time a target site upgrades its defenses.

Transformation for web data: parsing, normalization, and schema drift

Web content appears in shapes that traditional transformation logic was never built to handle. Raw HTML. Loosely structured JSON-LD tucked into a script tag. Plain text mixed with embedded tables. PDFs sitting behind a URL that redirects twice before you get the file.

Schema drift sits at the center of all of it. A site redesign, a CMS migration, a new A/B test variant, any one of these can quietly reshape the DOM and break a field mapping without throwing a single error. Pipelines that survive this are built assuming schema drift will happen, on a schedule nobody controls but the target website. They're built assuming it will happen, on a schedule nobody controls but the target website.

There are two broad approaches to handling this, and they trade off against each other directly. CSS selectors and path-based extraction are precise and fast, but they snap the moment the underlying structure shifts. Fine for internal sources you control. Fragile for anything on the public web where redesigns happen without warning.

LLM-powered extraction takes a different bet: infer what a field means from context, rather than where it sits in the DOM. That holds up better against layout changes, but it introduces non-determinism (the same page might get parsed slightly differently run to run) and adds latency you don't get with a simple selector match. One pipeline design analysis frames the strongest production setups as a blend: deterministic ETL controls doing the heavy lifting, with agentic reasoning layered in for the parts that need to bend without breaking.

Then there's normalization, which sounds boring until you're staring at four different date formats, three currency symbols, and a company name that shows up under five slightly different spellings across the sites you're pulling from. Web data doesn't just need parsing. It needs cleanup that assumes every site made its own private decisions about formatting, because they did.

Loading web data: destination design and format decisions that affect everything upstream

The load stage isn't the finish line, it's the design constraint that should have shaped everything before it. Decide what the destination needs to look like, and the transformation and extraction layers fall into place behind it, not the other way around.

Good storage design usually needs to serve more than one master at once:

• Audit and compliance work needs raw source records with provenance metadata attached, so someone can trace a number back to the page it came from.
• Reprocessing needs the ability to re-run transformation logic from the raw data without re-scraping the entire web all over again.
• Analytics needs a queryable structured format, whether that's a SQL warehouse or a data lake.
• Downstream automation needs clean, structured output that an LLM or an agent can consume directly, without a separate cleanup pass first.

The ELT trade-off becomes visible in practice at this point. When the warehouse on the other end is powerful and cheap to query, loading raw data first and transforming it in place, rather than transforming before load, buys flexibility. That matters especially early on, when the schema of a new web source is still being figured out and locking in a transformation too early means redoing it later anyway.

Being honest about how hard this stage actually is to automate matters. ELT-Bench tested 100 pipelines against 835 source tables and 203 data models, and the results were humbling. Even the best-performing agent in the benchmark, Spider-Agent running Claude-3.7-Sonnet with extended thinking, only correctly generated 3.9% of the data models, at an average cost of $4.30 and 89.3 steps per attempt. That's not a knock on the tooling, it's a signal. Automating the load and transform stage end-to-end remains unsolved, and pipeline design should account for that reality rather than assume some agent will handle it flawlessly next quarter.

Where Conventional Pipelines Break

Schema drift usually shows up as silently wrong data rather than an error message. That's the trap: the pipeline keeps running, keeps loading data, and the data is just quietly wrong. The pipeline keeps running, keeps loading data, and the data is just quietly wrong. Nobody gets paged for a null value that looks like it belongs there.

Conventional ETL setups tend to be brittle when a schema change hits, slow to recover once someone notices, and limited in how far they scale before the cracks show. One paper (published in Springer) reporting on agent-based pipeline implementations found a 76% improvement in pipeline breakage incidents, an 83% faster recovery time, and a 50% reduction in ETL processing time compared to conventional approaches. Those numbers mark the real gap between a legacy setup and a modern one, and it's a wide gap.

Four failure patterns occur repeatedly, and each one is worth designing against explicitly rather than discovering the hard way:

• A CSS selector stops matching. Extraction returns an empty string, the pipeline doesn't stop, and null values load straight into production.
• An anti-bot system returns a 200 status code with a challenge page instead of real content. The parser dutifully extracts garbage. No error gets raised, because as far as the HTTP layer is concerned, everything worked.
• A site serves cached or stale content instead of the current page. Freshness assumptions break without any signal that they did.
• JavaScript rendering finishes incomplete. The DOM gets scraped halfway through, and fields go missing several steps downstream, far from where the actual problem happened.

Monitoring web pipelines continuously: detecting changes before they become data quality incidents

Websites change without asking permission first. A monitoring layer that treats the web as a constant stream of small events, rather than a source that behaves the same way every time it's queried, is the difference between finding out a pipeline broke from an angry downstream user and catching it before the bad data ever leaves the building.

Monitoring needs to happen at every stage, not just at the end:

• At extraction: success rate, response time distribution, HTTP status codes, and content length compared against a baseline.
• At transformation: null rate broken out by field, type mismatch rate, and record counts checked against an expected range.
• At load: row counts, duplicate detection, and schema conformance checks at the destination.

Some pipelines shouldn't run on a timer. A competitor's price change, a new regulatory filing, a product page edit, these are events, and a pipeline built around a fixed cron job will always be a step behind them. The better pattern flips the relationship: change-detection monitoring watches the source and triggers extraction when something actually moves, instead of extraction running blind every six hours whether anything changed or not.

Coordinating scraping, crawling, and monitoring as three separate tools stitched together tends to create more overhead than it saves. Infrastructure that treats them as one connected system, where a crawl API can also fire a webhook the moment it detects a change, closes the loop between "something happened" and "the pipeline knows about it" without a human checking a dashboard in between.

Agentic ETL: how autonomous agents change pipeline design without replacing the architect

Agentic ETL means agents take on the parts of the pipeline that benefit from judgment. It means agents take on the parts of the pipeline that benefit from judgment, like inferring what a field means when the DOM shifts under it, while deterministic logic still handles the parts that need to behave the same way every single time.

That split matters because agents introduce something traditional ETL never had to budget for: non-determinism. An agent might parse the same page two different ways on two different runs, and the ELT-Bench numbers from earlier (3.9% of data models generated correctly, at a real dollar cost per pipeline) make clear this isn't a solved problem waiting to be adopted wholesale. Cost and reliability both need watching, not assuming.

The architect's job doesn't disappear here, it changes shape. Someone still has to decide which parts of the pipeline get deterministic rules and which parts get agentic reasoning, still has to set the monitoring that catches an agent's mistakes, and still has to design the storage layer that makes reprocessing possible when an agent gets something wrong. Autonomy in one layer raises the bar on discipline in every layer around it. That's not a caveat tacked onto agentic ETL, it's the actual design principle.