Handling Pagination and Infinite Scroll at Scale

Pagination and infinite scroll are two different engineering problems wearing the same UX costume. Numbered pages, "Load More" buttons, and endless scrolling feeds all look similar on screen, but the backend logic behind each one decides whether a scraper works cleanly, breaks quietly, or gets flagged and blocked. Get the pattern wrong, and the fix requires a rewrite.

How to detect which pagination pattern a site is using before writing any code

Before touching a scraping framework, open a browser and just watch. The URL bar tells you almost everything on the first pass: if it changes with a page number or offset parameter as you click through, that's numbered pagination, and it's the easiest of the three problems by a wide margin.

If the URL stays put, open the Network tab and watch what happens on scroll or on a "Load More" click. Look for XHR or fetch requests hitting some internal endpoint and returning JSON. That's the single most useful thing you'll find all day. Check the request parameters too. page, offset, cursor, after, or next_token will tell you exactly which pagination scheme is running under the hood.

Then look at the DOM itself. If new items get appended to an existing container after that network call, that's client-side rendering of API data rather than a server spitting out fresh HTML. That distinction changes the whole approach.

The one case that actually demands full browser automation: no URL change, no visible API call, and content that only shows up after scrolling. That's usually an Intersection Observer running scroll-triggered lazy loading, and there's no shortcut around rendering the page for real.

Here's the trap worth naming directly: a page can look exactly like infinite scroll while quietly running a clean, paginated JSON API behind the scenes. Check the Network tab before assuming a headless browser is needed, full stop. Skipping this step is a common way engineering time gets wasted on pagination work, and it's entirely avoidable.

Handling numbered pagination: URL patterns, parallelization, and where it breaks down

Numbered pagination is a graph traversal problem where every node already has an address. Each page sits at a predictable URL, so instead of guessing where content lives, a scraper just walks a known map.

There are two ways to walk it, and one is clearly better once volume matters. Following "Next" links is reliable but sequential, since page 6's URL doesn't exist until page 5 hands it over. Building URLs straight from a pattern, like ?page=N or &start=N, skips that dependency entirely and lets requests fire in parallel once the total page count is known or estimated. If the site's URL structure allows it, that second method is the only one worth building for real work. Following "Next" links one at a time works when there's no other choice, but shouldn't be treated as a design decision.

Finding the page count usually means parsing the last page number out of the pagination nav, assuming the site bothers to show one. If it doesn't, crawl forward until a page comes back empty or redirects to page 1. Either is a clean stop signal.

Numbered pagination breaks in three specific ways at scale. Offset drift is the sneaky one: if the underlying dataset changes mid-crawl, records shift position between pages, and items get skipped or duplicated with no error message telling anyone it happened. Deep pages also get slow, because offset-based database queries scan and throw away every row before the one they actually want; a deep page costs a lot more than page 4. And sequential, high-speed requests to the same URL template are about as obvious a bot signal as exists, which invites exactly the kind of attention nobody scraping at volume wants.

The fixes cost almost nothing relative to the payoff. Randomize request order when parallelizing, add jitter between calls, and deduplicate by item ID instead of trusting a page number to mean the same thing twice. An empty page, a redirect back to page 1, or a page that repeats the last one's content all mean the same thing. Stop.

Click-to-load and the API interception shortcut that changes the production calculus

The "Load More" button is a courtesy for the user, and underneath it, almost every click-to-load setup is just firing a plain HTTP request and getting JSON back. The button is theater; the API is the show.

The investigation process takes about two minutes. Click "Load More" once with the Network tab open. Find the XHR or fetch call that fires, copy it as a cURL command, and run it directly. If it returns JSON without needing the browser at all, that's the real target. From there, reverse-engineer the pagination parameter (usually page, offset, or cursor) and figure out the auth mechanism: cookie, header, or token.

This matters for one unglamorous reason: speed. A direct HTTP request to a JSON endpoint runs an order of magnitude faster than spinning up a browser render cycle for the same data. The response also arrives already structured, so there's no DOM parsing and no selector to babysit when the site redesigns its front end next quarter. And since nothing's driving a browser, everything parallelizes cleanly once the pagination scheme is known.

Sometimes the door's locked. Token rotation, signed request parameters, or session-bound auth can make hitting the API directly impractical. That's the one legitimate case for falling back to browser automation, the exception rather than the default. The rule holds regardless: check the Network tab first, every time, and treat a headless browser as the last resort, not the reflex.

Infinite scroll: the three viable engineering approaches and when each one applies

Infinite scroll has three real solutions, and picking the wrong one is how a scraper that worked fine in testing grinds to a halt in production.

API interception is still the first thing to check, and it pays off even more here than with click-to-load. The whole design goal of infinite scroll is to hide pagination from the user, not from the network. At volume (hundreds of thousands of records or more) this is the only approach that scales, tools like Olostep, a web scraping and crawling API built for AI teams, handle exactly this layer so developers never touch the underlying browser or proxy infrastructure. Rendering a browser for every scroll increment on that kind of dataset burns compute for no reason.

Scroll-triggered browser automation is the fallback when no clean API exists. Playwright, Puppeteer, and Selenium can all drive a real browser, but the scrolling itself usually needs to happen through injected JavaScript, since none of these tools have great native scroll controls. After each scroll, wait for a network-idle signal or a specific selector to appear, never a fixed timer. Fixed delays are fragile: too short and content gets missed, too long and time gets burned for nothing. Most sites trigger loading through the Intersection Observer API, which fires when the last visible list item enters the viewport, so scrolling to the bottom of the current DOM reliably sets it off. Cap the loop with a hard iteration limit, and check content hashes between scrolls; two identical scrolls in a row means the crawl hit bottom.

Scroll-until-network-idle is a cousin of the second approach: scroll continuously and wait for all network activity to settle, rather than watching for one specific selector. It's more forgiving on sites with unpredictable load times, but slower, since it's waiting on everything instead of just the one request that matters.

One failure mode cuts across all three, and it's worth naming directly. A 200 response and a page "load" event do not mean the content is actually sitting in the DOM. Single-page apps hydrate asynchronously, often well after the page technically finishes loading. Treat SPAs like the async systems they are, and wait for the selector that actually holds the data, not the browser's load event. The failure here is silent: no error, no crash, just an empty extraction that looks successful right up until someone checks the output.

Cursor-based vs. offset pagination: how the backend design shapes what a scraper can do

Offset pagination is simple to reason about and structurally shaky the moment the underlying data moves. Every request scans and discards all the rows before the one it wants, so query cost climbs with depth. Add or remove records mid-crawl and the offset window shifts underneath, silently skipping items or duplicating them with nothing flagging the error. The one real upside is that offset URLs are stable and cacheable, so a CDN can serve frequently requested pages fast without touching the database at all.

Cursor-based pagination solves the drift problem by pointing at a specific record instead of counting to the Nth one. Query cost stays flat regardless of depth, since the cursor works as an anchor, not a running tally. New records added after a cursor was issued don't shift anything that came before it. The tradeoff: getting an accurate total count requires a full-table scan, which is why cursor-paginated sites rarely show something like "results 1 to 20 of some exact total." Most can't, not without a lot of extra work just to print a number nobody strictly needs.

If a target uses offset pagination, don't fight it: crawl fast and deduplicate by item ID, because the data underneath will drift the moment the crawl pauses. Against a cursor API, save the cursor token between runs; resuming a paused crawl later is safe, since the cursor is a fixed marker rather than a moving position. Keyset pagination, the mechanism cursor pagination usually wraps around, is the more dependable target when both client and server sit under the same team's control, and it's worth treating as the default assumption for any modern API. Anyone building new infrastructure and reaching for offset pagination out of habit is choosing the fragile option on purpose.

One hybrid worth knowing: cursor pagination for the actual data feed, paired with a separate, capped count endpoint just for showing users a number on the page. It shows up more and more on large APIs, and it's a reasonable compromise between accuracy and performance.

Treating pagination state as a graph: loop detection and deduplication at production scale

Linear "next, next, next" logic works fine until a site redirects page 40 back to page 1, and suddenly the crawler loops forever with no error explaining why. Personalized feeds recycle overlapping content behind different cursor tokens. A/B tests generate pagination paths that look distinct but land on the same content. None of this is exotic; it's just what real, dynamic websites do on a normal Tuesday.

The fix is to treat pagination as a graph instead of a line. Every pagination state is a node, and every "next" link or cursor is a directed edge between nodes. For numbered pagination, the node key is the canonical URL plus its query parameters. For cursor-based APIs, it's the cursor token combined with whatever filters scope the result set. For infinite scroll, it's a hash of either the request parameters or the content that came back.

Loop detection then gets simple. Before issuing a new request, check if its node key is already in the visited set. If it is, stop. That's depth-first search with cycle detection, the same thing from the textbook, just applied to a crawl path instead of a tree diagram.

Content-hash deduplication adds a second layer for the messier cases, where cursor tokens differ but the content underneath overlaps, which happens constantly on fast-moving feeds. Hash the sorted item IDs on each page, keep a small rolling cache of recent hashes, and if the current page matches one already seen, treat it as a loop and move on.

Crawl-path logic (the graph traversal and loop detection) should live apart from extraction logic (actually parsing the page). They change at different speeds and for different reasons, and bundling them together just guarantees a site redesign breaks both at once. Distributed crawl queues with concurrency limits round this out, running many pagination paths in parallel without a thundering herd hammering the same endpoint at the same second. Built this way, pagination fails loudly: the scraper stops and says so, instead of cycling through the same twenty pages forever without telling anyone.

Anti-bot systems in 2025 and what they actually look for in pagination behavior

IP blocking and User-Agent checks are old news, easy to get around, and rarely what's catching scrapers anymore. Anti-bot systems score session behavior continuously now, watching scroll depth variation, timing between requests, interaction density, and whether a browser's fingerprint holds steady across the whole session.

The real tell is mechanical regularity, not any single bad signal. A real person's scroll pace, read time, and click rate wander around within a range, because people are inconsistent by nature. A scraper's numbers don't wander at all; they're metronomic, and that consistency is the giveaway. A convincing Chrome fingerprint sitting on top of robotic timing doesn't fix anything either, since mismatched signals get flagged just as fast as an outright fake browser would.

A few pagination-specific habits are practically an invitation to get blocked. Requesting pages at a fixed interval, scrolling to the exact same pixel at the bottom of the page every time with zero variation, and pulling every page in strict sequential order at top speed all read as machine behavior to anything watching. Add jitter to timing, and randomize request order wherever parallelization allows; both break the pattern up.

Cloudflare's shift matters here too, and it's a clear sign the ground moved. Cloudflare handles a massive share of global web traffic, and as of mid-2025 it started blocking AI-based scraping by default on newly configured sites, flipping the old assumption that the web stays open unless a site actively locks it down. New sites set up through Cloudflare now block AI crawlers from the start; access has to be granted, not just left unblocked by accident. Part of what drove that shift is the mismatch between how often AI crawlers fetch content and how little traffic they send back in return, a ratio lopsided enough that default-blocking became the industry's practical answer. Anyone still building scrapers on the assumption that the open web stays open by default is planning around a rule that no longer holds.