Handling Pagination and Infinite Scroll at Scale

A scraper that returns only part of the rows it requested by walking offset and limit parameters is looking at the wrong problem. The code runs fine, the parser extracts what it finds, and the output looks like a tidy, complete file. The actual issue sits one layer up: the page loads its first batch of data in the initial HTML, then quietly fetches everything else through follow-up XHR or Fetch requests that a single HTTP request never triggers, so what looks like a finished dataset is just the first slice of one. That is not a parsing failure. The parser did its job perfectly on incomplete input, which is the whole problem.

This is why the practitioner consensus for 2026 puts classification before code. Open the DevTools Network tab, identify which of the known pagination patterns the site is actually running, and only then decide what tool to build. The pattern determines the tool, not the other way around. Skipping that step risks the worst kind of bug: one that produces a plausible, wrong answer instead of an obvious crash. A cursor-based endpoint mistaken for offset pagination doesn't fail loudly, it just quietly produces gaps and duplicates that only show up when someone cross-references the data against a second source, by which point the crawl has already run, billed, and shipped.

The five pagination patterns

Every paginated, load-more, or infinite-scroll page on the web runs on one of five patterns, and the Network tab will tell you which one in under a minute, before a single line of scraping code gets written.

Numbered or URL pagination is the easiest to spot: the page number lives right in the URL, something like /reviews?page=3, and each page loads as its own self-contained document. Offset and limit pagination looks similar but works on a sliding window instead of a page count, with a request like ?offset=40&limit=20 asking for a specific slice of rows, and walking the list means incrementing that offset each time. Cursor-based pagination hands back an opaque token pointing at the next batch of results, one you cannot predict or construct yourself; the only way to get the next page is to read that token out of the previous response. A "Load More" button looks like a distinct pattern on the front end, but the request it fires is usually identical to the one infinite scroll fires, with the click simply standing in for the scroll event as the trigger. And infinite scroll itself fires that same kind of request automatically as the user nears the bottom of the page, with most modern implementations running cursor pagination as the scroll behavior.

Telling these apart takes about sixty seconds. Disable JavaScript and reload the page first, to see what the server renders without any help. Then re-enable JavaScript, open the Network tab, filter to Fetch and XHR requests, clear the log, and trigger one scroll or one click. Read the URL of whatever request shows up: a page or offset parameter indicates numbered or offset pagination, while a cursor, after, or next parameter indicates cursor-based pagination.

The right response to most infinite scroll: an HTTP request, not a browser

Once the pattern is identified, the next decision is the tool, and for infinite scroll specifically, the correct first move is almost never a browser. Reverse-engineering the hidden API request the page is already making lets a scraper call that endpoint directly, skipping the rendering step entirely, which makes it faster, cheaper, and less prone to breaking than running a full browser session.

Infinite scroll is a scroll listener or an IntersectionObserver that waits for the user to approach the bottom of the page, then fires an XHR call with parameters like offset=20, page=3, or a base64-encoded cursor string. Once that endpoint is identified, the scroll event itself stops mattering. The practitioner rule of thumb for 2026 holds that the vast majority of infinite scroll and load-more sites have a clean JSON API, and scraping that API directly is significantly faster and more reliable than browser automation.

One detail trips people up during this reverse-engineering step: plenty of endpoints check a Referer header before returning data, and the only way to spot that requirement is by reading the request details in the Network tab and copying the header over, not bolting it on later as an afterthought. None of this retires the headless browser from the toolkit. It just demotes it to the fallback role, reserved for cases where the endpoint is authenticated, signed, or otherwise impossible to call outside a full browser session.

Cursor pagination requires an opaque token

Cursor-based pagination has become the dominant pattern behind most modern APIs, and its one defining rule is that the token is opaque. It cannot be built from outside the response, which makes it more reliable than offset pagination and considerably less forgiving if a scraper gets the loop logic wrong.

Fetch the first URL, pull the cursor value out of that response, pass it into the next request, and repeat until the cursor field comes back empty or missing entirely. That is a read-then-pass pattern. It is not an increment pattern, and trying to treat it like one (guessing at the next token, or constructing it from some predictable formula) is the exact mistake that turns a clean crawl into a dataset full of silent gaps. Modern APIs behind social feeds and e-commerce infinite scroll pages return that next cursor directly in the response body, so the only correct move is to read it from there every time, never to construct it from URL math.

The opacity is the point, not an inconvenience to work around. Offset pagination has a structural weakness: if a row gets inserted into the dataset while a crawl is mid-run, every page after that insertion shifts, and the crawler ends up with duplicates or missing items it has no way of detecting on its own. A cursor sidesteps that problem entirely, because it always points at a consistent position in the dataset regardless of what else is being written to it at the same time.

Cursor pagination is sequential by nature, which comes with a tradeoff. Page N cannot be fetched without first completing page N-1, so there is no way to parallelize requests inside a single cursor chain. The only safe way to add parallelism is to launch multiple independent chains from separate starting points, a point that matters more once the crawl needs to move beyond a single list and into genuine scale (more on that shortly). One more piece belongs in every cursor loop from day one: deduplication by a stable canonical key, whether that is an item ID or a canonical URL, because any crash-and-restart will overlap with the last successfully completed page and start handing back items the crawler already has.

Stop conditions are not optional: how an uncapped loop becomes a broken dataset

Knowing how to fetch the next page solves only half the problem. Knowing when to stop matters just as much, and this is where a lot of otherwise well-built scrapers quietly go wrong. An uncapped loop either runs forever against a page that never actually ends, or it stops too early the first time it hits a transient error, and either failure produces a dataset that looks complete on the surface but is wrong underneath.

The most reliable stop signal comes straight from the API itself: a field like hasNextPage = false or next_cursor = null in the response body is as authoritative as a stop condition gets. Close behind that is an empty result set, where the response structure is intact but the array of items inside it has nothing in it. A non-200 response, or a response that looks fine on the surface but is actually a soft block, deserves special attention here: a tiny HTML body missing the selectors the scraper expects, or a page that reads like a "please verify you're human" template, is not a real end of the dataset and should trigger a retry, not get accepted as a legitimate stop signal. Mistaking a soft block for the end of the data is one of the most common ways a crawl quietly truncates itself, and it is almost invisible until someone checks the row count against what the site actually shows. A fourth signal is cycle detection, catching a repeated cursor value or a repeated content hash before it traps the loop in an endless circle, which matters most on sites with circular link structures that would otherwise loop forever without anyone noticing. A hard cap on total items or iterations acts as a simple backstop, so that even if every other check fails silently, the loop still has to stop eventually.

Modeling the whole crawl as a graph rather than a straight line helps make cycle detection concrete: computing a hash from the sorted item IDs on each page, and keeping a small cache of recent hashes to compare against, catches a repeating loop before it traps the crawler in it. For infinite scroll handled through a headless browser specifically, the stop condition looks a little different: scroll toward the bottom, wait for new rows to load, compare the item count before and after the scroll, and stop once that count holds steady across a scroll attempt, paired with a max-scroll ceiling so a page stuck on a loading spinner cannot trap the loop indefinitely. None of this is a performance nicety. A missing selector after an error is not proof the dataset has ended, and treating it as proof is how a completely functional-looking scrape ships with a chunk of the dataset simply missing.

Virtualized lists and ML-driven anti-bot systems: the two complications that force a browser anyway

Two separate complications can break the API-first approach described above, and they call for genuinely different fixes. One changes what a browser session can even see. The other changes whether any approach, API or browser, stays unblocked long enough to matter.

The first is virtualized lists. Plenty of modern apps built on React, Vue, or Next.js use virtualization libraries like react-window or @tanstack/virtual to render only the DOM nodes currently visible in the viewport, discarding the rest. That means even a fully loaded, fully rendered browser session might never expose the complete dataset at any single moment, because the DOM itself only ever contains a moving window of it. A naive scroll-and-parse script that assumes the DOM keeps growing as it scrolls will quietly miss rows that scrolled out of the rendered window and got discarded, which is a subtler failure than a plain network error because nothing throws an exception. The page just silently forgets what it already showed you.

The second complication is ML-driven anti-bot detection, and it is a different kind of problem entirely. These systems collect a long list of behavioral signals per session, things like mouse movement patterns, scroll velocity, typing cadence, click coordinates, and the order in which page elements get engaged, then run that behavioral fingerprint through machine learning models trained to tell human visitors from automated ones. At meaningful scale, a sizable share of commercial targets will run some version of this detection. Behavioral fingerprinting is something any high-volume crawl has to plan for. The response here is not to chase increasingly convincing human-behavior simulation inside a self-managed browser. The more durable fix is routing traffic through infrastructure built to handle session management, proxy rotation, and rendering as its core job, so the pagination logic described earlier in this piece stays untouched while the fetch layer underneath it does the harder work of staying unblocked.

Building a resumable, parallelizable crawl for paginated data at volume

A pagination loop that works cleanly for one list, one chain, one afternoon's worth of testing, does not automatically survive contact with volume, where three things that felt optional at small scale become structural requirements: resumability, parallelism, and deduplication.

Resumability means checkpointing the last page number or cursor value a run successfully completed, so that if the process dies halfway through, it restarts from that checkpoint instead of from page one. Pairing that checkpoint with a stable item ID lets the crawler drop the handful of duplicate rows that naturally overlap right at the restart boundary. This matters more for cursor chains than for numbered pagination, because there is no way to reconstruct the cursor value for page 400 out of thin air if that checkpoint gets lost. Losing it means starting the whole chain over from the beginning, with no shortcut available.

Parallelism works differently depending on the pattern. Because a cursor chain is sequential by definition, the only safe way to run work in parallel is to launch separate chains from separate independent starting points, rather than trying to split a single chain across concurrent requests, which just produces race conditions on the token. Deduplication must run across the entire crawl rather than page by page: a set of seen canonical keys, whether item IDs or canonical URLs, catches overlaps that happen across the whole run rather than within one response. Offset pagination is particularly exposed to this problem: any change to the underlying dataset mid-crawl can shift what shows up on a given offset and produce duplicates that a page-level check would never catch.

Rate limiting belongs in this list too, and not as a politeness gesture. Spacing requests out with randomized delays keeps the crawl predictable under anti-bot systems and avoids tripping rate limits that would otherwise leave an unnoticed gap in the dataset right where the block happened. A production-grade pagination loop, pulling all of this together, combines a hard cap on total pages, a seen-keys set for deduplication, a soft-block detector that raises an error rather than quietly accepting a block page as the end of data, and a per-page delay with some randomization built in.

The cost split between managed APIs and self-hosted browsers

Everything up to this point has been about building the thing correctly. What's left is deciding who runs it, an economic decision rather than a philosophical one. It turns on request volume, how complex the pagination pattern is, and what engineering time actually costs.

A 2026 analysis puts the crossover point around 500K–1M requests per month: below it, the time savings of a managed API outweigh the per-request cost premium, and above it, dedicated infrastructure starts making economic sense for stable, predictable patterns. Below that volume, the time saved by using a managed scraping API tends to outweigh its per-request premium, since someone still has to build, test, and maintain the equivalent infrastructure in-house for comparatively little benefit. Above that volume, for patterns that are stable and predictable, dedicated self-hosted infrastructure starts to make economic sense, because the fixed cost of building it gets spread across enough requests to justify the engineering time it took.

JavaScript rendering and premium proxy routing to bypass bot protections are not free in either model: on managed platforms they carry significant per-page cost multipliers versus plain HTTP requests, while in self-managed infrastructure they translate to RAM, CPU, and proxy spend that scales with volume. The "but the code is free" argument for self-hosting misses half the ledger: the code might cost nothing to download, but the compute, the proxies, and the maintenance hours it takes to keep a self-hosted crawler correctly classified, correctly looped, and correctly stopped are the real price tag, whichever side of that 500K-to-1M line a given project happens to sit on.