Wiring Page-Change Detectors to Webhook Endpoints Inside a Scraping Pipeline

An event-driven web scraping API with webhook support separates watching a page from extracting it. A change detector sends an HTTP event to a receiver, which triggers re-fetching, processing, and downstream action only when warranted. Wiring these parts together replaces wasteful application polling with a pipeline that acts on changes without waiting for a human.

Why polling loops are the wrong default for page-change pipelines

The classic polling cycle starts with cron, fetches a page, compares it with the previous version, and perhaps fires an alert. The bill arrives either way. A page that changes once a week but is fetched hourly returns the same results most of the time. PageCrawl's cost comparison illustrates how hourly checks through a scraping API accumulate credits month after month, even when the returned data stays unchanged.

Comparing pages also takes work to keep the comparison rules up to date. Timestamps, session IDs, ad placements, and CSRF tokens can make raw pages differ without changing the content anyone needs. A naive diff treats a rotating ad like breaking news, so the application has to keep rules that sort useful edits from background noise.

Even a correct diff can leave the job unfinished. Someone still has to read it and decide whether to act. Polling fetches pages needlessly, takes time to find changes, and may leave the reader with homework when an alert arrives. What does the correct architecture look like?

How event-driven architecture inverts the loop

The monitoring service watches the page, and the application waits for a useful event. A webhook is an HTTP POST sent automatically when that event occurs. The service initiates the next step. This inverts the application’s loop without pretending that the underlying page no longer needs scheduled checks.

PageCrawl's webhook guide draws the timing distinction clearly. A webhook fires on the check that detects a change, rather than waiting for the application’s next poll interval. The check schedule still governs discovery speed. Push delivery removes the extra wait after discovery.

That event becomes the starting point for work. It can trigger a re-fetch, re-extraction, re-embedding, or escalation without a person handling the initial handoff. Monitoring is therefore not an alerting accessory attached to a scraper. It decides when the scraper and everything downstream have a reason to run.

What change detection actually measures before a webhook fires

A webhook is only as useful as its trigger. The first detection layer normalizes page content and compares its hash with a stored hash. Verid's change detection API hashes visible text after removing scripts, styles, and comments. With that method, visible timestamps, rotating banners, and live counters still count as edits. Cleaned text is not automatically meaningful text.

Meter describes a different approach using a content hash, structural signature, and semantic similarity, with ads, timestamps, and layout shifts excluded. More granular detection then moves from whole-page comparison to extracted fields and predicates. The event can report previous values, current values, and which fields changed. Verid separates its full-page method from an AI prompt method that evaluates the page against a defined rule.

The final layer classifies the detected change. PageCrawl includes ai_summary, ai_priority_score from 0–100, and markdown_difference, allowing routing without another LLM call. Some monitors also evaluate a custom prompt into an important flag, logging lesser changes without escalation. Hash-only detection leaves classification to the consumer, while richer detection moves that work upstream. The payload is where detection quality becomes pipeline capability.

Configuring a change monitor — from URL to scheduled webhook

For implementation, the setup starts with the monitoring boundary. PageCrawl's agent tutorial describes creating a monitor by submitting a URL, a tracking mode such as tracking_mode: "content_only", and a frequency in minutes. Other documented modes cover specific elements, prices, visual changes, feeds, and structured data. The chosen scope determines what can wake the pipeline. Watching everything is not the same as watching the right thing.

Frequency sets the discovery window. The guide gives checks as often as every two minutes on paid plans and hourly on the free plan. Even a fast receiver has to wait for the next check to find a change. The checks must run as often as the pipeline needs to learn about edits.

The next POST configures the target webhook URL, selected events such as change_detected and monitor errors, and payload_fields. The tutorial selects title, contents, markdown_difference, ai_summary, ai_priority_score, and page. Monitors can be scoped per page or workspace tag, letting different groups reach different endpoints. Meter's SDK uses another setup sequence, generating a strategy from a plain-English description, creating a job, and then creating a schedule with a webhook_url.

Reading the webhook payload — what each field means for downstream routing

PageCrawl's payload reference makes the check id the deduplication key. The nested page object identifies the source through its id, name, url, slug, and monitor link. Its URL supplies the target for a re-fetch. The ISO 8601 changed_at timestamp works with the check ID to support replay ordering.

short_summary gives people a brief account, while ai_summary gives an LLM-generated description for alerts or audit logs. ai_priority_score shows how important the change is. markdown_difference gives the text diff to analyze or use to choose chunks, while difference and human_difference give the percentage changed and its readable equivalent.

page_screenshot_image gives a signed screenshot URL, and contents gives the current content, which may remove the need for another scrape. A status of ok describes a healthy monitor, while event_type: change_detected identifies a change and errors need a separate route.

Requesting unused content adds weight, but omitting routing fields forces another call. Meter's added, updated, and removed item arrays let the consumer decide what to do with each row. With the required context in the payload, the consumer can make its routing decisions without calling anything else.

Building the webhook receiver — three endpoint patterns and when to use each

An automation platform is the shortest route to existing SaaS tools. n8n, Zapier, Make, and Pipedream provide webhook triggers whose URLs become the monitor’s delivery targets. No server needs to be maintained. PageCrawl's guide illustrates a pricing workflow that extracts a new price, updates a Google Sheet or database, sends a Slack alert below a threshold, and creates a review task.

A custom server fits more involved internal work. A Flask or Express route accepts the POST, reads its JSON body, and hands the event to a processing function. Across languages, this pattern gives the pipeline room to route on fields, call internal services, and write to a database with custom logic.

A serverless function fits lightweight forwarding without a continuously running server. AWS Lambda, Google Cloud Functions, and Cloudflare Workers can receive events and forward them to Slack or a queue. The guide includes a Worker forwarding example. The practical choice is automation for SaaS integration, a custom server for complex internal routing, and serverless for lightweight forwarding at scale. RAG workflows usually need the custom or serverless path because embedding, vector-store updates, and chunk tracking go beyond a simple notification handoff.

Routing change events to the right downstream action

The primary routing hinge is ai_priority_score. High-score events can go to immediate automated action, while low-score events enter a review queue or are logged without action. Importance is only the first branch. A handler with page_type and change_scope context can then send documentation changes to re-ingestion and policy changes to human review.

For a RAG pipeline, markdown_difference identifies which chunks need attention. Re-embedding only those chunks avoids processing the whole page again. PageCrawl's agent tutorial explains why this keeps the vector database in step with a documentation source after it is first ingested. The event triggers maintenance when source content moves.

Competitive intelligence needs a different output from the same event. The handler identifies the page through page.url, extracts the new value from contents or the diff, and updates a pricing database. Escalation depends on the magnitude of the change. Compliance routing instead sends relevant diff text to an LLM with a compliance-analysis prompt, then stores the source URL, summary, priority score, and analysis in a structured audit record. For structured data, Meter's item arrays support propagation of only the rows that changed.

Making the receiver reliable — idempotency, retries, signing, and dead-letter queues

Production starts with the assumption that delivery can repeat. Deduplication on the event’s id prevents an already processed check from causing another embedding job or Slack alert. Next comes durable capture before expensive processing. The receiver stores the event, returns 200 after capture, and leaves a worker to process it asynchronously.

Retries need exponential backoff, jitter, and a bounded attempt count. Progressively longer delays can span seconds to several hours, while jitter spreads returning backlog traffic over time. Failure classification prevents pointless retries. Timeouts and 5xx responses are retryable, while 4xx responses other than 429 should go directly to a dead-letter queue.

Signing controls who may send events. HMAC-SHA256 signing with a per-endpoint secret, raw request bytes, and a timestamp provides the verification pattern. Parsed and reserialized JSON can change the bytes and break verification. A short acceptance window of around five minutes helps reject replays.

Finally, exhausted retries need somewhere visible to land. A dead-letter queue retains the payload, target endpoint, attempt history, and last response, and raises an alert. Its replay control allows redelivery once the consumer is healthy. Otherwise, a failed delivery disappears.

Combining scraping APIs with change detection so re-fetches happen only when warranted

PageCrawl's 2026 scraping API comparison separates reading a page from knowing when it changes, with different price curves for each job. A monitor supplies the signal, while a scraping API performs deeper extraction when that signal justifies it. Evaluating an event-driven web scraping API with webhook support requires this distinction alongside the presence of a callback URL.

In the combined pattern, scheduled monitoring detects an edit and POSTs the event to the receiver. The consumer then calls the scraping API for a full structured extraction of the changed page. Scraping credits are spent only on changed pages. However, when contents already supplies the needed input, another fetch adds no value to that workflow.

Meter describes a diff-first approach that POSTs only what changed and lets the consumer re-embed that part without a full re-crawl. What the next stage lacks determines the choice. A callback announcing that extraction is ready and an event announcing that a page changed serve different purposes. The latter decides whether extraction should run at all.

Wiring AI agents to change events via webhooks and MCP

The receiver can also be an AI agent runtime endpoint. PageCrawl monitors the page and sends the event, after which the agent reads ai_summary and markdown_difference, chooses an action, and executes it. That action may update a knowledge base, send an alert, or adjust a strategy. No fresh human prompt is needed to start the work.

Monitor creation can be autonomous as well as consumption. An agent that discovers a page can add it through the API or MCP server. MCP also offers a delivery alternative for runtimes without stable inbound endpoints. Its typed tools include list-monitors, get-check-diff, get-changes-since, add-page-monitor, trigger-check, and mark-changes-seen, among 34 tools total.

MCP pull-with-cursor provides replay-safe, cursor-guaranteed delivery, while webhook push offers lower latency, measured in seconds after the detecting check fires. This is not a return to repeatedly scraping unchanged pages. The agent consumes detected changes instead. Detection defines the signal, the payload supplies context, the receiver preserves delivery, and routing turns the event into action, and the same architecture now supplies an agent’s next job.