What is Webhook?

A webhook is an HTTP-based callback function that facilitates communication between two APIs. Unlike typical APIs where the client initiates the request (polling), a webhook is event-driven. When a specific event occurs in the source system—like a scraping job completing—the source system makes an HTTP POST request to a URL provided by the destination system. The payload is typically formatted as JSON and contains the relevant data or a pointer to it.

Polling requires your server to repeatedly ask, "Is the data ready?" This wastes compute cycles, consumes unnecessary bandwidth, and introduces latency (if you poll every 60 seconds, your data is on average 30 seconds stale). Webhooks invert this. Your server sits idle until the data is ready, at which point the scraping engine pushes it directly to you. It is the only scalable way to handle asynchronous jobs that might take anywhere from 5 seconds to 5 hours to complete.

Because webhook endpoints must be publicly accessible to receive data, they are vulnerable to spoofing. To secure them, the sender generates an HMAC (Hash-based Message Authentication Code) using the payload body and a shared secret, sending it in a custom header (e.g., X-Signature). The receiver recalculates the hash using their copy of the secret. If the hashes match, the payload is authentic and hasn't been tampered with in transit.

We treat webhook delivery as a first-class infrastructure component. Every outbound webhook is dispatched via a distributed message queue. If your endpoint is unreachable, we apply exponential backoff with jitter to prevent thundering herd problems when your infrastructure recovers. For massive datasets, we don't clog your endpoint with gigabytes of JSON; we upload the data to S3 and webhook you a secure, time-bound presigned URL.

The most common mistake engineers make is processing the webhook payload synchronously. If your endpoint receives a POST, parses the JSON, runs database inserts, and then returns a 200 OK, you will inevitably hit the sender's timeout threshold (usually 5-10 seconds). The sender assumes failure and retries, leading to duplicate data. Always accept the payload, push it to an internal queue, and return a 202 Accepted immediately.