What is Scrape Rate?

Scrape rate is measured at three levels simultaneously:

• Per IP — requests per second from a single exit node to a single domain. This is the detection-relevant unit.
• Per pipeline — aggregate requests per second across all IPs in the pool. This is the throughput unit.
• Per endpoint — requests per second to a specific URL pattern. Rate limits are often endpoint-specific, not domain-wide.

These three rates can differ dramatically. A pipeline running 20 IPs at 0.5 req/s each has a per-IP rate of 0.5 and a pipeline rate of 10. Both numbers matter for different reasons.

Detection risk doesn't scale linearly with rate — it scales faster. An IP making 1 req/s blends into normal user behaviour. The same IP making 5 req/s is in the top 0.01% of all users. Anti-bot classifiers flag velocity as one of the strongest bot signals precisely because real users almost never sustain high request rates.

This is why production pipelines distribute load across large IP pools at modest per-IP rates, rather than pushing a small number of IPs fast. The aggregate throughput can be identical; the detection profile is completely different.

A fixed scrape rate set at pipeline launch is wrong within days. Target sensitivity changes as anti-bot vendors update their models. IP reputation changes as IPs age. Server load changes by hour of day and day of week.

Production schedulers adjust rate continuously: reduce on block signal, increase gradually after a clean streak, back off during peak target traffic hours, and restore after cooldown. The goal is to always be just below the detection threshold, not to hit a configured number regardless of what's happening on the target.

Each pipeline has a configured throughput target and a per-IP rate ceiling. The scheduler allocates pool IPs to hit the throughput target while staying within the per-IP ceiling. When an IP is blocked or cooled down, the scheduler redistributes load to the remaining pool — the throughput target is maintained as long as enough IPs are available.

We publish a freshness SLO per pipeline: for a target with 50,000 pages at 5 req/s aggregate, a full refresh cycle takes ~2.8 hours. Clients see that number, not just a raw rate figure.

Many targets don't return 429. They return 200 with degraded content — a bot-wall page, a CAPTCHA challenge HTML body, or a silently empty data section. The HTTP layer looks clean; the data layer is broken.

Rate limit detection that only watches HTTP status codes misses the majority of actual rate limit events on sophisticated targets. The correct monitoring target is response body validation — hash the structure of successful responses and alert when the structure changes, regardless of status code.