What is Polite Crawling?

Polite crawling has three components that work together:

• Crawl-delay compliance — reading robots.txt and enforcing the declared minimum gap between requests
• Backoff on error signals — honouring Retry-After on 429 and 503 responses, or applying exponential backoff when no directive is present
• Concurrency control — limiting simultaneous connections per IP, since 10 parallel threads at 1 req/s is the same as 1 thread at 10 req/s from the server's perspective

All three must be active simultaneously. Honouring Crawl-delay while running 20 concurrent connections defeats the purpose entirely.

Before the first fetch, the crawler reads robots.txt and parses the Crawl-delay for its user-agent (and the wildcard * entry as fallback). That value becomes the minimum inter-request delay. A token bucket or leaky bucket rate limiter enforces it. On every response, the crawler checks for Retry-After. On 429 or 503, it backs off for at least the declared duration — or uses exponential backoff if no header is present — then resumes at a reduced rate. After N consecutive successful requests, it can step the rate back up to the calibrated maximum.

Crawl-delay declarations tell you the minimum. They don't tell you the maximum safe rate for your pipeline — a site might declare 1s but block aggressively at 0.5 req/s if the underlying server is under-provisioned. The correct approach is a ramp-up probe: start at 0.1 req/s, increase by 0.1 req/s every 60 seconds, and record the rate at which 429s first appear. Set the production rate at 80% of that threshold. This takes under 30 minutes and prevents bans that take days to clear.

Every DataFlirt pipeline starts with a rate calibration pass — ramp-up probe, threshold detection, production rate set at 80% of threshold. We apply ±20% jitter on every delay interval by default. Concurrency is always 1 connection per IP unless the target has been explicitly tested for higher concurrency. Our rate limiter is per-IP and per-domain independently — a session using multiple IPs enforces the rate limit on each IP separately, not aggregated across the pool.

Large CDN-backed sites have higher raw throughput capacity but more sophisticated rate detection. Cloudflare's rate limiting is per-IP, per-path-prefix, and per-ASN simultaneously — you can be within per-IP limits and still trigger a block at the path-prefix level if your crawler concentrates all fetches on /products/. Politeness on large sites means distributing load across paths and time windows, not just slowing down per-request. The detection is smarter; the strategy needs to match.