Crawl delay is the minimum wait interval a crawler enforces between consecutive requests to the same host — declared by the target in robots.txt or imposed by the crawler's own rate-limiter to avoid triggering detection and overloading origin servers. Ignore it and you get blocked; respect it naively across a 50-million-URL frontier and your pipeline takes six months to complete.
Crawl delay is not a courtesy setting — it's a scheduling constraint that determines throughput, detectability, and whether you finish the job before the data goes stale.
Ask a DataFlirt engineer →The <code>Crawl-delay</code> directive in robots.txt tells compliant bots how many seconds to wait between requests per host. Most serious targets set 2–10 seconds. At 5s delay on a single connection, crawling 1M pages takes 58 days — which is why production pipelines distribute across thousands of concurrent sessions, each honoring per-host delay independently.
robots.txt — the Crawl-delay: N directive, scoped to a User-agent groupnow - last_fetch_time >= crawl_delay. If the condition isn't met, the URL goes back into a hold state and the next available host slot is tried. This is why frontier design matters: a frontier that can only hold one host at a time will idle constantly on a 5s delay. Production frontiers hold thousands of host queues simultaneously so worker threads are never waiting on a single clock.
Retry-After: 60, the host slot locks for 60s — not 60 + 3s, just 60s (since 60 > 3). The two mechanisms are orthogonal: never mistake a rising 429 rate for a tuning problem with your delay setting. 429s mean you're overrunning server capacity or session fingerprint thresholds — not that your delay is slightly wrong.
robots.txt per host with a 6-hour TTL, and feed Crawl-delay values directly into our distributed scheduler's per-host token store (Redis). Our own politeness floor sits at 1.5 seconds — no host gets hit faster than that even if robots.txt is absent. For high-value targets where we've negotiated or observed aggressive rate-limiting, we apply custom per-host overrides. The 429 rate across our active pipelines runs below 0.2% — meaning the delay logic is calibrated well enough that targets almost never push back.
These three formulas are what every crawl scheduler solves at runtime. Get the delay budget wrong and you either block your own pipeline or saturate the target and trigger rate-limiting defenses.
Every host gets an independent delay clock. Here's a trace from our scheduler processing a single domain slot — from frontier pop to next-allowed-at update.
Crawl delay is rarely a single number from a single place. In practice, the effective delay applied to any given host is the maximum of several competing constraints — some declared, some inferred, some self-imposed.
not a global throttle.
Naive crawlers apply one rate limit globally. That's wrong — it either over-throttles small hosts or under-throttles large ones. DataFlirt runs per-host delay tokens in a distributed Redis clock so every origin gets exactly its declared wait, regardless of how many other hosts are being crawled simultaneously.
A snapshot of a single host slot as managed by our scheduler cluster.
Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.
About crawl delay, robots.txt compliance, rate-limiting, and how to keep a high-throughput pipeline polite without sacrificing speed.
Ask us directly →robots.txt is a convention, not an RFC or statute. Violating it won't get you arrested, but ignoring it aggressively enough to impair the server can expose you to CFAA claims in the US or IT Act liability in India. More practically, targets that see their delay directive ignored tend to upgrade to hard rate-limiting or block ranges.20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.