← Glossary / Retry-After Header

What is Retry-After Header?

Retry-After Header is an HTTP response header sent by a server alongside a 429 Too Many Requests or 503 Service Unavailable status code, indicating exactly how long a client must wait before making another request. For scraping pipelines, it is the definitive signal that you have hit a rate limit. Ignoring it guarantees an IP ban; respecting it allows your crawler to gracefully pause and resume without burning proxy reputation.

HTTP 429Rate LimitingProxy HealthBackoff LogicRFC 7231
// 02 — definitions

Pause and
resume.

The explicit backoff instruction from the target server, and why parsing it correctly is the difference between a temporary throttle and a permanent block.

Ask a DataFlirt engineer →

TL;DR

The Retry-After header tells your scraper exactly when it is safe to request again. It can be formatted as an integer (seconds) or an HTTP-date. Modern anti-bot systems use it as a compliance test: bots that ignore the header and keep hammering the endpoint are immediately blacklisted.

01Definition & structure
The Retry-After header is an HTTP response header indicating how long the user agent should wait before making a follow-up request. It is typically sent with a 503 (Service Unavailable) or a 429 (Too Many Requests) response. It is the server's explicit instruction on how to back off gracefully.
02Formats and parsing
It comes in two valid formats: a decimal integer indicating the number of seconds to wait (e.g., Retry-After: 120), or an HTTP-date string (e.g., Retry-After: Fri, 31 Dec 1999 23:59:59 GMT). Robust scraping infrastructure must parse both and handle timezone conversions correctly if the date format is used.
03The Thundering Herd problem
If a target rate-limits 500 of your concurrent workers simultaneously with a Retry-After: 60, and they all wake up exactly 60 seconds later, they will instantly trigger another 429 or crash the target. Adding random jitter (e.g., 60s + random(0, 15)s) prevents this synchronized traffic spike.
04How DataFlirt handles it
We treat Retry-After as a hard infrastructure constraint. Our distributed scheduler intercepts the header, parks the specific proxy-session pair in a delayed queue, and reallocates the worker compute to other tasks. We never drop a request; we just delay it compliantly while maintaining overall pipeline velocity.
05Did you know?
Some anti-bot vendors intentionally send a 429 with a very short Retry-After (e.g., 2 seconds) on the very first request of a new session. This is a honeypot to see if your client respects HTTP protocol standards. Naive scrapers ignore it and get permanently banned before they even fetch the HTML.
// 03 — backoff math

How long do
you actually wait?

DataFlirt's scheduler never trusts the raw header blindly. We apply a jittered multiplier to avoid the 'thundering herd' problem when thousands of suspended workers wake up simultaneously.

Effective wait time = Twait = Retry-After × (1.0 + Jitter)
Jitter is typically 0.1 to 0.3 to stagger reconnects. DataFlirt Scheduler
Date parsing fallback = Twait = Dateheader − Datecurrent
Used when the header is an HTTP-date instead of delta-seconds. RFC 7231
Penalty multiplier = P = 2violations
Exponential penalty applied if a worker accidentally breaches the wait window. Internal proxy router
// 04 — the wire trace

Hitting the wall
and backing off.

A worker thread encounters a 429 response, parses the Retry-After header, suspends the session, and successfully resumes.

HTTP 429delta-secondsworker suspend
edge.dataflirt.io — live
CAPTURED
// request 4,091
GET /api/v1/catalog?page=409
status: 429 Too Many Requests
Retry-After: 120
X-RateLimit-Remaining: 0

// scheduler intervention
action: suspend_worker
wait_target: 120s + 14s // adding jitter
proxy_session: preserved

// 134 seconds later...
action: resume_worker
GET /api/v1/catalog?page=409
status: 200 OK
X-RateLimit-Remaining: 999
// 05 — failure modes

Why retry logic
fails in production.

The most common reasons scrapers get banned despite attempting to respect rate limits.

PIPELINES MONITORED ·   300+ active
429s ANALYZED ·  ·  ·  ·  10M+ monthly
UPDATED ·  ·  ·  ·  ·  ·  2026-05-19
01

Ignoring the header entirely

100% ban rate · Default HTTP clients don't auto-pause
02

Thundering herd effect

High risk · All workers wake up at exactly T+0
03

Date parsing errors

Silent failure · Mishandling GMT/UTC HTTP-date formats
04

Proxy rotation mismatch

Session lost · Rotating IP before the retry window ends
05

Clock skew

Variable · Local server time drifts ahead of target
// 06 — our architecture

Don't just wait,

reallocate the worker.

When a DataFlirt worker receives a Retry-After header, sleeping the thread is a waste of compute. Instead, our scheduler parks the specific target-proxy pairing in a Redis-backed delayed queue and immediately assigns the worker to a different domain or a different proxy session. This ensures 100% compliance with the target's rate limit without degrading our overall pipeline throughput.

Worker reallocation state

Live trace of a worker thread handling a 429 response.

worker.id w-eu-west-04
target.status 429 Too Many Requests
retry_after 300s
action park_session
reallocation target_b_catalog
worker.utilization 98.4%
compliance.status strict_adherence

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

About rate limits, backoff strategies, and how DataFlirt maintains throughput when targets push back.

Ask us directly →
What happens if I ignore the Retry-After header? +
You fail the compliance test. Modern WAFs (Cloudflare, Akamai) use the 429 response as a trap. If you send another request before the timer expires, your IP or ASN is immediately moved from a temporary rate-limit bucket to a permanent blocklist.
Is Retry-After always in seconds? +
No. RFC 7231 allows it to be either an integer (delta-seconds) or an HTTP-date (e.g., Fri, 31 Dec 1999 23:59:59 GMT). Your scraper must be able to parse both formats. Failing to parse the date format usually results in the scraper defaulting to zero wait time, leading to a ban.
How does DataFlirt handle 429s at scale? +
We decouple the worker from the session. When a 429 hits, the specific proxy-target session is parked in a delayed queue for the exact duration of the Retry-After header plus a 10% jitter. The worker thread immediately picks up a different task. This maintains high throughput while guaranteeing zero limit violations.
Should I rotate my proxy when I get a Retry-After header? +
Usually, no. If you are scraping behind a login or need session continuity, rotating the proxy abandons the session. It is better to park the proxy, wait out the timer, and resume. If you are doing stateless scraping, you can rotate, but aggressive rotation after 429s can burn through your proxy pool's reputation.
What if the server sends a 429 but no Retry-After header? +
This is common with poorly configured APIs. In this case, you must implement your own exponential backoff strategy. Start with a 2-second wait, then 4, 8, 16, up to a maximum threshold. DataFlirt automatically falls back to a tuned exponential backoff if the header is missing.
Is it legal to scrape a site that rate-limits me? +
Rate limits are technical access controls, not inherently legal boundaries. However, aggressively bypassing rate limits by rotating thousands of IPs to ignore a 429 can be construed as unauthorized access or trespass to chattels in some jurisdictions. Respecting the Retry-After header demonstrates good-faith interaction with the target's infrastructure.
$ dataflirt scope --new-project --target=retry-after-header READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Scraping Errors

HTTP 429 Too Many Requests

The status code that triggers the need to parse the Retry-After header.

Read →
Network Layer

Rate Limiting

The server-side mechanism that restricts how many requests your scraper can make.

Read →
Web Scraping

Exponential Backoff

The fallback strategy you must use when a server sends a 429 without a Retry-After header.

Read →
Network Layer

X-RateLimit Header

The proactive headers that tell you how close you are to hitting the limit.

Read →