← Glossary / ERR_TOO_MANY_REDIRECTS

What is ERR_TOO_MANY_REDIRECTS?

ERR_TOO_MANY_REDIRECTS is a fatal network error triggered when an HTTP client exceeds its configured maximum redirect limit—typically 5 or 10 hops—while trying to reach a target URL. In web scraping, this rarely indicates a genuine site misconfiguration; instead, it is almost always a symptom of an anti-bot trap, a missing session cookie, or a geo-fencing loop. When a scraper gets caught in a redirect cycle, it burns proxy bandwidth, stalls worker threads, and ultimately drops the record from the dataset.

Scraping ErrorsRedirect LoopsNetwork LayerSession StateProxy Bandwidth
// 02 — definitions

The infinite
hop.

Why your scraper is bouncing between endpoints, and how to break the cycle before it exhausts your worker pool.

Ask a DataFlirt engineer →

TL;DR

ERR_TOO_MANY_REDIRECTS occurs when a server repeatedly responds with 301 or 302 status codes, pointing the client back to the original URL or into an endless chain. For scrapers, this is usually caused by failing to persist cookies across requests, triggering an infinite auth-redirect loop, or hitting a WAF that uses redirects as a crude bot challenge.

01Definition & structure
ERR_TOO_MANY_REDIRECTS is a network-level exception thrown by an HTTP client when it hits its configured maximum redirect threshold (e.g., maxRedirects). It occurs when a server responds with a 301, 302, 307, or 308 status code, instructing the client to fetch a new URL, but that new URL redirects again, forming a chain or an infinite loop.
02How it works in practice
When a scraper requests a protected endpoint, the server might issue a 302 redirect to a login page, attaching a session cookie via the Set-Cookie header. If the scraper's HTTP client is configured to follow redirects automatically but is not configured to persist cookies, it requests the login page without the cookie. The server sees an unauthenticated request and redirects to the login page again. The client follows, fails again, and repeats until the client's internal counter trips and throws the error.
03The proxy bandwidth drain
Redirect loops are uniquely punishing for scraping infrastructure because they consume premium residential proxy bandwidth without yielding any data. A scraper stuck in a 20-hop loop is performing 20 full DNS lookups, 20 TLS handshakes, and transferring headers 20 times. Multiplied across thousands of concurrent workers, a single misconfigured target can spike your infrastructure costs overnight.
04How DataFlirt handles it
We never allow our HTTP clients to auto-follow redirects. Our fetch layer intercepts every 30x response. We extract the Location header, parse any Set-Cookie directives into our session state, and evaluate the destination. If the destination matches a known WAF challenge pattern, we route the next request to a specialized solver. If it loops back to the origin, we immediately abort the request, flag the session as poisoned, and rotate the proxy.
05Did you know?
Some advanced anti-bot systems intentionally create "tarpit" redirect chains. They will issue 302 redirects to dynamically generated URLs (e.g., /verify?token=1 -> /verify?token=2) up to 50 times. A real browser will eventually complete the chain, but a naive scraper with a default redirect limit of 10 will crash, neatly filtering out automated traffic without ever serving a CAPTCHA.
// 03 — the redirect math

The cost of
following blindly.

Redirects aren't free. Every hop consumes proxy bandwidth, adds TLS negotiation latency, and increases the probability of a connection drop. DataFlirt caps automated redirect following to prevent worker starvation.

Redirect latency penalty = Ttotal = Σ (Tdns + Ttls + Tttfb) × Nhops
A 5-hop chain can easily add 2.5s to the total page load time. Network Layer Mechanics
Worker starvation risk = P(starve) = Nloops / Poolsize
Unbounded redirects tie up async workers indefinitely, halting pipeline throughput. DataFlirt infrastructure models
DataFlirt redirect cap = Maxhops = 3
If the target isn't reached in 3 hops, the session state is invalid. We abort. Internal SLO
// 04 — network trace

Anatomy of an
auth-loop trap.

A scraper attempting to access a protected endpoint without persisting the required session cookie, resulting in an infinite bounce between the login gate and the target.

HTTP 302Cookie rejectionWorker timeout
edge.dataflirt.io — live
CAPTURED
// Request 1: Target URL
GET /api/v1/inventory/pricing HTTP/2
Host: api.target.com
Response: 302 Found
Location: /login?next=/api/v1/inventory/pricing
Set-Cookie: session_id=xyz; HttpOnly

// Request 2: Login Gate (Scraper drops cookie)
GET /login?next=/api/v1/inventory/pricing HTTP/2
Host: api.target.com
Response: 302 Found
Location: /api/v1/inventory/pricing

// Request 3: Target URL (Still no cookie)
GET /api/v1/inventory/pricing HTTP/2
Response: 302 Found
Location: /login?next=/api/v1/inventory/pricing

// Client intervention
Error: ERR_TOO_MANY_REDIRECTS
Worker state: Terminated after 10 hops
// 05 — root causes

Why the server
keeps bouncing you.

Genuine site misconfigurations account for less than 5% of redirect loops in scraping. The vast majority are state-management failures on the client side.

SAMPLE SIZE ·  ·  ·  ·    1.2M errors
WINDOW ·  ·  ·  ·  ·  ·   30d trailing
UPDATED ·  ·  ·  ·  ·  ·  2026-05-19
01

Cookie persistence failure

68% of loops · State dropped between hops
02

Geo-fencing loops

15% of loops · Proxy IP clashes with requested locale
03

WAF JS challenges

9% of loops · Redirecting to a challenge page that fails
04

Mobile/Desktop toggle

5% of loops · User-Agent mismatching the endpoint
05

Trailing slash canonicals

3% of loops · Poorly configured backend routing
// 06 — state management

Stop following,

start inspecting the headers.

Blindly configuring your HTTP client to follow redirects is a rookie mistake. DataFlirt disables automatic redirect following at the network layer. Instead, we intercept every 301/302, inspect the Location and Set-Cookie headers, and programmatically decide whether the hop is legitimate or a trap. If a target demands a cookie, we persist it. If it loops, we quarantine the proxy and rotate the session.

Redirect Interceptor Log

DataFlirt edge worker handling a 302 response.

target.url /secure/data
response.status 302 Found
header.location /challenge/js
header.set_cookie __cf_bm=8a9f...
auto_follow false
action.taken intercept_and_solve
pipeline.status recovered

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

Common questions about redirect limits, cookie management, and how to prevent infinite loops in your scraping pipeline.

Ask us directly →
What is the default redirect limit in most HTTP clients? +
It varies, but it's usually too high for scraping. Python's requests defaults to 30, Axios defaults to 21, and Go's net/http defaults to 10. If a target hasn't served the payload within 3 hops, you are almost certainly caught in an auth or WAF loop. We recommend hardcapping your client at 3 to 5 hops.
How do I fix a cookie-based redirect loop? +
Use a CookieJar or a session object. When the server sends a 301/302, it often includes a Set-Cookie header. If your client follows the Location header but fails to attach that new cookie to the subsequent request, the server will just redirect you again. State persistence is mandatory.
Why does my scraper get a redirect loop but my browser doesn't? +
Browsers automatically handle cookies, execute JavaScript, and maintain state across the entire session. Your scraper is likely dropping a critical session token, or failing to execute a JS-based challenge that the server requires to break the loop and grant access.
Should I disable automatic redirects in my scraper? +
Yes. Set allow_redirects=False (Python) or maxRedirects: 0 (Node). Handle 30x responses manually. This allows you to capture cookies, inspect the destination URL, and detect traps before burning proxy bandwidth on an infinite loop.
How does DataFlirt handle geo-redirect loops? +
If a target forces US users to /en-us/ but your proxy is in the UK, the site might redirect you to /en-gb/, which your scraper then tries to force back to /en-us/. We map proxy ASN locales strictly to the target's expected region to prevent locale bouncing entirely.
Can a WAF use redirects to detect bots? +
Absolutely. Some WAFs issue a 302 to a temporary URL, set a tracking cookie, and expect the client to follow it. If the client follows too fast (sub-millisecond) or without the cookie, it's flagged as a bot. Intercepting the redirect allows you to add a realistic human delay before following.
$ dataflirt scope --new-project --target=err_too_many_redirects READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Scraping Errors

Redirect Loop Error

The broader category of infinite routing failures that cause worker starvation.

Read →
Scraping Errors

Max Redirects Exceeded

The specific exception thrown by HTTP clients when the hop limit is breached.

Read →
Web Scraping

Redirect Following

The automated client behavior that turns a single 302 into an infinite loop.

Read →
Anti-Scraping

Cookie Validation Failure

The root cause of most auth-based redirect loops in modern web scraping.

Read →