← Glossary / Proxy Health Check

What is Proxy Health Check?

Proxy health check is the continuous, automated validation of an IP node's viability before it is assigned to a scraping worker. It measures latency, connection success rate, target-specific block status, and bandwidth capacity. In a production proxy pool, health checks prevent dead or burned IPs from causing cascading timeouts, ensuring that your pipeline spends its execution time extracting data rather than waiting on dropped TCP handshakes.

IP ProxiesLatencyTCP HandshakePool ManagementRouting

// 02 — definitions

Test before
you route.

Why blindly trusting a proxy provider's uptime SLA is the fastest way to tank your pipeline's success rate.

Ask a DataFlirt engineer →

TL;DR

A proxy health check actively probes IPs in your pool against reference targets (like Cloudflare trace endpoints or Google) to verify they are alive, fast, and unblocked. Without aggressive health checking, dead nodes accumulate in your rotation, driving up error rates and inflating cloud compute costs as workers hang on dead sockets.

01Definition & structure

A proxy health check is an automated diagnostic routine that verifies an IP address is capable of successfully routing traffic before it is handed to a scraping worker. A complete health check validates three layers:

Transport: Can we establish a TCP connection and complete a TLS handshake within the timeout window?
Anonymity: Is the proxy leaking the origin IP via X-Forwarded-For or Via headers?
Target Viability: Is the IP currently blocked, rate-limited, or CAPTCHA-gated by the specific destination site?

Nodes that fail any critical check are quarantined or evicted from the active pool.

02Active vs. Passive checking

Active checking involves a dedicated background worker sending synthetic requests (like an HTTP HEAD to a fast CDN) to test the proxy. It guarantees the proxy is alive right now, but consumes bandwidth. Passive checking analyzes the HTTP status codes of actual scraping requests flowing through the proxy. If three real requests in a row return a 502 Bad Gateway, the passive monitor marks the proxy as dead. Production systems combine both.

03Target-specific validation

Health is relative. A datacenter IP might have 10ms latency and perfect uptime, making it "healthy" for scraping a poorly defended public directory. But if you route a request for a Cloudflare-protected site through that same IP, it will instantly return a 403. Advanced health checking systems maintain state per-target, ensuring that an IP is only marked "healthy" for the specific domains where its reputation is still intact.

04How DataFlirt handles it

We decouple health checking from data extraction. Our proxy gateway runs a continuous, asynchronous validation loop against our entire IP inventory. When your pipeline requests a URL, the gateway instantly assigns an IP that was verified healthy within the last 60 seconds. If a target-specific block occurs during a live scrape, the gateway intercepts the 403, evicts the IP for that target, and retries the request with a fresh node before returning the data to your worker.

05The "transparent proxy" trap

A common failure mode when buying cheap proxy lists is acquiring transparent proxies. These nodes pass basic TCP health checks perfectly, but they append your actual server IP to the HTTP headers. If your health check doesn't explicitly validate anonymity levels by echoing headers back to itself, you will route traffic through these nodes and instantly expose your scraping infrastructure's origin IP to the target.

// 03 — the math

Quantifying
node viability.

Health isn't binary. A node can be alive but too slow for a synchronous browser fetch, or fast but blocked by Akamai. DataFlirt calculates a composite health score per IP to determine routing eligibility.

Node Health Score = H = w₁(Uptime) + w₂(1/Latency) − w₃(Block_Rate)

Weighted composite. Drops below threshold trigger immediate eviction. DataFlirt routing logic

Timeout Probability = P(T) = 1 − e^{−(λ · t)}

Probability of a node failing within time t, given historical failure rate λ. Reliability engineering standard

Pool Viability Ratio = V = Healthy_Nodes / Total_Assigned_Nodes

V < 0.85 indicates upstream provider degradation or aggressive target blocking. DataFlirt infrastructure SLO

// 04 — health check trace

Validating a residential
exit node in 120ms.

A standard DataFlirt pre-flight health check. We test TCP connectivity, TLS negotiation, and target-specific block status before the IP enters the active routing table.

TCP/IPTLS 1.3Cloudflare Trace

edge.dataflirt.io — live

CAPTURED

// init health check worker
node.ip: "198.51.100.42"
node.asn: "AS7922 · Comcast Cable"

// phase 1: transport layer
tcp.handshake: 42ms OK
tls.negotiation: 78ms OK

// phase 2: anonymity check
headers.via: null
headers.x_forwarded_for: null
anonymity.level: "elite" OK

// phase 3: target-specific probe (target: e-commerce)
probe.url: "https://target.com/cdn-cgi/trace"
probe.status: 403 Forbidden
probe.cf_ray: "8a4f9b2...-IAD"

// routing decision
node.status: BURNED_FOR_TARGET
action: evict from target_pool_A, reassign to general_pool_B

// 05 — failure modes

Why proxies
fail checks.

The most common reasons an IP is marked unhealthy and evicted from the active rotation, based on DataFlirt's telemetry across 14M+ daily proxy checks.

DAILY CHECKS · · · · 14.2M

AVG EVICTION · · · · 12% of pool

UPDATED · · · · · · 2026-05-19

01

TCP connection timeout

offline node · Device powered off or network drop (common in residential)

02

Target-specific 403 / CAPTCHA

burned IP · IP reputation tanked for the specific destination domain

03

High latency / Jitter

congested link · Fails the 2000ms strict timeout threshold for browser jobs

04

SSL/TLS negotiation failure

interception · Upstream proxy attempting MITM or using outdated ciphers

05

Transparent proxy detection

leaking real IP · Injecting X-Forwarded-For headers, destroying anonymity

// 06 — our architecture

Check continuously,

route instantly, evict ruthlessly.

DataFlirt doesn't rely on upstream provider health metrics. We run an independent, asynchronous health-checking sidecar that probes every node in our residential and datacenter pools every 60 seconds. If a node fails a check, it is instantly evicted from the active Redis routing table. When a scraper requests an IP, it receives one that was verified healthy less than a minute ago. We absorb the latency of the health check so your extraction workers don't have to.

Proxy Routing State

Live snapshot of a residential node in the DataFlirt routing table.

node.id res-us-east-8842

pool.type residential · AT&T

last_check 14s ago

tcp.latency 42ms

target.status clean

bandwidth.util 94%

routing.state ACTIVE

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

// 07 — FAQ

Common
questions.

About active vs. passive checking, bandwidth costs, target-specific blocking, and how DataFlirt maintains a 99.9% healthy proxy rotation.

Ask us directly →

What is the difference between active and passive proxy health checks? +

Active checks send deliberate probe requests (e.g., to a fast CDN trace endpoint) on a schedule to verify the proxy is alive. Passive checks monitor the success/fail rate of actual scraping traffic flowing through the proxy. Production systems use both: active checks for baseline viability, and passive checks to detect target-specific blocks without generating artificial load.

Does aggressive health checking waste proxy bandwidth? +

Yes, if done poorly. If you pay per GB for residential proxies, downloading a 1MB test payload every minute will bankrupt you. We use zero-byte endpoints (like Cloudflare's /cdn-cgi/trace) or HTTP HEAD requests for active checks. The bandwidth consumed is negligible (bytes per request), but the reliability gained is massive.

How does DataFlirt handle target-specific blocks? +

An IP might be perfectly healthy for scraping Amazon but hard-blocked by LinkedIn. We maintain health scores per IP per target domain. If a node returns a 403 or CAPTCHA for Target A, it is evicted from Target A's routing pool but remains available for Target B. This maximizes the utility of the proxy pool without impacting success rates.

Is it legal to probe third-party targets for health checks? +

Sending a standard HTTP GET or HEAD request to a public endpoint to verify network routing is standard internet behavior. However, we avoid hitting the actual target's expensive infrastructure for baseline health checks to minimize unnecessary load. We use lightweight CDN trace endpoints for general health, and rely on passive monitoring of actual scraping traffic for target-specific block detection.

Why do residential proxies fail health checks so often? +

Residential proxies are real devices (phones, laptops, smart TVs) on consumer internet connections. They get turned off, lose Wi-Fi, or experience severe local network congestion. A 10-15% churn rate per hour is completely normal for a residential pool. This volatility is exactly why continuous health checking is mandatory — you cannot assume a residential IP will be alive 5 minutes from now.

What happens if a proxy dies mid-request? +

The scraper will throw a socket timeout or connection reset error. DataFlirt's fetch layer catches this, immediately marks the proxy as dead in the routing table, and transparently retries the request with a fresh, healthy IP from the pool. The downstream extraction logic never sees the failure.

$ dataflirt scope --new-project --target=proxy-health-check READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing

hello@dataflirt.com · Bengaluru · IST · typical reply < 4h