← Glossary / Keep-Alive Efficiency

What is Keep-Alive Efficiency?

Keep-alive efficiency measures the ratio of HTTP requests successfully routed over reused TCP/TLS connections versus those requiring a cold handshake. In high-throughput scraping pipelines, establishing a new connection for every request wastes CPU cycles, burns proxy bandwidth, and adds 100-300ms of latency per fetch. Optimising this ratio is the single highest-leverage infrastructure tweak for scaling requests per second without increasing hardware spend.

Scraping PerformanceTCP/TLSConnection PoolingLatencyHTTP/1.1

// 02 — definitions

Stop shaking
hands.

Why tearing down and rebuilding network sockets for every single GET request is destroying your pipeline's throughput.

Ask a DataFlirt engineer →

TL;DR

Keep-alive efficiency tracks how many requests share a single network socket. A cold TLS handshake takes three network round trips. If you fetch 1,000 pages with 1,000 new connections, you pay that latency tax 1,000 times. A highly efficient pipeline multiplexes those requests over a handful of persistent connections, cutting overhead by up to 80%.

01Definition & structure

Keep-alive efficiency is the measure of how effectively a scraping pipeline reuses established network sockets. When a client connects to a server via HTTPS, it must perform a DNS lookup, a TCP handshake, and a TLS negotiation. This process is computationally expensive and adds significant latency.

By sending the Connection: keep-alive header, the client asks the server to leave the socket open after the response is sent. Subsequent requests to the same host can then be transmitted immediately over the warm socket, bypassing the handshake phase entirely.

02How it works in practice

In a production scraper, keep-alive is managed by a connection pool. The pool maintains a set of open sockets to frequently accessed domains. When your code requests a URL, the HTTP client checks the pool. If a warm socket for that domain is available and idle, it is reused. If not, a new socket is created, used, and then returned to the pool.

Efficiency drops when the pool is misconfigured (e.g., max connections set too low) or when the target server enforces aggressive Keep-Alive: timeout=5 headers, forcing the pool to constantly prune and rebuild connections.

03The anti-bot tradeoff

While high keep-alive efficiency is great for throughput, it is a massive red flag for bot detection. Normal human browsing involves fetching an HTML document and its associated assets, resulting in maybe 20-50 requests per socket before the user idles and the connection drops.

A scraper routing 5,000 sequential API requests over a single TCP session is trivial to fingerprint. Security vendors monitor requests-per-session closely. Optimising a pipeline requires balancing maximum socket reuse against the target's specific behavioral thresholds.

04How DataFlirt handles it

We decouple the client's connection from the target's connection. Our edge nodes maintain highly tuned, persistent connection pools to target domains. We dynamically adjust the maximum requests per socket based on real-time WAF telemetry.

If a target flags connections after 100 requests, our pool automatically gracefully closes the socket at request 95 and seamlessly routes request 96 through a newly warmed socket on a different proxy IP. The client pipeline never sees the handshake latency or the IP rotation.

05The most common developer mistake

HTTP/1.1 enables keep-alive by default, leading many developers to assume their scripts are efficient. However, using convenience methods like requests.get() in Python creates an entirely new session object under the hood for every single call.

Because the session object is destroyed immediately after the request completes, the underlying socket is closed. To actually achieve keep-alive efficiency, you must explicitly instantiate a requests.Session() and use it for all requests in your loop.

// 03 - the math

Measuring socket
reuse.

Connection overhead is pure waste. DataFlirt's telemetry tracks these ratios per proxy node to ensure we aren't burning CPU on unnecessary cryptography.

Keep-Alive Ratio = Req_reused / (Req_reused + Req_cold)

Target > 0.90 for API scraping. Lower for HTML scraping due to IP rotation. Standard Network Telemetry

Handshake Latency Tax = Req_cold * (RTT_tcp + RTT_tls)

The total time wasted negotiating connections instead of transferring data. DataFlirt Performance Model

DataFlirt Pool Efficiency = Active_Sockets / Worker_Concurrency

Maintained at < 0.15. We use fewer sockets to serve more concurrent workers. Internal SLO

// 04 - connection pool trace

Cold starts vs.
warm sockets.

A trace from a DataFlirt worker node fetching 5 JSON payloads from a REST API. Notice the latency drop after the initial connection is established.

HTTP/1.1connection poolingTLS 1.3

edge.dataflirt.io — live

CAPTURED

// init connection pool
pool.size: 10 target: "api.target.com"

// req 1: cold start
socket.id: "sk_992a" state: NEW
timing.tcp: 42ms timing.tls: 88ms
timing.ttfb: 185ms status: 200 OK

// req 2: warm socket
socket.id: "sk_992a" state: REUSED
timing.tcp: 0ms timing.tls: 0ms
timing.ttfb: 55ms status: 200 OK

// req 3-5: warm socket
socket.id: "sk_992a" state: REUSED
timing.ttfb.avg: 53ms

// pool summary
requests.total: 5 handshakes: 1
keep_alive.efficiency: 80.0%

// 05 - efficiency killers

Why connections
get dropped.

Ranked by frequency across DataFlirt's infrastructure. Most keep-alive failures are caused by misconfigured client libraries or aggressive server-side timeouts.

PIPELINES MONITORED · 300+ active

AVG REUSE RATE · · · 82.4%

UPDATED · · · · · · 2026-05-19

01

Client session recreation

developer error · instantiating a new requests.Session() inside the loop

02

Server-side Keep-Alive timeout

target config · Nginx dropping idle sockets after 5 seconds

03

Proxy gateway termination

network layer · backconnect proxies forcing connection resets per request

04

WAF connection reset

anti-bot · Cloudflare dropping sockets that exceed request thresholds

05

HTTP/1.0 fallback

protocol error · legacy servers refusing persistent connections entirely

// 06 - our stack

Pool connections locally,

rotate them globally.

DataFlirt maintains persistent connection pools at the edge. When a client requests a page, we route it through an already-warm socket if the target allows it, or seamlessly spin up a new one if the proxy IP needs rotating. We absorb the TLS handshake latency so your extraction logic doesn't have to wait.

worker-pool-metrics

Live connection pool stats for a high-throughput JSON API pipeline.

target.host api.retail-target.com

pool.active_sockets 24optimal

requests.reused 14,205

handshakes.saved 14,205high efficiency

avg.latency.warm 42ms

avg.latency.cold 215ms

pool.status healthy

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

// 07 — FAQ

Common
questions.

About connection pooling, HTTP/2 multiplexing, proxy compatibility, and scaling throughput.

Ask us directly →

Why is my Python scraper so slow even with async? +

You are likely recreating your client session object inside your fetch loop. In Python, calling requests.get() or instantiating a new aiohttp.ClientSession() per URL forces a brand new TCP and TLS handshake every time. You must create the session once outside the loop and reuse it to benefit from keep-alive.

Do residential proxies support keep-alive? +

Often no, or very poorly. Most residential proxy networks use backconnect gateways that terminate your TCP connection at their load balancer, then open a separate connection to the target. Even if your local script uses keep-alive to the proxy, the proxy might not use keep-alive to the target.

How does HTTP/2 change keep-alive efficiency? +

HTTP/1.1 keep-alive is sequential - you must wait for request A to finish before sending request B on the same socket. HTTP/2 introduces multiplexing, allowing dozens of concurrent requests over a single TCP connection simultaneously. This makes connection reuse exponentially more valuable for scraping throughput.

What is the optimal connection pool size? +

It depends strictly on the target's rate limits and your proxy concurrency limits. A pool size of 100 is useless if the target's WAF bans any IP making more than 10 concurrent requests. We typically size pools dynamically based on the target's observed 429 (Too Many Requests) threshold.

How does DataFlirt balance keep-alive with IP rotation? +

We maintain separate warm pools per exit node. When a pipeline requires IP rotation, we don't tear down the socket - we simply route the next request to a different warm socket in the pool that is bound to a fresh residential IP. The handshakes happen asynchronously in the background.

Can keep-alive trigger anti-bot bans? +

Yes. Advanced anti-bot systems like Akamai and Cloudflare track the number of requests per TCP session. A human browser rarely makes 500 requests over a single socket. A scraper does. We monitor these thresholds and intentionally rotate sockets before the WAF flags the connection as mechanical.

$ dataflirt scope --new-project --target=keep-alive-efficiency READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing

hello@dataflirt.com · Bengaluru · IST · typical reply < 4h