← Glossary / Write Timeout

What is Write Timeout?

Write timeout is a network-layer error that occurs when a scraper fails to transmit its request payload to the target server within the configured time limit. Unlike read timeouts where you are waiting on the server to respond, a write timeout means your client or proxy cannot push bytes fast enough. In scraping pipelines, this typically strikes during heavy POST requests—like submitting massive GraphQL payloads, uploading files to bypass challenges, or routing through heavily throttled residential proxies.

Network ErrorTCP SocketProxy ThrottlingPOST Requests
// 02 — definitions

Bytes stuck
in transit.

When your scraper can connect to the server but cannot push the payload through the pipe fast enough before the clock runs out.

Ask a DataFlirt engineer →

TL;DR

A write timeout happens when the local TCP send buffer fills up because the network or proxy cannot transmit data to the remote server before the socket timer expires. It is almost exclusively a problem with POST/PUT requests routed over slow residential proxies with asymmetric bandwidth.

01Definition & structure

A write timeout is an error thrown by an HTTP client when it cannot push the entirety of a request payload to the network within a specified duration. At the OS level, this means the local TCP send buffer (SO_SNDBUF) is full, and the socket blocks until space frees up. If the timer expires before the buffer drains, the connection is aborted.

This error is entirely client-side. The server may be perfectly healthy, but the pipe between your scraper and the server is too narrow to accommodate the payload in the time allowed.

02How it works in practice

In web scraping, write timeouts rarely happen on GET requests because the payload is just a few hundred bytes of headers. They strike during heavy POST or PUT requests. Common scenarios include:

  • Submitting massive GraphQL mutation batches.
  • Uploading images to third-party CAPTCHA solving services.
  • Sending large JSON payloads to internal data ingestion APIs.

When routed through a slow proxy, the scraper writes data to the proxy faster than the proxy can forward it to the target. The proxy stops acknowledging packets, the scraper's TCP window closes, and the write operation times out.

03The residential proxy bottleneck

Residential proxies are the leading cause of write timeouts. Because they rely on consumer home internet connections, they suffer from extreme bandwidth asymmetry. A node might download HTML at 100 Mbps but upload your POST payload at 2 Mbps. If you attempt to send a 5MB payload with a standard 10-second write timeout, the math guarantees a failure before the transfer completes.

04How DataFlirt handles it

We avoid write timeouts through intelligent payload routing. Our infrastructure inspects the Content-Length of outbound requests. Payloads under 100KB are routed normally. Payloads over 100KB destined for residential networks are automatically converted to use Transfer-Encoding: chunked, which streams the data in smaller blocks. For massive payloads, we bypass residential nodes entirely and route through high-bandwidth ISP proxies to ensure symmetric transfer speeds.

05The silent killer: half-open connections

A poorly configured scraper without a strict write timeout can suffer from connection leaks. If a proxy silently drops the connection without sending a TCP RST packet, the scraper's socket remains half-open, waiting indefinitely to write the rest of its buffer. This leads to thread pool exhaustion and eventual scraper deadlock. Always configure explicit write timeouts, even if they are generous.

// 03 — the network math

Calculating write
capacity.

Write timeouts are a function of payload size, proxy upstream bandwidth, and the hard timeout limit set on the HTTP client. DataFlirt models these constraints to dynamically route heavy payloads.

Effective Upload Time = Payload_Size / min(Client_BW, Proxy_Upstream_BW)
The binding constraint is almost always the proxy's upstream bandwidth. Network Layer Basics
TCP Send Buffer = min(SO_SNDBUF, Receiver_Window)
If the buffer fills, the write operation blocks until the timer expires. OS Socket API
DataFlirt Chunk Size = Payload_Size / Target_Chunks
Dynamically sized between 8KB and 64KB based on proxy latency. Internal Routing Logic
// 04 — socket trace

A heavy POST request
hitting a proxy bottleneck.

Trace of a scraper attempting to send a 2.4MB GraphQL payload through a residential proxy. The first attempt times out; the second succeeds using adaptive chunking.

POSTGraphQLChunked Transfer
edge.dataflirt.io — live
CAPTURED
// attempt 1: standard POST over residential proxy
req.method: "POST" req.size: 2.4MB
proxy.node: "res_IN_mumbai_491"
socket.write_start: 0ms
socket.bytes_written: 512KB ... (12000ms)
socket.bytes_written: 1.2MB ... (28000ms)
error: write ETIMEDOUT (30000ms exceeded)

// attempt 2: adaptive chunking enabled
req.transfer_encoding: "chunked"
chunk.size: 64KB
socket.write_start: 0ms
chunk.1..38: transmitted successfully
req.status: 200 OK duration: 14200ms
// 05 — failure modes

Why the socket
gives up.

Ranked by frequency across DataFlirt's heavy-payload pipelines. Proxy upstream bandwidth is the dominant factor, especially on residential networks.

PIPELINES MONITORED ·   300+ active
POST REQUESTS ·  ·  ·  ·  12M/day
UPDATED ·  ·  ·  ·  ·  ·  2026-05-19
01

Residential proxy upstream limits

85% of timeouts · Asymmetric home internet connections throttle uploads
02

Massive JSON/GraphQL payloads

60% of timeouts · Batching too many operations into a single POST
03

Server-side tarpitting

45% of timeouts · Target intentionally reads bytes slowly to tie up concurrency
04

Suboptimal TCP window sizing

30% of timeouts · OS-level buffer constraints on the scraping worker
05

Proxy connection drops

15% of timeouts · Silent drops leaving the socket half-open
// 06 — payload routing

Chunked transfers,

bypassing proxy upstream limits.

Residential proxies are notorious for asymmetric bandwidth—great download speeds, abysmal upload speeds. When a scraper tries to POST a 3MB payload through a residential node, the write operation often times out before the server receives the full body. DataFlirt mitigates this by automatically switching to chunked transfer encoding for payloads over 500KB, streaming the data in smaller, manageable blocks that keep the proxy connection alive and prevent the socket from timing out.

POST payload delivery

Live metrics from a heavy GraphQL mutation routed through our proxy infrastructure.

payload.size 2.4 MB
proxy.type residential_IN
transfer.mode chunked
chunk.size 64 KB
write.duration 14.2s
socket.status closed cleanly
pipeline.state nominal

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

Common questions about write timeouts, proxy bandwidth, and how to configure HTTP clients for heavy payloads.

Ask us directly →
What is the difference between a read timeout and a write timeout? +
A read timeout occurs when your scraper has sent the request and is waiting for the server to reply, but the server takes too long. A write timeout occurs when your scraper is trying to send the request payload to the server, but the network or proxy is too slow to transmit the bytes before the timer expires.
Why do I only see write timeouts on residential proxies? +
Residential proxies route traffic through home internet connections, which are typically asymmetric. A home user might have 500 Mbps download speed but only 10 Mbps upload speed. When you send a large POST request, you are bound by that 10 Mbps upload limit, causing the write buffer to stall.
How do I fix a write timeout in Python? +
If you are using httpx, you can specify separate timeouts: timeout=httpx.Timeout(10.0, read=30.0, write=60.0, connect=10.0). If you are using requests, the standard timeout tuple only covers connect and read. You often have to rely on OS-level socket timeouts or switch to a more modern client for granular write control.
Will increasing the timeout always solve the problem? +
No. If the proxy silently drops idle connections, or if the target server has a hard limit on how long it will wait to receive a complete request body, increasing your client-side write timeout will just make your scraper fail slower. You need to reduce the payload size or use a faster proxy.
What is server-side tarpitting? +
Tarpitting is an anti-bot technique where the server intentionally reads your incoming request payload at an excruciatingly slow rate (e.g., 1 byte per second). This forces your scraper's socket to remain open, tying up your concurrency limits until you hit a write timeout.
How does DataFlirt handle large POST requests? +
We route heavy POST requests (like massive GraphQL batches) through ISP or datacenter proxies that offer symmetric bandwidth, reserving residential nodes strictly for GET requests. If a residential node must be used, we automatically apply chunked transfer encoding to stream the payload.
$ dataflirt scope --new-project --target=write-timeout READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Scraping Errors

Read Timeout

The inverse error: waiting for the server to send data after your request is complete.

Read →
Scraping Errors

Proxy Timeout

When the proxy server itself drops the connection before the target responds.

Read →
Scraping Errors

TCP Connection Timeout

When the initial handshake fails before any data can be written or read.

Read →
Network Layer

Chunked Transfer

Streaming data in blocks to bypass proxy limits and keep slow connections alive.

Read →