← Glossary / Transfer-Encoding

What is Transfer-Encoding?

Transfer-Encoding is an HTTP header that specifies the form of encoding used to safely transfer the payload body to the client. In scraping, it almost always means chunked, where the server sends data in a series of sized blocks rather than computing a final Content-Length. If your HTTP client doesn't handle chunked boundaries correctly, your pipeline will silently truncate JSON payloads or hang indefinitely waiting for a termination frame.

HTTP/1.1Chunked TransferNetwork LayerPayload ParsingStreaming
// 02 — definitions

Streaming
the payload.

How servers deliver large or dynamically generated responses without knowing the total size upfront, and why it breaks naive parsers.

Ask a DataFlirt engineer →

TL;DR

Transfer-Encoding (specifically chunked) allows a server to stream a response in discrete blocks, each prefixed by its hex size. It replaces the Content-Length header. Scraping pipelines must reassemble these chunks before parsing; failure to do so results in malformed JSON errors or truncated HTML DOMs.

01Definition & structure
Transfer-Encoding is an HTTP/1.1 header indicating that the payload has been transformed for safe transport. In modern web scraping, this almost exclusively means Transfer-Encoding: chunked. Instead of sending a Content-Length header, the server sends the body in discrete chunks. Each chunk starts with its size in hexadecimal, followed by a CRLF, the data itself, and another CRLF. The transmission ends with a chunk of size zero.
02How it works in practice
When a scraper hits an API that queries a massive database, the server doesn't wait for the query to finish to calculate the total byte size. It streams the JSON array as rows are returned. Your HTTP client must read the hex size, read exactly that many bytes, strip the framing, and append the data to a buffer. Only when the 0\r\n\r\n sequence is received is the payload considered complete and ready for standard JSON parsing.
03The Content-Length conflict
A response cannot legally contain both a Content-Length and a Transfer-Encoding header. If both are present, the HTTP specification dictates that Transfer-Encoding wins and the length header must be ignored. Malicious actors use this ambiguity for HTTP Request Smuggling, but in scraping, it usually just means a poorly configured upstream proxy is leaking headers.
04How DataFlirt handles it
We don't buffer large chunked responses in memory. Our extraction workers pipe the raw TCP stream directly into iterative JSON parsers (like ijson). This allows us to extract, validate, and write records to S3 while the HTTP request is still technically downloading. It reduces our memory footprint by over 80% on large catalog feeds and ensures partial data delivery even if the target server crashes mid-stream.
05Did you know?
HTTP/2 does not support Transfer-Encoding: chunked. The HTTP/2 protocol has its own built-in binary framing layer (DATA frames) that handles streaming natively. If your scraper receives a chunked header over an HTTP/2 connection, it means a reverse proxy (like Cloudflare or AWS ALB) down-translated the connection to HTTP/1.1 before delivering it to your client.
// 03 — the framing

How chunked
frames are sized.

The protocol overhead of chunked transfer is minimal, but the memory implications for a scraper buffering the whole response are massive. DataFlirt calculates buffer limits based on chunk velocity.

Chunk overhead = O = hex_length_bytes + 4
Each chunk adds the hex string size plus two CRLF sequences. RFC 7230
Total payload size = S = Σ chunk_sizei
The true size is only known after the terminal zero-chunk is received. HTTP/1.1 Spec
DataFlirt stream threshold = T = buffer_size / chunk_velocity
If T exceeds 500ms, we switch to disk-backed streaming to prevent OOM. Internal SLO
// 04 — wire trace

Reassembling a
chunked response.

A raw TCP trace of a chunked HTTP/1.1 response from a dynamic pricing API. Notice the hex size prefixes and the terminal zero-chunk.

HTTP/1.1chunkedTCP trace
edge.dataflirt.io — live
CAPTURED
// Response headers
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive

// Chunk 1 (25 bytes = 0x19)
19\r\n
{"status":"ok","prices":[\r\n

// Chunk 2 (31 bytes = 0x1F)
1F\r\n
{"id":101,"price":45.99},\r\n

// Chunk 3 (20 bytes = 0x14)
14\r\n
{"id":102,"price":...

// Terminal chunk
0\r\n
\r\n
// Client parser status
parser.state: reassembled
payload.bytes: 76
// 05 — failure modes

Where chunked
transfers break.

Ranked by frequency of occurrence in custom scraping scripts hitting dynamic APIs. Most failures stem from treating streaming data as a static block.

SAMPLE SIZE ·  ·  ·  ·    1.2M API requests
WINDOW ·  ·  ·  ·  ·  ·   30d trailing
UPDATED ·  ·  ·  ·  ·  ·  2026-05-19
01

Premature connection close

% of failures · TCP drop before the 0-chunk arrives
02

OOM on large payloads

% of failures · Buffering instead of streaming to disk
03

Malformed JSON

% of failures · Parsing before reassembly is complete
04

HTTP/2 down-translation

% of failures · Proxy stripping headers incorrectly
05

Trailing header parsing

% of failures · Failing to read headers after the 0-chunk
// 06 — streaming architecture

Don't buffer the ocean,

stream the drops.

When scraping 500MB JSON feeds or endless NDJSON streams, waiting for the terminal zero-chunk to begin parsing is an architectural failure. DataFlirt's extraction layer uses streaming JSON parsers that yield records as soon as the relevant bytes cross the wire. If the connection drops mid-transfer, we still deliver the records extracted up to that point, rather than throwing a fatal exception and losing an hour of compute.

stream-parser.config

Configuration for a worker handling a massive chunked NDJSON feed.

transfer.mode chunked
parser.engine ijson.parsestreaming
buffer.max_memory 64MB
buffer.overflow spill-to-disk
yield.behavior on-record-complete
error.incomplete yield-partials
pipeline.status active

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

About chunked encoding, HTTP protocol conflicts, streaming parsers, and how DataFlirt handles massive dynamic payloads.

Ask us directly →
Why do servers use chunked encoding instead of Content-Length? +
Servers use chunked encoding when the total size of the response isn't known at the time the headers are sent. This happens with dynamically generated content, database streaming, or large file generation. It allows the server to start sending bytes immediately rather than buffering the entire response in memory to calculate the Content-Length.
Can a response have both Content-Length and Transfer-Encoding? +
No. The HTTP/1.1 specification explicitly forbids this. If a message is received with both, the Transfer-Encoding overrides the Content-Length. Sending both is a common technique in HTTP Request Smuggling attacks, so modern WAFs and proxies will usually reject or sanitize such responses.
How does HTTP/2 handle chunked data? +
It doesn't. HTTP/2 uses its own binary framing mechanism (DATA frames) to stream content, making Transfer-Encoding: chunked obsolete. If you see a chunked header in an HTTP/2 response, it means a proxy or load balancer in the chain down-translated the HTTP/2 connection to HTTP/1.1 before it reached your client.
Why does my Python requests script hang on chunked responses? +
By default, requests.get() attempts to buffer the entire response into memory before returning. If the server is streaming data slowly (or tarpitting you), the script will hang. You need to use stream=True and iterate over the response using iter_content() or iter_lines() to process chunks as they arrive.
How does DataFlirt handle connection drops during a chunked transfer? +
Because we use streaming parsers, we extract and validate records continuously as chunks arrive. If the TCP connection drops before the terminal zero-chunk, we don't discard the data. We deliver the successfully parsed records, flag the job as a partial success, and attempt to resume via range requests if the target server supports them.
Is chunked encoding used for anti-bot detection? +
Yes, as a form of tarpitting. A WAF might accept your request but respond with Transfer-Encoding: chunked, sending one byte every 10 seconds. Naive scrapers will hold the connection open indefinitely, exhausting their thread pools. We enforce strict read timeouts and chunk velocity minimums to detect and sever these tarpit connections.
$ dataflirt scope --new-project --target=transfer-encoding READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Network Layer

Chunked Transfer

The specific encoding mechanism that breaks payloads into sized blocks.

Read →
Scraping Errors

Content-Length Mismatch

What happens when static size declarations conflict with actual payload bytes.

Read →
Network Layer

HTTP Pipelining

Sending multiple requests without waiting for responses, often complicating chunked parsing.

Read →
Anti-Bot Bypass

Tarpit Response

How WAFs use slow chunked transfers to exhaust your scraper's thread pool.

Read →