← Glossary / Missing Accept Header Block

What is Missing Accept Header Block?

A missing Accept header block is a zero-cost network filter used by Web Application Firewalls (WAFs) to drop HTTP requests that fail to declare what content types they can process. Real browsers always send a complex, highly specific Accept string on navigation requests. Naive scraping scripts using default HTTP libraries often omit it entirely or send a generic wildcard. For anti-bot systems, it's the cheapest way to identify and drop a script before spending compute on TLS or JavaScript fingerprinting.

WAF RulesHTTP HeadersBot DetectionNetwork LayerZero-Cost Filter
// 02 — definitions

The cheapest
bot filter.

Why edge networks check your headers before they even look at your IP address or TLS signature.

Ask a DataFlirt engineer →

TL;DR

A missing or generic Accept header is a dead giveaway that a request originated from a script, not a browser. WAFs like Cloudflare and Akamai use this as a first-line defense, returning a 403 Forbidden or 406 Not Acceptable without ever running a JS challenge. Fixing it requires matching the exact header string of your target browser.

01Definition & structure

A missing Accept header block occurs when a server or Web Application Firewall (WAF) rejects an HTTP request because it lacks a valid Accept header. The Accept header is part of content negotiation, telling the server what MIME types the client can process.

Real browsers send highly specific strings. For example, Chrome navigating to a webpage sends: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7. Default HTTP libraries (like Python's requests) often send */* or omit the header entirely, making them trivial to detect and block.

02How it works in practice

When a request hits an edge network like Cloudflare or Akamai, the WAF evaluates the headers before routing the request to the origin server. It runs a series of regex checks against the header block. If the User-Agent claims to be a modern browser but the Accept header is missing or generic, the WAF increments the request's anomaly score.

Because this check requires almost zero CPU cycles, it is used as a first-line defense to shed low-effort bot traffic before engaging more expensive defenses like JavaScript challenges or CAPTCHAs.

03The default library trap

Most scraping tutorials start with basic HTTP clients. By default, Python's requests sends Accept: */*. Go's net/http sends Accept-Encoding: gzip but often omits Accept. Node's axios sends Accept: application/json, text/plain, */*.

None of these match the complex, weighted strings sent by real browsers. If you are scraping a protected HTML endpoint using these defaults, you are broadcasting your identity as a script. You must explicitly override the headers to match your target browser profile.

04How DataFlirt handles it

We don't rely on static header dictionaries that go stale. Our proxy infrastructure dynamically binds the HTTP headers to the TLS fingerprint of the session. If a worker is assigned a Chrome 124 profile, the proxy automatically injects the exact Accept, Accept-Language, and Sec-Ch-Ua headers that Chrome 124 generates natively.

This ensures perfect coherence across the network layer, bypassing zero-cost WAF filters and ensuring our pipelines spend their time extracting data, not fighting 403 errors.

05The HTTP/2 pseudo-header complication

In HTTP/1.1, header order technically didn't matter. In HTTP/2, it absolutely does. Browsers send pseudo-headers (:method, :authority, :scheme, :path) in a specific sequence, followed by standard headers like Accept.

If you spoof the perfect Chrome Accept string but your HTTP/2 client sends the pseudo-headers in the wrong order, strict WAFs will still block you. The string must be correct, and the framing must match the engine.

// 03 — the header math

How WAFs score
header anomalies.

WAFs don't just check for presence; they calculate the distance between your headers and the expected baseline for your declared User-Agent.

Anomaly Score = S = w1(Missing) + w2(Order) + w3(UA Mismatch)
A score > 0.5 typically triggers an immediate block at the edge. Standard WAF heuristics
Chrome Navigation Accept = text/html,application/xhtml+xml...
The exact string changes by browser version and request type (fetch vs document). Chromium source
DataFlirt Header Coherence = C = Accept ∩ User-Agent ∩ Sec-Ch-Ua
Must equal 1.0 to pass strict WAFs. Mismatches flag the session. DataFlirt internal SLO
// 04 — waf edge trace

Dropping a naive
Python script.

A trace from an edge WAF evaluating an inbound request from a default Python requests library.

WAF RulesetHTTP/1.1Python/3.10
edge.dataflirt.io — live
CAPTURED
// inbound request
method: GET /products/12345
user-agent: python-requests/2.28.1
accept: */* // wildcard anomaly

// waf evaluation
rule.check: missing_browser_accept
rule.match: true // wildcard accept on navigation
rule.check: ua_anomaly
rule.match: true // non-browser UA

// action
action: BLOCK
response: 403 Forbidden
reason: "Request blocked by security rules"
// 05 — header failure modes

Where header
spoofing fails.

Just adding an Accept header isn't enough. WAFs look for coherence across the entire request bundle. These are the most common header-related block triggers.

WAF BLOCKS ·  ·  ·  ·  ·  12M/day
TOP CAUSE ·  ·  ·  ·  ·   Header Mismatch
01

Missing Accept entirely

98% block rate · Immediate drop by most CDNs
02

Generic wildcard (*/*)

85% block rate · Flags as non-browser script
03

Mismatch with User-Agent

70% block rate · Firefox UA with Chrome Accept
04

Incorrect header order

60% block rate · HTTP/2 pseudo-header sequence
05

Missing Sec-Fetch-*

45% block rate · Modern browser metadata absent
// 06 — our stack

Headers bound to TLS,

never hardcoded strings.

DataFlirt doesn't use static header dictionaries. Our edge proxy dynamically generates the Accept header, User-Agent, and Sec-Ch-Ua bundle directly from the TLS fingerprint profile assigned to the session. If the session is routed as Chrome 124 on macOS, the headers are byte-for-byte identical to a real Chrome 124 on macOS. This coherence is what bypasses zero-cost WAF filters and prevents early pipeline termination.

Header Coherence Check

Live validation of an outbound request from DataFlirt's proxy.

profile.id macOS-Chrome-124
tls.ja3 771,4865-4866...ok
header.user_agent Mozilla/5.0 (Macintosh...
header.accept text/html,application/xhtml+xml...ok
header.sec_ch_ua "Chromium";v="124"...
coherence.score 1.0pass
waf.action PASS

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

Common questions about Accept headers, WAF filtering, and how to structure requests to avoid zero-cost blocks.

Ask us directly →
What exactly is the Accept header? +
The Accept header is an HTTP request header that tells the server what content types (MIME types) the client can understand. For example, a browser requesting a webpage sends a complex string indicating it prefers HTML, but can also handle XML and various image formats. APIs typically expect application/json.
Why do WAFs block missing Accept headers? +
It's a zero-cost filter. Evaluating JavaScript challenges or TLS fingerprints requires compute. Checking if a string exists in the HTTP header block is practically free. Since every legitimate browser sends an Accept header, dropping requests without one is a highly accurate, low-cost way to shed bot traffic.
Can I just hardcode a Chrome Accept header in my script? +
Yes, but it's only a partial fix. WAFs check for coherence. If you send a Chrome Accept header but your User-Agent says Firefox, or your TLS handshake looks like Python's urllib, the anomaly score will still trigger a block. The entire request bundle must match.
How does HTTP/2 change header validation? +
In HTTP/2, headers are compressed using HPACK, and the order of pseudo-headers (like :method, :path, :authority) matters. Browsers send these in a very specific order. If your HTTP client sends the right Accept string but in the wrong HTTP/2 frame order, you will be flagged.
How does DataFlirt manage header updates? +
We run an automated fleet profiling system that captures real browser traffic daily. When Chrome releases a new version and alters its default Sec-Ch-Ua or Accept strings, our proxy profiles are updated automatically. Clients don't need to manage header dictionaries; the infrastructure handles it.
What HTTP status code indicates an Accept header block? +
It varies by WAF. Cloudflare often returns a 403 Forbidden. Some servers return a 406 Not Acceptable. In aggressive configurations, the WAF might just drop the TCP connection entirely (Connection Reset by Peer) to save bandwidth.
$ dataflirt scope --new-project --target=missing-accept-header-block READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Network Layer

HTTP Headers

The metadata key-value pairs that dictate how clients and servers communicate.

Read →
Network Layer

User-Agent String

The declared identity of the client, which must perfectly match the Accept header.

Read →
Network Layer

TLS Fingerprinting

The network-layer signature that WAFs check immediately after validating your headers.

Read →
Anti-Scraping

Web Application Firewall (WAF)

The edge security systems that implement missing header blocks to shed bot traffic.

Read →