← Glossary / Header Anomaly Detection

What is Header Anomaly Detection?

Header anomaly detection is a passive anti-bot technique that compares your incoming HTTP headers against the known fingerprint of the browser you claim to be. If your User-Agent says Chrome 124 but your request lacks sec-ch-ua or orders the HTTP/2 pseudo-headers like a Go HTTP client, the WAF flags the mismatch. It is the cheapest, fastest way for a target to drop naive scrapers before evaluating TLS or running JavaScript challenges.

Anti-botHTTP/2WAF RulesFingerprintingNetwork Layer
// 02 — definitions

The easiest way
to get caught.

Why sending a perfectly valid HTTP request is not enough if the headers do not match the browser you are pretending to be.

Ask a DataFlirt engineer →

TL;DR

Header anomaly detection evaluates the presence, order, casing, and values of HTTP headers. Modern bot managers like Akamai and Cloudflare maintain strict dictionaries of how real browsers format requests. A single missing Accept-Language header or a Python requests default can trigger an immediate 403 Forbidden.

01Definition & structure
Header anomaly detection is the process of inspecting an incoming HTTP request to ensure its headers are structurally coherent with the client it claims to be. A WAF checks:
  • Presence: Are mandatory headers like Accept-Language or sec-ch-ua missing?
  • Order: Are headers sent in the exact sequence the advertised browser uses?
  • Values: Does the Accept-Encoding list match the browser's decompression capabilities?
  • Framing: In HTTP/2, are the pseudo-headers ordered correctly?
Any deviation from the expected dictionary results in an anomaly score increase.
02How it works in practice
When a request hits a protected edge (like Cloudflare or DataDome), the WAF parses the headers before routing the request to the origin. It extracts the User-Agent, looks up the expected header signature for that specific browser version, and compares it against the actual request. If a Python script sends a Chrome User-Agent but orders its headers alphabetically (a common library default), the WAF drops the request with a 403 Forbidden. No CAPTCHA is served because the request is provably synthetic.
03The rise of Client Hints
Historically, spoofing the User-Agent was enough. With the introduction of Client Hints (sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform), browsers now send structured metadata about their environment. WAFs heavily rely on these headers because naive scrapers often forget to include them. A request claiming to be Chrome 120+ without Client Hints is an immediate, fatal anomaly.
04How DataFlirt handles it
We eliminate header anomalies by never guessing. DataFlirt's infrastructure binds every scraping session to a verified device profile. The network layer dynamically inherits the exact header dictionary, HTTP/2 pseudo-header sequence, and Client Hints directly from that profile. When a request leaves our edge, its headers are mathematically indistinguishable from the real browser it represents, ensuring a zero-anomaly score across all major bot managers.
05Did you know?
HTTP/2 requires all header keys to be strictly lowercase. However, some older scraping libraries attempt to send Title-Case headers (e.g., User-Agent instead of user-agent) over HTTP/2 connections. WAFs flag this casing mismatch instantly, as no modern browser violates the HTTP/2 specification.
// 03 — the math

How WAFs score
header entropy.

WAFs do not just look for bad headers; they look for the absence of expected good ones. DataFlirt's request engine models these exact penalty functions to ensure our header dictionaries perfectly align with the advertised User-Agent.

Header mismatch penalty = Score = Σ (wi · anomaly_flagi)
Weights vary. Missing sec-ch-ua on a Chrome UA is a fatal penalty. Standard WAF heuristic
HTTP/2 pseudo-header order = Ph2 = client_order == browser_expected ? 0 : 1
Browsers use :method, :authority, :scheme, :path. Go uses a different sequence. Akamai Bot Manager
DataFlirt header coherence = C = headers_sent ∩ UA_dictionary
C must equal 1.0. We never send a header the advertised browser wouldn't. Internal SLO
// 04 — what the WAF sees

A Python scraper
meets Akamai BMP.

A trace of a naive Python requests script attempting to spoof a Chrome User-Agent, and the exact header anomalies that trigger the block.

HTTP/2Akamai BMPPython requests
edge.dataflirt.io — live
CAPTURED
// inbound request
user-agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36..."
accept-encoding: "gzip, deflate" // anomaly: Chrome sends br, zstd
accept-language: missing // anomaly: real browsers always send this

// HTTP/2 framing analysis
h2.pseudo_order: ":method :scheme :path :authority" // anomaly: Python/Go default
h2.header_casing: "Title-Case" // anomaly: H2 requires lowercase

// sec-ch-ua client hints
sec-ch-ua: missing // anomaly: required for Chrome > 89
sec-fetch-site: missing

// classifier evaluation
anomaly_score: 98.4
decision: DENY
response: 403 Forbidden (Reference Error)
// 05 — detection vectors

Where header spoofing
falls apart.

The most common header anomalies that trigger WAF blocks across DataFlirt's monitored targets. Fixing the User-Agent is only step one.

SAMPLE SIZE ·  ·  ·  ·    1.2M blocked reqs
WAF TYPES ·  ·  ·  ·  ·   Cloudflare, Akamai
UPDATED ·  ·  ·  ·  ·  ·  2026-05-19
01

Missing Client Hints (sec-ch-ua)

Fatal · Claiming modern Chrome without sending sec-ch-* headers.
02

HTTP/2 Pseudo-Header Order

Fatal · Library defaults vs strict Browser sequences.
03

Accept-Encoding Mismatch

High risk · Missing Brotli (br) or Zstandard (zstd) on modern UAs.
04

Header Ordering

Medium risk · Browsers send headers in a strict, predictable sequence.
05

Missing Accept-Language

Medium risk · Almost all real user traffic includes locale preferences.
// 06 — our stack

Perfect coherence,

down to the HTTP/2 frame.

Spoofing headers manually is a losing game. WAFs update their browser dictionaries weekly. DataFlirt does not hardcode header strings; our request engine dynamically inherits the exact header dictionary, ordering, and HTTP/2 pseudo-header sequence directly from the underlying browser profile assigned to the session. If the session is Chrome 124 on macOS, the network layer emits the exact byte sequence Chrome 124 on macOS would emit.

Header Coherence Engine

Live validation of a DataFlirt outbound request against the assigned browser profile.

profile.ua Chrome 124 · macOS
h2.pseudo_order :m :a :s :pchrome-match
sec-ch-ua "Chromium";v="124"injected
accept-encoding gzip, deflate, br, zstd
header.order strict-browser-seq
waf.evaluation pass

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

About header dictionaries, HTTP/2 framing, client hints, and how DataFlirt maintains perfect network-layer coherence at scale.

Ask us directly →
Why does my scraper work in Postman but fail in Python? +
Postman and Python requests send different default headers. Postman includes Accept: */* and specific Accept-Encoding values, while Python sends User-Agent: python-requests/2.x. Even if you change the UA in Python, the underlying header order and missing browser-specific headers (like sec-fetch-dest) give you away immediately.
What are HTTP/2 pseudo-headers and why do they matter? +
In HTTP/2, the request line is replaced by pseudo-headers (:method, :authority, :scheme, :path). Browsers send these in a very specific order. Standard HTTP libraries (like Go's net/http or Python's httpx) use a different order. WAFs check this order before they even read your User-Agent.
Can I just copy the headers from my browser's network tab? +
Yes, for a single request. But browsers change their headers based on the request context (e.g., sec-fetch-dest: document for HTML, sec-fetch-dest: empty for APIs). Hardcoding a single header block for all requests will eventually trigger an anomaly flag when you request an API endpoint with document-level headers.
How do sec-ch-ua (Client Hints) affect scraping? +
Google introduced Client Hints to replace the User-Agent string. If your scraper sends a Chrome User-Agent > 89 but fails to send the corresponding sec-ch-ua, sec-ch-ua-mobile, and sec-ch-ua-platform headers, it is an instant anomaly. You must send both, and they must perfectly match.
How does DataFlirt handle header updates when browsers release new versions? +
We do not maintain static lists. Our fleet runs real browser engines that generate the headers natively. When we route traffic through our HTTP/2 proxy layer, we extract the exact header state from the headless instance and apply it to the wire-level request, ensuring 100% coherence without manual patching.
Does header anomaly detection apply to mobile app scraping? +
Absolutely. Mobile APIs expect headers generated by iOS/Android networking libraries (like CFNetwork or OkHttp). Sending a mobile app User-Agent with browser-like Accept headers or desktop HTTP/2 framing is a common mistake that triggers immediate API blocks.
$ dataflirt scope --new-project --target=header-anomaly-detection READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Network Layer

HTTP Headers

The fundamental key-value pairs that carry metadata, state, and identity in every web request.

Read →
Network Layer

User-Agent String

The primary identifier browsers use to announce themselves, and the most commonly spoofed header.

Read →
Network Layer

TLS Fingerprinting

The cryptographic equivalent of header anomalies, evaluated before the HTTP layer even connects.

Read →
Anti-Scraping

Akamai Bot Manager

One of the most aggressive WAFs for enforcing strict HTTP/2 pseudo-header and client hint coherence.

Read →