← Glossary / OWASP CRS Block

What is OWASP CRS Block?

OWASP CRS Block is a network-layer rejection triggered when an HTTP request violates the OWASP Core Rule Set — a standard library of signatures used by Web Application Firewalls (WAFs) like ModSecurity, Nginx, and AWS WAF. For scrapers, it usually means your payload, headers, or URL structure matched a known attack vector or a generic bot signature. It is the baseline defense layer that catches lazy scripts before they ever reach the application logic.

WAFModSecurityPayload InspectionHTTP HeadersAnomaly Scoring
// 02 — definitions

Tripped at
the firewall.

The mechanics of how generic WAF rules catch poorly constructed scraping requests before the application server even knows you exist.

Ask a DataFlirt engineer →

TL;DR

An OWASP CRS block happens when a WAF inspects your request and finds a pattern matching a known vulnerability or generic automation tool. It is rarely a sophisticated behavioral block. If you hit a CRS rule, you likely sent a malformed header, a suspicious User-Agent, or a payload that looks like an injection attack.

01Definition & structure
An OWASP CRS Block occurs when an inbound HTTP request triggers enough rules in the OWASP Core Rule Set to exceed the Web Application Firewall's anomaly threshold. The CRS is a collection of regular expressions and logic rules designed to catch common web attacks (SQLi, XSS, LFI) and protocol violations. For data extraction pipelines, CRS blocks are almost always the result of poor HTTP client configuration rather than behavioral detection.
02How anomaly scoring works
Instead of blocking immediately on a single suspicious pattern, modern WAFs use anomaly scoring. A missing Accept header might add 3 points. A generic curl User-Agent adds 5 points. If the WAF is configured to block at 5 points, the curl request is dropped instantly, while a custom script missing an Accept header might pass—unless it also triggers another minor rule.
03The Paranoia Level (PL) concept
CRS uses Paranoia Levels from 1 to 4. PL1 is the default and only blocks obvious attacks and blatant protocol violations. PL2 adds stricter regexes and header checks, often catching scrapers that don't perfectly mimic browser header order. PL3 and PL4 are extremely strict, often blocking legitimate traffic, and require extensive tuning. Most targets you scrape will run PL1 or PL2.
04How DataFlirt handles it
We treat CRS compliance as baseline table stakes. Our fetch infrastructure never relies on default HTTP client configurations. Every request is constructed with a complete, ordered header stack that matches the specific browser version advertised in the User-Agent. By ensuring absolute protocol hygiene, our requests generate an anomaly score of zero, bypassing CRS inspection entirely.
05Did you know: payload false positives
Scrapers often get blocked by CRS when submitting search forms or GraphQL queries that contain special characters (like ', ", or --). The WAF's regex engine misinterprets these characters as a SQL injection attempt (Rule 942). To avoid this, payloads must be properly URL-encoded, and sometimes query structures must be adjusted to avoid triggering generic injection signatures.
// 03 — the scoring model

How anomaly
scoring works.

Modern CRS deployments don't block on a single minor rule. They use anomaly scoring, accumulating points across multiple minor infractions until a threshold is breached.

Anomaly Score = S = Σ rule_weighti
Critical rules add 5 points, minor add 2. Default block threshold is usually 5. OWASP CRS Architecture
Paranoia Level (PL) = PL = 1 to 4
Higher PL = more aggressive rules evaluated = higher false positive rate for scrapers. WAF Configuration
DataFlirt Header Compliance = C = valid_headers / required_headers
Maintained at 1.0. Missing Accept or Accept-Language headers guarantee a PL2 block. Internal SLO
// 04 — waf inspection trace

A ModSecurity
rejection trace.

What happens when a naive scraper sends a request with a default Python requests User-Agent and a missing Accept header. The WAF scores the anomalies and drops the connection.

ModSecurityAnomaly ScoringHTTP 403
edge.dataflirt.io — live
CAPTURED
// inbound request
method: "GET /api/v1/products?q=shoes"
user_agent: "python-requests/2.28.1"
accept: missing

// CRS phase 1: request headers
rule.913100: MATCH // Found User-Agent associated with security scanner/script
score.add: 5
rule.920280: MATCH // Request Missing an Accept Header
score.add: 3

// CRS phase 2: request body/uri
rule.920320: PASS // Missing User-Agent (False, UA exists)

// evaluation
anomaly.total: 8
anomaly.threshold: 5
action: DENY
response: HTTP 403 Forbidden
// 05 — common triggers

Where the rules
catch you.

The most frequent OWASP CRS rules that trap scraping pipelines. These are almost entirely protocol and header compliance issues, not behavioral detection.

WAF DEPLOYMENTS ·  ·  ·   Millions globally
DEFAULT THRESHOLD ·  ·    5 points
UPDATED ·  ·  ·  ·  ·  ·  2026-05-19
01

Bad User-Agent (913100)

Critical · Default HTTP client strings (curl, requests, aiohttp)
02

Missing Accept Header (920280)

Warning · Lazy request construction missing standard browser headers
03

Protocol Violations (920 series)

Critical · Malformed HTTP/1.1 framing or invalid content-length
04

SQLi False Positives (942 series)

Critical · Scraping search endpoints with special characters
05

Missing Host Header (920280)

Warning · Direct IP access or poorly configured proxies
// 06 — our approach

Clean headers,

boring payloads, zero anomalies.

OWASP CRS isn't looking for human behavior; it is looking for broken HTTP clients and attack signatures. DataFlirt bypasses CRS blocks entirely by ensuring absolute protocol compliance. We don't just spoof User-Agents; we align the entire header stack, TLS fingerprint, and HTTP/2 framing to match the advertised client. When the WAF inspects the request, it sees a perfectly formed, boring browser. No anomalies, no points, no block.

WAF Compliance Trace

A DataFlirt request passing through a strict PL3 OWASP CRS deployment.

client.tls Chrome 124 signature
header.user_agent Mozilla/5.0 (Windows NT 10.0...)
header.accept text/html,application/xhtml+xml...
header.order Chromium standard
rule.913100 PASS
anomaly.score 0
waf.action ALLOW

Stay ahead of the pipeline

Data engineering
intel, weekly.

Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.

By submitting, you acknowledge our Privacy Notice.

// 07 — FAQ

Common
questions.

Common questions about WAF rules, anomaly scoring, and why your scraper is getting blocked before it even sees the target application.

Ask us directly →
What is the OWASP Core Rule Set? +
The OWASP ModSecurity Core Rule Set (CRS) is a set of generic attack detection rules for use with Web Application Firewalls. It protects web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. For scrapers, it acts as a strict bouncer checking for protocol compliance.
Why did my scraper get blocked if I wasn't attacking the site? +
CRS doesn't just block SQL injection; it blocks anomalies. If you use a default library like Python's requests, your User-Agent is flagged as a known automation tool. If you forget to include an Accept or Accept-Language header, you violate strict HTTP compliance rules. These accumulate points until you hit the block threshold.
How do I bypass an OWASP CRS block? +
Stop looking like a script. Ensure your HTTP headers exactly match what a real browser sends. This includes the User-Agent, Accept, Accept-Language, Accept-Encoding, and the exact order they appear in. Avoid sending payloads with unescaped special characters that might trigger SQLi or XSS regex filters.
Does CRS detect headless browsers? +
Generally, no. CRS operates at the network and HTTP protocol layer. It inspects headers, URIs, and payloads. It does not execute JavaScript or check navigator.webdriver. If your headless browser sends standard Chrome headers, CRS will pass it. Headless detection happens at the application or advanced bot-management layer (like Cloudflare or DataDome), not the CRS layer.
How does DataFlirt prevent CRS blocks at scale? +
We maintain strict protocol hygiene. Every request generated by our fleet uses a header profile dynamically bound to its TLS fingerprint. We never send default library User-Agents, and our payload encoders ensure search queries or form submissions are properly sanitized so they don't trip generic injection rules.
Is it legal to bypass a WAF block? +
Bypassing a WAF by sending well-formed, compliant HTTP requests to access public data is generally considered lawful, as you are simply acting like a standard web browser. However, attempting to bypass a WAF to exploit vulnerabilities (like SQLi) or access authenticated areas without permission violates the CFAA and similar statutes. We strictly access public data using compliant protocols.
$ dataflirt scope --new-project --target=owasp-crs-block READY

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.

Start a pipeline → View pricing
hello@dataflirt.com  ·  Bengaluru  ·  IST  ·  typical reply < 4h
// 08 — keep reading

Related glossary terms

Anti-Scraping

Web Application Firewall (WAF)

The infrastructure layer that executes the OWASP Core Rule Set.

Read →
Anti-Scraping

ModSecurity Rule Match

The specific engine that evaluates CRS rules against inbound HTTP traffic.

Read →
Anti-Scraping

Missing Accept Header Block

One of the most common CRS triggers for poorly configured scraping scripts.

Read →
Scraping Errors

HTTP 403 Forbidden

The standard HTTP response code returned when a CRS anomaly threshold is breached.

Read →