Decoded, web scraping.

The complete scraping dictionary — anti-bot, proxies, pipelines, AI agents & legal risk. Written for engineers.

// full glossary index

Web Scraping Core Concepts

51 terms
Web Scraping Core → HTML Scraping Core → Data Extraction Core → Web Crawling Core → Scraping Pipeline Core → DOM Parsing Core → HTML Parsing Core → CSS Selector Core → XPath Core → robots.txt Core → Request Throttling Core → Concurrent Scraping Core → Distributed Crawling Core → Screen Scraping Key → Full-Page Scraping Key → Incremental Scraping Key → Targeted Scraping Key → Focused Crawler Key → Crawl Delay Key → Link Extraction Key → XML Parsing Key → JSON Parsing Key → Regex in Scraping Key → Attribute Extraction Key → Sitemap Crawling Key → URL Normalization Key → URL Deduplication Key → Redirect Following Key → Scraping Session Key → Stateful Scraping Key → Parallel Crawling Key → Scraping Queue Key → Job Scheduling Key → Exponential Backoff Key → Data Freshness Key → Website Technology Fingerprinting Key → Deep Web Scraping Ref → Breadth-First Crawling Ref → Depth-First Crawling Ref → Polite Crawling Ref → URL Frontier Ref → Seed URL Ref → Tag Stripping Ref → Node Selection Ref → Recursive Crawling Ref → Crawl Exclusion Ref → Canonical URL Ref → Stateless Scraping Ref → JavaScript Framework Detection Ref → Surface Web Ref → Noindex Tag Ref →

HTTP and Network Layer

56 terms
HTTP Request Core → HTTP Response Core → HTTP Status Code Core → HTTP Headers Core → User-Agent String Core → Cookie Management Core → TLS Fingerprinting Core → Rate Limiting Core → Pagination Detection Core → AJAX Request Interception Core → REST API Scraping Core → GET Request Key → POST Request Key → Referer Header Key → Session Cookie Key → Cookie Jar Key → Bearer Token Key → API Key Authentication Key → OAuth Scraping Key → HTTPS Interception Key → TLS Handshake Key → Connection Pooling Key → Request Timeout Key → Response Body Key → Content-Type Header Key → Gzip Decompression Key → Throttle Headers Key → X-RateLimit Header Key → Retry-After Header Key → XHR Monitoring Key → Fetch API Interception Key → API Response Pagination Key → Cursor-Based Pagination Key → Offset Pagination Key → Page Token Pagination Key → Network Traffic Analysis Key → IP Whitelisting Key → IP Blacklisting Key → Request Batching Key → Response Caching Key → Accept-Language Header Ref → Persistent Cookie Ref → SSL Certificate Validation Ref → Transfer-Encoding Ref → Chunked Transfer Ref → WebSocket Scraping Ref → DNS Resolution Ref → Content Delivery Network Ref → Multiplexed Requests (HTTP/2) Ref → Cache Hit Rate Ref → Conditional GET (ETag) Ref → HTTP Pipelining Ref → Server-Sent Events Ref → SOAP API Parsing Ref → DNS Caching Ref → Edge Caching Ref →

Anti-Bot and Bypass Techniques

62 terms
CAPTCHA Core → reCAPTCHA v2 Core → reCAPTCHA v3 Core → hCaptcha Core → Cloudflare Bot Management Core → Cloudflare Challenge Page Core → Turnstile CAPTCHA Core → AI CAPTCHA Solver Core → Bot Detection Core → Browser Fingerprinting Core → Mouse Movement Simulation Core → Datacenter IP Detection Core → Bot Score Core → Anti-Detect Browser Core → Browser Profile Spoofing Core → User-Agent Rotation Core → Header Rotation Core → Stealth Mode Browsing Core → Headless Browser Detection Core → WebDriver Flag Removal Core → Playwright Stealth Core → Puppeteer Extra Stealth Core → Scraping Browser Core → Real Browser Rendering Core → JavaScript Challenge Core → Managed Challenge Core → Fake 200 OK Response Core → FunCaptcha Key → Image CAPTCHA Key → Canvas Fingerprinting Key → WebGL Fingerprinting Key → Keystroke Dynamics Key → Behavioral Biometrics Key → Honeypot Fields Key → Honeypot Links Key → ASN Blocking Key → Residential IP Detection Key → Risk Score Key → Cookie Rotation Key → Navigator Object Spoofing Key → Browser Integrity Check Key → Soft Block Key → Tarpit Response Key → Poisoned Data Response Key → Decoy Content Injection Key → Dynamic Class Name Obfuscation Key → Browser Token Validation Key → Obfuscated JavaScript Execution Required Key → Interaction Gate Key → Audio CAPTCHA Ref → CAPTCHA Farm Ref → Font Fingerprinting Ref → Audio Context Fingerprinting Ref → Screen Resolution Fingerprinting Ref → Timezone Detection Ref → Plugin Enumeration Ref → Hardware Concurrency Detection Ref → Device Memory Detection Ref → CSS-Based Text Hiding Ref → Client Puzzle Challenge Ref → Canary Token in HTML Ref → Language Detection Ref →

Proxies and IP Management

48 terms
Proxy Server Core → Residential Proxy Core → Datacenter Proxy Core → ISP Proxy Core → Mobile Proxy Core → Rotating Proxy Core → Sticky Proxy Session Core → Proxy Pool Core → Proxy Rotation Core → Geo-Targeted Proxy Core → IP Ban Core → IP Reputation Core → Shared Proxy Key → Dedicated Proxy Key → Backconnect Proxy Key → SOCKS5 Proxy Key → HTTP Proxy Key → HTTPS Proxy Key → Anonymous Proxy Key → Elite Proxy Key → City-Level Targeting Key → ASN Targeting Key → Peer-to-Peer Proxy Network Key → Proxy Authentication Key → Proxy Latency Key → Proxy Throughput Key → Bandwidth Throttling Key → Proxy Failover Key → Proxy Health Check Key → Concurrent Proxy Sessions Key → Proxy API Key → Subnet Blocking Key → IP Cooldown Period Key → IP Warming Key → Semi-Dedicated Proxy Ref → Transparent Proxy Ref → Carrier-Level Proxy Ref → Exit Node Ref → Proxy Chaining Ref → Proxy Scoring Ref → Proxy Benchmarking Ref → Proxy Dashboard Ref → Proxy Gateway Ref → Reverse Proxy Ref → Forward Proxy Ref → CIDR Block Ref → IPv4 Proxy Ref → IPv6 Proxy Ref →

Browsers and Rendering

42 terms
Headless Browser Core → Chromium Core → Playwright Core → Puppeteer Core → Selenium WebDriver Core → WebDriver Core → JavaScript Rendering Core → Dynamic Content Rendering Core → Cookie Consent Bypass Core → Selenium Grid Key → Firefox Headless Key → Shadow DOM Key → Browser Context Key → Incognito Mode Scraping Key → Browser Pool Key → Browser Instance Management Key → Remote Browser Key → Cloud Browser Key → Browser Session Management Key → Page Load Timing Key → DOMContentLoaded Event Key → Network Idle Detection Key → Screenshot Scraping Key → Visual Diff Detection Key → Scroll Emulation Key → Click Emulation Key → Form Filling Automation Key → Pop-Up Dismissal Key → Auto-Scroll Script Key → Resource Blocking Key → Selective Resource Loading Key → DOM Ready Time Key → WebKit Engine Ref → Blink Engine Ref → Web Components Ref → MutationObserver Ref → Multi-Tab Scraping Ref → Frame Switching Ref → Alert Handling Ref → Service Worker Interception Ref → WebdriverIO Ref → Page Object Model Ref →

Scraping Infrastructure and DevOps

50 terms
Containerized Scraper Core → Docker Scraping Image Core → Scrapy Framework Core → Scrapy Spider Core → Scrapy-Playwright Core → Scraper Orchestration Core → Scraper Monitoring Core → Worker Concurrency Core → Async Scraping Core → Kubernetes Scraping Cluster Key → Horizontal Scaling Key → Auto-Scaling Key → Task Queue Key → Message Queue Key → Redis Queue Key → Celery Worker Key → Scrapy Middleware Key → Scrapy Item Pipeline Key → Scrapy Splash Key → Cron Job Scheduling Key → Scraper Versioning Key → Scraper Testing Key → Mock HTTP Response Key → Scraper Logging Key → Error Rate Tracking Key → Retry Queue Key → Idempotent Scraping Key → Deduplication Queue Key → Scraper State Persistence Key → Cold Start Latency (Serverless) Key → Thread Pool Size Key → Event Loop Performance Key → Non-Blocking I/O Key → Coroutine-Based Scraping Key → Vertical Scaling Ref → Load Balancer Ref → RabbitMQ Ref → Apache Kafka Ref → Scrapy Feed Export Ref → Scrapy Settings Ref → Scrapy Downloader Ref → Apache Airflow Ref → Structured Logging Ref → Log Aggregation Ref → Grafana Dashboard Ref → Prometheus Metrics Ref → Dead Letter Queue Ref → Distributed Locking Ref → Autoscale Response Time Ref → Scraper Warm-Up Time Ref →

Scraper Maintenance and Change Detection

14 terms
Scraper Breakage Core → Selector Rot Core → Layout Change Detection Core → DOM Change Monitoring Core → Auto-Healing Scraper Core → Self-Healing Selector Core → AI-Assisted Selector Repair Core → Visual Regression Detection Key → Site Diff Monitoring Key → Field Availability Monitoring Key → Schema Change Alert Key → Scraper Migration Ref → Scraper Deprecation Ref → Scraper Runbook Ref →

Scraping Errors and Failure Codes

66 terms
HTTP 403 Forbidden Core → HTTP 429 Too Many Requests Core → HTTP 503 Service Unavailable Core → HTTP 520 Unknown Error (Cloudflare) Core → Empty Response Body Core → Proxy Connection Error Core → Proxy Timeout Core → Selector Not Found Error Core → StaleElementReferenceException Core → NoSuchElementException Core → TimeoutException (WebDriver) Core → WebDriverException Core → HTTP 407 Proxy Authentication Required Key → HTTP 502 Bad Gateway Key → HTTP 504 Gateway Timeout Key → HTTP 521 Web Server Down (Cloudflare) Key → HTTP 522 Connection Timed Out (Cloudflare) Key → HTTP 523 Origin Unreachable (Cloudflare) Key → HTTP 524 A Timeout Occurred (Cloudflare) Key → HTTP 401 Unauthorized Key → HTTP 400 Bad Request Key → HTTP 408 Request Timeout Key → HTTP 500 Internal Server Error Key → Partial Response Error Key → Truncated HTML Response Key → Response Encoding Error Key → Malformed JSON Response Key → Encoding Mismatch Key → Unexpected Redirect Chain Key → Redirect Loop Error Key → Max Redirects Exceeded Key → DNS Resolution Failure Key → Connection Refused Error Key → Connection Reset by Peer Key → TCP Connection Timeout Key → Socket Timeout Key → Read Timeout Key → Proxy Authentication Failure Key → Memory Leak in Scraper Key → Scraper Out-of-Memory Error Key → Infinite Pagination Loop Key → ElementClickInterceptedException Key → SessionNotCreatedException Key → NavigationTimedOut (Browser) Key → ERR_EMPTY_RESPONSE Key → ERR_CONNECTION_CLOSED Key → ERR_NAME_NOT_RESOLVED Key → ERR_TOO_MANY_REDIRECTS Key → Page Crash (Headless Browser) Key → Browser Context Destroyed Error Key → Scraper Deadlock Key → HTTP 525 SSL Handshake Failed (Cloudflare) Ref → HTTP 526 Invalid SSL Certificate (Cloudflare) Ref → HTTP 530 Origin DNS Error (Cloudflare) Ref → HTTP 451 Unavailable For Legal Reasons Ref → Malformed XML Response Ref → Content-Length Mismatch Ref → DNS NXDOMAIN Error Ref → DNS Timeout Ref → Write Timeout Ref → SSL Certificate Expired Error Ref → SSL Hostname Mismatch Ref → SOCKS5 Negotiation Failure Ref → ERR_CERT_AUTHORITY_INVALID Ref → HTTP 410 Gone Ref → HTTP 413 Payload Too Large Ref →

Anti-Scraping Platforms and Errors

62 terms
Cloudflare Ray ID Core → Cloudflare Error 1006 (Access Denied) Core → Cloudflare Error 1010 (Bad Browser Signature) Core → Cloudflare Error 1015 (Rate Limited) Core → Cloudflare Error 1020 (Access Denied by Firewall Rule) Core → Cloudflare Turnstile Failure Core → Cloudflare Firewall Rule Match Core → Cloudflare Bot Score Threshold Core → Cloudflare Super Bot Fight Mode Core → Cloudflare Under Attack Mode Core → Akamai Bot Manager Core → Akamai Sensor Data Block Core → PerimeterX / HUMAN Challenge Core → DataDome 403 Block Core → Imperva Block Page Core → Kasada Kami Challenge Core → Web Application Firewall (WAF) Core → Access Denied Page Core → Bot Detected Page Core → Interactive Challenge Core → Bad Bot Signature Match Core → Request Fingerprint Block Core → Header Anomaly Detection Core → Cloudflare Waiting Room Key → Cloudflare Workers Block Key → Cloudflare Verified Bot Allowlist Key → Cloudflare Error 1002 (Restricted) Key → Akamai Pragma Header Detection Key → Akamai Reference Number Error Key → PerimeterX Cookie Validation Key → DataDome Score Threshold Key → DataDome Device Check API Key → Imperva UTM Cookie Check Key → Imperva Reese84 Cookie Key → Shape Security Token Rotation Key → F5 Distributed Cloud Bot Block Key → Radware Bot Manager Block Key → Kasada TLS Fingerprint Check Key → Arkose Labs FunCaptcha Trigger Key → GeeTest CAPTCHA Block Key → AWS WAF Block Response Key → Google Cloud Armor Block Key → Nginx Rate Limit Response Key → Fail2Ban IP Block Key → ModSecurity Rule Match Key → Geo-Block Error Key → Country-Level Block Key → Missing Accept Header Block Key → Suspicious Referrer Block Key → Cookie Validation Failure Key → CSRF Token Mismatch Key → Anti-Automation Token Expiry Key → Session Invalidation Key → Signed Cookie Validation Key → HMAC Token Validation Failure Key → Cloudflare Error 1000 (DNS Resolution Error) Ref → Cloudflare Error 1016 (Origin DNS Error) Ref → Azure Front Door WAF Block Ref → Fastly WAF Block Ref → Apache mod_evasive Block Ref → OWASP CRS Block Ref → Cloudflare Error 1018 (Compute Zone Unavailable) Ref →

Scraping Performance

42 terms
Requests Per Second (RPS) Core → Data Extraction Speed Core → Scraper Latency Core → Success Rate Core → Block Rate Core → Retry Rate Core → Error Rate (Scraping) Core → Data Yield Rate Core → Parse Success Rate Core → Pages Per Minute Key → Time to First Byte (TTFB) Key → Page Load Time (Scraping Context) Key → Request Queue Depth Key → Connection Reuse Rate Key → Keep-Alive Efficiency Key → Image Blocking (Performance) Key → Parser Performance Key → LXML Parsing Speed Key → Memory Usage Per Worker Key → CPU Usage Per Scrape Job Key → Network Bandwidth Per Job Key → Cost Per Scraped Record Key → Scraper Efficiency Score Key → Field Extraction Completeness Key → Null Field Rate Key → Crawl Coverage Rate Key → Network Round-Trip Time Ref → DNS Lookup Time Ref → TCP Handshake Time Ref → TLS Negotiation Time Ref → Resource Deduplication Ref → Payload Size Optimization Ref → Font Blocking Ref → Ad Script Blocking Ref → JSON Decode Speed Ref → Disk I/O in Scraping Ref → Cost Per 1000 Requests (CPM) Ref → Bandwidth Consumption Tracking Ref → Egress Cost Optimization Ref → Cloud Compute Cost Per Scrape Ref → URL Discovery Rate Ref → Duplicate URL Rate Ref →

Authentication and Access Patterns

21 terms
Login-Wall Scraping Core → Form-Based Login Automation Core → Multi-Factor Authentication Handling Key → SSO (Single Sign-On) Scraping Key → JWT Token Extraction Key → Token Refresh Automation Key → API Token Rotation Key → Cookie-Based Auth Scraping Key → Account Warm-Up Key → Scraping Account Management Key → Paywall Redirect Key → SAML Flow Scraping Ref → Basic Auth Scraping Ref → Magic Link Authentication Ref → Passkey / WebAuthn Wall Ref → Age Verification Wall Ref → Terms of Service Gate Ref → Digest Authentication Ref → NTLM Authentication Ref → Client Certificate Authentication Ref → Session Hijacking (Research Context) Ref →

Site Structure and Content Patterns

35 terms
Paginated Listing Pages Core → Product Variant Scraping Core → Search Results Scraping Core → JSON-LD Extraction Core → Category Hierarchy Scraping Key → Faceted Search Scraping Key → Filter Parameter Scraping Key → Autocomplete API Scraping Key → Modal Content Scraping Key → Tab Content Scraping Key → Microdata Extraction Key → Open Graph Tag Extraction Key → Schema.org Markup Key → Structured Data Markup Key → Mobile-First Site Scraping Key → Geo-Specific Content Scraping Key → A/B Test Variant Detection Key → Personalized Content Detection Key → User-Specific Content Detection Key → Nested JSON Flattening Key → Boilerplate Removal Key → HTML Tag Stripping Key → Accordion Content Scraping Ref → Breadcrumb Extraction Ref → AMP Page Scraping Ref → PWA Scraping Ref → Responsive Breakpoint Handling Ref → Array Field Expansion Ref → HTML Entity Decoding Ref → Truncated Field Detection Ref → Cross-Field Validation Ref → Derived Field Computation Ref → Tooltip Content Scraping Ref → Hreflang Tag Extraction Ref → Markdown Stripping Ref →

Data Output Delivery and Formatting

25 terms
Output Field Mapping Key → Date Format Normalization Key → Currency Normalization Key → Data Delivery Frequency Key → Delta File Delivery Key → Diff-Based Data Delivery Key → Data Snapshot Key → Data Feed Subscription Key → Price String Parsing Key → Rating Extraction Key → Review Text Cleaning Key → CSV Storage Key → JSON Lines Format Key → S3 Data Delivery Key → Webhook Data Push Key → Data as a Service (DaaS) Key → Managed Data Feed Key → Custom Extraction Schema Key → Data Type Casting Ref → Unit Standardization Ref → Encoding Normalization Ref → Whitespace Normalization Ref → NDJSON Ref → Data Serialization Ref → SFTP Data Delivery Ref →

Data Cleaning and Transformation

29 terms
Deduplication Logic Core → Null Handling Core → Data Quality Core → Fuzzy Deduplication Key → Record Linkage Key → Entity Resolution Key → URL Canonicalization (Post-Scrape) Key → Noise Filtering Key → Data Wrangling Key → Data Normalization Key → Data Profiling Key → Data Accuracy Key → Data Completeness Key → Data Consistency Key → Data Timeliness Key → Data Validity Key → Schema Drift Detection Key → Data Downtime Key → Address Standardization Ref → Phone Number Normalization Ref → Email Validation Ref → Referential Integrity Check Ref → Missing Value Imputation Ref → Outlier Detection Ref → Data Uniqueness Ref → Great Expectations Ref → Data Unit Testing Ref → Stop Word Removal Ref → Soda Core Ref →

Data Engineering and Pipelines

58 terms
ETL Pipeline Core → ELT Pipeline Core → Data Ingestion Core → Real-Time Data Pipeline Core → Batch Processing Key → Stream Processing Key → Data Lake Key → Data Warehouse Key → Data Lakehouse Key → Apache Parquet Key → Data Schema Key → Upsert Operation Key → Pipeline Orchestration Key → Data Latency Key → Micro-Batch Processing Ref → Data Mart Ref → Raw Data Zone Ref → Bronze Layer Ref → Silver Layer Ref → Gold Layer Ref → Delta Lake Ref → Apache Iceberg Ref → Apache Avro Ref → Schema Registry Ref → Schema Evolution Ref → Data Catalog Ref → Data Lineage Ref → Data Provenance Ref → Data Versioning Ref → Change Data Capture (CDC) Ref → Apache Spark Ref → PySpark Ref → dbt (Data Build Tool) Ref → SQL Transformation Ref → Data Denormalization Ref → Data Partitioning Ref → Data Sharding Ref → Columnar Storage Ref → Data Compression Ref → Data Retention Policy Ref → DAG (Directed Acyclic Graph) Ref → Backfilling Ref → Data Reconciliation Ref → Data Audit Ref → Checksum Validation Ref → Pipeline Observability Ref → Data Contract Ref → Reverse ETL Ref → Data Activation Ref → Apache ORC Ref → Slowly Changing Dimension Ref → Apache Flink Ref → dbt Model Ref → Star Schema Ref → Snowflake Schema Ref → Fact Table Ref → Dimension Table Ref → Data Archiving Ref →

Databases and Storage

25 terms
Elasticsearch Key → Object Storage (S3) Key → PostgreSQL Ref → MongoDB Ref → Redis Ref → BigQuery Ref → Snowflake (Database) Ref → ClickHouse Ref → DuckDB Ref → Full-Text Search Ref → Query Optimization Ref → Upsert (INSERT ON CONFLICT) Ref → Bulk Insert Ref → Blob Storage Ref → DynamoDB Ref → TimescaleDB Ref → Vector Database Ref → Inverted Index Ref → Read Replica Ref → Materialized View Ref → Window Function Ref → CTE (Common Table Expression) Ref → JSONB Column Ref → Soft Delete Ref → Data Compaction Ref →

Data Governance and Compliance

14 terms
GDPR Compliance Key → Data Ownership Ref → Data Classification Ref → Data Access Control Ref → Audit Logging Ref → CCPA Compliance Ref → Master Data Management Ref → Column-Level Lineage Ref → Row-Level Security Ref → Column Masking Ref → Role-Based Access (RBAC) Ref → Attribute-Based Access (ABAC) Ref → Data Stewardship Ref → Data Sovereignty Ref →

AI and LLM in Scraping

28 terms
AI Scraping Agent Core → LLM-Powered Crawler Core → Named Entity Recognition (NER) Key → Information Extraction Key → Few-Shot Extraction Key → OCR (Optical Character Recognition) Key → Document AI Key → Multimodal Scraping Key → Sentiment Analysis Key → Relation Extraction Ref → Text Classification Ref → Zero-Shot Classification Ref → RAG (Retrieval-Augmented Generation) Ref → Semantic Search Ref → Vision-Language Model Ref → Synthetic Data Generation Ref → Fine-Tuning Data Preparation Ref → Opinion Mining Ref → Aspect-Based Sentiment Ref → Anomaly Detection Ref → Embedding Generation Ref → Data Augmentation Ref → Feature Engineering Ref → Label Studio Ref → Annotation Pipeline Ref → Topic Modeling (LDA) Ref → Text Vectorization Ref → Time Series Forecasting Ref →

Business Intelligence and Data-Led Growth

3 terms
Alternative Data Core → Web Data as a Signal Core → Data Mesh Ref →

SERP and Search Engine Scraping

1 terms
SERP API Core →

Mobile and App Scraping

7 terms
Mobile API Interception Key → Mobile User-Agent Spoofing Key → Mobile Emulation Key → Device Emulation in Chrome Key → APK Reverse Engineering (Legal Context) Ref → In-App Pricing Extraction Ref → Deep Link Extraction Ref →

Scraping Business and Commercial Models

5 terms
Data Enrichment API Key → Data Reselling Ref → IP Geolocation API Ref → Company Enrichment API Ref → Data Embargo Period Ref →

Developer Tools and Libraries

24 terms
BeautifulSoup Core → lxml Core → Apify SDK Core → Parsel Key → httpx Key → aiohttp Key → requests-html Key → Cheerio (Node.js) Key → Mitmproxy Key → Charles Proxy Key → Scrapy-Redis Key → MechanicalSoup Ref → Axios (Node.js) Ref → Colly (Go) Ref → Jsoup (Java) Ref → Puppeteer Sharp (.NET) Ref → curl Ref → Wget Ref → Fiddler Ref → Webhook Ref → REST API Data Delivery Ref → Postman (API Testing) Ref → HTTPie Ref → OpenAPI Spec Ref →
$ dataflirt pipeline --new --target=your-site READY

Know the terms.
Own the data.

20-minute scoping call. Pilot dataset within the week. Production within two. We scope, build, and operate the extraction pipeline.

Start a pipeline → View pricing
hello@dataflirt.com · Bengaluru · IST · typical reply < 4h