What is Web Scraping?

Web scraping is programmatic data extraction from websites. Every scraper, from a 20-line script to a distributed system, has three layers:

• Fetch — retrieve the page. Plain HTTP client (httpx, requests) or headless browser (Playwright) when JavaScript rendering is required. This layer absorbs all anti-bot friction.
• Parse — extract fields from the response. CSS selectors or XPath against HTML, key paths against JSON APIs, regex as a last resort.
• Store — validate against a schema and write records to their destination: files, object storage, a database, or a delivery feed.

The layers fail independently. A fetch failure is visible (4xx, 5xx, challenge page). A parse failure is often silent — the request succeeds, the selector misses, and the field is quietly empty.

A production run starts with a URL manifest — usually produced by a crawler. For each URL: select a proxy from the pool, send the request with a coherent header and TLS profile, check the response for challenge pages or decoy content, parse the fields, validate the record, write it. Failures go to a retry queue with backoff and a different exit IP.

The run ends with a reconciliation pass: records written vs URLs attempted, yield per field, cost per record. Web scraping at scale is less about the extraction code and more about this accounting — it's how you know the dataset you delivered is actually complete.

Most commercially valuable targets sit behind Cloudflare, Akamai, DataDome, or PerimeterX. These systems classify traffic before serving content, using signals across layers:

• Network — IP reputation, ASN type (datacenter vs residential), request rate per IP
• Protocol — TLS fingerprints (JA3/JA4), HTTP/2 frame ordering, header coherence
• Behaviour — JavaScript challenge results, mouse and timing entropy, navigation patterns

A scraper that gets the parse layer perfect but ignores these signals extracts nothing — the page it parses is a 403 or a challenge interstitial. Access engineering is half the discipline, and it's the half that changes monthly.

We separate access from extraction. The fetch layer is shared infrastructure — proxy pools, fingerprint management, challenge handling — maintained centrally, so an anti-bot escalation on one target gets fixed once, not per-pipeline. Extraction logic is per-target and schema-first: we define the output schema before writing a selector, and every record is validated against it at write time.

Our monitoring watches yield per field, not just request success. fetch.success_rate at 99% means nothing if price is empty on a third of records. When field yield drops, the pipeline alerts and quarantines the affected records — clients get a late dataset over a wrong one, every time.

The parsing code in a mature web scraping pipeline is typically under 10% of the codebase. The other 90% is access (proxy rotation, fingerprints, challenge handling), reliability (retries, backoff, dead-letter queues), and quality (schema validation, anomaly detection, reconciliation).

This is why "we'll just use BeautifulSoup" projects stall: the tutorial covers the 10%. It's also why LLM-based extraction hasn't eliminated the discipline — a model can replace your selectors, but it can't get you past Cloudflare, and at scale it costs more per page than the proxy that does. The hard part was never reading the HTML. It's getting the HTML, every day, without being noticed.