What is Full-Page Scraping?

Full-page scraping captures the entire rendered DOM of a page — every element, every injected widget, every lazy-loaded section — rather than targeting individual nodes. The pipeline stores the complete document and delegates field extraction to a downstream parser.

• capture layer — headless browser navigates to URL, waits for networkidle, scrolls to trigger lazy loads
• snapshot — full DOM serialised as MHTML or raw HTML, stored to object storage
• parser layer — schema inference runs offline against the stored snapshot
• change detection — subsequent runs diff the snapshot hash; only changed pages re-parse

The separation of capture from parsing is the key architectural decision — it makes full-page pipelines resilient to front-end redesigns.

A Playwright instance navigates to the target URL through a residential proxy, waits for networkidle (all XHR and fetch requests settled), then scrolls the viewport in steps to trigger intersection-observer-gated lazy loads. Once all widgets are hydrated, it serialises the DOM and uploads the snapshot. A separate parser job picks it up from S3, applies CSS selector chains or LLM-assisted field mapping, and writes structured records to the delivery sink. The capture and parse stages run independently — a parser failure doesn't lose the raw snapshot.

The biggest operational advantage of full-page scraping is that the raw snapshot survives schema changes. When a target redesigns its DOM, the existing parser breaks — but the snapshot still exists and can be re-parsed against a new selector map without re-fetching. Drift detection compares the structural fingerprint of successive snapshots (element count distributions, heading hierarchy, data-attribute patterns). A structural delta above a configurable threshold pauses delivery and triggers a selector review, preventing silent bad data from reaching the pipeline output.

We default to full-page capture for any new target in the first two weeks. It gives us a versioned archive of what the site looked like at each point in time — invaluable when a client disputes a data point. Once the schema is stable and selector coverage is validated at above 98%, we switch high-frequency fields to targeted extraction and keep full-page as a weekly ground-truth check. Every full-page snapshot is integrity-checked on ingest: DOM element count, content-length floor, and a hash diff against the previous run.

Detection is not correlated with how much of the page you extract — it's correlated with how your client looks on the wire. A full-page scraper running real Chrome on a residential IP with a clean JA3 is far less detectable than a targeted scraper using a Go HTTP client that leaks its TLS fingerprint on the first connection. The amount of DOM you consume after getting a 200 is irrelevant to anti-bot classifiers; the fingerprint you present before the HTML loads is everything.