What is Data Provenance?

Data provenance is the comprehensive metadata trail that documents the origin, context, and transformation history of a piece of data. In a scraping pipeline, it means every extracted record carries a payload detailing exactly how it was acquired.

A robust provenance block typically includes:

• source_url — The exact endpoint fetched, including query parameters.
• fetch_timestamp — The ISO 8601 timestamp of the HTTP response.
• raw_payload_hash — A SHA-256 hash of the raw HTML/JSON before parsing.
• pipeline_context — Scraper version, schema version, and worker ID.
• network_context — The proxy exit IP, ASN, and geographic region used.

Provenance is injected at the fetch layer. When the HTTP client receives a 200 OK, it immediately hashes the body and records the timestamp and proxy details. This metadata object is passed alongside the raw HTML to the extraction layer. The parser extracts the business fields (price, title, SKU) and appends the metadata object as a nested JSON field (e.g., _meta). When the record is written to the data warehouse, this metadata is flattened into hidden columns or stored as a JSONB blob for future auditing.

Provenance is your primary defense in data disputes. If a target site alleges you bypassed authentication to scrape private data, your provenance trail—specifically the raw payload hash linked to cold-stored HTML—proves the data was served publicly to an unauthenticated GET request. Without this audit trail, you have no cryptographic proof of the site's state at the time of extraction, leaving you vulnerable to claims of ToS violations or copyright infringement.

We enforce a strict data contract on every pipeline: no record is delivered without a complete _df_meta block. We store the raw HTML of every fetch in compressed S3 cold storage for 30 days, keyed by the raw_html_hash present in the delivered record. If a client spots an anomaly in their Snowflake instance, they can query the hash, and we can instantly retrieve the exact HTML document and scraper version that produced that specific row.

The biggest risk of ignoring provenance happens during aggregation. Imagine a pipeline scraping pricing from 10 different regional subdomains. If a product's price suddenly drops by 50% in your database, you need to know: did the global price drop, or did the scraper accidentally route the request through an Indian proxy and extract the INR price instead of USD? Without network and source URL provenance attached to that specific row, debugging this takes days instead of minutes.