What is Data Lineage?

Data lineage is the comprehensive map of a dataset's lifecycle. It records the origin of the data, what happens to it, and where it moves over time. In a modern data stack, lineage is typically represented as a Directed Acyclic Graph (DAG) where nodes are datasets or tables, and edges are the transformation jobs that move data between them.

A robust lineage system tracks:

• Origin: The exact source URL, API endpoint, or raw file.
• Transformations: Type coercions, string manipulations, and joins.
• Movement: Which systems the data passed through (e.g., Kafka → S3 → Snowflake).
• State: The schema version and scraper version active at the time of capture.

In a scraping context, lineage starts the moment an HTTP response is received. The raw HTML is saved to an object store, and a unique job_id is generated. As the HTML is parsed, the extraction script appends metadata to the structured record. When the record is loaded into a data warehouse, tools like dbt take over, tracking every SQL transformation applied to the raw data to produce the final analytical tables. If a dashboard shows a sudden spike in prices, the data engineer follows the lineage graph backward from the dashboard, to the dbt model, to the raw table, and finally to the exact HTML payload to see if the target site changed its layout.

Table-level lineage is a high-level view: it tells you that table_B is built using data from table_A. This is useful for impact analysis (e.g., "If I drop table_A, what breaks?").

Column-level lineage is granular: it tells you that table_B.discounted_price is derived specifically from table_A.raw_price minus table_A.discount_amount. For scraping pipelines, column-level lineage is critical because schema drift usually affects individual fields (like a changed CSS class for a price tag) rather than entire pages.

We build lineage into the pipeline architecture from day one. Every extraction job generates a unique run ID. The raw HTTP response is gzipped and stored in S3, keyed by that run ID. The extracted JSON records include a metadata block containing the run ID, the schema version, and the scraper version. When we deliver data to your warehouse, this metadata is preserved. If you ever question a data point, you have the cryptographic proof and the raw source material to audit our extraction logic.

Beyond debugging, data lineage is a regulatory necessity. Under frameworks like GDPR and CCPA, organizations must be able to demonstrate where personal data originated and how it propagated through their systems. If a user requests deletion, you must trace their data through every downstream table and cache. Without automated lineage, compliance audits become manual, error-prone forensic investigations.