What is Delta Lake?

Delta Lake is an open-source storage layer that brings reliability to data lakes. At its core, a Delta table is just a directory in object storage (like S3) containing two things: a collection of data files in Parquet format, and a _delta_log directory containing a transaction log.

The transaction log is the source of truth. It records every change made to the table — which files were added, which were removed, and what the schema looks like. When a query engine reads a Delta table, it first reads the log to determine exactly which Parquet files to scan, ignoring any files that are partially written or marked for deletion.

Cloud object stores like S3 are eventually consistent and don't support native file modifications. If a scraper crashes while writing a 5GB file, a downstream query might read corrupted, half-written data. Delta Lake provides ACID guarantees using optimistic concurrency control.

Writers create new Parquet files invisibly. Only when the write is complete do they attempt to commit a new entry to the transaction log. If two scrapers try to modify the same data simultaneously, Delta detects the conflict, allows one to succeed, and forces the other to retry. Readers are never blocked by writers.

Data lakes often turn into data swamps because anyone can write any file format with any columns. Delta Lake enforces schema on write. If a scraper tries to insert a string into a numeric price column, the transaction fails before any data is committed.

However, schemas in scraping inevitably change. Delta supports safe schema evolution. By setting a flag, you can explicitly allow new columns to be added to the table. Delta updates the metadata in the transaction log, and all historical records simply return null for the new column when queried.

We use Delta Lake as the delivery interface for enterprise clients. Instead of pushing raw JSON or CSV files that clients have to manually deduplicate, we maintain a Gold-tier Delta table in their S3 bucket. Our pipelines run MERGE operations to upsert new scraped records and update changed ones.

We run automated OPTIMIZE and VACUUM jobs weekly to compact small files and purge stale data outside the retention window, ensuring the client's Athena or Databricks queries remain lightning fast without ballooning their AWS storage bill.

Because Delta Lake doesn't immediately delete old Parquet files when records are updated, you get "time travel" out of the box. You can query the table exactly as it looked at a specific timestamp or version number.

This is invaluable for scraping pipelines. If a bad selector deployment corrupts a day's worth of pricing data, you don't need to restore from a backup. You simply run RESTORE TABLE catalog TO VERSION AS OF 142, and the transaction log reverts, instantly undoing the damage.