What is Columnar Storage?

Columnar storage organizes data on disk by column rather than by row. In a row-based system, reading the price of 10,000 products requires loading the entire row for all 10,000 products into memory. In a columnar system, the price values are stored contiguously, allowing the disk to read only the requested data.

This architecture is the foundation of modern analytical databases (OLAP) and data lakes. It is specifically designed for workloads that scan massive numbers of rows but only care about a subset of columns.

When a scraping pipeline delivers a dataset in a columnar format like Parquet, the file is divided into "row groups" (e.g., 100,000 records per group). Within each row group, the data is pivoted: all values for column A are written, followed by all values for column B.

The file footer contains metadata detailing the exact byte offsets for each column chunk, along with statistics like min/max values and null counts. When a query engine like Athena or BigQuery reads the file, it fetches the footer first, determines exactly which byte ranges contain the requested columns, and issues targeted HTTP GET requests to S3 to fetch only those bytes.

Because columnar storage groups identical data types together, it achieves extreme compression. A column of boolean values (in_stock) can be stored as a bitset. A column of repeated strings (category) can use dictionary encoding, storing the string "Electronics" once and referencing it with a 1-byte integer.

This type-specific encoding, combined with general-purpose compression algorithms like Snappy or Zstandard, routinely reduces the physical footprint of scraped data by 70–90% compared to raw JSON.

We treat columnar delivery as a first-class requirement for enterprise pipelines. While our extraction workers process data row-by-row, our delivery layer buffers these records and writes them to S3 or GCS as strongly-typed Parquet files.

We enforce schema contracts before writing, ensuring that a column defined as an integer never accidentally contains a string, which would break columnar encoding. This guarantees that when our clients point their data warehouse at our delivery buckets, the ingestion is fast, cheap, and error-free.

Many data teams start by dumping scraped JSON directly into a data lake. This is the "JSON trap." Because JSON is row-based and schema-on-read, every analytical query forces the engine to scan the entire dataset, parse the text, and extract the relevant fields on the fly.

As the dataset grows to terabytes, query costs skyrocket and performance plummets. Converting the pipeline delivery format from JSON to a columnar format is usually the single highest-ROI optimization a data engineering team can make.