What is Batch Processing?

Batch processing in the context of web scraping means executing a pipeline over a known, finite list of targets at a specific time. A batch job has a distinct start and end. It typically involves three phases: Discovery (finding all URLs to scrape), Fetch & Extract (downloading HTML and parsing fields), and Load (writing the final dataset to storage). Because the dataset is bounded, you can calculate completion percentages, guarantee deduplication, and validate the entire dataset's schema before delivering it to the client.

A scheduler (like Apache Airflow or a cron job) triggers the pipeline at a set interval — say, 02:00 UTC daily. The orchestrator spins up a cluster of worker nodes. A central queue distributes the URLs. The workers fetch the pages, extract the data, and write intermediate results to a temporary store. Once the queue is empty, a final aggregation step deduplicates the records, runs schema validation, and writes the output as a partitioned Parquet file to an S3 bucket. The workers are then spun down to save compute costs.

Batch jobs must be idempotent. If a job processing 5 million URLs crashes at 4.9 million, you cannot afford to start over — both for compute costs and because hitting the target site with another 4.9 million requests risks an IP ban. Idempotency is achieved by maintaining a persistent state of successfully processed URLs (often in Redis) and ensuring that writing the same extracted record twice does not corrupt the final dataset.

We treat batch pipelines as mission-critical ETL workloads. Our orchestrator dynamically scales worker concurrency based on the target's response times and our SLA window. If a target site slows down, we don't just blindly push more requests and cause a DoS; we throttle back and alert our on-call engineers if the SLA margin drops below 15%. Every batch delivery is atomic — clients never see partial or mid-run data.

The most common mistake junior engineers make when writing batch scrapers is accumulating the extracted data in memory (e.g., appending to a Python list) and writing it to disk only at the end of the script. This works for 10,000 records but causes an Out-Of-Memory (OOM) crash at 1,000,000 records. Production batch jobs stream their output to disk or a database continuously, keeping the worker's memory footprint flat regardless of the batch size.