System Design: Data Processing & Pipelines
Learn how large systems move, transform, and process data reliably at scale.
Batch vs. Stream Processing
Data pipelines process data either in scheduled batches or continuously as a stream. The right choice depends on how quickly downstream consumers need fresh results.
Batch Processing
Batch jobs process large volumes of accumulated data on a fixed schedule, such as hourly or nightly.
- Throughput: optimized for processing large datasets efficiently, not for low latency
- Tools: Hadoop, Spark, Airflow are common batch orchestration tools
- Tip: use batch when near-real-time results are not required
Stream Processing
Stream processing handles data continuously as individual events arrive, producing low-latency results.
- Latency: results are available seconds after an event occurs
- Tools: Kafka Streams, Flink, and Spark Streaming are common choices
- Tip: use streaming when the business needs near-real-time insight
ETL and ELT
ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) describe the order in which raw data is moved and reshaped before it becomes usable for analytics or downstream services.
ETL vs ELT
ETL transforms data before loading it into a warehouse; ELT loads raw data first and transforms it later using the warehouse's own compute.
- ETL: transformation happens in a separate processing layer before load
- ELT: transformation happens inside the destination system, often a cloud data warehouse
- Tip: ELT is common with modern warehouses like Snowflake or BigQuery that have cheap compute
Reliability: Queues, Backpressure, and Idempotency
Production pipelines must tolerate failures, slow consumers, and retries without corrupting or duplicating data.
Building Reliable Pipelines
Message queues, backpressure handling, and idempotent processing keep pipelines correct under failure and load.
- Message queues: decouple stages so a slow or failed consumer does not block producers
- Backpressure: downstream stages signal upstream stages to slow down when overwhelmed
- Idempotency: designing steps so re-processing the same event does not create duplicate side effects
- Tip: always design pipeline steps assuming any step can be retried at least once
Knowledge Check
1. What is the main difference between batch processing and stream processing?
2. What does ETL stand for?
3. Why are message queues commonly used between pipeline stages?
4. What is backpressure in a data pipeline?
5. Which property describes a pipeline that produces the same result even if a step is retried after a failure?