Batch vs streaming
- Batch — process a bounded chunk of data on a schedule (hourly/daily). Simple, high-throughput, cost-efficient; latency is the schedule interval. The default for analytics, training data, and reports.
- Streaming — process events continuously as they arrive, with sub-second-to-seconds latency. Necessary for real-time needs: fraud detection at transaction time, live dashboards, real-time features. More complex.
Choose by the latency requirement: if a delay of hours is fine, batch; if you need to act on events as they happen, stream.
Big-data processing: Spark
When data is too big for one machine, Spark processes it in parallel across a cluster. Concepts that come up:
- Partitioning — data is split across workers; good partitioning = balanced parallel work.
- Shuffles — operations that move data across the network (joins, groupBy) are the expensive part; minimize them.
- Data skew — one partition far larger than others → a straggler task that bottlenecks the whole job. A top Spark performance bug; fix with salting, broadcast joins for small tables, or repartitioning.
- Broadcast vs shuffle join — broadcasting a small table to every worker avoids a costly shuffle; the go-to optimization when one side of a join is small.
Streaming systems
- Kafka — the durable event log / backbone that decouples producers from consumers (also how CDC and event pipelines move data).
- Stream processing (Flink, Spark Structured Streaming) — windowing (tumbling/sliding), event-time vs processing-time, watermarks for late data, and exactly-once semantics (hard — usually you get at-least-once + idempotent consumers).
- CDC (Change Data Capture) — stream a database's changes (via Debezium/Kafka) to keep downstream systems in sync in near-real-time, instead of batch reloads.
For AI
Batch builds training/eval datasets and offline features; streaming powers real-time features (the fraud model needs the user's activity ) and low-latency data for online inference. Feature freshness is often a streaming problem.