Data Engineering Fundamentals

Batch vs streaming, Kafka, Spark, Airflow, data modeling, star schema vs data vault

Technical Reference & Key Concepts

Batch Processing

  • MapReduce: Split, process, combine. Good for large-scale offline processing.
  • Spark: In-memory DAG execution. Faster than MapReduce for iterative algorithms.
  • Airflow: DAG-based orchestration. Schedule, monitor, retry pipelines.

Streaming

  • Kafka: Durable log, partitioned, replicated. At-least-once by default.
  • Kafka Streams / Flink: Stateful stream processing. Exactly-once semantics available.
  • Kinesis: AWS-managed. Auto-scaling shards. Good for AWS-native stacks.

Data Modeling

  • Star Schema: Fact + dimension tables. Good for OLAP / reporting.
  • Data Vault: Hubs, Links, Satellites. Auditable, flexible, handles schema changes well.
  • Lakehouse: Data lake + warehouse capabilities (Delta Lake, Iceberg). ACID on object storage.