Data Engineering: Design a Pipeline

Interviewer

Design a real-time analytics pipeline. We need to ingest 10K events/second, process them with 1 minute latency, and power a dashboard.

Candidate

I'll break this into stages: Ingestion: Use Kafka as the event buffer. 10K events/second is modest for Kafka - 3 partitions on 3 brokers would handle it with replication. Processing: Kafka Streams or Apache Flink for the stream processing. Aggregate events into minute-level windows: count, sum, average, p95 latency, etc. Flink gives exactly-once semantics and handles backpressure automatically. Storage: Write aggregated results to a time-series database like ClickHouse or TimescaleDB. These are optimized for range scans and aggregation queries - perfect for dashboards. Serving: The dashboard queries ClickHouse directly. For sub-second queries, add a Redis cache of the latest aggregate for each metric. Refresh the cache every 60 seconds. Monitoring: Track Kafka consumer lag. If lag grows beyond 1 minute, alert. Also track Flink checkpoint duration - long checkpoints indicate backpressure.

Interviewer

How would you handle schema evolution? Events might add or remove fields over time.

Candidate

Use Avro or Protobuf with a Schema Registry. Producers register the schema with each message version. Consumers can read old messages because the registry stores all versions. In Flink, handle unknown fields gracefully - ignore them initially, then add support in a new version of the processing job with a savepoint restart. For ClickHouse, use a wide-column table or a JSON column for flexible fields that change frequently.