Streaming¶

Data that is processed as it arrives instead of in nightly batches: events on a log, changes read from a database, windows that close when a watermark says they are done. The hard parts are not the speed. They are ordering, late data, duplicates, and knowing how far behind you are.

flowchart LR
  SRC[(Sources:<br/>apps, devices, database logs)] --> LOG[Durable log<br/>Kafka, Kinesis, Pub/Sub, Event Hubs]
  LOG --> PROC[Stream processing<br/>windows, joins, watermarks]
  LOG --> SINK[Sinks:<br/>lake tables, warehouse, search]
  PROC --> SINK
  LAG{{Consumer lag, late data, duplicates}} -.watch.- LOG
  LAG -.watch.- PROC

The shape of a streaming system. A durable log decouples producers from consumers, processing turns events into results, and three things (lag, lateness and duplicates) decide whether the output can be trusted.

Learn¶

• Batch vs streaming - event time, watermarks, and when streaming is not worth it
• Queues vs streams - ordering, partitions, replay, and why a stream is a log
• Stream processing: windows and watermarks - when a window is finished, and what to do with late data
• Change data capture - turning a database's own log into a stream of changes
• Eventual consistency - why two readers can briefly disagree
• Idempotency explained - the property that makes at-least-once delivery safe

Compare¶

• Messaging and queues - Kafka, Kinesis, Pub/Sub, Event Hubs, SQS
• ETL and orchestration - where streaming jobs sit next to batch ones

Reference¶

• Architecture pattern: streaming analytics - Lambda vs Kappa, and the lakehouse hybrid
• Architecture pattern: CDC into a lakehouse - Debezium, DMS or Datastream into Delta or Iceberg
• Architecture pattern: event-driven architecture
• Troubleshooting: Kafka consumer lag - reading lag, skew, rebalances and offset resets, run on a real broker
• Troubleshooting: failing and late pipelines - triage, retries and backfills
• Troubleshooting: replication lag - the same "how far behind" question for a database replica

Build¶

• Build a streaming pipeline - Kafka into a Parquet lake, event-time windows, a watermark and a reconciliation
• Change data capture with Debezium into a lakehouse - PostgreSQL changes into Iceberg, with a replay test
• Build an Iceberg lakehouse locally - the table format the streams land in

Certify¶

• Confluent Certified Developer for Apache Kafka
• Confluent Certified Administrator for Apache Kafka
• AWS Data Engineer Associate (DEA-C01) - Kinesis and MSK streaming
• GCP Professional Data Engineer - Pub/Sub and Dataflow
• Azure Fabric Data Engineer (DP-700) - Eventstreams and Spark structured streaming

Related topics: Data engineering for the batch side of the same platform, and Databases for the stores that streams are read from and written to.