Skip to main content

Data Streaming

Data streaming has moved from a niche capability to a foundational layer of modern data platforms. Organizations that once relied solely on nightly batch jobs now operate real-time pipelines that power fraud detection, operational dashboards, recommendation engines, and IoT command centers. The shift is driven not only by user expectations for immediacy but also by the maturation of technologies that make continuous, reliable stream processing practical at scale.

This section of BigDataDevPro covers the messaging systems, processing engines, and architectural patterns that constitute a real-time data stack. It explores how event-driven architectures, exactly-once semantics, and change data capture work together to deliver fresh, trustworthy data to every downstream consumer — without compromising on reliability or operational cost.

What Is Data Streaming?​

Data streaming is the practice of processing data as it arrives, rather than waiting for a bounded batch to accumulate. A stream is an unbounded sequence of events, each carrying a payload and a timestamp. Stream processing systems consume these events, apply computations — filtering, joining, aggregating, enriching — and emit results continuously.

Key concepts:

  • Event: A single record representing something that happened, such as a user click, a sensor reading, or a database change.
  • Producer: An application or system that publishes events to a stream.
  • Consumer: An application or service that reads and processes events from a stream.
  • Topic: A named stream of events, partitioned for parallelism.
  • Partition: An ordered, immutable sequence of events within a topic. Partitions enable horizontal scaling and ordering guarantees.
  • Offset: A sequential identifier assigned to each event within a partition. Consumers track offsets to manage their position.
  • Event time vs. processing time: The time an event occurred at its source versus the time it was observed by the processing system. Handling the gap between these times is central to correct stream processing.

The fundamental difference from batch processing lies in the unbounded nature of the data. Batch systems process finite datasets with known beginnings and ends; streaming systems run indefinitely, maintaining state and handling late-arriving data without explicit boundaries.

Streaming Architecture in Modern Data Platforms​

A typical streaming architecture consists of four layers that decouple data production from consumption and processing:

  • Data sources: Application databases, mobile apps, web servers, IoT devices, and third-party APIs that generate event streams.
  • Event ingestion: Connectors and lightweight agents that capture data from sources and forward it reliably to the message platform. Kafka Connect and purpose-built CDC tools operate here.
  • Message platform: A durable, partitioned log (typically Apache Kafka) that buffers events, decouples producers from consumers, and provides replay capability.
  • Stream processing: Engines like Apache Flink or Spark Structured Streaming that read from the message platform, perform stateful or stateless transformations, and write results to sinks.
  • Storage, analytics, and applications: The sinks — lakehouse tables, real-time OLAP systems, vector databases, or microservices — that serve processed data to dashboards, models, and operational APIs.

This architecture allows organizations to add new consumers without impacting producers, to backfill historical data by replaying the log, and to scale each layer independently based on throughput demands.

Apache Kafka Fundamentals​

Apache Kafka is the most widely adopted distributed event streaming platform. It functions as a central nervous system for data in motion, connecting disparate systems with a high-throughput, low-latency, and durable log abstraction.

Kafka’s architecture is built around several core components:

  • Broker: A server that runs the Kafka process, stores partition data on disk, and serves client requests. A Kafka cluster consists of multiple brokers for scalability and fault tolerance.
  • Topic: A logical category or feed name under which events are published. Topics are subdivided into partitions.
  • Partition: The unit of parallelism and ordering. Events within a partition are strictly ordered by offset and are immutable once written.
  • Producer: Writes events to a topic. Producers can specify the partition (directly or via a key-based partitioner) or let Kafka assign round-robin.
  • Consumer: Reads events from topics. Consumers pull events and track their position using offsets.
  • Consumer Group: A set of consumers that cooperatively read from a topic. Each partition is consumed by exactly one consumer within a group, enabling horizontal scaling while preserving order.
  • Offset: A unique, monotonically increasing identifier for each event within a partition. Consumers commit offsets to Kafka or an external store to record progress.
  • Replication: Partitions are replicated across multiple brokers. One broker hosts the leader replica (handling all reads and writes), while followers replicate the data and can take over if the leader fails.

Kafka is used for multiple architectural purposes: as a durable message bus for asynchronous microservice communication, as the ingestion backbone for real-time analytics pipelines, and as a buffer that decouples stream processors from source systems.

Kafka Architecture and Design Concepts​

Understanding Kafka’s design trade-offs is essential for building production-grade streaming systems.

  • Partitioning strategy: Producers can hash on a message key to ensure all events with the same key land in the same partition, preserving per-key order. Random or round-robin strategies maximize throughput but lose ordering guarantees.
  • Ordering guarantees: Kafka guarantees total order within a partition, but not across partitions. Applications that require global ordering must use a single partition — at the cost of parallelism.
  • Replication and durability: Each partition has a configurable replication factor. Kafka uses an in-sync replica (ISR) set. Data is committed only after all ISRs acknowledge receipt. This trade-off between durability and latency is controlled via acks and min.insync.replicas settings.
  • Consumer scaling: The maximum parallelism of a consumer group equals the number of partitions. Adding more consumers than partitions leaves some consumers idle. This forces careful partition planning upfront.
  • Delivery semantics: Kafka supports at-least-once semantics by default. Exactly-once requires idempotent producers and transactional APIs, which span across multiple partitions and topics.

Stream Processing Engines​

While Kafka handles storage and transport of events, dedicated stream processing engines execute the transformation logic on those streams.

Apache Flink is a stateful, distributed stream processing engine designed for low-latency, high-throughput, exactly-once applications. It treats batch as a special case of streaming (bounded streams), unifying both paradigms under a single runtime.

Key capabilities:

  • Stateful stream processing: Flink maintains application state in local RocksDB instances backed by checkpointing, enabling complex event-driven logic and aggregations over long windows.
  • Event time processing: Flink natively supports event time, watermarks to handle lateness, and customizable windowing strategies (tumbling, sliding, session).
  • Exactly-once guarantees: Through distributed snapshots (checkpoints) based on the Chandy-Lamport algorithm, Flink provides exactly-once state consistency and can integrate with sinks that support transactional output.
  • Rich operator model: Flink’s DataStream API and Table API/SQL allow expressing complex streaming topologies, including multi-way joins, pattern matching (CEP), and async I/O.

Spark Structured Streaming​

Spark Structured Streaming brings streaming to the Spark ecosystem by treating streaming DataFrames as unbounded append-only tables. Built on the Catalyst optimizer and the Tungsten engine, it benefits from Spark’s mature batch infrastructure.

Key characteristics:

  • Micro-batch processing model: Structured Streaming divides the stream into small deterministic batches (default triggers as fast as possible). A continuous processing mode exists for lower latency but with tighter semantic constraints.
  • Integration with Spark ecosystem: Streams can be joined with batch data (DataFrames), written to the same lakehouse sinks, and queried through Spark SQL. This reduces architectural fragmentation for teams already invested in Spark.
  • Output modes: Append (new rows only), update (changed rows), and complete (full result table) — matching different sink requirements.

High-level comparison:

  • Flink is purpose-built for streaming with true event-time processing and lower latencies. It is preferred for complex, stateful, low-latency pipelines.
  • Spark Structured Streaming leverages existing Spark skills and infrastructure. It excels in environments where streaming is integrated with heavy batch processing on shared lakehouse storage.

Change Data Capture (CDC)​

Change Data Capture is the technique of tracking row-level changes (inserts, updates, deletes) in operational databases and propagating them as event streams. CDC enables near-real-time synchronization between OLTP and analytical systems without costly batch dumps or dual writes.

CDC tools like Debezium read the database transaction log (MySQL binlog, PostgreSQL WAL, Oracle LogMiner) and emit structured change events to Kafka. These events contain the before and after state of each row, along with metadata about the transaction.

Common use cases:

  • Data warehouse synchronization: Keeping a warehouse or lakehouse updated with the latest state of operational tables, often to feed BI dashboards or downstream aggregations.
  • Real-time analytics: Streaming CDC events directly into Flink or Spark for immediate enrichment and aggregation, bypassing batch ETL entirely.
  • Event-driven microservices: Triggering downstream services when specific business entities change — e.g., updating a search index when a product catalog entry is modified.

Streaming Data Processing Concepts​

Production stream processing requires mastering a set of concepts that directly affect correctness and performance.

  • Windowing: Dividing an infinite stream into finite chunks for aggregation. Common types include tumbling windows (fixed-size, non-overlapping), sliding windows (fixed-size, overlapping), and session windows (activity-based boundaries).
  • Stateful processing: Maintaining information across events — counters, session state, join buffers. State must be backed up via checkpointing to survive failures.
  • Watermarks: A notion of how late the system believes it is in event time. Watermarks trigger window computations and control how long the system waits for late data before producing results.
  • Late-arriving data: Events that arrive after the watermark has passed. Handling strategies include discarding, updating previously emitted results, or routing to a dead-letter stream.
  • Checkpointing and fault tolerance: Periodic snapshots of application state stored in durable storage. On failure, the system restores from the latest checkpoint and replays events from the saved offsets. This is the mechanism behind exactly-once guarantees.
  • Fault tolerance: Beyond checkpointing, it encompasses the ability of the system to reassign work when nodes fail, without data loss and with minimal disruption.

Delivery Semantics​

Delivery semantics define the guarantees a streaming system provides about message processing in the presence of failures.

  • At-most-once: Events are processed zero or one time. No duplicates, but data loss is possible on failure. Acceptable only for tolerant metrics or non-critical telemetry.
  • At-least-once: Events are processed one or more times. No data loss, but duplicates may occur. This is the default for most production systems because it can be mitigated through idempotent sinks and deduplication logic.
  • Exactly-once: Events are processed precisely once, even in the face of failures. This is the strongest guarantee, provided by systems like Kafka Streams, Flink, and (in certain modes) Spark Structured Streaming. It requires coordination between the processing engine’s checkpointing and the sink’s transactional output.

Achieving exactly-once in practice involves idempotent producers, transactional commits across partitions, and careful sink design (e.g., writing to an Iceberg table with upsert mode that supports deduplication via primary keys). The choice between at-least-once and exactly-once is a trade-off between simplicity, performance overhead, and the business cost of duplicates.

Building Production Streaming Pipelines​

A streaming pipeline that passes a local test is not ready for production. The following operational concerns are critical:

  • Scalability: Partitions must be sized to handle peak throughput. Processing jobs must scale out elastically. Kafka consumer groups and Flink task slots should align with partition counts.
  • Monitoring and observability: End-to-end latency, consumer lag, throughput per partition, checkpoint duration, and record error rates are essential metrics. Lag spikes indicate processing bottlenecks; checkpoint failures indicate state backend issues.
  • Error handling and dead letter streams: Poison events — malformed or unexpectedly large records — must be routed to a dead letter topic for inspection, not cause the entire pipeline to halt.
  • Schema management: Event schemas evolve. A schema registry (Confluent Schema Registry, AWS Glue Schema Registry) enforces compatibility and prevents breaking changes from shutting down consumers.
  • Data quality: Validate events at the point of ingestion and during processing. Flag schema violations, null fields, and out-of-range values with alerts.
  • Backpressure: Stream processors must signal backpressure to upstream systems to prevent overwhelming slower consumers. Kafka’s pull-based consumer model naturally handles this; Flink and Spark have their own flow control mechanisms.
  • Replay capability: Kafka’s durable log allows reprocessing historical events from a specific offset. This is critical for bug fixes, model retraining, and audit requirements.
  • Security: Encrypt data in transit (TLS), authenticate clients (SASL/SCRAM, mTLS), and authorize operations (ACLs). Integrate with enterprise identity providers.

Streaming and Event-Driven Architecture​

Streaming technology is not limited to analytics. It underpins event-driven architectures where microservices communicate asynchronously via events.

  • Event-driven architecture: Services publish business events to a topic when state changes. Other services subscribe and react independently. This decouples producers from consumers, enabling independent evolution and scaling.
  • Event sourcing: Instead of storing current state, the system persists a log of all state-changing events. Current state is derived by replaying the log. This pattern, often built on Kafka, provides a complete audit trail and enables temporal queries.
  • Asynchronous communication: Streaming decouples request/response flows, allowing systems to handle traffic spikes gracefully and process work in the background.

Streaming is appropriate when systems require loose coupling, when the same event must reach multiple consumers, or when real-time data freshness directly impacts business outcomes. It is not a universal replacement for batch, but it has become essential where latency-sensitive decisions are made.

Data Streaming Learning Path​

  1. Understand streaming fundamentals: Learn the vocabulary — events, topics, partitions, offsets — and the conceptual differences from batch processing.
  2. Learn messaging concepts: Study message durability, ordering guarantees, and consumer group semantics in Kafka.
  3. Master Apache Kafka: Build a mental model of broker clusters, replication, and delivery guarantees. Operate a topic from producer to consumer.
  4. Learn stream processing: Write stateful transformations in Flink or Structured Streaming. Understand windowing, watermarks, and checkpointing.
  5. Understand CDC: Set up a pipeline that captures database changes and streams them into a lakehouse or warehouse.
  6. Learn event-time processing: Build applications that handle late-arriving data and produce correct, time-windowed results.
  7. Design production streaming systems: Integrate monitoring, schema management, error handling, and security into your pipelines.

Explore the following articles in this section:

Connection to Other Sections​

Data Foundations — The Data Foundations section provides the distributed systems knowledge, processing paradigms, and data modeling concepts essential for understanding streaming. Partitioning, serialization, and consistency models are directly applied in Kafka topic design and Flink state management.

Data Processing — The Data Processing section covers batch engines like Spark, which share infrastructure and APIs with Spark Structured Streaming. Many platforms run batch and streaming workloads side by side, using the same lakehouse storage. Understanding both paradigms is necessary for designing unified, cost-effective architectures.

Data Architecture — Streaming components must be integrated into the broader platform. The Data Architecture section addresses how Kafka, Flink, and CDC fit into medallion architectures, data mesh topologies, and cloud-native reference architectures. It provides the system-level view needed to operate streaming at enterprise scale.