Module2
STREAM PROCESSING ARCHITECTURES
Architectural Building Blocks of Streaming Systems
Streaming data systems are designed with modular, loosely-coupled components that work
together to process continuous, high-velocity data. Each component plays a crucial role in
enabling real-time analytics and decision-making.
Producers: These are the originators of streaming data. Examples include:
IoT sensors (temperature, motion, etc.)
Web applications generating logs or user activity
Financial systems emitting transactions
Social media platforms
Producers write data to message brokers or directly to stream processors. They ensure that data
enters the pipeline in a time-sensitive, structured format.
Message Brokers / Transport Layer: This middleware transports data from producers to
consumers or processors. It provides buffering, ordering, delivery guarantees, and scalability.
Kafka: High-throughput, fault-tolerant log system
RabbitMQ: Lightweight message broker with support for complex routing
Amazon Kinesis / Azure Event Hubs: Managed cloud solutions
These brokers decouple systems to enable independent scaling and fault isolation.
Stream Processors: These are engines that process, transform, and analyze the data in motion.
Filtering & Transformation: Convert and clean data
Aggregations: Count, average, and summarize
Stateful Processing: Manage windows, joins, or user sessions
Examples include Apache Flink (stateful, low-latency), Spark Streaming (micro-batching), and
Storm (real-time DAGs).
Storage Systems: They persist raw or processed data for querying, backup, or batch analytics.
Short-Term: Redis, Kafka log, in-memory cache
Long-Term: HDFS, Cassandra, Amazon S3, time-series DBs
This storage allows for checkpointing, reprocessing, and integration with data lakes.
Consumers / Sinks: These use the processed output for actionable insights or downstream
applications.
Dashboards (Grafana, Kibana)
Machine Learning Pipelines
Alerting Systems
Data Warehouses and APIs
They ensure the results of streaming computations are accessible and usable in real-time.
Architecture Overview of Streaming Systems
A typical streaming system architecture resembles a pipeline flowing from data sources to
insights. The key layers are:
Data Ingestion Layer: Captures real-time data from external sources like APIs, sensors, logs,
etc., and formats it into readable streams.
Messaging Layer: This acts as the central nervous system for data flow. Kafka or similar
systems provide durable, replayable message queues and topics.
Stream Processing Engine: Performs core logic such as transformations, filtering, joining, and
windowed computations. These engines ensure event-time correctness, scalability, and
resilience.
Storage Layer: Stores data at various stages — raw input, intermediate states, or final outputs.
Cloud-native options and distributed file systems are commonly used.
Serving Layer: Interfaces that expose results to external users, applications, or business
systems. REST APIs, web dashboards, and notification systems fall into this layer.
These layers operate in a distributed fashion, with orchestration tools handling deployment and
monitoring.
Interaction with the System Architecture
Interactions between components in a streaming architecture are asynchronous and message-
driven, promoting scalability and fault tolerance.
Data Flow: Producers emit data → Message Broker queues it → Stream Processor reads
it → Output is stored/served.
Partitioning and Parallelism: Topics or data streams are divided into partitions to
enable parallel processing.
Backpressure Management: Systems use mechanisms to handle load spikes without
failure.
State Sharing: In stateful operations, processors may use shared or distributed state
stores.
Resilience: If a processor fails, brokers retain the data until it is reprocessed, ensuring no
loss.
These interactions are often implemented via RESTful APIs, RPCs, and message protocols like
Avro or Protobuf.
InfoSphere Streams Architecture
IBM InfoSphere Streams is a distributed computing platform for processing high-volume, high-
velocity data.
Key Components:
Streams Processing Language (SPL): Used to define operators and data flow.
Operators: Atomic units that perform tasks like filtering, parsing, or statistical analysis.
Toolkits: Libraries for machine learning, geospatial analytics, etc.
Processing Elements (PEs): Compiled blocks of logic that run distributed across nodes.
Support Services:
Streams Studio: An IDE to design, test, and deploy stream applications.
Resource Manager: Allocates and monitors system resources.
Metrics & Monitoring Services: Track performance, latency, and failures.
Streams Console: A dashboard for managing the system.
InfoSphere Streams supports dynamic scaling, operator migration, and real-time debugging. It is
enterprise-grade, fault-tolerant, and integrates well with existing IBM data solutions.
Functional and Non-Functional Design Principles and Patterns
Functional Design Principles:
Modular Composition: Break processing logic into reusable, composable units.
Declarative Programming: Define what to compute rather than how, promoting clarity.
Event Time Handling: Use event-time semantics and watermarking for accurate time-
based processing.
Windowing: Slice streams into fixed, sliding, or session-based windows.
Stateful Processing: Use keyed state, state stores, and check pointing.
Non-Functional Design Principles:
Latency Optimization: Process data near the source, use in-memory buffers, and reduce
serialization overhead.
Scalability: Employ partitioned topics, shard-aware logic, and stateless design.
Fault Tolerance: Enable check pointing, replay logs, and stateless restarts.
Observability: Incorporate logs, distributed tracing, and real-time metrics.
Security: Include encryption, access control, and audit trails.
Design Patterns for Streaming:
ETL Pipeline Pattern: Continuous extraction, transformation, and loading into
downstream stores.
Broadcast Pattern: Distribute the same data to multiple parallel consumers.
Sessionization Pattern: Group user activities into sessions for personalized insights.
Late Arrival Pattern: Buffer or delay processing to accommodate out-of-order or
delayed events.
Together, these principles and patterns ensure robustness, agility, and reliability in modern
stream processing systems.