Module 3
METHODS TO STREAM DATA PROCESSING
Algorithms for Data Analysis in Streaming Systems (25 Marks)
Streaming data systems are designed to process large volumes of real-time data with strict
constraints on latency, memory, and processing power. To meet these challenges, they utilize
sophisticated algorithms and architectural principles. This section explores key algorithms and
strategies used to analyze streaming data effectively and efficiently, ensuring both speed and
accuracy.
Accepting Constraints and Relaxing
Streaming systems must handle high-velocity data while balancing resource limitations and
performance:
Approximate Computation: Since storing and processing all data is impractical,
algorithms like Count-Min Sketch, Bloom Filters, and HyperLogLog provide
approximate results using compact data structures.
Relaxed Consistency Models: Strong consistency is often traded for eventual
consistency or snapshot isolation, increasing throughput.
Memory-Aware Algorithms: Designed to work under tight memory constraints by
discarding less relevant data or summarizing data streams.
Backpressure Mechanisms: Ensure stability by applying flow control to prevent data
overflow when downstream components are overwhelmed.
Fault Resilience via Relaxation: Systems are engineered to tolerate partial failures
through replay logs or micro-batching, providing "good enough" results until full
recovery.
These techniques strike a balance between speed, reliability, and resource constraints.
Sliding Window and Tumbling Window Timing Methods
Time windowing is essential in grouping real-time data for aggregation and analysis:
Tumbling Windows: Fixed-length windows with no overlap (e.g., every 5 minutes).
Used in fixed-interval summarization like transaction counts per minute.
Sliding Windows: Overlapping windows (e.g., 5-minute windows sliding every 1
minute). Capture trends and fluctuations in near real time.
Session Windows: Defined by user activity, close after a timeout period. Useful in user
behavior analysis.
Hopping Windows: Jump forward by fixed intervals with possible overlap.
Global Windows: Span the full dataset and are suitable when the data has natural
boundaries.
Time windows allow operators to perform real-time joins, aggregations, and anomaly detection.
Summarization Techniques
Streaming summarization helps compress data without sacrificing analytical value:
Reservoir Sampling: Maintains a representative sample of items from a continuous
stream.
Sketch-Based Techniques: Probabilistic structures (e.g., HyperLogLog for distinct
counts, Count-Min Sketch for frequency approximation) enable fast, low-memory
analytics.
Histograms & Quantiles: Summarize data distributions over time.
Incremental and Online Algorithms: Models continuously updated using new data
(e.g., online k-means, stochastic gradient descent).
Rolling Aggregations: Enable continuous computation of statistics like sum, mean, min,
max, or standard deviation over moving windows.
Principal Component Analysis (PCA): Can be adapted for real-time streams to reduce
dimensionality.
These methods ensure the system delivers actionable insights without storing every detail.
Communication Pattern Techniques
Effective communication patterns support scalability and distributed operation:
Publish-Subscribe: Decouples producers and consumers; suitable for broadcasting data
(e.g., Kafka topics).
Fan-out / Fan-in Patterns: One-to-many and many-to-one flows for replicating or
aggregating data.
Broadcasting / Multicasting: Enables wide distribution of common streams.
Pipelining: Passes data through multiple sequential transformations, ensuring modular
processing.
Shuffle Grouping: Distributes data evenly across workers to balance load.
Event Sourcing & CQRS: Separates write and read models to improve efficiency and
scaling.
Backpressure & Throttling: Help regulate data flow, avoiding overload in critical parts
of the pipeline.
Each pattern is chosen based on latency tolerance, fault tolerance, and throughput requirements.
Protocols to Send the Data to Clients
Delivering processed data efficiently is key in streaming pipelines:
WebSockets: Enable bidirectional, persistent connections—ideal for interactive
dashboards and alerts.
gRPC with HTTP/2: Binary streaming protocol offering low-latency communication
and multiplexed streams.
Server-Sent Events (SSE): Lightweight for unidirectional updates, suitable for browser
clients.
MQTT: A lightweight pub-sub protocol, often used in IoT environments for telemetry
data.
Kafka REST Proxy: HTTP interface to produce and consume Kafka topics.
CoAP (Constrained Application Protocol): Designed for constrained devices and lossy
networks.
AMQP / STOMP: Messaging protocols used in message brokers for reliable delivery.
These protocols ensure timely and reliable delivery of results to front-end systems or other
services.
Filtering the Streams
Filtering reduces noise, improves efficiency, and prioritizes important data:
Predicate Filtering: Simple conditions such as value > threshold or type =
'error'.
Pattern Matching: Detect sequences or anomalies using regex or finite automata.
Semantic Filtering: Uses content-based rules to assess importance.
Bloom Filters: Efficiently test for set membership in massive datasets.
Noise Filtering Algorithms: Techniques like Kalman or Gaussian filters reduce
unwanted variations in signal data.
Time-Based Filtering: Ignores old or stale data (e.g., TTL windows).
Machine Learning Filters: Use classifiers to detect relevant vs. irrelevant data streams
(e.g., spam detection in logs).
Smart filtering ensures system performance while maintaining data relevance.