Module 4
STREAM ANALYTICS
Stream Analytics: Data Pre-processing, Transformation, Modeling, and
Evaluation
Stream analytics focuses on real-time processing and analysis of continuously flowing data, such
as logs, sensor readings, transactions, and social media updates. Due to the dynamic and high-
speed nature of streaming data, traditional batch processing techniques fall short. Hence, stream
analytics relies on lightweight, incremental algorithms for both data preparation and modeling.
Data Pre-processing and Transformation
Effective data analysis begins with preparing the raw stream data. Since streaming systems deal
with noisy, unbounded, and high-velocity data, pre-processing and transformation are essential
for ensuring quality and relevance.
The Mining Process in Streaming Context
Unlike batch mining where data is static and accessible, stream mining must deal with:
One-pass processing (no revisiting data).
Real-time insights with low latency.
Continual model updating to adapt to changes (concept drift).
The mining process typically includes:
1. Ingestion of data from sources (sensors, logs, etc.)
2. Pre-processing and transformation (cleaning, filtering, summarizing).
3. Feature extraction and dimensionality reduction.
4. Model application or updates.
5. Result dissemination or alerting.
This pipeline must operate with bounded resources and strict time constraints.
Descriptive Statistics
Descriptive statistics provide essential insights into stream data. Metrics such as mean, standard
deviation, and variance help in understanding trends, detecting anomalies, and monitoring
system behavior.
Because storing the entire stream is infeasible, these statistics must be computed incrementally:
Running Mean and Variance: Calculated using online formulas (e.g., Welford’s
method).
Frequency Estimation: Achieved with compact structures like Count-Min Sketch.
Used in real-time dashboards and monitoring tools.
Sampling
Sampling reduces the volume of data by selecting a representative subset. This is crucial for:
Improving processing speed.
Enabling approximate queries.
Reducing memory and compute usage.
Streaming requires specialized techniques:
Reservoir Sampling: Randomly selects samples from an unbounded stream.
Sliding Window Sampling: Focuses on the most recent data, maintaining relevance.
These techniques maintain data utility without overwhelming system resources.
Quantization
Quantization transforms continuous data values into discrete bins or intervals. This step is critical
when:
Applying machine learning models that operate on categorical data.
Compressing data for transmission or storage.
Reducing noise in high-frequency signal streams.
For example, a temperature sensor may record precise values, but quantization can bucket them
as “Low”, “Medium”, or “High” for trend detection or rule-based monitoring.
Dimensionality Reduction
Streaming data often has high dimensionality (many features), especially in multimedia or IoT
domains. Processing all dimensions in real time can be computationally expensive.
Dimensionality reduction simplifies this by:
Removing redundant or irrelevant features.
Projecting data into a lower-dimensional space that preserves structure.
Techniques include:
Incremental PCA: Adapts PCA for streaming inputs.
Random Projection: A fast and probabilistic method for dimensionality reduction.
Feature hashing: Efficient for high-cardinality categorical variables.
Dimensionality reduction not only speeds up processing but also improves model generalization.
Modeling and Evaluation in Streaming Systems
Once pre-processing is complete, stream analytics moves into the modeling and evaluation stage.
Unlike static modeling in traditional systems, stream modeling must adapt on-the-fly, learning
and predicting as data evolves.
Offline Modeling and Online Evaluation
Many systems combine the best of both worlds:
Offline Modeling: Used to pre-train models on historical data. These models are more
accurate initially and can incorporate rich features and hyperparameter tuning.
Online Evaluation: Once deployed, models are updated incrementally using incoming
data.
Evaluation is also continuous:
Prequential Evaluation: A method where each new data point is first used for testing,
then for training. This mimics a live setting and enables ongoing accuracy tracking.
Models must adapt to concept drift—changes in the data distribution over time.
Data Stream Classification
Classification assigns a category label to each incoming data item. Applications include:
Email spam detection.
Network intrusion identification.
Social media sentiment analysis.
Stream classification requires:
Fast, incremental learning.
Limited memory usage.
Adaptability to concept drift.
Popular algorithms include:
Hoeffding Tree (VFDT): Makes statistically sound splits using a bounded number of
examples.
Online Naive Bayes: Updates class probabilities with each new instance.
Ensemble Methods: Multiple models are maintained and adapted based on performance.
Data Stream Clustering
Clustering is the task of grouping similar data points without prior labels. In a stream context,
clusters need to evolve over time, and outdated data must be forgotten.
Real-world applications include:
Real-time user segmentation.
Dynamic trend discovery in social networks.
Behavior monitoring in IoT systems.
Popular techniques:
Online K-Means: Centroids updated incrementally.
CluStream: Maintains micro-clusters which are summarized and merged over time.
DenStream: Detects dense regions and adapts to shifting data.
Data Stream Regression
Regression in stream analytics predicts continuous output values, such as:
Predicting stock prices.
Forecasting energy consumption.
Estimating traffic flow.
Key characteristics:
Online regression must learn incrementally.
Must cope with noise, delays, and non-stationary behavior.
Techniques:
Online Linear Regression: Coefficients are updated in real time.
Evolving Regression Trees: Trees adapt their structure with new data.
Fuzzy Regression: Models with uncertain boundaries, useful in imprecise environments.
Evaluation metrics such as MAE and RMSE are computed in an online manner.