1

Data Warehouse Definition

A Data Warehouse is a centralized repository that stores historical and integrated data from multiple sources
for reporting and business analysis and decision making.
Datawarehousing is the process of collecting, Integerating, storing and managing data from multiple sources
in a central repository.

Characteristics of Data Warehouse

1. Subject Oriented : Data is organized around subjects such as: Sales , Customer , Product Finance
Instead of applications. Example: Operational DB → Customer Orders ,Warehouse → Customer Analysis

2. Integrated : Data comes from multiple sources and is converted into a common [Link]: Date
format standardized , Currency standardized ,Naming conventions unified

3. Time Variant : Stores historical data over long periods. Example: Sales 2022 , Sales 2023 Sales 2024
Sales 2025 Operational DB generally stores only current data.

4. Non-Volatile: Once data enters warehouse it is: Read , Queried , Analyzed , It is generally not updated or
deleted. Operations: Load, Access , Not: Insert , Update Delete frequently.

Data Warehouse Architecture

Definition: Architecture describes the flow of data from source systems to end users for analysis.

Components: Data Sources → ETL → Data Warehouse → OLAP Server → Users/Analysts.

• Data Sources: Databases, ERP, CRM, Files, Web data.

• ETL: Extract (collect data), Transform (clean/convert data), Load (store into warehouse).
• OLAP Server: Performs Roll-up, Drill-down, Slice, Dice, Pivot operations.
• Front End Tools: Reporting, Dashboard and Dat a Mining tools.

Operational Database vs Data Warehouse

 2
OLTP OLAP
Operational DB Data Warehouse
Online Transaction Online Analytical
Transaction processing Analysis and decision making Processing Processing
Current data Historical data Current data Historical data
Frequent updates Rare updates
Insert/Update/Delete Analysis
Simple queries Complex queries
Application oriented Subject oriented Fast transactions Complex queries

OLAP (Online Analytical Processing) Example: Sales

Example: ATM
analysis
Definition: OLAP (Online Analytical Processing) is a technology used in Data Warehouses for
multidimensional analysis of large amounts of historical data to support decision making.

Purpose: Used for trend analysis, forecasting, business reporting and decision support.

Example: Analyzing sales based on Time, Product and Location dimensions.

Characteristics of OLAP

1. Multidimensional View: Data is viewed from multiple dimensions such as Time, Product, Customer
and Region.
2. Fast Analysis: Provides quick response for complex analytical queries.
3. Historical Data Analysis: Mainly works on historical data stored in Data Warehouse.
4. Supports Decision Making: Helps managers identify trends and patterns.
5. Complex Queries: Supports aggregation, summarization and statistical analysis.

Types of OLAP

1. MOLAP (Multidimensional OLAP) : Stores data in multidimensional cubes., Fast query performance.
Requires more storage.

2. ROLAP (Relational OLAP) : Stores data in relational tables. Can handle huge datasets. Slower than
MOLAP.

3. HOLAP (Hybrid OLAP) : Combination of MOLAP and ROLAP. Combines speed and scalability.

Data Mart : A Data Mart is a subset of Data Warehouse designed for a specific department such as Sales,
HR or Finance.

Metadata: Metadata means "Data about Data" and contains source information, schema details, data
types and transformation rules.
 3

Multidimensional Data Model

A Multidimensional Data Model organizes data into Facts and Dimensions for fast analytical processing in
Data Warehouses.

Components of Multidimensional Model

1. Fact Table: Central table containing numerical measures and foreign keys to dimension tables.
Examples: Sales Amount, Profit, Quantity Sold.
2. Dimension Table: Contains descriptive information about facts. Examples: Product, Customer,
Time, Location.
3. Measure: Numerical values on which analysis is performed. Examples: Revenue, Cost, Profit,
Quantity Sold.
4. Data Cube: Multidimensional representation of data using dimensions and measures. Example: Sales
analyzed by Product × Time × Location. Diagram: Data Cube Diagram.

Operations on Data Cube (OLAP Operations)

1. Roll-Up: Aggregates data by moving from lower level to higher level hierarchy. Example: City →
State → Country.
2. Drill-Down: Opposite of Roll-Up; moves from summary to detailed data. Example: Year → Month
→ Day.
3. Slice: Selects one value from a dimension to create a smaller cube. Example: Sales data for Year =
2025.
4. Dice: Selects multiple values from multiple dimensions. Example: Laptop and Mobile sales in Delhi
and Kolkata during 2025.
5. Pivot (Rotate): Rotates cube axes to provide different views of data. Example: Product vs Year →
Year vs Product.

Multidimensional Database Schemas

1. Star Schema: One Fact Table directly connected to multiple Dimension Tables. Characteristics:
Simple design, fewer joins, fast query processing, high redundancy. Example: Sales Fact → Product,
Customer, Time, Location. Relationship: One-to-Many. Diagram: Star Schema Diagram.
2. Snowflake Schema: Extension of Star Schema where Dimension Tables are normalized into multiple
related tables. Characteristics: Less redundancy, less storage, more joins, slower queries. Example:
Product → Category → Department. Diagram: Snowflake Schema Diagram.
3. Fact Constellation Schema (Galaxy Schema): Contains multiple Fact Tables sharing Dimension
Tables. Characteristics: Complex design, supports multiple business processes, highly scalable.
Example: Sales Fact and Shipping Fact sharing Product and Time dimensions. Diagram: Fact
Constellation Schema Diagram.

Star Schema Snowflake Schema Fact Constellation

One Fact Table One Fact Table Multiple Fact Tables

Denormalized Dimensions Normalized Dimensions Shared Dimensions

Fast Queries More Joins Complex Queries

More Redundancy Less Redundancy Highly Scalable

 4

ETL
ETL (Extract, Transform, Load) is the process of collecting data from multiple sources, converting it into a
suitable format and storing it into a Data Warehouse.

ETL Phases

1. Extract: Data is collected from various sources such as Databases, ERP Systems, CRM Systems, Files
and Web Sources.
2. Transform: Extracted data is cleaned, integrated and converted into a standard format. Operations
include Data Cleaning, Data Integration, Data Aggregation, Data Filtering and Data Conversion.
3. Load: Transformed data is loaded into the Data Warehouse for analysis and decision making.

Transform Phase of ETL

Definition: Transform phase is the process of converting extracted raw data into a clean, consistent and
suitable format before loading it into the Data Warehouse.

Operations Performed in Transform Phase

1. Data Cleaning: Removes missing, noisy, duplicate and inconsistent data.

2. Data Integration: Combines data from multiple sources into a unified format.
3. Data Transformation: Converts data types, formats and units into standard forms.
4. Data Aggregation: Summarizes detailed data into higher level information. Example: Daily Sales →
Monthly Sales.
5. Data Filtering: Removes unnecessary or irrelevant data.
6. Data Normalization: Scales values into a common range for analysis.
7. Data Generalization: Replaces low-level data with higher-level concepts. Example: Kolkata → West
Bengal → India.
8. Data Discretization: Converts continuous values into intervals. Example: Age 22 → Young.
9. Data Enrichment: Adds additional information from external sources.
10. Data Validation: Checks data for accuracy and correctness before loading.
 5

Data Mining
Definition: Data Mining is the process of discovering useful patterns, relationships, trends and knowledge
from large amounts of data using statistical, machine learning and database techniques.

Functions of Data Mining

1. Classification: Assigns data into predefined classes. Example: Spam or Not Spam.
2. Prediction: Predicts future values based on existing data. Example: Stock Price Prediction.
3. Clustering: Groups similar data objects together. Example: Customer Segmentation.
4. Association Analysis: Finds relationships among items. Example: Bread → Butter.
5. Sequential Pattern Mining: Finds patterns occurring in sequence. Example: Buy Mobile → Buy Cover
→ Buy Earphones.
6. Outlier Analysis: Detects abnormal or unusual data. Example: Credit Card Fraud Detection.
7. Evolution Analysis: Finds trends and changes over time. Example: Sales Growth Analysis .

Data Mining Process (KDD Process)

1. Data Cleaning: Removes noise and missing values.

2. Data Integration: Combines data from multiple sources.
3. Data Selection: Selects relevant data for mining.
4. Data Transformation: Converts data into suitable format.
5. Data Mining: Applies algorithms to extract patterns.
6. Pattern Evaluation: Identifies useful patterns.
7. Knowledge Presentation: Presents knowledge using reports and visualization.

Frequent Pattern Mining

Definition: Frequent Pattern Mining is the process of discovering patterns, itemsets, subsequences or
substructures that occur frequently in a dataset.

Types of Frequent Patterns

1. Frequent Itemset: Set of items that appear together frequently in transactions. Example: {Bread,
Butter}.
2. Sequential Pattern: Set of items occurring in a specific order. Example: Buy Mobile → Buy Cover →
Buy Earphones.
3. Structured Pattern: Frequent patterns in trees, graphs or networks. Example: Social Network
Connections.

Applications

1. Market Basket Analysis.

2. Product Recommendation Systems.
3. Cross Selling and Up Selling.
4. Website Click Analysis.
5. Fraud Detection.
 6

Association Rule Mining

Association Rule Mining is a technique used to discover relationships or associations between items in transaction
databases.
Applications
General Form of Association Rule
1. Market Basket Analysis.
X → Y, where X and Y are itemsets and X ∩ Y = Ø. Example 2. Product Placement.
3. Recommendation Systems.
Bread → Butter: Customers who buy Bread are likely to buy Butter. 4. Customer Purchase Analysis.

Measure of Quality

Support
Definition: Support measures how frequently an itemset appears in the database.

Formula: Support(X) = (Number of Transactions containing X) / (Total Transactions)

Formula: Support(X→Y) = (Number of Transactions containing X and Y) / (Total Transactions)

Example: If Bread and Butter appear together in 20 out of 100 transactions then Support = 20/100 = 20%.

Confidence
Definition: Confidence measures the probability that Y is purchased when X is purchased.

Formula: Confidence(X→Y) = Support(X∪Y) / Support(X)

Example: If 20 customers buy Bread and Butter together and 25 customers buy Bread then
Confidence(Bread→Butter) = 20/25 = 80%.

Lift
Definition: Lift measures the strength of association between two items.

Formula: Lift(X→Y) = Confidence(X→Y) / Support(Y)

Interpretation: Lift > 1 Positive Correlation, Lift = 1 Independent, Lift < 1 Negative Correlation

Correlation Analysis
Definition: Correlation Analysis measures the relationship between two variables or itemsets.

Types of Correlation

1. Positive Correlation: Increase in one variable increases the other. Example: Bread and Butter.
2. Negative Correlation: Increase in one variable decreases the other. Example: Umbrella and
Sunglasses.
3. No Correlation: Variables are independent.
 7

Market Basket Analysis

Definition: Technique used to analyze customer purchasing behavior by finding associations among
purchased [Link]: Customers buying Bread often buy Butter and [Link]: Product
Placement, Recommendation Systems, Store Layout Optimization.

Apriori Algorithm : Apriori Algorithm is a frequent itemset mining algorithm that generates candidate
itemsets level by level using the Apriori Property.

Apriori Property : If an itemset is frequent then all of its non-empty subsets must also be frequent.
Working Steps
Drawbacks of Apriori Algorithm
1. Find frequent 1-itemsets (L1).
2. Generate candidate 2-itemsets (C2). 1. Requires multiple database scans.
3. Remove infrequent itemsets using minimum support. 2. Generates large number of
4. Generate frequent 2-itemsets (L2). candidate itemsets.
5. Repeat until no new frequent itemsets are found. 3. High computational cost.
4. Poor performance for large datasets
Advantages

1. Simple and easy to understand.

2. Reduces search space using Apriori Property.
3. Suitable for market basket analysis.

FP Growth (Frequent Pattern Growth) is a frequent itemset mining algorithm that uses an FP Tree instead
of candidate generation.

Advantages: No candidate generation. , Faster than Apriori. Requires fewer database scans. Disadvantages :
FP Tree construction can be complex. Requires additional memory for tree storage.

Scalable Methods in Data Mining

Scalable Methods are data mining techniques that can efficiently process and analyze very large datasets without
significant increase in execution time or memory usage.

Characteristics of Scalable Methods : Efficient memory utilization.

1. Reduced execution time.

2. Supports parallel and distributed processing.

Types of Scalable Methods

1. Parallel Algorithms: Divide data among multiple processors

for simultaneous execution. Example: Parallel Apriori.
2. Distributed Algorithms: Data is processed on multiple
machines connected through a network. Example:
MapReduce based mining.
3. Sampling Methods: Analyze a representative subset
instead of the entire dataset.
4. Incremental Mining: Updates mining results using only
newly added data instead of reprocessing whole data.
5. Partitioning Methods: Divide database into smaller
partitions and mine each partition separately. FP TREE DIAGRAM
 8

Classification

Definition: Classification is a supervised learning technique that assigns data objects to predefined classes
using a model built from training data.

Working of Classification

1. Training Data with known class labels is provided.

2. Classification model is built using training data.
3. Model predicts class labels for new unseen data.

Classification Process : Training Dataset → Model Building → Testing Dataset → Predicted Class Label

Examples : Spam or Not Spam Email. , Loan Approved or Rejected. ,Disease Positive or Negative.

Prediction

Prediction is a supervised learning technique used to predict continuous or future values based on existing
data.

Prediction Process : Historical Data → Model Building → Prediction Model → Future Value Prediction

Examples

1. House Price Prediction.

2. Stock Price Prediction.
3. Weather Forecasting.
4. Sales Forecasting.

Evaluation of Classification

Evaluation of Classification is the process of measuring how accurately a classification model predicts class
labels for unseen data.

Types of Classification Outcomes

1. True Positive (TP): Model predicts Positive and actual class is also Positive.
2. True Negative (TN): Model predicts Negative and actual class is also Negative.
3. False Positive (FP): Model predicts Positive but actual class is Negative (Type-I Error).
4. False Negative (FN): Model predicts Negative but actual class is Positive (Type-II Error).

Confusion Matrix
Predicted Positive Predicted Negative

Actual Positive TP FN

Actual Negative FP TN

Accuracy Measure: Accuracy measures the percentage of correctly classified instances among all
instances.

Formula: Accuracy = (TP + TN) / (TP + TN + FP + FN)

 9

Interpretation: Higher Accuracy means better classification performance.

Other Accuracy Measures

1. Error Rate: Percentage of incorrectly classified instances.

Formula: Error Rate = (FP + FN) / (TP + TN + FP + FN)
2. Precision: Percentage of predicted positive cases that are actually positive.
Formula: Precision = TP / (TP + FP)
3. Recall (Sensitivity): Percentage of actual positive cases correctly identified.
Formula: Recall = TP / (TP + FN)
4. Specificity: Percentage of actual negative cases correctly identified.
Formula: Specificity = TN / (TN + FP)
5. F1-Score: Harmonic mean of Precision and Recall.
Formula: F1 = 2 × Precision × Recall / (Precision + Recall)

Cluster Analysis

Cluster Analysis is an unsupervised learning technique that groups similar data objects into the same cluster
and dissimilar objects into different clusters.
Applications
Characteristics of Cluster Analysis
1. Customer Segmentation.
1. Objects within the same cluster are highly similar. 2. Document Classification.
2. Objects belonging to different clusters are highly dissimilar. 3. Image Processing.
3. Does not require predefined class labels. 4. Market Analysis.
5. Medical Diagnosis.
Types of Data in Cluster Analysis

1. Interval Scaled Data: Continuous numerical values measured on a scale. Example: Height, Weight,
Temperature.
2. Binary Data: Data having only two possible values. Example: Yes/No, Male/Female.
3. Nominal Data: Categorical data without ordering. Example: Color, City, Department.
4. Ordinal Data: Categorical data with ordering. Example: Small, Medium, Large.
5. Ratio Scaled Data: Numerical data with a meaningful zero point. Example: Age, Salary, Distance.
6. Mixed Data: Combination of multiple data types.

Types of Clustering Methods

1. Partitioning Methods: Divide data into k clusters. Example: K-Means, K-Medoids.

2. Hierarchical Methods: Create hierarchy of clusters. Example: Agglomerative, Divisive.
3. Density Based Methods: Form clusters based on density of data points. Example: DBSCAN.
4. Grid Based Methods: Divide data space into grids. Example: STING.
5. Model Based Methods: Assume data follows a statistical model. Example: EM Algorithm.

Partitioning Methods

Partitioning Methods divide a dataset of n objects into k clusters, where each object belongs to exactly one
cluster and k is predefined.

Characteristics

1. Number of clusters (k) is specified in advance. 3. Tries to minimize intra-cluster distance

2. Each object belongs to only one cluster. and maximize inter-cluster distance.
4. Suitable for large datasets.
 10

Types of Partitioning Methods

1. K-Means: Uses centroid (mean) of cluster as cluster center and assigns objects to nearest centroid.
2. K-Medoids: Uses an actual data object (medoid) as cluster center instead of mean and is less
sensitive to outliers.
Disadvantages
Advantages
1. Number of clusters must be predefined.
1. Simple and easy to implement. 2. Sensitive to noise and outliers (especially K-Means).
2. Efficient for large datasets. 3. May produce different results for different initial
3. Fast convergence. centroids.

Hierarchical Methods

Hierarchical Methods create a hierarchy of clusters represented by a tree-like structure called Dendrogram.

Types of Hierarchical Methods

1. Agglomerative (Bottom-Up): Starts with each object as a separate cluster and repeatedly merges the
closest clusters until one cluster remains.
2. Divisive (Top-Down): Starts with all objects in one cluster and repeatedly splits clusters until each
object becomes a separate cluster.

Characteristics

1. Number of clusters need not be specified initially.

2. Produces hierarchical relationship among clusters. Disadvantages
3. Uses Dendrogram for visualization.
1. Computationally expensive for large
Advantages datasets.
2. Once merged or split, clusters cannot be
1. No need to specify number of clusters beforehand. undone.
2. Easy to understand and interpret. 3. Less scalable than partitioning methods.
3. Suitable for discovering hierarchical structures.

Dendrogram

Definition: A tree-like diagram used to represent the merging or splitting process in hierarchical clustering.
Steps of K-Means Algorithm

1. Select the number of clusters k.

2. Choose k initial centroids randomly.
3. Calculate the distance of each data point from all
centroids.
4. Assign each data point to the nearest centroid to form
clusters.
5. Recalculate the centroid of each cluster using the mean of
cluster members.
6. Repeat Steps 3, 4 and 5 until centroids no longer change or
maximum iterations are reached.

Formula for Centroid : Centroid = (Sum of all data points in cluster) / (Number of data points in cluster)
 11

Transactional Patterns

Definition: Transactional Patterns are patterns discovered from transaction databases that show
relationships among items purchased together in a transaction.

Example: {Bread, Butter} appears together in many transactions.

Applications: Market Basket Analysis, Product Recommendation, Cross Selling.

Temporal-Based Frequent Patterns

Definition: Temporal Frequent Patterns are frequent patterns that consider the time factor along with
occurrence frequency.

Example: Ice Cream sales increase during Summer every year.

Applications: Sales Forecasting, Stock Analysis, Weather Prediction.

Types of Temporal Patterns

1. Periodic Patterns: Patterns repeating at regular intervals. Example: Increased umbrella sales during
monsoon every year.
2. Seasonal Patterns: Patterns occurring in specific seasons. Example: Woolen clothes sales in winter.
3. Trend Patterns: Long-term increase or decrease in data values. Example: Increasing smartphone
sales every year.
4. Cyclic Patterns: Patterns repeating over long but non-fixed intervals. Example: Economic growth
cycles.

Mining Time Series Data

Mining Time Series Data is the process of discovering useful patterns, trends, similarities and relationships
from data collected over time intervals.

Time Series Data

Definition: A sequence of observations recorded at equal intervals of time.

Examples: Daily Temperature, Monthly Sales, Stock Prices, Rainfall Data.

Characteristics of Time Series Data

1. Data is time dependent.

2. Observations are recorded in chronological order.
3. Adjacent values are often correlated.

Components of Time Series Data

1. Trend (T): Long term increase or decrease in data values. Example: Increasing smartphone sales
every year.
2. Seasonal Variation (S): Patterns repeated at fixed intervals within a year. Example: Ice cream sales
increase in summer.
3. Cyclic Variation (C): Patterns repeated over long but irregular periods. Example: Economic cycles.
4. Irregular Variation (I): Random and unpredictable fluctuations caused by unexpected events.
 12

Time Series Model

Additive Model: Y = T + S + C + I
Multiplicative Model: Y = T × S × C × I Challenges in Mining Time Series Data

Applications 1. Large data volume.

2. Noise and missing values.
1. Stock Market Prediction. 3. High dimensionality.
2. Weather Forecasting. 4. Temporal dependency among
3. Sales Forecasting. observations.
4. Traffic Analysis. 5. Computational complexity.
5. Medical Monitoring.

Periodicity Analysis
Periodicity Analysis is the process of identifying patterns or events that repeat at regular time intervals in
time-series data.

Types of Periodicity

1. Full Periodicity: Entire pattern repeats at fixed intervals. Example: Monthly electricity bill
generation.
2. Partial Periodicity: Only some parts of the pattern repeat periodically. Example: Increased shopping
during weekends.

Applications
Advantages
1. Sales Forecasting.
1. Helps identify recurring events.
2. Weather Prediction.
2. Useful for forecasting future trends.
3. Stock Market Analysis.
3. Improves decision making.
4. Network Traffic Analysis.
5. Medical Monitoring.

Periodicity Detection Methods

1. Autocorrelation Analysis: Measures similarity between a time series and its delayed version to
detect repeating patterns.
2. Spectral Analysis: Uses frequency components to identify hidden periodic behavior in data.
3. Fourier Transform: Converts time-domain data into frequency-domain data for periodicity
detection.

Autocorrelation

Definition: Autocorrelation measures the correlation between observations of the same variable at different
time lags.
Example: Comparing today's sales with sales from 7 days ago to detect weekly patterns.

Spectral Analysis

Definition: Spectral Analysis identifies periodic patterns by analyzing the frequency components of time-
series data.
Application: Detection of seasonal and cyclic patterns.
 13

Decomposition in Time Series Analysis

Decomposition is the process of separating a time series into Trend, Seasonal, Cyclic and Irregular
components.
Trend Analysis

Definition: Trend Analysis is the process of identifying long-term increase, decrease or stable movement in
time-series data over a period of time.

Types of Trends

1. Upward Trend: Data values increase over time. Example: Growth of smartphone sales.
2. Downward Trend: Data values decrease over time. Example: Decline in DVD sales.
3. Horizontal Trend: Data values remain nearly constant over time. Example: Stable electricity
consumption.

Methods of Trend Analysis

1. Moving Average Method: Smooths short-term fluctuations to identify long-term trends.

2. Least Square Method: Fits a trend line to the data using regression.
3. Exponential Smoothing: Assigns more weight to recent observations.

Applications

1. Sales Forecasting.
2. Stock Market Analysis.
3. Population Growth Analysis.
4. Weather Prediction.

Similarity Search in Time Series Analysis

Definition: Similarity Search is the process of finding time-series sequences that are similar to a given
sequence or pattern.

Purpose

1. Identify similar trends and patterns.

2. Detect anomalies and unusual behavior.
3. Support prediction and forecasting.

Similarity Measures

1. Euclidean Distance: Measures straight-line distance between two sequences.

2. Dynamic Time Warping (DTW): Measures similarity even if sequences are shifted or stretched in
time.
3. Correlation Based Similarity: Measures similarity based on correlation coefficient.

Applications

1. Stock Market Analysis.

2. Speech Recognition.
3. Medical Signal Analysis (ECG, EEG).
4. Pattern Recognition.
 14

Mining Data Streams

Definition: Data Stream Mining is the process of extracting useful patterns and knowledge from continuous,
rapid and unbounded streams of data.

Data Stream

A Data Stream is a continuous flow of data generated in real time which cannot be stored entirely before
processing.
Examples: Sensor Data, Network Traffic, Stock Market Data, Social Media Feeds, IoT Devices.

Characteristics of Data Streams

Challenges in Data Stream Mining
1. Continuous and unbounded data flow. 1. Infinite data size.
2. High speed data arrival. 2. Limited memory availability.
3. Single scan processing. 3. High data arrival rate.
4. Limited memory and computation time. 4. Concept drift (data pattern changes over time).
5. Data cannot be stored completely. 5. Need for real-time processing.

Methodologies for Stream Data Processing

Techniques used to process continuous data streams efficiently in real time.

1. Sampling: Processes only a representative subset of stream data.

2. Load Shedding: Discards some incoming data when system is overloaded.
3. Sketching: Uses compact summary structures instead of storing complete data.
4. Aggregation: Stores summarized information instead of raw data.
5. Sliding Window Model: Processes only recent data within a fixed window.

Stream Data Systems

Systems designed to collect, process and analyze continuous streams of data in real time.

Characteristics
Examples
1. Real-time processing.
1. Apache Kafka
2. Low latency response.
2. Apache Storm
3. Scalability.
3. Apache Flink
4. Fault tolerance.
Apache Spark Streaming
Sliding Window Model

Definition: Technique that processes only recent data within a fixed time interval while discarding older
data.
Example: Analyze last 1 hour stock prices instead of all historical data.

Types

1. Fixed Window.
2. Sliding Window.
3. Landmark Window.
4. Damped Window.
 15

Frequent Pattern Mining in Data Streams

Frequent Pattern Mining in Data Streams is the process of discovering frequently occurring items or itemsets
from continuously arriving stream data in real time.
Methods
Characteristics
1. Sliding Window Model: Mines patterns from recent
1. Single pass over data.
data only.
2. Limited memory usage.
2. Landmark Window Model: Considers data from a
3. Real-time processing.
fixed starting point.
4. Handles continuously changing data.
3. Damped Window Model: Gives more importance to
Challenges recent data and less to older data.

1. Infinite data size. Applications

2. Limited storage and memory.
3. High speed data arrival. 1. Network Intrusion Detection.
4. Concept drift. 2. Stock Market Analysis.
3. Web Click Stream Analysis. [Link] Data Analysis

Sequential Pattern Mining in Data Streams

Sequential Pattern Mining in Data Streams discovers frequently occurring sequences from continuously
arriving stream data while preserving the order of events.

Example : Customer Purchase Sequence: Mobile → Mobile Cover → Earphones

Challenges
Applications
1. Infinite sequence generation.
2. Memory limitations. 1. Customer Behavior Analysis.
3. High processing speed requirement. 2. Web Usage Mining.
4. Concept drift. 3. Recommendation Systems.
4. Fraud Detection.
Classification of Dynamic Data Streams

Classification of Dynamic Data Streams is the process of classifying continuously changing stream data in
real time.

Characteristics

1. Continuous learning. Applications

2. Real-time prediction.
3. Adapts to changing patterns. 1. Spam Detection.
4. Handles concept drift. 2. Fraud Detection.
3. Network Intrusion Detection.
Challenges
4. Stock Market Prediction.
1. High speed data arrival.
2. Memory constraints.
3. Changing data distribution.
4. Imbalanced classes.
 16

Concept Drift
Definition: Concept Drift occurs when the statistical properties or patterns of stream data change over time causing
the existing model to become inaccurate.

Types

1. Sudden Drift: Immediate change in pattern.

2. Gradual Drift: Slow change over time.
3. Incremental Drift: Continuous small changes.
4. Recurring Drift: Old patterns reappear after some time.

Class Imbalance Problem

Class Imbalance Problem occurs when one class contains significantly more instances than the other
classes.

Example

Fraud Detection Dataset: Legitimate Transactions = 99% Fraud Transactions = 1%

Problems

1. Model becomes biased towards majority class.

2. Minority class prediction accuracy decreases.
3. Traditional accuracy measure becomes misleading.

Solutions

1. Oversampling: Increase minority class samples.

2. Undersampling: Reduce majority class samples.
3. SMOTE: Generate synthetic minority samples.
4. Cost Sensitive Learning: Assign higher cost to minority class errors.
5. Ensemble Learning: Use multiple classifiers for better performance.

Example

Cancer Detection: Healthy = 95% Cancer Patients = 5% A model predicting everyone as healthy gives 95%
accuracy but is practically useless.

Recent Trends in Distributed Warehousing and Data Mining

Modern technologies and approaches used to improve distributed storage and mining of large datasets.
Trends:

1. Cloud Data Warehousing. Advantages

2. Big Data Analytics.
3. Hadoop Ecosystem. 1. Better scalability.
4. Apache Spark. 2. Faster processing.
5. Real-Time Analytics. 3. Lower infrastructure cost.
6. Machine Learning Integration. 4. Real-time decision making.
7. AI Driven Data Mining.
8. Edge Computing.
 17

Graph Mining
Graph Mining is the process of discovering useful patterns, relationships and structures from graph data
consisting of vertices (nodes) and edges (links).

Components of Graph

1. Node (Vertex): Represents an object or entity. Example: Person in Facebook.

2. Edge (Link): Represents relationship between nodes. Example: Friendship between two users.

Types of Graph Patterns Applications

1. Frequent Subgraphs. 1. Social Network Analysis.

2. Graph Clustering. 2. Fraud Detection.
3. Graph Classification. 3. Recommendation Systems.
4. Community Detection. 4. Biological Network Analysis.
5. Web Link Analysis.
Social Network Analysis (SNA)

Social Network Analysis is the study of relationships and interactions among people, groups or
organizations using graph structures.
Applications
Components
1. Friend
1. Node: Individual person or organization. Recommendation.
2. Edge: Relationship such as friendship, follow or communication. 2. Influencer Detection.
3. Community Detection.
Measures 4. Viral Marketing.
5. Fraud Detection.
1. Degree Centrality: Number of direct connections of a node.
2. Betweenness Centrality: Measures how often a node lies on shortest paths.
3. Closeness Centrality: Measures how close a node is to all other nodes.
4. Eigenvector Centrality: Measures influence of a node based on connected nodes.

Web Mining

Definition: Web Mining is the process of extracting useful information and knowledge from web data, web
documents and web services.

Types of Web Mining

1. Web Content Mining: Extracts information from web pages, text, images, audio and video.
2. Web Structure Mining: Analyzes hyperlinks and document structure of web pages.
3. Web Usage Mining: Analyzes user browsing behavior using server logs and clickstreams.

Mining Web Page Layout Structure

Definition: Process of extracting information from page layout elements such as tables, frames and HTML
tags to understand page organization.

Applications : Information Extraction. , Search Engines. , Web Page Classification.

 18

Mining Web Link Structure

Process of analyzing hyperlinks among web pages to discover relationships and importance of pages.

Applications

1. Page Ranking.
2. Search Engines.
3. Community Detection.

Mining Multimedia Data on the Web

Definition: Process of extracting patterns and information from images, videos, audio and multimedia
content available on the web.

Applications

1. Image Retrieval.
2. Video Recommendation.
3. Face Recognition.

Automatic Classification of Web Documents

Definition: Process of automatically assigning web documents to predefined categories using machine
learning techniques.

Applications

1. Spam Detection.
2. News Categorization.
3. Document Organization.

Web Usage Mining

Definition: Process of discovering user access patterns from web logs and clickstream data.

Applications

1. Recommendation Systems.
2. Website Personalization.
3. Customer Behavior Analysis.
4. Targeted Advertising.

Distributed Data Mining (DDM)

Distributed Data Mining is the process of performing data mining on data distributed across multiple
databases, locations or systems.
Challenges
Characteristics
1. Data integration.
1. Data stored at multiple locations. 2. Data heterogeneity.
2. Parallel processing. 3. Security and privacy issues.
3. Improved scalability. 4. Communication overhead.
4. Reduced communication cost. 5. Synchronization problems.
 19

Page Topic
1 Data Warehouse Definition
1 Characteristics of Data Warehouse
1 Data Warehouse Architecture
2 Operational DB vs Data Warehouse
2 OLAP (Online Analytical Processing)
2 Types of OLAP (MOLAP, ROLAP, HOLAP)
2 Data Mart
2 Metadata
3 Multidimensional Data Model
3 Fact Table, Dimension Table, Measure
3 Data Cube
3 OLAP Operations (Roll-Up, Drill-Down, Slice, Dice, Pivot)
3 Star Schema
3 Snowflake Schema
3 Fact Constellation Schema
4 ETL
4 Transform Phase of ETL
5 Data Mining Introduction
5 Functions of Data Mining
5 KDD Process
5 Frequent Pattern Mining
6 Association Rule Mining
6 Support
6 Confidence
6 Lift
6 Correlation Analysis
7 Market Basket Analysis
7 Apriori Algorithm
7 FP Growth
7 Scalable Methods
8 Classification
8 Prediction
8 Evaluation of Classification
8 Confusion Matrix
8 Accuracy Measure
9 Precision, Recall, F1 Score
9 Cluster Analysis
9 Types of Data in Cluster Analysis
9 Types of Clustering Methods
9 Partitioning Methods
 20

Page Topic
10 K-Means Algorithm
10 Hierarchical Clustering
10 Dendrogram
11 Transactional Patterns
11 Temporal Based Frequent Patterns
11 Mining Time Series Data
11 Components of Time Series
12 Periodicity Analysis
12 Autocorrelation
12 Spectral Analysis
12 Challenges in Time Series Mining
13 Decomposition in Time Series
13 Trend Analysis
13 Similarity Search in Time Series
14 Mining Data Streams
14 Stream Data Processing Methodologies
14 Stream Data Systems
14 Sliding Window Model
15 Frequent Pattern Mining in Data Streams
15 Sequential Pattern Mining in Data Streams
15 Classification of Dynamic Data Streams
16 Concept Drift
16 Class Imbalance Problem
16 Recent Trends in Distributed Warehousing and Data Mining
17 Graph Mining
17 Social Network Analysis
17 Web Mining
17 Mining Web Page Layout Structure
18 Mining Web Link Structure
18 Mining Multimedia Data on the Web
18 Automatic Classification of Web Documents
18 Web Usage Mining
18 Distributed Data Mining