UNIT IV DATA PREPROCESSNG AND MINING TECHNIQUES:

Data Mining Functionalities – Data Preprocessing – Data Cleaning – Data Integration and
Transformation – Data Reduction – Data Discretization and Concept Hierarchy
Generation- Architecture of A Typical Data Mining Systems- Classification of Data Mining
Systems.

Association Rule Mining: - Efficient and Scalable Frequent Item set Mining Methods –
Mining Various Kinds of Association Rules – Association Mining to Correlation Analysis –
Pattern Mining in Multilevel, Multidimensional Space - Constraint-Based Association
Mining.

1. DATA PREPROCESSING

Data preprocessing is a crucial step in any data analysis or machine learning pipeline. It involves
transforming raw data into a clean and structured format suitable for analysis. The key steps in
data preprocessing include:
1. Data Collection

 Gather data from various sources (databases, CSV files, APIs, web scraping, etc.).

2. Data Cleaning

 Handle missing values (e.g., remove, impute, or replace).

 Remove duplicates.
 Fix incorrect or inconsistent data.
 Handle outliers (remove or transform).

3. Data Transformation

 Normalize or standardize numerical data.

 Encode categorical variables (one-hot encoding, label encoding, etc.).
 Apply feature scaling (Min-Max scaling, Z-score normalization).

4. Data Integration

 Merge datasets from different sources.

 Resolve schema mismatches.

5. Data Reduction

 Feature selection (remove irrelevant or redundant features).

 Feature extraction (PCA, LDA, etc.).
 Sampling (reduce dataset size while maintaining representativeness).

6. Data Splitting

 Divide data into training, validation, and test sets (e.g., 80-10-10 split).

7. Handling Imbalanced Data

 Use resampling techniques (oversampling, under sampling).

 Use synthetic data generation (SMOTE, ADASYN).

8. Feature Engineering

 Create new features from existing data.

 Transform variables for better model performance.
2. DATA CLEANING IN DATA PREPROCESSING
Data cleaning is the process of identifying and correcting errors, inconsistencies, and missing
values in a dataset to ensure accuracy and reliability for analysis or machine learning

Steps in Data Cleaning

1. Handling Missing Values

Remove Missing Data: If the missing values are too many, consider dropping the affected
rows/columns.

Imputation Techniques:

Numerical Data: Fill with mean, median, or mode.

Categorical Data: Fill with the most frequent category.

Advanced Methods: Use KNN imputation, regression, or deep learning.

import pandas as pd

from [Link] import SimpleImputer

df = pd.read_csv("[Link]")

# Fill missing numerical values with the median

imputer = SimpleImputer(strategy="median")

df[['column1']] = imputer.fit_transform(df[['column1']])

# Fill missing categorical values with the most frequent category

df['category_column'].fillna(df['category_column'].mode()[0], inplace=True)

Removing Duplicates

Identify and remove duplicate rows.

df = df.drop_duplicates()

Handling Outliers

Methods to detect outliers: Boxplot, Z-score, IQR method.

Fixing Outliers: Remove, transform, or cap values.
import numpy as np

# Using IQR to remove outliers

Q1 = df['column'].quantile(0.25)

Q3 = df['column'].quantile(0.75)

IQR = Q3 - Q1

df = df[(df['column'] >= (Q1 - 1.5 * IQR)) & (df['column'] <= (Q3 + 1.5 * IQR))]

Handling Data Type Errors

Convert incorrect data types.

df['price'] = pd.to_numeric(df['price'], errors='coerce') # Convert to numeric

Why is Data Cleaning Important?

* creases accuracy of machine learning models

* Reduces errors and inconsistencies
* Improves data reliability and usability

3. DATA INTEGRATION AND TRANSFORMATION

Data integration and transformation are key steps in preparing raw data for analysis. They ensure
data is merged, structured, and standardized for efficient processing in analytics or machine
learning models.

Data Integration (Merging Data from Multiple Sources)

A. Combining Multiple Datasets

Horizontal Merging (Join datasets by common columns, like customer_id).

Vertical Merging (Stack datasets with the same structure)

import pandas as pd

# Example: Merging sales and customer data on customer_id

df_sales = pd.read_csv("sales_data.csv")

df_customers = pd.read_csv("customer_data.csv")
df_merged = [Link](df_sales, df_customers, on="customer_id", how="inner") # Inner Join

Handling Schema Mismatches

 Standardize column names & formats across datasets.

 Resolve conflicts (e.g., different data types or missing values).

# Convert column names to lowercase

[Link] = [Link]()

# Convert date columns to a standard format

df["order_date"] = pd.to_datetime(df["order_date"])

Data Deduplication

 Remove duplicate records after integration.

Data Deduplication

Remove duplicate records after integration.

df = df.drop_duplicates()

Data Transformation (Reshaping Data for Better Analysis)

A. Handling Categorical Data

Label Encoding (Convert categories to numbers).

One-Hot Encoding (Create binary columns for each category).

from [Link] import LabelEncoder, OneHotEncoder

# Label Encoding

encoder = LabelEncoder()

df["category"] = encoder.fit_transform(df["category"])

# One-Hot Encoding

df = pd.get_dummies(df, columns=["category"])
Aggregation & Pivoting

Summarizing data for grouped insights.

# Summing total sales per customer

df_grouped = [Link]("customer_id")["sales_amount"].sum().reset_index()

# Pivoting Data

df_pivot = df.pivot_table(values="sales_amount", index="customer_id", columns="month",

aggfunc="sum")

# Summing total sales per customer

df_grouped = [Link]("customer_id")["sales_amount"].sum().reset_index()

# Pivoting Data

df_pivot = df.pivot_table(values="sales_amount", index="customer_id", columns="month",

aggfunc="sum")

Why is Data Integration & Transformation Important?

* Removes inconsistencies across multiple datasets.

* Prepares data for machine learning by ensuring a structured format.

* Improves data quality by handling missing, incorrect, or redundant data.

4. DATA REDUCTION IN DATA PREPROCESSING

Data Reduction is a technique used in data preprocessing to reduce the size of the dataset while
maintaining its integrity and analytical value. This helps in improving computational efficiency,
reducing storage requirements, and enhancing model performance.

Techniques for Data Reduction

A. Dimensionality Reduction

Reduces the number of features (columns) while preserving key information.

1. Principal Component Analysis (PCA)

Converts correlated features into a smaller set of uncorrelated features (principal components).
Helps in removing redundant information.

from [Link] import PCA

from [Link] import StandardScaler

# Standardize the data before PCA

scaler = StandardScaler()

df_scaled = scaler.fit_transform(df)

# Apply PCA to reduce dimensions to 2 components

pca = PCA(n_components=2)

df_pca = pca.fit_transform(df_scaled)

2. Linear Discriminant Analysis (LDA)

Similar to PCA but used for supervised learning (classification tasks).

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA(n_components=1) # Reduce to 1 feature

df_lda = lda.fit_transform(df, labels) # labels = target variable

3. Data Sampling

Reduces the number of rows while maintaining dataset representativeness.

1. Random Sampling

Selects a random subset of data.

df_sampled = [Link](frac=0.3, random_state=42) # 30% of data

2. stratified Sampling

Ensures balanced distribution across categories.

from sklearn.model_selection import train_test_split

df_train, df_test = train_test_split(df, test_size=0.2, stratify=df['category'])

3. Data Aggregation

Groups data to summarize information.

df_grouped = [Link]("customer_id")["sales_amount"].sum().reset_index()

Why is Data Reduction Important?

* Reduces storage and computation costs

* Enhances model performance by removing irrelevant data

* Prevents over fitting by eliminating unnecessary features

5. CLASSIFICATION OF DATA MINING SYSTEMS

Types of Databases Mined:

Data mining techniques can be applied to various types of databases based on the nature,
structure, and application of the data. Below are the main types of databases that can be mined:
[Link] Databases (RDBMS)

Stores data in structured tables with predefined schemas.

Uses SQL (Structured Query Language) for queries.

Data mining helps in customer segmentation, fraud detection, and trend analysis.

Examples:

MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server

Mining Techniques Used:

Association Rule Mining

Classification & Clustering

Regression Analysis

2. Transactional Databases

Stores real-time transactional data such as bank transactions, e-commerce purchases, and
financial records.

Data mining helps in fraud detection, risk analysis, and anomaly detection.

Examples:

Banking systems, Point-of-Sale (POS) databases

Mining Techniques Used:

1. Association Rule Mining (e.g., Market Basket Analysis)

[Link] Pattern Mining

[Link] Detection Algorithms

3. Time-Series Databases

Stores sequential data over time such as stock prices, weather data, and IoT sensor readings.

Used for trend forecasting, anomaly detection, and predictive analytics.

Examples:

InfluxDB, TimescaleDB, Apache Kafka

Mining Techniques Used:

[Link]-Series Forecasting (ARIMA, LSTM)

[Link] Detection

[Link] Analysis

Types of Knowledge Mined:

OLAP-Based Data Mining (Online Analytical Processing) – Uses multidimensional analysis for
decision-making.

Database-Oriented Data Mining – Extracts patterns from large databases.

Machine Learning-Based Data Mining – Uses AI techniques to train models.

Visualization-Based Data Mining – Represents data insights in charts, graphs, or dashboards.

Types of Techniques Mined:

Descriptive Data Mining – Focuses on summarizing and understanding patterns in data.

Predictive Data Mining – Uses machine learning algorithms to predict future outcomes.

Common Techniques:

1. Classification – Categorizes data into predefined classes (e.g., Decision Trees, Naïve Bayes,
SVM).

[Link] – Groups similar data points without predefined labels (e.g., K-Means, DBSCAN).

[Link] Rule Mining – Identifies relationships between variables (e.g., Market Basket
Analysis).

[Link] – Predicts numerical values (e.g., Linear Regression, Neural Networks).

Types of Application Mined

Business Intelligence Systems – Helps in customer segmentation, fraud detection, and risk
management.
Healthcare Data Mining – Predicts diseases, improves patient diagnosis, and analyzes medical
records.

Financial Data Mining – Used in stock market analysis, credit risk assessment, and fraud
detection.

E-commerce and Retail Data Mining – Analyzes customer preferences, recommends products,
and optimizes inventory.

6. ARCHITECTURE OF TYPICAL DATA MINING SYSTEMS

A data mining system follows a structured architecture to extract useful patterns and insights
from raw data. It consists of multiple layers that work together to process, store, and analyze data
efficiently.

1. Data Source:

The actual source of data is the Database, data warehouse, World Wide Web (WWW), text files,
and other documents. You need a huge amount of historical data for data mining to be
successful. Organizations typically store data in databases or data warehouses. Data warehouses
may comprise one or more databases, text files spreadsheets, or other repositories of data.
Sometimes, even plain text files or spreadsheets may contain information. Another primary
source of data is the World Wide Web or the internet.
2. Different processes:

Before passing the data to the database or data warehouse server, the data must be cleaned,
integrated, and selected. As the information comes from various sources and in different formats,
it can't be used directly for the data mining procedure because the data may not be complete and
accurate. So, the first data requires to be cleaned and unified. More information than needed will
be collected from various data sources, and only the data of interest will have to be selected and
passed to the server. These procedures are not as easy as we think. Several methods may be
performed on the data as part of selection, integration, and cleaning.

3. Database or Data Warehouse Server:

The database or data warehouse server consists of the original data that is ready to be processed.
Hence, the server is cause for retrieving the relevant data that is based on data mining as per user
request.

4 .Data Mining Engine:

The data mining engine is a major component of any data mining system. It contains several
modules for operating data mining tasks, including association, characterization, classification,
clustering, prediction, time-series analysis, etc.

In other words, we can say data mining is the root of our data mining architecture. It comprises
instruments and software used to obtain insights and knowledge from data collected from various
data sources and stored within the data warehouse.

5. Pattern Evaluation Module:

The Pattern evaluation module is primarily responsible for the measure of investigation of the
pattern by using a threshold value. It collaborates with the data mining engine to focus the search
on exciting patterns.

6. Graphical User Interface:

The graphical user interface (GUI) module communicates between the data mining system and
the user. This module helps the user to easily and efficiently use the system without knowing the
complexity of the process. This module cooperates with the data mining system when the user
specifies a query or a task and displays the results.

7. Knowledge Base:

The knowledge base is helpful in the entire process of data mining. It might be helpful to guide
the search or evaluate the stake of the result patterns. The knowledge base may even contain user
views and data from user experiences that might be helpful in the data mining process. The data
mining engine may receive inputs from the knowledge base to make the result more accurate and
reliable. The pattern assessment module regularly interacts with the knowledge base to get
inputs, and also update it.

7. DATA DISCRETIZATION AND CONCEPT HIERARCHY

GENERATION
What is Data Discretization?

Data discretization is the process of converting continuous numerical data into discrete
categorical bins or intervals. It is used to:
* Improve efficiency in machine learning models.
* Handle noisy data by grouping similar values.
* Simplify data representation for better interpretability.

Techniques for Data Discretization:

Binning (Equal-Width & Equal-Frequency)

Divides data into bins of equal size or frequency.

1. Equal-Width Binning

Splits data into bins with a fixed range size.

🔹 Example: If values range from 1 to 100, and we choose 5 bins, each bin will have a width
of (100-1)/5 = 19.8.

import pandas as pd

data = [5, 12, 23, 36, 47, 52, 68, 73, 85, 99]

df = [Link](data, columns=["Value"])

# Apply Equal-Width Binning

df["Equal_Width_Bin"] = [Link](df["Value"], bins=5, labels=["Very Low", "Low", "Medium",

"High", "Very High"])

print(df)

[Link]-Frequency Binning

Each bin contains approximately the same number of data points.

df["Equal_Frequency_Bin"] = [Link](df["Value"], q=5, labels=["Very Low", "Low",
"Medium", "High", "Very High"])

print(df)

Clustering-Based Discretization

Uses clustering algorithms like K-Means to group similar values.

from [Link] import KMeans

import numpy as np

data = [Link](data).reshape(-1, 1)

kmeans = KMeans(n_clusters=3)

df["Cluster"] = kmeans.fit_predict(data)

print(df)

Decision Tree-Based Discretization

Uses a decision tree to determine the best splits.

from [Link] import DecisionTreeClassifier

X = [Link](df["Value"]).reshape(-1, 1)

y = [0, 0, 1, 1, 1, 2, 2, 2, 3, 3] # Example labels for discretization

tree = DecisionTreeClassifier(max_depth=3)

[Link](X, y)

df["Tree_Bin"] = [Link](X)

print(df)

Concept Hierarchy Generation

What is Concept Hierarchy?

Concept hierarchy organizes data at different levels of abstraction to:

*Enable roll-up and drill-down analysis in OLAP.
* Help in data generalization for better insights.
*Improve classification accuracy by grouping similar values.

Examples of Concept Hierarchies:

Hierarchy for Age Grouping

(1-10) → Child

(11-20) → Teenager

(21-35) → Young Adult

(36-60) → Adult

(60+) → Senior

def age_group(age):

if age <= 10: return "Child"

elif age <= 20: return "Teenager"

elif age <= 35: return "Young Adult"

elif age <= 60: return "Adult"

else: return "Senior"

df["Age_Category"] = df["Value"].apply(age_group)

print(df)

Geographic Hierarchy

City → State → Country → Continent

geo_hierarchy = {

"New York": "USA",

"Los Angeles": "USA",

"London": "UK",

"Mumbai": "India"
}

df["Country"] = df["City"].map(geo_hierarchy)

Benefits of Concept Hierarchy

Improves data aggregation for reports.

* Supports multi-level analysis in Business Intelligence.
* Reduces complexity in machine learning models.

8. ASSOCIATION RULE MINING: - EFFICIENT AND

SCALABLE FREQUENT ITEM SET MINING METHODS:
Association Rule Mining is a fundamental technique in data mining used to identify
relationships between variables in large datasets. The most common application is in market
basket analysis, where businesses discover product purchase patterns.

To efficiently mine frequent itemsets (sets of items that appear together frequently), various
scalable algorithms have been developed.

1. Apriori Algorithm

Apriori is one of the earliest and most popular algorithms for frequent itemset mining.

Steps:

Generate Candidate Itemsets – Start with individual items and iteratively form larger itemsets.

Prune Infrequent Itemsets – Discard itemsets that do not meet a predefined minimum support
threshold.

Generate Association Rules – Identify strong relationships between items.

Efficiency Improvements:

Uses the "Apriori Property": If an itemset is frequent, then all its subsets must also be frequent.

Reduces the search space by eliminating non-frequent itemsets early.

Limitation: Requires multiple database scans, which can be slow for large datasets.

2. FP-Growth (Frequent Pattern Growth) Algorithm

FP-Growth is an improvement over Apriori that eliminates the need for multiple database scans.
Steps:

Build an FP-Tree – A compressed representation of the dataset.

Recursively Find Frequent Itemsets – Uses a divide-and-conquer approach to extract patterns

without generating candidate itemsets explicitly.

Efficiency Improvements:

Uses compact FP-Tree structure, reducing memory usage.

Avoids multiple database scans, making it faster than Apriori.

Limitation: More complex implementation and requires additional memory for the FP-Tree.

3. ECLAT (Equivalence Class Transformation)

ECLAT is a depth-first search approach that improves efficiency by using vertical data format
instead of horizontal.

Steps:

Transform Data into a Vertical Format – Each item is stored with a list of transactions in which it
appears.

Intersect Transaction Sets – Find common transactions among itemsets.

Efficiency Improvements:

Uses set intersections instead of database scans.

Works well with dense datasets.

Limitation: Not as efficient for very large datasets with sparse transactions.

Choosing the Right Algorithm:

Algorithm Strengths Weaknesses

Multiple database scans, slow on large
Apriori Easy to understand, widely used
datasets
FP-
Faster than Apriori, less memory-intensive Complex to implement
Growth
ECLAT Efficient for dense datasets, no candidate Not ideal for sparse datasets
Algorithm Strengths Weaknesses
generation

Scalability Improvements:

Parallel and Distributed Processing – Use Hadoop, Spark, or GPUs to mine large-scale datasets.

Sampling Techniques – Analyze a representative subset of data instead of the full dataset.

Incremental Mining – Update frequent itemsets dynamically as new data arrives, instead of
reprocessing everything.

9. MINING VARIOUS KINDS OF ASSOCIATION RULES IN

DATA MINING
Association Rule Mining (ARM) is a powerful technique for discovering hidden patterns in large
datasets. Traditional ARM focuses on frequent item sets and simple association rules, but real-
world applications require mining various types of association rules to capture more meaningful
relationships.

Types of Association Rules

1. Single-Dimensional vs. Multi-Dimensional Association Rules

 Single-Dimensional: Involves rules from a single attribute (e.g., Market Basket

Analysis: {Milk, Bread} → {Butter}).
 Multi-Dimensional: Incorporates multiple attributes in rules (e.g., {Age = 25-30, Income
= High} → {Buys Laptop}).
 Example: A supermarket might find that young professionals with high income are more
likely to buy organic food.

2. Quantitative Association Rules

 Handles continuous numeric attributes by discretizing them into intervals.

 Example: {Age = 20-30} ∧ {Salary = $50K-$70K} → {Buys SUV}.
 Use Case: Helps businesses target specific income groups for promotions.

3. Generalized Association Rules

 Leverages hierarchical relationships in data (e.g., category-level rules).

 Example: {Dairy Products} → {Bakery Products} instead of {Milk} → {Bread}.
 Optimization: Uses taxonomy trees to group items into higher-level categories.
4. Correlation-Based Association Rules

 Traditional ARM relies on support & confidence, but high confidence doesn’t imply
strong correlation.
 Uses correlation measures like Lift, Conviction, and Cosine Similarity to discover
strong relationships.
 Example: {Beer} → {Diapers} may have high support & confidence but must be
validated using correlation.

5. Sequential Association Rules (Sequential Pattern Mining)

 Finds time-ordered relationships in sequences.

 Example: {Buys Smartphone} → {Buys Accessories} → {Buys Warranty} (customer
behavior over time).
 Use Case: Used in e-commerce, web click stream analysis, and customer retention
strategies.

6. Weighted Association Rules

 Assigns different importance (weights) to items instead of treating all equally.

 Example: In medical data mining, {High Blood Pressure} → {Heart Disease} may have
higher weight than {Headache} → {Fever}.
 Use Case: Healthcare, fraud detection, and high-value transaction analysis.

7. Constraint-Based Association Rules

 Allows users to set conditions on rules (e.g., only finding rules with high profit or
involving specific items).
 Example: {Smartphone} → {Accessories} but only when accessories cost more than
$50.
 Use Case: Custom rule mining for specific business needs.

Different types of association rules allow for more meaningful insights beyond basic frequent
item set mining. Whether it's sequential, negative, weighted, or generalized rules, each type
serves a specific real-world application like retail, healthcare, finance, and fraud detection.
10. PATTERN MINING IN MULTILEVEL,
MULTIDIMENSIONAL SPACE – CONSTRAINT:
1. Introduction

Pattern mining in multilevel and multidimensional spaces extends traditional association rule
mining by considering hierarchical relationships and multiple attributes. Constraints are applied
to focus on meaningful patterns and reduce computational complexity.

2. Multilevel Pattern Mining

Multilevel pattern mining identifies patterns across different levels of abstraction within a
hierarchy.

Example: Retail Data Hierarchy

Scenario: Supermarket Sales Data

We have a dataset with the following attributes:

 Transaction ID
 Customer Age (Quantitative)
 Income Level (Categorical: Low, Medium, High)
 Product Category (Multilevel: Beverages → Tea → Green Tea)
 Purchased Items

Dataset (Sample Transactions)

Transaction ID Product Purchased Category Level 1 Category Level 2 Category Level 3

1001 Green Tea Beverages Tea Green Tea

1002 Black Tea Beverages Tea Black Tea

1003 Cheese Dairy Milk Products Cheese

1004 Whole Milk Dairy Milk Products Whole Milk

1005 Potato Chips Snacks Chips Potato Chips

Product Categories:

Level 1: Beverages → Dairy → Snacks

Level 2: Tea, Coffee → Milk, Cheese → Chips, Cookies

Level 3: Green Tea, Black Tea → Almond Milk, Whole Milk → Potato Chips, Corn Chips

Types of Multilevel Association Rules

Uniform Support Approach:

Uses the same minimum support for all levels.

Issue: Higher-level categories have more frequent items, causing loss of specific patterns at
lower levels.

Reduced Support Approach:

Lower support threshold for deeper levels.

Example: {Dairy} → {Cheese} (5%) vs. {Almond Milk} → {Organic Cheese} (1%).

Captures more detailed insights.

Item-Specific Support:

Different support values for different item categories.

Helps balance frequent and infrequent itemsets.

3. Multidimensional Pattern Mining

Multidimensional pattern mining considers multiple attributes beyond just items in

transactions.

Example: Supermarket Transactions

Single-dimensional rule: {Milk} → {Bread}

Multi-dimensional rule: {Age = 25-35, Income = High} → {Buys Organic Food}

Types of Dimensions

✔ Quantitative Dimensions: Age, Income, Price (Numeric values)

✔ Categorical Dimensions: Gender, Location, Product Type

Mining Strategies

Static Discretization: Pre-defines intervals for numerical attributes.

Example: Age groups → (0-20, 21-40, 41-60)

Dynamic Discretization: Creates intervals dynamically based on data distribution.

4. Constraint-Based Pattern Mining

Applying constraints helps filter patterns to focus only on interesting rules.

Types of Constraints

1. Knowledge-Based Constraints:

Domain experts define rules based on business logic.

Example: Only find rules involving "Luxury Products".

2. Data Constraints:

Set conditions on attributes like price range or location.

Example: {Price > $50} → {Premium Customer}.

3. Interestingness Constraints:

Use correlation measures like Lift (>1) to filter weak rules.

4. Length Constraints:

Limit rule size (e.g., only find 2-item rules).

Avoids overly complex rules like {A, B, C, D} → {E, F, G}.

5. Aggregation Constraints:

Use SUM, AVG, COUNT in quantitative rule mining.

Example: {Total Purchase > $500} → {Premium Membership}.

5. Real-World Applications

E-Commerce: Personalized recommendations based on multi-attribute data (e.g., Age, Purchase

History, Region).
Healthcare: Finding disease patterns across patient demographics and symptoms.
Finance: Fraud detection using transaction attributes (amount, location, frequency).

6. Conclusion

Multilevel and multidimensional pattern mining enhance traditional rule mining by

considering hierarchical relationships and multiple attributes. Constraints help focus on
meaningful patterns, improving efficiency and relevance.

11. ASSOCIATION MINING TO CORRELATION ANALYSIS:

1. Introduction

Association Rule Mining (ARM) is used to discover relationships between items in

transactional datasets using support, confidence, and lift. However, high confidence does not
always indicate a true relationship. Correlation Analysis validates these rules using statistical
methods like Pearson Correlation, Lift, and Chi-Square tests to measure the actual strength of
relationships.

2. Association Rule Mining (ARM)

ARM finds relationships between items in the form:

X⇒YX \Rightarrow YX⇒Y

where:

 Support: How often X and Y appear together.

 Confidence: Probability of buying Y given X.
 Lift: Strength of association beyond chance.

Example: Retail Transactions

Transaction ID Items Purchased

1001 Milk, Bread, Butter

1002 Milk, Bread

Transaction ID Items Purchased

1003 Milk, Butter

1004 Bread, Butter

1005 Milk, Bread, Butter, Eggs

ARM Rule Example:

Rule: {Milk, Bread} → {Butter}

 Support: 3/5 = 60%

 Confidence: 3/3 = 100%
 Lift: Lift=P(Milk,Bread,Butter)P(Milk,Bread)∗P(Butter)Lift = \frac{P(Milk, Bread,
Butter)}{P(Milk, Bread) * P(Butter)}Lift=P(Milk,Bread)∗P(Butter)P(Milk,Bread,Butter)
If Lift > 1, the rule is significant.

Issue: High confidence does not always mean true correlation.

3. Moving to Correlation Analysis

We apply correlation measures to check the true relationship between items.

(a) Pearson Correlation Coefficient (ρ)

Measures linear dependence between two items:

ρ(X,Y)=P(X,Y)−P(X)P(Y)P(X)(1−P(X))P(Y)(1−P(Y))ρ(X, Y) = \frac{P(X, Y) -
P(X)P(Y)}{\sqrt{P(X)(1-P(X)) P(Y)(1-P(Y))}}ρ(X,Y)=P(X)(1−P(X))P(Y)(1−P(Y))
P(X,Y)−P(X)P(Y)

 ρ > 0 → Positive correlation (bought together frequently).

 ρ = 0 → No correlation (independent).
 ρ < 0 → Negative correlation (rarely bought together).

Example Calculation for {Milk, Butter}:

 P(Milk) = 4/5 = 80%

 P(Butter) = 4/5 = 80%
 P(Milk, Butter) = 3/5 = 60%
 ρ(Milk, Butter) = 0.75 → Strong positive correlation ✅
(b) Lift Measure

Measures how much more likely items appear together than randomly:

Lift(X,Y)=P(X,Y)P(X)P(Y)Lift(X, Y) = \frac{P(X, Y)}{P(X) P(Y)}Lift(X,Y)=P(X)P(Y)P(X,Y)

 Lift > 1 → Items are positively correlated.

 Lift = 1 → No relationship.
 Lift < 1 → Negative correlation.

Example:

 {Milk, Butter} → Lift = 1.5 (Strong relationship).

 {Milk, Eggs} → Lift = 0.8 (Weak relationship).

(c) Chi-Square (χ²) Test

Tests whether two items appear together due to chance or dependency:

χ2=∑(Observed−Expected)2Expectedχ² = \sum \frac{(Observed -

Expected)^2}{Expected}χ2=∑Expected(Observed−Expected)2

 High χ² → Strong dependency.

 Low χ² → Weak or no association.

Use Case: Validates if high-confidence rules are statistically significant.

4. Combining ARM & Correlation Analysis

To improve association mining, we:

1. Filter rules where Lift > 1 (avoid random co-occurrence).

2. Check correlation (ρ > 0.5) for true relationships.
3. Use Chi-Square (χ²) tests to confirm statistical significance.

Final Rule Selection:

{Milk, Bread} → {Butter} (High Lift + Strong Correlation).
{Milk} → {Eggs} (High Confidence but Low Correlation).
5. Real-World Applications

✔Retail & E-Commerce → Product recommendations.

✔Healthcare → Finding disease co-occurrence patterns.
✔Fraud Detection → Identifying suspicious transactions.

6. Conclusion

ARM finds frequent patterns, but correlation analysis confirms true relationships.
Combining ARM with correlation methods reduces misleading rules.
Businesses can refine recommendations and strategies using this hybrid approach.