Unit – 2: Exploratory Data Analysis (EDA) for
Pattern Detection
1. Introduction to Exploratory Data Analysis (EDA)
Exploratory Data Analysis (EDA) is a fundamental step in data analysis and machine learning,
where the main goal is to understand the structure, patterns, relationships, and anomalies
present in the dataset before applying any model.
EDA helps in:
• Discovering hidden patterns
• Detecting outliers and anomalies
• Understanding relationships between variables
• Checking assumptions for further analysis
EDA is mostly performed using statistical methods and visualization techniques, which allow
analysts to summarize and interpret data effectively.
2. Data Visualization Techniques for Exploring Patterns
Data visualization is one of the most powerful tools in EDA because it allows us to represent data
graphically, making it easier to identify trends, patterns, and relationships.
2.1 Importance of Data Visualization
• Simplifies complex data
• Helps in quick pattern recognition
• Reveals hidden trends
• Improves decision-making
2.2 Common Visualization Techniques
a) Histogram
A histogram represents the distribution of a
numerical variable by dividing data into bins.
Theory:
• Shows frequency distribution
• Helps identify skewness and spread
• Useful for detecting outliers
b) Box Plot (Box-and-Whisker Plot)
A box plot summarizes data using five-
number summary:
• Minimum
• Q1 (First Quartile)
• Median
• Q3 (Third Quartile)
• Maximum
Theory:
• Helps in detecting outliers
• Shows data spread and central tendency
c) Scatter Plot
A scatter plot shows the relationship between two
variables.
Theory:
• Used to detect correlation
• Helps identify clusters and trends
• Indicates linear or non-linear relationships
d) Line Plot
• Used to show trends over time
• Common in time-series analysis
e) Bar Chart
• Represents categorical data
• Used for comparison between categories
2.3 Advanced Visualization
• Heatmaps (for correlation)
• Pair plots (multiple variable relationships)
• Violin plots (distribution + density)
3. Statistical Measures for Identifying Patterns and
Correlations
Statistical analysis is essential in EDA to quantify patterns and relationships.
3.1 Measures of Central Tendency
These measures describe the center of the dataset.
• Mean (Average): Sum of values / total values
• Median: Middle value
• Mode: Most frequent value
Theory:
• Mean is sensitive to outliers
• Median is robust for skewed data
3.2 Measures of Dispersion
These describe the spread or variability of data.
• Range: Max – Min
• Variance: Average squared deviation
• Standard Deviation: Square root of variance
Theory:
• High deviation → data widely spread
• Low deviation → data concentrated
3.3 Correlation Analysis
Correlation measures the strength and direction of relationship between variables.
• Positive correlation: Both variables increase together
• Negative correlation: One increases, other decreases
• Zero correlation: No relationship
Example: Height and weight (positive correlation)
3.4 Covariance
• Indicates direction of relationship
• Does not standardize the strength
3.5 Skewness and Kurtosis
• Skewness: Measures asymmetry of distribution
• Kurtosis: Measures peakness of distribution
3.6 Outlier Detection
Outliers are extreme values that differ from other observations.
Methods:
• Box plot
• Z-score
• IQR (Interquartile Range)
4. Feature Selection for Pattern Detection
Feature selection refers to the process of selecting the most relevant variables (features) from the
dataset.
4.1 Importance of Feature Selection
• Reduces model complexity
• Improves accuracy
• Removes irrelevant or redundant data
• Reduces overfitting
4.2 Types of Feature Selection Methods
a) Filter Methods
• Based on statistical tests
• Example: Correlation, Chi-square
b) Wrapper Methods
• Use machine learning models to evaluate features
• Example: Recursive Feature Elimination (RFE)
c) Embedded Methods
• Feature selection occurs during model training
• Example: Decision Trees
5. Feature Engineering for Pattern Detection
Feature engineering is the process of creating new features or transforming existing ones to
improve model performance.
5.1 Importance
• Improves predictive power
• Helps uncover hidden patterns
• Enhances model efficiency
5.2 Techniques of Feature Engineering
a) Handling Missing Values
• Mean/median imputation
• Removing rows
b) Encoding Categorical Data
• Label Encoding
• One-Hot Encoding
c) Feature Scaling
• Normalization
• Standardization
d) Creating New Features
• Combining existing features
• Example: BMI = weight / height²
e) Transformation
• Log transformation
• Polynomial features
6. Pattern Detection using EDA
EDA helps detect patterns such as:
• Trends (increase/decrease over time)
• Clusters (grouping of data points)
• Relationships (correlation between variables)
• Anomalies (outliers)
These patterns are essential for:
• Decision making
• Predictive modeling
• Data-driven insights