Data Analytics Fundamentals
Data Pre-processing
Handling Outliers in Data Preprocessing
When working with real-world data, not all values fit neatly within expected patterns.
Sometimes, certain data points stand out from the rest — these are known as outliers.
Handling them properly is an essential step in data preprocessing, as outliers can
significantly affect the performance and accuracy of your models.
What Are Outliers?
Outliers are values that differ greatly from most other observations in a dataset.
For example, if most students score between 60–80 marks, but one student scores 5 or 100,
those extreme values are outliers.
They can appear due to:
• Data entry errors (e.g., typing 1000 instead of 100)
• Measurement errors (e.g., faulty sensors)
• Natural variation (e.g., a genuine but rare event)
Types of Outliers
Outliers can be classified into various types based on their characteristics:
1. Global Outliers: Also known as point anomalies, these data points significantly differ
from the rest of the dataset.
2. Contextual Outliers: These are data points that are considered outliers in a specific
context. For example, a high temperature may be normal in summer but an outlier in
winter.
3. Collective Outliers: A collection of data points that deviate significantly from the rest
of the dataset, even if individual points within the collection are not outliers.
NOTES BY: DR. AMITAVA BISWAS 1
Why Are Outliers a Problem?
Outliers can:
• Distort statistical measures like mean and standard deviation.
(For instance, one extreme value can pull the average far away from the majority.)
• Mislead machine learning models, especially algorithms sensitive to scale or
distance (like regression or KNN).
• Affect data visualization, making patterns harder to see.
Detecting Outliers
There are several simple ways to identify outliers:
• Visual Methods:
o Box Plot: Shows data spread and highlights points beyond whiskers (potential
outliers).
o Scatter Plot: Reveals points that don’t follow the general trend.
• Statistical Methods:
o Z-Score Method:
Measures how far a data point is from the mean in terms of standard
deviations.
(𝑋 − mean)
𝑍=
std deviation
Typically, points with |Z| > 3 are considered outliers.
o IQR (Interquartile Range) Method:
Calculate IQR = Q3 − Q1.
Any point below Q1 − 1.5×IQR or above Q3 + 1.5×IQR is an outlier.
Handling Outliers
Once detected, you can handle outliers in several ways depending on their cause and
importance:
1. Remove Outliers:
If they are due to data errors or noise, you can remove them.
(Be cautious — don’t remove valuable rare events.)
2. Cap or Floor Outliers (Winsorizing):
Replace extreme values with a threshold (e.g., 1st or 99th percentile).
3. Transform Data:
Apply transformations (like log, square root, or Box-Cox) to reduce the effect of
outliers.
4. Use Robust Models:
Some algorithms (like decision trees or median-based models) are less sensitive to
outliers.
Methods for Outlier Detection in brief..
Outlier detection is a critical task in data analysis, crucial for ensuring the quality and
reliability of conclusions drawn from data. Different techniques are tailored for varying data
NOTES BY: DR. AMITAVA BISWAS 2
types and scenarios, ranging from statistical methods for general data sets to specialized
algorithms for spatial and temporal data. Some techniques are:
Standard Deviation Method
Standard Deviation Method is based on the assumption that the data follows a normal
distribution. Data points outside of three standard deviations from the mean are considered
outliers.
It is commonly used for univariate data analysis where the distribution can be assumed to be
approximately normal.
• Step 1: Calculate the average and standard deviation of the data set, if applicable.
• Step 2: Define the lower and upper bounds for outliers.
• Step 3: Identify outliers as data points that fall outside these bounds:
Example: Dataset: [1, 2, 2, 3, 1, 3, 10]. Find an outlier using the Standard Deviation Method.
(1+2+2+3+1+3+10) 22
Mean, 𝜇 = = ≈ 3.14
7 7
Standard Deviation, 𝑠 ≈
(1−3.14)2 +(2−3.14)2 +(2−3.14)2 +(3−3.14)2 +(1−3.14)2 +(3−3.14)2 +(10−3.14)2
√ ≈ 2.90
7
Lower and upper bounds for outliers:
• Lower bound = 3.142857 - 2 X 2.917 3.142857 - 5.834 = -2.691
• Upper bound = 3.142857 + 2 X 2.917 3.142857 + 5. 834 = 8.977
Any values outside [-2.691, 8.977] is an outlier, thus 10 is an outlier.
So, the data point 10 is identified as an outlier using the Standard Deviation Method.
IQR Method
The Interquartile Range (IQR) method focuses on the spread of the middle 50% of data. It
calculates the IQR as the difference between the 75th and 25th percentiles of the data and
identifies outliers as those points that fall below 1.5 times the IQR below the 25th percentile
or above 1.5 times the IQR above the 75th percentile. This method is robust to outliers and
does not assume a normal distribution.
• Step 1: Find Q1(25th percentage) and Q3(75th percentage)
• Step 2: IQR = Q3 - Q1.
• Step 3: Find Lower Bound: Q1 - 1.5 × IQR and Upper Bound Lower Bound: Q1 - 1.5 ×
IQR
It is suitable for datasets with skewed or non-normal distributions. Useful for identifying
outliers in datasets where the spread of the middle 50% of the data is more relevant than
the mean and standard deviation.
Example: Dataset X = {3,5,7,9,11,13,30}, find outlier using the IQR method.
• Q1 (25th percentile): Median of first half = Median of [3, 5, 7] = 5
• Q3 (75th percentile): Median of second half = Median of [11, 13, 30] = 13
IQR = Q3 − Q1 = 13 − 5 = 8
Lower Bound: Q1 - 1.5 × IQR = 5 - 1.5 × 8 = 5 - 12 = -7
NOTES BY: DR. AMITAVA BISWAS 3
Upper Bound: Q3 + 1.5 × IQR = 13 + 1.5 × 8 =13 + 12 = 25
Therfore the interval is -7 to 25. 30 lies outside the interval, therefore is an outlier.
Z-Score Method
The Z-score method calculates the number of standard deviations each data point is from
the mean. A Z-score threshold is set, commonly 3, and any data point with a Z-score
exceeding this threshold is considered an outlier. This method assumes a normal distribution
and is sensitive to extreme values in small datasets.
• Step 1: Calculate the mean.
• Step 2: Compute Standard Deviation
• Step 3:Calculate z-scores
• Step 4: Apply Threshold Rule: Mild outlier: |Z| > 2 and Extreme outlier: |Z| > 3
Suitable for datasets with large sample sizes and where the underlying distribution of the
data can be reasonably approximated by a normal distribution.
Example: X={4,5,5,6,7,8,20}, find outlier using the Z-score method.
4+5+5+6+7+8+20 55
Mean, 𝑋ˉ = = ≈ 7.86
7 7
(4−7.86)2 +(5−7.86)2 ×2+(6−7.86)2 +(7−7.86)2 +(8−7.86)2 +(20−7.86)2
Standard Deviation, 𝑠 ≈ √ ≈ 5.36
6
𝑋𝑖 −7.86
Z-scores: 𝑍𝑖 = 5.36
Z-score for all data points: 4: -0.72, 5: -0.53, 5: -0.53, 6: -0.35, 7: -0.16, 8: 0.03, 20: 2.26
Threshold Z > 2 := 20 is an outlier since |2.26| > 2
The choice of outlier detection technique depends on the characteristics of the data, the
underlying distribution, and the specific requirements of the analysis.
Challenges with Outlier Detection
Detecting outliers effectively poses several challenges:
• Determining the Threshold: Deciding the correct threshold that accurately separates
outliers from normal data is critical and difficult.
• Distinguishing Noise from Outliers: In datasets with high variability or noise, it can
be particularly challenging to differentiate between noise and actual outliers.
• Balancing Sensitivity: An overly aggressive approach to detecting outliers might
eliminate valid data, reducing the richness of the dataset.
Applications of Outlier Detection
Outlier detection plays a crucial role across various domains, enabling the identification of
anomalies that can indicate errors, fraud, or novel insights. Here are some key applications
of outlier detection with specific examples:
1. Financial Fraud Detection
• Fraud Detection: Outlier detection is extensively used in the financial sector to
identify fraudulent activities. For instance, credit card companies use outlier
NOTES BY: DR. AMITAVA BISWAS 4
detection algorithms to flag unusual spending patterns that may indicate stolen card
usage.
• Example: A credit card transaction for a large amount in a foreign country when the
cardholder usually makes small, local purchases could be flagged as an outlier,
triggering a fraud alert.
2. Cybersecurity
• Network Intrusion Detection: Outlier detection is critical in cybersecurity for
identifying unusual patterns of network traffic that could indicate a security breach.
• Example: A sudden increase in data transmission to an external IP address not
previously contacted by the network could be an outlier, suggesting a potential data
exfiltration attack.
3. AI/ML Modeling
• Data Cleaning: To prevent model skewing in training data
• In reducing Bias: Detects biased predictions.
4. Anomaly Detection in Big Data & Cloud Systems
• Cloud Security: To detect unauthorized access in large-scale cloud environments.
• Ensures integrity by flagging corrupted entries.
Example:
Suppose you are analyzing house prices, and one property is worth ₹10 crore while most
others are between ₹20–60 lakhs.
That one ₹10 crore house is an outlier — it could represent a luxury property, not an error.
So instead of deleting it blindly, you might analyze it separately or cap its value when
building a prediction model.
Feature scaling (StandardScaler, MinMaxScaler)
Feature scaling is a data preprocessing technique used in machine learning to standardize
the range of independent variables or features within a dataset. This is crucial because many
machine learning algorithms are sensitive to the scale of input features, and features with
vastly different scales can lead to biased learning or poor model performance.
StandardScaler (Standardization or Z-score normalization):
• What it does: StandardScaler transforms features to have a mean of 0 and a standard
deviation of 1. It achieves this by subtracting the mean of the feature and then
dividing by its standard deviation.
Python
X_scaled = (X - mean(X)) / std(X)
• When to use it:
o When the data is approximately normally distributed (Gaussian).
o With algorithms that assume features are centered around zero and have
similar variances, such as Linear Regression, Logistic Regression, Support
Vector Machines (SVMs), and Principal Component Analysis (PCA).
o When outliers are not a major concern, as it can be influenced by extreme
values.
NOTES BY: DR. AMITAVA BISWAS 5
MinMaxScaler (Normalization):
• What it does: MinMaxScaler scales features to a specified range, typically between 0
and 1. It achieves this by subtracting the minimum value of the feature and then
dividing by the range (maximum value - minimum value).
Python
X_scaled = (X - min(X)) / (max(X) - min(X))
• When to use it:
o When the data does not follow a normal distribution.
o With algorithms that are not sensitive to the distribution of data but benefit
from features being within a specific range, such as K-Nearest Neighbors
(KNN) and neural networks.
o When the presence of outliers is minimal or can be handled separately, as
MinMaxScaler is sensitive to extreme values due to its reliance on the
minimum and maximum.
Key Differences and Considerations:
• Distribution:
StandardScaler transforms the data to follow a standard normal distribution, while
MinMaxScaler scales data to a specific range without changing its original distribution shape.
• Outliers:
StandardScaler is less sensitive to outliers than MinMaxScaler, as it uses the mean and
standard deviation, which are less affected by extreme values than the min and max.
• Algorithm Compatibility:
The choice between StandardScaler and MinMaxScaler often depends on the specific
machine learning algorithm being used and its assumptions about the input data.
Comparison Table
Aspect StandardScaler MinMaxScaler
Formula (x – mean) / std (x – min) / (max – min)
Output Range No fixed range (mean=0, std=1) [0, 1] (or custom range)
Sensitive to Outliers Yes Yes, but can be worse if outliers exist
When to Use Normally distributed data Bounded or non-normal data
Example Use Case SVM, Logistic Regression Neural Networks, KNN
Encoding categorical variables (one-hot encoding, label encoding)
What is Categorical Data?
Categorical data, also known as nominal or ordinal data, is a type of data that consists of
values that fall into distinct categories or groups. Unlike numerical data, which represents
measurable quantities, categorical data represents qualitative or descriptive characteristics.
It is crucial to understand categorical data when working with machine learning models, as
most models require numerical inputs.
NOTES BY: DR. AMITAVA BISWAS 6
Examples of Categorical Data
Categorical variables can be represented as strings or labels and have a finite number of
possible values. Here are a few common examples:
• The city where a person lives (e.g., Delhi, Mumbai, Ahmedabad, Bangalore)
• The department a person works in (e.g., Finance, Human Resources, IT, Production)
• The highest degree a person has (e.g., High School, Diploma, Bachelor’s, Master’s,
PhD)
• The grades of a student (e.g., A+, A, B+, B, B-)
Types of Categorical Data
There are two main types of categorical data:
1. Ordinal Data
Ordinal data refers to categories that have an inherent order or ranking. When encoding
ordinal data, it’s essential to retain the information about the order in which the categories
are provided. For example, when considering the highest degree a person possesses, the
degree level provides vital information about their qualification, which can be an important
feature in determining their suitability for a job.
2. Nominal Data
Nominal data refers to categories that do not have an inherent order or ranking. When
encoding nominal data, the presence or absence of a feature is considered, but the order is
not relevant. For instance, when considering the city where a person lives, it’s important to
retain the information about which city they live in, but there is no particular order or
sequence among the cities (e.g., living in Delhi or Bangalore is equal).
By understanding the nature of categorical data and the distinction between ordinal and
nominal data, data scientists and machine learning practitioners can make informed
decisions about the appropriate encoding techniques to use, ensuring that the valuable
information contained within categorical variables is effectively captured and utilized by
their models.
Types of Encoding in Machine Learning
1. Identify Categorical Features:
• First, look at your data and find the features that contain non-numerical values like
text labels or categories (e.g., color, size, customer type). These are the features that
need encoding.
2. Choose an Encoding Technique:
• There are different encoding methods, each with its pros and cons. Here’s a quick
guide to some popular choices:
o One-Hot Encoding: Great for many categories, creates new binary features (1
for the category, 0 for others).
o Label Encoding: Simple, assigns a number to each category, but assumes
order matters (which might not be true).
NOTES BY: DR. AMITAVA BISWAS 7
o Ordinal Encoding: Similar to label encoding, but only use it if categories have
a natural order (like low, medium, high).
3. Apply the Encoding:
• Once you’ve chosen your technique, it’s time to apply it to your data. Many machine
learning libraries have built-in functions for encoding.
o For example, with one-hot encoding, you might create a new binary feature
for each category in your original feature.
4. Test and Refine (Optional):
• In some cases, you might want to try different encoding techniques and see how
they affect your machine learning model’s performance. This can help you find the
best approach for your specific data.
Now..
Encoding categorical variables is a crucial step in data preprocessing for machine learning, as
most algorithms require numerical input. Two common techniques are One-Hot Encoding
and Label Encoding are discussing here.
1. Label Encoding
Label Encoding is the simplest method.
Each unique category is assigned a numeric value.
Example:
City Encoded Value
Delhi 0
Mumbai 1
Chennai 2
Now, “Delhi”, “Mumbai”, and “Chennai” are replaced by 0, 1, and 2 respectively.
Advantages:
• Very simple to apply.
• Works well for ordinal data (data with an order, like “Low”, “Medium”, “High”).
Limitations:
• The model might mistakenly assume a numerical relationship between the values
(e.g., that Mumbai > Delhi because 1 > 0), which can mislead algorithms for nominal
data (no natural order, like city names).
Use Label Encoding mainly when categories have a clear ranking or order.
2. One-Hot Encoding
One-Hot Encoding solves the problem of “implied order” by creating separate binary
columns for each category.
Example:
City Delhi Mumbai Chennai
Delhi 1 0 0
NOTES BY: DR. AMITAVA BISWAS 8
City Delhi Mumbai Chennai
Mumbai 0 1 0
Chennai 0 0 1
Here, each city becomes its own column with values 0 or 1 (representing whether the row
belongs to that category).
Advantages:
• No false assumptions of order.
• Works well for most machine learning models.
Limitations:
• Increases the number of features (especially when a column has many unique
categories — known as the “curse of dimensionality”).
When to Use Which
Situation Best Encoding Method
Categories with order (e.g., Low,
Label Encoding
Medium, High)
Categories without order (e.g., City,
One-Hot Encoding
Color, Country)
Try Target Encoding or Frequency Encoding
Too many unique categories
(advanced techniques)
Choosing between One-Hot Encoding and Label Encoding:
• Nominal Data:
One-Hot Encoding is generally preferred to avoid introducing artificial ordinality.
• Ordinal Data:
Label Encoding is suitable, as it preserves the inherent order.
• High Cardinality:
If a nominal feature has a very large number of unique categories, One-Hot Encoding can
lead to a significant increase in dimensionality. In such cases, other techniques like Binary
Encoding or target-based encoding might be considered, or Label Encoding could be used if
the potential for misinterpretation by the model is managed.
• Model Type:
Tree-based models (e.g., Random Forest, Gradient Boosting) are often less sensitive to the
artificial ordinality introduced by Label Encoding on nominal data, while linear models are
more susceptible. Some advanced tree-based models (e.g., LightGBM) can handle
categorical variables directly without explicit encoding.
NOTES BY: DR. AMITAVA BISWAS 9