APPLIED DATA SCIENCE (ADS)

SEM – 8 | BE – COMPUTER ENGINEERING

MODULE 4
ANOMALY DETECTION
SYLLABUS Outliers · Causes of Outliers · Anomaly Detection Techniques · Outlier Detection
4.1 using Statistics

SYLLABUS Outlier Detection using Distance-based Methods · Outlier Detection using

4.2 Density-based Methods · SMOTE

Each question in exam is 10 marks — these notes cover all exam-oriented Q&A in full detail.
SECTION 4.1 — OUTLIERS, CAUSES & ANOMALY DETECTION
TECHNIQUES

4.1.1 What is an Anomaly / Outlier?

An anomaly is a data point that behaves very differently from the rest of the data. In a dataset, outliers
are stragglers — extremely high or low values that lie far away from all other observations. They are also
called abnormal observations, aberrant values, or rare events.

An outlier is a data point that DIFFERS SIGNIFICANTLY from other observations

DEFINITION in the dataset. It can distort statistical measures (mean, std. dev.) and affect the
performance of machine learning models.

Outliers are important because:

• They can indicate errors in data collection or processing (bad data).
• They can represent genuine extreme events (festival sale, natural disaster).
• They can signal fraud, system failures, or security breaches — the most critical cases.
• They distort statistical summaries like mean and standard deviation, leading to wrong analysis.

Example 1 — Simple Dataset

Consider: 2, 98, 101, 103, 106, 109, 112, 205
The values 2 and 205 are outliers — they lie far outside the main cluster of 98–112.

Example 2 — Daily Electricity Consumption

Day Units Consumed Status Reason
Monday 15 ✓ Normal Within range 14–17
Tuesday 16 ✓ Normal Within range 14–17
Wednesday 14 ✓ Normal Within range 14–17
Thursday 15 ✓ Normal Within range 14–17
Friday 17 ✓ Normal Within range 14–17
Saturday 16 ✓ Normal Within range 14–17
Sunday 45 ⚠ ANOMALY Extremely high — 2.6× the
average

📊 DIAGRAM: Figure 4.1 — Outlier Scatter Plot

Draw a 2D scatter plot (X-axis = data index, Y-axis = value). Plot 6 points clustered tightly near
y=15–17. Plot ONE point far above at y=45 (Sunday). Draw a dashed horizontal line at y=20 labeled
'Normal Threshold'. Mark the Sunday point with a circle and arrow labeled 'ANOMALY'.
4.1.2 Types of Anomalies / Outliers
There are three main types of anomalies. Understanding the type helps choose the right detection
method.

Type 1: Point Anomaly (Point Outlier)

A single individual data point is significantly different from the rest of the dataset.
DEFINITION
This is the most common type of anomaly.

• The anomaly occurs in isolation — one data point is far from the others.
• Example: Temperature readings: 25°C, 26°C, 27°C, 80°C, 26°C — the value 80°C is a point
anomaly.
• Example: Customer purchases: ₹500, ₹600, ₹550, ₹520, ₹15,000 — ₹15,000 is a point anomaly.
• Example: In the dataset {2, 98, 101, 103, 106, 109, 112, 205}, both 2 and 205 are point
anomalies.

📊 DIAGRAM: Figure — Point Anomaly (Line Graph)

Draw a line graph of 7 temperature readings over a week. Six points cluster near 25–27°C forming a
roughly flat line. One point (e.g. Thursday) spikes sharply to 80°C. Mark it with a large dot and label
'POINT ANOMALY'. Draw a dashed horizontal band around 24–28°C labeled 'Normal Range'.

Type 2: Contextual Anomaly (Conditional Anomaly)

A data point is anomalous only within a specific context or condition. The same
DEFINITION
value can be perfectly normal in a different context.

• Also called Conditional Anomaly — the context defines what is normal.

• Example: Temperature of 10°C is completely normal in December (winter) but would be an
alarming anomaly in June (summer).
• Example: A salary of ₹50,000/month is normal for a senior engineer but highly anomalous for an
intern.
• Example: A website receiving 10,000 visitors/hour is normal during a sale event but anomalous
on a Tuesday night.

📊 DIAGRAM: Figure — Contextual Anomaly (Two Bar Charts)

Draw TWO bar charts side by side. LEFT chart titled 'Winter (Dec)': bars show temperatures 5°C,
8°C, 10°C, 7°C, 9°C — highlight 10°C in GREEN labeled 'NORMAL'. RIGHT chart titled 'Summer
(June)': bars show temperatures 36°C, 38°C, 10°C, 37°C, 35°C — highlight 10°C in RED labeled
'CONTEXTUAL ANOMALY'. Add a note: 'Same value, different context'.

Type 3: Collective Anomaly

A group of data points that together form an anomaly, even though each individual
DEFINITION
point might appear normal on its own.
 • No single point looks unusual, but the pattern/group of points is abnormal.
• Example: Normal network traffic: 100, 110, 95, 105 packets/sec. Then suddenly a sequence: 500,
520, 530, 510 — this group indicates a DDoS cyber attack.
• Example: A series of ₹9,999 transactions (just below ₹10,000 reporting threshold) — each seems
normal, but the collective pattern signals fraudulent 'structuring'.
• Example: Abnormal heartbeat patterns in ECG — an individual beat may be borderline, but a
sustained pattern of unusual beats = cardiac anomaly.

📊 DIAGRAM: Figure — Collective Anomaly (Time-Series Line Graph)

Draw a time-series line graph (X=time, Y=network packets/sec). For time 1–10: line is flat/stable
around 100 units. From time 11–15: line suddenly jumps to 500–530 and stays elevated. Draw a
shaded rectangle around the 11–15 region and label it 'COLLECTIVE ANOMALY — DDoS Attack
Pattern'. The individual values in the attack window are not outrageous alone, but the pattern is.

Type What is Anomalous? Requires Example

Context?
Point Anomaly One individual data point No Temperature spike to
80°C in a series
Contextual Anomaly A point abnormal in one Yes 10°C normal in winter,
context anomalous in summer
Collective Anomaly A group/sequence of points Pattern matters Sudden sustained spike
together in network traffic

4.1.3 Why Detect Anomalies? (Real-World Importance)

Anomalies often indicate the most important events in a system — errors, fraud, failure, or rare
opportunities. Ignoring anomalies can lead to catastrophic outcomes.

Industry Type of Anomaly Consequence if Undetected

Banking & Finance Fraudulent credit card transaction Financial loss, customer harm
(₹2,00,000 abroad)
Healthcare Abnormal heart rate (200 bpm), Missed diagnosis, patient death
unusual lab values
Cybersecurity Unauthorized login, DDoS traffic Data breach, system downtime
spike
Manufacturing Faulty sensor reading, defective Product defects, line shutdown
machine behavior
E-Commerce Fake reviews, bot purchases, price Trust loss, revenue fraud
manipulation
Network Unusual bandwidth consumption at Intrusion, data exfiltration
3am

4.1.4 How Anomaly Detection Works — General Steps

The fundamental idea of anomaly detection is: understand what is NORMAL, then flag what deviates
significantly from that normal.

1. Collect Data — Gather sufficient historical data representing normal system behavior.
2. Understand Normal Pattern — Analyze the typical data: its distribution, mean, clusters, and
expected ranges.
3. Choose Detection Method — Select a statistical, distance-based, density-based, or ML-based
technique.
4. Measure Deviation — For each new data point, calculate how far it is from the learned normal
pattern.
5. Set Threshold — Define a boundary: beyond this threshold, a point is anomalous.
6. Flag & Report — Mark anomalous points and trigger alerts for investigation.
7. Act — Investigate the flagged point: is it an error, genuine anomaly, or fraud?

4.1.5 Causes of Outliers

Understanding WHY an outlier exists is crucial — it determines how to handle it (delete it, correct it, or
investigate it).

Cause 1: Data Entry and Measurement Errors

• Errors occur during manual data entry (typos) or from faulty sensors and instruments.
• These outliers are FALSE anomalies — they do not represent real events and should be
corrected or removed.
• Example: Height of adult men dataset. Heights: 1.59, 1.65, 1.71, 1.73, 1.72, ..., 10.81 m — the
value 10.81 m is clearly a data entry error (should be 1.081 m or similar).
• Example: Temperature sensor records 500°C instead of 50°C — sensor malfunction.
• Example: Income recorded as ₹50,000,000 instead of ₹50,000 — a decimal point typo.

Cause 2: Sampling Problems

• Occurs when a study accidentally collects data from outside the target population.
• Inferential statistics relies on samples being representative of the target population. A non-
representative sample introduces artificial outliers.
• Example: A study measuring average income accidentally includes one billionaire — that person's
income would be a massive outlier, making the mean meaningless.
• Example: Measuring average height of 18–25 year olds, but a 7-foot-tall basketball player slips
into the sample.
• The experimenter may also measure subjects under abnormal conditions — e.g., measuring
blood pressure during a panic attack.

Cause 3: Natural Variation (Genuine Outliers)

• Not all outliers are errors! Some are legitimate, real extreme events.
• All data distributions have a natural spread, and extreme values occur with low probability.
• In a Normal distribution: ~1 in 340 observations will be ≥ 3 standard deviations from the mean —
these are expected, not errors.
 • Example: A customer genuinely purchasing ₹50,000 worth of electronics during a Diwali sale —
real behavior, not fraud.
• Example: A company recording record profits in one quarter — real event, not a data error.
• These outliers must NOT be deleted — they represent valuable real-world variation.

Cause 4: Fraud or Abnormal Behavior

• Some outliers are intentionally generated by malicious activity — these are the MOST
IMPORTANT outliers to detect.
• Example: A credit card transaction of ₹2,00,000 in a foreign country when the cardholder is in
India — strong fraud signal.
• Example: Multiple failed login attempts from different geographic locations within seconds —
brute force cyber attack.
• Example: A medical scan showing a reading 10× the normal range — critical health anomaly.
• These outliers must NEVER be deleted — they need urgent investigation.

Cause Nature Example Action

Required
Data Entry / Measurement False anomaly — Height recorded as 10.8m Correct or
Error bad data delete
Sampling Problem False anomaly — Billionaire in average income Remove from
wrong population study sample
Natural Variation True anomaly — ₹50,000 Diwali purchase Keep — do
real event not delete
Fraud / Abnormal Behavior True anomaly — ₹2L credit card abroad Investigate
critical event urgently

4.1.6 Anomaly Detection Techniques — Overview

Several categories of techniques are used to detect outliers. Each has its own strengths, assumptions,
and best use cases.

# Category Key Idea When to Use Examples

1 Statistical Use mean, std dev, Data follows known Z-Score, IQR, Box
Methods quartiles to define normal distribution (normal/bell Plot, Mahalanobis
range curve)
2 Distance-Based Outlier = point far from its Normal data forms tight Euclidean
Methods nearest neighbors clusters Distance, KNN,
PCA
3 Density-Based Outlier = point in a low- Clusters of varying shapes LOF, DBSCAN
Methods density region compared and sizes
to neighbors
4 Machine Learning Learn what 'normal' looks Large, complex, high- Isolation Forest,
Methods like from training data dimensional datasets Autoencoders
5 Deep Learning Neural networks learn Unstructured data: LSTM,
Methods complex normal patterns images, video, audio Autoencoders,
GAN-based
 # Category Key Idea When to Use Examples
6 Clustering Outlier = point far from any Well-defined cluster K-Means based,
Methods cluster centroid structure DBSCAN
7 Ensemble Combine multiple methods Complex, diverse data Isolation Forest +
Methods for better robustness LOF combo

4.1.7 Challenges of Anomaly Detection

Despite its importance, anomaly detection is difficult due to several challenges:

8. Lack of Labelled Data: Most anomaly detection is unsupervised because labelled examples of
anomalies are rare, expensive, and hard to obtain. We cannot always know what an anomaly
looks like in advance.
9. Imbalanced Data: Anomalies occur very rarely — perhaps 0.1% of all data. This extreme class
imbalance makes it hard to train accurate detection models.
10. High False Positive Rates: Anomaly detection algorithms may flag too many normal points as
anomalies (false alarms), wasting investigation resources and causing alarm fatigue.
11. Concept Drift: The definition of 'normal' changes over time. A model trained on last year's data
may fail to detect new types of anomalies.
12. Interpretability: Some ML-based detectors (deep learning) are 'black boxes' — we cannot easily
explain WHY a point was flagged, making it hard to trust or act on the result.
13. Scalability: As dataset size grows to millions of records, running outlier detection algorithms in
real-time becomes computationally challenging.
14. Overfitting: A model trained on limited anomaly examples may memorize them and fail to detect
new, slightly different anomalies.

4.1.8 Outlier Detection Using Statistics

Statistical methods identify outliers by measuring how much a data point deviates from the expected
distribution. They are the simplest and most interpretable approach, best suited for univariate data with a
known distribution.

Method I — Z-Score (Standard Score)

The Z-score measures how many standard deviations a data point is away from
CONCEPT
the mean. A large absolute Z-score means the point is far from average.

Z = (X − μ) / σ

Where:
• X = the individual data point being tested
• μ (mu) = mean of the dataset
• σ (sigma) = standard deviation of the dataset
• Z = number of standard deviations from the mean
 If |Z| > 3 → the data point is an OUTLIER. This threshold corresponds to the
RULE
99.7% confidence interval (3-sigma rule) of the normal distribution.

Worked Example — Z-Score:

Dataset: 10, 12, 11, 13, 12, 40
15. Calculate Mean: μ = (10+12+11+13+12+40) / 6 = 98 / 6 ≈ 16.33
16. Calculate Variance: Σ(Xi−μ)² / n =
[(10−16.33)²+(12−16.33)²+(11−16.33)²+(13−16.33)²+(12−16.33)²+(40−16.33)²] / 6 ≈ 118.9
17. Standard Deviation: σ = √118.9 ≈ 10.9
18. Z-score for 40: Z = (40 − 16.33) / 10.9 ≈ 2.17 → Far from mean, flagged
19. Z-score for 10: Z = (10 − 16.33) / 10.9 ≈ −0.58 → Normal
20. Z-score for 12: Z = (12 − 16.33) / 10.9 ≈ −0.40 → Normal

Value Z-Score |Z| > 3? Classification

10 -0.58 No Normal
12 -0.40 No Normal
11 -0.49 No Normal
13 -0.30 No Normal
40 2.17 No (borderline) Suspected Outlier
Note: In a smaller 3σ may be too strict Use 2σ for small samples
dataset

Advantages of Z-Score:
• Simple, fast, and easy to understand and compute
• Works very well for normally distributed data
• Can be automated for real-time data streams

Limitations of Z-Score:
• Assumes data follows a normal (Gaussian) distribution — fails for skewed data
• The outlier itself inflates the mean and std. deviation, making it harder to detect (masking effect)
• Not robust for small datasets — every point heavily influences mean and std. dev.
• Only works for univariate (single-feature) data

📊 DIAGRAM: Figure — Z-Score Normal Distribution (Bell Curve)

Draw a bell curve (normal distribution). Mark the center as μ (mean). Draw vertical dashed lines at
μ±1σ, μ±2σ, and μ±3σ on both sides. Shade the area between μ±3σ as 'Normal Zone (99.7%)'. Mark
points beyond ±3σ on both tails in RED and label them 'OUTLIERS'. Add the text: '68% within 1σ |
95% within 2σ | 99.7% within 3σ'.

Method II — IQR (Interquartile Range) Method

The IQR method uses quartiles to define the spread of the middle 50% of data. It
CONCEPT
is robust — it is not affected by the outliers themselves, unlike the Z-score.
 IQR = Q3 − Q1
Lower Limit = Q1 − 1.5 × IQR
Upper Limit = Q3 + 1.5 × IQR

Where:
• Q1 (First Quartile) = 25th percentile — the value below which 25% of data falls
• Q3 (Third Quartile) = 75th percentile — the value below which 75% of data falls
• IQR = the range of the middle 50% of data (robust measure of spread)
• Any data point below Lower Limit OR above Upper Limit is an outlier

Step-by-Step IQR Process:

21. Sort the dataset in ascending order.
22. Find Q1: The median of the lower half of the data.
23. Find Q3: The median of the upper half of the data.
24. Calculate IQR = Q3 − Q1.
25. Calculate: Lower Limit = Q1 − 1.5 × IQR
26. Calculate: Upper Limit = Q3 + 1.5 × IQR
27. Any value < Lower Limit OR > Upper Limit is flagged as an outlier.

Worked Example — IQR:

Dataset: 10, 12, 14, 15, 18, 40
28. Sorted: 10, 12, 14, 15, 18, 40
29. Q1 = median of lower half {10, 12, 14} = 12
30. Q3 = median of upper half {15, 18, 40} = 18
31. IQR = 18 − 12 = 6
32. Lower Limit = 12 − (1.5 × 6) = 12 − 9 = 3
33. Upper Limit = 18 + (1.5 × 6) = 18 + 9 = 27
34. Check each value: 10 (>3 ✓ normal), 12 (✓), 14 (✓), 15 (✓), 18 (✓), 40 (>27 ⚠ OUTLIER)

Value 40 is an outlier because 40 > Upper Limit (27). All other values fall within
RESULT
the normal range [3, 27].

Advantages of IQR:
• Robust — the outlier itself does not affect Q1, Q3, or IQR (unlike Z-score)
• Makes no assumption about data distribution — works for skewed data too
• Simple to calculate and interpret

Limitations of IQR:
• Only considers the middle 50% of the data — may miss nuanced outliers
• Primarily univariate — not designed for multivariate analysis
 • The 1.5 multiplier is a convention, not absolute — may need adjustment for different domains

Method III — Box Plot (Visual IQR Method)

A Box Plot is a graphical representation of the IQR method. It visually shows the
CONCEPT
distribution, median, and outliers in one diagram.

• The box spans from Q1 to Q3 (the IQR), with a line inside marking the Median.
• Whiskers extend from Q1 downward and Q3 upward, to a maximum of 1.5×IQR from the box
edges.
• Any data points that fall beyond the whiskers are plotted as individual dots and are classified as
outliers.
• The whisker ends touch the last data point WITHIN 1.5×IQR — not necessarily exactly at
Q1±1.5×IQR.

📊 DIAGRAM: Figure — Box Plot (Most Important Diagram for IQR)

Draw a horizontal Box Plot with these clearly labeled components: • Minimum non-outlier value (left
end of left whisker) • Q1 (left edge of box) • Median (vertical line inside box) • Q3 (right edge of box) •
Maximum non-outlier value (right end of right whisker) • Lower Limit = Q1 − 1.5×IQR (left whisker
endpoint) • Upper Limit = Q3 + 1.5×IQR (right whisker endpoint) • 1–2 dots to the right of the right
whisker labeled 'OUTLIERS' Add annotations: 'IQR' with a double-headed arrow spanning the box,
'1.5×IQR' with arrows on each whisker.

Method IV — Mahalanobis Distance

Mahalanobis Distance measures how many standard deviations a data point is
CONCEPT from the mean of a multivariate dataset, accounting for correlations between
features.

Unlike Euclidean Distance which treats all dimensions equally, Mahalanobis Distance corrects for scale
differences and feature correlations, making it ideal for multivariate datasets.

Steps for Mahalanobis Distance Outlier Detection:

35. Compute the mean vector (μ) and covariance matrix (Σ) of the dataset.
36. For each data point x, compute the Mahalanobis distance: D(x) = √[(x−μ)ᵀ Σ⁻¹ (x−μ)]
37. Determine a threshold value based on chi-squared distribution.
38. Identify all points with D(x) greater than the threshold as outliers.

• Best used when features are correlated (e.g., height and weight, or temperature and humidity).
• Used in PCA-based outlier detection to find points far from the centroid in principal component
space.
SECTION 4.2 — DISTANCE-BASED, DENSITY-BASED
METHODS & SMOTE

4.2.1 Outlier Detection Using Distance-Based Methods

Distance-based methods detect outliers by measuring how far a data point is from its neighbors. The
core assumption is: normal data points form clusters and lie close to each other; outliers are isolated,
lying far from any cluster.

A data point is an outlier if it is significantly farther from its neighbors than the
CORE IDEA majority of other points. The larger the distance to the K nearest neighbors, the
more anomalous the point is.

General Steps for Distance-Based Outlier Detection:

39. For each data point p in the dataset:
a) Calculate the distance between p and all other data points.
b) Find the K nearest neighbors of p (the K closest points).
c) Calculate the average distance from p to its K nearest neighbors.
d) If this average distance is much larger than that of most other points, flag p as an outlier.
40. Set a threshold (manually or statistically). Points exceeding the threshold are outliers.

Distance Metric I: Euclidean Distance

Euclidean Distance is the straight-line geometric distance between two points in
DEFINITION n-dimensional space. It is the most commonly used distance metric in outlier
detection.

d(x, y) = √[ Σ (xi − yi)² ] for i = 1, 2, ..., n

In 2D (two features):

d(x, y) = √[ (x₁ − y₁)² + (x₂ − y₂)² ]

Where xi and yi are the values of the i-th feature in points x and y, and n is the number of features
(dimensions).

Mean of all Data Points:

Mean = (1/n) × Σ xi for i = 1, 2, ..., n

In outlier detection using Euclidean distance:

41. Calculate the mean of all data points.
 42. Compute the Euclidean distance from each point to the overall mean.
43. Set a threshold (e.g., using Z-score or IQR of these distances).
44. Points with distance greater than the threshold are flagged as outliers.

Worked Example — Customer Purchase Amounts:

Customer Purchase Amount Distance from Mean ≈ Interpretation
(₹) ₹843
A 500 ~343 Normal
B 520 ~323 Normal
C 510 ~333 Normal
D 530 ~313 Normal
E 505 ~338 Normal
F 5000 ~4157 OUTLIER — distance is
12× higher

All customers A–E spend ₹500–530 (tightly clustered). Customer F spends ₹5,000 — the Euclidean
distance to the mean is enormous, flagging F as an outlier.

📊 DIAGRAM: Figure — Euclidean Distance Outlier (1D Visualization)

Draw a number line (horizontal axis = purchase amount). Plot 5 points (A–E) clustered tightly
between 500–530. Plot one point (F) far to the right at 5000. Draw a double-headed arrow from the
cluster mean (~513) to point F, labeled 'd = 4157'. Also draw small arrows from mean to each of A–E
showing small distances (~300). Clearly annotate F as 'OUTLIER'.

Important Notes on Euclidean Distance:

• Euclidean distance assumes all features are equally important and have the same scale.
• If features have different scales (e.g., age in years vs. salary in thousands),
normalize/standardize the data first.
• In some cases, Mahalanobis distance is more appropriate as it accounts for feature correlations.
• Applications: Fraud Detection, Image Processing, Clustering, Market Basket Analysis.

4.2.2 K-Nearest Neighbor (KNN) Based Outlier Detection

KNN-based outlier detection is one of the most popular distance-based approaches. It defines an outlier
as a point whose K nearest neighbors are far away relative to most other points.

Algorithm — KNN Outlier Detection:

45. Choose a value of K (number of neighbors to consider). Typical values: K = 5, 10, or 20.
46. For each data point p, compute its distance to every other point in the dataset.
47. Sort these distances and select the K smallest (the K nearest neighbors).
48. Compute the outlier score: the average distance from p to its K nearest neighbors.
49. Compare this score across all points. Set a threshold.
 50. Points with an outlier score above the threshold are classified as outliers.

📊 DIAGRAM: Figure — KNN-Based Outlier Detection (2D Scatter Plot)

Draw a 2D scatter plot with: • CLUSTER 1: ~10 tightly packed points in the lower-left region. Pick
one point inside — draw K=3 short lines to its 3 nearest neighbors. Label these lines 'small
distances'. Label this point 'NORMAL (low avg dist)'. • CLUSTER 2: ~8 tightly packed points in the
upper-right region. • OUTLIER POINT: One isolated point in the middle-right area, far from both
clusters. Draw K=3 lines to its 3 nearest neighbors (which are far away in the clusters). Label these
lines 'large distances'. Label this point 'OUTLIER (high avg dist)'. Add a legend showing: filled circle
= normal, triangle = outlier.

Choosing K:
K Value Behavior Risk
Small K (e.g., 2–3) Sensitive to local variations; detects micro-clusters Too many false
positives
Medium K (e.g., 5–10) Balanced; recommended starting point Good balance
Large K (e.g., 20+) Stable but may miss subtle local anomalies May miss local outliers

Advantages of Distance-Based Methods:

• Simple and intuitive — easy to understand and implement
• No assumption about data distribution
• Works well when normal data forms tight clusters
• Useful in high-dimensional data

Limitations of Distance-Based Methods:

• Computationally expensive: requires O(n²) distance calculations for large datasets
• Choosing optimal K is difficult without domain knowledge
• Performance degrades in very high-dimensional spaces (curse of dimensionality)
• Euclidean distance does not handle correlated features well

4.2.3 PCA-Based Outlier Detection (Distance-Based)

PCA = Principal Component Analysis. A distance-based method that projects

FULL FORM high-dimensional data onto lower-dimensional principal components, then
identifies outliers as points far from the bulk of data in that space.

PCA-based outlier detection is especially powerful for high-dimensional datasets where direct distance
calculations are computationally expensive or misleading.

Steps to Perform PCA-Based Outlier Detection:

51. Standardize the data: Scale all features to have mean = 0 and standard deviation = 1. This
ensures all features have equal importance in PCA.
 52. Compute Principal Components: Find the orthogonal directions (eigenvectors of the covariance
matrix) that explain the most variance in the data. The first PC explains the most variance, the
second explains the next most, etc.
53. Project data: Transform each data point by projecting it onto the principal components to get
scores.
54. Compute Mahalanobis Distance: Measure the Mahalanobis distance of each data point in the
principal component space.
55. Set threshold: Points with high Mahalanobis distance are far from the data centroid in PC space
→ outliers.

📊 DIAGRAM: Figure — PCA-Based Outlier Detection (2D to 1D)

Draw a 2D scatter plot with an elongated elliptical cluster of points (showing correlation between X
and Y). Draw two arrows from the center: PC1 (long arrow along the main axis of the ellipse) and
PC2 (shorter arrow perpendicular to PC1). Show one point far from the cluster along the PC2
direction — label it 'OUTLIER in PC space'. Show another point within the ellipse labeled 'Normal'.
This illustrates that PCA finds the true structure of the data.

Advantages of PCA-Based Detection:

• Can detect outliers not visible with traditional methods (e.g., in correlated multi-feature space)
• Reduces computational cost by working in reduced-dimensional space
• Well-established mathematical foundation, supported by all major ML libraries

Disadvantages of PCA-Based Detection:

• Sensitive to scaling — must standardize data first
• Only captures linear relationships — cannot detect non-linear outliers
• Interpretation becomes harder in very high-dimensional PC space

Applications: Fraud Detection, Quality Control, Medical Diagnosis.

4.2.4 Outlier Detection Using Density-Based Methods

Density-based methods detect outliers by examining the local neighborhood density of data points.
Unlike distance-based methods (which only look at absolute distances), density-based methods compare
a point's density to its neighbors' densities — making them powerful for datasets with clusters of varying
sizes and shapes.

An outlier is a point that lies in a region of significantly lower density compared to

CORE IDEA its neighbors. Normal points are surrounded by many other points (high density).
Outliers are in sparse regions (low density).

Method I: LOF — Local Outlier Factor

LOF = Local Outlier Factor. An unsupervised density-based algorithm for
FULL FORM detecting outliers. It uses the concept of local density to compare each point to its
neighbors.

LOF Algorithm — How It Works:

 56. For each data point p, compute its distance to all other points.
57. Find k-nearest neighbors (k is a user-defined parameter).
58. Compute the k-distance of p: the distance from p to its k-th nearest neighbor.
59. Compute the reachability distance: reach_dist(p, o) = max(k-dist(o), d(p,o)) — this smooths out
statistical fluctuations.
60. Compute Local Reachability Density (LRD) of p: the inverse of the average reachability distance
of p from its k-neighbors.
61. Compute LOF score: ratio of average LRD of p's neighbors to p's own LRD.

LOF(p) = [Average LRD of k-neighbors of p] / LRD(p)

Interpretation:
• LOF(p) ≈ 1 → p has similar density to its neighbors → NORMAL point
• LOF(p) > 1 → p is in a lower-density region than its neighbors → Possible outlier
• LOF(p) >> 1 (e.g., >2 or >3) → p is significantly less dense than its neighbors → OUTLIER

Intuitive Example:
Imagine a busy market (dense cluster of shops). Most shops are surrounded by other shops — high
density, LOF≈1. Now imagine one isolated shop in an empty field 5 km away — very low density
compared to the market cluster. LOF >> 1 → outlier.

📊 DIAGRAM: Figure 4.2 — LOF Scatter Plot

Draw a 2D scatter plot with: • A dense cluster of ~15 small dots in the lower-left area. Draw small
circles around each dot (representing their LOF score circles — small, similar in size). Label one
'LOF ≈ 1 (Normal)'. • A moderate-density group of ~8 dots in the upper-right area with slightly larger
circles. Label 'LOF ≈ 1.2'. • 3 isolated dots scattered far from both clusters with very large circles.
Label these 'LOF >> 1 (OUTLIER)'. This is Figure 4.2 from your reference textbook.

Advantages of LOF:
• Unsupervised — no labelled training data required
• Handles datasets with clusters of different densities (unlike global methods)
• Can detect both local outliers (outlier within a nearby context) and global outliers

Disadvantages of LOF:
• The LOF threshold for 'outlier' varies across different datasets — no universal rule
• Computationally expensive for very large datasets
• Accuracy decreases in high-dimensional spaces

Applications of LOF:
• Fraud Detection
• Intrusion Detection (Cybersecurity)
• Network Security and traffic analysis

4.2.5 DBSCAN — Density-Based Spatial Clustering for Applications with Noise

 DBSCAN = Density-Based Spatial Clustering for Applications with Noise. A
FULL FORM density-based clustering algorithm that naturally identifies outliers (noise points)
without needing to specify the number of clusters.

DBSCAN is used when a dataset is large and multivariate — where IQR, Z-Score, or LOF may be
inefficient. It groups together points in dense regions and marks isolated points as noise (= outliers).

Key Parameters of DBSCAN:

Parameter Symbo Meaning Effect if Too Effect if Too
l Small Large
Epsilon (radius) ε The radius of the Small ε → too Large ε → all
neighbourhood around many points points merge
each data point become into one cluster
noise/outliers
Minimum Points minPts Minimum number of points Small minPts → Large minPts →
required within ε-radius to too many Core too few Core
form a dense region Points, clusters Points, many
(cluster) everywhere false outliers

DBSCAN Point Classifications:

Point Type Definition Color (in figures) Outlier?
Core Point Has ≥ minPoints within ε-radius Red No — part
(including itself). Forms the interior of of cluster
clusters.
Border Point Has < minPoints within ε-radius BUT Yellow No —
is within the ε-neighbourhood of a edge of
Core Point. cluster
Noise Point (Outlier) Is neither a Core Point nor a Border Purple YES —
Point. Completely isolated — no OUTLIER
dense region nearby.

DBSCAN Algorithm Steps:

62. Mark all points as UNVISITED.
63. Pick any unvisited point p. Mark p as VISITED.
64. Find all points within radius ε of p (the ε-neighbourhood of p).
If number of points in ε-neighbourhood ≥ minPoints:
a) p is a CORE POINT → Create a new cluster C.
b) Add all points in the ε-neighbourhood to C.
c) For each newly added point q in C, if q is also a Core Point, recursively add q's
neighbourhood to C too (cluster expansion).
If number of points in ε-neighbourhood < minPoints:
d) Mark p as NOISE (tentatively). It may be reclassified as a Border Point if it falls in another
Core Point's neighbourhood.
65. Repeat until all points are visited.
 66. All points remaining as NOISE are the OUTLIERS.

Worked Conceptual Example (minPoints = 3):

Consider 10 data points scattered on a 2D plane. DBSCAN draws a circle of radius ε around each point:
• Points A, B, C, D, E — densely clustered. Each has ≥ 3 points in its ε-circle (including itself). →
All are CORE POINTS (Red). They form Cluster 1.
• Point F — near Cluster 1 but only has 2 points in its ε-circle. BUT it falls inside the ε-circle of point
A (a Core Point). → BORDER POINT (Yellow).
• Point G — sits completely alone. Only 1 point (itself) in its ε-circle, and it's not near any Core
Point. → NOISE = OUTLIER (Purple).

📊 DIAGRAM: Figure 4.3 — DBSCAN: Raw Data Points

Draw a 2D scatter plot with ~10 data points scattered across the plane. Some are clustered together
(group of 6–7), two are near the cluster but not in it, and one is completely isolated. No circles drawn
yet. Label this as 'Step 1: Raw dataset before DBSCAN'. Title: 'Figure 4.3'.

📊 DIAGRAM: Figure 4.4 — DBSCAN: After Clustering (minPoints=3)

Same scatter plot as above but now: • Draw circles of equal radius ε around each point. • Core points
(have ≥3 points inside their circle): Color RED. Their circles overlap — this forms the cluster. • Border
points (inside a Core point's circle but own circle has <3 points): Color YELLOW. • Noise/Outlier point
(isolated, circle has only itself): Color PURPLE. • Draw the cluster boundary (outline of the merged
circles). • Label: 'Core Points (Red) — form cluster', 'Border Points (Yellow) — edge of cluster',
'Noise/Outlier (Purple) — isolated'. Title: Figure 4.4 — ε-neighbourhood with minPoints=3.

Why DBSCAN is Special — Arbitrary Shape Clusters:

Unlike K-Means (which only finds spherical clusters), DBSCAN can find clusters of ANY shape — L-
shaped, ring-shaped, crescent-shaped. Any dense region, regardless of shape, becomes a cluster.
Points outside all dense regions = outliers.

📊 DIAGRAM: Figure — DBSCAN vs K-Means (Arbitrary Shapes)

Draw two plots side by side: LEFT (K-Means): Two circular blobs of points. K-Means correctly
identifies them. But also show two crescent/ring-shaped point groups — K-Means fails badly here
(draws wrong boundaries). RIGHT (DBSCAN): The same crescent/ring-shaped groups. DBSCAN
correctly identifies them as two clusters. Isolated points far from crescents are marked as outliers
(noise).

Advantages of DBSCAN:
• Naturally identifies noise/outliers as points not belonging to any cluster
• Can detect clusters of arbitrary shape (unlike K-Means which assumes spherical clusters)
• Automatically determines the number of clusters — no need to specify K
• Time complexity: O(n log n) with spatial indexing — computationally efficient for large datasets
• Robust to noise and very effective in spatial data analysis

Disadvantages of DBSCAN:
• Not suitable for data with highly varying densities — a single ε cannot capture multiple density
levels
 • Sensitive to the choice of ε and minPoints — wrong values give poor results
• Requires careful parameter tuning using domain knowledge or grid search

Applications of DBSCAN:
• Fraud Detection in banking
• Network Intrusion Detection
• Medical Diagnosis (e.g. detecting abnormal cell clusters in MRI)
• Quality Control in Manufacturing

4.2.6 Comparison: Distance-Based vs Density-Based Methods

Feature Distance-Based (KNN) Density-Based (LOF /

DBSCAN)
Core Idea Outlier = far from K nearest Outlier = in low-density region
neighbors
Handles varying densities Poor — single distance Good — compares to LOCAL
threshold fails neighborhood density
Cluster shapes Works best with spherical Works with arbitrary shapes
clusters (DBSCAN)
Interpretability High — easy to understand Medium — density concept is
distances more complex
Computational Cost O(n²) — expensive for large data O(n log n) with indexing
(DBSCAN)
Key Parameter K (number of neighbors) ε (radius) + minPoints
(DBSCAN) / k (LOF)
Best Use Case Compact, well-separated Mixed-density, complex-shaped
clusters clusters
4.2.7 SMOTE — Synthetic Minority Over-Sampling Technique

The Problem: Class Imbalance

In many real-world classification datasets, one class has far more samples than another. This is called
class imbalance. When a machine learning model trains on imbalanced data, it learns to always predict
the majority class, achieving high raw accuracy while completely failing to detect the rare but critical
minority class.

Real-World Problem Majority Class Minority Class Class Ratio

Credit Card Fraud Legitimate transactions Fraudulent transactions 99:1
(99%) (1%)
Disease Diagnosis Healthy patients (95%) Sick patients (5%) 95:5
Spam Detection Normal emails (90%) Spam emails (10%) 90:10
Network Intrusion Normal traffic (99.9%) Attack traffic (0.1%) 999:1
Customer Churn Retained customers Churned customers 85:15
(85%) (15%)

If the model always predicts 'No Fraud', it gets 99% accuracy — but it catches 0%
PROBLEM of fraud! Accuracy is misleading. We need Recall and F1-Score to properly
evaluate on imbalanced data.

Why Simple Oversampling Is Not Enough

A naive fix is to duplicate minority class samples (random oversampling). This leads to:
• Overfitting — the model memorizes those exact few minority samples instead of learning the
general pattern.
• No new information is added — the model sees the same data multiple times.
• The decision boundary does not improve because no new regions of feature space are explored.

Instead of duplicating existing samples, SMOTE creates new SYNTHETIC

SMOTE (artificial) data points by INTERPOLATING between existing minority class
SOLUTION samples. This forces the model to learn a broader, more generalised decision
boundary.

SMOTE — Definition and Core Concept

SMOTE = Synthetic Minority Over-Sampling Technique. Proposed by Chawla et

FULL FORM al. (2002). A data augmentation technique that generates synthetic samples for
the minority class using linear interpolation between neighboring samples.
SMOTE Formula
X_new = X_i + rand(0, 1) × (X_neighbor − X_i)

Symbol Meaning
X_new New synthetic data point to be created
X_i The original minority class data point (starting point)
X_neighbor One of X_i's k nearest minority class neighbors (endpoint)
rand(0,1) A random number between 0 and 1 (determines position along the line
segment)
(X_neighbor − X_i) The direction vector from X_i to its neighbor
X_i + rand × (...) A random point along the line connecting X_i and X_neighbor

SMOTE Algorithm — Step by Step

67. Set the desired over-sampling ratio (e.g., generate 200% more minority samples).
68. For each minority class sample X_i:
a) Find k nearest neighbors of X_i from within the minority class only (default k=5).
b) Randomly select one neighbor, call it X_neighbor.
c) Generate a random number rand between 0 and 1.
d) Create new point: X_new = X_i + rand × (X_neighbor − X_i)
e) Add X_new to the dataset as a new minority class sample.
69. Repeat step 2 until the required number of synthetic samples is generated.
70. Combine original dataset with synthetic samples → balanced training dataset.
71. Train the ML model on this balanced dataset.

Worked Numerical Example

Given:
• Minority Point X_i = (2, 3)
• Nearest Neighbor X_neighbor = (4, 5)
• Random value rand = 0.5

Step-by-step calculation:
72. Difference vector: X_neighbor − X_i = (4−2, 5−3) = (2, 2)
73. Multiply by rand: rand × (2, 2) = 0.5 × (2, 2) = (1, 1)
74. New point: X_new = X_i + (1,1) = (2+1, 3+1) = (3, 4)

X_new = (2, 3) + 0.5 × [(4, 5) − (2, 3)] = (2,3) + (1,1) = (3, 4)

 New synthetic minority class point (3, 4) is created midway between original point
RESULT (2,3) and its neighbor (4,5). This is a brand-new point, not a copy of any existing
sample.

📊 DIAGRAM: Figure — SMOTE: Geometric Interpretation

Draw a 2D coordinate plane. • Plot point X_i = (2,3) — label it 'X_i (original)' • Plot X_neighbor = (4,5)
— label it 'X_neighbor' • Draw a line segment connecting (2,3) to (4,5) • Mark the midpoint (3,4) on
the line — label it 'X_new (synthetic, rand=0.5)' • Also show where rand=0.25 would land: (2.5, 3.5)
— label 'X_new if rand=0.25' • And rand=0.75: (3.5, 4.5) — label 'X_new if rand=0.75' • This shows
that SMOTE can create any point ALONG the line between two minority samples.

📊 DIAGRAM: Figure 4.5 — SMOTE Three-Stage Process

Draw THREE side-by-side 2D scatter plots: [Plot 1 — ORIGINAL DATASET] • Many black squares
(majority class) filling most of the space • Very few filled circles (minority class, e.g. 4–5 circles) •
Label: 'Original Dataset — Imbalanced (majority >> minority)' [Plot 2 — GENERATING SAMPLES] •
Same dataset, but now draw lines between minority circles • Show 3–4 new gray circles appearing
ON those lines • Lines between minority circles labeled 'interpolation' • Label: 'Generating Synthetic
Samples via SMOTE' [Plot 3 — RESAMPLED DATASET] • Same majority squares • Now many
more filled circles (original + synthetic combined) • Minority circles roughly equal in number to
majority squares • Label: 'Resampled Dataset — Balanced' This matches Figure 4.5 in your
textbook.

SMOTE Variants
Several variations of SMOTE have been developed to address specific limitations:

Variant Full Name / Key Idea When to Use

SMOTE-1 Generates samples only between minority and When minority class is
majority class boundary in feature space near majority class
boundary
SMOTE-2 Generates samples between a minority sample When minority class is
and nearest neighbors of each minority sample spread out
(broader coverage)
Borderline-SMOTE Uses ONLY minority samples near the decision When minority class
boundary. Avoids generating noisy samples in has clear boundary
safe interior regions. regions
Safe-Level SMOTE Addresses overfitting by generating samples less When overfitting is a
similar to existing minority samples, but still in the concern
same feature region
A-SMOTE (Adaptive) Adaptively adjusts the amount of oversampling for When imbalance varies
each minority sample based on the local degree of across regions of
imbalance feature space
SMOTE + ENN SMOTE followed by Edited Nearest Neighbors For cleaner decision
(ENN) to remove noisy borderline samples boundaries
SMOTE + Tomek Links SMOTE followed by Tomek Links removal to clean For well-separated
the boundary classes
A-SMOTE (Adaptive SMOTE) Algorithm:
75. Identify all minority class examples in the dataset.
76. Calculate a dynamic threshold value based on the degree of imbalance.
77. For each minority example, select k nearest neighbors from the minority class.
78. For each selected neighbor, calculate the distance and generate synthetic examples by
interpolating between them.
79. Repeat steps 3–4 until the threshold value is reached.
80. Combine original minority examples with all synthetic examples → balanced dataset.

Advantages of SMOTE
• Reduces class imbalance effectively, improving overall model fairness
• Significantly improves Recall and F1-Score for the minority class
• Avoids simple duplication → reduces overfitting compared to naive oversampling
• Generates new, meaningful data points that explore the feature space between existing samples
• Simple and easy to implement using standard ML libraries (imbalanced-learn in Python)
• Can be combined with under-sampling (removing majority class samples) for better results

Disadvantages of SMOTE
• Synthetic samples may not represent the true distribution of the minority class — can introduce
noise
• Sensitive to K: wrong K value can degrade performance
• Computationally expensive for very large datasets or high-dimensional feature spaces
• If minority class samples are scattered in majority class regions, SMOTE can generate noisy
borderline samples
• Does not consider the majority class — new synthetic points might overlap with majority class,
creating ambiguous boundaries

Applications of SMOTE

Application Domain Problem Why SMOTE Helps

Credit Card Fraud Detection Only 0.1% of transactions are Balances dataset so model can
fraudulent learn fraud patterns
Medical Diagnosis Rare diseases have very few Generates synthetic patient
patient records profiles to improve recall
Spam Email Detection Spam is minority in email streams Balances dataset for better spam
classifier training
Customer Churn Prediction Churned customers are minority Improves prediction of at-risk
customers
Network Intrusion Detection Attack events are rare vs. normal Balances normal vs. attack
traffic classes for intrusion model
COMPREHENSIVE COMPARISON — ALL ANOMALY
DETECTION METHODS

Method Category Key Formula / Advantages Limitations Applicatio

Mechanism ns
Z-Score Statistical Z=(X−μ)/σ; |Z|>3 → Simple, fast, Assumes normal Finance,
outlier normal dist. distribution QC
IQR Statistical IQR=Q3−Q1; outside Robust, no dist. Only univariate Any
Q1±1.5×IQR assumption domain
Box Plot Statistical Visual IQR: dots Visual, intuitive Only univariate EDA,
beyond whiskers reporting
Mahalanobis Distance D=√[(x−μ)ᵀΣ⁻¹(x−μ)] Handles feature Assumes linearity Multi-
correlation feature
data
Euclidean/ Distance d=√Σ(xi−yi)²; far from K No dist. O(n²) expensive Fraud,
KNN neighbors assumption, image
clusters proc.
PCA Distance Project to PC space, High-dim, finds Only linear High-dim
Mahal. dist. hidden patterns datasets
LOF Density LOF=avg_neigh_densit Handles variable Threshold varies Fraud,
y/own_density density cybersec
DBSCAN Density ε-neighbourhood + Arbitrary shapes, ε/minPts tuning Spatial,
minPoints finds noise fraud
SMOTE Resampling X_new=Xi+rand×(Xn−X Balances data, May generate noise Imbalanced
i) improves recall classif.

FEATURE SELECTION FOR ANOMALY DETECTION

The quality of features used can significantly impact anomaly detection performance.

Impact of Feature Quality:

• If features do not capture important characteristics, the system cannot distinguish normal from
anomalous data.
• Too many features → overfitting, high computational cost.
• Too few features → insufficient information for detection.
• Correlated features → redundancy and misleading distances.

Best Practices for Feature Selection:

81. Domain Knowledge: Use expert knowledge to identify the most relevant features for the specific
problem.
82. Univariate/Multivariate Analysis: Use correlation analysis, PCA, and mutual information to find
important features.
 83. Regularization: Use L1/L2 regularization techniques to select important features while avoiding
overfitting.
84. Experimentation: Test different feature subsets, measure detection performance on each.
85. Feature Engineering: Transform features to capture more meaningful information (e.g., normalize,
scale, or create interaction terms).

QUICK REFERENCE — KEY FORMULAS

Formula Method What It Means

Z = (X − μ) / σ Z-Score Std. deviations from mean. |Z|>3 →
outlier
IQR = Q3 − Q1 IQR Spread of middle 50% of data
Lower = Q1 − 1.5×IQR IQR Lower Limit Below this → outlier
Upper = Q3 + 1.5×IQR IQR Upper Limit Above this → outlier
d(x,y) = √Σ(xi−yi)² Euclidean Distance Straight-line distance in n-
dimensional space
Mean = (1/n) × Σxi Mean Formula Average of all data points
LOF(p) = Avg(LRD_neighbors) / Local Outlier Factor LOF>>1 → low density relative to
LRD(p) neighbors → outlier
X_new = Xi + rand(0,1) × (Xn−Xi) SMOTE New synthetic minority class point
via interpolation

For a 10-mark answer: Write Definition (2 marks) → Types or Algorithm Steps (3

marks) → Formula + Example (2 marks) → Diagram (1 mark) → Advantages +
EXAM TIP ★
Disadvantages (1 mark) → Applications (1 mark). Always draw the diagram — it
earns easy marks!