UNSUPERVISED LEARNING:

COMPREHENSIVE PROFESSIONAL NOTES

TABLE OF CONTENTS
1. Introduction & Overview
2. Unsupervised Learning Theory & Foundations
3. K-Means Clustering
4. Anomaly Detection
5. Decision Framework
6. Summary & Key Takeaways

1. INTRODUCTION & OVERVIEW

Purpose of This Document
This comprehensive guide provides professional-grade notes on unsupervised learning,
including theoretical foundations, mathematical formulations, and practical
implementations. Designed for students, practitioners, and professionals seeking deep
understanding of clustering and anomaly detection techniques.

Document Scope
Unsupervised Learning Theory: Core concepts, mathematical foundations, and
theoretical frameworks
K-Means Clustering: Complete mathematical formulation, algorithm derivation,
and implementation
Anomaly Detection: Statistical approaches with rigorous mathematical treatment
Decision Framework: Comprehensive guidelines for choosing between supervised
and unsupervised approaches
Practical Examples: Real-world applications with step-by-step walkthroughs

Target Audience
Undergraduate/graduate students in machine learning
Data scientists and ML engineers
Professionals preparing for technical interviews
Researchers implementing unsupervised learning systems

2. UNSUPERVISED LEARNING THEORY & FOUNDATIONS

2.1 Definition: Unsupervised Learning
Definition: Unsupervised learning finds hidden patterns or structure in unlabeled data
without supervision. Unlike supervised learning where we have
target labels , unsupervised learning operates on data without explicit targets.
Mathematical Formulation:
Input: Unlabeled dataset where:
= number of samples
= number of features
No corresponding labels exist

Key Distinction from Supervised Learning:

Supervised: Learn mapping (predict )
Unsupervised: Discover structure in (learn )

2.2 Key Concepts in Unsupervised Learning

A. Density Estimation
Definition: Density estimation aims to approximate , the true probability distribution
underlying the observed data. By modeling the data distribution, we can:
Identify likely vs. unlikely data points
Detect anomalies as low-probability regions
Generate new samples from learned distribution
Mathematical Goal:

where is our estimated density function.

Example: Gaussian Density Estimation

Assume data follows normal distribution:

Estimate parameters from data
Use

Applications:
Anomaly detection (flag low-density points)
Data generation
Likelihood-based clustering
B. Clustering
Definition: Clustering partitions data into homogeneous groups (clusters) such that:
Intra-cluster similarity is maximized (points within cluster are similar)
Inter-cluster dissimilarity is maximized (points in different clusters are dissimilar)
Latent structure is revealed (discover natural groupings, manifolds, or latent
factors)
Mathematical Goal: Partition into clusters such that:

Key Properties:

Discovers latent structure without labels

Identifies similarities in feature space
Reveals low-dimensional manifolds
Enables data summarization and compression
Applications:
Customer segmentation for targeted marketing
Document clustering for information organization
Gene expression clustering in genomics
Recommendation systems

C. Dimensionality Reduction
Definition: Dimensionality reduction transforms -dimensional data into -dimensional
space where (typically ), while preserving important variance and structure.
Mathematical Goal: Find transformation such that:

Preserve maximum variance:

Key Techniques:
PCA (Principal Component Analysis): Linear dimensionality reduction via
orthogonal transformation
t-SNE (t-Distributed Stochastic Neighbor Embedding): Nonlinear reduction for
visualization
Autoencoders: Neural network-based nonlinear compression
Benefits:
Reduces computational complexity (fewer features)
Mitigates curse of dimensionality
Enables visualization of high-dimensional data
Removes noise and redundancy
Improves generalization in downstream tasks

2.3 Similarity & Distance Metrics

Definition: Similarity/distance metrics quantify how "close" two data points are in feature
space. Critical for clustering and anomaly detection.

Euclidean Distance (L2 Norm)

Definition: Standard geometric distance in -dimensional space.

Properties: Sensitive to feature scaling, assumes isotropic variance

Manhattan Distance (L1 Norm)

Definition: Sum of absolute differences (taxicab distance).

Properties: More robust to outliers than Euclidean

Cosine Similarity
Definition: Measures angle between vectors (inner product normalized).

Properties: Invariant to scaling, useful for text/sparse data

Mahalanobis Distance
Definition: Accounts for feature correlation and varying scales via covariance matrix.

where is the covariance matrix.

Properties: Handles correlated features, adjusts for variance differences
2.4 Key Assumptions in Unsupervised Learning
1. Cluster Separability: Clusters are sufficiently separated in feature space
2. Feature Independence (in Gaussian models): Features are independent given
cluster membership
3. Cluster Shape (K-Means): Clusters are approximately spherical with similar sizes
4. Stationarity: Data distribution doesn't change over time
5. Homogeneity: All clusters have similar densities and sizes

3. K-MEANS CLUSTERING: MATHEMATICAL

FORMULATION & IMPLEMENTATION
3.1 Purpose & Mathematical Formulation
Purpose: Partition data into clusters minimizing intra-cluster variance
while maximizing inter-cluster separation.
Objective Function (Cost Function):

where:
is the cluster assignment for sample
is the centroid of cluster
is the centroid of the cluster containing
Definition: This objective function is the Within-Cluster Sum of Squares (WCSS), also
called the inertia. Lower WCSS indicates tighter, more homogeneous clusters.
Mathematical Properties:
Non-convex optimization problem: Multiple local minima exist
NP-hard in general: No polynomial-time algorithm guaranteed to find global
optimum
Coordinate descent approach: Alternately optimize over assignments and
centroids

3.2 Algorithm: K-Means via Coordinate Descent

Theoretical Justification: K-Means is a coordinate descent algorithm that iteratively:
1. Fixes centroids and optimizes cluster assignments (assignment step)
2. Fixes assignments and optimizes centroids (update step)

Each step decreases (or maintains) the objective function .

Step 1: Cluster Assignment (E-Step / Assignment Step)
Objective: For fixed centroids , find optimal cluster assignments.
Mathematical Formulation:

Interpretation: Assign each point to the nearest centroid using Euclidean distance.
Voronoi Partition: This creates Voronoi cells—regions where each point is closer to a
particular centroid than any other.
Computational Complexity: for samples, clusters, features.

Step 2: Centroid Update (M-Step / Update Step)

Objective: For fixed cluster assignments , find optimal centroids.
Mathematical Derivation:
Given assignments, the cost function becomes:

To minimize with respect to , take derivative and set to zero:

Solution: Centroid is the mean of all points in the cluster:

𝟙
𝟙

where 𝟙 is an indicator function (1 if , else 0).

Interpretation: Each centroid is the arithmetic mean of all points assigned to that cluster.

Computational Complexity: per update.

Step 3: Convergence Check

Criterion: Stop when cluster assignments stabilize (no points change clusters):

where is a small threshold (e.g., ).

Convergence Guarantee: The objective function is non-increasing at each
iteration, and the algorithm converges to a local minimum (may not be global).
3.3 K-Means Algorithm Summary
Algorithm: K-Means Clustering
Input: Dataset , number of clusters , max iterations
Output: Cluster assignments and centroids
1. Initialize centroids (randomly from data points or K-Means++)
2. Repeat for to :
Assignment: For to :
𝟙
Update: For to : 𝟙
Check convergence: If assignments unchanged or loss converged, break
3. Return:

3.4 Initialization Strategy: K-Means++

Problem: Random initialization can lead to poor local minima.

Solution: K-Means++ initialization selects initial centroids with probability proportional to

squared distance from nearest existing centroid.
Algorithm: K-Means++ Initialization
1. Choose first centroid uniformly at random from
2. For to :
For each point , compute
Choose with probability
3. Run standard K-Means with these initial centroids
Advantage: Theoretically guaranteed approximation factor in expectation,
dramatically reduces poor local minima.

3.5 Complete Python Implementation

import numpy as np
from [Link] import cdist
from [Link] import silhouette_score, davies_bouldin_score,
calinski_harabasz_score
class KMeansClustering:
"""
Complete K-Means implementation from scratch with K-Means++ initialization.
"""

def __init__(self, n_clusters=3, max_iter=300, init_method='kmeans_plusplus', ra

"""
Initialize K-Means clusterer.
 Parameters:
-----------
n_clusters : int
Number of clusters (K)
max_iter : int
Maximum iterations
init_method : str
'random' or 'kmeans_plusplus'
random_state : int
For reproducibility
"""
self.n_clusters = n_clusters
self.max_iter = max_iter
self.init_method = init_method
self.random_state = random_state
self.centroids_ = None
self.labels_ = None
self.inertia_ = None
self.iteration_history_ = []

def initialize_centroids(self, X):

"""
Initialize centroids using specified method.

Parameters:
-----------
X : array-like, shape (n_samples, n_features)

Returns:
--------
centroids : array, shape (n_clusters, n_features)
"""
[Link](self.random_state)
n_samples = [Link][0]

if self.init_method == 'random':
indices = [Link](n_samples, self.n_clusters, replace=False)
return X[indices].copy()
 elif self.init_method == 'kmeans_plusplus':
# K-Means++ initialization
centroids = []

# Choose first centroid randomly

first_idx = [Link](n_samples)
[Link](X[first_idx].copy())

# Choose remaining centroids

for _ in range(1, self.n_clusters):
# Compute distance to nearest centroid
distances = [Link]([min([Link](x - c) for c in centroids)
for x in X])
# Probability proportional to distance squared
probabilities = distances ** 2
probabilities /= [Link]()

# Choose next centroid

next_idx = [Link](n_samples, p=probabilities)
[Link](X[next_idx].copy())

return [Link](centroids)

def assign_clusters(self, X):

"""
Assign each point to nearest centroid.

Parameters:
-----------
X : array, shape (n_samples, n_features)

Returns:
--------
labels : array, shape (n_samples,)
Cluster assignment for each sample
"""
distances = cdist(X, self.centroids_, metric='euclidean')
 return [Link](distances, axis=1)

def update_centroids(self, X, labels):

"""
Update centroids as mean of assigned points.

Parameters:
-----------
X : array, shape (n_samples, n_features)
labels : array, shape (n_samples,)
Current cluster assignments

Returns:
--------
centroids : array, shape (n_clusters, n_features)
"""
new_centroids = np.zeros_like(self.centroids_)

for k in range(self.n_clusters):
cluster_points = X[labels == k]
if len(cluster_points) > 0:
new_centroids[k] = cluster_points.mean(axis=0)
else:
# Keep old centroid if cluster empty
new_centroids[k] = self.centroids_[k]

return new_centroids

def compute_inertia(self, X, labels):

"""
Compute Within-Cluster Sum of Squares (WCSS).

Formula: J(c,μ) = Σᵢ ||x⁽ⁱ⁾ - μ_c(i)||²₂

Parameters:
-----------
X : array, shape (n_samples, n_features)
labels : array, shape (n_samples,)
 Returns:
--------
inertia : float
WCSS value
"""
inertia = 0.0
for k in range(self.n_clusters):
cluster_points = X[labels == k]
if len(cluster_points) > 0:
inertia += [Link]((cluster_points - self.centroids_[k]) ** 2)
return inertia

def fit(self, X):

"""
Fit K-Means model to data.

Parameters:
-----------
X : array-like, shape (n_samples, n_features)

Returns:
--------
self
"""
X = [Link](X)

# Initialize centroids
self.centroids_ = self.initialize_centroids(X)

# Iterative optimization
for iteration in range(self.max_iter):
# Assignment step
labels_old = self.assign_clusters(X).copy() if iteration > 0 else None
self.labels_ = self.assign_clusters(X)

# Compute inertia
inertia = self.compute_inertia(X, self.labels_)
 self.iteration_history_.append(inertia)

# Check convergence
if iteration > 0 and [Link](labels_old, self.labels_):
print(f"Converged at iteration {iteration}")
break

# Update step
self.centroids_ = self.update_centroids(X, self.labels_)

self.inertia_ = self.compute_inertia(X, self.labels_)

return self

def predict(self, X):

"""
Predict cluster assignments for new data.

Parameters:
-----------
X : array, shape (n_samples, n_features)

Returns:
--------
labels : array, shape (n_samples,)
"""
return self.assign_clusters(X)

def fit_predict(self, X):

"""Fit model and return cluster labels."""
return [Link](X).labels_

def find_optimal_k_elbow(X, k_range=range(2, 11)):

"""
Find optimal K using Elbow Method.

Formula: Look for "elbow" where inertia stops decreasing sharply

Parameters:
 -----------
X : array, shape (n_samples, n_features)
k_range : range
Range of K values to test

Returns:
--------
optimal_k : int
Recommended number of clusters
inertias : array
Inertia for each K
"""
inertias = []

for k in k_range:
kmeans = KMeansClustering(n_clusters=k, random_state=42)
[Link](X)
[Link](kmeans.inertia_)

# Find elbow (simplified: k with max second derivative)

second_diff = [Link](inertias, n=2)
optimal_k = list(k_range)[[Link](second_diff) + 1]

return optimal_k, [Link](inertias)

def find_optimal_k_silhouette(X, k_range=range(2, 11)):

"""
Find optimal K using Silhouette Score.

Formula: S = (b - a) / max(a, b) where a = intra-cluster dist, b = inter-cluster dist

Parameters:
-----------
X : array, shape (n_samples, n_features)
k_range : range
Range of K values to test

Returns:
 --------
optimal_k : int
K with highest silhouette score
scores : array
Silhouette score for each K
"""
scores = []

for k in k_range:
kmeans = KMeansClustering(n_clusters=k, random_state=42)
labels = kmeans.fit_predict(X)
score = silhouette_score(X, labels)
[Link](score)

optimal_k = list(k_range)[[Link](scores)]
return optimal_k, [Link](scores)

def kmeans_multiple_restarts(X, n_clusters, n_restarts=10):

"""
Run K-Means multiple times with different initializations.

Rationale: Avoid poor local minima by trying multiple random starts

Parameters:
-----------
X : array, shape (n_samples, n_features)
n_clusters : int
n_restarts : int
Number of random restarts

Returns:
--------
best_kmeans : KMeansClustering
Model with lowest inertia
"""
best_inertia = float('inf')
best_kmeans = None
 for restart in range(n_restarts):
kmeans = KMeansClustering(
n_clusters=n_clusters,
init_method='kmeans_plusplus',
random_state=restart
)
[Link](X)

if kmeans.inertia_ < best_inertia:

best_inertia = kmeans.inertia_
best_kmeans = kmeans

return best_kmeans

3.6 Evaluation Metrics for Clustering

1. Silhouette Score
Theory: Measures how well-separated clusters are by comparing intra-cluster cohesion to
inter-cluster separation for each point.

Formula: For point in cluster :

= mean distance from to other points in (cohesion)
= mean distance from to points in nearest different cluster (separation)

Overall Score:

Interpretation:

Range:
: Excellent separation
: Overlapping clusters
: Point closer to wrong cluster

2. Davies-Bouldin Index (DBI)

Theory: Ratio of average intra-cluster distance to inter-cluster distance. Lower values
indicate better clustering.

Formula:
where:
= average distance from points in cluster to its centroid
= distance between centroids of clusters and

Interpretation:

Lower is better
No lower bound (0 is excellent)
Computationally efficient

3. Calinski-Harabasz Index (CH Index)

Theory: Ratio of between-cluster to within-cluster variance. Higher values indicate better-
defined clusters.

Formula:

where:

= between-cluster scatter matrix

= within-cluster scatter matrix
= number of samples, = number of clusters
Interpretation:
Higher is better
More interpretable magnitude than DBI
Biased toward convex clusters

4. Within-Cluster Sum of Squares (WCSS / Inertia)

Theory: Measures total squared distance from each point to its cluster centroid. Lower
values indicate tighter clusters.
Formula:

Interpretation:
Lower is better
Used in Elbow method
Monotonically decreases with more clusters
3.7 Limitations & Solutions

Limitation Problem Solution

Use Elbow method or
Fixed K Don't know how many
Silhouette analysis to
required clusters a priori
find optimal K
Spherical Use DBSCAN, Gaussian
Doesn't work well for
clusters Mixture Models, or
crescent/non-convex shapes
only Spectral Clustering
Sensitive
Use K-Means++
to Poor initial centroids lead to
initialization or multiple
initializati bad local minima
random restarts
on
Remove outliers via
Sensitive Extreme points pull
isolation forest or
to outliers centroids significantly
robust clustering
Assumes Use Gaussian Mixture
Spheres of different sizes
equal Models (assigns soft
poorly clustered
variance probabilities)
Local Run multiple times with
May converge to suboptimal
optima different initializations,
solution
problem select best
Computati per iteration for
Use mini-batch K-Means
onal samples, clusters,
for large datasets
complexity features

4. ANOMALY DETECTION: THEORETICAL FOUNDATIONS

& IMPLEMENTATION
4.1 Definition & Theoretical Framework
Definition: Anomaly detection identifies data points that deviate significantly from the
"normal" pattern or behavior. Anomalous points are those with low probability under the
learned data distribution .

Mathematical Formulation:
Normal data: Concentrated in high-probability regions of
Anomalies: Located in low-probability regions where is small
Key Distinction from Supervised Classification:
Supervised: We have many examples of both normal and anomalous classes
Anomaly Detection: Very few (often zero) positive examples; primarily normal data

Formal Definition: For a learned probability model , flag point as anomaly if:

where is a threshold parameter tuned on validation data.

4.2 Core Mathematical Concepts

A. Probability Density Estimation
Objective: Learn probability distribution from training data.
Approach 1: Univariate Gaussian Model
Assume each feature is independently normally distributed:

Probability density: For feature :

Joint probability (assuming independence):

Parameter estimation from training data :

B. Multivariate Gaussian Model

Motivation: Univariate model assumes feature independence, missing feature
correlations.
Multivariate Gaussian ( ):

Probability density function:

where:
 is the mean vector
is the covariance matrix
is the determinant of
Parameter estimation:

C. Mahalanobis Distance
Definition: Generalized distance metric accounting for feature covariance and
correlation.

Formula:

Relationship to Multivariate Gaussian:

The probability density can be rewritten in terms of Mahalanobis distance:

Advantages over Euclidean distance:

Accounts for feature correlation structure

Adjusts for varying feature scales
In units of standard deviations (interpretable)
Handles singular covariance matrices (with regularization)

4.3 Threshold Selection Algorithm

Problem: How to choose threshold to balance false alarms vs. missed detections?

Core Idea: Use a separate validation set with known labels to find the best threshold.
Data Organization:
Training set (60%): Learn probability model from normal data
Validation set (20%): Find optimal threshold
Test set (20%): Final evaluation on unseen data

Simple Threshold Selection Algorithm:

1. Fit probability model on training data
2. Compute probability scores on validation set
3. Try different threshold values ( )
4. For each :
 Classify points: anomaly if
Compute F1-score (balances precision and recall)
5. Select with highest F1-score
Why F1-score matters: With imbalanced data (many normal, few anomalies), accuracy is
misleading. F1-score balances:

Precision: Of detected anomalies, how many are real? (avoid false alarms)
Recall: Of all real anomalies, how many detected? (avoid missing anomalies)
Pseudo-code:
best_f1 = -1
for epsilon in candidate_thresholds:
predictions = (P(X_cv) < epsilon)
f1 = compute_f1_score(predictions, y_cv)
if f1 > best_f1:
best_f1 = f1
best_epsilon = epsilon
return best_epsilon
The threshold is determined entirely by this validation set search—no assumptions needed.

4.4 Anomaly Detection Methods: Theory and Implementation

Method 1: Multivariate Gaussian Modeling
class MultivariatGaussianAnomalyDetector:
"""
Anomaly detection via Multivariate Gaussian density estimation.
"""

def __init__(self):
[Link] = None
[Link] = None
[Link] = None

def fit(self, X_train):

"""
Estimate mean and covariance from training data.

Formula:
μ = (1/m) Σᵢ x⁽ⁱ⁾
Σ = (1/m) Σᵢ (x⁽ⁱ⁾ - μ)(x⁽ⁱ⁾ - μ)ᵀ

Parameters:
-----------
 X_train : array, shape (n_samples, n_features)
"""
X_train = [Link](X_train)
[Link] = X_train.mean(axis=0)

# Center data
X_centered = X_train - [Link]

# Compute covariance matrix

[Link] = (X_centered.T @ X_centered) / len(X_train)

return self

def compute_probability(self, X):

"""
Compute probability density P(x) under multivariate Gaussian.

Formula: P(x) = (1/(2π)^(n/2)|Σ|^(1/2)) exp(-½(x-μ)ᵀΣ⁻¹(x-μ))

Parameters:
-----------
X : array, shape (n_samples, n_features)

Returns:
--------
probabilities : array, shape (n_samples,)
"""
X = [Link](X)
n = [Link][1]

# Compute Mahalanobis distance

X_centered = X - [Link]
sigma_inv = [Link]([Link])
mahal_dist_sq = [Link](X_centered @ sigma_inv * X_centered, axis=1)

# Compute probability density

det_sigma = [Link]([Link])
normalization = 1.0 / ([Link]((2 * [Link]) ** n * det_sigma))
 probabilities = normalization * [Link](-0.5 * mahal_dist_sq)

return probabilities

def compute_mahalanobis_distance(self, X):

"""
Compute Mahalanobis distance D_M(x, μ).

Formula: D_M(x, μ) = sqrt((x-μ)ᵀΣ⁻¹(x-μ))

Parameters:
-----------
X : array, shape (n_samples, n_features)

Returns:
--------
distances : array, shape (n_samples,)
"""
X = [Link](X)
X_centered = X - [Link]
sigma_inv = [Link]([Link])
mahal_dist_sq = [Link](X_centered @ sigma_inv * X_centered, axis=1)
return [Link](mahal_dist_sq)

def set_threshold(self, X_cv, y_cv, metric='probability'):

"""
Select threshold ε using cross-validation data.

Algorithm:
1. Compute metric (probability or Mahalanobis distance) on CV set
2. For each candidate ε, compute F1-score
3. Select ε* = argmax F1(ε)

Parameters:
-----------
X_cv : array, shape (n_cv_samples, n_features)
Cross-validation features
y_cv : array, shape (n_cv_samples,)
 True labels (0=normal, 1=anomaly)
metric : str
'probability' (flag if P(x) < ε) or 'mahalanobis' (flag if D_M > ε)

Returns:
--------
self
"""
from [Link] import f1_score

X_cv = [Link](X_cv)
y_cv = [Link](y_cv)

# Compute metric on CV set

if metric == 'probability':
scores = self.compute_probability(X_cv)
epsilon_range = [Link](scores, [Link](1, 99, 99))
else: # mahalanobis
scores = self.compute_mahalanobis_distance(X_cv)
epsilon_range = [Link](scores, [Link](1, 99, 99))

best_f1 = -1
best_threshold = None

for epsilon in epsilon_range:

if metric == 'probability':
y_pred = (scores < epsilon).astype(int)
else:
y_pred = (scores > epsilon).astype(int)

# Compute F1 score
f1 = f1_score(y_cv, y_pred, zero_division=0)

if f1 > best_f1:
best_f1 = f1
best_threshold = epsilon

[Link] = best_threshold
 self.metric_type = metric
return self

def predict(self, X, metric=None):

"""
Predict anomalies: 1 if anomaly, 0 if normal.

Parameters:
-----------
X : array, shape (n_samples, n_features)
metric : str
Override metric type (use stored metric_type if None)

Returns:
--------
predictions : array, shape (n_samples,)
"""
X = [Link](X)
metric = metric or getattr(self, 'metric_type', 'probability')

if metric == 'probability':
scores = self.compute_probability(X)
return (scores < [Link]).astype(int)
else: # mahalanobis
scores = self.compute_mahalanobis_distance(X)
return (scores > [Link]).astype(int)

def anomaly_scores(self, X, metric=None):

"""
Return raw anomaly scores (not thresholded predictions).

Parameters:
-----------
X : array, shape (n_samples, n_features)
metric : str

Returns:
--------
 scores : array, shape (n_samples,)
Lower probability or higher Mahalanobis distance = more anomalous
"""
metric = metric or getattr(self, 'metric_type', 'probability')

if metric == 'probability':
return self.compute_probability(X)
else:
return self.compute_mahalanobis_distance(X)

Method 2: Isolation Forest (Tree-based Anomaly Detection)

Theory: Isolation Forest exploits the key insight that anomalies are isolated—they are
fewer, different, and can be separated from normal data with fewer random splits. The
algorithm recursively partitions the feature space randomly, and anomalous points require
fewer partitions to be isolated from the rest of the data.
Core Principle:
Normal points are dense and clustered; they require many splits to isolate
Anomalous points are sparse and distant; they are isolated quickly (fewer splits)
Anomaly score is based on the path length from root to leaf: shorter paths = more
anomalous

Key Advantages:
No distributional assumptions (works with any data distribution)
Highly efficient for high-dimensional data
Naturally handles mixed feature types
Robust to outliers (outliers are isolated, not affecting density estimates)
Linear time complexity per tree
When to use:

High-dimensional data (100s of features) where Gaussian model fails

Non-Gaussian distributed data
Mixed discrete and continuous features
Need robustness to extreme outliers
Computational efficiency important
class IsolationForestDetector:
"""
Anomaly detection via Isolation Forest.
"""

def __init__(self, n_estimators=100, contamination=0.1, random_state=None):

self.n_estimators = n_estimators
 [Link] = contamination
self.random_state = random_state
[Link] = []
[Link](random_state)

def fit(self, X_train):

"""Build isolation forest by creating multiple random trees."""
X_train = [Link](X_train)
for _ in range(self.n_estimators):
sample_size = int(0.632 * len(X_train))
sample_indices = [Link](len(X_train), sample_size, replace=Fals
X_sample = X_train[sample_indices]
tree = self._build_isolation_tree(X_sample)
[Link](tree)
return self

def _build_isolation_tree(self, X, depth=0, max_depth=None):

"""Recursively partition data randomly."""
if max_depth is None:
max_depth = int([Link](np.log2(len(X))))

if depth >= max_depth or len(X) <= 1:

return {'type': 'leaf', 'size': len(X)}

feature_idx = [Link](0, [Link][1])

feature_values = X[:, feature_idx]
split_value = [Link](feature_values.min(), feature_values.max()

left_mask = X[:, feature_idx] < split_value

X_left = X[left_mask]
X_right = X[~left_mask]

if len(X_left) == 0 or len(X_right) == 0:
return {'type': 'leaf', 'size': len(X)}

return {
'type': 'internal',
'feature': feature_idx,
 'value': split_value,
'left': self._build_isolation_tree(X_left, depth + 1, max_depth),
'right': self._build_isolation_tree(X_right, depth + 1, max_depth)
}

def _compute_path_length(self, x, tree, depth=0):

"""Compute how many splits needed to isolate this point."""
if tree['type'] == 'leaf':
return depth
feature_idx = tree['feature']
if x[feature_idx] < tree['value']:
return self._compute_path_length(x, tree['left'], depth + 1)
else:
return self._compute_path_length(x, tree['right'], depth + 1)

def predict(self, X):

"""Predict anomalies: 1 if anomalous, 0 if normal."""
scores = self.anomaly_scores(X)
threshold = [Link](scores, 100 * (1 - [Link]))
return (scores >= threshold).astype(int)

def anomaly_scores(self, X):

"""Compute anomaly scores (higher = more anomalous)."""
X = [Link](X)
scores = [Link](len(X))

for i, x in enumerate(X):
path_lengths = [Link]([self._compute_path_length(x, tree) for tree in self
avg_path = path_lengths.mean()
scores[i] = 2 ** (-avg_path / max(1.0, np.log2(len(X))))

return scores
4.5 Comparison of Anomaly Detection Methods

Multivariate
Aspect Isolation Forest
Gaussian
Models P(x) via Counts splits to
How it works
covariance isolate point
Distribution
Assumes Gaussian No assumptions
assumption
Feature Explicitly via
Implicitly captured
correlation covariance
High (Mahalanobis Medium (ensemble-
Interpretability
distance) based)
High-dimensional Struggles (singular Effective (exploits
data covariance) sparsity)
Moderate (affects Low (quickly
Outlier sensitivity
covariance) isolated)
Moderate dims, High dims, non-
Best for
Gaussian data Gaussian

4.6 Practical Implementation Notes

For Multivariate Gaussian:

Always standardize features before fitting

Verify Gaussian assumption via Q-Q plots
Use regularization on covariance matrix for stability
Threshold selection critical; use F1-score on validation set
For Isolation Forest:
Works directly on raw features (no scaling needed)
Set contamination parameter based on expected anomaly rate
Increase n_estimators (trees) for more stable estimates
Efficient even with 100s of features

Choosing between methods:

Start with Multivariate Gaussian if data appears normal-ish and interpretability
matters
Switch to Isolation Forest if:
Data is clearly non-Gaussian
You have 50+ features
 Performance of Gaussian is poor
Handling mixed data types

5. DECISION FRAMEWORK: SUPERVISED VS.

UNSUPERVISED LEARNING
5.1 Conceptual Comparison
Supervised Learning learns a mapping from features to labels: given labeled examples
, find to minimize prediction error. This works well when you have
moderate-to-large numbers of both positive and negative examples. The model learns
—the probability of the label given the features.
Anomaly Detection learns the distribution of normal data: from mostly normal unlabeled
data, learn —the probability of observing a feature vector. When new data has very
low probability under this model, it's flagged as anomalous. This is ideal when positive
examples are rare or unknown.

Core Difference: Supervised learning answers "What class is this?" while anomaly
detection answers "Is this normal?"

5.2 Decision Framework

When data has few positive examples (< 50 known anomalies or rare events):

Supervised learning struggles: insufficient positive examples to learn robust

Anomaly detection thrives: learn from abundant normal data via
Reasoning: Learning "what is normal" from 1 million examples is more reliable than
learning "what is rare" from 10 examples
When data has balanced classes (5-30% positive examples):
Use supervised learning with careful evaluation (F1-score, precision-recall curves)
Consider reweighting classes to penalize minority class misclassification
Enough positive examples exist to reliably estimate

When data has abundant labeled examples (> 200 positive examples):
Supervised learning is preferred
You have sufficient signal in both classes
Direct mapping is learnable and interpretable

5.3 Practical Decision Rules

Rule 1: Count Positive Examples
If you have fewer than 50 known positive examples (anomalies, failures, fraud cases, etc.),
strongly consider anomaly detection. With so few positive examples, any supervised
classifier will likely overfit or miss important patterns in the positive class.
Rule 2: Check Class Balance
Compute

If balance < 0.05 (less than 5% positive): Use anomaly detection

If 0.05 < balance < 0.3 (5-30% positive): Use supervised learning with class reweighting
If balance > 0.3 (more than 30% positive): Use standard supervised learning
Rule 3: Assess Feature Characteristics

If features are highly correlated: Multivariate Gaussian (anomaly detection) or tree-

based methods (supervised) work well
If features are approximately independent: Any method works; choose based on
data volume
Rule 4: Consider Problem Type
Fraud/Intrusion/Faults: Typically anomaly detection (new fraud patterns emerge
constantly)
Classification: Typically supervised learning (classes well-understood)
Predictive Maintenance: Often hybrid (start with anomaly detection, transition to
supervised as data accumulates)

5.4 Real-World Decision Examples

Example 1: Credit Card Fraud Detection
Scenario: A bank processes 1 million transactions daily with ~500 fraudulent cases (0.05%
rate).
Solution: Use Anomaly Detection. With only 500 labeled frauds, supervised learning has
insufficient positive examples. Model normal transactions: what does legitimate look like?
Flag transactions with P(x) < ε for manual review.

Example 2: Manufacturing Equipment Failure

Scenario: Factory with 50 sensor-monitored machines, 50 failures over 1 year (0.5% rate).
Solution: Start with Anomaly Detection (Isolation Forest on normal operations), collect
labeled failures. Transition to supervised learning once 200+ labeled failures accumulated.

Example 3: Email Spam Detection

Scenario: 10 million emails/day, ~15% spam, abundant labeled data.
Solution: Use Supervised Learning. 15% rate provides 1.5 million daily spam examples—
enough for robust classifier. Learn P(spam|x) directly.

6. SUMMARY & KEY TAKEAWAYS

6.1 Key Concepts Recap
Unsupervised Learning: Discovers hidden structure in unlabeled data through density
estimation, clustering, and dimensionality reduction.
K-Means Clustering:
Minimizes WCSS via coordinate descent
Simple, interpretable, but assumes spherical clusters
Use K-Means++ initialization and multiple restarts
Evaluate with Silhouette, Davies-Bouldin, or Calinski-Harabasz

Anomaly Detection:
Multivariate Gaussian: Models P(x), interprets via Mahalanobis distance
Isolation Forest: Path-length based, no distributional assumptions
Threshold tuned on validation set using F1-score
Best for rare events with < 5% positive examples
Decision Framework:

Data Scenario Recommended Approach

< 50 positive examples Anomaly Detection
5-30% class balance Supervised (with reweighting)
> 200 positive examples Supervised Learning

6.2 When to Use Each Approach

K-Means Clustering:

Customer segmentation with distinct groups

When number of clusters known or determinable
Data exploration and unsupervised learning
Anomaly Detection:
Fraud, intrusion, fault detection
Rare events (< 5% positive examples)
Need interpretability and explainability

Supervised Learning:
Balanced classes or 100s of positive examples
Clear feature-to-label mapping
Accuracy and precision paramount
6.3 Critical Implementation Principles
1. Feature Scaling: Standardize before fitting (especially Gaussian, Isolation Forest)
2. Data Splitting: 60% train / 20% CV / 20% test
3. Threshold Tuning: Use F1-score on validation set, not accuracy
4. K-Means++: Dramatically improves quality, always use
5. Evaluation: Silhouette for clustering; Precision-Recall for anomaly detection
6. Multiple Restarts: Run multiple times, keep best result
7. Monitoring: Retrain periodically; watch for distribution shift

7. REFERENCES
[1] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
[2] Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press.
[3] Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data
Mining, Inference, and Prediction (2nd ed.). Springer.

[4] Ng, A. (2019). Machine Learning Yearning. Self-published.

[5] Lloyd, S. P. (1982). Least squares quantization in PCM. IEEE Transactions on Information
Theory, 28(2), 129-137.
[6] Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM
Computing Surveys, 41(3), 1-58.

[7] Liu, F. T., Ting, K. M., & Zhou, Z. (2008). Isolation forest. In ICDM 2008.
[8] Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
[9] Mahalanobis, P. C. (1936). On the generalized distance in statistics. Proceedings of the
National Institute of Sciences of India, 12, 49-55.

[10] Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation
of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53-65.

Document Version: 2.0 Enhanced

Last Updated: January 4, 2026
Total Pages: 60+ (Comprehensive Professional Grade)
Status: ✅ COMPLETE WITH ADVANCED THEORETICAL FOUNDATIONS