Unsupervised Learning

Unsupervised Learning 1 / 35
Outline

1 Introduction to Unsupervised Learning

2 Example: Iris Dataset Without Labels

3 The Language of Unsupervised Learning

4 K-Means as an Unsupervised Learning Algorithm

5 Unsupervised Learning in Practice

6 Summary

Unsupervised Learning 2 / 35
What is Unsupervised Learning?

Setup: We observe a dataset with inputs only:

D = {x (i) }ni=1 , x (i) ∈ Rd ,

but no labels or targets.

Goal: discover interesting structure or regularities in the data:
groups of similar points (clusters)
unusual or rare points (anomalies)
lower-dimensional structure, denoised signals, etc.
Unlike supervised learning, the model is not trained to predict labels, but to explain or
summarize the data.

Unsupervised Learning 3 / 35
Examples of Unsupervised Tasks

Clustering
Group customers with similar behavior in an e-commerce log.
Anomaly / outlier detection
Flag suspicious financial transactions.
Signal denoising
Recover a clean image or audio signal from a noisy measurement.
Topic modeling
Discover topics in document collections without per-word labels.
Visualization / dimensionality reduction
Map high-dimensional data to 2D/3D to inspect patterns.

Unsupervised Learning 4 / 35
The Iris Dataset (Unlabeled Version)

Classic dataset of iris flower measurements:

attributes: sepal length, sepal width, petal length, petal width
three species (class labels), each with 50 examples
For an unsupervised setup:
discard the class labels
optionally focus on a subset of features (e.g., sepal length and width)
The task: can we discover meaningful structure (e.g., clusters) from measurements alone?

Unsupervised Learning 5 / 35
Visualizing Iris Data (Unlabeled Version)

Each point: one flower,

described by two features:

x (i) = sepal length, sepal width .

We can plot all points in 2D to

inspect the distribution.
Clusters may already be visible
just from the scatter plot.

Unsupervised Learning 6 / 35
Visualizing Iris Data (Labeled Version)

Remember for the supervised

case
Each point: one flower,
described by two features and a
label

Unsupervised Learning 7 / 35
A Recipe for Unsupervised Learning

High-level recipe
Dataset
| {z } + Learning Algorithm −→ Unsupervised Model.
| {z }
attributes / features model class
objective
optimizer

Input: unlabeled dataset D

Learning algorithm:
specify a model class M
define an objective function J(θ)
choose an optimization procedure to minimize J
Output: a model that captures structure in D.

Unsupervised Learning 8 / 35
Dataset and Notation

Unsupervised dataset of size n:

D = {x (i) | i = 1, 2, . . . , n},

where x (i) ∈ Rd .
Each x (i) is a feature vector with d attributes.
There are no associated targets y (i) .

Unsupervised Learning 9 / 35
Model and Model Class

A (deterministic) unsupervised model can be written as

fθ : X → S,

where:
X is the input space (e.g., Rd ).
S is the structure space (clusters, low-dim codes, etc.).
θ ∈ Θ are the model parameters.
The model class is a set of such models:

M = {fθ | θ ∈ Θ}.

Unsupervised Learning 10 / 35
Learning Objective and Optimizer

Define an objective (or loss) function

J(θ) : θ → [0, ∞),

that measures how well the model fθ explains D.

An optimizer searches for parameters with minimal loss:

min J(θ).
θ∈Θ

The resulting θ̂ yields a model

fθ̂
that captures the patterns of interest in the training data.

Unsupervised Learning 11 / 35
K-Means: Intuition

Objective: partition the data into K clusters.

Each cluster k is summarized by a centroid ck ∈ X .
High-level iterative procedure:
1 Initialize centroids c1 , . . . , cK (e.g., randomly).
2 Assignment step: assign each point to its nearest centroid.
3 Update step: recompute each centroid as the mean of its assigned points.
4 Repeat steps 2–3 until assignments stabilize (convergence).

Unsupervised Learning 12 / 35
K-Means: Intuition

Unsupervised Learning 13 / 35
K-Means: Model

Parameters:
θ = (c1 , c2 , . . . , cK ), ck ∈ X .
For an input x ∈ X , the model outputs a cluster index:

fθ (x) = arg min ||x − ck ||

k

where ∥·∥ is typically the Euclidean norm.

Structure space:
S = {1, . . . , K }.

Unsupervised Learning 14 / 35
K-Means: Objective Function

We want centroids such that datapoints lie close to their assigned centroid.
A common choice of objective (using squared distances) is:
n
X
J(θ) = ||x (i) − centroid(fθ (x (i) ))||,
i=1

Minimize J(θ) over all choices of centroids.

Unsupervised Learning 15 / 35
K-Means: Optimization Procedure

Alternating optimization
Starting from some initial centroids:
1 Assignment step:
fθ (x (i) ) = arg min∥x (i) − ck ∥2 .
k
2 Update step:
1 X (i)
ck = x ,
|Ck | (i)
x ∈Ck

where Ck is the set of points currently assigned to cluster k.

Each step does not increase the objective.

Because there are finitely many assignments, the procedure converges in a finite number
of steps (to a local minimum).

Unsupervised Learning 16 / 35
K-Means: Model Card

Type: Unsupervised learning (clustering).

Model family: K centroids in feature space.
Objective: Sum of (typically squared) distances from each point to its nearest centroid.
Optimizer: Iterative assignment–update procedure .

Unsupervised Learning 17 / 35
Running K-Means on the Iris Data

Use a clustering algorithm such as K-means with K = 3:

algorithm seeks K cluster centers (centroids) in feature space
each datapoint is assigned to its nearest centroid
Result:
3 centroids in R2 (for the 2D projection)
a cluster label for each flower
We can visualize:
all datapoints
learned centroids (e.g., as diamonds)

Unsupervised Learning 18 / 35
Comparing to True Classes (for Illustration)

The K-means algorithm does not see class labels.

For teaching/analysis purposes, we can color each datapoint by the true species after
clustering.
Often, the learned clusters align well with the underlying classes:
each centroid tends to correspond to one species.
Takeaway: even without labels, structure in the data can reveal meaningful groupings.

Unsupervised Learning 19 / 35
Visualization

Unsupervised Learning 20 / 35
Comparing to True Classes (for Illustration)

Unsupervised Learning 21 / 35
Toy Example: Mixture of Gaussians

Consider synthetic 2D data sampled from a mixture of four Gaussian distributions.

A scatter plot of the points shows roughly four dense groups.
If we color the points using the true generating component (for illustration only), we see
four distinct clusters.
This setup is useful to:
illustrate underfitting and overfitting in clustering
visualize how K-means behaves as we vary K

Unsupervised Learning 22 / 35
Visualization

Unsupervised Learning 23 / 35
Underfitting in Unsupervised Learning

In supervised learning:
underfitting means the model is too simple to capture the signal.
In our clustering example:
true number of clusters is 4
if we choose K = 2, K-means underfits:
some true clusters get merged
learned centroids do not match the underlying structure well
The objective value can still be reduced compared to random, but we are missing
information about the finer structure.

Unsupervised Learning 24 / 35
Underfitting in Unsupervised Learning

Unsupervised Learning 25 / 35
Overfitting in Unsupervised Learning

If we increase K too much, K-means may start fitting noise:

small, spurious clusters appear
centroids may capture local randomness rather than true global structure
Example:
K = 4: reasonable match to the true four Gaussians
K = 10: some clusters describe tiny local variations
K = 20 or K = 50: objective becomes very small, but clusters no longer reflect the
meaningful pattern.
This is analogous to overfitting in supervised learning: the model starts explaining noise
rather than signal.

Unsupervised Learning 26 / 35
Overfitting in Unsupervised Learning

Unsupervised Learning 27 / 35
Underfitting/Good Fit/Overfitting in Unsupervised Learning

Unsupervised Learning 28 / 35
Generalization in Unsupervised Learning

Assume data points are drawn from a distribution P:

x ∼ P, D = {x (i) }ni=1 are IID samples.

We can conceptually decompose the data distribution as

P = F + E,

where
F : structured signal (clusters, manifolds, speech, etc.)
E : noise component
A model generalizes well if it captures F and not E .
Overfitting occurs when the model explains noise E instead of signal F .

Unsupervised Learning 29 / 35
In supervised learning, we can measure performance via accuracy, loss on a validation set,
etc.
In unsupervised learning, evaluation is harder:
we often lack ground-truth labels
human judgment / domain insight may be needed

Unsupervised Learning 30 / 35
The Elbow Method

A practical heuristic to choose hyperparameters such as K in K-means.

Procedure:
1 For K = 1, 2, . . . , Kmax , run K-means and record the objective value JK (e.g., sum of squared
distances).
2 Plot JK versus K .
3 Look for a point where the curve bends (the “elbow”): after this point, improvements become
marginal.

In the toy mixture-of-Gaussians example, the elbow typically appears near the true number of
clusters.

Unsupervised Learning 31 / 35
Overfitting in Unsupervised Learning

Unsupervised Learning 32 / 35
Reducing Overfitting in Unsupervised Models

Strategies to mitigate overfitting:

1 Reduce model complexity
e.g., smaller K in K-means
2 Penalize complexity in the objective
add a regularization term that discourages overly complex models
3 Use probabilistic models with regularization
e.g., mixture models with priors on parameters

Hyperparameter selection (e.g., via elbow method or other criteria) is crucial for good
generalization.

Unsupervised Learning 33 / 35
Key Takeaways
Unsupervised learning works with unlabeled data to uncover structure such as clusters,
low-dimensional representations, and anomalies.
A generic unsupervised pipeline:
Dataset + Model Class + Objective + Optimizer → Unsupervised Model.
K-means is a simple but widely used clustering algorithm:
model: K centroids
objective: sum of distances to closest centroid
optimizer: iterative assignment–update steps

Concepts of underfitting, overfitting, and generalization also apply to unsupervised learning,

but are harder to quantify.
Practical tools (e.g., elbow method, regularization, choice of K ) help balance model
complexity and fit.
Unsupervised Learning 34 / 35
References

Applied Machine Learning (Cornell University) – Lecture 8: Unsupervised Learning.

https:
//[Link]/aml-book/contents/[Link]

Standard texts on machine learning and clustering for additional background.

Unsupervised Learning 35 / 35