University of Colorado Boulder
Unsupervised Learning
Geena Kim
Introduction
University of Colorado Boulder Geena Kim
Types of machine learning problems
Tasks
Learning Regression
- Predicts real-valued numbers
Supervised
- With labels Classification Binary class
- Predicts categories
Multi-class
ML Unsupervised
- Without labels
Reinforcement
- With feedback
University of Colorado Boulder Geena Kim
Why Unsupervised Learning
Learning
Supervised
- With labels
ML Unsupervised
- Without labels
Reinforcement
- With feedback
University of Colorado Boulder Geena Kim
Why Unsupervised Learning
Yann LeCun says about Unsupervised Learning…
Learning
in terms of data availability
Supervised
- With labels
ML Unsupervised
- Without labels
Reinforcement
- With feedback
University of Colorado Boulder Geena Kim
Goals of Unsupervised Learning
Not interested in prediction but to discover interesting things about the data
Informative visualization
Finding subgroups Clustering
Dimensionality Reduction
Preprocessing
Data synthesis
University of Colorado Boulder Geena Kim
Visualization by unsupervised learning
Image credit: [Link] and L van der Maaten et al (2008)
University of Colorado Boulder Geena Kim
Dimensionality Reduction
Projection to low-dimension Manifold learning
Image credit: ISLR textbook and Zhang et al (2010)
University of Colorado Boulder Geena Kim
Clustering
• Marketing and sales
• Social network analysis
• Genomics, Oncology
Image credit: T. Sørlie et al (2001)
University of Colorado Boulder Geena Kim
Applications: Recommender System
• Similarity based
• Learning latent features/Matrix Factorization
• Collaborative Filtering using Graph
University of Colorado Boulder Geena Kim
Data Generation
Image credit: Goodfellow et al(2014),
Ledig et al (2016),
Zhu et al (2017),
Han et al (2018)
University of Colorado Boulder Geena Kim
Self-supervision
• A generative model to reconstruct inputs (Autoencoders)
• Surrogate tasks in vision tasks
• Using clustering for graphs
• Pretraining for NLP tasks (GPT)
Image credit: Kingma & Welling (2013)
University of Colorado Boulder Geena Kim
Summary
• Unsupervised Learning
• Usage:
• Dimensionality reduction, pre-training, visualization
• Clustering (marketing, medicine, etc)
• Data generation
• Industrial applications such as Recommender systems
University of Colorado Boulder Geena Kim
Principal Component Analysis
University of Colorado Boulder Geena Kim
Dimensionality Reduction
Curse of dimensionality
Data become sparse
Features in high dimension tend to be redundant (and correlated)
Likely to overfit
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
PCA is a popular dimensionality reduction technique
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
PCA is a popular dimensionality reduction technique
Principal components
Normalized loading vectors
c1
c2
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
How to choose the principal components?
Method 1. Preserve the maximum variance
Method 2. Choose axis that minimizes the mean squared
distance between the original dataset and its projection
onto the axis
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
University of Colorado Boulder Geena Kim
Principal Component Analysis (PCA)
The best vector to project onto is called the 1st principal
component.
What properties should it have?
• Should capture largest variance in data
• Should probably be a unit vector
After we’ve found the first, look the second which:
• Captures largest amount of leftover variance
• Should probably be a unit vector
• Should be orthogonal to the one that came before it
University of Colorado Boulder Geena Kim
University of Colorado Boulder Geena Kim
University of Colorado Boulder Geena Kim
University of Colorado Boulder Geena Kim
University of Colorado Boulder Geena Kim
University of Colorado Boulder Geena Kim
How to find principal components
Define covariance matrix
X has zero mean
Eigenvectors of the covariance matrix
are the principal components
University of Colorado Boulder Geena Kim
Explained Variance Ratio
How many dimensions should we choose to use? What is explained variance?
What is explained variance ratio?
University of Colorado Boulder Geena Kim
PCA in sklearn
University of Colorado Boulder Geena Kim
PCA Applications
Principal Component Regression (PCR)
• Use transformed features
• The transformed features are uncorrelated
• Lower dimension helps
• Difficult to interpret
University of Colorado Boulder Geena Kim
PCA Applications
Eigenfaces, Face recognition
Turk, Matthew A; Pentland, Alex P (1991). Face recognition using eigenfaces
University of Colorado Boulder Geena Kim
Summary
• PCA as Dimensionality Reduction Techniques
• Finds axes that maximize the variance
• Explained variance ratio
• Feature selection
• Applications to PCR and face detection
University of Colorado Boulder Geena Kim