Pattern Recognition & Machine Learning
K Sri Rama Murty
IIT Hyderabad
ksrm@[Link]
K Sri Rama Murty (IITH) PRML 1 / 12
Prerequisites
Linear algebra
Calculus
Probability & random variables
Basic understanding of random processes & DSP
K Sri Rama Murty (IITH) PRML 2 / 12
Prerequisites
Linear algebra
Calculus
Probability & random variables
Basic understanding of random processes & DSP
EE2802 EE1010, EE1030, EE1080, EE2080, EE2000
EE5610 EE5609, EE5817
Similar courses
EE7390 - Renumbered as EE5610 (100% overlap)
EE5600, EE5601 & EE5602 (60-70% overlap)
K Sri Rama Murty (IITH) PRML 2 / 12
Pattern Recognition
Pattern
Dictionary: The regular way in which something happens or is done
Distinguishable regularity, structure, or arrangement of characteristics
in data that reveals underlying relationships or trends
K Sri Rama Murty (IITH) PRML 3 / 12
Pattern Recognition
Pattern
Dictionary: The regular way in which something happens or is done
Distinguishable regularity, structure, or arrangement of characteristics
in data that reveals underlying relationships or trends
Facial features, hand-written characters, spoken words, musical notes,
call records, share market trends, medical reports etc.,
K Sri Rama Murty (IITH) PRML 3 / 12
Pattern Recognition
Pattern
Dictionary: The regular way in which something happens or is done
Distinguishable regularity, structure, or arrangement of characteristics
in data that reveals underlying relationships or trends
Facial features, hand-written characters, spoken words, musical notes,
call records, share market trends, medical reports etc.,
Natural variations are inevitable - realizations don’t match exactly
Human beings are good at recognizing patterns but can’t explain how.
K Sri Rama Murty (IITH) PRML 3 / 12
Pattern Recognition
Pattern
Dictionary: The regular way in which something happens or is done
Distinguishable regularity, structure, or arrangement of characteristics
in data that reveals underlying relationships or trends
Facial features, hand-written characters, spoken words, musical notes,
call records, share market trends, medical reports etc.,
Natural variations are inevitable - realizations don’t match exactly
Human beings are good at recognizing patterns but can’t explain how.
Template matching approach to pattern recognition
Compare input data directly with stored templates
Matched filtering, KNN methods, dynamic-time warping, etc.
Works well on simple tasks, but is rigid and less adaptable
K Sri Rama Murty (IITH) PRML 3 / 12
Pattern Recognition
Pattern
Dictionary: The regular way in which something happens or is done
Distinguishable regularity, structure, or arrangement of characteristics
in data that reveals underlying relationships or trends
Facial features, hand-written characters, spoken words, musical notes,
call records, share market trends, medical reports etc.,
Natural variations are inevitable - realizations don’t match exactly
Human beings are good at recognizing patterns but can’t explain how.
Template matching approach to pattern recognition
Compare input data directly with stored templates
Matched filtering, KNN methods, dynamic-time warping, etc.
Works well on simple tasks, but is rigid and less adaptable
Statistical pattern recognition
Use data to train models that can generalize to unseen patterns.
Flexible and capable of handling complex, high-dimensional problems.
K Sri Rama Murty (IITH) PRML 3 / 12
Machine Learning
Data is at the core of machine learning
Design methods to extract valuable patterns from the data
K Sri Rama Murty (IITH) PRML 4 / 12
Machine Learning
Data is at the core of machine learning
Design methods to extract valuable patterns from the data
Model the underlying process that generates data
In regression, the model would describe a function that maps i/p to o/p
For example, regress weight of a person from his height
w = a0 + a1 h a = [a0 a1 ]T
Goal is to find good models that generalize well to yet unseen data
K Sri Rama Murty (IITH) PRML 4 / 12
Machine Learning
Data is at the core of machine learning
Design methods to extract valuable patterns from the data
Model the underlying process that generates data
In regression, the model would describe a function that maps i/p to o/p
For example, regress weight of a person from his height
w = a0 + a1 h a = [a0 a1 ]T
Goal is to find good models that generalize well to yet unseen data
Learning - a way to automatically find patterns/structure in data
Optimize the parameters of the model to approximate the data
Transforming experiences (data) into expertise
K Sri Rama Murty (IITH) PRML 4 / 12
Machine Learning
Data is at the core of machine learning
Design methods to extract valuable patterns from the data
Model the underlying process that generates data
In regression, the model would describe a function that maps i/p to o/p
For example, regress weight of a person from his height
w = a0 + a1 h a = [a0 a1 ]T
Goal is to find good models that generalize well to yet unseen data
Learning - a way to automatically find patterns/structure in data
Optimize the parameters of the model to approximate the data
Transforming experiences (data) into expertise
Design a parameterized model and let the learning algorithm find
the best set of parameters to approximate the observed data
K Sri Rama Murty (IITH) PRML 4 / 12
Learning Paradigms
Supervised learning (learning with a teacher)
Given training examples of input-output pairs, produce ”correct”
output for a new input
K Sri Rama Murty (IITH) PRML 5 / 12
Learning Paradigms
Supervised learning (learning with a teacher)
Given training examples of input-output pairs, produce ”correct”
output for a new input
Regression: The output takes continuous values
Classification: The output admits only a discrete set of values
3D image reconstruction, signal enhancement, disease detection,
character recognition
K Sri Rama Murty (IITH) PRML 5 / 12
Learning Paradigms
Supervised learning (learning with a teacher)
Given training examples of input-output pairs, produce ”correct”
output for a new input
Regression: The output takes continuous values
Classification: The output admits only a discrete set of values
3D image reconstruction, signal enhancement, disease detection,
character recognition
Unsupervised learning (learning without a teacher)
Given the data, discover the hidden structure in it.
Clustering, density estimation, novelty detection, compression,
K Sri Rama Murty (IITH) PRML 5 / 12
Learning Paradigms
Supervised learning (learning with a teacher)
Given training examples of input-output pairs, produce ”correct”
output for a new input
Regression: The output takes continuous values
Classification: The output admits only a discrete set of values
3D image reconstruction, signal enhancement, disease detection,
character recognition
Unsupervised learning (learning without a teacher)
Given the data, discover the hidden structure in it.
Clustering, density estimation, novelty detection, compression,
Reinforcement learning (learning with a critic)
Perform actions based on inputs from environment (exploit & explore)
Actions may be suitably rewarded or punished
Perform a sequence of actions that maximizes the reward
K Sri Rama Murty (IITH) PRML 5 / 12
Clustering: Which one is the best?
K Sri Rama Murty (IITH) PRML 6 / 12
Clustering: Which one is the best?
K Sri Rama Murty (IITH) PRML 6 / 12
Clustering: Which one is the best?
K Sri Rama Murty (IITH) PRML 6 / 12
Clustering: Which one is the best?
K Sri Rama Murty (IITH) PRML 6 / 12
Clustering: Which one is the best?
K Sri Rama Murty (IITH) PRML 6 / 12
Clustering: Which one is the best?
K Sri Rama Murty (IITH) PRML 6 / 12
Course Contents
K Sri Rama Murty (IITH) PRML 7 / 12
Course Contents
Linear models
Regression - Least squares estimate, Maximum-likelihood estimate,
Maximum a posteriori estimate, predictive distribution
K Sri Rama Murty (IITH) PRML 7 / 12
Course Contents
Linear models
Regression - Least squares estimate, Maximum-likelihood estimate,
Maximum a posteriori estimate, predictive distribution
Classification - linear discriminant functions, Fisher discriminant
analysis, perceptron, SVM, Bayes decision theory, generative vs
discriminative models - ML and MAP approaches, logistic regression
K Sri Rama Murty (IITH) PRML 7 / 12
Course Contents
Linear models
Regression - Least squares estimate, Maximum-likelihood estimate,
Maximum a posteriori estimate, predictive distribution
Classification - linear discriminant functions, Fisher discriminant
analysis, perceptron, SVM, Bayes decision theory, generative vs
discriminative models - ML and MAP approaches, logistic regression
Nonlinear models
Cover’s theorem, Nonlinear kernel transformation, fixed vs data-driven
kernels, feed-forward neural networks, error back-propagation, practical
issues in training DNNs
K Sri Rama Murty (IITH) PRML 7 / 12
Course Contents
Linear models
Regression - Least squares estimate, Maximum-likelihood estimate,
Maximum a posteriori estimate, predictive distribution
Classification - linear discriminant functions, Fisher discriminant
analysis, perceptron, SVM, Bayes decision theory, generative vs
discriminative models - ML and MAP approaches, logistic regression
Nonlinear models
Cover’s theorem, Nonlinear kernel transformation, fixed vs data-driven
kernels, feed-forward neural networks, error back-propagation, practical
issues in training DNNs
(If time permits) Unsupervised learning
Clustering, density estimation, Gaussian mixture models, dimensionality
reduction, autoencoders and its variants
K Sri Rama Murty (IITH) PRML 7 / 12
Course Contents - Sequence Models
K Sri Rama Murty (IITH) PRML 8 / 12
Course Contents - Sequence Models
Restaurant review
The food was good, not bad at all
The food was bad, not good at all
Cannot ignore the sequence information!
K Sri Rama Murty (IITH) PRML 8 / 12
Course Contents - Sequence Models
Restaurant review
The food was good, not bad at all
The food was bad, not good at all
Cannot ignore the sequence information!
Sequence models
Markov model, hidden Markov model (HMM), linear dynamical
systems, Kalman filters
Convolutional neural networks (CNN), recurrent neural networks
(RNN), architectures, long-short term memory (LSTM), gated
recurrent units (GRU), attention mechanism, autoregressive
feed-forward networks, and transformers
K Sri Rama Murty (IITH) PRML 8 / 12
Course Contents - Sequence Models
Restaurant review
The food was good, not bad at all
The food was bad, not good at all
Cannot ignore the sequence information!
Sequence models
Markov model, hidden Markov model (HMM), linear dynamical
systems, Kalman filters
Convolutional neural networks (CNN), recurrent neural networks
(RNN), architectures, long-short term memory (LSTM), gated
recurrent units (GRU), attention mechanism, autoregressive
feed-forward networks, and transformers
K Sri Rama Murty (IITH) PRML 8 / 12
Reference Books
K Sri Rama Murty (IITH) PRML 9 / 12
Evaluation Criteria
Lecture summaries - 10% (Next day 8 pm)
A handwritten summary should be submitted after every lecture
Complete all partial derivations
K Sri Rama Murty (IITH) PRML 10 / 12
Evaluation Criteria
Lecture summaries - 10% (Next day 8 pm)
A handwritten summary should be submitted after every lecture
Complete all partial derivations
Homework assignments - 10% (+7 days 8 pm)
K Sri Rama Murty (IITH) PRML 10 / 12
Evaluation Criteria
Lecture summaries - 10% (Next day 8 pm)
A handwritten summary should be submitted after every lecture
Complete all partial derivations
Homework assignments - 10% (+7 days 8 pm)
Programming assignments - 10% (+14 days 8 pm)
Programming assignments have to be done in Python.
The skeleton structure of the code will be posted on Google Colab
K Sri Rama Murty (IITH) PRML 10 / 12
Evaluation Criteria
Lecture summaries - 10% (Next day 8 pm)
A handwritten summary should be submitted after every lecture
Complete all partial derivations
Homework assignments - 10% (+7 days 8 pm)
Programming assignments - 10% (+14 days 8 pm)
Programming assignments have to be done in Python.
The skeleton structure of the code will be posted on Google Colab
Class tests - 30%
Surprise quizzes on completed topics
K Sri Rama Murty (IITH) PRML 10 / 12
Evaluation Criteria
Lecture summaries - 10% (Next day 8 pm)
A handwritten summary should be submitted after every lecture
Complete all partial derivations
Homework assignments - 10% (+7 days 8 pm)
Programming assignments - 10% (+14 days 8 pm)
Programming assignments have to be done in Python.
The skeleton structure of the code will be posted on Google Colab
Class tests - 30%
Surprise quizzes on completed topics
Endsem exam - 40%
K Sri Rama Murty (IITH) PRML 10 / 12
Evaluation Criteria
Lecture summaries - 10% (Next day 8 pm)
A handwritten summary should be submitted after every lecture
Complete all partial derivations
Homework assignments - 10% (+7 days 8 pm)
Programming assignments - 10% (+14 days 8 pm)
Programming assignments have to be done in Python.
The skeleton structure of the code will be posted on Google Colab
Class tests - 30%
Surprise quizzes on completed topics
Endsem exam - 40%
Attendance is not mandatory
However, a minimum of 80% attendance ensures a ”D” grade
K Sri Rama Murty (IITH) PRML 10 / 12
What is not covered?
Feature extraction is a critical step in building the ML models
Physical features refer to the attributes of the pattern
Eg. Kid vs adult classification: height, weight, hair colour, ...
In case of speech, articulatory positions, shapes, vocal-fold vibration,...
K Sri Rama Murty (IITH) PRML 11 / 12
What is not covered?
Feature extraction is a critical step in building the ML models
Physical features refer to the attributes of the pattern
Eg. Kid vs adult classification: height, weight, hair colour, ...
In case of speech, articulatory positions, shapes, vocal-fold vibration,...
Difficult to come up with an exhaustive list of features for a task
Moreover, data may not directly reveal these features
Given an image, identify kids and adults
Given a speech signal, recognize the sequence of words
K Sri Rama Murty (IITH) PRML 11 / 12
What is not covered?
Feature extraction is a critical step in building the ML models
Physical features refer to the attributes of the pattern
Eg. Kid vs adult classification: height, weight, hair colour, ...
In case of speech, articulatory positions, shapes, vocal-fold vibration,...
Difficult to come up with an exhaustive list of features for a task
Moreover, data may not directly reveal these features
Given an image, identify kids and adults
Given a speech signal, recognize the sequence of words
Representation captures abstractions of the physical features
Kid vs adult: height & width of bounding boxes along with depth
In the case of speech: pitch, vocal-tract resonances
K Sri Rama Murty (IITH) PRML 11 / 12
What is not covered?
Feature extraction is a critical step in building the ML models
Physical features refer to the attributes of the pattern
Eg. Kid vs adult classification: height, weight, hair colour, ...
In case of speech, articulatory positions, shapes, vocal-fold vibration,...
Difficult to come up with an exhaustive list of features for a task
Moreover, data may not directly reveal these features
Given an image, identify kids and adults
Given a speech signal, recognize the sequence of words
Representation captures abstractions of the physical features
Kid vs adult: height & width of bounding boxes along with depth
In the case of speech: pitch, vocal-tract resonances
Requires domain knowledge to extract such representations
K Sri Rama Murty (IITH) PRML 11 / 12
What is not covered?
Feature extraction is a critical step in building the ML models
Physical features refer to the attributes of the pattern
Eg. Kid vs adult classification: height, weight, hair colour, ...
In case of speech, articulatory positions, shapes, vocal-fold vibration,...
Difficult to come up with an exhaustive list of features for a task
Moreover, data may not directly reveal these features
Given an image, identify kids and adults
Given a speech signal, recognize the sequence of words
Representation captures abstractions of the physical features
Kid vs adult: height & width of bounding boxes along with depth
In the case of speech: pitch, vocal-tract resonances
Requires domain knowledge to extract such representations
Learn the representation from the data
K Sri Rama Murty (IITH) PRML 11 / 12
What is not covered?
Feature extraction is a critical step in building the ML models
Physical features refer to the attributes of the pattern
Eg. Kid vs adult classification: height, weight, hair colour, ...
In case of speech, articulatory positions, shapes, vocal-fold vibration,...
Difficult to come up with an exhaustive list of features for a task
Moreover, data may not directly reveal these features
Given an image, identify kids and adults
Given a speech signal, recognize the sequence of words
Representation captures abstractions of the physical features
Kid vs adult: height & width of bounding boxes along with depth
In the case of speech: pitch, vocal-tract resonances
Requires domain knowledge to extract such representations
Learn the representation from the data
Represent to Learn vs Learn to Represent
K Sri Rama Murty (IITH) PRML 11 / 12
Thank You !
K Sri Rama Murty (IITH) PRML 12 / 12