Machine Learning – CS643

Dr. Sheikh Faisal Rashid

• Assistant Professor: Computer
Science, UET Lahore
• Director: Artificial Intelligence
Research Lab (AIRL), KICS
• Vice President: Pakistan Pattern
Recognition Society (PPRS)
• Visiting Researcher DFKI, Germany
 shfaisal@[Link]

Department of Computer Science and Engineering UET, Lahore

Overview
 Pattern Classification
 Introduction

 Concepts

 Design Cycle
Pattern Classification
●More than a program
o Usually, we think a program is something written by an
experienced person.
Pattern Classification
●More than a program
o Usually, we think a program is something written by an
experienced person.
o Often, the program isn’t complete without “experience” of its
own.
Pattern Classification
●More than a program
o Usually, we think a program is something written by an experienced
person.
o Often, the program isn’t complete without “experience” of its own.
o The idea of writing programs that use data (experience) to create
better programs than people can write directly.
Pattern Classification
●Pattern classification systems make decisions
●Decisions are usually made autonomously
●Decisions are not pre-programmed
●Decision “rules” are derived from data
SE, AI, PR
● Software Engineering
o Manual creation of specifications, manual implementation, full control over details
of execution.
● Artificial Intelligence (Rule-Based Systems)
o Manual creation of specifications, specifications are directly executable (rule
interpreters, etc.). Details of execution are automated.
● Pattern Recognition, Machine Learning
o Programmer chooses category of application, but detailed specifications are
automatically derived from data. Execution is automated.
Fish Classification: An example
●“Sorting incoming Fish
on a conveyor according
to species using optical
sensing”
●Species
o Sea bass
o Salmon
Problem Analysis
● set up a camera and take some sample images to extract features
● feature types
o length (positive real number)
o lightness (positive real number)
o width (positive real number)
o number of fins (non-negative integer)
o shape of fins (one-of a set of possible categories)
o position of the mouth (one of a set of possible categories)
o ...
● base the decision of which kind of fish it is on these measurements
Overall classifier
Preprocessing
● raw camera image may be 1024 x 1024 pixels
o >1 million numbers
● contains lots of irrelevant data
o background
o dirt
o ...
● feature extraction
o data reduction—computational efficiency
o remove irrelevant variation
● feature measurements are passed to the classifier
Feature Vector
● collection of measurements like an “object” or “structure” or “database record”
● example
{ length = 21cm,
lightness = 0.73,
width = 8.3cm,
number_of_fins = 3,
shape_of_fins = {square, triangular, square},
position_of_the_mouth = {front} }
Dataset
Histogram
Decision Rule
Empirical Error Rate
Finding Good Features / Boundaries
 Decision Theory
18

 Threshold decision boundary and cost relationship

 Move our decision boundary toward smaller values of
lightness in order to minimize the cost (reduce the number
of sea bass that are classified salmon!)

Task of decision theory

 Multiple Features
19

 Adopt the lightness and add the width of the fish

Fish xT = [x1, x2]

Lightness Width
Multiple Features
Linear Decision Function
Generalization
Overfitting
Avoid overtraining
 Pattern Recognition Systems
25

Pattern Classification, Chapter 1

 Pattern Recognition Systems
26

 Sensing
 Use of a transducer (camera or microphone)
 PR system depends of the bandwidth, the resolution
sensitivity distortion of the transducer
 Segmentation and grouping
 Patterns should be well separated and should not
overlap

Pattern Classification, Chapter 1

 Pattern Recognition Systems
27

 Feature extraction
 Discriminative features
 Invariant features with respect to translation, rotation and scale.
 Classification
 Use a feature vector provided by a feature extractor to assign
the object to a category
 Post Processing
 Exploit context input dependent information other than from the
target pattern itself to improve performance
Pattern Classification, Chapter 1
 The Design Cycle
28

 Data collection
 Feature Choice
 Model Choice
 Training
 Evaluation
 Model Selection
 Computational Complexity
Pattern Classification, Chapter 1
 The Design Cycle
29

 Data Collection
 How do we know when we have collected an
adequately large and representative set of examples
for training and testing the system?
 The Design Cycle
30

 Feature Choice
 Depends on the characteristics of the problem
domain.
 Simple to extract, invariant to irrelevant
transformation insensitive to noise.
 The Design Cycle
31

 Model Choice
 Unsatisfiedwith the performance of our fish classifier
and want to jump to another class of model

Pattern Classification, Chapter 1

 The Design Cycle
32

 Training
 Use data to determine the classifier.
 Many different procedures for training classifiers and
choosing models

Pattern Classification, Chapter 1

 The Design Cycle
33

 Evaluation
 Measure the error rate (or performance and switch
from one set of features to another one

Pattern Classification, Chapter 1

The Design Cycle
 Cross-Validation

34
 The Design Cycle
35

 Computational Complexity
 What is the trade-off between computational ease
and performance?
 (How an algorithm scales as a function of the number
of features, patterns or categories?)

Pattern Classification, Chapter 1

Important terminology
 Training data
 Data samples
 Target vectors
 Learning / Training
 Machine takes training data and automatically learns mapping from data
samples to target vectors
 Test data
 Target vectors are concealed from the machine
 Machine predicts the target vectors based on previously learned model
 Accuracy can be evaluated by comparing the predicted vectors to the actual
vectors

 36
 Example
Handwritten Digit Recognition

 37
Problem Statement
 Consider a 28 x 28 pixel image
 Represented by a 784 dimensional vector x
 Goal: build a machine that takes the vector x as
input and produces the identity of digit 0,…,9 as
the output

 38
Machine Learning
 Tom Mitchel’s definition of Machine Learning:
A computer program is said to learn from
experience E with respect to some class of tasks T
and performance measure P, if its performance at
tasks in T, as measured by P, improves with
experience E.
Examples
Examples