भारतीय सू चना ौ यो गक सं थान नागपुर

Indian Institute of Information Technology Nagpur

An Institution of National Importance By An Act of Parliament

Introduction to Machine Learning

For Semester IV, B. Tech. Computer Science & Engineering

By – Dr. Amol Bhopale

1
 Course Material
• Class notes, ppts.
• Text Books
1. Machine Learning, Tom Mitchell, McGraw Hill, 1997.
2. Ethem Alpaydin, Introduction to Machine Learning, PHI,
2016

• Reference Books
1. T. Hastie, R. Tibshirani, J. Friedman. The Elements of
Statistical Learning, 2e, 2008.
2. Christopher Bishop. Pattern Recognition and Machine
Learning. 2e, 2006.
3. Richard O. Duda, Peter E. Hart, David G. Stork. Pattern
classification, Wiley, New York, 2001.
2
 What is Learning?
Learning
The ability to improve behavior based on the experience.

3
 What is Learning?
Learning
The ability to improve behavior based on the experience.

4
What is Learning?

Identify fruits in the image

5
 What is Learning?

Remember the names of the fruit

Ackee Mangosteen Rambutan Horned Finger Lime

Melon

6
What is Learning?

Identify Fruits in the image

7
 What is Learning?

How many Fruit names predicted correctly

Student 1 Student 2 Student 3

Correct 2 3 4

Incorrect 3 2 1

Performance 40% 60% 80%

8
 What is Machine Learning?

What we are expecting from Machine

Hey Machine! Sure
Can you Learn Human!
yourself? Feed me
data

9
 Machine Learning

A computer program is said to learn from experience E with respect to

some class of tasks T and performance measure P, if its performance at
tasks in T, as measured by P, improves with experience E

10
 Machine Learning to a Layman

You had a task : Identify Fruit Names

You Experienced: Remembered name and image of the fruit
Performance : How many fruit names you correctly Identified

11
 What is Machine Learning?
Machine Learning
Design of Algorithm that-
• Learn from data or build models using that data
• The learned model can be used to
• Detect patterns/structures/themes/trends etc. in the data
• Make predictions about future data and make decisions
• Modern ML algorithms are heavily “data-driven”
• No need to pre-define and hard-code all the rules (usually
infeasible/impossible anyway).
• The rules are not “static”; can adapt as the ML algorithm
ingests with more and more data.

12
Machine Learning vs Programming

13
 When to Use Machine Learning?
• Human expertise is absent
Example: navigating on mars
• Humans are unable to explain their expertise Example:
vision, speech, language
• Requirements and data change over time
Example: Tracking, Biometrics, Personalized
fingerprint recognition
• The problem or the data size is just too large
Example: Web Search
• When not to use it: If you can precisely/mathematically describe
how to solve the task. Just program it.
14
 Why Machine Learning?
• Machine Learning term first coined in 1959
• Computer Model based on Neural Network was created in
1943

15
 Why Machine Learning?
• Machine Learning term first coined in 1959
• Computer Model based on Neural Network was created in
1943

16
 Why Machine Learning?

DATA OPTIMIZED COMPUTING

ALGORITHMS POWER

17
 Why Machine Learning?

• Structured Data
• Unstructured Data
• “More than 300 million photos get uploaded per day.
• Every minute there are 510,000 comments posted
and 293,000 statuses updated”
DATA • “Over 2.5 quintillion bytes of data are created
every single day, and it's only going to grow from
there.
• By 2020, it is said that 1.7MB of data has been
created every second for every person on earth”
More than 80% data is unstructured.

18
 Why Machine Learning?

Python
Libraries: Pandas, Numpy, Sklearn, Keras
TensorFlow, PyTorch, Theano

OPTIMIZED
ALGORITHMS Less programming more science!

19
 Why Machine Learning?

Powerful CPUs
GPU
Parallel and Distributed Computing

COMPUTING
POWER

20
 Jargon Difference!

Machine Learning

Artificial Intelligence Deep Learning

Data Science

21
Jargon Difference!

22
 Types of Learning
• Supervised (inductive) learning: Training data includes desired
outputs.
• Unsupervised learning: Training data does not include desired
outputs, Find hidden/interesting structure in data.
• Semi-supervised learning: Training data includes a few desired
outputs
• Reinforcement learning: the learner interacts with the world
via “actions” and tries to find an optimal policy of behavior
with respect to “rewards” it receives from the environment

23
Types of Learning

24
A Typical Supervised Learning Workflow (for
Classification)

25
A Typical Supervised Learning Workflow (for
Classification)

26
A Typical Un-supervised Learning Workflow (for
Clustering)

27
 A Typical Un-supervised Learning Workflow (for
Clustering)

Note: Unsupervised Learning too

can have (and often has) a “test”
phase.
E.g., in this case, given a new
cat/dog image, predict which of the
two clusters it belongs to.

It can be done by assigning the

image to the cluster with closer
centroid
28
A Typical Reinforcement Learning Workflow

Agent’s goal is to learn a policy for

some task

Agent does the following repeatedly

• Senses/observes the environment
• Takes an action based on its current
policy
• Receives a reward for that action
• Updates its policy

There IS supervision, not explicit (as in

Supervised Learning) but rather implicit
(feedback based)

29
 Geometric View of Some Basic ML Problems
Regression

Supervised Learning:
Learn a line/curve (the “model”)
using training data consisting of
Input-output pairs (each output is a
real-valued number)

Use it to predict the outputs for new

“test” inputs

Classification

Supervised Learning: Learn a

linear/nonlinear separator (the
“model”) using training data
consisting of input-output pairs
Two-Class Multi-Class Two-Class (binary) Multi-Class Nonlinear
Use it to predict the labels for new (binary) Linear Nonlinear Classification
“test” inputs Linear Classification Classification
Classification
30
 Geometric View of Some Basic ML Problems

Clustering

Unsupervised Learning: Learn the

grouping structure for a given set
of unlabeled inputs

Dimensionality Reduction

Unsupervised Learning: Learn a

Low-dimensional representation for
a given set of high-dimensional
inputs

Two-dim to one-dim Three-dim to two-dim nonlinear

Note: DR also comes in supervised
linear projection projection (a.k.a. manifold
flavors (supervised DR)
learning)
31
 Machine Learning = Probability Density Estimation
Supervised Learning (“predict y given x ”) can be thought of as estimating p(y|x )

Labeled
“dog
Training ”
“dog
Data “dog
”
” Supervised ML p(class|image)
“cat”
“cat”
“cat”

Unsupervised Learning (“model x ”) can also be thought of as estimating p(x )

Unlabeled
Training p(image)
Unsupervised ML
Data

Harder for Unsupervised Learning because there is no supervision y

Other ML paradigms (e.g., Reinforcement Learning) can be thought of as learning prob. density

32
 Machine Learning = Function Approximation
Supervised Learning (“predict y given x ”) can be thought learning a function that maps x to y

Labeled “dog
Training “dog
”

”
Data “dog
Supervised : image class
”
“cat
ML
”
“cat”
“cat”

Unsupervised Learning (“model x ”) can also be thought of as learning a function that maps x to
some useful latent representation of x

Unlabeled
Training
Data latent representation
Unsupervised : image of image (e.g., cluster
ML
id or compressed
version)

Harder for Unsupervised Learning because there is no supervision y

Other ML paradigms (e.g., Reinforcement Learning) can be thought of as doing function approx.
33
 Machine Learning in the real-world
Broadly applicable in many domains (e.g., internet, robotics, healthcare and
biology, computer vision, NLP, databases, computer systems, finance, etc.)

34
Machine Learning helps Computer
Vision

35
Machine Learning helps Computer
Vision

36
Machine Learning helps NLP

37
Machine Learning helps NLP

38
Machine Learning helps NLP

39
Machine Learning helps NLP- Search and Info
Retrieval

40
Machine Learning meets Speech Processing
ML algorithms can learn to translate speech in real time

41
 Machine Learning helps Chemistry
ML algorithms can understand properties of molecules and learn to
synthesize new molecules

42
 Machine Learning helps Chemistry
ML algorithms can “read” databases of matetials and recreate the
Periodic Table within hours
“Recreated” Periodic Table

43
Machine Learning helps in Biology, E-commerce

44
Inductive Learning

45
 Inductive Learning
What is Inductive Learning Algorithm?
Inductive Learning Algorithm (ILA) is an iterative algorithm that is used for
generating a set of classification rules, which produces rules of the form “IF-THEN”,
for a set of examples, producing rules at each iteration and appending to the set of rules.
• In other words, the learner infers a general hypothesis from observed
instances.
• Also called as Deterministic Supervised Learning
• In this, first input x, (the verified value) given to a function f, and the output
is f(x).
• Then we can give different set of inputs (raw inputs) to the same function f,
and verify the output f(x).
• By using the outputs we generate (learn) the rules.

46
 Need for Inductive Learning
• It enables machines to generalize from limited examples to handle unseen
data.
• Real-world problems cannot be solved using predefined rules; inductive
learning helps discover patterns automatically.
• It supports prediction and decision-making in uncertain and dynamic
environments.
• It reduces the need for explicit programming by learning directly from data.
• It forms the foundation of most machine learning and AI applications such
as classification, regression, and pattern recognition.

47
 Importance of Inductive Learning in ML
Inductive learning forms the backbone of many machine learning algorithms,
allowing systems to adapt and improve from exposure to data, thereby enhancing
decision-making processes.

• It makes machine learning systems scalable and adaptable to new data.

• Most ML algorithms (decision trees, SVMs, neural networks) rely on inductive
inference.
• It is essential for prediction, classification, and pattern discovery in real-world
problems.

48
 Inductive Learning Process

Data Collection:
• Gather labeled examples representing the problem domain.
• Example: Emails labeled as spam or non-spam in a classification task.
Hypothesis Space:
• Define the set of possible hypotheses/models based on the chosen algorithm and
inductive bias.
Hypothesis Generation:
• Construct potential hypotheses using observed examples.
• Analyze instance features to identify patterns or relationships.
Hypothesis Evaluation:
• Assess hypotheses using evaluation metrics or validation techniques.
• Test predictive accuracy on new, unseen examples.
Hypothesis Refinement:
• Refine hypotheses iteratively based on evaluation feedback.
• Update models to enhance performance and generalization.
Generalization:
• Apply the final model to classify or predict new, unseen data instances.

49
Inductive Learning Process

50
 Inductive Learning
• Inductive learning or “Prediction”:
– Given examples of a function (X, F(X))
– Predict function F(X) for new examples X
This is the function which we are trying to learn.

• Classification
F(X) = Discrete
• Regression
F(X) = Continuous
• Probability estimation
F(X) = Probability(X):

Why it is called Inductive learning?

We are given some data and we are trying to do induction to
identify a function which can explain that data.
51
 Classification Learning
Task T
Input
• A set of instances d1, d2, ...., dn
• An instance has a set of features
• We can represent an instance as a vector
• d = <x1,x2, x3, ...., xn>

Output
• A set of predictions y1, y2, y3, ...., yc
• One of the fixed set of constant values
• Eg: {+1, -1}

Performance P - How accurately model predicts the output

Experience E - A set of labeled examples (x,y) where y is the true

label of x. 52
 Basic Terminologies
Types of features
1. Categorical - It will have finite number of categories and
classes.
Example
– Gender - Male, Female
– Age group: (0-12) children, (13-19) teenagers, (20-30)
adults, 31-60 working professionals, above 60 senior
citizen,
– Blood group: A, B, AB, O etc

2. Integer Valued
Example: Number of words in a text

3. Continuous - Are those which can take INFINITE number of

values.
Example: Age, height, weight, price etc. 53
 Basic Terminologies
• Feature: Distinct characters that can be used to describe each
object in a quantitative manner.
• Feature Vector: n-dimensional vector of numerical features
that represent some object.
• Feature Space:
– Suppose we have two features x1 and x2
– Two features will define two dimensional feature space
– In general n-features will define n-dimensional feature space.

• Instance Space X: Set of all possible objects that can be

described by features.
• Target Function: It is function we are trying to learn
• Training data set:
– Collection of examples observed by learning algorithms
– It is a used to discover potentially predictive relationship
54
 Basic Terminologies
Feature Space:
Properties that describe the
3.0
2.0
1.0
0.0 problem

0.0 1.0 2.0 3.0 4.0 5.0 6.0

55
 Basic Terminologies

Example:
<0.5,2.8,+>
+
+ + +
-
+ + - - -
+ - +
3.0

-
- + + -
- -
2.0

+ -
+ + -
1.0

-
0.0

0.0 1.0 2.0 3.0 4.0 5.0 6.0

56
 Basic Terminologies
Possible Functions
1. Slanted line with 2 parameters
y= mx +c
-> We need to define both intercept and slope.

2. Polynomial
quadratic function: ax² + bx +c
a,b,c- 3 parameters

3. Complex Function
Note:- We are interested in a function which not only fit the
training data but also works well with future or test data.

57
 Basic Terminologies
Representation of Function
• The representation of a hypothesis in machine learning
depends on the chosen features and the function class,
which together determine how input data is mapped to
output predictions.

• Features
Features are the measurable attributes or input variables used
to describe the data. They transform raw data into a suitable
form for learning.
• Function Class (Hypothesis Space)
The function class defines the set of all possible functions
(models) the learning algorithm can choose from, such as
linear functions, decision trees, or neural networks.

58
 Basic Terminologies
Representation

59
 Basic Terminologies
Representation

60
 Basic Terminologies

Hypothesis:
Function for labeling examples

Label: + + Label: -
+ ? + +
-
+ + - - -
+ ? - +
3.0

- ?
- + + -
- -
2.0

+ -
+ + - ?
1.0

-
0.0

0.0 1.0 2.0 3.0 4.0 5.0 6.0

61
 Hypothesis Space
• There could be many possible functions that explain the given training data.
• It is the set of legal hypothesis
• There could be multiple legal hypothesis or functions set of all such legal
hypothesis is called as hypothesis space.
• Eg: class 1: +ve, class 2: -ve

✔ Our objective is to come up with best hypothesis

✔ We denote the hypothesis space by H
✔ Output of learning algorithm will be h where h ∈ H

✔ One way to think about a supervised machine learning is as a device that

explores a "Hypothesis Space“.

62
 Hypothesis Space

Hypothesis Space:
Set of legal hypotheses

+
+ + +
-
+ + - - -
+ - +
3.0

-
- + + -
- -
2.0

+ -
+ + -
1.0

-
0.0

0.0 1.0 2.0 3.0 4.0 5.0 6.0

63
 Hypothesis Space
Target Function
It's a function which maps every input x to an output y, we denote it by f.

Our objective is to come up with a hypothesis h ∈ H that approximates

“f” based on the training data.

Input and output of a learning algorithm

Input - Training set, S
Output- Hypothesis, h where h ∈ H.

64
Hypothesis Space

65
 Inductive Learning In General
Inducing a general function from training examples.
Constructs a hypothesis h to agree with all the training examples
A hypothesis is consistent if it agrees (works will) with all training
examples.
A hypothesis is said to be generalized if it correctly predicts the value of
y for new examples.

Inductive learning hypothesis (Rule)

Any hypothesis used to approximate the target function well over a
sufficiently large set of training examples will approximate the target
function well over other unobserved examples-
If h works well on sufficiently large set of training example
Then it works well on observed data

66
Inductive Learning Algorithms

67
Applications of Inductive Learning

68
Challenges in Inductive Learning

69
 Supervised Learning
Given: <x, f(x)> for some unknown function f
Learn: A hypothesis H, that approximates f

Example Applications:
• Disease diagnosis
x: Properties of patient (e.g., symptoms, lab test results)
f(x): Predict disease
• Automated steering
x: Bitmap picture of road in front of car
f(x): Degrees to turn the steering wheel
• Credit risk assessment
x: Customer credit history and proposed purchase
f(x): Approve purchase or not
70
71
Learning = Representation + Evaluation
+ Optimization
• Combinations of just three elements
Representation Evaluation Optimization
Instances Accuracy Greedy search
Hyperplanes Precision/Recall Branch & bound
Decision trees Squared error Gradient descent
Sets of rules Likelihood Quasi-Newton
Neural networks Posterior prob. Linear progr.
Graphical models Margin Quadratic progr.
Etc. Etc. Etc.

72
 Inductive BIAS
As we can see that hypothesis space is very large. It is not possible to
look at every hypothesis individually to choose the best hypothesis.
So we put some restrictions on hypothesis.
If we restrict the hypothesis, it reflects a bias of the learning
algorithm

Bias Could be of two types

1. Restricted bias: Limits the hypothesis space
2. Preference bias: Impose ordering on hypothesis space.

Example of restriction bias: We may say that we are looking for a

linear function or we are looking for 3rd degree polynomial.
Example of preference bias: We may say that we are considering all
possible polynomials but we will prefer a polynomial of lower
degree.
73
 Generalization & Error
Coming up with a general function from training examples
When we do generalization some errors get introduced.
There are two components of generalization error
Bias error
Variance error

74
 Generalization & Error
Bias
This is the error introduced due to simplifying assumptions
made by a model
Simplified assumptions limit the model's capacity to learn.

Low Bias
Suggests less assumptions about the form of the target
function.

High Bias
Suggests more assumptions about the form of the target
function.

75
 Generalization & Error
Variance
Variance tells that how much a random variable is different
from its expected value.
If the machine learning model performs well with the training
dataset, but does not perform well with the test dataset, then
variance occurs.

Low variance.
Suggests small change to the estimated models with Changes
to the training dataset.

High variance.
Suggests large changes, to the estimated models with Changes
to the training dataset.

76
 Bias Variance Trade-off

Note: This is a regression problem.

Data is divided into train and test set
Not classes
Train

Test

Train Data
Test Data

77
 Bias Variance Trade-off

Complex Model
No error on training data
What about test set?

Train Data Overfit Model

Test Data

78
 Bias Variance Trade-off

Complex Model
No error on training data

What about test set?

Model performed well on train

Data but test error is high

Train Data Overfit Model

Test Data

79
 Bias Variance Trade-off

Complex Model
No error on training data

What about test set?

Model performed well on train

Data but test error is high

Train Data Overfit Model

Test Data

80
 Bias Variance Trade-off

Train error Low

High Variance
Test Error High

Error difference in train and test set is more

Hence, we don’t want an Overfitting model
Overfit Model

Overfit High Variance

Train Data
Test Data

81
 Bias Variance Trade-off

Simple Model
More error on training data
What about test set?

Train Data Underfit Model

Test Data

82
 Bias Variance Trade-off

Simple Model
More error on training data
What about test set?

Model is neither performing

Well on train data nor on Test

Train Data Underfit Model

Test Data

83
 Bias Variance Trade-off

Simple Model
More error on training data
What about test set?

Model is neither performing

Well on train data nor on Test

Train Data Underfit Model

Test Data

84
 Bias Variance Trade-off

Train error High

High Bias
Test Error High

Error difference in train and test set is Less

Hence, we don’t want an Underfit model

Underfit High Bias

Train Data
Test Data

85
 Bias Variance Trade-off

Train Data Overfit Model Underfit Model

Test Data

86
 Bias Variance Trade-off
High Low
Train Error
High Bias Low Bias
High Low
Test Error
High Variance Low Bias

Train Data
Test Data

87
 Over-fitting
• Over-fitting & under-fitting are two main errors/problems in the
machine learning model, which cause poor performance in
Machine Learning.
• Over-fitting occurs when the model fits more data than required,
and it tries to capture each and every data point fed to it.
Hence it starts capturing noise and inaccurate data from the
dataset, which degrades the performance of the model.
• An over-fitted model doesn't perform accurately with the
test/unseen dataset and can’t generalize well.
• An over-fitted model is said to have low bias and high variance.

88
 How to avoid Overfitting

• Using cross-validation
• Using Regularization techniques
• Implementing Ensemble Techniques.
• Picking a less parameterized/complex model
• Training the model with sufficient data
• Removing features
• Early stopping the training

89
 Under-fitting
• Model cannot create a mapping between the input and the target
variable
• Under-observing features leads to a higher error in the training
and unseen data samples.
• Under-fitting becomes obvious when the model is too simple and
cannot create a relationship between the input and the output.

90
 How to avoid Under-fitting

• Preprocessing the data to reduce noise in data

• More training to the model
• Increasing the number of features in the dataset
• Increasing the model complexity
• Increasing the training time of the model to get
better results.

91
92
 Over-fitting
Over-fitting during training

Model
error Error on
new data

Training error

Number of iterations

93
 Regularization and Over-fitting
Adding a regularizer:

Model
error Without regularizer
With regularizer

Number of iterations

94
 Cross-Validation
• Cross-validation involves partitioning your data into
distinct training and test subsets.

• The test set should never be used to train the model.

• The test set is then used to evaluate the model after

training.

95
 K-fold Cross-Validation
• To get more accurate estimates of performance you
can do this k times.
• Break the data into k equal-sized subsets Ai
• For each i in 1,…,k do:
– Train a model on all the other folds A1,…, Ai-1, Ai+1,…, Ak
– Test the model on Ai
• Compute the average performance of the k runs

96
5-fold Cross-Validation

97
 Occam’s Razor

• It is a principle of simplicity stating that:

Among multiple models explaining the data equally well, the simplest model should be
preferred.

Key points:
• Focuses on parsimony (simplicity)
• Avoids unnecessary assumptions or parameters
• Complex models may cause overfitting
• Simpler models generalize better on unseen data
• Widely used in science, statistics, and machine learning

98
 Occam’s Razor

How it is enforced in ML:

• Regularization (L1, L2)
• Early stopping
• Decision tree pruning
• Feature selection

99
 Learning as a search

• Learning can be viewed as the task of searching through a large

space of hypothesis implicitly defined the hypothesis
representation.
• The goal of this search is to find the hypothesis that best fits the
training examples and generalize well to unseen data.

100