24AM442-MACHINE LEARNING Unit: 1

DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND

DATA SCIENCE

Study Material for Academic Year 2025-26 (Even Semester)

COURSE NAME : MACHINE LEARNING

COURSE CODE : 24AM442
SEMESTER : IV

1
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

UNIT 2

SUPERVISED LEARNING
Linear Regression Models: Least squares, single & multiple
variables, Bayesian linear regression, gradient descent, Linear
Classification Models: Discriminant function – Perceptron
algorithm, Probabilistic discriminative model - Logistic regression,
Probabilistic generative model – Naive Bayes, Maximum margin
classifier – Support vector machine, Decision Tree(ID3), Random
Forests.

2
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Linear Regression Models:

Types of ML:

• Linear regression is a type of supervised machine-learning

algorithm that learns from the labelled datasets and maps the
data points with most optimized linear functions which can be
used for prediction on new datasets.
• The Linear regression algorithm assumes that there is a linear
relationship between the input and output, meaning the output
changes at a constant rate as the input changes.
• This relationship is represented by a straight line.
For example, take a model predicting house price based on size. Here
• Independent variable (input): House size because it's the factor
we control or observe.
• Dependent variable (output): House prize because it depends on
size of the house.
We use the independent variable to predict the dependent variable.

3
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Best Fit Line in Linear Regression:

• In linear regression, the best-fit line is the straight line that most
accurately represents the relationship between the independent
variable (input) and the dependent variable (output).
• It is the line that minimizes the difference between the actual
data points and the predicted values from the model.

i. Goal of the Best-Fit Line

The goal of linear regression is to find a straight line that
minimizes the error (the difference) between the observed data
points and the predicted values. This line helps us predict the
dependent variable for new, unseen data.

Here Y is called a dependent or target variable and X is called an

independent variable also known as the predictor of Y.
• θ1 represents the intercept, which is the value of Y when X = 0
• θ2 represents the slope, which shows how much Y changes for a
unit change in X

4
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

There are many types of functions or modules that can be used for
regression. A linear function is the simplest type of function. Here,
X may be a single feature or multiple features representing the
problem.

ii. Equation of the Best-Fit Line:

For simple linear regression (with one independent variable),
the best-fit line is represented by the equation y=mx+b
Where:
• y is the predicted value (dependent variable)
• x is the input (independent variable)
• m is the slope of the line (how much y changes when x changes)
• b is the intercept (the value of y when x = 0)

The best-fit line will be the one that optimizes the values of m
(slope) and b (intercept); so that the predicted y values are as close
as possible to the actual data points.

Least squares:
To find the best-fit line, we use a method called Least
Squares. The idea behind this method is to minimize the sum of
squared differences between the actual values (data points) and the
predicted values from the line. These differences are called
residuals.
The formula for residuals is:

5
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Where:
• yᵢ is the actual observed value
• y^ᵢ is the predicted value from the line for that xᵢ
The least squares method minimizes the sum of the squared
residuals:

Least Squares Method is used to derive a generalized

linear equation between two variables. When the value of the
dependent and independent variables they are represented as x and
y coordinates in a 2D Cartesian coordinate system. Initially, known
values are marked on a plot. The plot obtained at this point is called
a scatter plot.
Then, we try to represent all the marked points as a straight line
or a linear equation. The equation of such a line is obtained with the
help of the Least Square method. This is done to get the value of the
dependent variable for an independent variable for which the value
was initially unknown. This helps us to make predictions for the
value of a dependent variable.

6
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Least Square Method Graph:

Let us have a look at how the data points and the line of best fit
obtained from the Least Square method look when plotted on a
graph.

• The red points in the above plot represent the data points for
the sample data available.
• Independent variables are plotted as x-coordinates, and
dependent ones are plotted as y-coordinates.
• The equation of the line of best fit obtained from the Least
Square method is plotted as the red line in the graph.
We can conclude from the above graph how the Least Square
method helps us to find a line that best fits the given data points and
hence can be used to make further predictions about the value of the
dependent variable where it is not known initially.
How Do You Calculate Least Square?
To calculate the least squares solution, you typically need to:
i. Determine the equation of the line you believe best fits the data.
ii. Calculate the residuals (differences) between the observed

7
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

values and the values predicted by your model.

iii. Square each of these residuals and sum them up.
iv. Adjust the model to minimize this sum.

Real-Life Application of the Least Squares Method:

• Predicting housing prices using linear regression.
• Estimating trends in the financial market.

Least Square Method Solved Examples:

Problem 1: Find the line of best fit for the following data points using
the Least Square method: (x,y) = (1,3), (2,4), (4,8), (6,10), (8,15).
Solution:
Here, we have x as the independent variable and y as the dependent
variable. First, we calculate the means of x and y values denoted by
X and Y respectively.
X = (1+2+4+6+8)/5 = 4.2
Y = (3+4+8+10+15)/5 = 8

The slope of the line of best fit can be calculated from the formula

8
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

as follows:
m = (Σ (X - xi)*(Y - yi)) /Σ(X - xi)2
m = 55/32.8 = 1.68 (rounded upto 2 decimal places)
Now, the intercept will be calculated from the formula as follows:
c = Y - mX
c = 8 - 1.68*4.2 = 0.94
Thus, the equation of the line of best fit becomes, y = 1.68x + 0.94.

Problem 2: Find the line of best fit for the following data of
heights and weights of students of a school using the Least
Squares method:
• Height (in centimeters): [160, 162, 164, 166, 168]
• Weight (in kilograms): [52, 55, 57, 60, 61]
Solution:
Here, we denote Height as x (independent variable) and Weight as
y (dependent variable). Now, we calculate the means of x and y
values denoted by X and Y respectively.
X = (160 + 162 + 164 + 166 + 168 ) / 5 = 164
Y = (52 + 55 + 57 + 60 + 61) / 5 = 57

9
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Now, the slope of the line of best fit can be calculated from the
formula as follows:
m = (Σ (X - xi)✕(Y - yi)) / Σ(X - xi)2
m = 46/40 = 1.15
Now, the intercept will be calculated from the formula as follows:
c = Y - mX
c = 57 - 1.15*164 = -131.6
Thus, the equation of the line of best fit is, y = 1.15x - 131.6

Practice Questions on Least Square Method

Question 1: Find the equation of the best-fit line for the data:

x 1 2 3 4

y 2 3 5 7

Question 2: Fit a straight line using least squares for:

x 10 20 30 40

y 25 28 32 35

Question 3: For the data:

x -2 -1 0 1 2

y 4 1 0 1 4

(a) Find the least squares line.

10
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

(b) Comment on how well it fits the data.

Question 4: The following data represents the year and

population (in thousands):

x (year) 1 2 3 4 5

y (population in thousands) 50 54 57 60 65

Single & multiple variables:

Bayesian linear regression:
Real-world example: House price prediction
• Few sales data → high uncertainty
• Bayesian model says:
o “Price ≈ ₹50L ± ₹10L”
• As more data comes in:
o Uncertainty shrinks automatically
This is why Bayesian methods are gold in healthcare & economics.

When to Use Bayesian Regression?

• Small sample sizes: When data is scarce, Bayesian inference can
improve predictions.
• Strong prior knowledge: When domain expertise is available,
incorporating priors enhances model reliability.
• Handling uncertainty: If quantifying uncertainty in predictions
is essential.

11
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Why Bayesian regression:

Traditional regression models are typically assume no uncertainty
in data samples. Bayesian regression was introduced to model the
relationship between predictor and target variables, introducing
uncertainty into the modeling process.

Bayesian Regression: A Simple Example

Let’s show a very simple example to understand how Bayesian
regression works.
Assume we would like to estimate the price of a house based only
on one attribute: its square footage. The input attribute is denoted
by 𝑥1. A classical regression model for this problem would look
something like this:
ˆ𝑦 =150 ⋅𝑥1 + 50000
Where, the slope (m) is Rs.150 (price per [Link] and) and interceptor
(c) is Rs.50,000 (registration charge)
In Bayesian regression, the two model weights, 𝛽0 and 𝛽1, do not
take exact values like 50,000 and 150, respectively, but are instead
learned as probability distributions. For example:
• The bias term (also called intercept) 𝛽0 follows a normal
distribution (bell curve or Gaussian distribution), that is:
𝛽0 ∼𝑁(50000,50002)
****** What this actually means is

12
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

• Mean (μ) = 50,000

• Variance (σ²) = 50002=25,000,000
• Standard deviation (σ) = 5,000
This is a very wide normal distribution.
In normal distribution, the Empirical Rule (68-95-99.7),
stating approximately 68% of data falls within 1σ, 95% within
2σ, and 99.7% within 3σ of the mean.

• 68% of data values falls between: 45,000 to 55,000

• 95% of data values falls between: 40,000 to 60,000
• 99.7% of data values falls between: 35,000 to 65,000
*****
• Meanwhile, the slope or weight accompanying the single input
feature, 𝛽1, also becomes a normal distribution, namely:
𝛽1 ∼𝑁(150,202)
Recall that the two arguments defining a normal distribution are
its mean and variance. What it means (clearly)

13
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

• Mean (μ) = 150

• Variance (σ²) = 20² = 400
• Standard deviation (σ) = 20
So β1 is centered at 150, with a moderate spread of ±20.
68% of Β1 values lie between 130 and 170
95% of Β1 values lie between 110 and 190
99.7% of Β1 values lie between 90 and 210

How is the house price predicted then? Instead of returning a

single price prediction, the model applies inference by sampling from
these two distributions associated with weights, thereby producing
a range of possible predictions. For example, for a 100-square-
meter house, this is what two samples might look like:
1. Sample 1: 𝛽0 =45000, 𝛽1 =130, with
price ˆ𝑦 =45000 +130 ⋅100 =58,000 USD
2. Sample 2: 𝛽0 =55000, 𝛽1 =170, with
price ˆ𝑦 =55000 +170 ⋅100 =72,000 USD

By sampling a large number of times, we end up obtaining a

distribution of predicted prices, in which certain ranges of prices will
appear more frequently than others and, therefore, have a higher
probability than others. Based on this, we could end up formulating
predictions as confidence intervals, like:
𝑥1 𝛽 0 (C) 𝛽1 (m) ˆ𝑦
Buying [Link] Document Per [Link] price Total price
charge

14
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

100 65000 210

100 60000 190
100 55000 170 72000
100 50000 150 65000
100 45000 130 58000
100 40000 110
100 35000 90

If we consider σ value as Rs.5000, then 99.7% of data values falls

between 35,000 to 65,000. Then the ˆ𝑦 value will be calculated as
follows.
𝑥1 𝛽 0 (C) 𝛽1 (m) ˆ𝑦
Buying [Link] Document Per [Link] price Total price
charge
100 65000 150 80000
100 60000 150 75000
100 55000 150 70000
100 50000 150 65000
100 45000 150 60000
100 40000 150 55000
100 35000 150 50000

Gradient descent algorithm:

Example of Gradient Descent Algorithm: Let’s say you are
playing a game in which the players are at the top of a mountain and

15
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

asked to reach the lowest point

of the mountain. Additionally,
they are blindfolded. So, what
approach do you think would
make you reach the lake?
The best way is to
observe the ground and find
where the land descends. From that position, step in the descending
direction and iterate this process until we reach the lowest point.
Finding the lowest point in a hilly landscape.

Gradient means ‘slope’. Gradient descent is an iterative

optimization algorithm for finding the local minimum of a function.
To find the local minimum of
a function using gradient
descent, we must take steps
proportional to the negative of
the gradient (move away from
the gradient) of the function at
the current point. If we take
steps proportional to the
positive of the gradient
(moving towards the
gradient), we will approach a local maximum of the function, and the
procedure is called Gradient Ascent.
By moving step-by-step in the direction of the steepest

16
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

decrease in the loss function, it helps machine learning models learn

the best possible weights for better predictions.

What is a Cost Function?

It is a function that measures the performance of a model for
any given data. Cost Function quantifies the error between predicted
values and expected values and presents it in the form of a single real
number.
After making a hypothesis with initial parameters, we calculate
the Cost function. And with a goal to reduce the cost function, we
modify the parameters by using the Gradient descent algorithm
The goal of the gradient descent algorithm is to minimize the
given function (say, cost function). To achieve this goal, it performs
two steps iteratively:
1. Compute the gradient (slope), the first-order derivative of the
function at that point
2. Make a step (move) in the direction opposite to the gradient.
The opposite direction of the slope increases from the current
point by alpha times the gradient at that point
****Refer additional handwritten notes for Gradient Descent****

Linear Classification Models:

Classification is a supervised machine learning technique used to
predict labels or categories based on input data. The goal is to assign
each data point to a predefined class, such as spam vs. non-spam
emails or diseased vs. healthy patients.

17
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Types of Classification
i) Binary Classification
This is the simplest kind of classification. In binary
classification, the goal is to sort the data into two distinct
categories. Think of it like a simple choice between two
options. Imagine a system that sorts emails into either spam or
not spam. It works by looking at different features of the email
like certain keywords or sender details and decides whether
it’s spam or not. It only chooses between these two options.
ii) Multiclass Classification
Here, instead of just two categories, the data needs to be
sorted into more than two categories. The model picks the one
that best matches the input. Each data point belongs to only
one class or category. Think of an image recognition system
that sorts pictures of animals into categories like cat, dog and
bird.

iii) Multi-Label Classification

In multi-label classification single piece of data can belong to

18
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

multiple categories at once. Unlike multiclass classification

where each data point belongs to only one class, multi-label
classification allows datapoints to belong to multiple classes.
A movie recommendation system could tag a movie as both
action and comedy. The system checks various features (like
movie plot, actors or genre tags) and assigns multiple labels
to a single piece of data, rather than just one.

Linear Classification refers to categorizing a set of data points into

a seperate class based on a linear
combination of its explanatory
variables. The Data points can be
perfectly separable/ divisible by
using boundary line. Non-Linear
Classification refers to categorizing
those instances that are not linearly
separable. A linear classifier classifies large datasets quickly and
accurately.

Example for linear classifier: X1 and X2 are the internal mark and
external mark of students
X1 X2 Y=X1+X2-50 Class / Label
30 40 20 (+ve) Pass
25 35 10 (+ve) Pass
10 15 -25 (-ve) Fail
12 18 -20 (-ve) Fail

19
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Equation is Y=X1+X2-50 (The pass mark boundary is 50)

For a decision boundary, we always set y = 0
x1+x2−50=0 ; This is the line you draw.
According to the above equation if X1+X2=50 then X1=50-X2
S0, if X2=0, then X1 is 50
If X2=50, then X1 is 0
That’s it. Two points define a straight line.
• X-axis → x1
• Y-axis → x2
• Plot (0, 50) and (50, 0)
• Join them → decision boundary
• Line: x1+x2=50
• Region above line: x1+x2>50 → Class 1 (Pass)
• Region below line: x1+x2<50→ Class 0 (Fail)

This is linear classification in 2D.

• Some of the classifiers that use linear functions to separate

classes are

20
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

a) Linear Discriminant Classifiers,

b) Logistic Regression,
c) Perceptron, and
d) SVM (linear kernel).
• Some of the Non-linear Classifiers are as follows:
a) K-Nearest Neighbours
b) Random Forests,
c) Naive Bayes
d) Decision Tree Classification
e) Ensemble learning classifiers:
f) Kernel SVM
g) AdaBoost,
h) Bagging Classifier,
i) Voting Classifier,
j) Extra Trees Classifier
k) Multi-layer Artificial Neural Networks

Discriminant function:
Linear Discriminant Analysis (LDA) also known as Normal
Discriminant Analysis or Discriminant Function Analysis is
supervised classification problem that helps separate two or more
classes by converting higher-dimensional data space into a lower-
dimensional space. (2D into 1D)
For example, the below image shows 2 different classes with set of
black and green data points that are not linearly separable. It means,
there is no straight line that can separate 2 classes of data points

21
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

completely. The aim to classify 2 classes by LDA.

For converting higher dimension into lower dimension, LDA use

following 2 criteria to find a new axis (red dashed line).
i. Maximize the distance between means of 2 class
ii. Minimize the variation within each class variance.

The perpendicular distance from the decision boundary to the data

points shows how LDA reduces within-class variation and increases
class separability. The data points are then projected onto the new
axis, as shown in the figure below.

22
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

This shows how LDA creates a new axis to project the data and
separate two classes along a linear path. However, when class
distributions share the same mean, LDA cannot find a separating axis
and non-linear discriminant analysis is needed.

Steps to convert higher dimension into lower dimension:

Step 1: Compute the mean of both classes by

W is the class name

X is the data points of the class w
N1 is no. of data points of the class w
Step 2: Derive the covariance matrix (S1) of each class variable by

Step 3: Compute the ‘within class scatter matrix’ (Sw) by (S1+S2)

Step 4: Compute the ‘between class scatter matrix’ (S B)

Step 5: Compute Eigen values and Eigen vectors from the ‘within
classes (Sw)’ and ‘between class scatter matrix (SB)’

Eigen values can be calculated by

23
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Step 6: Once the Eigen values are calculated, next is to calculate

Eigen vectors. For that sort the Eigen values and select top ‘k’
values.
Step 7: Find the Eigen vectors (w1 and w2) corresponds to the top
‘k’ eigen vector by

Step 8: Obtain the LDA by taking the dot product of Eigen vector
and original data

Solved Example for LDA:

24
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

25
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Here, ‘T’ is Transpose (Changing row into coloumn and viceversa).

26
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

27
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

28
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

W* can be calculated directly as below,

Thus, 2D is reduced into 1D by LDA method.

29
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Thus 2 classes are completely separated using a line.

Perceptron algorithm:
Perceptron is a linear supervised machine learning algorithm.
It is used for binary classification. These algorithms analyze and
process the data to allow machines to recognize patterns, make
decisions, and interact effectively, mimicking human sensory
perception.
Key applications: Autonomous vehicles, facial recognition, and
augmented reality etc.,

Steps of Perceptron Learning Algorithm:

There are four significant steps in a perceptron learning algorithm:
i. First, multiply all input values with corresponding weight values
(slope) and then add them to determine the weighted sum.
Mathematically, we can calculate the weighted sum as follows:

30
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
 24AM442-MACHINE LEARNING Unit: 1

∑wi∗xi = x1∗w1+x2∗w2+…+wn∗xn.
Add another essential term called bias 'b' (interceptor) to the
weighted sum to improve the model performance.
∑wi∗xi+b.
ii. Next, an activation function is applied to this weighed sum,
producing a binary or a continuous-valued output.
Y=f(∑wi∗xi+b)
iii. Next, the difference between this output and the actual target value
is computed to get the error term, E, generally in terms of mean
squared error. The steps up to this form the forward propagation
part of the algorithm.
E=(Y−Yactual)2
iv. Then, optimize this error (loss function) using an optimization
algorithm. Generally, some form of gradient descent algorithm is
used to find the optimal values of the hyperparameters
like learning rate, weight, Bias, etc. This step forms the backward
propagation part of the algorithm.
An overview of this algorithm is illustrated in the following Figure:

31
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Perceptron Example:
Imagine a perceptron (in your brain).
The perceptron tries to decide if you should go to a music concert.
Is the artist good? Is the weather good?
What weights should these facts have?

Criteria Input Weight

Artists is Good x1 = 0 or 1 w1 = 0.7

Weather is Good x2 = 0 or 1 w2 = 0.6

Friend will Come x3 = 0 or 1 w3 = 0.5

Food is Served x4 = 0 or 1 w4 = 0.3

The venue is near x5 = 0 or 1 w5 = 0.4

The Perceptron Algorithm:

i. Set a threshold value
ii. Multiply all inputs with its weights
iii. Sum all the results
iv. Activate the output
i. Set a threshold value:
• Threshold = 1.5
ii. Multiply all inputs with its weights:
• x1 * w1 = 1 * 0.7 = 0.7
• x2 * w2 = 0 * 0.6 = 0
• x3 * w3 = 1 * 0.5 = 0.5
• x4 * w4 = 0 * 0.3 = 0
• x5 * w5 = 1 * 0.4 = 0.4

32
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

iii. Sum all the results:

• 0.7 + 0 + 0.5 + 0 + 0.4 = 1.6 (The Weighted Sum)
iv. Activate the Output:
• Return true if the sum > 1.5 ("Yes I will go to the Concert")

Note:
• If the weather weight is 0.6 for you, it might be different for
someone else. A higher weight means that the weather is more
important to them.
• If the threshold value is 1.5 for you, it might be different for
someone else. A lower threshold means they are more
wanting to go to any concert.

Probabilistic discriminative model and Probabilistic generative

model:
Probabilistic models:
• The probabilistic Model is a fundamental and popular
algorithm used for machine learning.
• They are used to represent the relationship between variables,
and they help us make predictions about future data.
• Probabilistic models aim is to learn patterns from data and
make predictions on new, unseen data.
• They capture the inherent uncertainty in data and incorporate it
into their predictions. Uncertainty in ML is about how much you
should trust the data or the model’s prediction. If you ignore it,
models look confident… and then fail hard in the real world.

33
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Uncertainty of data actually mean, It’s the unknowns + noise +

ambiguity baked into your dataset that make learning
imperfect. Even with a great model, messy data = shaky
predictions.
• This model algorithm uses simple rules like taking action if its
expected value is positive or taking action if its expected value
exceeds some threshold.
• Probability model is used when we want to predict a new
variable value based on previous variables or events. For
example, in machine learning, we can use a Bayesian inference
algorithm to find the best possible value for our prediction
based on past data.
• Applications of Probabilistic models are
o image and speech recognition,
o natural language processing,
o recommendation systems.

Categories of Probabilistic Models:

These models can be classified into the following categories:
• Generative models
• Discriminative models.
• Graphical models

Generative models:
Generative models aim to model the joint distribution of the
input and output variables. These models generate new data

34
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

based on the probability distribution of the original dataset.

Generative models are powerful because they can generate new
data that resembles the training data. Generative models have
been used successfully to create artworks, generate poetry and
music, simulate the evolution of biological species, and even
create artificial life.

Discriminative models:
Discriminative models are used for classification, where the
output is a binary (either true or false) prediction. The
discriminative model aims to model the conditional distribution
of the output variable given the input variable. They learn a
decision boundary that separates the different classes of the
output variable. Discriminative models are useful when the focus
is on making accurate predictions rather than generating new
data. They can be used for tasks such as image recognition,
speech recognition, and sentiment analysis. Discriminative
models have been applied successfully in many machine learning
applications, such as spam filtering, document classification, and
voice recognition. They have also been used successfully to
predict the outcomes of events such as earthquakes and natural
disasters.

Graphical models:
These models use graphical representations to show the
conditional dependence between variables. They are commonly

35
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

used for tasks such as image recognition, natural language

processing, and causal inference.
Naive Bayes:
Naïve synonym is innocent/simple. It is named as "Naive"
because it assumes the presence of one feature does not affect other
features. The "Bayes" part of the name refers to its basis in Bayes’
Theorem. Naive Bayes is a machine learning classification algorithm
that predicts the category of a data point using probability. It assumes
that all features are independent of each other. Naive Bayes performs
well in many real-world applications such as spam filtering,
document categorisation and sentiment analysis.
Consider a fictional dataset that describes the weather
conditions for playing a game of golf. Given the weather
conditions, each tuple classifies the conditions as fit(“Yes”) or
unfit(“No”) for playing golf. Here is a tabular representation of our
dataset.

36
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

The dataset is divided into two parts i.e feature matrix and
the response vector.
• Feature matrix contains all the vectors(rows) of dataset in which
each vector consists of the value of dependent features. In above
dataset, features are ‘Outlook’, ‘Temperature’, ‘Humidity’ and
‘Windy’.
• Response vector contains the value of class variable (prediction
or output) for each row of feature matrix. In above dataset, the
class variable name is ‘Play golf’.

37
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

38
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

4. Example: Weather Dataset

Let’s take a dataset used for predicting whether golf is played
based on:
• Outlook: Sunny, Rainy, Overcast
• Temperature: Hot, Mild, Cool
• Humidity: High, Normal
• Wind: True, False

Yes No P(yes) P(no)

Sunny 2 3 2/9 3/5
Overcast 4 0 4/9 0/5

39
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Rainy 3 2 3/9 2/5

Total 9 5 100% 100%

40
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

41
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Logistic regression:

42
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Logistic Regression is a supervised machine learning algorithm

used for classification problems. Unlike linear regression which
predicts continuous values it predicts the probability that an input
belongs to a specific class. For example, classifying animals into
categories like "cat," "dog" or "sheep."
• It is used for binary classification where the output can be one
of two possible categories such as Yes/No, True/False or 0/1.
• It uses sigmoid function to convert inputs into a probability
value between 0 and 1.

Understanding Sigmoid Function:

a) The sigmoid function is a important part of logistic regression
which is used to convert the raw output of the model into a
probability value between 0 and 1.
b) This function takes any real number and maps it into the range 0
to 1 forming an "S" shaped curve called the sigmoid curve or
logistic curve. Because probabilities must lie between 0 and 1, the
sigmoid function is perfect for this purpose.
c) In logistic regression, we use a threshold value usually 0.5 to
decide the class label.
• If the sigmoid output is same or above the threshold, the input
is classified as Class 1.
• If it is below the threshold, the input is classified as Class 0.
This approach helps to transform continuous input values into
meaningful class predictions.

43
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
 24AM442-MACHINE LEARNING Unit: 1

Types of Logistic Regression:

Logistic regression can be classified into three main types based on
the nature of the dependent variable:
i. Binomial Logistic Regression: This type is used when the
dependent variable has only two possible categories. Examples
include Yes/No, Pass/Fail or 0/1. It is the most common form of
logistic regression and is used for binary classification problems.
ii. Multinomial Logistic Regression: This is used when the
dependent variable has three or more possible categories that are
not ordered. For example, classifying animals into categories
like "cat," "dog" or "sheep." It extends the binary logistic
regression to handle multiple classes.
iii. Ordinal Logistic Regression: This type applies when the
dependent variable has three or more categories with a natural
order or ranking. Examples include ratings like "low,"
"medium" and "high." It takes the order of the categories into
account when modeling.

How does Logistic Regression work?

Logistic regression model transforms the linear regression function
continuous value output into categorical value output using a
sigmoid function which maps any real-valued set of independent
variables input into a value between 0 and 1. This function is
known as the logistic function.

44
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Simply this can be represented as the dot product of weight and bias.
z=mX+b
Logistic regression then applies the sigmoid function to z to convert
it into a probability between 0 and 1 which can be used to predict
the class.
Now we use the sigmoid function where the input will be z and we
find the probability between 0 and 1. i.e. predicted y.

45
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE
24AM442-MACHINE LEARNING Unit: 1

Maximum margin classifier:

Support vector machine:
Decision Tree(ID3):
Random Forests:

*******

46
Prof. Dr. K. SARAVANAN, Dept. of AI&DS, KCE