Unit 5
Supervised learning for regression analysis
Is there any cause and effect
relationship between the two
variables in case of the
bivariate population or
between one variable on one
side and two or more variables
on the other side in case of
multivariate population? If yes,
of what degree and in which
direction?
Table of contents
Linear regression,
logistic regression
Linear regression
Introduction
A type of supervised machine-learning algorithm
that learns from the labelled datasets and maps the
data points with most optimized linear functions
which can be used for prediction on new datasets.
For example we want to predict a student's exam
score based on how many hours they studied. We
observe that as students study more hours, their
scores go up. In the example of predicting exam
scores based on hours studied. Here
Independent variable (input): Hours studied
because it's the factor we control or observe.
Dependent variable (output): Exam score
because it depends on how many hours were
studied.
Features: Types:
Simplicity and Interpretability There are two main types of linear regression:
Predictive Ability Simple Linear Regression: Involves a single
independent variable and a single dependent
Basis for Other Models
variable. The equation is:
Efficiency
y = β0 + β1 * X
Analysis
Multiple Linear Regression: Involves multiple
independent variables. The equation is:
y = β0 + β1 * X1 + β2 * X2 + ... + βn * Xn
For classification the output(s) is nominal
In regression the output is continuous
● Function Approximation
Many models could be used – Simplest is linear
regression
● Fit data with the best hyper-plane which "goes
through" the points
● For each point the difference between the
predicted point and the actual observation is the
residue
The goal of linear regression is to find the best-fit line
that minimizes the error between the predicted and
actual values.
● For now, assume just one (input) Choose an objective function
independent variable x, and one For simple linear regression we choose sum
(output) dependent variable y squared error (SSE)
○ Multiple linear regression assumes
an input vector x S (predictedi – actuali)2 = S (residuei)2
○ Multivariate linear regression
Thus, find the line which minimizes the sum
assumes an output vector y
of the squared residues (e.g. least squares)
● We "fit" the points with a line (i.e.
hyper-plane) This exactly mimics the case assuming data
○ Which line should we use? points were sampled from the actual
hyperplane with Gaussian noise added
Example:Calculate the two regression equations Example:Obtain regression equation of Y on X
of X on Y and Y on X from the data given below, and estimate Y when X=55 from the following
taking deviations from a actual means of X and
Y.
Estimate the likely demand when the price is Rs.20.
logistic regression
A supervised learning classification algorithm z=(∑ni=1wixi)+b
used to predict the probability of a target
variable
The nature of target or dependent variable is
dichotomous, which means there would be only
two possible classes. (The dependent variable is
binary in nature having data coded as either 1
(stands for success/yes) or 0 (stands for
failure/no).)
Types:
Logistic regression can be classified into three main types based on the nature of the dependent variable:
Binomial Logistic Regression: This type is used when the dependent variable has only two possible
categories. Examples include Yes/No, Pass/Fail or 0/1. It is the most common form of logistic regression
and is used for binary classification problems.
Multinomial Logistic Regression: This is used when the dependent variable has three or more possible
categories that are not ordered. For example, classifying animals into categories like "cat," "dog" or
"sheep." It extends the binary logistic regression to handle multiple classes.
Ordinal Logistic Regression: This type applies when the dependent variable has three or more
categories with a natural order or ranking. Examples include ratings like "low," "medium" and "high." It
takes the order of the categories into account when modeling.
Evaluation metrics Example Given a set of 9 training data with
2-dimensional inputs and their corresponding
Accuracy class labels as follows.
Precision
Recall
F1 Score
Fit a Multi-Class Logistic Regression model to
the training data using the algorithm of Gradient
Descent? Provided that the learning rate is set
to be 0.05, the number of training epoch is set to
be 1 and the initial model parameters are set as
follows.