Classification vs.

Regression
Classification vs. Regression
Classification Regression
Classification predicts categorical labels based on Regression predicts continuous values based on
known labels. known values.

Advantages: Advantages:

Easy to interpret results, providing a clear A wide variety of relationships between variables,
classification for each input not limited to linear relationships

Useful for categorical output scenarios, such as Suitable for predicting continuous values
spam detection or image recognition

Disadvantages: Disadvantages:

Data may be linearly separable, which might not A chance of outliers significantly affecting the
always be true regression line and predictions

A risk of overfitting, especially with complex Linearity in the data

models, leading to poor generalization

SUPERVISED LEARNING CLASSIFICATION

Develop predictive models

based on both input and
output data REGRESSION

Combined Models
1: Support Vector Machines (SVMs)
Classification Regression
SVMs work by finding the hyperplane that best SVMs find the hyperplane that best fits the data, aiming to
separates different classes in classification tasks, maximiz- minimize the error between predicted
ing the margin between classes. and actual values.

Maximum
X2 Margin

Positive
Hyperplane

Maximum
Margin
Hyperplane

Negative Hyperplane Support

Vectors

X1

Data may be linearly separable, which might not A chance of outliers significantly affecting the
always be true regression line and predictions

A risk of overfitting, especially with complex Linearity in the data

models, leading to poor generalization
Classification vs. Regression
2: K-Nearest Neighbors (KNN)
Classification Regression
SVMs work by finding the hyperplane that best KNN calculates the average of the numerical
separates different classes in classification tasks, values of its k nearest neighbors to predict the
maximizing the margin between classes. value for a new data point.

K Nearest Neighbors

Class A
Class B
Pt

Class C

3: Decision Tree
Classification Regression
A decision tree divides the input space into A decision tree predicts the output value of a new
regions, each corresponding to a different class. It data point by averaging the target values of the
recursively splits the data based on the features to training samples in the leaf node to which the data
create distinct groups. point belongs.

Decision Node Root Node

Subtree Decision Node Decision Node

Lead Node Lead Node Lead Node Decision Node

Lead Node Lead Node

Classification vs. Regression
4: Random Forest
Classification Regression
A random forest is an ensemble of decision trees Each tree predicts a continuous value. The final
for classification, where each tree is trained on a prediction is the average of all tree predictions.
random subset of the data. Each tree “votes” for
the class, and the class with the most votes is the
final prediction.

Training Training Training

Data Data Data
1 2 n

Training
Set
Decision Decision Decision
Tree Tree Tree
1 2 n

Voting
Test Set
(averaging)

Prediction

5: Gradient Boosting
Classification Regression
Gradient boosting works by sequentially Gradient boosting in regression is similar to how
combining multiple weak classifiers (often decision it works in classification but predicts a continuous
trees), where each new classifier corrects the value. It sequentially fits new models to the
errors of its predecessor. A weighted sum makes residual errors made by the previous models,
the final predictions from all the classifiers. refining the prediction with each iteration until a
satisfactory model is achieved.

Data Data Data

Decision Decision Decision

Tree 1 Tree 2 Tree n

Prediction Prediction Prediction

Classification vs. Regression
6: Lasso Regression
Classification Regression
Lasso regression penalizes the absolute size of Lasso regression minimizes the sum of the
coefficients, shrinking some to zero. It helps in squared differences between the observed and
feature selection by excluding irrelevant features, predicted values, with an additional penalty for the
simplifying the model, and improving classification absolute size of coefficients. This encourages a
accuracy. simpler model by shrinking less influential
features to zero, aiding in feature selection and
reducing overfitting.

Training data points

Sales
Test data points

Advertisements

7: Ridge Regression
Classification Regression
Ridge regression penalizes the square of Ridge regression minimizes the sum of the squared
coefficients, pushing them toward zero. This helps differences between the observed and predicted
reduce model complexity and mitigate overfitting, values, with an additional penalty for the square of
leading to better generalization. coefficients. This helps reduce the impact of less
important features, making the model more
robust and less sensitive to noise in the data.

Least squares line

Ridge regression
y = ssr + λ slope 2

Size

Weight
Classification vs. Regression
Classification Models
1: Logistic Regression

Logistic regression models the probability of a binary outcome based on input variables.

Logistic Regression

User Age Yes

User Income

User Gender
No

Output Purchase | Yes or No

2: Linear Discriminant Analysis

Linear discriminant analysis is a method used for dimensionality reduction and classification. It
works by finding the linear combination of features that best separates multiple classes.

Bacteria 34 Bacteria
60
Discriminant scores

Virus Virus
32
CRP (mg/L)

40
30

20
28

0 26
36 38 40 42
Temp (C)
Classification vs. Regression
3: Naive Bayes Classifier

The Naive Bayes classifier calculates probabilities for each class and selects the one with the highest
probability.

Naive Bayes Classifier

Classifier

P(B|A) . P(A)
P (A|B) =
P(B)

Regression Models
1: Linear Regression

Linear regression works by finding the best-fitting straight line through a set of data points to predict
the relationship between the independent and dependent variables.

Fitted Line Plot

Price
350000

300000

250000

200000

150000

150000

50000
Size
1000 2000 3000 4000
Classification vs. Regression
2: Polynomial Regression

Polynomial regression works by fitting a polynomial equation to the data points to capture the non-
linear relationship between the independent and dependent variables.

Polynomial Model
Y

b 0 + b1x1+ b 2x1
y= 2

3: Gaussian Process Regression (GPR)

GPR uses probability to predict the values. This helps make predictions for our data and new data
points, along with a measure of uncertainty for each prediction.

20
f (χ) = χ sin(χ)
Predicton
15 95% confidence interval

Observations

10

-5

-10
0 2 4 6 8 10