Unit III: Supervised Learning Algorithms
Course: Machine Learning (Minor)
Level: [Link]. Second Year
Duration: 12 Hours
1. Introduction to Supervised Learning
Supervised learning is a class of Machine Learning techniques where the
model is trained using labeled data. Each training example consists of an
input vector X and a corresponding target output Y. The objective is to learn
a mapping function:
[ f: X Y ]
Characteristics
Availability of labeled dataset
Explicit target variable
Used for prediction and classification tasks
Types of Supervised Learning
Type Output Variable Examples
Regressi Continuous Price prediction,
on temperature
forecasting
Classific Discrete / Categorical Spam detection,
ation disease diagnosis
2. Regression Algorithms
Regression algorithms are used when the output variable is continuous.
2.1 Linear Regression
Definition
Linear Regression models the relationship between a dependent variable (y)
and an independent variable (x) using a straight line.
Mathematical Model
[ y = mx + c ]
Where: - (m) = slope of the line - (c) = y-intercept
Cost Function (Mean Squared Error)
[ J(m,c) = _{i=1}^{n} (y_i - _i)^2 ]
Diagram (Conceptual)
X-axis: Independent variable
Y-axis: Dependent variable
Best-fit straight line minimizing error
Algorithm: Simple Linear Regression
1. Initialize parameters (m) and (c)
2. Predict output using (y = mx + c)
3. Compute cost function
4. Update parameters using Gradient Descent
5. Repeat until convergence
Python Program: Linear Regression
import numpy as np
import [Link] as plt
from sklearn.linear_model import LinearRegression
X = [Link]([[1], [2], [3], [4], [5]])
y = [Link]([2, 4, 5, 4, 5])
model = LinearRegression()
[Link](X, y)
y_pred = [Link](X)
[Link](X, y)
[Link](X, y_pred)
[Link]('X')
[Link]('y')
[Link]('Simple Linear Regression')
[Link]()
2.2 Multiple Linear Regression
Definition
Multiple Linear Regression models the relationship between one dependent
variable and multiple independent variables.
Mathematical Model
[ y = b_0 + b_1x_1 + b_2x_2 + … + b_nx_n ]
Applications
House price prediction (area, rooms, location)
Sales forecasting
Python Program: Multiple Linear Regression
from [Link] import load_boston
from sklearn.linear_model import LinearRegression
X, y = load_boston(return_X_y=True)
model = LinearRegression()
[Link](X, y)
print('Coefficients:', model.coef_)
2.3 Polynomial Regression
Definition
Polynomial Regression models non-linear relationships by transforming input
features into polynomial features.
Mathematical Model
[ y = a_0 + a_1x + a_2x^2 + … + a_nx^n ]
Diagram (Conceptual)
Curved best-fit line instead of straight line
Python Program: Polynomial Regression
from [Link] import PolynomialFeatures
from sklearn.linear_model import LinearRegression
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
model = LinearRegression()
[Link](X_poly, y)
3. Classification Algorithms
Classification algorithms are used when the output variable is categorical.
3.1 k-Nearest Neighbors (k-NN)
Concept
k-NN is an instance-based learning algorithm that classifies a data point
based on the majority class of its nearest neighbors.
Steps
1. Choose value of k
2. Compute distance (Euclidean)
3. Select k nearest neighbors
4. Assign majority class
Distance Formula
[d= ]
Python Program: k-NN
from [Link] import KNeighborsClassifier
from [Link] import load_iris
from sklearn.model_selection import train_test_split
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y)
model = KNeighborsClassifier(n_neighbors=5)
[Link](X_train, y_train)
print('Accuracy:', [Link](X_test, y_test))
3.2 Naïve Bayes Classifier
Concept
Naïve Bayes is a probabilistic classifier based on Bayes’ Theorem with
independence assumption.
Bayes’ Theorem
[ P(A|B) = ]
Types
Gaussian Naïve Bayes
Multinomial Naïve Bayes
Python Program: Naïve Bayes
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
[Link](X_train, y_train)
print('Accuracy:', [Link](X_test, y_test))
3.3 Decision Trees
Concept
Decision Trees classify data using a tree-like structure of decisions.
Key Terms
Root node
Internal node
Leaf node
Splitting Criteria
Gini Index
Information Gain (Entropy)
Python Program: Decision Tree
from [Link] import DecisionTreeClassifier
model = DecisionTreeClassifier(criterion='entropy')
[Link](X_train, y_train)
print('Accuracy:', [Link](X_test, y_test))
4. Overfitting and Underfitting
Concept Description
Underfitting Model too simple, high bias
Overfitting Model too complex, high
variance
Diagram (Conceptual)
Underfit: Straight line
Good fit: Smooth curve
Overfit: Highly oscillating curve
5. Bias–Variance Trade-off
Bias
Error due to overly simple assumptions
Variance
Error due to model sensitivity to training data
Trade-off Diagram
X-axis: Model complexity
Y-axis: Error
Bias decreases, variance increases
6. Summary Table
Algorithm Type Output
Linear Regression Regression Continuous
Polynomial Regression Continuous
Regression
k-NN Classification Categorical
Naïve Bayes Classification Categorical
Decision Tree Classification Categorical
7. Exam-Oriented Points
Linear Regression assumes linear relationship
k-NN is a lazy learner
Naïve Bayes assumes feature independence
Decision Trees prone to overfitting
End of Unit III