Regression in Machine Learning

Regression is a machine learning technique used to predict a number (continuous value). It shows
a relationship between dependant (target) and independent (explanatory) variables. In simple
words, Regression = Predicting a numerical value using past data. So, whenever output is a
number, we use regression.

Basic Idea of Regression: Regression finds a relationship between Input (X) and Output (Y).
Example:
 Hours Studied (X)
 Marks Obtained (Y)
If hours increase → marks increase. Regression draws a best fit line to show this relation.

Why Do We Use Regression?

We use regression when we want to estimate or forecast future values.
Reasons:
 To know future trends
 To make business decisions
 To analyze relationships between variables
 To predict outcomes
Example:
 A school wants to predict next year admission count.
 A company wants to predict sales next month.

Field Use Example

Education Predict student marks
Business Sales prediction
Healthcare Predict patient recovery time
Temperature forecasting , or rainfall
Weather
forecasting
Real Estate House price prediction
Agriculture Crop yield prediction

Types of Regression
(A) Linear Regression
Most common and simple type.
Formula: Y = mX + c
Where:
 Y = Output (Marks)
 X = Input (Hours)
 m = Slope (rate of change)
 c = Intercept (starting value)

Example Data:
Hours (X) Marks (Y)
1 40
2 50
3 60
The line shows marks increasing with hours.
1
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
(B) Multiple Linear Regression
When more than one input is used.
Example: Marks = f(Study Hours, Attendance, Sleep)
Formula: Y = b0 + b1X1 + b2X2

(C) Polynomial Regression

Used when data is curved, not straight.
Example: Growth of plants over time.
Formula: Y = aX² + bX + c

Mathematical Example
Suppose we have data:
X (Hours) Y (Marks)
1 45
2 50
3 55
We assume equation: Y = 5X + 40
Check:
 X=1 → Y=45
 X=2 → Y=50
 X=3 → Y=55
So model works.

Important Terms

Term Meaning
Feature Input (X)
Target Output (Y)
Model Formula/Algorithm
Training Data Past Data
Prediction Future Output
Error Difference between actual & predicted

Advantages of Regression
Easy to understand, Good for forecasting, Fast calculation and Useful in many industries

Limitations
Works best with clean data, cannot handle complex patterns (Linear Regression) and Sensitive to
outliers

2
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
 Real Life Example Problem
Problem: Predict house price. Predict salary using years of experience.
Inputs: Area, Location, Number of rooms Experience Salary (₹)
Output: Price 1 20,000
Regression will learn from past house data 2 30,000
and predict new price.
3 40,000
Problem 1 – Student Marks Prediction
Problem: A teacher wants to predict
student marks based on study hours. Solution
Data: Salary increases 10,000 per year.
Equation: Y = 10000X + 10000
Hours (X) Marks (Y)
For 5 years’ experience:
1 40 Y = 10000×5 + 10000 = 60,000
2 50 Predicted Salary = ₹60,000
3 60
Solution Problem 4 – Temperature Prediction
We use Linear Regression formula: Problem
Y = mX + c Predict temperature over days.
Here we observe: Day Temp (°C)
 Every extra hour → marks increase 1 30
by 10 2 32
 So m = 10
3 34
 When X = 0, marks ≈ 30 → c = 30
Solution
Equation = 10X + 30 Temperature increases 2°C daily.
If student studies 4 hours: Y = 10×4 + 30 = Equation: Y = 2X + 28
70 marks For Day 5:
Y = 2×5 + 28 = 38°C
Problem 2 – House Price Prediction Predicted Temperature = 38°C
Problem
Predict house price based on area (sq ft). Problem 5 – Crop Yield Prediction
Problem
Area (X) Price (Y in Lakhs)
Predict crop yield using fertilizer amount.
500 10 Fertilizer (kg) Yield (tons)
1000 20 1 2
1500 30 2 4
Solution
3 6
Price increases 10 lakhs per 500 sq ft.
Equation: Y = 0.02X Solution
If area = 2000 sq ft: Y = 0.02 × 2000 = 40 Yield doubles with fertilizer.
Lakhs Equation: Y = 2X
So predicted price = 40 Lakhs If fertilizer = 5 kg:
Y = 2×5 = 10 tons
Problem 3 – Salary Prediction Predicted Yield = 10 tons

3
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
In all problems: X = Input (Feature), Y = Output (Target)
We find a pattern, then create a formula, Use formula to predict future values

Residual Error
Residual Error (Residual) is the difference between the actual value and the predicted value in
regression. Residual = Actual Value − Predicted Value (It shows how much the model is wrong
for one data point.) Difference Between Residual Error and Loss Function (Simple) Both are
related to error, but they are not the same. Residual = error of one prediction. Loss Function (Error
for all data points together.) = average/total error of all predictions. MSE is one of the loss function

What is a Loss Function?

A Loss Function is a mathematical formula that tells: “How wrong is the model?” It calculates
the difference between Actual value (y) and Predicted value (ŷ). If loss is small → Model is
good and If loss is large → Model is bad

Why Loss Function is Important?

 Model training tries to minimize loss
 Optimization algorithms (Gradient Descent) use loss
 All evaluation metrics come from loss
Machine Learning Goal: Minimize Loss → Improve Accuracy

Common Loss Functions in Regression

1. Mean Absolute Error (MAE) Loss
Formula: MAE = Σ |Actual − Predicted| / n
Nature
 Absolute difference
 Treats all errors equally
 Easy to understand
Example
Actual = [10, 20] and Predicted = [8, 18]
Errors = 2, 2
MAE = 2
2. Mean Squared Error (MSE) Loss
Formula: MSE = Σ (Actual − Predicted) ² / n
Nature
 Squares error
 Large errors punished more
 Used in training models
Example Errors = 2, 2
MSE = 4
3. Root Mean Squared Error (RMSE)
Formula: RMSE = √MSE
Nature
 Same unit as output
 Easier interpretation

4. Huber Loss (Advanced but Useful)

Idea
Combination of MAE + MSE.
 Small errors → squared

4
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
  Large errors → absolute
Used when outliers exist.

Relationship Between Loss and Performance Metrics

Loss Function Also Used As Metric? Purpose
MAE Yes Average error
MSE Yes Penalize large errors
RMSE Yes Real-unit error
R² No (Derived Metric) Fit quality

Loss Function measures how wrong the predictions are, and regression performance metrics are
different ways of viewing or summarizing that loss.

What is R² (R-Squared)?
R² (Coefficient of Determination) is a metric used in Regression to measure how well the model fits the
data. In simple words: R² tells how much of the actual variation is explained by the model.
Range of R²
 1 → Perfect model
 0 → Very poor model
 Negative → Worse than average prediction

Formula (Concept)
R² = 1 − (SSres / SStotal)
Where:
 SSres (Residual Sum of Squares) = Σ(Actual − Predicted)²
 SStotal (Total Sum of Squares) = Σ(Actual − Mean of Actual)²

Step-by-Step Numerical Example

Data
Actual (y) Predicted (ŷ)
3 2
5 4
7 6

Step 1 — Find Mean of Actual

Mean y = (3 + 5 + 7) / 3 = 5
Step 2 — Calculate SSres
[ (3−2)² + (5−4)² + (7−6)² = 1 + 1 + 1 = 3 ]
Step 3 — Calculate SStotal
[(3−5)² + (5−5)² + (7−5)² = 4 + 0 + 4 = 8 ]
Step 4 — Apply Formula
[R² = 1 − (3 / 8) = 1 − 0.375 = 0.625]
Interpretation: R² = 0.625 means and 62.5% of the data variation is explained by the
model.
How to Know “Best” R²?
 Closer to 1 → Better
 0.7 – 0.9 → Good model
 > 0.9 → Very strong fit
 < 0.5 → Weak model
But remember:
 Very high R² with many features may mean overfitting
 Always also check Adjusted R², MAE, RMSE

5
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
R² measures how well regression predictions match actual data — higher R² means better model
fit.

Regularization in Regression
Regularization is a technique used to control overfitting by adding a penalty to large coefficients
(weights) in a regression model.
Simple Meaning: We tell the model: “Do not become too complex. Stay simple and stable.”

Why Do We Need Regularization?

Without Regularization:
 Model memorizes training data
 High variance
 Poor performance on new data
 Very large weights
With Regularization:
 Controls complexity
 Reduces overfitting
 Improves prediction on unseen data
 Keeps weights small and stable

What is Overfitting?
Overfitting = Model fits training data perfectly but fails on new data.
Imagine drawing a curve that bends too much just to pass through every point — it becomes
unrealistic.

Where Do We Use Regularization?

 Linear Regression
 Polynomial Regression
 Logistic Regression
 Neural Networks / Deep Learning
 Any ML model where overfitting risk exists

Core Idea Behind Regularization

Regularization adds a penalty term to the loss function.
Normal Loss:
Loss = RSS (Error)
With Regularization:
Loss = RSS + Penalty
If weights become large → penalty increases → optimizer reduces weights.

IMPORTANT CLARITY POINT

Regularization DOES NOT change the equation type or polynomial degree.
It only reduces coefficient values.
 Degree = Structure of equation
 Coefficients = Numbers in equation

Types of Regularization
1. L1 Regularization (Lasso)
6
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
Formula
Loss = RSS + λ Σ |w|
Key Features
 Uses absolute values
 Some weights become exactly zero
 Performs Feature Selection

2. L2 Regularization (Ridge)
Formula
Loss = RSS + λ Σ w²
Key Features
 Uses square of weights
 Weights become small but not zero
 Keeps all features

3. Elastic Net
Formula
Loss = RSS + λ1 Σ |w| + λ2 Σ w²
Key Features
 Combination of L1 + L2
 Useful for many correlated features
 Balanced shrinkage

REFINED NUMERICAL EXAMPLES

Example 1 — Simple Linear Regression
Dataset
xy
12
24
36
True relation: y = 2x

Case A — Without Regularization

Assume model learns:
y = 7x
Predictions:
x Actual Predicted
12 7
24 14
36 21
RSS becomes very large → model unstable.

Case B — With L2 Regularization

Loss includes penalty:
If weight = 7
Penalty = 49
Optimizer reduces weight → new weight ≈ 2.2
New equation:
7
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
y = 2.2x
Predictions become close to real values → stable model.

Example 2 — Polynomial Regression (Degree Fixed)

Model Chosen
y = w1 x + w2 x²
Degree = 2 (fixed)

Without Regularization
y = 12x + 9x²
For x=2 → y=60 → unrealistic curve.

With Regularization
y = 2x + 1x²
Curve becomes smooth and closer to real data.
Degree did NOT change. Only coefficients changed.

Example 3 — Feature Selection (L1)

Suppose:
y = 3x1 + 4x2
After L1:
y = 2x1 + 0x2
Feature x2 removed automatically.

Comparison Table
Type Penalty Feature Removal Weight Nature
L1 (Lasso) w
L2 (Ridge) w² No All small
Elastic Net L1 + L2 Partial Balanced

Concept :
 No Regularization → Zig-zag curve → Overfitting
 Too Much Regularization → Very flat line → Underfitting
 Balanced Regularization → Smooth curve → Best performance

Real-Life Analogies
1. Speed Breaker 🚧
 No breaker → accident
 Too many breakers → slow travel
 Balanced → safe & efficient
2. Student Example
 Memorizing student → fails new questions
 Understanding student → solves new questions
Regularization makes the model like the understanding student.

8
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
So, Regularization adds a penalty to large coefficients so the model remains simple, stable, and
performs well on unseen data.

What is Logistic Regression?

Logistic Regression is a Machine Learning method used to predict categories (Yes/No,
True/False, 0/1). In simple words: Logistic Regression = Predicting a Class or Category, not
a number. Logistic Regression is a machine learning algorithm used to predict categories (0 or
1) using probability.

Difference Between Linear and Logistic Regression

Linear Regression Logistic Regression
Predicts Number Predicts Category
Example: Marks = 75 Example: Pass / Fail
Output is continuous Output is 0 or 1

Why Do We Use Logistic Regression?

We use it when output is two choices like:
 Yes / No
 Pass / Fail
 Spam / Not Spam
 Disease / No Disease
 Male / Female (classification example)

Where Is Logistic Regression Used?

Field Example
Education Pass or Fail prediction
Email Spam detection
Medical Disease detection
Banking Loan Approved or Not
Marketing Customer will buy or not

Basic Idea
Instead of giving a number like 75 or 80, it gives probability between 0 and 1.
 If probability > 0.5 → Class = 1 (Yes)
 If probability < 0.5 → Class = 0 (No)

Mathematical Formula (Simple)

Logistic Regression uses Sigmoid Function:
P = 1 / (1 + e^-(mx + c))
Do not worry about “e”.
Just remember:
 Output is always between 0 and 1

9
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India
  It converts numbers into probability

Example Problem – Pass/Fail

Study Hours Result
1 Fail (0)
2 Fail (0)
3 Pass (1)
4 Pass (1)
If a student studies 2.5 hours, model may give:
Probability = 0.6
Since 0.6 > 0.5 → Pass

Important Terms
Term Meaning
Probability Chance between 0 and 1
Sigmoid S-shaped curve
Threshold Decision limit (0.5)
Class Category (0 or 1)

Graph Idea
Linear Regression → Straight Line and Logistic Regression → S-Shaped Curve

Advantages
Simple and fast, Good for binary classification, and Easy to understand

Limitations
Only works well for simple data, Not good for very complex patterns and Mostly for two
classes

10
Dr. Rakesh Kumar Yadav, Associate Professor & Head, Department of CSE, Deputy Dean, Maharishi
School of Engineering & Technology, Maharishi University of Information Technology, Lucknow, U.P.
India