Regression
Bivariate – Multivariate Analysis
Regression analysis
• Statistical method used to identify and quantify the relationship
between a dependent variable (the outcome you want to predict)
and one or more independent variables (the predictors).
Regression
• Dependent Variable (Y): Also called the "response" or "target"
variable. It is the factor you are trying to understand or forecast.
• Independent Variable (X): Also called "explanatory" or "predictor"
variables. These are the factors you believe influence the
dependent variable.
• Regression Line: The "line of best fit" that represents the
relationship between variables. It minimizes the sum of squared
errors (the vertical distance between data points and the line).
What are Predictors?
• A predictor is a variable in a dataset that is used to predict or
explain the value of another variable, which is called the target or
dependent variable.
• Predictor vs Target
Predictors (X)
• Inputs used to make predictions
• Data the model learns from
• Examples: Age, Income, Location, Temperature
Target / Response (Y)
• What we want to predict
• The model’s final outcome
• Examples: Click (Yes/No), House Price, Disease Status
House Price Prediction
• Predictors (X): Size of house, Location, Age of building
• Target (Y): House Price
• Formula:
Price (Y) = f(Size, Location, Age)
Types of Regression
Simple Linear Regression
Predicts a continuous outcome using one independent variable
Equation: 𝑌 = 𝑎 + 𝑏𝑋 + 𝜀
𝑎: Intercept (value of Y when X = 0)
𝑏: Slope (change in Y per unit change in X)
Multiple Linear Regression
Uses two or more independent variables
Example: Predicting house price using size, location, and age
Types of Regression
Logistic Regression
• Used for classification problems
• Output is categorical (Yes/No, Pass/Fail)
Nonlinear Regression
• Models curved or complex relationships
• Examples: Polynomial, Exponential
Predictors
• Single Predictors
• Multiple Predictors
Single Predictor Model Simple & interpretable
Less accurate – ignores other factors
• “One cause → One effect”
Uses only ONE variable to predict an outcome.
Formula (Simple Regression):
Y = a + bX
Y → Outcome / Target (e.g., AC Sales)
X → Single predictor (e.g., Temperature)
a → Intercept
b → Effect of X on Y
Example: Predict AC Sales using only Temperature
Fitting a line to data
• A perfect linear relationship
• Predict y exactly from x using a
straight line
(e.g., y = 5 + 64.96x).
• In real life, this is rare — most
outcomes depend on many factors.
•
Example: Family income helps predict
college financial aid,
but it cannot predict it perfectly.
Three datasets where a linear model - even though
the data do not all fall exactly on the line.
Using linear regression to predict possum head
lengths
• 104 possums size measured in Australia
• Variables: Total length (cm) → predictor (x) Head length (mm) → response (y)
• Scatterplot shows a positive linear association
(longer possums → longer heads)
Linear Model for Possum Data (A least-squares
line)
• Regression equation:
ŷ = 41 + 0.59x
• For x = 80 cm → ŷ = 88.2 mm
• Interpretation:
• This is an average estimate for
possums of that length.
• Other factors may affect
predictions:
• Sex: males tend to be larger than
females
• Age: effect less clear ŷ = estimated head length,
x = total length
Other factors may affect predictions:
• Relationship between total length and head length of brushtail
possums, taking into consideration their sex or age.
Residuals
Goal of regression: Minimize residuals to get the best-fitting line
• Residual = Observed − Predicted value
• Data = Fit + Residual
• Interpretation:
• Point above line → positive residual
• Point below line → negative residual
• Smaller |residual| ⇒ better fit small negative residual (≈ −1)
large positive residual (≈ +7)
moderate negative residual (≈ −4)
Interpreting Residuals
• Highlighted points show different
residual sizes:
• Red circle: small negative residual (≈
−1)
• Gray diamond: large positive residual
(≈ +7)
• Pink triangle: moderate negative
residual (≈ −4)
• Residual size is measured by absolute
value |e|
• Since |−4| > |−1|, the pink triangle has a
larger error than the red circle.
Residual plot for the model predicting head length from total
length for brushtail possums
Residual Example
Model: ŷ = 41 + 0.59x
Observation: (76.0, 85.1)
Predicted: ŷ = 41 + 0.59(76) = 85.84
Residual:
𝑒 = 𝑦 − 𝑦ො = 85.1 − 85.84 = −0.74 mm
Interpretation:
Negative residual → Model overestimated head length.
Small residual ⇒ good fit.
Simple Linear
Regression
Sample Dataset for Linear Regression
Fuel Consumption
Engine Size Cylinders (independent CO₂ Emissions
(Comb) (independent
(independent variable) variable) (dependent variable)
variable)
2.0 4 8.5 196
2.4 4 9.6 221
1.5 4 5.9 136
3.5 6 11.1 255
3.5 6 10.6 244
3.5 6 10.0 230
3.5 6 10.1 232
3.7 6 11.1 255
3.7 6 11.6 267
2.4 4 9.2 ? (To be predicted)
• Thita 0 - Constant or intercept
• Thita 1 – coefficient (estimated by regression)
• X – Predictor(present in data)
Predictions with Linear Regression
Fuel Consumption
Engine Size Cylinders CO₂ Emissions
(Comb)
2.0 4 8.5 196
2.4 4 9.6 221
1.5 4 5.9 136
3.5 6 11.1 255
3.5 6 10.6 244
3.5 6 10.0 230
3.5 6 10.1 232
3.7 6 11.1 255
3.7 6 11.6 267
2.4 4 9.2 ? (To be predicted)
Solved Linear Regression
Table (Single Variable)
Engine Size CO₂ Emissions
2.0 196
2.4 221
1.5 136
3.5 255
3.5 244
3.5 230
3.5 232
3.7 255
3.7 267
2.4 ? (To be predicted)
Mx=3.0333 𝑀𝑦 = 226.2222
X Y (X - 𝑀𝑥 ) (Y - 𝑀𝑦 ) (X − 𝑀𝑥 ) 2 (X - 𝑀𝑥 ) (Y - 𝑀𝑦 )
2.0 196 -1.0333 -30.2222 1.0678 31.2296
2.4 221 -0.6333 -5.2222 0.4011 3.3074
1.5 136 -1.5333 -90.2222 2.3511 138.3407
3.5 255 0.4667 28.7778 0.2178 13.4296
3.5 244 0.4667 17.7778 0.2178 8.2963
3.5 230 0.4667 3.7778 0.2178 1.7630
3.5 232 0.4667 5.7778 0.2178 2.6963
3.7 255 0.6667 28.7778 0.4444 19.1852
3.7 267 0.6667 40.7778 0.4444 27.1852
Sum of
Σ Σ 0 0 Sum of Products
Squares
3.0333 226.2222 5.58 245.4333
Regression Formula
• Slope:
( 𝑋 − 𝑀𝑥 ሻ 𝑌 − 𝑀𝑦 245.4333
𝑏= 2 = ≈ 43.98
( 𝑋 − 𝑀𝑥 ሻ 5.58
• Intercept:
𝑎 = 𝑀𝑦 − 𝑏𝑀𝑥 = 226.2222 − 43.98 × 3.0333 ≈ 92.82
• Final Regression Equation Predict for Engine size X=2.4
• 𝑌 = 92.82 + 43.98𝑋
43.98*2.4 + 92.8 = 198.352
Evaluating Regression Model Accuracy
• Regression Sum of
Squares (SSR)
• Error Sum of Squares
(SSE)
• Total Sum of Squares
(SST) SST=SSR+SSE
• Coefficient of
Determination 𝑅2
• Adjusted 𝑅2
Total Sum of Squares (SST)
• Measures total variation in the dependent variable 𝑌
ˉ
2
𝑆𝑆𝑇 = ( 𝑌𝑖 − 𝑌ሻ
Regression Sum of Squares (SSR)
ˉ
2
𝑆𝑆𝑅 = ( 𝑌𝑖 − 𝑌ሻ
•Measures variation explained by the regression model
•𝑌𝑖 :predicted values from regression equation
Error Sum of Squares (SSE)
2
𝑆𝑆𝐸 = ( 𝑌𝑖 − 𝑌𝑖 ሻ
•Measures unexplained variation (error/residuals)
Relationship Between Variations
𝑆𝑆𝑇 = 𝑆𝑆𝑅 + 𝑆𝑆𝐸
Coefficient of Determination (𝑅 ሻ
2
Proportion of total variation in the dependent variable
explained by the model.
2
𝑆𝑆𝑅 𝑆𝑆𝐸
𝑅 = =1−
𝑆𝑆𝑇 𝑆𝑆𝑇
•Represents the proportion of variance
explained by predictors
Coefficient of Determination (𝑅 ሻ
2
• 𝑅2
• Values range from 0 to 1
• Higher R² ⇒ better fit
• Always increases (or stays same) when you add more
predictors -- even if the predictor is useless
• Limitation
• Can be misleading in multiple regression due to
overfitting
Adjusted 𝑅 2
Proportion of variance explained after adjusting for
number of predictors
𝑆𝑆𝐸/ 𝑛 − 𝑘 − 1
Adjusted 𝑅 = 1 −
2
𝑆𝑆𝑇/ 𝑛 − 1
where:
•𝑛= number of observations
•𝑘= number of independent variables
Adjusted 𝑅 2
• Penalizes unnecessary predictors
• Can increase or decrease
• More reliable for model comparison
Aspect R² Adjusted R²
Adjusted variance
Meaning Variance explained
explained
Effect of adding Increases only if
Always increases
predictors predictor is useful
Overfitting
No Yes
detection
Multiple
Used in Simple regression
regression
Model comparison Not reliable Reliable
Guided Practice - Think !!!
• If a model underestimates an observation, is the residual positive
or negative?
• If a model overestimates an observation, is the residual positive or
negative?
Guided Practice
• Compute the residuals for:
• Blue diamond: total length 85.0 cm, head length 98.6 mm
• Pink triangle: total length 95.5 cm, head length 94.0 mm
• Use the linear model:
Head length = 41 + 0.59 × total length
Describing linear relationships with correlation
Correlation
• Correlation which always takes values between -1 and 1,
describes the strength and direction of the linear relationship
between two variables.
Interpreting Correlation
• Perfect linear pattern → correlation = +1 or −1
• Strong positive relationship → near +1
• Strong negative relationship → near −1
• No linear relationship → near 0
Multiple Predictors Model Higher accuracy & real-world relevance
Widely used in business forecasting
“Many causes → One intelligent prediction”
Uses MULTIPLE variables together for prediction.
Formula (Multiple Regression):
Y = a + b₁X₁ + b₂X₂ + b₃X₃ + … + bₙXₙ
Y → Interest Rate (to be fixed)
X₁, X₂, X₃ … → Multiple predictors (Verified income, Bankruptcy,credit
checks, etc..)
b₁, b₂, b₃ … → Individual effects of each predictor
Indicator and categorical predictors
Indicator and categorical predictors
• The variable takes one of two values: 1 when the borrower has a
bankruptcy in their history and 0 otherwise. A slope of 0.74 means
that the model predicts a 0.74% higher interest rate for those
borrowers with a bankruptcy in their record.
Many predictors in a model
Output for the regression model, where interest rate is the
outcome and the variables listed are the predictors.
Model Selection
• Stepwise selection adds or removes predictors one at a time.
Techniques:
• Backward elimination: Start with all predictors, remove one at a time
until model cannot improve.
• Forward selection: Start with no predictors, add one at a time until no
further improvement.
Model Selection - Decision criterion:
Decision criterion: Often use adjusted R².
• Add/remove predictors based on which improves adjusted R².
• Stop when no further improvement occurs.
• Adjusted R²: Measures model strength and helps identify
predictors that add value to prediction accuracy.