Model Selection and Stepwise Regression
Feature Selection in multiple regression
Feature selection is a crucial step in the data preprocessing pipeline for regression tasks. It
involves identifying and selecting the most relevant features (or variables) that contribute to the
prediction of the target variable. This process helps in reducing the complexity of the model,
improving its performance, and making it more interpretable.
Feature selection is vital because not all features in a dataset are equally important. Some
features may be irrelevant or redundant, leading to overfitting and poor model performance. By
selecting only the most relevant features, you can:
● Reduce model complexity: Fewer features mean a simpler model, which is easier to
interpret and faster to train.
● Improve model performance: Removing irrelevant features can enhance the model's
predictive accuracy.
● Prevent overfitting: With fewer features, the model is less likely to learn noise from the
training data.
Correlation Analysis
Correlation analysis helps identify linear relationships between features and the target variable.
Features with high correlation to the target variable are typically considered more important for
the regression model. Similarly, pairs of features with high correlation to each other might
indicate redundancy, where only one feature may be necessary.
import pandas as pd
correlation_matrix = [Link]()
print(correlation_matrix["target_variable"].sort_values(ascending=False))
Univariate Selection
Univariate feature selection involves selecting features based on their individual relationship
with the target variable. This method uses statistical tests to determine the significance of each
feature.
from sklearn.feature_selection import SelectKBest, f_regression
# Assuming df is your DataFrame and 'target_variable' is the column you want to predict
X = [Link]("target_variable", axis=1)
y = df["target_variable"]
# Applying SelectKBest with ANOVA F-value
selector = SelectKBest(score_func=f_regression, k='all')
[Link](X, y)
# Displaying scores for each feature
feature_scores = [Link]({'Feature': [Link], 'Score': selector.scores_})
print(feature_scores.sort_values(by='Score', ascending=False))
Recursive Feature Elimination (RFE)
RFE is a recursive method that eliminates less important features in a step-by-step manner. It
works by fitting a model and removing the weakest feature(s) until the desired number of
features is reached.
from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression
# Assuming df is your DataFrame and 'target_variable' is the column you want to predict
X = [Link]("target_variable", axis=1)
y = df["target_variable"]
# Initialize the model (LinearRegression here, but you can use others)
model = LinearRegression()
# Applying RFE
rfe = RFE(estimator=model, n_features_to_select=5) # Change n_features_to_select as needed
[Link](X, y)
# Displaying ranking of features
feature_ranking = [Link]({'Feature': [Link], 'Ranking': rfe.ranking_})
print(feature_ranking.sort_values(by='Ranking'))
AIC (Akaike Information Criterion) is a metric used for model selection. It balances
goodness-of-fit and model complexity by penalizing extra parameters.
📌 AIC Formula: = Likelihood of the model AIC = 2k − 2 log(L)
where: K= Number of parameters (including intercept),
L= Likelihood of the model
📌Bayesian Information Criterion (BIC) is a model selection metric that
penalizes complexity:
BIC = k ln(n) − 2 log(L)
where: k = Number of parameters (including intercept)
n= Number of observations
l= Likelihood of the model
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import r2_score,mean_squared_error
import [Link] as plt
from sklearn.feature_selection import SelectKBest, chi2
import [Link] as sm
df=pd.read_csv("C:\\Users\\CSE-STAFF\\Downloads\\Student_Performance.csv")
print(df)
x1=[Link](columns=['Extracurricular Activities','Performance Index'])
#4 features selected
x2=df[['Hours Studied']] #only one feature selected
y=df["Performance Index"]
X1 = sm.add_constant(x1)
X2 = sm.add_constant(x2)
# Fit regression model
model1 = [Link](y, X1).fit() #for 4 features 1 output
model2 = [Link](y, X2).fit() #for 1 feature 1 output
# Print AIC value
print("AIC and bic for first model:", [Link], [Link])
print("AIC and bic for second model:", [Link],[Link])
Model Selection in Regression
Selecting the best regression model involves comparing different models based on statistical
metrics such as:
Adjusted R² (Accounts for the number of predictors/features)
AIC (Akaike Information Criterion) (Penalizes model complexity)
BIC (Bayesian Information Criterion) (Similar to AIC, but stronger penalty for complexity)
RMSE (Root Mean Square Error) (Measures prediction error)
Stepwise Regression
1. Stepwise regression is a method that automatically selects significant features in a
multiple regression model. it is used to find the smallest number of variables that can
explain the data.
2. Stepwise regression is a popular method for model selection because it can automatically
select the most important variables for the model and build a parsimonious model. This
can save time and effort for the data scientist or analyst, who does not have to manually
select the variables for the model.
3. Stepwise regression can also improve the model’s performance by reducing the number
of variables and eliminating any unnecessary or irrelevant variables. This can help to
prevent overfitting, which can occur when the model is too complex and does not
generalize well to new data.
4. Stepwise regression is a method for building a regression model by adding or removing
predictors in a step-by-step fashion. The goal of stepwise regression is to identify the
subset of predictors that provides the best predictive performance for the response
variable.
There are three main approaches:
1. Forward Selection
Start with no features
Add features one by one based on statistical significance
(lowest p-value or highest Adjusted R²).
Starts with no features and adds one feature at a time based on performance.
Stops when the specified number of features is reached or performance stops improving.
Python Code
from mlxtend.feature_selection import SequentialFeatureSelector as SFS
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Initialize model
model = LinearRegression()
# Perform forward selection (adds features one by one)
sfs = SFS(model,
k_features=5, # Selects 5 best features
forward=True, # Forward Selection
floating=False,
scoring='r2', # Uses R²
score cv=5) # 5-fold cross-validation [Link](X_train, y_train)
# Print selected features
print("Selected Features:", sfs.k_feature_names_
2. Backward Elimination
Start with all features
Remove least significant features based on highest p-value.
Starts with all features and removes the least important one at a time.
Stops when the specified number of features is reached or performance stops improving.
sbs = SFS(model, k_features=5, # Selects 5 best features
forward=False, # Backward Elimination
floating=False, scoring='r2', cv=5)
[Link](X_train, y_train)
# Print selected features print("Selected Features:", sbs.k_feature_names_)
3. Bidirectional (Stepwise Selection)
Combination of forward and backward, adding/removing features iteratively based on p-values
and AIC/BIC.
After each step, the forward and backward processes alternate, refining the selected feature set
by considering both additions and eliminations. This bidirectional approach allows the model to
dynamically adjust to the most useful subset of features.
SequentialFeatureSelector(model, n_features_to_select='auto', direction='both', cv=5)
Generalized Linear Model
In statistics, a generalized linear model (GLM) is a flexible generalization of ordinary linear
regression. The GLM generalized linear regression by allowing the linear model to be related to
the response variable via a link function and by allowing the magnitude of the variance of each
measurement to be a function of its predicted value.
Generalized Linear Models (GLMs) are a class of regression models that can be used to model a
wide range of relationships between a response variable and one or more predictor variables.
Unlike traditional linear regression models, which assume a linear relationship between the
response and predictor variables, GLMs allow for more flexible, non-linear relationships by
using a different underlying statistical distribution.
Below are some types of datasets and the corresponding distributions which would help us in
constructing the model for a particular type of data (The term data specified here refers to the
output data or the labels of the dataset).
1. Binary classification data – Bernoulli distribution
2. Real valued data – Gaussian distribution
3. Count-data – Poisson distribution
Why GLM?
Linear Regression model is not suitable if,
● The relationship between X and y is not linear. There exists some non-linear
relationship between them. For example, y increases exponentially as X
increases.
● Variance of errors in y (commonly called Homoscedasticity in Linear
Regression), is not constant, and varies with X.
● Response variable is not continuous, but discrete/categorical. Linear Regression
assumes normal distribution of the response variable, which can only be applied
on continuous data. If we try to build a linear regression model on a
discrete/binary y variable, then the linear regression model predicts negative
values for the corresponding response variable, which is inappropriate.
Components of G.L.M
[Link] Component/Probability Distribution:
It refers to the probability distribution, from the family of distributions, of the response
variable.
It specifies the probability distribution of the response variable; e.g., normal distribution
for 𝑌 in the classical regression model, or binomial distribution for 𝑌 in the binary logistic
regression model. This is the only random component in the model; there is not a separate
error term.
Specifies the probability distribution of the response variable Y, which comes from the
exponential family of distributions, such as:
Normal (Gaussian) → for continuous data (e.g., linear regression)
Binomial → for binary or proportion data (e.g., logistic regression)
Poisson → for count data (e.g., number of fraud cases)
[Link] Component/Linear Predictor -
It is just the linear combination of the Predictors and the regression coefficients.
It specifies the explanatory variables(x1,x2,....xk) in the model, more specifically, their
linear combination; e.g.(Y = β0 + β1X1 + β2X+....) as in a linear regression.
Defines the relationship between the predictor variables (X) and the response variable (Y)
using a linear combination:
Where:
X = predictor variable matrix
β = regression coefficient vector
η= linear predictor
η = Xβ
Link Function-
Transforms the expected value of the response variable, E(Y ) = μ, to the linear predictor:
g(μ) = η = Xβ
Identity function(μ = η)→ for normal distribution (linear regression)
Logit function log(μ/(1 − μ)) = η→ for binomial distribution (logistic regression)
Log function log(μ) = η→ for Poisson or gamma distributions
It specifies the link between the random and the systematic components.
It indicates how the expected value of the response relates to the linear combination of
explanatory variables.
probability Distribution Link Function
Normal Distribution Identity function
Binomial Distribution Logit/Sigmoid function
Poisson Distribution Log function (aka log-linear, log-link)
Simple Linear Regression, y= β0+β1X1
Multiple Linear Regression, y = β0+β1X1+β2X2
Binary Logistic Regression, for dichotomous or binary outcomes with
binomial distribution: Log odds= β0+β1X1+β2X2 Response variable has only 2
outcomes. Predictors can be continuous or categorical, and can also be
transformed.
Poisson Regression, for count based outcomes with poisson distribution:
Here count values are expressed as a linear combination of the explanatory
[Link] link is the link function.
log(λ)=β0+β1×1+β2×2, where λ is the average value of the count variableResponse
variable is a count value per unit of time and space Predictors can be continuous or
categorical, and can also be transformed.