Feature Generation and
Feature Selection
Prepared by:
Sadaqat Ali Khan Bangash
Feature Engineering
• Feature engineering is the pre-processing step of machine learning,
which is used to transform raw data into features that can be used for
creating a predictive model using Machine learning or statistical
Modelling.
• Feature engineering in machine learning aims to improve the
performance of models. In this topic, we will understand the details
about feature engineering in Machine learning.
• But before going into details, let's first understand what features are?
And What is the need for feature engineering?
What is a feature?
• Generally, all machine learning algorithms take input data to generate the
output.
• The input data remains in a tabular form consisting of rows (instances or
observations) and columns (variable or attributes), and these attributes are
often known as features.
• For example, an image is an instance in computer vision, but a line in the
image could be the feature.
• Similarly, in NLP, a document can be an observation, and the word count
could be the feature.
• So, we can say a feature is an attribute that impacts a problem or is useful
for the problem.
Feature Engineering Process in Machine Learning
1. Feature Generation/Feature Creation
• Feature generation is an essential step in machine learning and data
analysis.
• They involve preparing and refining the input data (features) to improve
the performance of a machine learning model.
• Feature generation involves creating new features from the existing data to
provide additional information that might be useful for a machine learning
model.
• The new features are created by mixing existing features using addition,
subtraction, and ration, and these new features have great flexibility.
…Continue
2. Transformations
• The transformation step of feature engineering involves adjusting the
predictor variable to improve the accuracy and performance of the
model.
• For example, it ensures that the model is flexible to take input of the
variety of data; it ensures that all the variables are on the same scale,
making the model easier to understand.
• It improves the model's accuracy and ensures that all the features are
within the acceptable range to avoid any computational error.
…Continue
3. Feature Extraction
• Feature extraction is an automated feature engineering process that
generates new variables by extracting them from the raw data.
• The main aim of this step is to reduce the volume of data so that it
can be easily used and managed for data modelling.
• Feature extraction methods include cluster analysis, text analytics,
edge detection algorithms, and principal components analysis (PCA).
…Continue
4. Feature Selection
• While developing the machine learning model, only a few variables in the dataset
are useful for building the model, and the rest features are either redundant or
irrelevant.
• If we input the dataset with all these redundant and irrelevant features, it may
negatively impact and reduce the overall performance and accuracy of the
model.
• Hence it is very important to identify and select the most appropriate features
from the data and remove the irrelevant or less important features, which is done
with the help of feature selection in machine learning.
• "Feature selection is a way of selecting the subset of the most relevant features
from the original features set by removing the redundant, irrelevant, or noisy
features."
Feature Selection Techniques/Feature Selection Algorithms
1. Filter Methods
• These methods rank features based on their statistical properties and
are independent of the machine learning algorithm you plan to use.
Common filter methods include:
• Chi-squared Test: Measures the independence between features and
the target variable for classification tasks.
• Information Gain / Mutual Information: Measures the amount of
information one feature provides about the other.
• Correlation Coefficient: Measures the linear relationship between
features.
…Continue
2. Wrapper Methods:
• These methods evaluate the performance of a machine learning
model using different subsets of features. Examples include:
• Recursive Feature Elimination (RFE): Starts with all features, trains
the model, and iteratively removes the least important feature until a
stopping criterion is met.
• Forward Selection: Starts with no features and adds one feature at a
time, choosing the one that provides the best improvement in model
performance.
…Continue
3. Embedded Methods:
• These methods incorporate feature selection within the model
training process itself. For example:
• LASSO (Least Absolute Shrinkage and Selection Operator): Penalizes
the absolute values of the feature coefficients, encouraging some
features to be exactly zero.
• Random Forest Feature Importance: Uses a Random Forest algorithm
to estimate feature importance by measuring how much a feature's
inclusion in decision trees decreases the impurity.
Have A Nice Day