Step by Step process of Feature Engineering
Introduction
Data Science is not a field where theoretical understanding helps you to start a
carrier. It totally depends on the projects you do and the practice you have done
that determines your probability of success. Feature engineering is a very important
aspect of machine learning and data science and should never be ignored. The
main goal of Feature engineering is to get the best results from the algorithms.
1. Why should we use Feature Engineering in data
science?
In Data Science, the performance of the model is depending on data preprocessing
and data handling. Suppose if we build a model without Handling data, we got an
accuracy of around 70%. By applying the Feature engineering on the same model
there is a chance to increase the performance from 70% to more.
Simply, by using Feature Engineering we improve the performance of the model.
2. Feature selection
Feature selection is nothing but a selection of required independent features.
Selecting the important independent features which have more relation with the
dependent feature will help to build a good model. There are some methods for
feature selection:
2.1 Correlation Matrix with Heatmap
Heatmap is a graphical representation of 2D (two-dimensional) data. Each data
value represents in a matrix.
Firstly, plot the pair plot between all independent features and dependent features.
It will give the relation between dependent and independent features. The relation
between the independent feature and the dependent feature is less than 0.2 then
choose that independent feature for building a model.
2.2 Univariate Selection
In this, Statistical tests can be used to select the independent features which have
the strongest relationship with the dependent feature. SelectKBest method can be
used with a suite of different statistical tests to select a specific number of features.
Which feature has the highest score will be more related to the dependent feature
and choose those features for the model.
2.3 ExtraTreesClassifier method
In this method, the ExtraTreesClassifier method will help to give the importance of
each independent feature with a dependent feature. Feature importance will give
you a score for each feature of your data, the higher the score more important or
relevant to the feature towards your output variable.
3. Handling Missing Values
In some datasets, we got the NA values in features. It is nothing but missing data.
By handling this type of data there are many ways:
In the missing value places, to replace the missing values with mean or median to
numerical data and for categorical data with mode.
Drop NA values entire rows.
Drop NA values entire features. (it helps if NA values are more than 50% in a
feature)
Replace NA values with 0.
If you choose to drop options, there is a chance to lose important information from
them. So better to choose to replace options.
4. Handling imbalanced data
Why need to handle imbalanced data? Because of to reduce overfitting and
underfitting problem.
suppose a feature has a factor level2(0 and 1). it consists of 1’s is 5% and 0’s is
95%. It is called imbalanced data.
Example:-
By preventing this problem there are some methods:
4.1 Under-sampling majority class
Under-sampling the majority class will resample the majority class points in the data
to make them equal to the minority class.
4.2 Over Sampling Minority class by duplication
Oversampling minority class will resample the minority class points in the data to
make them equal to the majority class.
4.3 Over Sampling minority class using Synthetic Minority Oversampling
Technique (SMOTE)
In this method, synthetic samples are generated for the minority class and equal to
the majority class.
5. Handling outliers
firstly, calculate the skewness of the features and check whether they are positively
skewed, negatively skewed, or normally skewed. Another method is to plot the
boxplot to features and check if any values are out of bounds or not. if there, they
are called outliers.
how to handle these outliers: –
first, calculate quantile values at 25% and 75%.
next, calculate the Interquartile range
IQR = Q3 – Q1
next, calculate the upper extreme and lower extreme values
lower extreme=Q1 – 1.5 * IQR
upper extreme=Q3– 1.5 * IQRe
lastly, check the values will lie above the upper extreme or below the lower
extreme. if it presents then remove them or replace them with mean, median, or any
quantile values.
Replace outliers with mean
Replace outliers with quantile values
Drop outliers
6. Binning
Binning is nothing but any data value within the range is made to fit into the bin. It is
important in your data exploration activity. We typically use it to transform
continuous variables into discrete ones.
Suppose if we have AGE feature in continuous and we need to divide the age into
groups as a feature then it will be useful.
7. Encoding:
Why this will apply? because in datasets we may contain object datatypes. for
building a model we need to have all features are in integer datatypes. so, Label
Encoder and OneHotEncoder are used to convert object datatype to integer
datatype.
Label Encoding
Before applying Label Encoding
After applying label encoding then apply the column transformer method to convert
labels to 0 and 1
One Hot Encoding:
By applying get_dummies we convert directly categorical to numerical
8. Feature scaling
Why this scaling is applying? because to reduce the variance effect and to
overcome the fitting problem. there are two types of scaling methods:
8.1 Standardization
When this method is used?. when all features are having high values, not 0 and 1.
It is a technique to standardize the independent features that present in a fixed
range to bring all values to the same magnitudes.
In standardization, the mean of the independent features is 0 and the standard
deviation is 1.
Method 1:
Method2:
After encoding feature labels are in 0 and 1. This may affect standardization. To
overcome this, we use Normalization.
8.2 Normalisation
Normalization also makes the training process less sensitive by the scale of the
features. This results in getting better coefficients after training.
Method 1: -MinMaxScaler
It is a method to rescales the feature to a hard and fast range of [0,1] by subtracting
the minimum value of the feature then dividing by the range.
Method 2: – Mean Normalization
It is a method to rescales the feature to a hard and fast range of [-1,1] with mean=0.
Feature Engineering with the help
of Data Visualization
Why Feature Engineering is important?
The features in your data will directly influence the
accuracy of your model. The better features gives good
accuracy on your test data. The better the features that you
choose, the better the results you will achieve.
Feature Reduction In ML
The General view is adding more features increases the
model/classifier’s performance.
Let us see why this is not the case:
Curse of Dimensionality:
Choosing more number of features will usually degrade
classifier’s performance.
CURSE OF DIMENSIONALITY
Limited training data
Limited Computational Resources
addition of irrelevant/unnecessary features leads to
increase in computational costs and decrease in
performance.
Feature Selection:
Having n initial features, it is possible to select (2^n )
combinations of features(2^n subsets are possible).
We just can’t go over all the possible (2^n) subsets.
Feature Selection is an optimization problem
Search the space of possible subsets.
Pick the one which best suits.
Evaluating the Feature Subset
Supervised Learning:
We evaluate using the estimated error in validation set when
trained on the selected subset.
Unsupervised Learning:
Look at the input and select the subset that has more
information.
Selecting the subset
Select all relevant and uncorrelated features.
Estimate classification/regression error by adding each
feature and select the one’s that gives ,maximum
improvement in validation error.
Drop all the features that have small impact on the error.
Univariate methods for selecting features
Pearson Correlation coefficient
ranges from -1 to +1 , if the value is near to +1 two features
are highly correlated so any one of the features can be
removed. if it is nearer to -1 two features are negatively high
correlated.
Let’s plot the correlation matrix using seaborn
import seaborn as sns
f, ax = [Link](figsize=(10,10))
[Link]([Link](), annot=True ,linewidth=0.5,
fmt='.1f',ax=ax);
If any two variables are having correlation coefficient over
0.7 , we can say both the features are highly correlated and
one can be removed.
We can check correlation between any two features using
Joint plot
[Link]([Link][:,'pressure'],
[Link][:,'grip_angle'],
kind ='regg',
color= 'blue'
);
looking at the plot one can say they are not well correlated,if
they are correlated the data points would have been well
classified as below.
Few more plots can be used to find if feature can well
classify the data by standardizing it.
1. Box-Plot
data1 = [Link](['state','timestamp'],axis =1)
data_std = (data1 - [Link]()) / [Link]()
data1 = [Link](data1,id_vars ="status",var_name =
"features",value_name="value")
[Link](figsize=(5,5))
[Link](x="features",y="value",hue="status",data = data1)
[Link](rotation=45);
2)Swarm-plot
[Link](style='whitegrid',palette='muted')
data =
train_data.drop(['Name','Ticket','Embarked','Sex','Cabin'],axis=1)
data_std = (data - [Link]()) / [Link]()
data = data_std
data = [Link](data,id_vars ="Survived",var_name =
"features",value_name="value")
[Link](figsize=(10,10))
[Link](figsize=(10,10))
[Link](x="features",y="value",hue="Survived",data = data)
[Link](rotation=45);
3)Violin-Plot
data =
train_data.drop(['Name','Ticket','Embarked','Sex','Cabin'],axis=1)
data_std = (data - [Link]()) / [Link]()
data = data_std
data = [Link](data,id_vars ="Survived",var_name =
"features",value_name="value")
[Link](figsize=(10,10))
[Link](x="features",y="value",hue="Survived",data = data)
[Link](rotation=60);