Module - 3

Feature Generation & Feature Selection

Extracting Meaning from Data

How do companies extract meaning from the data they have?

 What is feature selection?

 Feature selection is nothing but selecting the optimal and best

features for ML model so that the model give good performance.

 “We give good and get good”.

 Garbage in and Garbage out”.

 Need for feature selection?

i) While developing a Machine learning model, only a few

variables in the data set are useful for building the model and
the rest of the features are either redundant or irrelevant.
ii) If we input the data set with all these redundant and irrelevant
features, it may negatively impact and reduce the overall
performance and accuracy of the model.
iii) Hence, it is very important to identify and select the most
appropriate features from the data and remove the irrelevant
and less important features, which is done with the help of
feature selection Machine Learning.
iv) Feature selection is one of the important concept of ML, which
highly impacts the performance of the model.
 Need for feature selection? (contd.)
 Example: We have student data set.

Features are nothing but columns(attributes). Using this table

our objective is to analyze the performance of the students based
on these features.
 Clearly, we understand that Race and Religion are something
that will not contribute towards the performance.
 We see that here, Race and Religion are irrelevant features.
 Need for feature selection? (contd.)

 This is a very simple data set.

 Now assume that I feed all the feature and train the model,
which may give 60% accuracy and suppose I train the model giving
only the best features and it will give 80% accuracy.
 This is just relevant and best features that the model give me
good performance.
 What is really happening in the background is, suppose we have
all the features in the columns, while selecting the features, we
are selecting only relevant and best features and we left with only
the final features here.
 By selecting the relevant features and training the model based
on those features, we get really very good result.
 Why feature selection?

 Take a model. One model take 10 features and another use 5

features and both of them have same performance, then we
should really use the model which has 5 features, because it is
simpler, simpler to maintain, simpler to train, and less prone to
outfitting.

 To improve the model performance by choosing the right

features.
 Importance of feature selection

 It reduce the course of dimensionality as you work only with

relevant and optimal features.
 Course of dimensionality means when we working with lot &
lots of features with lots of columns and rows, it becomes very
difficult and complex data sets. So, we remove the irrelevant
features and we reduce the dimensionality.
 Another thing is that it minimizes the cost of computation
because we are working with less features.
 Another important thing is it helps in learning the model by
providing the best features to the training algorithm.
 Another thing is that it helps in achieving good accuracy.
 When to do Feature Selection?
 It should be done before hyper parameter tuning and model
selection. The reason being that if you are tuning the model, you
should change the feature set recursively.
 This feature selection is to be made before tuning and selection
of the model.
Feature selection is a way of selecting subset of the most relevant
features from the original feature set by removing the redundant,
irrelevant or Noisy features.
 Advantages of feature selection

There are several advantages of performing feature selection

before training ML model.

(i) Model with less number of features have higher explainability

and interpretability.
(ii) It is easier to implement ML models with reduced feature
space.
(iii) Lesser number of features leads to enhanced generalization
which in turn reduce overfitting.
(iv) Feature selection removes data redundancy.
(v) Training time of the models with fewer features is significantly
lower.
(vi) Models with fewer features are less prone to errors.
 Various feature selection Statistics (contd.)

 If our input and out put are both categorical, then we use Chi-
square test or mutual information.
 These are the statistical measures that we use in our models.
Then we have different input and output variables.
 Suppose, our input variable is numeric and output variable is
categorical , we apply ANOVA (Analysis of Variance) or Kindles
rank coefficient. Also the same statistical measures are being
applied when your input variables are categorical and out put
variable is numeric. In both these cases
 Various feature selection Statistics
 We have suppose x & y axes. X-axis is input variable, and y-axis
is output variable.
 These input and output variables shall be either categorical or
numeric.
 1) Feature selection method using Variance threshold

 Variance is the measure of spread from the mean.

 If variance is very large, our data is spread out far from the
mean, if variance is very small (say near to zero), the data is
very close to each other and very close to mean.

 If variance is near to zero, then we have a feature with constant

value.

 If a feature has variance near to zero, then that is likely not

predictive. That feature can be removed from the data set.
 Filter method

We use measure of importance to select top K best features.

2) Feature selection method using mutual information method

 How much information do I know about my target by observing

the features.

 We calculate the importance of all the features and we are

going to tell that these are the top K-number are the highest
important features.
 Feature selection techniques

There are two types of feature selection techniques.

1) Supervised feature selection technique

Supervised feature selection technique consider the target
variable and can be used for the labeled dataset.

2) Unsupervised feature selection technique

Unsupervised feature selection technique ignore the target
variable and can be used for unlabelled dataset.
 1) Supervised feature selection method

Under this there are three methods

2) Filters method
3) Embedded method
4) Wrappers method
 Filter Method
i) Information gain
ii) Chi-Square test
iii) Fisher’s score
iv) Missing values

Embedded method
v) Regularization L1, L2
vi) Random forest importance

Wrappers method
vii)Forward feature selection
viii) Backward feature elimination
ix) Exhaustive feature selection
x) Recursive feature elimination
 1) Filter method
 Filter methods are generally used as a preprocessing step.
 The selection of feature does not depend on ML algorithms,
instead, features are selected on the basis of their scores in various
statistical tests for their correlation with the outcome variable.
 The filter method filters out the irrelevant features and
redundant columns from the model by using different metrics
through ranking.
 the advantage of using filters method is that it needs low
computational time and does not over fit the data.

Selecting the Learning Perfor

Set of all
best subset Algorithm mance
features
 Filter method
How does Filter method works?
 We have some set of features. Out of it we have to select some
best features.
 We feed these features to the learning algorithm and then we
map the performance of this algorithm. That is how the feature
selection works.
 This technique is really very simple to understand but what
really happens in filter method is that this is just a preprocessing
step. We are preparing the data before feeding it to the algorithm.
 When we prepare the data well, that is preprocessing is done,
we get really very good performance by the model and really we
get very good results.
 In filter method there are various methods like 1) Information
gain, 2) Chi-square test, 3) Fisher’s score, 4) Missing value ratio.
 Filter method (contd.)

Some of the common techniques of filter method are as follows:

i) Information gain
ii) Chi-square test
iii) Fisher’s score
iv) Missing value ratio.
 1) Information Gain

 Information Gain feature selection is a method used for

selecting the best features from feature space.

 we will select highest information gain as the next split

attribute in the construction of the decision tree.

 When training a Decision Tree, we use information gain as a

metric for best split and hence construct the decision tree.
 Filters Method
Chi-Square test

We calculate Chi-square between each feature and the target.

 Select the desired number of features with best Chi-square
scores.
 Higher the Chi-square value , the more desired the feature will
be. That means the feature is more stronger in predicting the
outcome.
 When the Chi-square value is smaller then the feature is a weak
feature and the target value is less dependent on it.
 When to use Chi-Square test

 Developed by Pearson.
 This test determines the relationship between categorical
variables.
 This test is appropriate to use when
i) you want to test the hypothesis about one or more categorical
variables.
Ii) The sample was randomly selected from the population.
Iii) There are a minimum of 5 observations in each group or
combination of groups.
 It allows you to test when the frequency distribution is
significantly different from your expectations.
 Procedure to perform Chi-square test

1) Create a table of Observed and Expected frequencies.

2) Calculate the Chi-square value using the formula,

Where O is the observed frequency and

E is the expected frequency.
3) Establish the Hypothesis. H0 : for null hypothesis,

HA: for alternative hypothesis.

4) Find the critical value from chi-square critical table.
5) If Chi-square value > Chi-square critical value, Reject the null
hypothesis.
 Example On Chi-Square test

The table given below give the relationship between handedness

and gender in USA. This data is also known HANES data.
 Example: On Chi-Square test (contd.)

Step-1: Calculate expected value.

 Example: On Chi-Square test (contd.)
Step-2: Calculate Chi-Square value.
 Example: On Chi-Square test (contd.)

Step-3: Define the hypothesis.

Ho: Two categorical variables are not related in the population.

HA: The two categorical variables are related in the population.

With reference to the example,

H0: Handedness and gender are independent.

HA: Handedness and gender are not independent.
Step-4: Find the degrees of freedom
Foe categorical data,
Df= (r-1)*(c-1)=(3-1)*(2-1)=2
Chi-squared value=12
Degrees of freedom=2
Chi-square critical value= 5.991 when α=0.05

12> 5.991

The Null hypothesis is rejected.

It means, Based on the data,
the handedness and gender are not independent.
 3) Fisher’s Score
It returns the rank of the feature in decreasing order.
 It works on categorical data set.
It is one of the most widely used supervised feature selection
method.
 In Fisher's Score method of feature selection, compute Fisher
score for each feature and rank them according to their scores.
Estimate average Fisher score value called threshold value. The
feature value which is greater than the threshold value is selected.
The feature values below the mean F-Score are rejected.
 4) Missing value ratio

Missing Values Ratio is a feature selection technique used in

machine learning to identify and remove features from the dataset
that have a high percentage of missing values.
 2) Wrapper method

 In wrapper method, we try to use a subset of features and train

a model using them.
In this method, different combinations of features are made,
evaluated and compared with other combinations.
 On the basis of the output of the model, features are added or
removed, and with this feature set the model has trained again .
This methods are usually computationally very expensive.
 Wrapper method (contd.)

Some common techniques of wrapper method are as follows:

i) Forward feature selection

ii) Backward feature elimination
iii) Exhaustive feature selection
iv) Recursive feature elimination
 Wrapper method
1) Forward feature selection

(i) Starting from Scratch:

Begin with an empty set of features and iteratively add one
feature at a time.
(ii) Model Evaluation:
At each step, train and evaluate the machine learning model
using the selected features.
(iii) Stopping Criterion:
Continue until a predefined stopping criterion is met, such as a
maximum number of features or a significant drop in
performance.
 Wrapper method
2) Backward feature elimination

i) Starting with Everything:

Start with all available features.
ii) Iterative Removal:
In each iteration, remove the least important feature and
evaluate the model.
iii) Stopping Criterion:
Continue until a stopping condition is met.
 Wrapper method
3) Exhaustive feature Search

(i) Exploring All Possibilities:

Evaluate all possible combinations of features, which ensures
finding the best subset for model performance.

(ii) Computational Cost:

This can be computationally expensive, especially when the
feature space is very large.
 Wrapper method
4) Recursive feature elimination (RFE)
First decided how many features you want keep in the model.
(i) Ranking Features: Start with all features and rank them based on their
importance or contribution to the model.
(ii) Iterative Removal: In each iteration, remove the least important
feature(s).
(iii) Stopping Criterion: Continue until a desired number of features is
reached. Then we are going to check “Is the number of features
greater than the threshold say K=5. If the number of features is
greater than 5, again repeat the process by removing the weakest
feature.
Yes

All No. of
Remove
feat features No
weakest Done
ure >K
feature
s
 Feature selection using Recursive Feature Elimination
 Once we have the importance of each feature, we perform
feature selection using a procedure called Recursive Feature
Elimination. In this article, I’ll talk about the version that makes
use of the k-fold cross-validation.
 The idea is to fit the model, then remove the less relevant
feature and calculate the average value of some performance
metric in CV. Then we remove the second last important feature,
fit the model again and calculate the average performance. We
keep doing this approach until there are no features left. The set
of features that maximize the performance in CV is the set of
features we have to work with. Please note that the entire
procedure needs to work with the same values for the
hyperparameters.
 Advantages of Wrapper Methods
Wrapper methods offer several advantages:
1) Model-Specific Optimization: Wrapper methods are tailored to
the machine learning model they are optimizing, allowing them to
capture model-specific nuances and interactions among features.
2) Effective for Complex Models: They can be particularly useful
when working with complex models that exhibit non-linear
behaviour or intricate feature dependencies.
3) Feature Interaction: Wrapper methods can capture interactions
among features, which may not be evident through other feature
selection techniques like filter methods.
4) Performance Guarantee: Exhaustive search, though
computationally expensive, guarantees to find the best subset of
features in terms of model performance.
 Challenges in Wrapper method
Following are some of the challenges of wrapper methods:
1)Computational Cost: Some wrapper methods, especially
exhaustive search, can be computationally expensive, limiting
their applicability to large datasets.
2)Overfitting Risk: Without proper cross-validation and
regularization, wrapper methods may lead to overfitting the
model to the selected subset of features.
3) Model Choice: The choice of machine learning algorithm within
the wrapper can impact the results, so it is essential to consider
different models and their compatibility with the feature selection
process.
4)Data Quality: Wrapper methods rely heavily on the quality of
the dataset. No amount of feature selection can compensate for
poorly collected or noisy data.
 Conclusion
i) Wrapper methods in machine learning provide a powerful
framework for feature selection by optimizing a model’s
performance through the systematic evaluation of feature
subsets. They are particularly valuable when working with
complex models and when feature interactions play a crucial
role in the predictive task.
ii) However, wrapper methods should be used judiciously,
taking into account computational resources, the choice of
machine learning algorithm, and the quality of the dataset.
When employed wisely, wrapper methods can help enhance
model accuracy, reduce overfitting, and ultimately improve
the utility of machine learning models in solving real-world
problems.
 3) Embedded method
Embedded method combines the qualities of Filter and Wrapper
methods.
 It is implemented by algorithms that have their own built-in
feature selections.
 Embedded method combine the advantages of both Filter and
Wrapper methods by combining the interaction of features
along with low computation cost.
 These methods are also iterative, which evaluates each iteration
and optimally finds the most important features that contribute
the most to training in a particular iteration.
Selection of Best subset

Generate Learning algorithm

Set of all +
the subset
features Performance
 Decision Tree
Decision tree is a supervised learning technique that can be
used for both classification and regression problems, but
mostly it is preferred for solving classification problems.
Decision tree can be used for important feature selection of a
data set.
 It is a graphical representation of decision making process used
for feature selection.
 When we build a decision tree, some of the features of the data
set are going to be used in the decision nodes.
 The features used in the decision nodes can be thought of more
descriptive of the data sets.
 Features used in the decision nodes closer to the root are more
important than the lower in the tree.
 In order to build a tree we use CART (classification and
Regression Tree)algorithm.
 Decision Tree (contd. )

 A decision tree simply asks a question, and based on the

answer (Yes/No), it further split the tree into sub trees.
 Decision tree terminologies
Root node- It is a base node of a tree. The entire tree starts from a
root node. It is the first node of the tree. It represent he entire
population or sample or dataset.
Leaf node- Present at the end of the tree. These are the final
output nodes.
Splitting- the process of dividing the node into different parts on
the basis of some condition.
Branch- a sub tree formed by splitting the node.
Pruning- is the process of removing the unwanted branches from
the tree.
Parent/Child node- The root node of the tree is called the parent
node, and all other nodes branched from it is known as child
node.
 What are decision trees ?

Decision trees are a popular machine learning algorithm used for

both classification and regression tasks. They model decisions
based on the features of the data and their outcomes.
 How do decision trees play a role in feature selection?

 Decision trees select the ‘best’ feature for splitting at each node
based on information gain.
 Information gain measures the reduction in entropy(disorder) in
a set of data points.
 Features with higher information gain are considered more
important for splitting, thus aiding in feature selection.
 By recursively selecting features for splitting, decision trees
inherently prioritize the most relevant features for the model.
 Feature selection using Decision Tree
Feature selection using decision trees involves identifying the most
important features in a dataset based on their contribution to the
decision tree’s performance.
 Implementation: Feature Selection using Decision Tree

In this implementation, we are going to discuss a practical

approach to feature selection using decision trees, allowing for
more efficient and interpretable models by focusing on the most
relevant features. You can download the dataset from here.

Step 1: Importing Libraries

Step 2: Dataset Description
Step 3: Data Pre-processing
Step 4: Splitting the data
 Implementation: Feature Selection using Decision Tree (contd.)
Step 5: Scaling the data
Step 6: Training the Decision Tree Classifier
The DecisionTreeClassifier is trained with a maximum depth of 16
and a random state of 8, which helps control the randomness for
reproducibility.
Step 7: Feature selection
 The feature importance are calculated using the trained
classifier, indicating the relative importance of each feature in the
model’s decision-making process.
A threshold of 0.1 is set to select features with importance
greater than this value, potentially reducing the number of features
considered for the final model.
The selected features variable contains the names of the features
that meet the importance threshold, which can be used for further
analysis or model refinement.
 Implementation: Feature Selection using Decision Tree (contd.)
Step 8: Train a model using the selected features
Step 9: Comparing the accuracies
Output: 79% & 82%
Accuracy with all features: 0.7983333333333333
Accuracy with selected features: 0.8241666666666667
These accuracy scores provide insights into the performance of the
models. The accuracy score represents the proportion of correctly
classified instances out of the total instances in the test set.

Comparing the two accuracies:

The model trained with all features achieved an accuracy of
approximately 79.83%.
However, after feature selection, the model trained with selected
features achieved a higher accuracy of approximately 82.42%.
 Conclusion

Feature selection using decision trees offers a powerful and

intuitive approach to enhancing model performance and
interpretability. Following the outlined steps, we can easily select
features using decision trees to build more robust and efficient
models for various applications.
 CART Algorithm to construct decision tree

 Determine the attribute that best classifies the training data.

 Attribute with highest Information gain is considered to be best
feature.
How do you choose best attribute? Or How does the tree decide
where to split?
 It split based on Gini Index.
What is Gini Index?
Gini index measure the impurity or purity used in building decision
tree using CART algorithm.
What is information gain
Information gain is the decrease in entropy after a data set is split
on the basis of an attribute. Constructing a decision tree is all about
finding attribute that returns the highest Information gain. We are
selecting the node which give highest information gain.
 Reduction in Variance
 It is used in continuous target variables(regression problems).
 Lower variance is selected as the criteria to split the population.
 If he data is more pure then variance is less.

Chi-square value
It is an algorithm to find the statistical significance between the
difference between sub-nodes and parent nodes.
 Entropy
 Entropy is a metric used for calculating the Information Gain.
 Entropy is a metric which measure the impurity of a attribute.

Entropy(S)= -P(Yes)*log2P(Yes) –P(No)*log2P(No)

Where S: Sample space,
P(Yes):Probability of Yes, and
P(No): Probability of Nos

Note:
1) In the data, If number of YES = Number of No’s then P(S)=0.5.
2) In the data, if it contains all YES then P(S)= 0 or 1
3) In the data, if it contains all No then P(S)= 0 or 1.
4) All YES/NO means the data is highly pure or highly impure.
 Entropy
1) In the data, If number of YES = Number of No’s then P(S)=0.5.
2) In the data, if it contains all YES then P(S)= 0 or 1
3) In the data, if it contains all No then P(S)= 0 or 1.
 Impurity
 It is the degree of randomness.
 If the data is completely pure the value of purity os ZERO.
 If he data is completely impure then value of impurity is ZERO.
 The data set is divided into 2 parts YES & NO.
(I) When YES+NO=Total sample space, then
E(S) = -0.5 log20.5 - 0.5 log20.5 =1
(II) When YES = Total number of samples, P(Yes)=1
E(S) = -1* log21 – 1* log21 = 0
(III) When NO = Total number of samples, P(NO)=1
E(S) = -1* log21 – 1* log21 = 0
 Build a decision tree
Example:
Example(contd.)
Example (contd.)
Example(contd.)
Example(contd.)
Example(contd.)
 Pruning
 Pruning is nothing but cutting down the nodes in order to get
the optimal solution.
 Pruning reduces the complexity.
 Is tree based model better than Linear model?
If we are using Logistic regression for classification problem and
Linear regression for regression problem then why I should a
Tree?.
Ans: We can use any algorithm , it depends on the type of
problem you are solving.
(i) If the relationship between dependent and Independent
variable is well approximated by a linear model, then Linear
regression will out perform Tree based model.
(ii) If here is a high non-linearity and complex relationship
between dependent and independent variables, a Tree model
will out perform a classical regression model. A decision tree is
always do better than a linear model.
 Random Forest
What is a Random Forest?

 Random Forest is a supervised model that implements both

decision trees and bagging method.
 The idea is that the training dataset is resampled according to a
procedure called “bootstrap”.
 Each sample contains a random subset of the original columns
and is used to fit a decision tree.
 The number of models and the number of columns are
hyperparameters to be optimized.
 Random Forest
What is a Random Forest? (contd.)

 Finally, the predictions of the trees are mixed together

calculating the mean value (for regression) or using soft voting
(for classification).

 The idea of bagging is that, by averaging the outputs of the

single decision trees, the standard error decreases and so does
the variance of the model according to bias-variance trade-off.
That’s why Random Forest has become very famous in the last
years.
Variance
Variance, on the other hand, is the error due to the model’s
sensitivity to fluctuations in the training data. It’s the variability of
the model’s predictions for different instances of training data.
High variance occurs when a model learns the training data’s noise
and random fluctuations rather than the underlying pattern. As a
result, the model performs well on the training data but poorly on
the testing data, indicating overfitting.
 What is Bias?
Bias refers to the error due to overly simplistic assumptions in the
learning algorithm. These assumptions make the model easier to
comprehend and learn but might not capture the underlying
complexities of the data.
It is the error due to the model’s inability to represent the true
relationship between input and output accurately. When a model
has poor performance both on the training and testing data means
high bias because of the simple model, indicating underfitting.
A statistical model or a machine learning algorithm is said to have
underfitting when a model is too simple to capture data
complexities. It represents the inability of the model to learn the
training data effectively result in poor performance both on the
training and testing data. In simple terms, an underfit model’s are
inaccurate, especially when applied to new, unseen examples. It
mainly happens when we uses very simple model with overly
simplified assumptions. To address underfitting problem of the
model, we need to use more complex models, with enhanced
feature representation, and less regularization.
Note: The underfitting model has High bias and low variance.
 Reasons for Underfitting
The model is too simple, So it may be not capable to represent the
complexities in the data.
The input features which is used to train the model is not the
adequate representations of underlying factors influencing the
target variable. The size of the training dataset used is not
enough. Excessive regularization are used to prevent the
overfitting, which constraint the model to capture the data well.
Features are not scaled.

Techniques to Reduce Underfitting

i) Increase model complexity.
ii) Increase the number of features, performing feature
engineering.
iii) Remove noise from the data.
iv) Increase the number of epochs or increase the duration of
training to get better results.
 Overfitting in Machine Learning
A statistical model is said to be overfitted when the model does not make
accurate predictions on testing data. When a model gets trained with so much
data, it starts learning from the noise and inaccurate data entries in our data set.
And when testing with test data results in High variance. Then the model does
not categorize the data correctly, because of too many details and noise. The
causes of overfitting are the non-parametric and non-linear methods because
these types of machine learning algorithms have more freedom in building the
model based on the dataset and therefore they can really build unrealistic
models. A solution to avoid overfitting is using a linear algorithm if we have
linear data or using the parameters like the maximal depth if we are using
decision trees.

In a nutshell, overfitting is a problem where the evaluation of machine learning

algorithms on training data is different from unseen data.
Reasons for Overfitting:
High variance and low bias.
The model is too complex.
The size of the training data.
 Techniques to Reduce Overfitting

 Improving the quality of training data reduces overfitting by

focusing on meaningful patterns, mitigate the risk of fitting the
noise or irrelevant features.
 Increase the training data can improve the model’s ability to
generalize to unseen data and reduce the likelihood of
overfitting.
 Reduce model complexity.
 Good Fit in a Statistical Model
Ideally, the case when the model makes the predictions with 0
error, is said to have a good fit on the data. This situation is
achievable at a spot between overfitting and underfitting. In order
to understand it, we will have to look at the performance of our
model with the passage of time, while it is learning from the
training dataset.
With the passage of time, our model will keep on learning, and
thus the error for the model on the training and testing data will
keep on decreasing. If it will learn for too long, the model will
become more prone to overfitting due to the presence of noise
and less useful details. Hence the performance of our model will
decrease. In order to get a good fit, we will stop at a point just
before where the error starts increasing. At this point, the model is
said to have good skills in training datasets as well as our unseen
testing dataset.
 Random Forest
 It is an ensemble classifier made using many decision tree
models.
 Ensemble models combine the results from different models.
 Ensemble models use multiple machine learning algorithms to
obtain better predictive performance.
 When we are talking on Random forest, Random forest uses
multiple decision trees for prediction. So, we ensembling a lot of
decision trees to come up to our final outcome.
For example, in the figure,
 The entire data set is spit into 3 subsets, and each subset leads to
a particular decision tree.
 We have three decision trees and each decision tree will lead to
certain outcome.
 What random forest will do is, it will compile the results from all
the decision trees and then it will lead to a final outcome.
 So, it is the compilation of all the multiple decision trees.
 What is Random Forest?

 It is a collection of decision trees.

 Decision trees are built considering the whole data set and
considering all features.
 In random forest only a section of rows is selected at random
and a particular number of features which are selected at
random and trained upon. That is the decision trees are built
upon.
 The number of decision trees will be grown and each decision
tree results into certain out come.
 Random forest will compile the results of all those decision trees
to bring up the final result.
What is Random Forest?
 What is Random Forest? (contd.)
 In this figure, a particular instance actually has resulted in to three
different decision trees.
 Tree-1 results into a final outcome called class-A.
Tree-2 result into a final outcome called class-B, and finally
Tree-3 result into a final outcome called class-B.
 Random forest compile all these decision trees and it will go by the
majority voting.
 Here actually two decision trees are voted into favour of class-B,
therefore the final outcome is Class-B. That is how random forest
actually works.
How Random forest algorithm works
 How Random forest algorithm works
T is the total number of predictive variables that you have in your
data set.
M is the number of features selected.
D is the number of decision trees to be constructed.
V; Output: The class with the highest vote.
1) Out of T we select m features.
2) You select few features randomly and train the model. The
model will learn each time.
3) Model learn from all its previous experiences and now it is
more intelligent.
4) Construct many decision trees.
5) compile all the decision trees and get the result.
 Recommendation Engine(Recommendation system)

Recommendation engines are used for recommending the items

for users (customers).
(i) It recommend what movies would you like by knowing other
movies you liked?.
(ii) It recommend what book would you like by knowing your past
purchases?.
(iii) It recommend what kind of vacation you like knowing the past
vacation trips?
Building such a model is known as Recommendation Engine.
 Recommendation Engine(Recommendation system) (contd.)
To setup a recommendation engine, model consider the following
mapping .
Suppose U be the set of users and V be the set of items to
recommend. The edges could have assigned with positive or
negative weights.
The following bipartite graph shows users & items(TV shows) as
nodes.
You have some training data of past experience, using that data set,
you want to predict other performances for your users. That is
what essentially the Recommendation Engine is?.
 Algorithmic ingredients of a recommendation engine
1) Accuracy
The recommendation Engine would predict a given user’s rating
based on the movies previously watched and rated. It is important
to constantly monitor the accuracy of each result, as the algorithm
aims to predict ratings as close as possible to real ratings.
2) Coverage
The coverage represents the percentage of all the items that may
be recommended.
3) Popularity
Popularity refers to the percentage of times that a product was
bought or a content was consumed among all items. The greater,
the most popular.
 Algorithmic ingredients of a recommendation engine

4) Personalization
Each user should be suggested customized recommendations
depending on their preferences to optimize the engagement of the
users. The range of recommendations among the users reflects the
diversity of taste among the population.

5) Diversity
For example, You have purchased a pair of shoe over an e-
commerce website. On that sight login, you had a quick look at a
pair of shoes, and had recommended to you over and over again.
Instead, Amazon limit the number of items from the same
category, and recommend alternate categories like "shoe",
"dress", "shoe" etc.
 Dimensionality reduction

What is Predictive Modeling?

Predictive modeling is a probabilistic process that allows to

forecast outcomes, on the basis of some predictors.

These predictors are basically the features which play an

important role in deciding the outcome of the model.
 Dimensionality reduction
 Dimensionality reduction is a technique used to reduce the
number of features in a dataset by retaining the
important information of the original data. In other words, it is

a process of transforming high-dimensional data into a lower-

dimensional space preserving the essence of the original data.

In data science, high-dimensional data refers to data with a

large number of features or variables.
 Dimensionality reduction (contd.)
The advantages of dimensionality reduction are:
i) The complexity of the model increases with the increase of
number of features. In turn, the performance of the model
deteriorates as the number of features increases, and it
becomes more difficult to find a good solution.
ii) The high-dimensional data can also lead to model overfitting.
iii) The dimesionality reduction help in reducing the complexity
of the model and improving its performance of the model.
iv) It make easier to visualize the data.
Feature Selection:
Feature selection involves selecting a subset of the original
features that are most relevant to the problem at hand. The goal
is to reduce the dimensionality of the dataset while retaining the
most important features. There are several methods for feature
selection, including filter methods, wrapper methods, and
embedded methods. Filter methods rank the features based on
their relevance to the target variable, wrapper methods use the
model performance as the criteria for selecting features, and
embedded methods combine feature selection with the model
training process.
 Dimensionality reduction

Important techniques of dimensionality reduction are:

1) Singular value decomposition(SVD)

2) Principle component analysis (PCA)

 Principal Component Analysis(PCA)
PCA is a technique for reducing the dimensionality of datasets,
increasing interpretability with out information loss.
 Terminologies used in PCA
Views: The perspective through which data points are observed.
Dimension: Number of columns in a dataset are called the
dimension of that dataset.
Principal Component: New variables that are constructed as
linear combinations or mixtures of the initial variables.
Projections: The perpendicular distance between the principal
component and the data points.
 Important properties of PCA
1) Number of principal components is always less than or equal to
the number of attributes.
Example: From the table, Number of Principal Components<=3
 Important properties of PCA(contd.)

2) Principal components are orthogonal.

 Important properties of PCA(contd.)
3) The priority of principal components decreases as their numbers
increases. It means
If we are going to create one principal component, then everything
is summarised into one component.
If we are creating two principal component s then its priority
decreases because the value second component holds, decreases.
 How PCA works?
PCA perform the following operations in order to evaluate the
principal components for a given dataset.
1) Standardisation
2) Covariance matrix computation
3) Eigen values and Eigen vectors estimation
4) Feature vector.
 How PCA works? (contd.)
1) Standardisation

 The main aim of this step is to standardise the range of the

attributes so that each one of them lie within similar boundaries.
 This process involves removal of mean from the variable values
and scaling the data with respect to the Standard Deviation.
 This is done by using the formula,

Z=(Variable value - Mean)/ Standard Deviation

 How PCA works? (contd.)
2) Covariance Matrics computation
Covariance matrix is used to express the correlation between any
two or more attributes in a multidimensional dataset.
 How PCA works? (contd.)
Covariance matrics for more than two attributes in a
multidimensional dataset.
 Positive covariance indicate that the values of one variable is
directly proportional the other variable.
 Negative covariance indicate that the values of one variable is
inversely proportional to the other variable.
 3) Estimation of Eigen values and Eigen vectors
 [Link] and [Link] are the mathematical values that are extracted
from the covariance matrix.
 They are responsible for the generation of new set of variables
from the old set of variables which further leads to the
construction principal components.
 Eigen vectors do not change their directions after linear
transformations.
 Eigen values are the magnitude of the eigen vector.
 4) Feature vector
 It is a matrics that has Eigen vectors of the components that we
decide to keep as the columns.
 Here, we decide whether we must keep or discard the less
significant principal components that we have generated in the
above steps.