Random Forest Algorithm

Random forest, a popular machine learning algorithm developed by Leo Breiman and

Adele Cutler, merges the outputs of numerous decision trees to produce a single

outcome. Its popularity stems from its user-friendliness and versatility, making it

suitable for both classification and regression tasks.

Its widespread popularity stems from its user-friendly nature and adaptability, enabling it

to tackle both classification and regression problems effectively. The algorithm’s

strength lies in its ability to handle complex datasets and mitigate overfitting, making it a

valuable tool for various predictive tasks in machine learning.

One of the most important features of the Random Forest Algorithm is that it can handle

the data set containing continuous variables, as in the case of regression,

and categorical variables, as in the case of classification. It performs better for

classification and regression

Real-Life Analogy of Random Forest

Let’s dive into a real-life analogy to understand this concept further. A student named X

wants to choose a course after his 10+2, and he cant decide which course fit for his skill

set. So he decides to consult various people like his cousins, teachers, parents, degree

students, and working people. He asks them varied questions like why he should choose,

job opportunities with that course, course fee, etc. Finally, after consulting various

people about the course he decides to take the course suggested by most people.
Working of Random Forest Algorithm

Before understanding the working of the random forest algorithm in machine learning,

we must look into the ensemble learning technique. Ensemble simply means combining

multiple models. Thus a collection of models is used to make predictions rather than an

individual model.

Ensemble learning is a machine learning technique that combines the predictions

from multiple individual models to obtain a better predictive performance than any

single model.

Ensemble uses two types of methods:

***Random forest Classifier works on the Bagging principle.

Bagging

Bagging, also known as Bootstrap Aggregation, serves as the ensemble technique in the

Random Forest algorithm. Here are the steps involved in Bagging:

1. Selection of Subset: Bagging starts by choosing a random sample, or subset,

from the entire dataset.

2. Bootstrap Sampling: Each model is then created from these samples, called

Bootstrap Samples, which are taken from the original data with replacement. This

process is known as row sampling.

3. Bootstrapping: The step of row sampling with replacement is referred to as

bootstrapping.

4. Independent Model Training: Each model is trained independently on its

corresponding Bootstrap Sample. This training process generates results for each

model.

5. Majority Voting: The final output is determined by combining the results of all

models through majority voting. The most commonly predicted outcome among

the models is selected.

6. Aggregation: This step, which involves combining all the results and generating

the final output based on majority voting, is known as aggregation.

Now let’s look at an example by breaking it down with the help of the following figure.

Here the bootstrap sample is taken from actual data (Bootstrap sample 01, Bootstrap

sample 02, and Bootstrap sample 03) with a replacement which means there is a high

possibility that each sample won’t contain unique data. The model (Model 01, Model 02,

and Model 03) obtained from this bootstrap sample is trained independently. Each model

generates results as shown. Now the Happy emoji has a majority when compared to the

Sademoji. Thus based on majority voting final output is obtained as Happyemoji.

Steps Involved in Random Forest Algorithm

 Step 1: In this model, a subset of data points and a subset of features is selected

for constructing each decision tree. Simply put, n random records and m features

are taken from the data set having k number of records.

 Step 2: Individual decision trees are constructed for each sample.

 Step 3: Each decision tree will generate an output.

  Step 4: Final output is considered based on Majority Voting or Averaging for

Classification and regression, respectively.

For example:

Consider the fruit basket as the data as shown in the figure below. Now n number of

samples are taken from the fruit basket, and an individual decision tree is constructed

for each sample. Each decision tree will generate an output, as shown in the figure.

The final output is considered based on majority voting. In the below figure, you can

see that the majority decision tree gives output as an apple when compared to a

banana, so the final output is taken as an apple.

Important Features of Random Forest

Random Forest is distinguished by several key features that contribute to its

effectiveness and versatility:

 Diversity: Each decision tree in the Random Forest is built from a different

subset of data and features. This diversity helps in reducing overfitting and

improving the model’s generalization capability.

 Robustness: By averaging the results from multiple trees, Random Forest reduces

the variance and improves the robustness of the predictions.

 Handling of Missing Values: It can handle missing values internally by using

surrogate splits or by averaging results from other trees that do not have missing

values for the same data points.

 Feature Importance: It provides insights into the importance of each feature in

the prediction process. This can be particularly useful for feature selection and

understanding the underlying data patterns.

 Scalability: Random Forest can be parallelized because each tree is built

independently of the others. This makes it scalable to large datasets and high-

dimensional data.

 Versatility: It can be used for both classification and regression tasks. The

algorithm is also effective for tasks involving categorical and continuous

variables.

 Stability: Due to the ensemble nature, It is less sensitive to changes in the

training data compared to a single decision tree.

 Out-of-Bag Error Estimation: Random Forest provides an internal mechanism

for estimating the model error without the need for a separate validation set. This

is done using the out-of-bag (OOB) samples, which are not used in the

construction of each tree.

Difference Between Decision Tree and Random Forest

Random forest is a collection of decision trees; still, there are a lot of differences in their

behaviour.

Important Hyperparameters in Random Forest

Hyperparameters are used in random forests to either enhance the performance and
predictive power of models or to make the model faster.

Increase the Predictive Power

 n_estimators: Number of trees the algorithm builds before averaging the

predictions.

 max_features: Maximum number of features random forest considers splitting a

node.

 mini_sample_leaf: Determines the minimum number of leaves required to split

an internal node.

 criterion: How to split the node in each tree? (Entropy/Gini impurity/Log Loss)

 max_leaf_nodes: Maximum leaf nodes in each tree Increase the Speed

 n_jobs: it tells the engine how many processors it is allowed to use. If the value is

1, it can use only one processor, but if the value is -1, there is no limit.
  random_state: controls randomness of the sample. The model will always

produce the same results if it has a definite value of random state and has been

given the same hyperparameters and training data.

 oob_score: OOB means out of the bag. It is a random forest cross-validation

method. In this, one-third of the sample is not used to train the data; instead used

to evaluate its performance. These samples are called out-of-bag samples.

Out-Of-Bag Sample

In the example, you can observe that we repeated some animals while making the

sample, and some animals did not even occur once in the sample.

Here, Sample1 does not have Rat and Cow whereas sample 3 had all the animals equal to

the main training set.

While making the samples, data points were chosen randomly and with replacement, and

the data points which fail to be a part of that particular sample are known as OUT-OF-

BAG points.
Random Forest Implementation in R

Step 1: Install and Load Required Packages

Make sure you have the randomForest package installed.

[Link]("randomForest")
library(randomForest)
Step 2: Load the Dataset

The iris dataset contains information about 150 flowers, including features like sepal length,
sepal width, petal length, and petal width, along with their species.

data(iris)
head(iris)
Step 3: Split the Dataset into Training and Testing Sets

We’ll split the data into training (70%) and testing (30%) sets.

[Link](123)# For reproducibility

sample_index<-sample(1:nrow(iris),0.7*nrow(iris))
train_data<-iris[sample_index,]
test_data<-iris[-sample_index,]
Step 4: Train the Random Forest Model

We'll use Species as the target variable to predict the type of flower.

# Train the model

rf_model<-randomForest(Species ~ ., data =train_data,ntree=100,mtry=2, importance
=TRUE)
print(rf_model)

 Species ~ .specifiesSpecies as the target variable, and .indicates that all other columns
will be used as features.
 ntree = 100 specifies the number of trees in the forest.
 mtry = 2 sets the number of variables randomly sampled as candidates at each split.

Step 5: Evaluate the Model

After training, we can evaluate the model on the test data.

# Predict on test data

predictions<- predict(rf_model,test_data)

# Confusion Matrix
confusion_matrix<-table(predictions,test_data$Species)
print(confusion_matrix)
# Calculate accuracy
accuracy<-sum(diag(confusion_matrix))/sum(confusion_matrix)
print(paste("Accuracy:",round(accuracy *100,2),"%"))
Step 6: Feature Importance

Random Forests allow us to inspect the importance of each feature in making predictions.

importance(rf_model)
varImpPlot(rf_model)

 MeanDecreaseAccuracy:

 This value represents the overall decrease in model accuracy when the feature is
permuted across all species.
 A higher MeanDecreaseAccuracy means that the feature is more important for the
overall classification accuracy of the model. In this case, [Link] and
[Link] have high values, indicating that they are crucial for classifying all
species in the dataset.

 MeanDecreaseGini:

 This measures the importance of each feature based on the decrease in node impurity
(Gini impurity) averaged over all the trees in the forest.
 A higher MeanDecreaseGini means the feature contributes more to making splits in
the trees, improving the model's ability to classify the data. Here, [Link] has the
highest value, suggesting it’s the most important feature in improving the purity of the
classification.

# OOB predictions for each sample

oob_predictions<- rf_model$predicted

head(oob_predictions)

# Create confusion matrix using OOB predictions

table(oob_predictions, train_data$Species)
# Calculate OOB accuracy

oob_accuracy<- mean(oob_predictions == train_data$Species)

print(paste("OOB Accuracy:", round(oob_accuracy * 100, 2), "%"))

# OOB error rate per tree

oob_error_per_tree<- rf_model$[Link][, "OOB"]

plot(oob_error_per_tree, type = "l", main = "OOB Error Rate per Tree",

xlab = "Number of Trees", ylab = "OOB Error Rate")