Random Forest Algorithm
Random Forest is an ensemble machine learning algorithm that builds many decision trees and combines
their predictions to give more accurate and stable results.
Background: Decision Trees
A decision tree splits data into groups based on features to predict a class.
Pros
Simple to understand
Works with any type of data
Cons
High variance (unstable — changes with small data changes)
This is why we need Bagging and Random Forest.
Bagging (Bootstrap Aggregation)
Reduces variance by creating multiple datasets through sampling with replacement.
How it works:
1. Create B bootstrap samples
2. Train B different trees
3. Take majority vote for prediction
Benefit → Errors cancel out!
Random Forest (Improved Bagging)
Adds random feature selection so trees are less correlated → better accuracy.
� At each split:
Select m random features (typically √p)
Best split is chosen only from those features
Tree Prediction
1 Pass
2 Fail
3 Pass
4 Pass
5 Fail
How Random Forest Makes Final Prediction
Classification Formula
Random Forest uses Majority Voting:
Where:
Most common → Pass
Final Result = Pass
Regression Formula
Random Forest uses Averaging
Tree Price Prediction
1 10 lakh
2 12 lakh
3 11 lakh
Final = (10 + 12 + 11) / 3 = 11 lakh.
Gini Impurity Formula (Inside Each Tree):
Used to select the best split:
Where:
= probability of class i
Lower Gini = Better split
Example
7 Pass, 3 Fail → Total = 10
OOB Error (Out-of-Bag)
~33% unused data is used as test data → built-in accuracy estimator.
No need extra cross-validation!
Feature Importance:
Method Meaning
Gini Importance Measures impurity reduction
Permutation
Decrease in accuracy when feature is shuffled
Importance
Why Random Forest Works?
Wisdom of Crowd
Reduced Variance
Diversity of Trees (Data + Features)
Bagging vs Random Forest
Feature Bagging Random Forest
Features per split All available Random subset
Tree correlation Higher Lower
Accuracy Good Better
Real-Life Example: Loan Approval
Each tree predicts:
Approve � or Approve ✔
Final decision = majority vote → More reliable
Python Implementation
# Import tool
from [Link] import RandomForestClassifier
# Create model with 10 trees (n_estimators=10)
model = RandomForestClassifier(n_estimators=10) # 10 decision trees in forest
# Patient data: [Age, Blood Pressure, Cholesterol]
X = [[25, 120, 180], # Patient 1: 25yrs, BP 120, Cholesterol 180
[30, 130, 200], # Patient 2: 30yrs, BP 130, Cholesterol 200
[45, 140, 220], # Patient 3: 45yrs, BP 140, Cholesterol 220
[50, 150, 240]] # Patient 4: 50yrs, BP 150, Cholesterol 240
# Labels: 0=Healthy, 1=At Risk
y = [0, 0, 1, 1] # First 2 patients healthy, last 2 at risk
# Train model
[Link](X, y) # Model learns patterns from this data
# New patient to predict
new_patient = [[35, 135, 210]] # 35yrs, BP 135, Cholesterol 210
# Make prediction
result = [Link](new_patient) # result will be [0] or [1]
# Check prediction
if result[0] == 0:
print("Patient is: Healthy �") # If result is 0
else: # Print Healthy
print("Patient is: At Risk ��") ## If result is 1 ( Print At Risk)
What n_estimators=10 Does
ANALOGY:
Imagine you're buying a car �:
One friend's opinion = Single decision tree (might be biased)
10 friends' opinions = Random Forest with 10 trees (better decision!)
WHAT HAPPENS INTERNALLY:
(1)CREATES 10 DIFFERENT TREES:
# Behind the scenes:
Tree1 = DecisionTree(random_features=[Age, BP])
Tree2 = DecisionTree(random_features=[BP, Cholesterol])
Tree3 = DecisionTree(random_features=[Age, Cholesterol])
Tree4 = DecisionTree(random_features=[Age, BP])
Tree5 = DecisionTree(random_features=[BP, Cholesterol])
Tree6 = DecisionTree(random_features=[Age, Cholesterol])
Tree7 = DecisionTree(random_features=[Age, BP])
Tree8 = DecisionTree(random_features=[BP, Cholesterol])
Tree9 = DecisionTree(random_features=[Age, Cholesterol])
Tree10 = DecisionTree(random_features=[Age, BP, Cholesterol])
2. EACH TREE MAKES ITS OWN PREDICTION:
For patient [99, 250, 300]
Tree1: "At Risk" (looked at Age=99)
Tree2: "At Risk" (looked at BP=250)
Tree3: "At Risk" (looked at Cholesterol=300)
Tree4: "At Risk"
Tree5: "At Risk"
Tree6: "At Risk"
Tree7: "At Risk"
Tree8: "At Risk"
Tree9: "At Risk"
Tree10: "At Risk"
3. MAJORITY VOTING:
Votes for "At Risk": 10 out of 10
Votes for "Healthy": 0 out of 10
FINAL DECISION: "At Risk" (Unanimous!)
WHY USE 10 TREES (NOT 1 OR 100)?
Single Tree (n_estimators=1):
model = RandomForestClassifier(n_estimators=1) # BAD!
# Problem: One tree might make wrong decision
# Example: Tree only looks at BP, ignores Age and Cholesterol
# Risk of overfitting to training data
10 Trees (Good Balance):
model = RandomForestClassifier(n_estimators=10) # GOOD!
# Benefits:
# 1. Reduces overfitting (different trees see different aspects)
# 2. More reliable (majority voting)
# 3. Not too slow (10 trees train quickly)
# 4. Handles noise in data better
100 Trees (Too Many):
model = RandomForestClassifier(n_estimators=100) # OK but slower
# Benefits: Even more reliable
# Drawbacks: Slower training, more memory
# For small dataset (4 patients), 10 is enough!
HOW THE OUTPUT IS DECIDED:
TRAINING PATTERNS LEARNED:
The model learned from 4 examples:
Healthy Pattern: Young age (25-30) + Low numbers (BP: 120-130, Chol: 180-200)
At Risk Pattern: Older age (45-50) + High numbers (BP: 140-150, Chol: 220-240)
DECISION LOGIC:
When you input [99, 250, 300]:
Age 99 → Much higher than "At Risk" examples (45, 50)
BP 250 → Much higher than "At Risk" examples (140, 150)
Cholesterol 300 → Much higher than "At Risk" examples (220, 240)
ALL 3 values are EXTREMELY HIGH compared to training data!
MODEL'S THINKING PROCESS:
Question: "Is this patient similar to Healthy or At Risk cases?"
Comparison:
- Healthy cases: Age ~25-30, BP ~120-130, Chol ~180-200
- At Risk cases: Age ~45-50, BP ~140-150, Chol ~220-240
- New patient: Age=99, BP=250, Chol=300
Conclusion:
New patient is EVEN WORSE than "At Risk" cases!
Therefore: Definitely "At Risk "
OUTPUT BASIS:
Your patient gets "At Risk" because:
Age 99 is nearly DOUBLE the oldest "At Risk" patient (50)
BP 250 is 100 points HIGHER than "At Risk" BP (150)
Cholesterol 300 is 60 points HIGHER than "At Risk" cholesterol (240)
All 3 values are in the DANGER ZONE beyond training examples
Final Summary
Random Forest = Bagging + Random Feature Selection
Reduces Variance
Works extremely well on large and complex datasets!