Decisions Trees Have High Variance
Adding variance (i.e., complexity) is required for useful models.
Machine learning algorithms differ in the amount of variance that they can introduce.
Decision trees exhibit high variance.
Here’s the intuition – trees can radically change their shape based on changes in the training data…
Dataset #1 Dataset #2
Only for
teaching
purposes!
High Variance Leads to Overfitting
By default, scikit-learn decision trees will grow to be very complex.
Depending on the data, this can produce “interesting” models…
Real World Trees
In practice, trees are rarely used for “Production” scenarios.
They can be useful for certain ad hoc analysis scenarios (e.g., explaining what’s going on in the business).
However, if one is planning on using a decision tree for decision-making, optimal tuning is a must.
In the real world, decision trees are wildly useful – but not on their own.
The high variance of decision trees has been leveraged to create one of the most useful of all ML algorithms...
The mighty random forest!
Wisdom of the Crowd
“The three conditions for a group to be intelligent are diversity,
independence, and decentralization.” - James Surowiecki
ENSEMBLING:
Compiling predictions from multiple machine learning models in order to make more
accurate predictions than any individual model.
However, ensembling works best when the individual models are diverse and have
low correlation between their predictions.
GOOD NEWS! The Random Forest algorithm manufactures independence and
diversity across decision trees, making it an effective ensembling method.
Manufacturing Independence
The mighty random forest manufactures independence across decision trees via three mechanisms…
1 – Variance 2 - Bagging 3 – Feature Randomization
Blue
Blue
Orange
Orange
Green
Blue
Orange Red
Green Orange
Orange Red
Green Orange
Green Green
Blue
Randomizing Rows With Bagging
Trees will change their shape depending on the training data (i.e., high variance).
Bagging (i.e., “bootstrap aggregation”) takes a single training set and “manufactures” different datasets.
The magic of bagging works via random sampling with replacement…
Notice the
List has as many items as duplicates!
marbles in the jar.
Blue
Blue Blue Orange Red
Green Orange
Orange Orange
Orange Red
Orange Green
Green Green
Green
Blue
Let’s say I have a I reach into I write down the I repeat this I can create as
jar with 4 marbles the jar and drawn marble process, getting a many lists as I
– blue, green, red, draw a marble and put it back list that is different would like.
and orange. at random. in the jar. from the jar.
The marbles are the rows of your training set.
Bagging is Awesome
Bagging helps produce wildly different trees.
Bagging is also awesome because bags don’t have all the original training data…
Bag 0 Bag 1
Out of bag (OOB) data Out of bag (OOB) data
Feature Randomization
The mighty random forest doesn’t stop with bagging!
Random subsets of the features are provided to the decision trees…
Features Label
By default, the Let’s say that the RF is building The RF algorithm chooses 3
RandomForestClassifier (RF) a new tree from scratch. features at random to be
chooses the square root of the considered for the first split:
number of features at random:
14 = 3.7417 = 3
The decision tree then picks the
Only use the best split given the 3 features.
whole part.
Feature Randomization
Continuing this hypothetical example…
The RF algorithm picks these 3
The tree picks the best
random features for the first
split of the 3 features.
split of the tree.
The RF algorithm picks 3 more features at The RF algorithm picks 3 more features at
random for the left node split. random for the right node split.
The tree picks the best split of the 3 The tree picks the best split of the 3
features. features.
Feature Randomization
Continuing this hypothetical example…
The RF algorithm picks these 3
The tree picks the best
random features for the first
split of the 3 features.
split of the tree.
The RF algorithm picks 3 more features at The RF algorithm picks 3 more features at
random for the left node split. random for the right node split.
The tree picks the best split of the 3 The tree picks the best split of the 3
features. features.
The Dataset
The Dataset
Preparing Features
Preparing Features
Preparing Labels
Training a Mighty Random Forest
Get OOB Predictions
Visualize OOB Predictions
Visualize OOB Predictions