Ensemble Learning
and
Random Forests
DR. ANWAR M MIRZA
Ensemble Learning
Suppose you pose a complex question to thousands of random
people, then aggregate their answers. In many cases you will find
that this aggregated answer is better than an expert’s answer. This is
called the wisdom of the crowd. Similarly, if you aggregate the
predictions of a group of predictors (such as classifiers or regressors),
you will often get better predictions than with the best individual
predictor. A group of predictors is called an ensemble; thus, this
technique is called Ensemble Learning, and an Ensemble Learning
algorithm is called an Ensemble method.
Random Forest
As an example of an Ensemble method, you can train a group of
Decision Tree classifiers, each on a different random subset of the
training set.
To make predictions, you obtain the predictions of all the individual
trees, then predict the class that gets the most votes. Such an
ensemble of Decision Trees is called a Random Forest, and despite
its simplicity, this is one of the most powerful machine Learning
algorithms available today.
When to use Ensemble Learning
You will often use Ensemble methods near the end of a project, once
you have already built a few good predictors, to combine them into
an even better predictor.
In fact, the winning solutions in Machine Learning competitions
often involve several Ensemble methods (most famously in the
Netflix Prize competition – Kaggle Dataset Link here.).
Training Diverse Classifiers
Majority-vote classification
Example Dataset: Moons Dataset (chapter 5)
VotingClassifier in Scikit
from [Link] import RandomForestClassifier
from [Link] import VotingClassifier
from sklearn.linear_model import LogisticRegression
from [Link] import SVC
log_clf = LogisticRegression()
rnd_clf = RandomForestClassifier()
svm_clf = SVC()
voting_clf = VotingClassifier(
estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
voting='hard’)
# using the Moons dataset (chapter 5)
voting_clf.fit(X_train, y_train)
VotingClassifier in Scikit
The accuracy of each classifier can be found as follows:
>>> from [Link] import accuracy_score
>>> for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
... [Link](X_train, y_train)
... y_pred = [Link](X_test)
... print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
...
LogisticRegression 0.864
RandomForestClassifier 0.896
SVC 0.888
VotingClassifier 0.904
Bagging and Pasting
One way to get a diverse set of classifiers is to use very different
training algorithms, as just discussed in previous slides.
Another approach is to use the same training algorithm for every
predictor and train them on different random subsets of the training
set. When sampling is performed with replacement, this method is
called bagging (short for bootstrap aggregating). When sampling is
performed without replacement, it is called pasting.
Bagging – Sampling with
replacement
Source: [Link]
Pasting – Sampling without
replacement
Bagging and Pasting
Bagging and Pasting in Scikit
Bagging comparison
Bagging - Out of Bag (OOB)
Evaluations
There is no need to use cross-validation to evaluate model performance while
using Bagging.
By default, a BaggingClassifier samples m training instances with replacement
(bootstrap=True), where m is the size of the training set. This means that only
about 63% of the training instances are sampled on average for each predictor.
The remaining 37% of the training instances that are not sampled are called out-
of-bag (oob) instances. Note that they are not the same 37% for all predictors.
Since a predictor never sees the oob instances during training, it can be
evaluated on these instances, without the need for a separate validation set. You
can evaluate the ensemble itself by averaging out the oob evaluations of each
predictor.
Bagging – OOB Evlauation in
Scikit
bag_clf = BaggingClassifier(
DecisionTreeClassifier(random_state=42), n_estimators=500,
bootstrap=True, oob_score=True, random_state=40)
bag_clf.fit(X_train, y_train)
bag_clf.oob_score_
RandomForest
Random Forest is an ensemble of Decision Trees, generally trained via the
bagging method (or sometimes pasting), typically with max_samples set to the
size of the training set.
Boosting
Boosting (originally called hypothesis boosting) refers to any Ensemble method
that can combine several weak learners into a strong learner. The general idea of
most boosting methods is to train predictors sequentially, each trying to correct
its predecessor.
There are many boosting methods available, but by far the most popular are
AdaBoost (short for Adaptive Boosting) and Gradient Boosting.
AdaBoost
Boosting Example