Supervised Learning
Ghada Dahy Fathy Kamel
Department of Information Technology, Faculty of Computers
and Artificial Intelligence, Cairo University, Egypt
Module 3:Ensemble Learning
Agenda
1. Introduction
2. Majority Voting
3. Bagging
4. Boosting
5. Stacking
6. Random Forest
2
Introduction
1
Introduction
4
Introduction
Different Classifiers (1)
• Different Classifiers
• Conduct classification on a same set of class labels
• May use different input or have different parameters
• May produce different output for a certain example
Machine Learning Basics: 3. Ensemble Learning
Introduction
Different Classifiers (2)
• Performance
• Each of the classifiers is not perfect
• Complementary
• Examples which are not correctly classified by one classifier
may be correctly classified by the other classifiers
• Potential Improvements?
• Utilize the complementary property
Machine Learning Basics: 3. Ensemble Learning
Introduction
Ensembles of Classifiers
• Idea
• Combine the classifiers to improve the performance
• Ensembles of Classifiers
• Combine the classification results from different
classifiers to produce the final output
• Unweighted voting
• Weighted voting
Machine Learning Basics: 3. Ensemble Learning
Introduction
Example: Weather Forecast
Reality
1 X X X
2 X X X
3 X X X
4 X X
5 X X
Combine
Machine Learning Basics: 3. Ensemble Learning
Introduction
Ensemble Learning
• Ensemble Learning
• Relatively later field in machine learning
• Achieve state-of-the-art performance
• Central Issues in Ensemble Learning
• How to create classifiers with complementary
performances
• How to conduct voting
Machine Learning Basics: 3. Ensemble Learning
Introduction
Introduction
Majority Voting
2
M a j o r i t y Vo t i n g
M a j o r i t y Vo t i n g
M a j o r i t y Vo t i n g
M a j o r i t y Vo t i n g
M a j o r i t y Vo t i n g
M a j o r i t y Vo t i n g
Bagging
3
Bagging
Bagging
Bagging
Bagging
Boosting
3
Boosting
Strong and Weak Learners
• Strong Learner
• Take labeled data for training
• Produce a classifier which can be arbitrarily accurate
• Objective of machine learning
• Weak Learner
• Take labeled data for training
• Produce a classifier which is more accurate than
random guessing
Machine Learning Basics: 3. Ensemble Learning
Boosting
Construct Weak Classifiers
• Using Different Data Distribution
• Start with uniform weighting
• During each step of learning
• Increase weights of the examples which are not correctly learned by the weak learner
• Decrease weights of the examples which are correctly learned by the weak learner
• Idea
• Focus on difficult examples which are not correctly classified in the previous steps
Machine Learning Basics: 3. Ensemble Learning
Boosting
Machine Learning Basics: 3. Ensemble Learning
Boosting
Machine Learning Basics: 3. Ensemble Learning
Boosting
Machine Learning Basics: 3. Ensemble Learning
Boosting
Machine Learning Basics: 3. Ensemble Learning
Stacking
5
Stacking
Stacking
Stacking
Stacking
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import StackingClassifier
from sklearn.linear_model import LogisticRegression
from [Link] import DecisionTreeClassifier
from [Link] import KNeighborsClassifier
from [Link] import accuracy_score
data = load_iris()
X = [Link]
y = [Link]
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42)
base_models = [ ('dt', DecisionTreeClassifier()), ('knn', KNeighborsClassifier())]
meta_model = LogisticRegression()
stack_model = StackingClassifier(estimators=base_models,final_estimator=meta_model)
stack_model.fit(X_train, y_train)
y_pred = stack_model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)
Random Forest
6
Ransom Forest
Ransom Forest
Ransom Forest
Ransom Forest
Ransom Forest
Ransom Forest
Ransom Forest