CLASSIFICATION
Arvind Deshpande
11/19/2024 Arvind Deshpande (VJTI) 2
Techniques to Improve Classification Accuracy
• An ensemble for classification is a composite model,
made up of a combination of classifiers.
• The individual classifiers vote, and a class label prediction
is returned by the ensemble based on the collection of
votes.
• Ensembles tend to be more accurate than their
component classifiers.
11/19/2024 Arvind Deshpande (VJTI) 3
Ensemble methods -
11/19/2024 Arvind Deshpande (VJTI) 4
Decision boundary for a linearly separable problem (i.e., where
the actual decision boundary is a straight line).
A single decision tree An ensemble of decision trees
11/19/2024 Arvind Deshpande (VJTI) 5
Bagging
• Bootstrap aggregation
• For iteration i ( i = 1, 2, …., k), a training set, Di, of d
tuples is sampled with replacement from the original set of
tuples, D.
• A classifier model, Mi , is learned for each training set, Di .
• To classify an unknown tuple, X, each classifier, Mi ,
returns its class prediction, which counts as one vote.
• The bagged classifier, M, counts the votes and assigns
the class with the most votes to X.
• Bagging can be applied to the prediction of continuous
values by taking the average value of each prediction for
a given test tuple.
11/19/2024 Arvind Deshpande (VJTI) 6
Bagging
• The bagged classifier often has significantly greater
accuracy than a single classifier
• Derived from D, the original training data. It will not be
considerably worse and is more robust to the effects of
noisy data and overfitting.
• The increased accuracy occurs because the composite
model reduces the variance of the individual classifiers.
11/19/2024 Arvind Deshpande (VJTI) 7
Boosting
• In boosting, weights are also assigned to each training
tuple.
• A series of k classifiers is iteratively learned. After a
classifier, Mi , is learned, the weights are updated to allow
the subsequent classifier, Mi+1, to “pay more attention” to
the training tuples that were misclassified by Mi .
• The final boosted classifier, M, combines the votes of
each individual classifier, where the weight of each
classifier’s vote is a function of its accuracy.
11/19/2024 Arvind Deshpande (VJTI) 8
Bagging and Boosting
• Because of the way boosting focuses on the misclassified
tuples, it risks overfitting the resulting composite model to
such data.
• Therefore, sometimes the resulting “boosted” model may
be less accurate than a single model derived from the
same data.
• Bagging is less susceptible to model overfitting.
• While both can significantly improve accuracy in
comparison to a single model, boosting tends to achieve
greater accuracy.
11/19/2024 Arvind Deshpande (VJTI) 9
Random forest
• A supervised learning algorithm which is used for both
classification as well as regression. However, it is mainly
used for classification problems.
• Ensemble method - A machine learning technique that
combines several base models in order to produce one
optimal predictive model.
• Algorithm creates decision trees on data samples and
then gets the prediction from each of them and finally
selects the best solution by means of voting.
• This is better than a single decision tree because it
reduces the overfitting by averaging the result.
11/19/2024 Arvind Deshpande (VJTI) 10
Random forest
1. Start with the selection of random samples from a given
dataset.
2. Construct a decision tree for every sample.
3. It will get the prediction result from every decision tree.
4. Voting will be performed for every predicted result.
5. Select the most voted prediction result as the final
prediction result.
11/19/2024 Arvind Deshpande (VJTI) 11
Random forest
11/19/2024 Arvind Deshpande (VJTI) 12
Advantages
• Random forests overcome the problem of overfitting by
averaging or combining the results of different decision
trees.
• It works well for a large range of data items than a single
decision tree.
• Random forest has less variance than single decision
tree.
• Random forests are very flexible and possess very high
accuracy.
• It maintains a good accuracy even if a large proportion of
the data is missing.
• Scaling of data is not required in random forest algorithm.
It maintains good accuracy even providing data without
scaling.
11/19/2024 Arvind Deshpande (VJTI) 13
Disadvantages
• Complexity is the main disadvantage of random forest
algorithms.
• Construction of random forests are much harder and time-
consuming than decision trees.
• More computational resources are required to implement
random forest algorithm.
• It is less intuitive in case when we have a large collection
of decision trees.
• The prediction process using random forests is very time
consuming in comparison with other algorithms.