UNIT-3
CLASSIFICATION
• The Classification algorithm is a Supervised Learning technique that is used to
identify the category of new observations on the basis of training data.
• In Classification, a program learns from the given dataset or observations and
then classifies new observation into a number of classes or groups.
• Such as, Yes or No, 0 or 1, Spam or Not Spam, cat or dog, etc.
• Classes can be called as targets/labels or categories.
• The Classification algorithm is a Supervised learning technique, hence it takes
labeled input data, which means it contains input with the corresponding output.
• In classification algorithm, a discrete output function(y) is mapped to input
variable(x).
• 1. y=f(x), where y = categorical output
• The algorithm which implements the classification on a dataset is known as a
classifier.
• There are two types of Classifications:
• Binary Classifier: If the classification problem has only two possible outcomes, then it
is called as Binary Classifier.
• Examples: YES or NO, MALE or FEMALE, SPAM or NOT SPAM, CAT or DOG, etc.
• Multi-class Classifier: If a classification problem has more than two outcomes, then it
is called as Multi-class Classifier.
• Example: Classifications of types of crops, Classification of types of music.
TYPES OF ML CLASSIFICATION ALGORITHMS:
• Classification Algorithms can be further divided into the Mainly two category:
• o Linear Models
• o Logistic Regression
• o Support Vector Machines
• o Non-linear Models
• o K-Nearest Neighbours
• o Kernel SVM
• o Naive Bayes
• o Decision Tree Classification
• o Random Forest Classification
EVALUATING A CLASSIFICATION MODEL:
• Once our model is completed, it is necessary to evaluate its performance; either it
is a Classification or Regression model.
• So for evaluating a Classification model, we have the following ways:
• 1. Log Loss or Cross-Entropy Loss:
• o It is used for evaluating the performance of a classifier, whose output is a
probability value between the 0 and 1.
• o For a good binary Classification model, the value of log loss should be near to 0.
• o The value of log loss increases if the predicted value deviates from the actual
value.
• o The lower log loss represents the higher accuracy of the model.
2. CONFUSION MATRIX:
• The confusion matrix provides us a matrix/table as output and describes the
performance of the model. o It is also known as the error matrix.
• o The matrix consists of predictions result in a summarized form, which has a
total number of correct predictions and incorrect predictions.
• The matrix looks like as below table:
3. AUC-ROC CURVE:
• ROC curve stands for Receiver Operating Characteristics Curve and AUC stands for
Area Under the Curve.
• o It is a graph that shows the performance of the classification model at different
thresholds.
• o To visualize the performance of the multi-class classification model, we use the
AUCROC Curve.
• o The ROC curve is plotted with TPR and FPR, where TPR (True Positive Rate) on Yaxis
and FPR(False Positive Rate) on X-axis.
DECISION TREE INDUCTION ALGORITHM
• A machine researcher named J. Ross Quinlan in 1980 developed a decision
tree algorithm known as ID3 (Iterative Dichotomiser).
• Later, he presented C4.5, which was the successor of ID3. ID3 and C4.5 adopt
a greedy approach.
• In this algorithm, there is no backtracking; the trees are constructed in a top-
down recursive divide-and-conquer manner.
HOW DOES DECISION TREE ALGORITHM WORK?
• Let’s look at the steps that the decision tree algorithm follows.
• Step 1: The first node of the tree is always the root node and it consists of the
whole dataset. Let’s refer to the root node as S.
• Step2: Use attribute selection measures to find the best attribute.
• Step 3: Now divide the root node (S) into subsets that contain various possible
values of the best attribute that you found out in step 2.
• Step 4: Now generate a decision tree node that consists of the best attribute.
• Step 5: Now make new decision trees using the subsets that were created in
step – 3. Do this step recursively till the node cannot be divided anymore i.e
leaf node
ATTRIBUTE SELECTION MEASURES
• In general, there are different features/attributes in a dataset.
• We randomly cannot make a feature/attribute as the root node and then
follow the same for every decision node.
• Doing so would lead to a very wrong model that will make wrong decisions
based on un useful features.
• So to solve this problem, we have something called Attribute Selection
Measures.
• Here is a list of some attribute selection measures.
• [Link]
• 2.Information_gain
• 3.Gini_index
• 4.Gain_Ratio
• [Link] in Variance
• [Link]-Square
ENTROPY
• It is the measure of impurity (or) uncertainty in the data. It lies between 0 to 1
and is calculated using the below formula.
INFORMATION GAIN
• Information gain is simply the measure of change in entropy.
• The higher the information gain, the lower is the entropy.
• Thus for a model to be good, it should have high Information gain.
• A decision tree algorithm in general tries to maximize the value of information
gain, and an attribute/feature having the highest information gain is split first.
GINI INDEX
• It is a measure of purity or impurity while creating a decision tree.
• It is calculated by subtracting the sum of the squared probabilities of each
class from one.
• It is the same as entropy but is known to calculate quicker as compared to
entropy.
• CART ( Classification and regression tree ) uses the Gini index as an attribute
selection measure to select the best attribute/feature to split.
• The attribute with a lower Gini index is used as the best attribute to split.
UNDERSTANDING DECISION TREE WITH REAL LIFE USE
CASE:
Step 1. Start with the Whole Dataset
We begin with all the data which is treated as the root node
of the decision tree.
Step 2. Choose the Best Question (Attribute)
Pick the best question to divide the dataset. For example
ask: "What is the outlook?"
Possible answers: Sunny, Cloudy or Rainy.
• Step 3. Split the Data into Subsets
• Divide the dataset into groups based on the question:
• If Sunny go to one subset.
• If Cloudy go to another subset.
• If Rainy go to the last subset.
• Step 4. Split Further if Needed (Recursive Splitting)
• For each subset ask another question to refine the groups. For example If the
Sunny subset is mixed ask: "Is the humidity high or normal?"
• High humidity → "Swimming".
• Normal humidity → "Hiking".
• Step 5. Assign Final Decisions (Leaf Nodes)
• When a subset contains only one activity, stop splitting and assign it a label:
• Cloudy → "Hiking".
• Rainy → "Stay Inside".
• Sunny + High Humidity → "Swimming".
• Sunny + Normal Humidity → "Hiking".
• Step 6. Use the Tree for Predictions
• To predict an activity follow the branches of the tree. Example: If the outlook is Sunny and the humidity is High follow the tree:
• Start at Outlook.
• Take the branch for Sunny.
• Then go to Humidity and take the branch for High Humidity.
• Result: "Swimming".
• A decision tree works by breaking down data step by step asking the best possible questions at each point and stopping once it
reaches a clear decision. It's an easy and understandable way to make choices. Because of their simple and clear structure
decision trees are very helpful in machine learning for tasks like sorting data into categories or making predictions.