Classification in Data Mining
Classification in data mining is a supervised learning approach used to
assign data points into predefined classes based on their features. By
analysing labelled historical data, classification algorithms learn patterns
and relationships that enable them to categorize new, unseen data
accurately. Let's see some key characteristics about classification:
Predicts discrete, categorical outputs.
Learns from labelled datasets using supervised learning.
Identifies meaningful relationships among features.
Supports various algorithms based on rules, probability, distance or
boundaries.
Used widely for automation, risk detection and pattern recognition.
Classification - Classification is a data mining technique used to predict group membership of data
instances. –
Classification assigns items on a collection to target categories or classes.
- The goal of classification is to accurately predict the target class for each case in the data.
Types of Classification Techniques
Classification techniques can be divided into:Types of Classification Techniques
1. Binary Classification: Binary classification assigns data into one of
two possible categories. It is commonly used when the outcome is a
simple yes/no or true/false decision.
Used for tasks like spam vs. not spam, disease vs. no disease.
Simpler decision boundaries and lower computational complexity.
Can utilize algorithms like Logistic Regression, SVM or Decision Trees.
2. Multi-Class Classification: Multi-class classification deals with
problems where the output can belong to more than two categories,
requiring more complex decision boundaries.
Used in image classification, sentiment classification or product
categorization.
Models use strategies like One-vs-One or One-vs-All for separation.
Algorithms: Random Forests, Neural Networks, Multiclass SVMs.
Common Classification Algorithms
Decision Trees: Tree-structured models where internal nodes represent features,
branches represent rules, and leaf nodes represent outcomes.
K-Nearest Neighbors (KNN): Classifies instances based on the majority class of
their nearest neighbors in the feature space.
Bayesian Classifiers: Probabilistic models (e.g., Naive Bayes) that calculate the
probability of a data point belonging to a class.
Neural Networks: Complex, interconnected node structures designed for pattern
recognition.
Support Vector Machines (SVM): Models that find the optimal hyperplane to
separate classes
A Decision Tree helps us to make decisions by mapping out different
choices and their possible outcomes. It’s used in machine learning for
tasks like classification and prediction. In this article, we’ll see more about
Decision Trees, their types and other core concepts
A Decision Tree helps us make decisions by showing different options and
how they are related. It has a tree-like structure that starts with one main
question called the root node which represents the entire dataset. From
there, the tree branches out into different possibilities based on features in
the data.
Root Node: Starting point representing the whole dataset.
Branches: Lines connecting nodes showing the flow from one decision
to another.
Internal Nodes: Points where decisions are made based on data
features.
Leaf Nodes: End points of the tree where the final decision or prediction
is made.
There are mainly two types of Decision Trees based on the target variable:
1. Classification Trees: Used for predicting categorical outcomes like
spam or not spam. These trees split the data based on features to
classify data into predefined categories.
2. Regression Trees: Used for predicting continuous outcomes like
predicting house prices. Instead of assigning categories, it provides
numerical predictions based on the input features.
How Decision Trees Work?
1. Start with the Root Node: It begins with a main question at the root
node which is derived from the dataset’s features.
2. Ask Yes/No Questions: From the root, the tree asks a series of yes/no
questions to split the data into subsets based on specific attributes.
3. Branching Based on Answers: Each question leads to different
branches:
If the answer is yes, the tree follows one path.
If the answer is no, the tree follows another path.
4. Continue Splitting: This branching continues through further decisions
helps in reducing the data down step-by-step.
5. Reach the Leaf Node: The process ends when there are no more useful
questions to ask leading to the leaf node where the final decision or
prediction is made.
Let’s look at a simple example to understand how it works. Imagine we
need to decide whether to drink coffee based on the time of day and how
tired we feel. The tree first checks the time:
1. In the morning: It asks “Tired?”
If yes, the tree suggests drinking coffee.
If no, it says no coffee is needed.
2. In the afternoon: It asks again “Tired?”
If yes, it suggests drinking coffee.
If no, no coffee is needed.
Bayes Classification is a Supervised machine learning approach for
classification. It works on a probabilistic method based on Bayes
Theorem. It predicts the data point label or assigns the class on the basis
of heuristic and statistical data.
The above image denotes the relationship representation in a Bayes
Classifier.
The nodes represent features or variables in a Bayesian network.
These can either be discrete or continuous.
The links, on the other hand, denote the relationships between the
nodes, along with a probability distribution over variables.
Bayes Classification Workflow
Terminologies
Prior: Initial belief before evidence (e.g., % of spam emails). It is an
Input to the Classifier.
Likelihood: Evidence corresponding to a particular given class (e.g.,
frequency of “free” in spam). It is an Input to the Classifier.
Posterior: Updated belief after evidence. It is an Output from the
Classifier.
Evidence: Overall probability of observed features.
The above image demonstrates that Likelihood, Data, and Prior
probabilities are used as Input to the model. Bayes Theorem is used as the
mathematical principle. The resultant is Posterior Distribution.
Steps Involved in Classification
1. Collect Data: From training set, extract features X and target C.
2. Estimate Priors: Calculate P(C) for each class.
3. Estimate Likelihoods: For every feature X, compute P(X∣C).
4. Calculate Posterior: Use Bayes’ Theorem to get P(C∣X).
5. Predict Class: Assign the class with the highest posterior probability. It
calculates the posterior probability for each class and assigns the class
with the highest probability.
Advantages of Bayes Classifier
1. Handles small data well.
2. Can incorporate domain knowledge.
3. Probabilistic output allows threshold tuning.
4. Can model non-linear decision boundaries.
5. Adaptable to both discrete and continuous data.
IF-THEN Rules
Rule-based classifier makes use of a set of IF-THEN rules for classification.
We can express a rule in the following from −
IF condition THEN conclusion
Let us consider a rule R1,
R1: IF age = youth AND student = yes THEN buy_computer = yes
Points to remember −
The IF part of the rule is called rule antecedent or precondition.
The THEN part of the rule is called rule consequent.
The antecedent part the condition consist of one or more attribute tests
and these tests are logically ANDed.
The consequent part consists of class prediction.
Note − We can also write rule R1 as follows −
R1: (age = youth) ^ (student = yes))(buys computer = yes)
If the condition holds true for a given tuple, then the antecedent is
satisfied.
To extract a rule from a decision tree −
One rule is created for each path from the root to the leaf node.
To form a rule antecedent, each splitting criterion is logically ANDed.
The leaf node holds the class prediction, forming the rule consequent.
IF-THEN Rule
To define the IF-THEN rule, we can split it into two parts:
Rule Antecedent: This is the “if condition” part of the rule. This part is present in the
LHS(Left Hand Side). The antecedent can have one or more attributes as conditions,
with logic AND operator.
Rule Consequent: This is present in the rule's RHS(Right Hand Side). The rule
consequent consists of the class prediction.