GRADE 12 DATA SCIENCE FACULTY: Ms.

SUNITHA SUDHAKARAN

CLASSIFICATION
ALGORITHMS-I
CHAPTER-3
 CHAPTER 3-DECISION TREE
ALGORITHMS

LEARNING
OBJECTIVES
Students will be able to understand:
What is a Decision Tree?
How are Decision Trees used in Data
Science?
How to create a Decision Tree?
DECISION TREES EXPLAINED

 [Link]
DECISION TREES EXPLAINED

 [Link]
 [Link]
v=ad79nYk2keg&ab_channel=Simplilearn
 [Link]
Dg&ab_channel=LearnFree
HOW DOES A DECISION TREE
WORK
HOW DOES A DECISION TREE
WORK
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI A Decision tree is a diagrammatic

ON representation of the decision-making
process and has a tree-like structure.
APPLICATIO
Each internal node in the decision tree
N
denotes a question on choosing a
particular class.
CREATION
OF DT Every branch represents the outcome of
the test, and each leaf node holds a
class label.
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI  A decision tree is a type of supervised

ON learning algorithm that is commonly used in
machine learning to model and predict
outcomes based on input data.
APPLICATIO  It is a tree-like structure where each
N internal node represents a decision or test
on a specific feature or attribute, each
CREATION branch represents the outcome of that
OF DT decision, and each leaf node represents the
final decision or prediction.
 It is used for classification and regression
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI
ON

APPLICATIO
N

CREATION
OF DT
 CHAPTER 3-DECISION TREE
INTRODUCTION
ALGORITHMS

INTRODUCTI
ON

APPLICATIO
N

CREATION
OF DT
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI
ON

APPLICATIO
N

CREATION
OF DT
YOU'RE GIVEN DATA ABOUT WHETHER
SOMEONE BUYS COFFEE BASED ON THE TIME
OF DAY AND WHETHER THE WEATHER IS COLD.

Use the features: Cold Weather (Yes/No), Morning (Yes/No)

Draw a decision tree that predicts whether a person will Buy Coffee (Yes/No) based
on the table above.
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI
ON
 Decision Trees are considered to be one
of the most efficient classification
APPLICATIO techniques.
N  An even better way of using decision
trees is to use the Random Forest
CREATION algorithm which makes predictions based
OF DT on the outcomes of several decision
trees.
 A decision tree is like a flowchart that makes a decision — it asks
questions like “Is the income > \$50,000?” and moves down the
branches to predict something (like "Will they buy this product?").

 A random forest is just a group of many decision trees working together

to make better predictions.

 Simple Idea Behind Random Forest:

Don't trust just one tree — ask a whole forest and let them vote!"
 HOW IT WORKS

 1. Make many decision trees — each one is trained on a random

part of the data.
 2. Each tree gives its own prediction.
 3. The forest:

 For classification → takes a vote (majority wins).

 For regression → takes the average of all tree predictions.
EXAMPLE

 You want to predict if someone will like a movie.

 Tree 1 says: Yes

 Tree 2 says: No
 Tree 3 says: Yes
 Tree 4 says: Yes

 Majority says Yes → So the Random Forest predicts Yes

 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI
CREATION OF DECISION TREE-IRIS DATA SET
ON
 data=iris
APPLICATIO  print(data)
N  str(iris)
CREATION  dim(iris)
OF DT  summary(iris)
 #1 importing data set
 #2 preliminary analysis
 #3 to check for missing values of dataset
 #4 partitioning of data set into training data and testing data
 #5 creating of training data set
 #6 creating of testing data
#7 Creation of decision
#8 create new data for testing the model
#9 Get model prediction on new data
#10 calculating accuracy of the model 1- get model prediction on test data
#2- calculate accuracy


 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI
 CREATION OF DECISION TREE-IRIS DATA
ON
SET
APPLICATIO [Link](555)
N ind<-
sample(2,nrow(ind),replace=TRUE,prob=c(0.8,0.2)
CREATION )
OF DT Train<-iris[ind==1,]
Test<-iris[ind==2,]
 # 1. Create a random 1/2 “label” for every row of your
[Link]
i=sample(2,nrow(data),replace=TRUE,prob=c(0.8,0.2))
 #Draws a random value 1 or 2 for each of the
nrow(data) rows.
 #prob = c(0.8, 0.2) means about 80% of the entries
will be 1, and 20% will be 2.
 #i is therefore an integer vector of length = number of
rows in data
 #Creation of training data
 train=data[i==1,]
 #i == 1 yields a logical vector (TRUE /
FALSE) of the same length as nrow(data)
 print(head(train))
 print(dim(train))
test=data[i==2,]
print(head(test))
print(dim(test))
#data[i == 1, ] subsets rows: you
get roughly 80% of the original rows
(where i was 1) and all columns.
 library(party) # Loads the party package (used for
conditional inference trees)
 tree = ctree(Species ~ ., train) # Builds a decision tree model
 print(tree)
 plot(tree) # Plots the resulting tree
 CHAPTER 3-DECISION TREE ALGORITHMS

 CREATION OF DECISION TREE-IRIS DATA SET

INTRODUCTION
Tree<-ctree(Species~.,Train)
APPLICATION plot(Tree)
plot(Tree,type=‘simple’)
CREATION OF
[Link]
DT v=GCXsKNMDy1w&ab_channel=[Link]
ai
LINEAR MODEL

 You have a big sheet with red dots on one side and blue
crosses on the other.
 A linear model draws a straight line to separate them —
so each group stays on its own side.

 A linear model is like a smart ruler — it draws a

straight line to decide where things belong.
 A LINEAR MODEL TO PREDICT IF A
STUDENT WILL PASS OR FAIL
BASED ON:
 Study hours
 Attendance
 Previous test scores
The model adds up the values (like a formula),
and if the result is above a certain number, it
predicts Pass — otherwise, Fail.
It’s like drawing a straight line between students
who are likely to pass and those who may need
help.
BANKS USE LINEAR MODELS TO DECIDE IF
SOMEONE SHOULD GET A LOAN.
Input features:
 Income
 Age
 Credit score
 Loan amount requested
The model uses a formula (a straight-line rule) like:
Loan Score=(Income×w1)+(Credit Score×w2)+…\text{Loan Score} = (\
text{Income} \times w_1) + (\text{Credit Score} \times w_2) + \
dotsLoan Score=(Income×w1​)+(Credit Score×w2​)+…
If the score is above a line, the loan is approved. If not, it’s rejected
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

 Decision Trees and other tree-based learning algorithms

INTRODUCTI are considered to be one of the best and most used
ON supervised learning methods. They are important as they
are easy to visualize, understand, and have a high ease of
interpretation.
APPLICATIO
N
 Sometimes the trend in the data is not linear, so we cannot
apply linear classification techniques for these problems.
The linear approaches we’ve seen so far will not produce
CREATION accurate results.
OF DT  Forsuch cases, we need to build our models differently.
Decision trees are a good tool for classifying observations
when the trend is non-linear.
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI  Decision Trees are versatile as they can be used

ON to any kind of problem at hand - classification
or regression. Also, unlike linear models that we
APPLICATIO have studied earlier, decision trees map both
linear and non-linear relationships quite well.
N
 Decision tree outputs are very easy to
understand even for people from a non-
CREATION analytical background. They do not require any
OF DT statistical knowledge to read and interpret
them. Their graphical representation is very
intuitive, and users can easily relate to their
hypothesis.
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

INTRODUCTI
ON  Another major advantage of the
decision tree is that it can handle
APPLICATIO both numerical and categorical
N variables. Therefore, they require
fewer data-cleaning steps compared
CREATION to some other modeling techniques.
OF DT They are also not influenced much
by outliers and missing values to a
fair degree
 CHAPTER 3-DECISION TREE
INTRODUCTION ALGORITHMS

 Decision trees are used to solve both classification

and regression problems. However, there are certain
INTRODUCTI differences between them.
ON  1. Regression trees are used when the dependent
variable is continuous. Classification trees are used
APPLICATIO when the dependent variable is categorical.
N  2. In the case of the regression tree, the value of the
terminal nodes after training is the mean of the
observations. Thus, predictions on unseen data are
CREATION made using the mean.
OF DT  3. In the case of the classification tree, the value or
class of the terminal nodes after training is the mode
of the observations. Thus, predictions on unseen
data are made using the mode.
 Do you know a real-world decision tree that
helped save lives
 In late 1970, Lee Goldman, a U.S. Navy
cardiologist, developed a decision tree to
determine if a person was likely to have a
INTRODUCTION heart attack.
 Lee spent years developing and testing a
APPLICATION single model that would allow submarine
doctors to quickly evaluate possible heart
CREATION OF attack symptoms and determine if the
DT submarine had to resurface and evacuate the
chest pain sufferer.
 This visual and simplified approach to decision
making was one of the first uses of decision
To create a decision tree, you can follow the
steps below.

 Start with the main decision (root):

Place the main objective or question at the
INTRODUCTION top of the tree.
 Draw branches and nodes:
APPLICATION For each possible choice, draw a branch.
 Use square leaf node for further decisions.
 Use circle leaf node for uncertain outcomes.
CREATION OF
The leaf nodes represent the results of each decision.
DT
Add probabilities and values:
Research past data to estimate the
probability and expected value of each
decision. Write these on the branches.
new_data=list([Link]=c(5.1,5.
5,6.3),
[Link]=c(3.5,2.8,3.3),
[Link]=c(1.4,4.2,5.1),
[Link]=c(0.2,1.3,1.8))
 GET MODEL PREDICTIONS ON
TEST

datapredictions_test=predict(tree,new_
data=test, type="response")
predictions_test
GET MODEL PREDICTIONS ON
NEW DATA
predictions_newdata=predict(tree,n
ewdata=new_data,type="response")
predictions_newdata
CHECKING ACCURACY

accuracy=mean(predictions_test==t
est$Species)
accuracy
 [Link]
 [Link]
Entropy – "How messy or uncertain is the group?"
Entropy asks:
“How mixed up is this group? How much surprise is there when I
pick an item?”
•A group with only apples = Entropy = 0 → no surprise.
•A group with equal apples and oranges = Entropy = 1 →
maximum surprise.
Gini Impurity – "How often would we be wrong?"
Think of Gini as asking:
“If I randomly pick something from this group, how likely is it that
I pick the wrong label?”
•If the group has all the same label (e.g., all apples), Gini = 0 →
no chance of being wrong.
•If the group is half apples and half oranges, Gini = 0.5 → high
chance of being wrong.
MTCARS DATASET

 Motor Trend Car Road Tests

 Description
 The data was extracted from the 1974 Motor Trend US magazine,
and comprises fuel consumption and 10 aspects of automobile
design and performance for 32 automobiles (1973–74 models).
 WRITE CODE FOR THE BELOW
QUESTIONS

1. Write the R command to print the contents of the built-in `mtcars` dataset.

2. Write the R command to display the structure of the `mtcars` dataset.

3. Write the R command to display the names of all columns in the `mtcars`
dataset.

4. Write the R command to get a statistical summary of the `mtcars` dataset.

5. Write the R command to find the dimensions (number of rows and columns)
of the `mtcars` dataset.

6. Write the R command to calculate the number of missing values in each

column of the `mtcars` dataset and store the result in a new data frame
named `df`.
ANSWERS

 data=mtcars
 print(data)
 str(data)
 #name of the columns
 colnames(data)
 # Summary of data
 summary(data)
 # dimensions of data dim(data)
 df=[Link](num_missing=colSums([Link](data)))df
 WRITE CODE FOR THE BELOW
QUESTIONS
7. Write the R command to generate a random vector `ind` that splits the `data` into 80%
training and 20% testing sets.

8. Write the R command to display the `ind` vector created in the previous step.

9. Write the R command to create the training dataset using rows of `data` where `ind == 1`.

10. Write the R command to display the first 6 rows of the training dataset.

11. Write the R command to display the dimensions of the training dataset.

12. Write the R command to create the testing dataset using rows of `data` where `ind == 2`.

13. Write the R command to display the first 6 rows of the testing dataset.

14. Write the R command to display the dimensions of the testing dataset.
ind=sample(2,nrow(data),replace=TRUE,prob=c(0.8,0.2))
print(ind)
train=data[ind==1,]
print(head(train))
print(dim(train))
test=data[ind==2,]
print(head(test))
print(dim(test))
CLASSIFICATION ALGORITHM II
KNN
 WHAT IS K-NN?

Imagine you're in a library, and there's a new book placed in the shelf.
You don’t know which genre it belongs to — is it science fiction or
mystery?
So, you look at the 5 books placed closest to it.
You see that:
4 of them are labeled “Mystery”
1 is labeled “Science Fiction”
You might guess:
🟩 “This new book is probably a Mystery book.”
✅ That’s exactly how K-NN works!
 🎯 CORE IDEA OF K-NN:

“Things that are similar tend to

be close together.”
So, to guess the type of a new
object, K-NN looks at what types are
around it.
📦 EXAMPLE 1: CLASSIFICATION – BOXES IN A
WAREHOUSE

You want to know what category the last box belongs to.

Object Weight (kg) Height (cm) Category

Toy Car 1.2 10 Kids Item
Board Game 1.5 12 Kids Item
Tool Kit 5.0 25 Hardware Supply
Hammer 4.8 22 Hardware Supply
Toy cooking set 1.4 11 ???
K-NN looks at the nearest neighbors based on weight and height.
If most of the closest items are Kids Items,
it’ll predict the unknown one as a Kids Item too.
 CHOOSING K

 If K = 1 → Only 1 nearest neighbor is checked.

Risk: That neighbor might be a wrongly labeled item.
 If K = 3 → It checks 3 closest items and picks the
majority label.
This gives more reliable results.
 So, we usually test different values of K and pick the
one with the best results.
 HOW DOES K-NN WORK
INTERNALLY?
 It calculates the distance between the new object and
every object in the dataset.
 It selects the K closest items.
 It looks at their labels.
 It does a majority vote and assigns the most common
label to the new object.
 EXAMPLE

You have a graph with blue squares mostly on the left,

and orange stars mostly on the right.
Now, you add a new shape somewhere in between.
K-NN checks the K nearest shapes:
If 3 are stars and 2 are squares → it says it's a star
If 4 are squares and 1 is a star → it says it's a square
🧠 TWO KEY POINTS ABOUT K-NN:

 It doesn’t learn in advance – It just stores the data.

 It works only when you give it a new input – That’s when it
calculates and predicts.