0% found this document useful (0 votes)

4 views60 pages

Classification

The document outlines a course on classification in data science, focusing on techniques such as k-nearest neighbors and decision trees to classify iris species based on physical measurements. It explains the concepts of supervised learning, dataset splitting for training and testing, and the importance of model evaluation. Additionally, it discusses potential issues like overfitting and the visualization of decision trees for better understanding of classification models.

Uploaded by

Raju Kumar

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

4 views60 pages

Classification

Uploaded by

Raju Kumar

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

Classification

Data Science
COMP5122M
Lecturer: Duygu Sarikaya
What will you learn?
• Define what classification is
• Classify Iris Species using k-nearest neighbors
• Classify Iris Species using Decision Trees
• Building Decision Trees
• Entropy and Defining a Best Feature
Recap

Supervised Learning Unsupervised Learning

Quantitative Categorical
Response Response

Regression Classification Dimensionality Reduction Clustering

Nearest Neighbors Principal Component KMeans Clustering

Logistic Regression Analysis Agglomerative Clustering

Slide Ref: Principles and Techniques of Data Science, UC Berkeley

Recap
In “Unsupervised Learning”:
● Goal is to identify patterns in Unsupervised Learning
unlabeled data.
○ We do not have input/output pairs.

○ Note: Sometimes we may have Dimensionality Reduction Clustering

labels, but we’re just choosing to
ignore them (e.g. PCA on labeled
data).

Dimensionality Reduction.
● PCA.

Clustering.

Slide Ref: Principles and Techniques of Data Science, UC Berkeley

Recap
In “Supervised Learning”:
● Goal is to create a function that maps
inputs to outputs.
Supervised Learning ● Model is learned from example
input/output pairs. Each pair consists of:
Quantitative Categorical
Response Response ○ Input vector.

Regression Classification ○ Output value (label).

● Regression: Output value is quantitative.

● Classification: Output value is

categorical.

Slide Ref: Principles and Techniques of Data Science, UC Berkeley

A First Application: Classifying Iris Species

• A botanist is interested in distinguishing the species of some iris flowers that she
has found. She has collected some measurements associated with each iris: the
length and width of the petals and the length and width of the sepals, all measured
in centimeters

• Our goal is to build a machine learning model that can learn from the
measurements of these irises whose species is known, so that we can predict the
species for a new iris.
A First Application: Classifying Iris Species

• Because we have measurements for which we know the correct species of iris, this
is a supervised learning problem.

• We want to predict one of several options (the species of iris). This is an example of
a classification problem.

• The possible outputs (different species of irises) are called classes.

• Every iris in the dataset belongs to one of three classes, so this problem is a three-
class classification problem.

• The desired output for a single data point (an iris) is the species of this flower. For a
• particular data point, the species it belongs to is called its label.
A First Application: Classifying Iris Species

• Iris Plants Database

• ====================
• Notes
• ----
• Data Set Characteristics:
• :Number of Instances: 150 (50 in each of three classes)
• :Number of Attributes: 4 numeric, predictive attributes
• ...
• ----
A First Application: Classifying Iris Species
We want to build a machine learning model from this data that can predict the species
of iris for a new set of measurements.

How to measure success? Training and Testing Data

To assess the model’s performance, we show it new data (data that it hasn’t seen before)
for which we have labels.

This is usually done by splitting the labeled data we have collected (here, our 150 flower
measurements) into two parts.

One part of the data is used to build our machine learning model, and is called the training
data or training set.

The rest of the data will be used to assess how well the model works; this is called the test
data, test set, or hold-out set.
A First Application: Classifying Iris Species
Shuffling the dataset:
Using a test set containing only one of the three classes would not tell us much about how well our
model generalizes, so we shuffle our data to make sure the test data contains data from all classes.

Splitting the dataset

We can use 75% of the rows in the data as the training set, together with the corresponding labels for
this data. The remaining 25% of the data, together with the remaining labels, is declared as the test
set.

%75, %25
%80, %20
%70, %30
A First Application: Classifying Iris Species
The output of the train_test_split function (scikit-learn) is X_train, X_test, y_train, and y_test, which
are all NumPy arrays. X_train contains 75% of the rows of the dataset, and X_test contains the
remaining 25%:

print("X_train shape: {}".format(X_train.shape))

print("y_train shape: {}".format(y_train.shape))

X_train shape: (112, 4)

y_train shape: (112,)

print("X_test shape: {}".format(X_test.shape))

print("y_test shape: {}".format(y_test.shape))

X_test shape: (38, 4)

y_test shape: (38,)
Three classes seem
to be relatively well
separated using the
sepal and petal
measurements.
A First Application: Classifying Iris Species
Building Your First Model: k-Nearest Neighbors:

There are many classification algorithms in scikit-learn that we could use.

Here we will use a k-nearest neighbors classifier.

• Stores the training set.

• To make a prediction for a new data point, the algorithm finds the point in the training set that is
closest to the new point.

• Then it assigns the label of this training point to the new data point.

The k in k-nearest neighbors signifies that instead of using only the closest neighbor to the new data
point, we can consider any fixed number k of neighbors in the training (for example, the closest three
or five neighbors).

Then, we can make a prediction using the majority class among these neighbors.
A First Application: Classifying Iris Species

Ref: [Link]
A First Application: Classifying Iris Species
In K-NN, K is the number of nearest neighbors. The number of neighbors is the core
deciding factor. K is generally an odd number if the number of classes is 2. When K=1, then
the algorithm is known as the nearest neighbor algorithm. This is the simplest case.

In the below figure, suppose yellow colored “?” let's say P is the point, for which label needs
to predict. First, you find the one closest point to P and then the label of the nearest point
assigned to P.

[Link]
A First Application: Classifying Iris Species
Second, you find the k closest point to
P and then classify points by majority
vote of its K neighbors. Each object
votes for their class and the class with
the most votes is taken as the
prediction. For finding closest similar
points, we find the distance between
points using distance measures such
as Euclidean distance, Hamming
distance, Manhattan distance, and
Minkowski distance. The algorithm has
the following basic steps:

[Link] distance
[Link] closest neighbors
[Link] for labels

[Link]
A First Application: Classifying Iris Species
We formulated the task of predicting which species of iris a particular flower belongs to by using
physical measurements of the flower.

We used a dataset of measurements that was annotated by an expert with the correct species to
build our model, making this a supervised learning task.

There were three possible species, setosa, versicolor, or virginica, which made the task a three-class
classification problem.

The possible species are called classes in the classification problem, and the species of a single iris is
called its label.

We split our dataset into a training set, to build our model, and a test set, to evaluate how well our
model will generalize to new, previously unseen data.
Decision Trees
A Decision Tree is a very simple way to classify data. It is simply a tree of
questions that must be answered in sequence to yield a predicted classification.

How many
legs?
4
2 0

Can it talk? Snake Does it purr?

no yes yes no
Does it understand
Kangaroo Cat Dog
what it’s saying?
yes no

Human Parrot

Slide Ref: Principles and Techniques of Data Science, UC Berkeley

Classifying Iris Species : Using Petal Data Only
The plot below shows the width and length of the petals of each flower, with
the species annotated in the form of color.

We can build a decision tree manually just by looking at this picture.

Slide Ref: Principles and Techniques of Data Science, UC Berkeley

Example: Using Petal Data Only

Slide Ref: Principles and Techniques of Data Science, UC Berkeley

Example: Using Petal Data Only
Is petal_width < 0.75 and
petal_length < 2?
no yes