Introduction to Machine Learning &

Classification Part 1
Data Analytics with Python

1
Course Outline

 Weeks 1-8: Becoming a data analyst

 Data manipulation/ cleaning
 Data aggregation
 Data visualization

 Weeks 9-10: Becoming a data scientist

 Introductory machine learning:
Classification, Clustering, Regression

2
What is Machine Learning?

 A set of tools for extracting knowledge from data and making sense of complex
datasets
 An application of artificial intelligence (capability of a computer system to mimic
human cognitive functions)
 Interdisciplinary subject intersecting statistics, computer science, information
systems
 Aka statistical learning

3
AI, through History

[Link] 4
Also including Gen AI…. (NVIDIA DLI teaching kit)

GenAI (aka Generative AI) refers to a

subset of models and approaches
that have been trained specifically to
generate new data.

This is different to other,

un/supervised machine learning
models that are trained to classify
data samples, or connect input to
output values for regression

Examples of non-GenAI models:

▪ Convolutional Neural Networks
▪ Logistic Regression
▪ Random Forest

Examples of GenAI models:

▪ Large Language Models
▪ Diffusion Models
▪ Generative Adversarial Networks

5
Why care about machine learning?
 Provide descriptive and predictive knowledge to facilitate decisions, e.g.
 Which kinds of customers will return to make another purchase?
 What product should we recommend to a customer?
 How much inventory should we order to meet demand?

 Automate decisions
 E.g. Loan approvals based on creditworthiness, dynamic pricing,
recommendations
 Replace hand-coded rule systems [if this, do that, …]
 Reduce human error
6
Breaking down Machine Learning

Set of tools to model and understand complex datasets.

Two ways of learning:

1. Supervised Learning
2. Unsupervised Learning

7
What is Supervised Learning?

• Supervised Learning:
 Estimate/Predict an output (“correct labels”, Yi) based off of
one or more inputs Xi
 Identify patterns between input variables and output

 2 problem types:
1. Classification: predict results in a
categorical/qualitative output
 Example outputs: loan default (yes/no), diagnosis type
2. Regression: predict results with a
continuous/quantitative output
 Example outputs: income, value of house, stock price

8
Supervised Learning: Examples
 Classification:
 Spam Detection
 Customer Churn
 Disease diagnosis (presence or absence)

 Regression:
 House / Stock Price Predictions
 Demand Forecasting
 Temperature Prediction

9
What is Unsupervised Learning?
 Unsupervised Learning:
 We have inputs/measurements (Xi), but no
specific output/response (Yi) to relate the inputs to
 In other words, we lack a response variable to
“supervise” analysis
 Understand relationships between variables or
observations
 Dimensionality Reduction: simplify representation of
features
 Clustering: partition information into groups/clusters
10
Clustering: Examples

1. Market Segmentation: Identify distinct groups of potential customers

 Goal: Create groups to be reached with distinct marketing strategies
(e.g. frequent customers, price-sensitive customers, occasional customers)
 Data: Demographics (e.g. zip code, family income) and
behaviors/preferences (e.g., shopping habits)

2. Genomics: Group genes together

 Goal: identify genes with similar expressions
 Data: gene expressions (e.g. RNA sequencing)

11
What is Scikit-learn?
scikit-learn is a Python module integrating classic machine learning algorithms in the
tightly-knit world of scientific Python packages (numpy, scipy, matplotlib, statsmodel)

Tools for:
 Classification
 Clustering
 Dimensionality Reduction
 Model Selection
 Regression (but if you are a statistician, you’d prefer statsmodel)
 Preprocessing

12
Scikit-learn: [Link]

13
Classification
In more detail…

14
Classification

 Goal: Explain or predict categorical target (outcome) variable

 Examples: Purchase/no purchase, fraud/no fraud, creditworthy/not creditworthy…
 Data is prepared such that…
 Each row is a case (customer, student, applicant)
 Each column is a variable
 Target variable is often binary (0/1)

15
Classification in a visual….
• Some classifiers are interpretable
• Decision trees Classification Jargon:
• Logistic regression
• Some aren’t • Class: the target attribute that
• Support vector machines we want to predict
Class • Neural network
• Training set: the table
(DataFrame) used to learn
1 0.5 0.4 0 • Classifier: the entity that
Training set

Fit/Train
learns the differences
1.3 2.0 0.3 0
between classes
1.1 3.0 1.3 1 • Fit/train: the task of learning
CLASSIFIER • Predict: after training, the
0.1 2.5 0.3 1 task of predicting the class of
new objects

Test set
Predict
1.4 2.1 0.9 ?
1
16
Fit and predict in classification

• All models (classification and regression) implement at least two methods:

• fit(x, y) - Fit the model to the given dataset
• predict(x) - Predict the y values associated with the x values

17
Classification:
Decision Trees

18
To visualize decision trees – Windows
 Before we start, we need to install a couple of things which will make it possible to
visualize decision trees. (more detail in Jupyter file)
 Make sure your Anaconda environment has : pydotplus and graphviz
 If the Anaconda doesn't have the above two libraries, you can use the following command to install it into Anaconda.
Anaconda -> Environments -> base(root) -> Open Terminal:
 conda install -c conda-forge pydotplus
 conda install -c conda-forge graphviz
 If hit error
 conda install -c conda-forge graphviz python-graphviz

 For Mac users who experience issue installing graphviz, please try the following
steps:
• Go to this website: [Link] Once you launch the website, you will see a line of code under "install Homebrew"
Copy the code to your terminal, and run it (The code looks like this:
/bin/bash -c "$(curl -fsSL [Link]
 After installing Homebrew successfully, copy the code brew install graphviz to your terminal, and run the code

19
Decision Tree in a nutshell
GOAL: build a tree with splitting criteria to predict the class of an object

1. Recursively partition the training set with the goal of minimizing classification
errors, using the “most” helpful attribute
2. Many measures of node impurity to help choose the attribute to partition on:
 Gini impurity: 𝐺𝑖𝑛𝑖 𝑡 = 1 − σ𝑗[𝑃 𝑗 𝑡 2 ]
 Entropy: Entropy t = − σ𝑗[𝑃 𝑗 𝑡 × log(𝑃 𝑗 𝑡 )]
 Classification error: 𝐸𝑟𝑟𝑜𝑟 𝑡 = 1 − max 𝑃(𝑗|𝑡)
𝑗

Let’s see an example…

20
Example: Fountain owners

 Goal: Classify 24 households as owning or not owning fountains

 Attributes = Income, Lot Size

21
 Income Lot size Ownership
59.9 17.4 owner
84.6 16.1 owner
64.3 21.3 owner
60.8 20.2 owner
86.9 22.8 owner We want to build a tree that tells us the
109.3 18.8 owner difference between:
107.1 17.3 owner
82.5 21.6 owner
• Owners: those who own a fountain
68.9 19.8 owner • Non-owners: those who do not
92 19.9 owner
50.4 21.8 owner
80.4 19.7 owner
Training set
74.9 19.5 non-owner
52.3 20 non-owner
64.6 16.6 non-owner
42.8 19.8 non-owner
83.3 17 non-owner
48.9 16.8 non-owner
58.6 15.7 non-owner
65.9 17.5 non-owner
47.2 15.9 non-owner
32.8 18.6 non-owner
50.9 13.6 non-owner
62.2 13.9 non-owner
22
Here is the data set

23
 First split
Decision tree of depth 1

Lot >
18.7?

False True

nonowner owner

Suppose that the one above is

the final tree.

How many classification errors

does this tree make on the
training set?

24
First split
Decision tree of depth 1

Lot >
18.7?

False True

nonowner owner

Let’s split this node

further

25
 Second split
Decision tree of depth 1

Lot >
18.7?

False True
Income
owner
> 84?

False True

nonowner owner

Suppose that the one above is

the final tree.

How many classification errors

does this tree make on the
training set? 26
Second split
Decision tree of depth 2

Lot >
18.7?

False True
Income
owner
> 84?

False True

nonowner owner

Let’s split this node further

27
 Second split
Decision tree of depth 2

Lot >
18.7?

False True
Income Income
> 84? > 58?

False
False True True

nonowner owner

nonowner owner

How many classification errors

does this tree make on the
28
training set?
Training a decision tree in scikit-learn

To view tree

X Y

1 0.5 0.4 0
Fit/Train
1.3 2.0 0.3 0

1.1 3.0 1.3 1

0.1 2.5 0.3 1

29
Predicting Who Has a
Higher Income
Case study

30
Today’s data set

 [Link]
 This data was extracted from the 1994 Census bureau database by Ronny Kohavi
and Barry Becker (Data Mining and Visualization, Silicon Graphics). The
prediction task is to determine whether a person makes over $50K a year.

31
Steps

1. Data Cleaning
2. Building the Model
3. Interpreting our Results
4. Validating our Model

32
Data Cleaning

 Replace ? with [Link]

 Remove education
 Check for unique values
 Make dummy variables for categorical variables
 Make income a binary variable

33
Model Building

34
Interpreting the Tree

The splitting rule

(if this attribute <= 0.5, go to the left
child; else go to the right child)

35
Interpreting the Tree

The “diversity” or ”impurity” of the

population in this node. The tree is
built to minimize diversity in its
leaves

36
Interpreting the Tree

How many objects of the training

set fall into this node

37
Interpreting the Tree

How many belong to class 0 and

how many to class 1

38
Interpreting the Tree

The predicted class if this was a

leaf

39
Those not married civilly and with
low capital gain have low income 40
Those not married civilly and with
high capital gain have high income 41
Those married civilly and with low
education have low income 42
Those married civilly and with high
education have high income 43
[Link] = 12.5 is a good threshold to separate low income from high income, but it is more effective
44
for those married civilly (right part of the tree)
[Link] = 7073 is a good threshold to separate low income from high income, but it is more effective for
45
those NOT married civilly (left part of the tree)