Machine Learning

MKEM1713
Artificial Intelligence
 What is Data Science
Typical predictive analytic Managing the process
goals: that can transform
• who will win an hypotheses & data
Machine
election into actionable
Computer Learning
• what products will sell predictions
well together Science
• which loans will default
• which advertisements Data scientist is responsible
will be clicked on for:
Statistics
• acquiring data
• managing data
• choosing modelling
technique
• writing the code
• verifying results

Predictive Models
 What is Data Science

Some famous examples

Amazon’s product Google’s LinkedIn’s contact

recommendation advertisement recommendation
systems valuation systems system

Walmart’s
Twitter’s trending
consumer demand
topics
projection systems
Data Science Applications
Data Science Applications
Data Science Applications
 Overview of Machine Learning
• Refers to a vast set of tools for understanding data
• These tools can be classified as supervised or unsupervised
• Broadly speaking,
• supervised machine learning involves building a statistical model for
predicting, or inferring, an output based on one or more inputs.
Problems of this nature occur in fields as diverse as business, medicine,
astrophysics, and public policy
• with unsupervised machine learning, there are inputs but no
supervising output; nevertheless we can learn relationships &
structure from such data
Overview of Machine
Learning
Wage Data
Wage data, which contains income
survey information for males from the
central Atlantic region of the United
States
Left: wage as a function of age. On
average, wage increases with age until
about 60 years of age, at which point it
begins to decline
Center: wage as a function of year.
There is a slow but steady increase of
approximately $10,000 in the average
wage between 2003 and 2009
Right: Boxplots displaying wage as a
function of education, with 1
indicating the lowest level (no high
school diploma) & 5 the highest level Age Model Wage
(an advanced graduate degree). On
average, wage increases with the level
of education
Overview of Machine
Learning
Gene Expression Data
Left: Representation of the NCI60 gene
expression data set in a two-
dimensional space, 𝑍𝑍1 & 𝑍𝑍2 . Each point
corresponds to one of the 64 cell lines.
There appear to be 4 groups of cell
lines, which we have represented
using different colours
Right: Same as left panel except that
we have represented each of the 14
different types of cancer using a
different coloured symbol. Cell lines
corresponding to the same cancer type
tend to be nearby in the 2-dimensional
space
Dataset
𝒁𝒁𝟏𝟏 𝒁𝒁𝟐𝟐 Cancer
 What is Machine Learning
• Suppose that we are consultants hired by a client to provide
advice on how to improve sales of a particular product
• The Advertising data set consists of the sales of that product in
200 different markets, along with advertising budgets for the
product in each of those markets for 3 different media: TV,
radio and newspaper
What is Machine Learning?
The Advertising data set. The plot
displays sales, in thousands of units, as
a function of TV, radio & newspaper
budgets, in thousands of dollars, for
200 different markets. In each plot we
show the simple least squares fit of
sales to that variable. In other words,
each blue line represents a simple
model that can be used to predict
sales using TV, radio, & newspaper,
respectively
 What is Machine Learning
• It is not possible for our client to directly increase sales of the
product. On the other hand, they can control the advertising
expenditure in each of the 3 media
• Therefore, if we determine that there is an association
between advertising & sales, then we can instruct our client to
adjust advertising budgets, thereby indirectly increasing sales
• In other words, our goal is to develop an accurate model that
can be used to predict sales on the basis of the 3 media
budgets
• In this setting, the advertising budgets are input variables
(predictors) 𝑋𝑋 while sales is an output variable (response) 𝑌𝑌
• 𝑋𝑋1 - TV budget, 𝑋𝑋2 - radio budget, 𝑋𝑋3 - newspaper budget
 What is Machine Learning
• Generally, suppose we observe a quantitative response 𝑌𝑌 & 𝑝𝑝
different predictors, 𝑋𝑋1 , 𝑋𝑋2 , … , 𝑋𝑋𝑝𝑝
• We assume that there is some relationship between 𝑌𝑌 & 𝑋𝑋 =
(𝑋𝑋1 , 𝑋𝑋2 , … , 𝑋𝑋𝑝𝑝 ), i.e.
𝑌𝑌 = 𝑓𝑓 𝑋𝑋 + 𝜖𝜖
• Here 𝑓𝑓 is some fixed but unknown function of 𝑋𝑋1 , … , 𝑋𝑋𝑝𝑝 , & 𝜖𝜖 is
a random error term, which is independent of 𝑋𝑋 & has mean
zero
• In this formulation, 𝑓𝑓 represents the systematic information
that 𝑋𝑋 provides about 𝑌𝑌
What is Machine Learning?
The Income data set
Left: The red dots are the observed
values of income (in tens of thousands
of dollars) & years of education for 30
individuals
Right: The blue curve represents the
true underlying relationship between
income & years of education, which is
generally unknown (but is known in
this case because the data were
simulated). The black lines represent
the error associated with each
observation. Note that some errors are
positive (if an observation lies above
the blue curve) & some are negative (if
an observation lies below the curve).
Overall, these errors have
approximately mean zero
What is Machine Learning?
The plot displays income as a function
of years of education & seniority in the
Income data set. The blue surface
represents the true underlying
relationship between income & years
of education & seniority, which is
known since the data are simulated.
The red dots indicate the observed
values of these quantities for 30
individuals

In essence, machine learning refers to

a set of approaches for estimating
𝑓𝑓.
 Types of Machine Learning
Techniques
Most machine learning
problems fall into 1 of 2 Supervised
categories Unsupervised

• For each observation of the predictors 𝑥𝑥𝑖𝑖 , • For every observation 𝑖𝑖 = 1, . . . , 𝑛𝑛, we
𝑖𝑖 = 1, . . . , 𝑛𝑛 there is an associated response observe a vector 𝑥𝑥𝑖𝑖 but no associated
𝑦𝑦𝑖𝑖 response 𝑦𝑦𝑖𝑖
• Wish to fit a model that relates the • No response variable to predict
response to the predictors. • Referred to as unsupervised because we lack
• Aim: to accurately predict the response for a response variable that can supervise our
future observations (prediction) or to analysis
better understand the relationship • Aim: to understand the relationships
between the response & the predictors between the variables or between the
(inference) observations
• Methods: linear regression & logistic • Method: cluster analysis, or clustering. Goal:
regression to ascertain whether the observations fall
into relatively distinct groups
 Types of Machine Learning
Techniques
Machine
Learning

Prediction Reason Reason

Supervised Unsupervised Inference
Inference

Regression y is numeric
x Model y

Classification y is class
 Exercises
• Explain whether each scenario is a classification or regression problem, & indicate
whether we are most interested in inference or prediction.
• We collect a set of data on the top 500 firms in the US. For each firm we record
profit, number of employees, industry & the CEO salary. We are interested in
understanding which factors affect CEO salary.
• We are considering launching a new product & wish to know whether it will be
a success or a failure. We collect data on 20 similar products that were
previously launched. For each product we have recorded whether it was a
success or failure, price charged for the product, marketing budget,
competition price, & ten other variables.
• We are interesting in predicting the % change in the US dollar in relation to the
weekly changes in the world stock markets. Hence we collect weekly data for
all of 2012. For each week we record the % change in the dollar, the % change
in the US market, the % change in the British market, & the % change in the
German market.
 Exercises
• You will now think of some real-life applications for machine learning.
• Describe three real-life applications in which classification might be useful.
Describe the response, as well as the predictors. Is the goal of each application
inference or prediction? Explain your answer.
• Describe three real-life applications in which regression might be useful.
Describe the response, as well as the predictors. Is the goal of each application
inference or prediction? Explain your answer.
• Describe three real-life applications in which cluster analysis might be useful.
Flow of Creating & Evaluating Models
Model evaluation
• Quantifying the performance of a
model
• Must use a measure of model
performance that’s appropriate to
both the original business goal & the
chosen modelling technique

Predicting who would default on Predicting revenue lost to

loans (classification) defaulting loans (regression)
Accuracy RMSE

Precision
Flow of Creating & Evaluating Models
Model validation
• Generation of an assurance that the
model will work in production as it
worked during training
• Biggest cause of model validation
failures – not having enough training
data to represent the variety of what
may later be encountered in
production
 Test & Training Splits
• When you’re building a model to make predictions, you need
data to build the model (training set)
• You also need data to test whether the model makes correct
predictions on new data (test or hold-out set)

Training
Dataset Test set
set

Data that you feed to the model-building Data that you feed into the resulting
algorithm (regression, decision tree, etc.) model, to verify that the model’s
so that the algorithm can set the correct predictions are accurate
parameters to best predict the outcome
variable
 Evaluating Classification Models
• When building a model, the 1st thing to check is if the model
even works on the data it was trained from
• Example of classifying email into spam (email we in no way
want) & non-spam (email we want)
• Summary of classifier performance – confusion matrix (table
that summarizes the classifier’s predictions against the actual
known data categories)
Confusion matrix
Predicted condition Predicted condition

Negative Positive non-spam spam

True Negative TN FP True non-spam 246 17
condition condition
Positive FN TP spam 18 180
 Evaluating Classification Models
Measures of Classifier
Performance

Accuracy Precision Recall

• For a classifier, accuracy is defined as the number of items categorized correctly divided by
total number of items – what fraction of the time the classifier is correct
𝑇𝑇𝑇𝑇+𝑇𝑇𝑇𝑇
• Accuracy = = (cM[1,1] + cM[2,2]) / sum(cM) = 0.924
𝑇𝑇𝑇𝑇+𝐹𝐹𝐹𝐹+𝑇𝑇𝑇𝑇+𝐹𝐹𝐹𝐹
• The error of around 8% is unacceptably high for a spam filter!
Predicted condition Predicted condition

Negative Positive non-spam spam

True Negative TN FP True non-spam 246 17
condition condition
Positive FN TP spam 18 180
 Evaluating Classification Models
Measures of Classifier
Performance

Accuracy Precision Recall

• Precision is what fraction of the items the classifier flags as being in the class actually are in
the class - how often a positive indication turns out to be correct
𝑇𝑇𝑇𝑇
• Precision = = cM[2,2] / (cM[2,2] + cM[1,2]) = 0.914
𝑇𝑇𝑇𝑇+𝐹𝐹𝐹𝐹
• 91.4% precision means 8.6% of what was flagged as spam was in fact not spam!
• A measure of confirmation (when the classifier indicates positive, how often it is in fact
correct)
Predicted condition Predicted condition

Negative Positive non-spam spam

True Negative TN FP True non-spam 246 17
condition condition
Positive FN TP spam 18 180
 Evaluating Classification Models
Measures of Classifier
Performance

Accuracy Precision Recall

• Recall is what fraction of the things that are in the class are detected by the classifier
𝑇𝑇𝑇𝑇
• Recall = = cM[2,2] / (cM[2,2] + cM[2,1]) = 0.909
𝑇𝑇𝑇𝑇+𝐹𝐹𝐹𝐹
• A measure of utility (how much the classifier finds of what there actually is to find)

Predicted condition Predicted condition

Negative Positive non-spam spam

True Negative TN FP True non-spam 246 17
condition condition
Positive FN TP spam 18 180
 Validating Models
• Model evaluation: performance of the model on training data
• Biggest worry: validity of the model – will it show similar
quality on new data in production?
• Model validation: testing of a model on new data (test set)
 Validating Models
A common model problem: Overfitting
• An overfit model looks
great on the training
data & performs poorly
on new data
• Memorized the training
data instead of
discovering generalizable
rules or patterns
• Overfit model is bad:
– more complicated
than anything useful
– less accurate in
production
 Ensuring Model Quality
• The data used to build a model is not the best data for testing
the model’s performance
• Because this data was seen during model construction, &
model construction is optimizing your performance measure,
you tend to get exaggerated measures of performance on your
training data
• Perform all of your clever work on the training data alone, &
delay measuring your performance with respect to your test
data until as late as possible in your project – testing on held-
out data
Prediction with Decision
Trees

30
 Decision Trees
• Decision tree predict responses to data
• To predict a response, follow the decisions in the tree from the
root (beginning) node down to a leaf node. The leaf node
contains the response.
 Decision Trees
• This tree predicts classifications based on 2 predictors, x1 & x2
• To predict, start at the top node, represented by a triangle (Δ).
The 1st decision is whether x1<0.5. If so, follow the left branch,
& see that the tree classifies the data as type 0.
• If x1>=0.5, then follow the right branch to the lower-right
triangle node. Here the tree asks if x2<0.5. If so, then follow
the left branch to see that the tree classifies the data as type 0.
If not, then follow the right branch to see that the that the tree
classifies the data as type 1.
 Predict if John will play tennis
Day Outlook Humidity Wind Play
Training examples: 1 Sunny High Weak No New data:-
9 Yes / 5 No 2 Sunny High Strong No Outlook : Rain
Humidity: High
3 Overcast High Weak Yes Wind : Weak
4 Rain High Weak Yes Play :?
5 Rain Normal Weak Yes
6 Rain Normal Strong No
7 Overcast Normal Strong Yes
8 Sunny High Weak No
9 Sunny Normal Weak Yes
10 Rain Normal Weak Yes
11 Sunny Normal Strong Yes
12 Overcast High Strong Yes
13 Overcast Normal Weak Yes
14 Rain High Strong No
 Predict if John will play tennis
• Hard to guess
• Try to understand when John plays
• Divide & conquer:
• split into subsets
• are they pure? (all yes or all no)
• if yes: stop
• if not: repeat
• See which subset new data falls into
 Predict if John will play tennis
Day Outlook Humidity Wind Play
Training examples: 1 Sunny High Weak No New data:-
9 Yes / 5 No 2 Sunny High Strong No Outlook : Rain
Humidity: High
3 Overcast High Weak Yes Wind : Weak
4 Rain High Weak Yes Play :?
5 Rain Normal Weak Yes
6 Rain Normal Strong No
7 Overcast Normal Strong Yes
8 Sunny High Weak No
9 Sunny Normal Weak Yes
10 Rain Normal Weak Yes
11 Sunny Normal Strong Yes
12 Overcast High Strong Yes
13 Overcast Normal Weak Yes
14 Rain High Strong No
Predict if John will play tennis
Outlook 9 Yes / 5 No

Sunny Overcast Rain

Day Outlook Humidity Wind
3 Overcast High Weak
7 Overcast Normal Strong
12 Overcast High Strong
13 Overcast Normal Weak
2 Yes / 3 No 4 Yes / 0 No 3 Yes / 2 No
Day Outlook Humidity Wind Pure subset Day Outlook Humidity Wind
1 Sunny High Weak 4 Rain High Weak
2 Sunny High Strong 5 Rain Normal Weak
8 Sunny High Weak 6 Rain Normal Strong
9 Sunny Normal Weak 10 Rain Normal Weak
11 Sunny Normal Strong 14 Rain High Strong
Split further Split further
Predict if John will play tennis
Outlook 9 Yes / 5 No

Sunny Overcast Rain

4 Yes / 0 No
Pure subset

2 Yes / 3 No 3 Yes / 2 No
Day Outlook Humidity Wind Day Outlook Humidity Wind
1 Sunny High Weak 4 Rain High Weak
2 Sunny High Strong 5 Rain Normal Weak
8 Sunny High Weak 6 Rain Normal Strong
9 Sunny Normal Weak 10 Rain Normal Weak
11 Sunny Normal Strong 14 Rain High Strong
Split further Split further
 Predict if John will play tennis
Outlook 9 Yes / 5 No
Sunny Overcast
Humidity
Yes Rain

High Normal
Day Humidity Wind
1 High Weak
2 High Strong
8 High Weak
3 Yes / 2 No
Day Humidity Wind Day Outlook Humidity Wind
9 Normal Weak 4 Rain High Weak
11 Normal Strong 5 Rain Normal Weak
6 Rain Normal Strong
10 Rain Normal Weak
14 Rain High Strong
Split further
 Predict if John will play tennis
Outlook 9 Yes / 5 No
Sunny Overcast Rain
Humidity Wind
Yes
High Normal Weak Strong
Day Humidity Wind Day Humidity Wind
1 High Weak 6 Normal Strong
2 High Strong 14 High Strong
8 High Weak

Day Humidity Wind Day Humidity Wind

9 Normal Weak 4 High Weak
11 Normal Strong 5 Normal Weak
10 Normal Weak
 Predict if John will play tennis
Outlook 9 Yes / 5 No
Sunny Overcast Rain
Humidity Wind
High Normal Yes Weak Strong
No Yes Yes No

New data:-
Outlook : Rain
Humidity: High
Wind : Weak
Play :?
 Predict if John will play tennis
Outlook 9 Yes / 5 No
Sunny Overcast Rain
Humidity Wind
High Normal Yes Weak Strong
No Yes Yes No

New data:-
Outlook : Rain
Humidity: High
Wind : Weak
Play : Yes
 Predict if John will play tennis
Outlook 9 / 5
Sunny 2 / 3 Overcast 4 / 0 Rain 3 / 2
Humidity Wind
High 0 / 3 Normal 2/0 Yes Weak 3 / 0 Strong 0 / 2
No Yes Yes No

New data:-
Outlook : Rain
Humidity: High
Wind : Weak
Play : Yes
Prediction with K-
Nearest Neighbour

43
Using nearest neighbour methods
• The k-nearest neighbour (KNN) method scores an example by
finding the 𝐾𝐾 training examples nearest to the example & then
taking the average of their outcomes as the score
• Given a positive integer 𝐾𝐾 and a test observation 𝑥𝑥0 , the KNN
classifier first identifies the 𝐾𝐾 points in the training data that
are closest to 𝑥𝑥0 , represented by 𝒩𝒩0
• It then estimates the conditional probability for class 𝑗𝑗 as the
fraction of points in 𝒩𝒩0 whose response values equal 𝑗𝑗:

1
Pr 𝑌𝑌 = 𝑗𝑗 𝑋𝑋 = 𝑥𝑥0 = � 𝐼𝐼(𝑦𝑦𝑖𝑖 = 𝑗𝑗)
𝐾𝐾
𝑖𝑖∈𝒩𝒩0

44
• Here 𝐼𝐼(𝑦𝑦𝑖𝑖 = 𝑗𝑗) is an indicator variable that equals 1 if 𝑦𝑦𝑖𝑖 = 𝑗𝑗
and 0 if 𝑦𝑦𝑖𝑖 ≠ 𝑗𝑗
• KNN then classifies the test observation 𝑥𝑥0 to the class with
the largest probability

45
K-Nearest Neighbours
In the left-hand panel, we have plotted
a small training data set consisting of
six blue and six orange observations.
Our goal is to make a prediction for the
point labelled by the black cross.
Suppose that we choose 𝐾𝐾 = 3. Then
KNN will first identify the 3
observations that are closest to the
cross. This neighbourhood is shown as
a circle. It consists of 2 blue points and
1 orange point, resulting in estimated
probabilities of 2/3 for the blue class
and 1/3 for the orange class. Hence
KNN will predict that the black cross
belongs to the blue class. In the right-
hand panel of the figure we have
applied the KNN approach with 𝐾𝐾 = 3
at all of the possible values for 𝑋𝑋1 and
𝑋𝑋2 , and have drawn in the
corresponding KNN decision boundary

46
K-Nearest Neighbours
The choice of 𝐾𝐾 has a drastic effect on
the KNN classifier obtained

A comparison of the KNN decision

boundaries (solid black curves)
obtained using 𝐾𝐾 = 1 and 𝐾𝐾 = 100 on
the data. With 𝐾𝐾 = 1, the decision
boundary is overly flexible, while with
𝐾𝐾 = 100 it is not sufficiently flexible

For a good K , try something such that

you have a good chance of seeing 10
positive example in each
neighbourhood (Tutorial 4.2)

47
Prediction with Logistic
Regression

48
• Consider the Default data set, where the response
default falls into one of two categories, Yes or No
• Logistic regression models the probability that 𝑌𝑌
belongs to a particular category
• For the Default data, logistic regression models the
probability of default
• For example, the probability of default given
balance can be written as
Pr(𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 = 𝑌𝑌𝑌𝑌𝑌𝑌|𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏)
• The values of 𝑝𝑝(𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏), will range between 0 and
1
• Then for any given value of balance, a prediction
can be made for default

49
• For example, one might predict default = Yes for
any individual for whom 𝑝𝑝 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 > 0.5
• Alternatively, if a company wishes to be
conservative in predicting individuals who are at risk
for default, then they may choose to use a lower
threshold, such as 𝑝𝑝 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 > 0.1

50
Case Study
Classification using the Default data
Left: Estimated probability of default
using linear regression. Some
estimated probabilities are negative!
The orange ticks indicate the 0/1
values coded for default (No or Yes)
Right: Predicted probabilities of
default using logistic regression. All
probabilities lie between 0 and 1

51
The logistic model
• We model 𝑝𝑝(𝑋𝑋) using a function that gives outputs
between 0 and 1 for all values of 𝑋𝑋
• In logistic regression, we use the logistic function
𝑒𝑒 𝛽𝛽0 +𝛽𝛽1 𝑋𝑋
𝑝𝑝 𝑋𝑋 =
1 + 𝑒𝑒𝛽𝛽0 +𝛽𝛽1 𝑋𝑋
• To fit the model, we use a method called maximum
likelihood
• The logistic function will always produce an S-
shaped curve of this form, and so regardless of the
value of 𝑋𝑋, we will obtain a sensible prediction

52
• Regardless of the value of 𝑋𝑋, if 𝛽𝛽1 > 0 then
increasing 𝑋𝑋 will be associated with increasing 𝑝𝑝(𝑋𝑋),
and if 𝛽𝛽1 < 0 then increasing 𝑋𝑋 will be associated
with decreasing 𝑝𝑝(𝑋𝑋)

53
Estimating the regression coefficients
• Coefficients 𝛽𝛽0 and 𝛽𝛽1 are unknown, and must be
estimated based on the available training data
• The more general method of maximum likelihood is
preferred, since it has better statistical properties
• Intuition: we seek estimates for 𝛽𝛽0 and 𝛽𝛽1 such that
the predicted probability 𝑝𝑝(𝑥𝑥 ̂ 𝑖𝑖 ) of default for each
individual, corresponds as closely as possible to the
individual’s observed default status
• Try to find 𝛽𝛽̂0 and 𝛽𝛽̂1 such that plugging these
estimates into the model for 𝑝𝑝(𝑋𝑋), yields a number
close to 1 for all individuals who defaulted, and a
number close to 0 for all individuals who did not

54
• Maximum likelihood is a very general approach that
is used to fit many of the non-linear models
• In the linear regression setting, the least squares
approach is in fact a special case of maximum
likelihood

55
Making predictions
• Once the coefficients have been estimated, it is a
simple matter to compute the probability of default
for any given credit card balance
• For example, using the coefficient estimates given
in the table, we predict that the default probability
for an individual with a balance of $1,000 is

𝑝𝑝 𝑋𝑋 = 𝑒𝑒 𝛽𝛽 �1 𝑋𝑋
�0 +𝛽𝛽

1 + 𝑒𝑒𝛽𝛽�0 +𝛽𝛽�1 𝑋𝑋
= 𝑒𝑒 −10.6513+0.0055×1,000 = 0.00576
1 + 𝑒𝑒 −10.6513+0.0055×1,000

56
Making predictions
• which is below 1%. In contrast, the predicted
probability of default for an individual with a
balance of $2, 000 is much higher, and equals 0.586
or 58.6%

57
• We now consider the problem of predicting a
binary response using multiple predictors
𝑒𝑒 𝛽𝛽0 +𝛽𝛽1 𝑋𝑋1 +⋯+𝛽𝛽𝑝𝑝 𝑋𝑋𝑝𝑝
𝑝𝑝 𝑋𝑋 =
1 + 𝑒𝑒 𝛽𝛽0 +𝛽𝛽1 𝑋𝑋1 +⋯+𝛽𝛽𝑝𝑝 𝑋𝑋𝑝𝑝
• We use the maximum likelihood method to estimate
𝛽𝛽0 , 𝛽𝛽1 , … , 𝛽𝛽𝑝𝑝

58
• As student with a credit card balance of $1,500 &
an income of $40,000 has an estimated probability
of default of

• A non-student with the same balance & income

59
Unsupervised Learning

60
• The methods that we’ve discussed in previous
chapters build models to predict outcomes
• In this chapter – methods to discover unknown
relationships in data – unsupervised methods
• Unsupervised methods – no outcome to predict;
instead, discover patterns in data
• E.g.:
 find groups of customers with similar purchase patterns
 correlations between population movement &
socioeconomic factors
• Not ends in themselves; rather, ways of finding
relationships & patterns that can be used to build
predictive models

61
Cluster analysis
• One class of unsupervised methods
• Goal: to group data into clusters such that every
datum in a cluster is more similar to other datums in
the same cluster than is it to datums in other clusters

62
• For example, a company that offers guided tours
might want to cluster its clients by behaviour &
tastes:
 which countries they like to visit
 whether they prefer adventure tours, luxury tours or
educational tours
 what kinds of activities they participate in
 what sorts of sites they like to visit
• Such information can help the company design
attractive travel packages & target the appropriate
segments of their client base with them
• 2 approaches: hierarchical clustering & k-means
clustering

63
• In order to cluster, you need the notions of similarity
& dissimilarity
• Dissimilarity can be thought of as distance - points in
a cluster are closer to each other than they are to
the points in other clusters

64
• Different application areas will have different
notions of distance and dissimilarity.
• In this section, we’ll cover Euclidean distance

65
Euclidean distance
• The most common distance is Euclidean distance
• The Euclidean distance between two vectors x and
y is defined as
edist(𝑥𝑥, 𝑦𝑦) = (𝑥𝑥 1 − 𝑦𝑦[1])2 +(𝑥𝑥 2 − 𝑦𝑦[2])2 + ⋯

• This is the measure people tend to think of when

they think of “distance”
• Optimizing squared Euclidean distance is the basis
of k-means
• Euclidean distance only makes sense when all the
data is real-valued (quantitative)

66
• K-means is a popular clustering algorithm when the
data is all numeric and the distance metric is
squared Euclidean
• It’s fairly ad hoc and has the major disadvantage
that you must pick k in advance
• On the plus side, it’s easy to implement (one reason
it’s so popular) and can be faster than hierarchical
clustering on large datasets

67
The kmeans() function
• The function to run k-means in Matlab is kmeans().
The output of kmeans() are the cluster labels. The k-
means algorithm with k = 2:

68
 Questions
1. What is machine learning?
2. Explain machine learning techniques and its categories
3. Why do we need to evaluate our model?
4. Why do we need a portion of our data called test data?
5. Why do we need to validate our model?