How to build
Generalizable Machine Learning Models
using
Cross-Validation / Resampling Methods
CSE845 – Applied
Machine Learning
Model validation and verification
• Model verification and validation (V&V) is an enabling methodology
for the development of computational models that can be used to
make engineering predictions with quantified confidence.
• Model verification and validation are the primary processes for
quantifying and building credibility in numerical models.
Model Verification
• Verification of a model is the process of confirming that it is
correctly implemented with respect to the conceptual model (it
matches specifications and assumptions deemed acceptable for
the given purpose of application)
Model Validation
• Validation is the process of determining the degree to which a model
is an accurate representation of the real world from the perspective
of the intended uses of the model
Two Issues
• Model Selection
• Learning Algorithm
• How do we select the “optimal” parameter(s) for a given classification
problem?
• Validation
• Once we have chosen a model, how do we estimate its true error rate?
• The true error rate is the classifier’s error rate when tested on the ENTIRE
POPULATION
Building Machine Learning Models
• If we had access to an unlimited number of examples, these questions
would have a straightforward answer
• Choose the model that provides the lowest error rate on the entire
population
• And, of course, that error rate is the true error rate
• However, in real applications only a finite set of examples is available
• This number is usually smaller than we would hope for!
• Why? Data collection is a very expensive process
Model Training with Data
• One may be tempted to use the entire training data to select the
“optimal” classifier, then estimate the error rate
• This naïve approach has two fundamental problems
• The final model will normally overfit the training data: it will not be able to
generalize to new data. The problem of overfitting is more pronounced with
models that have a large number of parameters
• The error rate estimate will be overly optimistic (lower than the true error
rate). In fact, it is not uncommon to have 100% correct classification on
training data
Train-Test Split
• Split dataset into two groups
• Training set: used to train the classifier
• Test set: used to estimate the error rate of the trained classifier
• In problems where we have a sparse dataset we may not be able to afford
the “luxury” of setting aside a portion of the dataset for testing
• Since it is a single train-and-test experiment, the holdout estimate of error
rate will be misleading if we happen to get an “unfortunate” split
Resampling methods
• Cross Validation
• Random Subsampling
• K-Fold Cross-Validation
• Leave-one-out Cross-Validation
Main Idea
• Cross-validation is a resampling procedure used to
evaluate machine learning models on a limited data
sample
• The procedure has a single parameter called k that
refers to the number of groups that a given data sample
is to be split into
• Cross-validation is primarily used in applied machine
learning to estimate the skill of a machine learning
model on unseen data.
General Procedure
[Link] the dataset randomly.
[Link] the dataset into k groups
[Link] each unique group:
[Link] the group as a hold out or test data set
[Link] the remaining groups as a training data set
[Link] a model on the training set and evaluate it on the test set
[Link] the evaluation score and discard the model
[Link] the skill of the model using the sample of
model evaluation scores
[Link]
Example 1
• Imagine we have a data sample with 6 observations:
• [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]
• The first step is to pick a value for k in order to
determine the number of folds used to split the data.
• Here, we will use a value of k=3.
• That means we will shuffle the data and then split the
data into 3 groups. Because we have 6 observations,
each group will have an equal number of 2
observations.
Example 1
• [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]
• we will use a value of k=3, shuffle and Split Data
in 3 Groups
• Fold1: [0.5, 0.2]
• Fold2: [0.1, 0.3]
• Fold3: [0.4, 0.6]
Example 1
• Fold1: [0.5, 0.2]
• Fold2: [0.1, 0.3]
• Fold3: [0.4, 0.6]
• Three models are trained and evaluated with each fold
given a chance to be the held out test set
• For example:
• Model1: Trained on Fold1 + Fold2, Tested on Fold3
• Model2: Trained on Fold2 + Fold3, Tested on Fold1
• Model3: Trained on Fold1 + Fold3, Tested on Fold2
Example 2
K-Fold Cross-validation
• Create a K-fold partition of the the dataset
• For each of K experiments, use K-1 folds for training and a different
fold for testing
• This procedure is illustrated in the following figure for K=4
K-Fold Cross-validation
• K-Fold Cross validation is similar to Random Subsampling
• The advantage of K-Fold Cross validation is that all the examples in the
dataset are eventually used for both training and testing
• As before, the true error is estimated as the average error rate on test
examples
Leave-one-out Cross Validation
• Leave-one-out is the degenerate case of K-Fold Cross Validation,
where K is chosen as the total number of examples
• For a dataset with N examples, perform N experiments n For each
experiment use N-1 examples for training and the remaining example
for testing
Leave-one-out Cross Validation
• As usual, the true error is estimated as the average error rate on test
examples
Random Subsampling
• Random Subsampling performs K data splits of the entire dataset
• Each data split randomly selects a (fixed) number of examples without
replacement
• For each data split we retrain the classifier from scratch with the training
examples and then estimate Ei with the test examples
Random Subsampling
Implementation of K-
fold cross validation
Using Scikit-Learn
K-fold Cross-Validation
Import Packages
Load data
Test-Train Split and build a
regression model
K-fold Cross Validation
Output
Exercise
• Compute Cross-validated score for your Project.