UNDERSTANDING DATA

(MODULE 2)
Bivariate Data and Multivariate Data
• Bivariate data deals with causes of relationship among data between two variables.
Example:
• Consider data of temperature and sales of sweaters in a shop.

Here,
• The aim of bivariate analysis is to find relationships among variables.
• The relationships can then be used in comparisons, finding causes, and in further explorations using
graphical display of the data using any graph method called scatter plot.
Scatter plot
• It is used to visualize bivariate data.
• It is useful to plot two variables with or without nominal variables, to illustrate the trends, and also to show differences.
• It is a plot between explanatory and response variables.
• It is a 2D graph showing the relationship between two variables.
• The scatter plot indicates strength, shape, direction and the presence of Outliers.
• It is useful in exploratory data before calculating a correlation coefficient or fitting regression curve.
Bivariate Statistics
• Covariance and Correlation are examples of bivariate statistics.

Covariance
• is a measure of joint probability of random variables, say X and Y.
• It is defined as covariance(X, Y) or COV(X, Y)
• It is used to measure the variance between two dimensions.
• The formula for finding co-variance for specific x, and y are:
Correlation
• Correlation coefficient is a test for determining any association between two phenomena.
• It measures the strength and direction of a linear relationship between the x and y variables.
• The correlation indicates the relationship between dimensions using its sign.
1. If the value is positive, it indicates that the dimensions increase together.
2. If the value is negative, it indicates that while one-dimension increases, the other dimension
decreases.
3. If the value is zero, then it indicates that both the dimensions are independent of each other.
• If the given attributes are X = (x1, x2,…. xN) and Y=(y1, y2,……yN), then the correlation coefficient is
denoted as r, is given as:
Multivariate Statistics
• Multivariate data is the analysis of more than two observable variables
❖ Thousands of multiple measurements need to be conducted for one or more subjects.
• Multivariate analysis
1. Regression analysis
2. Principal component analysis
3. Path analysis

• The mean of multivariate data is a mean vector and the mean of the above three attributes is given
as (2, 5, 1.33)
Heat Map
• Heatmap is a graphical representation of 2D matrix.
• It takes a matrix as input and colors it.
❖ The darker colors indicate very large values
❖ The lighter colors indicate smaller values
• The advantage of this method is that by color shaping, larger values can be perceived well.

Example
1. In vehicle traffic data, heavy traffic regions can be differentiated from low traffic regions through
heatmap.
2. Patient data highlighting weight and health status is plotted. Here,
❖ X-axis is weights and Y-axis is patient counts
❖ he dark color regions highlight patients' weights vs patient counts in health status
PairPlot
• Pairplot or scatter matrix is a data visual technique for multivariate data.
• A scatter matrix consists of several pair-wise scatter plots of variables of the multivariate data.
• All the results are presented in a matrix format.
• By visual examination of the chart, one can easily find relationships among the variables such as
correlation between the variables.
Example: A random matrix of three columns is chosen and the relationships of the columns is plotted
as a pairplot (or scattermatrix).
Essential Mathematics for Multivariate Data
• Linear Algebra plays a major role as it deals with mathematics of data.
• Linear algebra deals with
1. Linear equations
2. Vectors
3. Matrices
4. Vector spaces
5. Transformations

Linear Systems and Gaussian Elimination for Multivariate Data

• A linear system of equation is a group of equations with unknown variables
• To facilitate the application of Gaussian elimination method following row operation are applied:
1. Swapping the rows
2. Multiplying or dividing a row by a constant
3. Replacing a row by adding or subtracting a multiple of another row to it
Matrix Decompositions
• Method to reduce matrix to its constituent parts is known as matrix factorization methods
• Popular matrix decomposition is called eigen decomposition
❖ Which deals with a way of reducing the matrix into eigen values and eigen vectors.
LU Decomposition
• It is a simplest matrix decompositions where the matrix A can be decomposed matrices:
A = LU
Here,
❖ L is the lower triangular matrix
❖ U is the upper triangular matrix

• The decomposition can be done using Gaussian elimination method.

Step 1: An identity matrix is augmented to the given matrix.
Step 2: Row operations and Gaussian elimination is applied to reduce the given matrix to get
matrices L and U.
Machine Learning and Importance of Probability and Statistics
• Statistics is heart of machine learning.
• Probability is needed for better understanding of machine learning.

Probability Distributions
• A probability distribution of a variable, say X, summarizes the probability associated with X's events.
• Distribution is a function that describes the relationship between the observations in a sample
space.

• Probability distributions are of two types:

1. Discrete probability distribution
2. Continuous probability distribution
Continuous Probability Distributions
• The relationships between the events for a continuous random variable and their probabilities is
called a continuous probability distribution.
• It is summarized as Probability Density Function (PDF).
• PDF calculates the probability of observing an instance.
• The plot of PDF shows the shape of the distribution.
• Continuous Probability Distributions types are
1. Normal distributions
2. Rectangular distributions
3. Exponential distributions
Normal Distribution
• Normal distribution is a continuous probability distribution.
• It is also known as gaussian distribution or bell-shaped curve distribution.
• It is the common distribution function. In normal distribution, data tends to be around a central
value with no bias on left or right.
• Example: The heights of the students, blood pressure of a population, and marks scored in a class
can be approximated using normal distribution.
Rectangular Distributions
• This is known as uniform distribution.
• It has equal probabilities for all values in the range a, b.
• It is given as
Exponential Distribution
• This is a continuous uniform distribution.
• This probability distribution is used to describe the time between events in a Poisson process.
• This distribution is helpful in modelling of time until an event occurs
• This is a special case of Gamma distribution with a fixed parameter of 1.
Discrete Distribution
• There are three type of Discrete distribution
1. Binomial
2. Poisson
3. Bernoulli

Binomial Distribution
• It has only two outcomes: success or failure
• This is called Bernoulli trial
Poisson Distribution
• This distribution is used to model the probability of a given number of events k.
• The mean rule λ is inclusive of previous events.
• Example:
1. Emails received
2. Number of customers visiting a shop
3. Number of phone calls received by the office
Bernoulli Distribution
• This distribution models an experiment whose outcome is binary
• The outcome is positive with p and negative with 1 – p
• PMF of this distribution is given as

where,
• The mean is p and variance is p(1-p)=q
Density Estimation
• Density estimation is the problem of estimating the density function from an observed data.
• The estimated density function, denoted as, p(x)
❖ Can be used to value directly for any unknown data, say x1 as p(x1).
• If its value is less than £1 then x1 is not an outlier or anomaly data.
❖ Else, it is categorized as an anomaly/ inconsistency data.
• There are two types of density estimation methods
1. Parametric density estimation
2. Non-parametric density estimation
Parametric Density Estimation
• It assumes that the data is from a known probabilistic distribution and can be estimated as
p(x | θ )
where,
❖ θ is the parameter.
❖ Maximum likelihood function is a parametric estimation method.

Maximum Likelihood Estimation

• Maximum Likelihood Estimation (MLE) is a probabilistic framework that can be used for density
estimation.
• This involves formulating a function called likelihood function which is the conditional probability of
observing the observed samples and distribution function with its parameters.
Example:
• If the observations are X = {x1, x2, …xn), then density estimation is the problem of choosing a PDF
with suitable parameters to describe the data.
• MLE treats this problem as a search or optimization problem where the probability should be
maximized for the joint probabilities of X and its parameter, θ.
Gaussian Mixture Model and Expectation –Maximization (EM) Algorithm
• Used for estimating the MLE in the presence of latent or missing variables.

Non-parametric Density Estimation

• A non-parametric estimation can be generative or discriminative.
• Parzen window is a generative estimation method that finds p(x | θ ) as conditional density.
• Discriminative methods directly compute p(θ | x) as posteriori probability.
• Parzen window and k-Nearest Neighbour (KNN) rule are examples of non-parametric density
estimation.
Feature Engineering and Dimensionality Reduction Techniques
• Feature engineering is about determining the subset of features that form an important part of the
input that improves the performance of the model, be it classification or any other model in machine
learning.
• Feature engineering deals with two problems
1. Feature Transformation
❖ Feature transformation is extraction of features and creating new features that may be helpful
in increasing performance.
❖ Example - the height and weight may give a new attribute called Body Mass Index (BMI).
2. Feature Selection
❖ Focuses on selection of features to reduce the time but not at the cost of reliability.
❖ The subset selection reduces the dataset size by removing irrelevant features and constructs a
minimum set of attributes for machine learning.
• The features can be removed based on two aspects:
1. Feature relevancy
• Features contribute more for classification than other features.
• Example - a mole on the face can help in face detection than common features like nose.
2. Feature redundancy
• Features are redundant.
• Example - when a database table has a field called Date of birth, then age field is not relevant as
age can be computed easily from date of birth. This helps in removing the column age that leads to
reduction of dimension one.
• So, the procedure is:
1. Generate all possible subsets
2. Evaluate the subsets and model performance
3. Evaluate the results for optimal feature selection
• Filter-based selection uses statistical measures for assessing features.
❖ In this approach, no learning algorithm is used.

• Wrapper-based methods use classifiers to identify the best features.

❖ These are selected and evaluated by the learning algorithms.
❖ Below are algorithm related to this
1. Stepwise Forward Selection
2. Stepwise backward Elimination
3. Principal Component Analysis
4. Linear Discriminant Analysis
5. Singular Value Decomposition
1. Stepwise Forward Selection
• This procedure starts with an empty set of attributes.
• Every time, an attribute is tested for statistical significance for best quality and is added to the
reduced set.
• This process is continued till a good reduced set of attributes is obtained.

2. Stepwise Backward Elimination

• This procedure starts with a complete set of attributes.
• At every stage, the procedure removes the worst attribute from the set leading to the reduced set.
3. Principal Component Analysis
• It transform a given set of measurements to a new set of features so that the features exhibit high
information packing properties.
• Consider a group of random vectors of the form:

• The mean vector of the set of random vectors is defined as:

mx = E(x)
❖ E refers to expected value of the population
• Covariance matrix can be calculated as
• For M random vectors, when M is large enough, the mean vector and covariance matrix can be
approximately calculated as:

• This covariance matrix is real and symmetric.

• If ei, and λi (where, i = 1, 2, ..., n) be the set of eigen vectors and corresponding eigen values of the
covariance matrix, the eigen values can be arranged in a descending order so that λi, ≥ λi+1 for i = 1,
2, ..., n - 1.
• The corresponding eigen vectors are calculated.
• Based on this, the transform kernel is constructed.
• Let the transform kernel be A. Then, the matrix rows are formed from the eigen vectors of the
covariance matrix.
• The mapping of the vectors x to y using the transformation can now be described as:
Y = A(x-mx)
• This transform is also called as Karhunen-Loeve or Hoteling transform.
• The original vector x can now be reconstructed as follows:
x = A T y + mx
• If K largest eigen values are used, the recovered information would be:
X= ATKy + mx
• The advantages of PCA reduces the attribute list by eliminating all irrelevant attributes.
• The PCA algorithm is as follows:
1. The target dataset x is obtained
2. The mean is subtracted from the dataset. Let the mean be m. Thus, the adjusted dataset is X-m.
The objective of this process is to transform the dataset with zero mean.
3. The covariance of dataset x is obtained. Let it be C.
4. Eigen values and eigen vectors of the covariance matrix are calculated.
5. The eigen vector of the highest eigen value is the principal component of the dataset.
The eigen values are arranged in a descending order. The feature vectors is formed.
Feature vector = {eigen vector1, eigen vector2,..., eigen vectorN }
6. Obtain the transpose of feature vector. Let it be A.
7. PCA transform is y = A x (x-m), where x is the input dataset, m is the mean and A is the transpose
of the feature vector.
• The original data can be retrieved using the formula given below
Linear Discriminant Analysis (LDA)
• LDA is to project higher dimension data to a line (lower dimension data).
❖ Let there be two classes c1 and c2
❖ Let μ1 and μ2 be the mean of the patterns of two classes.
❖ The mean of the class c1 and c2 can be computed as
Singular Value Decomposition
• Singular Value Decomposition (SVD) is a decomposition technique.
• Let A be the matrix, then the matrix A can be decomposed as:
A = USVT
Here,
❖ A is the given matrix of dimension m x n
❖ U is the orthogonal matrix whose dimension is mx n
❖ S is the diagonal matrix of dimension n x n
❖ V is the orthogonal matrix.
• The procedure for finding decomposition matrix is given as follows:
1. For a given matrix, find AAT
2. Find eigen values of AAT
3. Sort the eigen values in a descending order. Pack the eigen vectors as a matrix U.
4. Arrange the square root of the eigen values in diagonal. This matrix is diagonal matrix, S.
5. Find eigen values and eigen vectors for ATA. Find the eigen value and pack the eigen vector
as a matrix called V.
• The main advantage of SVD is compression.
• A matrix, say an image, can be decomposed and selectively only certain components can be
retained by making all other elements zero.
❖ This reduces the contents of image while retaining the quality of the image.
• SVD is useful in data reduction too.
Design of a Learning System
• A system that is built around a learning algorithm is called a learning system.
• The design focuses on below steps
1. Choosing a training experience
2. Choosing a target function
3. Representation of a target function
4. Function approximation
Training Experience
• Let us consider designing of a chess game.
• The training experience depends on the presence of a supervisor who can label all valid moves for a board
state.
• In the absence of a supervisor, the game agent plays against itself and learns the good moves
❖ If the training samples cover all scenarios.
❖ If the training samples and testing samples have the same distribution.

Determine the Target Function

• In this step, the type of knowledge that needs to be learnt is determined.
• In direct experience, a board move is selected and is determined whether it is a good move or not against
all other moves.
❖ If it is the best move, then it is chosen as: B-> M, where, B and M are legal moves.
• In indirect experience, all legal moves are accepted and a score is generated for each.
• The move with largest score is then chosen and executed.
Determine the Target Function Representation
• The representation of knowledge may be a table, collection of rules or a neural network.
• The linear combination of these factors can be coined as:
V = w0 + w1x1 + w2 x2 + w3x3
where,
• x1, x2, and x3, represent different board features
• w0, w1, w2, and w3, represent weights

Choosing an Approximation Algorithm for the Target Function

• The focus is to choose weights and fit the given training samples effectively.
• The aim is to reduce the error given as:
Introduction to Concept Learning
• Concept learning is a learning strategy of acquiring abstract knowledge or inferring a general
concept or deriving a category from the given training samples.
• Concept learning helps to classify an object that has a set of common, relevant features.
• It helps a learner compare and contrast categories based on the similarity and association of
positive and negative instances in the training data to classify an object.
• The learner tries to simplify by observing the common features from the training samples and then
apply this simplified model to the future samples.
❖ This task is also known as learning from experience.
• Example - Humans can identify different kinds of animals based on common relevant features and
categorize all animals based on specific sets of features.
• The special features that distinguish one animal from another can be called as a concept. This way
of learning categories for object and to recognize new instances of those categories is called as
concept learning.
• Concept learning requires three things:
1. Input - Training dataset which is a set of training instances, each labeled with the name of a
concept or category to which it belongs. Use this past experience to train and build the model.
2. Output - Target concept or Target function f. It is a mapping function f(x) from input x to output
y. It is to determine the specific features or common features to identify an object. In other
words, it is to find the hypothesis to determine the target concept.
For e.g., the specific set of features to identify an elephant from all animals.
1. Test - New instances to test the learned model.
1. Representation of hypothesis
• A hypothesis 'h' approximates a target function 'f' to represent the relationship between the
independent attributes and the dependent attribute of the training instances.
• The hypothesis is the predicted approximate model that best maps the inputs to outputs.
• Each hypothesis is represented as a conjunction of attribute conditions in the antecedent part.
• Example, (Tail = Short) ^ (Color = Black)....
• The set of hypothesis in the search space is called as hypotheses.
• Hypotheses are the plural form of hypothesis.
• Generally, 'H' is used to represent the hypotheses and 'h' is used to represent a candidate
hypothesis.
• Each attribute condition is the constraint on the attribute which is represented as attribute-value pair.
• In the antecedent of an attribute condition of a hypothesis, each attribute can take value as either '?'
or 'φ' or can hold a single value.
❖ "?" denotes that the attribute can take any value [e.g., Color = ?]
❖ "φ" denotes that the attribute cannot take any value [e.g., Horns = ]
❖ Single value denotes a specific single value from acceptable values of the attribute, i.e., the
attribute 'Tail' can take a value as 'short' [e.g., Tail = Short]
2. Hypothesis Space
• Hypothesis space is the set of all possible hypotheses that approximates the target function f.
• A machine learning algorithm would determine the best possible hypothesis that would best
describe the target function or best fit the outputs.
• Every machine learning algorithm would represent the hypothesis space in a different manner
about the function that maps the input variables to output variables.
• Example, a regression algorithm represents the hypothesis space as a linear function whereas a
decision tree algorithm represents the hypothesis space as a tree.
• The subset of hypothesis space that is consistent with all-observed training instances is called as
Version Space.
• Version space represents the only hypotheses that are used for the classification.
• Example, each of the attribute given in the below table has the following possible set of values.
3. Heuristic Space Search
• Heuristic search is a search strategy that finds an optimized hypothesis/solution to a problem by
iteratively improving the hypothesis/solution based on a given heuristic function or a cost measure.
• Heuristic search methods will generate a possible hypothesis that can be a solution in the
hypothesis space or a path from the initial state.
• This hypothesis will be tested with the target function or the goal state to see if it is a real solution.
• If the tested hypothesis is a real solution, then it will be selected.
• Example: Problem solved by heuristic search is the travelling salesman problem.
• Several commonly used heuristic search methods are hill climbing methods, constraint satisfaction
problems, best-first search, simulated-annealing, A* algorithm, and genetic algorithms.
4. Generalization and Specialization
• Through Generalization and Specialization,
❖ Hypothesis space can be searched for an approximate hypothesis that matches all positive
instances but does not match any negative instance.

Searching the Hypothesis Space

There are two ways of learning the hypothesis
1. Generalization – Specific to General learning
2. Specialization – General to specific learning
Generalization – Specific to General Learning
• Learning methodology will search through the hypothesis space for an approximate hypothesis by
generalizing the most specific hypothesis.
Specialization – General to Specific Learning
• This learning methodology will search through the hypothesis space for an approximate hypothesis
by specializing the most general hypothesis.
Hypothesis Space Search by Find-S Algorithm
• Find-S algorithm is to cover most specific hypothesis in H that is consistent with the positive
instances in the training dataset.
Limitations of Find-S Algorithm
1. Find-S algorithm tries to find a hypothesis that is consistent with positive instances, ignoring all
negative instances.
❖ As long as the training dataset is consistent, the hypothesis found by this algorithm may be
consistent.
2. The algorithm finds only one unique hypothesis, wherein there may be many other hypotheses that
are consistent with the training dataset.
3. Many times, the training dataset may contain some errors, hence such inconsistent data instances
can mislead this algorithm in determining the consistent hypothesis since it ignores negative
instances.
Version Spaces
• It consists of the subset of hypothesis from the hypothesis space that is consistent with all training
instances in the training dataset.

List Then-Eliminate Algorithm

• This learning algorithm is to initialize the version space to contain all hypotheses and then eliminate
any hypothesis that is found inconsistent with any training instances.
❖ Initially, the algorithm starts with a version space to contain all hypotheses scanning each
training instance.
❖ The hypotheses that are inconsistent with the training instance are eliminated.
❖ Finally, the algorithm outputs the list of remaining hypotheses that are all consistent.
Version Spaces and the Candidate Elimination Algorithm
• Algorithm computes version space by combination of two cases
1. Specific to General learning – Generalize S to include the positive example
2. General to Specific learning – Specialize G to exclude the negative example
• Using the Candidate Elimination algorithm,
• We can compute the version space containing all (and only those) hypotheses from H that are
consistent with the given observed sequence of training instances.
• The algorithm defines two boundaries
❖ 'general boundary' which is a set of all hypotheses that are the most general
❖ 'specific boundary' which is a set of all hypotheses that are the most specific
• Thus, the algorithm limits the version space to contain only those hypotheses that are most general
and most specific. Thus, it provides a compact representation of List-then algorithm
Generating Positive Hypothesis 'S’
• If it is a positive instance, refine S to include the positive instance.
❖ We need to generalize S to include the positive instance.
• The hypothesis is the conjunction of 'S' and positive instance.
• When generalizing,
❖ For the first positive instance, add to S all minimal generalizations such that S is filled with
attribute values of the positive instance.
❖ For the subsequent positive instances scanned, check the attribute value of the positive
instance and S obtained in the previous iteration.
❑ If the attribute values of positive instance and S are different
❑ fill that field value with a "?".
❑ If the attribute values of positive instance and S are same, no change is required.
❑ If it is a negative instance, it skips.
Generating Negative Hypothesis 'G’
• If it is a negative instance, refine G to exclude the negative instance. Then, prune G to exclude all
inconsistent hypotheses in G with the positive instance.
❖ Negative hypothesis indicates general hypothesis.
• If the attribute values of positive and negative instances are different,
❖ Then fill that field with positive instance value so that the hypothesis does not classify that
negative instance as true.
• If the attribute values of positive and negative instances are same,
❖ Then no need to update 'G' and fill that attribute value with a '?'.
Generating Version Space - [Consistent Hypothesis]
• We need to take the combination of sets in 'G' and check that with 'S’.
• When the combined set fields are matched with fields in 'S', then only that is included in the version
space as consistent hypothesis.
Modelling in Machine Learning
• A machine learning model is an abstraction of the training dataset that can perform a prediction on
new data.
• Training the model means feeding instances to the machine learning algorithm.
• Training datasets are used to fit and tune the model.
• After training a machine learning algorithm with the training data,
❖ A predictive model is generated as output to which a new data is fed to make predictions.
• The process of modelling means
❖ Training a machine learning algorithm with the training dataset, tuning it to increase
performance, validating it and making predictions for a new unseen data.
• The major concern in machine learning is
1. What model to select
2. How to train the model
3. Time required to train
4. The dataset to be used
5. What performance to expect

• Learning the parameters is the main goal in machine learning algorithms.

• There are two types of parameters
1. Model parameters
2. Hyperparameters
• Model Parameters - parameters that can be learnt directly from training data.
• Example
1. The coefficients used in regression model
2. Split attributes in decision tree model
3. Weights and biases in neural networks

• Hyperparameters are higher-level parameters which cannot be learnt directly.

• Example
1. Regularization lambda 2 used in regularized regression
2. Number of decision trees to include in a random forest
Machine Learning Process
• The four basic steps in the machine learning process are:
1. Choose a machine learning algorithm to suit the training data and the problem domain
2. Input the training dataset and train the machine learning algorithm to learn from the data and
capture the patterns in the data
3. Tune the parameters of the model to improve the accuracy of learning of the algorithm
4. Evaluate the learned model once the model is built

1. Model Selection and Model Evaluation

• Model Selection is a process of selecting one good enough model among different machine learning
models for the dataset or selecting different sets of features or hyperparameters for the same
machine learning model.
• Approaches used for selecting a machine learning model are:
1. Use resample methods and split the dataset as training, testing and validation datasets and
observe the performance of the model over all the phases.
❖ This approach is suitable for smaller datasets.
2. The simplest approach is to fit a model on the training dataset and to compute measures like error
or accuracy.
3. The use of probabilistic framework and quantification of the performance of the model as a score is
the third approach.
Re-sampling Methods
• Re-sampling is a technique to select a model by reconstructing the training dataset and test
dataset by randomly choosing instances by some method from the given dataset.
• This method involves selecting different instances repeatedly from a training dataset to tune a
model to improve the accuracy of a model.
• The common re-sampling model selection methods are
1. Random train/test splits
2. Cross-Validation (K-fold, LOOCV, etc.)
3. Bootstrap
Cross-Validation
• Cross-Validation is a method by which we can tune the model with only training dataset.
• It is a model evaluation approach by which we can set aside some data of the training dataset for
validation and fit the rest of the data to train the model.
• The popular cross-validation family of methods includes
1. Holdout method
2. K-fold cross-validation
3. Stratified cross-validation
4. Leave-One-Out Cross-Validation (LOOCV)
Holdout Method
• It is the simplest method of cross-validation.
• The dataset is split into two subsets called
1. Training dataset
2. Test dataset
• The model is trained using the training dataset and then evaluated using the test dataset.
• This holdout method can be applied for a single time which is called as single holdout method or it
can be repeated for more than once which is called as repeated holdout method.
• The average performance on the test dataset is estimated to evaluate the model.
K-fold Cross-Validation
• It is a cross-validation, which will split the training dataset into k equal folds/parts creating k-1
subsets of training set and one test subset.
❖ Out of the k folds, k-1 folds are used for training and one-fold is used for testing the model.
• This has to be performed for k iterations and during each iteration a different fold is selected for
testing.
• The average performance of the model on k iterations is the final estimate of the model
performance.
Stratified K-fold Cross-Validation
• Here, while splitting the dataset into k folds, each fold should contain the same proportion of
instances with a given categorical value.

Leave-One-Out Cross-Validation (LOOCV)

• This method repeatedly splits the n data instances of the dataset into training dataset containing
n-1 data instances and leaving one data instance for evaluating the model.
• This process is repeated n times and average test error is then estimated for the model.
• Example
• If the training dataset contains 100 data instances, then 99 instances are used for training and one
instance to test or evaluate the model.
❖ This process is repeated 100 times selecting a different instance as holdout instance for testing
in each iteration. This is called stratified cross-validation.
Model Performance
• One way to compute the metrics is to form a table called contingency table
• The metrics that can be derived from this contingency table are listed below:
Classifier Performance as Distance Measures
• The classifier accuracy can be plotted as a point.
• A point in the north-west is a better classifier.
❖ Euclid distance of two points of the two classifiers can give a performance measure.
• The value ranges from 0 to 1.
Visual Classifier Performance
• Curves to indicate performance of classifiers visually are:
1. Receiver Operating Characteristic (ROC) curve
2. Precision - Recall curves

• ROC curves are visual means of checking the accuracy and comparison of classifiers.
• ROC is a plot of sensitivity (True Positive Rate) and the 1-specificity (False Positive Rate) for a
given model.
• A sample ROC curve is shown below, where results of five classifiers are given.
❖ A is the ROC of an average classifier.
❖ The ideal classifier is E where the area under curve is 1.0.
❖ The rest of the classifiers B, C, D are categorized based on area under curve as good, better
and still better based on the area under curve values.
• Instead of predicting the label of a classifier, one can predict the probabilities of the model.
• Probabilities allow some better evaluation by functions that are called scoring functions or Scoring
rules.
• The area under curve (AUC) is one such score that can be used for classifier model evaluation.
• The integrated AUC is a measure of the model across threshold values.
• AUC indicates the accuracy of the model. A model is perfect if it has area under ROC curve as one.
• The AUC score 0 of a model indicates the wrong model.
• The approximate area under precision-recall curve also indicates the power of the model across
thresholds.
• A precision-recall curve is a plot of precision and recall for different threshold values. This curve is
useful if there is an imbalance in the classes where one class has more labels and other classes
have less samples.
• ROC is used when there is no class imbalance and precision-recall curves are used when there is a
moderate-to-large class imbalance.
Scoring Methods
• Its model selection is to combine the complexity of the model and performance of the model as a
score. Then, model selection is done by selecting the model that maximizes or minimizes the
score.
• Minimum Description Length (MDL) is a method for which aim is to describe target variable and
model in terms of bits.
• MDL is the principle of using minimum number of bits to represent the data and model.
• It is a variant of Occom Razor's principle that states that the model with the simplest explanation is
the best model.
• MDL recommends the selection of the hypothesis that minimizes the sum of two descriptions of data
and model.
• Let h be a learning model.
• Let L(k) is the number of bits used to represent the model and D is the number of predictions, then
the MDL is given as:
L(h) + L(D I h)
where,
• L(D|h) is the number of bits used to represent the predictions D based on the training set.

• MDL can be expressed in terms of negative log-likelihood also as:

MDL = -log(p(θ)) - log(p(y | x, θ))
where,
y is the target variable, x is the input and θ is the model parameters.