Machine Learning [BCS602]

Module-2

Bivariate Data And Multivariate Data

Bivariate Data involves two variables. Bivariate data deals with causes of relationships. The aim is
to find relationships among data. Consider the following Table 2.3, with data of the temperature in a
shop and sales of sweaters. Bivariate analysis is to find relationships among variables. The
relationships can then be used in comparisons, finding causes, and in further explorations. To do
that, graphical display of the data is necessary. One such graph method is called scatter plot.

Scatter plot is used to visualize bivariate data. It is useful to plot two variables with or without
nominal variables, to illustrate the trends, and also to show differences. It is a plot between
explanatory and response variables. It is a 2D graph showing the relationship between two
variables.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 1

 Machine Learning [BCS602]

Line graphs are similar to scatter plots. The Line Chart for sales data is shown in Figure 2.12.

2.6.1 Bivariate Statistics

Covariance and Correlation are examples of bivariate statistics. Covariance is a measure of joint
probability of random variables, say X and Y. Generally, random variables are represented in capital
letters. It is defined as covariance(X, Y) or COV(X, Y) and is used to measure the variance between
two dimensions. The formula for finding co-variance for specific x, and y are:

Here, xi and yi are data values from X and Y. E(X) and E(Y) are the mean values of xi and yi.
N is the number of given data. Also, the COV(X, Y) is same as COV(Y, X).
Example 2.6: Find the covariance of data X = {1, 2, 3, 4, 5} and Y = {1, 4, 9, 16, 25}.

The covariance between X and Y is 12. It can be normalized to a value between -1 and +1. This is
done by dividing it by the correlation of variables. This is called Pearson correlation coefficient.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 2

 Machine Learning [BCS602]

Sometimes, N - 1 is also can be used instead of N. In that case, the covariance is 60/4 = 15.
Correlation
The Pearson correlation coefficient is the most common test for determining any association
between two phenomena. It measures the strength and direction of a linear relationship between the
x and y variables.
1. If the value is positive, it indicates that the dimensions increase together.
2. If the value is negative, it indicates that while one-dimension increases, the other dimension
decreases.
3. If the value is zero, then it indicates that both the dimensions are independent of each other.
If the dimensions are correlated, then it is better to remove one dimension as it is a redundant
dimension.
If the given attributes are X = (x1, x2, ... , xN) and Y = (y1, y2, ... , yN), then the Pearson
correlation coefficient, that is denoted as r, is given as:

where, sX, sY are the standard deviati ons of X and Y.

2.7 MULTIVARIATE STATISTICS

In machine learning, almost all datasets are multivariable. Multivariate data is the analysis of more
than two observable variables, and often, thousands of multiple measurements need to be conducted
for one or more subjects.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 3

 Machine Learning [BCS602]

Multivariate data has three or more variables. The aim of the multivariate analysis is much more.
They are regression analysis, factor analysis and multivariate analysis of variance that are explained
in the subsequent chapters of this book.
Heatmap
Heatmap is a graphical representation of 2D matrix. It takes a matrix as input and colours it. The
darker colours indicate very large values and lighter colours indicate smaller values. The advantage
of this method is that humans perceive colours well. So, by colour shaping, larger values can be
perceived well. For example, in vehicle traffic data, heavy traffic regions can be differentiated from
low traffic regions through heatmap. In Figure 2.13, patient data highlighting weight and health
status is plotted. Here, X-axis is weights and Y- axis is patient counts. The dark colour regions
highlight patients’ weights vs patient counts in health status.

Pairplot
Pairplot or scatter matrix is a data visual technique for multivariate data. A scatter matrix consists of
several pair-wise scatter plots of variables of the multivariate data.
A random matrix of three columns is chosen and the relationships of the columns is plotted as a
pairplot (or scatter matrix) as shown below in Figure 2.14.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 4

 Machine Learning [BCS602]

Essential Mathematics for Multivariate Data

Machine learning involves many mathematical concepts from the domain of Linear algebra,
Statistics, Probability and Information theory. The subsequent sections discuss important aspects of
linear algebra and probability.
2.8.1 Linear Systems and Gaussian Elimination for Multivariate Data
A linear system of equations is a group of equations with unknown variables.
Let Ax = y, then the solution x is given as:

This is true if y is not zero and A is not zero. The logic can be extended for N-set of equations with
‘n’ unknown variables.
It means if A= and y=(y1 y2...yn), then the unknown variable x can be computed as:

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 5

 Machine Learning [BCS602]

unique solution, then the system is called consistent independent. If there are various solutions, then
the system is called consistent dependant. If there are no solutions and if the equations are
contradictory, then the system is called inconsistent.
For solving large number of system of equations, Gaussian elimination can be used. The procedure
for applying Gaussian elimination is given as follows:
1. Write the given matrix.
2. Append vector y to the matrix A. This matrix is called augmentation matrix.
3. Keep the element a11 as pivot and eliminate all a11 in second row using the matrix operation,

The same logic can be used to remove a11 in all other equations.
4 .Repeat the same logic and reduce it to reduced echelon form. Then, the unknown variable as:

5 .Then, the remaining unknown variables can be found by back - substitution as:

This part is called backward substitution.

To facilitate the application of Gaussian elimination method, the following row operations are
applied:
1. Swapping the rows
2. Multiplying or dividing a row by a constant
3. Replacing a row by adding or subtracting a multiple of another row to it These concepts
are illustrated in Example 2.8.

2.8.2 Matrix Decomposition

It is often necessary to reduce a matrix to its constituent parts so that complex matrix operations can
be performed. Then, the matrix A can be decomposed as:

matrix of eigen vectors, Λ is the diagonal matrix and QT is the transpose of matrix Q.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 6

 Machine Learning [BCS602]

LU Decomposition
One of the simplest matrix decomposition is LU decomposition where the matrix A can be
decomposed matrices: A = LU Here, L is the lower triangular matrix and U is the upper triangular
matrix. The decomposition can be done using Gaussian elimination method as discussed in the
previous section. First, an identity matrix is augmented to the given matrix. Then, row operations
and Gaussian elimination is applied to reduce the given matrix to get matrices L and U.
Example 2.9 illustrates the application of Gaussian elimination to get LU.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 7

 Machine Learning [BCS602]

Now, it can be observed that the first matrix is L as it is the lower triangular matrix whose values are
the determiners used in the reduction of equations above such as 3, 3 and 2/3. The second matrix is U,
the upper triangular matrix whose values are the values of the reduced matrix because of Gaussian
elimination.

2.8.3 Machine Learning and Importance of Probability and Statistics

Machine learning is linked with statistics and probability. Like linear algebra, statistics is the heart
of machine learning. The importance of statistics needs to be stressed as without statistics;
Probability Distributions
A probability distribution of a variable, say X, summarizes the probability associated with X’s
events. Distribution is a parameterized mathematical function. In other words, distribution is a
function that describes the relationship between the observations in a sample space. Consider a set
of data. The data is said to follow a distribution if it obeys a mathematical function that
characterizes that distribution. The function can be used to calculate the probability of individual
observations.
Probability distributions are of two types:
1. Discrete probability distribution
[Link] probability distribution
Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 8
 Machine Learning [BCS602]

The relationships between the events for a continuous random variable and their probabilities
Continuous Probability Distributions
Normal, Rectangular, and Exponential distributions fall under this category.
1. Normal Distribution – Normal distribution is a continuous probability distribution. This is
also known as gaussian distribution or bell-shaped curve distribution. It is the most common
distribution function. The shape of this distribution is a typical bell-shaped curve. In normal
distribution, data tends to be around a central value with no bias on left or right. The heights of the
students, blood pressure of a population, and marks scored in a class can be approximated using
normal distribution. PDF of the normal distribution is given as:

Here, m is mean and s is the standard deviation. Normal distribution is characterized by two
parameters – mean and variance.
One important concept associated with normal distribution is z-score. It can be computed as:

This is useful to normalize the data.

2. Rectangular Distribution – This is also known as uniform distribution. It has equal probabilities
for all values in the range a, b. The uniform distribution is given as follows:

3. Exponential Distribution – This is a continuous uniform distribution. This probability

distribution is used to describe the time between events in a Poisson process. Exponential
distribution is another special case of Gamma distribution with a fixed parameter of 1. This
distribution is helpful in modelling of time until an event occurs.
The PDF is given as follows:

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 9

 Machine Learning [BCS602]

Discrete Distribution Binomial, Poisson, and Bernoulli distributions fall under this category
1. Binomial Distribution – Binomial distribution is another distribution that is often encountered in
machine learning. It has only two outcomes: success or failure. This is also called Bernoulli trial.
The objective of this distribution is to find probability of getting success k out of n trials. The way
to get success out of k out of n number of trials is given as:

The binomial distribution function is given as follows, where p is the probability of success and
probability of failure is (1 - p). The probability of success in a certain number
of trials is given as:

Combining both, one gets PDF of binomial distribution as:

Here, p is the probability of each choice, k is the number of choices, and n is the total
number of choices. The mean of binomial distribution is given below:

And the variance is given as:

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 10

 Machine Learning [BCS602]

standard deviation is given as:

2. Poisson Distribution – It is another important distribution that is quite useful. Given an interval
of time, this distribution is used to model the probability of a given number of events k. The mean
rule l is inclusive of previous events. Some of the examples of Poisson distribution are number of
emails received, number of customers visiting a shop and the number of phone calls received by the
office.
The PDF of Poisson distribution is given as follows:

The mean is p and variance is p(1 - p) = q

Density Estimation
Let there be a set of observed values x1, x2, ... , xn from a larger set of data whose distribution is
not known. Density estimation is the problem of estimating the density function from an observed
data. There are two types of density estimation methods, namely parametric density estimation and
non-parametricdensity estimation.
Parametric Density Estimation
It assumes that the data is from a known probabilistic distribution and can be estimated as

Maximum likelihood function is a parametric estimation method.

Maximum Likelihood Estimation
Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 11
 Machine Learning [BCS602]

For a sample of observations, one can estimate the probability distribution. This is called density
estimation. Maximum Likelihood Estimation (MLE) is a probabilistic framework that can be used

for density estimation. This involves formulating a function called likelihood function which is the
conditional probability of observing the observed samples and distribution function with its
parameters. For example, if the observations are
X = {x1, x2, ... , xn}, then density estimation is the problem of choosing a PDF with suitable
parameters to describe the data. MLE treats this problem as a search or optimization problem
where the probability should be maximized for the joint probabilities of X and its parameter, theta.

If one assumes that the regression problem can be framed as predicting output y given input x, then
for p(y/x), the MLE framework can be applied as:

Here, h is the linear regression model. If Gaussian distribution is assumed as it is an obvious fact
that most of the data follow Gaussian distribution, then MLE can be stated as:

Here, b is the regression coefficient and xi is the given sample. One can maximize this function or
minimize the negative log likelihood function to provide a solution for linear regression problem.
The Eq. (2.37) yields the same answer of the least-square approach.
Gaussian Mixture Model and Expectation-Maximization (EM) Algorithm
In machine learning, clustering is one of the important tasks. MLE framework is quite useful for
designing model based methods for clustering data. A model is a statistical method and data is
assumed to be generated by a distribution model with its parameter, theta. There may be many

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 12

 Machine Learning [BCS602]

distributions involved and that is why it is called as mixture model.

Generally, there can be many unspecified distributions with different set of parameters. The EM
algorithm has two stages:

1. Expectation (E) Stage – In this stage, the expected PDF and its parameters are estimated for each
latent variable.
2. Maximization (M) stage – In this, the parameters are optimized using the MLE function. This
process is iterative, and the iteration is continued till all the latent variables are fitted by probability
distributions effectively along with the parameters.
Non-parametric Density Estimation
A non-parametric estimation can be generative or discriminative. Parzen window is a generative
estimation method that finds as conditional density. Discriminative methods directly compute
as posteriori probability. Parzen window and k-Nearest Neighbour (KNN) rule are examples of non-
parametric density estimation.

This window can be replaced by any other function too. If Gaussian function is used, then it is
called Gaussian density function.
KNN Estimation The KNN estimation is another non-parametric density estimation method. Here,
the initial parameter k is determined and based on that k-neighbours are determined. The probability
density function estimate is the average of the values that are returned by the neighbours.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 13

 Machine Learning [BCS602]

2.10 FEATURE ENGINEERING AND DIMENSIONALITY REDUCTION TECHNIQUES

Features are attributes. Feature engineering is about determining the subset of features that form an
important part of the input that improves the performance of the model, be it classification or any
other model in machine learning.
Feature engineering deals with two problems – Feature Transformation and Feature Selection.
Feature transformation is extraction of features and creating new features that may be helpful in
increasing performance. For example, the height and weight may give a new attribute called Body
Mass Index (BMI).
Feature subset selection is another important aspect of feature engineering that focuses on selection
of features to reduce the time but not at the cost of reliability.
The features can be removed based on two aspects:
1. Feature relevancy – Some features contribute more for classification than other features. For
example, a mole on the face can help in face detection than common features like nose. In simple
words, the features should be relevant.
2. Feature redundancy – Some features are redundant. For example, when a database table has a
field called Date of birth, then age field is not relevant as age can be computed easily from date of
birth.
So, the procedure is:
1. Generate all possible subsets
2. Evaluate the subsets and model performance
3. Evaluate the results for optimal feature selection
Filter-based selection uses statistical measures for assessing features. In this approach, no learning
algorithm is used. Correlation and information gain measures like mutual information and entropy
are all examples of this approach.
Wrapper-based methods use classifiers to identify the best features. These are selected and
evaluated by the learning algorithms. This procedure is computationally intensive but has superior
performance.
2.10.1 Stepwise Forward Selection
This procedure starts with an empty set of attributes. Every time, an attribute is tested for statistical
significance for best quality and is added to the reduced set. This process is continued till a good
reduced set of attributes is obtained.
2.10.2 Stepwise Backward Elimination
This procedure starts with a complete set of attributes. At every stage, the procedure removes the

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 14

 Machine Learning [BCS602]

worst attribute from the set, leading to the reduced set.

2.10.3 Principal Component Analysis

The Idea of the principal component analysis (PCA) or KL transform is to transform a given set of
measurements to a new set of features so that the features exhibit high information packing
properties. This leads to a reduced and compact set of features. Consider a group of random vectors
of the form:

The mean vector of the set of random vectors is defined as:

The operator E refers to the expected value of the population. This is calculated theoretically using
the probability density functions (PDF) of the elements xi and the joint probability density
functions between the elements xi and xj. From this, the covariance matrix can be calculated as:

The mapping of the v ectors x to y using the transformation can now be described as:

This transfo rm is also called as Karhunen - Loeve or Hoteling transform. The original vector x
can now be reconstructed as follows:

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 15

 Machine Learning [BCS602]

If K largest eigen values are used, the recovered information would be:

The PCA algorithm is as follows:

1. The target dataset x is obtained
2. The mean is subtracted from the dataset. Let the mean be m. Thus, the adjusted dataset is X – m.
The objective of this process is to transform the dataset with zero mean.
3. The covariance of dataset x is obtained. Let it be C.
4. Eigen values and eigen vectors of the covariance matrix are calculated.
5. The eigen vector of the highest eigen value is the principal component of the dataset. The eigen
values are arranged in a descending order. The feature vector is formed with these eigen vectors in
its columns.
Feature vector = {eigen vector1, eigen vector2, ... , eigen vectorn}
6. Obtain the transpose of feature vector. Let it be A.
7. PCA transform is y = A × (x – m), where x is the input dataset, m is the mean, and A is the
transpose of the feature vector.
The original data can be retrieved using the formula given below:

The new data is a dimensionaly reduced matrix that represents the original data.
Figure 2.15. The scree plot indicates that only 6 out of 246 attributes are important.

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 16

 Machine Learning [BCS602]

From Figure 2.15, one can infer the relevance of the attributes. The scree plot indicates that the first
attribute is more important than all other attributes.
2.10.4 Linear Discriminant Analysis
Linear Discriminant Analysis (LDA) is also a feature reduction technique like PCA. The focus of
LDA is to project higher dimension data to a line (lower dimension data). LDA is also used to
classify the data. Let there be two classes, c1 and c2. Let m1 and m2 be the mean of the patterns of
two classes. The mean of the class c1 and c2 can be computed as:

The aim of LDA is to optimize the function:

2.10.5 Singular Value Decomposition

2.10.5 Singular Value Decomposition

Singular Value Decomposition (SVD) is another useful decomposition technique. Let A be the
matrix, then the matrix A can be decomposed as:

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 17

 Machine Learning [BCS602]

Here, A is the given matrix of dimension m × n, U is the orthogonal matrix whose dimension is m ×
n, S is the diagonal matrix of dimension n × n, and V is the orthogonal matrix. The procedure for
finding decomposition matrix is given as follows:
1. For a given matrix, find AA^T
2. Find eigen values of AA^T
3. Sort the eigen values in a descending order. Pack the eigen vectors as a matrix U.
4. Arrange the square root of the eigen values in diagonal. This matrix is diagonal matrix, S.
5. Find eigen values and eigen vectors for A^TA. Find the eigen value and pack the eigen vector as a
matrix called V.
USV^ T. Here, U and V are orthogonal matrices. The columns of U and V are left and right singular
values, respectively. SVD is useful in compression, as one can decide to retain only a certain
component instead of the original matrix A as:

Based on the choice of retention, compression can be controlled.

BASICS OF LEARNING THEORY

Learning is a process by which one can acquire knowledge and construct new ideas or concepts
based on the experiences.
There are two kinds of problems – well-posed and ill-posed. Computers can solve only well-
posed problems, as these have well-defined specifications and have the following
components inherent to it.
1. Class of learning tasks (T) 2. A measure of performance (P) 3. A source of experience (E)
Classical and Adaptive ML systems.
Classic machines examine data inputs according to a predetermined set of rules, finding
patterns and relationships that can be used to generate predictions or choices. Support vector
machines, decision trees, and logistic regression are some of the most used classical machine-
learning techniques.
A class of machine learning techniques called adaptive machines, commonly referred to as
adaptive or deep learning, is created to automatically learn from data inputs without being
explicitly programmed. By learning hierarchical representations of the input, these
algorithms are able to handle more complex and unstructured data, such as photos, videos,
and natural language.
Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 18
 Machine Learning [BCS602]

Adaptive ML is the next generation of traditional ML – the new, the improved, the better. Even
though traditional ML witnessed significant progress.

3.1 DESIGN OF A LEARNING SYSTEM

INTRODUCTION TO CONCEPT LEARNING

3.4.1 Representation of a Hypothesis

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 19

 Machine Learning [BCS602]

3.4.2 Hypothesis Space

Hypothesis space is the set of all possible hypotheses that approximates the target function f.
The subset of hypothesis space that is consistent with all-observed training instances is called
as Version Space.
3.4.3 Heuristic Space Search
Heuristic search is a search strategy that finds an optimized hypothesis/solution to a problem by
iteratively improving the hypothesis/solution based on a given heuristic function or a cost measure.
3.4.4 Generalization and Specialization
Searching the Hypothesis Space
There are two ways of learning the hypothesis, consistent with all training instances from the
large hypothesis space.
1. Specialization – General to Specific learning
2. Generalization – Specific to General learning
Generalization – Specific to General Learning This learning methodology will search through the
hypothesis space for an approximate hypothesis by generalizing the most specific hypothesis.
Specialization – General to Specific Learning This learning methodology will search through the
hypothesis space for an approximate hypothesis by specializing the most general hypothesis.

3.4.5 Hypothesis Space Search by Find-S Algorithm

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 20

 Machine Learning [BCS602]

Limitations of Find-S Algorithm

3.4.6 Version Spaces

List-Then-Eliminate Algorithm

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 21

 Machine Learning [BCS602]

Candidate Elimination Algorithm

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 22

 Machine Learning [BCS602]

The diagrammatic representation of deriving the version space is shown below:

Deriving the Version Space

Dr. Manjunath R, HOD, Dept. of CSE, RRIT, Bengaluru 23