Module 2 BCS602

2.6 BIVARIATE DATA AND MULTIVARIATE DATA

Bivariate Data involves two variables. Bivariate data deals with causes of relationships. The aim is
to find relationships among data. Consider the following Table 2.3, with data of the temperature in
a shop and sales of sweaters.

Here, the aim of bivariate analysis is to find relationships among variables. The relationships can
then be used in comparisons, finding causes, and in further explorations. To do that, graphical
display of the data is necessary. One such graph method is called scatter plot.

Scatter plot is used to visualize bivariate data. It is useful to plot two variables with or without
nominal variables, to illustrate the trends, and also to show differences. It is a plot between
explanatory and response variables. It is a 2D graph showing the relationship between two
variables.

Line graphs are similar to scatter plots. The Line Chart for sales data is shown in Figure 2.12.

1
Module 2 BCS602

2.6.1 Bivariate Statistics

Covariance and Correlation are examples of bivariate statistics. Covariance is a measure of joint
probability of random variables, say X and Y. Generally, random variables are represented in
capital letters. It is defined as covariance(X, Y) or COV(X, Y) and is used to measure the variance
between two dimensions. The formula for finding co-variance for specific x, and y are:

Here, xi and yi are data values from X and Y. E(X) and E(Y) are the mean values of xi and yi.
N is the number of given data. Also, the COV(X, Y) is same as COV(Y, X).
Example 2.6: Find the covariance of data X = {1, 2, 3, 4, 5} and Y = {1, 4, 9, 16, 25}.

The covariance between X and Y is 12. It can be normalized to a value between -1 and +1. This
is done by dividing it by the correlation of variables. This is called Pearson correlation coefficient.
Sometimes, N - 1 is also can be used instead of N. In that case, the covariance is 60/4 = 15.

Correlation
The Pearson correlation coefficient is the most common test for determining any association
between two phenomena. It measures the strength and direction of a linear relationship between
the x and y variables.
1. If the value is positive, it indicates that the dimensions increase together.
2. If the value is negative, it indicates that while one-dimension increases, the other dimension
decreases.
3. If the value is zero, then it indicates that both the dimensions are independent of each
other.
If the dimensions are correlated, then it is better to remove one dimension as it is a redundant
dimension.
If the given attributes are X = (x1, x2, … , xN) and Y = (y1, y2, … , yN), then the Pearson correlation
coefficient, that is denoted as r, is given as:

where, sX, sY are the standard deviations of X and Y.

2.7 MULTIVARIATE STATISTICS

In machine learning, almost all datasets are multivariable. Multivariate data is the analysis of
more than two observable variables, and often, thousands of multiple measurements need to be
conducted for one or more subjects.

Multivariate data has three or more variables. The aim of the multivariate analysis is much
more. They are regression analysis, factor analysis and multivariate analysis of variance that are
explained in the subsequent chapters of this book.
2
Module 2 BCS602

Heatmap
Heatmap is a graphical representation of 2D matrix. It takes a matrix as input and colours it. The
darker colours indicate very large values and lighter colours indicate smaller values. The
advantage of this method is that humans perceive colours well. So, by colour shaping, larger values
can be perceived well. For example, in vehicle traffic data, heavy traffic regions can be
differentiated from low traffic regions through heatmap.

In Figure 2.13, patient data highlighting weight and health status is plotted. Here, X-axis
is weights and Y-axis is patient counts. The dark colour regions highlight patients’ weights vs
patient counts in health status.

Pairplot
Pairplot or scatter matrix is a data visual technique for multivariate data. A scatter matrix consists
of several pair-wise scatter plots of variables of the multivariate data.
A random matrix of three columns is chosen and the relationships of the columns is plotted
as a pairplot (or scatter matrix) as shown below in Figure 2.14.

3
Module 2 BCS602

2.8 ESSENTIAL MATHEMATICS FOR MULTIVARIATE DATA

Machine learning involves many mathematical concepts from the domain of Linear algebra,
Statistics, Probability and Information theory. The subsequent sections discuss important aspects
of linear algebra and probability.

2.8.1 Linear Systems and Gaussian Elimination for Multivariate Data

A linear system of equations is a group of equations with unknown variables.
Let Ax = y, then the solution x is given as:

This is true if y is not zero and A is not zero. The logic can be extended for N-set of equations
with ‘n’ unknown variables.
It means if A= and y=(y1 y2…yn), then the unknown variable x can be computed as:

If there is a unique solution, then the system is called consistent independent. If there are various
solutions, then the system is called consistent dependant. If there are no solutions and if the
equations are contradictory, then the system is called inconsistent.
For solving large number of system of equations, Gaussian elimination can be used. The
procedure for applying Gaussian elimination is given as follows:
1. Write the given matrix.
2. Append vector y to the matrix A. This matrix is called augmentation matrix.
3. Keep the element a11 as pivot and eliminate all a11 in second row using the matrix operation,

The same logic

can be used to remove a11 in all other equations.
4. Repeat the same logic and reduce it to reduced echelon form. Then, the unknown variable as:

5. Then, the remaining unknown variables can be found by back-substitution as:

This part is called backward substitution.

To facilitate the application of Gaussian elimination method, the following row operations are
applied:
1. Swapping the rows
2. Multiplying or dividing a row by a constant
3. Replacing a row by adding or subtracting a multiple of another row to it
These concepts are illustrated in Example 2.8.

4
Module 2 BCS602

2.8.2 Matrix Decomposition

It is often necessary to reduce a matrix to its constituent parts so that complex matrix operations
can be performed.
Then, the matrix A can be decomposed as:

where, Q is the matrix of eigen vectors, Λ is the diagonal matrix and QT is the transpose of matrix
Q.

LU Decomposition
One of the simplest matrix decomposition is LU decomposition where the matrix A can be
decomposed matrices: A = LU
Here, L is the lower triangular matrix and U is the upper triangular matrix. The decomposition
can be done using Gaussian elimination method as discussed in the previous section. First, an
identity matrix is augmented to the given matrix. Then, row operations and Gaussian elimination
is applied to reduce the given matrix to get matrices L and U.
Example 2.9 illustrates the application of Gaussian elimination to get LU.

5
Module 2 BCS602

Now, it can be observed that the first matrix is L as it is the lower triangular matrix whose
values are the determiners used in the reduction of equations above such as 3, 3 and 2/3.
The second matrix is U, the upper triangular matrix whose values are the values of the reduced
matrix because of Gaussian elimination.

2.8.3 Machine Learning and Importance of Probability and Statistics

Machine learning is linked with statistics and probability. Like linear algebra, statistics is the heart
of machine learning. The importance of statistics needs to be stressed as without statistics;

Probability Distributions
A probability distribution of a variable, say X, summarizes the probability associated with X’s
events. Distribution is a parameterized mathematical function. In other words, distribution is a
function that describes the relationship between the observations in a sample space.
Consider a set of data. The data is said to follow a distribution if it obeys a mathematical
function that characterizes that distribution. The function can be used to calculate the probability
of individual observations.
Probability distributions are of two types:
1. Discrete probability distribution
[Link] probability distribution
The relationships between the events for a continuous random variable and their probabilities

Continuous Probability Distributions Normal, Rectangular, and Exponential distributions

fall under this category.

1. Normal Distribution – Normal distribution is a continuous probability distribution.

This is also known as gaussian distribution or bell-shaped curve distribution. It is the
most common distribution function. The shape of this distribution is a typical bell-shaped

6
Module 2 BCS602

curve. In normal distribution, data tends to be around a central value with no bias on left
or right. The heights of the students, blood pressure of a population, and marks scored in
a class can be approximated using normal distribution.
PDF of the normal distribution is given as:

Here, m is mean and s is the standard deviation. Normal distribution is characterized

by two parameters – mean and variance.
One important concept associated with normal distribution is z-score. It can be
computed as:

This is useful to normalize the data.

2. Rectangular Distribution – This is also known as uniform distribution. It has equal

probabilities for all values in the range a, b. The uniform distribution is given as follows:

3. Exponential Distribution – This is a continuous uniform distribution. This probability

distribution is used to describe the time between events in a Poisson process. Exponential
distribution is another special case of Gamma distribution with a fixed parameter of 1.
This distribution is helpful in modelling of time until an event occurs.
The PDF is given as follows:

Discrete Distribution Binomial, Poisson, and Bernoulli distributions fall under this category.
1. Binomial Distribution – Binomial distribution is another distribution that is often encountered
in machine learning. It has only two outcomes: success or failure. This is also called
Bernoulli trial.
The objective of this distribution is to find probability of getting success k out of n trials.
The way to get success out of k out of n number of trials is given as:

The binomial distribution function is given as follows, where p is the probability of

success and probability of failure is (1 - p). The probability of success in a certain number
of trials is given as:

Combining both, one gets PDF of binomial distribution as:

Here, p is the probability of each choice, k is the number of choices, and n is the total
number of choices. The mean of binomial distribution is given below:

And the variance is given as:

7
Module 2 BCS602

Hence, the standard deviation is given as:

2. Poisson Distribution – It is another important distribution that is quite useful. Given an

interval of time, this distribution is used to model the probability of a given number of
events k. The mean rule l is inclusive of previous events. Some of the examples of Poisson
distribution are number of emails received, number of customers visiting a shop and the
number of phone calls received by the office.
The PDF of Poisson distribution is given as follows:

3. Bernoulli Distribution – This distribution models an experiment whose outcome is binary.

The outcome is positive with p and negative with 1 - p. The PMF of this distribution is
given as:

The mean is p and variance is p(1 - p) = q

Density Estimation
Let there be a set of observed values x1, x2, … , xn from a larger set of data whose distribution is
not known. Density estimation is the problem of estimating the density function from an observed
data.
There are two types of density estimation methods, namely parametric density estimation and
non-parametric density estimation.

Parametric Density Estimation It assumes that the data is from a known probabilistic distri-
bution and can be estimated as Maximum likelihood function
is a parametric estimation method.

Maximum Likelihood Estimation For a sample of observations, one can estimate the probability
distribution. This is called density estimation. Maximum Likelihood Estimation (MLE) is a
probabilistic framework that can be used for density estimation. This involves formulating
a function called likelihood function which is the conditional probability of observing the
observed samples and distribution function with its parameters. For example, if the observations
are X = {x1, x2, … , xn}, then density estimation is the problem of choosing a PDF with suitable
parameters to describe the data. MLE treats this problem as a search or optimization problem
where the probability should be maximized for the joint probabilities of X and its parameter, theta.

8
Module 2 BCS602

If one assumes that the regression problem can be framed as predicting output y given input x,
then for p(y/x), the MLE framework can be applied as:

Here, h is the linear regression model. If Gaussian distribution is assumed as it is an obvious

fact that most of the data follow Gaussian distribution, then MLE can be stated as:

Here, b is the regression coefficient and xi is the given sample. One can maximize this function
or minimize the negative log likelihood function to provide a solution for linear regression
problem. The Eq. (2.37) yields the same answer of the least-square approach.

Gaussian Mixture Model and Expectation-Maximization (EM) Algorithm In machine learning,

clustering is one of the important tasks. It is discussed in Chapter 13. MLE framework is quite
useful for designing model-based methods for clustering data. A model is a statistical method and
data is assumed to be generated by a distribution model with its parameter, theta. There may be
many distributions involved and that is why it is called as mixture model.

Generally, there can be many unspecified distributions with different set of parameters. The
EM algorithm has two stages:
1. Expectation (E) Stage – In this stage, the expected PDF and its parameters are estimated
for each latent variable.
2. Maximization (M) stage – In this, the parameters are optimized using the MLE function.
This process is iterative, and the iteration is continued till all the latent variables are fitted
by probability distributions effectively along with the parameters.

Non-parametric Density Estimation A non-parametric estimation can be generative

or discriminative. Parzen window is a generative estimation method that finds as
conditional density. Discriminative methods directly compute as posteriori probability.
Parzen window and k-Nearest Neighbour (KNN) rule are examples of non-parametric density

9
Module 2 BCS602

estimation.
This window can be replaced by any other function too. If Gaussian function is used, then it
is called Gaussian density function.
KNN Estimation The KNN estimation is another non-parametric density estimation method.
Here, the initial parameter k is determined and based on that k-neighbours are determined.
The probability density function estimate is the average of the values that are returned by
the neighbours.

FEATURE ENGINEERING AND DIMENSIONALITY REDUCTION TECHNIQUES

Features are attributes. Feature engineering is about determining the subset of features that form
an important part of the input that improves the performance of the model, be it classification or
any other model in machine learning.

Feature engineering deals with two problems – Feature Transformation and Feature Selection.
Feature transformation is extraction of features and creating new features that may be helpful in
increasing performance. For example, the height and weight may give a new attribute called Body
Mass Index (BMI).

Feature subset selection is another important aspect of feature engineering that focuses on
selection of features to reduce the time but not at the cost of reliability.
The features can be removed based on two aspects:
1. Feature relevancy – Some features contribute more for classification than other features.
For example, a mole on the face can help in face detection than common features like
nose. In simple words, the features should be relevant.
2. Feature redundancy – Some features are redundant. For example, when a database table
has a field called Date of birth, then age field is not relevant as age can be computed
easily from date of birth.
So, the procedure is:
1. Generate all possible subsets
2. Evaluate the subsets and model performance
3. Evaluate the results for optimal feature selection

Filter-based selection uses statistical measures for assessing features. In this approach,
no learning algorithm is used. Correlation and information gain measures like mutual information
and entropy are all examples of this approach.
10
Module 2 BCS602

Wrapper-based methods use classifiers to identify the best features. These are selected
and evaluated by the learning algorithms. This procedure is computationally intensive but has
superior performance.

2.8.4 Stepwise Forward Selection

This procedure starts with an empty set of attributes. Every time, an attribute is tested for
statistical significance for best quality and is added to the reduced set. This process is continued
till a good reduced set of attributes is obtained.

2.8.5 Stepwise Backward Elimination

This procedure starts with a complete set of attributes. At every stage, the procedure removes the
worst attribute from the set, leading to the reduced set.

2.8.6 Principal Component Analysis

The idea of the principal component analysis (PCA) or KL transform is to transform a given
set of measurements to a new set of features so that the features exhibit high information
packing properties. This leads to a reduced and compact set of features.
Consider a group of random vectors of the form:

The mean vector of the set of random vectors is defined as:

The operator E refers to the expected value of the population. This is calculated theoretically
using the probability density functions (PDF) of the elements xi and the joint probability density
functions between the elements xi and xj. From this, the covariance matrix can be calculated as:

The mapping of the vectors x to y using the transformation can now be described as:

This transform is also called as Karhunen-Loeve or Hoteling transform. The original vector x
can now be reconstructed as follows:

If K largest eigen values are used, the recovered information would be:

The PCA algorithm is as follows:

1. The target dataset x is obtained
2. The mean is subtracted from the dataset. Let the mean be m. Thus, the adjusted dataset is
X – m. The objective of this process is to transform the dataset with zero mean.
3. The covariance of dataset x is obtained. Let it be C.
4. Eigen values and eigen vectors of the covariance matrix are calculated.
5. The eigen vector of the highest eigen value is the principal component of the dataset.
The eigen values are arranged in a descending order. The feature vector is formed with these
11
Module 2 BCS602

eigen vectors in its columns.

Feature vector = {eigen vector1, eigen vector2, … , eigen vectorn}
6. Obtain the transpose of feature vector. Let it be A.
7. PCA transform is y = A × (x – m), where x is the input dataset, m is the mean, and A is the
transpose of the feature vector.
The original data can be retrieved using the formula given below:

The new data is a dimensionaly reduced matrix that represents the original data.

12
Module 2 BCS602

Figure 2.15. The scree plot indicates that only 6 out of 246 attributes are important.

From Figure 2.15, one can infer the relevance of the attributes. The scree plot indicates that
the first attribute is more important than all other attributes.

2.8.7 Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is also a feature reduction technique like PCA. The focus of
LDA is to project higher dimension data to a line (lower dimension data). LDA is also used to
classify the data. Let there be two classes, c1 and c2. Let m1 and m2 be the mean of the patterns of
two
classes. The mean of the class c1 and c2 can be computed as:

The aim of LDA is to optimize the function:

2.8.8 Singular Value Decomposition

Singular Value Decomposition (SVD) is another useful decomposition technique. Let A be the
matrix, then the matrix A can be decomposed as:

Here, A is the given matrix of dimension m × n, U is the orthogonal matrix whose dimension is
m × n, S is the diagonal matrix of dimension n × n, and V is the orthogonal matrix. The procedure
for finding decomposition matrix is given as follows:
1. For a given matrix, find AA^T

13
2. Find eigen values of AA^T
3. Sort the eigen values in a descending order. Pack the eigen vectors as a matrix U.
4. Arrange the square root of the eigen values in diagonal. This matrix is diagonal matrix, S.
5. Find eigen values and eigen vectors for A^TA. Find the eigen value and pack the eigen vector
as a matrix called V.
Thus, A = USV^T. Here, U and V are orthogonal matrices. The columns of U and V are left and
right singular values, respectively. SVD is useful in compression, as one can decide to retain only a
certain component instead of the original matrix A as:

Based on the choice of retention, the compression can be controlled.