MODULE 2

BIVARIATE AND MULTIVARIATE DATA

 Bivariate Data involves two variables

 The aim is to find relationships among data.
 Scatter plot and line graph is used to visualize bivariate data.
 It is useful to plot two variables with or without nominal variables, to illustrate
the trends, and also to show differences

BIVARIATE STATISTICS

 Covariance and Correlation are examples of bivariate statistics.

 Covariance is a measure of joint probability of random variables, say X and Y.
 Covariance is a measure of joint probability of random variables, say X and Y.
 Defined as COV(X, Y).
 The formula for finding co-variance for specific x, and y are:

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 1

 
 Here, xi and yi are data values from X and Y. E(X) and E(Y) are the mean
values of xi and yi, N is the number of given data Also, the COV(X, Y) is same
as COV(Y, X).

CORRELATION

 The Pearson correlation coefficient is the most common test for determining
any association between two phenomena.
 It measures the strength and direction of a linear relationship between the x and
y variables.
 The correlation indicates the relationship between dimensions using its sign.
1. If the value is positive, it indicates that the dimensions increase together.
2. If the value is negative, it indicates that while one-dimension increases,
the other dimension decreases.
3. If the value is zero, then it indicates that both the dimensions are
independent of each other.
 If the dimensions are correlated, then it is better to remove one dimension as it
is a redundant dimension.
 If the given attributes are X = (x1, x2, - xN) and Y = (y1, y2, - - - yN), then the
Pearson correlation coefficient, that is denoted as r, is given as:

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 2

MULTIVARIATE STATISTICS

 Multivariate data is the analysis of more than two observable variables

 The multivariate data is like bivariate data but may have more than two
dependant variables.
 Some of the multivariate analysis is regression analysis, principal component
analysis, and path analysis.

 Multivariate data has three or more variables.

HEATMAP

 Heatmap is a graphical representation of 2D matrix.

 It takes a matrix as input and colours it. The darker colours indicate very large
values and lighter colours indicate smaller values.
 The advantage of this method is that humans perceive colours well. So, by
colour shaping, larger values can be perceived well.
 For example, in vehicle traffic data, heavy traffic regions can be differentiated
from low traffic regions through heatmap.
 In Figure patient data highlighting weight and health status is plotted. Here, X-
axis is weights and Y-axis is patient counts. The dark colour regions highlight
patients' weights vs patient counts in health status.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 3

PAIRPLOT

 Pairplot or scatter matrix is a data visual technique for multivariate data. A

scatter matrix consists of several pair-wise scatter plots of variables of the
multivariate data.
 All the results are presented in a matrix format.
 By visual examination of the chart, one can easily find relationships among the
variables such as correlation between the variables.
 A random matrix of three columns is chosen and the relationship of the
columns is plotted as a pairplot or scattermatrix.

ESSENTIAL MATHEMATICS FOR MULTIVARIATE DATA

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 4

  Machine learning involves many mathematical concepts from the domain of
Linear algebra, Statistics, Probability and Information theory.
 While all branches of mathematics are crucial for machine learning, linear
algebra plays a major large role as it is the mathematics of data.
 Linear algebra deals with linear equations, vectors, matrices, vector spaces and
transformations.
 These are the driving forces of machine learning and machine learning cannot
exist without these data types.

Linear Systems and Gaussian Elimination for Multivariate Data

 A linear system of equations is a group of equations with unknown variables.

 This is true if y is not zero and A is not zero. The logic can be extended for N-
set of equations with 'n’ unknown variables.

 If there is a unique solution, then the system is called consistent independent.

 If there are various solutions, then the system is called consistent dependant.
 If there are no solutions and if the equations are contradictory, then the system
is called inconsistent.
 For solving large number of system of equations, Gaussian elimination can be
used. The procedure for applying Gaussian elimination is given as follows:
1. Write the given matrix.
2. Append vector y to the matrix A. This matrix is called augmentation
matrix.
3. Keep the element all as pivot and eliminate all a11 in second row using
the matrix operation.

4. Repeat the same logic and reduce it to reduced echelon form. Then, the
unknown variable as:

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 5

 5. Then, the remaining unknown variables can be found by back-
substitution as:

This part is called backward substitutions

 To facilitate the application of Gaussian elimination method, the following row

operations are applied:
1. Swapping the rows
2. Multiplying and dividing a row by a constant
3. Replacing a row by adding or subtracting a multiple of another row to it

Note: Problems on Gaussian Elimination Method Solved in Class

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 6

MATRIX DECOMPOSITION:

 It is often necessary to reduce a matrix to its constituent parts so that complex

matrix operations can be performed.

 The most popular matrix decomposition is called eigen

decomposition. It is a way of reducing the matrix into eigen values
and eigen vectors.
 Then, the matrix A can be decomposed as

 Where, Q is the matrix of eigen vectors, A is the diagonal matrix and

Q'is the transpose of matrix Q.

LU DECOMPOSITION

 One of the simplest matrix decompositions is LU decomposition where the

matrix A can be decomposed as

 Here, L is the lower triangular matrix and U is the upper triangular matrix.
 decomposition is done using Gaussian elimination method
 First, an identity matrix is augmented to the given matrix then, row operations
and Gaussian elimination is applied to reduce the given matrix to get matrices
L and U.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 7

  It can be cross verified that the multiplication of L and U yields the original
matrix A.

MACHINE LEARNING AND IMPORTANCE OF PROBABILITY AND

STATISTICS

 Machine learning is linked with statistics and probability. Like linear algebra,
statistics is the heart of machine learning

PROBABLITY DISTRIBUTION

 Distribution is a parameterized mathematical function. In other words,

distribution is a function that describes the relationship between the
observations in a sample space.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 8

  Consider a set of data. The data is said to follow a distribution if it obeys a
mathematical function that characterizes that distribution. The function can be
used to calculate the probability of individual observations.
 Probability distributions are of Hero types:
1. Discrete probability distribution
2. Continuous probability distribution

CONTINUOUS PROBABILITY DISTRIBUTION

 The relationships between the events for a continuous random variable and
their probabilities are called a continuous probability distribution.
 It is summarized as Probability Density Function (PDF)
 PDF calculates the probability of observing an instance
 Normal, Rectangular, and Exponential distributions fall under this category.

1. NORMAL DISTRIBUTION
 This is also known as gaussian distribution or bell-shaped curve
distribution.
 The shape of this distribution is a typical bell-shaped curve.
 In normal distribution, data tends to be around a central value with
no bias on left or right.
 The heights of the students, blood pressure of a population, and
marks scored in a class can be approximated using normal
distribution.
PDF of the normal distribution is given as:

 In normal distribution, mean, median and mode are same.

2. RECTANGULAR DISTRIBUTION
 This is also known as uniform distribution. It has equal probabilities
for all values in the range a,b.
 The uniform distribution is given as follows:

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 9

 3. EXPONENTIAL DISTRIBUTION
 This is a continuous uniform distribution. This probability
distribution is used to describe the time between events in a Poisson
process.
 Exponential distribution is another special case of Gamma
distribution with a fixed parameter of 1.
 This distribution is helpful in modelling of time until an event
occurs.
 The PDF is given as follows:

DISCRETE DISTRIBUTION

 Binomial, Poisson, and Bernoulli distributions fall under this category.

1. Binomial Distribution
 It has only two outcomes: success or failure. This is also called Bernoulli’s
trial.
 The objective of this distribution is to find probability of getting success k out
of n Trials. The way to get success out of k out of n number of trials is given
as:

 The binomial distribution function is given as follows, where p is the

probability of success and probability of failure is (1 — p). The probability of
success in a certain number of trials is given as:

 Combining both, one gets PDF of binomial distribution as

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 10

  Here, p is the probability of each choice, k is the number of choices, and n is
the total number of choices- The mean of binomial distribution is given below:

 And the variance is given as:

 Hence, the standard deviation is given as

2. POISSON DISTRIBUTION
 Given an interval of time, this distribution is used to model the probability of a
given number of events k.
 Some of the examples of Poisson distribution are number of emails received,
number of customers visiting a shop and the number of phone calls received by
the office.
 The PDF of Poisson distribution is given as follows:

 Here, x is the number of times the event occurs and λ is the mean number
of times an event occurs.

3. BEMOULLI DISTRIBUTION

 This distribution models an experiment whose outcome is binary. The outcome

is positive with p and negative with 1 — p.
 The PMF of this distribution is given as:

 The mean is p and variance is p(1 — p) = q

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 11

FEATURE ENGINEERING AND DIMENSIONALITY REDUCTION
TECHNIQUES

 Features are attributes.

 Feature engineering is about determining the subset of features that form an
important part of the input that improves the performance of the model
 Feature engineering deals with two problems
1. Feature Transformation
2. Feature Selection

FEATURE TRANSFORMATION

 Feature transformation is extraction of features and creating new features that

may be helpful in increasing performance.
 For example, the height and weight may give a new attribute called Body Mass
Index (BMI).

FEATURE SELECTION

 It focuses on selection of features to reduce the time but not at the cost of
reliability.
 The subset selection reduces the dataset size by removing irrelevant features
and constructs a minimum set of attributes for machine learning.
 If the dataset has n attributes, then time complexity is extremely high as n
dimensions need to be processed for the given dataset.
 For n attributes, there are 2" possible subsets.
 If the value of n is high, the problem becomes intractable. This is called ’curse
of dimensionality’.
 As the number of dimensions increases, the time complexity increases.
 The remedy is that some of the components that do not contribute much can be
deleted. This results in the reduction of dimensionality.
 The features can be removed based on two aspects:
1. Feature relevancy — some features contribute more for classification than
other features. For example, a mole on the face can help in face detection
than common features like nose.
2. Feature redundancy — some features are redundant. For example, when a
database table has a field called Date of birth, then age field is not relevant
as age can be computed easily from date of birth. This helps in removing
the column age that leads to reduction of dimension one.
 Filter-based selection uses statistical measures for assessing features. In this
approach, no learning algorithm is used.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 12

  Wrapper-based methods use classifiers to identify the best features. These are
selected and evaluated by the learning algorithms. This procedure is
computationally intensive but has superior performance.

STEPWISE FORWARD SELECTION

 This procedure starts with an empty set of attributes.

 Every time, an attribute is tested for statistical significance for best quality and
is added to the reduced set.
 This process is continued till a good reduced set of attributes is obtained.

STEPWISE FORWARD SELECTION

 This procedure starts with a complete set of attributes.

 At every stage, the procedure removes the worst attribute from the set, leading
to the reduced set.

Combined Approach Both forward and reverse methods can be combined so that the
procedure can add the best attribute and remove the worst attribute.

PRINCIPAL COMPONENT ANALYSIS

 Principal component analysis (PCA) or KL transform is to transform a given

set of measurements to a new set of features so that the features exhibit high
information packing properties.
 This leads to a reduced and compact set of features
 The advantages of PCA are immense. It reduces the attribute list by eliminating
all irrelevant attributes.
 The PCA algorithm is as follows:
1. The target dataset x is obtained
2. The mean is subtracted from the dataset. Let the mean be m. Thus, the
adjusted dataset is X-m. The objective of this process is to transform the
dataset with zero mean.
3. The covariance of dataset z is obtained. Let it be C.
4. Eigen values and eigen vectors of the covariance matrix are calculated.
5. The eigen vector of the highest eigen value is the principal component of
the dataset. The Eigen values are arranged in a descending order. The
feature vector is formed with these eigen vectors in its columns.

6. Obtain the transpose of feature vector. Let it be A.

7. PCA transform is y = A (x — m), where x is the input dataset, m is the
mean, and A is the transpose of the feature vector.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 13

 8. The original data can be retrieved using the formula given below:

 The new data is a dimensionally reduced matrix that represents the original
data. Therefore, PCA is effective in removing the attributes that do not
contribute.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 14

LINEAR DISCRIMINANT ANALYSIS:
Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 15
  Linear Discriminant Analysis (LDA) is also a feature reduction technique like
PCA.
 The focus of LDA is to project higher dimension data to a line (lower
dimension data)
 LDA is also used to classify the data.
 Let there be two classes, C1 and C2. Let µ1, and µ2 be the mean of the patterns
of two classes- The mean of the class C1and C2 can be computed as:

The aim of LDA is to optimize the function

 Where, V is the linear projection and σB and σ W are class scatter matrix and
within scatter matrix, respectively. For the two-class problem, these matrices are
given as:

SINGULAR VALUE DECOMPOSITION

 Singular Value Decomposition (SVD) is another useful decomposition

technique
 then the matrix A can be decomposed as:

 Here, A is the given matrix of dimension mx n , U is the orthogonal matrix whose

dimension is mxn, S is the diagonal matrix of dimension nx n , and V is the
orthogonal matrix.
 The procedure for finding decomposition matrix is given as follows:
1. For a given matrix, find AAT
2. Find eigen values of AAT

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 16

 3. Sort the eigen values in a descending order. Pack the eigen vectors as a
matrix U.
4. Arrange the square root of the eigen values in diagonal. This matrix is
diagonal matrix, S.
5. Find eigen values and eigen vectors for ATA. Find the eigen value and pack
the eigen vector as a matrix called V.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 17

  The main advantage of SVD is compression. A matrix, say an image, can be
decomposed and selectively only certain components can be retained by
making all other elements zero. This reduces the contents of image while
retaining the quality of the image. SVD is useful in data reduction too.

OVERFITTING:

 If we want to know how well the algorithm is generalising as it learns.

 we need to make sure that we do enough training that the algorithm generalises
well
 If we train for too long or too much, then we will overfit the data, which means
that we have learnt about the noise and inaccuracies in the data as well as the
actual function.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 18

  On the left of the figure the curve fits the overall trend of the data well (it has
generalised to the underlying general function),
 The training error would still not be that close to zero since it passes near, but
not through, the training data.
 As the network continues to learn, it will eventually produce a much more
complex model that has a lower training error (close to zero), meaning that it
has memorised the training examples, including any noise component of them,
so that is has overfitted the training data.
 We want to stop the learning process before the algorithm overfits, which
means that we need to know how well it is generalising at each timestep.
 We can’t use the training data for this, because we wouldn’t detect overfitting,
but we can’t use the testing data either, because we’re saving that for the final
tests.
 We will use a dataset called as validation set to validate the learning so far.
This is known as cross-validation in statistics.

TRAINING, TESTING, AND VALIDATION SETS

 We need three sets of data: the training set to actually train the algorithm, the
validation set to keep track of how well it is doing as it learns, and the test set
to produce the final results.

 The dataset is divided into three parts: training, testing, and validation sets.
 Typical splits include 50:25:25 (if you have a lot of data) or 60:20:20 (if data is
limited).
 If data is ordered by class (e.g., all class 1 data first, then class 2), simply
splitting it sequentially can cause poor results.
 To avoid this, data should be randomly shuffled before splitting or assigned
randomly to each set.
Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 19
  When data is scarce, using a separate validation set might reduce training
effectiveness.
 Instead, K-fold cross-validation is used, where the dataset is split into K subsets
and the model is trained multiple times, each time leaving out a different subset
for validation.
 In the extreme case, leave-one-out cross-validation (LOO-CV) is used, where
each data point is tested individually while training on all the remaining data.
 This ensures the model is validated on every single point, but it is
computationally expensive.
 More validation means better model evaluation, but it also requires training
multiple models, increasing computation time.
 A balance must be struck between data availability and computational
resources to achieve the best results.

THE CONFUSION MATRIX

 After training a model, we need to check if its predictions are good.

 For classification problems, we use a confusion matrix, while for regression
problems, we typically use sum-of-squares error to measure prediction
accuracy.
 Confusion matrix is a square matrix that compares actual (target) vs. predicted
classes. Rows represent actual classes, and columns represent predicted classes.
Each element (i, j) shows how many times class i was predicted as class j.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 20

  The leading diagonal (top-left to bottom-right) shows correct predictions.
Off-diagonal values represent misclassifications (errors). Example: If class C3
is misclassified as C1 twice, the matrix will show "2" in the respective position.
 Accuracy = (Sum of correct predictions) / (Total predictions). This is found
by dividing the sum of diagonal values by the sum of all matrix values.
 Accuracy is useful but not always enough. If classes are imbalanced (one class
has way more samples), accuracy can be misleading. Other metrics like
precision, recall, and F1-score help in better evaluating model performance.

ACCURACY METRICS

 Confusion Matrix: A table used for evaluating classification models. It

compares predicted labels with actual labels and contains four key values—
True Positives (TP), False Positives (FP), False Negatives (FN), and True
Negatives (TN).
 Accuracy: It is calculated as

which represents the proportion of correct predictions out of total predictions.

However, accuracy alone may not be sufficient in cases of imbalanced data.
 Sensitivity (Recall) & Specificity: Sensitivity (True Positive Rate): Measures
how well the model identifies actual positives, calculated as

 Specificity (True Negative Rate): Measures how well the model identifies
actual negatives, calculated as

 Precision: The proportion of correctly predicted positive cases among all

predicted positives, given by

 Recall: The proportion of actual positives correctly identified, given by

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 21

  F1 Score: A harmonic mean of precision and recall, given by

It balances precision and recall, making it useful for imbalanced datasets.

THE RECEIVER OPERATOR CHARACTERISTIC (ROC) CURVE

 The Receiver Operating Characteristic (ROC) curve is a graphical

representation of classifier performance. It plots the true positive rate (y-axis)
against the false positive rate (x-axis), helping evaluate and compare classifiers.
 A perfect classifier reaches the point (0,1), meaning 100% true positives and
0% false positives.
 A completely wrong classifier would be at (1,0).
 A classifier that performs at chance level lies along the diagonal from (0,0) to
(1,1), indicating random guessing.
 Instead of using a single point, the Area Under the Curve (AUC) is often
computed. It quantifies overall classifier performance, where a higher AUC
indicates better discrimination between classes.

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 22

  If only one point is available for a classifier, the ROC plot forms a trapezoidal
shape from (0,0) to (1,1). When multiple points are available (e.g., from
different datasets), they are plotted and connected in order.
 Using cross-validation (e.g., 10-fold cross-validation) generates multiple
classifiers, each with different test sets. The results can be ranked and plotted to
create a smoother ROC curve, allowing better comparison of classifiers.

UNBALANCED DATASETS

 When the dataset has an unequal number of positive and negative samples
(unbalanced dataset), accuracy can be misleading. A classifier might predict the
majority class more often and still achieve high accuracy, even if it performs
poorly on the minority class.
 Balanced accuracy is computed as the average of sensitivity (true positive
rate) and specificity (true negative rate).
 MCC is a better metric for evaluating classifiers on imbalanced datasets. It
takes into account true positives (TP), true negatives (TN), false positives (FP),
and false negatives (FN) using a formula that provides a more reliable
performance measure.

 For multiple classes, errors need to be distinguished for each class separately.
Specificity and recall differ for each class, so an approach where each class is
treated as positive while others are negative can be used to compute MCC or
other metrics.

MEASUREMENT PRECISION

 In machine learning, precision refers to the consistency of an algorithm’s

predictions. If similar inputs always produce similar outputs, the algorithm has
high precision. This is different from the traditional classification definition of
precision and is more about measuring the spread of predictions.
 An algorithm can be precise but not accurate. If it consistently predicts the
wrong output, it is "precisely wrong." Accuracy depends on how close
predictions are to the correct values, while precision only measures variability.
 Trueness measures how close predictions are to the actual values on average,
while precision measures the spread of predictions. A darts game analogy helps

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 23

 understand this—some throws may be clustered but far from the target (precise
but not true), while others may be spread out but cantered correctly (true but
not precise).

Sonia Fathima B.E, [Link] Dept of AIML AIT, CKM Page 24