MODULE -4 21CS752

Module 4
Understanding Data
Bivariate and Multivariate data, Multivariate statistics, Essential mathematics for Multivariate data,
Overview hypothesis, Feature engineering and dimensionality reduction techniques, Basics of Learning
Theory: Introduction to learning and its types, Introduction computation learning theory, Design of
learning system, Introduction concept learning. Similarity-based learning: Introduction to Similarity or
instance based learning, Nearest-neighbour learning, weighted k- Nearest - Neighbour algorithm.

CHAPTER -2
2.6 BIVARIATE DATA AND MULTIVARIATE DATA
Bivariate Data involves two variables. Bivariate data deals with causes of relationships. The aim is
to find relationships among data. Consider the following Table 2.3, with data of the temperature in
a shop and sales of sweaters.

Here, the aim of bivariate analysis is to find relationships among variables. The relationships can then be
used in comparisons, finding causes, and in further explorations. To do that, graphical display of the data is
necessary. One such graph method is called scatter plot.

Scatter plot is used to visualize bivariate data. It is useful to plot two variables with or without nominal
variables, to illustrate the trends, and also to show differences. It is a plot between explanatory and response
variables. It is a 2D graph showing the relationship between two variables. Line graphs are similar to scatter
plots. The Line Chart for sales data is shown in Figure 2.12.

2.6.1 Bivariate Statistics

Covariance and Correlation are examples of bivariate statistics. Covariance is a measure of joint probability
of random variables, say X and Y. Generally, random variables are represented in capital letters. It is defined
as covariance (X, Y) or COV (X, Y) and is used to measure the variance between two dimensions. The formula
[Link] M J, Professor, Dept of CSE, RNSIT
 MODULE -4 21CS752
for finding co-variance for specific x, and y are:

Here, xi and yi are data values from X and Y. E(X) and E(Y) are the mean values of xi and yi. N is the number
of given data. Also, the COV(X, Y) is same as COV(Y, X).

If the given attributes are X = (x1, x2, … , xN) and Y = (y1, y2, … , yN), then the Pearson correlation coefficient,
that is denoted as r, is given as: (σX, σY are the standard deviations of X and Y.)

2.7 MULTIVARIATE STATISTICS

In machine learning, almost all datasets are multivariable. Multivariate data is the analysis of more than two
observable variables, and often, thousands of multiple measurements need to be conducted for one or more
subjects. Multivariate data has three or more variables. The aim of the multivariate analysis is much more.
They are regression analysis, factor analysis and multivariate analysis of variance.

Heatmap A heat map is a graphical representation of data where individual values are represented by
colors. Heat maps are often used in data analysis and visualization to show patterns, density, or intensity of
data points in a two-dimensional grid.
Example: Let's consider a heat map to display the average temperatures (in °C) across different regions in
a country over a week. Each cell in the heat map will represent a temperature for a specific region on a
specific day. This is useful to quickly identify trends, such as higher temperatures in certain regions or
specific days with unusual weather patterns. The color gradient (from blue to red) indicates the
temperature range: cooler colors represent lower temperatures, while warmer colors represent higher
temperatures.

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752

Pairplot
Pairplot or scatter matrix is a data visual technique for multivariate data. A scatter matrix consists of several
pair-wise scatter plots of variables of the multivariate data. A random matrix of three columns is chosen and
the relationships of the columns is plotted as a pairplot (or scatter matrix) as shown in Figure 2.14.

2.8 ESSENTIAL MATHEMATICS FOR MULTIVARIATE DATA

Machine learning involves many mathematical concepts from the domain of Linear algebra, Statistics,
Probability and Information theory. The subsequent sections discuss important aspects of linear algebra
and probability.

2.8.1 Linear Systems and Gaussian Elimination for Multivariate Data

A linear system of equations is a group of equations with unknown variables. Let Ax = y, then the solution
x is given as: x= y/A= A-1y. This is true if y is not zero and A is not zero. The logic can be extended for N-
set of equations with ‘n’ unknown variables. It means if A= and y=(y1 y2…yn), then the unknown
variable x can be computed as: x= y/A= A-1y

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752

If there is a unique solution, then the system is called consistent independent. If there are various
solutions, then the system is called consistent dependant. If there are no solutions and if the equations are
contradictory, then the system is called inconsistent.

For solving large number of system of equations, Gaussian elimination can be used. The
procedure for applying Gaussian elimination is given as follows:
[Link] the given matrix.
[Link] vector y to the matrix A. This matrix is called augmentation matrix.
[Link] the element a11 as pivot and eliminate all a11 in second row using the matrix operation,

R2 - (a21/a11), here R2 is the 2nd row and (a21/a11) is called the multiplier.

The same logic can be used to remove a11 in all other equations.
[Link] the same logic and reduce it to reduced echelon form. Then, the unknown variable as:

[Link], the remaining unknown variables can be found by back-substitution as:

To facilitate the application of Gaussian elimination method, the following row operations are
applied:
[Link] the rows
[Link] or dividing a row by a constant
[Link] a row by adding or subtracting a multiple of another row to it
These concepts are illustrated in Example 2.8.

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752

2.8.2 Matrix Decomposition

It is often necessary to reduce a matrix to its constituent parts so that complex matrix operations can be
performed.
Then, the matrix A can be decomposed as: A=Q ^ QT

where, Q is the matrix of eigen vectors, Λ is the diagonal matrix and Q T is the transpose of matrix Q.

LU Decomposition
One of the simplest matrix decomposition is LU decomposition where the matrix A can be decomposed
matrices: A = LU. Here, L is the lower triangular matrix and U is the upper triangular matrix. The
decomposition can be done using Gaussian elimination method as discussed in the previous section. First,
an identity matrix is augmented to the given matrix. Then, row operations and Gaussian elimination is
applied to reduce the given matrix to get matrices L and U. Example 2.9 illustrates the application of
Gaussian elimination to get LU.

Now, it can be observed that the first matrix is L as it is the lower triangular matrix whose values are the
determiners used in the reduction of equations above such as 3, 3 and 2/3.
The second matrix is U, the upper triangular matrix whose values are the values of the reduced matrix
because of Gaussian elimination.

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752

Introduction to Machine Learning and Probability/Statistics

 Importance: Machine learning relies heavily on statistics and probability to make

predictions and analyze data.
 Statistics in ML: Key for understanding data patterns, measuring relationships, and
quantifying uncertainties.

Probability Distributions

 Definition: A probability distribution describes the likelihood of various outcomes for a variable XXX.
 Types:
o Discrete Probability Distributions: For countable events (e.g., binomial, Poisson).
o Continuous Probability Distributions: For measurable events on a continuum (e.g., normal,
exponential).

Continuous Probability Distributions

1. Normal Distribution (Gaussian Distribution)

 Shape: Bell curve, symmetric around the mean.

 Characteristics: Defined by mean μ and standard deviation σ.
 Probability Density Function (PDF)

 Applications: Common in natural data (e.g., heights, exam scores).

 Z-score: Standardizes data points. Z=X−μ/σ
2. Uniform Distribution (Rectangular Distribution)

 Definition: Equal probability for all outcomes within range [a,b].

 PDF :

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752
3. Exponential Distribution

Definition: Models time between events in a Poisson process

Discrete Probability Distributions

1 Binomial Distribution

 Definition: For trials with two outcomes (success/failure).

 Formula for Probability of k Successes in n Trials:

2 Poisson Distribution

 Definition: Models the number of events in a fixed interval of time.

 PDF

3 Bernoulli Distribution

 Definition: Models a single trial with two outcomes (success/failure).

 Probability Mass Function (PMF)

Density Estimation

 Goal: Estimate the probability density function (PDF) of data.

 Types:
o Parametric Density Estimation: Assumes a known distribution (e.g., Gaussian)
and estimates parameters.
o Non-Parametric Density Estimation: Does not assume a fixed distribution (e.g.,
Parzen window, k-Nearest Neighbors)

[Link] M J, Professor, Dept of CSE, RNSIT

MODULE -4 21CS752
Parametric Density Estimation

1 Maximum Likelihood Estimation (MLE)

 Definition: A method for estimating the parameters of a distribution by maximizing the

likelihood function.
 Likelihood Function: Maximize L(ϴ) for parameter ϴ

Gaussian Mixture Model (GMM) and Expectation-Maximization (EM) Algorithm

 GMM: A probabilistic model assuming data is generated from a mixture of Gaussian

distributions.
 EM Algorithm:
o E-Step: Estimate the distribution parameters for each latent variable.
o M-Step: Optimize parameters using MLE.
 Iteration: Repeat until convergence.

Non-Parametric Density Estimation Methods

1 Parzen Window

 Definition: A non-parametric technique that estimates the PDF based on local samples.
 Example: Uses a kernel function like Gaussian around each data point.

2 k-Nearest Neighbors (KNN)

 Definition: Estimates density by considering the kkk closest neighbors.

 Application: Frequently used in classification tasks.

2.9 Overview of Hypothesis Testing and Comparing Learning Methods

Overview of Hypothesis

Data collection alone is not enough. Data must be interpreted to give a conclusion. This assumption
of the outcome is called a hypothesis. Statistical methods are used to confirm or reject the hypothesis.
- Null Hypothesis (H0): The initial assumption or existing belief (often represents no effect or no
difference).
- Alternative Hypothesis (H1): Represents the hypothesis the researcher aims to establish.

Types of Hypothesis Tests

1. Parametric Tests: Based on parameters like mean and standard deviation (e.g., t-test, Z-test).
[Link] M J, Professor, Dept of CSE, RNSIT
MODULE -4 21CS752
2. Non-Parametric Tests: Dependent on data characteristics, like event independence or
distribution type.

Steps in Hypothesis Testing

1. Define null and alternate hypothesis.

2. Describe the hypothesis using parameters.
3. Choose the statistical test and set significance value (α).
4. Compute p-value (probability value).
5. Decide to accept or reject the hypothesis based on p-value and α.

Types of Errors in Hypothesis Testing

- Type I Error (False Positive): Incorrect rejection of a true null hypothesis.

- Type II Error (False Negative): Failure to reject a false null hypothesis.

Hypothesis Testing in Machine Learning

- Sample Error: Estimated error based on a sample dataset.

- True (Actual) Error: Error probability for a random instance; hard to calculate directly due to
large population size.
- Sample Error Formula: Given a sample S, sample error is the fraction of misclassified instances.

p-value

The p-value indicates the probability that the null hypothesis is true.
- If p-value ≤ α, reject H0 (null hypothesis).
- If p-value > α, accept H0.

Confidence Intervals

- Formula: Confidence Interval = 1 - α

- Example: For 90% confidence, we say there’s a 90% chance the true mean lies within the interval.
- Margin of Error: Given by x ± (z * s / sqrt(N))
- s is standard deviation, N is sample size, z is z-score for confidence level.

Comparing Learning Methods

Z-test

- Used for: Large sample sizes with known population variance.

- Z-statistic Formula: Z = (X̄ - μ) / (σ / sqrt(N))

[Link] M J, Professor, Dept of CSE, RNSIT

MODULE -4 21CS752
- X̄ : Sample mean, μ: Population mean, σ: Population standard deviation, N: Sample size.

t-test and Paired t-test

- t-test: Checks if the difference between sample means is significant.

- Formula: t = (X̄ - μ) / (s / sqrt(n))
- s: Sample standard deviation, n: Sample size.

Independent Two-Sample t-test:

- Compares two independent groups (e.g., Group A and B).
- Formula: t = (X̄ A - X̄ B) / sqrt(s^2 / NA + s^2 / NB)

Paired t-test

Used when samples are dependent (e.g., pre and post tests for the same subjects).
- Formula: t = (d̄ ) / (sd / sqrt(n))
- d̄ : Mean difference between pairs, sd: Standard deviation of differences, n: Number of pairs.

Chi-Square Test

Overview
The Chi-Square Test is a non-parametric test used to determine if there is a significant association
between observed and expected frequencies in categorical data. It’s often used for:
1. Goodness-of-fit: Testing if sample data matches an expected distribution.
2. Test of independence: Checking if two categorical variables are independent of each other.

This test is helpful in identifying duplications, redundancies, or dependencies between categories.

Key Concepts
Observed Frequency (O): The actual count of occurrences in each category.
Expected Frequency (E): The count expected if the null hypothesis is true.
Degree of Freedom (df): The number of categories minus one (C - 1), where C is the number of
categories.

Formula for Chi-Square Statistic

The formula for calculating the Chi-Square (χ²) statistic is:

χ² = Σ [(O - E)² / E]

Hypotheses
In the Chi-Square Test, we set up two hypotheses:
- Null Hypothesis (H₀): There is no significant difference between the observed and expected
frequencies.

[Link] M J, Professor, Dept of CSE, RNSIT

MODULE -4 21CS752
- Alternate Hypothesis (H₁): There is a significant difference between the observed and expected
frequencies.

Steps to Perform the Chi-Square Test

1. Set Hypotheses: Define H₀ and H₁.
2. Calculate Expected Frequency (E): Multiply the row total by the column total and divide by the
grand total.
3. Apply the Chi-Square Formula: Use χ² = Σ [(O - E)² / E]. -------------- (2.51)
4. Calculate Degrees of Freedom: df = C - 1, where C is the number of categories.
5. Compare p-value with Significance Level (α):
- If p-value ≤ α (e.g., 0.05), reject the null hypothesis.
- If p-value > α, fail to reject the null hypothesis.

Example: Chi-Square Test on Course Registration

Consider a class where 50 boys and 50 girls are given the option to register for a machine learning
course. The data on who registered is summarized in the table:

Solution
1. Set Hypotheses:

- H₀: There is no difference between boys and girls in course registration.

- H₁: There is a significant difference between boys and girls in course registration.

2. Calculate Expected Frequencies:

- Expected value for boys who registered = (Total boys × Total registered) / Grand Total.
- Repeat the process to calculate expected frequencies for each cell.

3. Apply the Chi-Square Formula: Calculate the Chi-Square statistic using χ² = Σ [(O - E)² / E].

4. Degree of Freedom: df = C - 1 = 2 - 1 = 1.

5. Interpret the p-value:

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752
- Let’s assume the calculated χ² statistic gives a p-value of 0.0412.
- Since 0.0412 < 0.05, we reject the null hypothesis, indicating a significant difference between
boys and girls in course registration.

Conclusion
Based on the Chi-Square Test, we conclude that there is a statistically significant difference between
boys and girls in terms of course registration for the machine learning class.

Summary Points
- Chi-Square Test helps compare observed vs. expected data frequencies.
- Use χ² = Σ [(O - E)² / E] formula for calculations.
- A p-value less than the significance level (e.g., 0.05) indicates a significant result.

2.10 FEATURE ENGINEERING AND DIMENSIONALITY REDUCTION TECHNIQUES

Features are attributes. Feature engineering is about determining the subset of features that form
an important part of the input that improves the performance of the model, be it classification or any other
model in machine learning.

Feature engineering deals with two problems – Feature Transformation and Feature Selection.
Feature transformation is extraction of features and creating new features that may be helpful in increasing
performance. For example, the height and weight may give a new attribute called Body Mass Index (BMI).

Feature subset selection is another important aspect of feature engineering that focuses on selection of
features to reduce the time but not at the cost of reliability.

The features can be removed based on two aspects:

[Link] relevancy – Some features contribute more for classification than other features. For
example, a mole on the face can help in face detection than common features like nose. In simple
words, the features should be relevant.
Feature redundancy – Some features are redundant. For example, when a database table has a field called
Date of birth, then age field is not relevant as age can be computed easily from date of birth.
So, the procedure is:
[Link] all possible subsets
[Link] the subsets and model performance
[Link] the results for optimal feature selection

Filter-based selection uses statistical measures for assessing features. In this approach, no learning
algorithm is used. Correlation and information gain measures like mutual information and entropy are all
examples of this approach.

Wrapper-based methods use classifiers to identify the best features. These are selected and evaluated by
the learning algorithms. This procedure is computationally intensive but has superior performance.

2.10.1 Stepwise Forward Selection

This procedure starts with an empty set of attributes. Every time, an attribute is tested for statistical
significance for best quality and is added to the reduced set. This process is continued till a good reduced
set of attributes is obtained.

2.10.2 Stepwise Backward Elimination

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752
This procedure starts with a complete set of attributes. At every stage, the procedure removes the worst
attribute from the set, leading to the reduced set.

2.10.3 Principal Component Analysis

The idea of the principal component analysis (PCA) or KL transform is to transform a given set of
measurements to a new set of features so that the features exhibit high information packing properties.
This leads to a reduced and compact set of features. Consider a group of random vectors of the form:

The mean vector of the set of random vectors is defined as:

The operator E refers to the expected value of the population. This is calculated theoretically using the
probability density functions (PDF) of the elements xi and the joint probability density functions between
the elements xi and xj. From this, the covariance matrix can be calculated as:

The mapping of the vectors x to y using the transformation can now be described as:

This transform is also called as Karhunen-Loeve or Hoteling transform. The original vector x
can now be reconstructed as follows:

If K largest eigen values are used, the recovered information would be:

The PCA algorithm is as follows:

[Link] target dataset x is obtained
[Link] mean is subtracted from the dataset. Let the mean be m. Thus, the adjusted dataset is X – m.
The objective of this process is to transform the dataset with zero mean.
[Link] covariance of dataset x is obtained. Let it be C.
[Link] values and eigen vectors of the covariance matrix are calculated.
[Link] eigen vector of the highest eigen value is the principal component of the dataset. The eigen
values are arranged in a descending order. The feature vector is formed with these eigen vectors in
its columns.
Feature vector = {eigen vector1, eigen vector2, … , eigen vectorn}
[Link] the transpose of feature vector. Let it be A.
[Link] transform is y = A × (x – m), where x is the input dataset, m is the mean, and A is the transpose
of the feature vector.
The original data can be retrieved using the formula given below:

The new data is a dimensionaly reduced matrix that represents the original data.
Figure 2.15. The scree plot indicates that only 6 out of 246 attributes are important.

From Figure 2.15, one can infer the relevance of the attributes. The scree plot indicates that
the first attribute is more important than all other attributes.

2.10.4 Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is also a feature reduction technique like PCA. The focus of LDA
is to project higher dimension data to a line (lower dimension data). LDA is also used to classify the
data. Let there be two classes, c1 and c2. Let m1 and m2 be the mean of the patterns of two classes.
The mean of the class c1 and c2 can be computed as:

[Link] M J, Professor, Dept of CSE, RNSIT

 MODULE -4 21CS752
The aim of LDA is to optimize the function:

2.10.5 Singular Value Decomposition

Singular Value Decomposition (SVD) is another useful decomposition technique. Let A be the
matrix, then the matrix A can be decomposed as:

Here, A is the given matrix of dimension m × n, U is the orthogonal matrix whose dimension is m × n, S is the
diagonal matrix of dimension n × n, and V is the orthogonal matrix. The procedure for finding decomposition
matrix is given as follows:
[Link] a given matrix, find AA^T
[Link] eigen values of AA^T
[Link] the eigen values in a descending order. Pack the eigen vectors as a matrix U.
[Link] the square root of the eigen values in diagonal. This matrix is diagonal matrix, S.
[Link] eigen values and eigen vectors for A^TA. Find the eigen value and pack the eigen vector as a
matrix called V.
Thus, A = USV^ T. Here, U and V are orthogonal matrices. The columns of U and V are left and right
singular values, respectively. SVD is useful in compression, as one can decide to retain only a certain
component instead of the original matrix A as:

Based on the choice of retention, the compression can be controlled.

[Link] M J, Professor, Dept of CSE, RNSIT