Module 2

Bivariate Data and Multivariate Data

Bivariate data involves 2 variables and focuses on finding relationships between them.
Example:
Temperature in a shop and sales data.
Temp (°C) Sales of sweets

5 200

10 150

20 140

To analyze relationships, a scatter plot is used.

 It helps visualize trends, differences and relationships between two variables.
 Shows strength, shape, direction and presence of outliers.
 Useful for exploratory data before calculating correlation.
Line graphs are similar to scatter plots.

Bivariate Statistics
Covariance and Correlation are examples of bivariate statistics.
Covariance
Denoted by: COV(X,Y)COV(X,Y)
It is used to measure the variance between two dimensions.
Formula
COV(X,Y)=1N∑i=1N(xi−E(X))(yi−E(Y))COV(X,Y)=N1i=1∑N(xi−E(X))(yi−E(Y))
Where
 xixi and yiyi → data values from X and Y
 E(X)E(X) and E(Y)E(Y) → mean values of xixi and yiyi
 NN → number of data points
Example
Find the covariance of data
X = {1,2,3,4,5}
Y = {1,4,9,16,25}
Mean:
E(X)=155=3E(X)=515=3E(Y)=555=11E(Y)=555=11COV(X,Y)=(1−3)(1−11)+(2−3)(4−11)+(3−3)
(9−11)+(4−3)(16−11)+(5−3)(25−11)5COV(X,Y)=5(1−3)(1−11)+(2−3)(4−11)+(3−3)(9−11)+(4−3)
(16−11)+(5−3)(25−11)COV(X,Y)=12COV(X,Y)=12

Correlation
Correlation is denoted by r.
Pearson correlation coefficient measures the strength and direction of the linear relationship
between two variables.
Formula
r=COV(X,Y)σxσyr=σxσyCOV(X,Y)
Where
σxσx = standard deviation of X
σyσy = standard deviation of Y
Notes
If r>0r>0 → X and Y increase together
If r<0r<0 → one increases while the other decreases
If r=0r=0 → variables are independent
Example
X = {1,2,3,4,5}
Y = {1,4,9,16,25}
μx=3μx=3μy=11μy=11
Standard deviation:
σx=(1−3)2+(2−3)2+(3−3)2+(4−3)2+(5−3)25σx=5(1−3)2+(2−3)2+(3−3)2+(4−3)2+(5−3)2σx=1.41σx
=1.41σy=(1−11)2+(4−11)2+(9−11)2+(16−11)2+(25−11)25σy
=5(1−11)2+(4−11)2+(9−11)2+(16−11)2+(25−11)2σy=8.6486σy
=8.6486COV(X,Y)=12COV(X,Y)=12r=12(1.41)(8.6486)r=(1.41)(8.6486)12r=0.984r=0.984

Multivariate Statistics
Multivariate data involves more than two variables and is commonly used in machine learning.
Similar to bivariate data but with multiple dependent variables.
Common analysis techniques
 Regression analysis
 Principal Component Analysis (PCA)
 Path analysis
Example
Id Attribute1 Attribute2 Attribute3

1 2 5 2

2 1 4 1

3 3 6 3

Example data has 3 attributes and their mean is called a centroid.

Variance of multivariate data is represented as a covariance matrix.

Heatmap
A graphical representation of a 2-D matrix using colors.
 Darker colors → larger values
 Lighter colors → smaller values
Helps in perceiving patterns easily.
Example:
 Traffic analysis (high vs low traffic regions)
 Health data (weight vs health status)

Pairplot
A visualization tool for multivariate data.
 Consists of scatter plots for each pair of variables.
 Helps in finding relationships and correlations.
Example:
A random matrix of 3 columns plotted as a pairplot to show relationships.

Essential Mathematics for Multivariate Data

Machine learning requires concepts from:
 Linear algebra
 Statistics
 Probability
 Information theory
Linear algebra is essential for ML because it deals with:
 Linear equations
 Vectors
 Matrices
  Transformations

Linear Systems and Gaussian Elimination

A linear system of equations is a group of equations with unknown variables.
Written as:
Ax=bAx=b
Solved as:
x=A−1bx=A−1b
(if A is non-zero)
Types of solutions
 Unique solution → consistent and independent
 Multiple solutions → consistent and dependent
 No solution → inconsistent
Gaussian Elimination Steps
1. Convert system into an augmented matrix.
2. Use pivot element to eliminate variables in other rows.
3. Convert to row echelon form.
4. Use back substitution to find unknowns.
Example system:
2x1+4x2=62x1+4x2=64x1+3x2=74x1+3x2=7
Result:
x1=1x1=1x2=1x2=1

Matrix Decomposition
Matrix decomposition breaks a matrix into smaller parts for easier calculations.
Eigen Decomposition
A matrix is decomposed into eigenvalues and eigenvectors.
A=QΛQTA=QΛQT
Where
Q → eigenvector matrix
Λ → diagonal matrix

LU Decomposition
Splits a matrix into:
A=LUA=LU
Where
L → lower triangular matrix
U → upper triangular matrix
Steps
1. Start with identity matrix
2. Apply Gaussian elimination
3. Reduce matrix to obtain L and U

Machine Learning and Dependence on Probability and Statistics

ML is closely linked to probability and statistics.
Probability helps in understanding data distributions.
Statistics assists data analysts.
Many ML datasets follow different probability distributions.

Probability Distributions
A probability distribution describes how values of a variable are distributed.
It is a mathematical function that determines the probability of different values in a dataset.
Types
1. Discrete probability distribution – countable values
2. Continuous probability distribution – infinite possible values
Key Concepts
Probability Density Function (PDF)
Determines probability for continuous variables.
Cumulative Distribution Function (CDF)
Probability that a variable is ≤ a given value.
Probability Mass Function (PMF)
Used for discrete distributions.

Continuous Probability Distributions

Normal Distribution (Gaussian)
 Bell-shaped curve
 Mean = median = mode
 Used for heights, blood pressure, test scores
Formula:
f(x;μ,σ)=12πσ2e−(x−μ)22σ2f(x;μ,σ)=2πσ21e−2σ2(x−μ)2
Z-score:
z=x−μσz=σx−μ

Uniform Distribution
All values in range [a,b] have equal probability.
Used for:
 fair dice rolls
 random number generation
P(x)=1b−aP(x)=b−a1

Exponential Distribution
Used for modelling time until an event occurs.
Example: waiting time for a bus.
f(x,λ)=λe−λxf(x,λ)=λe−λx
Mean:
1λλ1
Standard deviation:
1λλ1

Discrete Probability Distributions

Binomial Distribution
Models probability of x successes in n trials.
Each trial has two outcomes:
success (p) or failure (q)
P(X=x)=(nx)pxqn−xP(X=x)=(xn)pxqn−x
Mean = npnp
Variance = npqnpq

Poisson Distribution
Models number of events in a fixed time.
Example: number of emails per hour.
P(X=x)=e−λλxx!P(X=x)=x!e−λλx
Mean = λλ
Standard deviation = λλ

Bernoulli Distribution
Models experiment with two outcomes.
Used in binary classification.
f(x;p)={1−p,x=0p,x=1f(x;p)={1−p,p,x=0x=1
Mean = pp
Variance = pqpq

Density Estimation
Let
X={x1,x2,...xn}X={x1,x2,...xn}
be observed values from an unknown distribution.
Density estimation estimates the probability density function from these observations.
If p(x)p(x) is small → x may be an anomaly.
Two types
1. Parametric density estimation
2. Non-parametric density estimation

Parametric Density Estimation

Assumes data follows a known probability distribution p(x∣θ)p(x∣θ).
Where θθ is the parameter set.

Maximum Likelihood Estimation (MLE)

MLE estimates parameters by maximizing the likelihood function.
Likelihood:
L(X;θ)=∏i=1np(xi;θ)L(X;θ)=i=1∏np(xi;θ)
Goal:
max⁡L(X;θ)maxL(X;θ)

Log Likelihood Function

Since products can become very small numerically, we take logarithm:
max⁡∑i=1nlog⁡p(xi;θ)maxi=1∑nlogp(xi;θ)
Sometimes we minimize negative log likelihood (NLL).

MLE in Regression (Gaussian Assumption)

Assume output variable follows Gaussian distribution.
p(y∣x)=12πσ2e−(y−h(x))22σ2p(y∣x)=2πσ21e−2σ2(y−h(x))2
Where
h(x)h(x) = model function (linear regression)
h(x)=β0+β1xh(x)=β0+β1x

Expectation Maximization (EM Algorithm)

Used for MLE when data has latent variables.
Example: Gaussian Mixture Model (GMM).
Steps
1. E-Step: estimate expected PDF and parameters
2. M-Step: optimize parameters using MLE
Repeat until parameters converge.

Non-Parametric Density Estimation

Does not assume a specific distribution.

Parzen Window Estimation

Given n samples.
Let k be number of points in region R.
p=knp=nk
Density estimate:
p(x)=knVp(x)=nVk
Where V is volume of region.
For hypercube:
V=hdV=hd

K-Nearest Neighbour Density Estimation

Uses k nearest neighbours instead of fixed region.
Density estimate is the average value determined by nearest neighbours.

Feature Engineering and Dimensionality Reduction

Feature Engineering
Selecting or transforming features to improve model performance.
Feature transformation
Creates new features.
Example: BMI from height and weight.
Feature selection
Choosing relevant features and removing unnecessary ones.

Feature Subset Selection

Reduces dataset by removing irrelevant and redundant features.
Avoids curse of dimensionality.
Uses optimization to find best subset.

Feature Removal Criteria

Relevancy
Important features contribute more.
Redundancy
Features providing duplicate information can be removed.
Example: age and date of birth.

Selection Process
1. Generate possible subsets
2. Evaluate subsets and model performance
3. Select best subset

Selection Methods
Filter based
Uses statistical measures (correlation, mutual information).
Wrapper based
Uses classifiers for feature selection.
More accurate but computationally expensive.

Algorithms
Stepwise Forward Selection
Starts with no attributes and adds the best one each step.
Stepwise Backward Elimination
Starts with all attributes and removes the worst one each step.
Combined approach
Adds best attribute and removes worst attribute.

Principal Component Analysis (PCA)

Reduces dimensions by transforming data into new features with maximum variance.
Steps
1. Subtract mean from dataset
2. Compute covariance matrix
3. Calculate eigenvalues and eigenvectors
4. Select top eigenvectors
5. Transform dataset
Linear Discriminant Analysis (LDA)
Reduces dimensions while preserving class separability.
Steps
1. Compute class means
2. Compute between-class and within-class scatter matrices
3. Find eigenvalues and eigenvectors
4. Transform data

Singular Value Decomposition (SVD)

Represents matrix A as
A=USVTA=USVT
Where
U and V → orthogonal matrices
S → diagonal matrix with singular values
Steps include computing eigenvalues and eigenvectors and forming matrices.