Bivariate data and Multivariate data
• Bivariate data involves two variables
• Bivariate data deals with causes of relationships. The aim is to find
relationships among data
Bivariate Ststistics
• Covariance and Correlation are examples of bivariate statistics
• Covariance is a measure of joint probability of random variables, say X and Y.
• Generally random variables are represented in capital letters
• It is defined as covariance (X,Y) or COV(X,Y) and it is used to measure the variance between two
dimensions
• Here xi and yi are data values from X and Y. E(X) and E(Y) are the mean values of xi and yi. N is
the number of given data
• COV (X,Y) is same as COV (Y,X)
Correlation
• The pearson correlation coefficient is the most common test for
determining any association between two phenomena
• It measures the strength and direction of a linear relationship between
the x and y variables
– If the value is positive, it indicates that the dimensions increase together
– If the value is negative, it indicates that while one dimension increases, the other
dimension decreases
– If the value is zero, then it indicates that both the dimensions are independent of each
other
MULIVARIATE STATISTICS
Feature Engineering and Dimensionality reduction
techniques
• Features are attributes, feature engineering is about
determining the subset of features that form an important
part of the input that improves the performance of the
model, be it classification or any other model in ML
• Feature engineering deals with two problems:
– Feature transformation : it is extraction of features and creating new features
that may be helpful in increasing performance. Example: height and weight
may give new attribute called BMI
– Feature selection: it focuses on selection of features to reduce the time but
not at the cost of reliability. It removes irrelevant features
• Some of the components that do not contribute much can be
deleted dimensionality reduction
Two methods to remove features
• Feature relevancy: some features contribute more for
classification than other features. Ex: a mole on the face can
help in face detection than common features like nose
– The feature should be relevant
– The relevancy of the features can be determined based on information
measures such as mutual information, correlation based features like
correlation coefficient and distance measure
• Feature redundancy: some features are redundant. Ex: a
database table has a field called date of birth, then age field is
not relevant as age can be computed easily from DOB
Dimensionality reduction methods
1. Stepwise forward selection: Stepwise forward selection is a feature
selection technique that starts with an empty set of features and adds them
one by one based on a chosen criterion (e.g., minimizing error or
maximizing accuracy).
• Suppose we have a dataset where we want to predict a target variable (Y)
using three features: X1 (temperature), X2 (Humidity) , X3 (Wind Speed)
– Stepwise Forward Selection Process:
– Start with an empty model (no features).
– Evaluate each feature individually and pick the best one:
– Model with X1 → Error = 8
– Model with X2 → Error = 5 ✅ (Best)
– Model with X3 → Error = 6
– Select X2 (Humidity), as it gives the lowest error.
– Add another feature to the selected one and evaluate:
– Model with X2 + X1 → Error = 3 ✅ (Best)
– Model with X2 + X3 → Error = 4
– Select X1 (Temperature), as it improves performance the most.
– Try adding the remaining feature (X3):
– Model with X2 + X1 + X3 → Error = 3 (No improvement)
– Stop selection since adding X3 does not reduce the error further
– Final Selected Features: X2 (Humidity) and X1 (Temperature
2. Stepwise Backward Elimination: Backward elimination starts with all
features included in the model and removes the least significant one step by
step until only important features remain.
• Example: Suppose we have a dataset where we want to predict a target
variable (Y) using three features: X1 (temperature), X2 (Humidity) , X3
(Wind Speed)
– We start with all features and remove the least useful one based on a performance
metric (e.g., p-value in regression or error in machine learning).
– Start with all features: Model with X1 + X2 + X3 → Error = 3
– Evaluate the impact of removing each feature:
– Remove X1 (Model with X2 + X3) → Error = 3.5
– Remove X2 (Model with X1 + X3) → Error = 4
– Remove X3 (Model with X1 + X2) → Error = 2.8 ✅ (Best)
– Remove X3 (Wind Speed), as its removal improves performance.
– Repeat for the remaining features (X1 and X2):
– Model with X1 + X2 → Error = 2.8
– Removing any more features increases the error → STOP.
– Final Selected Features: X1 (Temperature) and X2 (Humidity)
– Backward elimination ensures that only the most important features remain in the
model while discarding irrelevant ones.
• 3. Principal Component Analysis (PCA): the idea of PCA or KL transform
(Karhunen-Loève expansion ) is to transform a given set of measurements
to a new set of features so that the features exhibit high information
packing properties. This leads to reduced and compact set of features
• PCA algorithm:
– The target dataset x is obtained
– The mean is subtracted from the dataset. Let the mean be m. Thus, the
adjusted dataset is X-m. the objective of this process is to transform the
dataset with zero mean
– The covariance of dataset x is obtained. Let it be C
– Eigen values and eigen vectors of the covariance matrix are calculated
– The eigen vector of the highest eigen value is the principal component of the
dataset. The eigen values are arranged in a descending order . The feature
vector is formed with these eigen vectors in its columns
• Feature vector= {eigen vector1, eigen vector2………}
– Obtain the transpose of feature vector. Let it be A.
– PCA transform is y=A x (x-m), where x is the input dataset, m is the mean and
A is the transpose of the feature vector