3.

Numeric Data

Chapter 3: Numeric data (9 hrs)

1. Multivariate Linear Regression

a) Matrix formulation and OLS estimation
b) Measures of fit: R-squared and Adjusted R-squared
c) Multi-collinearity and variance-inflation factors
2. Non-parametric Regression: Nadaraya-Watson Kernel regression
a) Derivation of the estimator
b) Rules of setting the appropriate bandwidth size
3. Principal Component Analysis
a) Mathematical formulation and relevant derivations
b) Interpreting the principal components

57
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

1. Numeric data
Numeric data consists of values that represent quantities. These values can be used for
performing mathematical operations such as addition, subtraction, averaging, correlation,
regression, etc.
Examples: height (170 cm), weight (55 kg), marks (82), income (Rs. 50,000), temperature
(24.5°C).

Types of Numeric Data

There are two major types:

i) Discrete Numeric Data

 Takes countable values.

 No fractions allowed.
 Examples:
o Number of students (25)
o Number of cars in parking (48)

ii) Continuous Numeric Data

 Takes measurable values and can have decimals.

 Infinite possible values within a range.
 Examples:
o Height (162.5 cm)
o Time (3.74 seconds)
o Temperature (28.3°C)

Multivariate Data
Multivariate data is data involving multiple variables (two or more) recorded for each
observation or individual.
Examples:
 A student's record: (height, weight, marks, attendance)
 A company’s sales data: (sales, profit, number of customers)
 A country’s development indicators: (GDP, literacy rate, life expectancy, inflation)
Each observation has multiple measurements.

58
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Types of Multivariate Data

1. Numeric multivariate data: e.g., height, weight, age, income

2. Categorical multivariate data: e.g., gender, color, nationality
3. Mixed-type multivariate data: e.g., income (numeric) + education (categorical)

Examples from Real Life

Example 1: Health dataset

Person Height Weight BP Cholesterol

A 165 60 120 190

B 172 72 135 210

Each person has multiple features → multivariate.

Example 2: Education dataset

Student Maths Science English

S1 80 75 70

S2 85 78 75

Structure of Multivariate Data

Usually presented as a data matrix:

Where:
 n = number of observations
 p = number of variables
 Each row = one observation
 Each column = one variable

59
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

2. Introduction to Linear Regression

Linear regression is a powerful tool used to predict a continuous dependent variable based on one or
more independent variables. It establishes a linear relationship between the dependent variable
(target) and one or more independent variables (predictors). In our case, we are interested in
predicting CO2 emissions from a car based on its engine size, among other potential factors.

The general equation of linear regression is:

3. Multivariate Linear Regression

It is the extension of simple linear regression.
Multivariate linear regression is a statistical technique that models the relationship between
multiple independent variables and multiple dependent variables.
Multivariate linear regression is designed for situations where you need to predict more than
one outcome at the same time.

60
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data
It helps answer: How does Y change when several predictors change at the same time?

Examples:

 Predicting salary based on experience, education, age

 Predicting house price using size, number of rooms, location score
 Predicting marks using study hours, attendance, IQ score

Why Multivariate Regression? (Uses)

1. Prediction
E.g., predicting house price using size, location, age.
2. Modeling relationships
Understand how hours studied, IQ, and sleep affect exam score.
3. Feature importance
Identify which variables have strongest influence.
4. Control for multiple factors
Example: UNICEF MICS data — predicting child nutrition controlling for income,
mother’s education, sanitation.
5. Forecasting
Useful for time-series + cross-sectional data.

Model Representation
The multivariate linear regression model with p predictors is:
Y = β0 + β1X1 + β2X2 + ⋯ + βpXp + ε
Where:
 Y = dependent variable
 X1,X2,…,Xp = predictors
 β0 = intercept
 β1,…,βp = regression coefficients
 ε = error term

61
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

a) Matrix formulation and OLS estimation

To simplify computations, the model is written in matrix form:

Y=Xβ+ε
Where:

 Y is an n×p matrix of dependent variables (n observations, p response variables)

 X is an n×(k+1) matrix of independent variables (including intercept)
 B is a (k+1)×p matrix of regression coefficients
 E is an n×p matrix of error terms

62
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Example:

Given dataset, represent this into matrix form to model: Y = β0 + β1X1 + β2X2 + ε

Obs X1 X2 Y

1 2 1 5

2 4 2 9

3 3 5 12

Solution:

Step 1: Form the X matrix

Always include:
 A column of 1s for the intercept
 Columns for each predictor

Step 2 — Form the Y vector

Step 3 — Parameter vector

Final Matrix Regression Model

63
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Ordinary Least Squares (OLS) estimation:

 OLS is the most commonly used method for estimating the parameters of a
linear regression model.s
 It finds the best-fitting regression line by minimizing the sum of squared
errors between the observed values and the predicted values.
 In multivariate regression with several predictors:

 OLS uses matrix algebra to estimate all coefficients at once.

OLS estimator is:

Example:

Obs X1 X2 Y

1 2 1 5

2 4 2 9

3 3 5 12

1. Form the design matrix X (include intercept column of 1s) and response vector Y

2. Compute X′X and X′Y

64
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

3. OLS formula

So we need:

Now,

Final estimated regression equation

Assignment:
A real estate analyst wants to predict house prices (in $1000s) based on:
 X1 = Size of the house in hundreds of sq. ft.
 X2 = Number of bedrooms
The dataset collected is:

House Size (X1) Bedrooms (X2) Price (Y)

1 2 1 150

2 3 2 200

3 4 3 250

4 5 3 300

5 6 4 350
Task: Find the estimated regression equation, using matrix formulation and OLS
estimation.

65
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

b) Measures of fit: R-squared and Adjusted R-squared

In regression analysis, after estimating the model, we want to know:

How well does the model explain the variation in the dependent variable (y)?

Two common measures are:

 R-squared (Coefficient of Determination)
 Adjusted R-squared

i) R-squared (Coefficient of Determination)

R-squared (R2) or the coefficient of determination, is the statistical measure of the
variance of the regression (best fit) line from the actual data points. In a generalized
linear regression model the model accuracy ranging from 75 - 95% is considered as a
good prediction model, whereas 100% accuracy means overfitting.

Where:
 Numerator is sum of squares of the residuals (the difference between the
observed and predicted values).
 and Denominator is the total sum of squares (the variance of the observed data).

R-squared values range from 0 to 1:

R-squared quantifies how well a model fits the data. Higher values indicate a better
fit, while lower values suggest the model is less effective.
 An R-squared of 0 indicates that the model explains none of the variability of the
dependent variable.
 An R-squared of 1 indicates that the model explains all the variability of the
dependent variable.

66
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

 R-squared = 0.0: The predicted points are randomly scattered, indicating the
model has no predictive power. The independent variable (x) does not explain any
variation in the dependent variable (y).
 R-squared = 0.5: The predicted points show a moderate relationship with the
actual data. The model explains some variation in y, but a lot remains unexplained.
 R-squared = 1.0: The predicted points perfectly align with the actual data,
indicating a perfect fit. The independent variable explains all the variation in y.

ii) Adjusted R-squared

 Adjusted R-squared is a performance metrics which can be termed as a more
refined version of R-squared which priorities the input features that correlates
with the target variable.
 It takes into account the number of predictors in the model and whether they are
significant.
 While R-squared always increases when more predictors are added, Adjusted R-
squared increases only if the new predictors genuinely improve the model.
 It prevents overfitting by balancing the model’s performance with its complexity.
Formula for Adjusted R-squared is:

Where:
 n = number of data points (observations)
 k = number of predictors (features)

 A higher Adjusted R-squared indicates that the model fits the data well without
including unnecessary predictors, suggesting that the chosen features are

67
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data
meaningful and contribute to explaining the variability in the dependent variable.
On the other hand, a lower or negative Adjusted R-squared suggests that adding
more predictors does not improve the model’s performance and may even harm
it. This occurs when extra predictors add noise rather than value.

Both models have the same adjusted R-squared value of 0.94 explaining significant
variation in the dependent variable:
 Well-Fitted Model: The predicted line aligns well with the data, capturing the
true relationship between X and y.
 Overfitted Model: Though it has a high Adjusted R-squared (0.94), the line is too
wiggly, indicating the model is overfitting by learning noise instead of the true
pattern.

Key Differences Between R-squared and Adjusted R-squared

1. The value of R-squared increases when we increase an independent factor , whereas

the value of Adjusted R-square increases only when the independent factor is
necessary for the dependent factor.
2. The value of R-square can not be negative, whereas the value of Adjusted R-squared
value can be negative.
3. R-squared favors complex models without penalizing for irrelevant predictors, while
Adjusted R-squared penalizes unnecessary predictors, promoting simpler, more
generalizable models.

68
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Numerical:
A researcher is studying how study hours (X₁) and attendance rate (X₂) influence exam score
(Y) of 5 students. The dataset is:

Student X₁ (Hours) X₂ (%) Y (Score)

1 2 70 50

2 3 80 60

3 4 75 65

4 5 85 80

5 6 90 85

The regression model fitted is: ̂

Y = 5 + 6X1 + 0.3X2

1. Compute the predicted values ̂

Y

2. Compute the Total Sum of Squares (SST)

3. Compute the Regression Sum of Squares (SSR)
4. Compute the Residual Sum of Squares (SSE)
5. Calculate R²
6. Calculate Adjusted R²

Solution:

1. Compute Predicted Values 𝑌̂

̂ = 5 + 6X1 + 0.3X2
Y

Student X₁ X₂ Y ̂
𝐘

1 2 70 50 (5 + 12 + 21 = 38)

2 3 80 60 (5 + 18 + 24 = 47)

3 4 75 65 (5 + 24 + 22.5 = 51.5)

4 5 85 80 (5 + 30 + 25.5 = 60.5)

(5 + 36 + 27 =
5 6 90 85
68)

69
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

2. Compute Total Sum of Squares (SST)

The sum of squares total (SST) or the total sum of squares (TSS) is the sum of
squared differences between the observed dependent variables and the overall mean.

̅:
First find 𝐘

Now compute:

Y ̅
Y−𝐘 ̅)2
(Y−𝐘

50 -18 324

60 -8 64

65 -3 9

80 12 144

85 17 289

SST=324+64+9+144+289= 830

3. Compute Regression Sum of Squares (SSR)

The sum of squares due to regression (SSR) or explained sum of squares (ESS) is the
sum of the differences between the predicted value and the mean of the dependent
variable. In other words, it describes how well our line fits the data.

Compute deviations:

𝐘̂ ̂−68
𝐘 ̂−68)2
(𝐘
38 –30 900
47 –21 441
51.5 –16.5 272.25
60.5 –7.5 56.25
68 0 0

SSR =900+441+272.25+56.25+0 =1669.5

70
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

4. Calculate SSE

The sum of squares error (SSE) or residual sum of squares (RSS, where residual means
remaining or unexplained) is the difference between the observed and predicted values.

Y ̂
𝐘 ̂
Residual e=Y−𝐘 e2

50 38 12 144

60 47 13 169

65 51.5 13.5 182.25

80 60.5 19.5 380.25

85 68 17 289

SSE = 144+169+182.25+380.25+289 = 1164.5

5. Calculate R²

This is greater than 1, which cannot happen normally, meaning:

 The assumed regression equation is not correctly fitted, OR
 Coefficients were not obtained using OLS.

6. Calculate Adjusted R²

71
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Final Results

 R² = 2.01 (invalid—indicates poor model specification)

 Adjusted R² = –1.81 (indicates regression is not meaningful)

Assignment:

 A researcher wants to predict exam score (Y) using study hours (X₁) and attendance
(%) (X₂). The sample (n = 6) is:

Student X₁ = Hours X₂ = Attendance (%) Y = Score

1 2 50 37.0

2 3 60 47.0

3 4 65 54.5

4 5 70 60.0

5 6 80 68.0

6 7 90 79.0

 An OLS regression of Y on X₁ and X₂ gives the estimated model (coefficients obtained

by solving β^=(X′X)−1 X′y):
 Y^ = −0.324 + 3.189 X1 + 0.630 X2
 Task: compute predicted values Y^, SST, SSR, SSE, then R² and Adjusted R².

Python Implementation:
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from [Link] import r2_score

# Sample dataset: Predictor (X) and Dependent Variable (y)

[Link](42)
X = [Link](100, 2) # Two predictors
y = 3.5 * X[:, 0] + 1.8 * X[:, 1] + [Link](0, 0.5, 100) # y depends on both
predictors

model = LinearRegression()
72
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

[Link](X, y)

# Predict y using the trained model

y_pred = [Link](X)

# Evaluation - R-squared:

r2 = r2_score(y, y_pred)
print(f"R-squared: {r2:.4f}")

# Evaluation - Adjusted R-squared:

n = [Link][0] # Number of observations

p = [Link][1] # Number of predictors
adjusted_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)
print(f"Adjusted R-squared: {adjusted_r2:.4f}")

73
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

c) Multi-collinearity and variance-inflation factors

Multi-collinearity
Multicollinearity is a statistical phenomenon that occurs when two or more independent
variables in a regression model are highly correlated, indicating a strong linear relationship
among the predictor variables.
Multicollinearity occurs in a multiple regression model when two or more predictor
variables are highly correlated with each other.
Mathematically, if

and this linear relationship is strong, then variable Xj is nearly a linear combination of other
predictors → multicollinearity.

For example, in a regression model, variables such as height and weight or household
income and water consumption often show high correlation.
 If the correlation between two variables is +/- 1.0, then the variables are said to
be perfectly collinear.

Problems Caused:

 Coefficients become sensitive to small changes in the model or data.

 Interpretation of the coefficients becomes unreliable
 Increases standard errors of the coefficients

Why Does Multicollinearity Occur? (Causes)

1. Naturally occurring correlation

e.g., height and weight, income and spending.
2. Derived variables included together
e.g., using both "total marks" and "average marks".
3. Dummy-variable trap
Using all categories of a categorical variable instead of (k – 1).
4. Small dataset
Few observations magnify correlation.

74
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Why is Multicollinearity a Problem?

 While multicollinearity does not reduce prediction accuracy, it creates problems in

interpreting regression coefficients:

 This issue complicates regression analysis by making it difficult to accurately

determine the individual effects of each independent variable on the dependent
variable.
 The presence of multicollinearity can lead to unstable and unreliable coefficient
estimates, making it challenging to interpret the results and draw meaningful
conclusions from the model.
 Detecting and addressing multicollinearity is crucial to ensure the validity and
robustness of regression models.

Strategies to address multicollinearity:

These methods help simplify the model and make sure it provides reliable and interpretable
results.
1. Increase the sample size: to improve model accuracy, making it easier to
differentiate between the effects of different predictors.
2. Remove highly correlated predictors using Variance Inflation Factor (VIF),
which tells you if certain variables are highly correlated. If the VIF is too high,
consider removing one of the correlated predictors to improve model stability.
3. Combine correlated variables and combine them into a single, more meaningful
predictor. This can be done using techniques like Principal Component Analysis
(PCA) or factor analysis, which help reduce redundancy by creating a new variable
that represents the combined information.

Variance-inflation factors
The VIF is a common and effective way to detect multicollinearity. It measures how much
the variance of an estimated regression coefficient is increased due to the correlation among
the predictors. Here’s how to interpret VIF values:
o VIF = 1: Indicates no correlation.
o 1 < VIF < 5: Suggests moderate correlation.

75
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data
o VIF > 5: Indicates high correlation, problematic; needs attention
o VIF > 10: This signals serious multicollinearity

VIF for predictor Xj is:

Where:

 Rj2 = R² from regressing Xj on all other predictors.

Numerical:
Given the following R2 values, compute VIF of each predictor. Identify which
predictor suffers from multicollinearity.

Predictor R² from regressing on other predictors

X₁ 0.80

X₂ 0.55

X₃ 0.92

Solution:
We have:

Given:

Now:

76
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Which predictor suffers multicollinearity?

4. Non-parametric Regression: Nadaraya-Watson Kernel

regression
Non-Parametric Regression:
 Non-parametric regression is used when the relationship between the predictor X and
the response Y is unknown or too complex for a simple linear model.
 In non-parametric regression, we do not assume a fixed formula (like a straight line
or curve).
 Instead of fitting a fixed equation (like Y = β0+β1X), we estimate the relationship
directly from the data.
 This makes non-parametric methods flexible and good when the relationship between
inputs and outputs is unknown.

The Nadaraya–Watson (N–W) estimator:

 The Nadaraya–Watson (N–W) estimator is one of the simplest and most widely used
methods in non-parametric regression.
 The Nadaraya–Watson estimator is a way to predict the value of Y for a given X
without assuming a straight line or any specific formula.

 The Nadaraya–Watson method predicts the value at a point by:

o Looking at nearby data points
o Giving more importance to closer points
o Giving less importance to faraway points
 So, it works like smart averaging.

77
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Simple real-life example:

Suppose you want to predict today’s temperature at 12 PM.

You look at:

 Yesterday at 12 PM
 Last week at 12 PM
 Other nearby times

You trust closer times more than distant ones.

Nadaraya–Watson does the same with data.

Analogy: Temperature around a City

Imagine you want to estimate the temperature at noon in the city center, but you only
have temperature readings from nearby weather stations.
 Stations close to the city center → more reliable for your estimate → higher
weight.
 Stations far away → less reliable → lower weight.
You compute a weighted average of all station temperatures.

That’s exactly what Nadaraya–Watson does for data points.

 It’s like averaging nearby points — but smarter, because closer points count
more.
 Non-parametric → No assumption of a straight line or specific curve.
 Bandwidth matters → It decides how “wide” your neighborhood is:
o Wide → smooth, general trend
o Narrow → follows every tiny fluctuation

Formula in Simple Terms

78
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Example Analogy: Temperature around a City

Imagine you want to estimate the temperature at noon in the city center, but you only
have temperature readings from nearby weather stations.
 Stations close to the city center → more reliable for your estimate → higher
weight.
 Stations far away → less reliable → lower weight.

You compute a weighted average of all station temperatures.

Station Distance (km) Temperature (°C) Weight (closer = higher)

1 25 0.5

3 24 0.3

5 22 0.2

Weighted average:

This is exactly how Nadaraya–Watson estimates Y for a given X.

a) Derivation of the estimator

 In the Nadaraya–Watson regression, the estimator is the formula we use to

estimate the unknown regression function at any point x.

 Simply, it is the predicted value of Y at a specific X=x, calculated as a weighted

average of the observed Y values, where points closer to x get higher weight.

 The Estimator Formula:

79
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Derivation:

We are given n observations:

Assume the regression model:

Step 1: Define the Target Function

We want to estimate the regression function:

Step 2: Using conditional densities, the Conditional Expectation is:

Step 3: Kernel estimate of marginal density is given by:

80
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Kernel estimate of numerator is:

Step 4: Substitute into m(x):

Hence, final form (Nadaraya-Watson Estimator) is:

b) Rules of setting the appropriate bandwidth size

The bandwidth parameter h plays a crucial role in determining the smoothness of the
estimated regression function. A smaller bandwidth results in a less smooth estimate that
is more sensitive to the data points, while a larger bandwidth produces a smoother estimate.
Choosing h is a bias–variance trade-off.

1. Data Density Rule:

 Dense data region → use smaller h

 Sparse data region → use larger h

2. Sample Size Rule:

 Small dataset → use larger h (to reduce noise)

 Large dataset → use smaller h (enough neighbors available)

81
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

3. Problem Complexity Rule

 Highly nonlinear data → smaller h

 Smooth trend data → larger h

4. Rule of Thumb(Silverman's rule):

 One common method for selecting the bandwidth is the rule of thumb, which is
based on minimizing the Mean Squared Error (MSE).
 A typical heuristic for bandwidth selection is:

where:
 σ = standard deviation of X
 n = number of samples

5. Cross-validation:

 Cross-validation is another commonly used method to select the optimal bandwidth.

 In this method, the data is split into training and test sets, and the bandwidth that
minimizes the cross-validation error is chosen.
 Try different values of h.
 Choose the one that minimizes prediction error (e.g., mean squared error)

6. Heuristic Bandwidth Selection:

 In practice, a heuristic selection process might be used based on trial and error.
 For example:- If the data is highly variable, a larger bandwidth is chosen to smooth
out fluctuations.
 If the data is relatively smooth, a smaller bandwidth might be used to capture finer
details of the relationship between the variables

82
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

5. Principal Component Analysis

PCA is a dimensionality reduction technique that transforms a dataset with many variables
into a smaller set of new variables called principal components (PCs). PCA technique helps
us to reduce the number of features in a dataset while keeping the most important
information. It changes complex datasets by transforming correlated features into a smaller
set of uncorrelated components.
 These components keep as much information (variance) as possible.
 They are uncorrelated with each other.
 They are ordered:
o PC1 = direction with maximum variance
o PC2 = second highest variance (orthogonal to PC1)
o etc.

83
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

a) Mathematical formulation and relevant derivations

Step 1: Start with data

Suppose we have a dataset with n observations and p variables
We write it as a matrix:

Step 2: Standardize each data (Z-score)

For each variable, Subtract mean and divide by standard deviation.

This ensures variables are on the same scale, especially important when variables
are in different units (marks, height, age, etc.)

This produces the standardized matrix Z.

Step 3: Compute the covariance matrix

The covariance matrix shows how variables vary together. Each element Sij shows
how variables i and j vary together.

84
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Manual computation is done as:

Step 4: Solve the Eigenvalue–Eigenvector Problem

We solve:

o Eigenvectors → tell us how each principal component is formed

o Eigenvalues → tell us how important each component is

We sort eigenvalues in decreasing order:

The eigenvector corresponding to λ1 is Principal Component 1 (PC1).

 Large eigenvalue → component captures a lot of variation

 Eigenvector → tells how the component is formed from original variables

85
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Step 5: Construct the Principal Components

A principal component is a linear combination of variables.
For the k-th principal component:

This produces new variables, each being a linear combination:

86
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Numerical Example:
Compute PC1 for below dataset:

Solution:
Step 1: Start with data

Obs Systolic (X₁) Diastolic (X₂)

1 126 78

2 128 80

3 128 82

4 130 82

5 132 86

Step 2: Standardize the data (Z-score)

Compute Mean:

Compute sample standard deviations:

87
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

Standardize each variable (z-scores)

Step 3: Compute the covariance matrix

The covariance between variable 1 and variable 2 is:

Similarly,

Compute products Zi1Zi2 (rounded):

Sum of products ≈1.4890+0.1889−0.0473+0.0710+2.0817 = 3.7833

88
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data
So,

Thus the correlation matrix (covariance of Z) is approximately

Step 4: Solve the Eigenvalue–Eigenvector Problem

To calculate the eigenvalues of the covariance matrix, we solve the characteristic
equation:

Solve the quadratic equation

Eigen values
λ1= 1.9455 and λ2 = 0.0545

Find Eigenvector for λ1=1.9455

We solve:

89
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

V2 = 0.9997 V1
V1 = V2 / 0.9997
So the Eigen vector is:
𝑥1 1
V1 = [ ]=[ ]
𝑥2 0.9997
Normalizing this to unit vector, we get:
1 1

V1= [
√12 +𝑜.99972
0.9997
1.414
] = [0.9997]= [0.707]
0.706
√12 +0.99972 1.414

Similarly, if we compute second Eigen vector , we get :

0.707
V2 = [ ]
−0.7071

Step 5: Construct the Principal Components

The principal components are projections of the original data onto these eigenvectors.
PC1 (First Principal Component):
The correct PC1 score for each observation is:
PC1 = 0.707 × ZSystolic + 0.706 × ZDiastolic

Obs Z-Systolic Z-Diastolic PC1 = 0.707·ZS + 0.706·ZD

= 0.707(−1.2280)+0.706(−1.2131)
1 –1.2280 –1.2131 = −0.8678−0.8554
= −1.7232

= 0.707(−0.3507)+0.706(−0.5390)
2 –0.3507 –0.5390 = −0.2476−0.3807
= −0.6283

90
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data

= 0.707(−0.3507)+0.706(0.1349)
3 –0.3507 0.1349 = −0.2476+0.0952
= −0.1524

= 0.707(0.5261)+0.706(0.1349)
4 0.5261 0.1349 = 0.3720+0.0952
= 0.4672

= 0.707(1.4033)+0.706(1.4832)
1.4033
5 1.4832 = 0.9921+1.0460
= 2.0381

b) Interpreting the principal components

What a Principal Component Is

 Think of a principal component as a new variable created from your original
variables.
 Its job is to summarize the data by capturing as much variation (differences) as
possible.
 PC1 captures the most variation, PC2 the next most, and so on.

How PCs Are Made

 Each PC is a combination of the original variables.
 Example:
PC1=0.707×Systolic+0.706×Diastolic

 The numbers (0.707, 0.706) are called loadings.

 They tell us how much each original variable contributes to that PC.

How to Read the Loadings

 Big numbers → strong contribution: If a variable has a bigger loading, it matters

more for that PC.
 Sign (+ or −):

91
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026
 3. Numeric Data
o Same sign → variables increase together along this PC.
o Opposite signs → one increases while the other decreases.
 In our example, Systolic and Diastolic have similar positive loadings → both
increase together.

Understanding PC Scores
 Once we have PCs, we can calculate a score for each observation.
 Example: For a person with high Systolic and Diastolic BP, the PC1 score will be
high.
 This tells us where each person lies along the “main pattern” captured by the
PC.

Why This Is Useful

 Simplifies data: Instead of many variables, we can focus on a few PCs.
 Finds patterns: Observations with similar PC scores are similar in terms of the
original variables.
 Helps visualization: We can plot PC1 vs PC2 to see clusters, trends, or outliers.

Simple example in words:

 If PC1 = combination of Systolic & Diastolic BP:
o High PC1 → high blood pressure
o Low PC1 → low blood pressure
 PC1 tells us the main pattern of blood pressure in the dataset.

***

92
Compiled by: Er. Shiva Ram Dam | Data Science and Analytics | BE | PU | 2026