Chapter 1

Linear Models

Introduction to Linear Models

Linear models form the foundation of many mathematical and statistical methods used in
engineering, data science, and machine learning. They describe relationships between a dependent
variable (response) and one or more independent variables (predictors) using a linear function.
Despite their simplicity, linear models are powerful tools for prediction, inference, and
interpretation.

A linear model assumes that the dependent variable 𝑦 can be expressed as a linear combination
of input variables 𝑥 1 , 𝑥 2 , . . . , 𝑥 𝑛 and an error term 𝜖:

𝑦 = 𝛽0 + 𝛽1 𝑥 1 + 𝛽2 𝑥 2 + ⋯ + 𝛽𝑛 𝑥 𝑛 + 𝜖

Where:

• 𝑦: Dependent (response) variable

• 𝑥 𝑖: Independent (predictor) variables

• 𝛽𝑖 : Coefficients or parameters of the model

• 𝜖: Random error term accounting for noise or unexplained variation

• The term linear refers to linearity in the parameters 𝛽𝑖 , not necessarily in the variables 𝑥 𝑖.

1.1 Types of Linear Models

Simple Linear Regression:

Involves one predictor variable: 𝑦 = 𝛽0 + 𝛽1 𝑥 + 𝜖

This model represents a straight-line relationship between 𝑥and 𝑦.

Multiple Linear Regression:

Involves two or more predictors: 𝑦 = 𝛽0 + 𝛽1 𝑥1 + 𝛽2 𝑥 2 + ⋯ + 𝛽𝑛 𝑥 𝑛 + 𝜖

Used to model more complex relationships.

Page 1 of 57
Polynomial Regression:

Extends linear regression by including polynomial terms of the predictors to model nonlinear
relationships.

Generalized Linear Models (GLM):

Extends linear models to handle non-normal response variables by applying a link function g(⋅):

𝑔(𝔼[𝑌]) = 𝛽0 + 𝛽1 𝑥1 + ⋯ + 𝛽𝑛 𝑥 𝑛

Examples:

• Logistic Regression (for binary data)

• Poisson Regression (for count data)

Hierarchical or Mixed-Effects Models:

Used when data are organized at more than one level (e.g., patients within hospitals), allowing for
both fixed and random effects in the modeling.

1.2 Assumptions of Linear Models

For the model to produce valid inferences, the following assumptions are made:

1. Linearity: The relationship between predictors and the response is linear.

2. Independence: Observations are independent of each other.

3. Homoscedasticity: The variance of residuals is constant.

4. Normality: The residuals are normally distributed.

5. No Multicollinearity: Predictors should not be highly correlated.

Violating these assumptions can lead to biased or inefficient estimates.

1.3 Estimation of Parameters

The most common method for estimating parameters 𝛽𝑖 is the Least Squares Method, which
minimizes the sum of squared residuals:

Page 2 of 57
 𝑛

min⁡ ∑(𝑦𝑖 − 𝑦̂𝑖 )2 = min⁡ (𝑌 − 𝑋𝛽)𝑇 (𝑌 − 𝑋𝛽)

𝛽 𝛽
𝑖=1

Where:

• 𝑌= vector of observed values

• 𝑋= matrix of predictor variables

• 𝛽= vector of coefficients

The solution is obtained by: 𝛽̂ = (𝑋 𝑇 𝑋)−1 𝑋 𝑇 𝑌

1.4 Model Evaluation Metrics

After fitting a model, its performance is evaluated using statistical measures such as

• R-squared (𝑅 2 ): Proportion of variance in the dependent variable explained by the model.

• Adjusted 𝑅 2 : Adjusted for the number of predictors used.

• Mean Squared Error (MSE): Average of squared differences between observed and
predicted values.

• Root Mean Squared Error (RMSE): Square root of MSE, in the same units as the
response variable.

• p-values & F-statistic: Used for hypothesis testing of model parameters.

1.5 Applications of Linear Models in Engineering

Linear models are widely used across various fields, including

• Social Sciences: To analyze relationships between socio-economic factors and outcomes.

• Health Sciences: For understanding the impact of risk factors on health outcomes.

• Economics: To explore the relationship between economic indicators.

• Marketing: In assessing the effect of different marketing strategies on sales

• Engineering fields:

i. Predictive Maintenance: Estimating machine wear and failure probabilities.

ii. Signal Processing: Linear filters and transformations.

Page 3 of 57
 iii. Control Systems: Modeling system behavior with linear approximations.

iv. Data Analytics: Trend analysis and forecasting.

v. Software Engineering: Effort estimation (e.g., COCOMO model).

vi. Modeling the load–deflection relationship in structures.

vii. Estimating output power in electrical circuits.

viii. Predicting system performance metrics in computer engineering.

1.6 Advantages and Limitations

Advantages

• Simple and interpretable

• Fast to compute

• Works well for linearly separable data

• Provides insights into variable importance

Limitations

• Poor performance on non-linear relationships

• Sensitive to outliers

• Relies on strong statistical assumptions

2. Modeling and Measurement Scales

A scale is a device or an object used to measure or quantify any event or another object.

Understanding measurement scales is essential in linear modeling, as the choice of scale affects
the type of statistical analysis that can be conducted and the interpretations that can be made.
Linear models are designed to work with different types of data, and the measurement scales of
the variables in these models significantly influence analytical processes. Here’s a breakdown of
the types of measurement scales and how they relate to linear modeling.

A model is a simplified representation of a real-world process that helps in understanding,

prediction, and control. Modeling in engineering can be

• Physical models: scaled-down prototypes.

• Mathematical models: using equations and functions.

Page 4 of 57
 • Statistical models: based on data and probabilistic assumptions.

2.1 Measurement Scales

Data in models are measured using four fundamental scales

Scale Type Characteristics Example

Nominal Categories without order Type of processor (Intel, AMD)
Ordinal Ordered categories CPU speed ratings (slow, medium, fast)
Interval Equal intervals, no true zero Temperature in °C
Ratio Equal intervals, true zero Voltage, resistance, or time

Each measurement scale dictates which statistical tools are valid. For instance, correlation and
regression require interval or ratio data.

Nominal Scale:

Definition: This scale categorizes data without a specific order. Each category is mutually
exclusive and there is no ranking among the categories.

Examples: Gender (male, female), marital status (single, married, divorced), and types of vehicles
(sedan, SUV, truck).

Use in Linear Models: While nominal variables cannot be used directly in traditional linear
regression, they can be included in models using techniques like one-hot encoding or creating
dummy variables. For example, if gender is included in a linear model, it would typically be coded
as 0 and 1(e.g., male = 0, female = 1).

Ordinal Scale:

Definition: This scale ranks data in a specific order but does not quantify the difference between
the ranks. It indicates which items have more or less of a certain characteristic.

Examples: Likert scales (e.g., rating satisfaction from 1 to 5), education level (high school,
bachelor's, master's).

Use in Linear Models: Ordinal data can be included in linear models, but care should be taken
when interpreting coefficients, as the differences between ordinal categories may not be equal.
Transformations or specific modeling techniques, such as ordinal logistic regression, may be more
appropriate in some cases.

Page 5 of 57
Interval Scale:

Definition: This scale has both order and equal intervals between values but lacks a true zero point.
The absence of a true zero means that ratios are not meaningful.

Examples: Temperature in Celsius or Fahrenheit, IQ scores.

Use in Linear Models: Interval data can be used directly in linear regression models as predictor
or outcome variables. Since distances between values are meaningful, the coefficients of the linear
model can be interpreted in terms of changes in the dependent variable associated with changes in
the independent variable.

Ratio Scale:

Definition: This scale has order, equal intervals, and a meaningful zero point, allowing for the
comparison of absolute magnitudes. Ratios of numbers on a ratio scale are also meaningful.

Examples: Height, weight, age, income.

Use in Linear Models: Ratio data is also used directly in linear regression models. With a true zero,
the interpretation of coefficients is straightforward, allowing for both differences and ratios to be
communicated clearly.

2.2 Implications for Linear Modeling

Variable Selection: When building a linear model, it is important to recognize the type of variables
(nominal, ordinal, interval, ratio) and choose predictors that will effectively represent the
relationships being modeled.

Model Specification: Based on the measurement scales of your variables, you may need to choose
how to encode nominal data or whether to treat ordinal data differently.

Interpretation of Results: The way you interpret the coefficients depends on the measurement
scale. For example, coefficients for interval or ratio scales provide information about the change
in the dependent variable for a unit change in the independent variable, while coefficients for
nominal scales represent the difference between groups.

Assumptions: Many linear modeling techniques (e.g., ordinary least squares regression) assume
homoscedasticity, normality of residuals, and linearity. Understanding the measurement scales
helps when checking and validating these assumptions.

Page 6 of 57
3. Measures of Central Tendency

Measures of central tendency summarize data with a single representative value.

3.1 Mean
1
The arithmetic mean is the most common measure: 𝑋ˉ = 𝑛 ∑𝑛𝑖=1 𝑋𝑖

Example:
4+6+8+10 +12
For resistance values 𝑅 = [4,6,8,10,12]: 𝑅ˉ = =8
5

3.2 Median

The median is the middle value when data are sorted.

Example: The median of [4, 6, 8, 10, 12] is 8.

3.3 Mode

The mode is the value occurring most frequently.

Example: In [2, 4, 4, 6, 8], the mode = 4.

3.4 Comparison

• The mean is sensitive to outliers.

• The median is robust against extreme values.

• The mode is useful for categorical data.

4. Graphs

In data analysis, visualization is key to understanding the patterns and relationships within the
data. Plotting techniques can help reveal important insights and guide decision making. In this
section, we will explore different types of plots such as univariate, bivariate, and multivariate plots.

Page 7 of 57
4.1 Univariate Plots:

Univariate plots are used to visualize the distribution of a single variable. They are useful for
identifying patterns, outliers, and skewness in the data. Common univariate plots include
histograms, density plots, and box plots.

4.1.1 Univariate Plots for Categorical Data:

Univariate plots for categorical data are used to visualize the distribution of a single categorical
variable. They are useful for identifying patterns, frequencies, and proportions in the data.
Common univariate plots for categorical data include bar plots, count plots, and pie charts.

a) Bar Plots:

A bar plot displays the frequency or proportion of each category in a categorical variable. Each bar
represents a category, with the height of the bar representing the frequency or proportion of
observations in that category.

b) Count Plots:

A count plot is a type of bar plot that displays the count of observations in each category. It is
similar to a bar plot, but instead of displaying the frequency or proportion, it displays the actual
count of observations.

Page 8 of 57
c) Pie Charts:

A pie chart displays the proportion of each category in a categorical variable. Each slice of the pie
represents a category, with the angle of the slice representing the proportion of observations in that
category.

4.1.2 Univariate Plot For Numerical Data:

Univariate plots for numerical data are used to visualize the distribution of a single numerical
variable. They are useful for identifying patterns, outliers, and skewness in the data. Common
univariate plots for numerical data include histograms, density plots, and box plots.

Page 9 of 57
a) Histograms:

A histogram is a bar chart that displays the frequency of a numerical variable’s values. It is created
by dividing the data into intervals, called bins, and counting the number of observations that fall
within each bin.

b) Density Plots:

A density plot is a smoothed version of a histogram. It uses a kernel density estimation (KDE) to
estimate the probability density function of the data. Density plots are useful for comparing the
distribution of two or more datasets.

Page 10 of 57
c) Box Plots:

A box plot, also known as a whisker plot, displays the distribution of a numerical variable using
five number summaries: the minimum, first quartile, median, third quartile, and maximum. It also
shows any potential outliers.

4.2 Bivariate Graphs:

When two variables are analyzed together, the resulting visualization is called a bivariate graph.
Bivariate plots are used to visualize the relationship between two variables. They are useful for
identifying trends, patterns, and correlations. Common bivariate plots include scatter plots, line
plots, and bar plots.

a) Scatter Plots:

A scatter plot displays the relationship between two continuous variables. Each point on the plot
represents an observation, with the x-axis representing one variable and the y-axis representing the
other.

Page 11 of 57
b) Line Plots:

A line plot displays the relationship between two continuous variables, with one variable
represented on the x-axis and the other on the y-axis. Line plots are useful for showing trends over
time.

c) Bar Plots:

A bar plot displays the relationship between two categorical variables. Each bar represents a
category, with the height of the bar representing the value of the second variable.

Page 12 of 57
d) Bubble Chart:

Similar to a scatter plot, but each point is represented by a bubble. The size of the bubble reflects
a third variable, adding another dimension to the analysis.

e) Contour Plot :

A graph used to show the density of data points in two dimensions. It consists of contour lines
representing levels of equal value, helpful for visualizing the relationship between two continuous
variables.

Page 13 of 57
f) Polygon or Line Area Chart:

A chart that displays trends over time by connecting points with a line and filling the area beneath
the line. It provides insights similar to a line graph but emphasizes the magnitude of change.

5. Covariance:

The changes between the two variables, such that change in one variable is equal to change in
another variable. This is the property of a function of maintaining its form when the variables are
linearly transformed. Covariance is measured in units, which are calculated by multiplying the
units of the two variables.

5.1 Types of Covariance:

Positive Covariance: the variables show similar behavior.

Negative Covariance: both the variables move in the opposite direction

Covariance measures how two variables vary together.

𝑛
1
Cov(𝑋, 𝑌) = ∑(𝑋𝑖 − 𝑋ˉ)(𝑌𝑖 − 𝑌ˉ)
𝑛−1
𝑖=1

i. If Cov(X, Y) > 0, variables move in the same direction; if < 0, they move oppositely.

Page 14 of 57
 ii. If Cov(X, Y) is greater than zero, then we can say that the covariance for any two variables
is positive and both the variables move in the same direction.
iii. If Cov(X, Y) is less than zero, then we can say that the covariance for any two variables is
negative and both the variables move in the opposite direction.
iv. If Cov(X, Y) is zero, then we can say that there is no relation between two variables.

Example:

X Y
2 5
4 10
6 12
8 20

𝑋ˉ = 5, 𝑌ˉ = 11.75
1
Cov(𝑋, 𝑌) = [(2 − 5)(5 − 11.75) + (4 − 5)(10 − 11.75) + (6 − 5)(12 − 11.75) + (8
3
− 5)(20 − 11.75)]
1
= [20.25 + 1.75 + 0.25 + 24.75] = 15.67
3
A positive covariance indicates X and Y increase together.

6. Z-Scores and Correlation

6.1 Z-Score:

In statistics and data analysis, the Z-score (also known as the standard score) is a crucial concept
used to describe the position of a data point relative to the mean of a data set. It measures how
many standard deviations a value is from the mean. Z-scores are widely used in data
normalization, hypothesis testing, and probability calculations in engineering, computer science,
and data analytics.

Page 15 of 57
Understanding Z-scores helps engineers and data scientists compare values from different datasets,
detect outliers, and standardize features for machine learning algorithms.

The Z-score is defined mathematically as:

𝑋−𝜇
𝑍=
𝜎

Where:

• 𝑋= the raw score (data point)

• 𝜇= the mean of the dataset

• 𝜎= the standard deviation of the dataset

6.2 Interpretation of Z-Scores

Z-Score Interpretation
𝑍 =0 The data point is exactly equal to the mean.
𝑍 =1 The data point is one standard deviation above the mean.
𝑍 = −1 The data point is one standard deviation below the mean.
𝑍 > 3or 𝑍 < −3 The data point is considered an outlier.

Z-scores are especially important when working with normally distributed data, where:

• About 68% of the data lies within ±1 standard deviation.

• About 95% lies within ±2 standard deviations.

• About 99.7% lies within ±3 standard deviations.

This is known as the Empirical Rule or 68–95–99.7 Rule.

Example 1:

Suppose the exam scores of a class are normally distributed with:

• Mean 𝜇 = 70

• Standard deviation 𝜎 = 10

• A student scored 𝑋 = 85

Finding the Z-score and interpret it:

Page 16 of 57
 𝑋 − 𝜇 85 − 70 15
𝑍= = = = 1.5
𝜎 10 10

Interpretation:
The student’s score is 1.5 standard deviations above the mean. This means the student performed
better than the majority of students in the class.

Example 2:

In another test, the mean score is 𝜇 = 60, standard deviation 𝜎 = 5, and a student scored 𝑋 = 50.
𝑋 − 𝜇 50 − 60
𝑍= = = −2
𝜎 5

Interpretation:
This student’s score is 2 standard deviations below the mean, which indicates a below-average
performance.

6.3 Application of Z-Scores in Engineering and Computer Science:

Z-scores are widely used in various engineering and computational contexts.

(a) Data Normalization:

In machine learning and data preprocessing, features are standardized using Z-scores to bring them
to a common scale, improving algorithm performance.
𝑋−𝜇
𝑋normalized =
𝜎

(b) Outlier Detection:

Z-scores help identify outliers in datasets. If ∣ 𝑍 ∣> 3, the data point is typically considered an
outlier.

(c) Quality Control and Process Engineering:

Used to determine if a process measurement lies within acceptable limits of variation (for example,
Six Sigma process control).
Page 17 of 57
(d) Probability and Hypothesis Testing:

Z-scores are used to compute probabilities from the standard normal distribution and perform
Z-tests to compare sample means.

6.4 Relationship with Standard Normal Distribution

When we convert raw data into Z-scores, the data follows a Standard Normal Distribution,
which has:

• Mean (𝜇) = 0

• Standard deviation (𝜎) = 1

This transformation simplifies probability calculations because all normal distributions can be
converted into this standardized form.

Example: Finding Probability Using Z-Score

Suppose we have exam scores that follow a normal distribution with 𝜇 = 70and 𝜎 = 10. What is
the probability that a student scores less than 80?

Find the Z-score

80 − 70
𝑍= =1
10
Using the Z-table, the cumulative probability for 𝑍 = 1 is 0.8413.

Interpretation:
There is an 84.13% chance that a randomly selected student scores less than 80.

6.5 Advantages of Using Z-Score

1. Provides a standardized measure for comparison.

2. Enables normalization of data for statistical analysis.

3. Facilitates outlier detection and data quality control.

4. Converts any normal distribution to the standard normal distribution.

5. Useful in hypothesis testing and predictive modeling.

Page 18 of 57
6.6 Limitations of Z-Score

1. Assumes data is normally distributed; not suitable for skewed data.

2. Sensitive to outliers, which can distort mean and standard deviation.

3. Doesn’t provide information about the shape of the distribution.

The Z-score is a fundamental statistical concept that quantifies how far a data point lies from the
mean in terms of standard deviations. It plays a critical role in data analysis, probability
computation, and machine learning preprocessing. For engineers and computer scientists,
mastering Z-score concepts allows for effective data standardization, anomaly detection, and
interpretation of results within various analytical contexts.

6.7 Correlation Coefficient

Correlation estimates the depth of the relationship between variables. It is the estimated measure
of covariance and is dimensionless. In other words, the correlation coefficient is a constant value
always and does not have any units.

The Pearson correlation coefficient (r) measures the strength and direction of a linear
relationship:
Cov(𝑋, 𝑌)
𝑟=
𝜎𝑋 𝜎𝑌
• 𝑟 = 1: perfect positive correlation

• 𝑟 = −1: perfect negative correlation

• 𝑟 = 0: no correlation

Where:
• 𝑟 = correlation between the variables X and Y

• Cov(X,Y) = covariance between the variables X and Y

• 𝜎𝑋 = standard deviation of the X variable

• 𝜎𝑌 = standard deviation of the Y variable

Based on the value of correlation coefficient, we can estimate the type of correlation between the
given two variables. Also, the graphical representation of correlation among two variables is given
in the below figure.

Page 19 of 57
6.8 Covariance vs Correlation

Example 1:

Number of observations = 6

Mean of X = 17.67

Mean of Y = 25.5

Page 20 of 57
 X 2 8 18 20 28 30
Y 5 12 18 23 45 50

Cov(X, Y) = (⅙) [(2 – 17.67)(5 – 25.5) + (8 – 17.67)(12 – 25.5) + (18 – 17.67)(18 – 25.5) + (20 –
17.67)(23 25.5) + (28 – 17.67)(45 – 25.5) + (30 – 17.67)(50 – 25.5)] = 157.8

Example 2:

Given covariance 15.67, if 𝜎𝑋 = 2.58 and 𝜎𝑌 = 6.29,

15.67
𝑟= = 0.96
2.58 × 6.29

This indicates a strong positive linear relationship.

Example 3:

A student appeared for two tests. He secured 80 in the first and 75 in the second. The mean and
deviation for the first were 70 and 15 respectively, while for the second it was 64 and 12
respectively. What conclusion can you make on comparing the student's performance for both
exams?

Answer: The student performed better in the second test.

Example 4:

The mean temperature of 60 airports was recorded to be 65 degrees with a standard deviation of 5
degrees. If an airport records a temperature of 68 degrees what percentage of temperatures lie
below this value.

Answer: Around 72.6% of temperatures lie below 68 degrees

7. Ordinary Least Squares (OLS)

The Ordinary Least Squares (OLS) method is one of the most fundamental techniques in
statistical modeling and regression analysis. It is widely used to estimate the parameters of a linear
regression model, which describes the relationship between one dependent variable and one or
more independent variables.

Page 21 of 57
OLS minimizes the sum of the squared differences (errors) between observed values and the
values predicted by a linear function. It is a cornerstone method in data science, machine learning,
econometrics, and engineering data analysis.

7.1 Concept of Ordinary Least Squares

The OLS method assumes that the relationship between the dependent variable 𝑌 and the
independent variables 𝑋1 , 𝑋2 , . . . , 𝑋𝑛 is linear, and can be expressed as:

𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + ⋯ + 𝛽𝑛 𝑋𝑛 + 𝜖

Where:

• 𝑌: Dependent (response) variable

• 𝑋𝑖 : Independent (predictor) variables

• 𝛽𝑖 : Coefficients (parameters) to be estimated

• 𝜖: Random error term

̂0 , 𝛽
The goal of OLS is to find estimates 𝛽 ̂1 , … , 𝛽
̂𝑛 that minimize the sum of squared residuals
𝑚 𝑚

𝑆 = ∑(𝑌𝑖 − 𝑌̂𝑖 ) = ∑(𝑌𝑖 − 𝛽0 − 𝛽1 𝑋1𝑖 − ⋯ − 𝛽𝑛 𝑋𝑛𝑖 )2

2

𝑖 =1 𝑖 =1

7.2 Mathematical Derivation

For a simple linear regression with one independent variable 𝑋

𝑌𝑖 = 𝛽0 + 𝛽1 𝑋𝑖 + 𝜖𝑖

The OLS estimators of 𝛽0 and 𝛽1 are obtained by minimizing

𝑛

𝑆 = ∑(𝑌𝑖 − 𝛽0 − 𝛽1 𝑋𝑖 )2
𝑖=1

To find the minimum, take partial derivatives of 𝑆 with respect to 𝛽0 and 𝛽1 , and set them equal to
zero
∂𝑆
= −2∑(𝑌𝑖 − 𝛽0 − 𝛽1 𝑋𝑖 ) = 0
∂𝛽0
∂𝑆
= −2∑𝑋𝑖 (𝑌𝑖 − 𝛽0 − 𝛽1 𝑋𝑖 ) = 0
∂𝛽1

Page 22 of 57
Solving these two equations gives

∑(𝑋𝑖 − 𝑋ˉ )(𝑌𝑖 − 𝑌ˉ)

̂1 =
𝛽
∑(𝑋𝑖 − 𝑋ˉ)2
̂0 = 𝑌ˉ − 𝛽
𝛽 ̂1 𝑋ˉ

Thus, the estimated regression line becomes

𝑌̂ = 𝛽
̂0 + 𝛽
̂1 𝑋

7.3 Matrix Form of OLS

For multiple linear regression with several predictors, the OLS model can be expressed in matrix
form

𝑌 = 𝑋𝛽 + 𝜖

Where:

• 𝑌= 𝑛 × 1 vector of dependent variable values

• 𝑋= 𝑛 × (𝑝 + 1)matrix of input variables (including a column of 1’s for the intercept)

• 𝛽= (𝑝 + 1) × 1vector of coefficients

• 𝜖= 𝑛 × 1vector of errors

The OLS estimator for 𝛽 is

𝛽̂ = (𝑋 𝑇 𝑋)−1 𝑋 𝑇 𝑌

Example 1: Simple Linear Regression

Let’s consider a dataset of hours studied (X) and marks obtained (Y):

Hours (X) Marks (Y)

1 2
2 4
3 5
4 4
5 5

Page 23 of 57
Step 1: Compute means

𝑋ˉ = 3, 𝑌ˉ = 4

̂1
Step 2: Compute 𝛽

∑(𝑋𝑖 − 𝑋ˉ )(𝑌𝑖 − 𝑌ˉ)

̂1 =
𝛽
∑(𝑋𝑖 − 𝑋ˉ )2
(1 − 3)(2 − 4) + (2 − 3)(4 − 4) + (3 − 3)(5 − 4) + (4 − 3)(4 − 4) + (5 − 3)(5 − 4)
̂1 =
𝛽
(1 − 3)2 + (2 − 3)2 + (3 − 3)2 + (4 − 3)2 + (5 − 3)2
6
= = 0.6
10

̂0
Step 3: Compute 𝛽
̂0 = 𝑌ˉ − 𝛽
𝛽 ̂1 𝑋ˉ = 4 − (0.6 × 3) = 2.2

Hence, the regression line is:

𝑌̂ = 2.2 + 0.6𝑋

Interpretation:
For every additional hour studied, marks increase by 0.6 on average.

Example 2: Multiple Linear Regression (Conceptual)

Suppose we are modeling CPU performance (Y) based on clock speed (X₁) and RAM size (X₂).
The OLS model would be

𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + 𝜖

The parameters 𝛽0 , 𝛽1 , 𝛽2 are estimated using

𝛽̂ = (𝑋 𝑇 𝑋)−1 𝑋 𝑇 𝑌

Such models help engineers predict system performance or resource utilization using statistical
data.

Page 24 of 57
7.4 Assumptions of OLS

OLS works effectively under the following assumptions

1. Linearity: The relationship between dependent and independent variables is linear.

2. Independence: Observations are independent of each other.

3. Homoscedasticity: The variance of residuals is constant across all values of X.

4. Normality: The residuals follow a normal distribution.

5. No Multicollinearity: Independent variables are not highly correlated.

Violations of these assumptions may lead to biased or inefficient estimates.

7.5 Evaluation Metrics

After fitting an OLS model, its performance is typically evaluated using

1. R-squared (𝑅 2 ) – Measures the proportion of variance in Y explained by X.

2. Adjusted R-squared – Adjusts 𝑅 2 for the number of predictors.

3. Mean Squared Error (MSE):

1
𝑀𝑆𝐸 = ∑(𝑌𝑖 − 𝑌̂𝑖 )2
𝑛

4. Root Mean Squared Error (RMSE):

𝑅𝑀𝑆𝐸 = √𝑀𝑆𝐸

5. F-test and t-test – Used to check model and coefficient significance.

7.6 The t-Test:

The t-test is a statistical test used to determine whether there is a significant difference between
the means of two groups. It is mainly used when the sample size is small (usually 𝑛 < 30) and the
population standard deviation is unknown.

Types of t-tests

1. One-sample t-test – compares the sample mean with a known population mean.
Page 25 of 57
 2. Independent two-sample t-test – compares the means of two independent samples.

3. Paired t-test – compares means from the same group at different times or under different
conditions.

Formula For a one-sample t-test:

𝑋ˉ − 𝜇0
𝑡=
𝑠/√ 𝑛

Where:

• 𝑋ˉ = sample mean
• 𝜇0 = population mean (assumed value)

• 𝑠= sample standard deviation

• 𝑛= sample size

Problem:
An engineer wants to check whether two machines produce bolts of the same average length.

Machine Sample Size (n) Mean (mm) Standard Deviation (mm)

A 10 50.5 0.3

B 10 50.2 0.4

Hypotheses:

𝐻0 : 𝜇𝐴 = 𝜇𝐵 (no difference)
𝐻1 : 𝜇𝐴 ≠ 𝜇𝐵

Calculation:

𝑋ˉ1 − 𝑋ˉ 2
𝑡=
𝑠2 𝑠2
√ 1+ 2
𝑛1 𝑛2
50.5 − 50.2 0.3 0.3
𝑡= = = = 1.90
2 2 √ 0.009 + 0.016 0.158
√0.3 + 0.4
10 10

Page 26 of 57
Decision:
At 5% significance level (𝑡0.05,18 = 2.10), since 1.90 < 2.10,
→ Fail to reject 𝐻0 .
Thus, there is no significant difference between the two machines’ output lengths.

7.7 The F-Test

The F-test is used to compare two population variances and to determine if they are significantly
different. It is also used as the basis of ANOVA (Analysis of Variance) to test multiple group
means.
𝑠21
Formula: 𝐹 =
𝑠22

Where:

• 𝑠12 = variance of sample 1 (larger variance placed in the numerator)

• 𝑠22 = variance of sample 2

The test statistic 𝐹follows an F-distribution with (𝑛1 − 1)and (𝑛2 − 1)degrees of freedom.

Problem:
Two different alloys are tested for tensile strength variability.

Alloy Sample Size (n) Variance (𝑠 2 )

X 9 25

Y 8 12

Hypotheses:

𝐻0 : 𝜎𝑋2 = 𝜎𝑌2
𝐻1 : 𝜎𝑋2 ≠ 𝜎𝑌2
Calculation:

𝑠𝑋2 25
𝐹= = = 2.083
𝑠𝑌2 12

Page 27 of 57
Degrees of freedom: 𝑣1 = 8, 𝑣2 = 7
At 5% significance level, 𝐹0.05,8,7 = 3.73

Decision:
Since 𝐹 = 2.083 < 3.73,
→ Fail to reject 𝐻0 .
Thus, there is no significant difference between the variances of the two alloys.

7.8 Relationship Between t-test and F-test

Comparison t-Test F-Test

Purpose Compares means Compares variances

Distribution Used Student’s t-distribution F-distribution

Example Use Checking if two machines produce Checking if two machines have
same mean size equal variability

Relation 𝑡 2 = 𝐹(for two samples) F = t²

7.9 Applications in Engineering

• Quality control: Testing the consistency of materials or machine performance.

• Design of experiments (DOE): Comparing different treatments or processes.

• Reliability testing: Checking whether variability between batches is significant.

• Product testing: Comparing performance means between designs.

Both t-tests and F-tests are vital tools in engineering statistics for comparing means and variances,
respectively. They help engineers make objective, data-driven decisions when evaluating
materials, designs, and manufacturing processes. Correct application ensures reliable conclusions
and improved engineering outcomes.

7.10 Applications in Engineering and Computer Science

OLS has wide applications, such as:

• Software Engineering: Predicting project completion time or software defects.

Page 28 of 57
 • Machine Learning: Basis for linear regression, ridge regression, and LASSO.

• Data Analytics: Forecasting trends and relationships in datasets.

• Signal Processing: Noise reduction and linear filtering.

• Control Systems: Modeling linear dynamic systems.

7.11 Advantages and Limitations

Advantages:

• Simple and easy to interpret.

• Computationally efficient.

• Provides closed-form parameter estimates.

• Widely used in real-world engineering and data applications.

Limitations:

• Sensitive to outliers.

• Assumes linearity and homoscedasticity.

• Cannot handle multicollinearity or non-linear patterns directly.

The Ordinary Least Squares (OLS) method is a fundamental tool in statistical modeling,
providing a simple yet powerful way to estimate relationships between variables. It serves as the
basis for many advanced modeling techniques used in modern data science, artificial intelligence,
and engineering systems analysis. A clear understanding of OLS equips engineers with the ability
to interpret data, detect trends, and make predictions with mathematical precision.

Example:

X Y
1 2
2 3
3 5
4 4

𝑋ˉ = 2.5, 𝑌ˉ = 3.5

Page 29 of 57
 (1 − 2.5)(2 − 3.5) + (2 − 2.5)(3 − 3.5) + (3 − 2.5)(5 − 3.5) + (4 − 2.5)(4 − 3.5)
𝛽̂1 =
(1 − 2.5)2 + (2 − 2.5)2 + (3 − 2.5)2 + (4 − 2.5)2
4.5
= = 0.9
5
𝛽̂0 = 3.5 − 0.9(2.5) = 1.25

So, 𝑌̂ = 1.25 + 0.9𝑋

8. Sampling Distributions and Statistical Inference

In engineering and scientific studies, we often deal with large populations of data, such as network
traffic, production quality, system response times, or sensor readings. Analyzing the entire
population is often impractical or impossible, so we rely on sampling (selecting a representative
subset of the population).

Sampling Distributions and Statistical Inference provide the mathematical foundation for
making predictions or decisions about populations based on sample data.

8.1 Sampling Distributions

A sampling distribution is the probability distribution of a given statistic (like the mean, variance,
or proportion) based on a random sample drawn from a population.

In other words, if we take multiple samples from a population and compute a statistic (e.g., the
sample mean) for each, the distribution of those statistics forms the sampling distribution.

Commonly Used Statistics are,

1
• Sample mean: 𝑋ˉ = ∑𝑛𝑖=1 𝑋𝑖
𝑛

1 𝑛
• Sample variance: 𝑆 2 = 𝑛−1 ∑𝑖 =1(𝑋𝑖 − 𝑋ˉ)2
𝑥
• Sample proportion: 𝑝̂ = 𝑛

8.2 Properties of the Sampling Distribution of the Mean

If we draw samples of size 𝑛 from a population with mean 𝜇 and variance 𝜎 2 :

𝐸(𝑋ˉ) = 𝜇
𝜎2
𝑉𝑎𝑟(𝑋ˉ ) =
𝑛

Page 30 of 57
8.3 Central Limit Theorem (CLT)

The Central Limit Theorem states that, regardless of the shape of the population distribution, the
distribution of the sample mean 𝑋ˉ approaches a normal distribution as the sample size 𝑛 becomes
large (typically 𝑛 ≥ 30).

𝜎2
𝑋ˉ ∼ 𝑁(𝜇, )
𝑛

Example 1:

A sensor in a network node records the packet transmission time, which is normally distributed
with mean 𝜇 = 50ms and standard deviation 𝜎 = 10ms.

If we take a sample of 𝑛 = 25 measurements

Find:

1. The mean and standard deviation of the sample mean.

2. The probability that the sample mean is between 48 ms and 52 ms.

Solution:

𝐸(𝑋ˉ) = 𝜇 = 50
𝜎 10
𝑆𝐷(𝑋ˉ ) = = =2
√ 𝑛 √ 25

Convert to Z-scores:
48 − 50 52 − 50
𝑃(48 < 𝑋ˉ < 52) = 𝑃( <𝑍 < )
2 2
= 𝑃(−1 < 𝑍 < 1) = 0.6826

Interpretation:
There is a 68.26% chance that the sample mean transmission time is between 48 ms and 52 ms.

8.4 Statistical Inference

Page 31 of 57
Statistical inference involves using sample data to draw conclusions or make decisions about a
population. It includes two major processes

• Estimation (Point and Interval)

• Hypothesis Testing

8.5 Estimation

1. Point Estimation

A point estimate is a single value used to approximate a population parameter.

Example:
Sample mean 𝑋ˉ is a point estimate of population mean 𝜇.

2. Interval Estimation (Confidence Intervals)

A confidence interval (CI) gives a range of values that is likely to contain the population
parameter with a certain level of confidence (e.g., 95%).
𝜎
𝐶𝐼 = 𝑋ˉ ± 𝑍𝛼/2
√𝑛

If 𝜎is unknown, use the t-distribution:

𝑠
𝐶𝐼 = 𝑋ˉ ± 𝑡𝛼/2,(𝑛−1)
√𝑛

Example : Confidence Interval for Mean

A sample of 10 resistors has an average resistance of 100 Ω with a standard deviation of 5 Ω. Find
the 95% confidence interval for the true mean resistance.

𝑛 = 10, 𝑋ˉ = 100, 𝑠 = 5, 𝑡0.025 ,9 = 2.262

5
𝐶𝐼 = 100 ± 2.262 × = 100 ± 3.58
√ 10
⇒ (96.42,103.58)

Interpretation:
We are 95% confident that the true mean resistance lies between 96.42 Ω and 103.58 Ω.

Page 32 of 57
8.6 Hypothesis Testing

Hypothesis testing helps engineers decide whether sample data provides enough evidence to
support or reject a claim about the population.

Steps in Hypothesis Testing

1. State the hypotheses:

o Null hypothesis 𝐻0 : No difference or no effect.

o Alternative hypothesis 𝐻1 : There is a difference/effect.

2. Select significance level (α): Commonly 0.05 or 0.01.

3. Choose the appropriate test statistic: 𝑍, 𝑡, 𝜒 2 , etc.

4. Compute the test statistic from sample data.

5. Make a decision:

o Reject 𝐻0 if ∣ 𝑡𝑒𝑠𝑡 𝑠𝑡𝑎𝑡𝑖𝑠𝑡𝑖𝑐 ∣> 𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 𝑣𝑎𝑙𝑢𝑒.

o Otherwise, fail to reject 𝐻0 .

Example: Hypothesis Test for Mean

A manufacturer claims that the mean lifetime of a battery is 500 hours. A random sample of 16
batteries has a mean of 490 hours and standard deviation of 20 hours. Test the claim at a 5%
significance level.

𝐻0 : 𝜇 = 500𝐻1 : 𝜇 ≠ 500
𝑛 = 16, 𝑋ˉ = 490, 𝑠 = 20
𝑋ˉ − 𝜇 490 − 500
𝑡= = = −2.0
𝑠/√ 𝑛 20/4

From the t-table, 𝑡0.025 ,15 = 2.131

∣ 𝑡 ∣= 2.0 < 2.131 ⇒ Fail to reject 𝐻0

Conclusion:
There is no significant evidence that the mean lifetime differs from 500 hours.

Page 33 of 57
8.7 Applications in Engineering

Field Example Application

Computer Networks Estimating average packet delay or loss rate
Manufacturing Testing mean diameter of produced components
Software Engineering Analyzing average response time of algorithms
Data Science / AI Estimating model performance metrics using sampling
Electronics Verifying circuit tolerance levels with hypothesis testing

Conclusion

Sampling distributions and statistical inference are powerful tools for engineers to make data-
driven decisions under uncertainty. Understanding these concepts allows engineers to:

• Analyze small samples to infer about large systems.

• Quantify uncertainty using confidence intervals.

• Test system or process hypotheses with controlled error rates.

In essence, these methods are essential for ensuring quality assurance, reliability, and
optimization in engineering systems.

Example

Suppose the lifetimes of capacitors follow a distribution with mean 𝜇 = 2000 hours and 𝜎 =
400hours.

If 𝑛 = 64capacitors are tested, find the probability that their average lifetime exceeds 2100
hours.

𝑋ˉ − 𝜇 2100 − 2000
𝑍= = =2
𝜎/√ 𝑛 400/8

From normal tables, 𝑃(𝑍 > 2) = 0.0228.

Hence, the probability that the average lifetime exceeds 2100 hours is 2.28%.

Relationship Between Confidence Intervals and Hypothesis Testing

Page 34 of 57
• Decision-Making: The results from both methods can lead to similar conclusions. If a confidence
interval for a parameter does not contain the null hypothesis value, it suggests that the null
hypothesis can be rejected at a corresponding significance level.

• Complementary Tools: A confidence interval provides a range of plausible values for the
population parameter, while hypothesis testing provides a binary decision (reject/not reject the null
hypothesis).

• Interconnected Procedures: For instance, constructing a confidence interval for a population

mean often involves an underlying hypothesis test regarding the mean.
• Effect Size and Practicality: Confidence intervals give more information regarding the magnitude
of an effect, whereas hypothesis testing tells whether an effect is statistically significant

8.8 Confidence Intervals and Hypothesis Testing

Confidence Intervals (CI)

A confidence interval provides a range of plausible values for a population parameter (e.g., mean,
proportion).
𝜎
For a normal distribution with known 𝜎: 𝐶𝐼 = 𝑋ˉ ± 𝑍𝛼/2 √𝑛

𝑠
If 𝜎is unknown, we use the t-distribution: 𝐶𝐼 = 𝑋ˉ ± 𝑡𝛼/2, 𝑛−1
√𝑛

where:

• 𝑍𝛼/2 or 𝑡𝛼/2 = critical value

• 𝑠= sample standard deviation

• Confidence level = 1 − 𝛼

Example of Confidence Interval

Suppose 25 CPU speeds are measured, with a sample mean of 3.2 GHz and sample standard
deviation 0.4 GHz. Find the 95% confidence interval for the true mean CPU speed.
𝑠 0.4
𝐶𝐼 = 𝑋ˉ ± 𝑡𝛼/2, 𝑛−1 = 3.2 ± (2.064) = 3.2 ± 0.165
√𝑛 5

Page 35 of 57
Thus, the 95% CI is (3.035 GHz, 3.365 GHz).

Interpretation: We are 95% confident that the true mean CPU speed lies within this range.

9. Hypothesis Testing

A hypothesis is an assumption or claim about a population parameter (like the mean, proportion,
or variance).
Hypothesis testing is a statistical procedure used to determine whether the data collected from a
sample provides enough evidence to support or reject this claim.

Null Hypothesis (H₀)

The null hypothesis (H₀) is the default assumption or claim that there is no difference, no effect,
or no change.

Example: 𝐻0 : 𝜇 = 100(The mean is 100)

Alternative Hypothesis (H₁ or Hₐ)

The alternative hypothesis (H₁) is what you want to test or prove. It contradicts the null hypothesis.

Example: 𝐻1 : 𝜇 ≠ 100(The mean is not 100)

Level of Significance (α)

The level of significance (α) is the probability of rejecting a true null hypothesis (Type I error).
Common choices: 𝛼 = 0.05,0.01,0.10

Test Statistic

A test statistic is a standardized value calculated from the sample data that is used to decide whether
to reject the null hypothesis.

9.1 Types of Hypothesis Tests

Page 36 of 57
 Test Type When to Use Distribution
Z-test Population variance known or large sample (n ≥ 30) Standard Normal (Z)
t-test Population variance unknown, small sample (n < 30) Student’s t-
distribution
χ²-test (Chi- For variance or goodness-of-fit tests Chi-square (χ²)
square)
F-test To compare two variances F-distribution

Steps in Hypothesis Testing

1. State the hypotheses

𝐻0 : Null hypothesis
𝐻1 : Alternative hypothesis

2. Select the significance level (α)

(e.g., 0.05 or 5%)

3. Select the test statistic

Depending on data type (Z, t, χ², F).

4. Calculate the test statistic from sample data.

5. Find the critical value from the corresponding statistical table.

6. Decision Rule:

o If ∣ 𝑡𝑒𝑠𝑡 𝑠𝑡𝑎𝑡𝑖𝑠𝑡𝑖𝑐 ∣> 𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 𝑣𝑎𝑙𝑢𝑒, reject 𝐻0 .

o Otherwise, fail to reject 𝐻0 .

7. Conclusion:
State the interpretation of the result in the problem’s context.

Example 1: Z-Test for Population Mean

Problem:

A manufacturer claims that the mean lifetime of a type of LED bulb is 10,000 hours. A random
sample of 36 bulbs has an average lifetime of 9,800 hours, with a standard deviation of 600 hours.
Test at a 5% significance level whether the claim is valid.

Solution:

Page 37 of 57
Step 1:
𝐻0 : 𝜇 = 10000
𝐻1 : 𝜇 ≠ 10000(two-tailed test)

Step 2:
Significance level 𝛼 = 0.05

Step 3:
Population standard deviation unknown, but 𝑛 = 36(large sample) ⇒ Z-test

Step 4:

𝑋ˉ − 𝜇
𝑍=
𝑠 /√ 𝑛
9800 − 10000 −200
𝑍= = = −2
600/√ 36 100

Step 5:
From Z-table, for 𝛼 = 0.05, 𝑍0.025 = ±1.96

Step 6:
∣ 𝑍 ∣= 2 > 1.96 ⇒ Reject 𝐻0

Step 7: Conclusion:
There is sufficient evidence to conclude that the mean lifetime is significantly different from
10,000 hours.

Example 2: t-Test for Mean

Problem:

A new algorithm claims to reduce average execution time below 50 ms. A random sample of 10
runs gives an average time of 47 ms with a standard deviation of 4 ms. Test at the 1% significance
level.

Solution:

Step 1:
𝐻0 : 𝜇 = 50
𝐻1 : 𝜇 < 50(left-tailed test)

Step 2:

Page 38 of 57
 𝛼 = 0.01

Step 3:
Population standard deviation unknown, 𝑛 < 30⇒ t-test

Step 4:

𝑋ˉ − 𝜇 47 − 50 −3
𝑡= = = = −2.374
𝑠 /√ 𝑛 4/√ 10 1.264

Step 5:
Degrees of freedom 𝑑𝑓 = 𝑛 − 1 = 9
From t-table, 𝑡0.01 ,9 = −2.821

Step 6:
∣ 𝑡 ∣= 2.374 < 2.821 ⇒ Fail to reject 𝐻0

Step 7: Conclusion:
There is not enough evidence to conclude that the new algorithm significantly reduces execution
time below 50 ms.

Example 3: χ²-Test for Variance

Problem:

A machine produces metal rods with a target variance of 0.04 𝑐𝑚2 . A sample of 10 rods has a
sample variance of 0.08 𝑐𝑚2 . Test at 5% significance level if the machine’s variance is still within
acceptable limits.

Solution:

Step 1:
𝐻0 : 𝜎 2 = 0.04
𝐻1 : 𝜎 2 ≠ 0.04

Step 2:
𝛼 = 0.05, 𝑛 = 10

Step 3:
Use Chi-square test:

2
(𝑛 − 1)𝑠 2 (10 − 1)(0.08)
𝜒 = = = 18
𝜎2 0.04
Page 39 of 57
Step 4:
Degrees of freedom 𝑑𝑓 = 9
From χ² table:
2 2
𝜒0.025 ,9 = 19.02, 𝜒0.975 ,9 = 2.70

Step 5:
2.70 < 18 < 19.02 ⇒ Fail to reject 𝐻0

Step 6: Conclusion:
The variance has not significantly changed from the target value of 0.04 𝑐𝑚2 .

Example 4: F-Test for Equality of Variances

Problem:

Two different servers are tested for response time variability.

Server A: 𝑛1 = 8, 𝑠12 = 0.25
Server B: 𝑛2 = 10, 𝑠22 = 0.10
Test at 5% significance whether their variances differ.

Solution:

𝐻0 : 𝜎12 = 𝜎22
𝐻1 : 𝜎12 ≠ 𝜎22
𝑠12 0.25
𝐹= 2= = 2.5
𝑠2 0.10

Degrees of freedom:
𝑑𝑓1 = 7, 𝑑𝑓2 = 9

From F-table, 𝐹0 .025 ,7,9 = 4.03

𝐹 = 2.5 < 4.03 ⇒ Fail to reject 𝐻0

Conclusion:
There is no significant difference in variance between the two servers’ response times.

Real-World Engineering Applications

Field Example
Page 40 of 57
 Computer Networks Testing if new routing algorithm improves packet delivery time
Software Engineering Verifying if code optimization reduces CPU usage
Manufacturing Checking if new process affects mean product length
Data Science / AI Comparing model accuracies on two datasets
Electronics Testing if the mean voltage output differs from specification

Conclusion

Hypothesis testing provides a systematic way to make objective, data-driven decisions in

engineering.
It helps engineers:

• Validate performance improvements

• Maintain product quality

• Control process variability

• Compare algorithms or systems with statistical confidence

By properly setting hypotheses and interpreting results, engineers can make reliable inferences
and minimize decision errors.

9.2 Z-Test for Mean (σ Known)

𝑋ˉ − 𝜇0
𝑍=
𝜎/√ 𝑛

If ∣ 𝑍 ∣> 𝑍𝛼/2 , reject 𝐻0 .

Example:

Claim: The mean latency of a computer network is 100 ms.

A sample of 50 pings gives 𝑋ˉ = 105ms, 𝜎 = 10ms, and 𝛼 = 0.05.
105 − 100
𝑍= = 3.54
10/√ 50

𝑍0.025 = 1.96; since 3.54 > 1.96, reject 𝐻0 .

Hence, the mean latency is significantly greater than 100 ms.

9.3 t-Test for Mean (σ Unknown)

Page 41 of 57
 𝑋ˉ − 𝜇0
𝑡=
𝑠/√ 𝑛

Compare with 𝑡𝛼/2, 𝑛−1 from the t-table.

Example:

A sample of 16 processors has an average power usage of 65 W and 𝑠 = 5W. Test 𝐻0 : 𝜇 = 60at
5% significance.
65 − 60
𝑡= =4
5/4

𝑡0.025 ,15 = 2.131; since 4 > 2.131, reject 𝐻0 .

Thus, the true mean power usage is greater than 60 W.

10. Type I and Type II Errors

In hypothesis testing, we make decisions based on sample data, but every decision carries a risk
of error.
Even with perfect experimental design and calculations, there is always a chance of making an
incorrect conclusion.

The two kinds of possible mistakes are known as Type I and Type II errors.
Understanding these errors is vital for engineers, because design choices, quality control, and
algorithm evaluation often depend on statistical decisions.

Decision / Reality H₀ True H₀ False

Reject H₀ Type I Error (α) Correct Decision (Power = 1 − β)
Fail to Reject H₀ Correct Decision Type II Error (β)

10.1 Type I Error (α)

Definition: A Type I error occurs when the null hypothesis ((H_0)) is rejected when it is actually
true. In simpler terms, it is the mistake of finding evidence for an effect or difference that does
not actually exist.

• Symbol: The probability of making a Type I error is denoted by (alpha), often referred to as the
"significance level." Common choices for (alpha) are 0.05 (5%) and 0.01 (1%).

Page 42 of 57
• Example: Suppose a medical test is designed to detect a disease. A Type I error would occur if
the test indicates that a person has the disease (rejecting the null hypothesis that they do not have
it) when in fact they are healthy

• Definition: Rejecting the null hypothesis when it is actually true.

• Also called: False Positive.

Symbol: 𝛼(alpha)
Interpretation: Probability of rejecting a true H₀.

𝑃(Type I Error) = 𝛼

Example (Engineering Context):

A software engineer tests whether a new algorithm reduces latency.

• H₀: The mean latency = 50 ms

• H₁: The mean latency < 50 ms

If the test concludes that latency is reduced when it actually isn’t, a Type I error has occurred.

Consequence: The engineer might wrongly adopt a less efficient algorithm.

10.2 Type II Error (β)

Definition: A Type II error occurs when the null hypothesis ((H_0)) is not rejected when it is
actually false. In other words, it is the mistake of failing to find evidence for an effect or
difference that does exist.

• Symbol: The probability of making a Type II error is denoted by (beta). The power of a test
(which is the probability of correctly rejecting a false null hypothesis) is given by (1 - beta).

• Example: Continuing with the medical test example, a Type II error would occur if the test
indicates that a person does not have the disease (failing to reject the null hypothesis) when in
fact they do have it.

• Definition: Failing to reject the null hypothesis when it is actually false.

• Also called: False Negative.

Symbol: 𝛽(beta)
Interpretation: Probability of failing to detect a real effect.

𝑃(Type II Error) = 𝛽

Page 43 of 57
Example (Engineering Context):
If the new algorithm truly reduces latency, but the test fails to detect it, a Type II error occurs.

Consequence: The engineer misses a valuable improvement.

10.3 Relationship between α, β, and Power

• Power of a test is the probability of correctly rejecting a false null hypothesis:

Power = 1 − 𝛽

• Reducing one error generally increases the other:

o If we make α smaller (reduce the chance of false positives), we usually increase β
(miss real effects).

o Hence, engineers must balance α and β based on application importance.

Example of Trade-off:

Situation Desired Control

Safety-critical systems (e.g., aircraft sensor Minimize Type II error → don’t miss real faults
fault detection)
Quality control in manufacturing Minimize Type I error → don’t reject good
products unnecessarily

Example 1: Type I Error

Problem:

A manufacturer claims the mean voltage of a circuit board is 5 V.

A quality engineer tests 100 boards at 5% significance.

Page 44 of 57
 • H₀: μ = 5 V

• H₁: μ ≠ 5 V

• α = 0.05

If the test statistic exceeds the critical range (say due to random variation) and the engineer
rejects H₀ even though μ = 5 V in reality, that is a Type I error.

Probability of this mistake: 0.05 (5%).

Consequence: Rejecting a batch of perfectly good boards.

Example 2: Type II Error

Problem:

Suppose the true mean voltage has dropped to 4.8 V, but the test does not detect this change
(fails to reject H₀).

This is a Type II error (β).

Consequence: Faulty products continue to be used, potentially damaging downstream circuits.

The power of the test (1 − β) represents how effectively the test can detect such deviations.

Example 3: Practical Comparison (Algorithm Testing)

An engineer compares two data-compression algorithms:

Algorithm A (old) vs. Algorithm B (new).

• H₀: Mean compression ratio (B) = Mean compression ratio (A)

• H₁: Mean compression ratio (B) > Mean compression ratio (A)

Possible Outcome Meaning Type of Error

Correctly conclude B is better when it truly is True Positive —
Conclude B is better when it’s not False Positive Type I
Conclude B shows no improvement when it actually is better False Negative Type II

Trade-off:
Setting α = 0.01 reduces false positives (Type I) but increases β, making it harder to detect small
improvements.

Page 45 of 57
10.4 Factors Affecting Type I and Type II Errors

Factor Effect on Errors

Sample size (n) Increasing n decreases both α and β (more reliable test).
Significance level (α) Smaller α decreases Type I error but increases Type II
error.
True difference between means Larger effect size → smaller β (easier detection).
(effect size)
Test variability (σ) Less variability → smaller β (more power).

10.5 Managing and Balancing Errors

Engineers choose α and β based on application needs:

• Medical or safety systems: prioritize low β (don’t miss real issues).

• Production quality control: prioritize low α (don’t reject good units).

• Software benchmarking: may balance both errors moderately (α = 0.05).

Increasing sample size is often the best way to reduce both errors simultaneously.

10.6 Summary Table

Error Symbol Occurs When Known As Example Consequence

Type
Type I α Reject H₀ when it is False Accepting a faulty conclusion that a
true Positive system improved
Type II β Fail to reject H₀ when False Missing an actual improvement
it is false Negative
Power 1−β Correctly reject false True Detecting an improvement
H₀ Detection accurately

Example

In quality control, 𝐻0 : mean diameter = 10 mm.

If we set 𝛼 = 0.05, we have a 5% risk of rejecting a batch that actually meets specifications
(Type I).
If 𝛽 = 0.10, there is a 10% risk of accepting a defective batch (Type II).

Page 46 of 57
11. Multiple Regression

Multiple Regression is an extension of simple linear regression that models the relationship
between one dependent variable (Y) and two or more independent variables (X₁, X₂, …, Xₖ).

The general form of the multiple regression equation is:

𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + ⋯ + 𝛽𝑘 𝑋𝑘 + 𝜀

Where:

• 𝑌: Dependent variable (response)

• 𝑋1 , 𝑋2 , … , 𝑋𝑘: Independent (predictor) variables

• 𝛽0 : Intercept (value of Y when all Xs = 0)

• 𝛽1 , 𝛽2 , … , 𝛽𝑘 : Regression coefficients

• 𝜀: Random error term

Objectives of Multiple Regression

1. To predict the value of a dependent variable using multiple predictors.

2. To quantify the relationship between the dependent and independent variables.

3. To understand how each independent variable individually affects the outcome.

4. To control for confounding variables when analyzing real-world systems.

Assumptions of Multiple Regression

For valid results, the following assumptions must hold:

1. Linearity: Relationship between predictors and response is linear.

2. Independence: Observations are independent.

3. Homoscedasticity: Constant variance of residuals (errors).

4. Normality: Errors are normally distributed.

5. No Multicollinearity: Predictors are not highly correlated with each other.

Page 47 of 57
11.1 Estimation of Regression Coefficients

The regression coefficients are estimated using the Least Squares Method, which minimizes the
sum of squared residuals (differences between observed and predicted Y values).

Minimize ∑(𝑌𝑖 − 𝑌̂𝑖 )2

In matrix form: 𝐘 = 𝐗𝜷 + 𝜺
̂ = (𝐗 𝑇 𝐗)−1 𝐗 𝑇 𝐘
Where, 𝜷

This equation gives the best linear unbiased estimates (BLUE) of regression coefficients under
the classical assumptions.

Example 1: Predicting CPU Load

Problem Statement:

A computer engineer wants to predict the CPU load (Y) of a server based on:
• X₁ = Number of active processes

• X₂ = Average network usage (MB/s)

The observed data for 5 times intervals is:

Time X₁ (Processes) X₂ (Network MB/s) Y (CPU Load %)

1 10 50 40
2 20 60 50
3 30 70 60
4 40 80 70
5 50 90 80

Step 1: Model Form

𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + 𝜀

Step 2: Compute Regression Coefficients

Using software (like Excel, Python, or R), we find:

𝑌̂ = 10 + 0.8𝑋1 + 0.2𝑋2

Step 3: Interpretation

• Intercept (β₀ = 10): When both X₁ and X₂ are zero, expected CPU load is 10%.
Page 48 of 57
 • β₁ = 0.8: Each additional process increases CPU load by 0.8%, holding X₂ constant.

• β₂ = 0.2: Each additional 1 MB/s of network usage increases CPU load by 0.2%, holding
X₁ constant.

Step 4: Prediction Example

For X₁ = 25 processes and X₂ = 75 MB/s,

𝑌̂ = 10 + 0.8(25) + 0.2(75) = 10 + 20 + 15 = 45

Predicted CPU load = 45%

Example 2: Software Performance Estimation

A software engineer measures execution time (Y, in ms) as a function of:

• X₁ = Input size (MB)

• X₂ = Memory usage (MB)

The regression model obtained is:

𝑌 = 5 + 0.3𝑋1 + 0.05𝑋2

Interpretation:

• The base execution time (intercept) = 5 ms

• For each additional MB of input size, time increases by 0.3 ms

• For each additional MB of memory usage, time increases by 0.05 ms

If 𝑋1 = 100MB and 𝑋2 = 200MB,

𝑌̂ = 5 + 0.3(100) + 0.05(200) = 5 + 30 + 10 = 45 ms

Predicted execution time = 45 milliseconds

11.2 Goodness of Fit

The Coefficient of Determination (R²) measures how well the regression model explains the
variability in the dependent variable.
𝑆𝑆𝑟𝑒𝑠
𝑅2 = 1 −
𝑆𝑆𝑡𝑜𝑡

Page 49 of 57
 • 𝑅 2 = 1: Perfect fit

• 𝑅 2 = 0: No relationship

An Adjusted R² accounts for the number of predictors:

2
(1 − 𝑅 2 )(𝑛 − 1)
𝑅𝑎𝑑𝑗 = 1−
𝑛−𝑘 − 1

where
n = number of observations, k = number of predictors.

11.3 Hypothesis Testing of Regression Coefficients

To test whether a predictor significantly affects Y:

𝐻0 : 𝛽𝑖 = 0vs.𝐻1 : 𝛽𝑖 ≠ 0

Compute t-statistic:

𝛽̂𝑖
𝑡=
𝑆𝐸(𝛽̂𝑖 )

If |t| > tₐ/₂, reject H₀ (the variable significantly affects Y).

Example 3: Predicting Product Quality

A production engineer models product quality score (Y) based on:

• X₁ = Temperature (°C)

• X₂ = Pressure (bar)

• X₃ = Processing time (minutes)

Model:

𝑌 = 50 + 0.5𝑋1 + 2𝑋2 − 1.5𝑋3

Interpretation:

• Increasing temperature by 1°C increases quality score by 0.5 (if other factors constant).

• Increasing pressure by 1 bar increases quality by 2 points.

Page 50 of 57
 • Increasing processing time by 1 minute decreases quality by 1.5 points.

For 𝑋1 = 40, 𝑋2 = 5, 𝑋3 = 10:

𝑌̂ = 50 + 0.5(40) + 2(5) − 1.5(10) = 50 + 20 + 10 − 15 = 65

Predicted product quality = 65

11.4 Applications of Multiple Regression in Engineering

Field Example
Computer Predicting network throughput based on bandwidth, latency, and packet
Networks loss
Machine Learning Feature-based prediction of target variables (e.g., linear regression
model)
Software Estimating software defects based on lines of code, complexity, and
Engineering testing time
Electronics Modeling circuit output voltage as a function of input voltage,
temperature, and resistance
Manufacturing Predicting product strength using pressure, material ratio, and cooling
rate

11.5 Advantages and Limitations

Advantages:

• Handles multiple factors simultaneously

• Measures relative importance of variables

• Useful for prediction and control in engineering systems

Limitations:

• Assumes linear relationships

• Sensitive to multicollinearity

• Requires large sample size for reliable estimates

11.6 Matrix Form

In matrix notation, the model is written as:

𝐘 = 𝐗𝛃 + 𝛜

Page 51 of 57
Where:

• 𝐘is an 𝑛 × 1vector of responses

• 𝐗is an 𝑛 × (𝑘 + 1)matrix of predictors (including a column of ones for the intercept)

• 𝜷is a (𝑘 + 1) × 1vector of coefficients

• 𝛜is an 𝑛 × 1vector of errors

11.7 Ordinary Least Squares (OLS) Solution

The least squares estimator of 𝜷is obtained by minimizing the sum of squared residuals:
̂ = (𝐗 𝑇 𝐗)−1 𝐗 𝑇 𝐘
𝜷

The fitted values are:

̂ = 𝐗𝜷
𝐘 ̂

and residuals:
̂
𝐞 = 𝐘 −𝐘

Example

An engineer wants to model CPU performance (𝑌) as a function of clock speed (𝑋1) and number
of cores (𝑋2 ).

CPU Speed (GHz) Cores Performance Score

1 2.0 2 200
2 2.5 4 280
3 3.0 4 320
4 3.5 6 400
5 4.0 8 460
The estimated model (using OLS computation or software) yields:

𝑌̂ = 50 + 70𝑋1 + 25𝑋2

Interpretation:

• Increasing clock speed by 1 GHz increases performance by 70 units.

• Adding one core increases performance by 25 units, keeping other factors constant.

Page 52 of 57
12. Miscellaneous

12.1 Coefficient of Determination (R²)

The R-squared value indicates how well the model fits the data:
𝑆𝑆𝑟𝑒𝑠
𝑅2 = 1 −
𝑆𝑆𝑡𝑜𝑡

Where:

• 𝑆𝑆𝑟𝑒𝑠 = ∑(𝑌𝑖 − 𝑌̂𝑖 )2

• 𝑆𝑆𝑡𝑜𝑡 = ∑(𝑌𝑖 − 𝑌ˉ)2

If 𝑅 2 = 0.95, 95% of the variation in Y is explained by the predictors.

12.2 Adjusted R²

Adding more predictors can artificially inflate 𝑅 2 . The adjusted R² compensates for this:

2
(1 − 𝑅 2 )(𝑛 − 1)
𝑅𝑎𝑑𝑗 = 1−
𝑛−𝑘 − 1

where 𝑘is the number of predictors.

12.3 Significance Testing of Regression Coefficients

To check if a variable significantly affects Y, we perform a t-test for each coefficient:

𝛽̂𝑖
𝑡𝑖 =
𝑆𝐸(𝛽̂𝑖 )

If ∣ 𝑡𝑖 ∣> 𝑡𝛼/2,𝑛−𝑘−1 , reject 𝐻0 : 𝛽𝑖 = 0.

12.4 Autocorrelation

Autocorrelation is the correlation of a signal or variable with a lagged version of itself over
successive time intervals. It is widely used in time series analysis and signal processing.

Page 53 of 57
For a stationary discrete-time series 𝑋𝑡 , the autocorrelation function (ACF) at lag 𝑘is:
𝐶𝑜𝑣(𝑋𝑡 , 𝑋𝑡−𝑘 ) 𝐸[(𝑋𝑡 − 𝜇)(𝑋𝑡−𝑘 − 𝜇)]
𝜌𝑘 = =
𝑉𝑎𝑟(𝑋𝑡 ) 𝜎2

Where:

• 𝜇 = 𝐸[𝑋𝑡 ]is the mean

• 𝜎 2 = 𝑉𝑎𝑟(𝑋𝑡 )is the variance

• 𝑘 = 0,1,2, …is the lag

12.5 Properties of Autocorrelation

1. 𝜌0 = 1(autocorrelation at lag 0 is always 1)

2. ∣ 𝜌𝑘 ∣≤ 1

3. 𝜌−𝑘 = 𝜌𝑘 (symmetry)

Example

Suppose we have a signal:

𝑋 = [2,4,6,8,10]

The mean 𝜇 = 6, variance 𝜎 2 = 8.

For lag 𝑘 = 1:
5
1
𝐶𝑜𝑣(𝑋𝑡 , 𝑋𝑡−1 ) = ∑(𝑋𝑡 − 𝜇)(𝑋𝑡−1 − 𝜇)
𝑛−1
𝑡 =2
1
= [(4 − 6)(2 − 6) + (6 − 6)(4 − 6) + (8 − 6)(6 − 6) + (10 − 6)(8 − 6)]
4
1 1
= [(−2)(−4) + 0 ∗ (−2) + 2 ∗ 0 + 4 ∗ 2] = [8 + 0 + 0 + 8] = 4
4 4
4
𝜌1 = = 0.5
8

Page 54 of 57
12.6 Cross-Correlation

Cross-correlation measures the correlation between two different signals or time series 𝑋𝑡 and
𝑌𝑡 at different lags.
𝐶𝑜𝑣(𝑋𝑡 , 𝑌𝑡−𝑘 ) 𝐸[(𝑋𝑡 − 𝜇𝑋 )(𝑌𝑡−𝑘 − 𝜇𝑌 )]
𝑟𝑋𝑌 (𝑘) = =
𝜎𝑋 𝜎𝑌 𝜎𝑋 𝜎𝑌

• Useful for signal alignment, lag detection, and pattern matching.

• 𝑘 > 0: Y lags behind X

• 𝑘 < 0: Y leads X

Example

Let 𝑋 = [1,2,3,4], 𝑌 = [2,3,4,5], lag 𝑘 = 1:

4
1
𝐶𝑜𝑣(𝑋𝑡 , 𝑌𝑡 −1 ) = ∑(𝑋𝑡 − 𝑋ˉ )(𝑌𝑡−1 − 𝑌ˉ)
𝑛−1
𝑡 =2
𝑋ˉ = 2.5, 𝑌ˉ = 3
1
𝐶𝑜𝑣 = [(2 − 2.5)(2 − 3) + (3 − 2.5)(3 − 3) + (4 − 2.5)(4 − 3)]
3
1 1 2
= [(−0.5)(−1) + (0.5)(0) + (1.5)(1)] = [0.5 + 0 + 1.5] = ≈ 0.667
3 3 3

12.7 Covariance Functions

Covariance measures the linear dependence between two variables 𝑋and 𝑌:

𝑛
1
𝐶𝑜𝑣(𝑋, 𝑌) = 𝐸[(𝑋 − 𝜇𝑋 )(𝑌 − 𝜇𝑌 )] = ∑(𝑋𝑖 − 𝑋ˉ )(𝑌𝑖 − 𝑌ˉ)
𝑛
𝑖=1

• 𝐶𝑜𝑣(𝑋, 𝑌) > 0→ X and Y tend to increase together

• 𝐶𝑜𝑣(𝑋, 𝑌) < 0→ X increases while Y decreases

• 𝐶𝑜𝑣(𝑋, 𝑌) = 0→ no linear relationship

Page 55 of 57
Example

Let 𝑋 = [1,2,3], 𝑌 = [4,5,6]:

𝑋ˉ = 2, 𝑌ˉ = 5
1
𝐶𝑜𝑣(𝑋, 𝑌) = [(1 − 2)(4 − 5) + (2 − 2)(5 − 5) + (3 − 2)(6 − 5)]
3
1 2
= [1 + 0 + 1] = ≈ 0.667
3 3

12.8 Covariance Matrix

In multivariate analysis, we often work with vectors of variables 𝑋1 , 𝑋2 , . . . , 𝑋𝑛 .

The covariance matrix Σis an 𝑛 × 𝑛symmetric matrix:

𝑉𝑎𝑟(𝑋1 ) 𝐶𝑜𝑣(𝑋1 , 𝑋2 ) … 𝐶𝑜𝑣(𝑋1 , 𝑋𝑛 )
𝐶𝑜𝑣(𝑋2 , 𝑋1 ) 𝑉𝑎𝑟(𝑋2 ) … 𝐶𝑜𝑣(𝑋2 , 𝑋𝑛 )
Σ=[ ]
⋮ ⋮ ⋱ ⋮
𝐶𝑜𝑣(𝑋𝑛 , 𝑋1 ) 𝐶𝑜𝑣(𝑋𝑛 , 𝑋2 ) … 𝑉𝑎𝑟(𝑋𝑛 )

• Diagonal: variances of individual variables

• Off-diagonal: covariances between variables

12.9 Correlation Matrix

The correlation matrix 𝑅standardizes the covariance matrix:

1 𝜌12 … 𝜌1𝑛
𝜌21 1 … 𝜌2𝑛 𝐶𝑜𝑣(𝑋𝑖 , 𝑋𝑗 )
𝑅=[ ], 𝜌𝑖𝑗 =
⋮ ⋮ ⋱ ⋮ 𝜎𝑋𝑖 𝜎𝑋𝑗
𝜌𝑛1 𝜌𝑛2 … 1

• 𝑅values range from -1 to 1

• Measures strength and direction of linear relationships

Example

Suppose 𝑋 = [1,2,3], 𝑌 = [2,4,6], 𝑍 = [3,6,9]

𝐶𝑜𝑣(𝑋, 𝑌) = 1, 𝐶𝑜𝑣(𝑋, 𝑍) = 2, 𝐶𝑜𝑣(𝑌, 𝑍) = 4

Page 56 of 57
Variance: 𝑉𝑎𝑟(𝑋) = 1, 𝑉𝑎𝑟(𝑌) = 4, 𝑉𝑎𝑟(𝑍) = 8
1 2
1 = 0.5 ≈ 0.707
√1 ∗ 4 √1 ∗ 8
𝑅=[ 4 ]
0.5 1 = 0.707
√4 ∗ 8
0.707 0.707 1

Page 57 of 57