USTHB : 1st ING ST 2024/2025 Probability and Statistics
Chapter 2
Simple linear regression
When we observe two quantitative variables on the same individuals, we can
investigate a potential linear relationship (a connection if it exists) between these two
variables, and study it.
Regression provides an expression of this relationship in the form of a mathematical
function.
Correlation provides information about the strength of the relationship.
Example 1: We are interested in the height 𝑿 and weight 𝒀 of 𝟏𝟎 teenage athletes.
Individuals 𝟏 𝟐 𝟑 𝟒 𝟓 𝟔 𝟕 𝟖 9 10
Height 𝒙𝒊 𝟏𝟕𝟓 𝟏𝟖𝟎 𝟏𝟕𝟎 𝟏𝟕𝟔 𝟏𝟕𝟏 𝟏𝟔𝟕 𝟏𝟕𝟐 𝟏𝟕𝟓 𝟏𝟖𝟓 𝟏𝟔𝟓
Weight 𝒚𝒊 𝟕𝟎 𝟕𝟔 𝟔𝟕 𝟕𝟐 𝟔𝟖 𝟔𝟕 𝟔𝟗 𝟕𝟒 𝟖𝟖 𝟔𝟎
By plotting the data (𝒀 with respect to 𝑿), we can potentially observe a "trend". This
graphical representation is called a scatter graph, or more commonly a scatter plot or a
scatter chart, due to its shape. The shape of the scatter plot indicates the nature of the
relationship (if any) between the two statistical variables. When the scatter plot (𝒙𝒊 , 𝒚𝒊 ) is
approximately linear, one can consider expressing the relationship between 𝑿 and 𝒀 can be
expressed by a linear equation, such that 𝒀 = 𝒂𝑿 + 𝒃.
For example, the plot corresponding to the table is shown in Figure 2.1. The scatter
plot has a linear increasing appearance. It indicates an increasing linear relationship
between X and Y of the type 𝒀 = 𝒂𝑿 + 𝒃, (where, 𝒂 > 𝟎).
The strength of the linear relationship between 𝑿 and 𝒀 is determined (measured)
by the covariance, which is given by
𝒏 𝒏
𝟏 𝟏
𝑪𝒐𝒗 𝑿, 𝒀 = 𝒙 𝒊 − 𝑿 𝒚𝒊 − 𝒀 = 𝒙𝒊 𝒚𝒊 − 𝑿𝒀.
𝒏 𝒏
𝒊=𝟏 𝒊=𝟏
We have the following results:
𝑪𝒐𝒗 𝑿, 𝒀 > 0 ⟺ 𝑿 et 𝒀 vary in the same direction,
Chapter 1 Page 1
USTHB : 1st ING ST 2023/2024 Probability and Statistics
𝑪𝒐𝒗 𝑿, 𝒀 < 0 ⟺ 𝑿 et 𝒀 vary in the opposite direction,
𝑪𝒐𝒗 𝑿, 𝒀 = 𝑪𝒐𝒗 𝒀, 𝑿 𝒆𝒕 𝑪𝒐𝒗 𝑿, 𝑿 = 𝑽𝒂𝒓(𝑿).
Figure 2.1: Scatter plot of Y with respect to X
Linear regression consists of finding the best-fitting straight line through the points.
However, mathematically, there is only one line that solves the problem, it is the regression
line. Its equation is given by:
𝒀 = 𝒂𝑿 + 𝒃,
𝑪𝒐𝒗 𝑿,𝒀
where 𝒂 = and 𝒃 = 𝒀 − 𝒂𝑿.
𝑽𝒂𝒓(𝑿)
For our previous example, the calculations would yield to: 𝒂 = 𝟏. 𝟏𝟕𝟕𝟗 and 𝒃 =
−𝟏𝟑𝟑. 𝟑𝟖, so 𝒀 = 𝟏. 𝟏𝟕𝟕𝟗𝑿 − 𝟏𝟑𝟑. 𝟑𝟖, as shown in the following representation:
Chapter 1 Page 2
USTHB : 1st ING ST 2023/2024 Probability and Statistics
Figure 2.2: Regression line of Y with respect to X
The correlation coefficient is defined as:
𝑪𝒐𝒗 𝑿, 𝒀
𝑹= ,
𝝈𝑿 𝝈𝒀
Where 𝝈𝑿 = 𝑽𝒂𝒓(𝑿) and 𝝈𝒀 = 𝑽𝒂𝒓(𝒀).
Additionally, we have the following:
−𝟏 ≤ 𝑹 ≤ 𝟏 ⇔ 𝑹 ≤ 𝟏,
If 𝑹 > 0 ⇔ 𝑪𝒐𝒗 𝑿, 𝒀 > 0 , 𝑿 𝒆𝒕 𝒀 vary in the same direction,
If 𝑹 < 0 ⇔ 𝑪𝒐𝒗 𝑿, 𝒀 < 0 , 𝑿 𝒆𝒕 𝒀 vary in opposite directions,
When 𝑹 ≈ 𝟏,then we can conclude a strong linear relationship between the
variables X and Y of the form 𝒀 = 𝒂𝑿 + 𝒃.
The quality of linear regression is assessed through the coefficient 𝑹𝟐 . The closer it
is to 𝟏 (usually ≥ 𝟎. 𝟕𝟓 in practice), the better the linear regression fits the collected
data.
In our example, we find 𝑹𝟐 = 𝟎. 𝟗𝟎𝟓𝟔 indicating a very strong linear relationship
between weight and height. In this case, we can make ‘prediction’, for instance, calculating
that the weight of a 𝟏𝟗𝟓 cm tall teenager would be 𝟗𝟔. 𝟑𝟏 kg.
Chapter 1 Page 3