Methods
Least Squares
Ibrahim
Kambaima
2026
Overview
• Least Squares
• Geometry of Least Squares
• Projections
• Algebraic Properties of OLS
• Partitioned Regression
Least Squares
The matrix form of the linear model
• We write the linear model in matrix form as
𝒚𝒚 = 𝑿𝑿𝜷𝜷 + 𝜺𝜺
• By picking any coefficient 𝜷𝜷 � 1 we can produce a (vector) of fitted
values
� = 𝑿𝑿𝜷𝜷
𝒚𝒚 �1
• Corresponding to those fitted values we get a vector of residuals
�
𝒆𝒆 = 𝒚𝒚 − 𝒚𝒚
• The Residual Sum of Squares is
𝒆𝒆′ 𝒆𝒆 = 𝑒𝑒12 + 𝑒𝑒22 + ⋯ + 𝑒𝑒𝑛𝑛2
Ordinary Least Squares
• The Ordinary Least Squares criterion is to pick that coefficient vector
� that minimises the Residual Sum of Squares
𝜷𝜷
• We can show by vector differentiation that the solution to the OLS
problem is given by
� = 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝒚𝒚
𝜷𝜷
Example 1
• Model is
𝑦𝑦𝑖𝑖 = 𝑥𝑥𝑖𝑖 𝛽𝛽 + 𝜀𝜀𝑖𝑖
• Note: no intercept, only one x variable and only one parameter
• Two observations:
1 4
𝒚𝒚 = , 𝒙𝒙 =
4 3
• OLS solution is
𝛽𝛽̂ = 𝒙𝒙′ 𝒙𝒙 −1 𝒙𝒙′ 𝒚𝒚
4 −1 1
= 4 3 4 3
3 4
1
= ∗ 16 = 0.64
25
Example 2
• Model is
𝑦𝑦𝑖𝑖 = 𝛽𝛽1 + 𝑥𝑥𝑖𝑖 𝛽𝛽2 + 𝜀𝜀𝑖𝑖
• Intercept plus one x-variable. Two parameters.
• Three observations:
3 2
𝒚𝒚 = 1 , 𝒙𝒙 = 1
2 0
• Because of the intercept, the data matrix is
1 2
𝑿𝑿 = 1 1
1 0
Example 2 OLS solution
• The OLS solution is
� = 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝒚𝒚
𝜷𝜷
1 2 −1 3
1 1 1 1 1 1
= 1 1 1
2 1 0 2 1 0
1 0 −1 2
3 3 6
=
3 5 7
1 5 −3 6
=
6 −3 3 7
1.5
=
0.5
• i.e. 𝛽𝛽̂1 = 1.5 and 𝛽𝛽̂2 = 0.5
Geometry of Least Squares
Example 1 again
8
• Data is given by:
1 4
6
𝒚𝒚 = , 𝒙𝒙 =
4 3 (1,4)
observation 2
• Here 𝒚𝒚 and 𝒙𝒙 are just (4,3)
4
vectors and they can be
plotted like any other two-
2
dimensional vectors in the
plane
0
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2)
Show vectors as
directed arrows
8
• Data is given by:
1 4
6
𝒚𝒚 = , 𝒙𝒙 =
4 3 (1,4)
observation 2
• We want to pick 𝛽𝛽̂ so that (4,3)
4
𝒙𝒙𝛽𝛽̂ is as close to 𝒚𝒚 as
possible
2
• Different values of 𝛽𝛽̂ will
0
give different fitted values
and different residual
-2
vectors -2 0 2 4
observation 1
6 8
(y1,y2) (x1,x2)
Set 𝛽𝛽̂ = 1.5
8
6
𝒙𝒙𝛽𝛽̂ =
4.5
6
(6,4.5)
• Fitted value is blue arrow (1,4)
observation 2
4
• Residual is
1 6 −5
𝒆𝒆 = − =
2
4 4.5 −0.5
• That vector is represented
0
by the orange arrow
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (x1b1,x2b1)
Set 𝛽𝛽̂ = 2
8
̂ 8 (8,6)
𝒙𝒙𝛽𝛽 =
6
6
• Fitted value is blue arrow (1,4)
observation 2
4
• Residual is
1 8 −7
𝒆𝒆 = − =
2
4 6 −2
• That vector is represented
0
by the orange arrow
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (x1b2,x2b2)
Set 𝛽𝛽̂ = 0.25
8
1
𝒙𝒙𝛽𝛽̂ =
0.75
6
• Fitted value is blue arrow (1,4)
observation 2
4
• Residual is
1 1 0
𝒆𝒆 = − =
2
4 0.75 3.25
(1,0.75)
• That vector is represented
0
by the orange arrow
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (x1b3,x2b3)
Set 𝛽𝛽̂ = −0.25
8
−1
𝒙𝒙𝛽𝛽̂ =
−0.75
6
• Fitted value is blue arrow (1,4)
observation 2
4
• Residual is
1 −1 2
𝒆𝒆 = − =
2
4 −0.75 4.75
• That vector is represented
0
by the orange arrow
(-1,-0.75)
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (x1b4,x2b4)
Set 𝛽𝛽̂ = 0.64
8
̂ 2.56
𝒙𝒙𝛽𝛽 =
1.92
6
• Fitted value is blue arrow (1,4)
observation 2
4
• Residual is
1 2.56 −1.56
𝒆𝒆 = − =
2
4 1.92 2.08 (2.56,1.92)
• That vector is represented
0
by the orange arrow
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (yh1,yh2)
All possible values of 𝛽𝛽̂
8
• As we change 𝛽𝛽̂ we trace out a
line through 𝒙𝒙
6
• Corresponding to each point
along the line there is a (1,4)
corresponding residual vector
observation 2
4
• OLS criterion is to pick that 𝛽𝛽̂ that (4,3)
minimises
2
𝑒𝑒12 + 𝑒𝑒22
• But note that in this space that is
just the square of the length of
0
the residual vector
• i.e. the OLS criterion tells us to
-2
pick the point on the line that is -2 0 2 4 6 8
as close as possible to 𝒚𝒚 observation 1
(y1,y2) (x1,x2) (yh1,yh2)
OLS solution 𝛽𝛽̂ = 0.64
8
• That is the point where the
residual vector is at right
6
angles to the line through 𝒙𝒙 (1,4)
• Two vectors are at right angles
observation 2
4
when their dot product (inner
product) is zero
2
• The dot product is the sum of (2.56,1.92)
the element by element
product, i.e.
0
𝒂𝒂. 𝒃𝒃 = 𝑎𝑎1 𝑏𝑏1 + ⋯ + 𝑎𝑎𝑛𝑛 𝑏𝑏𝑛𝑛
• This can also be written as
-2
𝒂𝒂′ 𝒃𝒃 -2 0 2 4 6 8
• i.e. we want 𝒙𝒙′ 𝒆𝒆 =0 observation 1
(y1,y2) (x1,x2) (yh1,yh2)
OLS solution 𝛽𝛽̂ = 0.64
8
• But 𝒙𝒙′ 𝒆𝒆 = 𝒙𝒙𝒙 𝒚𝒚 − 𝒙𝒙𝛽𝛽̂
• Setting this equal to zero we
6
get the condition (1,4)
𝒙𝒙′ 𝒚𝒚 − 𝒙𝒙′ 𝒙𝒙𝛽𝛽̂ = 0
observation 2
4
𝒙𝒙′ 𝒚𝒚 = 𝒙𝒙′ 𝒙𝒙𝛽𝛽̂
i.e.
2
𝛽𝛽̂ = 𝒙𝒙′ 𝒙𝒙 −1 𝒙𝒙′ 𝒚𝒚 (2.56,1.92)
• This is the same equation that
0
we got via differentiation
• This procedure generalises
-2
• Although it is hard to visualise -2 0 2 4 6 8
when we get beyond two observation 1
observations
(y1,y2) (x1,x2) (yh1,yh2)
Example 2
• Model is
𝑦𝑦𝑖𝑖 = 𝛽𝛽1 + 𝑥𝑥𝑖𝑖 𝛽𝛽2 + 𝜀𝜀𝑖𝑖
• 3 observations
3 2
𝒚𝒚 = 1 , 𝒙𝒙 = 1
2 0
• The fitted values are given by the vector equation
1 2
1 𝛽𝛽̂1 + 1 𝛽𝛽̂2
3 1 2 1 0
• The vectors 1 , 1 and 1 can all be plotted in the “usual” three dimensional vector
space. 2 1 0
1 2
• As we pick different values for 𝛽𝛽̂1 and 𝛽𝛽̂2 , the sum 1 𝛽𝛽̂1 + 1 𝛽𝛽̂2 maps out a plane
1 0
(3,1,2)
(2.5,2,1.5)
(1,1,1)
Observation 3
(2,1,0)
Observation 2
Observation 1
(3,1,2)
(2.5,2,1.5)
(1,1,1)
(2,1,0)
Observation 3
Observation 1
Observation 2
(3,1,2)
(2.5,2,1.5)
(1,1,1)
(2,1,0)
Observation 3
Observation 2 Observation 1
OLS solution
• Again we pick that vector in the plane that is “closest” to 𝒚𝒚
• That is again equivalent to dropping a perpendicular from 𝒚𝒚 into that space
1
• It means that the residual vector 𝒆𝒆 has to be at right angles to both the 1 and
2 1
1 vectors
0
• The condition that 1 1 1 𝒆𝒆 = 0 and 2 1 0 𝒆𝒆 = 0 can be compactly
written as
𝑿𝑿′ 𝒆𝒆 = 𝟎𝟎
i.e.
𝑿𝑿′ 𝒚𝒚 − 𝑿𝑿𝜷𝜷 � = 𝟎𝟎
�
𝑿𝑿′ 𝒚𝒚 = 𝑿𝑿′ 𝑿𝑿𝜷𝜷
� = 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝒚𝒚
𝜷𝜷
This is not accidental
• When we try to make the length of the residual vector as small as
possible, we are trying to minimise
𝒆𝒆 = 𝑒𝑒12 + 𝑒𝑒22 + ⋯ + 𝑒𝑒𝑛𝑛2
• So the idea of minimising the residual sum of squares arises naturally
if we think of 𝒚𝒚 as a vector in ℝ𝑛𝑛 and the vector of fitted values 𝒚𝒚
�=
� as likewise a vector in ℝ𝑛𝑛 and our problem is to pick 𝜷𝜷
𝑿𝑿𝜷𝜷 � so as to
minimise the distance between them, i.e. minimise
�
𝒚𝒚 − 𝒚𝒚
• So the OLS solution is to pick 𝜷𝜷� so as to ensure that 𝑿𝑿′ 𝒆𝒆 = 𝟎𝟎
Projections
Dropping a perpendicular
8
• The process of dropping a
perpendicular from any point 𝒚𝒚 into
6
the space “spanned” by the
explanatory variables 𝒙𝒙1 , 𝒙𝒙2 , … , 𝒙𝒙𝑘𝑘 is
an example of a projection
observation 2
4
• The equation for any fitted value is
given by
2
� = 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝒚𝒚
� = 𝑿𝑿𝜷𝜷
𝒚𝒚
• The matrix 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ is called the
0
P matrix (or hat matrix)
• It represents the process that
“converts” 𝒚𝒚 values into 𝒚𝒚
�
-2
• Observe that it only depends on 𝑿𝑿 -2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (yh1,yh2)
Properties of the P matrix
8
• If we “project” a point that is
already inside the space, it just
6
stays where it is
• Mathematically 𝑷𝑷𝑷𝑷 = 𝑷𝑷
observation 2
4
• Check:
𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′
2
= 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′
= 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑰𝑰𝑿𝑿′
0
= 𝑷𝑷
• We say that the P matrix is
-2
-2 0 2 4 6 8
idempotent observation 1
(y1,y2) (x1,x2) (yh1,yh2)
The same process also creates residuals
8
• The residual vector is given by
� = 𝒚𝒚 − 𝑷𝑷𝑷𝑷
𝒆𝒆 = 𝒚𝒚 − 𝒚𝒚
6
= 𝑰𝑰 − 𝑷𝑷 𝒚𝒚
observation 2
• The matrix 𝑴𝑴 = 𝑰𝑰 − 𝑷𝑷 is another
4
important matrix
• It is the “Residual Maker”
2
𝒆𝒆 = 𝑴𝑴𝑴𝑴
• It is also a type of projection – onto
0
the space at right angles to the space
spanned by 𝒙𝒙1 , 𝒙𝒙2 , … , 𝒙𝒙𝑘𝑘
-2
• That space is called the orthogonal -2 0 2 4 6 8
complement observation 1
(y1,y2) (x1,x2) (yh1,yh2)
Creating the residuals
8
• We can show this by plotting the
residual vectors in the usual way,
6
i.e. with origin at zero (1,4)
observation 2
4
• e.g. in example 1 the residual
vector was
2
−1.56 (2.56,1.92)
2.08
0
-2
-2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (yh1,yh2) (e1,e2)
Properties of the M matrix
8
• It is evident that again if the point 𝒚𝒚
has been moved into this space, that
6
it won’t move a second time
observation 2
• i.e. the M matrix is also idempotent
4
𝑴𝑴𝑴𝑴 = 𝑰𝑰 − 𝑷𝑷 𝑰𝑰 − 𝑷𝑷
= 𝑰𝑰 − 𝑷𝑷 − 𝑷𝑷 + 𝑷𝑷2
2
• We know that P is idempotent
0
• So
𝑴𝑴𝑴𝑴 = 𝑰𝑰 − 𝑷𝑷 − 𝑷𝑷 + 𝑷𝑷
-2
= 𝑰𝑰 − 𝑷𝑷 -2 0 2 4 6 8
= 𝑴𝑴 observation 1
(y1,y2) (x1,x2) (yh1,yh2) (e1,e2)
Relationship between M and P
8
• By definition
𝑴𝑴 + 𝑷𝑷 = 𝑰𝑰
6
• The matrices “annihilate” each other
observation 2
𝑴𝑴𝑴𝑴 = 𝑷𝑷𝑷𝑷 = 𝟎𝟎
4
• Interpretation:
2
• 𝑷𝑷𝑷𝑷 are fitted values and 𝑴𝑴 makes
residuals – so 𝑴𝑴𝑴𝑴𝑴𝑴 are the residuals we
get when regressing fitted values on X.
0
• 𝑴𝑴𝑴𝑴 are residuals so 𝑷𝑷𝑷𝑷𝑷𝑷 are the fitted
values we get when regressing residuals
-2
on X – all zeros! -2 0 2 4 6 8
observation 1
(y1,y2) (x1,x2) (yh1,yh2) (e1,e2)
Further properties
• M also annihilates the X matrix
𝑴𝑴𝑴𝑴 = 𝑰𝑰 − 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿𝑿 𝑿𝑿
= 𝑿𝑿 − 𝑿𝑿 𝑿𝑿′ 𝑿𝑿 −1 𝑿𝑿′ 𝑿𝑿
= 𝑿𝑿 − 𝑿𝑿 = 𝟎𝟎
• Implication for the linear model:
𝒆𝒆 = 𝑴𝑴𝑴𝑴 = 𝑴𝑴 𝑿𝑿𝜷𝜷 + 𝜺𝜺 = 𝑴𝑴𝑴𝑴𝜷𝜷 + 𝑴𝑴𝑴𝑴 = 𝑴𝑴𝑴𝑴
𝒆𝒆 = 𝑴𝑴𝜺𝜺
Algebraic properties of OLS
Properties that follow from nature of OLS
solution:
1. The OLS estimator is a linear function of the dependent variable
2. The fitted values are a linear function of the dependent variable
3. The residuals are orthogonal to the explanatory variables
• If there is an intercept in the model then “orthogonal to” is the same as
“uncorrelated with”
4. The residuals are orthogonal to the fitted values
5. The residuals are a linear function of the errors 𝜺𝜺
6. The average of the residuals is zero if there is an intercept in the model
7. The average of the fitted values is 𝑦𝑦� if there is an intercept in the model
8. The OLS solution is invariant to linear (nonsingular) transformations of
the data
• The fitted values are not affected if the explanatory variables are transformed
• If the dependent variable is transformed, then the fitted values are transformed in
the same way, i.e. the interpretation of the answer does not change