Generalized Inverses in Regression Models
Janet van Niekerk
Janet van Niekerk Generalized Inverses in Regression 1 / 17
Motivation
Consider the linear model
y = X β + ε, ε ∼ (0, σ 2 I ).
The normal equations:
X ⊤ X β = X ⊤ y.
If X ⊤ X is invertible:
β̂ = (X ⊤ X )−1 X ⊤ y.
What if X ⊤ X is singular?
Janet van Niekerk Generalized Inverses in Regression 2 / 17
Motivation
Consider the linear model
y = X β + ε, ε ∼ (0, σ 2 I ).
The normal equations:
X ⊤ X β = X ⊤ y.
If X ⊤ X is invertible:
β̂ = (X ⊤ X )−1 X ⊤ y.
What if X ⊤ X is singular?
Janet van Niekerk Generalized Inverses in Regression 2 / 17
Motivation
Rank deficiency arises when:
Perfect multicollinearity
p>n
Over-parameterised models
Constraints or identifiability issues
Janet van Niekerk Generalized Inverses in Regression 3 / 17
Definition: Generalized Inverse
A matrix G is a generalized inverse of A if
AGA = A.
Janet van Niekerk Generalized Inverses in Regression 4 / 17
Definition: Generalized Inverse
A matrix G is a generalized inverse of A if
AGA = A.
Interpretation:
Preserves the range of A
Does not require invertibility
Not necessarily unique
Janet van Niekerk Generalized Inverses in Regression 4 / 17
Moore–Penrose Pseudoinverse
The Moore–Penrose inverse A+ satisfies:
1. AA+ A = A
2. A+ AA+ = A+
3. (AA+ )⊤ = AA+
4. (A+ A)⊤ = A+ A
Janet van Niekerk Generalized Inverses in Regression 5 / 17
Moore–Penrose Pseudoinverse
The Moore–Penrose inverse A+ satisfies:
1. AA+ A = A
2. A+ AA+ = A+
3. (AA+ )⊤ = AA+
4. (A+ A)⊤ = A+ A
Properties:
Unique
Produces minimum-norm solution
Orthogonal projections
Janet van Niekerk Generalized Inverses in Regression 5 / 17
SVD Representation
Singular value decomposition:
X = UΣV ⊤ .
Then
X + = V Σ+ U ⊤ ,
where
+ 1
Σ = diag for σi > 0.
σi
Janet van Niekerk Generalized Inverses in Regression 6 / 17
SVD Representation
Singular value decomposition:
X = UΣV ⊤ .
Then
X + = V Σ+ U ⊤ ,
where
+ 1
Σ = diag for σi > 0.
σi
Key idea:
We invert singular values, not matrices.
Janet van Niekerk Generalized Inverses in Regression 6 / 17
Cases
Overdetermined system
Underdetermined system
Collinearity
Intrinsic models
Janet van Niekerk Generalized Inverses in Regression 7 / 17
Overdetermined system
More equations than unknowns
Both solutions coincide.
We never computed (A⊤ A)−1
The SVD explicitly inverts singular values.
The solution equals the projection of b onto the column space of A.
(Next week)
Janet van Niekerk Generalized Inverses in Regression 8 / 17
Underdetermined system
Infinite solutions exist.
Moore–Penrose selects the minimum norm solution.
Janet van Niekerk Generalized Inverses in Regression 9 / 17
Collinearity
Coefficients are not unique.
Fitted values ARE unique.
Inference lives in column space.
Janet van Niekerk Generalized Inverses in Regression 10 / 17
Intrinsic models
Q is singular.
Corresponds to intrinsic Gaussian model.
Precision matrix defines differences, not absolute level.
Generalized inverse defines covariance up to constant.
Singular precision = invariance under translation.
Janet van Niekerk Generalized Inverses in Regression 11 / 17
Discussion
We never invert full matrices in practice.
We invert singular values.
Rank deficiency encodes invariance.
Pseudoinverse chooses minimal norm.
Ridge is stabilized inversion.
Statistical identifiability is geometric.
Janet van Niekerk Generalized Inverses in Regression 12 / 17
Glossary
What is the Null Space?
Let A be a matrix. The null space of A is the set of all vectors x such that
Ax = 0.
It is denoted by
N (A) = {x : Ax = 0}.
Intuitive Meaning
Think of a matrix A as a machine.
You put a vector x into the machine, and it produces an output Ax.
The null space consists of all inputs that the machine completely erases,
meaning they produce zero as output.
The null space is the collection of all directions that a matrix
cannot see.
Janet van Niekerk Generalized Inverses in Regression 13 / 17
Glossary
QR Decomposition of a Matrix
Let A be an n × p matrix with n ≥ p.
The QR decomposition of A is a factorization of the form
A = QR,
where
Q is an n × p matrix with orthonormal columns,
Q ⊤ Q = Ip ,
R is a p × p upper triangular matrix.
Janet van Niekerk Generalized Inverses in Regression 14 / 17
Geometric Interpretation
The QR decomposition expresses the columns of A as orthogonal
directions.
The columns of Q form an orthonormal basis for the column space of
A.
The matrix R contains the coordinates of the original columns of A
relative to this orthonormal basis.
Thus, QR separates:
Direction (contained in Q),
Scaling and linear combination structure (contained in R).
Janet van Niekerk Generalized Inverses in Regression 15 / 17
Rank
Let A be an m × n matrix.
The rank of A, denoted rank(A), is the number of linearly independent
columns (or rows) of A.
Equivalently, the rank is:
The dimension of the column space of A,
The dimension of the row space of A,
The maximum number of linearly independent columns (or rows).
It always holds that
rank(A) ≤ min(m, n).
Full Rank
If A is m × n and rank(A) = n (with m ≥ n), then A has full column
rank.
If A is n × n and rank(A) = n, then A is invertible.
Janet van Niekerk Generalized Inverses in Regression 16 / 17
Singular matrix
A square matrix A of size n × n is called singular if it is not invertible.
This happens if and only if
rank(A) < n.
Equivalently, A is singular if any of the following hold:
det(A) = 0,
The columns of A are linearly dependent,
There exists a nonzero vector x such that Ax = 0,
The null space N (A) contains nonzero vectors.
Janet van Niekerk Generalized Inverses in Regression 17 / 17