lOMoARcPSD|57238726
1 Matrices and Linear Algebra
1.1 Partitioning - MM1.1
Matrices can be partitioned into smaller (rectangular) submatrices and manip-
ulations can be done using the subpieces provided that the dimensions in the
partitions permit any matrix multiplications.
1.2 Transposes - MM 1.2
1. (X T )T = X.
2. (XY )T = Y T X T =
6 XT Y T !
3. A matrix X is symmetric if and only if X T = X.
1.3 Inverses - MM 1.3
1. X is singular if and only if it has a 0 determinant, |X| = 0.
2. If X is nonsingular, then X −1 is nonsingular and (X −1 )−1 = X.
3. If X, Y are nonsingular, thenXY is nonsingular and (XY )−1 = Y −1 X −1 =
6
X −1 Y −1 .
4. If X is nonsingular, then X T is nonsingular and (X T )−1 = (X −1 )T .
1.4 Orthogonality - 1.3
X is an orthogonal matrix if and only if the columns (or rows) of X form an
orthonormal set.
1.5 Eigenthings - MM 1.4
1. The eigenvalues of a matrix by solve the characteristic equation (this is a
polynomial in λ) |A − λI| = 0.
2. Find one eigenvector for each eigenvalue as a non-zero solution, x, of the
homogeneous system ()A − λI)x = 0.
3. If A is (real and) symmetric, then its eigenvalues are all real, and its
eigenvectors are orthogonal.
4. If P is an orthogonal matrix of the same size as A, then the eigenvalues
of P T AP are the same as the eigenvalues of A.
5. Let A be a symmetric k × k matrix. Then an orthogonal matrix P con-
sisting of eigenvectors exists such that
λ1 0 ... 0
0 λ2 . . . 0
P T AP = . ,
.. ..
.. . .
0 0 ... λk
where λi , i = 1, 2, . . . , k, are the eigenvalues of A.
1
lOMoARcPSD|57238726
1.6 Linear Independence and rank - MM 1.4
1. Vectors x1 , x2 , · · · , xk are linearly depdendent if there exists some num-
bers a1 , a2 , . . . , ak , which are not all zero, such that
a1 x1 + a2 x2 + . . . + ak xk = 0.
2. We define the rank of a matrix is the the greatest number of linearly
independent columns
3. For any matrix X we have r(X) = r(X T ) = r(X T X).
4. If X is k × k, then X is nonsingular if and only if r(X) = k.
5. If X is n × k, P is n × n and nonsingular, and Q is k × k and nonsingular,
then r(X) = r(P X) = r(XQ).
6. The rank of a diagonal matrix is equal to the number of nonzero diagonal
entries in the matrix.
7. r(XY ) ≤ r(X), r(Y ).
1.7 Idempotence - MM 1.5
A square matrix A is idempotent if A2 = A
1.8 Trace MM - 1.5
1. The trace of a square k × k matrix X, denoted by tr(X), is the sum of its
Pk
diagonal entries: tr(X) = i=1 xii .
2. If c is a scalar, tr(cX) = c tr(X).
3. tr(X ± Y ) = tr(X) ± tr(Y ).
4. If XY and Y X both exist, tr(XY ) = tr(Y X).
1.9 Theorems
1. The eigenvalues of idempotent matrices are always either 0 or 1.
2. If A is a symmetric and idempotent matrix, r(A) = tr(A).
3. Let A1 , A2 , . . . , Am be a collection of symmetric k × k matrices. Then the
following are equivalent:
• There exists an orthogonal matrix P such that P T Ai P is diagonal
for all i = 1, 2, . . . , m;
• Ai Aj = Aj Ai for every pair i, j = 1, 2, . . . , m.
4. Let A1 , A2 , . . . , Am be a collection of symmetric k × k matrices. Then any
two of the following conditions implies the third:
• All Ai , i = 1, 2, . . . , m are idempotent;
Pm
• i=1 Ai is idempotent;
2
lOMoARcPSD|57238726
• Ai Aj = 0 for i =
6 j.
5. Let A1 , A2 , . . . , Am be a collection of symmetricPk × k matrices.
Pm If the
m
conditions in the previous result are true, then r ( i=1 Ai ) = i=1 r (Ai ) .
1.10 Quadratic Forms - MM 2.1
1. Let A be a k×k matrix and y a k×1 vector containing variables, q = yT Ay
is called a quadratic form in y, and A is called the matrix of the quadratic
form.
Pk Pk
2. q = i=1 j=1 aij yi yj .
3. If yT Ay > 0 for all y =6 0, then we say that the quadratic form yT Ay is
positive definite; we also say that the matrix A is positive definite.
4. If yT Ay ≥ 0 for all y, then we say that the quadratic form yT Ay is positive
semi-definite; we also say that the matrix A is positive semi-definite.
5. A symmetric matrix A is positive (semi-)definite if and only if its eigen-
values are all (non-negative) strictly positive.
1.11 Differentiation by vectors - MM 2.2
1. Suppose we have a vector of variables y = (y1 , y2 , . . . , yk )T , and some
scalar function of them: z =f (y). We define the derivative of z with
∂z/∂y1
∂z/∂y2
respect to y as follows: ∂∂zy = .
..
.
∂z/∂yk
∂z
2. If z = aT y where a is a vector of constants, then ∂y = a.
∂z
3. If z = yT y, then ∂y = 2y.
∂z
4. If z = yT Ay, then ∂y = Ay + AT y. In particular, if A is symmetric, then
∂z
∂y = 2Ay.