Basic Maths

[Link]
Associate Dean/Research
shanthi@[Link]
School of Computing
SASTRA Deemed to be University
Linear Algebra
Scalars,Vectors,Matrices and Tensors
Multiplying Matrices and Vectors
Identity and Inverse Matrices
Linear Dependence and Span
Norms
Special Kinds of Matrices and Vectors
Eigen decomposition
Singular Value Decomposition
The Moore-Penrose Pseudo inverse
The Trace Operator
The Determinant
Example: Principal Components Analysis
[Link] SASTRA 1 / 36
Scalars: A scalar is just a single number, in contrast to most of
the other objects studied in linear algebra, which are usually
arrays of multiple numbers.
Vectors: A vector is an array of numbers. The numbers are
arranged in order. We can identify each individual number by its
index in that ordering. Typically we give vectors lowercase
names in bold typeface, such as x.
Matrices: A matrix is a 2-D array of numbers, so each element is
identified by two indices instead of just one.
Tensors: In some cases we will need an array with more than two
axes. In the general case, an array of numbers arranged on a
regular grid with a variable number of axes is known as a tensor.
We denote a tensor named “A” with this typeface: A. We
identify the element of A at coordinates (i, j, k) by writing Ai,j,k.

[Link] SASTRA 2 / 36
[Link] SASTRA 3 / 36
Operators

A scalar can be thought of as a matrix with only a single entry.

From this, we can see that a scalar is its own transpose: a = aT .
We can add matrices to each other, as long as they have the
same shape, just by adding their corresponding elements: C = A
+ B where Ci,j = Ai,j + Bi,j.
We can also add a scalar to a matrix or multiply a matrix by a
scalar, just by performing that operation on each element of a
matrix: D = a · B + c where Di,j =a·Bi,j +c.

[Link] SASTRA 4 / 36
If A is of shape m × n and B is of shape n × p, then C is of
shape m × p. We can write the matrix product just by placing
two or more matrices together, for example, C = AB.
Note that the standard product of two matrices is not just a
matrix containing the product of the individual elements. Such
an operation exists and is called the element-wise
J product, or
Hadamard product, and is denoted as A B.
Singular:A square matrix with linearly dependent columns is
known as singular.
If A is not square or is square but singular, solving the equation
is still possible, but we cannot use the method of matrix
inversion to find the solution.

[Link] SASTRA 5 / 36
Norm:Measure the size of vectors using a function called a norm.
Formally, the Lp norm
L2 Norm: With p = 2, is known as the Euclidean norm, which is
simply the Euclidean distance from the origin to the point
identified by x.
L∞ norm: Also known as the max norm. This norm simplifies to
the absolute value of the element with the largest magnitude in
the vector.

[Link] SASTRA 6 / 36
Examples

[Link] SASTRA 7 / 36
Examples

[Link] SASTRA 8 / 36
Orthogonal: A vector x and a vector y are orthogonal to each
other if x T y = 0. If both vectors have nonzero norm, this means
that they are at a 90 degree angle to each other. In R n , at most
n vectors may be mutually orthogonal with nonzero norm.
Orthonormal: If the vectors not only are orthogonal but also
have unit norm, we call them orthonormal.
Orthogonal Matrix: An orthogonal matrix is a square matrix
whose rows are mutually orthonormal and whose columns are
mutually orthonormal.
AT A = AAT = I
A−1 = AT

[Link] SASTRA 9 / 36
Orthogonal Matrix Properties

All identity matrices are an orthogonal matrix.

The product of two orthogonal matrices is also an orthogonal
matrix.
The collection of the orthogonal matrix of order n x n, in a
group, is called an orthogonal group and is denoted by ‘O’.
The transpose of the orthogonal matrix is also orthogonal.
The inverse of the orthogonal matrix, which is A-1 is also an
orthogonal matrix.
The determinant of the orthogonal matrix has a value of 1.
The eigenvalues of the orthogonal matrix also have a value as 1,
and its eigenvectors would also be orthogonal and real.

[Link] SASTRA 10 / 36
Eigen Decomposition

Decompose a matrix into a set of eigenvectors and eigenvalues.

An eigenvector of a square matrix A is a nonzero vector v such
that multiplication by A alters only the scale of v: Av = λv .
The scalar λ is known as the eigenvalue corresponding to this
eigenvector. (One can also find a left eigenvector such that
v T A = λv T , but we are usually concerned with right
eigenvectors.)
The eigendecomposition of A is then given by A = Vdiag (λ)V 1 .

[Link] SASTRA 11 / 36
[Link] SASTRA 12 / 36
[Link] SASTRA 13 / 36
[Link] SASTRA 14 / 36
[Link] SASTRA 15 / 36
[Link] SASTRA 16 / 36
[Link] SASTRA 17 / 36
[Link] SASTRA 18 / 36
[Link] SASTRA 19 / 36
[Link] SASTRA 20 / 36
[Link] SASTRA 21 / 36
SVD
The singular value decomposition (SVD) provides another way
to factorize a matrix, into singular vectors and singular values.
Eigendecomposition involves analyzing a matrix A to discover a
matrix V of eigenvectors and a vector of eigenvalues λ such that
we can rewrite A A = Vdiag (λ)V 1 .
A as a product of three matrices: A = UDV T
Suppose that A is an m × n matrix. Then U is defined to be an
m × mmatrix D to be an m × n matrix, and V to be an n × n
matrix.
Each of these matrices is defined to have a special structure.
The matrices U and V are both defined to be orthogonal
matrices.
The matrix D is defined to be a diagonal matrix. Note that D is
not necessarily square.
[Link] SASTRA 22 / 36
[Link] SASTRA 23 / 36
Trace operator

The trace operatorPgives the sum of all the diagonal entries of a

matrix: Tr (A) = i Ai,i .
Trace operator provides an alternative
√ way of writing the
Frobenius norm of a matrix: AF = AAT
Trace operator is invariant to the transpose operator:
Tr (A) = Tr (AT ).
The trace of a square matrix composed of many factors is also
invariant to moving the last factor into the first position, if the
shapes of the corresponding matrices allow the resulting product
to be defined: or more generally,
Tr (ABC ) = Tr (CAB) = Tr (BCA)

[Link] SASTRA 24 / 36
[Link] SASTRA 25 / 36
Determinant

The determinant of a square matrix, denoted det(A), is a

function that maps matrices to real scalars.
The determinant is equal to the product of all the eigenvalues of
the matrix.
The absolute value of the determinant can be thought of as a
measure of how much multiplication by the matrix expands or
contracts space.
If the determinant is 0, then space is contracted completely
along at least one dimension, causing it to lose all its volume.
If the determinant is 1, then the transformation preserves
volume.

[Link] SASTRA 26 / 36
[Link] SASTRA 27 / 36
[Link] SASTRA 28 / 36
[Link] SASTRA 29 / 36
[Link] SASTRA 30 / 36
[Link] SASTRA 31 / 36
[Link] SASTRA 32 / 36
[Link] SASTRA 33 / 36
[Link] SASTRA 34 / 36
[Link] SASTRA 35 / 36
Probability Theory Information Theory
RandomVariables
ProbabilityDistributions
MarginalProbability
ConditionalProbability
TheChainRuleofConditionalProbabilities
Independence and Conditional Independence
Expectation,VarianceandCovariance
CommonProbabilityDistributions
UsefulPropertiesofCommonFunctions
Bayes’Rule
TechnicalDetailsofContinuousVariables
InformationTheory
StructuredProbabilisticModels
[Link] SASTRA 36 / 36