Machine learning

Lecture 4: Math review

Professor Shaker El-Sappagh
[Link]@[Link]
Spring 2024
Outline
1. Linear algebra
2. Calculus
3. Probability
4. Optimization
1. Linear algebra
Data
Data – vector - matrix
Vectors

• Vector definition
▪ Physics: vector is an arrow with direction and length.
▪ Computer science: vector is a one-dimensional array of ordered real-valued scalars. Vector starts from the origin.
▪ Mathematics: vector is a quantity possessing both magnitude and direction, represented by an arrow indicating the
direction, and the length of which is proportional to the magnitude. The two basic operations on vector are addition and
scaler multiplication (scaling).
• Vectors are written in column form or in row form
▪ Denoted by bold-font lower-case letters
1
7 𝐱= 1 7 0 1 𝑇
𝐱=
0
1

• For a general form vector with 𝑛 elements, the vector lies in the 𝑛-dimensional space 𝐱 ∈ ℝ𝑛
𝑥1
𝑥
𝐱= 2
⋮
𝑥𝑛
Geometry of Vectors
Vectors

• First interpretation of a vector: point in space

▪ E.g., in 2D we can visualize the data points with respect to a coordinate
origin

• Second interpretation of a vector: direction in space

▪ E.g., the vector 𝐯 = 3, 2 𝑇 has a direction of 3 steps to the right and 2
steps up
▪ The notation 𝐯 is sometimes used to indicate that the vectors have a
direction
▪ All vectors in the figure have the same direction

• Vector addition
▪ We add the coordinates, and follow the directions given by the two
vectors that are added
Geometry of Vectors

• The geometric interpretation of vectors as points in space allow us to consider a

training set of input examples in ML as a collection of points in space
▪ Hence, classification can be viewed as discovering how to separate two clusters of
points belonging to different classes (left picture)
o Rather than distinguishing images containing cars, planes, buildings, for example
▪ Or, it can help to visualize zero-centering and normalization of training data (right
picture)
Basic notation
Basic notation
One dimensional array of numbers

Two-dimensional array of numbers

Matrix represents a particular process of transforming one vector (or the entire vector space) into another:
rotating, scaling, stretching, etc.
[Link]
Dot product

a.b = (length of the projection of a on b)*(length of b)

Dot product is positive if the two vectors are pointing in similar directions, zero if they are
perpendicular, and negative if the two vectors are pointing in nearly opposite directions.
Dot product
Dot Product and Angles
Vectors

• Dot product of vectors, 𝐮 ∙ 𝐯 = 𝐮𝑇 𝐯 = σ𝑖 𝑢𝑖 ∙ 𝑣𝑖

▪ It is also referred to as inner product, or scalar product of vectors
▪ The dot product 𝐮 ∙ 𝐯 is also often denoted by 𝐮, 𝐯

• The dot product is a symmetric operation, 𝐮 ∙ 𝐯 = 𝐮𝑇 𝐯 = 𝐯 𝑇 𝐮 = 𝐯 ∙ 𝐮

• Geometric interpretation of a dot product:
angle between two vectors
▪ I.e., dot product 𝐯 ∙ 𝐰 over the norms of the
vectors is cos 𝜃
𝐮∙𝐯
𝐮∙𝐯= 𝐮 𝐯 𝑐𝑜𝑠 𝜃 cos𝜃 =
𝐮 𝐯

• If two vectors are orthogonal: 𝜃 = 90°, i.e., cos(𝜃) = 0, then 𝐮 ∙ 𝐯 = 0

𝐮∙𝐯
• Also, in ML the term cos𝜃 = is sometimes employed as a measure of
𝐮 𝐯
closeness of two vectors/data instances, and it is referred to as cosine similarity

Picture from: [Link]

Dot product

, this is when we are interested in directions only.

The length of a vector is defined as the square root of the dot product of the vector by itself.
Linear and quadratic functions
Linear functions as decision boundary and dot products
Hyperplanes

• Hyperplane is a subspace whose dimension is one less than that of its ambient space
▪ In a 2D space, a hyperplane is a straight line (i.e., 1D)
▪ In a 3D, a hyperplane is a plane (i.e., 2D)
▪ In a d-dimensional vector space, a hyperplane has 𝑑 − 1 dimensions, and divides the space into two half-spaces
• Hyperplane is a generalization of a concept of plane in high-dimensional space
• In ML, hyperplanes are decision boundaries used for linear classification
▪ Data points falling on either sides of the hyperplane are attributed to different classes
More general linear functions

𝑓 𝑥1, 𝑥2, 𝑥3, 𝑥4 = 𝑥1, 𝑥2, 𝑥3, 𝑥4 . (3, 0, −2, 0)

𝑥1 4, −1,0,0 . 𝑋 4 −1 0 0 𝑥1
𝑥2
𝑓 ⇒ 0,0,1,0 . 𝑋 = 0 0 1 0 𝑥2
𝑥3
−1,0,0,6 . 𝑋 −1 0 0 6 𝑥3
𝑥4
Basic notation
Identity matrix
Diagonal matrix
Adding and subtracting… vectors and matrices
Matrices

• Addition or subtraction ( A  B )i , j = A i , j  B i , j

• Scalar multiplication
( cA )i , j = c  Ai , j

( AB )i , j = A i ,1B 1, j + A i ,2B 2, j + + A i ,n B n, j
• Matrix multiplication

▪ Defined only if the number of columns of the left matrix is the same as the number of rows of the right matrix
▪ Note that 𝐀𝐁 ≠ 𝐁𝐀
Vector-Vector Product
Matrix-Vector Product

• By matrix-vector multiplication: Matrix makes linear transformation of a vector x to another

vector y. Transformation is the movement of the vector, and linear means that the grid is
parallel and evenly spaced. In any transformation, the origin must not change.
Matrix-Vector Product
Matrix-Matrix Multiplication

Matrix-Matrix Multiplication is a composition of a set of transformations.

Matrix-Matrix Multiplication (properties)
Transpose of a matrix
Transpose of a matrix
Matrices are associative but not commutative
Symmetric and diagonal squared matrices
Trace
Norm
Examples of norm
Examples of norm
Matrix norm
Linear Independence

The basis of a vector space is a set of linearly independent vectors

that span the full space.
Rank of a Matrix

Matrix B has 𝐫𝐚𝐧𝐤 𝐁 = 𝟏, since the two columns are linearly dependent. 2 −1
𝐁=
4 −2
Matrix C has 𝐫𝐚𝐧𝐤 𝐂 = 𝟐, since it has two linearly independent columns:
I.e., 𝐜4 = −1 ∙ 𝐜1 , 𝐜5 = −1 ∙ 𝐜3 , 𝐜2 = 3 ∙ 𝐜1 +3 ∙ 𝐜3

1 3 0 −1 0
𝐂 = −1 0 1 1 −1
0 −3 1 0 −1
2 3 −1 −2 1
Properties of rank
The Inverse of a Square Matrix

(i.e., 𝐫𝐚𝐧𝐤 𝑨 = 𝒏)
All 𝑎𝑖 must be not zero. If so, invert each 𝑎𝑖 (i.e., 1/ 𝑎𝑖 )
Pseudo-Inverse of a Matrix
Matrices

• Pseudo-inverse of an (m, n) matrix

▪ Also known as Moore-Penrose pseudo-inverse

• For matrices that are not square, the inverse does not exist
▪ Therefore, a pseudo-inverse is used
−1 𝑇
• If 𝑚 > 𝑛, then the pseudo-inverse is 𝐀† = 𝐀T 𝐀 𝐀 and 𝐀† 𝐀 = 𝐈
−1
• If 𝑚 < 𝑛, then the pseudo-inverse is 𝐀† = 𝐀𝑇 𝐀𝐀T and 𝐀𝐀† = 𝐈
Orthogonal Matrices
Span and Projection

• Give examples of the 2-D and 3-D vector span.

• Special case is if the two vectors are zeros (Span is only one point), or the two
vectors are in the same direction (span is a line).
Range
Null space or kernel of a matrix

It is the set of all solutions to the homogeneous equation Ax = 0

The Determinant
The Determinant: Intuition

• For two-dimensional matrices, S generally The set S

has the shape of a parallelogram.
• In our example, the value of the
determinant is |𝐴|= -7, so the area of the
parallelogram is 7.
• In three dimensions, the set S corresponds
to an object known as a parallelepiped.
The Determinant: Properties
The Determinant: Properties

The Determinant of the diagonal matrix is the product of values of the diagonal.
𝑎 0 0
A= 0 𝑏 0 Det(A) = abc, plot it.
0 0 𝑐
The Determinant: Formula
The Determinant: Examples
Quadratic Forms
A matrix represents squared set of numbers (same rows and columns), it also represent a quadratic function.

𝒙𝑻 𝑰𝒙 = 𝒙𝑻 𝒙, so replace I with any arbitrary matrix A.

Quadratic Forms
Quadratic Forms
When is a square matrix “positive"?
Positive semidefinite matrices
Positive semidefinite matrices

𝑥1
⇒ 𝑥12 + 4𝑥1 𝑥2 + 𝑥22 is this positive quadratic quantity?
𝑥2
Try x1=1 and x2=-1.

So, even all values are positive, it is not positive semidefinite matrix.
Positive semidefinite matrices

If all the diagonal values are >= 0

Positive semidefinite matrices

Yes. If c>=0.

Yes.
Checking if a matrix is PSD
General rule
Positive Semidefinite Matrices

It should be obvious that if A is positive definite, then -A is negative definite and vice
versa.
Likewise, if A is positive semidefinite then -A is negative semidefinite and vice versa.
If A is indefinite, then so is -A.
Positive Semidefinite Matrices

We will comeback to this point later when we

discuss about optimization.
Eigenvalues and Eigenvectors
As said before, matrix transforms the vector space to another vector space, but there are some vectors
which are not transformed but scaled. These vectors are called eigenvectors, and the scale is called
eigenvalues.
Eigenvalues and Eigenvectors
Eigenvalues and Eigenvectors
Properties of eigenvalues and eigenvectors
Eigenvalues and Eigenvectors of Symmetric Matrices

So, Eigenvectors define new bases (new coordinates) for the data, which simplifies the data
representation.

Example:
Example
Example:

[Link]
/[Link]

[Link]
_Algebra/A_First_Course_in_Linear_Algebra_(K
uttler)/07%3A_Spectral_Theory/7.01%3A_Eige
nvalues_and_Eigenvectors_of_a_Matrix
2. Calculus
Differential Calculus

• For a function 𝑓: ℝ → ℝ, the derivative of f is defined as

𝑓 𝑥+ℎ −𝑓 𝑥
𝑓 ′ 𝑥 = lim
ℎ→0 ℎ
• If 𝑓 ′ 𝑎 exists, f is said to be differentiable at a
• If f ‘ 𝑐 is differentiable for ∀𝑐 ∈ 𝑎, 𝑏 , then f is differentiable on this interval
▪ We can also interpret the derivative 𝑓′(𝑥) as the instantaneous rate of change of 𝑓(𝑥) with respect to x
▪ I.e., for a small change in x, what is the rate of change of 𝑓(𝑥)

• Given 𝑦 = 𝑓(𝑥), where x is an independent variable and y is a dependent variable, the following expressions are
equivalent:

′
𝑑𝑦 𝑑𝑓
′
𝑑
𝑓 𝑥 =𝑓 = = = 𝑓 𝑥 = 𝐷𝑓 𝑥 = 𝐷𝑥 𝑓(𝑥)
𝑑𝑥 𝑑𝑥 𝑑𝑥
𝑑
• The symbols , D, and 𝐷𝑥 are differentiation operators that indicate operation of differentiation
𝑑𝑥
Differential Calculus

• The following rules are used for computing the derivatives of explicit functions
Higher Order Derivatives

• The derivative of the first derivative of a function 𝑓 𝑥 is the second derivative of 𝑓 𝑥

𝑑2 𝑓 𝑑 𝑑𝑓
=
𝑑𝑥 2 𝑑𝑥 𝑑𝑥
• The second derivative quantifies how the rate of change of 𝑓 𝑥 is changing
▪ E.g., in physics, if the function describes the displacement of an object, the first derivative gives the velocity of the object
(i.e., the rate of change of the position)
▪ The second derivative gives the acceleration of the object (i.e., the rate of change of the velocity)

• If we apply the differentiation operation any number of times, we obtain the n-th derivative of 𝑓 𝑥
𝑛
𝑛
𝑑𝑛 𝑓 𝑑
𝑓 𝑥 = 𝑛= 𝑓 𝑥
𝑑𝑥 𝑑𝑥
Partial Derivatives
• So far, we looked at functions of a single variable, where 𝑓: ℝ → ℝ
• Functions that depend on many variables are called multivariate functions
• Let 𝑦 = 𝑓 𝐱 = 𝑓(𝑥1 , 𝑥2 , … , 𝑥𝑛 ) be a multivariate function with n variables
▪ The input is an n-dimensional vector 𝐱 = 𝑥1 , 𝑥2 , … , 𝑥𝑛 𝑇 and the output is a scalar y
▪ The mapping is 𝑓: ℝ𝑛 → ℝ

• The partial derivative of y with respect to its ith parameter 𝑥𝑖 is

𝜕𝑦 𝑓(𝑥1 , 𝑥2 , … , 𝑥𝑖 +ℎ, … , 𝑥𝑛 ) − 𝑓(𝑥1 , 𝑥2 , … , 𝑥𝑖 , … , 𝑥𝑛 )

= lim
𝜕𝑥𝑖 ℎ→0 ℎ
𝜕𝑦
• To calculate (𝜕 pronounced “del” or we can just say “partial derivative”), we can treat 𝑥1 , 𝑥2 , … , 𝑥𝑖−1 ,
𝜕𝑥𝑖
𝑥𝑖+1 … , 𝑥𝑛 as constants and calculate the derivative of y only with respect to 𝑥𝑖
• For notation of partial derivatives, the following are equivalent:

𝜕𝑦 𝜕𝑓 𝜕
= = 𝑓 𝐱 = 𝑓𝑥𝑖 = 𝑓𝑖 = 𝐷𝑖 𝑓 = 𝐷𝑥𝑖 𝑓
𝜕𝑥𝑖 𝜕𝑥𝑖 𝜕𝑥𝑖
Integral Calculus
• For a function 𝑓(𝑥) defined on the domain [𝑎, 𝑏], the definite integral of the function is denoted
𝑏

න 𝑓 𝑥 𝑑𝑥
𝑎
• Geometric interpretation of the integral is the area between the horizontal axis and the graph of 𝑓(𝑥) between
the points a and b
▪ In this figure, the integral is the sum of blue areas (where 𝑓 𝑥 > 0) minus the pink area (where 𝑓 𝑥 < 0)
Gradient
• We can concatenate partial derivatives of a multivariate function with respect to all its input variables to
obtain the gradient vector of the function
• The gradient of the multivariate function 𝑓(𝐱) with respect to the n-dimensional input vector 𝐱
= 𝑥1 , 𝑥2 , … , 𝑥𝑛 𝑇 , is a vector of n partial derivatives
𝑇
𝜕𝑓 𝐱 𝜕𝑓 𝐱 𝜕𝑓 𝐱
𝛻𝐱 𝑓 𝐱 = , ,…,
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
• When there is no ambiguity, the notations 𝛻𝑓 𝐱 or 𝛻𝐱 𝑓 are often used for the gradient instead of 𝛻𝐱 𝑓 𝐱
▪ The symbol for the gradient is the Greek letter 𝛻 (pronounced “nabla”), although 𝛻𝐱 𝑓 𝐱 is more often it is
pronounced “gradient of f with respect to x”
• In ML, the gradient descent algorithm relies on the opposite direction of the gradient of the loss function ℒ
with respect to the model parameters 𝜃 𝛻𝜃 ℒ for minimizing the loss function
▪ Adversarial examples can be created by adding perturbation in the direction of the gradient of the loss ℒ with
respect to input examples 𝑥 𝛻𝑥 ℒ for maximizing the loss function
Jacobian Matrix
• The concept of derivatives can be further generalized to vector-valued functions (or, vector fields) 𝑓: ℝ𝑛 → ℝ𝑚
• For an n-dimensional input vector 𝐱 = 𝑥1 , 𝑥2 , … , 𝑥𝑛 𝑇 ∈ ℝ𝑛 , the vector of functions is given as
𝑇
𝐟 𝐱 = 𝑓1 𝐱 , 𝑓2 𝐱 , … , 𝑓𝑚 𝐱 ∈ ℝ𝑚
• The matrix of first-order partial derivates of the vector-valued function 𝐟 𝐱 is an 𝑚 × 𝑛 matrix called a Jacobian

𝜕𝑓1 𝐱 𝜕𝑓1 𝐱
⋯
𝜕𝑥1 𝜕𝑥𝑛
𝐉= ⋮ ⋱ ⋮
𝜕𝑓𝑚 𝐱 𝜕𝑓𝑚 𝐱
…
𝜕𝑥1 𝜕𝑥𝑛
▪ For example, in robotics a robot Jacobian matrix gives the partial derivatives of the translational and angular velocities of
the robot end-effector with respect to the joints (i.e., axes) velocities
Matrix Calculus: The gradient
Matrix Calculus: The gradient
Hessian Matrix
• To calculate the second-order partial derivatives of multivariate functions, we need to calculate the derivatives
for all combination of input variables
• That is, for a function 𝑓(𝐱) with an n-dimensional input vector 𝐱 = 𝑥1 , 𝑥2 , … , 𝑥𝑛 𝑇 , there are 𝑛2 second partial
derivatives for any choice of i and j
𝜕2𝑓 𝜕 𝜕𝑓
=
𝜕𝑥𝑖 𝜕𝑥𝑗 𝜕𝑥𝑖 𝜕𝑥𝑗
• The second partial derivatives are assembled in a matrix called the Hessian

𝜕2𝑓 𝜕2𝑓
⋯
𝜕𝑥1 𝜕𝑥1 𝜕𝑥1 𝜕𝑥𝑛
𝐇𝑓 = ⋮ ⋱ ⋮
2 2
𝜕 𝑓 𝜕 𝑓
…
𝜕𝑥𝑛 𝜕𝑥1 𝜕𝑥𝑛 𝜕𝑥𝑛

• Computing and storing the Hessian matrix for functions with high-dimensional inputs can be computationally
prohibitive
▪ E.g., the loss function for a ResNet50 model with approximately 23 million parameters, has a Hessian of 23 M × 23 M
= 529 T (trillion) parameters
Matrix Calculus: Hessian
Matrix Calculus: Gradients of Linear Functions
Matrix Calculus: Gradients of Quadratic Function
3. Probability
Probability (study of uncertainty)
Axioms of Probability
Properties of Probability
Probability or sample spaces

S = {(1,1),(1,2),…,(6,6)},
S = {1,2,3,4,5,6} X {1,2,3,4,5,6}
S = {1,2,3,4,5,6}2

No. of outcomes is 36
All are equally likely
So, probability of each outcome is 1/36.
All sum to 1.
Events

Event A ⊂ 𝑺
A = {(z1, z2): z1+z2 = 10}
A= {(4,6), (5,5), (6,4)}
P(A) = 3 X 1/36 = 1/12
Multiple Events

Experiment: (Coin #, toss 1, toss 2, toss 3, toss 4)

S = {1,2,3,…, 10} {H, T} {H, T} {H, T} {H, T}

We have two events:

A: Piking the bad coin: {10, __, __, __, __}

B: Getting four tails: {__, T, T, T, T}

P(both A and B occurs) = P(A ∩ B) = P(A)*P(B) = 1/10 * 1 = 1/10

Conditional Probability

P(A|B) =P(A ∩ B)/P(B)

P(A ∩ B) = 1/10
P(B) = P(all four tosses are tails)
= P(bad coin, all tails) + P(not bad coin, all tails)
𝟏 𝟏 𝟏 𝟏
= 1/10 * 1 + 9/10 * (𝟐 × 𝟐
×
𝟐
× ) = 25/160
𝟐

P(A|B) =P(A ∩ B)/P(B) = 1/10 / 25/160 = 0.64

Bayes’ Rule
Bayes’ Theorem
Probability

• Bayes’ theorem – allows to calculate conditional probabilities for one variable when conditional probabilities for
another variable are known
𝑃 𝑌| 𝑋 𝑃 𝑋
𝑃 𝑋| 𝑌 =
𝑃 𝑌

• Also known as Bayes’ rule

• Multiplication rule for the joint distribution is used: 𝑃 𝑋, 𝑌 = 𝑃 𝑌| 𝑋 𝑃 𝑋
• By symmetry, we also have: 𝑃 𝑌, 𝑋 = 𝑃 𝑋| 𝑌 𝑃 𝑌

• The terms are referred to as:

▪ 𝑃 𝑋 , the prior probability, the initial degree of belief for 𝑋
▪ 𝑃 𝑋| 𝑌 , the posterior probability, the degree of belief after incorporating the knowledge of 𝑌
▪ 𝑃 𝑌| 𝑋 , the likelihood of 𝑌 given 𝑋
▪ P(Y), the evidence
𝐥𝐢𝐤𝐞𝐥𝐢𝐡𝐨𝐨𝐝 × 𝐩𝐫𝐢𝐨𝐫 𝐩𝐫𝐨𝐛𝐚𝐛𝐢𝐥𝐢𝐭𝐲
▪ Bayes’ theorem: posterior probability =
𝐞𝐯𝐢𝐝𝐞𝐧𝐜𝐞
Bayes’ Rule
Conditional Probability and Bayes’ Rule
Law of total probability and Bayes' theorem

B1 B2 … Bn
Such that Bi are:
(1) mutually excusive and A∩B1 A ∩ B2 A ∩ Bn
(2) span the entire sample space.
Law of total probability and Bayes' theorem
Example:
Chain Rule
Independence
Random variables
Random variables
The distribution of a random variable

X = {0, 1}
P(X=0) = P(die = 1,2) = 1/3
P(X=1) = P(die = 3,4,5,6) = 2/3
Expected value or mean
• Once we have a distribution of a random variable, we can compute the
expected value or mean.
• Expected value of RV X is the weighted average of all the possible values that
X can take, where each value is weighted by its probability of occurring.
A property of expected values
Variance
Variance
Properties of Variance
Alternative formula for variance

Can be derived from the

previous equation.
Probability Mass Function (PMF)
Cumulative Distribution Function (CDF)
Probability Density Function (PDF)
Some common distributions (Discrete distributions)
Some common distributions (Continuous distributions)
Some common distributions (Summary)
Independence of multiple random variables
Dependance
Positive correlation
Types of correlation
Pearson (1903): fathers
and sons

He tried to quantify the

correlation with a specific
number.

Correlation coefficient: the degree

of correlation
Correlation
coefficient:
pictures

So, what is the

formula?
Covariance and correlation

[-std(X)std(Y), std(X)std(Y)]

Just normalization to be [-1,1]

Properties Involving Covariance
Thank you
 Notation

• 𝑎, 𝑏, 𝑐 Scalar (integer or real)

• 𝐱, 𝐲, 𝐳 Vector (bold-font, lower case)
• 𝐀, 𝐁, 𝐂 Matrix (bold-font, upper-case)
• A, B, C Tensor ((bold-font, upper-case)
• 𝑋, 𝑌, 𝑍 Random variable (normal font, upper-case)
• 𝑎∈𝒜 Set membership: 𝑎 is member of set 𝒜
• 𝒜 Cardinality: number of items in set 𝒜
• 𝐯 Norm of vector 𝐯
• 𝐮 ∙ 𝐯 or 𝐮, 𝐯 Dot product of vectors 𝐮 and 𝐯
• ℝ Set of real numbers
• ℝ𝑛 Real numbers space of dimension n
• 𝑦 = 𝑓 𝑥 or 𝑥 ↦ 𝑓 𝑥 Function (map): assign a unique value 𝑓(𝑥) to each input
value 𝑥
• 𝑓: ℝ𝑛 → ℝ Function (map): map an n-dimensional vector into a scalar
 Notation

• 𝐀⊙𝐁 Element-wise product of matrices A and B

• 𝐀† Pseudo-inverse of matrix A
𝑑𝑛 𝑓
• n-th derivative of function f with respect to x
𝑑𝑥 𝑛
• 𝛻𝐱 𝑓 𝐱 Gradient of function f with respect to x
• 𝐇𝑓 Hessian matrix of function f
• 𝑋~𝑃 Random variable 𝑋 has distribution 𝑃
• 𝑃 𝑋|𝑌 Probability of 𝑋 given 𝑌
• 𝒩 𝜇, 𝜎 2 Gaussian distribution with mean 𝜇 and variance 𝜎 2
• 𝔼𝑋~𝑃 𝑓 𝑋 Expectation of 𝑓 𝑋 with respect to 𝑃 𝑋
• Var 𝑓 𝑋 Variance of 𝑓 𝑋
• Cov 𝑓 𝑋 , 𝑔 𝑌 Covariance of 𝑓 𝑋 and 𝑔 𝑌
• corr 𝑋, 𝑌 Correlation coefficient for 𝑋 and 𝑌
• 𝐷𝐾𝐿 𝑃||𝑄 Kullback-Leibler divergence for distributions 𝑃 and 𝑄
• 𝐶𝐸 𝑃, 𝑄 Cross-entropy for distributions 𝑃 and 𝑄
 Probability Distributions

• Bernoulli distribution
▪ Binary random variable 𝑋 with states 0, 1 𝑝 = 0.3
▪ The random variable can encodes a coin flip
which comes up 1 with probability p and 0
with probability 1 − 𝑝
▪ Notation: 𝑋 ∼ 𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖 𝑝

• Uniform distribution
▪ The probability of each value 𝑖 ∈ 1,2, … , 𝑛 is
1
𝑝𝑖 =
𝑛
▪ Notation: 𝑋 ∼ 𝑈 𝑛
▪ Figure: 𝑛 = 5, 𝑝 = 0.2
 Probability Distributions

• Binomial distribution
▪ Performing a sequence of n independent 𝑛 = 10, 𝑝 = 0.2
experiments, each of which has probability p of
succeeding, where 𝑝 ∈ 0, 1
▪ The probability of getting k successes in n trials
𝑛 𝑘
is 𝑃 𝑋 = 𝑘 = 𝑝 1 − 𝑝 𝑛−𝑘
𝑘
▪ Notation: 𝑋 ∼ 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝑛, 𝑝
• Poisson distribution
▪ A number of events occurring independently in
a fixed interval of time with a known rate 𝜆
𝜆=5
▪ A discrete random variable 𝑋 with states 𝑘
𝜆𝑋 ∙𝑒 −𝜆
∈ 0, 1, 2, … has probability 𝑃 𝑋 = 𝑘 =
𝑋!
▪ The rate 𝜆 is the average number of occurrences
of the event
▪ Notation: 𝑋 ∼ 𝑃𝑜𝑖𝑠𝑠𝑜𝑛 𝜆
 Probability Distributions

• Gaussian distribution
▪ The most well-studied distribution
o Referred to as normal distribution or informally bell-shaped distribution
▪ Defined with the mean 𝜇 and variance 𝜎 2
▪ Notation: 𝑋 ∼ 𝒩 𝜇, 𝜎 2
▪ For a random variable 𝑋 with n independent measurements, the density is
1 𝑥−𝜇 2
−
𝑃𝑋 𝑥 = 𝑒 2𝜎 2
2𝜋𝜎 2