Jize Zhang

HKUST CEE
September 14, 2022
 Next Class: Awesome Guest Lecture on RL

Reinforcement Learning for Smart Building Control

by Professor Zhe WANG

• Same classroom, but we start from a lil later at 1PM (so 1-3pm)

2
 Literature Review

• Paper review (1-2 pages), on a paper you choose & I agree. Template will be provided in Canvas.

• Counts for a total of 10 points in the grading system.

• Find a topic of current interest in the intersection of machine learning and your research domain
(relatively well cited, within 10 years, not a review paper).

• Send me a potential list of papers and I will provide feedback ASAP. After we converge, please
proceed to write the review.

• Due: next Friday Sept. 23th midnight.

3
 Literature Review

• Due: next Friday Sept. 23th midnight.

• The review typically includes several aspects:

– Is the motivation good and important enough?

– Is the proposed idea novel enough for the motivation?
– Is the experimental validation (main results, ablation study) convincing enough?
– Is the paper presentation clear to show necessary details?
– Possible structure: summarization + pros + cons + suggestions

4
 Content

• Fundamentals of Probability Theory

• Discrete random variables

• Continuous random variables

• Random vectors

5
 Some References

• Chris Bishops’ PRML book, Chapter 2.

• Kevin Murphy’s, Machine Learning: A probabilistic perspective, Chapter 2。

6
 Probability
Fundamental

7
 Some Notations

• Outcome & experiment: two terms we will use. An experiment is any activity whose outcome is
unknown or random:
– Getting a head when we toss a coin.

We further associate probabilities to these events by defining the event and the sample space.

• Sample space: the collection of all possible outcomes of an experiment.

– A sample space for a single throw of a die will be {1,2,3,4,5,6}. One of these is bound to occur if we throw a die.
The sample space exhausts all the possibilities that can happen when that experiment is performed.

• Event: a subset of the sample space.

– An event of getting a face card from poker would be E= {J,Q,K}.

8
 Probability axioms

• For any event A, the associated probability is a non-negative real number:

0 ≤ p (A)
• The probability that at least one of all the possible outcomes of the sample space S will
occur is 1:
p (S) = 1
Prof. Andrey
Kolmogorov
• If two events A and B are mutually exclusive (i.e., 𝐴𝐴 ∩ 𝐵𝐵 = ∅), then the probability of
either A or B occurring is the probability of A occurring plus the probability
of B occurring:

p (A 
= B) p (A) + p (B)
9
 Interpretation: Frequentist vs Bayesian

• Frequentist Probability: Long run frequencies of ‘events’ … if

we could repeatedly conduct an experiment an infinite number
of times, the proportion of experiments where the event E
would happen.

Relative frequency (ratio)

• Example event “a package sent within HK for 2nd day delivery
actually arrives within one day”. Now let’s repeat the
experiment …

• As the data size becomes larger, the relative frequency will

stabilize and converge to a value, which we call it the
probability of that event, P(E).

10
 Interpretation: Frequentist vs Bayesian

• Bayesian Probability: Quantify our uncertainty about something, e.g., number

between 0 and 1 to which our belief that an event E occurs
– Even about events that do not have long term frequencies:
How possible will the polar ice cap melt by 2030?

• Example event “a package sent within HK for 2nd day delivery actually arrives within
one day”. I will have some initial belief about this. After observing a few packages, my
will be modified to take into account of observations (e.g., three packages have arrived
on time in a row, the probability of package arrive on time should be high).

• But the probability rules are the same, regardless of which view you choose.

11
Is Coin Toss Random or Deterministic?

12
Interpret Probability in a Bayesian Manner

Now let’s generalize!

13
 Interpret Probability in a Bayesian Manner

• A: a logical (yet uncertain) sentence. For example, there was once life on Mars.

• B: another logical sentence, as long as it is not contradicting A. For example, there was
once intelligent life on Mars.

• I: all information we have.

And some notation shortcut:
Not A: ¬
A
A or B is true: AB
A and B is true: AB

14
Interpret Probability in a Bayesian Manner

15
Interpret Probability in a Bayesian Manner

16
Some Rules

17
 Rule I: Sum Rule

The probability of an event

and its opposite is equal to
one.

=1 p( S =
¬
S ) p( S ) + p( ¬S )

18
 Rule II: Product Rule

p ( AB | I ) = p ( A | BI ) p ( B | I )

The probability of A and B is the probability of A

given that B is true times the probability of B is
true.

If one wishes to verify the truth of two logic

arguments AB, one can first verify B and then verify
A assuming B (i.e., conditional probability)

19
 Example

Consider that we are drawing balls from the box

without putting back. The box contains 10 balls, 6 of
which are red and 4 are blue.

Let’s find the probability that we draw a blue ball in

the first draw (A) and a red ball in the second draw
(B).

20
Rule III: Bayes’ Rule

The Bayes’ Rule connects two conditional probabilities:

p ( B | AI ) p ( A | I )
p ( A | BI ) =
p( B | I )
Proof:

21
Example

What is the probability of there being a fire when you see smoke? You know these facts:

•Fires happens 1% of the time when you see smoke

•Smoke is likely to occur 10% of the time

•Smoke is likely to occur 90% of the times when there is a fire

22
Other Useful Rules

Extending the sum rule, we can do the following for any two logical sentences A and B:
p ( A + B | I )= p ( A | I ) + p ( B | I ) − p ( AB | I )
Proof visually:

23
Other Useful Rules

Consider the sequence of n logical sentences B1 to Bn such that:

One of them is definitely true: p ( B1 + B2 +  + Bn | I ) =

1

while they are mutually exclusive: p ( Bi B j | I ) = 0

We can do the following:

This links together all probabilities (e.g., p(A), p(Bi), p(ABi) and p(A|Bi)).
24
Other Useful Rules

25
Discrete Random Variables

26
 Notations

• A discrete random variable is a function X giving the result of an uncertain experiment.

• Discrete if X takes finite values e.g., 0, 1, 2 …

• Continuous if it takes infinite real values (e.g., arbitrary value from the interval [0, 1]).

• We usually use upper case to represent the random variables X, Y, Z and lower case for their values
(x, y, z).

27
 Probability Mass Function (PMF)

• The probability mass function (PMF) of X , p(X) describes the probability that X takes the value x,
written as 𝑝𝑝 X = 𝑥𝑥 or 𝑝𝑝 𝑥𝑥 for simplification.

• Example: the random variable corresponding to the result of a coin toss. Without loss of generality,
we can map heads to the number 0 and tails to the number 1.

• We need to specify the probability of one of these events, as the probability of the other one is
trivially defined. For a fair coin we have: 𝑝𝑝 X = 0 = 𝑝𝑝 X = 1 =?

28
 Probability Mass Function (PMF): a visualization

• The probability mass function (PMF) of X , p(X) describes the probability that X takes the value x,
written as 𝑝𝑝 X = 𝑥𝑥 or 𝑝𝑝 𝑥𝑥 for simplification. Below we give a simple visualization for a RV that
can take four different values.

29
 Probability Mass Function (PMF): properties

• Non-negative: 𝑝𝑝 𝑥𝑥 > 0.

• Normalized: ∑𝑥𝑥 𝑝𝑝 𝑥𝑥 = 1 (why?).

• Without loss of generality, we can say that 𝑝𝑝 X = 𝑥𝑥1 𝑜𝑜𝑜𝑜 𝑥𝑥2 = 𝑝𝑝 X = 𝑥𝑥1 + 𝑝𝑝(X = 𝑥𝑥2)

30
 Expectation

• The expectation of a random variable X is defined to be:

– Think of it as some rough indicator about the value of the random variable that one should “expect” to get. However,
take this interpretation with a grain of salt (think about coin toss).

• Example: what is the expectation for a fair coin flipping, i.e., when: 𝑝𝑝 X = 0 = 𝑝𝑝 X = 1 = 0.5?

31
 Variance

• The variance of a random variable X is defined as the expectation of the square deviation from its
expectation, i.e.:

– Think of it as some rough indicator about the spread of the random variable around its expectation.

• Example: what is the variance for a fair coin flipping, i.e., when: 𝑝𝑝 X = 0 = 𝑝𝑝 X = 1 = 0.5?

32
 Bernoulli Distribution

• Bernoulli distribution generalizes the concept of a coin toss. Think of it as the result of
an experiment with two possible outcomes 0 and 1. One just needs to specify the
probability of one outcome, typically the probability of zero.

• So mathematically to describe Bernoulli RV, this is all we need:

𝑝𝑝 X = 1 = p

• And we can denote X ~ Bernoulli(p).

• Expectation
• Variance

33
 From Bernoulli to Binomial Distribution

• The Bernoulli distribution describes the coin flipping procedure when we flip it once.
But what happens if we run more than once?

• Here comes the Binomial distribution generalizes the concept of a coin toss:

– The experiment consists of a sequence of n identical and independent Bernoulli experiments called
trials, where n is fixed in advance

– Each trial outcome is a Bernoulli RV -- each trial can result in only one of 2 possible outcomes (let’s
denote them as success and failure).

– Trials are independent -- the outcome on any particular trial does not influence the outcome on other
trial.

34
 From Bernoulli to Binomial Distribution

• The Binomial distribution counts the total number of successes in the given n trials.

• PMF of a binomial distribution X~Bin(n, p), i.e., the probability of k successes, when
we repeat the Bernoulli trial for n times, each time with a success rate of p:

• Expecation: np
• Variance: np(1-p)

35
 Poisson Distribution

• The Poisson distribution models the number of times an event occurs in an interval (physically,
of space or time, let’s say time).
• A bit similar to Binomial: recall that Binomial tells us the number of successes during n trials.

• Poisson assumptions:
– The occurrence of one event does not affect the probability that a second event will occur. That is, events occur
independently.
– The average rate at which events occur is independent of any occurrences. For simplicity, this is usually assumed
to be constant,
PMF:

Where λ = the average number of occurrence during that interval.

36
 Poisson Distribution

As the parameter λ becomes bigger, the PMF of Poisson looks more and more like a bell curve;
it actually converges to a normal distribution with mean λ and standard deviation λ (will talk about
this latter) …

37
Continuous Random Variables

38
 Continuous Random Variables

• A continuous random variable means its possible values comprises infinite values (e.g.,
contain an interval).

• Example: measurement of physical quantities

• In principle variables such as height, weight, and temperature are continuous, in

practice the limitations of our measuring instruments restrict us to a discrete (though
sometimes very finely subdivided) world.

• However, continuous models often approximate real-world situations very well

39
 Cumulative Distribution Function (CDF)

• Let X be a continuous random variable.

• Its cumulative distribution function (CDF) F(x) gives the probability that X is smaller
than x:
F=( x) : P( X ≤ x)

• An example CDF function:

• Several properties:
– it is an increasing function for x
– When x = -∞, F(x) = 0
– When x = +∞, F(x) = 1

40
 Probability Density Function (PDF)

• Let X be a continuous random variable.

P ( x ≤ X ≤ x + ∆x)
• Its probability density function (PDF) p(x) is defined by: p ( x) :=
for small Δx. ∆x

• It is related with the derivative of CDF.

+∞
• Properties: always non-negative 𝑝𝑝(𝑥𝑥) ≥ 0, and integrate up-to-one ∫−∞ 𝑝𝑝 𝑥𝑥 𝑑𝑑𝑑𝑑 = 1

41
 Expectation and Variance

• The expectation of a random variable X is defined to be:

– Just like the discrete case, except that we change summation to integration.

• The variance of a random variable X is defined as:

42
 Uniform Distribution

Models a random variable that equal-likely takes value within an interval [a, b].

• The PDF of uniform distribution: 𝑝𝑝 𝐱𝐱 =

• The CDF of uniform distribution: F 𝐱𝐱 =

• The expectation of uniform distribution: E[x] =

• The variance of uniform distribution: V[x] =

43
 Gaussian Distribution

• Most popular distribution!

• As your dataset becomes larger, Gaussian naturally

appears. Fits many natural phenomena like age, height,
test-scores, IQ scores and so on.

• Analytical tractability. Many parametric methods rely on

Gaussian distribution (e.g., least square regression).

• Interpretable: conclusions and summaries derived from

such analysis are intuitive and easy to explain to audience.

44
 Gaussian Distribution

• PDF:

• Mean and variance, we should already know;

• CDF does not have an analytical form (can numerically

compute)

• Commonly we refer to a Gaussian RV with µ = 0 and σ =

1 as standard Normal RV, usually denoted by the letter Z

45
 Gaussian Distribution

• Inappropriate for skewness, fat tails, multimodality, etc.

46
Standard Normal to Arbitrary Gaussian Distribution

47
 Quantiles

• In the old days we look up the standard

normal table for CDFs.

48
Quantiles to Empirical Rules

49
 Exponential Distribution

• Often used to model the time elapsed between some independent events.

• The PDF of an exponential distribution X~Exp(λ):

• λ > 0 is the parameter of the distribution, often called the rate parameter. The RV is
defined on the interval [0, ∞).

• Expectation: E[X] = 1/ λ and Variance V[X] = 1/ λ2

50
 Beta Distribution

• In contrast to other distributions we mentioned (apart from uniform), it

is also defined on an interval [0, 1].

• The PDF of a Beta distribution X~Beta(α, β) is parameterized by two

shape parameters α, β:

• Recall the binomial distribution. PDF looks very similar!

• Difference between Binomial and Beta is that the former models the
number of successes (k), the latter models the probability (x) of success.

51
Relationship between Multiple
Random Variables

52
 Two Random Variables

• Now suppose that we have two random variables X and Y;

• We will define the joint mass function of the pair (x, y) as the probability that X = x and
Y = y:
𝑝𝑝 𝑥𝑥, 𝑦𝑦 = 𝑝𝑝(𝑋𝑋 = 𝑥𝑥, 𝑌𝑌 = 𝑦𝑦)

• Sum rule: 𝑝𝑝 𝑋𝑋 = 𝑥𝑥𝑥𝑥 = ∑𝑦𝑦 𝑝𝑝(𝑋𝑋 = 𝑥𝑥𝑥𝑥, 𝑌𝑌 = 𝑦𝑦)

• Product rule: 𝑝𝑝 𝑋𝑋 = 𝑥𝑥, 𝑌𝑌 = 𝑦𝑦 = 𝑝𝑝 𝑌𝑌 = 𝑦𝑦 𝑋𝑋 = 𝑥𝑥 𝑝𝑝(𝑋𝑋 = 𝑥𝑥)

• Similar to joint density function, when you swap summation for integration.

• We often call p(x) and p(y) the marginal probability functions.

53
 Covariance

• Covariance operator measures how correlated X and Y are:

• Positive: correlated; Negative: anti-correlated; Zero: uncorrelated.

• A useful property of the covariance operator is that it can give tell you something about
the variance of the sum of two random variables. It is:

• Correlation coefficient: provides a measure of covariance on a standard scale, by

dividing the calculated covariance by standard deviation

54
 Graphical Illustration on Correlation

• We should get some intuitive feeling about correlation/anti-correlation and uncorrelation.

55
Independent Random Variables

56
 Independent Random Variables

• We have two random variables x and y

• We say that they are independent if and only if conditioning on one does not tell you
anything about the other: 𝑝𝑝 𝑥𝑥|𝑦𝑦 = 𝑝𝑝(𝑥𝑥)

• You can check reversely, that the definition is consistent: i.e., 𝑝𝑝 y|𝑥𝑥 = 𝑝𝑝(𝑦𝑦)

• Proof:

57
 Some Properties

• The joint PDF factorizes: 𝑝𝑝 𝑥𝑥, 𝑦𝑦 = 𝑝𝑝 𝑥𝑥 𝑝𝑝(𝑦𝑦)

• Expectation of the product E(XY) is the product of the expectation E(X)E(Y):

• Covariance is zero C[X,Y] = 0 -- i.e., uncorrelated… I might leave this proof for
homework!

• But the reverse is not true! Uncorrelated does not imply independency.

58
 Some Properties

• But the reverse is not true! Uncorrelated does not imply independency.

59
Random Vectors

60
 From Random Variables to Random Vectors

• Suppose that we have N random variables X1, X2, …, XN The term X = (X1, X2, …, XN)
is a random vector (denoted by bold, usually).
• We will refer to their PDF as the joint PDF:
𝑝𝑝 𝐱𝐱 = 𝑝𝑝(𝑥𝑥1, 𝑥𝑥2, … , 𝑥𝑥𝑥𝑥)
• Expectation:

• Covariance: let X and Y be N- and M-dimensional random vectors. The covariance of

X and Y is the N by M matrix consisting of all covariances between components of X
and Y:

61
 Multivariate Gaussian

• Notation:

• PDF:

• Invariant under linear transformations:

• Marginal normality: all marginal distributions are Gaussian.

• Zeros correlation is independent: random variables are independent iff their covariance is 0.

62
Multivariate Gaussian: Uncorrelated & Correlated Examples

…. Suppose that we have N random variables X1, X2, …, XN The

term X = (X1, X2, …, XN) is a random vector (denoted by bold,
usually).

63
 Multivariate Gaussian: Conditional Examples

Let us assume that x1 and x2 are correlated

Gaussian, with mean 1 and 2, variance 2 and 4, and
covariance of 0.9 (weakly correlated).

Now I plot their joint PDF as the right hand figure:

Assume that I observe x2 = -2, and I wish to infer the

conditional PDF of x1 given the information of x2:

It is still a Gaussian distribution (x1|x2), but can you

guess its mean and variance?

64
Multivariate Gaussian -- Conditional Distribution

65
Multivariate Gaussian: Conditional Examples

66