AMCE235164 - Advanced Mathematics

for Computer Engineering

Probability

Dr. Dung T. Le

[Link]
Probability Axioms
Probability P : S  { A | A  S }  [0, 1] satisfies
• Non-negativity: for every event A
P  A  0
• Normalization:
PS   1
• Additivity: for mutually exclusive events Ai
 
 
P  ∪    P  Ai 
 i 1  i1

© Dung T. Le 2
Mutually Exclusive Events
• Mutually exclusive events are events that can not occur at
the same time
• They have no outcomes in common
– For example, flipping a coin and get both head and tail

© Dung T. Le 3
Conditional Probability
• The probability of B given A, or the probability of B
conditioned (or conditional) on A is defined as
P A  B
P  B | A 
P  A
• This formula can be rewritten as multiplicative rule of
probability
P  A  B   P  B | A  P  A
• In some cases events are related. Thus, if an event A has
occurred then an event B can be known more

© Dung T. Le 4
Conditional Probability – Example (cont.)

Roll a dice

• Event A: observe an even number  A = {2, 4, 6}

• Event B: observe a number less than 4  B = {1, 2, 3}

• If we know nothing else, then P(B) = 3/6 = 1/2

• But if we know A has occurred, then P(B|A) = 1/3

© Dung T. Le 5
Conditional Probability – Example (cont.)

Two fair dice are thrown. Given that the first shows 3, what is
the probability that the total exceeds 6?
• Event A: the first die shows 3
• Event B: the total exceeds 6

A   3, b  :1  b  6 , B   a, b  : a  b  6
A  B  {(3, 4),(3,5),(3,6)}
• Hence,
P  A  B | A B | 3
P  B | A     1/ 2
P  A | A| 6

© Dung T. Le 6
Independence
• Events are not always be related.

• Event A and B are independent if and only if

P  B | A  P  B 
or
P  A  B   P  A  P  B 

© Dung T. Le 7
Independence – Example
Choose a card at random from a pack of 52
playing cards, each being picked with equal
probability 1/52. Prove that the suit of the
chosen card is independent of its rank.
4 1 13 1
P  king    , P (spade)  
52 13 52 4
• Alternatively,
1 1 1
P  spade king      P (spade)  P (king)
52 4 13

© Dung T. Le 8
Sample Spaces Have Inconvenience?
• Samples spaces are not always numbers, e.g., words, letters
– Coin flip: S = {Head, Tail}
– Drawing a card: S = {Ace of Spade, 2 of Spade, King of Club, …}

• What happens when you are dealing with six different

outcomes counted over 1000 individual trials?
• We need something that allows us to visualize the
distribution of such items in a meaningful way that charts and
lists of sample spaces and events couldn't possibly accomplish
 RANDOM VARIABLE

© Dung T. Le 9
Why Use a Random Variable?
• In many experiments, it is easier to deal with a summary
variable than with the original probability structure
In an opinion poll, we ask 5 people whether agree or disagree
with a certain issue
– Record a “1” for agree and “0” for disagree  The sample space for
this experiment has 25 elements

• Suppose we only interest in the number of people who agree

– Define X = number of “1” is recorded out of 5  Easier to deal with
this sample space (has only 6 elements)

© Dung T. Le 10
Random Variables
• A random variable is a measureable function X: S   that
{  S : X()  x} for each x  .

• The value of a random variable

– is determined by the outcome of an experiment
– takes on a new value each time the experiment is perform

© Dung T. Le 11
Random Variables – Example
• Roll two dice. Let X = number of sixes
 Possible values of X = {0, 1, 2}

• Throw two coins. Let X = number of heads

 Possible values of X = {0, 1, 2}

• Take an 60-minute exam. Let X = time to complete this exam

 Possible values of X = {x: 0  x  60}


P  X  a   P   S | X    a 

P  a  X  b   P   S | a  X    b 
© Dung T. Le 12
Types of Random Variables
• Discrete Random Variable: A random variable whose set of
possible values is a discrete set
– Number of calls
– People in line
– Mistakes per page
• Continuous Random Variable: A random variable whose set
of possible values is a continuous set
– Length
– Time
– Volume

© Dung T. Le 13
Types of Random Variables
• Roll two dice. Let X = number of sixes
 Possible values of X = {0, 1, 2}  discrete

• Throw two coins. Let X = number of heads

 Possible values of X = {0, 1, 2}  discrete

• Take an 60-minute exam. Let X = time to complete this exam

 Possible values of X = {x: 0  x  1}  continuous

© Dung T. Le 14
Cumulative Distribution Function (CDF)

• Cumulative distribution function (CDF) F: R  [0, 1] of a

real-value random variable X, evaluated at x is defined as

F  x  P  X  x

• The probability that X lies in (a, b], where a < b is

P  a  X  b   FX  b   FX  a 

© Dung T. Le 15
Probability Mass Function (PMF)
• Probability mass function (pmf) p: R  [0, 1] of a discrete
random variable X is defined as

pX  x   P  X  x 

• The probabilities associated with all values must be non-

negative and sum up to 1

 p  x  1
x
X and p X  x   0

© Dung T. Le 16
PMF – Examples
• Roll two dice. Let X = number of sixes  X = {0, 1, 2}
P(X = 0) = 25/36
P(X = 1) = 10/36
P(X = 2) = 1/36
• Throw two coins. Let X = number of heads  X = {0, 1, 2}
P(X = 0) = 1/4
P(X = 1) = 2/4
P(X = 2) = 1/4

© Dung T. Le 17
PMF – Examples (cont.)
• A pmf with P(X = 0) = 1/8, P(X = 1) = 3/8, P(X = 2) = 3/8, P(X = 3) = 1/8

F(0) = P(X  0) = P(X = 0) = 1/8

F(1) = P(X  1) = P(X = 0) + P(X = 1) = 1/2
F(2) = P(X  2) = P(X = 0) + P(X = 1) + P(X = 2) = 7/8
F(3) = P(X  3) = P(X = 0) + P(X = 1) + P(X = 2) + P(X = 3) = 1

© Dung T. Le 18
PMF – Examples (cont.)
• Suppose that 10% of all households have no children, 30%
have one child, 40% have two children, and 20% have three
children. Select a household at random and let X = number of
children. What is the pmf of X?

© Dung T. Le 19
PMF – Examples (cont.)
• We may list each value
P(X = 0) = 0.1
P(X = 1) = 0.3
P(X = 2) = 0.4
P(X = 3) = 0.2

or we may present it as a chart

x P(X = x)
0 0.1
1 0.3
2 0.4
3 0.2

© Dung T. Le 20
PMF – Examples (cont.)
or we may present it as a stick graph

or we may present it as a histogram

© Dung T. Le 21
Mean of a Discrete Random Variable

• Mean is the weighted average of all of its values. The weights

are the probabilities
E  X      x  P  x
x

• The mean is also called the expected value. However, that

does not mean that it is literally the value that we expect to
see

© Dung T. Le 22
Mean – Example
• Recall the example where X was the number of children in a
household.
x P(X = x)
0 0.1
1 0.3
2 0.4
3 0.2

• Find the mean of X .

© Dung T. Le 23
Mean – Example (cont.)
• Multiply each x by the corresponding probability.
x P(X = x) xP(X = x)
0 0.1 0
1 0.3 0.3
2 0.4 0.8
3 0.2 0.6

 = 1.7

• Add up the column of products to get the mean.

© Dung T. Le 24
Variance of a Discrete Random Variable

• Variance is the average of the square deviations from the

mean
V  X      x     P  x
2 2

V  X    2   x2  P  x    2  E  X 2   E 2  X 
x

• Standard variance
  2

© Dung T. Le 25
Variance – Example
• Recall the example where X was the number of children in a
household
x P(X = x)
0 0.1
1 0.3
2 0.4
3 0.2

V  X   E  X 2   E2  X 

  0 2  0.1  12  0.3  2 2  0.4  32  0.2   1.7 2  0.81

© Dung T. Le 26
Distribution of Discrete Random Variables

• Discrete Uniform Distribution

1
p X  x   , n  b  a  1; a, b integers with b  a
n
• Bernoulli Distribution

p X  x   p 1  p  , for x  0,1
x 1 x

• Binomial Distribution
n n x
p X  x     1  p  , for n  0,1, 2,...
 x
• Geometric Distribution
p X  x   p 1  p  , for x  0,1, 2,...
x 1

© Dung T. Le 27
Probability Density Function (PDF)
• A continuous random variable has the probability density
function (pdf) if
b
P  a  X  b    f X  x dx
a

• The CDF of X
x
FX  x   P  X  x    f X  u du


• Some properties
P  X  x  0
d
fX  x  F  x
dx

© Dung T. Le 28
PDF – Example
• A bus arrives at a bus stop every 30 minutes. If a person
arrives at the bus stop at a random time, what is the
probability that the person will have to wait less than 10
minutes for the next bus?

© Dung T. Le 29
PDF – Example (cont.)
• Let W = wait time in minutes. The pdf of W is a uniform
distribution on the interval [0, 30].

1
fW  w   , 0  x  30
30

• This is called a continuous uniform distribution

10 10 1 1
P W  10    fW  w dw   dw 
0 0 30 3
© Dung T. Le 30
Properties of PDF
• P  a  X  b   0 for all possible interval [a, b]

•  f X  x dx  1


• For a specific value of interest x0, the CDF is

x0
F  x0   P  X  x0    f X  x dx


• If x1 and x2 are specific values of interest, then

x2
P  x1  X  x2    f X  x dx  F  x2   F  x1 
x1

© Dung T. Le 31
Expected Value of a Continuous R.V.

• Recall the expected value of a discrete random variable

E  X      x  P  x
x

• Expected value of a continuous random variable


E  X      x  f X  x dx


© Dung T. Le 32
Variance of a Continuous R.V.
• Recall the variance of a discrete random variable
V  X      x     P  x
2 2

• Variance of a continuous random variable


V X     x    f X  x dx
2 2


© Dung T. Le 33
Proof of V(x) of a Continuous R.V.

 

V X   E x   x   f X  x dx
2 2


  f X  x dx

 x 2
 2  x   2

  
  x f X  x dx  2   xf X  x dx  
2 2
 f X  x dx
  

  x 2 f X  x dx  2  2   2


  x 2 f X  x dx   2  E  x 2    2




© Dung T. Le 34
Summary of Mean and Variance
• E  g  X    the expectation of g  X 
 g  x p  x, discrete r.v.
 x

  g  x  f  x  dx, continuous r.v.

•   E  X   the true mean of X
  xp  x  , discrete r.v.

 x
  xf  x  dx, continuous r.v.


•  2  V  X   E  X   2   the variance of X
   x   2 p  x  , discrete r.v.
 x

   x   2 f  x  dx, continuous r.v.


•    2  V  X   the standard deviation of X

© Dung T. Le 35
Exercises
• Let X be a random variable with PDF given by
cx 2 , x 1
fX  x  
0, otherwise
Find the constant c.
Find E(X) and Var (X ).
Find P(X  1/2).
• Answer:
c = 3/2; E(X ) = 0; Var(X ) = 3/5.

© Dung T. Le 36
Exercises (cont.)
• Let X be a random variable with PDF given by
1 x
fX  x  e
2
If Y = X 2, find the CDF of Y.
• Answer:

FY  y   1  e  y

© Dung T. Le 37
Exercises (cont.)
• Let X and Y be two independent random variables. Suppose
that we know Var(2X  Y) = 6 and Var(X + 2Y) = 9. Find
Var(X ) and Var(Y ).

• Answer:
Var(X ) = 1;; Var(Y ) = 2..

© Dung T. Le 38
Exercises (cont.)
• Let X be a discrete random variable with the following PMF
0.1, for x  0
0.4, for x  1

PX  x   0.3, for x  2
0.2, for x  3

0, otherwise
Find E(X ).
Find Var(X ).
If Y = (X  2)2, find E(Y ).

• Answer: E(X ) = 1.6; Var(X ) = 0.84; E(Y ) = 1.

© Dung T. Le 39
Normal Distribution
• ‘Bell-shaped’ and symmetrical
• Mean (average), median (midpoint), mode are equal
• ‘Middle spread’ is 1.33 
• Random variable has infinite range

© Dung T. Le 40
Importance of Normal Distribution
• Describes many random processes or continuous phenomena
• Can be used to approximate discrete probability distributions
– Example: Binomial

• Basis for Classical Statistical Inference

© Dung T. Le 41
Useful Properties of Normal Distribution

• About half of “weight” below mean (because symmetrical)

• About 68% of probability within 1 standard deviation of mean
(at change in curve)

• About 95% of probability

within 2 standard deviations
• More than 99% of probability
within 3 standard deviations

© Dung T. Le 42
PDF of Normal Distribution
• General form of the PDF of normal distribution is
2
1  x 
1   
f  x  e 2  

 2

x = value of random variable

 = mean of x
 = standard deviation

•  = 3.14159, e = 2.71828

© Dung T. Le 43
Notation of Normal Distribution
• X  N(, ) = random variable X is normally distributed with
mean  and standard deviation .

• For example,
X  N(2, 1)
X  N(2, 2)
X  N(2, 1)
X  N( 2, 2)

© Dung T. Le 44
Standardize the Normal Distribution

• Standard normal distribution, aka., z-distribution is Z  N(0, 1)

X 
Z


© Dung T. Le 45
Area Under the Curve in Z  N(0, 1)
1
1 z  u2
P( Z  z ) 
2 

e 2
du

© Dung T. Le 46
Joint Distribution of R.V.s
• Sometimes more than one measurement (r.v.) is taken on
each member of the sample space
• Joint behavior of two random variables (continuous or
discrete), X and Y determined by their joint cumulative
distribution function
FX ,Y  x, y   P  X  x, Y  y 

• n-dimensional case
FX1 ,..., X n  x1 ,..., xn   P  X 1  x1 ,..., X n  xn 

© Dung T. Le 47
Discrete Case
• Suppose X, Y are discrete random variables defined on the
same probability space
• The joint probability mass function of two discrete random
variables X and Y is the function pX,Y(x,y) defined for all pairs
of real numbers x and y by

p X ,Y  x, y   P  X  x and Y  y 

• For a joint pmf pX,Y(x,y)

p X ,Y  x, y   0 and  p  x, y   1
x y
X ,Y

© Dung T. Le 48
Joint PMF – Example
Toss a coin three times. Define X: number of heads on first toss,
Y: total number of heads
• Sample space S = {TTT, TTH, THT, HTT, THH, HTH, HHT, HHH}
• The joint distribution of X and Y is
x y 0 1 2 3
0 1/8 2/8 1/8 0 1/2
1 0 1/8 2/8 1/8 1/2
1/8 3/8 3/8 1/8

• PMF of X: sum the appropriate rows

• PMF of Y: sum the appropriate columns
© Dung T. Le 49
Marginal Probability Function
• The marginal probability mass function for X is
p X  x    p X ,Y  x, y 
y
• The marginal probability mass function for Y is
pY  y    p X ,Y  x, y 
x
• If X1,…, Xn are discrete random variables on the same sample space with
joint probability function
p X1 ,..., X n  x1 ,..., xn   P  X  x1 ,..., X m  xn 
• The marginal probability function for X1 is
p X1  x1    p X1 ,..., X n  x1 ,..., xn 
x2 ,..., xn
• The two-dimensional marginal probability function for X1 and X2 is
p X1 X 2  x1 , x2    p X1 ,..., X n  x1 , x2 , x3 ,..., xn 
x3 ,..., xn

© Dung T. Le 50
Marginal Probability Function – Example

Toss a coin three times. Define X: number of heads on first toss,

Y: total number of heads
• Sample space S = {TTT, TTH, THT, HTT, THH, HTH, HHT, HHH}
• The joint distribution of X and Y is
x y 0 1 2 3
0 1/8 2/8 1/8 0 1/2
1 0 1/8 2/8 1/8 1/2
1/8 3/8 3/8 1/8

• p X  x    p X ,Y  x, y   ? pY  y    p X ,Y  x, y   ?
y x

© Dung T. Le 51
Independence of Random Variables
• Random variables X and Y are independent if the events (X 
A) and (Y  B) are independent

• Theorem:
Two discrete random variables X and Y with joint pmf pX,Y(x,y)
and marginal mass function pX(x) and pY(y), are independent
if and only if
p X ,Y  x, y   p X  x  pY  y 

© Dung T. Le 52
Conditional Joint Probability
• Given the joint pmf of X and Y,
P  X  x and Y  y 
P X  x | Y  y 
P Y  y 
and
P  X  x and Y  y 
P Y  y | X  x  
P  X  x

© Dung T. Le 53
Conditional Joint Probability – Example

Toss a coin three times. Define X: the number of heads on the

first toss, Y: the total number of heads
• Sample space S = {TTT, TTH, THT, HTT, THH, HTH, HHT, HHH}
• The joint distribution of X and Y is
x y 0 1 2 3
0 1/8 2/8 1/8 0 1/2
1 0 1/8 2/8 1/8 1/2
1/8 3/8 3/8 1/8

• P Y  2 | X  1  ? P  X  1| Y  1  ?

© Dung T. Le 54
Continuous Case
• Random variables X and Y are (jointly) continuous if there is a non-
negative function fX,Y(x,y) such that
P   X , Y   A    f X ,Y  x, y dxdy
A

for any “reasonable” two-dimensional set A

• fX,Y(x,y) is called a joint density function for (X, Y)
• If A   X , Y  : X  x, Y  y , the joint CDF of X, Y is
A   X , Y  : X  x, Y  y

• From the fundamental theorem of calculus,

2 2
f X ,Y  x, y   FX ,Y  x, y   FX ,Y  x, y 
xy yx

© Dung T. Le 55
Properties of Joint Density Function

• f X ,Y  x, y   0 for all x, y  R

• Its integral over R2 is

 
  f X ,Y  x, y  dxdy  1
 

• For n random variables, the joint pdf assigns a probability for

each possible combination of values:

f  x1 , x2 ,..., xn   0

 ... f  x , x ,..., x dx ...dx

R R
1 2 n 1 n 1

© Dung T. Le 56
Joint Density Function – Example
• Consider the following bivariate density function

12 2
  x  xy 
f X ,Y  x, y    7
 0
• It is a valid density function

f X ,Y  x, y   0 for all x, y  R
 
  f X ,Y  x, y  dxdy  1
 
• Compute

 x  xy dydx
12 2
1 x
P X  Y    f X ,Y  x, y dxdy  
0 0 7
x y

© Dung T. Le 57
Properties of Joint Distribution Function

For random variables X, Y, FX,Y: R2  [0, 1] given by

• xlim FX ,Y  x, y   0

y 

• lim FX ,Y  x, y   1
x 
y 

• FX ,Y  x, y  is non-decreasing in each variable, i.e.,

FX ,Y  x1 , y1   FX ,Y  x2 , y2 
if x1  x2 and y1  y2
• lim FX ,Y  x, y   FY  y  and lim FX ,Y  x, y   FX  x 
x  y 

© Dung T. Le 58
Marginal Density
• The marginal (cumulative) distribution function of X is
x 
FX  x   P  X  x     f X ,Y  u , y dydx
 

• The marginal density of X is then


fX  x  F '
X  x    f X ,Y  x, y dy
• Similarly the marginal density of Y is

fY  y    f X ,Y  x, y dx


© Dung T. Le 59
Generalization to Higher Dimension

Suppose X, Y, Z are jointly continuous random variables with

density f(x,y,z), then
• Marginal density of X is given by
 
fX  x    f X ,Y ,Z  x, y, z dydz
 

• Marginal density of X, Y is given by


f X ,Y  x, y    f X ,Y ,Z  x, y, z  dz


© Dung T. Le 60
Covariance
• Variables may change in relation to each other
• Covariance measures how much the movement in one variable predicts
the movement in a corresponding variable
Cov  X , Y   E  X   X Y  Y  
• Alternative formula
Cov  X , Y   E  XY   E  X  E Y 
• Variance of a sum
Var  X  Y   Var  X   Var Y   2Cov  X , Y 
• Claim: Covariance is bilinear
Cov  aX  b, cY  d   E  aX  E  aX    cY  E  cY   
 E  ac  X   X Y  Y    acCov  X , Y 
© Dung T. Le 61
Meaning of the Sign of Covariance
• Given Y = aX + b. Then Cov(X,Y) = Cov(X, aX + b) = aVar(X)

• If a > 0, above the average in X goes with above the average in Y

• If a < 0, above the average in X goes with below the average in Y
• Cov(X,Y) = 0 means that there is no linear trend which connects X
and Y

© Dung T. Le 62
Meaning of the Value of Covariance
• Let HI be height in inches and HC be the height in centimeters
Cov  H C ,W   Cov  2.54 H I ,W   2.54Cov  H I ,W 

 the value depends on the units an is not very informative!

© Dung T. Le 63
Covariance and Correlation
• Define the correlation coefficient as
 X  E  X  Y  E Y  
  Corr  X , Y   E   
 SD  X  SD Y  
where E(X) = mean of X; SD(X) = standard deviation of X
• Using the linearity of expectation, we get
Cov  X , Y 
1    1
SD  X  SD Y 

© Dung T. Le 64
Covariance and Correlation
• Notice that
  aX  b, cY  d     XY  a, b  0 
•  This new quantity is independent of the change in scale
So it’s value is quite informative

© Dung T. Le 65
Covariance and Independence
• X and Y are uncorrelated if and only if any of the following
hold
Cov  X , Y   0
Corr  X , Y   0
E  X , Y   E  X  E Y 

• In particular, if X and Y are independent they are

uncorrelated

© Dung T. Le 66
Covariance and Correlation – Example

Toss a coin three times. Define X : number of heads on first toss,

Y: total number of heads
• Sample space S = {TTT, TTH, THT, HTT, THH, HTH, HHT, HHH}
• The joint distribution of X and Y is
x y 0 1 2 3
0 1/8 2/8 1/8 0 1/2
1 0 1/8 2/8 1/8 1/2
1/8 3/8 3/8 1/8

Cov  X , Y 
Cov  X , Y   ?  ?
 XY
© Dung T. Le 67
Questions & Answers

© Dung T. Le 68