Probability

Course Notes–Week 11

Course Notes–Week 11 Probability 1 / 34

Overview

Some Properties of Expectation

Covariance of two Random Variables
Correlation
Skewness
Kurtosis

Course Notes–Week 11 Probability 2 / 34

Properties of Expectation

Consider a function g (X ) of a single random variable X . For instance, if X is an oven

temperature in degrees centigrade, then

9
g (X ) = X + 32
5
is the same temperature in degrees Fahrenheit. The expectation of the function g (X ) is again
the sum of the products value × probability.

Course Notes–Week 11 Probability 3 / 34

Properties of Expectation

In the discrete case, where X has probability distribution f (x)

X
E [g (X )] = g (xi ) f (xi )
xi

In the continuous case, where X has probability density function f (x)

Z ∞
E [g (X )] = g (x)f (x)dx
−∞

Course Notes–Week 11 Probability 4 / 34

Properties of Expectation

If X has mean µ = E (X ), then taking g (x) = (x − µ)2 , we have E [g (X )] = E (X − µ)2 ,

which is just the variance σ 2 of X

Var(X ) = E (X − µ)2

Course Notes–Week 11 Probability 5 / 34

Properties of Expectation

When g (x) = ax + b, for given constants a and b, then random variable g (X ) has
expectation
Z ∞ Z ∞ Z ∞
E (aX + b) = (ax + b)f (x)dx = a xf (x)dx + b f (x)dx
−∞ −∞ −∞
= aE (X ) + b

and variance
Z ∞
Var(aX + b) = (ax + b − aµX − b)2 f (x)dx
−∞
Z ∞
=a 2
(x − µX )2 f (x)dx
−∞
2
= a Var(X )

Course Notes–Week 11 Probability 6 / 34

Properties of Expectation

Result:
For given constants a and b

E (aX + b) = aE (X ) + b

Var(aX + b) = a2 Var(X )

Course Notes–Week 11 Probability 7 / 34

The mean and standard deviation of a standardized random variable

Example 1.1
Let X have mean µ and standard deviation σ. Use the properties of
expectation to show that the standardized random variable
X −µ
Z=
σ
has mean 0 and standard deviation 1.

Course Notes–Week 11 Probability 8 / 34

Solution of Example 1.1

Solution: Since Z is of the form

X −µ 1 µ
Z= = X − = aX + b
σ σ σ
where a = 1/σ and b = −µ/σ,
1 µ µ µ
E (Z ) = E (X ) − = − = 0
σ σ σ σ
and the variance of Z is
 2
1 σ2
Var(X ) = 2 = 1
σ σ
because E (X ) = µ and Var(X ) = σ 2 .

Course Notes–Week 11 Probability 9 / 34

Example 1.2
Suppose the daily amount of electricity X required for a plating process has mean 10 and
standard deviation 3 kilowatt-hours. If the cost of electricity is 20 dollars per kilowatt hour,
find the mean, variance, and standard deviation of the daily cost electricity.

Solution: The daily cost of electricity,

g (X ) = 20X ,

has mean
20E (X ) = 20 × 10 = 200
dollars and variance
(20)2 Var(X ) = 202 · 32 = 3600.
Its standard deviation is √
3600 = 60
dollars.
Course Notes–Week 11 Probability 10 / 34
Functions of Several Random Variables

Given any collection of k random variables, the function

Y = g (X1 , X2 , . . . , Xk )

Examples include Y = X1 − X2 when g (x1 , x2 ) = x1 − x2 and

Y = 2X1 + 3X2 when g (x1 , x2 ) = 2x1 + 3x2 .

Course Notes–Week 11 Probability 11 / 34

Expectation of g (X1 , . . . , Xk )

Definition 1.1
The random variable g (X1 , X2 , . . . , Xk ) has expected value, or mean, which is the sum of the
products value × probability.

In the discrete case,

XX X
E [g (X1 , X2 , . . . , Xk )] = ··· g (x1 , x2 , . . . , xk ) f (x1 , x2 , . . . , xk )
x1 x2 xk

In the continuous case,

Z ∞ Z ∞ Z ∞
E [g (X1 , X2 , . . . , Xk )] = ... g (x1 , x2 , . . . , xk ) f (x1 , x2 , . . . , xk ) dx1 dx2 · · · dxk
−∞ −∞ −∞

Course Notes–Week 11 Probability 12 / 34

Covariance of two Random Variables

Several important properties of expectation can be deduced from this definition.

Taking g (x1 , x2 ) = (x1 − µ1 ) (x2 − µ2 ), we see that the product (x1 − µ1 ) (x2 − µ2 ) will
be positive if both values x1 and x2 are above their respective means or both are below
their respective means. Otherwise it will be negative.
The expected value E [(X1 − µ1 ) (X2 − µ2 )] will tend to be positive when large X1 and X2
tend to occur together and small X1 and X2 tend to occur together, with high probability.
This measure E [(X1 − µ1 ) (X2 − µ2 )] of joint variation is called the population
covariance of X1 and X2 .

Cov (X1 , X2 ) = E [(X1 − µ1 ) (X2 − µ2 )]

Course Notes–Week 11 Probability 13 / 34

Covariance when Independent

If X1 and X2 are independent so f (x1 , x2 ) = f1 (x1 ) f2 (x2 ),

Z ∞ Z ∞
E [(X1 − µ1 ) (X2 − µ2 )] = (x1 − µ1 ) (x2 − µ2 ) f (x1 , x2 ) dx1 dx2
Z−∞
∞
−∞
Z ∞
= (x1 − µ1 ) f1 (x1 ) dx1 · (x2 − µ2 ) f2 (x2 ) dx2
−∞ −∞
=0

Result
If X1 and X2 are independent, then

Cov (X1 , X2 ) = 0.

Course Notes–Week 11 Probability 14 / 34

Linear Combination: Expectation

Result: Further, the expectation of a linear combination of two independent random

variables Y = a1 X1 + a2 X2 is

E (Y ) = E (a1 X1 + a2 X2 )
Z ∞Z ∞
= (a1 x1 + a2 x2 ) f1 (x)f2 (x2 ) dx1 dx2
−∞ −∞
Z ∞ Z ∞ Z ∞ Z ∞
= a1 x1 f1 (x1 ) dx1 f2 (x2 ) dx2 + a2 f1 (x1 ) dx1 x2 f2 (x2 ) dx2
−∞ −∞ −∞ −∞
= a1 E (X1 ) + a2 E (X2 )

This result holds even if the two random variables are not independent.

Course Notes–Week 11 Probability 15 / 34

Linear Combination: Variance

Also,

Var(Y ) = E (Y − µY )2
h i
= E (a1 X1 + a2 X2 − a1 µ1 − a2 µ2 )2
h i
= E (a1 (X1 − µ1 ) + a2 (X2 − µ2 ))2
h i
= E a12 (X1 − µ1 )2 + a22 (X2 − µ2 )2 + 2a1 a2 (X1 − µ1 ) (X2 − µ2 )
h i h i
= a12 E (X1 − µ1 )2 + a22 E (X2 − µ2 )2 + 2a1 a2 E [(X1 − µ1 ) (X2 − µ2 )]
= a12 Var (X1 ) + a22 Var (X2 ) + 2a1 a2 Cov (X1 , X2 )
= a12 Var (X1 ) + a22 Var (X2 )
since the third term is zero because we assumed X1 and X2 are independent.

Course Notes–Week 11 Probability 16 / 34

Extension to k Variables

These properties hold for any number of random variables whether they are continuous or
discrete.
Let Xi have mean µi and variance σi2 for i = 1, 2, . . . , k.
The linear combination Y = a1 X1 + a2 X2 + · · · + ak Xk has

E (a1 X1 + a2 X2 + · · · + ak Xk ) = a1 E (X1 ) + a2 E (X2 ) + · · · + ak E (Xk )

or
k
X
µY = ai µ i
i=1

Course Notes–Week 11 Probability 17 / 34

Variance for Independent Variables

When the random variables are independent,

Var (a1 X1 + a2 X2 + · · · + ak Xk ) = a12 Var (X1 ) + a22 Var (X2 ) + · · · + ak2 Var (Xk )
or
n
X
σY2 = ai2 σi2
i=1

Course Notes–Week 11 Probability 18 / 34

Example 2.1
Let X1 have mean µ1 and variance σ12 and let X2 have mean µ2 and variance σ22 . Find the
mean and variance of X1 − X2 and X1 + X2 if X1 and X2 are independent.

Note that X1 − X2 is of the form a1 X1 + a2 X2 with a1 = −a2 = 1 so it has mean

1 · µ1 + (−1)µ2 = µ1 − µ2
and variance

(1)2 · σ12 + (−1)2 σ22 = σ12 + σ22

Since X1 + X2 corresponds to the case with a1 = a2 = 1, it has mean

1 · µ1 + 1 · µ2 = µ1 + µ2
and variance

12 · σ12 + 12 · σ22 = σ12 + σ22

Course Notes–Week 11 Probability 19 / 34
Example 2.2
Find the mean and variance of 2X1 + X2 − 5, if X1 has mean 4 and variance 9, while X2
has mean −2 and variance 6, and the two are independent.
(a) E (2X1 + X2 − 5)
(b) Var(2X1 + X2 − 5)

Course Notes–Week 11 Probability 20 / 34

Example 2.3 Mean and Variance of the Sample Mean
Let the n random variables X1 , X2 , . . . , Xn be independent and each have mean µ and
variance σ 2 .
n
1X
Show that for the sample mean X̄ = Xi :
n
i=1
(a) Mean: µX̄ = E (X̄ ) = µ
σ2
(b) Variance: σX̄2 = Var(X̄ ) = n

Course Notes–Week 11 Probability 21 / 34

Solution:

(a) The sample mean

X1 + X2 + · · · + Xn 1 1 1
X̄ = = X1 + X2 + · · · + Xn
n n n n
is a linear combination with constants ai = 1/n for i = 1, 2, . . . , n.
Consequently,
n
1 1 1 X1 1
E (X̄ ) = E (X1 ) + E (X2 ) + · · · + E (Xn ) = µ = nµ = µ
n n n n n
i=1

so the expected value or mean of X̄ is the same as the mean of each observation.

Course Notes–Week 11 Probability 22 / 34

Solution:

(b) The variance of X̄ is

 2  2  2
1 1 1
Var(X̄ ) = Var (X1 ) + Var (X2 ) + · · · + Var (Xn )
n n n
n  2
X 1
= σ2
n
i=1
 2
1
= nσ 2
n
σ2
=
n

Course Notes–Week 11 Probability 23 / 34

Example 3.1
Let the n random variables X1 , X2 , . . . , Xn be independent and each have the same distribution
with mean µ and variance σ 2 .
Use the properties of expectation to show that σ 2 is the mean, or expectation, of the sample
variance
n
X
2
Xi − X̄ /(n − 1)
i=1

Course Notes–Week 11 Probability 24 / 34

Solution:

We write

2
2
Xi − X̄ = Xi − µ + µ − X̄ = (Xi − µ)2 + (µ − X̄ )2 + 2 (Xi − µ) (µ − X̄ )
so the numerator of the sample variance is
n n n n
X
2 X 2
X
2
X
Xi − X̄ = (Xi − µ) + (µ − X̄ ) + 2 (Xi − µ) (µ − X̄ )
i=1 i=1 i=1 i=1
Pn
and the last term equals −2(X̄ − µ) i=1 (Xi − µ) = −2n(X̄ − µ)2 .

Course Notes–Week 11 Probability 25 / 34

Solution:

Consequently,
n n
X
2 X
Xi − X̄ = (Xi − µ)2 − n(X̄ − µ)2
i=1 i=1
2
Now E (Xi − µ) = Var (Xi ) = σ 2 and, by Example 34,

E (X̄ ) = µ and E (X̄ − µ)2 = Var(X̄ ) = σ 2 /n.

Taking expectation term by term and summing,
" n # n
X
2 X σ2
E Xi − X̄ = σ2 − n = (n − 1)σ 2
n
i=1 i=1

Dividing both sides by n − 1, we conclude that σ 2 is the expected value of the sample variance.

Course Notes–Week 11 Probability 26 / 34

The sample variance S 2 is itself a random variable, because it is computed from random data.

n
2 1 X
S = (Xi − X̄ )2 .
n−1
i=1

We conclude that the expected value of the sample variance is

E (S 2 ) = σ 2 .

When the population is normally distributed, the scaled sample variance follows a chi-square
distribution:
(n − 1)S 2
∼ χ2n−1 .
σ2
Note: The chi-square distribution is a special case of the Gamma distribution. In fact,
 
k
χ2k ∼Γ ,2 .
2
Course Notes–Week 11 Probability 27 / 34
Correlation

Definition 4.1
X and Y are two random variable have the means µx and µy , have the standart deviations σX
and σY respectively. The ratio
σXY
ρ=
σX σY
is called correlation coefficient.
Cov(X , Y ) E [(X − µX ) (Y − µY )]
ρ= =r h
σX σY i h i
E (X − µX )2 E (Y − µY )2

Course Notes–Week 11 Probability 27 / 34

Correlation — Interpretation
The correlation coefficient ρ measures the strength and direction of the linear relationship
between two random variables X and Y .
A positive correlation (ρ > 0) indicates that X and Y tend to increase or decrease
together. When one variable is large (small), the other is also likely to be large (small).
A negative correlation (ρ < 0) indicates an inverse relationship: when one variable is
large, the other tends to be small. The stronger the magnitude of ρ, the clearer this
opposing pattern becomes.
The value of ρ always lies between −1 and 1:
−1 ≤ ρ ≤ 1.
Values of ρ close to +1 represent a strong positive linear relationship, whereas values
close to −1 represent a strong negative linear relationship. Values near 0 indicate a weak
or no linear relationship.
When ρ = 0, the variables are said to be uncorrelated, meaning there is no linear
association. However,
Course Notes–Week 11 note that they may still be dependent in a nonlinear way.
Probability 28 / 34
Example 4.1
Suppose the daily amount of electricity X required for a plating process has mean 10 and
standard deviation 3 kilowatt-hours. If the cost of electricity is 20 dollars per kilowatt-hour,
find the mean, variance, and standard deviation of the daily cost of electricity.

Solution: The daily cost of electricity is

g (X ) = 20X .

The mean cost is

E [g (X )] = 20E (X ) = 20 × 10 = 200.
The variance is
Var(g (X )) = (20)2 Var(X ) = 202 · 32 = 3600.
The standard deviation is √
3600 = 60.

Course Notes–Week 11 Probability 29 / 34

Correlation between X and 20X

Correlation between electricity usage and cost: Since cost is a linear transformation
of usage,
g (X ) = 20X ,
the two variables move together perfectly. When X increases, g (X ) increases
proportionally.

Result
If Y = 20X and Var(X ) > 0, then the correlation between X and Y is

ρX ,Y = 1.

Course Notes–Week 11 Probability 30 / 34

Proof
We have
Cov(X , 20X ) = 20 Cov(X , X ) = 20 Var(X ),
and
Var(20X ) = 202 Var(X ) = 400 Var(X ).
Thus
Cov(X , 20X ) 20 Var(X ) 20 Var(X )
ρX ,20X = p =p = = 1.
Var(X ) Var(20X ) Var(X ) · 400 Var(X ) 20 Var(X )

This represents a perfect positive linear relationship.

Course Notes–Week 11 Probability 31 / 34

Skewness and Kurtosis

The third moment about the mean E (X − µ)3 describes the symmetry or skewness
(lack of symmetry) of a distribution. This characteristic is often measured by the quantity
"  #
E (X − µ)3 X −µ 3
=E .
σ3 σ

The fourth moment about the mean E (X − µ)4 describes whether a distribution is
peaked or flat, also called the kurtosis of the distribution. It is commonly measured by
"  #
E (X − µ)4 X −µ 4
=E .
σ4 σ

Course Notes–Week 11 Probability 32 / 34

Skewness and Kurtosis Overview

Skewness Comparison Normal vs Low/High Kurtosis

Course Notes–Week 11 Probability 33 / 34

Skewness:
Positive skewness (right–skewed): The tail extends to the right; most observations are
concentrated on the left.
Negative skewness (left–skewed): The tail extends to the left; most observations are
concentrated on the right.
The skewness of the normal distribution is 0.
Kurtosis:
Positive kurtosis (leptokurtic): More peaked than the normal distribution with heavier
tails.
Negative kurtosis (platykurtic): Flatter than the normal distribution with lighter tails.
The kurtosis of the normal distribution is 3 (excess kurtosis = 0).

Course Notes–Week 11 Probability 34 / 34