Elements of Probability note

Ashikul Islam
December 2024

Chapter: Mathematical Expectation

Question-1: Define Mathematical Expectation with suitable example.

Answer:
Case i : For discrete Random variable

If X is a Discrete random variable which can take finite of infinite sequence of

different possible values 𝑥1 , 𝑥2 , … 𝑥𝑛 with corresponding probabilities
𝑓(𝑥1 ), 𝑓(𝑥2 ), … 𝑓(𝑥𝑛 )
,then the mathematical expectation of the random variable X, denoted by μ,
is defined by

𝛍 = 𝐄(𝑿) = ∑ 𝒙 ⋅ 𝑷(𝑿 = 𝒙)
𝒙

Example: Consider a random variable X representing the outcome of a fair six-sided

die. The possible outcomes are 1, 2, 3, 4, 5, 6, each with equal probability 𝑃(𝑋 =
1
𝑥) = 6 .
The expected value is calculated as:
6
1 1 1 1 1 1
𝐸(𝑋) = ∑ 𝑥 ⋅ 𝑃(𝑋 = 𝑥) = 1 ⋅ + 2 ⋅ + 3 ⋅ + 4 ⋅ + 5 ⋅ + 6 ⋅ .
6 6 6 6 6 6
𝑥=1
Simplifying:
1 + 2 + 3 + 4 + 5 + 6 21
𝐸(𝑋) = = = 3.5
6 6
Thus, the expected value of X is 3.5.

Case ii : For continuous Random Variable

1
 If X is a continuous random variable with pdf f(x) , then the mathematical
expectation of X is defined by

∞
𝛍 = 𝐄(𝐗) = ∫ 𝐱𝐟(𝐱)𝐝𝐱
−∞

Example: Let X be a continuous random variable with a probability density

function:
2𝑥, 0 < 𝑥 < 1
𝑓(𝑥) = {
0, 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
The expected value is calculated as :
∞
𝐸(𝑋) = ∫ 𝑥 ⋅ 𝑓(𝑥) 𝑑𝑥
−∞

Substituting the given 𝑓(𝑥):

1 1
𝐸(𝑋) = ∫ 𝑥 ⋅ 2𝑥 𝑑𝑥 = ∫ 2𝑥 2 𝑑𝑥.
0 0
Evaluating the integral:
1 1
2
𝑥3 1 2
𝐸(𝑋) = 2 ∫ 𝑥 𝑑𝑥 = 2 [ ] = 2 ⋅ =
0 3 0 3 3
2
Thus, the expected value of 𝑋 is 3

Properties of Expectation:

Theorem 1 : If X is a Random Variable with expectation E[X], then

𝐄[𝐚𝐗 + 𝐛] = 𝐚 𝐄[𝐗] + 𝐛
For any numerical constants a and b.

Theorem 2: The expected value of the sum of difference o two or more functions of
a random variable X is the sum or difference of the expected values of the function,
that is
E[g(X) ± h(X)] = E[ g(X)] + E[h(X)]
Where both g(X) and h(X) are functions of X.

2
 Theorem 3: If X is a Random Variable and g1 (X), g 2 (X), g 3 (X), … g k (X) be the k
functions of X , then
𝐄[ 𝐠 𝟏 (𝐗) + 𝐠 𝟐 (𝐗) + 𝐠 𝟑 (𝐗) + ⋯ + 𝐠 𝐤 (𝐗)]
= 𝐄[ 𝐠 𝟏 (𝐗)] + 𝐄[ 𝐠 𝟐 (𝐗)] + 𝐄[ 𝐠 𝟑 (𝐗)] + ⋯ + 𝐄[ 𝐠 𝒌 (𝐗)]

Variance:

If X is a random variable with mean 𝜇, then the variance of X, denoted by Var [X], is
defined by,
𝐯𝐚𝐫[𝑿] = 𝐄[𝑿 − 𝑬(𝑿)]𝟐 = 𝐄[𝑿 − 𝛍]𝟐

∑(𝑥𝑖 − 𝜇)2 𝑓(𝑥𝑖 ) , If X is a discrete Random Variable

={
∫(𝑥 − 𝜇)2 𝑓(𝑥)𝑑𝑥 , If X is a continuous Random Variable

Properties of variance:

Theorem 1. Variance of the random variable X is zero if and only if there exists a
constant c such that 𝐏[𝐗 = 𝐜] = 𝟏; Var (constant) = 0 .
Theorem 2. For any constants a and b , 𝐯𝐚𝐫[ 𝐚𝐗 + 𝐛] = 𝒂𝟐 𝒗𝒂𝒓[𝑿].

Chebyshev’s Inequality: History and Basics

Historical Background
Chebyshev’s inequality is named after the Russian mathematician Pafnuty
Lvovich Chebyshev, who developed it in the 19th century. It is one of
the fundamental results in probability theory and statistics. While Chebyshev
formulated the inequality, the concept was independently discovered and
refined by other mathematicians, including Bienaymé. This is why the
inequality is sometimes referred to as the Bienaymé–Chebyshev inequality.
The inequality is a cornerstone of probability theory because it provides a way

3
 to bound the probability of large deviations for any random variable with finite
variance, without requiring the variable to follow a specific distribution (e.g.,
normal distribution).

Basic concepts:
Chebyshev's inequality provides a lower bound on the probability that the value of a
random variable falls within a certain number of standard deviations from its mean. It
applies to any random variable with a well-defined mean (μ) and variance(𝛔𝟐 ) ,
regardless of the distribution.

The inequality states that the proportion of values that lie within k standard deviations
𝟏
of the mean is at least 𝟏 − for k >1 .
𝐤𝟐

Formula:
For a random variable X with mean, 𝝁 = 𝑬[𝑿] and variance , 𝝈𝟐 =
𝑽𝒂𝒓(𝑿), Chebyshev's inequality is written as:
𝟏
𝑷(|𝑿 − 𝝁| ≥ 𝒌𝝈) ≤ 𝟐 , 𝒌 > 𝟎
𝒌
Alternatively, the probability that X is within k standard deviations of the mean is at
1
least 1 − .
𝑘2
𝟏
𝐏(|𝐗 − 𝛍| < 𝐤𝛔) ≥ 𝟏 − , 𝐤 > 𝟏
𝐤𝟐

Interpretation:

❖ Chebyshev's inequality makes no assumption about the shape of the distribution. It is

applicable to any distribution with finite variance.
❖ The inequality is often used to describe the spread of data and to provide bounds for
outliers in datasets

Chapter: Moments and Generating Functions

In statistics and probability theory, moments are quantitative measures used to

describe the shape and properties of a distribution or a dataset. Moments provide

4
 valuable insights into characteristics such as the center, spread, and shape of the data.
Mathematically, moments are based on the powers of the deviations of data points or
random variables from a central value, usually the mean.

Types of moments:

Raw Moments (about the origin):

• These are calculated with reference to the origin (zero).

• The r-th raw moment is defined as,

𝛍′𝐫 = 𝐄[𝐗 𝐫 ]

where X is the random variable and r is a positive integer.

Central Moments (about the mean):

• These are calculated with reference to the mean of the distribution.

• The r-th central moment is defined as:

𝛍𝐫 = 𝐄[(𝐗 − 𝛍)𝐫 ]

where 𝜇 = 𝐸(𝑋) is the mean of X

Key moments and their significance:

1. First Moment:

• The first raw moment is the mean (𝜇 = 𝐸[𝑋]).

• The first central moment is always zero because,

𝛍𝟏 = 𝐄[𝑿 − 𝛍] = 𝟎

2. Second Moment:

• The second central moment is the variance (𝜎 2 ):

𝛍𝟐 = 𝑬[(𝑿 − 𝛍)𝟐 ]

5
 • The square root of the variance is the standard deviation, which measures the
spread of the data.

3. Third Moment:

• The third central moment (𝜇3 ) indicates skewness, describing the asymmetry of
the distribution:

𝛍𝟑 𝟐
𝛃𝟏 = Skewness =
𝛍𝟐 𝟑

The coefficients of skewness ,denoted by 𝛄𝟏 ,is defined as √𝛃𝟏

4. Fourth Moment:

• The fourth central moment (𝜇4 ) is related to kurtosis, which measures the
"tailedness" or peakedness of the distribution.

𝛍𝟒
𝛃𝟐 =
𝛍𝟐𝟐

The coefficients of kurtosis, denoted by 𝜸𝟐 = 𝜷𝟐 − 𝟑. 𝜷𝟏 𝒂𝒏𝒅 𝜷𝟐 𝒐𝒓 𝜸𝟏 𝒂𝒏𝒅 𝜸𝟐

sometimes measure the shape characteristics of a distribution.

Applications of moments:

1. Descriptive Statistics: Moments provide essential summary statistics like mean,

variance, skewness, and kurtosis.
2. Probability Distributions: Moments help in characterizing and understanding the
properties of distributions.
3. Statistical Modeling: Moments are used in fitting distributions to data and in
hypothesis testing.
4. Engineering and Physics: Moments are used in mechanics to describe mass
distribution and rotation.

Moments offer a mathematical framework to summarize and analyze data or random

variables comprehensively.

6
7