Probability, Statistics and Stochastics Processes (MAL2010)

Lecture 1
Data → Analysis → Utilizing Data
We begin by considering two primary modelling approaches:
• Deterministic Models: No uncertainty is involved. Examples include:

V = IR, F = ma

• Statistical Models: Incorporate uncertainty. Examples include:

Relation between Temperature and Atmospheric Pressure

Relation between Weight and Height
Relation between Income and Expenditure

A statistical formulation can be expressed as:

w = f (u) + ϵ

where ϵ represents uncertainty.

Remark 1. A mathematical structure is used to define a statistical model.

As behaviour changes, uncertainty tends to increase. The target in modelling is to:

1. Reduce error

2. Increase accuracy

1 Statistics
Statistics can be divided into three broad categories:

1.1 Descriptive Statistics

Provides a historical perspective of data. Examples include:
• Measures of central tendency: Mean, Median, Mode
• Measures of dispersion: Range, Variance, Covariance
• Data visualization: Bar Chart, Pie Chart, Box-plot, Scatter diagram

1
1.2 Predictive Statistics
• Analyzing historical data for prediction
• Using a mathematical structure for prediction to understand the behaviour of the
entire population

Population

Sample

After modelling, we test the model to ensure reliability.

1.3 Prescriptive / Inferential Statistics

Focused on:
• Testing
• Validation

2 Probability Theory

2.1 Understanding Probability

Probability is synonymous with chance.

Definition 1 (Empirical Definition). Probability is defined as:

Favourable outcomes
Total outcomes

This is the classical definition, not the rigorous mathematical one.

2.2 Mathematical Structure of Probability

A Probability Function can be illustrated using examples:
1 1
Example 1. Tossing a coin ⇒ P (Head) = 2
Throwing a die ⇒ P (Each face) = 6

2
2.3 Random Experiment
• Number of possible outcomes is fixed.
• In a deterministic experiment, the number of outcomes is fixed and known.
• For a random experiment, the set of outcomes is fixed, but the actual outcome is not
fixed.
• Repetition under identical conditions is possible.

2.4 Sample Space

Definition 2. The Sample Space S is the set of all possible outcomes.

Example 2. 1. Weekday of your birthday: S = {Mon, . . . , Sun}

2. Height of a student in a class

3. Set of all even numbers

S may be finite, countably infinite, or uncountable. Example: S = {2, 4, 6, . . . }.

2.5 Events
Definition 3. An Event is any subset of the sample space S. The empty set ∅ is also an
event.

2.6 Axiomatic Definition of Probability

A probability is a function:
P : F → [0, 1]
satisfying:

1. P (S) = 1

2. If A ∩ B = ∅, then:
P (A ∪ B) = P (A) + P (B)

2.7 Power Set

The Power Set P(S) is the set of all subsets of S.

• Contains all subsets of S

• Contains ∅ and S
• If A ∈ P(S), then Ac ∈ P(S)

3
If |S| = ∞, then:
|P(S)| = 2∞ = ∞

4
Lecture 2

2.8 Probability Function

Definition 4. A probability function is a mapping:

P : F → [0, 1]

used to quantify uncertainty in experiments.

Example 3. Analyzing data of students in IITJ:

• Heights
• Weights
• Health center visits
• Academic perceptions
Each variable exhibits different levels of uncertainty. For example, in height analysis:

Minimum height = 3 ft, Maximum height = 7 ft

We can use bar charts or histograms to understand the uncertainty.

2.9 Axioms of Probability

Let S be the sample space. Then P : F → [0, 1] be such that

1. P (S) = 1

2. If A1 , A2 , · · · ⊂ S are such that Ai ∩ Aj = ∅ for i ̸= j, then:

!
[ X
P Ai = P (Ai )
i i

2.10 Properties of the Power Set

For the power set P(S):

1. ∅ ∈ P(S)

2. S ∈ P(S)

3. If A ∈ P(S), then Ac ∈ P(S)

4. If A, B ∈ P(S), then A ∪ B ∈ P(S) and A ∩ B ∈ P(S)

5
2.11 Sigma Algebra
Definition 5. A sigma algebra F is a set of subsets of S such that:

1. ∅ ∈ F
2. If A ∈ F, then Ac ∈ F
3. If A, B ∈ F, then A ∪ B ∈ F
Remark 2 (De Morgan’s Law). Union and complement closure imply intersection closure:
A ∩ B = (Ac ∪ B c )c ∈ F
Proposition 1. The power set P(S) is maximal sigma algebra.
Example 4. S = {1, 2, . . . , 6}, throwing a die. Let A = {1, 3, 5}. Define F = {∅, S, A, Ac }.
Example 5. S = {1, 2, 3, 4, 5, 6}, A = {2, 4, 6}, B = {3, 6}. Find the smallest sigma algebra
F containing A and B.
Example 6. F = {∅, S} is the minimal sigma algebra.

2.12 Properties of Probability Functions

1. P (Ac ) = 1 − P (A)
2. P (S) = 1
3. P (A ∪ Ac ) = 1 =⇒ P (∅) = 0 if A = S

2.13 Set Operations in Probability

Using union, intersection, and complement:
A ∪ B : “A or B”, A ∩ B : “A and B”, Ac : “not A”

If A ∩ B = ∅, then A and B are mutually exclusive.

If A ∩ B ̸= ∅, then they are not mutually exclusive and:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B)

Proof. Write A = (A ∩ B) ∪ (A ∩ B c ) and B = (B ∩ A) ∪ (B ∩ Ac ). Adding:

P (A) + P (B) = P (A ∩ B) + P (A ∩ B c ) + P (B ∩ A) + P (B ∩ Ac )
Simplifying gives the required formula.
Theorem 1. For three events:
P (A ∪ B ∪ C) = P (A) + P (B) + P (C) − P (A ∩ B) − P (B ∩ C) − P (C ∩ A) + P (A ∩ B ∩ C)

6
2.14 Independence and Conditional Probability
If A and B are independent:
P (A ∩ B) = P (A)P (B)
If dependent:
P (A ∩ B) = P (A)P (B|A)
where P (B|A) is the conditional probability of B given A.

• The process of conditioning reduces the sample space. For example, searching for a
student in IITJ given that the student is in the Lecture Hall Complex (LHC).

Lecture 3

2.15 Events and Joint Occurrence

Let A ⊂ S (sample space). The joint occurrence of events A and B is given by A ∩ B.

If independent: P (A ∩ B) = P (A)P (B)

If dependent: P (A ∩ B) = P (A)P (B|A)

2.16 Conditional Probability

If B has occurred first and then A occurs:
P (A ∩ B)
P (A|B) =
P (B)

where:
P (A) = Probability of A, P (A|B) = Probability of A given B.

Example 7. A new student in IITJ is finding a class. The probability:

P (Finding class | Right-hand side) = . . .

Given more and more information, uncertainty reduces. The process of updating information
is called conditioning.

2.17 Multiplication Rule

P (A ∩ B) = P (B)P (A|B)
As A occurs first, the updation rule increases in relevance.

7
2.18 Axioms for Conditional Probability
Check if P (A|B) satisfies probability axioms:

1. P (S|B) = 1
2. If A1 and A2 are mutually exclusive then P (A1 ∪ A2 | B) = P (A1 | B) + P (A2 | B)

Thus, conditional probability satisfies all the axioms of probability.

2.19 Law of Total Probability

If B1 , B2 , . . . , Bn is a partition of S, then for any A ⊂ S:
X
P (A) = P (A|Bi )P (Bi )
i
The partition of S mean S = ∪Bi and Bi ∩ Bj = ϕ, for all i ̸= j.
Example 8. Searching for a student in IITJ: partition IITJ into subsets like hostel, mess,
etc., and search in each section separately.
Example 9 (Factory with 3 Units). Three production units:
• Unit 1: 50% capacity, 5% defective
• Unit 2: 30% capacity, 3% defective
• Unit 3: 20% capacity, 1% defective

1 2 3

50%, 5% 30%, 3% 20%, 1%

The probability that a product is defective:

P (D) = P (D|1)P (1) + P (D|2)P (2) + P (D|3)P (3)
Example 10 (Binary Communication Channel). The channel receives 0 and 1 with proba-
bilities 60% and 40% respectively. Correct transmission rates: 95% for 0, 90% for 1.

Transmitted Received Probability

0 0 0.95
0 1 0.05
1 0 0.10
1 1 0.90

We can compute:
P (0 received) = P (0 received | 0 transmitted)P (0) + P (0 received | 1 transmitted)P (1)

This approach can be extended to n sequential occurrences.

8
2.20 Bayes’ Theorem
If an item is found defective, we can ask: “Where did it come from?” — the place of
maximum likely.

P (I ∩ D) P (D|I)P (I)
P (I|D) = =P
P (D) i P (D|Ii )P (Ii )

Theorem 2 (Bayes’ Theorem).

P (A ∩ B) = P (A)P (B|A) = P (B)P (A|B)

which implies:
P (A)P (B|A)
P (A|B) =
P (B)

Lecture 4

3 Conditional Probability and Bayesian Inference

3.1 Updation Rule (Conditional Probability)

Definitions:
• Unconditional Probability: P (A) — Probability of event A without any condition.
• Conditional Probability: P (A | B) — Probability of event A given that event B
has occurred. This represents the updated belief after incorporating new information.
Example: In a manufacturing context — suppose we want to determine whether more or
fewer items will be sold given the current economic condition.

3.2 Concept of Prior & Posterior

• Prior Probability: P (A) — Initial belief about event A, before observing new evi-
dence.
• Posterior Probability: P (A | B) — Updated belief about A after observing evidence
B.

Bayes’ Theorem:
P (A ∩ B) P (A) · P (B | A)
P (A | B) = =
P (B) P (B)
Bayes’ theorem requires knowledge of priors and likelihoods.

9
Example: Manufacturing Defects from Multiple Units

Unit Proportion of Production Defect Probability

I 60% 0.01
II 10% 0.10
III 30% 0.20

Let:
D : Defective item found
I1 , I2 , I3 : Events that the item came from Unit I, II, III respectively
Given:
P (I1 ) = 0.6, P (I2 ) = 0.1, P (I3 ) = 0.3
P (D | I1 ) = 0.01, P (D | I2 ) = 0.10, P (D | I3 ) = 0.20

Step 1: Compute Total Probability of Defect (Using the law of total probability)
3
X
P (D) = P (Ii ) · P (D | Ii )
i=1

P (D) = (0.6)(0.01) + (0.1)(0.10) + (0.3)(0.20) = 0.006 + 0.01 + 0.06 = 0.076

Step 2: Compute Posterior Probabilities

Using Bayes’ Theorem:

0.6 · 0.01
P (I1 | D) = ≈ 0.079
0.076
0.1 · 0.10
P (I2 | D) = ≈ 0.132
0.076
0.3 · 0.20
P (I3 | D) = ≈ 0.789
0.076
Although, P (I3 ) = 0.3 (not very high) but the updated probability P (I3 |D) = 0.789 (maxi-
mum).

3.3 Bayes’ Theorem Overview

P (D | A) · P (A)
P (A | D) =
P (D)
Where:
• P (A | D): Posterior probability
• P (D | A): Likelihood
• P (A): Prior probability

10
 • P (D): Marginal probability of data
Applications include:
• Bayesian Networks
• Naive Bayes Classifier
• Email spam filtering

4 Independence of Events
Two events A and B are sain to be independent if:
P (A ∩ B) = P (A) · P (B)
If not independent:
P (A ∩ B) = P (A) · P (B | A) = P (B) · P (A | B)
Alternatively:
P (B | A) = P (B)

Example (Independent Events)

If:
P (A) = 0.8, P (B) = 0.9
then:
P (A ∩ B) = 0.8 · 0.9 = 0.72

Mutual Independence of Three Events

Events A, B and C are mutually independent if:
P (A ∩ B ∩ C) = P (A)P (B)P (C)
P (A ∩ B) = P (A)P (B), P (B ∩ C) = P (B)P (C), P (A ∩ C) = P (A)P (C)

5 Probability Theory Structure

A probability space is a triple:
(S, F, P )
where:
• S: Sample space
• F : Sigma-algebra of events
• P : Probability measure

11
Examples:
{H, T }, {Even, Odd}, {1, 2, 3, 4, 5, 6}

6 Random Variables
A Random Variable (RV) is:
X:S→R
Example: Coin Toss
S = {H, T }, X(H) = 0, X(T ) = 1
1 1
P (X = 0) = , P (X = 1) =
2 2

Example: Tossing a Coin 3 Times

S = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}
Define:
X = number of Heads
Possible values:
X ∈ {0, 1, 2, 3},
1 3 3 1
P (X = 0) = P (X = 1) = P (X = 2) = P (X = 3) =
8 8 8 8

Lecture 5

7 Random Variable
The sample space, defined as the set of all possible outcomes of an experiment, represents
the set of uncertainty.
Depending upon the underlying random experiment, we can have different types of sample
spaces:
• S is finite.
• S is countably infinite.
• S is uncountable.

12
Examples
1. Number of games to win the first match:
• Minimum: 1
• Maximum: Unknown upper limit

2. Number of bikers wearing helmets:

• Minimum: 0
• Maximum: Unknown

3. Number of patients visiting a doctor:

• Minimum: 0
• Maximum: Unknown

4. Lifetime / Survival time:

• Uncountable
• Represented in terms of an interval (a, b)

Example: Random Variable from Sample Space

Consider 5 coins and 5 matches won. Let X ∈ {0, 1, 2, 3, 5} represent the number of matches
won.
Elements from the sample space are mapped to real numbers via a mathematical function,
which is defined as a random variable:

X:S→R

Example: Tossing a Coin Twice

Sample space:
S = {HH, HT, TH, TT}
Let X = number of heads:
X ∈ {0, 1, 2}

Support of a random variable: The set of values of the random variable, corresponding
to which we have non-zero probability.

Can X = 3?
Yes, but with probability P = 0.

13
Probability Distribution of X
X 0 1 2
PX 14 12 14

Here,
P (X = 0) = P (TT) = P (1st toss T ∩ 2nd toss T)
Since the tosses are independent:
P (TT) = P (T ) · P (T )
In general:
P (A ∩ B) = P (A) · P (B)

Remark
If we map elements from S to R, the resulting function is a random variable.

7.1 Types of Sample Spaces and Random Variables

• S binary: X ∈ {0, 1}
• S finite: X ∈ {0, 1, . . . , n}
• S infinite countable: X ∈ {0, 1, 2, . . . }
• S uncountable: X ∈ (a, b)

Sample spaces can be:

• Finite
• Countably infinite
• Uncountable
In such cases, if X takes discrete values, we have a Discrete Random Variable.

7.2 Discrete Random Variables

A Discrete Random Variable X answers the question “How many?” or represents count-
ing/categorizing outcomes. The values of X are taken from the set of real numbers (may be
ranked/nominal).
Let:
X : {x1 , x2 , . . . , xn }, P (X = xi ) = pi
Here, pi is given by:
pi = P (X = xi ) (Probability mass function, PMF)

Conditions:

14
 1. pi ≥ 0, ∀i
n
X
2. pi = 1
i=1

7.3 Defining a Random Variable and Probability Distribution

For defining any random variable:

x: −1, 0, 15, 20 (any real values)

For probability distribution:

p: a, b, c, d
where a > 0, b > 0, c > 0, d > 0 and a + b + c + d = 1.
For the same random variable, we can have different probability distributions. Example:
1 1 1 1 1 3 1 1 1 3 3 1
     
, , , , , , , , , , ,
4 4 4 4 8 8 4 4 8 8 8 8
P
where pi = 1.

7.4 Distribution: Capturing Behaviour of Uncertainty

A distribution captures the behaviour or information of uncertainty.

• Every uncertainty can be explained with an underlying distribution.

• The task is to write the random variable and the underlying probability distribution.
• Values of a random variable can be:
– Categories
– Ordinal (e.g., 0–1)
– Ranked
• We can have a probability distribution for these values.

Example
India vs England – 5 matches. Let X = number of wins by India:

X ∈ {0, 1, 2, 3, 4, 5}

The probability PX can be obtained from historical/statistical data.

15
7.5 Experiments and Data Collection
Consider an experiment: “How many successes in n trials?” We can determine probabilities:
• From historical data (statistics)
• From simulation
• By processing experimental observations
In practice, we have:
X: x1 , x2 , . . . , xn (observations)
P : p1 , p2 , . . . , pn (probabilities)

7.6 Cumulative Distribution Function (CDF) / Probability Law

The CDF accumulates probabilities:

FX (x) = P (X ≤ x)

Example: Let X ∈ {0, 1, 2, 3} with:

1 3 3 1
P (X = 0) = , P (X = 1) = , P (X = 2) = , P (X = 3) =
8 8 8 8
Then:
1 3 4
FX (1) = P (X ≤ 1) = + =
8 8 8
FX (−5) = P (X ≤ −5) = 0
1
FX (0.2) = P (X ≤ 0.2) =
8
1 3 3 7
FX (2) = P (X ≤ 2) = + + =
8 8 8 8
The CDF can be defined for every real number.

Graphical Representation
The CDF is a step function with jumps at each point xi where P (X = xi ) > 0.
• Jumps correspond to the probability mass function (PMF).
• Maximum jump size = 1
• Minimum jump size = 0

16
7.7 Properties of CDFs
1. lim FX (x) = 0, lim FX (x) = 1
x→−∞ x→+∞

2. FX (x) is non-decreasing.

3. FX (x) is right-continuous:
FX (x) = FX (x+ )
It may have left-hand limits (LHL) but is not necessarily left-continuous (for discrete
RVs, it has jumps).

7.8 CDF Defines a Distribution

If fX (x) is given by: 


 0,x≤0
1,

0<x<1

fX (x) = 23



 4
, 1≤x<2
1, x ≥ 2



We check if it satisfies CDF properties:

• Non-decreasing ✓
• FX (−∞) = 0, FX (+∞) = 1 ✓
• Right-continuity at all points ✓ except at x = 0 (must be verified)
If a function is not right-continuous at some point, it is not a valid CDF.

8 Relationship Between CDF and Probability Distri-

bution

Points of Discontinuity
For a discrete random variable X, the points of discontinuity of the cumulative distribu-
tion function (CDF) correspond to the values in the support of X.
Example:
Points of discontinuity: 0, 1, 2
1 1 1
PX (0) = , PX (1) = , PX (2) =
2 4 4

17
CDF ↔ Probability Distribution
Given the CDF FX (x), the probability mass function (PMF) for a discrete random variable
can be obtained as:
pX (x) = FX (x+ ) − FX (x− )
where:
• FX (x+ ) = limt→x+ FX (t) (right-hand limit)
• FX (x− ) = limt→x− FX (t) (left-hand limit)
These jumps in the CDF correspond to the probabilities P (X = x).

8.1 From Sample Space to Random Variable

The sample space S (set of uncertainties) can be mapped to a discrete random variable X.
Once X is defined, we can specify its probability distribution without explicitly working with
the full probability space (S, F, P ) in detail.

18
Lecture 6
Random Variables and Distribution Functions

2. Random Variables (R.V)

A Random Variable is a function that assigns a real number to each outcome of a random
experiment.

Types:

1. Discrete Random Variable

• Takes values from a finite or countable set:
X : x1 , x2 , . . . , xn
with probabilities:
n
X
pi = P (X = xi ), pi = 1
i=1
• Probability distribution is defined by the Probability Mass Function (PMF).
2. Continuous Random Variable
• Takes values from a continuum (measured, not counted).
• No jumps; continuous throughout.
• Probability at a point is zero:
P (X = x) = 0
• The CDF is continuous everywhere.

3. Cumulative Distribution Function (CDF)

For any random variable X:
FX (x) = P (X ≤ x)
Relation: Distribution → Distribution Function → CDF → Probability Distribution.

Example (Discrete Case):

If:
X ∈ {0, 1, 2}
with:
1 1 1
P (X = 0) = , P (X = 1) = , P (X = 2) =
3 3 3
The CDF will be a step function with jumps at x = 0, 1, 2.

19
4. Properties of CDF F (x):
1. limx→−∞ F (x) = 0, limx→∞ F (x) = 1
2. F (x) is non-decreasing.
3. F (x) is right-continuous.
4. For a discrete R.V.:
P (X = x) = F (x) − F (x− )
(Point probability or PMF)

Summary Table:
Feature Discrete R.V. Continuous R.V.
Values Finite / Countable Infinite (measurable)
PMF Yes No
PDF No Yes
P (X = x) > 0 possible Always 0
CDF Type Step function Continuous function

Continuous Random Variables and Probability Density Function

(PDF)
• For a continuous random variable, every single point has probability zero:
P (X = a) = 0
• Example: Probability of hitting a specific point on a board by a blindfolded person.
The board contains infinitely many points, hence probability at any exact point is zero.
• Continuous ⇒ Non-existence of probability for a single point.
Example (Real Line): On the number line, points are densely packed — probability at
one point is zero.

Continuity as Approximation of Discrete Case

• Tossing 60 coins in 1 minute is discrete.
• Tossing 60 coins in 1 second ⇒ can be approximated as continuous.

Probability Over Intervals

For a continuous random variable, probability is assigned to intervals, not points:
P (a < X < b) = P (a ≤ X ≤ b) = P (a ≤ X < b) = P (a < X ≤ b)
The difference between open and closed intervals is irrelevant since P (X = a) = 0.

20
Probability Density Function (PDF)
We replace the Probability Mass Function (PMF) for discrete variables with the Prob-
ability Density Function for continuous variables.

• Definition: A function f (x) is said to be a probability density function (PDF) of a

continuous random variable X if:

f (x) ≥ 0 for all x (non-negative)

and Z ∞
f (x) dx = 1
−∞

• The probability distribution of X is given by f (x) over a ≤ x ≤ b, provided:

Z b
f (x) dx = 1 (total probability)
a

Example

2x, 0 < x < 1,
f (x) =
0, otherwise.
Here f (x) is a valid PDF.
Probability between k1 and k2 :
Z k2
P (k1 ≤ X ≤ k2 ) = 2x dx
k1

Example:
1 3 Z 3/4
 
P <X< = 2x dx
4 4 1/4

Relationship Between PDF and CDF

• The probability for a continuous random variable X in an interval [a, b] is given by the
area under the PDF curve between a and b:
Z b
P (a ≤ X ≤ b) = f (x) dx
a

• The Cumulative Distribution Function (CDF) of X is defined as:

Z k
FX (k) = P (X ≤ k) = f (x) dx
−∞

where f (x) is the Probability Density Function (PDF).

21
From PDF to CDF
Z x
FX (x) = f (t) dt
−∞

The CDF is obtained by integrating the PDF from −∞ to x.

From CDF to PDF

If F (x) is differentiable, then:
d
f (x) = FX (x)
dx
This follows from the Leibniz rule for differentiation under the integral sign.

22
Lecture 7

Example


0 < x < 1,
x,
3
f (x) = kx , 1 ≤ x ≤ 2,



0, otherwise.
Solution: Using the definition of PDF, we have
Z ∞
f (x)dx = 1
−∞
Z 1 Z 2
2
xdx + kx3 dx = 1 =⇒ k =
0 1 15
Now CDF is 


 0, x < 0,
 x2 ,

0 ≤ x ≤ 1,

FX (x) = 12 1 4



 2
+ 30 (x − 1), 1 ≤ x ≤ 2,


1, otherwise.

Functions of Random Variables

Let X has probability distribution

X:0 1 2 3
1 3 3 1
PX :
8 8 8 8
What is the probability distribution of Y = g(X)?.
Let Y = X 2

Y :0 1 4 9
1 3 3 1
PY :
8 8 8 8
Consider the following distribution

X : −1 1 1
1 1 1
PX :
3 3 3
Let Y = X 2

Y :0 1

23
 1 2
PY :
3 3
Let X be a Continuous Random Variables with probability density function fX (x) then what
is the pdf of Y = g(X)?
 
FY (y) = P (Y ≤ y) = P g(X) ≤ y .
Can we convert it into known distribution fX (x)?

 
FY (y) = P X ≤ g −1 (y) (is g invertible?)
= FX (g −1 (y)).
d
fY (y) = FY (y).
dy
Since FY (y) = FX (g −1 (y)), we have

  d  −1 
fY (y) = fX g −1 (y) · g (y) (is g differentiable?)
dy

Example

1, 0 < x < 2,
2
fX (x) =
0, otherwise.
Let Y = X 2 , find fY (y)?.

Solution:
FY (y) = P (Y ≤ y)
 
= P X2 ≤ y
 √ 
=P X≤ Y .
√
Here FY (y) = FX ( Y ). Now, we get
d √
FX ( Y )
fY (y) =
dY
√ 1
= fX ( y). √
2 Y
1
= √ , 0 < y < 4.
4 Y
Theorem 3. Let X be a continuous random variable with probability density function fX (x)
and Y = g(X) be a monotonic and differentiable function. Then the probability density
function of Y is given by
  d −1
fY (y) = fX g −1 (y) g (y) , y ∈ R.
dy

24
Proof. Since g(X) is one-to-one and continuous, it is either strictly monotonically increasing
or decreasing. Assume that it is strictly monotonic increasing. The cdf of Y is given by

FY (y) = P (Y ≤ y) = P (g(X) ≤ y) = P (X ≤ g −1 (y)) = FX (g −1 (y)).

Hence, the pdf of Y is
d   d  
fY (y) = FY (y) = fX g −1 (y) · g −1 (y) .
dy dy
 
In this case, because g is increasing, d
dy
g −1 (y) > 0. Hence we can write d
dy
g −1 (y) =

d
| dy g −1 (y)|.
Suppose g is strictly decreasing function. Then

FY (y) = P (X ≥ g −1 (y)) = 1 − FX (g −1 (y)).

Hence, the pdf of Y is

d   d  −1 
fY (y) = FY (y) = fX g −1 (y) · − g (y) .
dy dy
  
But since g is decreasing d
dy
g −1 (y) < 0 and, hence − dy
d
g −1 (y) = | dy
d
g −1 (y)|. Thus, it is
true for both cases.

Example
We consider fX (x) = 21 , 0 < x < 2 and Y = eX , then find fY (y)?.

Solution: Since the function Y = eX is increasing function in given domain so by using

theorem 3, we get
  d  −1  1 1
fY (y) = fX g −1 (y) · g (y) = fX (ln y) · = , 1 < y < e2 .
dy y 2y

25
Lecture 8

Example
Consider fX (x) = 1, 0 < x < 1 and Y = X 3 , then find fY (y)?.
Solution: Here, Y is a strictly increasing and differentiable function in the given domain.
Then from theorem 3, we have
  d  −1  1 1
fY (y) = fX g −1 (y) · g (y) = fX (y 1/3 ) · y −2/3 , = 2/3 , 0 < y < 1.
dy 3 3y

1
Verification: To check whether fY (y) = y −2/3 is a valid pdf or not.
3
To verify this

1. Total probability: Z 1
fY (y) dy = 1.
0
Here 1
Z 1 Z 1
1 −2/3 1

fY (y) dy = y dy = 3y 1/3 = 1.
0 0 3 3 0

Therefore, fY (y) is a valid probability density function.

Note Many times, a function g(x) may not be one-to-one (1–1). For example, consider the
function Y = x2 . In that case, we see that the region x 7→ Y is the function’s domain and
range. So what we do is look at the inverse image for a given Y . And if there are two inverse
images, then we split the region, i.e., the domain of x into two disjoint regions, so that both
of them map x from each part of the domain to the full range. For that, we have a following
result.

Result: Let X be a continuous random variable with pdf fX (x). Suppose Y = g(X) be
piecewise monotonic function of X. If g1 , g2 , · · · , gn are piecewise monotonic component of
g such that they all map to same range of Y . Then
n
d −1
fX (gi−1 (y))|
X
fY (y) = (g (y))|
i=1 dy i

Example
Let fX (x) = 21 , −1 < x < 1 and Y = X 2 .

Solution: Y = X 2 is neither increasing nor decreasing in (−1, 1). From −1 < x < 1, this
maps to 0 ≤ y < 1. For a given Y (which is positive), we have two inverse images:
√ √
x = − y, −1 < x < 0 and x = + y, 0 < x < 1.

26
So, if we consider two portions of the domain, both are mapped to same range. So, the
idea here is that in each part of the domain, the function will be one-to-one i.e. if we are
√ √
considering only one inverse image say + y or − y) then the function is either increasing
or decreasing. Therefore, we calculate the density in each region separately and add it. So,
we have
√ d√ √ d √
fY (y) = fX ( y)| y| + fX (− y)| (− y)|
dy dy
1 1 1 1
= √ + √
22 y 22 y
1
= √ , 0 < y < 1.
2 y

Example
1
Let fX (x) = , −1 < x < 2 and Y = X 2 .
3
Solution:
From above results, we have

√ d√ √ d √
fX ( y) y + fX (− y) (− y) , 0 < y < 1,



dy dy

fY (y) =
 √ d√
fX ( y) y , 1 < y < 4.



dy

1 d√ 1
Since fX (x) = and y = √ , we get
3 dy 2 y

1


 3√y , 0 < y < 1,





fY (y) = 
1
 6√y ,


 1 < y < 4.

Example
Let fX (x) = π1 , 0 < x < π and Y = sin X.

27
Lecture 9

Expectation of a random variable

Definition 6. Let X be a random variable. If X is a discrete random variable with pmf
p(x) and X
|x|p(x) < ∞,
x

then the expectation of X is X

E(X) = xp(x).
x

If X is a continuous random variable with pdf f (x) and

Z ∞
|x|f (x)dx < ∞,
−∞

then the expectation of X is Z ∞

E(X) = xf (x)dx.
−∞

Example
1. Let the random variable X of the discrete type have the pmf given by

X:0 1 2 3
1 3 3 1
PX :
8 8 8 8
we have
1 3 3 1 3
E(X) = (0 × ) + (1 × ) + (2 × ) + (3 × ) = .
8 8 8 8 2
2. Let the random variable X of the continuous type have the pdf given by

f (x) = 3x2 , 0 < x < 1.

Then Z 1
3
E(X) = x · 3x2 dx = .
0 4
Definition 7. Suppose XR is a continuous random variable with pdf fX (x) and let Y = g(X)
∞
for some function g. If −∞ |g(x)|fX (x)dx < ∞, then the expectation of Y exists and it is
given by Z ∞
E(g(X)) = g(x)fX (x)dx.
−∞

28
 • Case:1 If g(X) = X r , then rth moment about the origin is
n
E(X r ) = xri pi ,
X
(for discrete random variable)
i=1
Z ∞
r
E(X ) = xr f (x)dx, (for continuous random variable)
−∞

• Case:2 If g(X) = (X − a)r , then rth moment about the point a is

h i n
E (X − a)r = (xi − a)r pi ,
X
(for discrete random variable)
i=1
Z ∞
r
E(X ) = (x − a)r f (x)dx, (for continuous random variable)
−∞

Note: If r = 1, then it represents mean.

• Case:3 Second moment about mean
h i n h i2
E (X − E(X))2 =
X
xi − E(X) pi .
i=1

Let E(X) = µ =mean, then we have

h i n h i2
E (X − E(X))2 =
X
xi − µ p i .
i=1

or h i Z ∞
2
E (X − E(X)) = (x − µ)2 f (x)dx.
−∞

Variance
h i
The variance (measure of randomness) of X is defined to be E (X − E(X))2 . It is denoted
by σ 2 or by Var(X).
h i h i
Var(X) = E (X − E(X))2 = E (X 2 − 2E(X)X + (E(X))2 ) ;

since E is a linear operator,

Var(X) = E(X 2 ) − 2(E(X))2 + (E(X))2

So,  2
Var(X) = E(X 2 ) − E(X) .
Note: Var(X) ≥ 0.

29
Example
X : −1 0 1
1 1 1
PX : .
3 3 3
By solving this E(X) = 0 and E(X 2 ) = 32 . Therefore

2
Var(X) = σ 2 = .
3

Standard Deviation
q
σ= Var(X).

Convex Function
A function h : R → R is said to be convex on an interval I ⊆ R if
 
h λx + (1 − λ)y ≤ λf (x) + (1 − λ)f (y),

for all x, y ∈ I and λ ∈ [0, 1].

Jensen’s Inequality
Let h : R → R be a convex function, and let X be a random variable. Then
 
h E(X) ≤ E(h(X)).

Note: In general,
h(E(X)) ̸= E(h(X)).

Examples:
1 1
 
X E(X)
E(e ) ̸= e , E ̸= .
X E(X)

30
Lecture 10

Properties of Expectation
1. For constants a, b ∈ R,
E[aX + b] = a E[X] + b.

2. For constants a, b ∈ R,
 2
Var(aX + b) = E[(aX + b)2 ] − E[aX + b] = a2 Var(X).

Moments
• The first moment is the expectation:

E[X].

• The second central moment is the variance:

Var(X).

• In general, the r-th central moment about the mean is

h i
µr = E (X − E[X])r .

Existence of Moments
If E[X r ] exists for some r, then E[X s ] also exists for all s ≤ r.

Moment Generating Function (MGF)

Let X be a continuous random variable with pdf fX (x). The Moment Generating Function
(MGF) of X, denoted by MX (t), is defined as:
 
MX (t) = E etX ,
provided the expectation exists.

31
Example 1
X : −1, 0, 1
PX : 14 , 14 , 1
2

Mean:
1
E(X) =
4
Second Moment:
3
E(X 2 ) =
4
Variance:  2
3 1 11
Var(X) = − =
4 4 16

Moment Generating Function

 
MX (t) = E etX = etx PX
X

MX (t) = e−t · 1
4
+ e0·t · 1
4
+ e1·t · 1
2

1 1 1
MX (t) = e−t + + et
4 4 2
The pdf is 
1, 0 < x < 2,
f (x) =  2
0, otherwise.

The MGF is Z 2
1
MX (t) = etx · dx.
0 2
#2
1 etx
" !
1 e2t − 1
MX (t) = · = , t ̸= 0.
2 t 0
2 t

Thus,
e2t − 1
MX (t) = , t ̸= 0.
2t

32
Example 2
The pdf is
f (x) = e−x , x > 0.

The MGF is Z ∞ Z ∞
MX (t) = e−x etx dx = e−(1−t)x dx.
0 0

This integral converges only when t < 1.

∞
−1 −(1−t)x 1

MX (t) = e = , t < 1.
1−t 0 1−t
So,
1
MX (t) = , t < 1.
1−t
For t < 1, the MGF of X exists and we can expand:
" #
t2 X 2 t3 X 3
MX (t) = E[etX ] = E 1 + tX + + + ··· .
2! 3!

t2
MX (t) = 1 + t E[X] + E[X 2 ] + · · ·
2!
From the series expansion,
d
E[X] = MX (t) ,
dt t=0

dn
E[X n ] = MX (t) .
dtn t=0

Result
If X and Y are two random variables such that

MX (t) = MY (t),

then X and Y are identically distributed, that is, both X and Y follow same probability
distribution.

33
Example
Let
X : −1, 0, 1, P (X) = 13 , 31 , 13 .

Then,
E[X] = (−1) · 31 + 0 · 13 + (1) · 1
3
= 0.

The moment generating function (MGF) may not exist for all values of t. For some
distributions, the integral Z ∞
MX (t) = etx f (x) dx
−∞

diverges for certain t.

Thus,
|MX (t)| ≰ ∞ (MGF need not be always bounded).

Characteristic Function (CF)

The characteristic function of a random variable X is defined as
Z ∞
φX (t) = E[eitX ] = eitx f (x) dx.
−∞

Property
Since |eitx | = 1, Z ∞ Z ∞
|φX (t)| ≤ |eitx |f (x) dx = f (x) dx = 1.
−∞ −∞

Hence,
|φX (t)| ≤ 1 for all t.

Note:
• MGFs may not exist always.
• Characteristic functions always exist.

34
Lecture 11

Property of MGF
If
MX (t) = MY (t),
then X and Y are identically distributed.

Basic Properties
1. MX (0) = 1

2. For a linear transformation Y = aX + b:

MaX+b (t) = E[et(aX+b) ] = ebt MX (at).

Characteristic Function (CF)

The characteristic function of a random variable X is defined as

φX (t) = E[eitX ] = MX (it) (if MGF exists).

• Characteristic function always exists.

• |φX (t)| ≤ 1.

Z ∞
φX (t) = eitx f (x) dx
−∞

From inversion,
1 Z ∞ −itx
f (x) = e φX (t) dt
2π −∞
For X ∼ Exp(1),
1
MX (t) = , t < 1.
1−t
Thus,
1
φX (t) = MX (it) = .
1 − it
This CF can be inverted to recover the pdf of the exponential random variable.
We had earlier:
1 Z∞ 1
f (x) = e−ixt dt
2π −∞ 1 − it

35
Moment Inequalities

Result 1 (Markov’s Inequality)

Let X be a continuous random variable with pdf fX (x). Let h(x) be a non-negative function.
Then,
E[h(X)]
P (h(X) ≥ ε) ≤ , ε > 0.
ε

Example:
Suppose the salary of people is a random variable X, but the exact distribution is unknown.
If the average salary is given as E[X], then
E[X]
P (Salary ≥ 50, 000) ≤ .
50, 000

Proof
Z ∞
E[h(X)] = h(x)f (x) dx
−∞

Split into two parts:

Z Z
E[h(X)] = h(x)f (x) dx + h(x)f (x) dx
h(x)≥ε h(x)<ε

Note that the second integral is always non-negative, and for the first integral, since h(x) ≥ ε,
Z
h(x)f (x) dx ≥ ε P (h(X) ≥ ε).
h(x)≥ε

Thus,
E[h(X)] ≥ ε P (h(X) ≥ ε).

E[h(X)]
=⇒ P (h(X) ≥ ε) ≤ .
ε
This proves Markov’s Inequality.
We know: Z
E[h(X)] ≥ h(x)f (x) dx
h(x)≥ε

Since h(x) ≥ ε on this domain,

Z
E[h(X)] ≥ ε f (x) dx
h(x)≥ε

36
 =⇒ E[h(X)] ≥ ε P (h(X) ≥ ε).

Thus,
E[h(X)]
P (h(X) ≥ ε) ≤ .
ε
Equivalently,
E[h(X)]
P (h(X) < ε) ≥ 1 − .
ε

Application
Markov’s Inequality provides a fundamental tool for bounding probabilities. It is the basis
for deriving further inequalities, such as Chebyshev’s Inequality.

Markov Inequality:
If we take
h(X) = |X|r , r > 0,
then for any ε > 0:
E[|X|r ]
P (|X|r ≥ εr ) ≤ .
εr
Equivalently,
E[|X|r ]
P (|X| ≥ ε) ≤ .
εr
This is the general form of Markov’s Inequality.
We start with the general inequality:
 
  E |X|r
Pr |X| ≥ ε ≤ , for r > 0.
εr

This can also be written as:

  E (|X|r )
Pr |X| ≥ ε ≤ .
εr
Remark: This shows that if we know any moment of X, then we can bound its tail proba-
bility. It is not necessary to only use the first moment.

37
Chebyshev’s Inequality
Suppose X is a random variable with expectation E(X) = µ and variance Var(X) = σ 2 .
We choose the function h(X) = (X − µ)2 . Then, for any k > 0,
 
  E (X − µ)2 σ2 1
Pr (X − µ)2 ≥ k 2 σ 2 ≤ = = 2.
k2σ2 2
k σ 2 k

Hence,
  1
Pr |X − µ| ≥ kσ ≤ .
k2
Equivalently,
  1
Pr |X − µ| < kσ ≥ 1 − .
k2
For k = 2,
  1 3
Pr |X − µ| < 2σ ≥ 1 − = .
4 4

38
Lecture 12

Standard Discrete Distributions

1. Bernoulli Distribution (Binary outcomes)

• Random variable:
X ∈ {0, 1}
• Probability mass function (pmf):

P (X = 1) = p, P (X = 0) = 1 − p

x 0 1
P (X = x) 1 − p p
General form:
P (X = r) = pr (1 − p)1−r , r ∈ {0, 1}
• Expectation:
E(X) = p
• Variance:
Var(X) = p(1 − p)
• Moment generating function (MGF):
 
MX (t) = E etX = (1 − p) + pet

Remark: A Bernoulli trial is a trial where there are exactly two possible outcomes (suc-
cess/failure).

Binomial Distribution
Consider a random variable X = number of successes out of n Bernoulli trials. Here, n is
the total number of trials in an experiment.

The experiment is set as:

1. All the trials are independent.

2. All the trials are identical, i.e., the probability of success in each trial is the same,
denoted by p.

39
Let Xi ∼ Bernoulli(p), where

P (Xi = 1) = p, P (Xi = 0) = 1 − p.

Then,
X = X 1 + X2 + · · · + Xn
is the sum of n Bernoulli trials, called a Binomial random variable.

X ∈ {0, 1, 2, . . . , n}

The pmf of X is given by:

!
n r
P (X = r) = p (1 − p) n−r , r = 0, 1, 2, . . . , n
r

P (X = 0) = (1 − p)n
!
n
P (X = 1) = p(1 − p)n−1
1

• A random variable X is said to follow a Binomial distribution with parameters n

and p, denoted as
X ∼ Binomial(n, p).

Example
Suppose India vs. Australia, 5 matches. Let

X ∼ Binomial(5, p).

Then, !
5 r
P (X = r) = p (1 − p)5−r , r = 0, 1, 2, . . . , 5.
r

• P (X ≤ 2) ̸= P (X < 2)
• P (X ≤ 3) ̸= P (X < 3) = P (X ≤ 2)

n
!
n r
p (1 − p)n−r .
X
E(X) = r
r=0 r

Simplifying, we get
E(X) = np.

40
 Var(X) = E(X 2 ) − (E(X))2 = np(1 − p).

n
!
tX tr n r
p (1 − p)n−r .
X
MX (t) = E(e ) = e
r=0 r

Simplifying,  n
MX (t) = pet + (1 − p) .

Poisson Distribution as a Limit of Binomial

Consider
Xn ∼ Binomial(n, p).

Conditions
• n is large (n → ∞),
• p is small (p → 0),
• such that λ = np remains finite.

Binomial pmf
!
n r
P (X = r) = p (1 − p)n−r .
r

Substitute p = nλ :
!r !n−r
n(n − 1) · · · (n − r + 1) λ λ
P (X = r) = 1− .
r! n n

Limit
As n → ∞,
n(n − 1) · · · (n − r + 1)
→ 1,
nr
and !n
λ
1− → e−λ .
n

Thus,
e−λ λr
P (X = r) = , r = 0, 1, 2, . . .
r!

41
Lecture 13

Poisson’s Distribution
Let X be a discrete random variable. X is said to follow Poisson’s distribution with param-
eter λ if

λr e−λ
P (X = r) = , r = 0, 1, 2, . . .
r!

∞
X λr e−λ
=1
r=0 r!

If the underlying situation is Binomial and X has no upper bound.

In Practice
• X: no. of occurrences of an event in unit time interval.
• λ: average no. of occurrences in unit time interval (in a hour).

Example
Whether a car passes or not, Bernoulli distribution because

1 → car (X = 1), 0 → no car (X = 0)

Average no. of accidents on a road in a week = 3, i.e. E(X) = 3.

Example 1
Let X: no. of accidents in one week.
It is again Bernoulli as:

Accident (X) → 1, No accident (X) → 0

P (X = 5)

For 5 accidents in one week:

35 e−3
P (X = 5) =
5!

42
Example 2
Let X: no. of patients having cancer.

Yes → 1, No → 0

Since no upper limit, ⇒ Poisson distribution.

Question 2
Probability of 5 accidents in 2 weeks:

λ=6

65 e−6
P (Y = 5) =
5!

Note
If n is large and p is small, then the results generated from Binomial and Poisson are almost
same.

Recurrence Formula
λr e−λ
P (X = r) =
r!

λ
P (X = r + 1) = P (X = r)
r+1

Expectation and MGF

∞
X λr e−λ
E(X) = r· =λ
r=0 r!

V ar(X) = λ

43
Moment Generating Function
∞
λr e−λ
ert ·
X
MX (t) =
r=0 r!

t t
= e−λ · eλe = eλ(e −1)

t
MX (t) = eλ(e −1)

Lecture 14

Geometric Distribution
Definition 8 (Number of failures before the first success). Let the random variable X be
the number of failures before the first success occurs. The probability mass function (PMF)
is given by:
P (X = r) = (1 − p)r p for r = 0, 1, 2, . . .

Definition 9 (Number of trials to get the first success). Let the random variable X be the
number of trials required to get the first success. The PMF is given by:

P (X = r) = (1 − p)r−1 p for r = 1, 2, 3, . . .

Note 8.1. We will use the second definition (number of trials) for the following examples
and properties. Let q = 1 − p.

Example 11 (UPSC Qualification). Let the probability of qualifying for the UPSC exam in
a single attempt be p = 0.3. Let X be the number of attempts needed to qualify. What is the
probability that exactly 4 attempts are needed?

Solution. Here, r = 4, p = 0.3, and q = 0.7.

P (X = 4) = q 4−1 p = (0.7)3 (0.3) = 0.343 × 0.3 = 0.1029

8.1.1 Key Properties and Formulas

For X ∼ Geometric(p) based on the number of trials:

• Expected Value: E[X] = ∞ r−1
= p1
P
r=1 rpq
• Variance: Var(X) = E[X 2 ] − (E[X])2 = pq2

44
 • Moment Generating Function (MGF):
∞
tX
etr pq r−1
X
MX (t) = E[e ] =
r=1
∞
= pet (qet )r−1
X

r=1
t
pe
= , provided qet < 1
1 − qet
Property 8.1 (Memoryless Property). The Geometric distribution is the only discrete dis-
tribution that has the memoryless property.

P (X > m + n | X > m) = P (X > n)

This means that, given the first m trials were failures, the probability of having at least n
more failures is the same as the original probability of having at least n failures.

Proof of the Memoryless Property. The cumulative distribution function (CDF) is FX (r) =
P (X ≤ r) = 1 − q r .
Therefore, the survival function is P (X > r) = 1 − P (X ≤ r) = q r .
By the definition of conditional probability:
P ((X > m + n) ∩ (X > m))
P (X > m + n | X > m) =
P (X > m)
P (X > m + n)
= (since if X > m + n, it must be that X > m)
P (X > m)
q m+n
= m
q
n
=q
= P (X > n)

8.2 The Negative Binomial (Pascal) Distribution

Definition 10. Let the random variable X be the number of trials required to achieve a total
of k successes. The PMF is given by:
!
r−1 k
P (X = r) = p (1 − p)r−k for r = k, k + 1, k + 2, . . .
k−1

The logic is that the rth trial must be the k th success, and the preceding r − 1 trials must
contain exactly k − 1 successes.
Note 8.2. The Negative Binomial distribution does not satisfy the memoryless property.

45
Standard Continuous Distributions

The Uniform Distribution

A random variable X follows a Uniform distribution on the interval [a, b], denoted X ∼
U [a, b], if it is equally likely to take any value in this interval.
• Probability Density Function (PDF):

 1 for a ≤ x ≤ b
b−a
fX (x) =
0 otherwise
• Cumulative Distribution Function (CDF):

0 for x < a


x−a
FX (x) =  b−a
for a ≤ x ≤ b


1 for x > b
a+b
• Expected Value: E[X] = 2
2
• Variance: Var(X) = (b−a)
12

The Exponential Distribution

The Exponential distribution is the continuous analogue of the Geometric distribution and
is often used to model the time until an event occurs.
• Probability Density Function (PDF): Let the rate parameter be λ > 0.

λe−λx for x ≥ 0
fX (x) =
0 otherwise
Sometimes, it is parameterized with the scale parameter β = 1/λ:
1
fX (x) = e−x/β for x ≥ 0
β

Lecture 15

Gamma Function
The Gamma function, Γ(n), is useful for integration.
Γ(n) = 0∞ e−x xn−1 dx
R
•
• For a positive integer n, Γ(n) = (n − 1)!
• A useful property is Γ(n) = (n − 1)Γ(n − 1)
A general form: 0∞ e−ax xn−1 dx = Γ(n)
R
• an

46
Gamma Distribution
Let X ∼ G(α, β). The PDF is:
1
fX (x) = xα−1 e−x/β , x > 0, α > 0, β > 0
Γ(α)β α

The Gamma distribution is a family of distributions.

Special Cases
• When α = 1, the Gamma distribution becomes the Exponential distribution:
1 1
fX (x) = 1
x1−1 e−x/β = e−x/β
Γ(1)β β

• When α = 2:
1 x
fX (x) = 2
x2−1 e−x/β = 2 e−x/β
Γ(2)β β
Mean: E(X) = αβ

Moments and MGF

The hth moment for the exponential distribution is:

h 1 Z ∞ h −x/λ
E(X ) = x e dx
λ 0
For the Gamma distribution:
• Mean: E(X) = αβ
• Second Moment: E(X 2 ) = α(α + 1)β 2
• Variance: V ar(X) = E(X 2 ) − [E(X)]2 = αβ 2
1 1
• MGF: MX (t) = (1−βt) α , for t < β

Chi-Square (χ2 ) Distribution

The Chi-square distribution is a special case of the Gamma distribution where α = ν/2 and
β = 2.
1
fX (x) = xν/2−1 e−x/2 , x ≥ 0
Γ(ν/2)2ν/2
Here, ν is the degree of freedom.
• Mean: E(X) = ν
• Variance: V ar(X) = 2ν

47
Lecture 16

Gaussian (Normal) Distribution

A continuous random variable X follows a normal distribution with a mean µ and variance
σ 2 If its probability density function (pdf) is.

1 1 x−µ 2
fX (x) = √ e− 2 ( σ )
σ 2π
For a normal distribution:
• Mean (E(X)) = µ
!
Z ∞
1 (x − µ)2
Z ∞
E(X) = xf (x) dx = x · √ exp − dx
−∞ −∞ σ 2π 2σ 2
X−µ
– Put σ
=Z
!
1 Z∞ z2
=√ (µ + σz) exp −
2π −∞ 2

! !
µ Z∞ z2 σ Z∞ z2
=√ exp − dz + √ z exp − dz
2π −∞ 2 2π −∞ 2

R∞ z
 2
 √
– Where −∞ exp − 2 dz = 2π
R∞  2

– −∞ z exp − Z2 dz = 0
=⇒ E(X) = µ
• Variance = σ 2
1 2 2
• MGF = E(eXt )= e µt+ 2 σ t

Standardization of a random Variable

Let X be a random variable with mean µ = E[X] and variance σ 2 = Var(X) > 0, The
standardization transforms X into a new variable Z and is defined as:

X −µ
Z=
σ
and follows a standard normal distribution:
Z ∼ N (0, 1).

48
For a random variable X following a normal distribution with X ∼ N (µ, σ 2 ) the transfor-
mation of density function on normalization is fX (x) →
− fZ (z), and it becomes:

d 1 −Z 2
fz (z) = fX (µ + σZ) (µ + σZ) = √ e 2
dZ 2π

Properties
• Z has mean 0:
X −µ 1
 
E[Z] = E = (E[X] − µ) = 0.
σ σ
• Z has variance 1:
X −µ 1
 
V ar(Z) = V ar = 2 V ar(X) = 1.
σ σ
• Z has Coefficient of Skewness 0:
E[Z 3 ] = 0
• Z has Coefficient of Kurtosis 0:
E[Z 4 ] = 3

49
Lecture 17

Example
In a class, the marks obtained by students in a subject follow a normal distribution with
mean 35 and variance 16. Find the probabilities:
(1)P (X < 40)
(2)P (30 < X < 40)
(3)P (X > 45)

Since, X ∼ N (µ, σ 2 ) with µ = 35 and σ = 4

(1)P (X < 40) = P ( X−µ

σ
< 40−35
4
)

= P (Z < 1.25) = 0.894

X−µ
(2)P (35 < X < 40) = P ( 30−35
4
< σ
< 40−35
4
)

= P (−1.25 < Z < 1.25)

= P (Z < 1.25) − P (Z < −1.25)

= P (Z < 1.25) − P (Z > 1.25)

= P (Z < 1.25) − (1 − P (Z < 1.25)) = 2P (Z < 1.25) − 1 = 0.788

(3)P (X > 45) = P ( X−µ
σ
> 45−35
4
)

= P (Z > 2.5)

= 1 − P (Z < 2.5) = 1 − 0.9938 = 0.0062

Representation of normal distribution curve

Example 1:
Let P (Z ≤ a + b) = 0.7 and P (Z ≤ 2a − b) = 0.4, find a and b ?

Since, P (Z ≤ a + b) = 0.7 =⇒ a + b = Z0.3 and

P (Z ≤ 2a − b) = 0.4 =⇒ a + b = Z0.6

50
 Figure 1: P (Z > Zp ) = p or P (Z ≤ Zp ) = 1 − p

From the Z-table Z0.3 = 0.52 and Z0.6 = −0.26, solve for a and b

Example 2:
Find out µ and σ if it is given that the P (X > 30) = 0.4, P (X < 18) = 0.05 and X ∼
N (µ, σ 2 ).
30−µ 30−µ
Since, P (X > 30) = 0.4 =⇒ P (Z > σ
) = 0.4 =⇒ σ
= Z0.4 and
18−µ 18−µ
P (X < 18) = 0.05 =⇒ P (X > 18) = 0.95 =⇒ P (Z > σ
) = 0.95 =⇒ σ
= Z0.95

30−µ 18−µ
From the Z-table, σ
= 0.25 and σ
= 1.65, solve for µ and σ

Other Variants of Normal Distribution

Let X ∼ N (µ, σ 2 ), then

Variant 1: The linear Form of X, i.e Y = aX + b also follows normal distribution with
mean µa + b and variance a2 σ 2

Since,
MY (t) = Max+b (t) = E(e(ax+b)t ) = ebt MX (at)
1 2 (at)2
= ebt eµ(at)+ 2 σ
1 2 (at)2
= e(µa+b)t+ 2 σ

51
Comparing with the MGF of the Normal distribution, Mean= µa + b and Variance = a2 σ 2
Variant 2:Log Normal Distribution
If the log of a normal distribution follows a normal distribution

Let Y = eX =⇒ X = ln Y, Y ∈ (0, ∞)

Mean:
The mean of X is E[X] = E[eY ].
Using the moment-generating function (MGF) of Y :
1 2 t2
MY (t) = E[etY ] = eµt+ 2 σ ,

we set t = 1 to get
σ2
E[X] = E[eY ] = MY (1) = eµ+ 2 .

Variance:
The variance of X is
Var(X) = E[X 2 ] − (E[X])2 .

We have
2
E[X 2 ] = E[e2Y ] = MY (2) = e2µ+2σ .

Thus,  2
σ2
2 2 2
 2
 2
Var(X) = e2µ+2σ − eµ+ 2 = e2µ+2σ − e2µ+σ = eσ − 1 e2µ+σ .

Overall mean and Variance:

σ2
 2
 2
E[X] = eµ+ 2 , Var(X) = eσ − 1 e2µ+σ .

52
Lecture 18

Multiple Random Variables

When dealing with more than one random variable simultaneously, we study their joint
behaviour. This is important in probability, statistics, finance, and machine learning,
where outcomes often depend on several random factors.

• Single random variable: Described by a probability distribution (PMF, PDF, or

CDF).
• Multiple random variables: Require joint distributions to capture dependence.

Discrete Random Variable

Let X and Y , are two discrete random variables having values x1 , x2 , ...., xn and y1 , y2 , .......yn .
Then the joint probability is defined as

X\Y y1 y2 y3 ··· ym P (X = xi )
x1 p11 p12 p13 ··· p1m p1∗
x2 p22 p22 p23 ··· p2m p2∗
x3 p32 p32 p33 ··· p3m p3∗
.. .. .. .. .. .. ..
. . . . . . .
xn pn1 pn2 pn3 · · · pnm pn∗
P (Y = yj ) p·1 p·2 p·3 · · · p·m 1

and the joint probability mass function (pmf) is given as

pX,Y (x, y) = P (X = x, Y = y).

and if it stratifies the Properties:

a) pX,Y (x, y) ≥ 0,
XX
b) pX,Y (x, y) = 1.
x y

Marginal Distributions
Marginal probability mass function (PMF) of x and y is given as
X X
pX (x) = p(x, y), pY (y) = p(x, y).
y x

53
Conditional Distributions
Conditional PMF of x given y is defiend as

p(x, y)
P (X = x | Y = y) = , pY (y) > 0.
pY (y)

and conditional PMF of y given x is defiend as

p(x, y)
P (Y = y | X = x) = , pX (x) > 0.
pX (x)

Independence
X and Y are independent if

p(x, y) = pX (x) pY (y) (discrete),

and Let X, Y, Z be independent, then

P (X = xi , Y = yj , Z = zk ) = P (X = xi )P (Y = yj )P (Z = zk ) ∀i, j, k

Expectation
For discrete case: XX
E[g(X, Y )] = g(x, y) p(x, y).
x y

Example
Consider the experiment of tossing three fair coins simultaneously. Define the random
variables:

• X = the number of heads obtained,

• Y = the absolute difference between the number of heads and tails, i.e.,

Y = |#Heads − #Tails|.

Sample Space
S = {HHH, HHT, HT H, T HH, HT T, T HT, T T H, T T T }

54
Values of X and Y

Outcome X (#Heads) Y = |#Heads - #Tails|

HHH 3 3
HHT 2 1
HTH 2 1
THH 2 1
HTT 1 1
THT 1 1
TTH 1 1
TTT 0 3

Joint Probability Table P (X, Y )

X\Y 0 1 3
0 0 0 1/8
1 0 3/8 0
2 0 3/8 0
3 0 0 1/8

Marginal Probabilities
1 3 3 1
P (X = 0, Y = 3) = , P (X = 1, Y = 1) = , P (X = 2, Y = 1) = , P (X = 3, Y = 3) =
8 8 8 8

6 2
P (Y = 0, X = 0) = 0, P (Y = 1) = , P (Y = 3) =
8 8

Conditional PMF
P (Y | X)
P (X = x, Y = y)
P (Y = y | X = x) =
P (X = x)

X|Y 0 1 3
0 0 0 1
1 0 1 0
2 0 1 0
3 0 0 1

55
P (X | Y )
P (X = x, Y = y)
P (X = x | Y = y) =
P (Y = y)

Y |X 0 1 2 3
0 0 0 0 0
1 0 1/2 1/2 0
3 1/2 0 0 1/2

Expectation
X X
E[X] = xP (X = x) = 1.5, E[Y ] = yP (Y = y) = 1.5
x y

56
Lecture 19

Discrete Random Variable

1. XX
E(g(x, y)) = g(xi , yj ) pij
i j

2. X XX
E(g(x)) = g(xi ) pi+ = g(xi ) pij
i i j

3.
X
E(g(x) | Y = y) = g(xi ) pi|j
i
n
X
= g(xi ) P (X = xi | Y = y)
i=1

= h(y)
E(g(x) | y) = h(y)

Continuous Random Variable

Let X and Y be two random variables, then the joint probability density function of the
pair of X and Y is denoted by f (x, y), where
a) fRR(x, y) > 0.
b) f (x, y) dy dx.
Example:
f (x, y) = 4xy, 0 < x < 1, 0 < y < 1

ZZ
p((x, y) ∈ R) = f (x, y) dy dx

(x, y) ∈ R

57
Marginal Density
The marginal densities of X and Y are obtained by integrating the joint density over the
other variable:
Z ∞
fX (x) = fX,Y (x, y) dy,
−∞
Z ∞
fY (y) = fX,Y (x, y) dx.
−∞

These represent the individual distributions of X and Y irrespective of the other variable.

Conditional Density
The conditional density of X given Y = y is defined as
fX,Y (x, y)

, if fY (y) > 0,



fX|Y (x | y) =  fY (y)


0, if fY (y) = 0.

Similarly, the conditional density of Y given X = x is

fX,Y (x, y)

, if fX (x) > 0,



fY |X (y | x) = fX (x)



0, if fX (x) = 0.

Example:
We are given the joint density
f (x, y) = c(x + y 2 ), 0 < x < 1, 0 < y < 1.

(a) Find c
Since f (x, y) is a pdf, Z 1Z 1
f (x, y) dx dy = 1.
0 0
Z 1Z 1 Z 1
1

2 2
c(x + y ) dx dy = c +y dy.
0 0 0 2
" #1
1 y3 1 1 5
   
=c y+ =c + =c .
2 3 0
2 3 6
Thus,
6
c= .
5
58
(b) Marginal density of X
Z 1 Z 1
6
fX (x) = f (x, y) dy = (x + y 2 ) dy.
0 0 5
6 1 6 1
   
= x(1) + = x+ , 0 < x < 1.
5 3 5 3

(c) Marginal density of Y

Z 1 Z 1
6
fY (y) = f (x, y) dx = (x + y 2 ) dx.
0 0 5
6 1
 
= + y2 , 0 < y < 1.
5 2

(d) Conditional density of X given Y = y

6
f (x, y) 5
(x + y2) x + y2
fX|Y (x | y) = =  = 1 , 0 < x < 1.
fY (y) 6 1
+ y2 2
+ y2
5 2

(e) Conditional density of Y given X = x

6
f (x, y) (x + y 2 ) x + y2
fY |X (y | x) = = 65   = , 0 < y < 1.
fX (x) x + 1 x + 31
5 3

Final Results

6
c= ,
5
6 
fX (x) = x + 13 , 0 < x < 1,
5
6 1 2

fY (y) = + y , 0 < y < 1,
5 2
x + y2
fX|Y (x | y) = 1 , 0 < x < 1,
2
+ y2
x + y2
fY |X (y | x) = , 0 < y < 1.
x + 13

Discrete vs. Continuous Random Variables

A random variable can be classified as either discrete or continuous, depending on the
type of values it can take and the way probabilities are assigned.

59
 Aspect Discrete Continuous
Definition Takes a countable number of dis-
Takes an uncountable (con-
tinct values tinuum) number of values
Example Number of heads in 10 coin tosses,
Height of a person, time
number of students in a class taken to run a race
Probability Assign- Probabilities assigned to individ-
Probability for exact values
ment ual values: P (X = xi ) is 0; probability is defined
over intervals
Probability Function Probability Mass Function (pmf): Probability Density Func-
p(x) = P (X = x) tion (pdf): f (x)
P (a ≤ X ≤ b) = ab f (x) dx
Pb
Probability Computa- P (a ≤ X ≤ b) = x=a p(x)
R

tion
Cumulative Distribu- F (x) = P (X ≤ x) = t≤x p(t) (x) = P (X ≤ x) =
P
F
Rx
tion Function (CDF) −∞ f (t) dt
Graph Representation Bar graph (pmf) Smooth curve (pdf)
P R∞
Total Probability x p(x) = 1 −∞ f (x) dx = 1

Table 1: Comparison between Discrete and Continuous Random Variables

Expectation with Conditional Density

For any measurable function g(x), the conditional expectation is
Z ∞
E[g(X) | Y = y] = g(x) fX|Y (x | y) dx.
−∞

Independence
The random variables X and Y are independent if and only if

fX,Y (x, y) = fX (x) fY (y), ∀ x, y.

Properties:
Z
E[x] = xf (x) dx
Z
E[X|Y ] = xf (x|y) dx
ZZ
E[g(x, y)] = g(x, y)f (x, y) dx dy
x,y

60
Example:
f (x, y) = cxy, 0≤x≤y≤1

Z 1Z 1
cxy dy dx = 1
0 x

Z 1Z 1
cxy dx dy = 1
0 y

Solution
We are given
f (x, y) = cxy, 0 ≤ x ≤ y ≤ 1.

(a) Find c
Since f (x, y) is a probability density function,
Z 1Z y
cxy dx dy = 1.
0 0

First integrate with respect to x:

Z y
x2 y cy 3
cxy dx = cy · = .
0 2 0 2
Now integrate with respect to y:
Z 1
cy 3 c 1 c
dy = · = .
0 2 2 4 8
Thus,
c
=1 =⇒ c = 8.
8

(b) Marginal density of X

Z 1 Z 1
fX (x) = f (x, y) dy = 8xy dy.
x x
!
Z 1
12 x2
= 8x y dy = 8x − .
x 2 2
fX (x) = 4x(1 − x2 ), 0 ≤ x ≤ 1.

61
(c) Marginal density of Y
Z y Z y
fY (y) = f (x, y) dx = 8xy dx.
0 0
Z y
y2
= 8y x dx = 8y · .
0 2
fY (y) = 4y 3 , 0 ≤ y ≤ 1.

(d) Conditional density of X given Y = y

f (x, y) 8xy 2x
fX|Y (x | y) = = 3 = 2, 0 ≤ x ≤ y ≤ 1.
fY (y) 4y y

(e) Conditional density of Y given X = x

f (x, y) 8xy 2y
fY |X (y | x) = = = , x ≤ y ≤ 1.
fX (x) 4x(1 − x )
2 1 − x2

Final Results

c = 8,
fX (x) = 4x(1 − x2 ), 0 ≤ x ≤ 1,
fY (y) = 4y 3 , 0 ≤ y ≤ 1,
2x
fX|Y (x | y) = 2 , 0 ≤ x ≤ y ≤ 1,
y
2y
fY |X (y | x) = , x ≤ y ≤ 1.
1 − x2

62
Lecture 20
We are given the joint PDF

f (x, y) = k, 0 < x < 1, 0 < y < 1, x ≥ 2y,

Solution

1. Find k
The support region is 0 < y < 0.5, 2y ≤ x < 1.
Z 0.5 Z 1 Z 0.5 h i0.5
1= k dx dy = k (1 − 2y) dy = k y − y 2 = k · 0.25
y=0 x=2y 0 0

⇒ k = 4.

So
f (x, y) = 4, 0 < y < 0.5, 2y ≤ x < 1.

2. Marginal PDF of Y
For 0 < y < 0.5: Z 1
fY (y) = 4 dx = 4(1 − 2y).
x=2y

For y ≥ 0.5, fY (y) = 0.

Thus 
4(1 − 2y), 0 < y < 0.5,
fY (y) =
0, otherwise.

3. Marginal PDF of X
For 0 < x < 1, 0 < y ≤ x/2:
Z x/2
fX (x) = 4 dy = 2x.
y=0

So 
2x, 0 < x < 1,
fX (x) =
0, otherwise.

63
4. Conditional PDF f (x | y)
For 0 < y < 0.5:
f (x, y) 4 1
f (x | y) = = = , 2y ≤ x < 1.
fY (y) 4(1 − 2y) 1 − 2y
So 
 1 , 2y ≤ x < 1,
f (x | y) =  1−2y
0, otherwise.

Independence
The random variables X and Y are independent if and only if
fX,Y (x, y) = fX (x) fY (y), ∀ x, y.

Dependence
The random variables X and Y are dependent then
fX,Y (x, y) = fX (x) f (y|x), ∀ x, y.

Example 1
Find whether the given function is independent or not.
f (x, y) = 4xy, 0 < x < 1, 0<y<1

Example 2
Find whether the given function is independent or not.
f (x, y) = x + y, 0 < x < 1, 0<y<1

Example 3
Find whether the given function is independent or not.
f (x, y) = e−(x+y) , 0 < x, 0<y
Theorem 4. Let (X, Y ) be continuous random variables with joint probability density func-
tion f (x, y) and support of x and y. Then X and Y are independent if and only if there
exist functions g(x) and h(y) such that:
f (x, y) = g(x) h(y) for all (x, y),

64
Proof. (⇒) If X and Y are independent:
By definition of independence,

f (x, y) = fX (x) fY (y) for all (x, y) ∈ R2 .

Let g(x) = fX (x), h(y) = fY (y). Then f (x, y) = g(x)h(y).

(⇐) Suppose f (x, y) = g(x)h(y)

We first show g(x) is proportional to fX (x) and h(y) is proportional to fY (y).
The marginal pdf of X is:
Z Z Z
fX (x) = f (x, y) dy = g(x)h(y) dy = g(x) · h(y) dy.

Similarly, the marginal pdf of Y is:

Z Z Z
fY (y) = f (x, y) dx = g(x)h(y) dx = h(y) · g(x) dx.
RR
Now, since f (x, y) dx dy = 1, we have:
Z Z Z  Z 
1= g(x)h(y) dy dx = g(x) dx h(y) dy .

Now check the product of marginals:

Z Z
fX (x)fY (y) = [g(x) · h(y) dy] · [h(y) · g(x) dx] = 1 · f (x, y).

Thus f (x, y) = fX (x)fY (y) for all (x, y). Hence X and Y are independent.

65
Lecture 21

Joint Distribution and Conditional Probabilities

Example
Let X, Y be two continuous random variables with joint probability density function (PDF):

fX,Y (x, y) = x + y, 0 < x < 1, 0 < y < 1.

1. Marginal Distributions

The marginal PDFs are obtained by integration:

Z 1 Z 1
1
fX (x) = fX,Y (x, y) dy = (x + y) dy = x + , 0 < x < 1,
0 0 2
Z 1 Z 1
1
fY (y) = fX,Y (x, y) dx = (x + y) dx = y + , 0 < y < 1.
0 0 2
—

2. Conditional Distributions

The conditional PDFs are:

fX,Y (x, y) x+y

fX|Y (x | y) = = , 0 < x < 1,
fY (y) y + 12

fX,Y (x, y) x+y

fY |X (y | x) = = , 0 < y < 1.
fX (x) x + 12
—

3. Probability Calculations

(a) Joint probability in a rectangle

  Z 1/2 Z 1/4
P 0 < X < 21 , 1
3
<Y < 1
4
= fX,Y (x, y) dy dx.
0 1/3

66
(b) Marginal probability of X
Z 1/2
P (0 < X < 12 ) = fX (x) dx.
0

(c) Conditional probability

 
1

1 3 1
 P 3
< Y < 34 , 0 < X < 1
2
P <Y < 0<X< = .
3 4 2
P (0 < X < 12 )

(d) Point probability

 
P X = 12 , 1
3
<Y < 3
4
= 0,
since point probabilities are zero for continuous random variables.

(e) Marginal point probability

P (X = 12 ) = 0.

(f) Conditional probability with a point event

We cannot directly compute  

1 3 1
P 3
<Y < 4
X= 2
,
0
because it leads to a 0
form. Instead, one can approximate:
  Z 3/4
1 3 1
P 3
<Y < 4
X= 2
= fY |X (y | x = 12 ) dy.
1/3

This is an example of using continuity of the conditional distribution.

—

Expectations and Variances in Transformation Form

1. Definitions
For a function g(x): Z ∞
E[g(X)] = g(x)fX (x) dx.
−∞

67
For joint random variables:
Z ∞ Z ∞
E[g(X, Y )] = g(x, y)fX,Y (x, y) dx dy.
−∞ −∞

Conditional expectation:
Z ∞
E[g(X) | Y = y] = g(x)fX|Y (x | y) dx.
−∞

2. Variance
Var(X) = E[X 2 ] − (E[X])2 ,

 2
Var(X | Y ) = E[X 2 | Y ] − E[X | Y ] .

Covariance and Correlation

Covariance
Definition: For two random variables X and Y , the covariance is defined as
h i
Cov(X, Y ) = E (X − E[X])(Y − E[Y ]) .

This can be simplified as

Cov(X, Y ) = E[XY ] − E[X]E[Y ].

Special Case: Variance

If we set Y = X, then

Cov(X, X) = E[X 2 ] − (E[X])2 = Var(X).

Interpretation
• Cov(X, Y ) > 0: Positive association between X and Y (they tend to move in the same
direction).
• Cov(X, Y ) < 0: Negative association between X and Y (they tend to move in opposite
directions).
• Cov(X, Y ) = 0: No linear association between X and Y .

68
Graphical Interpretation
Given data points (Xi , Yi ) such as:
X = {1, 2, 3, 4, . . . }, Y = {6, 8, 9, . . . },
we can plot (X, Y ) pairs. The relative movement of X̄ and Ȳ (sample means) shows whether
X and Y increase together (positive covariance) or in opposite directions (negative covari-
ance).
—

Correlation Coefficient
Definition: The correlation coefficient ρX,Y (also called Pearson correlation) is a normalized
measure of linear association:
Cov(X, Y )
ρX,Y = q q .
Var(X) Var(Y )

Properties
• −1 ≤ ρX,Y ≤ 1.
• ρX,Y = 1: Perfect positive linear relationship.
• ρX,Y = −1: Perfect negative linear relationship.
• ρX,Y = 0: No linear relationship (but variables may still be dependent).

Connection to Cauchy–Schwarz Inequality

The normalization of covariance into correlation follows from the Cauchy–Schwarz inequality:
q q
| Cov(X, Y )| ≤ Var(X) · Var(Y ).
Dividing both sides gives |ρX,Y | ≤ 1.
—

Summary
• Covariance measures the direction of linear association.
• Correlation standardizes covariance, giving both direction and strength of linear asso-
ciation.
• Variance is a special case of covariance when X = Y .

69
Example A: fX,Y (x, y) = x + y on [0, 1]2
Assume
fX,Y (x, y) = x + y, 0 < x < 1, 0 < y < 1,
and fX,Y (x, y) = 0 elsewhere. (Note this integrates to 1 on [0, 1]2 .)

Marginals
Z 1
fX (x) = (x + y) dy = x + 21 , 0 < x < 1,
0
Z 1
fY (y) = (x + y) dx = y + 12 , 0 < y < 1.
0

Conditionals
x+y
fX|Y (x | y) = , 0 < x < 1,
y + 12
x+y
fY |X (y | x) = , 0 < y < 1.
x + 12

Expectations
Z 1 Z 1 
7

1
E[X] = xfX (x) dx = x x+ 2
dx = .
0 0 12
By symmetry,
7
E[Y ] = .
12
Joint expectation: ZZ
1
E[XY ] = xy(x + y) dx dy = .
[0,1]2 3

Conditional expectation and variance (as functions of y)

Z 1
x+y 3y + 2
E[X | Y = y] = x· 1 dx = .
0 y+2 3(2y + 1)
Z 1
x+y 4y + 3
E[X 2 | Y = y] = x2 · 1 dx = .
0 y+2 6(2y + 1)

Thus
 2 6y 2 + 6y + 1
V(X | Y = y) = E[X 2 | Y = y] − E[X | Y = y] = .
18(4y 2 + 4y + 1)

70
Variance, covariance and correlation
Compute full variances:
11 11
V(X) = E[X 2 ] − (E[X])2 = , V(Y ) = .
144 144
Covariance:
1 7 7 1
C(X, Y ) = E[XY ] − E[X] E[Y ] = − · =− .
3 12 12 144
Correlation:
1
− 144 1
ρX,Y = q =− ≈ −0.0909.
11
· 11 11
144 144

Example B: Triangular domain 0 < x < y < 1 with

fX,Y (x, y) = k(x + y)
Define
fX,Y (x, y) = k(x + y), 0 < x < y < 1,
and fX,Y (x, y) = 0 outside this triangular region. First find the normalization constant k.

Normalization
Integrate k(x + y) over 0 < x < y < 1:
Z 1 Z y
ZZ
k
k(x + y) dx dy = k (x + y) dx dy = .
0<x<y<1 y=0 x=0 2
Set equal to 1 gives k/2 = 1 ⇒ k = 2.
Hence
fX,Y (x, y) = 2(x + y), 0 < x < y < 1.

Marginals
For fixed x (with 0 < x < 1), y ranges x to 1:
Z 1
fX (x) = 2(x + y) dy = −3x2 + 2x + 1, 0 < x < 1.
y=x

For fixed y (with 0 < y < 1), x ranges 0 to y:

Z y
fY (y) = 2(x + y) dx = 3y 2 , 0 < y < 1.
x=0
R1 R1
(One may check 0 fX (x) dx = 0 fY (y) dy = 1.)

71
Conditionals
For 0 < x < y < 1:
2(x + y)
fX|Y (x | y) = , 0 < x < y,
3y 2
2(x + y)
fY |X (y | x) = , x < y < 1.
−3x2 + 2x + 1

Expectations
Compute E[X], E[Y ], E[XY ] over the triangular region:
ZZ
5
E[X] = x · 2(x + y) dx dy =
,
0<x<y<1 12
ZZ
3
E[Y ] = y · 2(x + y) dx dy = ,
0<x<y<1 4
ZZ
1
E[XY ] = xy · 2(x + y) dx dy = .
0<x<y<1 3

Conditional expectations
Z y
2(x + y) 5y
E[X | Y = y] = x· 2
dx = , 0 < y < 1.
0 3y 9
Z 1
2(x + y) 5x2 + 5x + 2
E[Y | X = x] = y· dy = , 0 < x < 1.
x −3x2 + 2x + 1 3(3x + 1)

Conditional variance (example for X | Y )

2
Z y
2(x + y) 7y 2
E[X | Y = y] = x2 · dx = ,
0 3y 2 18
so
7y 2 13y 2
2
5y

V(X | Y = y) = − = .
18 9 162

Variance, covariance and correlation

Full variances:
43 3
V(X) = E[X 2 ] − (E[X])2 = , V(Y ) = .
720 80
Covariance:
1
C(X, Y ) = E[XY ] − E[X] E[Y ] = .
48

72
Correlation:
√
1/48 5 129
ρX,Y = q ≈ 0.440225 (exact simplified form ).
43
· 3 129
720 80

73
Lecture 22

Multiple Random Variables- Several Properties

If we have two random variables X and Y , we can define their joint probability density
function (pdf) as:
fX,Y (x, y)
From this joint pdf, we can compute several summary statistics describing their joint behav-
ior.

Five-Point Summary
From fX,Y (x, y), we can derive the following five important parameters:

E[X], E[Y ], Var(X), Var(Y ), E[XY ]

Note: To compute E[XY ], we require the joint pdf fX,Y (x, y); we cannot compute it using
the marginal distributions alone.

Mean Vector
The mean vector for the two random variables is defined as:
" #
E[X]
µ=
E[Y ]

Covariance
The covariance between X and Y is given by:

Cov(X, Y ) = E[XY ] − E[X]E[Y ]

Covariance measures the direction of linear association between two random variables.

• Cov(X, Y ) > 0: Positive linear association

• Cov(X, Y ) < 0: Negative linear association
• Cov(X, Y ) = 0: No linear association

Remark: The term “linear” is crucial—covariance captures only linear dependence. Non-
linear dependencies may still exist even if covariance is zero.

74
Limitation of Covariance
Covariance does not indicate the strength of the relationship and is not restricted to a
specific range. To overcome this limitation, we use the correlation coefficient.

Correlation Coefficient
The correlation coefficient (also called the Pearson correlation) is defined as:
Cov(X, Y )
ρX,Y = q
Var(X) Var(Y )
It is a normalized measure of linear association that satisfies:
−1 ≤ ρX,Y ≤ 1

Interpretation
• ρX,Y = 1: Perfect positive linear relationship
• ρX,Y = −1: Perfect negative linear relationship
• ρX,Y = 0: No linear relationship (though a non-linear relationship may exist)

Important Note: A zero correlation (ρ = 0) implies no linear relationship, but there

could still be a non-linear dependency between X and Y . Thus, the interpretation of “no
relationship” is only valid in the linear sense.

Independence and Scatter Plot

To examine the relationship between X and Y :

1. Compute ρX,Y
2. Check its significance statistically (e.g., via hypothesis testing)
3. Visualize using a scatter diagram

If ρX,Y = 0, there is no linear relationship, but independence cannot be inferred purely from
this statistic.

Connection to Machine Learning

The concept of measuring association between random variables marks a key foundation for
data-driven learning. The distinction between linear and non-linear relationships forms the
basis for model selection in machine learning, where understanding data dependency is
essential.

75
Perfect Linear Relationship
Let
Y = aX + b
represent a perfect linear relationship between two random variables X and Y .
The correlation between X and Y is defined as:
Cov(X, Y )
ρX,Y = q
Var(X)Var(Y )

We know that:
Cov(X, aX + b) = a Cov(X, X) = a Var(X)
and
Var(aX + b) = a2 Var(X)

Hence, 
a Var(X) a +1, a > 0
ρX,Y =q = =
Var(X) · a2 Var(X) |a| −1, a < 0

Interpretation: If a > 0, X and Y have a perfect positive linear relationship; if a < 0,

they have a perfect negative linear relationship.

Transformation Properties
(A) Expectation of Linear Combination

E[aX + bY + c] = aE[X] + bE[Y ] + c

(B) Variance of Sum

Var(X + Y ) = E[(X + Y )2 ] − [E(X + Y )]2 = Var(X) + Var(Y ) + 2 Cov(X, Y )

This property leads to the concept of diversification in finance. When two datasets (or
assets) are combined, the total variability depends on their covariance. This concept forms
the mathematical basis for portfolio theory (Markowitz Model, you can explore more but
MAL2010 is limited to intro only).

(C) Variance of Linear Combination

Var(aX + bY + c) = a2 Var(X) + b2 Var(Y ) + 2ab Cov(X, Y )

76
(D) Variance of Difference

Var(X − Y ) = Var(X) + Var(Y ) − 2 Cov(X, Y )

(E) Covariance of Linear Combination

Cov(aX + b, cY + d) = ac Cov(X, Y )

Numerical Example
Given:
E[X] = 2, E[Y ] = 1, Var(X) = 4, Var(Y ) = 9, E[XY ] = −2

Then:
Cov(X, Y ) = E[XY ] − E[X]E[Y ] = −2 − (2)(1) = −4

The correlation is:

Cov(X, Y ) −4 2
ρX,Y = q =√ =−
Var(X)Var(Y ) 4×9 3

Interpretation: There exists a negative linear relationship between X and Y with

moderate strength.
Now, consider:

Cov(2X + 1, 3Y − 1) = (2)(3)Cov(X, Y ) = 6(−4) = −24

Covariance Matrix
For two random variables X and Y , the covariance matrix is defined as:
" #
Var(X) Cov(X, Y )
Σ=
Cov(Y, X) Var(Y )

Using variance–covariance notation:

" #
2
σX σXY
Σ=
σXY σY2

Properties:
• Generalizable to n variables also.
• Variance is always non-negative: Var(X) ≥ 0
• Covariance matrix is symmetric: Σ = ΣT
• Covariance matrix is positive semi-definite

77
Independence of Random Variables
If X and Y are independent:
fX,Y (x, y) = fX (x) fY (y)
Then: ZZ
E[XY ] = xy fX (x)fY (y) dx dy = E[X]E[Y ]
Hence:
Cov(X, Y ) = 0

Conclusion: Independence implies zero covariance, but the converse is not necessarily true.

Example: Nonlinear Relationship with Zero Covariance

Let X ∼ U (−1, 1) with pdf:
1
fX (x) = , −1 ≤ x ≤ 1
2
and let Y = X 2 .
Then E[X] = 0, but:

Cov(X, Y ) = E[XY ] − E[X]E[Y ] = E[X 3 ] − 0 = 0

Even though Y is a deterministic function of X (parabolic relation), covariance is zero.

Hence:
Cov(X, Y ) = 0 ̸⇒ X and Y are independent
Covariance captures only linear dependence, not nonlinear relationships.

Key Takeaways
• Perfect linear relationship: ρ = ±1
• Zero correlation implies no linear relation, but not necessarily independence
• Variance–covariance structure forms the basis for multivariate data representation
• Independence assumptions simplifies computation: E[XY ] = E[X]E[Y ]

78
Lecture 23

Transformation of Two Random Variables

Let X and Y be two continuous random variables with joint probability density function
(pdf):
fX,Y (x, y)
The objective is to study the transformation of these random variables into a new pair (U, V ).

Transformation Equations
Define two new random variables:

U = g1 (X, Y ), V = g2 (X, Y )

The transformation functions g1 and g2 should satisfy the following conditions:

• The transformation is one-to-one (invertible)
• The partial derivatives involved are continuous
• The Jacobian determinant of the transformation is nonzero
If the transformation is invertible, then there exist inverse functions:

X = h1 (U, V ), Y = h2 (U, V )

Objective
Our goal is to find the joint pdf of the transformed random variables (U, V ), denoted by
fU,V (u, v), from the given fX,Y (x, y).

Principle of Transformation
The fundamental requirement is that the total probability remains invariant under transfor-
mation: ZZ ZZ
fX,Y (x, y) dy dx = fU,V (u, v) dv du
RXY RU V

where RXY and RU V denote the regions of support for (X, Y ) and (U, V ), respectively.

Jacobian Transformation
The change of variables from (x, y) to (u, v) is carried out using the Jacobian of the trans-
formation.

79
Definition: Jacobian

The Jacobian of transformation from (U, V ) to (X, Y ) is given by:

∂x ∂x
∂u ∂v
J=
∂y ∂y
∂u ∂v
Then, the joint pdf of (U, V ) is expressed as:
 
fU,V (u, v) = fX,Y h1 (u, v), h2 (u, v) × |J|
where |J| denotes the absolute value of the Jacobian determinant.

Example: Cartesian to Polar Transformation

Let:
X = r cos θ, Y = r sin θ
Then the Jacobian is:
∂x ∂x
∂r ∂θ cos θ −r sin θ
J= = = r(cos2 θ + sin2 θ) = r
∂y ∂y sin θ r cos θ
∂r ∂θ
Thus,
dx dy = |J| dr dθ = r dr dθ

Hence, if fX,Y (x, y) is known, then the pdf of (R, Θ) is:

fR,Θ (r, θ) = fX,Y (r cos θ, r sin θ) · r

General Expression for Transformation

For a general transformation:
U = g1 (X, Y ), V = g2 (X, Y )
and its inverse:
X = h1 (U, V ), Y = h2 (U, V )
the corresponding density function is:
∂h1 ∂h1
 
∂u ∂v
fU,V (u, v) = fX,Y h1 (u, v), h2 (u, v) ×
∂h2 ∂h2
∂u ∂v
The region of integration (support) must also be transformed accordingly.

80
Real-World Interpretation
A physical example of such transformation occurs in mechanics: Consider two particles
moving randomly with velocity components X and Y . If we define:
√ −1

Y

U = velocity magnitude = X2 +Y 2, V = angle of motion = tan
X
then the transformation from (X, Y ) to (U, V ) allows us to derive the joint density of kinetic
energy and direction.

Summary of Conditions for Valid Transformation

For a valid transformation from (X, Y ) to (U, V ):

1. The functions g1 (x, y) and g2 (x, y) must be one-to-one and differentiable.

2. The inverse functions h1 (u, v), h2 (u, v) must exist.

3. The Jacobian determinant

∂x ∂x
∂u ∂v
J= ̸= 0
∂y ∂y
∂u ∂v
must exist and be nonzero.

4. The transformed pdf fU,V (u, v) should integrate to 1 over the support of (U, V ).

Takeaway
• Transformations help in expressing complex relationships between random variables in
simpler coordinates.
• The Jacobian determinant acts as a scaling factor that preserves total probability.
• Common transformations include Cartesian to Polar, and in higher dimensions, trans-
formations to Spherical or Cylindrical coordinates.

Transformation of Two Random Variables — Worked

Examples

Example 1: Circularly symmetric density (polar coordinates)

Suppose the joint pdf is given (candidate) by
1  
fX,Y (x, y) = exp − x2 − y 2 , x, y ∈ R.
2π

81
Remark (normalization). Check the integral
ZZ
1 −(x2 +y2 ) 1 Z 2πZ ∞ −r2 1 1 1
e dx dy = e r dr dθ = · 2π · = ,
R2 2π 2π 0 0 2π 2 2
so the constant 1/(2π) does not normalize this pdf to 1. Two commonly used normalized
forms are:
1 −(x2 +y2 ) 1 − 1 (x2 +y2 )
(A) fX,Y (x, y) = e , (B) fX,Y (x, y) = e 2 .
π 2π
Both of the above integrate to 1 on R2 . (Form B is the standard bivariate normal with
independent N (0, 1) components.)
Now transform to polar coordinates:

X = r cos θ, Y = r sin θ, r ≥ 0, θ ∈ [0, 2π).

Jacobian: |J| = r. Thus for case (A) the transformed joint density is
1 −r2
fR,Θ (r, θ) = fX,Y (r cos θ, r sin θ) · r = e · r, r ≥ 0, θ ∈ [0, 2π).
π

Example 2: Exponential-type pdf with transform

Let
fX,Y (x, y) = e−(x+y) , x > 0, y > 0.
Define
Y
U = X + Y, V = .
X +Y
Solve for the inverse:
X = U (1 − V ), Y = U V.
Jacobian:
∂x/∂u ∂x/∂v 1 − v −u
J= = = u(1 − v) + uv = u.
∂y/∂u ∂y/∂v v u
So |J| = u. The transformed joint pdf is
 
fU,V (u, v) = fX,Y u(1 − v), uv · |J| = e−u · u, u > 0, 0 < v < 1.
 
Note this factors as fU,V (u, v) = ue−u · 1, so U and V are independent, with

fU (u) = ue−u , u > 0 (Gamma(2,1) pdf), fV (v) = 1, 0 < v < 1.

R∞ R1
Verify normalization: 0 ue−u du = 1 and 0 1 dv = 1, so total mass is 1.

82
Example 3: Transformation Example
If we have the joint density over X, Y as

fXY (x, y) = 1,

where x ∈ (0, 1) and y ∈ (1, 2), and we let

u = x + y, v = x − y,

then we can write

u+v u−v
x= , y= .
2 2
Now, the Jacobian is calculated as
∂x ∂x 1 1
J = ∂u ∂v = 2 2 = − 1 .
∂y ∂y 1 1 2
−
∂u ∂v 2 2
Hence,
1
|J| = .
2
Now, fU V (u, v) = fXY (x, y) × |J| = 12 , and we must find the range of u and v.
The given rectangular region for x, y is:

0 ≤ x ≤ 1, 1 ≤ y ≤ 2.

We compute the transformed corner points:

(0, 1) → (1, −1), (1, 1) → (2, 0), (1, 2) → (3, −1), (0, 2) → (2, −2).

Thus, the region in the u, v-plane is a quadrilateral with vertices (1, −1), (2, 0), (3, −1), (2, −2).
We can also describe the region analytically as:

− u ≤ v ≤ u − 2, 1 ≤ u ≤ 2,

u − 4 ≤ v ≤ 2 − u, 2 ≤ u ≤ 3.

Hence,
1,

(u, v) in the region defined above,

fU V (u, v) = 2
0,

otherwise.

To verify validity, we can integrate to check normalization:

Z 2
1 Z 3
1
[(u − 2) − (−u)] du + [(2 − u) − (u − 4)] du = 1.
1 2 2 2

83
If we have to find the marginal with respect to u, then it will be





u − 1, 1 ≤ u ≤ 2,

fU (u) =

3 − u, 2 ≤ u ≤ 3,


0,

otherwise.

Similarly, for fV (v), we can go back to the graphical representation of the rectangular region,
plot the limits accordingly, and then calculate the marginal with respect to v.

84
Lecture 24
Topic: Conditional Expectation, Regression,
and Covariance Structure

1. Mean Vector and Covariance Matrix

1.1 (a) For Two Random Variables

Let " #
X1
X= .
X2
The mean vector is " # " #
E[X1 ] µ1
E[X] = = .
E[X2 ] µ2
The covariance matrix is defined as:
" #
h
T
i Var(X1 ) Cov(X1 , X2 )
Σ = E (X − E[X])(X − E[X]) = .
Cov(X2 , X1 ) Var(X2 )

1.2 (b) For Three Random Variables

Let    
X1 µ1
X2  ,
X= µ2  .
E[X] = 
 

X3 µ3
Then  
Var(X1 ) Cov(X1 , X2 ) Cov(X1 , X3 )
 
Cov(X2 , X1 )
Σ= Var(X2 ) Cov(X2 , X3 )
.
Cov(X3 , X1 ) Cov(X3 , X2 ) Var(X3 )
Notes:
• Σ is symmetric: Σ = ΣT .
• Σ is positive semi-definite.

2. Correlation Coefficient
For two random variables X and Y ,
Cov(X, Y )
ρXY =
σX σY

85
 2
where σX = Var(X) and σY2 = Var(Y ).

Theorem 5. For any two random variables X, Y with finite and nonzero variances,

−1 ≤ ρXY ≤ 1,

where
Cov(X, Y )
ρXY = .
σX σY

Proof. We use the fact that for any real numbers a, b ∈ R,

Var(aX + Y ) ≥ 0.

Expanding the variance,

Var(aX + Y ) = a2 Var(X) + 2a Cov(X, Y ) + Var(Y ).

This quadratic in a must be nonnegative for all a, hence its discriminant must be less than
or equal to zero.

∆ = (2 Cov(X, Y ))2 − 4 Var(X) Var(Y ) ≤ 0.

Simplifying,
Cov(X, Y )2 ≤ Var(X) Var(Y ).
2 2
Dividing both sides by σX σY ,
!2
Cov(X, Y )
≤ 1.
σX σY
Hence,
−1 ≤ ρXY ≤ 1.

Equality condition: Equality occurs when the discriminant ∆ = 0, i.e., when

Y = aX + b

for some real constants a, b, meaning Y is a perfect linear function of X.

2.1 Zero Correlation Does Not Imply Independence

Statement: If X and Y are independent, then ρXY = 0(Prove yourself). However, the
converse is not true: uncorrelated variables may still be dependent.

86
Example 12 (Counterexample). Let X be uniformly distributed on the interval [−1, 1], i.e.,

1,

−1 ≤ x ≤ 1,

fX (x) = 2
0,

otherwise.

Define a new variable Y = X 2 .

Then:
1
E[X] = 0, E[Y ] = E[X 2 ] = .
3
Also,
E[XY ] = E[X 3 ] = 0,
since X 3 is an odd function symmetric about zero.
Hence,
Cov(X, Y ) = E[XY ] − E[X]E[Y ] = 0.
Thus, the correlation coefficient is
ρXY = 0.

However, Y = X 2 is completely determined by X; that is, Y depends nonlinearly on X.

Therefore, X and Y are dependent, even though their correlation is zero.

Uncorrelated does not imply independent.

3. Conditional Expectation
Definition 11. Let X and Y be random variables. The conditional expectation of Y given
X is the random function
E(Y | X) = g(X),
where for each fixed value x,
X


 y P (Y = y | X = x), discrete Y,
g(x) := E(Y | X = x) = Zy ∞
y fY |X (y | x) dy, continuous Y.



−∞

fX,Y (x, y)
where fY |X (y|x) = .
fX (x)
Remark 3. E(Y | X) is a measurable function of X. It is itself a random variable (a
function of X).

87
3.1 Properties
h i
• E E(Y | X) = E(Y ).
• For any (suitable) function h,
h i h i
E h(X) E(Y | X) = E h(X)Y .

• E(Y | X) minimizes mean squared prediction error among all functions of X.

4. Best Predictor (MSE sense)

Consider prediction of Y by a function g(X). The mean square error (MSE) is
h i
MSE(g) := E (Y − g(X))2 .

Theorem 6. Among all measurable functions g(X), the conditional expectation g ∗ (X) :=
E(Y | X) minimizes the MSE:
h i
g ∗ = arg min
g
E (Y − g(X))2 .

Sketch of proof. Write

   
Y − g(X) = Y − E(Y | X) + E(Y | X) − g(X) .

Square and take expectation:

h i h i h i
E[(Y −g(X))2 ] = E (Y −E(Y | X))2 +E (E(Y | X)−g(X))2 +2E (Y −E(Y | X))(E(Y | X)−g(X)) .

The cross-term is zero because conditioning on X gives

h i  h i 
E (Y −E(Y | X))(E(Y | X)−g(X)) = E E (Y −E(Y | X)) | X (E(Y | X)−g(X)) = 0.

Thus h i h i
E[(Y − g(X))2 ] = E (Y − E(Y | X))2 + E (E(Y | X) − g(X))2 .
The first term does not depend on g; the second term is minimized (to zero) by taking
g = E(Y | X).

5. Simple Linear Regression (conditional mean as linear

function)
A commonly used parametric form for the conditional mean is

E(Y | X = x) = α + βx.

88
This motivates the regression model
Y = α + βX + ε,
where ε is a zero-mean error satisfying E(ε | X) = 0.
The parameters α, β are chosen to minimize
h i
E (Y − α − βX)2 .

5.1 Solution for the linear least-squares coefficients

Minimize w.r.t. α, β. Equivalently, solve the normal equations (set derivatives to zero):

E[Y − α − βX] = 0,
h i
E X(Y − α − βX) = 0.
From the first, α + βE(X) = E(Y ). From the second,
E[XY ] − αE[X] − βE[X 2 ] = 0.
Solving gives
Cov(X, Y )
β= , α = E(Y ) − βE(X).
Var(X)
Equivalently, the linear conditional mean can be written as
Cov(X, Y )  
E(Y | X) = E(Y ) + X − E(X) .
Var(X)

5.2 Results:
• Cov(X, ε) = 0.
• Var(Y ) = β 2 Var(X) + Var(ε).
• If the correlation ρ(X, Y ) = 0, then β = 0 and the best linear predictor is constant;
but ρ = 0 does not imply independence.

6. Example (joint density f (x, y) = k(x + y) on [0, 1]2)

Let
f (x, y) = k(x + y), 0 ≤ x ≤ 1, 0 ≤ y ≤ 1,
and f (x, y) = 0 elsewhere. First find k.

ZZ Z 1Z 1 Z 1   
1 1 1
1= f (x, y) dy dx = k (x + y) dy dx = k x+ 2
dx = k 2
+ 2
= k · 1.
[0,1]2 0 0 0

Hence k = 1. Thus f (x, y) = x + y on the unit square.

89
6.1 Marginal of X
Z 1
fX (x) = (x + y) dy = x + 21 , 0 ≤ x ≤ 1.
0

6.2 Conditional density fY |X (y | x)

f (x, y) x+y
fY |X (y | x) = = , 0 ≤ y ≤ 1.
fX (x) x + 12

6.3 Conditional expectation E(Y | X = x)

Compute Z 1
x+y 1 Z 1 
E(Y | X = x) = y· 1 dy = 1 xy + y 2 dy.
0 x+ 2 x+ 2 0

Now Z 1
1 Z 1
1
xy dy = x · , y 2 dy = ,
0 2 0 3
so
3x + 2
x/2 + 1/3 6 3x + 2
E(Y | X = x) = = 1 = .
x + 1/2 x+ 6x + 3
2
Therefore, the conditional expectation (best predictor) is

3X + 2
E(Y | X) = (0 ≤ X ≤ 1).
6X + 3

Remark 4. This example demonstrates computing a conditional expectation from a joint

density: first normalize the joint pdf, find the marginal fX , then compute fY |X and integrate
yfY |X (y | x).

7. Geometric viewpoint (optional)

The conditional expectation E(Y | X) can be interpreted as the orthogonal projection of
Y onto the space of functions measurable w.r.t. X. The linear regression predictor (least-
squares linear) is the projection of Y onto the linear span of {1, X}. Projection properties
explain many identities (e.g. residual orthogonality).

90
Lecture 25
Topic: Regression and Bivariate Normal Dis-
tribution

1. Prediction and Regression

Let X and Y be two random variables. We denote the conditional expectation

E(Y |X) = Yp ,

where Yp is the predicted value of Y for a given X.

• If Y is categorical, the process is called classification.

• If Y is continuous, the process is called regression.

Thus,
E(Y |X) = Yp = best predictor of Y.

X −→ g(X) −→ Yp .
If g(X) is linear, it leads to linear regression.

2. Parametric and Non-Parametric Estimation

Given data (xi , yi ), estimation can be:
• Parametric: assume a specific form of the density f (x, y) (e.g., Normal).
• Non-Parametric: no assumption on density form.

3. Example: Conditional Expectation

Given joint density
f (x, y) = (x + y), 0 < x < 1, 0 < y < 1,
we compute Z 1
E(Y |X = x) = yf (y|x) dy,
0
where Z 1
x+y
fX (x) = (x + y)dy = x + 21 , fY |X (y|x) = .
0 x + 12

91
Hence,
x 1
1 Z 1
2
+ 3 x/2 + 1/3
E(Y |X = x) = 1 y(x + y) dy = 1 = .
x+ 2 0 x+ 2
x + 1/2

4. Simple Linear Regression Model

Assume a linear relation:
Yp = α + βX.
The objective is to choose α and β such that the prediction error is minimum.
The total squared error:
n n
ε2 = (yi − yp (i))2 = (yi − α − βxi )2 .
X X

i=1 i=1

We minimize ε2 with respect to α and β.

5. Least Squares Estimation

Set partial derivatives to zero:
∂ε2 ∂ε2
= 0, = 0.
∂α ∂β
This gives the normal equations:
X
(yi − α − βxi ) = 0,
X
xi (yi − α − βxi ) = 0.

Simplifying: P P
 yi
 = nα + β xi ,
 P xi y i P P 2
=α xi + β x.

i

Solving for α and β:

xi yi − ( xi )( yi )
P P P
n
β= , α = ȳ − β x̄.
n x2i − ( xi )2
P P

Thus, the regression line is

Yp = α + βX.

92
5.1 Graphical Interpretation of Simple Linear Regression

(A) (B)
10 10
Best fit: y = 0.96x + 0.10
(x1 , y1 )

r1 = y1 − b
y1
5 5
y

y
(x1 , b
y1 )

0 0

0 2 4 6 8 10 0 2 4 6 8 10
x x

Figure 2: (A) Scatter plot showing observed data points, candidate regression lines, and the
best fit line with a confidence band. (B) Illustration of a single observation and its residual
r1 = y1 − yb1 projected onto the regression line.

6. Relation with Covariance

Cov(X, Y )
β= .
Var(X)

Hence,
Cov(X, Y )
Yp = E(Y ) + (X − E(X)).
Var(X)

7. Bivariate Normal Distribution

If (X, Y ) jointly follow a normal distribution with parameters (µ1 , µ2 , σ1 , σ2 , ρ), we write:

(X, Y ) ∼ BN (µ1 , µ2 , σ1 , σ2 , ρ).

The joint probability density function is:

" ( )#
1 1 (x − µ1 )2 (x − µ1 )(y − µ2 ) (y − µ2 )2
f (x, y) = √ exp − − 2ρ + .
2πσ1 σ2 1 − ρ 2 2(1 − ρ2 ) σ12 σ1 σ2 σ22

x, y ∈ R.

93
8. Independence and Correlation in the Bivariate Nor-
mal Case
If ρ = 0, " !#
1 1 (x − µ1 )2 (y − µ2 )2
f (x, y) = exp − + .
2πσ1 σ2 2 σ12 σ22
This can be factorized as
f (x, y) = g(x) h(y),
showing that X and Y are independent.
However, note:
ρ = 0 ̸⇒ X ⊥ Y (in general),
but
ρ=0⇒X⊥Y only for the bivariate normal case.

Remark 5. For the bivariate normal distribution, the only possible dependence between X
and Y is linear.

94
Lecture 26
Topic: Bivariate Normal Distribution and Con-
ditional Expectation

1. Conditional Expectation and Bivariate Normal Dis-

tribution
Recall: In regression, we model the expected value of Y given X as
E(Y |X) = g(X)
For a linear relation,
Ŷp = α + βX
where
Cov(X, Y )
β=
Var(X)

2. Bivariate Normal Distribution

If (X, Y ) jointly follow a bivariate normal distribution, it is denoted as
(X, Y ) ∼ BN (µ1 , µ2 , σ1 , σ2 , ρ)
where ρ is the correlation coefficient between X and Y .
The joint probability density function (p.d.f.) is
" ( )#
1 1 (x − µ1 )2 (x − µ1 )(y − µ2 ) (y − µ2 )2
f (x, y) = √ exp − − 2ρ +
2πσ1 σ2 1 − ρ 2 2(1 − ρ2 ) σ12 σ1 σ2 σ22

3. Marginal Distributions
If (X, Y ) ∼ BN (µ1 , µ2 , σ12 , σ22 , ρ), then
X ∼ N (µ1 , σ12 ) and Y ∼ N (µ2 , σ22 )

3.1 Proof: Marginal distributions of the Bivariate Normal

Let (X, Y ) ∼ BN (µ1 , µ2 , σ12 , σ22 , ρ) with joint p.d.f.

" ( )#
1 1 (x − µ1 )2 (x − µ1 )(y − µ2 ) (y − µ2 )2
f (x, y) = √ exp − − 2ρ + .
2πσ1 σ2 1−ρ 2 2(1 − ρ2 ) σ12 σ1 σ2 σ22

95
The marginal density of X is obtained by integrating out Y :
Z ∞
fX (x) = f (x, y) dy.
−∞

Step 1: Complete the square in y:

(y − µ2 )2 (x − µ1 )(y − µ2 ) 1 σ2 2 ρ2 (x − µ1 )2
 
− 2ρ = y − µ 2 − ρ (x − µ 1 ) − .
σ22 σ1 σ2 σ22 σ1 σ12

Step 2: Substitute and simplify the exponent:

" #
1 (x − µ1 )2
fX (x) = √ exp −
2πσ1 σ2 1 − ρ2 2σ12
  2 
Z ∞
1 y − µ2 − ρ σσ21 (x − µ1 ) 
× exp−  dy.

−∞ 2(1 − ρ2 ) σ22

Step 3: Recognize the remaining integral as a normalizing constant:

σ
(y − µ2 − ρ σ21 (x − µ1 ))2
Z ∞ " #
1 q
exp − dy = 2π(1 − ρ2 ) σ2 .
−∞ 2(1 − ρ2 ) σ22

Step 4: Substitute back:

" #
1 (x − µ1 )2
fX (x) = √ exp − .
2πσ1 2σ12

Thus,
X ∼ N (µ1 , σ12 ) .
By symmetry, integrating out X gives

Y ∼ N (µ2 , σ22 ) .

4. Conditional Distribution of Y |X
The conditional distribution of Y given X = x is also normal:
σ2
 
2 2
Y |X = x ∼ N µ2 + ρ (x − µ1 ), σ2 (1 − ρ )
σ1

Therefore,
σ2
E(Y |X = x) = µ2 + ρ (x − µ1 )
σ1
and
Var(Y |X = x) = σ22 (1 − ρ2 )

96
4.1 Derivation: conditional law Y | X = x
The conditional density is
f (x, y)
f (y | x) = .
fX (x)

When we divide the joint density by fX (x) those x-only terms cancel, leaving the conditional
density in the Gaussian form
( 2 )
1 1 σ2

f (y | x) = q exp − 2
y − µ2 − ρ (x − µ1 ) .
2π σ22 (1 − ρ2 ) 2 σ2 (1 − ρ )
2 σ1

Therefore Y | X = x is normal with

σ2
E(Y | X = x) = µ2 + ρ (x − µ1 ) , Var(Y | X = x) = σ22 (1 − ρ2 ) .
σ1

NOTE: the conditional mean is linear in x; the conditional variance does not depend on x
and is reduced by the factor 1 − ρ2 .

5. Regression Function from Conditional Expectation

From the above,
σ2 σ2
E(Y |X) = (µ2 − ρ µ1 ) + ρ X
σ1 σ1
Comparing with Ŷ = α + βX,
σ2 σ2
α = µ2 − ρ µ1 , β=ρ
σ1 σ1
Hence, for a bivariate normal model, the regression function is linear in X.

6. Independence and Correlation

If ρ = 0, then
f (x, y) = fX (x) fY (y)
which implies that X and Y are independent.

X ⊥ Y ⇐⇒ ρ = 0

97
7. Example
Let
(X, Y ) ∼ BN (0, 1, 1, 4, 0.5)
Then
2
E(Y |X = x) = 1 + 0.5 (x − 0) = 1 + x
1
and
Var(Y |X = x) = 4(1 − 0.25) = 3
So
Y |X = x ∼ N (1 + x, 3)

8. Sum of Normal Variables

If X and Y are jointly normal, then any linear combination is also normal. For example,
 
Z = 2X + Y ⇒ Z ∼ N 2E(X) + E(Y ), 4 Var(X) + Var(Y ) + 4 Cov(X, Y )

9. Multivariate Normal Distribution

For n variables X1 , X2 , . . . , Xn , the multivariate normal density is given by
1 1
 
f (x) = exp − (x − µ)T Σ−1 (x − µ)
(2π)n/2 |Σ|1/2 2

where µ and Σ are the mean vector and the covariance matrix, respectively.

98
Lecture 27
Topic: Statistical Inference and Estimation
Population and Sample
• A population refers to the entire collection of individuals or observations of interest.
• Since it is often not possible to observe the whole population, we select a sample.
• The sample is used to estimate characteristics (parameters) of the population.

Population → Sample → Estimator

Before collection: a sample is considered a collection of random variables. After collec-

tion: the sample becomes a set of observed values.

Definition of a Random Sample

A random sample of size n is a collection of i.i.d. (independent and identically distributed)
random variables:
X1 , X2 , . . . , Xn
where each Xi follows the same distribution f (xi ; θ) and is independent of the others.

n
Y
f (x1 , x2 , . . . , xn ) = f (xi ; θ)
i=1

Example
For 50 students, let Xi represent whether the ith student plays cricket:

1,if the student plays cricket,
Xi = 
0, otherwise.

Then X1 , X2 , . . . , X50 form a random sample of Bernoulli variables.

Estimation
Given a random sample (X1 , X2 , . . . , Xn ):
• If the form of f (xi ; θ) is known, the estimation is called parametric.
• If the form of f (xi ; θ) is unknown, the estimation is non-parametric.
Goal: To estimate the population parameter θ using the sample.

99
Statistics and Estimators
• A statistic is any function of the random sample:

T = g(X1 , X2 , . . . , Xn )

and it should not involve any unknown parameter.

• A statistic that is used to estimate a population parameter is called an estimator.
• The estimate is the numerical value of the estimator based on observed data.

Central Limit Theorem (CLT)

Let X1 , X2 , . . . , Xn be a random sample such that

E(Xi ) = µ, V ar(Xi ) = σ 2 .

Then as n → ∞, Pn
X − nµ
Z= i=1
√i −→ N (0, 1).
nσ

That is, the standardized sum of a large number of independent random variables tends
toward a standard normal distribution.

Moment Generating Function (MGF) Proof of CLT (for Xi ∼ N (0, 1))

Pn
X
Z= √ i
i=1
n
Then,
 P √  n  √  √
MZ (t) = E(etZ ) = E et Xi / n
E etXi / n
= [MXi (t/ n)]n
Y
=
i=1

2 /2
Since MXi (t) = et for Xi ∼ N (0, 1),
h 2 /(2n)
in 2 /2
MZ (t) = et = et

Hence,
Z ∼ N (0, 1).

100
Lecture 28
Topic: Sampling, Estimation and Properties of
Estimators
Sampling Framework
Let X1 , X2 , . . . , Xn be a random sample such that
Xi ∼ f (xi ; θ),
where θ is an unknown population parameter.
The objective is to construct an estimator g(X1 , X2 , . . . , Xn ) of θ.

Central Limit Theorem (CLT) — Revisited

If X1 , X2 , . . . , Xn are independent random variables such that
E(Xi ) = µ, V ar(Xi ) = σ 2 ,
then for large n (n ≥ 30),
Pn Pn
i=1 Xi − E( i=1 Xi )
q Pn ∼ N (0, 1).
V ar( i=1 Xi )

Since X  X 
E Xi = nµ, V ar Xi = nσ 2 ,
we have
Xi − nµ
P
Z= √ ∼ N (0, 1).
nσ

Example using CLT

Suppose Xi ∼ B(1, 41 ), i.e., Xi is a Bernoulli variable with p = 14 .
Then for n = 80,
80
!
X 1
E Xi = 80 × = 20,
i=1 4
80
!
X 1 3
V ar Xi = 80 × × = 15.
i=1 4 4

Hence,
80
X
Xi ∼ N (20, 15).
i=1

101
To find: !
X 18 − 20 27 − 20
P (18 ≤ Xi ≤ 27) = P √ ≤Z≤ √ .
15 15

Another Example
Let X1 , X2 , . . . , X100 be a random sample with

E(Xi ) = 6.5, V ar(Xi ) = 4.

Then, X
P (50 ≤ Xi ≤ 60) =?
P
can be solved using CLT by standardizing the variable Xi .
Solution:
X  X 
E Xi = 100(6.5) = 650, V ar Xi = 100(4) = 400,
√
SD = 400 = 20.

Using the Central Limit Theorem,

Xi − 650
P
Z= .
20

Hence,
50 − 650 60 − 650
X  
P (50 ≤ Xi ≤ 60) = P ≤Z≤ = P (−30 ≤ Z ≤ −29.5).
20 20

Since these Z-values are far into the left tail of the standard normal curve,

P (−30 ≤ Z ≤ −29.5) ≈ 0.

Therefore,
X
P (50 ≤ Xi ≤ 60) ≈ 0.

Estimation of Parameters
If X1 , X2 , . . . , Xn ∼ f (xi ; θ), then θ represents the population parameter.
An estimator is a function of the sample:

g(X1 , X2 , . . . , Xn ) → θ̂

which is used to estimate the actual parameter θ.

102
Properties of a Good Estimator
(i) Unbiasedness

An estimator g(X1 , X2 , . . . , Xn ) is called unbiased for θ if

E[g(X1 , X2 , . . . , Xn )] = θ.

Example: Let X1 , X2 , . . . , Xn ∼ N (µ, σ 2 ).

Consider:
X1 + X 2 X1 + X2 + · · · + Xn
g1 = , g2 = .
2 n
Then,
1 1X
 
E(g1 ) = [E(X1 ) + E(X2 )] = µ, E(g2 ) = E Xi = µ.
2 n
Hence, both g1 and g2 are unbiased estimators of µ.

(ii) Efficiency

Among two unbiased estimators g1 and g2 of θ, the estimator with smaller variance is said
to be more efficient.

V ar(g1 ) < V ar(g2 ) ⇒ g1 is more efficient.

Example:
X1 + X 2 1 1
 
V ar(g1 ) = V ar = [V ar(X1 ) + V ar(X2 )] = σ 2 ,
2 4 2
2
X1 + X2 + · · · + Xn σ
 
V ar(g2 ) = V ar = .
n n
Hence g2 is more efficient than g1 .

(iii) Minimum Variance Unbiased Estimator (MVUE)

If an estimator is unbiased and has the smallest possible variance among all unbiased esti-
mators, it is called a Minimum Variance Unbiased Estimator (MVUE).
If this property holds for all values of θ, it is called the Uniform Minimum Variance
Unbiased Estimator (UMVUE).

103
Lecture 29
Topic: Sample Statistics and Chi-Square Dis-
tribution
Random Sample and Estimators
Let X1 , X2 , . . . , Xn be a random sample from a population with p.d.f. f (xi ; θ), where θ is
an unknown parameter.
The estimator is defined as a statistic:

g(X1 , X2 , . . . , Xn ) → θ̂.

If
E[θ̂] = θ,
then θ̂ is said to be an unbiased estimator of θ.
Among unbiased estimators, if
V ar(g1 ) ≤ V ar(g2 ),
then g1 is said to be a more efficient estimator than g2 .

Sample Statistics
(i) Sample Mean
n
1X
X̄ = Xi
n i=1

(ii) Sample Variance

n
2 1 X
S = (Xi − X̄)2
n − 1 i=1

Properties of Sample Mean

Let X1 , X2 , . . . , Xn be a random sample such that Xi ∼ N (µ, σ 2 ).
Then:
1X 1X
 
E(X̄) = E Xi = E(Xi ) = µ,
n n
σ2
V ar(X̄) = .
n
104
Hence, X̄ is an unbiased estimator of µ.
Moreover,
X̄ − µ
√ ∼ N (0, 1).
σ/ n

Case 1: Large Samples

If the sample size n is large, by the Central Limit Theorem:
Xi − nµ
P
√ → N (0, 1),
nσ

or equivalently,
X̄ − µ
√ → N (0, 1).
σ/ n

Case 2: Small Samples

When the sample size n is small, the variability of X̄ depends on the sample variance S 2 .
The sample variance is defined as
n
1 X
S2 = (Xi − X̄)2 .
n − 1 i=1

Expanding (Xi − X̄)2 :

(Xi − X̄)2 = [(Xi − µ) − (X̄ − µ)]2 = (Xi − µ)2 + (X̄ − µ)2 − 2(Xi − µ)(X̄ − µ).

Summing over i and simplifying:

n n
(Xi − X̄)2 = (Xi − µ)2 − n(X̄ − µ)2 .
X X

i=1 i=1

Taking expectation:

1 1 σ2
E(S 2 ) = [nσ 2 − nV ar(X̄)] = [nσ 2 − n ] = σ 2 .
n−1 n−1 n

Hence,
S 2 is an unbiased estimator of σ 2 .

105
Distribution of Sample Variance
If Xi ∼ N (µ, σ 2 ), then
Xi − µ
Zi = ∼ N (0, 1).
σ
Now define
Y = Z 2.

Then the p.d.f. of Y is

1
fY (y) = √ e−y/2 , y > 0.
2πy

This is the Chi-Square distribution with 1 degree of freedom (χ21 ).

Chi-Square Distribution for Sample Variance

If Y1 , Y2 , . . . , Yn are i.i.d. χ2 (1) random variables, then
n
Yi ∼ χ2 (n).
X

i=1

Hence, for a normal sample X1 , X2 , . . . , Xn ∼ N (µ, σ 2 ),

(n − 1)S 2
∼ χ2(n−1) .
σ2

106