Study Guide: Random Variables and
Stochastic Processes
Author: Arush Prakash
Module 4: Random Variables and Stochastic Processes
This module introduces the concepts of probability, random variables, and stochastic
processes, which are crucial for analyzing and designing systems that deal with uncertain or
noisy signals, common in communication systems.
Key Concepts Covered:
● Probability:
○ Basic Definitions: Experiment, Sample Space, Event.
○ Axioms of Probability:
■ P(A)≥0 for any event A.
■ P(S)=1 (where S is the sample space).
■ For mutually exclusive events A1,A2,…, P(A1∪A2∪…)=P(A1)+P(A2)+….
○ Conditional Probability: P(A∣B)=P(B)P(A∩B)(Probability of A given B).
○ Statistical Independence: Events A and B are independent if P(A∩B)=P(A)P(B),
or equivalently, P(A∣B)=P(A).
● Random Variables (RVs):
○ A function that maps outcomes from a sample space to real numbers.
○ Discrete Random Variables: Take on a finite or countably infinite number of
values.
■ Probability Mass Function (PMF): PX(xi)=P(X=xi). Sum of all PMF values is
1.
○ Continuous Random Variables: Can take any value within a continuous range.
■ Probability Density Function (PDF): fX(x) such that P(a≤X≤b)=∫abfX(x)dx.
Integral of PDF over all space is 1. fX(x)≥0.
○ Cumulative Distribution Function (CDF): FX(x)=P(X≤x).
■ Properties: 0≤FX(x)≤1, non-decreasing, FX(−∞)=0, FX(∞)=1.
■ Relationship between PDF and CDF: fX(x)=dxdFX(x).
● Joint Probability:
○ Joint CDF: FXY(x,y)=P(X≤x,Y≤y).
○ Joint PDF (for continuous RVs): fXY(x,y) such that P((X,Y)∈R)=∬RfXY(x,y)dxdy.
○ Joint PMF (for discrete RVs): PXY(xi,yj)=P(X=xi,Y=yj).
● Average Value of Random Variables (Expectation):
○ Mean (Expected Value): E[X]
■ For discrete RV: E[X]=∑ixiPX(xi)
■ For continuous RV: E[X]=∫−∞∞xfX(x)dx
○ Expected Value of a Function of RV: E[g(X)]
■ For discrete RV: E[g(X)]=∑ig(xi)PX(xi)
■ For continuous RV: E[g(X)]=∫−∞∞g(x)fX(x)dx
● Variance of a Random Variable:
○ σX2=Var(X)=E[(X−E[X])2]=E[X2]−(E[X])2
○ Standard Deviation: σX=Var(X)
● Common Probability Distributions:
○ Discrete: Bernoulli, Binomial, Poisson.
○ Continuous: Uniform, Gaussian (Normal), Exponential.
■ Gaussian PDF: fX(x)=σ2π1e−(x−μ)2/(2σ2) where μ is mean and σ2 is
variance.
● Stochastic Processes (Random Processes):
○ A collection of random variables indexed by time, X(t).
○ Stationary Processes: Statistical properties do not change with time.
■ Strict-Sense Stationary (SSS): All statistical properties (joint PDFs of any
order) are time-invariant.
■ Wide-Sense Stationary (WSS): Mean is constant, and autocorrelation
depends only on the time difference τ=t2−t1.
■ Mean: E[X(t)]=μX(constant)
■ Autocorrelation: RX(t1,t2)=E[X(t1)X(t2)]=RX(t2−t1)=RX(τ)
○ Ergodic Processes: Time averages are equal to ensemble averages. This allows
estimation of statistical properties from a single realization of the process.
Important Formulas:
● Mean: E[X]=∫−∞∞xfX(x)dx
● Variance: Var(X)=E[X2]−(E[X])2
● Autocorrelation (for WSS): RX(τ)=E[X(t)X(t+τ)]
● Power Spectral Density (PSD) for WSS processes: SX(ω)=F{RX(τ)} (Wiener-Khinchin
Theorem)
Study Tips:
● Understand Definitions: Be very clear on the definitions of PMF, PDF, CDF, mean, and
variance.
● Practice Calculations: Work through examples of calculating these statistical
measures for different random variables.
● Gaussian Distribution: Pay special attention to the Gaussian (Normal) distribution as it
is ubiquitous in noise modeling.
● Distinguish Stationarity: Understand the differences between SSS and WSS processes
and their implications.
● Basic Probability: Ensure your fundamentals of probability (conditional probability,
independence) are solid.
This module provides the probabilistic framework necessary to analyze signals corrupted by
noise and to understand the performance of communication and signal processing systems in
real-world scenarios.