Foundations of Applied Probability &
Statistics
With Direct Applications to Machine Learning and Deep Learning
Abstract
This document serves as a comprehensive study guide covering basic to advanced concepts in
probability, random variables, and statistics. Every core mathematical concept is paired with its
direct application in modern Machine Learning (ML) and Deep Learning (DL) contexts.
Module 1: Basic Probability & Discrete Random Variables
Probability Spaces & Conditional Probability
Probability deals with quantifying uncertainty. A probability space consists of a sample space Ω (all
possible outcomes), a set of events, and a probability measure P.
• Conditional Probability: The probability of event A given that B has occurred.
P(A ∩ B)
P(A|B) =
P(B)
• Bayes’ Theorem: Reverses conditional probability.
P(A|B)P(B)
P(B|A) =
P(A)
ML & Deep Learning Connection: Naive Bayes & Classification
When building classifiers (e.g., predicting if a loan applicant will default), P(Class|Data) is what we
want to predict. Bayes’ rule allows us to calculate this using P(Data|Class) (Likelihood) and P(Class)
(Prior).
Discrete Random Variables
A random variable X maps outcomes to real numbers. If X takes countable values, it is discrete.
• Binomial Distribution: Number of successes in n independent Bernoulli trials.
n k
P(X = k) = p (1 − p)n−k
k
• Poisson Distribution: Probability of a given number of events occurring in a fixed interval.
λ k e−λ
P(X = k) =
k!
• Expectation (Mean): µ = E[X] = ∑ xi P(xi )
• Variance: σ 2 = Var(X) = E[X 2 ] − (E[X])2
ML & Deep Learning Connection: Binary Cross-Entropy
In binary classification, the output is modeled as a Bernoulli distribution. The Binary Cross-Entropy
(BCE) loss function used in neural networks is derived directly by minimizing the negative log-
likelihood of this distribution.
1
Module 2: Continuous Probability Distributions
Continuous random variables take on an infinite number of values (e.g., weight, time). Probabilities are
defined by a Probability Density Function (PDF), f (x).
The Normal (Gaussian) Distribution
The most important distribution due to the Central Limit Theorem.
2 !
1 1 x−µ
f (x) = √ exp −
σ 2π 2 σ
• Exponential Distribution: Models time between events. f (x) = λ e−λ x for x ≥ 0.
ML & Deep Learning Connection: Weight Initialization & VAEs
When initializing weights in deep neural networks, we draw them from a Normal Distribution to pre-
vent vanishing/exploding gradients. Furthermore, Generative AI like Variational Autoencoders (VAEs)
force their latent space to follow a standard Gaussian distribution (N(0, 1)).
Modules 3 & 4: Bivariate Distributions & Basic Statistics
Measures of Central Tendency and Dispersion
• Skewness: Measures the asymmetry of the probability distribution.
• Kurtosis: Measures the ”tailedness” of the distribution.
• Covariance: Measures how two variables change together.
Cov(X,Y ) = E[(X − µX )(Y − µY )]
• Correlation Coefficient (r): Standardized covariance, scaling between -1 and 1.
Cov(X,Y )
r=
σX σY
ML & Deep Learning Connection: Feature Engineering & PCA
High correlation between two input features means they contain redundant information. In ML, we
often use correlation matrices to filter out redundant features before passing them to a model, or use
Principal Component Analysis (PCA) to orthogonalize the dataset.
Module 5: Applied Statistics & Curve Fitting
Method of Least Squares
Used to fit a curve to a set of data points by minimizing the sum of the squares of the offsets (residuals)
of the points from the curve. For a line y = mx + c, we minimize:
n
S = ∑ (yi − (mxi + c))2
i=1
2
ML & Deep Learning Connection: Linear Regression & Neural Network Loss
The ”Sum of Squared Residuals” is exactly the Mean Squared Error (MSE) loss function used in
Deep Learning for regression tasks. Training a model via Gradient Descent is just the algorithmic
way to minimize this exact function when analytic solutions (like standard Least Squares) become too
computationally expensive.
Module 6: Small Samples & Hypothesis Testing
Hypothesis testing allows us to make inferences about populations based on sample data.
Key Tests
• Student’s t-test: Used to compare the means of two groups when the sample size is small (n < 30)
and population variance is unknown.
x̄ − µ
t= √
s/ n
• Chi-Square (χ 2 ) Test: Used for categorical data to assess goodness of fit or independence of
attributes.
(Oi − Ei )2
χ2 = ∑
Ei
(Where O is observed frequency and E is expected frequency)
ML & Deep Learning Connection: A/B Testing & Model Evaluation
How do you prove a new ML algorithm actually performs better than the old one, rather than just
getting lucky on a specific test set? You use a t-test on the accuracy metrics of both models over
multiple validation folds to prove statistical significance.
Consistent practice with these foundational formulas will severely reduce the ”black box” nature of
modern Artificial Intelligence frameworks.