Fundamentals of Data Science:

Statistics and Machine Learning with Python

Unit I  Introduction to Data Science and Applied Statistics

Lecture Notes
Scope of the Unit (15 hours). Introduction to Data Science and Applied Statistics; De-
scriptive statisticsmean, median, mode, range, standard deviation, covariance; Sampling and
sample design; Condence intervals and margin of error; Probability distributionsBernoulli,
Binomial, Poisson and Normal.

These notes are written to be read continuously, like a book. Each topic develops from rst
principles, states the key results as formal theorems with proofs where a proof exists, illustrates
the ideas with fully worked examples and short Python demonstrations, and ends with a graded
set of exercises ranging from routine drill to more demanding problems.
Contents

1 Introduction to Data Science and Applied Statistics 4

1.1 What is Data Science? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1.1 The Data Science Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Populations, Samples, and the Language of Statistics . . . . . . . . . . . . . . . . 5

1.3 Types of Data and Scales of Measurement . . . . . . . . . . . . . . . . . . . . . . 5

2 Descriptive Statistics 7
2.1 Measures of Central Tendency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.1 The Arithmetic Mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.2 The Median . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.3 The Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1.4 Relationships Among Mean, Median, and Mode . . . . . . . . . . . . . . . 9

2.2 Measures of Dispersion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.1 The Range . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.2 Variance and Standard Deviation . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.3 The Coecient of Variation . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3 Covariance: How Two Variables Move Together . . . . . . . . . . . . . . . . . . . 12

2.4 Descriptive Statistics in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Sampling and Sample Design 15

3.1 Why Sample at All? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Sampling and Non-Sampling Errors . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3 Probability Sampling Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3.1 Simple Random Sampling (SRS) . . . . . . . . . . . . . . . . . . . . . . . 16

3.3.2 Stratied Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3.3 Systematic Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3.4 Cluster Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2
Fundamentals of Data Science Unit I

3.3.5 Non-Probability Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.4 The Sampling Distribution of the Mean . . . . . . . . . . . . . . . . . . . . . . . 17

3.5 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4 Condence Intervals and Margin of Error 20

4.1 Point and Interval Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.2 Condence Interval for a Mean, Population Variance Known . . . . . . . . . . . . 20

4.3 Condence Interval for a Mean, Population Variance Unknown . . . . . . . . . . 22

4.4 Condence Interval for a Proportion . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.5 Choosing the Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5 Probability Distributions 25
5.1 Random Variables, Expectation, and Variance . . . . . . . . . . . . . . . . . . . . 25

5.2 The Bernoulli Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.3 The Binomial Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5.4 The Poisson Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.5 The Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.6 Summary of the Four Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.7 Probability Distributions in Python . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Further Reading 33

3
 Chapter 1
Introduction to Data Science and Applied
Statistics

1.1 What is Data Science?

Data science is the discipline concerned with extracting knowledge, understanding, and action-
able decisions from data. It sits at the intersection of three older subjects: statistics, which
supplies the theory of uncertainty, inference, and experimental design; computer science, which
supplies algorithms, data structures, and the computational machinery to handle data at scale;
and the domain of application (biology, economics, engineering, medicine, the social sciences),
which supplies the questions worth asking and the meaning of the answers. A data scientist is
therefore someone who can pose a question in a form that data can answer, gather and clean
the relevant data, choose and t an appropriate model, andcruciallycommunicate the result
honestly, together with the uncertainty that surrounds it.

It is helpful to distinguish data science from the two subjects it draws on most heavily.
Classical statistics tends to begin with a carefully designed experiment or survey and a model
chosen in advance; the emphasis is on inference what can we conclude about a population
from a sample, and with what condence. Machine learning tends to begin with a large, often
observational, data set and asks a predictive questiongiven past data, how well can we forecast
an unseen case. Data science is the umbrella under which both live, and applied statistics is
the connective tissue: the descriptive summaries, sampling ideas, condence statements, and
probability models that any responsible use of data must rest on. This unit builds exactly that
foundation.

1.1.1 The Data Science Lifecycle

Real projects rarely proceed in a straight line, but it is useful to name the stages that recur.
Figure 1.1 shows the cycle. Oneframes the problem, deciding what is to be predicted or estimated
and what would count as success. One then acquires data and cleans ithandling missing values,
correcting types, removing duplicatesa stage that in practice consumes the majority of the
eort. Next comes exploratory data analysis, in which descriptive statistics and visualisation
reveal the shape of the data. One then models : tting a statistical model or a machine-learning
algorithm, and evaluating it against held-out data or against the uncertainty implied by sampling
theory. Finally one communicates and deploys, and the insight gained usually reframes the
original question, so the cycle repeats.

4
Fundamentals of Data Science Unit I

Frame the Acquire & Explore (EDA)

problem clean data

Communicate Evaluate Model

& deploy
Figure 1.1: The data science lifecycle. The loop back from communication to framing captures
the fact that every nished analysis suggests the next question.

1.2 Populations, Samples, and the Language of Statistics

Almost every statistical statement is about a population : the entire collection of individuals,
objects, or measurements that we care aboutall voters in a country, all resistors produced by
a factory, all possible outcomes of an experiment repeated indenitely. A numerical fact about
the whole population, such as its true mean, is a parameter. Parameters are usually unknown
and unknowable, because measuring the whole population is impossible or uneconomical. What
we can obtain is a sample : a subset of the population that we actually observe. A numerical
fact computed from the sample, such as the sample mean, is a statistic. The central logic of
inferential statistics is to use a statistic (which we can compute) to say something reliable about
a parameter (which we cannot). By long convention, parameters are written with Greek letters
and the corresponding statistics with Roman letters, as summarised in Table 1.1.

Table 1.1: Common parameters and their sample counterparts.

Quantity Population parameter Sample statistic Read as

Mean µ x̄ mu, x-bar

Variance σ2 s2 sigma squared
Standard deviation σ s sigma, ess
Proportion p p̂ pee, p-hat
Correlation ρ r rho
Size N n

1.3 Types of Data and Scales of Measurement

Before summarising data we must know what kind of data we have, because the permissible
summaries depend on it. The rst split is between qualitative (categorical) data, which record a
quality or categoryblood group, brand of phone, pass/failand quantitative (numerical) data,
which record a magnitude. Quantitative data are further split into discrete data, which arise
from counting and take isolated values (number of children, number of defects), and continuous
data, which arise from measuring and can in principle take any value in an interval (height,
time, temperature).

A ner and more consequential classication is due to the psychologist S. S. Stevens, who
distinguished four scales of measurement. On a nominal scale, values are mere labels with
no ordereye colour, religion, city of birth; the only meaningful operations are counting and
comparison for equality. On an ordinal scale, values can be ranked but the gaps between them
are not necessarily equalthe grades A, B, C, or a survey response of poor, fair, good, excellent;
here the median makes sense but the mean does not, strictly speaking. On an interval scale,

5
Fundamentals of Data Science Unit I

dierences are meaningful and equal, but there is no true zero, so ratios are meaningless
temperature in Celsius is the classic case, where 40◦ C is not twice as hot as 20◦ C. Finally, on
a ratio scale there is a genuine zero, so both dierences and ratios make sensemass, length,
and duration are ratio quantities, and it is meaningful to say one object is twice as heavy as
another. Recognising the scale of each variable tells you immediately which of the descriptive
measures of the next chapter you are entitled to use.

Remark 1.1. The scale is a property of the meaning of the numbers, not of the symbols. Postal
codes are written as numerals but are nominal; you would never average them. Conversely, a
ve-point Likert scale is often treated as if it were interval so that means can be reported, a
convenient but debatable practice you should be aware of.

6
 Chapter 2
Descriptive Statistics

Descriptive statistics is the art of reducing a mass of numbers to a few gures that capture
its essential features without lying about them. Two families of summary dominate: measures
of central tendency, which locate the centre of the data, and measures of dispersion, which
quantify how spread out the data are around that centre. A third idea, covariance, describes
how two variables move together. We treat each in turn, stating the dening formulas, proving
the properties that justify their use, and working through numerical and Python examples.

Throughout, suppose we have observed n numerical values x1 , x2 , . . . , xn on a single variable.

2.1 Measures of Central Tendency

2.1.1 The Arithmetic Mean
Denition 2.1 (Arithmetic mean). The arithmetic mean of the values x1 , . . . , xn is

n
1X
x̄ = xi .
n
i=1

For data grouped into distinct values x1 , . . . , x k occurring with frequencies f1 , . . . , fk (so that
weighted
P
i fi = n), the mean is the average
Pk
fi xi
x̄ = Pi=1
k
.
i=1 fi

The mean is the balance point of the data: if we imagine the values as equal masses placed
along a rod at their numerical positions, the mean is the point at which the rod balances. This
physical picture is made precise by the rst property below.

Theorem 2.2 (The mean makes deviations cancel) . The sum of the deviations of the observa-
tions about their mean is zero: n
X
(xi − x̄) = 0.
i=1

Proof.
P
Expanding the sum and using i xi = nx̄ (which is just the denition of the mean
rearranged),
n
X n
X n
X
(xi − x̄) = xi − x̄ = nx̄ − nx̄ = 0.
i=1 i=1 i=1

7
Fundamentals of Data Science Unit I

The next theorem explains why the mean, and the associated squared distances, sit at the
heart of so much of statistics: the mean is the single number that is closest to all the data in
the least-squares sense.

Theorem 2.3 (Least-squares optimality of the mean) . Among all real numbers c, the sum of
squared deviations
n
X
S(c) = (xi − c)2
i=1
is minimised uniquely at c = x̄.

Proof. S is a dierentiable function of c. Dierentiating,

n
X n
X X 
′
S (c) = 2(xi − c)(−1) = −2 (xi − c) = −2 xi − nc .
i=1 i=1 i
′ x̄. Since S ′′ (c)
P
Setting S (c) =0 gives i xi = nc, i.e. c= = 2n > 0 for all c, this stationary
point is a strict global minimum. Hence the mean is the unique minimiser.

Remark 2.4. Theorem 2.3 is the seed of the method of least squares and of linear regression:
whenever we t a model by minimising a sum of squared errors, we are generalising the fact that
the best constant summary of a data set is its mean.

A useful computational property is that the mean transforms simply under a change of scale
and origin. If every observation is replaced by yi = axi + b (for constants a, b), then ȳ = ax̄ + b;
the proof is immediate from linearity of the summation. This is the basis of the coding or
assumed-mean shortcut once used for hand computation.

2.1.2 The Median

Denition 2.5 (Median) . Arrange the observations in increasing order as the order statistics
x(1) ≤ x(2) ≤ · · · ≤ x(n) . The median M is the middle value:

x( n+1 ) , n odd,
2
M = 1 
 x n +x n
2 (2) ( 2 +1) , n even.

The median divides the ordered data into two halves of equal count. Its great virtue is
robustness : because it depends only on the middle of the order statistics, it is unaected by
extreme values. If the largest observation in a data set were replaced by one a thousand times
larger, the mean would move substantially but the median would not move at all. This makes
the median the preferred measure of location for skewed data such as incomes or house prices.
The median has its own optimality property, dual to Theorem 2.3.

Theorem 2.6 (Least-absolute-deviations optimality of the median) . The function A(c) =

n
− c| is minimised when c is a median of the data.
P
i=1 |xi

Proof sketch. c rightward by a small amount δ > 0. Every observation lying to

Consider moving
the left of c δ , and every observation to the right has its
has its absolute deviation increased by
deviation decreased by δ . If L points lie left of c and R lie right, the net change in A is (L − R)δ .
Thus A decreases as long as R > L and increases once L > R; the minimum is reached when the
counts on the two sides balance, which is exactly the dening property of the median. (When n
is even, any point between the two middle order statistics achieves the minimum, which is why
the median is conventionally taken as their average.)

8
Fundamentals of Data Science Unit I

2.1.3 The Mode

Denition 2.7 (Mode). The mode is the value that occurs with the greatest frequency. A data
set may have one mode ( unimodal ), two (bimodal ), or several (multimodal ); if every value occurs
equally often the mode is not dened.

The mode is the only measure of central tendency that applies to purely nominal datathe
modal blood group in a class is a perfectly sensible statement, whereas the mean blood group
is nonsense. For continuous data the mode is read from a histogram or a tted density as the
location of the peak.

2.1.4 Relationships Among Mean, Median, and Mode

The three measures coincide for a perfectly symmetric, unimodal distribution. For a skewed
distribution they separate in a predictable order: in a right-skewed (positively skewed) distri-
bution the long tail on the right pulls the mean above the median, which in turn lies above the
mode, so mode < median < mean; for left skew the inequalities reverse. Figure 2.1 illustrates the
right-skewed case. For moderately skewed distributions the three are connected by the empirical
relation
Mode ≈ 3 Median − 2 Mean,
often attributed to Karl Pearson, which is a handy way to estimate one measure when the other
two are known.
density

Mode
MedianMean
x

Figure 2.1: A right-skewed distribution. The long right tail drags the mean furthest to the right,
the mode stays at the peak, and the median lies between them.

Example 2.1 (Central tendency by hand) . The number of goals scored by a team in its last
nine matches was
2, 1, 3, 0, 2, 5, 2, 1, 2.
2+1+3+0+2+5+2+1+2 18
Mean. x̄ = = = 2 goals.
9 9
Median. Sorted: 0, 1, 1, 2, 2, 2, 2, 3, 5. With n = 9 (odd) the median is the 5th value, M = 2.
Mode. The value 2 occurs four times, more than any other, so the mode is 2.
Here all three measures agree, reecting the near-symmetry of this small data set.

9
Fundamentals of Data Science Unit I

2.2 Measures of Dispersion

Two data sets can share the same mean yet be utterly dierent: the numbers 50, 50, 50 and
0, 50, 100 both average 50, but the rst is perfectly consistent while the second is wildly variable.
A measure of central tendency alone is therefore never enough; we also need a measure of
dispersion or spread.

2.2.1 The Range

Denition 2.8 (Range). The range is the dierence between the largest and smallest observa-
tions,
Range = x(n) − x(1) .

The range is trivial to compute and to understand, and for that reason it is widely used in
quality control (where it drives the R-chart). Its weakness is that it uses only two of the n
observations and is therefore extremely sensitive to outliers and gives no information about how
the bulk of the data are distributed.

2.2.2 Variance and Standard Deviation

A better idea uses every observation by measuring its deviation from the mean. Because the
deviations sum to zero (Theorem 2.2), we cannot simply average them; squaring rst removes
the sign.

Denition 2.9 (Variance and standard deviation). The population variance of x1 , . . . , x N (the
whole population) is
N
1 X 1 X
σ2 = (xi − µ)2 , µ= xi ,
N N
i=1 i
√
and the population standard deviation is σ = σ 2 . For a sample of size n, the sample variance
is
n
1 X
s2 = (xi − x̄)2 ,
n−1
i=1
√
with sample standard deviation s= s2 .

The standard deviation is expressed in the same units as the data (unlike the variance, whose
units are squared), which is why it is the spread measure usually quoted. The mysterious divisor
n−1 in the sample variance, rather than n, is called Bessel's correction ; we prove in Chapter 3
2
that it makes s an unbiased 2
estimator of σ . Intuitively, the deviations are taken about x̄
rather than the unknown µ, and x̄ is itself pulled towards the data, so the squared deviations
underestimate the true spread; dividing by n−1 inates the estimate by just the right amount
to compensate.

For computation by hand or in a single pass, the following identity is invaluable.

Theorem 2.10 (Computational formula for variance).

n n n P 2
X X X ( i xi )
(xi − x̄)2 = x2i − nx̄2 = x2i − .
n
i=1 i=1 i=1

10
Fundamentals of Data Science Unit I

Proof. Expand the square inside the sum and split it:

X X
X 2 X
(xi − x̄)2 = x2i − 2x̄xi + x̄2 = xi − 2x̄ xi + nx̄2 .
i i i i

−2x̄ · nx̄ = −2nx̄2 , giving

P
Now i xi = nx̄, so the middle term is

X X
x2i − 2nx̄2 + nx̄2 = x2i − nx̄2 .
i i

The variance also behaves predictably under a linear change of variable, a fact we use
constantlyfor example when standardising to z -scores.

Theorem 2.11 (Variance under a linear transformation). If yi = axi + b for constants a, b, then
the variance of the y 's satises

s2y = a2 s2x , equivalently sy = |a| sx .

In particular, adding a constant b leaves the variance unchanged, and multiplying by a scales the
standard deviation by |a|.

Proof. Since ȳ = ax̄ + b, the deviation of yi is yi − ȳ = (axi + b) − (ax̄ + b) = a(xi − x̄). Squaring
and summing,
X X X
(yi − ȳ)2 = a2 (xi − x̄)2 = a2 (xi − x̄)2 ,
i i i

and dividing by n−1 gives s2y = a2 s2x . Taking square roots gives sy = |a|sx , the absolute value
ensuring a non-negative standard deviation.

Example 2.2 (Variance and standard deviation) . Compute the sample variance and standard
deviation of 4, 8, 6, 5, 12.
x2i = 16+64+36+25+144 = 285.
P P
First xi = 4+8+6+5+12 = 35, so x̄ = 35/5 = 7. Next
By Theorem 2.10,
X 352
(xi − x̄)2 = 285 − = 285 − 245 = 40.
5
i
40 √
Hence s2 = = 10 and s= 10 ≈ 3.16. Had we treated these ve numbers as a whole
5−1
population, we would instead divide by N =5 to get σ 2 = 40/5 = 8 and σ ≈ 2.83.

2.2.3 The Coecient of Variation

Standard deviation is an absolute measure of spread and cannot be compared fairly across data
sets measured in dierent units or of very dierent magnitudes. The coecient of variation,
s
CV = × 100%,
x̄
expresses the standard deviation as a percentage of the mean and so provides a relative, unit-free
measure. A series with a smaller CV is the more consistent. It is meaningful only for ratio-scale
data with a positive mean.

11
Fundamentals of Data Science Unit I

2.3 Covariance: How Two Variables Move Together

So far each observation carried a single number. Suppose now that on each of n individuals we
record two variables, giving paired data (x1 , y1 ), . . . , (xn , yn )for instance the height and weight
of each person. Covariance measures the direction of the linear relationship between them.
Denition 2.12 (Covariance). The sample covariance of the pairs (xi , yi ) is

n
1 X
Cov(x, y) = sxy = (xi − x̄)(yi − ȳ).
n−1
i=1

The sign of the covariance tells the story. If large x tends to accompany large y (and small
with small), the products (xi − x̄)(yi − ȳ) are predominantly positive and the covariance is
positive: the variables move together. If large x tends to accompany small y, the products are
predominantly negative and the covariance is negative. A covariance near zero indicates no
linear associationthough, as we stress below, it does not rule out a non-linear one. Note that
Cov(x, x) = s2x : covariance of a variable with itself is just its variance, so covariance genuinely
generalises variance.

As with the variance, there is a shortcut formula and a linearity property.

Theorem 2.13 (Computational formula for covariance).

n n P P
X X ( i xi ) ( i yi )
(xi − x̄)(yi − ȳ) = x i yi − .
n
i=1 i=1

Proof. Expanding the product,

X X
X X X
(xi − x̄)(yi − ȳ) = xi yi − x̄yi − ȳxi + x̄ ȳ = xi yi − x̄ yi − ȳ xi + nx̄ ȳ.
i i i i i
P P
Using i yi = nȳ and xi
iP = nx̄, the last three terms combine: −x̄(nȳ)− ȳ(nx̄)+nx̄ȳ = −nx̄ȳ .
1 P P
Therefore the sum equals i xi yi −nx̄ȳ , and substituting x̄ȳ = n2 ( i xi )( i yi ) and simplifying
gives the stated form.

The magnitude of covariance depends on the units of x and y and so is hard to interpret on
its own. Dividing by the two standard deviations removes the units and rescales to the interval
[−1, 1], giving Pearson's correlation coecient
sxy
r= .
sx sy

The bound −1 ≤ r ≤ 1 is a consequence of the CauchySchwarz inequality; r = ±1 exactly

when the points lie on a straight line. We treat correlation more fully in later units, but record
here the one warning that can never be repeated too often.

Remark 2.14 (Correlation is not causation, and zero correlation is not independence). A large
|r| signals a strong linear association, not that one variable causes the other; both may be driven
by a hidden third variable. Conversely, r =0 means only that there is no linear trend. The
points on a perfect parabola y= x2 symmetric about zero have exactly zero covariance, yet y is
completely determined by x. Always plot the data.

12
Fundamentals of Data Science Unit I

Example 2.3 (Covariance and correlation). For the ve students below, x is hours studied and
y is the test score.
x 1 2 3 4 5
y 2 4 5 4 5
P P P
Then x = 15, y = 20, xy = 1·2 + 2·4 + 3·5 + 4·4 + 5·5 = 2 + 8 + 15 + 16 + 25 = 66,
with n = 5. By Theorem 2.13,
X 15 · 20 6
(xi − x̄)(yi − ȳ) = 66 − = 66 − 60 = 6, so sxy = = 1.5.
5 4
The positive covariance conrms that more study hours go with higher scores. (Computing
√
sx = 10/2 ≈ 1.58 and sy ≈ 1.30 gives r ≈ 0.73, a fairly strong positive linear relationship.)

2.4 Descriptive Statistics in Python

Every measure of this chapter is available in one line using NumPy and pandas. The snippet
below reproduces the variance example and computes the full set of summaries. Note the ddof
(delta degrees of freedom) argument: ddof=0 divides by n (population), ddof=1 divides by
n−1 (sample); pandas uses the sample convention by default while NumPy uses the population
convention, a frequent source of confusion.

1 import numpy as np
2 import pandas as pd
3
4 x = np . array ([4 , 8 , 6 , 5 , 12])
5
6 print ( " mean : " , np . mean ( x ) ) # 7.0
7 print ( " median : " , np . median ( x ) ) # 6.0
8 print ( " range : " , x . max () - x . min () ) # 8
9 print ( " var ( population , ddof =0) : " , np . var (x , ddof =0) ) # 8.0
10 print ( " var ( sample , ddof =1) : " , np . var (x , ddof =1) ) # 10.0
11 print ( " std ( sample , ddof =1) : " , np . std (x , ddof =1) ) # 3.162...
12
13 # Mode and a full summary via pandas
14 s = pd . Series ([2 , 1 , 3 , 0 , 2 , 5 , 2 , 1 , 2])
15 print ( " mode : " , s . mode () . tolist () ) # [2]
16 print ( s . describe () ) # count , mean , std , min ,
quartiles , max
17
18 # Covariance and correlation matrices
19 xx = np . array ([1 , 2 , 3 , 4 , 5])
20 yy = np . array ([2 , 4 , 5 , 4 , 5])
21 print ( " covariance : " , np . cov ( xx , yy , ddof =1) [0 , 1]) # 1.5
22 print ( " correlation : " , np . corrcoef ( xx , yy ) [0 , 1]) # 0.73...

Exercises
Exercise 2.1 (Routine). ◦
The daily maximum temperature ( C) over a week was 31, 33, 30, 34,
33, 35, and 33. Find the mean, median, mode, and range.

Exercise 2.2 (Routine) . Using the computational formula of Theorem 2.10, nd the sample
variance and standard deviation of 10, 12, 14, 16, 18. Then verify Theorem 2.11 by subtracting
10 from each value and recomputing.

13
Fundamentals of Data Science Unit I

Exercise 2.3 (Moderate) . A factory records the number of defective items in 10 batches:
3, 0, 2, 5, 1, 4, 2, 3, 2, 8. Compute the mean and median, and explain, with reference to the value
8, why the median is the more representative summary here. What is the coecient of variation?

Exercise 2.4 (Moderate) . Prove that if a constant b is added to every observation, the mean
increases by b but the variance is unchanged. Which measures of central tendency and dispersion
are, and are not, aected by (i) a change of origin and (ii) a change of scale?

Exercise 2.5 (Moderate). The pairs (x, y) are (2, 6), (4, 5), (6, 7), (8, 10), (10, 12). Compute the
covariance and Pearson correlation coecient, and interpret the sign and rough strength of the
relationship.

Exercise 2.6 (Challenging). 2 2 2

P P
Show, without calculus, that i (xi −c) = i (xi − x̄) +n(x̄−c)
for any constant c. Deduce Theorem 2.3 directly from this identity, and explain what the extra
term n(x̄ − c)2 represents.

Exercise 2.7 (Challenging) . A data set of n values has mean x̄ and standard deviation s. A
single new observation equal to x̄ is added. Show that the mean is unchanged and that the new
standard deviation is strictly smaller than s (assuming s > 0). Interpret the result.

14
 Chapter 3
Sampling and Sample Design

We have seen how to summarise data once we have it. But where does the data come from, and
why should a summary of a mere sample tell us anything about the population we care about?
This chapter answers those questions. Good conclusions rest on good samples, and a sample is
good not because it is large but because it is drawn in a way that lets probability theory speak
on its behalf.

3.1 Why Sample at All?

A census examines every member of a population; a sample survey examines only a part. One
might think a census is always better, but sampling is usually preferred, for four reasons. It
is far cheaper and faster, so results are available while they are still relevant. It is sometimes
the only possibility, as when testing is destructive (one cannot burn every match to check that
matches light, or a factory would have nothing to sell). And, perhaps surprisingly, a well-
designed sample can be more accurate than a census, because the smaller scale permits better
training of investigators, more careful measurement, and closer supervision, reducing the human
errors that plague a vast operation.

The population one wishes to study is thetarget population. The list or procedure from which
the sample is actually drawn is the sampling frame. The two are ideally identical but rarely are:
a telephone survey has as its frame only those with listed numbers, and any mismatch between
frame and target population is a source of bias no amount of data can cure.

3.2 Sampling and Non-Sampling Errors

Two quite dierent kinds of error aict a survey. Sampling error is the discrepancy between a
sample statistic and the true population parameter that arises simply because we observed a part
rather than the whole; dierent samples give dierent values. Sampling error is unavoidable,
but it is measurable its typical size is exactly what the standard error and condence interval
of the next chapter quantifyand it shrinks as the sample size grows. Non-sampling error,
by contrast, is any error not due to sampling: a faulty frame, non-response, poorly worded
or leading questions, data-entry mistakes, interviewer bias, respondents who lie. Non-sampling
error is far more insidious because increasing the sample size does not reduce ita bigger badly-
designed survey is merely a more condently wrong oneand because it is hard to quantify.
Much of the craft of sample design is directed at controlling non-sampling error.

15
Fundamentals of Data Science Unit I

3.3 Probability Sampling Methods

A sampling method is a probability sampling method if every unit in the population has a
known, non-zero probability of selection, xed in advance and independent of the investigator's
judgement. Only probability sampling permits valid inference, because only then can we compute
the sampling distribution of our estimates. There are four workhorse designs.

3.3.1 Simple Random Sampling (SRS)

In simple random sampling without replacement, every possible subset of n units from the N in
the population is equally likely to be the chosen sample. Equivalently, units are drawn one at
a time, each remaining unit equally likely at each draw. SRS is the conceptual baseline against
which all other designs are compared and the model underlying the theory of the following
chapters. Its practical drawback is that it requires a complete frame and can, by chance, produce
a sample that misses an important subgroup.

3.3.2 Stratied Sampling

When the population falls into distinct, internally homogeneous groups called strata (for example
age bands, or geographic regions), stratied sampling draws an independent simple random
sample from each stratum and combines them. This guarantees representation of every stratum
and, when the strata are genuinely more homogeneous within than between, yields estimates
more precise than SRS of the same total size. Allocation of the total sample across strata may be
proportional (each stratum contributes in proportion to its size) or optimal (Neyman allocation,
which also accounts for the variability within each stratum).

3.3.3 Systematic Sampling

Systematic sampling selects every kth unit from an ordered frame after a random start, where
k ≈ N/n is the sampling interval. It is simple to execute and spreads the sample evenly through
the frame. Its hidden danger is periodicity : if the frame has a cyclic pattern whose period is
a multiple of k, the sample can be badly biasedsampling every 7th day, for instance, would
always land on the same weekday.

3.3.4 Cluster Sampling

Cluster sampling divides the population into naturally occurring groups called clusters (city
blocks, schools, villages), randomly selects some whole clusters, and surveys every unit within
the chosen clusters (or subsamples them, in multi-stage sampling). Its purpose is economy :
when the population is geographically dispersed, it is far cheaper to survey everyone in a few
selected villages than a scattered SRS across all of them. The price is usually a loss of precision,
because units within a cluster tend to be similar and so carry less independent information.

The contrast between stratied and cluster sampling is worth xing in mind. In stratied
sampling we sample from every group and want the groups internally homogeneous ; in cluster
sampling we sample only some groups and want each cluster to be a miniature of the whole.
Figure 3.1 contrasts the four designs.

16
Fundamentals of Data Science Unit I

Simple random Stratied Systematic Cluster

Figure 3.1: Four probability sampling designs; lled circles are selected units. SRS scatters the
sample; stratied takes some from every horizontal stratum; systematic takes every k th unit;
cluster takes all units inside chosen clusters.

3.3.5 Non-Probability Sampling

For completeness we note the methods that do not assign known selection probabilities: con-
venience sampling (whoever is easiest to reach), judgement or purposive sampling (the expert
picks typical units), quota sampling (ll xed quotas per group by any means), and snowball
sampling (respondents recruit further respondents, useful for hidden populations). These are
cheap and sometimes the only feasible route, but they support no valid probability statement:
one cannot legitimately attach a condence interval or margin of error to a convenience sample,
however large.

3.4 The Sampling Distribution of the Mean

Fix attention on the sample mean X̄ . Because the sample is random, X̄ is itself a random vari-
able: repeat the sampling and you get a dierent value each time. The probability distribution
of X̄ over all possible samples is its sampling distribution, and understanding it is the whole
basis of statistical inference. Two facts pin down its centre and spread.

Theorem 3.1 (Mean and standard error of X̄ ). Let X1 , . . . , Xn be independent draws from a
population with mean µ and variance σ 2 (as in simple random sampling from a large population,
or sampling with replacement). Then
σ2 σ
E[X̄] = µ, Var(X̄) = , SD(X̄) = √ .
n n
√
The quantity σ/ n is called the standard error of the mean.

Proof. unbiased
1 Pn 1
By linearity of expectation, E[X̄] = n i=1 E[Xi ] = n (nµ) = µ, so X̄ is an
estimator of µ: on average it is right. For the variance, independence makes the variance of a
sum the sum of the variances, and the scaling rule Var(aX) = a2 Var(X) gives

 P 
1 1 Pn 1 σ2
Var(X̄) = Var n i Xi = n2 i=1 Var(Xi ) = n2
(nσ 2 ) = n .

Taking the square root gives the standard error.

√
The single most important consequence is the appearance of n, not n, in the denominator:
to halve the standard error one must quadruple the sample size. Precision improves with sample
size, but with diminishing returns. This square-root law governs the economics of every survey.

We can now redeem the promise of Chapter 2 and explain Bessel's correction.

17
Fundamentals of Data Science Unit I

Theorem 3.2 (Unbiasedness of the sample variance) . Under the assumptions of Theorem 3.1,
the sample variance satises E[S 2 ] = σ 2 .
1 Pn
S2 = n−1 i=1 (Xi − X̄)2

Proof. Write Xi − X̄ = (Xi − µ) − (X̄ − µ) and expand the sum of squares:

n
X n
X
(Xi − X̄)2 = (Xi − µ)2 − n(X̄ − µ)2 ,
i=1 i=1

the cross term vanishing exactly as in Theorem 2.10. Take expectations. By denition E[(Xi −
µ)2 ] = σ 2 , so the rst sum has expectation nσ 2 . By Theorem 3.1, E[(X̄ −µ)2 ] = Var(X̄) = σ 2 /n,
2 2
so the second term has expectation n · σ /n = σ . Therefore

" #
X
E (Xi − X̄)2 = nσ 2 − σ 2 = (n − 1)σ 2 ,
i

n−1 2
and dividing by n−1 gives E[S 2 ] = σ 2 . Dividing instead by n would give n σ , an underestimate
which is precisely why we use n − 1.

3.5 The Central Limit Theorem

Theorem 3.1 tells us the centre and spread of the sampling distribution of X̄ but not its shape.
The remarkable answer, and arguably the most important theorem in all of applied statistics, is
that the shape is approximately normal the bell curve of the next chapteralmost regardless
of the shape of the population.

Theorem 3.3 (Central Limit Theorem). Let X1 , X2 , . . . be independent and identically dis-
tributed random variables with nite mean µ and nite variance σ 2 > 0. Then the standardised
sample mean converges in distribution to a standard normal variable as n → ∞:
X̄ − µ d
Zn = √ −−→ N (0, 1).
σ/ n

Equivalently, for large n the sample mean X̄ is approximately N µ, σ 2 /n .

A complete proof requires characteristic functions and is beyond this unit, but the content
of the theorem is easy to appreciate and to demonstrate. Whatever the populationuniform,
skewed, even discretethe distribution of the average of many independent draws piles up into
√
a bell shape centred at µ with spread σ/ n. A common rule of thumb is that n ≥ 30 suces
for the approximation to be good, fewer if the population is already roughly symmetric, more if
it is heavily skewed. It is the Central Limit Theorem that licenses the normal-based condence
intervals of the next chapter even when the underlying data are not themselves normal.

Example 3.1 (Standard error in practice). A machine lls bottles with a nominal 500 ml. The
ll volume has standard deviation
√ √ σ = 8 ml. If we sample n = 16 bottles, the sample mean ll
has standard error σ/ n = 8/ 16 = 2 ml. To cut this standard error to 1 ml we would need
√
σ/ n = 1, i.e. n = 64: quadrupling the sample to halve the error, exactly as the square-root
law predicts.

1 import numpy as np
2 rng = np . random . default_rng (0)
3

18
Fundamentals of Data Science Unit I

4 # Population is strongly right - skewed ( exponential ) , NOT normal .

5 pop_mean , n , reps = 1.0 , 30 , 20000
6 means = [ rng . exponential ( pop_mean , n ) . mean () for _ in range ( reps ) ]
7 means = np . array ( means )
8
9 print ( " Mean of sample means : " , means . mean () ) # ~1.0 (= mu )
10 print ( " SD of sample means : " , means . std ( ddof =1) ) # ~1/ sqrt (30)
=0.183
11 # A histogram of ` means ` is bell - shaped despite the skewed population
:
12 # this is the Central Limit Theorem in action .

Exercises
Exercise 3.1 (Routine) . Distinguish, with one example each, between (a) a parameter and a
statistic, and (b) sampling error and non-sampling error. Why does increasing the sample size
help with one but not the other?

Exercise 3.2 (Routine) . A university has 8000 students spread across four faculties. Describe
how you would draw a sample of 400 students using (i) simple random, (ii) stratied, and (iii)
systematic sampling. State one advantage of the stratied design here.

Exercise 3.3 (Moderate) . A population has standard deviation σ = 20. Find the standard
error of the mean for samples of size 25, 100, and 400. By what factor must the sample size
grow to reduce the standard error to one-fth of its value at n = 25?

Exercise 3.4 (Moderate) . Explain why a very large convenience sample (say, an online poll
answered by 50,000 self-selected visitors) can give a worse estimate of public opinion than a
carefully drawn probability sample of 1,000. Frame your answer in terms of sampling versus
non-sampling error.

Exercise 3.5 (Challenging)

P. Using the identity in the proof of Theorem 3.2, show directly
1 n−1 2
that the estimator σ̃ 2 = n i (Xi − X̄)
2 has expectation
n σ , and hence is biased. By what
multiplicative factor is it too small on average, and what happens to this factor as n → ∞?

Exercise 3.6 (Challenging) . The Central Limit Theorem concerns the mean. Design a small
simulation (in words or code) to investigate whether the sample median of n draws from a skewed
population is also approximately normal for large n. What features would you look at in the
histogram of medians?

19
 Chapter 4
Condence Intervals and Margin of Error

A single number estimated from a samplea point estimate is almost never exactly equal to
the parameter it targets, and reported on its own it hides how much it might be o. Responsible
statistics reports instead an interval of plausible values together with a statement of how con-
dent we are that the interval captures the truth. This chapter builds such condence intervals
and denes the margin of error that so often appears (and is so often misunderstood) in news
reports of polls.

4.1 Point and Interval Estimation

Given a sample, an estimator is a rule that produces a valuefor example use X̄ to estimate
µ, or use p̂, the sample proportion, to estimate p. We saw in Theorem 3.1 that X̄ is unbiased
√
and has standard error σ/ n; the standard error is the natural unit of uncertainty. An interval
estimator pairs the point estimate with a margin on either side. We want to choose that margin
so that, over repeated sampling, the interval contains the true parameter a specied proportion
of the timesay 95%. That proportion is the condence level, written 1 − α, so that α is the
probability the interval misses.

4.2 Condence Interval for a Mean, Population Variance Known

Suppose rst, as an idealisation, that the population standard deviation σ is known and that
X̄ is (at least approximately) normally distributedeither because the population is normal, or
because n is large enough for the Central Limit Theorem to apply. Standardising,

X̄ − µ
Z= √ ∼ N (0, 1).
σ/ n

Let zα/2 be the value with area α/2 to its right under the standard normal curve (so z0.025 = 1.96
for 95% condence). Then, by the symmetry of the normal,
 
X̄ − µ
P −zα/2 ≤ √ ≤ zα/2 = 1 − α.
σ/ n

Rearranging the inequality to isolate µ gives the interval.

20
Fundamentals of Data Science Unit I

Theorem 4.1 (z -interval for the mean) . If X̄ ∼ N (µ, σ 2 /n) with σ known, a 100(1 − α)%
condence interval for µ is
σ  
X̄ ± zα/2 √ i.e. X̄ − zα/2 √σn , X̄ + zα/2 √σn .
n

√
Proof.


Starting from P − zα/2 ≤ (X̄ − µ)/(σ/ n) ≤ zα/2 = 1 − α, multiply through by the
√
positive quantity σ/ n:
 
P −zα/2 √σn ≤ X̄ − µ ≤ zα/2 √σn = 1 − α.

Subtract X̄ throughout and multiply by −1 (which reverses the inequalities), giving

 
P X̄ − zα/2 √σn ≤ µ ≤ X̄ + zα/2 √σn = 1 − α,

which is the stated interval.

The half-width of this interval,

σ
E = zα/2 √ ,
n
is the margin of error : the maximum distance, at the stated condence, between the point
estimate and the true mean. Note its three leversit grows with the condence level (through
zα/2 ), grows with the population spread σ, and shrinks with the square root of the sample size.

Remark 4.2 (What  95% condent does and does not mean). The condence level is a property
of the procedure, not of any one interval. It means: if we were to repeat the whole sampling-
and-interval-building process many times, about 95% of the intervals produced would contain
the true µ. For the one interval we actually computed, µ is either in it or notthere is no
probability left over. It is therefore wrong to say there is a 95% probability that µ lies in this
particular interval. Figure 4.1 makes the correct picture concrete.

3.0 (true mean)

misses 3.0

estimate

Figure 4.1: Ten 95% condence intervals from ten dierent samples. Each is centred on its own
sample mean (dot); most cover the xed true mean µ, but by chance one does not. Over the
long run, about 95% cover µ.

21
Fundamentals of Data Science Unit I

4.3 Condence Interval for a Mean, Population Variance Un-

known
In practice σ is almost never known; we must estimate it by the sample standard deviation s.
Replacing σ by s introduces extra uncertainty, and the standardised quantity no longer follows
the normal distribution but Student's t-distribution with n − 1 degrees of freedom:

X̄ − µ
T = √ ∼ tn−1 .
s/ n

The t-distribution is symmetric and bell-shaped like the normal but has heavier tails, reecting
the added uncertainty from estimating σ; as n→∞ it converges to the standard normal. The
condence interval takes the same form as before with z replaced by the appropriate t critical
value.

Theorem 4.3 (t-interval for the mean). For a sample from a normal population with σ unknown,
a 100(1 − α)% condence interval for µ is
s
X̄ ± tα/2, n−1 √ ,
n

where tα/2, n−1 is the upper-α/2 critical value of the t-distribution with n − 1 degrees of freedom.
Example 4.1 t-interval). A sample of n = 16 steel rods has mean length x̄ = 50.2 cm and
(A
sample standard deviation s = 0.8 cm. Construct a 95% condence interval for the mean length.
With n − 1 = 15 degrees of freedom, the critical value is t0.025,15 = 2.131. The margin of error is

0.8
E = 2.131 × √ = 2.131 × 0.2 = 0.426 cm,
16
so the interval is 50.2 ± 0.43, (49.77, 50.63) cm. We are 95% condent the true mean rod
i.e.
length lies in this range. (Had we wrongly used z = 1.96, we would have obtained the slightly
too-narrow margin 0.392, understating our uncertainty.)

4.4 Condence Interval for a Proportion

Opinion polls estimate a population proportion pthe fraction who favour a candidate, sayby
the sample proportion p̂ = X/n, where X is the number of successes in n trials. For large
n
pthe Central Limit Theorem makes p̂ approximately normal with mean p and standard error
p(1 − p)/n; estimating p by p̂ in the standard error gives the Wald interval.

Theorem 4.4 (Condence interval for a proportion). For large n, an approximate 100(1 − α)%
condence interval for p is r
p̂(1 − p̂)
p̂ ± zα/2 .
n
p
The margin of error of a poll is exactly E = zα/2 p̂(1 − p̂)/n. Because p̂(1 − p̂) is largest
p √
when p̂ = 0.5, the worst-case margin of error at 95% condence is 1.96 0.25/n = 0.98/ n,
which is why a poll of about
√ n = 1000 people is so often quoted with a margin of error of roughly
±3%: 0.98/ 1000 ≈ 0.031.

22
Fundamentals of Data Science Unit I

Example 4.2 (A poll) . In a survey of n = 600 voters, 342 say they support a proposal, so
p̂ = 342/600 = 0.57. A 95% condence interval for the true support p is
r
0.57 × 0.43
0.57 ± 1.96 = 0.57 ± 1.96(0.0202) = 0.57 ± 0.040 = (0.530, 0.610).
600
The margin of error is about 4 percentage points. Because the entire interval lies above 0.5, we
can be 95% condent the proposal has majority support.

4.5 Choosing the Sample Size

Because the margin of error depends on n, we can turn the calculation around and ask: how
√
large a sample is needed to achieve a desired margin E? Solving E = zα/2 σ/ n for n gives, for
a mean,
z
α/2 σ
2
n= ,
E
up
p
always rounding to the next whole number. For a proportion, solving E = zα/2 p(1 − p)/n
gives
2 p(1 − p)
zα/2
n= ,
E2
and in the absence of any prior guess for p one uses the conservative worst case p = 0.5, which
maximises p(1 − p) = 0.25 and so guarantees the margin whatever the truth turns out to be.

Example 4.3 (Sample size for a poll) . How many voters must be sampled so that a 95%
condence interval for the support proportion has margin of error at most E = 0.02 (two
percentage points)? Using the conservative p(1 − p) = 0.25 and z0.025 = 1.96,
1.962 × 0.25 0.9604
n= 2
= = 2401.
0.02 0.0004
So about 2400 respondents are requirednote that improving the margin from the usual 3% to
2% more than doubles the needed sample, the square-root law again.

1 import numpy as np
2 from scipy import stats
3
4 # t - interval for a mean ( sigma unknown )
5 x = np . array ([50.1 , 49.8 , 50.5 , 50.3 , 49.9 , 50.4 , 50.0 , 50.6])
6 n = len ( x ) ; xbar = x . mean () ; s = x . std ( ddof =1)
7 tcrit = stats . t . ppf (0.975 , df =n -1) # 95% -> upper 2.5%
8 E = tcrit * s / np . sqrt ( n )
9 print ( f " 95% CI for mean : ({ xbar - E :.3 f } , { xbar + E :.3 f }) " )
10
11 # CI for a proportion
12 X , N = 342 , 600
13 phat = X / N
14 z = stats . norm . ppf (0.975) # 1.96
15 E = z * np . sqrt ( phat *(1 - phat ) / N )
16 print ( f " 95% CI for p : ({ phat - E :.3 f } , { phat + E :.3 f }) , margin = { E :.3 f
}")
17
18 # Sample size for margin 0.02 at 95% , worst case p =0.5
19 E_target = 0.02
20 n_needed = ( z **2 * 0.25) / E_target **2
21 print ( " n needed : " , int ( np . ceil ( n_needed ) ) ) # 2401

23
Fundamentals of Data Science Unit I

Exercises
Exercise 4.1 (Routine) . A sample of n = 64 light bulbs from a large batch has mean life
x̄ = 1200 hours. The population standard deviation is known to be σ = 100 hours. Construct
a 95% condence interval for the mean life, and state the margin of error.

Exercise 4.2 (Routine) . Explain the dierence between a point estimate and an interval es-
timate, and state precisely what the  95% in a 95% condence interval refers to. Why is it
incorrect to say that a particular computed interval has a 95% probability of containing µ?

Exercise 4.3 (Moderate). A sample of 25 measurements has x̄ = 8.4 and s = 1.5. Assuming a
normal population, build a 99% condence interval for µ. (Use t0.005,24 = 2.797.) Compare its
width with that of the corresponding 95% interval and explain the dierence.

Exercise 4.4 (Moderate) . In a quality audit, 45 of 500 inspected components are defective.
Give a 95% condence interval for the true defect rate p, and state the margin of error. If
management wants the margin reduced to 1% (with no prior estimate of p), how large a sample
is required?

Exercise 4.5 (Challenging) . Show algebraically that, for a xed condence level and xed n,
the margin of error of a proportion is maximised at p̂ = 0.5. (Hint: maximise g(p) = p(1 − p)
on [0, 1].) Explain the practical consequence for reporting worst-case poll margins.

Exercise 4.6 (Challenging). A researcher wants to estimate a population mean to within E=2
units with 95% condence. A small pilot study suggests σ ≈ 12. Find the required sample size.
Then discuss how the required size changes if (i) the condence level is raised to 99%, and (ii)
the pilot has underestimated σ, the true value being 15.

24
 Chapter 5
Probability Distributions

The condence intervals of the last chapter leaned on the normal distribution, and the Central
Limit Theorem hinted that a handful of standard distributions govern an enormous range of
phenomena. This chapter studies the four that a data scientist meets most often: the Bernoulli
and Binomial distributions for counts of successes, the Poisson distribution for counts of rare
events, and the Normal distribution for measurements. We rst recall the language of random
variables.

5.1 Random Variables, Expectation, and Variance

A random variable X is a numerical outcome of a random experiment. It is discrete if it takes
isolated values (typically counts) and continuous if it can take any value in an interval (typically
measurements).

A discrete random variable is described by its

P probability mass function (pmf ) p(x) = P(X =
x), which must satisfy p(x) ≥ 0 and
x p(x) = 1. A continuous
R∞ random variable is described
by its probability density function (pdf ) f (x) ≥ 0 with f (x) dx = 1; for a continuous
−∞
variable, probability is area under the density, P(a ≤ X ≤ b) =
Rb
a f (x) dx, and the probability
of any single exact value is zero. In both cases the cumulative distribution function (cdf ) is
F (x) = P(X ≤ x).
The expectation (mean) and variance of X generalise the descriptive mean and variance of
Chapter 2 from data to distributions:

(P
x x p(x) (discrete)
σ 2 = Var(X) = E (X − µ)2 = E[X 2 ] − µ2 .
 
µ = E[X] = R
x f (x) dx (continuous),

The last identity, the distributional twin of Theorem 2.10, and the linearity rules E[aX + b] =
aE[X] + b and Var(aX + b) = a2 Var(X), are used repeatedly below.

5.2 The Bernoulli Distribution

The simplest non-trivial random variable models a single trial with two outcomes, generically
success (coded 1) and failure (coded 0)a coin toss, whether a component is defective,
whether a customer clicks.

25
Fundamentals of Data Science Unit I

Denition 5.1 (Bernoulli distribution) . A random variable X has the Bernoulli distribution
with parameter p ∈ [0, 1], written X ∼ Bernoulli(p), if it takes the value 1 with probability p
and 0 with probability 1 − p; that is, its pmf is

p(x) = px (1 − p)1−x , x ∈ {0, 1}.

Proposition 5.2 (Mean and variance of the Bernoulli). If X ∼ Bernoulli(p) then E[X] = p and
Var(X) = p(1 − p).

Proof. Directly, E[X] = 1 · p + 0 · (1 − p) = p. Since X ∈ {0, 1} we have X 2 = X , so E[X 2 ] =

E[X] = p, and
2 2 2
hence Var(X) = E[X ] − (E[X]) = p − p = p(1 − p).

The variance p(1 − p) is largest at p = 12 (maximum uncertainty, as with a fair coin) and
vanishes at p = 0 or p = 1 (a certain outcome has no variability)the same expression that
gave the worst-case poll margin in Chapter 4.

5.3 The Binomial Distribution

Now repeat a Bernoulli trial n times, independently and with the same success probability p,
and count the total number of successes. That count is the binomial random variable  the
number of heads in n tosses, the number of defective items in a batch of n, the number of clicks
among n visitors.

Denition 5.3 (Binomial distribution) . Let X be the number of successes in n independent

Bernoulli(p) trials. Then X has the binomial distribution with parameters n and p, written
X ∼ Bin(n, p), with pmf

 
n x
p(x) = p (1 − p)n−x , x = 0, 1, 2, . . . , n,
x

n

n!
where
x = is the binomial coecient.
x! (n − x)!

The formula is easy to read o. Any particular sequence with x
successes and n−x failures
n
has probability px (1 − p)n−x by independence; and there are
x distinct positions in which
the x successes can occur, so we multiply. That the probabilities sum to one is the binomial
Pn n
px (1 − p)n−x = (p + (1 − p))n = 1,


theorem: x=0 x which is where the distribution gets its
name. Figure 5.1 shows the shape for two choices of p.

The mean and variance could be found by grinding through the dening sums, but there is a
far more elegant route that also illuminates why the answers take the form they do.

Theorem 5.4 (Mean and variance of the binomial). If X ∼ Bin(n, p) then

E[X] = np, Var(X) = np(1 − p).

Proof. Write X = X1 + X2 + · · · + Xn , where Xi ∼ Bernoulli(p) indicates success on the ith trial;

this decomposition is exactly what count the successes means. By linearity of expectation and
Proposition 5.2,
n
X n
X
E[X] = E[Xi ] = p = np.
i=1 i=1

26
Fundamentals of Data Science Unit I

Bin(10, 0.5) Bin(10, 0.2)

0.3
0.2
0.2
p(x)

p(x)
0.1
0.1

0 0
0 2 4 6 8 10 0 2 4 6 8 10
x x

Figure 5.1: The binomial pmf. With p = 0.5 (left) it is symmetric about n/2; with p = 0.2
(right) it is right-skewed and concentrated near the smaller values.

Because the trials are independent, the variance of the sum is the sum of the variances, so again
by Proposition 5.2,

n
X n
X
Var(X) = Var(Xi ) = p(1 − p) = np(1 − p).
i=1 i=1

Remark 5.5 (Recurrence for fast computation) . Successive binomial probabilities satisfy the
ratio
p(x + 1) n−x p
= · ,
p(x) x+1 1−p
so one can start from p(0) = (1 − p)n and generate the whole distribution by repeated multi-
plication, avoiding factorials altogetheruseful both for hand computation and for numerically
stable code.

Example 5.1 (A binomial calculation) . A fair die is rolled 5 times; success is rolling a six,
so p = 1/6 and n = 5. What is the probability of at least one six? It is easiest to use the
complement:
 
5 1 0

5 5

5 5

3125
P(X ≥ 1) = 1 − P(X = 0) = 1 − 6 6 =1− 6 =1− 7776 ≈ 0.598.
0
1 5
The expected number of sixes is E[X] = np = 5/6 ≈ 0.83, with variance np(1 − p) = 5 · 6 · 6 =
25
36 ≈ 0.69.

5.4 The Poisson Distribution

The Poisson distribution models the number of times a random event occurs in a xed interval
of time or space, when the events happen independently at a constant average ratethe number
of calls arriving at a switchboard per minute, of typos per page, of radioactive decays per second,
of accidents at a junction per month.

Denition 5.6 (Poisson distribution). A random variable X has the Poisson distribution with
parameter λ > 0, written X ∼ Poisson(λ), if its pmf is

e−λ λx
p(x) = , x = 0, 1, 2, . . .
x!
Here λ is the mean number of occurrences in the interval.

27
Fundamentals of Data Science Unit I

P∞ x /x!
That the probabilities sum to one uses the exponential series x=0 λ = eλ :
∞ −λ x ∞
X e λ −λ
X λx
=e = e−λ eλ = 1.
x! x!
x=0 x=0

Theorem 5.7 (Mean and variance of the Poisson) . If X ∼ Poisson(λ) then E[X] = λ and
Var(X) = λ. The Poisson distribution is the notable case in which the mean and the variance
are equal.

Proof. For the mean, drop the x=0 term (which contributes nothing) and cancel one factor of
x:
∞ ∞ ∞
X e−λ λx X e−λ λx X λx−1
E[X] = x = = λe−λ = λe−λ eλ = λ,
x! (x − 1)! (x − 1)!
x=1 x=1 x=1

the inner sum being eλ after the substitution k = x − 1. For the variance it is convenient to
compute E[X(X − 1)] rst, cancelling two factors:
∞ ∞
X e−λ λx X λx−2
E[X(X − 1)] = x(x − 1) = λ2 e−λ = λ2 e−λ eλ = λ2 .
x! (x − 2)!
x=2 x=2

Then E[X 2 ] = E[X(X−1)]+E[X] = λ2 +λ, so Var(X) = E[X 2 ]−(E[X])2 = (λ2 +λ)−λ2 = λ.

The Poisson distribution is intimately related to the binomial: it is what the binomial becomes
when there are very many trials, each with a very small success probability, but a moderate
expected number of successes. This law of rare events is why the Poisson governs counts of
rare occurrences.

Theorem 5.8 (Poisson limit of the binomial) . Suppose

Xn ∼ Bin(n, pn ) where n → ∞ and
pn → 0 in such a way that npn → λ for a constant λ > 0. Then for each xed x = 0, 1, 2, . . .

e−λ λx
P(Xn = x) −→ .
x!

Proof. Put pn = λ/n (this satises npn = λ; the general case is a routine modication). The
binomial pmf is

λ −x
   x 
λ n−x λx λ n
    
n λ n!
P(Xn = x) = 1− = · · 1− 1− .
x n n x! (n − x)! nx n n

Group the factors as

λx λ n λ −x
     
n!
1− 1− .
x! (n − x)! nx n n
| {z } | {z } | {z }
(I) (II) (III)

n(n − 1) · · · (n − x + 1)
As n→∞ with x xed: term (I) = → 1, being a product of x factors
nx
each tending to 1; term (II) → e
−λ n
by the standard limit (1 − a/n) → e
−a ; and term (III) → 1
x
λ −λ
since λ/n → 0. Hence the whole expression tends to e , as claimed.
x!

In practice the Poisson gives an excellent approximation to the binomial whenever n is large
(say n ≥ 50) and p small (say p ≤ 0.1), with λ = np. Figure 5.2 shows the Poisson pmf for two
rates.

28
Fundamentals of Data Science Unit I

Poisson(2) Poisson(5)

0.15
0.2
p(x)

p(x)
0.1
0.1
5 · 10−2

0 0
0 2 4 6 8 0 3 6 9 12
x x

Figure 5.2: The Poisson pmf. For small λ it is strongly right-skewed; as λ grows it becomes
more symmetric and bell-shaped, foreshadowing its normal approximation.

Example 5.2 (A Poisson calculation). A call centre receives on average λ = 3 calls per minute.
What is the probability of exactly 5 calls in a given minute, and of no calls? Directly,

e−3 35 e−3 · 243

P(X = 5) = = ≈ 0.1008, P(X = 0) = e−3 ≈ 0.0498.
5! 120
So there is about a 10% chance of ve calls and a 5% chance of a completely quiet minute.

5.5 The Normal Distribution

The normal (or Gaussian) distribution is the continuous, bell-shaped distribution that the Cen-
tral Limit Theorem places at the centre of statistics. Heights, measurement errors, examination
scores, and countless averages are approximately normal, andby Theorem 3.3so is almost
any sample mean.

Denition 5.9 (Normal distribution). X has

A continuous random variable the normal distri-
bution with mean µ and variance σ2, X ∼ N (µ, σ 2 ), if its pdf is
written

(x − µ)2
 
1
f (x) = √ exp − , −∞ < x < ∞.
σ 2π 2σ 2

The special case µ = 0, σ = 1 is the standard normal, whose variable is conventionally called Z
and whose cdf is denoted Φ(z).

The curve is symmetric about µ, where it attains its single peak, so for the normal the mean,
median, and mode coincide. It has inection points exactly one standard deviation on either
side of the mean, at x = µ ± σ, which gives σ a direct visual meaning as the width of the bell.
The parameter µ shifts the curve left or right without changing its shape, while σ stretches or
compresses it. The total area under the curve is one, as every density must be.

Any normal variable can be converted to the standard normal by subtracting its mean and
dividing by its standard deviationthe operation we already used implicitly in Chapter 4.

X −µ
Theorem 5.10 (Standardisation). If X ∼ N (µ, σ2 ) then Z = ∼ N (0, 1). Consequently,
σ
for any a < b,    
b−µ a−µ
P(a ≤ X ≤ b) = Φ −Φ .
σ σ

29
Fundamentals of Data Science Unit I

Proof sketch. Z is a linear function of X, so by the linearity rules E[Z] = (E[X] − µ)/σ = 0 and
Var(Z) = Var(X)/σ 2 = 1. That Z remains normal follows because a linear transformation of a
normal variable is normal (a change of variables in the density conrms it). Hence Z ∼ N (0, 1),
a−µ b−µ
and rewriting the event a≤X≤b as
σ ≤Z≤ σ gives the probability in terms of Φ.

The value z = (x − µ)/σ is the z -score of x: it measures how many standard deviations x
lies above (z > 0) or below (z < 0) the mean, and makes observations from dierent normal
distributions directly comparable. Areas under the standard normal are read from tables of Φ or
computed in software. A particularly memorable consequence is the empirical rule, illustrated
in Figure 5.3.

Proposition 5.11 (The 689599.7 rule). For X ∼ N (µ, σ2 ), approximately 68% of the proba-
bility lies within one standard deviation of the mean, 95% within two, and 99.7% within three:

P(|X − µ| ≤ σ) ≈ 0.68, P(|X − µ| ≤ 2σ) ≈ 0.95, P(|X − µ| ≤ 3σ) ≈ 0.997.

density

68%

95%
99.7%
µ − 3σ µ − 2σ µ−σ µ µ+σ µ + 2σ µ + 3σ

x (in units of σ from µ)

Figure 5.3: The normal curve and the empirical (689599.7) rule. The shaded central band
holds about two-thirds of the area within one σ; essentially all the area lies within three.

Finally, the binomial itself becomes approximately normal when n is large, an instance of
the Central Limit Theorem (a binomial is n independent Bernoulli variables).
If
a sum of
X ∼ Bin(n, p) with both np ≥ 5 and n(1 − p) ≥ 5, then X is approximately N np, np(1 − p) .
Because we approximate a discrete distribution by a continuous one, we apply a continuity
correction, replacing the integer event X = k by the interval k− 1
2 ≤ X ≤ k+ 1
2 before
standardising.

Example 5.3 (Normal probability and a normal approximation) . (a) Adult male heights are
approximately N (µ = 170, σ2 = 82 ) cm. The probability that a randomly chosen man is taller
than 182 cm is, by standardising,

 
182 − 170
P(X > 182) = P Z > = P(Z > 1.5) = 1 − Φ(1.5) ≈ 1 − 0.9332 = 0.0668.
8

(b) A fair coin is tossed n

p= 100 times; approximate
√ the probability of at least 60 heads. Here
µ = np = 50 and σ = np(1 − p) = 25 = 5. With the continuity correction,  X ≥ 60
becomes  X ≥ 59.5, so

 
59.5 − 50
P(X ≥ 60) ≈ P Z ≥ = P(Z ≥ 1.9) = 1 − Φ(1.9) ≈ 0.0287.
5

30
Fundamentals of Data Science Unit I

5.6 Summary of the Four Distributions

Table 5.1 collects the denitions and moments derived above for quick reference. Notice the
family relationships: a binomial is a sum of n independent Bernoullis; a Poisson is the limit of a
binomial with many rare trials; and both the binomial and the Poisson become approximately
normal as their means grow, by the Central Limit Theorem.

Table 5.1: The four standard distributions of Unit I.

Distribution Type pmf / pdf Mean Variance

Bernoulli(p) discrete px (1 − p)1−x , x ∈ {0, 1} p p(1 − p)

n x n−x


Bin(n, p) discrete
x p (1 − p) np np(1 − p)
e−λ λx
Poisson(λ) discrete λ λ
x!
1 −(x−µ) 2 /2σ 2
N (µ, σ 2 ) continuous √ e µ σ2
σ 2π

5.7 Probability Distributions in Python

The [Link] module provides every distribution with a uniform interface: pmf or pdf for
the mass/density, cdf for cumulative probabilities, ppf for quantiles (the inverse cdf ), and rvs
for random draws.

1 from scipy import stats

2
3 # Binomial : P ( at least one six in 5 rolls of a fair die )
4 print (1 - stats . binom . pmf (0 , n =5 , p =1/6) ) # 0.598...
5 print ( " mean , var : " , stats . binom . stats ( n =5 , p =1/6 , moments = " mv " ) )
6
7 # Poisson : P ( exactly 5 calls ) when lambda = 3
8 print ( stats . poisson . pmf (5 , mu =3) ) # 0.1008...
9
10 # Normal : P ( X > 182) for N (170 , 8^2)
11 print (1 - stats . norm . cdf (182 , loc =170 , scale =8) ) # 0.0668...
12
13 # Standard - normal critical value used for 95% intervals
14 print ( stats . norm . ppf (0.975) ) # 1.959...
15
16 # Poisson as a limit of the binomial ( Theorem 5.10) :
17 import numpy as np
18 for n in [10 , 100 , 1000]:
19 approx = stats . binom . pmf (2 , n =n , p =3/ n ) # np = 3 fixed
20 print (n , round ( approx , 5) ) # -> Poisson (3) .
pmf (2) =0.2240
21 print ( " target : " , round ( stats . poisson . pmf (2 , mu =3) , 5) )

31
Fundamentals of Data Science Unit I

Exercises
Exercise 5.1 (Routine) . A biased coin has probability p = 0.3 of landing heads. It is tossed
8 times. Find the probability of (a) exactly 3 heads, (b) at most 1 head, and (c) the expected
number of heads and its variance.

Exercise 5.2 (Routine) . Accidents at a road junction occur at an average rate of λ=2 per
week. Assuming a Poisson model, nd the probability of (a) no accidents in a given week, (b)
exactly two accidents, and (c) more than three accidents.

Exercise 5.3 (Routine). The marks in an examination are approximately N (µ = 60, σ = 12).
What proportion of students score (a) above 75, (b) between 54 and 72? Using the empirical
rule, within what symmetric range about the mean do about 95% of marks lie?

Exercise 5.4 (Moderate). A manufacturer knows that 2% of its microchips are defective. In a
shipment of 200 chips, use the Poisson approximation to the binomial (Theorem 5.8) to estimate
the probability that at most 3 are defective. Identify the value of λ you use and justify that the
approximation is appropriate.

p(x + 1) n−x p
Exercise 5.5 (Moderate) . Prove the recurrence relation = · for the
p(x) x+1 1−p
binomial pmf stated in the Remark of Section 5.3. Use it to build the full distribution of
Bin(4, 0.5) starting from p(0).

Exercise 5.6 (Moderate) . √X ∼ Poisson(λ), show using the meanvariance results that
For
its coecient of variation is 1/ λ. Interpret what this says about the relative variability of
rare-event counts as the average rate increases.

Exercise 5.7 (Challenging). A fair die is rolled 180 times. Let X be the number of sixes. Using
the normal approximation to the binomial with continuity correction, estimate P(25 ≤ X ≤ 35).
State the mean and standard deviation you use and explain why the continuity correction is
needed.

Exercise 5.8 (Challenging). Starting from the denition, prove that if X1 ∼ Poisson(λ1 ) and

Pk2 ) are independent, then X1 + X2 ∼ Poisson(λ1 + λ2 ). (Hint: compute P(X1 +

X2 ∼ Poisson(λ
X2 = k) = j=0 P(X1 = j)P(X2 = k − j) and recognise a binomial expansion.) Explain
intuitively why the sum of two independent Poisson counts is again Poisson.

Exercise 5.9 (Challenging)

√
. An unbiased estimator debate. Recall from Chapter 3 that X̄
estimates µ with standard error σ/ n. Combine this with the Central Limit Theorem to explain,
√
in your own words and with a sketch, why the condence interval X̄ ± 1.96 σ/ n of Chapter 4
has approximately 95% coverage even when the underlying population is a skewed distribution
such as the Poisson, provided n is reasonably large.

32
Further Reading

The following texts extend the material of this unit; the rst two are widely used Indian text-
books that treat the descriptive and probability content in detail, while the remainder give
complementary and computational perspectives.

ˆ S. C. Gupta and V. K. Kapoor, Fundamentals of Mathematical Statistics, Sultan Chand &

Sons, New Delhi (latest revised edition). A comprehensive standard reference for descriptive
statistics, probability, and distributions, with many solved problems.

ˆ J. Medhi, Statistical Methods: An Introductory Text, New Age International, New Delhi. A
clear introduction to descriptive statistics, sampling, and probability distributions suited to
rst courses.

ˆ S. M. Ross, A First Course in Probability, 10th ed., Pearson, 2018/2019. A rigorous and
readable development of random variables and the standard distributions, with the Poisson
limit and central limit theorems.

ˆ D. Freedman, R. Pisani, and R. Purves, Statistics, 4th ed., W. W. Norton, 2007. An ex-
ceptionally intuitive account of sampling, condence intervals, and the meaning of the box
model, with minimal formalism.

ˆ J. VanderPlas, Python Data Science Handbook, 2nd ed., O'Reilly Media, 2023. A practical
guide to NumPy, pandas, and the scientic-Python stack used for the computations in these
notes; the full text is also freely available online.

End of Unit I.

33