Data and Sampling Distributions

By
Dr G S Madhan Kumar

September 1, 2025

Data and Sampling Distributions September 1, 2025 1 / 107

Chapter Overview: Data and Sampling Distributions

Focus: Why sampling is essential, even with big data.

Key Ideas:
Random sampling reduces bias and improves data quality.
Population vs. Sample – we usually work with samples, not the full
population.
Sampling distributions explain how statistics vary across samples.
Central Limit Theorem: sample means tend to be normally distributed.
Standard error measures uncertainty in sample estimates.
Bootstrap and confidence intervals help quantify variability.
Common probability distributions: Normal, t, Binomial, Chi-square, F,
Poisson.
What We Learn: Tools to measure uncertainty, avoid bias, and
model real-world data.

Data and Sampling Distributions September 1, 2025 2 / 107

Why Sampling Matters Even in Big Data

A common misconception is that the era of big data has eliminated the need for
sampling. In reality, the huge amount of data available today is often of mixed
quality and relevance. Using all of it directly can be inefficient and may introduce
bias.
Sampling is therefore still essential. By selecting smaller, representative subsets of
data, we can:
Build and test predictive models more efficiently.
Conduct experiments, such as comparing the effect of different web page
designs on user clicks.
Focus on improving data quality, rather than being overwhelmed by data
quantity.

Data and Sampling Distributions September 1, 2025 3 / 107

In statistics, the population refers to all possible data points, which
are assumed to follow some underlying but often unknown
distribution.
A sample, on the other hand, is the observed data we actually work
with.
Traditional statistics: Focused on population assumptions.
Modern statistics: Focuses on sampling and real data instead.
Example: Coin flip (binomial distribution) → buy/don’t buy,
fraud/no fraud, click/no click.

Data and Sampling Distributions September 1, 2025 4 / 107

Population vs. Sample

In statistics, a sample is a subset of data taken from a larger data set, which is
called the population.
Unlike in biology, a population in statistics is not a group of people or organisms.
Instead, it refers to a large, well-defined, and sometimes even theoretical or imagi-
nary set of data.
Since it is often impossible or impractical to study the entire population, we use
samples to make inferences about it.

Data and Sampling Distributions September 1, 2025 5 / 107

Population versus Sample

Data and Sampling Distributions September 1, 2025 6 / 107

Random Sampling

Random sampling is a method where every member of the population has an

equal chance of being chosen for the sample.
The resulting subset of data is called a simple random sample.
Sampling can be performed in two ways:
With replacement: After being chosen, each observation is placed back
into the population and may be selected again.
Without replacement: Once chosen, an observation is not returned to the
population, and cannot be selected again.

Data and Sampling Distributions September 1, 2025 7 / 107

Data Quality and Representativeness

When building models or estimates, data quality often matters more than data
quantity.
In data science, quality includes:
Completeness of data
Consistency of format
Cleanliness of records
Accuracy of individual values

Statistics adds one more vital aspect: representativeness, which ensures that the
chosen sample truly reflects the characteristics of the entire population.

Data and Sampling Distributions September 1, 2025 8 / 107

The Literary Digest Poll of 1936

A famous example of sample bias comes from the 1936 U.S. Presidential election.
The magazine Literary Digest conducted a poll of over 10 million people, in-
cluding its subscribers and names from automobile and telephone directories.
The results predicted a landslide victory for Alf Landon over Franklin D. Roo-
sevelt.
In reality, Roosevelt won easily. The prediction failed because the poll’s sample
was biased.

Data and Sampling Distributions September 1, 2025 9 / 107

Why Gallup Was Right

George Gallup, founder of the Gallup Poll, surveyed only about 2,000 people but
did so carefully using methods to ensure the sample was representative of the
U.S. population.
Key reason for Digest’s failure:
Their sample included mostly wealthy individuals (magazine subscribers, car
owners, telephone owners).
This group did not represent the average U.S. voter in 1936.

Lesson: Quantity of data is not enough. What matters more is the quality and
representativeness of the sample.

Data and Sampling Distributions September 1, 2025 10 / 107

Sample Bias: Literary Digest vs. Gallup

Literary Digest Poll Gallup Poll

10 million responses Only 2,000 responses
Mostly wealthy: Carefully selected,
subscribers, car owners, telephone owners representative of U.S. voters
Predicted: Landon wins Predicted: Roosevelt wins

Actual Outcome: Roosevelt won in a landslide

Data and Sampling Distributions September 1, 2025 11 / 107

Sample Bias: Representative vs. Biased Samples

Population
Large group of individuals (diverse characteristics)

Representative Sample Biased Sample

Smaller group Smaller group
Reflects population diversity Overrepresents some, ignores others

Data and Sampling Distributions September 1, 2025 12 / 107

Understanding Bias in Statistics

Bias refers to systematic errors introduced by the way data is measured or sampled.
Unlike random errors, which occur unpredictably, bias consistently shifts results in
a particular direction.
Key distinction:
Random error: Natural variation that occurs by chance. It averages out
over many observations.
Bias: A consistent deviation caused by flawed measurement, sampling, or
model assumptions. It does not disappear with more data.

Bias can be visible (detectable by comparison with benchmarks) or invisible (hidden

inside the process). Often, bias suggests that a model is misspecified or that
important variables have been left out.

Data and Sampling Distributions September 1, 2025 13 / 107

Scatterplot of shots from a gun with true aim

Data and Sampling Distributions September 1, 2025 14 / 107

Scatterplot of shots from a gun with biased aim

Data and Sampling Distributions September 1, 2025 15 / 107

Random Selection: Motivation

To avoid the problem of sample bias that led the Literary Digest to wrongly predict
a Landon victory, George Gallup used scientifically chosen methods to achieve a
sample that was representative of the U.S. electorate.
Key point: At the heart of all modern sampling methods lies the principle of
random selection.

Data and Sampling Distributions September 1, 2025 16 / 107

Random Sampling: Defining the Population

Random sampling is not always straightforward. A crucial step is to define the

accessible population.
Example: Customer survey
Who counts as a customer?
All past purchasers?
Refund cases?
Internal test purchases?
Resellers, billing agents, end customers?
The definition of the population strongly affects representativeness.

Data and Sampling Distributions September 1, 2025 17 / 107

Sampling Procedure: Database Example

Once the population is defined, a sampling procedure must be chosen.

Case 1: Sampling from a Database (static data)
Example: Select 100 customers at random from the company database.
Each customer has an equal chance of being chosen.
Works well when data is already stored and fixed.

Data and Sampling Distributions September 1, 2025 18 / 107

Sampling Procedure: Streaming/Flow Example

Case 2: Sampling from a Flow (real-time data)

Example: Sampling real-time website visitors or transactions.
Timing matters:
Weekday vs. weekend visitors may behave differently.
Morning vs. late-night visitors may represent different user groups.
Important when data arrives continuously and user behavior changes with
time.

Data and Sampling Distributions September 1, 2025 19 / 107

Stratified Sampling: Concept

Sometimes a population contains important subgroups (strata) that must be rep-

resented.
Steps in Stratified Sampling:
Divide the population into meaningful subgroups (strata).
Randomly select samples from each subgroup.

Why use it?

A simple random sample may underrepresent smaller groups.
Stratified sampling ensures that all groups are included fairly.

Data and Sampling Distributions September 1, 2025 20 / 107

Stratified Sampling: Example

Political Polling Example:

U.S. electorate contains different racial/ethnic groups.
If we take a simple random sample:
Whites may dominate the sample.
Blacks and Hispanics may be underrepresented.
Stratified sampling ensures that all groups (whites, blacks, Hispanics) are
proportionately represented in the sample.

Key Point: Stratified sampling = “sample a little from everyone.”

Data and Sampling Distributions September 1, 2025 21 / 107

Smaller Can Be Better in Data Science

In the era of big data, it is a common misconception that more data is always
better. In fact, random sampling can reduce bias and make the analysis more
efficient. Working with a smaller dataset allows us to carefully examine the data
in ways that are not practical with millions of records.
With a smaller, well-chosen dataset, we can:
Check for missing values and understand where information is incomplete.
Inspect outliers and determine whether they are errors or contain important
insights.
Ensure data consistency, such as verifying that formats for dates, IDs, or
measurements are uniform.

Exploring millions of records manually is slow and often impossible, but sampling
enables closer inspection and higher quality analysis.

Data and Sampling Distributions September 1, 2025 22 / 107

When Does Dataset Size Matter?

Big data is not always necessary, but it becomes essential when the data is both
large and sparse.
What does sparse mean?
Sparse data contains mostly empty or zero values.
Example: Google search queries can be represented as a huge table:
Rows = individual search queries.
Columns = words in the English language (150,000+ words).
Cells = 1 if a word appears in the query, 0 otherwise.
Since most queries only have a few words, the vast majority of entries are
zero.

Data and Sampling Distributions September 1, 2025 23 / 107

Big Data and Rare Queries

Big data is most useful when we deal with rare events or unusual queries.
If the query is common (e.g., “weather today”), we don’t need billions of
records—patterns appear quickly.
If the query is rare (e.g., “Ricky Ricardo and Little Red Riding Hood”), it
might only appear once in millions or billions of records.
Only with huge amounts of data can we gather enough examples of such
rare queries to return good results.

Data and Sampling Distributions September 1, 2025 24 / 107

Example: Search Engines

Google processes trillions of searches every year.

For popular queries, useful results are easy to generate with relatively little
data.
For rare and specific queries, however, massive amounts of data are required.
Example: Without trillions of stored queries, Google could not return
accurate results for unusual searches like:
“Ricky Ricardo and Little Red Riding Hood”

Data and Sampling Distributions September 1, 2025 25 / 107

Key Point: The Long Tail of Big Data

The true value of big data is not just in answering common and popular
queries.
Its power lies in handling the “long tail”—rare, unusual, or highly specific
queries.
Big data makes it possible to give accurate and useful answers even when
events are extremely rare.

Data and Sampling Distributions September 1, 2025 26 / 107

Sample Mean vs. Population Mean

Statisticians distinguish between:

x̄ (x-bar) = sample mean (average from a subset).
µ (mu) = population mean (average from the whole population).

Why separate symbols?

We observe and calculate from samples.
We infer about the population from those samples.
Keeping them distinct avoids confusion.

Data and Sampling Distributions September 1, 2025 27 / 107

Selection Bias

What is Selection Bias?

Occurs when data is chosen in a way that produces misleading results.
Can happen consciously (e.g., cherry-picking data) or unconsciously (e.g.,
stopping an experiment early).
Final conclusions reflect the way data was selected, not the true underlying
reality.

Why is it dangerous in Data Science?

Large datasets encourage searching for patterns without a clear hypothesis.
The more we search through the data, the more likely it becomes that we
will find patterns that appear interesting but are actually due to chance.
As statisticians say: ”If you torture the data long enough, sooner or later it
will confess.”

Data and Sampling Distributions September 1, 2025 28 / 107

Selection Bias: Coin Toss Example
Selection bias can make random chance look meaningful. This can be seen with
the example of coin tosses:
Case A: Hypothesis Testing
One person claims: “I can flip 10 heads in a row.”
We test the claim directly: they flip 10 times and succeed.
This is surprising, since the probability is only 1/1000.
If true, it may suggest real skill or some special factor.

Case B: Data Snooping (Selection Bias)

Imagine 20,000 people each flip a coin 10 times.
By chance alone, it is almost guaranteed that at least one person will get 10
heads.
If we only look at that person afterwards, it looks impressive, but it is just
luck.
This is selection bias: choosing extreme cases after the fact makes chance
events look meaningful.
Data and Sampling Distributions September 1, 2025 29 / 107
Why is the Probability about 1/1000?

1
Each coin flip has a probability of 2 of landing on heads.
To get 10 heads in a row:
1 10 1


P(10 Heads) = 2 = 1024 ≈ 0.001

Data and Sampling Distributions September 1, 2025 30 / 107

Hypothesis Testing vs. Data Snooping (Selection Bias)

Hypothesis Testing (Good)

You decide your question first.
Then you collect data to test that question.
If the result is unusual, you can trust it.
Example:
A company says: “Our new ad will increase sales compared to the old
one.”
They run an A/B test: half the customers see the old ad, half see the
new ad.
Then they compare sales between the two groups.
If sales go up, it is strong evidence the new ad works.

Data and Sampling Distributions September 1, 2025 31 / 107

Data Snooping / Selection Bias (Bad)
You do not decide the question first.
Instead, you look at lots of data until you find something interesting.
Often, this is just coincidence, not a real effect.
Example:
The same company collects all customer data and tries many
comparisons: men, women, young, old, different cities, etc.
After many tests, they notice: “The ad works great for left-handed
customers in New York!”
This pattern may just be luck, not reality.
This is selection bias: finding a “result” only because you searched
long enough.

Data and Sampling Distributions September 1, 2025 32 / 107

The Vast Search Effect (Selection Bias) (John Elder)
What is it?
The vast search effect happens when we test too many things on the same
dataset.
If we keep running many models or asking many questions, we will
eventually find something that looks interesting.
But often, that “interesting” result is not real—it is just a random fluke.

Example:
Imagine flipping a coin 1,000 times.
If you divide the flips into many groups, one group might look like it gives
heads almost every time.
This does not mean the coin is special—it just happened by chance.
In data science, the same thing can happen: searching long enough will
reveal patterns that are not truly meaningful.
Data and Sampling Distributions September 1, 2025 33 / 107
Guarding Against Selection Bias

There are several techniques to reduce the risk of selection bias in data science:
Holdout sets: Keep aside a portion of the data that is never used during
training. Test results only on this untouched data.
Multiple holdouts: Sometimes more than one holdout set is needed for
added confidence.
Target shuffling (permutation tests): Randomly shuffle the outcome
labels and rerun the model. If the same “interesting” pattern still appears, it
was likely due to chance.
Avoid cherry-picking: Do not selectively choose samples, time periods, or
stop experiments early just because the results look favorable.

By applying these methods, data scientists can ensure that the insights they find
are real and not artifacts of biased selection.

Data and Sampling Distributions September 1, 2025 34 / 107

Holdout Sets, Multiple Holdouts, Target Shuffling

1. Holdout Set
A holdout set means splitting data into train and test.
You train your model on one part and test on the other.
Purpose: check performance on unseen data.

2. Multiple Holdouts
Instead of one train/test split, create several splits (e.g., 70/30, 80/20,
different random seeds).
Each split gives slightly different results.
Purpose: reduce dependency on one random split.

Data and Sampling Distributions September 1, 2025 35 / 107

Holdout Sets, Multiple Holdouts, Target Shuffling

3. Target Shuffling (Permutation Tests)

A different approach: randomly shuffle the target labels (e.g., fraud
vs. not fraud).
Train models on this shuffled data.
If the model still performs well, the original result may be due to
chance.
Purpose: test whether the relationship found is real or just noise.

Data and Sampling Distributions September 1, 2025 36 / 107

Regression to the Mean

Definition:
Regression to the mean means that when something is unusually high or low
the first time we measure it, the next time it usually comes closer to the
average.

Why it Happens:
Results depend on two factors:
Skill (true ability)
Luck (random chance)
Extreme results often include both skill and a lot of luck.
On the next try, skill remains, but luck usually evens out → the result moves
closer to the average.

Data and Sampling Distributions September 1, 2025 37 / 107

Example: Rookie of the Year
What is a Rookie?
A rookie is a player in their first year of professional sports.
“Rookie of the Year” is an award for the best-performing first-year player.

Rookie of the Year, Sophomore Slump:

In the first year, one rookie outperforms all others.
This is due to both skill and good luck.
In the second year, the skill remains, but the luck often disappears.
Performance drops closer to the average → this is regression to the mean.

Other Example: Height

Very tall fathers tend to have tall children.
But on average, the children are not as tall as the father.
This happens because extreme values move back toward the average over
time.
Data and Sampling Distributions September 1, 2025 38 / 107
Sampling Distribution of a Statistic

A sampling distribution is the distribution of a sample statistic (e.g.,

mean, median) over many samples drawn from the same population.
Different samples usually give slightly different results.
This variation is called sampling variability.

Data and Sampling Distributions September 1, 2025 39 / 107

Why Sampling Distribution Matters

In classical statistics, we often have only a small sample but want to make
inferences about a large population.
Since our estimate (mean, proportion, etc.) depends on the sample, it may
differ if we take another sample.
The sampling distribution tells us how much variation we can expect.

Data and Sampling Distributions September 1, 2025 40 / 107

Key Properties of Sampling Distribution

The sampling distribution is usually more regular and bell-shaped than

the raw data.
Sample size matters:
Larger samples → less variability, narrower distribution.
Smaller samples → more variability, wider distribution.

Data and Sampling Distributions September 1, 2025 41 / 107

Sampling distribution — idea

Imagine you want to know the average height of students in your school,
but you don’t measure everyone. Instead, you take samples (for example,
measure 10 students, then another 10, etc.). Each time you take a sample,
you get a slightly different average. All these different averages together
form a sampling distribution.

Data and Sampling Distributions September 1, 2025 42 / 107

What happens when you repeat sampling

Each sample gives an average. Different samples give different averages

because the students picked are different. The collection of those averages
is the sampling distribution — it shows how much the sample averages vary.

Data and Sampling Distributions September 1, 2025 43 / 107

Larger sample size (e.g., 100 students)

Each sample’s average is more accurate.

Different samples don’t vary much from each other.
The sampling distribution is narrow and tight.

Data and Sampling Distributions September 1, 2025 44 / 107

Smaller sample size (e.g., 5 students)

Each sample’s average may swing a lot (one very tall or short student
changes the result).
Different samples vary more.
The sampling distribution is wide and spread out.

Data and Sampling Distributions September 1, 2025 45 / 107

In short

Bigger samples give more stable and consistent results; smaller sam-
ples give noisier and more spread-out results.

Data and Sampling Distributions September 1, 2025 46 / 107

Raw Data

Histogram of annual incomes of 1,000 loan applicants.

Data is messy and spread out (some earn a lot, some less).
Shows high variability.

Data and Sampling Distributions September 1, 2025 47 / 107

Sample Means of 5

Take 5 incomes at a time, find their average.

Repeat this 1,000 times.
Histogram of these averages is less spread out.
Variability decreases compared to raw data.

Data and Sampling Distributions September 1, 2025 48 / 107

Sample Means of 20

Take 20 incomes at a time, find their average.

Repeat this 1,000 times.
Histogram becomes very tight and narrow.
Averages are clustered closely around the true mean.

Data and Sampling Distributions September 1, 2025 49 / 107

Key Lesson

With bigger samples, averages become more stable.

Smaller samples show more bounce and spread.
Raw data (sample size = 1) is the most variable.
In short: more data ⇒ less variability in sample means.

Data and Sampling Distributions September 1, 2025 50 / 107

Histogram

Data and Sampling Distributions September 1, 2025 51 / 107

Central Limit Theorem (CLT)

When we take means from many samples, they form a bell-shaped

curve (normal distribution).
This happens even if the original data is not normal, provided the
sample size is large enough and the data is not too extreme.

Data and Sampling Distributions September 1, 2025 52 / 107

Why CLT Matters

CLT allows us to use normal-approximation formulas (like the

t-distribution).
These are used to calculate sampling distributions for inference:
Confidence intervals
Hypothesis tests
This is why CLT receives a lot of attention in traditional statistics.

Data and Sampling Distributions September 1, 2025 53 / 107

CLT in Data Science

In practice, data scientists rely less on formal hypothesis tests and

confidence intervals.
The bootstrap provides an alternative for inference.
Therefore, CLT is conceptually important but less central in data
science practice.

Data and Sampling Distributions September 1, 2025 54 / 107

Standard Error (SE)

The Standard Error measures how much a sample statistic (like the
mean) is expected to vary across samples.
Bigger samples ⇒ smaller SE (more stable averages).
Smaller samples ⇒ larger SE (averages bounce more).

Data and Sampling Distributions September 1, 2025 55 / 107

Standard Error Formula

Formula
s
SE = √
n

s = sample standard deviation.

n = sample size.
As n increases, SE decreases.

Data and Sampling Distributions September 1, 2025 56 / 107

Square Root of n Rule

SE decreases as sample size n increases.

To reduce SE by half, you must increase the sample size by a factor of
4.
This relationship is often called the square root of n rule.

Data and Sampling Distributions September 1, 2025 57 / 107

Understanding SE without CLT

1 Collect many new samples from the population.

2 Calculate the statistic (e.g., mean) for each sample.
3 Compute the standard deviation of those statistics.
4 That value is the Standard Error.

Data and Sampling Distributions September 1, 2025 58 / 107

Standard Error in Practice

Collecting new samples repeatedly is not practical.

Instead, use the bootstrap:
Resample from your existing data (with replacement).
Compute the statistic many times.
Use the spread of these bootstrap statistics as your SE.
Works for many statistics and avoids relying on CLT.

Data and Sampling Distributions September 1, 2025 59 / 107

The Bootstrap (Better Explained)

Normally, to understand how much a statistic (like a mean) might

vary, you would take many new samples from the population.
But in reality, we usually only have one dataset.
Solution: Treat your dataset as if it were the population.
Randomly resample from it with replacement (so the same point
may appear more than once).
Each resample looks a little different from the original dataset.
For each resample, calculate your statistic (mean, median, regression
slope, etc.).
Repeat this process many times (hundreds or thousands).
The collection of these resample statistics forms the bootstrap
sampling distribution.

Data and Sampling Distributions September 1, 2025 60 / 107

Why Bootstrap is Useful

Provides estimates of variability, confidence intervals, and standard

error without needing extra data.
Unlike the Central Limit Theorem, it does not require normality
assumptions.
Simple, flexible, and works for almost any statistic.

Data and Sampling Distributions September 1, 2025 61 / 107

Bootstrap Example

Suppose we have a dataset of exam scores: {60, 70, 80, 90, 100}.
We want to know how variable the mean score is.
Steps:
1 Randomly resample 5 scores with replacement, e.g., {70, 70, 80, 100,
60}.
2 Compute the mean of this resample.
3 Repeat thousands of times to create many means.
4 Look at the spread (standard deviation) of these means — this is the
bootstrap estimate of SE.
Result: We get a distribution of means, even though we only had one
dataset.

Data and Sampling Distributions September 1, 2025 62 / 107

The Idea of the Bootstrap

Original dataset is treated as if it were the entire population.

We create new datasets (called bootstrap samples) by sampling
with replacement from the original.
Each resample is the same size as the original dataset, but may
contain duplicates.
For each resample, calculate the statistic of interest (mean, median,
regression coefficient, etc.).
Repeating this process many times gives us a distribution of the
statistic: the bootstrap sampling distribution.

Data and Sampling Distributions September 1, 2025 63 / 107

Bootstrap Algorithm (for the Mean)

1 Draw a sample value, record it, then replace it.

2 Repeat until you have n values (same size as original sample).
3 Compute the mean of these n values.
4 Repeat steps 1–3, R times (hundreds or thousands).
5 Use the R means to estimate:
Standard Error (SE)
Confidence Intervals
Sampling variability

Data and Sampling Distributions September 1, 2025 64 / 107

Bootstrap Example

Suppose our original sample is: {2, 3, 6, 7}.

Resample with replacement (size 4): e.g., {3, 7, 7, 2} → mean = 4.75.
Another resample: {2, 2, 6, 7} → mean = 4.25.
Another resample: {3, 3, 6, 6} → mean = 4.5.
After many resamples, we have a distribution of means.
This distribution approximates the sampling distribution of the
mean.

Data and Sampling Distributions September 1, 2025 65 / 107

Bootstrap with Multivariate Data

In real datasets, we usually have many variables (columns). Each row

represents one complete observation (a person, a record, etc.).
In the bootstrap, we resample entire rows (not individual values) to
create new datasets.
We then fit our model on each bootstrapped dataset. This helps us
to:
See how much the model parameters (e.g., regression coefficients) vary.
Check the stability and reliability of the model.
Sometimes even improve prediction accuracy.
Example: Decision Trees
Train many trees on different bootstrap samples.
For regression: average the predictions.
For classification: take the majority vote.
This process is called Bagging (short for Bootstrap Aggregating).

Data and Sampling Distributions September 1, 2025 66 / 107

Why the Bootstrap Matters

For the mean, theoretical sampling distributions have been well

known since 1908.
For many other metrics (medians, regression coefficients, error rates),
no closed-form sampling distribution exists.
Bootstrap allows:
Estimation of variability and confidence intervals.
Sample size determination (by experimenting with different n).

Data and Sampling Distributions September 1, 2025 67 / 107

Resampling vs. Bootstrapping

Resampling is a broad term that may include:

Bootstrapping (sampling with replacement).
Permutation tests (sampling often without replacement).
Bootstrap specifically refers to:
Sampling with replacement from the observed dataset.
Treating the sample as the “population” for inference.

Data and Sampling Distributions September 1, 2025 68 / 107

Confidence Intervals

When we calculate a statistic (e.g., sample mean), it will not

perfectly equal the true population value.
To understand uncertainty, we use:
Frequency tables, histograms, and boxplots (visual methods).
Standard errors (numerical measure of variability).
Confidence Intervals (CIs).
A Confidence Interval gives a range of plausible values for the
population parameter.
Example: A 95% CI means that if we repeated sampling many times,
about 95% of the intervals would contain the true value.

Data and Sampling Distributions September 1, 2025 69 / 107

Why Confidence Intervals?

People prefer single numbers (point estimates), but these hide

uncertainty.
A Confidence Interval (CI) presents a range of plausible values
instead of one number.
This makes the uncertainty explicit and prevents overconfidence.
Each CI has a coverage level, e.g., 90% or 95%.
Example: A 95% CI means that if we repeated the sampling process
many times, about 95% of the intervals would capture the true
population value.

Data and Sampling Distributions September 1, 2025 70 / 107

Bootstrap Confidence Interval Algorithm

1 Given a dataset of size n, draw a resample of size n (with

replacement).
2 Compute the statistic of interest (e.g., mean, median, regression
slope).
3 Repeat steps 1–2 many times (say R resamples).
4 Collect the R bootstrap statistics into a distribution.
5 For an x% CI:
Trim 100−x
2 % of values from each tail of the distribution.
The remaining middle range gives the CI.

Data and Sampling Distributions September 1, 2025 71 / 107

What is Normal Distribution?

It is a bell-shaped curve in statistics.

Most of the data points lie around the center (mean), and fewer
points are found as you move away from the center.
Example: If you measure the height of many people, most will be
around average height, while very few will be very short or very tall.

Data and Sampling Distributions September 1, 2025 72 / 107

Simple Example

Think about exam scores (out of 100) of many students:

Average = 50
Most students score around 50 (say between 40–60).
Fewer students score very low (below 20) or very high (above 80).
If you draw this data, it will form a bell-shaped curve → normal
distribution.

Data and Sampling Distributions September 1, 2025 73 / 107

Key Features

Symmetrical – The left side and right side of the curve are mirror
images.
Mean = Median = Mode – The peak of the curve is the average
value.
Spread depends on Standard Deviation (σ):
Small σ → curve is tall and narrow (data is tightly packed).
Large σ → curve is wide and flat (data is more spread out).

Data and Sampling Distributions September 1, 2025 74 / 107

68–95–99.7 Rule (Empirical Rule)

About 68% of data lies within 1 standard deviation of the mean.

About 95% lies within 2 standard deviations.
About 99.7% lies within 3 standard deviations.

µ
µ−σ µ+σ

Data and Sampling Distributions September 1, 2025 75 / 107

Standard Deviation (σ)

It tells us how spread out the data is from the average (mean).
Small σ → data is tightly packed near the mean.
Large σ → data is more spread out.

Data and Sampling Distributions September 1, 2025 76 / 107

Normal Distribution Visualization

Data and Sampling Distributions September 1, 2025 77 / 107

1 Standard Deviation from the Mean

Imagine the mean is the center (average value).

If you go 1 σ above the mean and 1 σ below the mean, you cover the
range:
[Mean − σ, Mean + σ]
In Normal Distribution: About 68% of all values fall within 1 standard
deviation of the mean.

Data and Sampling Distributions September 1, 2025 78 / 107

Example: Students’ Exam Marks

Suppose:
Mean = 50
Standard deviation (σ) = 10
Then:
1 standard deviation range = [50 – 10, 50 + 10] = [40, 60]
Meaning: About 68% of students scored between 40 and 60 marks.

Key Point
Within 1 standard deviation from the mean means most of the data
(around 68%) lies in that range.

Data and Sampling Distributions September 1, 2025 79 / 107

Example: Exam Marks of 1000 Students

Mean = 70, = 10
1 Within ±1 (68%) → 60–80 → 680 students
2 Within ±2 (95%) → 50–90 → 950 students
3 Within ±3 (99.7%) → 40–100 → 997 students
Only 3 students scored below 40 or above 100.

Data and Sampling Distributions September 1, 2025 80 / 107

Interpretation of Exam Scores Example

Most students’ marks cluster near ∼ 70.

As you move 1σ, 2σ, 3σ away from the mean, fewer students fall in
those tails — matching the 68–95–99.7 pattern for a normal-like
distribution.
Small differences from the exact percentages are normal because we
used a finite sample.

Data and Sampling Distributions September 1, 2025 81 / 107

Normal Distribution Visualization

Data and Sampling Distributions September 1, 2025 82 / 107

1. Standard Normal Distribution (Z-distribution)
A special case of the normal distribution.
Mean (µ) = 0, Standard deviation (σ) = 1.
Any normal distribution can be converted into a standard normal
distribution using:
X −µ
Z=
σ
where:
X = value from the data
µ = mean of the data
σ = standard deviation
This process is called standardization.
It allows us to use Z -tables to calculate probabilities.
Example: If exam marks ∼ N (70, 10), then a student scoring 80 has:
80 − 70
Z= =1
10
⇒ The student is 1 standard deviation above average.
Data and Sampling Distributions September 1, 2025 83 / 107
2. QQ-Plots (Quantile-Quantile Plots)

A graphical tool to check if your data follows a normal distribution.

Steps:
1 Sort your data values.
2 Compare them with values from a perfect normal distribution.
3 Plot: Theoretical quantiles (x-axis) vs Sample quantiles (y-axis).
Interpretation:
Points close to the diagonal ⇒ data is approximately normal.
Curved (S-shape or bowed) pattern ⇒ data is not normal.

Data and Sampling Distributions September 1, 2025 84 / 107

Real-Life Example: Exam Marks

Take the 1000 exam marks.

Convert them to Z -scores ⇒ Standard Normal.
Draw a QQ-plot:
If data is close to normal ⇒ points roughly follow a straight line.

Data and Sampling Distributions September 1, 2025 85 / 107

Z-Score Table (Standard Normal Distribution)

The Z-Table shows the cumulative probability (area under the curve to the left of
Z ).

Z 0.00 0.01 0.02 0.03

0.0 0.5000 0.5040 0.5080 0.5120
0.5 0.6915 0.6950 0.6985 0.7020
1.0 0.8413 0.8438 0.8461 0.8485
1.5 0.9332 0.9345 0.9357 0.9370
2.0 0.9772 0.9778 0.9783 0.9788

Example: Z = 1.0 ⇒ 0.8413

Meaning: 84.13% of values lie below 1 standard deviation above the mean.

Data and Sampling Distributions September 1, 2025 86 / 107

Z-Score Table (Z-Table)

A Z-table gives the area (probability) under the standard normal

curve (µ = 0, σ = 1) up to a particular Z value.
It answers: “What percentage of the data lies to the left of this
Z-score?”
Why useful?
Once data is converted into Z-scores, we can use the same table for any
normal distribution.

Data and Sampling Distributions September 1, 2025 87 / 107

Example
Exam marks follow N(70, 10). Student score = 80

80 − 70
Z= =1
10
From Z-table: P(Z ≤ 1.00) = 0.8413
84.13% of students scored below 80
15.87% scored above 80

Data and Sampling Distributions September 1, 2025 88 / 107

QQ-Plot (Quantile-Quantile Plot)

A picture test for normality.

Steps:
1 Take your data (e.g., exam scores).
2 Compare it to what “perfectly normal data” would look like.
3 Plot:
X-axis → Theoretical (perfectly normal) values
Y-axis → Your actual data values
Interpretation:
Dots along a straight line ⇒ data is normal.
Dots bending away ⇒ data is not normal.

Data and Sampling Distributions September 1, 2025 89 / 107

✓Example with Exam Marks

If we make a QQ-plot of student exam scores:

If marks follow a normal curve, dots will line up diagonally.
If many students scored 100 (ceiling effect), the top dots bend away
from the line.

Data and Sampling Distributions September 1, 2025 90 / 107

high-voltage Summary

Standard Normal: A way to put all normal distributions on the same

scale (mean 0, SD 1).
QQ-Plot: A quick visual check to see if your data looks normal.

Data and Sampling Distributions September 1, 2025 91 / 107

Standard Normal Distribution Visualization

Data and Sampling Distributions September 1, 2025 92 / 107

QQ-Plots

Data and Sampling Distributions September 1, 2025 93 / 107

What is a Long-Tailed Distribution?

A distribution with a tail that stretches far out compared to a normal

(bell-shaped) curve.
Meaning:
Most data points are still near the center.
But there are more extreme values (outliers) than you’d expect under a
normal distribution.
The “tail” is the part of the curve that goes far to the left or right.

Data and Sampling Distributions September 1, 2025 94 / 107

Key Features

1 Heavier tails than normal distribution ⇒ unusual, extreme events are

more likely.
2 Not symmetric sometimes ⇒ can be skewed (one side longer).
3 Examples:
Pareto distribution (wealth distribution: few rich, many poor).
Cauchy distribution.
t-distribution (with small degrees of freedom).

Data and Sampling Distributions September 1, 2025 95 / 107

Real-Life Examples

Wealth / Income: Most people earn average incomes, but a few

earn extremely high incomes ⇒ long right tail.
Earthquake magnitudes: Small quakes are common, very large ones
are rare but possible.
Stock market returns: Most days have small ups/downs, but
occasionally huge crashes or rallies happen.

Data and Sampling Distributions September 1, 2025 96 / 107

Why it Matters?

Assuming data is “normal” when it actually has long tails ⇒

underestimates risk of extreme events.
Example: In finance, using normal distribution underestimates
probability of a market crash.

Data and Sampling Distributions September 1, 2025 97 / 107

In Short

Summary
A long-tailed distribution is one where extreme values (outliers) are more
likely than in the normal distribution.

Data and Sampling Distributions September 1, 2025 98 / 107

Long-Tailed Distribution

Data and Sampling Distributions September 1, 2025 99 / 107

Normal vs Long-Tailed Distributions
Here’s the graph
Blue curve → Normal distribution (most data near the mean, very
thin tails).
Red dashed curve → Long-tailed distribution (Student’s t with 3
degrees of freedom). Notice the fatter tails, meaning extreme values
are more likely.
Example:
In exam marks, if scores follow a normal distribution, very high/low
marks are rare.
But if marks follow a long-tailed distribution, you’d see more students
with extremely high or low scores than expected.
Why it matters:
Long-tailed models are important in finance, natural disasters, or
income studies.
They capture rare but impactful events better than the normal
distribution.
Data and Sampling Distributions September 1, 2025 100 / 107
Data and Sampling Distributions September 1, 2025 101 / 107
Data and Sampling Distributions September 1, 2025 102 / 107
Data and Sampling Distributions September 1, 2025 103 / 107
Data and Sampling Distributions September 1, 2025 104 / 107
Data and Sampling Distributions September 1, 2025 105 / 107
Data and Sampling Distributions September 1, 2025 106 / 107
Data and Sampling Distributions September 1, 2025 107 / 107