Data and Sampling Distributions

Sampling, Populations, and Distributions

1. Why Sampling Is Still Important (Even in the Era of Big Data)

Many people wrongly believe that because we have “big data,” sampling is no
longer needed.
This is not true.

Even with huge amounts of data, we still need sampling because:

 All data is not useful – big data contains noise, errors, and irrelevant
information.
 Sampling saves time and cost – working with the entire dataset is
often slow and expensive.
 Models are usually trained on samples – even in large companies,
machine learning models are first tested using samples.
 To reduce bias – properly selected samples can represent the
population better than random large data.

Example:
A website wants to test which new homepage design gets more clicks.
Instead of showing it to ALL users, they sample 5% of visitors and test design
A vs B.
This saves time and avoids risk.

2. Understanding Populations and Samples

In statistics:

 Population = The entire group we are interested in

(Example: All customers of Amazon)
 Sample = A small subset taken from the population
(Example: 2,000 customers selected randomly)

2.1 Population Distribution (Left Side of Figure 2-1)

 Every population has a distribution (shape of its data).

  This distribution is unknown in real life.
 Traditional statistics focused on making assumptions about this shape
to do calculations.

2.2 Sample Data & Empirical Distribution (Right Side of Figure 2-1)

 In real-world data science, we directly work with the sample data we

have.
 Its distribution is called the empirical distribution
(It is based on observed data, not theory).

2.3 Sampling Procedure (Arrow connecting both sides)

 To go from population → sample, we use sampling methods

(e.g., simple random sampling, stratified sampling, systematic sampling)

Modern data science focuses on the sample and the data actually available,
not on theoretical assumptions about the entire population.

3. When Population Models Matter

Even though data science usually focuses on sample data, some cases require
population-level modeling.

3.1 Example: Coin Toss

  A coin toss follows a binomial distribution.
 Example outcomes: heads/tails, yes/no, click/no click, fraud/no fraud.
 These physical or natural processes can be mathematically modeled.

3.2 Real-Life Examples of Binomial Situations

 Customer buys a product (buy/not buy)

 Fraud detection (fraud/not fraud)
 Email spam filtering (spam/not spam)
 Patient has disease or not (positive/negative)

In such cases, knowing the population distribution (binomial) helps us make

predictions and compute probabilities.

4. Simple Illustrative Example

Example: Click-through rate for an ad

Suppose the true click rate for an online ad is unknown.

Population:
All users who will see the ad in one month.

Sample:
We take 2,000 users and test the ad.

If 120 people click the ad,

Sample click rate = 120/2000 = 6%

We use this sample to estimate the population click rate.

Random Sampling and Sample Bias

1. What Is a Sample and a Population?

In statistics:

 Population → The entire set of data or items you want to study.

(It can be real, theoretical, or imaginary.)
 Sample → A small subset taken from that population to study or
analyze.
Example:
A college has 10,000 students (population).
We select 200 students to conduct a survey (sample).

2. What Is Random Sampling?

Random sampling means every member of the population has an equal

chance of being selected.

This method avoids favouritism and increases fairness.

There are two types:

a) Sampling With Replacement

 After a person/data point is selected, it is put back and may be selected

again.
 Each draw is independent.

Example:
Selecting 3 playing cards with replacement
→ You may pick the same card multiple times.

b) Sampling Without Replacement

 After selection, the item is not returned to the population.

 Cannot be selected again.

Example:
Selecting 3 playing cards without replacement
→ You will get 3 different cards.

Simple Random Sample (SRS)

When random sampling is done without stratifying or dividing into groups,

the result is called a simple random sample.

3. Why Data Quality Matters More Than Quantity

In data science, more data is not always better if the data is:

 incomplete
 inconsistent
 incorrect
  not representative

High-quality samples often give better results than large but biased datasets.

4. Sample Bias

Sample bias occurs when a sample does NOT represent the population in a
meaningful way.

This usually happens when:

 Only certain types of people are selected

 Selection is not random
 Certain groups are over-represented or under-represented

⭐Classic Example: Literary Digest Poll (1936)

 Literary Digest surveyed 10 million people.

 Predicted Alf Landon would win US presidential election.
 Result was completely wrong.

Why?
Their sample was biased:

 They polled their magazine subscribers (rich people).

 They used lists of people with telephones and cars (also wealthy).
 In 1936, poorer people (who favored Roosevelt) were left out.

George Gallup

 Polled only 2,000 carefully selected people.

 Used proper random selection.
 Correctly predicted Roosevelt’s victory.

⭐Lesson

Large sample + bad sampling method = wrong results

Small sample + good random sampling = accurate results

5. What Causes Sample Bias?

Sample bias happens when:

 The sampling method favors certain groups

  The sample systematically differs from the population
 The difference is meaningful and persistent

Examples:

 Surveying only morning gym-goers → Bias toward fitness enthusiasts

 Asking only college students about political opinions → Age bias
 Online surveys → Internet users only

6. Important Key Terms

Sample

A subset selected from a larger dataset (population).

Population

The entire dataset or the complete group being studied.

N (Population Size) / n (Sample Size)

N = total number in population

n = total number in sample

Random Sampling

Every member of the population has an equal chance of being selected.

Simple Random Sample (SRS)

Random sampling without dividing the population into groups.

Stratum (Plural: Strata)

A subgroup of the population with similar characteristics.

Example:
Male and female groups in a college = two strata.

Stratified Sampling

Dividing the population into strata and selecting random samples from each.

Example:
Select 50 male and 50 female students randomly → stratified sampling.
Bias

Systematic error.

Sample Bias

A sample that does not represent the population due to non-random selection.

Bias
What Is Bias in Statistics?

In statistics, bias means a systematic error — an error that consistently

pushes results in one direction.
It is not due to random chance.
It happens because the measurement process or sampling method is
flawed.

Bias causes the results of an experiment or model to be consistently wrong

in the same predictable way.

Difference Between Random Error and Bias

To understand bias, imagine shooting bullets at a target.

1. Random Error (Unbiased Process)

 Shots fall around the center, but not exactly in the middle.
 Some go left, some right, some up, some down.
 There is no pattern.
 This shows no bias, only random variation.

(Refers to Figure 2-2: cluster around the center but spread out.)

2. Biased Process

 Shots still have random variation.

 BUT they mostly fall toward one side (e.g., upper-right corner).
 This means shots are consistently shifted in a direction.

(Refers to Figure 2-3: cluster is shifted upwards and to the right.)

This consistent shift is bias.

Key Idea

Unbiased = errors are random

Biased = errors have a direction (systematic)

Why Is Bias Dangerous?

Bias causes a model or measurement to be consistently wrong, even if the

sample size is large.

A large dataset cannot fix bias.

Sources of Bias

Bias can be:

a) Observable Bias

We can clearly see that results are wrong or shifted.

Example:
A thermometer is always showing 2°C more than actual temperature.

b) Invisible Bias

We cannot directly observe it.

We notice it only when:

 Predictions are off from real-known values

 Benchmark comparison fails
 A pattern appears consistently

For example, a model predicting customer churn that is always

underestimating by 5%.

Causes of Bias in Models

Bias often indicates problems such as:

1. Model Misspecification
o Wrong model chosen
o Using a linear model for a nonlinear problem
o Using too few features
2. Missing Important Variables (Omitted Variable Bias)
o Not including income in a model predicting spending
o Not including weather when predicting electricity usage
3. Sampling Bias
o Sample is not representative of population
o Only surveying people online
4. Measurement Bias
o Faulty instruments
o Incorrect data entry
 o Misrecorded values

Real-Life Examples of Bias

Example 1: Faulty Scale

A weighing machine always adds +2 kg.

Even after many measurements, the values will be systematically wrong.

Example 2: Biased Survey

You ask only city mall visitors about fitness habits.

Result → overestimates fitness levels because rich, urban people visit malls.

Example 3: Model Bias

A housing price model predicts:

 Always ₹50,000 lower for expensive houses

 Always ₹10,000 higher for cheap houses

This systematic shift indicates bias.

Example

Example:
A machine learning model predicts employee salaries. The actual salary is
₹50,000. The model repeatedly predicts values like ₹44,000, ₹45,000,
₹46,000. Even though predictions vary, they are always below the true value.
This shows negative bias.

Random Selection
Random selection is one of the most important concepts in sampling because
it helps us avoid sample bias and ensures that the sample represents the
population properly. A famous historical example is how the Literary Digest
poll failed in 1936 by choosing a biased sample, while George Gallup
succeeded by using scientific random sampling.
1. Why Random Selection Is Important

George Gallup used random sampling to predict the correct winner of the
1936 US presidential election.
Although he used only 2,000 people, his sample was much more accurate
than the 10 million biased responses collected by Literary Digest.

Key Message:

Good random selection is better than large biased samples.

2. Why Random Sampling Is Difficult in Practice

Random sampling sounds easy, but in real-world data science, it requires

clear definitions and carefully designed procedures.

To perform random sampling, we first must define the accessible

population, meaning:
Who exactly should be considered part of the population?

3. Defining the Population Properly

Example: Customer Survey

Suppose a company wants to conduct a small customer survey to understand

buying behavior.

Before sampling, they must define who is a “customer.”

Questions they must answer:

 Do we include customers who bought only once?

 Do we include very old customers who haven’t bought recently?
 Do we include customers who made product returns?
 What about corporate customers or resellers?
 Do we count internal test purchases made by company staff?

Without a proper definition, the resulting sample will be biased.

Exam Example:

If you include only recent customers, the survey will be biased toward active
users and will not represent long-term customers.
4. Deciding the Sampling Procedure

A simple instruction like “select 100 customers at random” still requires more
detail.

Case 1: Real-Time Flow Sampling (Web Visitors or Transactions)

If sampling from a continuous flow (like website visitors), the time of

sampling matters:

 Customers at 10:00 AM on weekdays may be office workers

 Customers at 10:00 PM on weekends may be teenagers or students

If you choose only one time window, the sample becomes biased.
So, sampling must be spread across different times.

5. Stratified Sampling (Very Important Technique)

Sometimes, simple random sampling does not give enough representation for
smaller groups.

Stratified sampling is used when the population is divided into subgroups

(strata) such as:

 Gender groups
 Income levels
 Age categories
 Ethnic groups (Whites, Blacks, Hispanics)
 Customer types (new, returning, premium)

In stratified sampling:

1. Divide population into strata

2. Take random samples from each stratum
3. Combine the results → final representative sample

This ensures that all groups—even minorities—are included.

Example of Stratified Sampling

Goal: Estimate voting preferences of Americans by ethnic group.

Population contains:
  60% White
 15% Black
 25% Hispanic

If we take a simple random sample of 1,000 people:

 Whites ≈ 600
 Blacks ≈ 150
 Hispanics ≈ 250

But suppose we need equal-sized groups for precise comparison.

Using stratified sampling, we can sample:

 300 Whites
 300 Blacks
 300 Hispanics

Each group is now equally represented, even if the population sizes differ.

6. Why Stratified Sampling Helps

 Prevents small groups from being ignored

 Improves accuracy of results for subgroups
 Reduces sampling bias
 Allows comparison between groups

Example:

Comparison of health outcomes between men and women → stratifying by

gender makes results more reliable.

Size Versus Quality: When Does Size

Matter?
In the age of big data, many people assume that more data automatically
means better results.
However, in statistics and data science, data quality is often more
important than data quantity.
A smaller, carefully selected sample can often give better insights than a
massive dataset filled with noise, missing values, and biases.

Why Smaller Data Can Be Better

When we work with a random sample, instead of the entire dataset:

1. Less Bias
Random sampling eliminates systematic errors that can occur when
huge datasets are collected from biased sources.
2. Better Data Quality Checks
o Missing values can be verified
o Outliers can be examined
o Mistakes can be corrected
Doing this for millions of records is impossible, but it is possible
for a few thousand.
3. Easier Data Visualization
Graphs like scatterplots or boxplots become unreadable with millions of
points.
A smaller sample makes visual exploration easier.
4. Faster Processing & Better Understanding
Smaller datasets allow analysts to explore every variable manually,
leading to deeper understanding.

Example
Imagine a dataset of 50 million customer transactions.
Suppose we want to detect fraudulent transactions.

 If we use all 50 million records → checking missing values or outliers

would take enormous time.
 If we take a random sample of 10,000:
o Missing values can be manually checked
o Outliers (e.g., unusually high transactions) can be investigated
o Plots like histograms or boxplots can be drawn clearly

This smaller sample gives cleaner insights and still remains representative if
chosen correctly.
When Do We Actually Need Massive Data?
Big data becomes necessary when the data is:

1. Very Large AND Very Sparse

A sparse dataset means most values are zeros.

Example: Google Search Data

 Every search query is like a row

 Every possible English word is like a column
 Cell = 1 if the query contains that word, else 0

There are:
✔ Over 150,000 English words
✔ Over 1 trillion search queries per year
✔ Most query-word entries are 0 → extremely sparse

To predict the best search result, Google needs:

 Massive volume
 Rare combinations of search terms
 Records repeated across millions of users

This cannot be replaced with a small sample.

Why Big Data Helps in Search Engines

When a user types a very rare query like:

“Ricky Ricardo and Little Red Riding Hood”

In early internet days:

 Results would show unrelated pages about Ricky Ricardo

 Or separate pages on “Little Red Riding Hood”
 Because hardly anyone searched for this exact combination

Today, due to trillions of searches, Google can find:

 The exact I Love Lucy episode where Ricky tells the Red Riding Hood
story.
Even if only a few thousand people ever typed this exact phrase, Google
could find it because it collected trillions of search data points.

The key idea:

To find rare patterns (1 in a million events), we need huge amounts of
data.

Why Sampling Does NOT Help Here

Random sampling cannot capture extremely rare events because:

 If something occurs once in a million, a random sample of 10,000 will

almost never include it.
 Big data allows us to capture the "long tail" of rare queries.

Sample Mean vs Population Mean

In statistics, we often work with samples instead of whole populations
because studying the entire population is usually impossible or too expensive.
To avoid confusion between what we observe and what we estimate,
statisticians use different symbols for sample and population means.

Population Mean ( μ )
 The population mean is the true average of the entire population.
 Symbol: μ (mu)
 It is usually unknown because we rarely have data for every member of
a population.

Example (Exam-Friendly):

 Suppose the average height of all adults in a country is 165 cm.

 This true value is μ.
 We usually do not know this number because measuring every adult is
impossible.
Sample Mean ( x̄ )
 The sample mean is the average of a sample, which is a smaller subset
taken from the population.
 Symbol: x̄ (x-bar)
 This is the value we actually calculate from data.

Why Do We Use Different Symbols?

Statisticians want to keep the difference clear:

Concept What It Represents Symbol

Population Mean True average of entire population μ
Average of the sample we
Sample Mean x
collected

Reason for distinction:

 x̄ (sample mean) is observed directly from sample data.

 μ (population mean) is not observed; it is inferred or estimated using
the sample.

Keeping separate symbols helps avoid confusion between:

 What we know x
 What we wish to estimate μ
Selection Bias
Selection bias occurs when the data used for analysis is not chosen
randomly, but instead selected in a way that distorts the results.
This selection may be intentional or unintentional, but it leads to
misleading conclusions.

Statisticians often warn:

➡ “If you torture the data long enough, it will confess.”
Meaning: if you search long enough in large datasets, you will always find
some pattern—even if it is meaningless.

Why Selection Bias Happens

There are two main reasons:

1. Starting with a hypothesis and testing it properly

If we design a correct experiment (random sampling, proper control groups),

then our conclusions are reliable.

2. Searching for patterns in available data (data snooping)

When analysts explore huge datasets without a clear plan, they often find
patterns by luck, not because the pattern is real.

This is the danger:

➡ A pattern found accidentally may look meaningful but has no real
significance.

Thought Experiment
Case 1: Proper Experiment

Someone claims they can flip a coin and get 10 heads in a row.
You ask them to do it immediately → They flip 10 heads.

 Probability of 10 heads = 1/2;: = 1 in 1024

 Very unlikely → You assume they have special skill.

Case 2: Large Group + After-the-Fact Selection

A sports stadium announcer asks 20,000 people to each flip a coin 10 times.
Those who get 10 heads must report.

Probability that at least one person gets 10 heads:

 Very high (greater than 99%)

Now the result is not special.

It is simply because many people tried.

Conclusion:

 Selecting after seeing results = selection bias

 Selecting before seeing results = valid experiment

These two situations illustrate the difference between real effects vs

chance discoveries.

Why Selection Bias Is Dangerous in Data

Science
Modern data science involves:

 Millions of rows
 Thousands of variables
 Hundreds of model configurations

This means:
➡ The more you search, the more “interesting results” you will find by
chance.

John Elder (a major researcher) calls this the:

“Vast Search Effect”

If you keep running models repeatedly, some model will show a strong
pattern—even if the pattern is completely random or false.
How to Protect Against Selection Bias
1. Use Holdout Sets

Divide data into:

 Training set
 Validation set
 Test (holdout) set

Patterns found in training must also appear in the holdout set.

If not → The pattern is random, not real.

2. Use Multiple Holdouts if Needed

Especially when tuning many models.

3. Use Target Shuffling (Permutation Test)

Proposed by John Elder.

How it works:

 Randomly shuffle the target variable (e.g., fraud / no fraud)

 Train the model on this shuffled data
 If the model still finds strong patterns → The original pattern was
probably fake

This technique tests whether the discovered pattern is stronger than

chance.

Common Types of Selection Bias

1. Nonrandom Sampling
Choosing data that does not represent the population.
Example: Surveying only wealthy people → biased sample.
2. Cherry-Picking Data
Selecting only the data that supports a conclusion.
Example: A company shows only the months where profits increased.
3. Biased Time Intervals
Choosing specific dates that exaggerate a trend.
Example: Analyzing stock performance only during a market boom.
 4. Stopping an Experiment Early
Ending an experiment when results look good—even if it’s just random
variation.
5. Data Snooping / P-hacking
Running many tests until something “interesting” appears.

Regression to the Mean

Regression to the mean is a statistical phenomenon where extreme values
are usually followed by more average (central) values in the next
measurement.

It happens naturally in repeated measurements and often leads people to

make incorrect conclusions.

Core Meaning
If someone performs extremely well or extremely poorly at one time, their
next performance is likely to be:

➡ Closer to the average

➡ Not as extreme
➡ Because luck or natural fluctuation changes

This is NOT because improvement or decline is guaranteed — it's simply

because extreme results contain both skill and luck, and luck does not
repeat consistently.

Sports Example
Consider the “Rookie of the Year, Sophomore Slump” phenomenon.

 A rookie athlete wins “Rookie of the Year” because he performs better

than all others.
 His performance is a combination of:
o Skill
o Luck

Next year:

 The skill remains the same

  But the luck usually decreases
 So performance gets closer to average → regression to the mean

This decline is not due to poorer skill, but due to the natural reduction of
lucky breaks.

Skill + Luck Concept

Let performance = Skill + Luck

 In one season, an extreme performance probably means:

o High skill
o Plus unusually high luck

Next season:

 Skill remains
 Luck tends to return to normal, so:
o Extreme performance → becomes more average
o This is regression to the mean

Historical Origin (Galton)

Francis Galton (1886) first explained regression to the mean when studying
height:

 Very tall fathers tend to have sons who are tall, but slightly shorter
 Very short parents tend to have children who are short, but less short
 Children’s heights move toward the average of the population

This was the earliest scientific explanation of regression to the mean.

Important Insight
Regression to the mean is a form of selection bias.
When we select an extreme case, we unintentionally select:

 Skill
 Plus a lot of luck

But when we measure again, luck is usually lower → results look worse.
This often misleads people into thinking:

 “The player got worse”

 “The coaching worked”
 “The treatment improved the score”

When in fact, it’s just natural regression.

Everyday Examples
1 Students' Test Scores

A student scoring extremely high on one exam often scores lower on the next
exam because:

 First exam = skill + luck

 Second exam = skill + normal luck
2 Stock Market

If a stock has an unusually high performance one year, next year performance
usually moves toward average market returns.

3 Medical Measurements

Patients with extremely high blood pressure readings often show lower
readings in the next measurement even without treatment.

Why? Because extreme values swing back toward average due to natural
variability.

Sampling Distribution of a Statistic

When we take a sample from a population, we calculate some statistic from
that sample—such as the sample mean, sample median, sample
proportion, or sample variance.

But if we took another sample, we would get a slightly different value.

This happens because each sample contains different members of the
population.

✔ Definition

The sampling distribution of a statistic is the distribution of that statistic

(like the mean) over many repeated samples drawn from the same
population.

✔ Why It Matters

Because:

 We almost never have access to the full population.

 We rely on samples.
 But samples vary → this variation must be understood for estimation
and inference.

Key Idea: Sampling Variability

Different samples → different results

This is called sampling variability.
Example (exam-friendly):

Imagine the population is all incomes of loan applicants.

If we take 20 different samples of size 100 each, each sample mean income
will be different.

Important Observation
The sampling distribution of the sample mean is:

 More regular (smooth) than the original data distribution

 More bell-shaped, even if the data itself is skewed
 Narrower when sample size increases

➡ Larger sample size → less variability → narrower sampling distribution.

Example
Take income data of Lending Club loan applicants.

Construct:

1. 1,000 raw income values

2. 1,000 sample means (each mean from 5 incomes)
3. 1,000 sample means (each mean from 20 incomes)

When you plot histograms:

 The raw values are spread out and skewed

 Means of 5 values are smoother
 Means of 20 values form a clear bell curve
Central Limit Theorem (CLT)
The Central Limit Theorem is one of the most important results in statistics.

✔ Definition (Simple Form)

When we take many sample means from a population:

➡ The distribution of these sample means will be approximately normal

(bell-shaped)
➡ Even if the original population is not normal
➡ As long as the sample size is sufficiently large usually n ≥ 30

This is true for:

 skewed populations
 heavy-tailed populations
 any distribution that is not too extreme
Why CLT Is Important

It allows us to:

 build confidence intervals

 perform hypothesis tests
 use normal and t-distribution formulas
even when the underlying population distribution is unknown.

In Data Science

CLT is still conceptually important,

but confidence intervals and hypothesis tests are less central in real-life
ML work.

Example for CLT

Suppose the population of incomes is very skewed.
If we take repeated samples of size 50 and calculate the means:

 The histogram of sample means will look normal

 Even though the income distribution itself is skewed

This is the Central Limit Theorem in action.

Standard Error (SE)

The standard error tells us how much a statistic (like the mean) varies from
sample to sample.

✔ Definition

Standard Error = the standard deviation of the sampling distribution.

Key Insights
1. As sample size increases → SE decreases
Means become more stable with larger samples.
2. Square-root rule:
To cut SE in half, you must quadruple the sample size.
3. SE explains why big samples give more precise estimates.

Estimating Standard Error in Real Life

We usually cannot take many samples from the population.
So what do we do?

✔ Approach 1 (Ideal but impossible)

1. Take many new samples

2. Compute mean of each
3. Calculate SD of these means → SE

This is not practical.

✔ Approach 2 (Practical and modern) — Bootstrap

The bootstrap is used:

 Take many resamples with replacement from the original sample

 Compute the statistic each time
 Calculate the SD of the bootstrap statistics → SE

Bootstrap works for almost any statistic:

 mean
 median
 regression coefficient
 proportion
 difference in means

And it does not require normality or the CLT.

The Bootstrap
The bootstrap is a modern and powerful statistical method used to estimate
the sampling distribution of a statistic without collecting new data.

It works by resampling the existing sample WITH replacement, many

times, and recalculating the statistic for each resample.

This method is extremely useful when:

 We have small samples

 We cannot draw new samples from the population
 The sampling distribution is unknown
 Classical formulas (like normal distribution assumptions) do not apply

Why Do We Need the Bootstrap?

In theory, to understand sampling variability, we would draw:

 Many new samples

 Calculate the statistic each time
  Look at the distribution

But in real life:

 We cannot keep collecting new data

 Data collection is expensive, time-consuming, or impossible

The bootstrap solves this by pretending our sample itself is the population.

Conceptual Idea
Imagine taking your original sample and copying it millions of times to
create a giant population.

Then:

 Randomly draw samples from this imagined population

 Compute statistics (mean, median, model coefficients, etc.)
 Build a sampling distribution

But in practice, we don’t actually make millions of copies.

We simply sample WITH replacement from the original sample.
This behaves the same way as sampling from the imaginary large population.

Bootstrap Algorithm
To bootstrap a statistic (such as the sample mean) for a sample of size n:

1. Draw one value at random from the sample

2. Record it and put it back (sampling with replacement)
3. Repeat steps 1–2 until you have n values → one bootstrap sample
4. Compute the statistic (e.g., mean) of this bootstrap sample
5. Repeat steps 1–4 R times
o R = number of bootstrap iterations (often 1,000 or 10,000)

This yields R bootstrap statistics, which can be used to:

 Estimate standard error

 Plot a histogram or boxplot
 Build confidence intervals

What Does the Bootstrap Give Us?

After performing R iterations, we obtain a bootstrap distribution of the
statistic.

From this distribution we can estimate:

✔ 1. Standard Error

The standard deviation of the bootstrap results is the estimated standard

error.

✔ 2. Confidence Intervals

Use percentiles of the bootstrap distribution (e.g., 2.5% and 97.5%) for a 95%
CI.

✔ 3. Shape of the Sampling Distribution

We can visualize it (histogram or boxplot).

✔ 4. Variability of Model Parameters

Useful for:

 Regression coefficients
 Decision tree parameters
 Machine learning models

Example
Suppose we have a small sample of incomes:

[40, 50, 60, 55, 45] (n = 5)

To bootstrap the mean:

Bootstrap sample 1:

Random picks (with replacement):

50, 40, 40, 60, 55 → mean = 49

Bootstrap sample 2:

45, 60, 45, 50, 50 → mean = 50

Bootstrap sample 3:

60, 60, 40, 55, 60 → mean = 55

Repeat this 1,000 times.

Then:

 Standard deviation of the 1,000 means → standard error

 Middle 95% of these means → 95% confidence interval

This gives us a practical estimate of sampling variability without collecting

new data.

Advantages of the Bootstrap

✔ No assumptions of normality
✔ Works for any statistic: mean, median, proportion, regression coefficients
✔ Easy to implement computationally
✔ Very useful in modern data science

When is the Bootstrap used?

 When formulas for standard error are unknown or unreliable
 For small samples
 For skewed or non-normal data
 In machine learning for model stability
 When confidence intervals from theory do not apply

Resampling Versus Bootstrapping

In statistics and data science, resampling refers to a broad class of methods
that
repeatedly draw samples from data in order to assess variability,
uncertainty, or
significance of a result.

Bootstrapping is a specific type of resampling method, but the two terms

are not exactly the same.
Resampling (General Term)

Resampling is an umbrella term that includes several techniques in which

data is
drawn repeatedly from an observed dataset.

✔ Purpose of Resampling

 Estimate uncertainty of statistics

 Compare groups
 Test hypotheses
 Validate models

✔ Common Resampling Methods

 Bootstrap
 Permutation tests
 Randomization tests
 Cross-validation

Resampling may be done:

 With replacement
 Without replacement
(depending on the method)

Bootstrapping (Specific Method)

Bootstrapping is a particular resampling technique where:

➡ Samples are drawn with replacement

➡ Sampling is done only from the observed data
➡ The sample size remains the same as the original dataset

The goal is to estimate:

 Sampling distribution
 Standard error
 Confidence intervals

✔ Key Rule
Bootstrapping always uses sampling with replacem ent

Key Differences
Feature Resampling Bootstrapping
Meaning General category Specific method
Sampling with replacement May or may not Always
Sampling without replacement Possible No
Uses original sample Yes Yes
Includes permutation tests Yes No
Main goal Variability, testing, validation Standard error, CI

Important Exam Point

 All bootstrapping is resampling

 Not all resampling is bootstrapping

Confidence Intervals
In statistics, a sample estimate (such as a mean or proportion) is only an
approximation of the true population value. Since samples vary, the estimate
contains uncertainty.

A confidence interval (CI) expresses this uncertainty by giving a range of

values instead of a single number.

What is a Confidence Interval?

A confidence interval is a range of values, calculated from sample data, that

is likely to contain the true population parameter.

Instead of saying:

“The mean income is ₹62,231”

We say:

“The mean income is between ₹58,000 and ₹66,000 with 90% confidence”

This helps avoid overconfidence in point estimates.

Level of Confidence

Every confidence interval has a confidence level, usually:

 90%
 95%
 99%

✔ Meaning of Confidence Level

A 90% confidence interval means:

If we repeat the same sampling procedure many times, about 90% of the
calculated intervals will contain the true population value.

It does not mean:

 There is a 90% chance that the true value lies in this specific interval
(The true value is fixed; the interval varies.)

Why Confidence Intervals are Important

 Single numbers hide uncertainty

 Confidence intervals:
o Show possible error in estimation
o Help in better decision-making
o Are more honest and informative

Bootstrap Confidence Interval

The bootstrap is a practical way to construct confidence intervals, especially

when assumptions about normality are not valid.

Algorithm for Bootstrap Confidence Interval

Given a sample of size n:

1. Draw a random sample of size n with replacement from the original

data
2. Compute the statistic of interest (e.g., mean)
3. Repeat steps 1–2 R times (e.g., 1,000 times)
4. Sort the R bootstrap statistics
5. For an x% confidence interval:
o Remove 100 − x /2 % from the lower end
 o Remove 100 − x /2 % from the upper end
6. The remaining range is the confidence interval

Example (Exam-Friendly)

Sample size = 20
Sample mean income = ₹62,231

After bootstrapping:

 Lower 5% cutoff = ₹58,000

 Upper 95% cutoff = ₹66,000

90% Confidence Interval:

[ ₹58,000, ₹66,000]

This means:

We are 90% confident that the true mean income lies within this range.
Effect of Confidence Level and Sample Size

✔ Confidence Level

 Higher confidence → wider interval

 Lower confidence → narrower interval

✔ Sample Size

 Larger sample → narrower interval

 Smaller sample → wider interval

This is because:

 More data → less uncertainty

 More confidence → need more safety margin

Formula-Based Confidence Intervals

Traditional statistics often uses:

 t-distribution (for small samples)

 normal distribution (for large samples)

These rely on assumptions about data distribution.

In modern data science, bootstrap confidence intervals are preferred

because:

 Fewer assumptions
 Work for many statistics and models

Normal Distribution
The normal distribution is a bell-shaped, symmetric distribution that
plays a central role in traditional statistics. It is mainly important because
sample statistics (such as sample means) often follow a normal distribution.

Although real-world raw data is often not normally distributed, the normal
distribution is very useful for:
  Error analysis
 Sampling distributions
 Confidence intervals
 Hypothesis testing

Properties of Normal Distribution

 Symmetric about the mean

 Mean = Median = Mode
 Defined by:
o Mean μ
o Standard deviation σ

✔ Empirical Rule

In a normal distribution:

 68% of data lies within ±1σ of the mean

 95% of data lies within ±2σ of the mean
 99.7% lies within ±3σ of the mean

This rule helps estimate probabilities quickly.

Standard Normal Distribution (z-
Distribution)
A standard normal distribution is a normal distribution with:

 Mean = 0
 Standard deviation = 1

To convert any normal variable into standard normal form, we use

standardization.

Standardization (z-Score)

Standardization converts data values into z-scores, which represent how

many
standard deviations a value is from the mean.

Interpretation of z-Score

 z = 0 → value equals the mean

 z = +2 → value is 2 standard deviations above mean
 z = −1 → value is 1 standard deviation below mean

QQ-Plot (Quantile-Quantile Plot)

A QQ-Plot is a graphical tool used to check whether a dataset follows a
normal distribution.
How QQ-Plot Works

 Data values are first standardized (z-scores)

 Values are sorted from lowest to highest
 Each value is plotted against the corresponding theoretical normal
quantile

✔ Axes:

 x-axis: theoretical normal quantiles

 y-axis: observed z-scores

Interpreting a QQ-Plot

 Points close to the straight diagonal line → data is approximately

normal
 Points deviate strongly from the line → data is not normal

✔ Example:

If data is generated from a normal distribution, the QQ-plot points will lie
almost exactly on the line.

Long-Tailed Distributions
In practice, most real-world data is not normally distributed.
Many datasets have long tails, meaning:

 Extreme values occur more often than expected under a normal

distribution
 Both symmetric and skewed distributions can have long tails

What Are Tails?

 Tails are the extreme low and high values in a distribution

 Long tails indicate high variability and outliers

Examples of Long-Tailed Data

 Income and wealth data (right-skewed)

 Stock market returns
 Web traffic and click data
 Insurance losses

Normal vs Long-Tailed Behavior

Under a normal distribution:

 Extreme values are very rare

Under long-tailed distributions:

 Extreme events are much more common

This explains why:

 Market crashes
 Rare failures
 Unusual events
occur more often than normal theory predicts.

Student’s t-distribution
The Student’s t-distribution (or simply t-distribution) is a probability distribution that is very
important in statistics, especially when dealing with small sample sizes.
 -distribution looks similar to the normal distribution (bell-shaped curve).

fatter tails — meaning it gives more probability to extreme values.

-distribution gets closer and closer to the normal

distribution.

When it is used

The t-distribution is mainly used when:

1. The population standard deviation is unknown.

2. The sample size is small (n < 30).
3. You want to make inferences about a population mean.

Key properties

 Centered at 0, just like the standard normal distribution.

 Defined by a parameter called degrees of freedom (df), usually related to sample
size:

df=n−1

 With lower df, the curve is wider and has heavier tails.
 With higher df, it becomes almost identical to the normal distribution.

Example

Suppose you take a sample of 10 students’ exam scores to estimate the population mean.

 Since the sample is small and the population standard deviation is unknown, you
use the t-distribution for confidence intervals or hypothesis tests instead of the
normal distribution.

In short:

 Small samples + unknown population σ → use t-distribution.

 Large samples or known σ → normal distribution is fine.
Here’s the graph

 Blue curve → Standard Normal Distribution.

 Red/Green/Orange dashed curves → Student’s t-distributions with df = 2, 5, 30.

Notice how:

 With low df (2, 5), the t-distribution has fatter tails (more probability for extreme
values).
 With large df (30), the t-distribution is almost the same as the normal distribution.

For the sample

[72, 75, 78, 70, 74, 69, 80, 77, 73, 71]

1) Sample size and mean

 n=10
 Sum 72+75+78+70+74+69+80+77+73+71 = 739
 Sample mean

xˉ=739/10=73.9

2) Sample standard deviation ss

Use the (unbiased) sample formula with denominator n−1.

1. Compute deviations xi−xˉ for each data point and square them.
I’ve put the full list in the “Deviations table” you can open in the interface.
2. Sum of squared deviations

∑ xi−xˉ 2=116.9

3. Sample variance

s2=∑ xi−xˉ 2/n−1=116.9/9=12.988889

4. Sample standard deviation

s=square root (12.988889) =3.604010

3) Standard error (SE) of the mean

SE=s/square root(n)=3.604010/square root(10)=1.139688

4) Critical values for 95% two-sided CI

 z critical (normal distribution): z 0.975=1.959964
 t critical with df=n−1=9 : t 0.975,9=2.262157

“0.975” because a 95% two-sided CI leaves 2.5% in each tail.)

5) Confidence intervals
General form:

xˉ± critical ×SE

Using normal (z)

73.9±1.959964×1.139688=(71.666252, 76.133748 ≈ 71.67, 76.13

 Using Student’s t (df=9)

73.9±2.262157×1.139688=(71.321846, 76.478154 ≈ 71.32, 76.48

What this shows

  The t-interval is wider than the z-interval because it uses a larger critical value to
reflect the extra uncertainty from a small sample and unknown population standard
deviation.
 As n grows, t critical → z critical, and the two intervals become nearly identical.

Here’s the comparison

 Blue bar (z CI) → Narrower interval 71.67 to 76.13 .

 Red bar (t CI, df=9) → Wider interval 71.32 to 76.48 .
 Black line → Sample mean 73.9 .

The t-distribution “stretches” the interval to account for extra uncertainty with a small
sample size and unknown population σ.

standard error becomes much smaller because:

SE=s/square root(n)=3.6/square root(100) =0.36

 z 0.975=1.96
 t 0.975,99≈1.984 almost the same as z .

 Normal (z):

74±1.96×0.36=(73.29, 74.71
 t (df = 99):

74±1.984×0.36=(73.29, 74.71)

The two intervals are now virtually identical.

 This shows why for large n, the t-distribution ≈ normal distribution, and many people just
use the normal approximation.

Important Points about Student’s t-distribution

1. Origin of the name “Student”

o Published in 1908 in the journal Biometrika by William Sealy Gosset.
o He used the pseudonym “Student” because his employer (Guinness
Brewery didn’t want competitors to know it was using advanced statistical
methods.

2. Gosset’s motivation
o He wanted to answer:
“What is the sampling distribution of the mean of a sample, drawn from a
larger population?”
o In other words: If I take small random samples from a population, what does
the distribution of those sample means look like?

3. His experiment
o He used a dataset of 3,000 measurements criminals’ heights and finger
lengths).
o He repeatedly drew small samples (size = 4).
o For each sample, he computed the mean and standardized it (z-scores).
o Then he plotted the frequency of these standardized sample means.

4. Discovery of the t-distribution

o Gosset compared his sample results to a new theoretical function he
derived.
o That function matched his experimental data very well → this function
became known as the Student’s t-distribution.

5. Key insight
o The t-distribution describes the variability of sample means when
sample sizes are small and the population standard deviation is unknown.
o It explained why small-sample estimates needed wider ranges (heavier
tails).
Critical values are obtained.

1. Why 0.975?
For a 95% two-sided confidence interval, we want 2.5% in each tail of the distribution:

Left tail=0.025, Right tail=0.975

So the critical value is the point where the cumulative distribution function (CDF) equals
0.975.

2. Normal distribution (z-critical)

 The standard normal distribution has mean = 0, std = 1.
 We look up the 97.5th percentile (0.975 quantile) in the Z-table.
 Or, mathematically:

Z 0.975=Φ−1(0.975)

where Φ−1= inverse CDF of the normal distribution.

Value:

Z 0.975=1.959964 ≈ 1.96

3. t-distribution (t-critical)
 Degrees of freedom = df=n−1.
Here, n=10, so df=9.
 We want the same quantile: 0.975 (97.5th percentile).
 Using the t-distribution’s inverse CDF:

t 0.975,9=Ft,9−1(0.975)

Value from a t-table or software:

t 0.975,9=2.262157

This is larger than 1.96 because the t-distribution has fatter tails — we need a “bigger
cutoff” to capture 95% of the area.
4. General Rule
 For 95% CI (two-sided): use quantile at 0.975.
 For 90% CI: use quantile at 0.95.
 For 99% CI: use quantile at 0.995.

1. Reading from the Z-table (Standard

Normal Distribution)
We want:

z0.975

That means: the cumulative probability (area under the curve up to z) = 0.975.

Step A: Open a Z-table

 The Z-table lists probabilities for different z-values.

 Usually rows give the first 2 digits (like 1.9) and columns give the 2nd decimal (like
0.06).

Step B: Look for 0.975 in the table

 Find the closest probability to 0.9750 inside the table.

 You’ll see it at row 1.9 and column 0.06.

So:

z=1.96

Example (95% CI).

Mini Z-table (Standard Normal)

We are looking for cumulative probability = 0.975.
The Z-table gives P Z≤z

z 0.00 0.01 0.02 0.03 0.04 0.05 0.06

1.9 0.9713 0.9719 0.9726 0.9732 0.9738 0.9744 0.9750

At row = 1.9 and column = 0.06, the value is 0.9750.

So:
z 0.975=1.96

2. Reading from the t-table (Student’s t

distribution)
We want:

t 0.975,9

That means: the value of t with df = 9 that leaves 2.5% in the upper tail.

Step A: Open a t-table

 Across the top row, you’ll see confidence levels like one-tail 0.025, two-tail 0.05).
 Along the left column, you’ll see degrees of freedom df .

Step B: Locate df = 9

 Move across the row for df = 9 until you’re under the column for two-tail = 0.05
(which corresponds to 95% CI).

Step C: Read the number

 The value is about 2.262.

So:

t 0.975,9=2.262

Mini t-table (Student’s t distribution)

We want df = 9 and two-tailed 0.05 (which means 95% CI, or 0.025 in each tail).

df One-tail 0.025 Two-tail 0.05

df One-tail 0.025 Two-tail 0.05

8 2.306 2.306

9 2.262 2.262

10 2.228 2.228

At df = 9 under two-tail 0.05, the value is 2.262.

So:

t 0.975,9=2.262

Binomial Distribution
The Binomial Distribution is used when we are dealing with yes/no outcomes (also
called binary outcomes).

Key Idea

 You have a fixed number of trials (say n).

 Each trial has only two outcomes → Yes/No, Success/Failure, 1/0.
 Each trial has the same probability of success (p).
 The trials are independent what happens in one doesn’t affect the others .

The Binomial Distribution tells us the probability of getting exactly k successes out of n
trials.

Formula:

the probability mass function (PMF) of the Binomial distribution.

Example
Suppose:

 n=5 tosses of a coin

 p=0.5 (fair coin)
 Find P(X=2) (probability of exactly 2 heads).

So the probability of getting exactly 2 heads is 31.25%.

Here’s the Binomial distribution for n=5,p=0.5:

 The x-axis shows the number of heads (k) in 5 coin tosses.

 The y-axis shows the probability P(X=k).
 Notice it’s symmetric, centered around k=2 and k=3.

For example, P(X=2)=0.3125 (the tallest bar).

Here’s the Binomial distribution for n=20,p=0.5 5:

 The peak is now around k=10 (since the expected value = n⋅p=10).
 The bars form a bell-shaped curve, closely resembling the Normal distribution.
 As n increases further (e.g., n=50,100), the binomial gets smoother and looks almost
identical to a normal distribution.
 This happens because of the Central Limit Theorem, which says:

If you have many independent trials, the sum (or average) of them tends to follow a
Normal distribution, regardless of the original distribution.

Since the Binomial is the sum of Bernoulli trials (success/failure), it also follows this
principle.

This is why we often use the Normal approximation to the Binomial when n is large.

Success vs. Failure

 “Success” doesn’t mean good → it just means the outcome we’re interested in
tracking.
 Example:
o In fraud detection, a fraudulent transaction = 1 (success).
o In medical studies, a patient surviving = 1 (success).
o In marketing, a customer buying = 1 (success).

In short:
The Binomial Distribution models the probability of getting a certain number of
successes (1s) in a fixed number of independent yes/no trials, given the probability of
success in each trial.

R Functions

dbinom(x, size=n, p=p) → gives probability of exactly x successes.

Example:

dbinom(x=2, size=5, p=0.1)

 → 0.0729 7.29% .

 pbinom(x, size=n, p=p) → gives probability of x OR FEWER successes.

Example:
 pbinom(2, size=5, p=0.1)

→ 0.9914 99.14% .

Python Equivalent (SciPy)

PMF (Probability Mass Function) = probability of exactly x successes.

from [Link] import binom

[Link] 2, n=5, p=0.1 # → 0.0729

CDF (Cumulative Distribution Function) = probability of x or fewer successes.

[Link] 2, n=5, p=0.1 # → 0.9914

Chi-Square Distribution
Poisson and Related Distributions
Many real-world processes generate events randomly over time or space.
Examples include:

 Visitors arriving at a website

 Cars arriving at a toll booth
  Defects in fabric per square meter
 Typographical errors per 100 lines of code

Such processes are modeled using Poisson, Exponential, and Weibull

distributions.

Poisson Distribution

The Poisson distribution describes the number of events occurring in a

fixed interval of time or space, when events occur randomly but at a
constant average rate.

✔ When to Use Poisson Distribution

 Events occur independently

 Events occur at a constant average rate
 Two events cannot occur at exactly the same instant
 We count number of events, not time between them

Parameter of Poisson Distribution

 λ (lambda) = average number of events per interval

 Mean = λ
 Variance = λ

Practical Use

Poisson distribution is widely used in:

 Queueing systems
 Call center capacity planning
 Network traffic analysis
 Manufacturing defect analysis

If customer service calls arrive at an average of 2 calls per minute, the

number of calls per minute follows a Poisson distribution with λ = 2.

R code:

rpois(100, lambda=2)

Python code:

from scipy import stats

[Link](2, size=100)

This simulates 100 time intervals, each showing the number of calls received
per minute.

Exponential Distribution

While the Poisson distribution models number of events, the exponential

distribution models the time between consecutive events.

✔ When to Use Exponential Distribution

 Events occur randomly

 Rate of events is constant
 We measure waiting time, not count

⭐Parameter

 Uses the same λ (rate) as Poisson

 Mean time between events = 1 / λ

⭐Examples

 Time between website visits

 Time between incoming phone calls
 Time required to complete a service task
 Time until next failure (if rate is constant)

R code:

rexp(n=100, rate=0.2)

Python code:

from scipy import stats

[Link](0.2, size=100)

This simulates 100 waiting times when the average arrival rate is 0.2 events
per minute.

Key Assumption

For both Poisson and Exponential distributions:

The event rate λ must remain constant during the interval.

In practice, this assumption is valid only for short, homogeneous time

periods.

Weibull Distribution

In many real-world cases, event rate changes over time, especially in

mechanical systems.

Example:

 Machines are more likely to fail as they age

In such cases, Poisson and exponential distributions are not suitable.

The Weibull distribution is used instead.

Key Features of Weibull Distribution

 Extends the exponential distribution

 Allows the event rate to change over time
 Widely used in reliability engineering and failure analysis

Parameters of Weibull Distribution

1. β (beta) – shape parameter

o β > 1 → failure rate increases over time
o β < 1 → failure rate decreases over time
o β = 1 → constant rate same as exponential
2. η (eta) – scale parameter
o Also called characteristic life
o Represents typical lifetime of a component

Applications

 Time-to-failure analysis
 Reliability studies
 Mechanical and industrial systems
 Survival analysis

R code:

rweibull(100, 1.5, 5000)

Python code:

from scipy import stats

stats.weibull_min.rvs(1.5, scale=5000, size=100)

This simulates 100 lifetimes with:

 Increasing failure rate β = 1.5

 Characteristic life = 5000 units