Sampling and Randomization

Marc Shotland
MIT
Outline

• Distributions and expected value

• Sampling distributions
– population distribution
– sampling distribution
– law of large numbers/central limit theorem
– standard deviation and standard error
• Sampling and Randomization strategies
– Sampling in practice
– Randomization in practice

J-PAL | S AMPLING AND S AMPLE S IZE 2

Outline

• Distributions and expected value

• Sampling distributions
– population distribution
– sampling distribution
– law of large numbers/central limit theorem
– standard deviation and standard error
• Sampling and Randomization strategies
– Sampling in practice
– Randomization in practice

J-PAL | S AMPLING AND S AMPLE S IZE 3

What’s the expected result?

• If you were to roll a die once, what is the “expected

result”? (i.e. the average)

J-PAL | S AMPLING AND S AMPLE S IZE 4

Roll a die once. Register your result

A. 1
100%
B. 2
C. 3
D. 4
E. 5
F. 6

0% 0% 0% 0% 0%

A. B. C. D. E. F.
J-PAL | S AMPLING AND S AMPLE S IZE 5
Roll one die 10 times.
Register your result each time
A. 1
40% 40%
B. 2
C. 3
D. 4
E. 5
20%
F. 6

0% 0% 0%

A. B. C. D. E. F.
J-PAL | S AMPLING AND S AMPLE S IZE 6
Rolling 1 die:
possible results & probability

1/6

0
1 2 3 4 5 6

J-PAL | S AMPLING AND S AMPLE S IZE 7

What is the “expected value” of a die?

A. 1
B. 2 13% 13% 13% 13% 13% 13% 13% 13%

C. 3
D. 4
E. 5
F. 6
G. All of the above
H. None of the above

A. B. C. D. E. F. G. H.
J-PAL | S AMPLING AND S AMPLE S IZE 8
Rolling 1 die:
“Expected Value”

1/6

0 3.5
1 2 3 4 5 6

J-PAL | S AMPLING AND S AMPLE S IZE 9

What’s the expected result?

• If you were to roll two dice once, what is the expected

value of the sum?

J-PAL | S AMPLING AND S AMPLE S IZE 10

Rolling 2 dice:
Possible totals & likelihood
1/4

1/6

0
2 3 4 5 6 7 8 9 10 11 12
Frequency 1/36 1/18 1/12 1/9 5/36 1/6 5/36 1/9 1/12 1/18 1/36

J-PAL | S AMPLING AND S AMPLE S IZE 11

Rolling 2 dice:
11 possible totals, 36 permutations
Die 1

2 3 4 5 6 7

3 4 5 6 7 8

4 5 6 7 8 9
Die 2

5 6 7 8 9 10

6 7 8 9 10 11

7 8 9 10 11 12

J-PAL | S AMPLING AND S AMPLE S IZE 12

Rolling 2 dice:
Average value of two dice & likelihood

1/4

1/6
Likelihood

0
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Frequency 1/36 1/18 1/12 1/9 5/36 1/6 5/36 1/9 1/12 1/18 1/36

J-PAL | S AMPLING AND S AMPLE S IZE 13

Outcomes and Permutations

• Putting together permutations, you get:

1. All possible outcomes
2. The likelihood of each of those outcomes
• Each column represents one possible outcome (average
result)
• Each block within a column represents one possible
permutation (to obtain that average)

2.5

J-PAL | S AMPLING AND S AMPLE S IZE 14

Rolling 3 dice, taking the average:
16 results 318, 216 permutations
1/6

0
1 1 1/3 1 2/3 2 2 1/3 2 2/3 3 3 1/3 3 2/3 4 4 1/3 4 2/3 5 5 1/3 5 2/3 6
Frequency 0% 1% 3% 5% 7% 10% 12% 13% 13% 12% 10% 7% 5% 3% 1% 0%

J-PAL | S AMPLING AND S AMPLE S IZE 15

What is the “expected value” of the
average of three dice?
A. 1
B. 2
C. 3
D. 4
E. 5
F. 6
G. All of the above
H. None of the above

0% 0% 0% 0% 0% 0% 0% 0%

A. B. C. D. E. F. G. H.
J-PAL | S AMPLING AND S AMPLE S IZE 16
Rolling 4 dice:
21 results, 1296 permutations

14%

12%

10%

8%

6%

4%

2%

0%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

J-PAL | S AMPLING AND S AMPLE S IZE 17

Rolling 5 dice:
26 results, 7776 permutations
12%

10%

8%

6%

4%

2%

0%
1 2 3 4 5 6

J-PAL | S AMPLING AND S AMPLE S IZE 18

Rolling 10 dice:
51 results, >60 million permutations
6.0%

5.0%

4.0%

3.0%

2.0%

1.0%

0.0%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

Looks like a bell curve, or a normal distribution

19
Rolling 30 dice:
151 results, 2 x 10 23 permutations
3.5%

3.0%

2.5%

2.0%

1.5%

1.0%

0.5%

0.0%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

>95% of all rolls will yield an average between 3 and 4 20

Rolling 100 dice:
501 results, 6 x 1077 permutations
2.0%

1.5%

1.0%

0.5%

0.0%
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

>99% of all rolls will yield an average between 3 and 4

Rolling dice: 2 lessons

1. The more dice you roll, the

closer most averages are to the true average
(the distribution gets “tighter”)
-THE LAW OF LARGE NUMBERS-
2. The more dice you roll, the
more the distribution of possible averages
(the sampling distribution)
looks like a bell curve (a normal distribution)
-THE CENTRAL LIMIT THEOREM-

J-PAL | S AMPLING AND S AMPLE S IZE 22

Which of these is more accurate?

I. II.

A. I.
B. II.
C. Don’t know

0% 0% 0%

A. B. C.
J-PAL | S AMPLING AND S AMPLE S IZE 27
Accuracy versus Precision

Precision (Sample Size)

estimates

truth

Accuracy (Randomization)

J-PAL | S AMPLING AND S AMPLE S IZE 28

Accuracy versus Precision

estimates truth
Precision (Sample Size) truth

estimates

truth truth
estimates

estimates

Accuracy (Randomization)

J-PAL | S AMPLING AND S AMPLE S IZE 29

THE basic questions in statistics

• How confident can you be in your results?

•  How big does your sample need to be?

J-PAL | S AMPLING AND S AMPLE S IZE 30

Outline

• Distributions, expected value

• Sampling distributions
– population distribution
– sampling distribution
– law of large numbers/central limit theorem
– standard deviation and standard error
• Sampling and Randomization strategies
– Sampling in practice
– Randomization in practice

J-PAL | S AMPLING AND S AMPLE S IZE 31

Example: Balsakhi Program

J-PAL | S AMPLING AND S AMPLE S IZE 32

Baseline test scores
500

450

400

350

300

250
frequency
200

150

100

50

0
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 33

Mean = 26
500

450

400

350

300

250 26
frequency
200 mean

150

100

50

0
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 34

Standard Deviation = 20
500 600
1 Standard
450 Deviation
500
400

350
400
300

250 26 300 frequency

sd
200
mean
200
150

100
100
50

0 0
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 35

Let’s do an experiment

• Take 1 Random test score from the pile of 16,000 tests

• Write down the value
• Put the test back
• Do these three steps again
• And again
• 8,000 times
• This is like a random sample of 8,000 (with replacement)

J-PAL | S AMPLING AND S AMPLE S IZE 36

Population v. sampling distribution

500 4.0%

450
3.5%
400
3.0%
350
2.5%
300

250 2.0%
frequency
200 freq (N=1)
1.5%
150
1.0%
100
0.5%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

This is the distribution of my sample of 8,000 students 37

What can we say about this sample?

500 4.0%

450
3.5%
400
3.0%
350
26 2.5%
300

250 2.0% mean

200 frequency
1.5%
freq (N=1)
150
1.0%
100
0.5%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

Good, the average of the sample is about 26… 38

But…

• … I remember that as my sample goes, up, isn’t the

sampling distribution supposed to turn into a bell curve?
• (Central Limit Theorem)
• Is it that my sample isn’t large enough?

J-PAL | S AMPLING AND S AMPLE S IZE 39

One limitation of statistical theory is that it
assumes the population distribution is
normally distributed

A. True
B. False
C. Depends
D. Don’t know

0% 0% 0% 0%

A. B. C. D.
J-PAL | S AMPLING AND S AMPLE S IZE 40
The sampling distribution may not be normal
if the population distribution is skewed

A. True
B. False
C. Depends
D. Don’t know

0% 0% 0% 0%

A. B. C. D.
J-PAL | S AMPLING AND S AMPLE S IZE 41
Outline

• Sampling distributions
– population distribution
– sampling distribution
– law of large numbers/central limit theorem
– standard deviation and standard error
– with clusters
• Practical Strategies for Sampling and Randomization
– Sampling in practice
– Randomization in practice

J-PAL | S AMPLING AND S AMPLE S IZE 42

How do we get from here…

500

400
This is the distribution
300
of the population
200
(Population Distribution)
100

0
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

To here… This is the distribution of

Means from all Random
Samples (Sampling
distribution)

J-PAL | S AMPLING AND S AMPLE S IZE 43

Draw 10 random students, take the
average, plot it: Do this 5 & 10 times.

Frequency of Means With 5 Samples

10
8
6
4
2
0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51

Frequency of Means With 10 Samples

10
8
6
4
2
0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51

J-PAL | S AMPLING AND S AMPLE S IZE 44

 Draw 10 random students:
50 and 100 times
Frequency of Means With 50 Samples
10

0
1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152
Frequency of Means with 100 Samples
10

0
1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152

J-PAL | S AMPLING AND S AMPLE S IZE 45

 Draw 10 random students:
500 and 1000 times
Frequency of Means With 500 Samples
80
70
60
50
40
30
20
10
0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51

Frequency of Means With 1000 Samples

80
70
60
50
40
30
20
10
0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51
J-PAL | S AMPLING AND S AMPLE S IZE 46
Draw 10 Random students

• This is like a sample size of 10

• What happens if we take a sample size of 50?

J-PAL | S AMPLING AND S AMPLE S IZE 47

What happens to the sampling distribution if
we draw a sample size of 50 instead of 10,
and plot the mean (thousands of times)?
25% 25% 25% 25%
A. The bell curve will be
narrower
B. The bell curve will be
wider
C. The variance will
increase
D. Neither. The underlying
sampling distribution
does not change.

A. B. C. D.
J-PAL | S AMPLING AND S AMPLE S IZE 48
 Draws of 10 Draws of 50

Frequency of Means With 5 Frequency of Means With 5

Samples Samples
10 10

8 8
6 6
4 4
2 2
0 0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52

Frequency of Means With 10

Frequency of Means With 10 Samples
Samples 10
10 8
8
6
6
4 4
2 2
0 0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52
J-PAL | S AMPLING AND S AMPLE S IZE 49
 Draws of 10 Draws of 50
Frequency of Means With 50 Frequency of Means With 50
Samples Samples
14
14
12
10
8 9
6
4 4
2
0 -1
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52

Frequency of Means with 100 Frequency of Means With 100

Samples Samples
20
20
15
15
10 10
5 5

0 0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52
50
J-PAL | S AMPLING AND S AMPLE S IZE
 Draws of 10 Draws of 50

Frequency of Means With 500 Frequency of Means With 500

Samples Samples
90 100

70 80
60
50
40
30
20
10
0
-10 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52

Frequency of Means With 1000

Frequency of Means With 1000 Samples
Samples 200
160
150
110
100
60 50

10 0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 51
-40
J-PAL | S AMPLING AND S AMPLE S IZE
Outline

• Sampling distributions
– population distribution
– sampling distribution
– law of large numbers/central limit theorem
– standard deviation and standard error
– with clusters
• Practical Strategies for Sampling and Randomization
– Sampling in practice
– Randomization in practice

J-PAL | S AMPLING AND S AMPLE S IZE 52

Population & sampling distribution:
Draw 1 random student (from 8,000)
500 4.0%

450
3.5%
400
3.0%
350
2.5%
300
26
250 2.0% mean
frequency
200
1.5% freq (N=1)
150
1.0%
100
0.5%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 53

Sampling Distribution:
Draw 4 random students (N=4)
500 4.5%

450 4.0%

400
3.5%
350
3.0%
300
2.5%
26
250 mean
2.0% frequency
200
freq (N=4)
1.5%
150
1.0%
100

50 0.5%

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 54

Law of Large Numbers : N=9
500 7.0%

450
6.0%
400

350 5.0%

300
4.0%
250 26 mean
3.0% frequency
200
freq (N=9)

150 2.0%

100
1.0%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 55

 Law of Large Numbers: N =100
500 25.0%

450

400 20.0%

350

300 15.0%

250 26
mean
frequency
200 10.0%
freq (N=100)

150

100 5.0%

50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 56

Central Limit Theorem: N=1
500 4.0%

450
3.5%
400
3.0%
350
2.5%
300
26 mean
250 2.0%
frequency
200 dist_1
1.5%
150 freq (N=1)
1.0%
100
0.5%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

The yellow line is a theoretical distribution 57

Central Limit Theorem : N=4
500 4.5%

450 4.0%

400
3.5%
350
3.0%
300
2.5%
26 mean
250
frequency
2.0%
200 dist_4
1.5% freq (N=4)
150
1.0%
100

50 0.5%

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 58

Central Limit Theorem : N=9
500 7.0%

450
6.0%
400

350 5.0%

300
4.0%
mean
250 26
frequency
3.0%
200 dist_9
freq (N=9)
150 2.0%

100
1.0%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 59

Central Limit Theorem : N =100
500 25.0%

450

400 20.0%

350

300 15.0%
mean
250 26
frequency
200 10.0% dist_100
freq (N=100)
150

100 5.0%

50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 60

Standard deviation/error

• What’s the difference between the standard deviation

and the standard error?
• The standard error = the standard deviation of the
sampling distributions

J-PAL | S AMPLING AND S AMPLE S IZE 62

Variance and Standard Deviation
∑ 𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉 − 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 2
2
Variance = 𝜎𝜎 =
𝑁𝑁

𝜎𝜎 2 = 400

Standard Deviation = 𝜎𝜎 = 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉

𝜎𝜎 = 20

Standard Error = 𝜎𝜎� 𝑁𝑁

SE = 20� 𝑁𝑁

J-PAL | S AMPLING AND S AMPLE S IZE 63

Standard Deviation & Standard Error
500 4.0%

450
3.5%
400
3.0%
350
2.5%
300
26 mean
250 2.0%
frequency
200 sd
1.5%
dist_1
150
1.0%
100
0.5%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 64

Sample size ↑ x4, SE ↓ ½

500 4.5%
SD = 20 SE = 10
450 4.0%

400
3.5%
350
3.0%
300
2.5% mean
250 26 frequency
2.0% sd
200
se4
1.5%
150 dist_4

1.0%
100

50 0.5%

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 65

Sample size ↑ x9, SE ↓ ?
500 7.0%

450
6.0%
400

350 5.0%

300
4.0%
mean
250 26
frequency
3.0% sd
200
se9
150 2.0% dist_9

100
1.0%
50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 66

Sample size ↑ x100, SE ↓?
500 25.0%

450

400 20.0%

350

300 15.0%
mean
250 26 frequency
sd
200 10.0%
se100
150 dist_100

100 5.0%

50

0 0.0%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
test scores

J-PAL | S AMPLING AND S AMPLE S IZE 67

Increasing the sample size by a factor of
4 results in
20% 20% 20% 20% 20%
A. Making the standard
error 1/4th the size
B. Making the standard
error half the size
C. Doubling the standard
error
D. Quadrupling the
standard error
E. Doubling the standard
deviation

A. B. C. D. E.
J-PAL | S AMPLING AND S AMPLE S IZE 68