Scribd
Upload
10 views10 pages

Bootstrap and Permutation Tests Explained

Lab 9 discusses resampling methods, particularly the bootstrap technique, which involves resampling from an existing sample to assess the stability of sample means. It highlights the use cases for bootstrap methods, such as limited data and non-normal distributions, while also noting the assumption that the sample must be representative of the population. Additionally, permutation tests are introduced as a way to estimate p-values without relying on distribution assumptions by creating a null distribution through random arrangements of data.

Uploaded by

Neeraj Yadav
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
10 views10 pages

Bootstrap and Permutation Tests Explained

Lab 9 discusses resampling methods, particularly the bootstrap technique, which involves resampling from an existing sample to assess the stability of sample means. It highlights the use cases for bootstrap methods, such as limited data and non-normal distributions, while also noting the assumption that the sample must be representative of the population. Additionally, permutation tests are introduced as a way to estimate p-values without relying on distribution assumptions by creating a null distribution through random arrangements of data.

Uploaded by

Neeraj Yadav
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd

Lab 9: Resampling methods

Enter the bootstrap


Bradley Efron The computer
The procedure is quite straightforward
• To obtain a sense of how stable a sample
mean is (how it distributes).
• Simply *resample* (with replacement!) from
the existing sample.
• Do this n times (10k  100k  1m)
• Calculate the standard deviation of these
resampled means.
• Compare with the figure of interest.
Bootstrap logic

37 577
3 577 73757
?
Use cases
• When there is only limited data (controversial)
• When the underlying distribution is not
normal and/or not known
• When estimating sample means of rare events
Bootstrap drawback/assumption
• This works if and only if the sample truly is
representative of the population.
• In other words, if something didn’t happen in the
sample, it can’t ever happen.
• “The data we have is the only data that can ever be”
• So small samples necessarily over- or underestimate
the probability of rare events.
It seems like a miracle

But it works (usually)!


Permutation tests
• Classical tests (e.g. t-test) assume that the data is
distributed in a certain way.
• If the distribution of the data is not what is assumed,
the reported p-value by the test is not the real p-
value (!)
• Permutation tests use the actual data to estimate
how likely a given result is.
• Logic: We pretend we lost the labels (which group
data came from), then create a null distribution (by
random arrangement of groups) and compare with
Example
• A rat is stressed out for 2 weeks
• 10 neurons are then taken out
• 5 we treat with Ketamine and 5 we don't treat at all
• We then count the number of dendritic spines of each
neuron
• Hypothesis: Ketamine works by growing dendritic spines
• K = [117 123 111 101 121]
• C = [98 104 106 92 88]
• Test statistic: sum(K) – sum(C)
• We can now calculate exact p-value by determining the
null distribution through resampling methods.
The result

You might also like