Section 8: Fisher’s exact test
1
A brief theoretical review
Fisher’s exact test is a statistical significance test used
in the analysis of contingency in order to verify if data
in a 2 × 2 contingency table confirm the hypothesis (H0 )
which holds that the two categorical variables have no
association with each other.
Although in practice it is employed when sample sizes
are small, it is valid for all sample sizes. It is named
after its inventor, R. A. Fisher, and is one of a class
of “exact” tests, so called because the significance of
the deviation from a null hypothesis can be calculated
exactly, rather than relying on an approximation that
becomes exact in the limit as the sample size grows to
infinity, as with many statistical tests.
With large samples, a chi-squared test can be used. Ho-
wever, the significance value it provides is only an appro-
ximation, because the sampling distribution of the test
statistic that is calculated is only approximately equal
to the theoretical chi-squared distribution. The appro-
ximation is inadequate when sample sizes are small, or
the data are very unequally distributed among the cells
of the table, resulting in the cell counts predicted on the
null hypothesis (the “expected values”) being low.
2
When has Fisher’s exact test to be used?
The usual rule of thumb for deciding whether the chi-
squared approximation is good enough is that the chi-
squared test is not suitable when the expected values
in any of the cells of a contingency table are below 5,
or below 10 when there is only one degree of freedom
(this rule is now known to be overly conservative). In
fact, for small, sparse, or unbalanced data, the exact
and asymptotic p-values can be quite different and may
lead to opposite conclusions concerning the hypothesis
of interest. For contingency tables with a large sample
size and well-balanced numbers in each cell of the table,
Fisher’s exact test is not accurate, and the chi-square
test is preferred.
Exact computations are based on the statistical theory
of exact conditional inference for contingency tables
Fisher’s exact test is definitely appropriate when the row
totals and column totals are both fixed by design. once
margins have been fixed the joint frequencies distribu-
tion will not depend on nuisance parameters (as total
sample size,row and column margins) but only on the
strength of association between A and B through the
cross product ratio Ψ.
3
To understand how Fisher’s exact test works, it is es-
sential to understand what a contingency table is and
how it is used. In the simplest example, there are only
two variables to be compared in a contingency table.
Usually, these are categorical variables.
To verify independence is equivalent to verify:
H0 : Ψ = 1 H0 : Ψ = 1 H0 : Ψ = 1
or or
H1 : Ψ 6= 1 H1 : Ψ > 1 H1 : Ψ < 1
Under the independence assumption, that is if the null
hypothesis H0 : Ψ = 1 holds, with margin fixed by de-
sign, the table will follow an hypergeometric standard
distribution; so, in order to conduct the conditioned test
with a significance level α it will be necessary:
1. to order all possible tables that have the same mar-
gins of the observed ones on the basis of their
probability under the null hypothesis;
2. to evaluate αobs as the sum of evidence provided by
the observed data (or any more extreme table) for
the null hypothesis.
3. the smaller the value of αobs , the greater the evi-
dence for rejecting the null hypothesis.
4
Fisher’s exact test in R: [Link]()
The [Link]() performs Fisher’s exact test for te-
sting the null of independence of rows and columns in
a contingency table with fixed marginals.
As first argument you can pass both a matrix or two
vectors having same length.
If x is a matrix, it is taken as a two-dimensional contin-
gency table, and hence its entries should be nonnegative
integers.
The second mandatory argument is the type of alterna-
tive hypothesis. The alternative for a one-sided test is
based on the odds ratio, so alternative = “greater” is a
test of the odds ratio being bigger than or. Two-sided
tests are based on the probabilities of the tables, and
take as “more extreme” all tables with probabilities less
than or equal to that of the observed table, the p-value
being the sum of such probabilities.
Examples:
1. x<-matrix(c(1, 9,11, 3),nrow=2,byrow=T,
dimnames=list(Diet=c("Yes","No"),
Gender=c("M","F")))
[Link](x)
[Link](x,alternative="less")
2. TeaTasting <-matrix(c(3, 1, 1, 3),nrow=2,
dimnames=list(Guess=c("Milk","Tea"),
Truth =c("Milk","Tea")))
[Link](TeaTasting, alternative = "greater")
5
Now, an example of evaluation of αobs by “hand”
Let’s to observe the table
tabobs<-matrix(c(3,2,1,9),2,2,byrow=T)
p3<-dhyper(3,5,10,4)
tables with the same marginals will be
tab1<-matrix(c(4,1,0,10),2,2,byrow=T)
tab2<-matrix(c(2,3,2,8),2,2,byrow=T)
tab3<-matrix(c(1,4,3,7),2,2,byrow=T)
tab4<-matrix(c(0,5,4,6),2,2,byrow=T)
The probability of observing each of the previous tables
under the independence hypothesis will be:
p4<-dhyper(4,5,10,4)
p2<-dhyper(2,5,10,4)
p1<-dhyper(1,5,10,4)
p0<-dhyper(0,5,10,4)
plot(0:4,c(p0,p1,p2,p3,p4),type="h")
points(0:4,c(p0,p1,p2,p3,p4),pch=20)
If the alternative hypothesis is two-sided, then:
αobs = p3 + p4
as the table with n11 = 4 has observed probability lower
than p4. So, in this particular example the observed
6
p-value will be equal both in the case of two-sided or
positive association (“grater”) hypothesis:
p3+p4
[Link](taboss,alt="greater")$[Link]
[Link](taboss,alt="[Link]")$[Link]
On the contrary, if the alternative hypothesis holds a
negative association then
αobs = p3 + p2 + p1 + p0
p3+p2+p1+p0
[Link](taboss,alt="less")$[Link]
Despite the fact that Fisher’s test gives exact p-values,
some authors have argued that it is conservative, i.e.
that its actual rejection rate is below the nominal signi-
ficance level.
In order to demonstrate it lt us to simulate tables from
an independence model:
x1<-c(0,0,1,1)
x2<-c(0,1,0,1)
mu<-exp(5+.5*x1-.2*x2)
mu
fisher<-rep(NA,1000)
chisq<-rep(NA,1000)
for(i in 1:1000){
y<-rpois(4, mu)
tab<-matrix(y,nrow=2)
fisher[i]<-[Link](tab)$p
chisq[i]<-[Link](tab)$[Link]
}
mean(fisher<=0.05) #ampiezza reale del test
The p-value cumulative function emphasizes this aspect:
plot(sort(fisher),(1:length(fisher))/length(fisher),
bty="l",type="s", xlab="p-valore",ylab="probabilita")
abline(0,1,lty=2) #la f.r. teorica di riferimento
The cumulative function of p-value of the observed is
under the theoretical one (it is known that the under the
null hypothesis the p-value follows an uniform distribu-
tion) this means that the actual rejection rate is below
7
the nominal significance level (0.05), the true probabi-
lity of incorrectly rejecting the null hypothesis is never
greater than the nominal level.
Fisher’s exact test: exercises
• Exercise 1
Data records sex and whether or not the individual
has perfect pitch for 99 conservatory of music stu-
dents. Use Fisher’s exact test of the null hypothesis
that sex is independent of having perfect pitch.
The data can be tabulated as follows.
• Exercise 2
The following table shows the results of a retrospec-
tive study comparing radiation therapy with surgery
in treating cancer of the larynx. The response indi-
cates whether the cancer was controlled for at least
two years following treatment.
8
• Report and interpret the P-value for Fisher’s exact
test with H1 : Ψ > 1 and H1 : Ψ 6= 1. Using the
function [Link](), explain how the P-values
were calculated.
• Using R (e.g. write a function that computes the
hypergeometric probability for each entry in the
first cell of the 2 × 2 table), plot the (discrete)
distribution of the P-value from Fisher’s exact test.