CHAPTER 5

Basic concepts for valid testing

5.1. Bayes p-values and prior-corrected p-values

A commonly encountered null hypothesis about an unknown quantity U is that it is equal to 0 or some

other fixed value u0 . That null hypothesis is the claim that U = u0 . The estimated prior probability of that
b (U = u0 ) for short.
null hypothesis is then written as Pr

The Bayes factor, abbreviated by BF, is the constant of proportionality between the prior odds and the

posterior odds of a null hypothesis (Section 3.2.1). It is a way to measure the amount of evidence supporting

a null hypothesis over its alternative hypothesis.

If you declared a discovery of an effect, then that probability is equal to the probability that your dis-

covery is false. That is called the local false discovery rate. The symbol for that is Pr (false discovery | p),

which is a short way to say, “the probability that you would make a false discovery if you rejected the null
b (false discovery | p) is an estimate
hypothesis, given that the p-value for that null hypothesis is equal to p.” Pr

of the posterior probability that the null hypothesis that U = u0 is true:

b (false discovery | p) = Pr
b (U = u0 | p) = 1
(5.1.1) Pr .
1−Pr(U=u0)
b
1+
BF ×Pr(U=u
b 0)

Section 3.2.1 explains where its version of that equation that comes from. That version is labeled equation

(3.2.1).
b (false discovery | p), unlike BF alone, accounts for Pr
Pr b (U = u0 ). However, from the standard frequen-

b (false discovery | p) is not a valid p-value, whereas BF is a valid

tist paradigm of statistical inference, Pr
b (U = u0 ) in a single number?
p-value. Can we have both validity and dependence on Pr
b (U = u0 ) is the Bayes p-value, which is
Yes, a valid p-value that accounts for Pr

 1

 if BF < 1
Pr(U=u0 )+(1−Pr(U=u0 ))/ BF
pBayes =
b b
(5.1.2) .

1 if BF ≥ 1


b (false discovery | p) and pBayes together on the y-axis for each value Pr
E XERCISE 5.1. Plot Pr b (U = u0 )

on the x-axis from 0 to 1, with separate a separate plot for each of these values of the Bayes factor:

(1) BF = 2
67
 5.2. ESTIMATING THE BAYES FACTOR FROM A P-VALUE 68

(2) BF = 1

(3) BF = 0.05

(4) BF = 0.01

E XERCISE 5.2. What general patterns do you notice in the plots you made for Exercise 5.1?

5.2. Estimating the Bayes factor from a p-value

One way to estimate the Bayes factor is to transform a p-value using a function called a calibrator. An

example of a calibrator is the function given by

c= 0.5
(5.2.1) BF .
p−1/3 −1

c or by the value of the Bayes factor estimated by a different calibrator, pBayes

Whenever BF is replaced by BF

is called a prior-corrected p-value.

E XAMPLE 5.1. If p = 0.05 and Pr

b (U = u0 ) = 10/11, then this R code may be used to estimate the Bayes

b (false discovery | p)
factor using the calibrator in equation (5.2.1) and to calculate the corresponding values of Pr

and pBayes by substitution into equations (5.1.1) and (5.1.2):

> p = 0 . 0 5 ; BF = 0 . 5 / ( p ^ ( − 1 / 3 ) − 1 ) ; p r i o r 0 = 1 0 / 1 1 ;

r o u n d ( c ( " p− v a l u e " = p , " Bayes f a c t o r " = BF ,

" p r i o r p r o b a b i l i t y o f t h e n u l l h y p o t h e s i s "= p r i o r 0 ,

" p o s t e r i o r p r o b a b i l i t y o f t h e n u l l h y p o t h e s i s " = 1 / ( 1 + ( 1 − p r i o r 0 ) / ( BF * p r i o r 0 ) ) ,

" Bayes p− v a l u e " = 1 / ( p r i o r 0 +(1 − p r i o r 0 ) / BF ) ) , d i g i t s = 3 )

p− v a l u e

0.050

Bayes f a c t o r

0.292

p r i o r p r o b a b i l i t y of the n u l l hypothesis

0.909

p o s t e r i o r p r o b a b i l i t y of the n u l l hypothesis

0.745

Bayes p− v a l u e

0.819
 5.2. ESTIMATING THE BAYES FACTOR FROM A P-VALUE 69

Similarly, for p = 0.005 and Pr

b (U = u0 ) = 50%:

> p = 0 . 0 0 5 ; BF = 0 . 5 / ( p ^ ( − 1 / 3 ) − 1 ) ; p r i o r 0 = 5 0 / 1 0 0 ;

r o u n d ( c ( " p− v a l u e " = p , " Bayes f a c t o r " = BF ,

" p r i o r p r o b a b i l i t y o f t h e n u l l h y p o t h e s i s "= p r i o r 0 ,

" p o s t e r i o r p r o b a b i l i t y o f t h e n u l l h y p o t h e s i s " = 1 / ( 1 + ( 1 − p r i o r 0 ) / ( BF * p r i o r 0 ) ) ,

" Bayes p− v a l u e " = 1 / ( p r i o r 0 +(1 − p r i o r 0 ) / BF ) ) , d i g i t s = 3 )

p− v a l u e

0.050

Bayes f a c t o r

0.103

p r i o r p r o b a b i l i t y of the n u l l hypothesis

0.500

p o s t e r i o r p r o b a b i l i t y of the n u l l hypothesis

0.093

Bayes p− v a l u e

0.187

E XERCISE 5.3. Given a null hypothesis for which you have a p-value and prior probability, complete

the following subexercises using equation (5.2.1). Hint: modify the code of Example 5.1 using your p-value

and prior probability, and then paste the modified code into a Posit cloud workspace.1

(1) What are the numeric values of these?

(a) The estimated Bayes factor, BF

c

b (U = u0 | p)
(b) The estimated posterior probability of the null hypothesis, Pr

(c) The prior-corrected p-value, a case of the Bayes p-value, pBayes

(2) What do your answers to subexercise (1) tell you about how certain or uncertain you can be about

whether or not the null hypothesis is true? How would you explain the results in a post to social

media?

b (false discovery | p) and pBayes to-

b (U = u0 ) = 0 and equation (5.2.1), plot Pr
E XERCISE 5.4. Using Pr

gether on the y-axis for each value p on the x-axis from 0 to 1, with separate a separate plot for each of these

values of the p-value:

1[Link]
 5.3. FURTHER READING 70

(1) p = 0.5

(2) p = 0.1

(3) p = 0.05

(4) p = 0.005

E XERCISE 5.5. What general patterns do you notice in the plots you made for Exercise 5.4?

b (U = u0 ) = 50%.
E XERCISE 5.6. Complete Exercise 5.4 except using Pr

E XERCISE 5.7. What general patterns do you notice in the plots you made for Exercise 5.6?

5.3. Further reading

Equation (5.1.2) gives a valid p-value not only when BF is a Bayes factor but also when 1/ BF is an

e-value [104, Prop. 2.2].

The quantity of equation 5.1.2 is a special case of the Bayes p-value defined more generally by Bickel

[27]. In addition, the calibrator of equation (5.2.1) may be replaced by the reciprocal of any p-to-e calibrator

meeting the conditions of Vovk and Wang [104, Prop. 2.1], potentially yielding a different prior-corrected

p-value for each p-to-e calibrator [27].