Section 4

Introduction to Statistical Inference

Helper or hinderer?
We will begin this section with a prominent Psychology study about natural tendencies of infants,
published in 2007. Infants were shown a play that showed one toy/shape playing the helping role,
while a different toy was given the hindering role. Afterward, infants were given the choice to pick
one of the two toys to play with.
In a pilot run for this study, 16 ten-month old infants were recruited for the study, and 14 of them
chose the helper toy. Is this evidence that infants have natural tendencies toward helpful actions, or
was it just dumb luck? Maybe these infants didn’t internalize anything from the play they were
shown, and randomness alone led to 14 of the 16 infants picking the helper toy.
We can test this idea statistically by asking a probability question: if the infants truly were picking
these toys at random and were not influenced by the play, how likely was it for at least 14 of these
infants to pick that toy? Using this probability, we can evaluate whether the naïve hypothesis we
made that infants pick the toys with equal probability is potentially valid. A larger probability
would seem to show that this data could plausibly occur under random chance, where a small
probability would indicate that maybe this assumption is not a good one.
This line of thinking describes the idea/process of hypothesis testing:
1. Make an assumption about some parameter. (in this case, a population proportion p)
2. Collect data-based evidence to potentially challenge that assumption.
3. Based on the data, evaluate whether the initial assumption is reasonable.
To address this kind of simulation as a class, let’s begin by collecting data-based evidence through
simulation. We will first make the assumption that the babies are just picking toys with no
preference randomly, that is, the proportion of times they pick the helpful toy is 0.5. In this case, we
can simulate outcomes using a coin, let’s say heads represents a baby picking the helper toy, and
tails represents the hinderer toy. Try flipping 16 coins now to simulate what would happen in a
similarly sized sample! When you’ve completed this, enter your result into the form provided by
your instructor. You’ll look at the compiled results as a class once everyone has completed this!
Simulating the coin experiment
To be able to collect more sets of 16 coin flips to emulate what babies with no toy preference might
do, we can use a tool to simulate this. Follow the link below to open this tool:

[Link]
You’ll be presented with a simulation device that looks like the image
to the left. This box represents what one individual outcome of your
probability experiment looks like. For the case of the babies (or a
coin), this is what the choice of one baby would be. When we use this
device, we will always use 0’s and 1’s to represent outcomes. This is
because the device will sum up all of the outcomes to get a result, so
results of 0 are not counted and results of 1 are. For the choice of one
baby here, we will thus have two outcomes inside the box: a 1 to
represent the “helper” toy being picked, and a 0 to represent the
“hinderer” toy being picked.
We don’t just have one baby though, we have 16! To account for the
number of times you use the device, or the sample size, change the
“Select ___ items” at the top to 16. If we click “start” at the top, we will
simulate one set of 16 coin flips, or a set of 16 babies’ choices,
assuming they have no preference. But just simulating one set or one
sample is not enough, we need to
do this many times to build out a
full distribution. Let’s try to collect 1000 samples total! Make sure
to set the speed at “Fastest” so it will do these simulations quickly.
Doing this will build out a full distribution for the possible
outcomes. The precise distribution you simulate may look
different than the one someone else made due to the random
nature of simulations, but they should be very similar. We’d like to
assess how likely it was for babies with no preference to choose
the helper toy 14 out of 16 times. To do this easily, we can turn on
a “movable line” to our plot (see the image on the right for where
to do this.) Once we create this movable line, we divide up and
count all of the results where at least 14 or more helper toys are
picked. Doing so makes a distribution like the one shown below:
From the simulation, we can see that a result of 14 or more occurred 3 out of 1000 times, or 0.3% of
the time. This indicates that if the 16 babies had no preference for either toy, it is incredibly
unlikely to see at least 14 of them pick the helper toy just by chance. Thus, it seems that the
researchers hypothesis that babies have a natural tendency for helpful behaviors seems supported
by this simulation!
Hypothesis testing terminology
The simulation we carried out in class and using the simulation software is what is known as a
hypothesis test. We know that the researchers hypothesized that babies had helpful tendencies, but
in statistical testing, we usually write out these hypotheses in terms of a parameter. Here, we were
testing a proportion, specifically, the proportion of babies that preferred the helper toy, so we
typically write out our hypothesis in terms of this proportion, p.
We don’t just write out one hypothesis – we write out two! One hypothesis represents the
assumption that we made in carrying out the simulation: that the babies had no preference for
either toy. This is referred to as the ___________ hypothesis. The other hypothesis reflects the
researcher’s hypothesis and basis for doing the statistical test. This is the ____________________
hypothesis. Using appropriate notation, we would write out these hypotheses as follows:

H0: ________________________________

Ha: ________________________________

The 0.3% figure that we found is called a ______________________. Generally, this is defined as the
probability of getting a result as extreme or more, assuming your null hypothesis is true. This
matches what we did, as we assumed the babies had no preference for either toy to be true, and
found the probability of getting 14 or more babies to pick the helper toy.
This measure represents the level of evidence that we have against our null hypothesis. The lower
the probability, the less plausible it is to get our data if this hypothesis were to be true, giving
evidence that the null hypothesis is not a good hypothesis. But the higher the probability, the more
plausible it is to get our data under this hypothesis, giving credibility to the null hypothesis. It is
often difficult to determine what constitutes “enough” evidence to go against a null hypothesis. Let’s
try carrying out an example using the simulation tool for yourself now!
Example: The scenario from your nightmares has come true – you’re taking a multiple
choice exam but are very unprepared to take the test. There are 50 questions on the exam
and each question has four choices. Your score comes back and you get 17 questions
correct. While the grade for this exam is not ideal, this is better than 25% of the questions
which is better than what you would expect from random guessing. But could you have
obtained a score like this even if you were randomly guessing? Use the simulation tool to
find the p-value in this scenario. (250 samples should be enough – this takes some time!)
The results of this test bring up an important question: what p-values constitute enough evidence to
go against your null hypothesis? A 10% chance isn’t super likely, but it does happen 1 in 10 times,
which shows that it is still somewhat plausible to occur. 10% is not a particularly high probability,
but I’m sure we’ve all experienced weather forecasts with a 10% chance of rain where it rains all
day.
To make decisions for a hypothesis test, we need to determine what constitutes enough evidence to
go against our null hypothesis. What we determine as “enough” can change depending on the
context of our test and how important the decision we make is. But as a rough guideline, this is how
statisticians typically interpret the strength of their evidence in testing:

p-value Interpretation
> 0.1 No evidence against the null hypothesis
0.05 – 0.1 Some/weak evidence against the null hypothesis
0.01 – 0.05 Moderate evidence against the null hypothesis
0.001 – 0.01 Strong evidence against the null hypothesis
< 0.001 Very strong evidence against the null hypothesis
When conducting a hypothesis test, researchers often set a cutoff value for what constitutes enough
evidence for their purposes. This value, denoted α, is referred to as the ________________________________.
Based on the p-value we get, we would make decisions about our test as follows:
If ________________________________, Reject the null hypothesis (H0)

If ________________________________, Fail to reject the null hypothesis (H0)

Typically, α is often set to 0.05 as a “default” value. This is a standard originally set by statistician
RA Fisher nearly 100 years ago, and was done so rather arbitrarily. Yet even today, many research
journals that accept quantitative work use 0.05 as the default level of significance for tests. This
creates a bit of a false dichotomy – 0.051 and 0.049 are very similar levels of evidence for p-values,
but we would interpret them very differently according to the way we have defined the decision of
a test. If we conduct a hypothesis test and get a p-value that is very close to our significance level,
rather than make a broad decision based on a borderline result, we should try to to replicate the
study to see if how our results might differ with a new sample.
Question: Why do we use the terms “reject” and “fail to reject” for hypothesis testing? Why
wouldn’t we “accept” our null hypothesis based upon the results of a test?
Two-sided alternatives

Example: You have found a coin on the ground that has seen
better days and are curious to determine if it is still a fair coin
when flipped – that is, heads and tails are still equally likely to
occur. You flip the coin 100 times and get 58 heads. Does this
result still seem plausible if the coin were still fair? Use the
simulation tool to determine a p-value.

An interesting consideration when conducting this test is that if the coin is truly not fair due to the
bend, we don’t know whether heads or tails is more likely to occur. What does this mean about our
p-value, and what results are considered “as extreme or more” than what we observed?

In this test, we would write out our hypotheses as follows:

H0: ________________________________

Ha: ________________________________
Errors in hypothesis testing
The difficulty in making statistical decisions is that we never have the ability to prove anything.
While we saw that 14 out of 16 infants choosing the helper toy was strong evidence toward the idea
that infants were making the decision consciously and not randomly, it’s always possible that it
could have happened by random chance. Similarly, if we flip 16 coins, it’s unlikely to get 14 heads to
come up, but it is definitely possible.
Thus, it’s possible that these infants really didn’t consciously choose the helper toy 14 times, and
this was just statistical improbability. If we assume this to be the case, then going on to conclude
that they were making this choice consciously and preferring helpful behaviors would be an error.
The type of error that could have been made in this case would be called a ___________________________,
where you reject H0 when H0 was actually true. On the flip side, if you fail to reject H0 and Ha is
actually true, this would be called a _______________________. The table that follows shows the possible
outcomes of a hypothesis test:
Decision
Fail to Reject H0 Reject H0

H0
The true
hypothesis
Ha

The probability that a type I error occurs has already been defined – it’s α. Why? If the null is
actually true, then our null distribution that we would simulate not only reflects the null, it would
reflect actual samples we would obtain in the real world too. Because these match, the p-values we
get match the actual probabilities of obtaining a result like that. Getting a low p-value would result
in an error here (since the null is true!) so the percentage of results that produce a p-value lower
than the significance level would just be the significance level itself.
As a result, we can think about setting α at the beginning of a test as a way to say how often we are
comfortable with rejecting the null incorrectly. Hypothesis tests are often used to make real-world
decisions, so when setting α, you should think about what the consequences are of being wrong. If
your hypothesis tests will make a decision regarding releasing a new prescription to the public, and
you were testing for its side effects being potentially deadly, a lower α value might be a good idea. If
your consequences are less dire, then a higher value of α is potentially acceptable.
We use the symbol β to represent the probability of a type II error. While α is something that we
know and set at the outset of a hypothesis test, β is not easily calculated. One common
misconception is that α and β are complementary probabilities, but they are not. The probability of
a type I error is based on the assumption that the null is true, where type II error is based on the
assumption that the alternative is true, so they can’t be complementary because they are based on
different assumptions!
The complement of β does have its own definition though: the quantity 1 – β is the _______________.
This value tells you a probability of rejecting the null hypothesis when the alternative is true, a
probability of utmost importance to researchers. Hypothesis testing is designed so that the
researcher’s theory is reflected in the alternative hypothesis. Collecting data can be expensive, so
before conducting the test, they will want to know what the likelihood is that that they will get
results that match their theories.
If one is interested in increasing the power of a test, there are two ways to accomplish this:

• ______________________: This is a somewhat artificial way of increasing power, but by making

more p-values able to reject the null hypothesis, you also make it more likely that you do so
correctly. The cost here is that you also increase the probability that you reject the null
hypothesis when the null is true too.

• ______________________: By increasing this, you get more information about your population,
and thus make better and more informed decisions. This makes it more likely to distinguish
your data from the null hypothesis.
To illustrate these relationships, try using the applet at the link below:

[Link]
Using R to conduct a hypothesis test

At this point, you might have realized that all of the probabilities that we have computed thus far
are just like binomial probabilities we did last section, as they are independent, have two outcomes,
and have a fixed sample size/probability. So why did we go through the simulations? To see the
probability models that we use to generate them and emphasize that they assume a null hypothesis
is true! But rather than carry tests out using these simulations every time, we can also just compute
a binomial probability. Remember from last section that we can use the pbinom function in R to
compute these probabilities.

Example: Using the pbinom function, find the p-value for the helper-hinderer scenario
where 14 of the 16 babies used the helper toy.

Another way we can compute these probabilities quickly is using the [Link] function.

[Link](x, n, p=p0, alternative=ALT)

The values of x and n are your count and sample size, and p0 is the null value. For the alternative
(ALT), specify one of three options: “less”, “greater”, or “[Link]” depending on the
direction of your test.

Example: Using the [Link] function, find the p-value for the helper-hinderer scenario
where 14 of the 16 babies used the helper toy.
Practical Significance
It’s important to remember that the purpose of hypothesis testing is to see if your data fits within
the null hypothesis or not. We noted earlier that you can increase the power of your test by
increasing the sample size, but sometimes this may go too far. Consider the next example:
Example: Trident’s advertisements always claim that 4 out of 5 dentists recommend Trident
gum to their patients over other brands of chewing gum. Someone thought that they might
be overstating how many dentists really recommend their gum, so they took a random
sample of 4000 dentists in the US and asked them if they would recommend Trident gum to
their patients. 3132 of those dentists said that they would recommend Trident. Evaluate
Trident’s claim based on this data with α = 0.01.

According to this test, we would conclude that Trident’s claim is false. But is that fair to say? If we
check the percentage of dentists that recommended Trident, we get 3132/4000 = 78.3%. The
hypothesis test that we conducted told us that if 80% of all dentists recommend Trident, then
seeing the data that we obtained of 78.3% at such a high sample size is very unlikely. But does that
make Trident’s claims in their commercials invalid? A difference of 1.7% is not terribly meaningful,
and for advertising purposes, rounding to an easy fraction is better for delivering their message. It’s
important to keep in mind that hypothesis testing can only tell us if the null hypothesis is not
consistent with our data, and that the actual difference between the data and the null hypothesis
might not hold any practical importance. You may be able to have a more powerful test with a
larger sample size, but that larger sample size may not end up telling you results that are
meaningful, even if they are statistically significant.
To wrap up this section, let’s finish with one more example that does all steps of a hypothesis test
using R, and interprets the results of the test. This example highlights another useful thing about
using R to compute p-values from a hypothesis test: we can define null hypotheses using any
general proportion, rather than using the “lottery ball” machine in the simulation tool that requires
exact fractions.
Example: The 2010 census showed that 55.3% of US households do not have any children. A
random sample of 500 US households is taken now to assess if this population proportion
has changed since 2010. Of the 500 households, 305 had no children. Use a α = 0.05 to carry
out the test.
Write out hypotheses:
H0: ________________________________

Ha: ________________________________
Conduct the test and evaluate the evidence:

Interpret the p-value:

Make a decision for the test:

Draw conclusions:
Additional Practice:
Example: In Western Countries, only about 12% of the population identifies as “left-
handed.” While part of hand preference may be environmentally conditioned, genetics may
also play a role. One theory posits that red-headed people are more likely to be left-handed!
Based on genetic theory, we’d like to see if red-headed people might be more likely to be
left-handed than the general population. Let’s say that we take a random sample of 125 red-
headed people. We found that 40 of them had a preference for their left-hand. Conduct a full
hypothesis test for this scenario:
Write out hypotheses:
H0: ________________________________

Ha: ________________________________
Conduct the test and evaluate the evidence:

Interpret the p-value:

Make a decision for the test:

Draw conclusions: