Experimental methods

Experimental methods are research designs in which the researcher explicitly and
intentionally induces exogenous variation in the intervention assignment to facilitate
causal inference. The experimental method involves the manipulation of variables to
establish cause-and-effect relationships. The key features are controlled methods and
the random allocation of participants into controlled and experimental groups. This
method relies on controlled research methods and random assignment of study subjects
to test a hypothesis.

The increased popularity of the experimental method was due in part to the rise in the
cognitive approach to mass communication, a perspective that derives from the field of
psychology, where experiments are the main source of data gathering.

We examine only the more basic techniques in this chapter, with discussions of the
controlled laboratory experiment, quasiexperimental designs, and field experiments.

There are several reasons to use the experimental method:

1. Evidence of causality.
2. Control is an advantage of the experimental method. Researchers have control
over the environment, the variables, and the subjects.
3. Cost.
4. Replication.
The experimental technique, however, is not perfect. It has three major disadvantages.
1. Artificiality.
2. Researcher (experimenter) bias.
3. Limited scope. Finally, some research questions simply do not lend themselves to the
experimental approach. Many of the more interesting research topics in mass media are
concerned with the collective behavior of perhaps millions of people. Experiments on
this scale are much too massive to conduct.
CONDUCTING EXPERIMENTAL RESEARCH
The experimental method involves both manipulation and observation. In the simplest
form of an experiment, researchers manipulate the independent variable and then
observe the responses of subjects on the dependent variable. Although every
experiment is different, most researchers agree that the following eight steps should be
followed when conducting an experiment:
1. Select the setting. Many experiments are best conducted in a laboratory or in another
environment under the direct control of the researcher.
2. Select the experimental design. The appropriate design depends on the nature of the
hypothesis or the research question, types of variables to be manipulated or measured,
availability of subjects, and amount of resources available.
3. Operationalize the variables. In the experimental approach, independent variables are
usually operationalized in terms of the manipulation done to create them. Dependent
variables are operationalized by constructing scales or rules for categorizing
observations of behavior.
4. Decide how to manipulate the independent variable. To manipulate the independent
variable (or variables), a set of specific instructions, events, or stimuli is developed for
presentation to the experimental subjects.
There are two types of manipulations: straightforward and staged. In a straightforward
manipulation, written materials, verbal instructions, or other stimuli are presented to
the subjects.
In a staged manipulation, researchers construct events and circumstances that enable
them to manipulate the independent variable. Staged manipulations can be relatively
simple or rather elaborate. They frequently involve the use of a confederate, a person
who pretends to be a subject but who is actually part of the manipulation.
5. Select and assign subjects to experimental conditions. Recall from Chapter 1 that to
ensure external validity, experimental subjects should be selected randomly from the
population under investigation. The various random sampling techniques discussed in
Chapter 4 are appropriate for selecting subjects for experimental studies.
6. Conduct a pilot study. A pilot study with a small number of subjects will reveal
problems and allow the researcher to make a manipulation check—a test to determine
if the manipulation of the independent variable actually has the intended effect.
7. Administer the experiment. After the problems are corrected and the manipulation is
checked, the main phase of data collection begins. Experimental manipulations can be
carried out on either individuals or groups. The dependent variable is measured, and the
subjects are debriefed. During debriefing, the researcher explains the purpose and the
implications of the research. If the manipulation required deception, the researcher
must explain why and how the deception was used. (See Chapter 3 for more about
deception and other ethical problems of research.)
8. Analyze and interpret the results. The subjects’ scores on the dependent variable(s)
are tabulated, and the data are analyzed.

CONTROL OF CONFOUNDING VARIABLES

As discussed in previous chapters, researchers must ensure the internal validity of their
research by controlling for the effects of confounding variables (extraneous variables)
that might contaminate their findings. These variables can be controlled through the
environment, experimental manipulations, experimental design, or assignment of
subjects. This section concentrates on techniques used to ensure that the groups in an
experiment are comparable before the experimental treatment is administered. Group
comparability is an important consideration because it helps to rule out possible
alternative explanations based on the natural differences among people.

Perhaps an example will illustrate this point. Suppose a researcher wants to determine
whether different versions of a TV commercial’s musical soundtrack have different
impacts on what is remembered from that ad. The researcher uses a media research
class as the sample and assigns an ad with a rap soundtrack to the students in the first
three rows. Students in the last three rows view the same ad but hear a heavy metal
soundtrack. Both groups are then given a recall test, and the results show that the
group that heard the rap soundtrack remembered more information from the ad. How
confident should the researcher be of this conclusion? Not too confident: it is entirely
possible that the people who sat in the first three rows of the class are different from
those who sat in the back. The “fronters” might have chosen the front seats because
they are more intelligent, more attentive, and more alert than the “backers,” who might
have sat in the rear of the room so that they could sleep, talk, or otherwise amuse
themselves. Consequently, the superior performance of the “fronters” might be due to
these factors rather than to the effectiveness of the soundtrack. How can researchers
ensure that their groups are equivalent? There are three main techniques: randomizing,
matching, and including the confounding variable in the design.
Randomization
A powerful technique for eliminating the influence of extraneous variables is
randomization: randomly assigning subjects to various treatment groups. Random
assignment means that each subject has an equal chance of being assigned to each
treatment group. This method works because the variables to be controlled are
distributed in approximately the same way in all groups. In the preceding example,
suppose that the researcher had randomly
assigned students to one group or the other instead of assigning them according to
where they sat. It is highly probable that the average level of intelligence in the two
groups would have been the same, as would have been their
levels of attentiveness and alertness, thus ruling out those variables as alternative
explanations. In addition, randomization would equalize some other confounding factors
that the researcher might have overlooked. Random assignment would avoid the
presence of a disproportionate number of men or women in one or the other group,
which skews the results. The same might be said for geographic background:
Randomization would provide for proportionate numbers of urban versus rural residents
in each group.
Matching
Another way to control for the impact of confounding variables is to match subjects on
characteristics that may relate to the dependent variable. There are two primary
methods of matching. The first, matching by constancy, makes a variable uniform for all
of the experimental groups. For example, let’s say a researcher is interested in assessing
the impact of playing two types of video games on aggressiveness in children. Past
research strongly suggests that gender is related to the levels and types of aggressive
acts performed. To match the sample by constancy and control for gender effects, the
researcher may decide to perform the experiment using only boys or only girls in the
sample. The second type of matching involves matching by pairing. In this method,
subjects are paired on a similar value of a relevant variable before being assigned to
different groups. Using the video game example, suppose the researcher suspects that a
subject’s prior level of aggressive tendencies has an impact on how that subject is
affected by violent video games. The researcher would administer a test of
aggressiveness to all subjects and calculate their scores. For simplicity’s sake, let’s say
that there are only three possible scores on this test: low, medium, and high. The
researcher would find two people who scored high on this test, pair them up, and then
assign one at random to one treatment group and the other to the second. A similar
procedure would be followed for those scoring low and medium. When finished, the
researcher would be confident that equal numbers of high-, medium-, and low-
aggression subjects were placed in each treatment group. This process, of course, is not
necessarily restricted to pairs. If an experiment had three groups, subjects could be
matched as triplets and then randomly assigned to groups. In addition to gaining control
over confounding variables, matching subjects increases
the sensitivity of the experimental design. Because the treatment groups become more
homogeneous, smaller differences that might have been obscured by individual
variations can be detected. On the other hand, this method does have some
disadvantages. Matching by constancy limits the generalizability of the study and
restricts the size of the population available for sampling. Both forms of matching also
require at least some prior knowledge about the subjects and may require the extra
effort of a pretest.
Including the Confounding Variable in the Design
Another way to control the impact of confounding variables in an experiment and to
increase the sensitivity of the experiment is to incorporate the confounding variable(s)
into the design. For instance, let’s return to the video game example. Instead of
controlling for the effects of gender by restricting the study to only boys or to only girls,
the researcher might include gender as an independent variable. After the sample is
divided by gender, each male or female would be randomly assigned to a condition. The
resulting design would have four groups: males who play video game A, males who play
video game B, females who play video game A, and females who play video game
B. (Note that this is an example of the factorial design described later in this chapter.)
An added benefit of this design is that it can provide information about the
interaction—the combined effects of the confounding variable and independent variable
of interest. Again, there are disadvantages to this method. Including another factor in
the design increases the number of subjects needed for the experiment and also
increases the time and energy necessary to complete it.