Control Problems in Research

Week 5
 Today’s question
 Last week, we understood what
experiments are (not)
 A robust experiment allows
strong causal conclusions to be
made. To this end, we control for
variables which can compromise
our conclusions!
 What are some of the “noises”
that can muddy our conclusions
and how do we control for them?
Two basic research designs

Experim
ent

Between Within-
-subjects subjects
 Between-subjects
Design
 Assuming there are two levels of
the IV, participants in Group A
are not the same participants as
those in Group B. Independent
groups are formed.
 That is participants will receive
either A or B.
 Particularly useful when working
with subject variables; when
participating in one level makes it
impossible for people to
Three types of independent groups in between-
participate in the other levels
subjects design.
 Disadvantage: differences IV Manipulated
between the conditions could be Randomly-assigned independent groups
due to the independent variables, Matched groups
but they might also be due to Subject Variables
differences between the two Non-equivalent (naturalistic) groups
Within-subjects
Design

 Each participant is
exposed to each level of
the independent variable
in a within-subjects
design. Assuming there
are two levels of the IV,
each participant receives
both levels A and B
 In within-subjects design,
multiple measurements
are taken from the same
participant.
 They eliminate equivalent
group problem, but
Design & control
problems
 Each design poses its own
(unique) problems with
controls

 Between subjects
 Creating equivalent
groups

 Within subjects
 Minimizing sequence
effects
Creating
equivalent
groups in
between-
subjects
designs
How are equivalent
independent/between-
subjects groups formed?

 Random assignment
(not the same as
random selection in
surveys that aim to
get representative
samples)

 Matched on key
variables
 Creating
equivalen  Every person volunteering for the study has an
equal chance of being placed in any groups to be
ce groups: formed

Random  The goal is to take any individual difference

assignmen factors that could bias the study and spread them
evenly throughout the different groups
t
 Ideally, both groups should be equivalent on all
traits. Although there is no guarantee of an even
spread, but the larger number of participants, the
better your chances.

 Largely, used in laboratory research

How to  Simple randomization - Coin flip,
rolling a die, or using random
randomly number table

assign  Can lead to unequal groups

participants
?
How to  Block Randomization: a procedure
ensuring that each condition of the
randomly study has a participant randomly
assign assigned to it before any condition is
repeated a second time.
participants?  Suppose an experiment has five
conditions (labelled as A, B, C, D,
and E). One “block” is made up of a
random order of all five conditions:
 Guarantees similarly-sized groups
Creating Random assignment requires large number of


subjects to even out individual differences

equivalen  Sometimes this is impossible or impractical

ce groups:  Patients with rare diseases are difficult to find,
whereas healthy controls are abundant
Matching
 Solution: Match the controls on key variables,
and then distribute participants randomly to
the different groups in the experiment.
 In this example, anxiety level will be the key
matching variable.
Matching  The variable you choose to match
should be the one that is
theoretically most important.

 It should be closely related to the

DV
Controlling for
sequence
effects in
within-subjects
or repeated-
measures
designs

[Link]

What would be the problem if Coke is always

presented first, and Pepsi is always presented
second?
 Sequence or Order Effects

 It is possible that a prior measurement will influence a later

measurement – order or sequence effect.
 The typical way to control sequence effects in a within-subjects
design is to use more than one sequence, a strategy known as
counterbalancing.
 Counterbalancing
 There are two general categories of counterbalancing, depending on whether
participants are tested in each experimental condition just one time or are
tested more than once per condition.
 Testing once per condition
 Complete counterbalancing
 Partial counterbalancing
 Testing more than once per condition
 Reverse counterbalancing
 Block randomization
 Complete Counterbalancing: Two
stimuli
Subject 1: AB
Subject 2: BA
Subject 3: AB
Subject 4: BA A B
…

[AB, BA] combination is for two stimuli.

 Complete Counterbalancing: Three stimuli

 What if you have

three levels? How
many combinations
of the conditions can
you have?

A B C
 Complete Counterbalancing: Three stimuli

Subject 1: ABC
Subject 2: BAC
Subject 3: CAB
Subject 4: BCA A B C
Subject 5: CBA
Number of combinations = C! (where C is number of
Subject 6: ACB conditions)

(C! means “three factorial” in this case)

That is, 3! = 1 × 2 × 3
>3 stimuli: Partial Counterbalancing
 3! is small, but when you have 4!, 5!, etc., the number of combinations quickly
outnumber your resources (participants)
 Partial Counterbalancing: Whenever a subset of the total number of sequences
is used, the result is called partial counterbalancing or, sometimes, ‘‘incomplete’’
counterbalancing.
 For Partial Counterbalancing, use Latin square (see textbook for description). With
a perfectly balanced Latin square, you are assured that (a) every condition of the
study occurs equally often in every sequential position, and (b) every condition
precedes and follows every other condition exactly once.
 For the first row use the rule: 1,2,N,3,N-1,4,N-2,5,N-3.6
Second row: Add 1 to each element (N+1=1)
 Reverse counterbalancing: Testing
More Than Once per Condition
 In many studies it is reasonable, even necessary, for participants to
experience each condition more than one time.
 Reverse counterbalancing: the experimenter presents the conditions
in one order, and then presents them again in the reverse order.

Stroop effect study used reverse counterbalancing (ABBA)

Block Randomization: Testing More
Than Once Per Condition
 Suppose you were conducting a study in which you wanted to see if
participants would be more affected by the illusion when it was presented
vertically than when shown horizontally or at a 45◦ angle. Four conditions
of the study are assigned to the letters A, B, C, and D. You can show
these conditions multiple times to the same participant
 A = horizontal
 B=45◦ to the left
 C = 45◦ to the right
 D = vertical
 Within each block, the order of conditions is randomized. Participants
would encounter all four conditions in a randomized order, then all four
again but in a block with a new randomized order, and so on for as many
blocks of four as needed.
 A reverse counterbalancing would look like this: A–B–C–D — D–C–B–A . A
Control
Problems
in
Developme
ntal
Research
Two kinds of research designs:
cross-sectional (between-
subjects) or longitudinal (an
ultra-long within subjects)
Control Problems: Cohort Effect

With a cross-
sectional design That can only be
(recruiting old vs. done with a
young), you longitudinal
cannot rule out study.
cohort effects.
 Summary
 Broadly, there are two kinds of basic experimental research
designs: between-subjects and within-subjects.
 Each design poses its own (unique) problems in relation
controls, and there there are different ways to control for them,
depending on the type of design.
 Between-group: creating equivalent groups (random assignment
and matching)
 Within-group: sequence effects (counterbalancing)