Research Design

 Specific methods and procedures used to answer a research

question

Research Question Types

 Exploratory, Descriptive, Relational

Research Design Categories:

 Experimental Research Design
 Quasi-Experimental Research Design
 Nonexperimental Research Design
o Each type of design is distinguished by the level of control that
is established in the design
Control
 In research design is
o The manipulation of a variable

o Holding all other variables constant

 When control is low; neither criterion is met

 When control is high; both criteria are met

Experimental Research Designs

 Use of methods and procedures to make observations in which the
researcher fully controls the conditions and experiences of
participants by applying three required elements of control:
o Randomization, manipulation and comparison/control

 Strength
o Only research design capable of demonstrating cause and
effect
 Conditions and experiences under full control of the
researcher
 Limitation
o Behavior that occurs under controlled conditions may not be
the same as behavior that occurs in a natural environment

Quasi-Experimental Research Design

  Use of methods and procedures to make observations in a study that
is structured similar to an experiment, but the conditions and
experiences of participants are not under the full control of the
researcher
o Study Lacks Random Assignment

 IE assignment was done by school, not by student

o When the factor is not manipulated (quasi-independent)

o OR when the research design lacks a comparison/control group

 Quasi-Independent Variable
o Any factor in which the levels of that factor are pre-existing

 Characteristics such as gender, ability level, ethnicity

o Variable with levels to which participants are not randomly
assigned and that differentiates the groups of conditions being
compared in a research study
 Limitation
o Researchers do not have full control to manipulate the
independent variable and therefore cannot demonstrate cause
and effect
 Quasi-independent variable

Nonexperimental Research Design

  Use of methods and procedures to make observations in which the
behavior or event is observed "as is" or without an intervention from
the researcher
o The conditions or experiences of participants are not
manipulated
 Manipulation occurs when the researcher creates the conditions in
which participants are observed
 Strength
o Can be used to make observations in settings in which the
behaviors and events being observed naturally operate
 Limitation
o Lacks the control needed to demonstrate cause and effect

 We can speculate about potential causes for the

observations we make; but we cannot know for sure
without greater control

Just because a research design does not explain cause and effect; does
not mean it is worse. Designs are well adapted and help add to an
understanding of the behaviors/events one observes since we cannot
always bring behaviors and events under full control of a researcher.

Internal Validity
 The extent to which a research design includes enough control of the
conditions and experiences of participants that it can demonstrate a
single unambiguous explanation for a manipulation (aka cause and
effect)
o The more control the higher internal validity

o Experimental designs have the greatest control therefore the

highest internal validity
o Nonexperimental research designs typically have the least
control therefore lowest internal validity

External Validity
 The extent to which observations made in a study generalize beyond
the specific manipulations or constraints in the study
o The more an observation generalizes beyond the specific
conditions or constraints; the higher the external validity
  The fewer the constraints or more natural settings within
which observations are made = higher

Experiment
 Any study that demonstrates cause
o Methods and strict procedures used in an experimental
research design to specifically control the conditions under
which observations are made to isolate cause-and-effect
relationships between variables
 Three elements of control that researchers use to draw cause-and-
effect conclusions
o Randomization

 Random sampling and random assignment

 Use of methods for selecting individuals to participate in
a study and assigning them to groups such that as each
individual has an equal chance of being selected to
participate and assigned a group
o Manipulation

 (Of) variables that operate in an experiment

o Comparison/Control

 A Control Group

Independent Variable or Factor

 The variable that is manipulated in an experiment
o The levels of the variable remain unchanged (or
"Independent") between groups in an experiment
 The presumed Cause
 Levels of a factor
o Specific conditions or groups created by manipulating the
factor

Individual Differences
 Unique characteristics of participants in a sample that can differ
from once participant to another
Confound or Confounding Variable
 A variable not accounted for in a research study that could be
causing or associated with observed changes in the independent
variable

Dependent Variable
 The variable that is believed to change in the presence of the
independent variable. The presumed "effect"

Extraneous Variable
 One that is not controlled for in the study that affects the dependent
variable

In an experiment the independent variable is manipulated to create

groups and the dependent variable is measure in each group.
Internal Validity Threats
 Factors that threaten internal validity vary systematically with the
levels of an independent variable
o Think of it as a confound

 History and Maturation

o History Effect

 An unanticipated event that co-occurs with a treatment

or manipulation in a study
o Maturation

 A participant's physiological or psychological state

changes over time during a study
 Internal events that exist within an individual NOT
external
 

 Regression and Testing Effects

o Regression towards the mean

 A change or shift in a participant's performance toward a

level/score that is closer or more typical to their true
potential or mean ability on some measure, after
previously scoring unusually high or low on the same
measure
 Usually occurs when participants are from the
bottom or top percentile
o Testing Effect

 Improved performance on a test or measure the second

time it is taken due to the experience of taking the test
 Emphasis on improving performance; whereas
regression can be increase or decrease
 o

 Instrumentation and Measurement

o Instrumentation

 The measurement of the dependent variable changes

due to low reliability of the instrument
 Dependent variable changes due to an error during
the course of a research study
 Can be problematic when it is inherently prone to error;
such as low reliability
 Also be due to inadequate training to use the measure
 o

 Attrition or Experimental Mortality

o Attrition

 Participant does not show up for a study at a scheduled

time or fails to complete the study
o Heterogenous Attrition

 Rates of attrition are different between groups in a study

o Homogeneous Attrition

 Rates of attrition are the same in groups

 It can threaten the external validity
 o

 Environmental Factors
o Characteristics or dynamics of the study itself and the actions
of the researcher that can pose a threat
 Only poses a threat when it varies systemically with the
levels of an independent variable
 Treatment Factors
o Includes

 Treatment Replications
 Delivering the treatment for a long enough period
of time after which the effect is expected to occur
 Diffusion of Treatment
 Treatment inadvertently gets delivered to the
control group
 Two ways
 Person delivers treatment in the view of
all participants, both treatment and
control
 Participants of different conditions talk
to each other about the study
 Participant Effects
  Participants alter their behavior as a result of
simply being selected to participate
 Social Desirability
 Participants change their behavior to put
themselves in the best light
 Respond in not true ways, but socially
desirable ones
 More likely for less socially desirable
topics
 Hawthorne Effect
 Participants are selected as part of the
treatment group and change their behavior
because they believe they are getting special
treatment
 John Henry Effect (Compensatory Rivalry Effect)
 Participant who is assigned to the control
group who tries harder because they were
not selected to receive the special treatment
 Resentful Demoralization
 Participant assigned to the control group feel
demoralized or resentful because they were
not selected for the treatment group
 They were less motivated

Threats to External Validity

 Limit the extent to which observations made by a researcher
generalize beyond the constraints of the study
o Factor that is held constant becomes the constraint to which
observations are limited
 Population Validity
 o Extent to which results observed will generalize to the target
population from which a sample was selected
 Homogeneous attrition
 Same number of participants from different
conditions of a study does not show up at a
scheduled time or fails to complete a study
 For instance, those who drop out may be
different from those who participate.
Therefore, the results would only be limited
to those who show up to participate and may
not generalize to those who don't.
o Key concern

 An effect observed in study will only occur in that study

 Sampling methods and participant characteristics
 Ecological Validity
o Extent to which results observed will generalize across
settings or environments
 Natural setting has high ecological validity because
conducted in the same setting in which the
behavior/event being measured would normally operate
o Generally high as long as observations are not dependent on
specific features of the research settings itself
 Temporal Validity
o Extent to which results observed will generalize across time
and at different points in time
 How our explanations of behaviors change over time
o Key concern

 Timing of measurements
 Changes in our understanding
 Outcome Validity
o Extent to which results observed in a study will generalize
across different but related dependent variables
 High outcome validity means generalize a result beyond
the specific measures used in a study
 Treatment Validity
 o Extent to which treatment can be generalized; whether it can
be implemented as it is conceptualized to other individuals
 Thorough description of different components of the
treatment, who delivered it, and dosage
 Interventions that are difficult to implement/extend
over a long period of time may be more difficult to
generalize


Mundane Realism
 Extent to which a research setting physically resembles or looks like
the natural or real-world environment being simulated
o If setting looks real to participants, high mundane realism

Experimental Realism
 Extent to which an experimental situations feels real
o The psychological aspects of a research setting are meaningful
or feel real to participants

Increasing the mundane and experimental realism of a study will increase

the external validity of the study