Web vs.
Lab
John H. Krantz
Hanover College
Approach
Not focus on technology
Focus on the research question
What is best way to answer question
Approach
Web is a tool
to use in determining best way to ask question
Much as field experiments have become a
tool:
e.g. Milgram, S. , Bickman, L., & Berkowitz, L.
(1969). Note on the drawing power of crowds of
different sizes. Journal of Personality and Social
Psychology, 13(2), 79-82.
While this example is trivia it shows an example of
a study not possible in the laboratory
Approach
What this implies:
For some questions web is a superior method to
lab
May depend on your situation as we all use what
we have available
Characteristics of Web vs.
Lab
Web Lab
Distributed Confined
User or remotely Experimenter controlled
controlled environment environment or at least
and location and some determinable
factors are unknowable Accidental sampling
Accidental sampling of generally but random
large population sampling of restricted
Potentially anonymous population is possible
Requires computer
Experimenter is known to
be present or near
May not have computer
Characteristics of Web vs.
Lab
Similarities
A certain act of belief is needed to accept self-
report responses
Cannot control state and previous state or events
of participant
This is an issue in control
Random assignment is possible
Reasons for Use – Musch & Reips
(2000)
In descending order (7 very important – 1 not
important)
Large N 5.5
High statistical power 4.5
Speed data collection/participants
from other countries 3.6
External validity 3.4
Low cost 3.2
Reasons for Use
Possible responses not possible on paper
Dynamic and interactive (but any computer would
do)
Access to special populations
More precise replication
Reduction of experimenter effects
Comparability to local sample
New set of variables to study
Reasons for Concern
Ethical issues
Informed consent/debriefing
Data security
Data validity
Subject falsification/collaboration
Loss of experimental control
Equipment/environment Variation
Reasons for Concern
Computers are necessary
Lack of ability to clarify instructions and
informed consent
New class of variables that may impact
responses
Does this last one sound familiar?
Importance of Problems
Faced
Musch & Reips (2000)
In descending order
Lack of control of behavior 3.6
Lack of control of motivation 3.4
Inability to ask questions 3.3
No control over hardware 2.9
Nonrepresentative sample 2.8
Ethical problems 1.5
Issues regarding the sample
Web is stated to be not representative
Questions
How many labs have representative sample?
Do we know what person variables are significant
or do we control out of the assumption of a
possible effect?
The greater diversity of samples (Krantz & Dalal,
2000 and “College Sophomores”) may allow this
to become a research question instead of
assumption
The Web Study In Context
Relevant Dimensions
Methodological Issues
Subject-Related Issues
Ethical Issues
A Decision Guide
Relevant Dimensions –
Method
Statistical Power
Refers to noisy data
Not in all cases will noise be better on Web
Ruppertsberg et al (2000).
Web, n = 151, s = 12.5, SE = 1.02
Control, n = 14, s = 7.4, SE = 1.9
Effect Size or Robustness of the Effect
Basically is it easy to obtain the result
Relevant Dimensions –
Method
Sample Bias
Is the effect sensitive to the characteristics of the
sample?
External Validity
Does this effect or relationship happen outside of
laboratory
Does this effect or relationship happen away from
a computer
Relevant Dimensions –
Method
Need for Control
Stimulus
Problematic Dimensions
Color
Luminance
Temporal
If can keep on one page
Relative spatial dimensions are fine
Example Study: Column
Taper
Description:
Doric columns appear to lean when vertical
Apparently due to taper of column width
Method
Three columns. Adjust so out two same position
as inner (vertical or horizontal)
Validation
Results
Manipulation Validation
6
y = 0.99x
5
4
Measured Angle
0
0 1 2 3 4 5
Target Angle
Vertical Results 250
Tilt at Perceived Vertical (arcmin)
200
150
y = 386.88x - 3.36
r2 = 0.87
100
Web Vert
50
Lab Vert
0 Linear (Web Vert)
-0.4 -0.2 0 0.2 0.4 Linear (Lab Vert)
-50
-100 y = 325.75x - 11.72
r2 = 0.99
-150
-200
Log Column Proportion
Relevant Dimensions –
Method
Need for Control
Measurement
Need will be sensitive to type of measurement
Think of all question order effects
Now type of response objects
Environment
Effect of Web Known
Has the method been examined on Web
Time for data collection
Response time for results
Relevant Dimensions –
Subject
Special Populations
Ethnic groups
Age
Education
Other targeted populations (Buchanan, 2000)
Self-Monitoring: found groups predicted high and low on
scale to validate response
Relevant Dimensions –
Subject
Interaction with Subjects
Need for Clarification
As part of research design
Demand Characteristics
Web removes us, but our design might still
provide some hints
Relevant Dimensions – Ethics
Sensitivity of Ethical Issues
Deception
Potential for harm
Effects of Fraud by Participants on Data
Faking responses
Participant Collaboration
A Decision Guide - Method
Sensitivity of Study to/Need:
Low High
Statistical Power Either Web
Effect Size Depends1 Either
Sample bias Either Usually Lab2
Ecological Either Depends3
Validity
Decision Guide – Method
Low High
Control Either Lab
Known Valid on Lab at least as Web
Web check
Time for Data Web Either
Decision Guide – Subject
Low High
Special Either Web, in most
Populations cases
Subjects from Either Web
other countries
Interaction with Either Lab
subjects
Demand Either Web
Possibility
Decision Guide – Ethics
Low High
Sensitivity to Either Lab
Ethics
Effects of Fraud Either Lab
Factors not Included in Decision
Guide
Did not include ease or cost
Also not important factor in early researchers,
Musch & Reips, 2000
These factors should not drive design criteria.
Conclusions
Web and lab distinct environments
Each method has strengths
Match method to research question
Still useful to examine methods
Study locally as well as globally
Differences do not necessarily imply web is less
valid than lab
Differences could be meaningful
Differences could imply problem in lab envirnment