Qualities of research:
1. Theoretical- questions based on theories of how the world works
2. Empirical- based on our observations
3. Nomothetic- the search for generalizable principles that are true of all
individuals in a group or population vs. idiographic-study of an
individual
4. Probabilistic- inferences are base on probabilities
Deductive reasoning: general-specific
Inductive reasoning: specific-general
Key aspects of variables:
- An attribute that describes a person, place, thing, or idea
- The value of a variable can vary
- Can by quantitative or qualitative
- Exhaustive and mutually exclusive
Cause
- Inus condition: an insufficient but non-redundant (adds unique
prediction) part of an unnecessary, but sufficient condition
o Example: high fat diet inus condition for a heart attack
Effect
- Experiment: manipulate and then measure what happens (DV)
- Counterfactual: what would the outcome be if we didn’t do the
manipulation
- Effect: difference between what happened and what would have
happened
Cause-effect relationship:
- Temporal precedence
- Covariation of cause and effect
- No plausible alternative explanations (counterfactuals)
o Consider control groups or multiple groups within experiment
- Often bidirectional
Causal relationship
- Temporal precedence
- Cause is related to effect
- Decreased likelihood that other causes explain effect
- Mediators: what about the mechanisms that explain the relationship
between a cause and effect
- Moderators: understand situations in which the cause does or doesn’t
produce the effect
Research fallacies
- Ecological fallacy: make a conclusion about individuals based on
analyses of group data
- Exception fallacy: make a conclusion based on an exceptional case
Justice validity checks:
1. Generalizability test- would the observed effect hold in under-enrolled
groups?
2. Measurement test- do tools perform equally well across diverse
populations?
3. Mechanism test- are there genetic or biological differences missed by
excluding key populations?
Internal validity:
- Approximate truth about inferences regarding cause-effect or causal
relationships
- Have evidence that what you did in the study caused what you
observed
- Key question is whether observed changes can be attributed to your
program/intervention and not to other possible causes
Threats to internal validity: All possible causes explaining a causal
relationship between IV and DV; could occur without manipulation
1. Ambiguous temporal precedence: typically an issue with correlational
studies, also with quasi-experimental designs
2. Selection: systematic, non-random differences across participant
groups that could cause the effect
3. History: events that occur during the study that could cause the effect
4. Maturation: naturally occurring changes could be confused with a
treatment effect
5. Regression to the mean: a statistical phenomenon where, following an
extreme measurement or event, subsequent measurements tend to be
closer to the average
6. Attrition: loss of participants over course of study
7. Testing: completion of a measure can affect subsequent completions of
that measure, which is mistaken for a treatment effect
8. Instrumentation: an aspect of a measure may change over time or over
conditions in a way that is mistaken for a treatment effect
9. Additive and interactive effects:
a. selection-history threat
b. selection-maturation: may be episodic or naturally improves
c. selection-testing: experimental group can have extra pretest
d. selection-instrumentation: change only happening in one group
e. selection-attrition:
f. selection-regression: chronic pain
10. Social threats to validity:
a. Compensatory rivalry: I’ll show them!
b. Resentful demoralization: FU threat
c. Compensatory equalization of treatments: research team
contaminates conditions by providing parts of treatment to
comparison condition
d. Experimenter expectancies: researcher biases the study
influencing outcome
11. Design type:
a. Experimental designs: random assignment ideal for minimizing
threats to internal validity
b. Quasi experimental designs: group differences likely
Construct validity:
- Making inferences from the particular study variables to the theoretical
constructs they represent
- Clear description of person, setting, treatment and outcome
constructs, select variables that match constructs of interest, examine
comparability between constructs and variables, revise construct
description if necessary
- Challenges:
o Constructs typically consist of multiple components
o No perfect agreement between the variable and the theoretical
construct
o Measurement of variables that represent constructs of interest
Translation validity:
- Face validity: does the measure assess the construct
- Content validity: what constitutes assessment
Criterion-related validity:
1. Predictive validity-
2. Concurrent validity
3. Convergent validity
4. Discriminant validity
Reliability: quality of measurement
1. Inter-rater reliability- consistency of estimates across raters
2. Test-retest reliability- consistency between administrations at different
time points
3. Parallel-forms reliability- create two parallel forms of the measure and
examine consistency between the forms
4. Internal consistency reliability- estimate the consistency of the results
within the measure’s items
Threats to construct validity:
1. Inadequate preoperational explication of constructs
2. Confounding constructs and levels of constructs
3. Mono-operation bias- use of only one treatment, in one place, at one
timepoint
4. Mono-method bias- using only one measure to assess your outcome
variable
5. Interaction of different treatments- the treatment condition involves
other treatments, not just the intended treatment
6. Interaction of testing and treatment- testing makes participants more
open and responsive to treatment
7. Restricted generalizability across constructs- fail to consider or
measure the potential negative influences of a treatment
8. Poor reliability and measurement error
9. Measurement non-variance
Social threats to construct validity:
1. Hypothesis guessing: the participants guess of what the study is about
influences their behavior our outcome variable
2. Evaluation apprehension: personal characteristics relevant to
evaluation influence responses on an outcome variable
External Validity:
- Inferences about the extent to which a causal relationship holds over
variations in person, settings, treatments and outcomes
Strategies to enhance external validity:
- Demonstrate IV-DV effect with different types of treatments
- Demonstrate IV-DV effect with different outcome variables
- Demonstrate IV-DV effect with different subsamples of participants
- Demonstrate IV-DV effect with different settings
- Secondary data analysis
- Combine data with another researchers
- Meta analyses
Threats to external validity:
- Interaction of the causal relationship with units:
o An effect found with certain kinds of participants might not hold
up if other kinds of participants had been examined
- Interaction of the causal relationship over treatment variations:
o A variation of the treatment is used
o When a treatment is combined with another treatment
o When only part of the treatment is used
- Interaction of the causal relationship with outcomes:
o An effect found with a certain kind of outcome might not hold up
if other kinds of outcomes were examined
- Interaction of the causal relationship with settings:
o An effect found in one setting might not hold up if other settings
had been examined
- Context dependent mediation:
o Units, treatments, settings, outcome
Checklist for external validity:
- Settings eligible, participating, declining and excluded
- Description of usual care.
- Practitioners’ expertise and recruitment.
- Run-in period and exclusion by adverse events.
- Proper implementation and background for intervention.
- Allowance/prohibition of co-interventions used in clinical practice.
- Baseline differences to target settings.
- Participants eligible, enrolling, declining and excluded
- Baseline characteristics and risk, including co-morbidity for both
intervention and control group.
- Geographical aspects.
- Relevant and patient-relevant outcomes
- Baseline differences to target population, including prevalence &
incidence.
- Treatments delivered as intended (percentage)
- Monitoring of patients.
- Treatments completed and dropouts.
- Change or relapse by condition and follow-up/dropout
- Settings continuing, modifying or discontinuing after research project
- Maintenance and long-term effect
- Reporting adverse events.
- Potential barriers and limitations in implementing.
- External validity beyond eligibility criteria.
- Administrative policy.
- Regulatory status.
- Clear description of treatment alternatives, including the intervention
studied.
- Conflict of interests.
- Literature study carried out beforehand.
- Data sources and management.
- Valid outcome measure/presentation of effect size.
- Valid choice of statistics.
- Internally valid.
- Availability of necessary technologies.
- Sufficient sample size (subgroups).
- Absolute risk of a poor outcome in the control group.
- Adequacy of non-trial treatment – both intended and actual.
- Effect of intervention on most relevant components of compo-site
outcomes.
- Who measured follow-up.
- Frequency of follow-up.
- Adequacy of the length of follow-up
Samples
- Theoretical population: who do you want to generalize to
- Study population: what population can you get access to
- Sampling frame: how can you get access to them
- Sample: who is in study
Random sampling/probability sampling:
- Ideal b/c it simplifies external validity inferences
- Uses some form of random selection
- Random sampling and random assignment improves confidence in
generalizability and causal inference
Types of probability sampling:
- Simple: use a random numbers table or random number generator or
comparable procedure to select sample
- Stratified random sampling: divide population into subgroups and then
take a simple random sample in each subgroup
- Cluster random sampling:
- Multi stage sampling:
Nonprobability sampling:
- Not based on the rationale of probability theory: harder to know how
well sample represents the population
- Accidental, haphazard or convenience, purposive sampling
Purposive sampling:
- Modal instance sampling
o Most typical
o Case studies, smaller studies
- Expert sampling
o Often for qualitative research
- Quota sampling
o For each group
- Heterogeneity sampling
o As different as possible
- Snowball sampling
o Specific sample ask them to refer to similar people in
network/community
- Respondent-driven sampling
Efficacy:
Effectiveness:
Implementation research:
- Systematic study of methods that support the application of research
findings and other evidence-based knowledge into policy and practice
- Evaluates how various interventions or approaches are adopted and
applied in real world settings
- RE-AIM: Reach, effectiveness, adoption, implementation, maintenance
- EPIS: Exploration, preparation, implementation, sustainment
- CFIR: Consolidated framework for implementation research
Statistical conclusion validity:
Threats to statistical conclusion validity:
- Two kinds of errors about relationships
1. Conclude there is no relationship when in fact there is
a. Type I error
2. Conclude there is a relationship when in fact there is not
a. Type II error
- Low reliability of measures
- Poor reliability of treatment implementation
- Random irrelevancies in the setting
- Heterogeneity of respondents
- Low statistical power
- Fishing and the error rate problem
o Finding relationship by chance, when there is none
- Restricted range: ceiling and floor effects
- Violated assumptions of statistical tests
- Inaccurate effect size estimation
Reduce threats to statistical conclusion validity:
- Good reliability of measures
- Good treatment of implementation
- Data cleaning/preparation
- Examine data for relevant assumptions
- Adequate statistical power
- Adjust for multiple testing
How to know if have adequate power:
- Sample size
- Effect size
- Alpha level (usually 0.05)
- Power (usually .80)
o 1-B (reject H0 when H0 is false)
Steps to determine sample size:
- Identify design
o Experimental design
- Identify effect size
- Identify analysis
- Use G-power to determine sample size
Considerations that affect sample size and power calculations:
- Attrition
o Power analysis for longitudinal studies should always account for
attrition
- Clustering
o Less power when observations are not independent because
each contributes less unique information
- Cross-over/treatment contamination
o Less power because the effect size gets smaller
- Interim data “looks”
o Risk of false positives, must be preplanned, readjust power
analysis
- Increase power:
o Increase sample size, increase strength of treatment, improve
measurement, recruit homogeneous participants, equal sample
sizes in each condition, within subjects design, use covariates in
analysis
o Just because high power and can detect significant associations,
does NOT mean the associations are clinically meaningful
- Correcting for multiple testing:
o When conducting multiple analyses, should adjust the error rate
(significance level) to reflect the number of analyses doing
Stricter p-value cut off
Bonferroni correction (very strict)
False detection rate correction
Randomized experiments:
- Manipulations are assigned to experimental units by chance
- Creates 2 or more groups that are similar on average (probably)
o Differences between groups can be attributed to manipulation,
not group differences
Nuisance variables:
Simpsons paradox:
Experimental designs:
- No pre-test
o
- Pre and post test
o
- Longitudinal designs
- Solomon four-group design
- Switching replications design
Control group options:
- No treatment
- Waitlist control
- Attention control (sham/placebo treatment, psychoeducation)
- Usual care/standard of care
- Enhanced usual care
Other comparision:
- Comparison/alternative treatment
o High power is required
- Multiple conditions
- Factorial designs
Fractional factorial designs:
- Only a subset of conditions examined
- 5 or more factors
- Overhead costs associated with new experimental conditions are
relatively high
- Primarily interested in main effects, possible a few selected lower-order
interactions
- Most of the remaining effects are expected to be negligible in size
- Multiphase optimization strategy
Cross over design:
- Randomize the order that participants receive treatment
o May need a wash out period
Moderation:
Mediation:
Moderated mediation:
- Effect of a fourth variable
Mediated moderation:
- Initial moderated effect is mediated pathways
Common problems with experiments:
- Recruitment
o Target population not well defined
o Ways to ensure diversity- compensation, outreach/community
events
- Randomization
o Ensuring participants understand randomization
o Independent person create randomization scheme
o Automate detailed procedures if possible
- Differences by condition on some variable
o Testing for differences
o Type I error
o Covariates
- Implementation:
o Ensure treatment is delivered as intended
o Consider how to measure each implementation
- Analysis:
o Intent to treat
o Per protocol
o As treated
- Attrition:
o Non-adherence
o Drop out
Types of randomization:
- Matching or stratifying randomization
- Block randomization
Analyzing attrition:
- MCAR: missing completely at random
o Missing not related to any variables in the dataset
- MAR: missing at random
o Missing related to some other variables in the dataset
- MNAR: missing not at random
o Missing related to scores on itself
- Analysing missing data patterns:
o Dummy code variables
o Chi square
o T-tests
Addressing attrition statistically:
- Missing data remains missing (if MCAR)
o Complete case analysis- can lower N
- Drop the variable
- Imputation
o Replace with fixed value, mean/median, data from similar
participant
o Backward/forward fill
o Machine learning
- Full information maximum likelihood estimation
Ecological momentary assessment (EMA):
- Structured self-report diary technique in which individuals provide
information based on their situational context, feelings, thoughts, and
behavioral patterns in the flow of daily life
- Surveys typically completed multiple times throughout the day for
several days or weeks
- Prospective, real-time, repeated-measures sampling
- Within individual data
Why EMA:
- Reduces retrospective assessment bias
- Ecological validity
- Captures the experiencing self
- Temporal resolution: precedence and course
- Real time monitoring and treatment
- Roots in ecological psychology- behavior can be understood when
investigated in the context in which it occurs
Sampling designs:
- Interval-contingent sampling: predetermined intervals of time
- Signal-contingent sampling: random unpredictable moments
- Event-contingent sampling: following an event of interest
- Daily diary: an interval-contingent sampling in which assessments
occur only once
- Hybrid designs: some questions asked at random intervals, others are
event-contingent
EMA and causal inference:
- Experimental designs
o Fine grain evaluation of treatment effect
o Extending treatment into everyday life- EMI
- Quasi-experimental and observational designs
o Fine grain evaluation of treatment effect
o Event based EMI’s
- Between person and within person analyses
EMA measures:
- Questionnaire length most burdensome
- Items often adapted from standard, validated questionnaires
- Measurement reactivity needs to be evaluated
- Passive measures
- Ambulatory assessments
EMA challenges and responsibilities:
- Risk monitoring: challenges of monitoring in real time and knowing
when to intervene
o Safety protocol
- Informed consent: explicate all data being collecting
- Inclusivity
- Participant burden and appropriate compensation
- Data privacy