Introduction to Experiments
Elham, Namrata, Eveline & Sujith
Theory of Science and Research Methodology | Seminar Assignment 1
1
Today's scope
1) Introduction to experiments (lecture)
2) Guest speaker: Myriam Aries
3) Activity
2
What is an experiment?
An experiment tests cause and effect.
Researchers manipulate one or more variables while controlling
others to see how this affects an outcome.
→ Used to test hypotheses and draw conclusions from empirical data.
3
Research subjects vary
Can involve, e.g., human subjects, materials, or systems.
Examples:
• Testing material strength under different temperatures
• Evaluating algorithm performance with different types of data
• Measuring satisfaction with different prototype designs
4
Basic structure
Independent Dependent
Manipulation
variable variable
Study method Different materials Exam score
5
Key building blocks
Component Description Example
Study method (flashcards vs.
Independent variable What you change
reading)
Dependent variable What you measure (the outcome) Exam score
Control variables The variables you hold constant Study time, lighting
The participants receiving the
Experimental group(s) Flashcard group
treatment
Reading group (if comparing the
two study methods)
Baseline that we're measuring
Control group
against
No-study group (if testing
effectiveness of the methods)
6
Design types
Between-subjects design
• Different participants in each condition
• One group tests prototype A, one group tests prototype B
Within-subjects design
• Same participants in each condition
• One group tests both prototype A and prototype B
Mixed design
• Combination of both
• Each group tests a different prototype (between), but all participants
complete multiple task types (within)
7
1) Between-subjects design 2) Within-subjects design
Prototype A Prototype B Prototype A Prototype B
3) Mixed design
Prototype A Prototype B
Task A Task B Task A Task B
8
Better for within or between?
Be guided by your research question.
RQ1: How does light affect people in the morning vs. evening?
RQ2: Does expertise affect satisfaction with the same interface?
9
We can't control everything
• Individual differences (knowledge, mood, skills)
• Environmental factors (noise, light, time of day)
• Psychological effects (expected behavior or responses)
• Researcher influence (unintentional reactions)
10
When (not) to choose an experiment
Suitable if you want to...
Test a cause-and-effect relationship
Manipulate one or more variables
Compare conditions systematically
Avoid if...
You can't control key variables
Manipulation is unethical or not practical
11
Types of experiments
▪ Laboratory experiments
▪ Field experiments
▪ Natural experiments
▪ Quasi experiments
12
Laboratory experiments
• Conducted in a highly controlled environment where the researcher
manipulates the independent variable (IV) to observe its effect on the
dependent variable (DV).
Advantages: Disadvantages:
High internal validity Lack ecological validity
High reliability Demand characteristics can bias results
Why Laboratory experiments? Ideal when the goal is to establish causal
relationships and achieve high control and precision in measurement.
Example: Testing the effect of laser power on the hardness of AM titanium alloy
under controlled lab conditions
13
Field experiments
• Experiments are carried out in a natural setting, in which the
researcher manipulates the independent variable to see the effects of it
on the dependent variable.
Advantages: Disadvantages:
Higher realism Less control over external variables
Higher ecological validity Harder to replicate
Why Field experiments? Some issues are too big and too complex to study
in a lab or on paper.
Example: Testing memory performance under dynamic lighting that follows
circadian rhythms in hospitals.
14
Natural experiments
• In natural experiments, the researcher takes advantage of a naturally
occurring independent variable.
Advantages: Disadvantages:
Very high ecological validity No control over independent variables
Research on sensitive topics Rare with low replicability
No random allocation
Why natural experiments? Some variables can’t be ethically or
artificially manipulated. It's a good opportunity for policy evaluation.
Example: Studying the effects of melting glaciers on global warming or Evaluating
how a certain traffic law changes the number of accidents.
15
Quasi experiments
• Quasi experiments have an independent variable that is based on an
existing difference between people (ex: mental health)
Advantages: Disadvantages:
If in lab setting, good control No control over independent
. over external variables . variables
Moderate replication No random allocation
Why quasi experiments? In many real-world contexts, random assignment is
not possible.
Example: Memory performance between people with and without major
depressive disorder.
16
When to use each type of experiments?
If you need... Use... Because...
• Maximum control Laboratory experiment You have high control over
• Clear causality variables and can randomize
participants
• Realistic behavior Field experiments You conduct a somewhat
Still have some control controlled experiment in a real-
world environment.
• To study pre-existing Quasi experiments You can’t randomly assign but
groups ethically can still compare conditions
• To study real events or Natural experiments The independent variable
policies you can’t control changes naturally, giving you
ethical real-world data.
17
Ethics
• Ensure consent from participants.
• Avoid harm or discomfort.
• Maintain confidentiality and anonymity.
• Ensure honesty and transparency in data collection and reporting.
• Seek ethical approval.
18
Sample Size
• Sample size refers to the number of observations or participants included in a
study.
• It affects the accuracy, reliability, and generalizability of research findings.
• Too small a sample → results may be unreliable or biased.
• Too large a sample → may be costly and unnecessary.
• Factors influencing sample size:
o Desired confidence level and margin of error
o Expected effect size
o Population variability
o Study design and analysis method
19
Validity
• How well a study measures what it intends Types of Validity
to measure.
• Ensures results are accurate, credible and
applicable.
20
Internal Validity
• Indicates how confidently we can say that one variable caused the
change in another.
• Focuses on the accuracy of cause-and-effect relationships within the
study.
• High internal validity = well-controlled study with minimal bias.
• Did one variable actually cause the observed effect?
21
External Validity
• Refers to how well the study’s results can be generalized beyond the
study.
• Types of external validity:
oStatistical generalisation: from sample → population
oTheoretical generalisation: from study results → theory
oEcological generalisation: from study → real-life contexts
• High external validity = findings are applicable beyond the research
context.
• Can results apply elsewhere?
22
Content Validity
• Assesses whether a test or measurement covers all aspects of the
concept being studied.
• Ensures complete representation of the topic area.
• High content validity = the measure fully captures the concept’s
domain.
• Does the measure include all relevant aspects?
23
Construct Validity
• Examines whether a test or instrument truly measures the theoretical
concept (construct) it claims to measure.
• Connects the measurement to the underlying theory.
• High construct validity = strong link between concept and
measurement.
• Does it truly measure the intended concept?
24
Criterion Validity
• Evaluates how well a measure correlates with an existing standard or
predicts a relevant outcome.
• Focuses on practical performance and predictive accuracy.
• High criterion validity = strong agreement with established or future
results.
• Does it correlate with or predict a valid standard?
25
Data Collection
&
Data Management
26
Introduction
What is Data ?
To obtain information
Data are individual pieces of
factual information recorded and Purpose of To keep on record
used for the purpose of analysis. Data
Collection To make decision
To pass information
What is Data collection?
Data collection is the process of
gathering and analyzing Mainly there are two types of Data collection :
information from various sources Primary Secondary
to address research questions. Data Data
27
Methods of Data Collection
Primary Data Secondary Data
Primary data v/s
▪ Real time ▪ Past data
Secondary data
▪ Sure about the sources ▪ Not sure about sources
▪ Can avoid bias ▪ Refining the research problem
▪ Can answer research questions ▪ Bias can’t be ruled out
▪ Cost and time ▪ Cheaper and no time limitation
▪ Flexible ▪ Less flexible
Table: Considerations in relation to the choice of suitable techniques
Säfsten, K. & Gustavsson, M. (2024). Research methodology 2.0 : for engineers and other problem-solvers 28
Methods of Data Collection
Primary Data can be collected through Secondary Data can be collected through
Observation Interviews Questionnaires Workshop Documents
▪Direct ▪ Personal ▪ Mail
▪Indirect - Structured ▪ Online
▪Structured - Unstructured
▪Unstructured - Semi-structured
▪Participant ▪ Telephonic
▪Non-
participant
Säfsten, K. & Gustavsson, M. (2024). Research methodology 2.0 : for engineers and other problem-solvers 29
(Research) Data Management
Why is it important? Benefits of Data management
▪ FAIR
▪ Increases the impact of your
▪ CARE
research
▪ Avoiding duplication of work
How to manage data ? ▪ Making it easier to share
▪ Ensuring research integrity and
▪ Document the data validation of results
▪ Organize the data ▪ Ensures accountability
▪ Data protection
▪ Preservation and data
accessibility
[Link] 30
Quantitative data
• Measurements • Structured surveys
oSensors and instrument readings oNumerical ratings
oSoftware logs and Error rates oRanking tasks
oPerformance tasks oFrequencies or counts
oTest duration
• Structured observations
oFrequencies or counts
oDuration measurements
oBehavioral checklists
31
Quantitative data
Why quantitative data?
• To measure and compare objectively
• To identify patterns, correlations, and trends
• To summarize large amounts of data efficiently
Quantitative data analysis: What have you collected:
• Descriptive statistical analysis • Numerical variables:
o Interval No true zero
• Regression test o Ratio Start from true zero
• Comparison test • Categorial variables:
• Correlation test o Nominal ex: blood type
o Ordinal ex: Pain severity scale
32
Descriptive statistical analysis
Describes and summarizes data
• Measures of central tendency: Mean, Median, Mode
• Measures of dispersion: Range, Variance, Standard
deviation
When to use:
• At the Start of data analysis
• When you want to summarize experimental results
33
Regression Analysis
• Regression is a method for examining and modelling relationships
between variables.
Simple linear regression Multiple linear regression Logistic linear regression
IV IV
IV DV
IV DV IV DV
Metric
Metric Nominal
IV IV
When to use:
• When you want to test relationships between continuous variables
• When you want to make predictions
34
Comparison Analysis
• Comparison Analysis look for differences among group means.
• Two groups: Paired T-test, independent T-test, Welch's t test,…
• More groups: ANOVA, MANOVA, Friedman Test,…
o Post-hoc tests:
▪ Tukey’s HSD
▪ Dunnett's test
When to use:
• Comparing differences between test and control group
• When comparing the same people under different conditions
35
Correlation Analysis
• Correlation analysis measures the strength and direction of the
relationship between two continuous or ordinal variables.
• Pearson r correlation, Kendall rank correlation, Spearman rank
correlation,...
When to use:
• When you want to explore relationships between variables
• When you want to identify which variables are related
36
Statistical Analysis Cheat Sheet
37
Qualitative data
• Open-ended survey questions
• Post-experiment interviews
• Think-aloud recordings
• Observation notes
Why should we use it?
• To understand why or how something occurs
• To capture experiences, beliefs, and meaning
• As a complement to quantitative data in mixed methods
38
Thematic analysis
Data Codes Themes Interpretation
▪ Familiarize yourself with the data (read, re-read, take notes)
▪ Generate codes (words/phrases describing the data)
▪ Identify themes (codes turned into broader categories)
▪ Review and refine (iterative process)
▪ Name and describe themes
39
Content analysis
• Similar to thematic analysis
• Allows us to quantify (e.g., looking at frequencies)
• Usually has main categories and subcategories
• A subcategory can't belong to two main categories
• A main category need at least two subcategories
40
Approaches to qualitative analysis
Inductive
Data Codes Theory
• Themes or categories emerge from the data
• Exploring interview data without pre-defined categories
41
Approaches to qualitative analysis
Deductive
Theory Data Codes
• Guided by existing theory
• Coding user comments based on known usability heuristics
42
Manifest vs. latent
Manifest
• Focuses on explicit content (what is literally said or done)
• Suitable for survey data
Latent
• Focuses on underlying meaning (what is implied)
• Suitable for rich interviews
43
44
Challenges
Subjectivity
• Interpretation depending on the researcher's perspective
Data volume and complexity
• Time-consuming to transcribe and analyze
Losing track
• Easy to drift away from the RQs
Generalization
• Context-specific, difficult to generalize
45
Thank you
for listening!
Questions?
References
1. Säfsten, K. & Gustavsson, M. (2024). Research methodology 2.0 : for engineers and other problem-solvers
2. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.
3. Calisi, R. M., & Bentley, G. E. (2009). Lab and field experiments: Are they the same animal? Hormones and Behavior, 56(1), 1–10.
4. Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of advanced nursing, 62(1), 107-115.
5. Field, A. (2024). Discovering statistics using IBM SPSS statistics. Sage publications limited.
6. Kampenes, V. B., Dybå, T., Hannay, J. E., & Sjøberg, D. I. K. (2009). A systematic review of quasi-experiments in software engineering. Information and Software
Technology, 51(1), 71–82.
7. Xia, T., Zhao, F., & Nianogo, R. A. (2022). Interventions in hypertension: Systematic review and meta-analysis of natural and quasi-experiments. Clinical
Hypertension, 28(13).
8. Diamond, J. (1986). Laboratory experiments, field experiments, and natural experiments. Community ecology, 3-22.
9. Lee, T. (2024, May 28). Field experiments, explained. University of Chicago News. [Link]
10. Network of the National Library of Medicine. (2022, September 6). Quantitative data.
11. Bevans, R. (2023, June 22). Choosing the right statistical test: Types & examples.
12. Ott, R. L., & Longnecker, M. (2010). An introduction to statistical methods and data analysis. Cengage Learning Inc.
13. [Link]
47