RESEARCH METHODOLOGY

Comprehensive Study Notes

Topics Covered
1. Hypothesis Testing: Level of Significance, Sample Size, Power & Efficiency
2. Standard Errors and Confidence Intervals for Proportions & Probabilities
3. Sampling Methods and Minimizing Errors
4. Experimental Designs: Pre-experimental, True-experimental, Quasi-experimental, Statistical
5. Levels of Measurement of Variables (Nominal, Ordinal, Interval, Ratio Scales)
6. Establishing Reliability, Validity and Norms
7. Tests of Significance: Parametric and Non-parametric Tests

Based on Kothari's Research Methodology and standard references

SECTION 1: HYPOTHESIS TESTING — CORE CONCEPTS

1.1 Definition and Nature of a Hypothesis

A hypothesis is a specific, testable statement about the expected relationship between variables
or about the characteristics of a population. It is a provisional answer to a research question,
formulated before data collection begins, and serves as the logical backbone of scientific inquiry.
Every empirical study begins with a problem that is translated into a hypothesis, which then
guides the design, methodology, and analysis of the research.
A well-framed hypothesis does not merely describe — it predicts. It specifies what the researcher
expects to observe if a particular theoretical explanation is correct. This predictive nature is what
makes the hypothesis central to hypothesis testing: we either gather enough evidence to reject or
fail to reject the stated null hypothesis.

Types of Hypotheses
• Null Hypothesis (H₀): The hypothesis of no effect, no difference, or no relationship. It
asserts that any observed differences in the sample are due to chance variation, not a real
effect. The null hypothesis is the one formally subjected to statistical testing. Example:
'There is no significant difference in blood pressure between the treatment and control
groups.'
• Alternative Hypothesis (H₁ or Hₐ): The hypothesis that contradicts H₀ — it asserts that a
real effect, difference, or relationship exists. It represents what the researcher expects to
find. Example: 'The treatment group will show a significantly greater reduction in blood
pressure than the control group.'
• Directional (One-tailed) Hypothesis: Specifies not only that a difference exists but also
the direction of the difference. Used when theory or prior evidence strongly supports a
specific direction. Example: 'Students using active learning methods will score higher than
students using passive methods.' The statistical test is concentrated in one tail of the
distribution.
• Non-directional (Two-tailed) Hypothesis: Predicts a difference or relationship but does
not specify direction. Used when the direction is theoretically uncertain. Example: 'There
will be a significant difference in scores between active and passive learning groups.' The
test is split across both tails of the distribution.

Characteristics of a Good Hypothesis

• Clearly and precisely stated — unambiguous language
• Empirically testable — can be verified or refuted through observation and measurement
• Theoretically grounded — linked to existing knowledge or theoretical frameworks
• Specific — involves specific variables, populations, and expected relationships
• Stated in simple declarative form, not as a question
• Parsimonious — as simple as possible while still being accurate
1.2 The Hypothesis Testing Process
Hypothesis testing is a formalized statistical procedure used to decide, on the basis of sample
data, whether there is enough evidence to reject the null hypothesis. The procedure follows a
structured sequence of steps:
• Step 1: Formulate hypotheses: State H₀ and H₁ clearly and ensure they are mutually
exclusive and exhaustive.
• Step 2: Set the significance level (α): Decide on the threshold probability (commonly α =
0.05) before data collection.
• Step 3: Select the appropriate test statistic: Choose based on data type, distribution,
number of groups, and research design (e.g., t-test, ANOVA, chi-square, Mann-Whitney
U).
• Step 4: Determine the critical region: Identify the critical value from the relevant
distribution table; define the rejection region.
• Step 5: Collect data and compute the test statistic: Calculate the value (e.g., t-value,
F-value, χ²) from the sample data.
• Step 6: Make a statistical decision: If the computed test statistic falls in the rejection
region (or if p < α), reject H₀. Otherwise, fail to reject H₀.
• Step 7: Draw substantive conclusions: Interpret the statistical decision within the real-
world context of the research question.

1.3 Level of Significance (α — Alpha)

The level of significance (α) is the probability of rejecting the null hypothesis when it is actually
true. It defines the threshold of risk the researcher is willing to accept for making a Type I Error
(false positive). This value is set by the researcher before conducting the study, based on the
consequences of making an incorrect decision.

Commonly Used Significance Levels

Significance Level (α) Interpretation and Context
α = 0.05 (5%) The most common threshold in social sciences, psychology,
education, and behavioral research. Means there is at most a
5% probability of falsely rejecting H₀.
α = 0.01 (1%) More stringent standard used in medical and clinical research.
Only a 1% chance of a false positive is acceptable.
α = 0.001 (0.1%) Extremely conservative — used in large-scale genomic studies
and high-stakes biomedical research.
α = 0.10 (10%) More lenient — used in pilot studies, exploratory research, or
preliminary analyses where false positives carry low risk.
Type I and Type II Errors
Two types of decision errors can occur in hypothesis testing. Understanding them is critical
because the significance level directly controls one type of error while affecting the other.

Error Type Definition Control Strategy

Type I Error (α) Rejecting H₀ when it is actually Set α at a low value (e.g., 0.01).
TRUE — a false positive. Use replication to confirm
Concluding an effect exists when findings.
it does not.
Type II Error (β) Failing to reject H₀ when it is Increase sample size. Increase
actually FALSE — a false power. Reduce measurement
negative. Missing a real effect. error.
Power (1 − β) The probability of correctly Increase N, increase α, reduce σ,
rejecting H₀ when it is false — use more sensitive designs.
correctly detecting a real effect.

NOTE: Inverse Relationship of Errors

Reducing α (being stricter about accepting results as significant) automatically increases β (the
chance of missing a real effect). Researchers must balance these two risks based on the
consequences: in clinical trials, a Type I Error (treating people with an ineffective drug) and a
Type II Error (missing an effective treatment) both carry serious consequences.

1.4 Sample Size Estimation

Sample size estimation is arguably the most important planning step in any quantitative research
study. An adequate sample size ensures that the study has sufficient statistical power to detect a
meaningful effect if one truly exists, while avoiding the waste of resources associated with an
unnecessarily large sample. Sample size calculations should be performed before data collection
begins.

Key Determinants of Sample Size

• Effect Size (δ or d): The magnitude of the difference or relationship of interest. Smaller
effect sizes require larger samples to detect. Effect size is typically quantified using
Cohen's d for mean differences, Pearson r for correlations, or Cohen's f for ANOVA.
• Level of Significance (α): A more stringent alpha (e.g., 0.01 vs. 0.05) requires a larger
sample to detect the same effect at the same power level.
• Desired Statistical Power (1 − β): Higher required power necessitates a larger sample.
The standard minimum is 80% power (β = 0.20). Clinical trials may require 90% or higher.
• Population Variability (σ): Greater variability in the population means more data is
needed to estimate parameters precisely. Usually estimated from pilot studies or literature.
• Test directionality: Two-tailed tests require slightly larger samples than one-tailed tests
to achieve the same power, because the rejection region is split.
Cohen's Effect Size Conventions
Effect Size (Cohen's d) Interpretation and Practical Meaning
Small (d = 0.20) Subtle difference, barely perceptible in real life. Requires large
sample (e.g., ~394 per group for 80% power at α=0.05).
Medium (d = 0.50) Moderate difference, noticeable to an observer. Requires
moderate sample (e.g., ~64 per group).
Large (d = 0.80) Obvious, easily observable difference. Requires smaller sample
(e.g., ~26 per group).

Sample Size Formula: Comparing Two Means

n (per group) = 2 × (Zα/2 + Zβ)² × σ² / δ²
Where: Zα/2 = z-critical value for the chosen significance level (1.96 for α=0.05); Zβ = z-critical
value for desired power (0.842 for 80% power); σ² = population variance (or estimate from pilot); δ
= minimum clinically or practically meaningful difference to detect.

Sample Size Formula: Estimating a Proportion

n = Z²α/2 × p(1 − p) / e²
Where p = estimated proportion (use 0.5 for maximum conservatism), e = desired margin of error.
For a 95% CI and 5% margin: n = (1.96)² × 0.25 / (0.05)² = 3.8416 × 0.25 / 0.0025 ≈ 384.

KEY POINT: Finite Population Correction

When the sample is more than 5–10% of the population, apply the finite population correction:
n_adjusted = n / [1 + (n − 1)/N], where N = population size. This reduces the required sample
size.

1.5 Statistical Power

Statistical power is the probability that a hypothesis test will correctly reject the null hypothesis
when the alternative hypothesis is true — the probability of detecting a real effect that actually
exists in the population. Power is defined as 1 − β, where β is the probability of a Type II Error.
A power of 0.80 (80%) is the widely accepted minimum standard in behavioral and social
sciences research. This means the test will detect a real effect 80% of the time it exists, and miss
it 20% of the time. Higher power (e.g., 0.90 or 0.95) is preferred in clinical or high-stakes
research.

Factors That Increase Statistical Power

Factor How It Increases Power
Larger sample size (N) Reduces sampling error; narrows the sampling distribution;
 Factor How It Increases Power
increases sensitivity
Larger effect size A bigger true difference is easier to detect — it stands further
from H₀
Higher α (less stringent) Relaxing from α=0.01 to α=0.05 moves the critical value closer
to H₀, increasing the rejection region (but raises Type I risk)
Reduced measurement error More precise measures reduce within-group variability;
improves signal-to-noise ratio
One-tailed test Concentrates the entire rejection region in one tail — increases
power if direction is correct
Within-subjects / repeated Controls individual differences; dramatically reduces error
measures design variance
Using covariates (ANCOVA) Statistically controls for sources of variance; increases
precision

1.6 Statistical Efficiency

Efficiency refers to the ability of a statistical test to use the available information in the data
optimally — to detect an effect with the smallest possible sample size. A more efficient test
reaches the same power with a smaller sample compared to a less efficient test.
Parametric tests (e.g., independent t-test) are more efficient than their non-parametric
counterparts (e.g., Mann-Whitney U) when the assumptions of parametric tests (especially
normality) are met. This is because parametric tests use the actual values of observations, while
non-parametric tests convert data to ranks and therefore lose some information.
• Asymptotic Relative Efficiency (ARE): A formal comparison of efficiency between two
tests as sample size approaches infinity. The Mann-Whitney U test has an ARE of
approximately 95.5% relative to the t-test under normality — meaning it requires only
slightly larger samples. Under non-normal distributions, non-parametric tests can be more
efficient than parametric alternatives.

1.7 Standard Errors

A standard error (SE) is the standard deviation of the sampling distribution of a statistic. It
quantifies how much the sample statistic (e.g., sample mean, sample proportion) is expected to
vary from sample to sample, and therefore reflects the precision of the statistic as an estimate of
the corresponding population parameter.
The standard error of the mean (SEM) is the most commonly encountered standard error. It
answers the question: if we drew many samples of size n from the population and computed the
mean each time, how spread out would those sample means be?
Standard Error of the Mean
SEM = σ / √n (or estimated as s / √n when σ is unknown)
This shows that SEM decreases as sample size increases — larger samples give more precise
estimates of the population mean. It also increases with greater population variability. The Central
Limit Theorem guarantees that the sampling distribution of the mean approaches normality as n
increases (generally n ≥ 30), regardless of the population's distribution.

Standard Errors for Common Statistics

Statistic Standard Error Formula
Sample mean (x̄ ) SE = s / √n
Sample proportion (p̂ ) SE = √[p̂ (1 − p̂ ) / n]
Difference of two means SE = √[(s₁²/n₁) + (s₂²/n₂)]
Difference of two proportions SE = √[(p₁q₁/n₁) + (p₂q₂/n₂)]
Pearson correlation r SE = √[(1 − r²) / (n − 2)]
Regression coefficient b SE = s_y|x / √[Σ(x − x̄ )²]
Sample median SE ≈ 1.2533 × (s / √n) [for normal distributions]

NOTE: Standard Error vs. Standard Deviation

Standard deviation (SD) describes the spread of individual observations within a single sample.
Standard error (SE) describes the precision of a sample statistic as an estimate of the population
parameter — it is the SD of the sampling distribution of that statistic. As N increases, SD remains
relatively stable while SE decreases.

1.8 Confidence Intervals for Proportions and Probabilities

A confidence interval (CI) is a range of plausible values for a population parameter, constructed
from sample data, that will contain the true population parameter with a specified level of
confidence (e.g., 95%). It goes beyond a point estimate by quantifying the uncertainty around the
estimate.
The width of a CI reflects the precision of the estimate: a narrow CI indicates a precise estimate; a
wide CI indicates considerable uncertainty. Confidence intervals are more informative than p-
values alone because they communicate both statistical significance and the magnitude of effects.

Interpreting a 95% Confidence Interval

The correct frequentist interpretation: if we were to repeat the sampling and CI construction
procedure 100 times, approximately 95 of the 100 resulting intervals would contain the true
population parameter. It does NOT mean there is a 95% probability that the parameter lies within
this specific interval — once constructed, the interval either contains the parameter or it does not.
CI for a Population Proportion
CI = p̂ ± Zα/2 × √[p̂ (1 − p̂ ) / n]
For 95% CI: Zα/2 = 1.96. For 99% CI: Zα/2 = 2.576. Used when the sample size is large (np ≥ 5
and n(1-p) ≥ 5).

Example: In a study, 270 out of 500 participants reported improvement after therapy. p̂ = 270/500
= 0.54. SE = √(0.54 × 0.46 / 500) = √(0.000497) = 0.0223. 95% CI = 0.54 ± 1.96 × 0.0223 = 0.54
± 0.044 → [0.496, 0.584]. We are 95% confident that the true proportion showing improvement is
between 49.6% and 58.4%.

CI for a Population Mean

CI = x̄ ± tα/2,(n−1) × (s / √n)
The t-distribution is used (rather than z) when the population standard deviation (σ) is unknown,
which is the norm in practice. As n increases, the t-distribution converges to the standard normal
(z) distribution. For n > 120, t and z are virtually identical.

Factors Affecting CI Width

Factor Effect on Confidence Interval Width
Larger sample size (n ↑) Narrower CI — more precision
Higher variability (s ↑) Wider CI — less precision
Higher confidence level (95% Wider CI — greater certainty comes at cost of precision
→ 99%)
Larger Zα/2 or t-value Wider CI
Smaller Zα/2 (one-tailed) Narrower CI on one side
SECTION 2: SAMPLING METHODS AND MINIMIZING ERRORS

2.1 Fundamentals of Sampling

Sampling is the process of selecting a subset (sample) from a larger group (population) in order to
make inferences about population characteristics. Because studying an entire population is
usually impractical, too costly, or simply impossible — especially in large or hard-to-access
populations — carefully selected samples serve as the basis for generalizations about the whole
population.
The validity of any inference drawn from a sample depends critically on how well the sample
represents the population. A biased sample leads to biased conclusions regardless of how
sophisticated the analysis. Sampling theory provides the mathematical foundation for estimating
population parameters, quantifying uncertainty, and designing efficient surveys and studies.

Key Sampling Terminology

• Population (Universe): The entire set of individuals, objects, or measurements sharing a
defined characteristic that the researcher wishes to draw conclusions about. Must be
clearly defined before sampling begins.
• Sample: A subset of the population selected for direct observation or measurement.
• Sampling Frame: The complete list or enumeration of all units in the population from
which the sample is drawn (e.g., voter registration rolls, hospital admission records,
university enrollment lists). The sampling frame should ideally match the target population
exactly.
• Sampling Unit: The element or group of elements constituting the unit of selection at
each stage (e.g., individual person, household, school, clinic).
• Parameter: A fixed numerical characteristic of the population (e.g., population mean μ,
population proportion π). Usually unknown; estimated from sample statistics.
• Statistic: A numerical characteristic computed from sample data (e.g., sample mean x̄ ,
sample proportion p̂ ) used to estimate the population parameter.
• Representativeness: The degree to which the sample accurately reflects the key
characteristics of the population. The primary goal of good sampling.

2.2 Probability Sampling Methods

In probability sampling, every unit in the population has a known, non-zero probability of being
selected. This enables unbiased estimation of population parameters and allows computation of
sampling error. Probability sampling is the foundation of valid statistical inference.
A. Simple Random Sampling (SRS)
Every individual in the population has an equal and independent probability of being selected.
Selection is typically done via random number tables, lottery methods, or computer-generated
random numbers (e.g., using RAND function in Excel or [Link] in Python).
• Advantages: Theoretically unbiased; each sample has an equal probability; forms the
basis for most sampling theory
• Disadvantages: Requires a complete and accurate sampling frame; inefficient for large,
geographically dispersed populations; may miss small subgroups (minority representation
not guaranteed)
• Best for: Homogeneous populations with accessible, complete lists (e.g., employees of a
single organization)

B. Systematic Sampling
Participants are selected at fixed, regular intervals from a list. A random starting point (between 1
and k) is chosen, then every k-th unit is selected, where k = N/n (sampling interval, N = population
size, n = required sample size).
• Example: N = 2000, n = 100, k = 20. Start randomly at 14; then select units 14, 34, 54,
74...
• Advantages: Simpler to execute than SRS; ensures even coverage across the population
list; does not require a completely numbered list
• Disadvantages: If the population list has a periodic pattern (periodicity) that coincides with
the sampling interval k, systematic bias can occur (e.g., if every 20th unit on a list is
always a supervisor, and k = 20)

C. Stratified Random Sampling

The population is first divided into distinct, non-overlapping, homogeneous subgroups called
strata based on a variable related to the research (e.g., age, gender, income level, geographical
region, diagnosis). A simple random sample is then drawn independently from each stratum.
• Proportionate Stratified Sampling: Each stratum is sampled in proportion to its size in
the population. If females constitute 60% of the population, 60% of the total sample is
drawn from females. Ensures proportional representation of all strata.
• Disproportionate Stratified Sampling: Strata are sampled in proportions that differ from
their population sizes, often to oversample rare but important groups. Appropriate when
some strata are too small for proportionate sampling to give adequate representation.
Requires weighting during analysis.
• Advantages: More representative than SRS; enables separate analysis of each stratum;
increases statistical efficiency; reduces sampling error within strata
• Disadvantages: Requires prior knowledge of stratification variables; more complex design
and analysis; costly if many strata
D. Cluster Sampling
The population is divided into naturally existing groups called clusters (e.g., schools, villages,
hospitals, city blocks). A random sample of clusters is selected, and all individuals within selected
clusters (single-stage) or a random sample of individuals within clusters (two-stage) are studied.
• Single-stage: All elements within selected clusters are included
• Two-stage (Multi-stage): First randomly select clusters, then randomly select individuals
within chosen clusters
• Advantages: Cost-effective and practical for large, geographically dispersed populations;
does not require a complete listing of all individuals in advance
• Disadvantages: Higher sampling error than SRS for the same sample size (individuals
within clusters tend to be more similar to each other — intraclass correlation reduces
effective sample size); results in Design Effect (DEFF) > 1

E. Multi-Stage Sampling
A more complex variant where sampling is done in multiple successive stages, typically
combining different methods at each stage. Widely used in large national surveys (e.g., NFHS —
National Family Health Survey in India). Example stages: States → Districts → Villages →
Households → Individuals.

2.3 Non-Probability Sampling Methods

In non-probability sampling, not every member of the population has a known or equal chance of
being selected. These methods are more commonly used in qualitative research, exploratory
studies, and situations where a probability sample is impossible or unnecessary. Results cannot
be validly generalized using probability theory.

A. Convenience (Accidental) Sampling

Individuals who are readily available and accessible are selected. The selection is based on
convenience rather than any systematic procedure. Example: Administering a questionnaire to
students who happen to be in the library; recruiting volunteers through social media.
• Advantage: Quick, cheap, requires minimal resources
• Disadvantage: Highly susceptible to bias; unrepresentative; results cannot be generalized
to the population

B. Purposive (Judgmental) Sampling

The researcher deliberately selects individuals based on their particular expertise, knowledge, or
characteristics that are specifically relevant to the study. The sample is chosen with a purpose —
to include the 'best' informants for the research question.
• Used in: Qualitative research, case studies, expert opinion studies, focus groups
• Example: Selecting experienced clinicians to study clinical decision-making processes
• Disadvantage: Highly dependent on researcher's judgment; prone to bias
C. Quota Sampling
Similar to stratified sampling but without random selection. The population is divided into
subgroups, and a non-random sample is taken from each subgroup until a predetermined quota is
fulfilled. The interviewer fills the quota by any convenient means.
• Advantage: Ensures representation of key subgroups without a complete sampling frame;
fast
• Disadvantage: Within-quota selection is not random — cannot calculate valid standard
errors or make statistical inferences

D. Snowball (Network / Chain-Referral) Sampling

Initial participants are recruited, and they then refer other eligible individuals from their social
networks. The sample grows like a 'snowball' rolling down a hill. Especially useful for hard-to-
reach, hidden, or stigmatized populations.
• Used for: People who use drugs, undocumented immigrants, survivors of rare conditions,
certain professional groups
• Disadvantage: Heavy dependence on initial contacts; network-based bias; may miss
isolated individuals

2.4 Sampling Errors and Non-Sampling Errors

Any difference between the sample estimate and the true population parameter is an error. Errors
in survey research fall into two broad categories:

Sampling Error (Variable Error)

Sampling error is the natural, unavoidable variability that arises because a sample — rather than
the entire population — is measured. Even with perfect sampling methodology, different samples
from the same population will yield different estimates. The magnitude of sampling error depends
on sample size (it decreases as n increases) and population variability.
Sampling error is quantified by the standard error of the statistic. It can be reduced but never
eliminated (unless the entire population is surveyed — a census). It forms the basis of confidence
intervals.

Non-Sampling Errors (Systematic Errors)

Non-sampling errors arise from sources other than the sampling process itself. They are
potentially more serious because they do not decrease with larger samples — in fact, they can
increase.

Type of Non-Sampling Error Description and Examples

Coverage Error (Frame Error) The sampling frame excludes some members of the population
or includes non-members. Example: Using a telephone landline
directory excludes mobile-only users and the poor.
Type of Non-Sampling Error Description and Examples
Non-response Error Selected individuals fail to respond or participate. If non-
respondents systematically differ from respondents (e.g., the
sickest patients are least likely to complete a health survey),
estimates are biased.
Measurement Error Inaccurate measurement due to poorly worded questions,
interviewer behavior, response format issues, or instrument
problems. Example: Leading questions that push respondents
toward a particular answer.
Response Bias (Social Participants give inaccurate answers because they want to
Desirability) appear favorable. Example: Underreporting drug use or alcohol
consumption on surveys.
Interviewer Bias The interviewer's characteristics (gender, race, tone, manner)
influence how participants respond. Example: Male interviewers
getting different answers to sensitive questions than female
interviewers.
Processing Error Errors introduced during data coding, entry, transcription, or
analysis. Example: Miscoding '2' (Female) as '1' (Male) during
data entry.
Order / Context Effects The order of questions influences responses. Example: Rating
personal happiness after questions about recent negative
events.

2.5 Strategies to Minimize Sampling Errors

• Use probability sampling wherever feasible to ensure calculable, unbiased estimates
• Increase sample size — the most direct way to reduce random sampling error
• Use stratified sampling to ensure all important subgroups are represented proportionally
• Carefully verify and update the sampling frame before sampling begins
• Use multiple follow-up attempts (phone, mail, in-person) to reduce non-response rates
• Compare early vs. late respondents on key variables — late respondents approximate
non-respondents
• Ensure anonymous/confidential data collection to reduce social desirability bias
• Standardize all data collection procedures through interviewer training and protocol
manuals
• Pilot test the questionnaire to detect ambiguous items, unclear instructions, or ordering
effects
• Use double data entry verification or automated data validation to minimize processing
errors
SECTION 3: EXPERIMENTAL DESIGNS

3.1 Introduction and Key Concepts

An experimental design is a structured plan that specifies how participants are assigned to
conditions, how variables are measured and controlled, how treatments are applied, and how
resulting data will be analyzed. The primary purpose of experimental design is to establish causal
relationships between the independent variable (the manipulated cause) and the dependent
variable (the measured outcome) while systematically controlling for confounding variables.
The logic of experimentation rests on three foundational principles: (1) Manipulation — the
researcher actively manipulates the independent variable; (2) Control — extraneous variables are
held constant or balanced across conditions; (3) Randomization — participants are randomly
assigned to conditions to distribute unmeasured individual differences evenly across groups.

Key Design Terminology

• Independent Variable (IV): The variable manipulated by the researcher — the presumed
cause. E.g., type of treatment, teaching method, drug dosage.
• Dependent Variable (DV): The outcome variable measured to assess the effect of the IV.
E.g., test score, symptom severity, reaction time.
• Control Group: A comparison group that does not receive the experimental treatment;
provides the baseline against which effects are judged.
• Experimental Group: The group that receives the active treatment/intervention.
• Random Assignment (Randomization): Allocating participants to groups by chance,
ensuring every individual has an equal probability of being in any group. This is what
distinguishes true from quasi-experiments.
• Internal Validity: The degree to which observed changes in the DV can confidently be
attributed to manipulation of the IV rather than to confounding factors.
• External Validity: The generalizability of findings beyond the specific study to other
populations, settings, treatments, and time periods.
• Threats to Internal Validity: History (external events), maturation (natural change),
testing effects, instrumentation changes, statistical regression to the mean, selection bias,
experimental mortality (attrition).

3.2 Pre-Experimental Designs

Pre-experimental designs are the weakest category in terms of establishing internal validity. They
lack either a control group, random assignment, or both. They provide minimal control over
threats to internal validity and are best suited for preliminary, descriptive, or exploratory
investigations.
3.2.1 One-Shot Case Study (X → O)
A single group is exposed to a treatment (X), and a single post-treatment measurement (O) is
taken. There is no pre-test and no comparison group.
• Notation: X O
• Example: A workplace delivers a one-day safety training; employees are then assessed on
safety knowledge.
• Major weaknesses: No baseline for comparison; cannot determine what scores would
have been without treatment; no ability to establish causation.

3.2.2 One-Group Pretest-Posttest Design (O₁ X O₂)

A single group is measured both before (O₁) and after (O₂) the treatment. The change O₂ − O₁ is
taken as evidence of the treatment's effect.
• Notation: O₁ X O₂
• Example: Measuring depression scores (PHQ-9) before and after a 6-week mindfulness
intervention in a single group of patients.
• Threats to internal validity: History (other events between O₁ and O₂), maturation
(participants naturally improve over time), testing effect (familiarity with the test increases
scores), regression to the mean (initially extreme scorers tend to score closer to average
on retest), instrumentation changes.

3.2.3 Static Group Comparison

Two existing, non-equivalent groups are compared after one group has received a treatment.
There is no random assignment and no pre-test.
• Notation: X O₁ (treatment group) / — O₂ (comparison group)
• Example: Comparing test performance of a class that received a new curriculum vs.
another class that received the standard curriculum.
• Threat: Selection bias — the groups may have differed before the treatment; without a pre-
test, this cannot be assessed.

3.3 True Experimental Designs

True experimental designs are considered the 'gold standard' for establishing causal
relationships. They are characterized by: (1) random assignment of participants to conditions; (2)
presence of at least one control group; (3) direct manipulation of the independent variable. These
three features together maximize internal validity.

3.3.1 Randomized Posttest-Only Control Group Design

Participants are randomly assigned to either the experimental or control group. The experimental
group receives treatment (X); both groups are measured post-treatment. Random assignment
ensures baseline equivalence so that post-test differences reflect the treatment.
• Notation: R X O₁ (experimental) / R — O₂ (control)
 • Example: Randomly assigning patients to a new antidepressant vs. placebo; measuring
depression scores after 8 weeks.
• Advantage: Controls all major threats to internal validity through randomization. Avoids
pre-test sensitization.

3.3.2 Randomized Pretest-Posttest Control Group Design

The classic randomized controlled experiment. Random assignment, pre-tests on both groups,
treatment administered to the experimental group, post-tests on both groups.
• Notation: R O₁ X O₂ (experimental) / R O₃ — O₄ (control)
• The pre-test verifies baseline equivalence (despite randomization, it provides additional
assurance); the change score (O₂−O₁) vs. (O₄−O₃) is the treatment effect.
• Limitation: Pre-testing may sensitize participants to the treatment (reactivity); adding a pre-
test increases the time and cost of the study.

3.3.3 Solomon Four-Group Design

An elegant extension of the randomized pretest-posttest design that adds two additional groups
without pre-tests. It allows researchers to separately assess the effect of pre-testing itself and test
for interactions between pre-testing and treatment.
• Group 1: R O₁ X O₂ (pre-tested, treated)
• Group 2: R O₃ — O₄ (pre-tested, control)
• Group 3: R X O₅ (not pre-tested, treated)
• Group 4: R — O₆ (not pre-tested, control)
• Advantage: Maximum control over threats including testing effects and testing-treatment
interaction. Provides the strongest evidence for causal inference.
• Disadvantage: Requires four groups, hence a very large sample; expensive and complex.

3.3.4 Factorial Design

A design involving two or more independent variables (factors), with each combination of factor
levels forming a separate experimental condition. Factorial designs allow the simultaneous
examination of main effects of each independent variable and the interaction effects between
variables.
• A 2×2 factorial: 2 levels of Factor A × 2 levels of Factor B = 4 conditions
• Main effect: The effect of one factor averaged across all levels of the other factor
• Interaction effect: When the effect of one IV depends on the level of another IV. E.g., a
treatment might be effective for females but not for males — that is an interaction between
treatment and gender.
• Advantage: More efficient than running separate experiments; reveals interactions invisible
in single-factor designs
• Analysis: Factorial ANOVA (two-way or higher-order ANOVA)
3.4 Quasi-Experimental Designs
Quasi-experimental designs resemble true experiments in that an independent variable is
manipulated and outcomes are measured, but they lack full random assignment of participants to
conditions. They are used extensively in educational, social, health, and policy research where
random assignment is impractical or ethically unacceptable.
While quasi-experiments sacrifice some internal validity compared to true experiments, they often
gain substantially in external validity (they study real-world conditions). Researchers must
carefully consider and rule out alternative explanations (threats to internal validity) when
interpreting quasi-experimental results.

3.4.1 Nonequivalent Control Group Design

Two pre-existing groups (classrooms, clinics, communities) are used — one as the experimental
group and one as a comparison group — without random assignment. Both groups receive pre-
tests and post-tests.
• Notation: O₁ X O₂ (experimental) / O₃ — O₄ (control)
• Difference from true experiment: Groups are not randomly assigned; they are pre-existing
intact groups.
• Example: A new school-based mental health program is implemented in School A
(experimental); School B continues as usual (comparison). Both are assessed before and
after.
• Key threat: Selection bias — the groups may differ systematically before the intervention
on variables related to the outcome. The pre-test helps detect this.

3.4.2 Interrupted Time Series Design

Multiple measurements of the same outcome are taken on the same group (or population) over an
extended period. A treatment or intervention is introduced at a known point in time (interrupting
the series), and subsequent data are compared to the pre-intervention trend.
• Notation: O₁ O₂ O₃ O₄ O₅ X O₆ O₇ O₈ O₉ O₁₀
• Effect of treatment: Evidenced by a change in level (intercept shift) or a change in slope
(trend change) at the point of intervention.
• Example: Monthly rates of road fatalities measured over 5 years before and after
introduction of mandatory seatbelt legislation.
• Strength: Multiple pre-intervention data points establish a clear, stable baseline trend;
powerful for evaluating policy interventions.
• Threat: History — other events coinciding with the intervention could explain the observed
discontinuity.

3.4.3 Regression Discontinuity Design

Participants are assigned to conditions (treatment vs. control) based on whether they score above
or below a predetermined cutoff on a continuous assignment variable (e.g., an entrance exam
score, an income threshold, a clinical severity score). A sharp discontinuity in the outcome
variable at the cutoff point provides evidence of a causal treatment effect.
 • Example: Students scoring below 50 on a diagnostic test receive a remedial tutoring
program; those scoring 50+ proceed normally. If post-intervention test scores show a
sharp improvement specifically at the 50-point boundary, this suggests the tutoring
program was effective.
• Advantage: Can provide internally valid causal estimates despite lack of random
assignment; particularly powerful for policy evaluation.
• Disadvantage: Only valid if the assignment variable is measured continuously and the
cutoff is applied rigorously; the result applies mainly to individuals near the cutoff.

3.4.4 Counterbalanced (Within-Subjects) Quasi-Experimental Design

All participants receive all treatments, but in different orders (counterbalanced across participants
or groups). This controls for order effects by distributing them evenly. The Latin Square
arrangement is commonly used.
• Example: Four groups each receive treatments A, B, C, D in different orders: ABCD,
BCDA, CDAB, DABC. Each treatment appears once in each position.
• Advantage: Every participant serves as their own control; powerful for detecting small
effects.
• Threat: Carryover effects — the effect of one treatment may persist into subsequent
conditions.

3.5 Statistical Experimental Designs

Statistical designs refer specifically to the arrangement of treatments in relation to known sources
of variability (nuisance factors or blocking factors), and how the analysis partitions variance
among these sources. These designs are most developed in agricultural research but are widely
applied in clinical and social science research.

3.5.1 Completely Randomized Design (CRD)

Experimental units are randomly assigned to treatment groups without any restrictions or
blocking. All variation not accounted for by treatments is treated as random error.
• Analysis: One-way ANOVA
• Best for: Homogeneous experimental units in controlled laboratory settings
• Limitation: If experimental units are heterogeneous, CRD has high error variance; other
designs are more efficient

3.5.2 Randomized Block Design (RBD)

Experimental units are first grouped into blocks based on a known source of nuisance variation
(e.g., age groups, litter mates, hospitals, school classes). Within each block, treatments are
randomly assigned. This controls for between-block variation, reducing error variance and
increasing precision.
• Analysis: Two-way ANOVA (treatments × blocks)
 • Advantage: More efficient than CRD when the blocking variable is substantially correlated
with the DV
• Example: Testing three teaching methods (A, B, C) in five schools (blocks). Each school
uses all three methods with different student groups — school-to-school differences are
'blocked out'.

3.5.3 Latin Square Design

Controls for two sources of nuisance variation simultaneously (row effect and column effect).
Each treatment appears exactly once in each row and once in each column. This design is highly
efficient for its size but requires the number of rows, columns, and treatments to be equal.
• Example: A 4×4 Latin square for 4 treatments (A, B, C, D) across 4 time periods (rows)
and 4 subjects (columns). Each treatment appears once in each time period and once for
each subject.
• Limitation: Assumes no interaction between row factors, column factors, and treatments;
also, degrees of freedom for error are limited
SECTION 4: LEVELS OF MEASUREMENT OF VARIABLES

4.1 Introduction
The level of measurement (or scale of measurement) of a variable refers to the mathematical
properties of the values assigned to observations. Stanley Smith Stevens (1946) introduced the
landmark classification of four scales of measurement — Nominal, Ordinal, Interval, and Ratio —
commonly known by the acronym NOIR. This hierarchy from lowest to highest determines the
types of mathematical operations and statistical procedures that are appropriate, and
misidentifying the scale can lead to inappropriate analyses and misleading conclusions.

4.2 Nominal Scale

The nominal scale is the simplest and lowest level of measurement. Numbers or symbols are
assigned to objects or persons merely as labels to classify them into distinct, mutually exclusive
categories. The numbers have no quantitative meaning — they are arbitrary identifiers. All
categories are equivalent; none is 'greater than' or 'less than' another.

Properties of Nominal Scale

• Classification only: objects are sorted into distinct, non-overlapping categories
• No ordering: categories have no inherent order, rank, or magnitude
• No equal intervals: the concept of equal distances between categories is meaningless
• No true zero: zero (if used) is just another arbitrary category label
• Only equivalence and non-equivalence are meaningful: either two things belong to the
same category or different categories

Examples
• Gender: Male (1), Female (2), Non-binary (3) — the numbers are purely arbitrary codes
• Blood type: A, B, AB, O
• Diagnostic categories: Major Depressive Disorder, Bipolar Disorder, Schizophrenia (DSM-
5 categories)
• Marital status: Single, Married, Divorced, Widowed, Separated
• Nationality, ethnicity, religion, political party affiliation
• ICD-10 and DSM-5 diagnostic codes

Appropriate Statistical Procedures

• Mode (the category with the highest frequency — the only meaningful measure of central
tendency)
• Frequency counts, proportions, and percentages
 • Chi-square test for goodness-of-fit or test of independence
• Phi coefficient (φ) for 2×2 tables; Cramér's V for larger tables (measures association
between two nominal variables)
• Lambda (λ) — measure of proportional reduction in prediction error for nominal data

4.3 Ordinal Scale

The ordinal scale retains the classification function of the nominal scale but adds the property of
rank ordering — objects or persons can be meaningfully arranged from most to least (or least to
most) with respect to the measured attribute. The critical limitation is that the intervals between
adjacent ranks are NOT necessarily equal — rank 1 and rank 2 may be very close, while rank 2
and rank 3 may be far apart. Only the order of categories is meaningful, not the magnitude of
difference between them.

Properties of Ordinal Scale

• Classification into categories (same as nominal)
• Ranked ordering: one category is greater or lesser than another
• Unequal intervals: distances between adjacent ranks are unknown and may vary
• No true zero point
• Relative positions are meaningful; differences between positions are not

Examples
• Likert-type scales: 'Strongly Disagree (1) – Disagree (2) – Neutral (3) – Agree (4) –
Strongly Agree (5)'. We know 4 > 3 > 2 but cannot claim the difference between 4 and 5
equals the difference between 1 and 2.
• Academic letter grades: A > B > C > D > F (but 'A' is not exactly twice as good as 'B')
• Clinical severity ratings: Mild, Moderate, Severe, Very Severe
• Socioeconomic status: Lower class, Middle class, Upper class
• Military/professional ranks: Private, Corporal, Sergeant, Lieutenant, Captain, Major
• Competition rankings: 1st, 2nd, 3rd place in a race

Appropriate Statistical Procedures

• Median and percentiles / quartiles (appropriate measures of central tendency)
• Range and interquartile range (IQR) as measures of spread
• Spearman's rank-order correlation coefficient (ρ) and Kendall's tau (τ)
• Non-parametric tests: Mann-Whitney U, Wilcoxon Signed-Rank, Kruskal-Wallis H,
Friedman test
• Mode is also applicable
4.4 Interval Scale
The interval scale adds the critical property of equal intervals to the properties of ordinal scales.
The distance between any two adjacent values is the same throughout the entire scale, making
addition and subtraction of scores meaningful. However, there is no true (absolute) zero point —
the zero is arbitrary and does not represent the complete absence of the attribute being
measured. This means ratios between values are NOT meaningful on an interval scale.

Properties of Interval Scale

• Classification and ordering (same as ordinal)
• Equal intervals between adjacent scale points — differences are meaningful
• No absolute zero — zero is an arbitrary reference point
• Ratios are NOT meaningful: 40°C is not 'twice as hot' as 20°C
• Addition and subtraction of scores are meaningful; multiplication and division are not

Examples
• Temperature (Celsius or Fahrenheit): 0°C is NOT the absence of heat; the difference
between 10°C and 20°C equals the difference between 30°C and 40°C.
• IQ scores: Standardized with mean 100 and SD 15. An IQ of 0 does not mean zero
intelligence.
• Standardized psychological test scores (e.g., depression inventory total scores, anxiety
scale scores)
• Calendar years: The year 0 CE does not represent the beginning of time.
• Attitude scale scores when validated as reflecting equal intervals

Appropriate Statistical Procedures

• Mean and standard deviation (appropriate measures of central tendency and dispersion)
• Pearson product-moment correlation (r)
• t-tests (one-sample, independent, paired)
• Analysis of Variance (ANOVA)
• Regression analysis (simple and multiple)
• Factor analysis

4.5 Ratio Scale

The ratio scale is the highest and most informative level of measurement. It possesses all the
properties of the interval scale and additionally has a TRUE ABSOLUTE ZERO POINT — zero
represents the complete and genuine absence of the attribute being measured. This makes ratios
between values fully meaningful: a value of 60 is genuinely twice as large as a value of 30.
Properties of Ratio Scale
• Classification, ordering, and equal intervals (all properties of interval scale)
• True absolute zero: zero = complete absence of the attribute
• All four arithmetic operations (addition, subtraction, multiplication, division) are meaningful
• Ratios are meaningful: 60 kg is twice 30 kg; 4 hours is half of 8 hours

Examples
• Height, weight, length — all physical measurements with a true zero
• Reaction time (in milliseconds): 0 ms would mean instantaneous response
• Number of errors on a task: 0 errors = no errors
• Income in rupees or dollars: 0 income = no income
• Blood pressure, heart rate, respiratory rate
• Age: 40 years is genuinely twice as old as 20 years
• Dosage in mg/kg, enzyme activity levels

Appropriate Statistical Procedures

• All parametric statistics applicable to interval data, plus:
• Geometric mean (meaningful because ratios are meaningful)
• Coefficient of variation (CV = SD/Mean × 100%) — meaningful as a relative measure of
variability
• All multivariate techniques (MANOVA, discriminant analysis, structural equation modeling)

4.6 Summary Comparison Table

Property Nominal / Ordinal / Interval / Example of Each
Ratio
Classification ✓/✓/✓/✓ Blood type / Pain level / IQ /
(Categories) Weight
Ordered (Ranked) ✗/✓/✓/✓ — / Mild>Mod>Severe / — /
60kg>30kg
Equal intervals ✗/✗/✓/✓ — / — / Temp in °C / Height in
cm
Absolute zero ✗/✗/✗/✓ — / — / — / Weight=0 means
none
Meaningful ratios ✗/✗/✗/✓ — / — / — / 40kg = 2×20kg
Best central tendency Mode / Median / Mean / Mean
Best correlation Phi / Spearman ρ / Pearson r /
Pearson r
SECTION 5: ESTABLISHING RELIABILITY, VALIDITY, AND
NORMS

5.1 Reliability
Reliability refers to the consistency, stability, and dependability of a measurement instrument or
procedure. A reliable instrument produces the same (or closely similar) results when applied
repeatedly under the same conditions, when administered by different raters, or when different
equivalent forms of the test are used. Reliability is a necessary prerequisite for validity: a measure
cannot be valid if it is not reliable, but reliability alone does not guarantee validity.
Reliability is formally defined in Classical Test Theory (CTT) as: X = T + E, where X = observed
score, T = true score, E = error of measurement. Reliability is the proportion of observed score
variance attributable to true score variance: r_xx = σ²T / σ²X. A reliability coefficient of 1.0 would
mean all variation in scores reflects true differences; 0 would mean all variation is error.

5.1.1 Test-Retest Reliability (Coefficient of Stability)

The same instrument is administered to the same group of participants on two separate
occasions, with an appropriate time interval between administrations. The Pearson correlation (r)
between scores at Time 1 and Time 2 is the reliability coefficient.
• Measures stability of scores over time
• Appropriate time interval: Long enough to prevent memory effects (usually 2–6 weeks);
short enough that the underlying attribute has not genuinely changed
• Best suited for: Stable psychological traits (e.g., personality, intelligence, chronic anxiety)
• Not appropriate for: Rapidly fluctuating states (e.g., mood, acute pain)
• Acceptable coefficient: r ≥ 0.80

5.1.2 Parallel Forms Reliability (Coefficient of Equivalence)

Two equivalent forms of the same test (same content domain, same difficulty, same number and
format of items) are developed and administered to the same group. The correlation between
scores on Form A and Form B is the reliability coefficient.
• Eliminates practice and memory effects (unlike test-retest)
• Advantage: Provides cleaner measure of reliability without time effects
• Disadvantage: Extremely difficult and expensive to construct two truly equivalent forms;
any differences in form difficulty reduce the coefficient

5.1.3 Internal Consistency Reliability

Measures the extent to which all items within a single test are measuring the same underlying
construct. Based on a single test administration and analysis of the inter-item relationships.
Split-Half Reliability
The test is split into two halves (commonly odd vs. even items, or first half vs. second half), and
scores on the two halves are correlated. Because the reliability coefficient applies to only half the
test length, the Spearman-Brown prophecy formula is applied to estimate the full-length test
reliability:
r_SB = 2r_halves / (1 + r_halves)
Example: If split-half r = 0.70, then r_SB = 2(0.70)/(1+0.70) = 1.40/1.70 = 0.82

Cronbach's Alpha (α)

The most widely used measure of internal consistency. Mathematically, it is equivalent to the
mean of all possible split-half reliabilities. It ranges from 0 (no reliability) to 1 (perfect reliability).
For continuous or polytomous items (e.g., Likert scales):
α = (k / (k−1)) × (1 − Σσ²ᵢ / σ²ₓ)
Where k = number of items, Σσ²ᵢ = sum of individual item variances, σ²ₓ = total score variance.
• α ≥ 0.90: Excellent — suitable for high-stakes diagnostic use
• α = 0.80–0.89: Good — appropriate for most research
• α = 0.70–0.79: Acceptable — minimum for research purposes
• α = 0.60–0.69: Questionable — borderline, only for exploratory research
• α < 0.60: Poor — scale needs significant revision

Kuder-Richardson KR-20 and KR-21

Used when items are dichotomous (scored right/wrong, yes/no). KR-20 is the exact analogue of
Cronbach's alpha for binary items; KR-21 is a simplified version that assumes all items have
equal difficulty.
KR-20 = (k / (k−1)) × (1 − Σ(pᵢqᵢ) / σ²ₓ)
Where pᵢ = proportion answering item i correctly, qᵢ = 1 − pᵢ.

5.1.4 Inter-Rater (Inter-Observer) Reliability

Assesses the degree of agreement between two or more independent raters or observers who
independently score the same set of responses, behaviors, or test items. This is critical when
subjective judgment is involved (e.g., scoring open-ended interview responses, coding behavioral
observations, rating essay quality).
• Percentage Agreement: Simple proportion of cases on which raters agree. However, this
does not correct for chance agreement and may be misleadingly high when one rating
category is very common.
• Cohen's Kappa (κ): Corrects for chance agreement. κ = (P_o − P_c) / (1 − P_c), where
P_o = observed agreement, P_c = expected chance agreement. Ranges from −1 (perfect
disagreement) to +1 (perfect agreement). κ ≥ 0.61 = substantial; κ ≥ 0.81 = almost perfect.
• Intraclass Correlation Coefficient (ICC): Used when ratings are on a continuous scale
and there are more than two raters. Several forms exist depending on whether raters are
fixed or random effects.
5.2 Validity
Validity refers to the degree to which a test or measurement instrument actually measures what it
purports to measure — whether it is capturing the intended construct accurately and completely.
While reliability concerns consistency, validity concerns accuracy and meaningfulness. The two
concepts are related: validity is bounded by reliability (the maximum possible validity coefficient =
√reliability), but a reliable test is not necessarily valid.

5.2.1 Content Validity

Content validity is the degree to which the items of a test adequately and representatively sample
the full domain or universe of content relevant to the construct being measured. It is established
primarily through systematic expert judgment and logical analysis rather than through statistical
procedures.
• Face Validity: The most superficial form: does the test appear to measure what it claims,
based on surface inspection? This is important for gaining participant cooperation but
provides weak evidence of validity.
• Content Validity Ratio (CVR — Lawshe, 1975): A systematic quantitative approach
using a panel of subject matter experts who rate each item as 'Essential,' 'Useful but not
essential,' or 'Not necessary.' CVR = (ne − N/2) / (N/2), where ne = number of experts
rating the item 'Essential,' N = total number of experts. Minimum CVR values depend on
panel size (e.g., minimum CVR = 0.99 for 5 experts; 0.62 for 14 experts).
• Content Validity Index (CVI): Average of item-level CVRs across all items in the test. CVI
≥ 0.80 is generally considered acceptable.

5.2.2 Criterion-Related Validity

Criterion-related validity is established by correlating test scores with an external criterion — a
direct, independent, and relevant measure of the characteristic the test is designed to assess. It is
assessed statistically.
• Concurrent Validity: The test score and criterion measure are obtained at approximately
the same time. Assesses whether the test correlates with a currently available criterion.
Example: A new computerized diagnostic test for depression is validated against
structured clinical interview (SCID) diagnoses made at the same time.
• Predictive Validity: The test is administered first, and the criterion is measured at a later
time. Assesses the test's ability to forecast future performance. Example: University
entrance examination scores predicting academic GPA obtained 2 years later.

5.2.3 Construct Validity

Construct validity is the most fundamental and theoretically important form of validity. It addresses
the question: does this test actually measure the psychological, social, or theoretical construct it
claims to measure? It is established through the gradual accumulation of multiple lines of
evidence across many studies over time — it is never established by a single study.
 • Convergent Validity: The test should correlate positively and substantially with other
measures that theoretically assess the same or highly similar constructs. Example: A new
measure of anxiety should correlate highly with established anxiety measures like the
GAD-7 or BAI.
• Discriminant (Divergent) Validity: The test should NOT correlate substantially with
measures of theoretically different, unrelated constructs. Example: A measure of anxiety
should not correlate highly with general intelligence scores.
• Multi-trait, Multi-method Matrix (MTMM): Campbell and Fiske's (1959) approach:
measuring multiple constructs using multiple methods. Convergent validity is supported by
high correlations within the same construct across methods; discriminant validity is
supported by low correlations between different constructs.
• Known-Groups Validity: The test distinguishes meaningfully between groups known to
differ on the construct. Example: A diagnostic depression scale should produce
significantly higher scores in clinically depressed patients than in healthy controls.
• Factor Validity: Factor analysis of the test items reveals a factor structure that is
theoretically consistent with the construct model. Confirmatory factor analysis (CFA) is
preferred over exploratory factor analysis (EFA) for this purpose.

5.2.4 External (Ecological) Validity

The extent to which findings from a study can be generalized to other populations, settings,
treatments, and time periods beyond the specific conditions of the study. High experimental
control often reduces ecological validity — laboratory conditions may not mirror real-world
conditions. Naturalistic studies often have higher ecological validity but lower internal validity.

NOTE: Relationship Between Reliability and Validity

A test can be reliable without being valid (consistent but inaccurate, like a scale that consistently
overestimates weight). But a test cannot be valid without being reliable (an inconsistent measure
cannot accurately capture the construct). Mathematically: Maximum Validity ≤ √(Reliability). Thus
reliability sets an upper bound on validity.

5.3 Norms
Norms are performance standards derived empirically from a carefully defined, representative
standardization sample. They allow a raw test score to be interpreted meaningfully by comparing
it to the performance of a relevant reference group. Without norms, a raw score like '72 on a
depression scale' conveys nothing about whether 72 represents high, average, or low depression.

Types of Norms
• Age Norms: Express the average performance at each age level. A child's raw score is
compared to the average for children of the same chronological age. Widely used in
developmental testing. Example: The Binet intelligence scale originally expressed
performance as a mental age.
 • Grade Norms: Express the average performance at each school grade level. Used in
educational achievement tests. Example: A grade norm of 4.5 means the student is
performing at the level of a student midway through 4th grade.
• Percentile Norms (Percentile Ranks): Indicate the percentage of individuals in the
normative sample who scored at or below a given score. A PR of 80 means the examinee
scored higher than 80% of the normative group. Percentile ranks are ordinal — they
compress differences at the extremes and exaggerate differences near the median.
• Standard Score Norms: Express scores in units of standard deviation from the group
mean. Common standard score systems include: z-scores (mean=0, SD=1), T-scores
(mean=50, SD=10), IQ-type scores (mean=100, SD=15), SAT scores (mean=500,
SD=100, older scale), Stanines (mean=5, SD=2, 9-point scale).
• Local vs. National Norms: National norms are based on a nationally representative
sample. Local norms are derived from a specific subgroup (e.g., students within a state or
district). Local norms are more appropriate when comparisons within a specific group are
more meaningful than comparisons to the national population.

Developing a Normative Sample — Key Requirements

• Sufficiently large: typically n ≥ 200 per age/grade group in standardized educational tests;
larger samples for broadly used clinical tests
• Representative: stratified by age, gender, geographic region, socioeconomic status,
educational attainment, and other relevant variables
• Clearly defined: the population to which norms apply must be precisely specified
• Periodically updated: norms become outdated as populations change. The Flynn Effect
(rising IQ scores across generations) demonstrates that norms must be re-standardized
every 15–20 years or scores will be systematically overestimated.

Standard Score Conversion Table

Score System Mean | SD | Score Range
z-score 0 | 1 | Approximately −3 to +3
T-score 50 | 10 | 20–80
IQ (Wechsler, Binet) 100 | 15 | 40–160
SAT (older scale) 500 | 100 | 200–800
Stanine 5 | ~2 | 1–9
Normal Curve Equivalent 50 | 21.06 | 1–99
(NCE)
SECTION 6: TESTS OF SIGNIFICANCE — PARAMETRIC AND
NON-PARAMETRIC

6.1 Introduction and Classification

Tests of significance (also called inferential statistical tests) are procedures that use sample data
to make decisions about population parameters or to test hypotheses about population
characteristics. They evaluate the probability that observed results could have arisen by chance if
the null hypothesis were true. When this probability (the p-value) is sufficiently small (less than α),
the result is declared statistically significant and the null hypothesis is rejected.
Inferential tests fall into two broad families: parametric tests and non-parametric (distribution-free)
tests. The choice between them depends on the level of measurement of the data, the distribution
of the data, and the sample size.

Parametric Tests Non-Parametric (Distribution-Free) Tests

Assume data come from a No assumptions about the shape of the population distribution
normally distributed population
Require interval or ratio level Appropriate for nominal or ordinal data; can also be used for
data continuous data that violates normality
More statistically powerful Less powerful than parametric tests under normality (but can
when assumptions are met be more powerful under non-normality)
Sensitive to outliers and Robust to outliers and distributional violations
violations of assumptions
Estimate population Based on ranks, frequencies, or signs — not actual data values
parameters (mean, variance)
Examples: t-test, ANOVA, Examples: Mann-Whitney U, Kruskal-Wallis, Spearman ρ, chi-
Pearson r, regression square

6.2 Parametric Tests

6.2.1 Student's t-Test

The t-test is used to test hypotheses about population means. It is based on the t-distribution (a
symmetric, bell-shaped distribution with heavier tails than the normal distribution, which accounts
for the additional uncertainty of estimating σ from a sample). As sample size increases, the t-
distribution approaches the normal distribution.

A. One-Sample t-Test
Tests whether the mean of a single sample differs significantly from a known or hypothesized
population mean (μ₀).
t = (x̄ − μ₀) / (s / √n), df = n − 1
• Example: A psychologist tests whether the mean BDI score of a group of patients (x̄ = 22)
differs significantly from the population mean for outpatients (μ₀ = 18).
• Assumptions: Data should be approximately normally distributed (or n ≥ 30 by CLT);
observations should be independent.

B. Independent Samples t-Test (Two-Sample t-Test)

Tests whether the means of two independent (unrelated) groups differ significantly from each
other.
t = (x̄ ₁ − x̄ ₂) / SE(x̄ ₁ − x̄ ₂), df = n₁ + n₂ − 2
SE(x̄ ₁ − x̄ ₂) = sp × √(1/n₁ + 1/n₂)
Where sp = pooled standard deviation = √[(( n₁−1)s₁² + (n₂−1)s₂²) / (n₁+n₂−2)]
• Example: Comparing mean anxiety scores between a CBT group and a waitlist control
group.
• Assumptions: Normality within each group; homogeneity of variance (tested with Levene's
F-test); independence of observations.
• When variances are unequal (Levene's p < 0.05): Use Welch's t-test (separate variances
t-test) with adjusted degrees of freedom.

C. Paired Samples t-Test (Dependent t-Test)

Tests whether the mean difference between paired observations (same participants measured
twice, or matched pairs) differs significantly from zero.
t = D̄ / (s_D / √n), df = n − 1
Where D̄ = mean of difference scores (D = X₂ − X₁), s_D = standard deviation of difference
scores, n = number of pairs.
• Example: Comparing PHQ-9 depression scores in the same patients before and after 12
weeks of antidepressant treatment.
• Advantage over independent t-test: By comparing each person to themselves, individual
differences are controlled, reducing error variance and increasing power.

6.2.2 Analysis of Variance (ANOVA)

ANOVA is used when comparing means of three or more independent groups. It tests whether at
least one group mean differs significantly from the others. Rather than performing multiple
pairwise t-tests (which inflates the Type I error rate), ANOVA partitions the total variance in the
data into variance due to the treatment (between-groups) and variance due to random error
(within-groups).

One-Way ANOVA
 F = MS_Between / MS_Within = (SS_Between / df_Between) / (SS_Within /
df_Within)
• df_Between = k − 1 (k = number of groups)
• df_Within = N − k (N = total sample size)
• A significant F tells us that at least one mean differs — but not which groups differ. Post-
hoc tests are needed.

Post-Hoc Tests
Post-Hoc Test When to Use and Properties
Tukey's HSD (Honestly Appropriate when all pairwise comparisons are of interest and
Significant Difference) group sizes are equal. Controls family-wise error rate.
Bonferroni correction Divide α by the number of comparisons. Conservative but
widely applicable; can be used for any number of planned or
post-hoc comparisons.
Scheffé test Most conservative; suitable when testing complex (non-
pairwise) comparisons. Useful when some comparisons are
planned and some are exploratory.
Fisher's LSD (Least Significant Least conservative; only appropriate after a significant omnibus
Difference) F-test. Controls experiment-wise error only weakly.
Duncan's Multiple Range Test Less conservative than Tukey's; used when identifying rank
ordering of means.

Assumptions of ANOVA
• Independence of observations
• Normality of the dependent variable within each group (robust to mild violations)
• Homogeneity of variance across groups (Levene's test; robust to moderate violations if
group sizes are equal)

Two-Way ANOVA
Tests the effects of two independent variables and their interaction on a continuous dependent
variable. Partitions variance into: main effect of Factor A, main effect of Factor B, A×B interaction
effect, and within-cell error.

6.2.3 Pearson Product-Moment Correlation Coefficient (r)

Measures the strength and direction of the linear association between two continuous (interval or
ratio level) variables. r ranges from −1 to +1.
r = Σ[(xᵢ − x̄ )(yᵢ − ȳ)] / √[Σ(xᵢ − x̄ )² × Σ(yᵢ − ȳ)²]
• r = +1: Perfect positive linear relationship; r = 0: No linear relationship; r = −1: Perfect
negative linear relationship
 • r² (coefficient of determination): Proportion of variance in Y explained by X. E.g., r = 0.60
means r² = 0.36 → 36% shared variance.
• Significance test: t = r√(n−2) / √(1−r²), df = n−2
• Assumptions: Linearity of the relationship; bivariate normality; no severe outliers

6.2.4 Linear Regression

Regression analysis predicts the value of a dependent variable (Y) from one or more independent
variables (X). Simple linear regression: Ŷ = a + bX, where a = Y-intercept, b = regression
coefficient (slope).
• b represents the expected change in Y for each one-unit increase in X
• Significance of b tested using t = b/SE_b; overall model significance tested with F-test
• Multiple regression: Ŷ = a + b₁X₁ + b₂X₂ + ... + b_kX_k — each bᵢ represents the effect of
Xᵢ controlling for all other predictors
• R² = proportion of variance in Y explained by all predictors combined

6.2.5 Z-Test for Proportions

Tests whether a sample proportion differs significantly from a hypothesized population proportion,
or whether two sample proportions differ significantly. Appropriate for large samples (np ≥ 5 and
nq ≥ 5).
z = (p̂ − p₀) / √[p₀(1−p₀) / n]

6.3 Non-Parametric Tests

Non-parametric (distribution-free) tests make no assumptions about the underlying population
distribution. They are used when data violate parametric assumptions, when data are nominal or
ordinal, or when sample sizes are too small to verify normality. Although generally less powerful
than parametric tests under normality, non-parametric tests can be more powerful when the
normality assumption is violated.

6.3.1 Chi-Square Test (χ²)

The chi-square test is the most widely used non-parametric test. It compares observed
frequencies in categories against expected frequencies.
χ² = Σ [(Oᵢ − Eᵢ)² / Eᵢ]
• Goodness-of-Fit Test: Tests whether an observed frequency distribution fits a
hypothesized (theoretical) distribution. df = k − 1. Example: Testing whether the
distribution of blood types in a sample matches expected population frequencies.
• Test of Independence (Contingency Table): Tests whether two categorical variables are
statistically independent or associated. df = (r−1)(c−1). Example: Testing whether gender
(male/female) is associated with choice of therapy type (CBT/DBT/psychodynamic).
• Important assumption: Expected frequency in each cell must be ≥ 5. If violated, use
Fisher's Exact Test (for 2×2 tables).
6.3.2 Mann-Whitney U Test (Wilcoxon Rank-Sum Test)
The non-parametric equivalent of the independent samples t-test. Tests whether two independent
groups tend to have different population distributions (specifically, whether one group tends to
produce higher values than the other). Does not require normality.
Procedure: Pool all observations, rank them from lowest (1) to highest, then separately sum the
ranks for each group. The U statistic is computed from these rank sums.
U₁ = n₁n₂ + n₁(n₁+1)/2 − R₁
• Take the smaller of U₁ and U₂; compare to critical value table (or use normal
approximation for large samples)
• Example: Comparing subjective pain ratings (1–10 ordinal scale) between patients
receiving two different analgesic treatments

6.3.3 Wilcoxon Signed-Rank Test

The non-parametric equivalent of the paired samples t-test. Used to compare two related samples
(same individuals measured twice, or matched pairs) when the distribution of difference scores is
not normal.
Procedure: Calculate difference scores (D = X₂ − X₁), rank the absolute values of the differences
(ignoring zeros), then assign the sign (+ or −) of the original difference to each rank. T = the
smaller of the sum of positive ranks and the sum of negative ranks.
• Under H₀: the sum of positive and negative ranks should be approximately equal
• Example: Comparing self-rated quality of life scores in the same patients before and after
a rehabilitation program, when difference scores are skewed

6.3.4 Kruskal-Wallis H Test

The non-parametric equivalent of one-way ANOVA. Tests for differences among three or more
independent groups on an ordinal or non-normally distributed continuous variable.
H = [12 / (N(N+1))] × Σⱼ[Rⱼ² / nⱼ] − 3(N+1)
• Rⱼ = sum of ranks for group j; nⱼ = sample size of group j; N = total sample size
• H follows the chi-square distribution with df = k − 1 (for n ≥ 5 per group)
• Post-hoc pairwise comparisons: Dunn's test with Bonferroni correction
• Example: Comparing job satisfaction scores (rated on a 5-point Likert scale) across four
departments of a hospital

6.3.5 Friedman Test

The non-parametric equivalent of one-way repeated-measures ANOVA. Used when the same
participants are measured under three or more conditions and the data are at least ordinal.
• Procedure: Within each participant, rank their scores across the k conditions. Sum the
ranks for each condition across all participants. The Friedman statistic follows a chi-square
distribution.
 • Example: Participants rate the acceptability of three different medication regimes under a
crossover design, and ratings are ordinal

6.3.6 Spearman's Rank-Order Correlation Coefficient (ρ)

The non-parametric equivalent of Pearson's r. Measures the strength and direction of the
monotonic (not necessarily linear) relationship between two ordinal variables, or continuous
variables when Pearson's assumptions are violated.
ρ = 1 − [6 Σdᵢ²] / [n(n² − 1)]
Where dᵢ = difference between ranks of the i-th pair. Like Pearson's r, ρ ranges from −1 to +1.
• Tested for significance using t = ρ√(n−2) / √(1−ρ²), df = n−2
• Example: Correlating students' class rank in Mathematics with their rank in Physics

6.3.7 Sign Test

The simplest non-parametric test for paired data. Only considers the direction (sign) of differences
between paired observations — ignores their magnitude entirely. Tests whether the median of
differences equals zero (i.e., H₀: median difference = 0).
• Procedure: Count the number of positive differences (n₊) and negative differences (n₋);
ignore ties. Under H₀, n₊ and n₋ should each be about half the total. Use binomial
distribution (or z-approximation for large n).
• Less powerful than Wilcoxon Signed-Rank test because it discards magnitude information;
used only when differences cannot be meaningfully ranked

6.3.8 Kendall's Tau (τ)

Another non-parametric measure of ordinal association, like Spearman's ρ but based on
concordant and discordant pairs. Preferred over Spearman's ρ in some contexts because its
sampling distribution is better understood and it generalizes more naturally to partial correlation
(Kendall's partial tau).
τ = (C − D) / [n(n−1)/2]
Where C = number of concordant pairs, D = number of discordant pairs.

6.4 Comprehensive Summary: Parametric vs. Non-Parametric

Research Situation Parametric Test Non-Parametric Equivalent
One sample vs. known One-sample t-test One-sample Wilcoxon / Sign test
population mean
Two independent group Independent t-test Mann-Whitney U test
means
Two paired/related Paired t-test Wilcoxon Signed-Rank test
means
Research Situation Parametric Test Non-Parametric Equivalent
3+ independent group One-way ANOVA Kruskal-Wallis H test
means
3+ related/repeated Repeated-measures ANOVA Friedman test
measures
Linear association (2 Pearson's r Spearman's ρ or Kendall's τ
continuous vars)
Association: 2 — (use chi-square) Chi-square test of independence
categorical variables
Association: 2×2 table, — Fisher's Exact Test
small n
Predict Y from one X Simple linear regression —
(continuous)
Predict Y from multiple Multiple regression —
Xs

6.5 Choosing Between Parametric and Non-Parametric Tests

• Use Parametric When:: Data are interval or ratio; the population distribution is
approximately normal (or n ≥ 30 by CLT); variances are homogeneous; no severe outliers;
you want maximum statistical power.
• Use Non-Parametric When:: Data are ordinal or nominal; sample size is small (n < 20)
and normality cannot be assumed; data are severely skewed or contain outliers that
cannot be removed; the outcome is a rank or category; a normality test (Shapiro-Wilk,
Kolmogorov-Smirnov) is significant (p < 0.05).
QUICK REFERENCE: KEY FORMULAS AND TABLES

Essential Formulas
Formula Expression
Standard Error of Mean SEM = s / √n
95% CI for Mean x̄ ± 1.96 × (s/√n) [or t × SEM for small n]
95% CI for Proportion p̂ ± 1.96 × √[p̂ (1-p̂ )/n]
z-score z = (X − μ) / σ
T-score T = 10z + 50
Pearson r r = Σ(x-x̄ )(y-ȳ) / √[Σ(x-x̄ )²·Σ(y-ȳ)²]
Spearman ρ ρ = 1 - 6Σd²/[n(n²-1)]
Chi-Square χ² = Σ(O-E)²/E
Cronbach's Alpha α = k/(k-1) × (1 - Σσ²ᵢ/σ²ₓ)
Spearman-Brown r_sb = 2r / (1+r)
F ratio (ANOVA) F = MS_between / MS_within
Cohen's d d = (μ₁-μ₂) / σ_pooled
CVR (Lawshe) CVR = (ne - N/2) / (N/2)
Power Power = 1 − β
Max Validity Validity ≤ √(Reliability)
KR-20 KR-20 = (k/(k-1)) × (1 - Σpᵢqᵢ/σ²)
Mann-Whitney U U = n₁n₂ + n₁(n₁+1)/2 − R₁

Critical z and t Values

Value z or t
z for 90% CI (α=0.10, two- z = 1.645
tailed)
z for 95% CI (α=0.05, two- z = 1.960
tailed)
z for 99% CI (α=0.01, two- z = 2.576
tailed)
Value z or t
z for 80% power (β = 0.20) z = 0.842
z for 90% power (β = 0.10) z = 1.282
z for 95% power (β = 0.05) z = 1.645
Chi-square critical value (df=1, χ² = 3.841
α=0.05)
Chi-square critical value (df=2, χ² = 5.991
α=0.05)
Chi-square critical value (df=3, χ² = 7.815
α=0.05)

Reliability and Effect Size Benchmarks

Benchmark Value / Interpretation
Excellent reliability (Cronbach's α ≥ 0.90
α)
Good reliability α = 0.80 – 0.89
Acceptable reliability α = 0.70 – 0.79
Poor reliability α < 0.60
Cohen's κ: Substantial κ = 0.61 – 0.80
agreement
Cohen's κ: Almost perfect κ = 0.81 – 1.00
agreement
Small effect size (Cohen's d) d = 0.20
Medium effect size (Cohen's d) d = 0.50
Large effect size (Cohen's d) d = 0.80
Small effect for ANOVA f = 0.10
(Cohen's f)
Medium effect for ANOVA f = 0.25
(Cohen's f)
Large effect for ANOVA f = 0.40
(Cohen's f)
Minimum content validity (CVI) CVI ≥ 0.80

End of Research Methodology Comprehensive Notes

All sections covered: Hypothesis Testing • Sampling • Experimental Designs • Measurement Scales •
Reliability & Validity & Norms • Parametric & Non-Parametric Tests