Reliability, Validity, and

Sampling in Research
Based on Ranjit Kumar (2014) & Shaughnessy et al. (2015)

By: Dr. Syeda Ayat e Zainab Ali

Introduction to Reliability and Validity

Two Pillars of Measurement Quality Foundation for Scientific Credibility

Reliability and validity are the two pillars of Without reliability, measurements fluctuate randomly;
measurement quality in research, ensuring that the without validity, results do not reflect the intended
data collected is both consistent and meaningful. Kumar concept. Together, they ensure research instruments
(2014, p. 155) explains that reliability refers to the generate trustworthy findings that can inform scientific
stability or consistency of measurement, while conclusions and practice.
validity refers to the accuracy or truthfulness of the
measurement.
Reliability: Concept and
Importance
Reliability ensures that a measuring instrument produces the same
results under consistent conditions. Kumar (2014, p. 156) emphasises
that reliable data is free from random error. For example, if a stress
questionnaire produces vastly different results for the same person
within a short time frame, it lacks reliability.
Reliable tools allow researchers to differentiate between real changes in
a construct and random noise, making replication and generalisation
possible. Thus, reliability provides the foundation for scientific
credibility.
Types of Reliability (I)

Test–retest reliability Parallel-forms reliability

Test–retest reliability examines the stability of an Parallel-forms reliability involves creating two
instrument across time. A high correlation between equivalent versions of an instrument and correlating
scores from two administrations of the same test scores from both. Kumar (2014, p. 158) notes this
indicates consistency (Kumar, 2014, p. 157). This is method reduces memory effects but requires careful
essential when measuring stable constructs like design of truly equivalent tests.
personality or intelligence.
Types of Reliability (II)
Split-half reliability Inter-rater reliability Internal consistency
reliability
Split-half reliability divides Inter-rater reliability assesses
items within a test into two sets the extent of agreement between Internal consistency
(e.g., odd vs. even items) and different observers recording the reliability, often measured by
compares the results. A high same behaviour. For instance, if Cronbach's alpha, tests whether
correlation suggests that all items two teachers independently rate items in a scale consistently
measure the same construct student engagement similarly, measure the same underlying
consistently (Kumar, 2014, p. reliability is high. This is construct. Kumar stresses this is
158). especially important in vital for attitude scales and
qualitative and observational psychological tests.
research.
Threats to Reliability
Kumar (2014, p. 160) highlights threats that reduce reliability.
Respondent-related issues include fatigue, lack of interest, or
misunderstanding of items, leading to inconsistent responses.

Instrument-related issues include vague wording, poorly designed

scales, or malfunctioning devices that yield fluctuating results.

Situational issues include inconsistent administration conditions—such

as noisy environments or variations in researcher instructions—which
can distort responses. Reliable research requires strict control to reduce
these threats.
Validity: Concept and Importance
Validity addresses whether a measurement instrument truly measures
what it is supposed to. Kumar (2014, p. 161) explains that a tool can be
reliable but invalid. For instance, a clock that always runs 5 minutes slow
is reliable but not valid.
In social and behavioural sciences, validity is crucial because constructs
like "anxiety" or "leadership" are abstract and require careful
operationalisation. Validity ensures results are accurate, interpretable,
and relevant.
Types of Validity (I)

Face validity Content validity

Face validity refers to the degree to which an Content validity examines whether the instrument fully
instrument appears to measure what it claims, based on represents the construct. For example, a depression scale
surface judgement. While not scientific proof, face validity should include items on mood, energy, sleep, and
increases participant acceptance (Kumar, 2014, p. 162). cognition. Missing domains reduce content validity.
Types of Validity (II)

Construct validity Criterion validity

Construct validity determines whether the instrument Criterion validity assesses how well a measure
truly measures the theoretical construct. This requires correlates with an external criterion. Predictive validity
statistical testing and comparisons with related constructs checks whether it forecasts future outcomes (e.g., SAT
(Kumar, 2014, p. 163). For example, an intelligence test predicting college GPA), while concurrent validity checks
should correlate with academic performance but not with agreement with an existing tool (Kumar, 2014, p. 165).
unrelated variables.
Threats to Validity

Selection bias History and maturation effects Testing and instrumentation

Validity can be threatened by selection History effects (external events Other threats include testing effects
bias, when non-random sampling skews influencing outcomes) and maturation (participants improving due to repeated
results (Kumar, 2014, p. 166). effects (natural participant changes exposure) and instrumentation
over time) may also distort findings. changes (shifts in measurement tools
or observers). Controlling these threats
is essential to ensure valid conclusions.
Reliability and Validity
Together
Reliability and validity are distinct but interconnected. Reliability
ensures consistency, while validity ensures accuracy (Kumar, 2014, p.
168).

A measure can be reliable without being valid, but it cannot be valid

unless it is reliable. For example, a broken thermometer may
consistently show 25°C (reliable) but inaccurately (not valid).

Researchers must therefore test both properties before using data for
decision-making or theory building.
Population vs. Sample

Population Sample
A population refers to the entire group of individuals to A sample is the subgroup selected from this population for
whom the researcher wishes to generalise findings study. Sampling is necessary because studying entire
(Shaughnessy et al., 2015, p. 129). For example, all populations is often impractical or impossible.
university students in a country.

The representativeness of the sample determines the accuracy of generalisations. Poorly chosen samples lead to biased or misleading c
Importance of Sampling
• Sampling allows researchers to study smaller groups efficiently while still making valid inferences about larger populations
(Shaughnessy et al., 2015, p. 131).
• A well-chosen sample reduces costs, saves time, and improves feasibility without sacrificing accuracy.
• However, sampling requires careful design—biased samples compromise both internal validity (causal conclusions) and
external validity (generalisability).
Probability Sampling: Simple Random Sampling
In simple random sampling, every member of the population has an
equal chance of being chosen (Shaughnessy et al., 2015, p. 133).

This method minimises bias and ensures representativeness, making it

ideal for quantitative research.

However, it requires a complete list of the population, which is not always

available. It can also be costly and time-consuming in large populations.
Probability Sampling: Stratified and Cluster Sampling

Stratified sampling Cluster sampling

Stratified sampling divides the population into strata Cluster sampling selects intact groups (e.g.,
(e.g., gender, age groups) and randomly samples from classrooms, neighbourhoods) and studies all or some
each, ensuring proportional representation (Shaughnessy individuals within them. This reduces costs for large
et al., 2015, p. 135). populations but may introduce cluster-level bias.

Both techniques enhance representativeness but require careful planning.

Non-Probability Sampling: Convenience and
Purposive

Convenience sampling Purposive sampling

Convenience sampling selects participants who are Purposive sampling selects individuals deliberately
easily available, such as students in a classroom chosen for their relevance to the study. For instance,
(Shaughnessy et al., 2015, p. 139). While practical, it interviewing only teachers about classroom management.
carries high risk of bias. This is useful in qualitative research where depth is
prioritised over generalisability.
Non-Probability Sampling:
Quota and Snowball
Quota sampling Snowball sampling

Quota sampling ensures Snowball sampling recruits

specific proportions of subgroups participants through referrals,
are included (e.g., 50% men, 50% useful for hidden or hard-to-reach
women), though participants are populations like drug users or
not chosen randomly refugees.
(Shaughnessy et al., 2015, p.
141).
While these methods provide access, they limit the ability to generalise
findings confidently.
Sampling Bias
Sampling bias occurs when the sample systematically differs
from the population, reducing representativeness (Shaughnessy
et al., 2015, p. 143).

• Common causes include poor sampling frames, non-random

selection, and non-response bias (certain types of people refusing
• participation).
Sampling bias threatens external validity by producing skewed results
that cannot be generalised beyond the sample.
Sample Size Considerations

↑ £ ⚖
Larger samples Cost factor Balance needed
Reduce sampling error and increase Larger samples are costlier and harder Between practicality and statistical power
representativeness to manage

Determining sample size involves balancing practicality with statistical power (Shaughnessy et al., 2015, p. 145).

The ideal sample size depends on population variability, the precision required, and available resources. Researchers often use
power analysis to estimate appropriate sample size.
Conclusion: Reliability, Validity, and Sampling as
Research Foundations
Reliability, validity, and sampling together form the backbone of research methodology. Reliable and valid instruments
ensure data quality, while sound sampling ensures representativeness.

Kumar (2014, p. 210) and Shaughnessy et al. (2015, p. 147) both stress that neglecting these elements undermines the entire research

By carefully addressing measurement quality and sampling design, researchers produce trustworthy findings that can be
generalised and applied responsibly.