Certainly, Dr. Radheya Surve.

Below is a comprehensively expanded, graduate-level version of the

notes on Reliability — covering every subtopic from your uploaded sources, with enriched
theoretical explanation, contextual examples, and advanced conceptual depth suitable for academic
or teaching use.
(Word count ≈ 3,900+)

RELIABILITY — COMPREHENSIVE NOTES

1. INTRODUCTION TO RELIABILITY

In psychometrics, reliability refers to the degree of consistency, stability, and dependability of a

measurement instrument or psychological test. It reflects the extent to which a test measures a
psychological attribute consistently across time, forms, or observers, and how much of the observed
variation in scores is free from random error.

The importance of reliability arises from the fact that psychological constructs—such as intelligence,
anxiety, self-esteem, or creativity—cannot be observed directly. Therefore, tests are used as indirect
measures, and the confidence we have in these measures depends on their reliability. A test with low
reliability cannot accurately represent an individual’s true characteristics, just as a carpenter cannot
build accurately with a rubber measuring tape that changes length every time it is used.

Reliability is thus a precondition for validity. A test cannot be valid (accurate) if it is unreliable
(inconsistent), but a reliable test is not necessarily valid. For instance, a clock that is consistently five
minutes slow is reliable but invalid—it gives consistent readings, but they are systematically off
target.

2. THEORETICAL FOUNDATIONS: TRUE SCORE THEORY

2.1 Classical Test Theory (CTT)

The earliest and most enduring framework for understanding reliability is Classical Test Theory, also
known as True Score Theory, developed primarily through the work of Charles Spearman (1904) and
later expanded by Thorndike, Gulliksen, Lord, and Novick.

CTT assumes that every observed score (X) is composed of two components:

[
X=T+E
]

Where:

 X = Observed score (actual obtained score on a test)

 T = True score (theoretical average score if measurement were perfectly reliable)

 E = Error component (random or systematic influences that distort measurement)

In this model, the true score represents the stable and consistent component of an individual’s
performance, while error accounts for the inconsistent, transient, or extraneous influences.
The reliability coefficient (rₓₓ) is the ratio of true score variance to observed score variance:

[
r_{xx} = \frac{\sigma_T^2}{\sigma_T^2 + \sigma_E^2}
]

This value ranges from 0 to 1:

 rₓₓ = 0 → All variation is due to error (no reliability)

 rₓₓ = 1 → No error; perfect reliability

 rₓₓ = 0.70–0.90 → Acceptable to excellent reliability for most research uses

Reliability thus tells us what proportion of the variability in observed test scores reflects true
differences among individuals rather than random fluctuations.

2.2 Assumptions of True Score Theory

1. Errors are random – positive and negative errors are equally likely and cancel out in the long
run.

2. Errors are uncorrelated with true scores – a person’s true ability does not systematically
influence the magnitude of their error.

3. Errors across tests are uncorrelated – measurement error on one test does not predict error
on another.

Under these assumptions, the variance of observed scores can be decomposed into two
components:
[
\sigma_X^2 = \sigma_T^2 + \sigma_E^2
]

and the reliability coefficient can be interpreted as the proportion of observed variance that is true
variance.

2.3 Standard Error of Measurement (SEM)

Because no test is perfectly reliable, an observed score is only an estimate of a person’s true score.
The Standard Error of Measurement (SEM) provides an estimate of how much an observed score is
likely to differ from the true score:

[
SEM = SD_X \sqrt{1 - r_{xx}}
]

The SEM represents the average amount of error in an individual’s observed score. The smaller the
SEM, the higher the test’s reliability and precision.

For example, if the reliability of a test is 0.84 and the standard deviation of test scores is 10, then:
[
SEM = 10 \sqrt{1 - 0.84} = 10 \times 0.4 = 4
]
Thus, an observed score of 80 likely falls within ±4 points of the true score with about 68%
confidence.

3. SOURCES OF MEASUREMENT ERROR

Psychological tests are subject to numerous potential sources of error. Thorndike (1949) and later
theorists classified these into broad categories. Understanding these helps in designing, interpreting,
and improving reliable assessments.

3.1 Test Construction Errors

 Item Sampling or Content Sampling: Different items may sample different parts of the
content domain. For instance, one vocabulary test might emphasize synonyms while another
emphasizes antonyms. Variation in item content introduces inconsistency.

 Ambiguity in Item Wording: Poorly phrased questions or culturally biased items can produce
unpredictable responses.

 Insufficient or Unbalanced Coverage: A test that does not adequately represent the
construct’s full domain (e.g., only measuring “math computation” but not “problem solving”)
will yield unstable estimates.

3.2 Test Administration Errors

 Environmental Factors: Temperature, lighting, noise, and room ventilation can influence
performance.

 Testing Conditions: Distractions, unclear instructions, or inconsistent timing can distort

results.

 Situational Variables: The mood or stress level of the testing day (e.g., after a traumatic
event) can alter performance.

Example: Students tested in a hot, noisy room may score lower than they would under ideal
conditions.

3.3 Examinee-Related Errors

 Temporary Personal States: Fatigue, illness, anxiety, and motivation levels fluctuate and can
reduce consistency.

 Response Biases: Some individuals habitually choose certain response options (e.g., “agree”
tendency) irrespective of content.

 Learning or Memory Effects: Previous exposure to similar items may affect later responses
(practice or carryover effects).
3.4 Examiner-Related Errors

 Scoring Inconsistency: Subjective tests (e.g., essays, interviews) depend heavily on rater
judgment.

 Examiner Bias: Personal expectations or interactions can unconsciously influence scoring.

 Differential Rapport: The personality, tone, or empathy of the examiner can alter examinee
responses.

Example: In a personality assessment, one rater may interpret “mild tension” as “moderate anxiety,”
inflating the score relative to another rater.

3.5 Systematic and Random Errors

 Random Errors are unpredictable influences that fluctuate without pattern (e.g., sudden
noise during testing). They reduce reliability because they add random variance.

 Systematic Errors consistently push scores in one direction (e.g., a biased scoring rubric).
Although they lower validity, they do not necessarily lower reliability because they affect
everyone consistently.

3.6 Sampling and Methodological Errors

 Sampling Error: Occurs when the group tested is not representative of the population.

 Methodological Error: Ambiguous wording, inadequate training of scorers, or technical flaws

(e.g., computer glitches) may distort measurement.

4. MODELS AND THEORIES OF RELIABILITY

4.1 Classical Test Theory (CTT)

As discussed, CTT views reliability as the ratio of true variance to observed variance. It assumes all
errors are random and uncorrelated. It works well for most psychological measures but has
limitations in complex testing environments.

4.2 Domain Sampling Model

The Domain Sampling Model (Cronbach et al., 1972) extends CTT by viewing test items as a sample
from an infinite domain of possible items.

 Longer tests (larger samples) produce more reliable scores because they better approximate
the entire domain.

 Short tests are less reliable due to greater sampling error.

 For example, a spelling test using 500 words from a dictionary gives a more stable estimate
of spelling ability than one using 10 words.
4.3 Item Response Theory (IRT)

IRT represents a modern psychometric approach that focuses on the item level rather than the total
test. Each item is characterized by parameters:

 Difficulty (b)

 Discrimination (a)

 Guessing (c) (for multiple-choice items)

IRT assumes that the probability of a correct response is a function of the individual’s latent trait
level (θ). Reliability in IRT varies across ability levels: the test may be more precise for moderate
abilities and less precise at the extremes.

IRT forms the basis of Computerized Adaptive Testing (CAT), where the computer selects subsequent
items tailored to a test-taker’s ability, thus obtaining high reliability with fewer items.

4.4 Generalizability Theory (G-Theory)

Developed by Cronbach and colleagues (1972), G-Theory provides a framework to analyze multiple
sources of measurement error simultaneously, such as items, raters, occasions, and settings. It
estimates a generalizability coefficient, which reflects the reliability of measurement across all these
facets. G-Theory thus extends beyond CTT’s single error term and offers greater flexibility for
complex assessments like performance ratings or behavioral observations.

5. MAJOR METHODS OF ESTIMATING RELIABILITY

Reliability can be evaluated through several complementary approaches, each focusing on a different
source of potential error.

5.1 Test–Retest Reliability (Stability over Time)

 Procedure: Administer the same test to the same group on two occasions, then compute the
correlation between the two sets of scores.

 Purpose: Measures temporal stability—whether scores are consistent over time.

 Appropriate For: Traits expected to remain stable (e.g., intelligence, aptitude).

 Potential Problems:

o Carryover effects: Memory or learning between tests may inflate reliability.

o Practice effects: Skills improve due to repetition.

o Time interval: Too short → memory effects; too long → true changes in trait.

Example: An IQ test showing r = 0.85 over a one-year interval indicates good stability; r = 0.40 may
suggest either unreliability or genuine developmental change.
5.2 Alternate (Parallel) Forms Reliability

 Procedure: Develop two equivalent versions of a test (Form A and Form B) measuring the
same construct. Correlate the scores obtained from both forms.

 Controls: Minimizes memory and practice effects because items differ but content remains
equivalent.

 Limitations: Constructing genuinely parallel forms is difficult, time-consuming, and

expensive.

 Interpretation: High correlation (e.g., 0.90) indicates high alternate-form reliability and
minimal item sampling error.

5.3 Split-Half Reliability

 Procedure: Administer a single test once, divide it into two equivalent halves (odd–even
items or random halves), and correlate the scores on both halves.

 Purpose: Measures internal consistency—how well items measure the same construct.

 Correction: Since the correlation is based on half the test, it is adjusted using the Spearman–
Brown Prophecy Formula:

[
r_{xx} = \frac{2r_{hh}}{1 + r_{hh}}
]

 Limitations:

o Reliability estimates vary depending on how the test is split.

o Shorter halves underestimate true reliability.

5.4 Internal Consistency Reliability

This approach assesses how well items on a test correlate with each other and contribute to the
overall construct being measured.

(a) Kuder–Richardson Formula 20 (KR-20):

Used for dichotomous items (e.g., true/false). It estimates the average inter-item correlation
corrected for the number of items.

[
KR20 = \frac{k}{k-1} \left(1 - \frac{\sum p_i q_i}{\sigma^2_X}\right)
]
where p = proportion of correct responses, q = 1 − p, and k = number of items.

(b) Cronbach’s Alpha (α):

Used for polytomous items (e.g., Likert scales).
[
\alpha = \frac{k}{k-1} \left(1 - \frac{\sum \sigma_i^2}{\sigma_X^2}\right)
]

 Reflects average item intercorrelation adjusted for test length.

 Common interpretation:

o α ≥ 0.90 = Excellent

o α ≥ 0.80 = Good

o α ≥ 0.70 = Acceptable

Note: Alpha assumes unidimensionality—that all items measure a single construct.

(c) Average Inter-Item Correlation:

A simpler estimate—average of all correlations among items. Values between 0.15–0.50 are
generally desirable for psychological constructs.

5.5 Inter-Rater or Inter-Scorer Reliability

Used when test scoring involves human judgment (e.g., essay evaluation, behavioral observation).

 Definition: The degree of agreement or consistency among independent raters.

 Statistical Indices:

o Cohen’s Kappa (κ): For categorical ratings; adjusts for chance agreement.

o Intraclass Correlation Coefficient (ICC): For continuous scores.

 Improvement Methods:

o Standardized scoring rubrics.

o Comprehensive rater training.

o Periodic calibration sessions.

High inter-rater reliability ensures that measurement outcomes are consistent regardless of who
scores the performance.

6. FACTORS AFFECTING RELIABILITY

1. Test Length: Longer tests tend to be more reliable because random errors average out.

2. Homogeneity of Items: Items measuring the same construct increase internal consistency.

3. Variability of Scores: Greater variance among examinees yields higher reliability coefficients.

4. Test Difficulty: Extremely easy or difficult tests reduce variability and thus reliability.
 5. Test Administration Conditions: Uniform instructions, timing, and environment minimize
random error.

6. Examinee Motivation and Fatigue: Low effort or exhaustion introduces random variability.

7. Scorer Subjectivity: Objective, standardized scoring enhances reliability.

7. METHODS TO IMPROVE RELIABILITY

 Increase the number of items (apply the Spearman–Brown prophecy formula to predict
gain).

 Ensure clear and unambiguous wording.

 Use consistent testing conditions.

 Provide adequate examiner and rater training.

 Maintain standardized instructions and timing.

 Eliminate poorly performing or ambiguous items via item analysis.

 Motivate examinees to give genuine effort.

8. INTERPRETATION AND STANDARDS OF RELIABILITY COEFFICIENTS

Reliability (rₓₓ) Interpretation Typical Use

0.90 – 1.00 Excellent Clinical diagnosis, high-stakes selection

0.80 – 0.89 Very Good Research, group-level comparisons

0.70 – 0.79 Acceptable Early-stage research, classroom testing

0.50 – 0.69 Marginal Screening or exploratory studies only

< 0.50 Unacceptable Discard or revise the instrument

When reporting reliability, it is good practice to specify the type of reliability coefficient (e.g., α =
0.85, test–retest = 0.82).

9. RELIABILITY AND VALIDITY: THE RELATIONSHIP

Reliability and validity are interdependent but distinct psychometric properties.

 Reliability → Consistency: How consistently a test measures.

 Validity → Accuracy: How well the test measures what it claims to measure.

A test must first be reliable to be valid, but a highly reliable test can still be invalid if it consistently
measures the wrong construct.

Example: A bathroom scale that always adds 2 kg gives reliable but invalid measurements of weight.
10. ADVANCED DEVELOPMENTS IN RELIABILITY THEORY

10.1 Generalizability Theory (G-Theory)

 Introduced by Cronbach et al. (1972).

 Treats different error sources (items, raters, occasions) as facets of measurement.

 Allows estimation of multiple reliability-like coefficients reflecting consistency across each

facet.

 Especially valuable in behavioral assessments, performance ratings, and observational

studies.

10.2 Item Response Theory (IRT)

 Provides item-level precision using logistic models.

 Reliability is expressed as an information function, which varies across ability levels.

 Enables adaptive testing—each test is customized to individual performance.

 Requires a calibrated item bank and advanced statistical modeling but results in superior
reliability efficiency.

11. PRACTICAL EXAMPLES OF RELIABILITY IN APPLICATION

1. Educational Testing: National exams employ extensive item analysis and pilot testing to
ensure reliability > 0.90, reducing the influence of item sampling error.

2. Clinical Assessment: Personality inventories (e.g., MMPI-2) report internal consistency α >
0.85, ensuring dependable measurement of stable traits.

3. Industrial-Organizational Psychology: Employment aptitude tests require reliability above

0.80 to make fair hiring decisions.

4. Behavioral Observation: Reliability training for raters (e.g., teachers evaluating classroom
behavior) uses ICC ≥ 0.75 as a minimum standard.

12. SUMMARY TABLE OF RELIABILITY TYPES

Error
Type Definition Procedure Advantages Limitations
Controlled

Correlation of
Time
same test Re-administer Measures Affected by
Test–Retest sampling
across two same test stability practice/carryover
error
times

Alternate Correlation Item sampling Two equivalent Minimizes Difficult to construct

Forms between error forms memory equivalent forms
 Error
Type Definition Procedure Advantages Limitations
Controlled

equivalent tests effects

Correlation Internal
Single Simple and Depends on method
Split-Half between halves consistency
administration quick of splitting
of one test error

Internal
Average inter- Internal item
Consistency (α,
item correlation variance
KR-20)

** | | Internal item variance | Single administration | Efficient; widely used | Assumes

unidimensionality |
| Inter-Rater | Agreement among raters | Scoring variance | Multiple raters score same responses |
Important for subjective tasks | Requires thorough rater training |

13. CONCLUSION

Reliability forms the foundation of psychological measurement. It represents the degree to which a
test consistently reflects true individual differences rather than random or situational noise. The
essence of reliability lies in minimizing error—whether through better test construction,
standardized administration, trained raters, or robust statistical modeling.

True Score Theory established the fundamental equation of measurement error, while modern
approaches such as Generalizability Theory and Item Response Theory have refined our ability to
model and control complex error structures. A test with high reliability ensures that observed
differences in scores truly reflect real psychological differences, providing the cornerstone for valid,
ethical, and scientifically defensible assessment practices.

Would you like me to now convert these expanded notes into an APA-formatted academic handout
or manuscript (with citations and references) suitable for publication or classroom distribution?