Reliability

Theories of reliability
1) Classical Test Theory: Classical test theory of reliability was developed by Spearman. It
assumes that any observed score is equal to the true score plus the error score. According to
this theory, each person has a true score which would be obtained if there are no errors in
measurement. But, this is an ideal situation. Errors in measurement are common and can
result from faults in the measuring instruments as well as the characteristics of individuals or
the situations which can affect test score. It further states that errors in measurement are
not systematic but rather are random. This means that the causes of errors in measurement
are so varied and complex that these errors act like random variables. These errors can be
positive or negative and they are not correlated with true scores.

2) Generalizability Theory: This is another alternative approach for measuring the consistency
of test scores. According to this theory, the factors that determine the amount of difference
in measurement are distinct for internal consistency methods than for test retest. It
identifies systematic as well as random sources of inconsistency that can contribute to the
error score. It acknowledges that apart from the random sources of error, there are also
certain specific systematic sources of inconsistency in measurement. According to this
theory, the test scores can differ based on when and how the test was taken, and this will
further determine the generalizability of the test scores and the meaning of the test
reliability.

What is Reliability?
Reliability is an essential part of research study. It determines the accuracy and precision of the test,
which are two important components of conducting a scientific research. Reliability refers to the
consistency of scores. This means, that a test is said to be reliable when consistent scores can be
produced over a period of time. Another measure of reliability is based on the correlation between
the scores obtained on different set of items of the same test.

This points us to the two main components of reliability:

1) Temporal Stability: This refers to consistency of scores obtained from a single test on two
different occasions. Hence, the consistency of scores based on testing and retesting is called
temporal stability. The correlation coefficient of temporal stability is called the coefficient of
stability.
2) Internal stability: This refers to the scores obtained from two equivalent sets of items of
single test after a single administration. The correlation coefficient of internal consistency is
called alpha coefficient.

For any statistical measurement to be considered as reliable, it must indicate both coefficient of
stability and coefficient of alpha.

Characteristics of Reliability
1) Reliability involves consistency. As discussed above, reliability is centred on the concept of
consistency, both related to production of consistent scores across a period of time as well
as the consistency of correlation between the items of the test.
 2) Reliability is concerned with the test scores not with the test itself. Any test may have a
number of reliabilities based on whom it is being conducted on or where it is being
conducted. Thus, reliability is centred on the scores of the test.
3) Reliability does not always translate to validity . It is possible that tests scores are
consistent, and thus reliable but this does not necessarily suggest that the test is valid or
measuring what it is supposed to measure.

Types of Reliability
1) Test Retest Reliability
Test-retest reliability measures the consistency of results when you repeat the same test on
the same sample at a different point in time. You use it when you are measuring something
that you expect to stay constant in your sample. Many factors can influence your results at
different points in time: for example, respondents might experience different moods, or
external conditions might affect their ability to respond accurately.
Test-retest reliability can be used to assess how well a method resists these factors over
time. The smaller the difference between the two sets of results, the higher the test-retest
reliability. To measure test-retest reliability, you conduct the same test on the same group of
people at two different points in time. Then you calculate the correlation between the two
sets of results.

2) Interrater Reliability
Interrater reliability (also called inter observer reliability) measures the degree of agreement
between different people observing or assessing the same thing. You use it when data is
collected by researchers assigning ratings, scores or categories to one or more variables.
People are subjective, and interrater reliability takes this into account to suggest that
different observers’ perceptions of situations and phenomena naturally differ. Reliable
research aims to minimize subjectivity as much as possible so that a different researcher
could replicate the same results. At the same time it allows in identifying and avoiding any
biases that may be reflected by a single researcher, making the study more objective.
Thus, multiple researchers involved in data collection or analysis can ensure in keeping a
check on one another.
To measure interrater reliability, different researchers conduct the same measurement or
observation on the same sample. Then you calculate the correlation between their different
sets of results. If all the researchers give similar ratings, the test has high interrater
reliability.

3) Parallel forms reliability

Parallel forms reliability measures the correlation between two equivalent versions of a test.
You use it when you have two different assessment tools or sets of questions designed
to measure the same thing. The most common way to measure parallel forms reliability is to
produce a large set of questions to evaluate the same thing, then divide these randomly into
two question sets.
The measurement of this reliability involves a same group of respondents to answer both
sets, and you calculate the correlation between the results. High correlation between the
two indicates high parallel forms reliability.
4) Internal Consistency
Internal consistency assesses the correlation between multiple items in a test that are
intended to measure the same construct. You can calculate internal consistency without
repeating the test or involving other researchers, so it’s a good way of assessing reliability
when you only have one data set.
This form of reliability is important because when you devise a set of questions or ratings
that will be combined into an overall score, you have to make sure that all of the items really
do reflect the same thing. If responses to different items contradict one another, the test
might be unreliable.
Two common methods are used to measure internal consistency.
Average inter-item correlation: For a set of measures designed to assess the same
construct, you calculate the correlation between the results of all possible pairs of items and
then calculate the average.
Split-half reliability: You randomly split a set of measures into two sets. After testing the
entire set on the respondents, you calculate the correlation between the two sets of
responses.