Chapter 6 Item Analysis and Validation

Lesson 6.2 Validation and Validity

________________________________________________________________________

6.2 Validation and Validity

After performing the item analysis and revising the items which need revision, the next step is to
validate the questionnaire. The purpose of validation is to determine the characteristics of the
whole test itself, namely, the validity and reliability of the test. Validation is the process of
collecting and analyzing evidence to support the meaningfulness and usefulness of the test.

Validity- is the extent to which a test measures what it purports to measure or as referring to
the appropriateness, correctness, meaningfulness and usefulness of the specific decisions a
teacher makes based on the test results. These two definitions of validity differ in the sense that
the first definition refers to the test itself while the second refers to the decisions made by the
teacher based on the test. A test is valid when it is aligned with the learning outcome.

There are three main types of evidence that may be collected: content-related evidence of
validity, criterion-related evidence of validity and construct-related evidence of validity.

Content-related evidence of validity refers to the content and format of the instrument. How
appropriate is the content? How comprehensive? Does it logically get at the intended variable?
How adequately does the sample of items or questions represent the content to be assessed?

Criterion-related evidence of validity refers to the relationship between scores obtained using
the instrument and scores obtained using one or more other tests (often called criterion). How
strong is this relationship? How well do such scores estimate present or predict future
performance of a certain type?

Construct-related evidence of validity refers to the nature of the psychological construct or

characteristic being measured by the test. How well does a measure of the construct explain
differences in the behavior of the individuals or their performance on a certain task?

The usual procedure for determining content validity may be described as follows:
• The teacher writes out the objectives of the test based on the Table of Specifications and
then gives these together with the test to at least two (2) experts along with a description
of the intended test takers.
• The experts look at the objectives, read over the items in the test and place a check mark
in front of each question or item that they feel does not measure one or more objectives:
They also place a check mark in front of each objective not assessed by any item in the
test. The teacher then rewrites any item checked and resubmits to the experts and/or
writes new items to cover those objectives not covered by the existing test. This
 continues until the experts approve of all items and also until the experts agree that all
of the objectives are sufficiently covered by the test.

• In order to obtain evidence of criterion-related validity, the teacher usually compares

scores on the test in question with the scores on some other independent criterion test
which presumably has already high validity. For example, if a test is designed to measure
mathematics ability of students and it correlates highly with a standardized mathematics
achievement test (external criterion), then we say we have high criterion-related
evidence of validity. In particular, this type of criterion-related validity is called its
concurrent validity. Another type of criterion-related validity is called predictive validity
wherein the test scores in the instrument are correlated with scores on a later
performance (criterion measure) of the students. For example, the mathematics ability
test constructed by the teacher may be correlated with their later performance in a
Division-wide mathematics achievement test.

• In summary content validity refers to how will the test items reflect the knowledge
actually required for a given topic area (e.g. math). It requires the use of recognized
subject matter experts to evaluate whether test items assess defined outcomes. Does a
pre-employment test measure effectively and comprehensively the abilities required to
perform the job? Does an English grammar test measure effectively the ability to write
good English? Criterion-related validity is also known as concrete validity because
criterion validity refers to a test's correlation with a concrete outcome. In the case of pre-
employment test, the two variables that are compared are test scores and employee
performance.

• There are 2 main types of criterion validity-concurrent validity and predictive validity.
Concurrent validity refers to a comparison between the measure in question and an
outcome assessed at the same time. An example of concurrent validity is a comparison
of the score with NAT Math exam with course grades in Grade 12 Math. In predictive
validity, we ask this question: Do the scores in NAT Math exam predict the Math grade in
Grade 12?

6.3. Reliability
Reliability -refers to the consistency of the scores obtained - how consistent they are for each
individual from one administration of an instrument to another and from one set of items to
another.

Types of Reliability
1. Internal Reliability- assesses the consistency of results across items within a test.
• Split-half method is effective for large questionnaires or test with several items
measuring the same construct. To improve the reliability of the test employing this
method, items with low correlations are either removed or modified
• Cronbach alpha is a better than split-half because it gives the average of all the split-half
reliabilities. It measures how well items in a scale (1-Strongly disagree to 5-strongly
agree) correlate with one another (Salvucci et al., 1997)
• Kuder-Richardson 20/21 formula is applicable to dichotomous items (0/1). Items of the
test are scored 1 if marked correctly, otherwise zero.
2. External Reliability- gauges the extent to which a measure varies from one use to another.
It measures how consistently a test produces similar results over repeated administrations or
under different conditions.

Reliability and validity are related concepts. If an instrument is unreliable, it cannot get valid
outcomes. As reliability improves, validity may improve (or it may not). However, if an instrument
is shown scientifically to be valid then it is almost certain that it is also reliable.

Predictive validity compares the question with an outcome assessed at a later time. An example
of predictive validity is a comparison of scores in the National Achievement Test (NAT) with first
semester grade point average (GPA) in college. Do NAT scores predict college performance?
Construct validity refers to the ability of a test to measure what it is supposed to measure. As
researcher, you intend to measure depression but you actually measure anxiety so your research
gets compromised.

The following table is a standard followed almost universally in educational test and
measurement

Reliability Interpretation
0.90 and above Excellent reliability, at the level of the best standardized tests
0.80-0.90 Very good for a classroom test
0.70-.0.80 Good for a classroom test; in the range of most. There are probably a
few items which could be improved.
0.60-0.70 Somewhat low. This test needs to be supplemented by other measures
(e.g., more tests) to determine grades. There are probably some items
which could be improved.
0.50-0.60 Suggests need for revision of test, unless it is quite short (ten or fewer
items). The test definitely needs to be supplemented by other measures
(e.g., more tests) for grading.
0.50 and below Questionable reliability. This test should not contribute heavily to the
course grade, and it needs revision.