Dr. K. A.
Korb, UniJos 1/28/2
Introduction
• Tests are used to:
a) Make important life decisions about individuals
Test Development: b) Conduct research
• For both purposes, a good test is vital to ensure life
Scale Construction decisions are helpful and research conclusions are
truthful
• A poorly designed test leads to test scores that are
Dr. K. A. Korb inaccurate, which will lead to wrong decisions
UniJos • Developing a good test is VERY important.
• Developing a good test takes a lot of time, thought,
and preparation
Process of Developing a Test Test Conceptualization
• Test Conceptualization: Idea for a test • Step 1: Describe purpose and rationale for test
– What the test will measure and for what purpose will
• Test Construction: Drafting test items the test will be used?
• Test Tryout • Step 2: Describe the target population for the test
• Item Analysis • Step 3: Clearly define the key variable of interest
• Test Revision • Step 4: Create item specifications
• Step 5: Choose item format
• Step 6: Specify administration and scoring
procedures
Dr. K. A. Korb, UniJos 1/28/2
Test Conceptualization: Test Conceptualization:
Purpose and Rationale Target Population
• What is the test designed to measure? • Relevant characteristics to consider:
• What is the purpose for the test?
– Age
• Who will use the test?
– Educational Status
• Who will take the test?
• How will the test be administered? – Language (which language and level of language
ability)
• What is the ideal format for the test given the test
taker characteristics and time/financial considerations? – Literacy Level
• Is there any potential harm that can result from test – Disabilities
administration?
• How will test scores be meaningful?
Test Conceptualization: Test Conceptualization:
Define Variable Item Specifications
• Develop by referencing: • List of major content areas to be included in
– Theory the test
– Empirical Literature – Also can include number of items
– Cultural Definitions • For clinical tests, the DSM-V (or other
– DSM-V or other diagnostic criteria guidelines for diagnosis) can provide
guidelines for item specifications
• Otherwise, develop item specifications from
theory or definitions of variable
Dr. K. A. Korb, UniJos 1/28/2
Separation Anxiety Disorder (SAD) Test Conceptualization
Diagnostic Criteria (DSM-IV-TR) Types of Instruments
• Evidenced by three or more of the following: • Self-Report: Participants report their own demographic
– Recurrent excessive distress when separation from home or major characteristics, attitudes, beliefs, knowledge, feelings,
attachment figures occurs or is anticipated and behavior
– Persistent and excessive worry about losing or possible harm to major • Can take the form of Questionnaire or Interview
attachment figures
• Performance Assessment: Directly assess performance
– Persistent and excessive worry that an unpleasant event will lead to
separation from a major attachment figure (e.g., kidnapping) on a contrived task
– Persistent reluctance or refusal to go to school or elsewhere because • Observation Schedule: Researchers observe participants’
of fear of separation behavior under specified conditions
– Persistently and excessively fearful or reluctant to be alone without • Checklist: Identify the frequency or presence of
major attachment figures behaviors or characteristics
– Persistent reluctance or refusal to go to sleep without being near a
major attachment figure or to sleep away from home. • Examination/Test: Test participants’ knowledge of a
– Repeated nightmares with the theme of separation topic
– Repeated complaints of physical symptoms (e.g., headaches, nausea)
when separation from major attachment figures occurs or is
anticipated
Self-Report Questionnaire
• Self-Report: Participants are asked to report • Advantages:
their own attitudes, beliefs, knowledge, – Easy to administer and score
feelings, and behavior – Can be administered to larger numbers of test-
– Self-report can be either a questionnaire or takers
interview • Disadvantages:
– Attitudes, beliefs, values, interests, knowledge, – The data is typically not as in-depth as an
feelings, and some types of behavior are typically interview
best assessed by self-report for most target
– Requires a high level of literacy
populations
Dr. K. A. Korb, UniJos 1/28/2
Interview Types of Items for Self-Report
• Advantages: • Open-ended: Test-takers give their own response
– Gathers rich, in-depth information • Advantages:
– More in-depth information
– Can ask test-taker to expand on unclear responses • Disadvantages:
– Can be used with any test-taker (e.g., cannot read) – Test-takers need to have good language/literacy skills
• Disadvantages: – Test users have to categorize and code responses
(scoring is subjective)
– Takes time – Subject to inter-rater reliability issues
– Typically subject to issues of inter-rater reliability – Test-takers’ responses may not be understandable
(Questionnaires)
– Test-takers often skip these items on questionnaires
Types of Items for Self-Report Sample Items
• Close-ended: Test-takers select their response from • Open-Ended: What causes you anxiety?
given categories on the test
• Advantages: • Close-Ended: What causes you anxiety?
– Easy and objective scoring a) Public speaking
– Response is clear to the test user
• Disadvantages: b) Large crowds
– Test developer must anticipate most common responses c) Examinations
– Test developer must develop response categories that the
target population understands d) Driving
– Test developer must develop response categories that are
exhaustive (all participants have a place to appropriately
respond)
Dr. K. A. Korb, UniJos 1/28/2
1 2 3 4 5
Test Item Formats
Strongly 2 3 4 Strongly
Disagree Agree
• Rating Scale: Grouping of words, statements,
or symbols on which judgments concerning Not at all
true
2 3 4 Very true
the strength of a particular trait, attitude, or
Not at all 2 3 4 Very
emotion are indicated by the test-taker. important important
Not at all 2 3 4 Extremely
Strongly Strongly
Disagree Disagree Neutral Agree Agree
1 2 3 4 5 6
1 2 3 4 5
0 1 2 3 4
Test Item Formats
Never Rarely Sometimes Most of Always
the time
• Paired Comparisons: Test-takers are
Never Yearly Monthly Weekly Daily
presented with pairs of stimuli that they are
asked to compare and select one.
Rarely Monthly Weekly 1-2 times Every • Useful to place test-takers into categories
a week Day
• Avoids the problem of test-takers agreeing (or
1-29 30 minutes 1-2 hours 3 hours
0
minutes minutes – 1 hour or more
disagreeing) to every category
Dr. K. A. Korb, UniJos 1/28/2
Paired Comparisons Example: 5 Love Test Format:
Languages Questionnaire Checklists
1. A) I like to receive notes of affirmation • Checklists identify the frequency or presence
B) I like to be hugged of behaviors or characteristics
2. A) I like to spend one-to-one time with a person – Adjective checklist: Tick list of adjectives that
who is special to me. apply to yourself (or who is being rated)
• Parents’ perceptions of children
B) I feel loved when someone gives practical help
• Clients’ perceptions of therapist
to me.
• Employees’ perceptions of Managers
3. A) I like it when people give me gifts. – Psychological Processing Checklist–Revised (PPC–
B) I like leisurely visits with friends and loved R)(learning difficulties)
ones. – PTSD Checklist (PCL-5)
Test Format:
Adjective Checklist
Checklists
1. Absentminded 8. Adaptable [Link] • Hare Psychopathy Checklist uses a semi-
structured interview to identify the presence or
2. Adventurous 9. Assured [Link] absence of psychopathological symptoms such
3. Articulate [Link] [Link] as:
– Grandiosity
4. Acquiescent [Link] [Link] – Pathological lying
5. Artistic [Link] [Link] – Cunning and manipulating
– Lack of remorse
6. Active [Link] [Link] – Impulsiveness
7. Assertive [Link] [Link] – Failure to accept responsibility for own actions
– Lack of realistic long-term goals
Dr. K. A. Korb, UniJos 1/28/2
Test Format: Test Format:
Checklists Performance Assessment
Indicate if you have personally done in any behavior within the • Directly assess performance on tasks
past three years that you have been in school.
1. Copy your continuous assessment answer from another
• Professional performance
student. YES NO – Practicals in Medical Boards
2. Read another student’s answers on their exam script during – PSY 804
the exam. YES NO • Neuropsychological performance (e.g., memory,
3. Ask another student for an answer during the exam. YES NO attention)
4. Write answers on an exam script beforehand to carry it into – Doors and People (Memory)
the hall. YES NO
– Wechsler Memory Scale
5. Trade your script with a student in the exam so they write
your answer. YES NO – Iowa Gambling Task (real-life decision-making)
6. Write answers or notes on your body, clothing, or personal – Stroop Task (reaction time)
belongings. YES NO
7. Use your handset to get answers. YES NO
Test Format:
Observation
• Measures behaviors only
• Examples
– Parent-Child Interaction Assessment-II (PCIA)
– Parent-Child Early Relational Assessment
– Dyadic Parent-Child Interaction Coding System-II
Dr. K. A. Korb, UniJos 1/28/2
Test Format: Item Construction
Observation Developing Good Items
• Consider: • Use clear words so all participants understand
– Where and when will the behaviors be observed? • Be precise and specific in the wording of each
– What possible situation will the test-taker will be
item so different test-takers do not interpret the
item differently
exposed to?
• Ensure the item is not biased or leading toward
– How long will test-takers be observed? one response.
– How will behaviors be recorded? • Each item should have only one point (avoid
• Duration: Length of time that behaviors occurs double-barreled items)
• Frequency Count • Ensure that each test-taker has a place to
• Interval: Set a stopwatch for a specific interval of time, respond
then record what behavior the test-taker is engaged in
Administration and Scoring Guidelines Test Try-Out
• Standardized assessment requires very clear • Try items out on people similar in critical
administration guidelines so all participants respects to the people for whom the test was
complete the assessment under similar designed
conditions • Sample size: No fewer than 5 participants
– Critical for performance and observation (preferably at least 10 participants) for every 1
assessments item on the test
• Scoring guidelines should provide specific • Conduct under conditions as identical as
score interpretations based on empirical possible to conditions under which the final
research test will be administered