RESEARCH INSTRUMENTS
gibson manhuru 8/20/13
Data collection methods
a) Observation…
Is watching people, programs, events,
communities, etc.
Involves all 5 senses: sight, hearing,
smell, touch, and taste
observation includes more than just
“seeing”
gibson manhuru 8/20/13
Observation is used
To provide information about real-life
situations and circumstances
To assess what is happening
Because you cannot rely on participants’
willingness and ability to furnish
information
gibson manhuru 8/20/13
When is observation useful?
When you want direct information
When you are trying to understand an
ongoing behavior, process, unfolding
situation, or event
When there is physical evidence, products,
or outcomes that can be readily seen
When written or other data collection
methods seem inappropriate
gibson manhuru 8/20/13
Observations
Advantages Disadvantages
Most direct measure of May require training
behavior Observer’s presence may
Provides direct create artificial situation
information Potential for bias
Easy to complete, saves Potential to overlook
time meaningful aspects
Can be used in natural Potential for
misinterpretation
or experimental settings
Difficult to analyze
gibson manhuru 8/20/13
Observation – Purpose, benefits
Can see things in their natural context
Can see things that may escape conscious
awareness, things that are not seen by others
Can discover things no else has ever really
paid attention to, things that are taken for
granted
gibson manhuru 8/20/13
Cont’d
Can learn about things people may be
unwilling to talk about
Inconspicuous
– least potential for
generating observer effects
Least intrusive of all methods
Can be totally creative – has flexibility to
yield insight into new realities or new ways
of looking at old realities
gibson manhuru 8/20/13
Observation – Limitations
Potential for bias and effect of culture on what
1.
you observe and interpret
Reliability
2.
Ease of categorization
Usually you do not rely on observation alone;
combine your observations with another method
to provide a more thorough account of your
program.
gibson manhuru 8/20/13
Observation – Ethical issues
Unobtrusiveness is its greatest strength;
also potential for abuse in invasion of
privacy
Can venture into places and gather data
almost anywhere
Covert – overt
Always
consider ethics and human
subjects protection.
gibson manhuru 8/20/13
Continuing…….
Structured -looking for
Unstructured-looking at
Observing what does not happen may be
as important as observing what does
happen.
gibson manhuru 8/20/13
Steps in planning for observation
Determine who/what will be observed.
Determine
aspects that will be observed
(characteristics, attributes, behaviors, etc.).
Determine where and when observations will be
made.
Develop the observation record sheet.
gibson manhuru 8/20/13
Cont’d
Pilot test the observation record sheet.
Train
the observers and have them
practice.
Collect the information.
Analyze
and interpret the collected
information.
Write up and use your findings.
gibson manhuru 8/20/13
b) interviews
Interviewing is:
Talking and listening to people
Verbally asking program participants the
program evaluation questions and hearing
the participant’s point of view in his or her
own words.
gibson manhuru 8/20/13
Cont’d
Interviews can be either structured or
unstructured, in person or over the
telephone.
Done face-to-face or over the phone
Individual; group
gibson manhuru 8/20/13
Interviews are useful…
When the subject is sensitive
When people are likely to be inhibited in
speaking about the topic in front of others
When people have a low reading ability
When bringing a group of people together
is difficult (e.g., in rural areas)
gibson manhuru 8/20/13
Interviews
Verbally asking program participants the
program evaluation questions and hearing
the participant’s point of view in his or her
own words.
Interviews can be either structured or
unstructured, in person or over the
telephone.
gibson manhuru 8/20/13
Interviews
Advantages
Disadvantages
deep and free costly in time and
response personnel
flexible, adaptable requires skill
glimpse into may be difficult to
respondent’s tone, summarize responses
Gestures possible biases:
ability to probe, interviewer, respondent,
follow-up situation
gibson manhuru 8/20/13
Interview type: structured
Uses script and questionnaire
No flexibility in wording or order of
questions
Closed response option
Open response option
gibson manhuru 8/20/13
Type: guided
Outline of topics or issues to cover
May vary wording or order of questions
Fairly conversational and informal
gibson manhuru 8/20/13
Type: Conversational interview
May not know that an interview is taking place
Spontaneous
Questions emerge from the situation and what
is said
Topics or questions are not predetermined
Individualized and relevant to situation
gibson manhuru 8/20/13
Interviewing tips
Keep language pitched to that of respondent
Avoid long questions
Create comfort
Establish time frame for interview
Avoid leading questions
Sequence topics
Be respectful
Listen carefully
gibson manhuru 8/20/13
Questionnaires are
Data collection instruments used to collect
standardized information that can be
expressed numerically or through short
answers
Basic instruments of surveys and structured
interviews
gibson manhuru 8/20/13
Cont’d
Appropriate when…
you want information from many people
you have some understanding of the
situation and can ask meaningful
questions
information
is sensitive or private −
anonymous questionnaires may reduce
bias
gibson manhuru 8/20/13
Questionnaires
Advantages and disadvantages
Advantages Disadvantages
might not get careful
can reach large
feedback
numbers wording can bias client’s
Provide for anonymity response
response rate is often
relatively inexpensive
low
easy to analyze literacy demands
gibson manhuru 8/20/13
When should a questionnaire
be used?
Respondents
can provide useful
information about the topic.
You know what it is you want to know and
are reasonably sure that you can ask
standardized questions to get the
information.
gibson manhuru 8/20/13
Cont’d
Respondents can be relied upon to provide
the information you need (perhaps with
incentives).
This
means they can comprehend the
questions and respond properly, they are
truthful, and they are motivated enough to
respond carefully.
gibson manhuru 8/20/13
Good questionnaires are NOT
EASY!
Developing a good questionnaire, takes
time, time, and more time.
Multiple (even a dozen!) drafts may be
involved before the questionnaire is ready.
It’s important to involve others in writing
the questionnaire.
gibson manhuru 8/20/13
Questionnaire design Considerations
Kind of information: What do you want to
know? Is the information already available?
Wording of questions and responses
Formatting the questionnaire
Pre-testing
Cover letters and introductions
When/where will the questionnaire be
distributed? How will returns be managed? etc
gibson manhuru 8/20/13
Questionnaire design
Is the information already available?
Don’t ask a question unless
it has a use.
Eliminate the “nice to know.”
What will you do with each piece of
information gathered?
gibson manhuru 8/20/13
Continuing…
Write questions through your respondent’s
eyes.
Will the question be seen as reasonable?
Will it infringe on the respondent’s privacy?
Will the respondent be able and willing to
answer the question?
Be
selective and realistic when writing
questions.
gibson manhuru 8/20/13
6 Steps In Developing Effective Questionnaires
Decide what information you need.
Determine sample – respondents.
Develop accurate, user-friendly
questionnaire.
Develop plan for distribution, return, and
follow-up.
Provide clear instructions and a good cover
letter.
Pilot test.
gibson manhuru 8/20/13
Step 1: What information is needed?
Be specific
Need to know vs. would like to know
Check to see if information exists
elsewhere
What do you want to be able to say:
counts, percentages, relationships,
narratives
gibson manhuru 8/20/13
Step 2: Sample
Who will complete the questionnaire?
What
do you know about their
preferences, abilities, and cultural
characteristics that may affect the way they
respond?
gibson manhuru 8/20/13
Step 3: Develop questionnaire
Make sure questions cover information needed.
Word questions carefully.
Consider cultural nuances.
Sequence questions appropriately.
Attend to formatting.
Write clear, complete directions.
Review to see if it is user-friendly; consider the
respondent.
Make the questionnaire attractive.
Work as a team.
Plan on writing several draft questionnaires.
gibson manhuru 8/20/13
Step 4: Plan distribution, return, follow-up
Distribution: when, where
At meetings, sites, through mail, email, internet
Return: when, where
Return to individual, collection box
Return envelope addressed/stamped
Return envelope addressed only
Follow-up
gibson manhuru 8/20/13
Step 5: Cover Letter − Explanation
Purpose of questionnaire – how information will be used
Why they are being asked to fill it out
Importance of their response
How and when to respond
Whether response will be anonymous or confidential
Your appreciation
Promise results, if appropriate
Signature − sponsorship
gibson manhuru 8/20/13
Step 6: Pilot test
Always
With people as similar to respondents as possible
Do they understand the questions? The
instructions?
Do questions mean same thing to all?
Do questions elicit the information you want?
How long does it take?
Revise as necessary
gibson manhuru 8/20/13
Types of questions
Open-ended questions − allow respondents
to provide their own answers
Closed-ended questions − list answers and
respondents select either one or multiple
responses
gibson manhuru 8/20/13
Open-ended questions
Do not provide any specific responses from
which the participant would choose.
Allow respondents to express their own
ideas and opinions.
gibson manhuru 8/20/13
Open-ended questions
Pros and cons
Pros: Cons:
Can get unintended or More difficult to answer
May be harder to
unanticipated results
Wide variety of answers categorize for
Answers in interpretation
More difficult for people
participants’ “voices”
who don’t write much
gibson manhuru 8/20/13
Open-ended questions
Examples:
What communication skills did you learn in
this workshop that you will use with your
children?
What benefits do you receive from this
organization?
gibson manhuru 8/20/13
Closed-ended questions
Provide specific answers from which the
participant must choose.
Sometimes called “forced choice.”
Response possibilities include: one best
answer, multiple responses, rating, or
ranking scale.
gibson manhuru 8/20/13
Closed-ended questions
Pro Cons
Easy to analyze Chance of none of the
responses choices being appropriate
Stimulates recall Biases response to what
you’re looking for
Misses unintended
outcomes
gibson manhuru 8/20/13
Closed-ended questions
Examples:
What communication skills did you learn in
this workshop that you will use with your
children?
What benefits do you receive from this
organization?
gibson manhuru 8/20/13
Closed-ended questions
Provide specific answers from which the
participant must choose.
Sometimes called “forced choice.”
Response possibilities include: one best
answer, multiple responses, rating, or
ranking scale.
gibson manhuru 8/20/13
Closed-ended questions
Pros Cons
Easy to analyze Chance of none of the
responses choices being appropriate
Stimulates recall Biases response to what
you’re looking for
Misses unintended
outcomes
gibson manhuru 8/20/13
Closed-ended questions
Example − one best answer:
What does the word “nutrition” mean to you?
(Circle one number.)
Getting enough vitamins
The food you eat and how your body uses it
Having to eat foods I don’t like
Having good health
gibson manhuru 8/20/13
Closed-ended questions
Example − multiple responses:
Of the communication skills taught in this
workshop, which will you use with your children?
active listening
acknowledge feelings
ask more open-ended questions
provide one-on-one time for discussion
negotiation
other_____________________
gibson manhuru 8/20/13
Closed-ended questions
Example − Rating Scale
To what extent do you agree or disagree with
the new zoning code? (Circle one.)
1 . Strongly disagree
2 . Mildly disagree
3 . Neither agree or disagree
4 . Mildly agree
5 . Strongly agree
gibson manhuru 8/20/13
When wording the questions, consider…
The particular people for whom the
questionnaire is being designed
The particular purpose of the questionnaire
How questions will be placed in relation to
each other in the questionnaire
gibson manhuru 8/20/13
Use clear, specific, simple wording.
Match vocabulary and reading skills of your
respondents.
Are any words confusing? Do any words have
a double meaning?
Avoid the use of abbreviations and jargon.
gibson manhuru 8/20/13
Example:
Use clear, specific, simple wording.
Avoid jargon or technical language.
Jargon:
What kind of post-litigation concerns have
you and your ex-spouse had?
Better:
Since having your visitation rights set by a
judge, what other concerns have you and
your ex-spouse had about visitation?
gibson manhuru 8/20/13
Include all necessary information.
Avoid vague questions and answers.
Avoid ambiguous words or phrases.
Avoid questions that may be too specific.
Avoid making assumptions.
gibson manhuru 8/20/13
Example: Vague questions
Vague:
How will this seminar help you?
gibson manhuru 8/20/13
Example: Vague questions
Better:
What skills did you learn in this seminar that will
help you follow the child custody arrangements set by
the court?
how to negotiate changes or with my ex-spouse
how to explain visitation arrangements to my children
steps to requesting a change in arrangements from
the court
how to separate child support from visitation disputes
gibson manhuru 8/20/13
Avoid leading questions
Biased questions
Influence people to respond in a certain
way
Make assumptions about the respondent
Use language that has strong positive or
negative appeal
gibson manhuru 8/20/13
Example: Leading questions
Leading:
Do you think this seminar will help you stop
fighting with your spouse about the children?
Better:
How do you think this seminar will help you
work with your spouse to address your
children’s concerns?
gibson manhuru 8/20/13
Avoid double-barreled questions.
Ask one question at a time.
Avoid ambiguity − questions that have
multiple responses.
gibson manhuru 8/20/13
Example: Double-barreled question
Double:
How will this seminar help you communicate better
with your children and their grandparents about your
divorce?
Better:
How will this seminar help you communicate with
your children about your divorce?
How will this seminar help you communicate with
your children’s grandparents about their relationship
with their grandchildren?
gibson manhuru 8/20/13
Make the response categories clear, logical,
and mutually exclusive.
Only one possible answer
Similar-sized categories
Responses in a logical order
gibson manhuru 8/20/13
Example: Clear, logical, and
mutually exclusive responses
Example: Better spacing, logic,
and mutually exclusive:
Poor spacing
Children’s Ages
and logic:
Children’s Ages under 1 year of age
0−1 1−3 years of age
1−3 4−6 years of age
3−6 7−9 years of age
7−12 10−12 years of age
13−18
13−15 years of age
16−18 years of age
gibson manhuru 8/20/13
Example: Vague quantifier
Vague:
How often did you attend an Extension-
sponsored workshop during the past year?
a. Never
b. Rarely
c. Several times
d. Many times
gibson manhuru 8/20/13
Example: Vague quantifier
Better:
How often did you attend an Extension-
sponsored workshop during the past year?
a. Not at all
b. One to two times
c. Three to five times
d. More than five times
gibson manhuru 8/20/13
Types of rating scales
Category scales
Numeric scales
Semantic differentials
gibson manhuru 8/20/13
Rating scales
Ordered options to gauge difference of opinion.
Keep the order of choices the same throughout the
form.
Odd number of options allows people to select a
middle option.
Even number forces respondents to take sides.
Simpler is better.
gibson manhuru 8/20/13
Category/Rating scales
Use words or phrases to express a range
of choices.
The number of categories depends on the
amount of differentiation.
Three, four, or five categories are most
common.
gibson manhuru 8/20/13
Category/Rating scales
Balance the scale with an equal number of positive
and negative options.
“No opinion” or “uncertain” are not part of a scale.
They are usually placed off to the side or in a
separate column.
All choices should refer to the same thing/concept.
gibson manhuru 8/20/13
Category/Rating scales − Example
Poor: Better:
Not worth my time
Not at all interested
Slightly interested
Slightly interested
Moderately interested
Moderately interested
Very interested
Very interested
gibson manhuru 8/20/13
Rating scales − Words
Not much
A little Not much
Some
Some Little
A great deal
A lot
Somewhat
Much
A great deal
gibson manhuru 8/20/13
Rating scales − Words
Never Extremely poor
Seldom Below average
Average
Often
Above average
Always
Excellent
gibson manhuru 8/20/13
Rating scales − Words
Strongly disagree Disagree Completely disagree
Disagree Neither agree nor Mostly disagree
Agree disagree Slightly disagree
Strongly agree Agree Slightly agree
Uncertain Mostly agree
Completely agree
gibson manhuru 8/20/13
Formatting considerations
Overall appearance
Length of the questionnaire
Order of questions
Demographic data collection
gibson manhuru 8/20/13
Formatting − Overall appearance
Use an easy-to-read typeface.
Leave plenty of white space.
Separate
different components of a
questionnaire by using different type styles.
Use arrows to show respondents where to go.
gibson manhuru 8/20/13
Formatting − Length of the questionnaire
Shorter questionnaires usually generate higher
return.
Include enough items to be thorough but don’t
over-burden the respondent.
Length is not usually as important as other
formatting characteristics.
gibson manhuru 8/20/13
Formatting − Order of questions
Introduction − Include
questionnaire’s sponsor, purpose,
use, confidentiality, etc.
Include instructions for how to
answer the questions (e.g., Circle
one; Check all that apply).
gibson manhuru 8/20/13
Formatting − Order of questions
Arrange questions so they flow naturally.
Place demographic questions at the end of
the questionnaire.
Be consistent with numbers, format, and
scales.
gibson manhuru 8/20/13
Formatting − Order of questions
Start with the easiest questions − avoid
controversial topics.
Address important topics early.
Move from specific questions to general
questions.
Move from closed-ended to open-ended
questions.
gibson manhuru 8/20/13
Formatting − Demographic data
collection
Only include questions about demographic
data that you will use.
You
may want to preface demographic
questions with the purpose for collecting the
information.
You may need to state that providing this
information is optional and/or explain how it
affects program eligibility.
gibson manhuru 8/20/13
Formatting − Demographic
data collection
Age Residence
Gender
Previous contact with organization
Ethnicity
Marital status Prior knowledge of topic
Family size First-time participant vs. repeats
Occupation
How you learned about the program
Education
Employment status
gibson manhuru 8/20/13
Pre-test the questionnaire
ALWAYS
ALWAYS
ALWAYS
gibson manhuru 8/20/13
What you want to find out in a
pretest:
Does each question measure what it is
supposed to measure?
Are all the words understood?
Are questions interpreted in the same way
by all respondents?
Are all response options appropriate?
Is there an answer that applies to each
respondent?
gibson manhuru 8/20/13
Pre-testing questions
Are the answers respondents can choose
from correct?
Are some responses missing?
Does the questionnaire create a positive
impression
does it motivate people to answer it?
Does any aspect of the questionnaire
suggest bias?, etc
gibson manhuru 8/20/13
Pre-testing steps
Select reviewers who are similar to the
respondents and who will be critical. (Also ask
your colleagues to review it.)
Ask them to complete the questionnaire as if
it were “for real.”
Obtain feedback on the form and content of
the questionnaire and the cover letter. Was
anything confusing, difficult to answer, de-
motivating?
gibson manhuru 8/20/13
Pre-testing steps, continued
Assess whether the questions produce the
information you need.
Try the tabulation and analysis
procedures.
Revise.
If necessary, repeat these steps to pre-test
the revised version.
gibson manhuru 8/20/13
Revise and revise…
A quality questionnaire is almost never
written in one sitting.
A
quality questionnaire goes through
multiple revisions (maybe a dozen!) before
it is ready.
Remember – a list of questions is just the
starting point. There are many factors that
affect response.
gibson manhuru 8/20/13
Choices: Timing of data collection
When will data be collected?
Before and after the program
At one time
At various times during the course of
the program
Continuously through the program
Over time − longitudinally
gibson manhuru 8/20/13
A good cover letter will include
information about…
Purpose and importance of the survey
Survey sponsor − use letterhead
Why
the respondent was selected to
participate
Benefit(s) of completing survey
Assurance of anonymity or confidentiality
gibson manhuru 8/20/13
A good cover letter will include
information about…
How results will be used
Instruction for returning the survey
When to respond
How to obtain survey results
Contact information
gibson manhuru 8/20/13
Cover letters − Tips
Personalize the letter in salutation or
signature
Hand-sign the letter
Express appreciation for their participation
Include pre-addressed, stamped return
envelope
gibson manhuru 8/20/13
END