Table 5-4.

Pros and Cons of Data Collection Tools

Tool Type Pros Cons Tips on Validity and Reliability

Interview • Collects rich, qualitative data. • Is time consuming. • Write clear interview guide in advance.
• Allows follow-on questions. • Requires a lot of data synthesis. • Have a subject matter expert review
• Allows observation of body language. • Because data are very subjective, and revise the form.
normally requires corroboration of • Tell the interviewee in advance what
information from other sources. the purpose of the interview is and
provide a general description of what
types of information will be examined.

Observation • Is the best way to see how work gets • The people being observed may adjust • Make sure the people being observed
done. their behavior while being observed. know in advance why they were select-
• Can often detect self-created job aids • The people being observed may feel ed and what information is sought.
and other distinct advantages that uncomfortable being observed. • Don’t be perceived as a “spy” by stand-
exemplary performers possess. • It is a time-consuming method that will ing over people while observing them.
• Is the only method that allows the require a lot of data analysis later. • The best way to get a feel for the job

Chapter 5: Data Collection and Assessment

WLP practitioner to truly see the while observing is for the WLP practi-
barriers that are present on the front tioner to ask to try to do the job
line. (assuming it is safe to do so).

Focus Group • Allows respondents to build on each • It can easily deteriorate into a “whining • Carefully choose participants according
other’s ideas. session.” to a set of criteria.
• Enables the consolidation of individual • If working with an organization that has • Avoid mixing individual contributors
responses. low trust, participants may refuse to with their supervisors.
speak in front of others. • Advise participants in advance about
• It does not work well if participants the ground rules (especially on confi-
feel intimidated by other participants dentiality).
(e.g., the “big boss” is in the room and • Record session and use private room.
no one wants to speak). • Have a recorder to take notes.
• Use a structured approach, such as
nominal group technique.
99

(continued on next page)

100

Section II: Assessing and Analyzing Needs

Table 5-4. Pros and Cons of Data Collection Tools (continued)

Survey • Reaches a large population. • It is very difficult to predict how the • Buy whenever possible.
• Automates data collection if online respondent will interpret questions. • Pilot test, review, revise, and test
survey tools are used. • It does not allow a lot of follow-on, again.
• Tests assumptions over a large probing questions. • Use a simple split-half reliability
population. • It tends to have very low response check.*
rates.

Test • Is the best way to determine learning • It requires considerable skill to build a • Buy whenever possible.
gains. valid and reliable test. • Pilot test, review, revise, and test
• Is quick and easy to administer. • Test anxiety may skew the results. again.
• Carries a lot of weight when studying • It is difficult to administer in situations • Use a simple split-half reliability
learning systems. where language is a barrier. check.*

Document • Is normally the only method for gather- • Documents can be hard to access. • Only review documents that are
Review ing hard data on performance issues. • Documents must be true and accurate. directly linked to the research question.
• Is difficult to argue with if the data • Review of some documents may be • Get written permission to review any
come from valid performance records. perceived as a violation of privacy. sensitive documents.
• Is quick compared with more qualita- • Gather data on how much confidence
tive forms of data generation. the client and stakeholders have in the
• Is less subject to interpretation. accuracy of these documents (e.g., ask
employees if they believe that the stan-
dard operating procedure is accurate).

*A split-half reliability check allows the researcher to determine if the phrasing of the question has an influence on how the respondent answers the question.
Two versions of the survey or test are provided. In one version, the questions are phrased in the affirmative, while in the other version the questions are
phrased in the negative. The responses are then compared to see if the way the questions were worded affected the response.