Data Collection Methods

Data types and formats

2
Structured data
Structured data is data that is highly organised and located in a
fixed field within a defined record or file.
This includes data contained in relational databases or
spreadsheets.
Easy to input, easy to store and easy to analyse
Follow rules and accessed using SQL

3
Semi and un-structured data
Not everything in business fits neatly into a predefined field and that’s
where semi-structured and unstructured data comes in
80% of business-relevant information
It is often text heavy but may also contain dates, numbers and different
types of data such as images or audio files.
Semi-structured data is a hybrid. This is data that may have some
structure that can be used for analysis but large chunks are unstructured
o A linkedin post can be categorised by author, date or length but the content is
generally unstructured
o Word processing software includes metadata detailing the author’s name, when it
was created and amended but the content of the document is still unstructured.

4
Internal data
Most businesses are already data rich
Problem: they are insight poor and don’t often know how to use
the data they have, never mind utilise the treasure trove of
external data that also exists.
As a rule of thumb, internal data is usually easier and cheaper
to access because it is owned and controlled by the business.
o financial records, customer feedback, transaction history, employee surveys,
hR data, etc.

5
External data
external data as the name would suggest is any data that exists
outside your business which is held either publicly or privately by
another organisation.
if data is public then you can collect it for free, pay a third party for it
or hire a third party to collect it for you.
Private data is usually something you would need to source and pay
for from another business or third party data supplier. external data
might include weather data, social media profile data, trend data or
government-held data such as census information

6
Capability to analyze a variety of formats
it is the explosion in semi-structured and unstructured data along with the
technological capacity to store and analyse that data that now makes
analytics so exciting. we now have the capability to analyse a variety of
formats including:
o Database/spreadsheet format
o images
o text
o numbers
o Video
o Audio
o Sensor data.

7
Data collection methods

8
Introduction

Data Collection Strategies

Characteristics of Good Measures
Quantitative and Qualitative Data
Tools for Collecting Data

9
Data Collection Strategies
No one best way: decision depends on
oWhat you need to know: numbers or stories
oWhere the data reside: environment, files, people
oResources and time available
oComplexity of the data to be collected
oFrequency of data collection
oIntended forms of data analysis

10
Rules for Collecting Data
Use multiple data collection methods
Use available data, but need to know
o how the measures were defined
o how the data were collected and cleaned
o the extent of missing data
o how accuracy of the data was ensured

11
Rules for Collecting Data
If must collect original data:
o be sensitive to burden on others
o pre-test, pre-test, pre-test
o establish procedures and follow them (protocol)
o maintain accurate records of definitions and coding
o verify accuracy of coding, data input

12
Structured Approach
All data collected in the same way
Especially important for multi-site and cluster evaluations so you can
compare
Important when you need to make comparisons with alternate
interventions

13
Use Structured Approach When:

need to address extent questions

have a large sample or population
know what needs to be measured
need to show results numerically
need to make comparisons across different sites or
interventions

14
Semi-structured Approach
Systematic and follow general procedures but data are not collected
in exactly the same way every time
More open and fluid
Does not follow a rigid script
o may ask for more detail
o people can tell what they want in their own way

15
Use Semi-structured Approach when:

conducting exploratory work

seeking understanding, themes, and/or issues
need narratives or stories
want in-depth, rich, “backstage” information
seek to understand results of data that are unexpected

16
Characteristics of Good Measures
Is the measure relevant?
Is the measure credible?
Is the measure valid?
Is the measure reliable?

17
Quantitative Approach
Data in numerical form
Data that can be precisely measured
o age, cost, length, height, area, volume, weight, speed, time, and temperature
Harder to develop
Easier to analyze

18
Qualitative Approach
Data that deal with description
Data that can be observed or self-reported, but not always
precisely measured
Less structured, easier to develop
Can provide “rich data” — detailed and widely applicable
Is challenging to analyze
Is labor intensive to collect
Usually generates longer reports

19
 Which Data?
If you: Then Use:
- want to conduct statistical analysis
- want to be precise Quantitative
- know what you want to measure
- want to cover a large group
- want narrative or in-depth information
- are not sure what you are able to measure Qualitative
- do not need to quantify the results

20
Obtrusive vs. Unobtrusive Methods
Obtrusive Unobtrusive
Data collection methods that Data collection methods that
directly obtain information from do not collect information
those being evaluated directly from evaluees
e.g. interviews, surveys, focus e.g., document analysis,
groups GoogleEarth, observation at a
distance, trash of the stars

21
How to Decide on Data Collection Approach

Choice depends on the situation

Each technique is more appropriate in some situations than others
Caution: All techniques are subject to bias

22
 Triangulation to Increase Accuracy of Data
Triangulation of methods
o collection of same information using
different methods
Triangulation of sources
o collection of same information from a
variety of sources
Triangulation of evaluators
o collection of same information from more
than one evaluator

23
Data Collection Tools
1. Participatory Methods
2. Records and Secondary Data
3. Observation
4. Surveys and Interviews
5. Focus Groups
6. Diaries, Journals, Self-reported Checklists
7. Expert Judgment
8. Delphi Technique
9. Other Tools

24
Tool 1: Participatory Methods
Involve groups or communities heavily in data collection
Examples:
o community meetings
o mapping
o transect walks

25
Community Meetings
One of the most common participatory methods
Must be well organized
o agree on purpose
o establish ground rules
 who will speak
 time allotted for speakers
 format for questions and answers

26
Mapping
Drawing or using existing maps
Useful tool to involve stakeholders
o increases understanding of the community
o generates discussions, verifies secondary sources of information,
perceived changes
Types of mapping:
o natural resources, social, health, individual or civic assets,
wealth, land use, demographics

27
Transect Walks

Evaluator walks around community observing people, surroundings,

and resources

Need good observation skills

Walk a transect line through a map of a community — line should go

through all zones of the community

28
Tool 2: Records and Secondary Data
Examples of sources:
o files/records
o computer data bases
o industry or government reports
o other reports or prior evaluations
o census data and household survey data
o electronic mailing lists and discussion groups
o documents (budgets, organizational charts, policies and procedures, maps,
monitoring reports)
o newspapers and television reports

29
Use Existing Data Sets

Key issues: validity, reliability, accuracy, response rates,

data dictionaries, and missing data rates

30
Advantage/Challenge: Available Data
Advantages Often less expensive and faster than
collecting the original data again

Challenges There may be coding errors or other

problems. Data may not be exactly
what is needed. You may have
difficulty getting access. You have to
verify validity and reliability of data

31
Tool 3: Observation
See what is happening
otraffic patterns
oland use patterns
olayout of city and rural areas
oquality of housing
ocondition of roads
oconditions of buildings
owho goes to a health clinic

32
Observation is Helpful when:
need direct information
trying to understand ongoing behavior
there is physical evidence, products, or outputs than can be observed
need to provide alternative when other data collection is infeasible or
inappropriate

33
Degree of Structure of Observations
Structured: determine, before the observation, precisely what will be
observed before the observation
Unstructured: select the method depending upon the situation with
no pre-conceived ideas or a plan on what to observe
Semi-structured: a general idea of what to observe but no specific
plan

34
Google Earth
Maps and satellite images for complex or pinpointed regional
searches
Has an Advanced version and an Earth Outreach version
Web site for Google Earth
o [Link]

35
Ways to Record Information from Observations
Observation guide
o printed form with space to record
Recording sheet or checklist
o Yes/no options; tallies, rating scales
Field notes
o least structured, recorded in narrative, descriptive style

36
 Advantages and Challenges: Observation

Advantages Collects data on actual vs. self-

reported behavior or perceptions. It is
real-time vs. retrospective

Challenges Observer bias, potentially unreliable;

interpretation and coding challenges;
sampling can be a problem; can be
labor intensive; low response rates

37
Tool 4: Surveys and Interviews

Excellent for asking people about: perceptions, opinions, ideas

Less accurate for measuring behavior

Sample should be representative of the whole

Big problem with response rates

38
Structures for Surveys
Structured:
o Precisely worded with a range of pre-determined responses that
the respondent can select
o Everyone asked exactly the same questions in exactly the same
way, given exactly the same choices
Semi-structured
o Asks same general set of questions but answers to the questions
are predominantly open-ended

39
Structured vs. Semi-structured Surveys

Structured harder to develop

easier to complete
easier to analyze
more efficient when working with large numbers
Semi- easier to develop: open ended questions
structured more difficult to complete: burdensome for people to
complete as a self-administrated questionnaire
harder to analyze but provide a richer source of data,
interpretation of open-ended responses subject to
bias
40
Modes of Survey Administration
Telephone surveys
Self-administered questionnaires distributed by mail, e-mail, or
websites
Administered questionnaires, common in the development context
In development context, often issues of language and translation

41
Mail / Phone / Internet Surveys
Literacy issues
Consider accessibility
o reliability of postal service
o turn-around time
Consider bias
o What population segment has telephone access? Internet access?

42
 Advantages and Challenges of Surveys

Advantages Best when you want to know what people

think, believe, or perceive, only they can
tell you that
Challenges People may not accurately recall their
behavior or may be reluctant to reveal
their behavior if it is illegal or stigmatized.
What people think they do or say they do
is not always the same as what they
actually do.
43
Interviews

Often semi-structured
Used to explore complex issues in depth
Forgiving of mistakes: unclear questions can be clarified
during the interview and changed for subsequent
interviews
Can provide evaluators with an intuitive sense of the
situation

44
Challenges of Interviews
Can be expensive, labor intensive, and time consuming
Selective hearing on the part of the interviewer may miss information
that does not conform to pre-existing beliefs
Cultural sensitivity: e.g., gender issues

45
Tool 5: Focus Groups
Type of qualitative research where small homogenous groups of
people are brought together to informally discuss specific topics
under the guidance of a moderator
Purpose: to identify issues and themes, not just interesting
information, and not “counts”

46
Focus Groups are inappropriate when:
language barriers are insurmountable
evaluator has little control over the situation
trust cannot be established
free expression cannot be ensured
confidentiality cannot be assured

47
Focus Group Process
Phase Action

1 Opening Ice-breaker; explain purpose; ground rules;

introductions
2 Warm- Relate experience; stimulate group interaction;
up start with least threatening and simplest questions
3 Main Move to more threatening or sensitive and
body complex questions; elicit deep responses; connect
emergent data to complex, broad participation
4 Closure End with closure-type questions; summarize and
refine; present theories, etc; invite final comments
or insights; thank participants
48
Advantages and Challenges of Focus Groups
Advantages Can be conducted relatively quickly and
easily; may take less staff time than in-depth,
in-person interviews; allow flexibility to make
changes in process and questions; can
explore different perspectives; can be fun

Challenges Analysis is time consuming; participants not

be representative of population, possibly
biasing the data; group may be influenced by
moderator or dominant group members

49
Tool 6: Diaries and Self-Reported Checklists
Use when you want to capture information about events in people’s
daily lives
Participants capture experiences in real-time not later in a
questionnaire
Used to supplement other data collection

50
Guidelines for Diaries or Journals
Step Process

1 Recruit people face-to-face

• encourage participation, appeal to altruism, assure
confidentiality, provide incentive
2 Provide a booklet to each participant
• cover page with clear instructions, definitions, example
• short memory-joggers, explain terms, comments on last
page , calendar
3 Consider the time-period for collecting data
• if too long, may become burdensome or tedious
• if too short may miss the behavior or event
51
Self-reported Checklists
Cross between a questionnaire and a diary
The evaluator specifies a list of behaviors or events and asks the
respondents to complete the checklist
Done over a period of time to capture the event or behavior
More quantitative approach than diary

52
Advantages and Challenges of Diaries and Self-
reported Checklists
Advantages Can capture in-depth, detailed data that might be
otherwise forgotten
Can collect data on how people use their time
Can collect sensitive information
Supplements interviews provide richer data

Challenges Requires some literacy

May change behavior
Require commitment and self-discipline
Data may be incomplete or inaccurate
Poor handwriting, difficult to understand phrases

53
Tool 7: Expert Judgment
 Use of experts, one-  Can be structured or
on-one or as a panel unstructured
o e.g., Government task o Issues in selecting experts
forces, Advisory Groups

54
Selecting Experts
Establish criteria for selecting experts not only on recognition as
expert but also based on:
o areas of expertise
o diverse perspectives
o diverse political views
o diverse technical expertise

55
Advantages and Challenges of Expert Judgment

Advantages Fast, relatively inexpensive

Challenges Weak for impact evaluation

May be based mostly on perceptions
Value of data depends on how
credible the experts are perceived to
be

56
Tool 8: Delphi Technique
Enables experts to engage remotely in a dialogue and
reach consensus, often about priorities
Experts asked specific questions; often rank choices
Responses go to a central source, are summarized and
fed back to the experts without attribution
Experts can agree or argue with others’ comments
Process may be iterative

57
Advantages and Challenges of Delphi Technique
Advantages Allows participants to remain anonymous
Is inexpensive
Is free of social pressure, personality influence,
and individual dominance
Is conducive to independent thinking
Allows sharing of information
Challenges May not be representative
Has tendency to eliminate extreme positions
Requires skill in written communication
Requires time and participant commitment

58
Other Measurement Tools
scales (weight) health testing tools:
tape measure o i.e. blood pressure
stop watches aptitude and
achievement tests
chemical tests:
i.e. quality of water citizen report cards

59
Data Collection Summary

Choose more than one data collection technique

No “best” tool

Do not let the tool drive your work but rather choose the right tool to
address the evaluation question

60