Chapter 3

Designing Complex Development

Programs

Abstract Approaches to designing development programs vary. Some emphasize

stakeholder involvement; others focus on a program’s environment; still others
emphasize the sequence of change depicted in program results (change frame-
works). This chapter examines the use of such change frameworks as the Logical
framework approach (LFA), Theory of change approach (ToCA), and Participatory
impact pathways approach (PIPA) for designing complex development programs
and the extent to which such frameworks enable stakeholders to explicate and
question implicit assumptions.

Introduction

Program design (or planning) involves the selection of an integrated set of

purposeful activities that will be undertaken to achieve one or several related
objectives. Decisions made during this process can be political in the sense that
they involve compromises and tradeoffs among various interests, though the
differences may be couched in technical terms (Abonyi and Howard 1980). Here
the term program design includes issues that have to be decided prior to selection
of activities and problem definition. One of the crucial aspects of designing
complex programs, as Chap. 2 suggests, is unearthing the assumptions that shape
the program designs. Whatever process stakeholders may take in program design
or in devising a plan to implement the design, unearthing the assumptions
underlying these processes is vital to ensuring sound program structure and to
uncloaking potential problems. Surfacing ‘‘taken-for-granted’’ assumptions must
be high on the list; failure to make these assumptions explicit could be fatal to
program success.

A. M. Nkwake, Working with Assumptions in International Development 25

Program Evaluation, DOI: 10.1007/978-1-4614-4797-9_3,
 Springer Science+Business Media New York 2013
26 3 Designing Complex Development Programs

Program design approaches can be grouped into three categories:

(a) Stakeholder-focused approaches: These emphasize why and how to engage
optimal involvement of stakeholders, such as beneficiaries and program
sponsors, in designing programs. Hummelbrunner (2010) refers to these
approaches as social frameworks.
(b) Change-perspective: These encompass perspective(s) that can be applied to
any kind of change framework, especially systems thinking.
(c) Outcome-focused approaches: These concentrate on program outcomes as the
framework for their planning (including simple linear processes like logical
framework approaches and complex reiterated processes).
The first two—stakeholder-focused approaches and change perspectives are
described briefly below. The remaining discussion in this chapter chiefly addresses
outcome-focused approaches, with particular emphasis on the LFA.

Stakeholder-Focused Approaches

Stakeholder-focused frameworks tend to emphasize the roles of various actors in

the change process. An example is Social network analysis (SNA), a set of
techniques for analyzing social systems, used to understand networks and their
participants. SNA helps program designers to visualize, measure and simulate
relationships (See Owen-Smith and Powell 2004; Ahuja 2000; Moody and White
2003; Chandrasekhar et al. 2011; Freeman 2006; Wasserman and Faust 1994;
Hogan et al. 2007). SNA also allows designers to analyze these relationships in
visual, as well as mathematical terms (Hummelbrunner 2010). Programs involving
multiple actors and operating at different levels of scale can benefit substantially
from the SNA technique.
Another framework that emphasizes the role of stakeholders in a change pro-
cess is Outcome mapping (OM). Developed by the International Development
Research Centre (IDRC) in Ottawa, Canada, OM guides program stakeholders
through an iterative process to identify their desired change(s) and to work col-
laboratively to bring it (them) about. It focuses on changes in the behaviors,
relationships, and actions or activities of the people, groups, and organizations
with whom a development program is working directly or seeking to influence—
and of the program(s) being influenced by these interactions (See Deprez and
Ongevalle 2006a, b; Earl et al. 2001; Kibel 2000; Rockwell and Bennett 2000).
OM focuses on one specific type of outcome: changes in the behavior of ‘boundary
partners’ with whom the program works directly and anticipates opportunities for
mutual interaction and influence (Hummelbrunner 2010).
Change Perspective 27

Change Perspective

A change perspective is not a change framework per se; it is a way of thinking that
can be applied to several different frameworks. A common perspective is ‘‘systems
thinking,’’ an approach that has been popularized by Williams and others (e.g.,
Williams and Iman 2008; Trochim et al. 2006; Leischow et al. 2008). Johnson
et al. (2011) argue that systems thinking is a form of evaluative thinking that
involves mining ‘‘systems literature for its heuristic value’’ as opposed to using
systems methodologies (e.g., system dynamics, network analysis) in evaluation
work. Richard (2009) argues that systems concepts (such as interrelationships,
system dynamics, perspectives, boundaries, etc.) can be particularly useful to
evaluators in situations where rigorous rethinking, reframing, and unpacking of
complex realities and assumptions are needed. Applying systems concepts in
designing programs, according to Richard, yields several benefits:
• more effective ways of dealing with complexity and complex situations.
• development of new ways of understanding situations.
• enabling of attention to coalitions by project evaluators.
• focus on properties that emerge unexpectedly.
• help in identifying leverage points.
• capacity to enable stakeholders to recognize evolutionary nature of programs.

Outcome-Focused Frameworks

These frameworks focus on the sequence of changes that are anticipated to be

created by the implementation of a program. Davies refers to these as ‘‘theories of
change.’’ (Davies 2004:103) describes five categories of change-focused
processes:
• linear processes with varying numbers of stages, e.g., logical frameworks.
• linear processes with branching structures (e.g., when revisions are made on a
logical frameworks to include more steps, outputs, and outcomes).
• simple Parallel Processes (nested Logical Frameworks): output of a program’s
Logical Framework can be represented as the ‘purposes’ in component projects.
Differencesintheapproacharethendocumentedineachproject,attheiroutputlevels.
• interacting Parallel Processes: data on activities and outcomes are aggregated
into larger units, such as localities or responsible staff, rather than longer term
outcomes in the lives of beneficiaries.
• reiterated Processes: These create linkages between goals at different levels in
the hierarchical structure (projects and objectives).
The present discussion focuses on the logical frameworks approach, which has
been the most commonly applied to program planning and evaluation.
28 3 Designing Complex Development Programs

The LFA and Assumptions: Does It Adequately

Reflect Complexity?

In the last 5 decades, the LFA has come to play a central role in the planning and
management of development (e.g. Ambrose and Roduner 2009; Bakewell and
Garbutt 2005; Davies 2005; Gasper 1997; 2000, etc.). With its origins in United
States military, space and development agencies, the LFA was later adopted and
adapted by European development organizations and United Nations agencies.
It has become a major standard in applications to donors for grants.
Its distinguishing feature is the use of the four-by-four matrix–the log frame.
A log frame is a summarized description of a design of an intervention or program
showing how inputs will be transformed into outputs and outcomes. But there is
more to the LFA approach than the log frame. The matrix simply summarizes the
main elements of the project and connects them to each other. The LFA is an
overarching process by which the elements that go into the matrix are formulated—
the wider planning procedures of problem analysis, the development of objectives
and indicators, and identification of risks and assumption, which then feed into the
overall program plan (Bakewell and Garbutt 2005). LFA is a widely used (some-
times misused) model for planning; its roles will be explored in subsequent sections
and it will also serve as a point of reference in discussing other models.
The log frame has a reasonably standard form. A typical example of the format
is given in Table 3.1. The content of the four-by-four matrix is straightforward:
(i) the vertical column is a hierarchy of objectives: activities deliver outputs that
contribute to outcomes, which help to bring about the overall goal (Bakewell and
Garbutt 2005). In some variations, the four rows are called activities, results/
outputs, purpose(s), and overall objectives (Gasper 2000, p. 18). (ii) The horizontal
row demonstrates how progress against each objective can be assessed (indicators
and means of verification) and the external factors (assumptions and risks) that
might affect whether reaching these objectives will contribute to the next level.

The ZOPP

One offspring of the logical framework approach is ZOPP (Zielorientierte Pro-

jektplanung or GOPP—Goal-Oriented Project Planning), which has been used and
promoted by GTZ (German Technical Corporation) ZOPP is an extension of the
LFA. The ZOPP’s output is a planning matrix—the logical project framework—
that summarizes and structures the main elements of a project and highlights logical
linkages among intended inputs, planned activities, and expected results. The ZOPP
approach is used for essentially all German-funded projects and is a prerequisite for
funding approval. The acronym ‘ZOPP’ is sometimes used to encompass all of the
logical framework methodologies in deference to its principal initiators, the
German development agencies, and particularly the GTZ (World Bank 2009).
 The ZOPP

Table 3.1 European commission’s version of the project matrix

Intervention logic Objectively verifiable indicators Sources of Assumptions
verification
1. Overall objective Indicators Data sources for This cell is empty in the EC version but some versions put
The longer term benefits to (target- Measures (direct or indirect) to indicators for here: Important events, conditions, or decisions necessary
group) beneficiaries and wider verify to what extent the overall for sustaining objectives in the long run
benefits to other groups overall objective is fulfilled objective
2. Project purpose Indicators Data sources for 1. Assumptions
Benefits to be received by the Measures (direct or indirect) to indicators for Important events, conditions, or decisions outside the control
project beneficiaries or target verify to what extent the project purpose of the project which must prevail for the overall objective
group project purpose is fulfilled. to be attained
3. Results Indicators Data sources for 2. Assumptions
Services to be delivered to the Measures (direct or indirect) to indicators for Important events, conditions, or decisions outside the control
intended beneficiaries or target verify to what extent the results of the project management, necessary for the achievement
group results are produced of the project purpose
4. Activities 5. Inputs Data sources for 3. Assumptions
The activities that have to be Goods and services necessary to activity Important events, conditions, or decisions outside the control
undertaken by the project in undertake the activities monitoring of the project management, necessary for the production
order to produce the outputs of the results
4. Preconditions
Adapted from European Commission (1999)
29
30 3 Designing Complex Development Programs

ZOPP includes various subparts used to clarify projects, and the project’s logical
framework itself is often required by development agencies in their project appraisal.
There are five logical levels of the ZOPP in a standard project cycle (see Box 3.1).

Box 3.1 Levels of the ZOPP Cycle

1. Pre-ZOPP: an in-house exercise by agencies in preparation for a project.

2. Appraisal ZOPP: an in-house appraisal for preparing Terms of Reference
of a project.
3. Partner ZOPP: prepared in-country; coordination of conclusions, and
recommendations with staff of project country
4. Take-off ZOPP: prepared in-country; preparation of the plan of opera-
tions with personnel responsible for project execution counterpart
authorities.
5. Replanting ZOPP: prepared in-country; adjustments during project
implementation.
Other ZOPPs are recommended annually in projects to update planning
as needed. The ZOPP has two phases: analysis and project planning. The
analysis phase has 4 sub-steps, with the identification of ‘real’ problems as
the driver for the exercises (World Bank 2009).

Other Variations of the LFA

Over the years, the use of the LFA has evolved to see many variations.
Bakewell and Garbutt (2005) in their study on use and misuse of this
approach have identified LFA as a formal procedure, a way of thinking, and
a brand, as well as a system. (See Box 3.2).

Box 3.2 Variations in LFA

LFA as a Formal Procedure

For some, the LFA is a formal procedure for planning projects, and in some
cases also providing the base for the monitoring and evaluation system. The
LFA sets out a number of standard steps to be completed, which may include
some form of participatory problem assessment and identification of aims
and objectives, some form of risk assessment, and so forth. The output of the
Box 3.2 Variations in LFA 31

LFA, the project or program plan is usually required to be summarized in a

logical framework.

LFA as a Way of Thinking

LFA is understood more as a set of ideas and a way of thinking. The value
lies in working through a hierarchy of objectives, ensuring that you consider
appropriate indicators, and take account of the risks and assumptions. How
one goes about this process can be quite flexible and does not necessarily
need to be captured in a matrix.

LFA as a Brand or Label

A third sense in which the term LFA is used is as a way of branding or labeling
one’s planning approach to make it acceptable to donors. As one NGO
respondent stated after describing the organization’s approach to planning,
‘‘We only call it LFA because the donor requires us to use the LFA.’’

LFA as a System

The major problem with the LFA is how it is used in practice. While many
accepted that the logic and systematic approach it encouraged are very
important, actually putting it into practice in their management systems
presented major challenges.

Another variation of the logical framework is for programs to focus on

mapping out program results without necessarily specifying assumptions
indicators or means of verification. An example of a behavior change
communication program is shown below (Fig. 3.1).

How is the LFA Helpful in Dealing with Complexity

and Assumptions?

Although the logical framework has become universally known, it is far from
universally liked. It has been the subject of much criticism over the years, con-
cerning both its theoretical basis and its application in practice (Bakewell and
Garbutt 2005). Gasper (2000) argues that the LFA approach has been a practice
with relatively little accompanying theory, in sharp contrast to cost benefit analysis
or even participatory appraisal (Gasper 2000).
32 3 Designing Complex Development Programs

INPUTS PROCESS OUTPUTS OUTCOMES IMPACT

Distribute/Dissemi- Schools receive

Schools have
nate various prog- various
Develop & print integrated and
ram messages, program messages,
various functional children
guidelines Increased ability
guidelines & related guidelines and clubs.
and materials. of pupils to
program materials/ materials.
avoid risky Increased number
messages. of children who
situations and to
refuse sext delay sex
School teachers until marriage
Conduct tailor-made trained Children have
training for school in counseling and improved
based counseling guidance of pupils knowledge on risky
& guidance. as well situations and how to
as use program avoid them.
materials. Increased
Reproduce CIE kits, communication of
teachers’ Train additional pupils with parents on
handbooks, and teachers on use of delaying sexual debut
Training Manuals programmaterials for
and log books. wider coverage. Action oriented Caregivers have
meetings held with improved knowledge
various stakeholders. on importance of
delayed sexual debut,
Conduct action positive parenting,
Oriented and how to support
meetings with their children at
caregivers, Action plans to different development
Identify and provide school communities, address risky levels. Increased capacity
resourceful PTCs, Districts and situations produced of Ugandan
persons/speakers other and implemented. education system
on various topics stakeholders. to set up &
related to HIV sustain safemodel
prevention. schools.
Organize and
conduct tailor - Program Grants
made talk shows awarded to and
on various received by
HIV prevention model schools. School-based HIV
related topics. prevention activities
strengthened and
School Program best practices
Grants Set up & Disseminate transferred from
mechanism for Grant model to non-model
acquisition by schools. Talk shows conducted schools.
in schools.

Fig. 3.1 A logic model for a behavior change communication program

One of LFA’s main pitfalls lies in the identification of problems as a starting

point. It starts with the perspective that the proverbial glass is half-empty rather
than half-full. The ‘glass half empty’ perspective then drives the process, which
usually results in a limited vision of potentials and solutions. Those who have
worked with appreciative inquiry may not find the problem-led perspective an
appropriate way to engage with communities or even to start any engagement
(Gasper 2000).
Another area of criticism surrounds its flexibility. There is an understanding
among users that development programs are too complex to fit in a four-by-four
matrix. This pitfall is closely linked to the fact that for most people, the LFA is the
matrix (and by reference the program model) and what cannot make it into the
matrix is probably not worth knowing about the intervention. In struggling to
describe the intervention through the vertical and horizontal logics, users find it
rigid—limiting exploration, adaptive thinking, and further innovation. Program
managers like summaries and perhaps this is why donors insist on the log frame. It
gives them the sense of having a coherent summary of the major program ele-
ments, but it does not give them an overview of the problems, actors, assets, and
linkages, to name just a few missing elements. Bakewell and Garbutt’s review
insists that the LFA is mostly top-down.
How is the LFA Helpful in Dealing with Complexity and Assumptions? 33

A simplistic characterization of the prevailing attitudes to the LFA runs as

follows: ‘‘donors insist on it, while NGOs use it under sufferance. All recognize
that it has many weaknesses, but there is a common view that despite these
weaknesses, it is the best of a bad bunch of options available for planning and
monitoring development work; Hence it carries on being widely used against all
objections’’ (Bakewell and Garbutt 2005, p. 1).

Lessons from the LFA

Its pitfall notwithstanding, LFA has taught—or should have taught—two impor-
tant lessons to development students and practitioners, including evaluators. First,
the log frame gets program planners to think carefully through what they are
planning to do. Planners have to consider systematically how their proposed
activities might contribute to the desired goal through delivering outputs and
outcomes. This has been a key in engaging clarity on steps that comprise program
activities. This clarity, however, comes with simplification. The second lesson
concerns the explication of assumptions. These two lessons are discussed below.

Simplification or an Extension of Clarity

Much criticism regarding the LFA centers around its tendency to simplify complex
conditions and programs. Although these criticisms have basis, it is worth noting
that ‘‘simplification’’ is not a new notion. For a long time in development and
management science, people have used various tools to simplify complex situa-
tions, not as a way of overlooking their complexity but as a key step in under-
standing the puzzle. Examples of other such tools include maps, dashboards, and
similar devices.
The example of dashboards is worth exploring a little further: A dashboard in
analysis and planning, like the dashboard in an automobile, is meant to organize
and present information in a way that is easy to read and use. This inevitably
involves a simplification. The fact that the dashboard does not provide ALL the
information does not mean that everything that does not appear on the dashboard is
not useful or does not exist.
Bob Williams (2011:1) narrates an interesting experience with perceptions of
the log frames. Facilitating a workshop on log frames, Bob started by putting a toy
elephant on the table and asked participants to tell him what it was. Most people
said that it was an elephant. So he asked what an elephant does. Among other
responses, participants replied, ‘‘It provides transportation.’’ ‘‘It moves logs.’’ ‘‘It
performs tricks in circuses.’’ He asked, ‘‘Does this elephant do this in any con-
ceivable way?’’ His students acknowledged, ‘‘No … it’s a model of an elephant.’’
The model does not provide a simplified form of reality (e.g., moves only tiny logs
very short distances); it merely represents reality. It’s a metaphor for an elephant.
34 3 Designing Complex Development Programs

The model at best reminds us what we’ve heard elephants might do, maybe even
seen them doing, but it doesn’t actually reflect that behavior.’’
It can be argued that a greater form of simplification would be to assume that
we can and should understand a complex world without any aid that initially
guides us to its core elements. Although simplification can aid understanding and
help to focus on the most important pieces of the puzzle, there is a danger of
overlooking other pieces outside the ‘‘dashboard’’. An example of this is the
tendency to overlook or ignore unexpected results. Other examples of this nature
of log frame misuse are cited by Gasper (2000, p. 21).
• A logic less frame: where a pre-existing log frame format is used to accom-
modate a pre-existing design rather than to help create a logical design in an
appropriate format. The failing can be counteracted; a log frame format can be
used from an early stage in project design, with its series of means-ends links
derived from a systematic prior cause and effect analysis.
• Lack frame: Many LFA users have underestimated that a ‘frame’ includes some
things and leaves others out, and that a ‘frame-work’ is to help the required
work, not substitute for it.
• Lock frame: After an LFA has been prepared; it tends to be fixed and not
updated, and thus becomes a lock-frame. Ironically, this seems more likely the
more care has gone into an LF and the more people consulted. In principle, this
failing can be counteracted if staff are permitted, willing and able to do the extra
work of updating.
Perhaps the simplification pitfall in the use of log frame arises most frequently
when users take it to be more than a simplification and abstraction. In doing this,
the results chain turns out to be a mechanistic set of causes and effects operating in
a self-contained predictable system and environment.
As Rowland puts it (2003, p. 5)
…the tool as a whole has attracted much criticism over the years, but this seems to be one
of its strength: to have a tool which systematically encourages you to become aware of the
assumptions being made throughout a planning process seems like an excellent idea.

The existence of the assumptions column, however, cannot substitute for the
awareness and understanding of the assumptions that are needed in order to fill it
in that requires an openness of mind which the tool itself cannot provide. It also
requires a willingness to revise assumptions when they prove inaccurate and on the
part of institutions, a willingness not only to allow revisions, but to welcome them
as evidence of learning and experience. The value of a log frame thus varies from
one use to another. Program design frameworks such as the theory of change
approach discussed later in this chapter emphasize the need for clarity on how
program resources are expected to transform into desired results. The difference is
that the LFA does the same with a matrix, which makes it appear linear. More
often than not, this is a false depiction of the transformation process. A more
appropriate representation of the ‘‘messiness’’ of the complex transformation
processes is shown in Fig. 3.2.
Simplification or an Extension of Clarity 35

Potential paths to goal–no

initial consensus required

Initial Desired Desired

future future
situation Desired situation
future situation
situation

Initial action
agreed by
consensus

Fig. 3.2 Linear and nonlinear pathways to goal (adopted from Bakewell and Garbutt 2005)

The Necessity of Explicit Assumptions

Assumptions, potentially one of the most important aspects of the log frame, are
also perhaps the most misused. Assumptions are specified in the fourth column of
the log frame. The aim of specifying the assumptions is to identify the external
factors whose presence or absence will (or are likely to) affect the success of the
project. It will have become apparent during the analysis phase that the project
alone cannot achieve all objectives identified in the log frame. These objectives
will affect the project’s implementation and long-term sustainability, but they lie
outside its control. These conditions must be met, however, if the project is to
succeed (European Communities 1999). Assumptions in the log frame are stated in
terms of the desired situation (Jackson 2000). The main condition in assumptions
is that they are outside the control of project managers; they should be outside of
the project’s intervention, yet critical to the project’s success (European Com-
munities 1999).
Preconditions differ from assumptions in that they must be met before a project
can commence. For example, without the implementation of certain policy mea-
sures by a key partner, the project rationale may be undermined. The likelihood of
these conditions being met should be estimated as part of assessing the riskiness of
the project. Some preconditions and assumptions will be more critical to project
success than others. These characteristics are represented in Fig. 3.3 an algorithm
for assumptions.
It is important for each program stakeholder to write out his/her assumptions as
an initial step in the process, then discuss them as a group. This not only increases
clarity about stakeholders’ views of how the process will work, but it helps them
individually and as a group to identify major and minor assumptions, to judge the
likelihood of the assumptions being sustained or invalidated in practice, and
identify the assumptions as more and less critical to achieving the desired result(s).
It may be necessary to make this an iterative process to help identify group
consensus.
36 3 Designing Complex Development Programs

Is the
external
factor
Yes Will it be
important? realized?

Do not
include in Almost
No the Log certainly
frame

Include
as
Likely
assumpti
on

Is it possible to Unlikely
Yes redesign the
project to
influence the
external factor?

No

Redesign the
project to add
activities or
The project is
results; or
not technically
reformulate the
feasible
purpose

Fig. 3.3 An algorithm for assumptions (adopted from Jackson 2000)

Additionally, this process should make it easier to evaluate performance of such

interventions. But depending on the rigor of the initial process, this may be more
rhetoric than reality. Most of the assumptions we find in project log frames are not
really helpful. They are mostly such truisms as, ‘‘We hope the project will achieve
its purpose if the donor agrees to fund it;’’ or ‘‘There will not be outbreaks of civil
wars,’’ or ‘‘The community will be willing to participate.’’ The table below pro-
vides examples of ill-designed assumptions gone awry.
It is true that if there is an outbreak of a civil war, an intervention aimed at
increasing farmers’ incomes by providing with them seeds is unlikely to succeed.
Simplification or an Extension of Clarity 37

Improved
health status

a
Improved
Increase Water and
health services Sanitation
facilities
d

c
More
trained
TBAs More
Knowledge Increased Adequate More
More knowledge More
on Health immunization water sanitation
issues coverage functional of proper
health supply facilities
facilities sanitation

Fig. 3.4 Results chain of a health project

Linkage Assumptions stated Some additional questions Some observations

area you would ask
A&B Depending on how ‘‘improved Why do we believe that in this This plan is silent about other
health status’’ is defined, community, health depends aspects that influence peoples
increasing access to health mostly on availability of health, especially those
services and water and facilities? related to the individual’s
sanitation facilities would responsibility for wellbeing
improve the health status of
community project
participants.
C Trained Traditional Birth Why do we think that training This plan is silent about
Attendants remain in the TBAs, increasing health quality of health services and
communities where they are awareness, immunization how this could affect demand.
posted coverage and the number of Could it be an issue?
Peace prevails communities functional health facilities are The service oriented approach
are willing to be involved the most appropriate ways of is more aligned with curative
raisings health services than preventive health; does
delivery? the narrative clarify this
What else might hinder the choice?
effectiveness of these
approaches and how would
these limitations be
addressed?
D This linkage shows that a
combination of ‘software’
(education for behavioral
change) and ‘hard ware’
(facilities) is needed for better
sanitation. Not sure if the
narrative recognizes that there
could be barriers to behavior
change and how this would be
resolved among the more as
well as less literate members
of the community.
38 3 Designing Complex Development Programs

It is worthwhile, however, to note the assumption that farmers who have more
seeds will have more income. Depending on the course of the civil war, the
assumption may hold true in general, for a major portion of the country, or only in
scattered areas. Depending on the farmers’ skill with a new crop within any given
area, incomes may increase, decrease, or remain stable. This can be a starting point
for all the assumptions that actually have to be true in order to achieve the desired
outcome. For example, the farmers have to have safe storage for the seed until
planting time; they need to know how to plant these particular seeds properly and
how to care for the growing crops. If the crop is a new one, the farmer(s) and their
family(ies) must have some idea about what its uses are. The assumptions need to
be spelled out. If an assumption is not explicated, it cannot be explored for con-
sequences, good, or ill.
Figure 3.4 shows an example of a project that hopes to improve the health
status of people in a community by increasing health services delivered, which will
be achieved through training more providers and developing more facilities,
among other activities. In this project’s log frame, the key assumptions listed were
‘‘No disease outbreaks’’ and ‘‘Support of government’s health ministry.’’ We
should ask, ‘‘Why have providing health services and sanitation facilities been
identified as the best ways to improve health in this community? Why choose these
approaches over others?’’ Assumptions should not be wishes or hopes; they should
explore in realistic ways what conditions are necessary and sufficient for success in
meeting the project’s goals and what changes—for better or worse—have some
chance of occurring. The current assumptions column does not help us explicate
the assumptions, let alone explore them.

What Else Might Hinder the Effectiveness of These

Approaches (Or Support Their Success), and What Have We
Thought About the Likelihood of Those Factors?

Figure 3.5 shows a project that seeks to improve household food security by
increasing household food availability, as well as reducing post-harvest crop waste.
Why do we believe that the food security question in this community is about
household food availability and post-harvest waste? What else (other than the stated
assumptions) might hinder the effectiveness of this approach and what have we
thought around those factors. Why do we believe that the best way to improve
household food availability in this community is by increasing acreage, incomes, and
access to markets? Why do we choose these approaches over others? Is this rationale
clear in the narrative? How do these pieces work together to bring about the intended
change? To extend the analysis, does the project plan to measure village by village or
province-wide? What differences could this make in the assumptions?
What Else Might Hinder the Effectiveness of These Approaches 39

Improved
household
Food
Security
b

a
Increased Reduced
household post
food
availability harvest waste
d
More
cultivation
c
by
household
Increased More Increased
household More More
access to knowledge
incomes community drying
markets about
stores floors
storage

Fig. 3.5 Results chain of a food security project

Linkage Assumptions Some additional questions you would Some observations

area stated ask (check consistence between
narrative and log frame)
A&B Communities are Why do we believe that the food Increasing household incomes could
wiling to adopt security question in this community is flow from increased agricultural
new technologies about household food availability and production and access to markets
Communities will post harvest waste? What else (other
make material than the stated assumptions) might
contributions hinder the effectiveness of this
approaches and what have we thought
around those factors)
C As above Why do we believe that the best way to Land tenure might be an issue here
improve household food availability in
this community is by increasing
acreage, incomes, and access to
markets? (Why do we choose these
approached over others? Is this rational
clear in the narrative? How do these
three pieces work together to bring
about increase d food availability?)
D As above This linkage shows that a combination
of ‘software’ (education for behavioral
change) and ‘hard ware’ (facilities) is
needed for better sanitation. Not sure if
the narrative recognizes that there
could be barriers to behavior change
and how this would be resolved among
the more as well as less literate
members of the community.
40 3 Designing Complex Development Programs

Gasper (2000) presents some interesting arguments about the superficial

treatment of assumptions in log frames. First, he argues that it is not the problem of
the log frame itself as it is the problem of the users (Gasper 2000, p. 17):
LFA then at least shows up casualness about assumptions and prepares the way for more
serious treatment.

But he also argues that there is a problem with the LFA:

‘Yet it (the LFA) literally marginalizes assumptions analysis, locating it in the final column
of the matrix. The fourth column is the second one to be defined when constructing a
logical framework without noting the incongruity. The bulk of users who came to the
assumptions column fourth tended to be tired and on too different a mental track, after an
arduous traverse of indicators’’.

Another difficulty has been suggested in the way the LFA defines assumptions.
They are considered external factors beyond the control of the project and outside
the project intervention. Knowing such assumptions is critical, but may not help
evaluators in explaining program performance. In the next section, we review
another evaluation approach that takes a different perspective on explicating
assumptions. This approach, the Theory of Change approach, is spawning a lot of
interest, even some prominence, in the evaluation community.
Another weakness in the use of log frames is the inclusion of what Powell
(2011) calls ‘‘redundant results’’ and Bury (2011) terms a ‘‘category error’’. This
arises when a subsidiary result or outcome is placed below a higher level (com-
posite) result or outcome, to which it should contribute, and yet in fact it is simply
part of its definition. For example, a result such as increased awareness of HIV
among schoolchildren and their parents might have below it the results ‘‘increased
awareness of HIV among schoolchildren and increased awareness of HIV among
parents.’’ These two subsidiary results are redundant and should be removed. If
they are the same thing as the composite result, they cannot cause it. Bury (2011)
suggests that a good results framework ensures that the lower level results are (a)
all necessary to achieve the higher level result, (b) sufficient to achieve the higher
level result, and (c) plausibly cause achievement of the higher level result, rather
than merely being categories thereof. Anderson (2004) talks of domino effects in a
results framework. Domino results are those that will emanate from a result
without having to do anything additional. For example, a result above parents’
awareness of HIV may suggest that a result leads to reduction in risky behavior,’’
which is not part of the definition of ‘increasing parents’ awareness of HIV.’’
Powell (2011) identifies a number of scenarios where subsidiary results are not
logically independent of composite results:
(a) When they exactly overlap: If subsidiary results overlap exactly with the
composite result, it is better to do away with the redundant lower layer and
specify those results in the definition of the composite result.
(b) A composite result can include content that is not covered in the subsidiary
results. If the result ‘‘increased parents’ and children’s awareness of HIV’’ were
changed to ‘‘increased awareness of HIV among educational stakeholders’’,
What Else Might Hinder the Effectiveness of These Approaches 41

the awareness levels of parents and children become part of the outcome, but
the result also refers to teachers, principals, education authorities, etc. Now, do
we assume that the subsidiary results (or any other results elsewhere in the
framework) can make a causal contribution to this additional outcome among
parents? If no, then there may be a component of this result that will have no
programming to change and which should just be deleted along with the
entirely redundant subsidiary level. If yes, because it is believed that the
children and their parents will influence other stakeholders, then it should
be clarified that there is an assumption that parents’ and children’s awareness
will be multiplied by passing that awareness along to other stakeholders.
(c) The subsidiary results can include content that is not covered in any of the
composite results. So if the result ‘‘Increased parent and children’s awareness
of HIV was changed to ‘‘increased awareness of Sexually Transmitted
Infections among school children’’, parents’ awareness of HIV is still part of
the definition, but it also includes components that are not covered. It may be
assumed that this additional content in the subsidiary results can make a causal
contribution to the composite (or to any other results in the framework). If no,
then there is content in the subsidiary results that does not lead to higher level
change and which should probably just be deleted (and then the subsidiary
level is entirely redundant and we can delete that too). If yes, perhaps because
the assumption is deemed valid that children’s knowledge of other STDs will
influence or reinforce their or other stakeholders’ awareness of HIV, again the
basis of this expectation should be clarified and logically presented.

The Theory of Change Approach

‘Theories of Change’ (ToC) has proved a popular approach for both evaluators and
commissioners of evaluations for complex social policy programs (e.g., Coote et al.
2004; Judge 2000; Mason and Barnes 2007; Jacobs et al. 2002; Weiss 2000). This
approach has provided program developers with useful tools for working through
designs of complex change processes, by helping them to clarify and reflect on how
selected programming approaches and strategies can or should elicit desired results.
Much work on popularizing the ToC Approach (ToCA) has been done by The
Aspen Institute’s’ Roundtable on Comprehensive Community Initiatives (RTCCI)
and the Kellogg Foundation’s work on community-building and improvement.
These two organizations have approached the ToCA in similar ways.
According to RTCCI, a theory of change explains how a group of early and
intermediate accomplishments sets the stage for producing long-range results.
A more complete theory of change articulates the assumptions about the process
through which change will occur and specifies the ways in which all of the
required early and intermediate outcomes related to achieving the desired long-
term change will be brought about and documented as they occur (Anderson 2004,
p. 3). RTCCI’s approach to theory of change requires stakeholders to be precise
42 3 Designing Complex Development Programs

Long-term
Explanations for Outcome
WHY

Necessary Necessary
Pre- Pre-
Condition Outcomes
Condition
that must
be
achieved
Necessary Necessary Necessary BEFORE
Pre- Pre- Pre- long-term
Condition Condition Condition can be achieved

Fig. 3.6 Pathways of change (adapted from Clark and Anderson 2004)

and clear about the type of changes they want to achieve. Participants in theory of
change workshops are asked to predict exactly who or what is going to change,
over what period of time, and by how much, at every step in an often complex
process. ‘‘We ask them to specify how and why they expect change to happen in a
particular way. We also ask how they are going to bring their resources to bear on
creating early and intermediate changes that add up to their ultimate goal. Simple
questions, in theory…but difficult to answer in practice’’ (Anderson 2004).
RTCCI’s ToC has four components:
(1) A pathway of change that illustrates the relationships among a variety of
outcomes that are each considered preconditions of the long-term goal.
(2) Indicators that are defined to be specific enough to measure results—success
or failure or some intermediate state.
(3) Interventions that are used to bring about each of the preconditions on the
pathway, and at each step of the pathway.
(4) Assumptions that explain why the whole theory makes sense.

Pathway of Change

A pathway of change is a map that illustrates the relationship(s) between actions and
outcomes and also shows how outcomes are related to each other over the lifespan of
the initiative (See Fig. 3.6). The long-term goal of the initiative appears at the top of
the map, and the outcomes that must be produced in order to achieve the goal are
arranged in sequence with the earliest at the bottom and the final outcomes at the top
of the list. This map is then read from the bottom to the top, suggesting that the earliest
outcomes (at the bottom) are needed to arrive at the next level, and outcomes at the
middle level are needed to reach the top (Anderson 2004).
Outcome and Precondition 43

Outcome and Precondition

For RTCCI, everything in the pathway of change is a precondition to the long-term

goal. That is, the outcomes on the path are all required to reach the goal. Without
each of them in place, we assume the goal cannot be attained. It is intended that this
logic helps to weed out extra outcomes that may be nice but unnecessary to achieve
the intended goal. It is not necessarily different from the results chain conceptu-
alized in a log frame. What is different is that the ToCA asks whether these out-
comes or preconditions are necessary and sufficient to reach the long-term goal.

Indicators

Indicators, another component that ToCA and LFA have in common, should help
in recognizing success within the pathway of change. The difference is that with in
ToCA, the indicator (for both the goal and outcomes), might include more detail
that ‘‘operationalizes its definition’’ (Anderson 2004). The indicators are opera-
tionalized by these questions: (a) who or what is the target population for change?
(b) How much change has to occur in this indicator for us to claim to have
successfully reached the outcome? (c) How long will it take to bring about the
necessary change in this indicator in the target population? These questions are
essential in elaborating indicators as a component of the program’s ToC.

Intervention

This component describes the types of actions that would be required to bring
about each of the preconditions on the pathway of change. This may well be
similar to inputs and outputs on the log frame. The difference might be that in the
ToCA, an intervention might be as simple as a single activity or as complex as an
entire program. Each outcome in the pathway of change is linked to a specific
intervention, revealing the often complex web of activity that is required to bring
about the desired long-term community change (Anderson 2004).

Assumptions

This is another component that ToCA and LFA have in common, though the
strategies conceptualize their roles differently. In the ToCA, assumptions explain
both the connections between the preconditions for long-term change occurring in
the early and intermediate stages and the expectations about how and why pro-
posed interventions will bring them about (Anderson 2004). In LFA, the
44 3 Designing Complex Development Programs

Table 3.2 Comparing log frame and theory of change

Log frame Theory of change
Logic models graphically illustrate program Theories of Change link outcomes and
components, and creating one helps activities to explain HOW and WHY the
stakeholders clearly identify outcomes, desired change is expected to come about
inputs, and activities
Logic models usually start with a program and Theories of change may start with a program,
illustrate its components but are best when starting with a goal,
before deciding what programmatic
approaches are needed
Logic models require identifying program Theories of change also require justifications at
components, so you can see at a glance if each step—you have to articulate the
outcomes are out of sync with inputs and hypothesis about why something will cause
activities, but they does not show WHY something else (it is a causal model,
activities are expected to produce outcomes remember!)
Logic Models does not always identify Theories of Change require identifying
indicators (evidence to measure whether indicators that will appropriately monitor
outcomes are met or not) (show that they desired changes/outcomes.
identify indicators differently)
Logic Models are great when you need to: Theories of Change are best when you need to:
Show someone something they can understand Design a complex initiative and want to have a
at a glance rigorous plan for success
Demonstrate you have identified the basic Evaluate appropriate outcomes at the right time
inputs, outputs, and outcomes for your work and the right sequence
Summarize a complex theory into basic Explain why an initiative worked or did not
categories work, and what exactly went wrong
Adapted from Clark and Anderson (2004)

assumptions are always unstated, despite the assumptions column. In LFA,

assumptions are those factors that are external to the program which could affect
the success of the program. These two framings of the assumptions concept are
both important. It seems, though, that those developing these development pro-
grams have been more schooled to the concept that assumptions are external, and
that alone does not help evaluators understand programs performance. ToCA gives
more attention to this long-overlooked issue of assumptions that affect, or could
affect, the internal structure and operations of the program. These assumptions
may be supported by research, or ‘‘best practices,’’ that can strengthen the case for
the plausibility of theory and the likelihood that stated goals will be accomplished.
Three main types of assumptions underlie a theory of change:
1. Assumptions about why each precondition is necessary to achieve the result in
the pathway of change and why the set of preconditions is sufficient to bring
about the long-term outcome.
2. Assumptions from social science theory that connect program activities to
specific outcomes for specific populations or communities. This may include
findings from ‘‘best practice’’ research as well as evidence from academic (or
basic) research.
Assumptions 45

3. External assumptions about the context/environment in which the program or

intervention will be implemented.
When a theory of change is built around the wrong assumptions about the
project’s context or environment, even the most elaborate pathway of change can
fall apart once the incorrect assumptions manifest themselves—not to mention that
a plan based on faulty assumptions is not likely to bring about the desired goal.
Therefore, it is important to have the stakeholder group think critically about what
they consider to be true about the operating environment as well as the links in the
theory of change before they sign off on their theory (Anderson 2004).
RTCCI offers some probing questions that participants may use to elaborate and
check assumptions in the pathways of change:
• When you look at the total picture, do you believe that the theory makes sense?
• Do the preconditions make sense as the logical steps toward the long-term
outcome?
• How will we be able to bring about the outcomes at the levels we have
predicted?
• Is there anything going on in the real world that may make it difficult to get this
theory off the ground the way we have planned it?
• Is this theory of change PLAUSIBLE? Have we created a compelling story
about the pathway of change that would lead to the long-term goal in this
community?
• Is this theory of change FEASIBLE? Do we have the capacities and resources to
implement the strategies that would be required to produce the outcomes in the
pathway of change?
• Is this theory TESTABLE? Have we specified how success will be measured
clearly enough that we can recognize progress toward our goal when we see it?
Have we defined indicators for each outcome in clear terms that a researcher or
evaluator can use to produce a research plan?

Theory of Change Versus Log Frame

There has been much discussion (and confusion) about differences between log
frame and ToC. The table above summarizes the main differences (See Table 3.2).
An illustration by Anderson and Gina (2004) helps to further illuminate the
difference between a ToC and log frame. How often do students need to attend an
after-school program in order for their reading tests scores to improve? A Logic
Model would tell you that the after—school program is an activity and improved
reading scores are an outcome. It might tell you that attendance at the after—
school program is an intermediate outcome. But it would not tell you that
‘‘Students need to attend after-school programs at least three days per week for a
minimum of 60 days, and the curriculum must focus on love of reading and
literacy in order for test scores to rise.’’
46 3 Designing Complex Development Programs

Participatory Impact Pathways Analysis

Participatory impact pathways analysis (PIPA) is an approach to planning, mon-

itoring, and evaluation developed for use with complex research-for-development
projects (See Douthwaite et al. 2003, 2007, 2008; Renger and Titcomb 2002).
PIPA begins with a participatory workshop where stakeholders make explicit their
assumptions about how their project will make an impact, and then produce an
‘‘outcomes logic model’’, and an ‘‘impact logic model.’’
These two logic models provide an ex-ante framework of impact predictions
that can also be used in priority setting and ex-post impact assessment. PIPA
engages stakeholders in a structured participatory process, promoting learning and
providing a framework for ‘‘action research’’ on processes of change (Alvarez
et al. 2010).
The PIPA approach enables project staff and stakeholders to describe jointly the
project’s theories of action, develop logic models, and use them for project
planning and evaluation. The term impact pathways is synonymous with the terms
‘‘theories of action’’ and ‘‘program theory’’. Similar to ToCA and LFA, PIPA
encourages stakeholders to think through how project activity outcomes can lead
to intermediate and long-term changes. With the PIPA approach, a project’s
impact pathways are described in two ways:
1. A problem or objectives tree which describes the linear logic, showing a chain
of results or sequence of changes expected from a project’s outputs, ultimately
leading to the goal.
2. Network maps, showing how project stakeholders work together, influence
each other, and influence the general environment; all of which influence
attainment of project results.
The approach utilizes a workshop process that is designed to help participants
raise, describe, and discuss their hypotheses for how project activities and outputs
could eventually contribute to desired goals. Within the workshop, stakeholders
are facilitated to develop the two perspectives shown in Fig. 3.7, and then integrate
them through developing an outcomes logic model that describes the project
strategies, outputs, and outcomes necessary to achieve the project vision. The
outcomes logic model links the outcomes to the actor or group of actors that will
bring them about.
The description of these hypotheses is a description of the project’s impact
pathways. Participants develop these pathways by:
1. Clarifying, reaching mutual understanding, and communicating their project’s
intervention logic and its potential for achieving impact;
2. Understanding other projects working in the same program, and identifying
areas for collaboration;
3. Generating a feeling of common purpose and better programmatic integration
(when more than one project is represented in the workshop);
Participatory Impact Pathways Analysis 47

Identifying Linear Logic

1. Problem (Sequence of changes from
Tree outputs to goal)
(Understanding Identifying the
the needed evolving network
change) of stakeholders
2. What the 3. Vision needed to achieve
project will (where the the vision
produce project is
going

4. “Now” Integration of
Network map 6 “Future” the linear
(Existing Network Map
(Necessary logic and the
Relationships network map
relationships for
essential for achieving the
Vision)

5. Key
Changes

8. Timeline, targets
and milestones
7. Outcomes Logic (Linking activities to
Model (the outcomes outcomes, sets targets
that the project will and milestones, and
help achieve -how and formsthe basis ofan
with whom) evaluation plan)

Fig. 3.7 The PIPA workshop (adapted from Alvarez et al. 2010)

4. Developing a narrative describing the project’s intervention logic and possible

future impacts (thus a form of ex-ante impact assessment);
5. Developing a framework for subsequent monitoring and evaluation.
There are six main steps in the process:
Step 1: Participants develop a problem tree which links the problems that the
project is directly addressing with the social, environmental, and economic con-
ditions that it wishes to improve.
Step 2: The problem tree is turned into an outcome tree that represents the
situation after problems have been addressed.
Step 3: The outcome tree is converted into an outcomes logic model that
describes changes in knowledge and attitudes and skills and practices.
Step 4: Strategies to bring about these changes are devised and articulated.
Step 5: A monitoring and evaluation plan and a schedule of activities are
developed.
Step 6: The plan is executed.
The changes listed in the outcomes logic model are ones that are possible, at a
stretch, to achieve within the timeframe of the project. They generally describe the
48 3 Designing Complex Development Programs

expected direct benefits of the project and can be evaluated through the monitoring
and evaluation system described above.

Comparing PIPA, LFA, and ToC Approaches

All three approaches—PIPA, LFA, and ToC—emphasize participatory processes

in which stakeholders identify problems and devise strategies for achieving out-
comes and outlining pathways by which these outcomes would be achieved.
However, completing formats of a traditional logical framework is difficult to do in
groups and is thus often dominated by one or two people (Douthwaite et al. 2007).
Groups are less constrained when they use flow diagrams used in PIPA and ToC
processes to map pathways of change; the skills of a facilitator not withstanding. In
the PIPA workshop, participants only attempt to create a logic model once the
underlying impact pathways have been discussed and agreed.
The impact logic model is a flowchart that shows the chains of outcomes that
link outputs to eventual developmental impacts. The other element that makes
PIPA useful is the use of network maps to complement logic models in explaining
impact pathways. A net work map is a flow chart that shows the chains of out-
comes that link outputs to eventual developmental impacts. The network maps
give additional detail to the causative theory, given the principle that the inter-
actions among stakeholders are key to the scale-up and scale out of intervention
effects (Douthwaite et al. 2007). Network maps are used to explore how stake-
holders are linked to and influence each other, and how the project aims to change
the existing networks (Douthwaite et al. 2007).
All three approaches—LFA, ToC, and PIPA—provide opportunities for
unearthing stakeholder assumptions. The LFA approach tends to focus more on
external assumptions—the factors beyond the control of project implementers that
influence the success of the project. While those are important assumptions to
consider, the assumptions related to how program outputs will be transformed into
desired long-term changes are often not unearthed, and these are more dealt with
by the ToC and PIPA approaches. The reflection on both the results and rela-
tionships aspects of planned change with in PIPA provides many opportunities for
unearthing stakeholder assumptions.
It is also important to note that creativity and facilitation skills account for
much of what any of these three approaches can provide in designing complex
development programs; sometimes it may necessitate a mixture of more than one
design approach depending on the complexity of program’s internal and external
environment. Just as the choice of a program design approach needs to respond to
contextual complexity; the choice of program evaluation methodology needs to be
appropriate to a program’s complexity. The next chapter discusses the need to
navigate the evaluation methodology toolbox guided by the ‘‘appropriateness’’
principle.
References 49

References

Ahuja, G. (2000). Collaboration networks, structural holes, and innovation: A longitudinal study.
Administration Science Quarterly, 45(1), 425–455.
Abonyi, G., & Howard, N. (1980). A Boolean approach to interactive program planning.
Management Science, 26(7), 719–735.
Alvarez, S., Douthwaite, B., Thiele, G., Mackay, R., Co0 rdoba, D., & Tehelen, K. (2010).
Participatory impact pathways analysis: a practical method for project planning and
evaluation. Development in Practice, 20(8) 946–958.
Ambrose, K. & Roduner, D. (2009). A conceptual fusion of the logical framework approach
logical framework approach and outcome mapping. OM Ideas Paper No. 1.
Anderson, A. (2004). Theory of change as a tool for strategic planning: a report on early
experiences. Aspen Institute: Washington, DC.
Bury, B. (2011). Response to Steve Powell on illogical frameworks, composite results and
logframe bloat. Retrieved on December 12 2011, from [Link].
Chandrasekhar, A. G., Kinnan, C. & Larreguy, H. (2011). Informal insurance, social networks,
and savings access: Evidence from a lab experiment in the field. MIT Working Paper.
Clark, H., & Anderson, A. (2004, November). Theories of change and logic models: Telling them
apart. Presentation at American evaluation association annual conference. Atlanta, Georgia.
Coote, A., Allen, J., & Woodhead, D. (2004). Finding out what works. London: Kings Fund.
Douthwaite, B., Schulz, S., Olanrewaju, A. S., & Ellis-Jones, J. (2003). Impact pathway
evaluation: an approach for achieving and attributing impact in complex systems. Agricultural
Systems, 78, 243–265.
Douthwaite, B., Alvarez, S., Cook, S., Davies, R., George, P., Howell, J., et al. (2007).
Participatory impact pathways analysis: a practical application of program theory in research-
for-development. The Canadian Journal of Program Evaluation, 22(2), 127–159.
Douthwaite, B., Alvarez, S., Thiele, G., & Mackay, R. (2008). Participatory impact pathways
analysis: A practical method for project planning and evaluation ILAC Brief May 17 2008
The Institutional Learning and Change (ILAC) Initiative ([Link]).
Hummelbrunner, R. (2010) Beyond logframe: Critique, variations and alternatives in Nobuko
Fujita, beyond logframe; Using systems concepts in evaluation, Foundation for advanced
studies on international development, Tokyo, pp. 1–34.
Bakewell, O., & Garbutt, A. (2005). The use and abuse of the logical framework a review of
international development NGOs’ experiences. Stockholm: SIDA.
Davies, R. (2004). Scale. Complexity and the representation of theories of change, evaluation,
10(1), 101–121.
Davies, R. (2005). Moving from logical to network frameworks: A modular matrix approach to
representing and evaluating complex programs, in Creative Commons Attribution-
NonCommercial-ShareAlike site available online at: [Link]
search?q=cache:cs98bp1tuXgJ:[Link]/docs/
[Link]+network+software+Visualyzer&hl=en&ct=clnk&cd=2&gl=uk
Deprez, S., Van Ongevalle, J. (2006a). Building reflection and learning into educational programmes,
introducing outcome mapping—the case of St2eep. Conference Proceedings, International
Conference on Strengthening Innovation and Quality in Education, Leuven, Belgium.
Deprez, S., Van Ongevalle, J. (2006b) Outcome mapping: Learning the way forward—an
alternative way to plan, monitor and evaluate programmes. Conference Proceedings,
Environmental Education Association of Southern Africa. Harare, Zimbabwe.
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection
into development programs. Ottawa, ON: International Development Research Centre.
European Communities (EC) (1999). Project cycle management training handbook. Brussels:
Author.
Gasper, D. (1997). ‘Logical frameworks’: A critical assessment. Working Paper no. 278. The
Hague: Institute of Social Studies.
50 3 Designing Complex Development Programs

Freeman, L. (2006). The development of social network analysis. Vancouver: Empirical Press.
Gasper, D. (2000). Evaluating the ‘logical framework approach’ towards learning-oriented
development Evaluation. Public Administration and Development., 20(1), 17–28.
Jackson, B. (2000). Project designing projects and project evaluations using the logical
framework approach. Retrieved October 12 2011, from [Link]/
[Link].
Judge, K. (2000). Testing evaluation to the limits: The case of english health action zones’.
Journal of Health Services Research and Policy, 5(1), 1–3.
Hogan, B., Carrasco, J., & Wellman, B. (2007). Visualizing personal networks: Working with
participant-aided sociograms. Field Methods, 19(2), 116–144.
Jacobs, B., Mulroy, S., & Sime, C. (2002). Theories of change and community involvement in
North Staffordshire health action zone. In L. Bauld & K. Judge (Eds.), Learning from health
action zones. Chichester: Aeneas Press.
Johnson, M. A., Casillas, W., Brown, J. U., & Trochim, W. (2011, November). Using systems
thinking in evaluation capacity building: The systems evaluation protocol. Paper Presented
American evaluation association annual conference, Anaheim CA.
Kibel, B. (2000). Outcome engineering toolbox: User manual. Retrieved June 15 2011, from
Pacific Institute for Research and Evaluation Web site: [Link]
[Link].
Leischow, S. J., Best, A., Trochim, W. M., Clark, P. I., Gallagher, R. S., Marcus, S., et al. (2008).
Systems thinking to improve the public’s health. American Journal of Preventive Medicine,
35(2), 196–203.
Mason, P., & Barnes, M. (2007). Constructing theories of change: Methods and sources.
Evaluation, 13(2), 151–170.
Moody, J., & White, D. R. (2003). Structural cohesion and embeddedness: A hierarchical concept
of social groups. American Sociological Review, 68(1), 103–127.
Owen-Smith, J., & Powell, W. W. (2004). Knowledge networks as channels and conduits: The
effects of spillovers in the Boston Biotechnology community. Organization Science, 15(1),
5–21.
Powell, S. (2011). Illogical frameworks, composite results and log frame bloat. Retrieved
December 12 2011, from [Link].
Renger, R., & Titcomb, A. (2002). A three-step approach to teaching logic models. American
Journal of Evaluation, 23, 493–503.
Richard, R. F. (2009, November). The logic model and systems thinking: Can they co-exist?
Paper presented at the American Evaluation Association Conference. Orlando FL.
Rockwell, S. K., & Bennett, C. F. (2000). Targeting outcomes of programs. Retrieved July, 2011
from University of Nebraska, Institute of Agriculture and Natural Resources. [Link]
edu/TOP/
Rowlands, J. (2003) Beyond the comfort zone: some issues, questions, and challenges in thinking about
development approaches and methods. In D. Eade (ed.), Development methods and approaches:
Critical reflections. A development in practice reader (pp. 1–20). Oxfam GB, London.
Trochim, W. M., Cabrera, D. A., Milstein, B., Gallagher, R. S., & Leischow, S. J. (2006).
Practical challenges of systems thinking and modeling in public health. American Journal of
Public Health, 96, 538–546.
Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications.
Cambridge: Cambridge University Press.
Williams, B., & Iman, I. (2008). (eds.). Systems concepts in evaluation: An expert anthology.
Point Reye, CA: Edge Press of Inverness.
William, B. (2011). Comments on a blog. [Link]
Weiss, C. (2000). Which links in which theories shall we evaluate? New Directions for
Evaluation, 87(Fall), 35–45.
World Bank (2009). Interactive community planning-upgrading Urban communities. Retrieved
September 23 2011, from [Link]
[Link].