PROGRAMME

INDICATORS
Some programme indicators are designed to be collected directly at programme level. Others are designed
to capture, summarise or aggregate information collected from projects or partners within a programme.
In these cases it is important to be clear about how information should be transmitted between the
different levels, and who is responsible for making sure it happens.

Indicators are developed, collected, analysed and reported Case 1: Independent indicators
in most social development projects. They can also be
developed and used at programme level. Programmes may
range from a collection of a few projects working on a
common theme to large global, regional, country or sector
programmes.

Sometimes, programme indicators are not designed to be

collected directly, but instead rely on information that is
collected at project or partner levels, and then transmitted
up to programme level. If a civil society organisation (CSO)
plans to use this kind of programme indicator, it is
important that it develops processes that enable
information to be transmitted between the different levels
of the programme. The risk otherwise is that nobody takes
responsibility for collecting the indicators, and the
monitoring and evaluation (M&E) system for the Programme indicators Project indicators
programme falls into disuse as a result.
• % of people in an • # of water points
area living within installed
The five different cases outlined in this paper illustrate 2km of a protected • # of water committees
ways in which programme indicators can be linked to water source supported
project indicators. The principles can be applied within any
type of programme, ranging from a small programme
working in one locality through to a large global, regional,
national or sector programme. The first three cases are the Case 2: Framing or basket indicators
most common.
Framing or basket indicators, sometimes also known as
bucket indicators, are not specific and cannot be collected
There is no standard terminology for directly at programme level. Instead they are used to
describing different types of programme identify broad areas or domains of change. More specific
indicators. They are known by different indicators at project level are then captured and
names in different organisations. summarised under the framing indicators.

In the example shown on the following page, the

programme-level framing indicator is CSOs influence
Case 1: Independent indicators government practices at local levels. This kind of indicator
cannot be collected directly. Instead, it relies on specific,
In the first case, programme and project indicators are
tangible indicators being developed and reported at project
collected independently. Programme indicators are
level, such as:
collected by programme staff using programme resources.
In the example shown in the diagram above, the % of • # of times government invites CSOs to attend
people in an area living within 2km of a protected water meetings to discuss policy.
source could be collected at programme level via a survey, • Evidence of CSO submissions being copied into
or by using a system of water point mapping. Different government policy documents.
projects might then define, collect and analyse their own • Successful lobbying of local government by CSO.
indicators, such as the number of water points installed or
Evidence produced through these three project indicators
number of water committees supported. Even though the
could all be collected under the one framing indicator at
results at programme level may depend partly on activities
programme level, thereby providing examples of the type
carried out at project level, the two sets of indicators are
of change that a programme is achieving.
not formally linked.
© INTRAC 2023
 Case 2: Framing or basket indicators Case 3: Aggregated indicators
Indicators from different projects can also be added up at
programme level to generate an aggregated indicator. In
Specific indicators from projects (B, C & D) are
collected together under programme indicator A. this case the indicator must be defined in exactly the same
way at both project and programme level (see diagram
below). An example of an aggregated indicator is # of
farmers trained in land management. Aggregated indicators
are always quantitative.

If a CSO intends to use an aggregated indicator, it needs to

ensure that the indicator is applied in exactly the same way
throughout all the projects contributing to the programme
indicator. In practice this means that:
• the same indicator definition needs to be used;
• all relevant projects need to collect information on
the indicator;
• the same (or similar) tools and methods need to
be used to collect the information; and
Programme indicator Project indicators • the quality of information collection needs to be
• CSOs influence • # of times governments consistent.
government invite CSOs to attend
practices at local meetings to discuss The risk otherwise is that different numbers are added
levels (A) policy (B)
together to produce a meaningless total. In the example
• Evidence of CSO
submissions being provided above, it would not make sense to add together
copied into government numbers from different projects if there were different
policy documents (C) interpretations of what was meant by ‘farmers’, ‘training’
• Successful lobbying of or ‘land management’.
local government by
CSO (D) If a programme wants to aggregated impact- or outcome-
level indicators there are two more conditions. First, the
change covered by the indicator needs to be assessed over
It is important to note that framing indicators are rarely similar timescales. And second, the contribution of projects
used to measure programme change. Instead, they are to the change should be broadly similar. For example, it
used to generate examples or illustrations of the type of would not make sense to add together an indicator such as
changes that are occurring within a programme. A framing
indicator is basically a way of collecting together and
summarising a number of specific changes under a
common theme. Case 3: Aggregated indicators

Project indicators feeding into a programme-level framing

indicator may be either quantitative or qualitative. In fact, Indicators from projects are aggregated together
they need not necessarily be indicators at all, and to generate programme indicator (A). All
information on unexpected or negative changes can also be indicators are defined in exactly the same way.
mapped onto a framing indicator. One advantage of using
framing indicators is that it leaves project staff free to
collect indicators which are useful to them, without A
worrying about complying with programme requirements.
Indeed, there is no theoretical reason why project staff
should be aware of the existence of the programme-level
framing indicator at all!

Programme framing indicators may be defined as

qualitative indicators as shown in the example above (CSOs A A A
influence government practices at local levels). However,
some programmes may choose to turn them into
quantitative indicators by counting the number of cases Example indicators at both programme and project
that occur. In that case the framing indicator could equally level
well be defined as: • # of local government decisions influenced by CSOs
• # and description of cases where CSOs influence
• # of children (boys/girls) enrolled in new schools
government practices at local levels.
• # of farmers with increased crop yields > 30%
• # of examples showing how CSOs have influenced
• # and % children 0-23 months who are underweight
government practices at local levels. (weight for age), by gender

© INTRAC 2023
 Case 4: Translated indicators
Conditions for aggregating indicators from project to
programme level Translated indicators are less common, and can be seen as
a halfway house between cases 2 & 3. Projects still report
For all indicators:
on different indicators (as in case 2). However, those
➔ The same indicator definitions need to be used
indicators are then ‘translated’ into an aggregated indicator
➔ All relevant projects need to collect information on the
(as in case 3) later in the process. This can either be done
indicators
by the project staff themselves, or, more commonly, later
➔ The same (or similar) tools and methods need to be used
on by programme staff.
➔ The quality of information collection needs to be
consistent In the example shown below, three different project
For impact- and outcome-level indicators: indicators are all converted into a common indicator before
➔ The change covered by the indicator needs to be being aggregated. For instance, programme staff could
reported over similar timescales measure or estimate the number of households with access
➔ the contribution of projects to the change should be
to newly installed water points (indicator B). It should also
broadly similar
be possible to measure or estimate the number of
households being served by wells treated for
contamination (indicator C). It might be harder to estimate
the number of households with increased access to an
# and % of women who are engaged in meaningful improved water source as a result of increased awareness
employment from two different projects if one was based of hygiene around water sources (indicator D), but it would
on in-depth, direct support from a CSO over many years, still be possible. After translation, the three numbers could
and the other involved limited, indirect capacity building then be aggregated under the common programme
support in a project that was only just beginning. indicator # of households with access to an improved water
source (A).
CSOs often want to use aggregated indicators to enable
them to summarise performance across different
interventions in different locations. However, aggregating
indicators is never as easy as it sounds. In practice, it is Case 4: Translated indicators
essential to ensure that indicators are comprehensively
defined at the start of a programme, so that all project staff
can work to the same indicator definitions. Different indicators from projects (B, C & D) are
translated into a common indicator (A) and then
A major implication of this approach is that all relevant aggregated to programme level.
partners and projects need to be told (or persuaded) to use
the indicators, whether or not they are of any use to them
in their own work. At best, this can result in unnecessary A
administrative burdens. At worst, it might mean projects
and/or partners do not bother to ensure that information is
collected accurately.

Aggregation is more commonly applied at activity or output A A A

levels; for example counting the numbers of schools built,
credit groups supported, or intended beneficiaries reached
across a portfolio. It is less commonly used at outcome
level, and is used mainly in situations where there are
B C D
industry standard indicators, and when the time gap
between outputs and outcomes is relatively short (e.g.
survival rates for operations or weight increase in nutrition
programmes). Programme indicator Project indicators
• # of households • # of safe water points
It is much harder to aggregate indicators if trying to with access to an installed in region (B)
measure complex or intangible outcomes, or if there are improved water • # of wells that have been
long timescales between activities carried out and desired source (A) treated to remove
changes. Consequently, aggregation is rarely applied at contamination (C)
• # of communities with
outcome level in complex sectors of work such as
increased awareness of
governance, conflict resolution or the mobilisation of civil hygiene around water
society, where outcomes (and therefore indicators) need to sources (D)
be more specific to the context, and contribution is harder
to assess.

© INTRAC 2023
CSOs use translated indicators within social development in
Case 5: Cluster indicators
many different ways. Some of the most common are as
follows.

▪ In cost-benefit analysis or social return on investment, Different indicators from projects (A, B & C) are
different kinds of social benefits are routinely brought up to programme level and reported as
programme indicators.
translated into monetary values to enable aggregation
and comparison.
▪ Health programmes often convert different indicators
collected within projects into DALYs (disability-adjusted A B C
life years) – understood as the additional number of
years of healthy life for a person.
▪ Rating indicators may be used to aggregate and
compare performance based on different project
indicators. For example, project performance can be
translated into a RAG (red, amber, green) rating, or A B C
expressed on a scale of ‘1’ to ‘5’, and then aggregated
into a programme indicator.
Programme indicators Project indicators
The main advantage of using translated indicators is that
some level of aggregation can be achieved even if project • # and description of • # and description of
government policies government policies
indicators have not been defined or collected in the same that have been that have been
way from the start of a programme. However, using influenced by CSOs influenced by CSOs
translated indicators is not always easy, and can require a (A) (A)
lot of time and energy. As with aggregation, it should not • # of advocacy • # of advocacy
be undertaken lightly. campaigns designed campaigns designed
and implemented (B) and implemented (B)
• # of meetings held • # of meetings held
with local government with local government
Case 5: Cluster indicators officials (C) officials(C)
In the final case, some programmes define indicators that
are essentially project indicators which have been brought
up to programme level. This is often done in programmes
which bring together a group of projects for marketing or “If a CSO wishes to link
communications purposes, or for management indicators across different
convenience. In the example provided in the diagram
levels of a programme then it
opposite, three different indicators collected by projects
have been brought up to programme level. Each indicator, needs to create the
when reported at programme level, only covers the work of mechanisms to do so. It will
an individual project.
not happen on its own.
Indicators are not magnets.”
Summary
Programme indicators are not always defined and used in
the same way as project indicators. In some cases a great
deal of thought needs to go into developing programme
indicators and ensuring they are linked to indicators at
project or partner levels. Programmes often use a
combination of the five cases described in this paper to
provide a range of information, some of which is reliant
on information coming through project indicators and
some of which can be collected independently at
programme level.

Further reading and resources

Further information on how to aggregate and summarise information across different levels of an organisation can be found in
the M&E Universe paper on aggregation and summarisation. This can be accessed by clicking on the link below.

© INTRAC 2023
The material in this paper is mostly taken from a paper called “Summarisation of Portfolio Change” by Nigel Simister, published
in 2016. This can be found on INTRAC’s website at [Link]
[Link].

Author(s): INTRAC is a values-based, not-for-profit organisation with a mission to strengthen civil society
Nigel Simister organisations. Since 1991, INTRAC has contributed significantly to the body of knowledge on
monitoring and evaluation. Our approach to M&E is practical and founded on core principles. We
encourage appropriate M&E, based on understanding what works in different contexts, and we work
with people to develop their own M&E approaches and tools, based on their needs.

M&E Training & Consultancy M&E Universe

INTRAC’s team of M&E specialists offer consultancy and For more papers in
training in all aspects of M&E, from core skills development the M&E Universe
through to the design of complex M&E systems series click the
Email: info@[Link] home button