I

A guide to support Value

for Money (VfM) analysis
for public managers
July 2021

The
1
Chartered Institute of 
A guide to support value for Money (VfM)
Public Finance
analysis & managers
for public Accountancy
 This guide complements a VfM toolkit
which has been published separately.
Both were developed under a
collaborative project between
Government Outcomes Lab (GO Lab)
and CIPFA, the Chartered Institute of
Public Finance and Accountancy.

The project was funded by the

Engagement Fellowship grant from the
University of Oxford’s Higher Education
Innovation Fund (HEIF).

Both the guide and toolkit are in

beta versions and are open for public
comments. We welcome your feedback
to revise and improve them in future
iterations.

2 A guide to support value for Money (VfM)

analysis for public managers
 Project team
Dr Mehdi Shiva (economist, Jeffrey Matsu (Chief
GO Lab) managed and co- Economist, CIPFA) co-designed
designed the project, assisted the project, led policy outreach,
with drafting the guide and co-organised and co-facilitated
the toolkit, co-organised and the roundtable, reviewed and
co-facilitated the roundtable, edited the toolkit and the guide.
arranged user experience
interviews, reviewed and edited
the toolkit and the guide.

Mr Chevano Baker (GO Lab) Ms Ruby Dickson (GO Lab)

drafted the first version of the revised the toolkit and the guide,
guide and the toolkit, provided developed the case study,
analytical support, and assisted recorded the video instruction,
with presentations, organising and assisted with presentations,
events and user experience organising events and user
interviews. experience interviews.

Dr Mara Airoldi (GO Lab)

reviewed and edited the guide
and the toolkit, and provided
oversight for the project.

Further information
email M
 [Link]@[Link] or
[Link]@[Link]

3 A guide to support value for Money (VfM)

analysis for public managers
 Acknowledgement
The project team are grateful to the experts who
participated in the roundtable. We are especially
thankful to Neil Stanworth, Nicola Lycett, Bilal Dunoo,
Janice Hadfield and Janet Bowlzer for using the toolkit
with real data and sharing their experience and
feedback. We also appreciate all the help and support
from both the GO Lab and CIPFA teams.

4 A guide
A guide
to support
to support
prospective
value for Money
Value for
(VfM)
Money (VfM)
analysis
analysis
forfor
public
public
managers
managers(beta)
 Contents

I Overview 6
IV The SIB tab 22

II Introduction to
prospective VfM
9
V Appendix 23

assessments

III VfM assessment:

a step-by-step
14

guide to the 4Es

framework

5 A guide to support value for Money (VfM)

analysis for public managers
 I

Overview
This guide is aimed at public managers planning to assess Value for Money
(VfM) of outcomes-based contract (OBC) programmes, or any other type of
programme with an outcome-focus, using prospective information. This involves
assessing economic validity of the programme with respect to ‘doing nothing’ as
well as the closest comparator.

This guide will: This guide will not:

• Describe what VfM represents in • Provide a framework to support the

public provision of social services with choice of service delivery model (ie
a special focus on outcome-based sourcing or procurement strategy).
contracts (OBCs). In particular this The delivery model assessment,
guide emphasises the link between which used to be known as ‘make vs
economy and effectiveness criteria. buy’, is part of another framework that
we are planning to develop. The UK
• Promote thinking about longer-term
effects of interventions, such as Government’s Sourcing Playbook also
outcomes and impact, at the design/ provides some guidance on that.
planning stage of programmes. • Provide a comprehensive stand-alone
This means that having a good source for decision making. There are
appreciation for efficiency is helpful many other considerations you need to
but not necessary, especially when take into account apart from economic
outcomes are both identifiable and validity.
measurable.

• Explain how it could be used to What When

appraise public programmes with
VfM Guide & Proposal/
respect to anticipated costs and value
Toolkit Planning
of them using prospective information. Stages
• Introduce our VfM toolkit (Excel
Workbook), which is a step-by- Who How
step and structured framework
for conducting a VfM assessment. Public Qualitiative and
managers Quantitative
Notably, this toolkit at the current Assessments
version is intended for learning and
self-assessment and should not be
taken as the sole source for decision Why
making. The toolkit promotes a
To assess the optimal use of resources
wholesome approach and takes to achieve the intended outcome
advantage of the full spectrum of
VfM criteria. Figure 1: Intended context and purpose of
the VfM Guide and toolkit.

6 A guide to support value for Money (VfM)

analysis for public managers
 I | Overview

I.i. How is this guide [Link]. What does VfM mean in

different from other VfM general public accounting
guides? contexts?
This guide supports outcome-orientation, The term ‘Value for Money’ (VfM) is
uses prospective (ex-ante) information used widely among decision makers,
to inform decision making at planning but its meaning varies just as widely
stage, and is specified at a ‘programme’ depending on the user. Often it is simply
level. It is written both as a stand-alone used interchangeably with ‘efficiency’ –
guide to VfM assessments and the steps best use of inputs to get the maximum
within them, and also as a companion to output. CIPFA’s VfM toolkit, for instance,
the GO Lab-CIPFA VfM toolkit. This guide assists UK local authorities to identify
support ‘value boost’ as an important opportunities for improving efficiency
consideration at the planning stage, along by comparing their spending and key
with ‘cost control’. This perspective is performance indicators to those of
opposed to the ‘cost squeeze’ approach. similar councils.
VfM assessments often involve trade-offs
between the 4Es. For instance, a narrow However, the definition of VfM is much
focus on economy could lead to cost- broader in both national and international
minimisation at the expense of justifiably frameworks. For example, the official
better outcomes. Balancing these criteria UK definition from the National Audit
to ensure that VfM considers the overall Office (NAO) describes VfM as “the
benefits and costs of an intervention optimal use of resources to achieve
is paramount. intended outcomes”. The NAO outlines
four core criteria against which VfM is
Both this guide and the toolkit are assessed (the 4Es): economy, efficiency,
optimised for discrete programmes effectiveness and equity. The extent
with known – projected or estimated to which a VfM assessment covers all
– parameters, namely timelines, cost four criteria depends on the context
drivers, outputs and outcomes. For the of the programme. It might be that an
beta version, we have specially focused assessment of efficiency is more relevant
on social and health programmes and if accurate information on outcomes
test-run the toolkit on multiple occasions. is not available. However, the equity
We believe, however, that our toolkit consideration is often neglected.
is applicable to broader use for public
programmes. In case there are limitations
to using this for other policy domains, we
would like to hear your thoughts.

7 A guide to support value for Money (VfM)

analysis for public managers
 I | Overview

[Link]. What does VfM This makes assessment of the

represent in an OBC effectiveness criterion more

context?
straightforward, as outcomes are
already identified. We like to encourage
identifying outcomes even for other types
VfM poses an evaluative question
of service delivery methods (like in-house
about how well resources are used
or fee-for service). In OBCs, outcomes
in the delivery of a programme and
are identifiable, mainly quantifiable and
whether the usage is justified. A VfM
measurable. Hence a VfM assessment
assessment seeks to assess the use of
for OBCs helps to determine how to
resources in maximising the outcomes
maximise those outcomes for given inputs
of a programme so that more informed,
and outputs. All of these differences
evidence-based decisions can be made.
provide a facilitating opportunity for
running a thorough VfM assessment with
For OBCs in particular, a clearly
more confidence.
articulated VfM assessment will also
facilitate planning clear monitoring,
evaluation and outcomes verification
strategies, as payment is fully or partially
attached to the success of them.

8 A guide to support value for Money (VfM)

analysis for public managers
 II

Introduction to
prospective VfM
assessments
This guide aims to provide an overview of things to consider when preparing
a prospective evaluation plan for OBCs. The purpose of a prospective VfM
assessment is mainly to support decision-making at the design/planning/
proposal stage, similar to feasibility analysis or preparing a business case.
This would help decision makers to better justify the use of OBCs in their
intended programmes and/or adjust particular aspects to achieve VfM.

There are certain factors you will need • Prospective VfM assessment
to consider when conducting VfM using requires handling both historic and
prospective information: projected data, which may require a
certain level of data skills. Our toolkit
• The VfM assessment will require data
supports high-level data analysis
on the programme’s costs, benefits or
but leaves the detailed estimates to
value, and price of outcomes. These
internal capacity. In cases where data
may be retrieved from budgeted
is not quantifiable (eg outcomes) and/
data or estimations/projections
or comparable (eg 'complex products’,
using ‘historic’ information, as actual
where there are multiple individuals,
evidence can only be obtained after
groups and/or goals), judgement may
implementation.
need to rely on qualitative measures.
• Appropriate benchmarks are helpful However, the results of a mixed
to use as comparators. It may be method approach (qualitative and
challenging to find information on quantitative) are much more reliable
outcomes given their novelty, and to fill data gaps, cross-check findings
consequently lack of comparator and provide validity on data used to
data and information. CIPFA’s VfM make informed judgements.
toolkit, mentioned above, contains a
• Generating prospective estimates
rich dataset of various performance
of outcomes and impacts is even
indicators and unit costs for local
more challenging under uncertainty,
councils, which you may use
for example in fragile environments
for benchmarking. If there is no
where unexpected events are likely to
comparator (ie a programme that
occur. Also, in face of crisis (such as a
shares the same outcomes), then one
pandemic), the results might diverge
solution is using average cost and
substantially from the expectations.
performance values of a set of similar
Sometimes it is helpful to additionally
programmes within the sector that
work out a set of best and worst-case
are not commissioned using OBCs.
scenarios to be better prepared.

9 A guide to support value for Money (VfM)

analysis for public managers
 II | Introduction to prospective
VfM assessments

In the following, we describe how to prepare for a VfM assessment. This guide is
structured in a way to match the toolkit formatting. Please note that headlines are
labelled to correspond to the toolkit, namely letter ‘Z’ corresponds to miscellaneous,
‘A’ to Economy, ‘B’ to Efficiency, ‘C’ to Effectiveness, and ‘D’ to Equity.

II.A.i. Stages of a VfM assessment

A VfM assessment using prospective information can be conducted in two stages:
(1) VfM framework design and (2) VfM assessment. Each stage consists of three
sequential steps as shown in Figure 2.

VfM Framework design VfM Assessment

VfM Determine Analysis,

Theory of Gather
criteria and evidence synthesis and Reporting
change evidence
standards needed judgement

Figure 2: Stages in prospective VfM evaluation (adapted from King and OPM, 2018) King, 2019.

[Link]: Theory of change Outputs are the goods or services that

the programme activities produce (eg
A ‘theory of change’ (ToC) describes number of sessions held) while outcomes
the sequential steps of programme are the eventual goals of the programme
development, namely inputs, activities, (eg improved mental health). Impacts
outputs, outcomes and impact. See are longer-term effects of a programme
New Philanthropy Capital’s practical and could be measured using ‘impact
guide and the Social Impact Bonds measurement’ methods (see GO Lab’s
(SIBs) provider toolkit for how to create Impact Wayfinder to find the most
a ToC. Prior to assessing the VfM of relevant resources tailored for any
your programme, you should generate context). Also, see GO Lab’s setting and
and review the programme’s theory of measuring outcomes guide to learn
change to understand how and why a further about outcomes.
desired outcome/impact is likely to occur
in a given context.

From a commissioning standpoint, it

is important to distinguish between
inputs, outputs and outcomes. Inputs are
resources invested in the programme to
deliver the outputs (eg number of staff
employed for a programme).

10 A guide to support value for Money (VfM)

analysis for public managers
 II | Introduction to prospective
VfM assessments

As the below diagram indicates, each

criterion of VfM in connected to specific
stages of ToC, and there are different
methods of evaluation depending on the
evaluation plan.

spending fairly

Equity

spending less spending well spending wisely

VfM
Criteria Economy Efficiency Effectiveness

Theory of
Issue Input Activity Output Outcome Impact
Change

Cost-efficiency

Evaluation Cost-effectiveness
Methods

Cost-benefit

Figure 3: Augmented VfM theory of change.

11 A guide to support value for Money (VfM)

analysis for public managers
 II | Introduction to prospective
VfM assessments

[Link]: Evaluation methods Each evaluation method could be

prospective, mid-term or retrospective.
There are different ways to evaluate Our toolkit calculates and presents
a programme. Most common methods prospective measures for all the above
are cost efficiency, cost-effectiveness methods on the summary tab. For
analysis (CEA) and cost-benefit analysis more detailed guidance on conducting
(CBA). The right choice of evaluation these evaluations, see the Treasury’s
method depends on the aims and Green Book, J-PAL’s Guide to Cost
objectives of the programmes as well as Effectiveness and Greater Manchester
the available information and context. For Combined Authority’s CBA model.
example, CEA might be better than CBA Additionally, GO Lab’s Impact Wayfinder
when comparing interventions that you tool can help you find appropriate impact
cannot monetise. However, CBA may be evaluations according to the focus of
better when making decisions between your organisation and the scale of your
programmes with either the same or programme.
different outcomes. We encourage
avoiding cost-efficiency analysis alone
when other methods are possible.

[Link]: VfM criteria and standards

Criteria selection
First, select the criteria for identifying the evidence needed, interpreting the evidence
and then making clear judgements about performance. The UK government’s 4Es
(economy, efficiency, effectiveness and equity) framework – also referred to as 3Es,
as ‘equity’ is sometimes perceived as optional – is generally used as the criteria to
assess VfM. Programme-specific definitions and more detailed sub-criteria can then
be used to link this framework with the theory of change of a specific programme.

Standards
Define the levels of performance for each programme-specific criterion. We used
a scoring system in our toolkit for qualitative assessment ranging from ‘poor’ to
‘excellent’ to support comparison against standards (see this as an example for VfM
assessment with standards).

Spending less Spending well Spending wisely Spending fairly

Minimising the The relationship The relationship The extent to
cost of resources between the between the which services are
used or required output from goods intended and available to and
(inputs). or services and actual results of reach all people
the resources to public spending that they are
produce them. (outcomes). intended to.

Economy Efficiency Effectiveness Equity

Figure 4: 4Es Framework.

12 A guide to support value for Money (VfM)

analysis for public managers
 II | Introduction to prospective
VfM assessments

II.A.v: Evidence assessment [Link]: Analysis, synthesis

and judgement
Identify the evidence needed to
support the VfM assessment. Evidence Examine each type of evidence
falls into two broad categories: separately, then synthesise the
qualitative (descriptive/narrative terms) components to reach a collective
and quantitative (indicator-based judgement about the level of performance
measurement). The preceding steps against the criteria and standards
(ie the programme’s theory of change, defined. It can be helpful to use a
criteria and standards) should ensure standardised process such as the 4Es
that the evidence is relatively robust and framework, which we will elaborate on
supports the evaluative judgements throughout the rest of this guide.
appropriately. The data checklist sheet in
our VfM Toolkit provides a starting point
for the evidence you are likely to require.
[Link]: Reporting

[Link]: Gather evidence

Prepare a VfM assessment report as a
matter of record. Our VfM toolkit provides
a handy printable summary sheet that
Collate the evidence (eg data on cost can serve this purpose.
drivers, payment mechanism, cash
flows outcome measurement, prices,
etc) required to address each criterion,
ensuring balance between the 4Es
wherever possible. Moreover, assess
the quality of this evidence and its utility
in capturing value for money of the
programme. See our VfM toolkit for more
details on the prescribed data to make
such calculations.

13 A guide to support value for Money (VfM)

analysis for public managers
 III

VfM assessment: a
step-by-step guide to
the 4Es framework
This guide accompanies an Excel-based VfM toolkit and walkthrough video,
accessed through here, that can help you go through all these steps. This section
of the guide follows the same structure as the toolkit, so you can refer to the
same section number on both the guide and the toolkit and refer here for more
detailed guidance on specific portions of the toolkit.

Worksheet Description

Intro This worksheet gives an introduction and overview of the VfM

process.

Summary Start here to fill out the basic details. This printable worksheet
gives a summary of your VfM metrics, along with an optional
benchmark summary.

Data checklist An overview of the various data points in the VfM analysis,
which you should attempt to gather in order to complete the
assessment.

Quantitative A tool to assist with a high-level quantitative assessment of your

Calculator inputs, outputs, outcomes and equity metrics.

Qualitative A self-assessment instrument to assure the quality of

Assessment programme. This will output an aggregated set of qualitative
metrics.

SIB specific Further analytical tools to assist with evaluating social impact
estimates bonds.

Table 1: Toolkit components.

14 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

III.A. Economy: Are inputs of appropriate quality bought

at a minimised price?
In this stage, you should expect to analyse and justify the key cost-drivers of the
programme. Assess the payment mechanism (structure, unit, timing, frequency and
length). Since the evaluation is done ex-ante, you will need to estimate the costs. GO
Lab’s pricing outcomes guide discusses how to estimate costs in OBCs and SIBs.

Inputs (eg time, staff, consultants, raw materials, capital, etc) should be procured at the
least cost for the relevant level of quality (these are separate from the service delivery
costs that are shouldered by the provider/investor).

Please refer to the step-by-step figures below for a description of both quantitative
and qualitative efficiency assessments.

III.A.i: Quantitative assessment

1 2 3 4 5
Gather the Measure the Find unit costs
Adjust figures Compare unit
budgeted values average unit cost figures for similar
to account for costs against
of the key cost for each of your programmes
temporal factors time-adjusted
drivers (eg staff, key cost drivers to use as
if necessary comparator
procurement, comparators
figures
M&A, etc.)

Figure 5: Quantitative assessment steps for economy.

[Link]: Qualitative assessment

1 6
Is there a robust counterfactual to
Is resource allocation linked to previous
monitor performance established
performance data in a similar context?
before implementation?

2 7
Are the costs of data collection/
analysis/validation appropriate (and Are service providers only paid for
in proportion) to the intervention and results achieved?
the ultimate benefits of the data?

3 7.1
Are the reasons for using an outcome-
Is budget planned to be monitored
based payment scheme over
regularly?
alternatives justified?

4 7.2
Are the reasons for using social impact
Is the sourcing strategy justified?
bond (SIB) justified?

5
Are procurement guidelines for
selecting providers followed?

Figure 6: Qualitative assessment steps for economy.

15 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

Additional considerations for economy in III.B: Efficiency: How well

outcomes-based contracting are inputs converted into
• Keep in mind that the majority of outputs?
costs in OBCs are related to ‘outcome
Efficiency might be your next key
payments’, but there are still other
concern — if your programme is explicitly
drivers especially around monitoring,
concerned with outcomes over outputs,
management and evaluation.
please see ‘Additional considerations’
• The schedule between intervention, below within this section. After assessing
impact measurement and outcomes the various inputs into your programme,
payment is critical to the feasibility and the associated costs, look to your ToC
of an outcomes-based contract and (see above sections for detail) to identify
thereby improving the VfM case. your key outputs. These will likely be
According to Social Finance, investors specific goods or services procured within
typically prefer an OBC or SIB that the confines of the programme.
matures within a time horizon of
around five years. For example, a mental health counselling
programme might include a month-long
• Regarding SIBs, the earlier
course of intensive counselling as one
investment and returns can be
output, or alternatively might choose to
generated, the lower the ‘cost of
specify that an hour of therapy provided
capital’ required by investors. Early
is the appropriate unit. For a programme
payments may incentivise investors
providing housing benefits to rough
to ‘recycle’ those payments to
sleepers, an output might be a night
fund intervention costs during the
provided in a shelter, or perhaps a month
remaining term of the programme.
of subsidised rent. It is up to the user to
This reduces the initial capital
define the appropriate units of output.
requirement and consequently
reduces the ‘cost of capital’. A SIB’s
You should evaluate the expected
cost of capital may vary due to
quantity of outputs for your programme,
several other factors as well as the
given the projected inputs — for example,
timing of payments. The SIB specific
how many referred potential beneficiaries
estimates sheet in our toolkit provides
will actually complete a month-long
additional analytical tools to help
course of therapy? How many rough
guide your thinking around cash
sleepers will you provide with housing,
flow assessments and the timing of
and for how many nights or months? Try
outcome payments.
to specify the success rate: how many
outputs will you expect per referred
participant? 100%? 70%? The answer to
this question can seriously impact your
expected VfM.

16 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

III.B.i: Quantitative assessment

1 2 3 4 5
Calculate the Calculate the per Estimate Calculate the Compare costs
per participant participant cost chances of cost adjusted for per output
benefits success based the success rate against benefits
on historical per output
data, experience,
and analysis

Figure 7: Quantitative assessment steps for efficiency.

[Link]: Qualitative Keep in mind:

assessment
• You can supplement this assessment
with indicators such as standards of
After specifying the quantitative
target outputs achieved, percentage
indicators of efficiency, you should
of eligible persons achieving
also perform a qualitative assessment
targets, etc. These indicators should
to ensure that your programme and
compare well with those of similar
processes can sustainably hit your
programmes. If you do not anticipate
efficiency targets.
hitting the targets, ensure that you
provide justifications.

1
Are assumptions (from the activities
to the output) consistent with the • Another issue affecting efficiency
programme’s theory of change? of SIBs compared to either OBCs or

2
Do we have clear and realistic conventional fee for service might
milestones, timelines and targets be lack of competition in the supply
built on a baseline? market due to the limited number

3
Have different alternatives for of investors and/or providers. It is
delivering the programme and therefore highly recommended to
respective benefits and costs been
encourage competition between
considered?
potential providers as this will tend to

4
Are there systems in place for promote efficiency.
systematically monitoring, evaluation
and programme management?

5
Are there monitoring tools and
planning in place to mitigate risks
and make timely adjustments?

6
What economies of scale have you
been able to identify in the delivery
of your programme?

Figure 8: Qualitative assessment steps for efficiency.

17 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

[Link]: Additional III.C: Effectiveness: How

considerations for efficiency well do those outputs
in outcomes-based convert into outcomes?
contracting
In the third E, turn again to your ToC
• An assessment of efficiency for and identify the main outcomes of
OBCs depends on the context. your programme or programme. These
Typically, making the distinction will likely be the ‘ultimate goal’ of the
between output and outcome should programme. As with outputs, the user
be possible, eg in the majority of is free to define the outcomes, and to
early intervention and preventative identify the relevant targets and units.
programs where the target is of an
For example, a mental health programme
‘output’ nature linked to some future
may strive to reduce self-assessed
‘outcome’. However, there may be
depression for a target period. Such
instances where distinguishing
a programme might set the outcome
between output and outcome in an
target as one participant reporting
OBC is more challenging owing to the
10% fewer days of ‘feeling severely
type of indicators used for outcome
depressed’ in a month in a regular survey.
payment. When this happens, the
An employment related programme
distinction between ‘effectiveness’
might strive for participants to find
and ‘efficiency’ is less evident.
gainful employment, so perhaps an
• Comparing expected costs and outcome might be one participant
outputs could provide some successfully landing a job, or alternatively,
understanding of how well the demonstrating attendance at work for
services are priced and being one month.
delivered, both in their own right
and relative to the conventional
methods of commissioning. The III.C.i: Quantitative
assessment
ratio of output to costs could be
a measure for comparison. In this
context, an assessment of efficiency
You should define measurable outcomes
for OBCs may be more relevant if you
with clear benchmarks and should
do not have sufficient information
attempt to project the number of
on outcomes. It might also be that
outcomes seen per output. For example, if
you choose to do an assessment of
the output of an employment programme
efficiency to get an understanding
is a two-week training session followed
of what the programme will do
by guided placement, then perhaps
differently to achieve greater
the programme expects that 80% of
innovative and efficient service
participants will land a job within the
delivery.
next month.

18 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

In the GO Lab-CIPFA toolkit, the quantitative assessment for effectiveness allows

you to enter your predictions for the average ‘conversion rate’ between outputs and
outcomes. You can use this as a handy learning tool to quickly assess whether your
project will produce the intended outcomes, and what parameters you may want to
tweak. Feel free to adjust the predictions and the pairing of outputs to outcomes as you
please, to reflect the structure of your programme.

The summary tab of the toolkit will give some useful metrics given your projected
inputs. For example, you can refer to the benefit-cost ratio (BCR) and cost-
effectiveness ratio (CER), both of which may be useful for your project or programme.
Please refer above to learn more about how these measures are used.

1 2 3 4
Calculate the Estimate the rate Estimate either Express in
percentage of of outputs that benefit to monetary
the budget tied will be translated cost ratio or terms the
to outcomes into actual effectiveness to wider financial
outcomes cost ratio benefits of the
programme

Figure 9: Quantitative assessment steps for effectiveness.

[Link]: Qualitative
assessment 1
Are the programme’s objectives
realistic and clearly mentioned?

2
Another important assessment of Is there a clear theory of change
based on evidence?
effectiveness should be qualitative.

3
You should think carefully about your
Are the outcome and impact
proposed programme, and identify any indicators relevant and robust?
risks or blind spots that may arise when

4
converting outputs into outcomes. Our Does the programme have leverage,
replication potential and offer wider
toolkit can give a helpful walkthrough of benefits?
some potential questions, but feel free to

5
add your own as necessary. Can you demonstrate the
additionality of the progamme’s
activities?

6
Will the programme be sustainable
over the long term?

7
Will the programme generate
important learning?

Figure 10: Qualitative assessment steps

for effectiveness.

19 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

[Link]: Additional • Note that unlike outputs, the outcome

considerations for payer or the providers do not exercise

effectiveness in outcomes-
direct control over outcomes. As such,

based contracting
there are some uncertainties that
you, as a public manager, may not
be able to control. So prepare for the
• If the programme benefits can unexpected, draw different scenarios
be translated into monetary and come up with plans to minimise
value, cost-benefit calculation the risks.
(total cost of outcome divided by
total value of outcomes) can be
conducted; otherwise, a simple cost- III.D: Equity: How well
do the activities reach
effectiveness calculation (total cost of

all people that they are

outcomes divided by total number of
participants) is sufficient. See our VfM
toolkit for more details on how to do intended to?
the calculation.
VfM assessments have recently started to
• Your programme may generate wider
include considerations of equity as a main
benefits and there may be benefits
criterion. You may decide if you want to
and costs that are realised later
make equity a standalone consideration
in time. See our pricing outcomes
or deal with it when assessing for
guide for further discussion on these
economy, efficiency and effectiveness. In
benefits and costs.
this guide, we adapt the People in Need's
• Later payments are more likely to VfM Equity Quantitative Analysis tool
align with long-term policy goals as we attempt to provide a quantitative
due to the information of outcome’s assessment of equity for OBCs. Possible
achievement and sustainability. Later categories of equity you may wish to
payments also enable the outcome consider include vulnerable targeting,
payer to use money for alternative minority outreach, disabled and other
uses in the interim and provide outreach.
additional benefits if inflation is high
and is not included in the pricing.
Chapter A6 of HM Treasury's Green
Book outlines how these timings
benefits can be quantified using
discounting.

20 A guide to support value for Money (VfM)

analysis for public managers
 III | VfM assessment: a step-by-step
guide to the 4Es framework

III.D.i: Quantitative
assessment

1 2 3 4
Identify equity Link the estimated Identify benchmarks Compare the
components for the outcomes to equity using available programme indicator
programme (eg age, components and evidence (eg national and the baselines (ie
gender, economic calculate programme or regional data) difference between
status) level equity indicator the programme’s
(eg out of total and the baseline
paricipants, X% are equity indicator)
girls)

Figure 11: Quantitative assessment steps for equity

[Link]: Qualitative
assessment

1
Does commisioning by OBCs
impact equity relative to alternative
methods?

2
Were all relevant stakeholders
involved in the programme design?

3
Was there a robust targeting
criterion in the programme design?

4
Will there be functional complaint
and feedback mechanisms?

5
Are there equity considerations
at economy, efficiency and
effectiveness levels?

Figure 12: Qualitative assessment steps for equity.

21 A guide to support value for Money (VfM)

analysis for public managers
 IV

The SIB tab

A SIB is a financial instrument that raises One is a ‘cash flow assessment’ table
capital and links financial returns to the and figure, which takes money inflow and
success of pre-defined, measurable outflow for the investor and calculates
social outcomes only after those results ROI and IRR. The second is a ‘timeline
have been achieved and evaluated. In assessment’ table and figure that
the meantime, private investors provide collates information of payments by the
operating funds and are repaid with a commissioner across time. Rule of thumb
modest return if the initiative achieves is more risk-share for the public sector the
its goals. Given the differences between earlier the payments, and vice versa.
SIBs and other types of service provision
methods, we aim to add some SIB specific
analysis in future iterations of the toolkit.
For the time being, there are two SIB
specific items on this experimental tab.

22 A guide to support value for Money (VfM)

analysis for public managers
 V

Appendix
Further description of qualitative Is there a robust counterfactual to
steps monitor performance established before

[Link]: Economy
implementation? Performance should be
evaluated against a robust counterfactual
in the form of a control group. This would
Is resource allocation linked to previous
strengthen an organisation’s evidence
performance data in a similar context?
base for outcome-based payment
Assess whether the programme’s
effectiveness as a delivery mechanism.
budget is justified using data from similar
programmes.
Are providers only paid for results
achieved? Assess whether data
Are the costs of data collection/
management systems and processes
analysis/validation appropriate (and
are able to capture accurate, reliable and
in proportion) to the intervention and
timely information on outcomes achieved
the ultimate benefits of the data? Data
and the number of participants attached
collection should not be too costly. For
to the programme to ensure providers are
programmes where these costs are
only paid for results achieved.
unavoidably higher, they must be
clearly justified.
Are the reasons for using an outcome-
based payment scheme over
Is budget planned to be monitored
alternatives justified? The success of
regularly? Check if the budget will be
an outcome-based payment scheme
monitored on at least a monthly basis.
depends on the environment in which the
Proper budget monitoring helps to
programme is operating. For instance,
minimise costs.
an outcome-based payment scheme
Is the sourcing strategy justified? A is most likely to succeed if outcomes
good sourcing strategy ensures the can be measured and attributed to the
proper identification, assessment and interventions. The economy condition of
engagement of suppliers for the planned VfM may be undermined if an outcome-
activities. Consider how competitive the based payment scheme is applied
supplier market is and whether better inappropriately given that there will
prices can be leveraged. We are planning be additional costs to monitor the risks
to develop a framework to support public associated with the quality of
managers with sourcing strategies. the programme.

Are procurement guidelines for

selecting providers followed? A robust
and competitive procurement process
ensures significant cost reductions and
management, and that risks to outputs/
outcomes are identified, assessed and
minimised.

23 A guide to support value for Money (VfM)

analysis for public managers
 V | Appendix

Are the reasons for using social impact Are there systems in place for
bond (SIB) justified? Assess whether systematically monitoring, evaluating
adding an investor to the model improves and managing the programme?
performance and how long it will take Consider whether there are the means to
to generate outcome payments. The systematically monitor progress against
feasibility of a SIB depends on two set outputs and targets as well as the
conditions: delivering cashable resource progress and quality of activities to be
savings and financing innovation. These implemented.
two conditions may be satisfied if better
outcomes can be achieved at lower costs Are there monitoring tools and planning
through a new set of services funded by in place to mitigate risks and make
a SIB compared with alternative forms of timely adjustments? Examine whether
commissioning (Social Finance, 2013). the risk analysis and mitigation strategy
cover the key threats and overall risk
level of the programme as well as any
[Link]: Efficiency perverse incentives (eg to carry on with
the cheaper/quicker approach rather
Are assumptions (from the activities than the most suitable). For instance, if
to the output) consistent with the outcomes seem harder to achieve, either
programme’s theory of change? Examine because the cohort is difficult to help or
whether sufficient activities (quality because the desired level of improvement
and quantity) are planned to produce is high, then the probability of achieving
the output and whether targets for the outcomes will be lower, and the contract
output are appropriate. Assess whether will be deemed riskier – and risk demands
indicators are described accurately and compensation.
are inexpensive, reproducible and usable
as a means for monitoring. What economies of scale have you
been able to identify in the delivery
Do we have clear and realistic of your programme? Check if the cost
milestones, timelines and targets built per participant reduces if you were to
on a baseline? Assess the likelihood increase the scale of the programme. If
of the programme delivery in a timely so, provide justifications for why a larger
manner, in line with output indicators and programme is not targeted.
their expected targets while responding
to contextual changes.

Have different alternatives for delivering

the programme and respective benefits
and costs been considered? Examine
whether the programme’s objective
and expected outputs justify the cost
compared to similar programmes.
Alternatives could provide the means to
achieve the same for less cost. Moreover,
performance might be greater for the
same cost.

24 A guide to support value for Money (VfM)

analysis for public managers
 V | Appendix

[Link]: Effectiveness Does the programme have leverage,

replication potential and offer wider
Are the programme’s objectives realistic benefits? Assess the leverage of other
and clearly mentioned? Checking activities and the wider financial and
this allows you to assess whether the non-financial benefits of the programme.
programme’s objectives can be achieved Is there any evidence that shows
as the effect of the programme. Examine significant potential for expansion or
whether the main assumptions from replication?
the output to the programme’s objective
are accurately perceived. Consider if the Can you demonstrate the additionality
output planned is sufficient to achieve of the programme’s activities? Is there
the programme objectives as well as the a plan in place to measure ‘additionality’
factors that impede the achievement of and check ‘attribution’? Changes could
the programme objective. happen over time due to external
factors. Understand what would happen
Is there a clear theory of change based without the programme and whether
on evidence? The theory of change there are externalities that will accrue to
should show that outputs are necessary participants as a result of the planned
and sufficient to deliver the desired activities.
outcomes. Identify the elements of the
theory of change that are the weakest Will the programme be sustainable
and assess whether the programme’s over the long term? Consider factors (eg
activities can overcome these behavioural change approach, feasibility
weaknesses. Follow Step 10 of PIN’s VFM study, exit strategy) that ensure a better
Guide approach to identify the strength of chance of outputs turning into outcomes.
the evidence to show how effective each
output is at reaching the outcome. Will the programme generate important
learning? Examine whether the
Are the outcomes and impact indicators programme contributes to a knowledge
relevant and robust? Relevant indicators gap and how important lessons will be
are clear, rule-driven, causally linked and shared to stakeholders.
accurately express the programme’s
objective.

25 A guide to support value for Money (VfM)

analysis for public managers
 V | Appendix

[Link]: Equity Will there be functional complaint and

feedback mechanisms? Complaints
Does commissioning by OBCs impact and feedback mechanisms generate
equity relative to alternative methods? evidence on critical needs and provide
Consider any perverse incentives that opportunities for the implementation to
occur in an OBC design that might be adapt accordingly.
avoided in a conventional contract and
vice versa. Are there equity considerations at an
economy, efficiency and effectiveness
Were all relevant stakeholders involved levels? This may include sustainable
in the programme design? Participatory procurement (the cheapest sustainable
planning and decision-making ensure option rather than the cheapest ones) or
that all stakeholders (including the most local economy enhancement (employing
marginalised groups, target groups, community members rather than more
providers) identify core equity challenges expensive non-local options).
and how to address them.

Was there a robust targeting

criterion in the programme design? A
robust targeting criterion justifies the
selection of participants and enhances
a programme’s reach to the most
vulnerable groups (eg those at socio-
economic disadvantage, people with
disabilities, gender sensitivities).

26 A guide to support value for Money (VfM)

analysis for public managers
 I

77 Mansell Street, London E1 8AN

+44 (0)20 7543 5600

The Chartered Institute of Public Finance and Accountancy.

Registered with the Charity Commissioners of England and Wales No 231060
Registered with the Office of the Scottish Charity Regulator No SC037963.

[Link]
27 A guide to support value for Money (VfM)
analysis for public managers