A response to the Independent Commission for Aid Impact
The Independent Commission for Aid Impact (ICAI) released its performance review of DFID’s approach to value for money (VFM) on 20 February 2018. This note responds to the recommendations in ICAI’s report, by outlining OPM’s Approach to Assessing Value for Money (King & OPM, 2018) and how this approach might be used to strengthen DFID’s approach to VFM.
OPM’s approach to assessing value for money
In 2016, we teamed up with Julian King, an evaluation specialist, who worked with staff from across the company to develop the basis of a robust and distinct OPM approach to assessing VFM. The approach, published in January 2018, involves making transparent, evidence-based judgements about how well resources are being used, and whether the value derived is sufficient to justify the investment. It combines good evaluation practice (explicit evaluative reasoning) with economic analysis, to provide a framework for:
- organising evidence of performance and VFM;
- interpreting the evidence on an agreed basis; and
- presenting a clear and robust performance story.
The methodology was successfully piloted during annual reviews of DFID’s Sub-National Governance (SNG) programme in Pakistan and MUVA, a women’s economic empowerment programme in Mozambique. Since then we have applied the approach on upwards of a dozen different development project and programmes, spanning a range of clients, countries, sectors, and budgets. It has been well received by funding agencies, partner governments, and project teams, who in particular appreciate the use of explicit evaluative reasoning.
ICAI’s performance review of DFID’s approach to VFM
During 2017, ICAI conducted a performance review of DFID’s VFM approach in programme and portfolio management. ICAI’s report acknowledges that DFID has established itself as a global champion on VFM, advocating for the VFM agenda with multilateral agencies and partners, and embedding VFM into its performance management processes. The review found that the VFM approach has, over time, placed greater emphasis on the quality as well as the quantity of results. The inclusion of equity in VFM assessments was also acknowledged as a strength, as “reaching marginalised groups may entail additional effort and cost”.
The review found that the focus on VFM had led to programmes finding ways to improve VFM, most commonly through cost savings but also in some cases by identifying under-performing programmes and taking remedial action and, less frequently, by looking at “new and better ways of doing things”.
Areas to improve, and OPM’s response
The review identified gaps and weaknesses, and made recommendations to address these. ICAI’s recommendations, and our response, are as follows.
ICAI recommends: DFID country offices should articulate cross-cutting value for money objectives at the country portfolio level, and should report periodically on progress at that level.
This was the area where ICAI found the most significant gaps, with no system for reporting and capturing results at the country portfolio level. We agree that VFM assessment should focus not only on individual programmes, but on how they work together to deliver lasting impact. The approach described in OPM’s VFM Guide can be used at multiple levels from projects or programmes to whole portfolios. The process of evaluative reasoning remains the same while the criteria, standards, and types of evidence are tailored to context. A VFM assessment at country portfolio level, that provides clear answers to important questions about performance and VFM, has the capacity to support sound decisions that can lead to improvements in the impact and value achieved by UK aid over time.
ICAI recommends: Drawing on its experience with introducing adaptive programming, DFID should encourage programmes to experiment with different ways of delivering results more cost-effectively, particularly for more complex programming.
ICAI noted that despite DFID’s support for adaptive programming, its approach to VFM had yet to reflect the importance of experimenting and adapting in achieving VFM. Our approach to VFM was designed to support better VFM assessment in complex adaptive programmes, where adaptive and incremental approaches are needed to gain traction — and where, for example, there is no linear relationship between outputs and outcomes; where ‘off the shelf’ VFM indicators, such as cost per unit of output or outcomes do not yield meaningful data; and where responsiveness to an evolving context is a more important determinant of VFM than delivery of anticipated outputs.
With MUVA, for example, the VFM framework identified and assessed three dimensions of effectiveness:
- effectiveness as an urban female economic empowerment programme;
- effectiveness as a learning programme; and
- effectiveness as an influencing programme.
Detailed sub-criteria, indicators, and narrative were specified for each of these. The framework explicitly recognised that some of the projects and approaches being tested had not been tried before, while others build on an existing evidence base — and that part of the value of MUVA is derived from learning, including providing evidence about what works, what does not work, and why.
ICAI recommends: DFID should ensure that principles of development effectiveness — such as ensuring partner country leadership, building national capacity, and empowering beneficiaries — are more explicit in its value for money approach. Programmes should reflect these principles in their value for money frameworks, and (where appropriate) incorporate qualitative indicators of progress at that level.
ICAI found that “DFID’s results system is not currently oriented towards measuring or reporting on long-term transformative change”. For example, the results system does not measure “the contribution of UK aid to catalysing wider development processes, such as enhancing the ability of its partner countries to finance and lead their own development”, or other complex objectives.
Our approach involves developing criteria and standards that are tailored to the programme context and agreed in advance of the VFM assessment — providing a clear statement of what aspects of programme management, delivery, and results are relevant to good performance and VFM. Principles such as partner country leadership, building national capacity, and empowering beneficiaries can be built into the VFM framework, providing an agreed set of filters for interpreting the evidence.
For example, in the SNG Programme, the VFM assessment examined not only the delivery of technical assistance and ‘products’ (such as draft legislation, rules and guidelines, and training) but also the extent to which these deliverables were picked up by provincial and district governments, and the extent to which they in turn influenced the flow of funds to districts to meet identified needs. The VFM assessment used contribution tracing, a cutting edge approach to determining the contribution of the SNG programme to long-term outcomes, in the absence of a measured counterfactual. The approach tests evidence and analyses causal claims along a detailed theory of change.
ICAI recommends: DFID should be more explicit about the assumptions underlying the economic case in its business cases, and ensure that these are taken into account in programme monitoring. Delivery plans should specify points in the programme cycle when the economic case should be fully assessed. Senior responsible owners should also determine whether a reassessment is needed following material changes in the programme, results targets, or context.
ICAI noted, “value for money is assessed in the business case for each new programme, using a mixture of quantified cost-benefit analysis and narrative justification”. However, the assumptions behind the original economic case are not routinely checked during the life of the programme.
Our approach to VFM is based on a common understanding of the theory of change which underpins a project or programme, and its periodic reconfirmation. This enables a mix of quantitative and qualitative evidence to be tracked against original expectations, to determine whether adjustments are needed to the programme and its measurement. Evaluation criteria and standards are set in consultation with senior responsibe owners and programme teams, with reference to the business case and other foundational documents.
Furthermore, our approach incorporates cost-benefit analysis and/or other forms of economic analysis where relevant to track progress toward the anticipated return on investment, based on actual versus expected costs and outcomes. Our approach places equal emphasis on VFM assessment at specific points in time together and forward-looking judgements about ‘direction of travel’ to support assessments of whether changes in context or results merit changes in design, delivery modalities, and targets.
ICAI recommends: Annual review scores should include an assessment of whether programmes are likely to achieve their intended outcomes in a cost-effective way. DFID should consider introducing further quality assurance into the setting and adjustment of logframe targets.
ICAI found that DFID’s VFM approach tends to emphasise “controlling costs and holding implementers to account for efficient delivery” and that annual review scores relate principally to progress at output level, and not to achievement of outcomes or VFM as a whole.
We agree that economy and efficiency, though often easier to measure than effectiveness and equity, are not sufficient to provide a determination of VFM. We recognise that care is needed to avoid a bias toward cost-cutting rather than value maximisation. For example, focusing on minimising costs can miss an opportunity to enhance VFM by spending slightly more and gaining disproportionately greater outcomes.
Linear project-management tools like logframes can run into problems when selecting indicators, if what is measurable is mistaken for what is important. The evaluation-specific approach describes in OPM’s VFM Guide reverses this — by first identifying what is important, and then determining how it should be measured and evaluated. Where VFM frameworks are developed as part of the inception phase, the evaluation-specific approach described in our guide can support the determination of meaningful logframe targets.
Our approach offers a disciplined and pragmatic approach to analysing VFM, and enables VFM assessment to evaluate the quality and quantity of outcomes, even when those outcomes are hard to measure. The use of explicit VFM criteria and standards provides a clear basis for making sound judgements about performance and VFM using all of the available evidence, bearing in mind the strengths and limitations of each evidence source.
OPM’s VFM Guide can be downloaded from our website, and please contact David Hoole (email@example.com) or Stephanie Allan (firstname.lastname@example.org) in our Public Financial Management team for more information.