Getting the Best We Can Buy: Three Solutions to Improve the Use of Value for Money Evidence in Global Development

The need for effective evidence-informed priority-setting in global development is more urgent than ever, with widespread global challenges and reduced funding due to both COVID-19 related public spending and economic slowdowns. Policymakers and donor partners must make difficult and consequential choices on how best to spend multi-billion-dollar aid funds, because it is impossible to do everything. Yet failing to achieve the most with the money is a missed opportunity, with real-world consequences in benefits foregone. Which is the better buy; mosquito nets or cancer drugs? Structured lesson plans for teachers or simply providing new textbooks? Or perhaps just giving money directly? This blog explores three key barriers to using value for money evidence in global development and offers three solutions to overcome these challenges.

What is value for money evidence for global development, and how is it currently used?

So far, priority setting in international development has focused more on measuring impact and identifying “what works.” Too often the benefits are considered without an essential piece of information for any purchase, at the individual or national level: cost. Evidence on value for money can help assess which interventions offer the best impact in proportion to the resources required; the greatest so-called “bang for your buck”. How should we spend the money available to us? On the things that give us the most of whatever it is we value, be that healthy people, educated children, quality jobs, protected environments, or – realistically – a mix of these benefits. Indeed, there is arguably a moral imperative to do so, with lives and livelihoods lost as a result of cost-ineffective investments.

There are a range of economic evaluation frameworks that can be used to rigorously make value for money calculations - cost-effectiveness analysis and cost-benefit analysis perhaps chief among them. Moreover, there are a range of methods that can be used to estimate costs and effects, from empirical experiments such as randomised trials to pure modelling approaches that input relevant data to simulate the scenarios of interest.  This kind of information must be considered alongside other factors – such as equity, human rights, public acceptability, and feasibility – but it is uniquely well placed to identify investments that offer the greatest overall development gains.

Efforts to synthesise and use value for money evidence in global development are on the rise. In education, recent work has led to a newly proposed consolidated measure of impact, important for making cost-effectiveness comparisons, and a recent synthesis of the available value for money evidence. In global health, early initiatives to rank and compare global health interventions by cost-effectiveness led to the Disease Control Priorities Project (now on its third iteration) as well as the International Decision Support Initiative’s work to strengthen evidence-based priority-setting within countries. In recent years, the UK Foreign Commonwealth and Development Office developed “Best Buys” notes which attempt to summarise the value for money evidence within sectors from health to economic development, and worked with the Global Education Evidence Advisory Panel and the World Bank on publishing the Smart Buys in improving learning. Taking a different approach but with the same broad aims, USAID has piloted a series of cash benchmarking studies that seek to establish a minimum standard of cost-effectiveness for its programs. Lastly, initiatives such as Givewell and the Effective Altruism movement have emerged to support philanthropic giving based on value for money approaches.

However, despite isolated successes, the use of good quality value for money evidence to inform global development prioritisation is still the exception, not the rule. So, what are the key barriers to using value for money evidence in global development, and how can we overcome them?

Three critical challenges impede the effective use of evidence on value for money in international development

1. Evidence is scarce, difficult to find and difficult to interpret. Many areas of development simply have very little analysis of the comparative value for money of alternative actions. A review of recent education studies in Africa found that fewer than one in three included cost-effectiveness analysis. Where limited evidence does exist, it is time consuming to search databases of academic literature to find it. And even if relevant evidence is identified, it is difficult to apply findings from one setting to another, since key drivers of costs or effects can change substantially. For example, wages in Malawi may be very different from those in South Africa. It can also be difficult to make the comparisons between studies necessary for prioritization if the methods to collect and analyse differ between the studies, as they too often do. The result is a thin, patchy evidence base that is difficult to use to truly inform policy.

2. There is a lack of clear demand and processes for use. Global development donors have multiple overlapping reasons for providing aid. Proposed programs therefore must meet multiple criteria, and the comparative cost-effectiveness of outcomes is seldom chief among these; this is even more likely to be true when the data are hard to find or considered unreliable, or inapplicable to a new context. The UK government was an early champion of using value for money evidence and is perhaps foremost among development donors in seeking its use. However, requiring its inclusion in business cases, while a positive step, has led principally to the need to find supportive evidence for existing priorities rather than genuine evidence-informed priority-setting.

3. Sectoral silos. A perennial problem in both research and policy, work on value for money in global health, for example, tends to be almost hermetically sealed from other development sectors, such as climate or education. This can result in missed opportunities for cross-sectoral learning on evaluation methods but also potentially an undervaluing of investments which cut across sectors, such as school feeding or cash transfers, which have documented health and education benefits.

Three key solutions to get bang for your buck in global development

1. Donors should invest in a global development value for money evidence platform. Technical advances in natural language processing are increasingly used to produce automated or semi-automated systematic reviews and evidence maps for large bodies of evidence. Developing a comprehensive evidence map or “one-stop-shop” for value for money evidence in global development would radically lower barriers to identifying relevant evidence that already exists and serve as a platform for dedicated evidence synthesis or prioritisation analyses. It could also foster a cross-sectoral community of practice, advancing methods to improve both the comparability and transferability of evaluations that are produced.

2. Researchers should always include cost analysis in impact evaluations. Measuring intervention impact is often complex and resource intensive. Building in a plan to rigorously estimate costs alongside impact is typically not a significant additional burden and doing so would hugely increase the available evidence on value for money. Decades ago, most clinical trials did not consider cost, but today integration of economic evaluations in clinical trials is all but compulsory. University of California at Berkeley’s Center for Effective Global Action’s (CEGA) is collaborating with 3ie and others to improve the frequency and quality of costing in development impact evaluations. Research funders will need to make this a required component of future research and provide support to research teams to skill-up on these methods. Some donors are heeding the call. USAID recently instituted a new requirement that all its impact evaluations include cost analysis. A recent call for education evaluations at the World Bank mandates cost data, and others are doing the same.

3. Decision makers should establish processes to use evidence for prioritisation, not only advocacy or post-hoc justification. Ultimately, use of value for money evidence in development decision making starts with decision makers not only wanting to use this evidence, but also articulating this demand as part of strategic plans and embedding processes for evidence use in routine decision making. The UK government should articulate the role of value for money evidence in the forthcoming International Development Strategy and establish a mechanism for considering value for money evidence in team and departmental prioritisation before the point of drafting a business case. USAID operational policy currently notes that staff might consider comparative cost-effectiveness (among other sources of evidence) as part of program design, but there is little sense of how frequently this is done or how it is weighed relative to other requirements. USAID should first take stock of how often, and under what circumstances, comparative value-for-money evidence is brought to bear in program design and then set institutional targets to increase its use. While the examples addressed here are both national donors, the broad principle applies equally to multilateral organisations; boards could do more to push funders to establish processes ensuring investments are value for money.

Overall, while counting the cost is not always the most popular perspective in global development, it is essential to achieve the best possible outcomes and should be considered central to good stewardship of aid resources. Currently value for money evidence is thin, fragmented and poorly used. While encouraging efforts are underway to improve this, more investment is required, including in the creation of a platform to ensure good quality evidence can be found when it’s needed. But efforts to expand and improve evidence are insufficient in the absence of commitment to evidence use. It is incumbent upon donors to advance strategic opportunities to embed value for money in decision making to get the best they can buy.