Results vs. Claims: Strengthening the UK’s Aid Watchdog

The UK’s aid watchdog, Independent Commission for Aid Impact (ICAI), has just entered its third four-year phase. ICAI is a major asset in ensuring aid is well-spent. Having reviewed ICAI’s prior work, we think ICAI could should focus more on “results claims”—that is, whether the estimates of expected benefits that underpin decision-making by Ministers are well-evidenced. Below we lay out why and how.

The value of an independent evaluator

ICAI have substantially improved the scrutiny of aid spending in the UK, triggering improvements in both new and existing programmes which our previous report showed covered £28bn of aid across 65 reviews. HM Treasury confirmed to Parliament last week (Q37) that ICAI reports will inform spending review allocations. ICAI is also an asset for aid transparency. It has access to information that’s invisible or too difficult to gather for the public, which it publishes in a clear and comprehensible format.

Other countries are adopting the UK’s innovation. Sweden is considering their approach, while France, who are  steadily increasing their level of aid to 0.55 percent in 2022, is explicitly aiming to follow ICAI’s model.

In the UK, ICAI faces new challenges—including indications that UK aid quality is falling, as well as a major increase in the amount spent outside DFID, from 13 to 28 percent since 2010.

ICAI’s new head, Tamsyn Barton very sensibly started her tenure with a consultation on its third phase. She has already responded positively to important issues: assessing aid across Government, incorporating beneficiaries feedback on projects, retaining ICAI’s scoring, committing to publishing ICAI’s literature reviews, and increasing focus on results.

How ICAI can drive even more impact

A critical juncture for aid effectiveness occurs when Ministers decide which aid projects to undertake or extend, based on proposals (business cases). There are innumerable possible uses of aid, with vastly different impacts. Choosing well between them relies on having good estimates of the likely impacts, or “expected benefits” of each project.

Without someone independently checking these claims, there is an incentive to exaggerate. If evidence on the effectiveness of your proposal is scarce, then, rather than being honest about uncertainty, there is an incentive to present intermediate outputs, such as how many people a programme will “reach,” as benefits. When it comes to actually setting targets for implementing organisations, if the estimate was a bit high, you can always water it down once approved. Indeed, ICAI has found a disconnect between estimates in business cases and project documents. Taken together, this means that aid programmes don’t always compete to be selected on which will have the best impact, they could compete on which can spin the best, and there is a potential risk of a race to overstate benefits and certainty.

This matters because the most effective interventions in health can be thousands of times more effective than the least. In education, even among interventions generally considered to be good value, the difference in effectiveness can be 50 or 100 times. The difference might be even greater for harder to measure programmes. Although estimates of their expected benefits will have a greater range and use more assumptions, they require just as much scrutiny.

What ICAI should do next: check results claims

We think that ICAI can help fix this problem, by using its unique independence to check these claims. By doing so, for every programme sample it reviews, ICAI would create strong incentives for each project that would lead to a much stronger and more honest decision-making process.

Where ICAI have focussed on these results, they have uncovered major issues. In particular, last year a review on maternal mortality made a splash when it challenged claims about how many maternal deaths DFID had prevented.

To date, ICAI’s results and impacts work have focussed more on claims made after programmes have been implemented, so we don’t know whether the selection or design of the programme was also based on weak evidence.

So, each ICAI review should check that:

1. Claims about expected benefits in business cases are meaningful

2. These claims are based on reasonable evidence and assumptions

3. These claims are consistent with targets and indicators in project documents

4. Evaluations are rigorous and unbiased, and results feed back into programme design

Most ICAI reviews take a sample of aid projects, and we propose that the ex-ante results claims for this sample are also examined. This would work for upcoming reviews on UK aid to Ghana and the African Development Bank, though perhaps not for some others, like the review on sexual exploitation. In its recent review of the Newton Fund, the above four steps were taken very quickly—because the Fund was, incredibly, approved without any idea of the expected benefits.

We think a focus on results claims in every review would have major benefits.

First, it would make the selection of projects better informed. We think that this is a particularly important decision point in the policy cycle, that has not received enough scrutiny so far.

Figure 1. Overview of policy cycle, ICAI focus and potential to influence decisions

Image

Second, it would strengthen the incentive for teams to generate rigorous evidence and use it in programme design.

Third, it would give policymakers and the public a clear indication of whether they can trust the claims being made about the impact of aid programmes. Since voters don’t experience aid spending directly, this extra oversight is important.

Why ICAI?

Checking results claims credibly requires independence. Departments do have procedures to quality assure programmes, but they are not consistent across departments, and cannot be independent. Within government there are incentives that might make results claims less reliable. ICAI is the only independent aid-focussed body that could do this.

Another of ICAI’s assets is its close links to Parliament’s Development Select Committee. Indeed, we’d like this to go further with ICAI reporting to Parliament—like the National Audit Office—rather than being an advisory public body with its mandate and head set by the Development Secretary.

We think that a focus on the results claims that inform decisions about project selection can significantly improve aid impact. We hope that ICAI—and the Development Select Committee—will ensure a results-claim focus in all reviews going forward as the norm.

This blog is a summary of recommendations that we submitted to ICAI’s public consultation, and discussed with ICAI officials and other stakeholders.