New Analysis: Global Fund Must Back Up Impact Claims with Transparent, Rigorous Methods

London – The Global Fund to Fight AIDS, Tuberculosis and Malaria asserts that it and its partners have saved 27 million lives—but researchers say that more evidence and data collected using rigorous and transparent methods is needed to back up that claim, in a new analysis published in The Lancet this week.

The analysis, written by Dr. Rocco Friebel of The London School of Economics and Political Science (LSE) and Rachel Silverman, Amanda Glassman, and Dr. Kalipso Chalkidou of the Center for Global Development (CGD), calls on The Global Fund to publish more robust data and be more transparent in its official impact reports, and highlights ways the financing organisation can improve its evidence base.

The authors note that The Global Fund likely has had a positive impact, having distributed over $39 billion of aid funding around the world, but suggest improvements that The Global Fund and others could adopt for their impact measurement and reporting, including:

• More rigorous performance evaluations: Rather than relying on theoretical, modelled results alone, more rigorous performance evaluations would help assess whether The Global Fund’s grant recipients are delivering the services and products being financed.

• Structuring grants to prioritize evidence from the start: Taking this step would help ensure the most effective projects and products are being funded.

• Expanding the scope of impact evaluation: The Global Fund should expand the scope of its evaluations and refocus on areas where evidence is most needed, like particularly large projects and programs using new and untested strategies.

• Publishing the data and models used to estimate impact: Where empirical evidence isn’t available, The Global Fund should at least make the data and models used to estimate impact available to researchers and the public.

The analysis also details current shortcomings of The Global Fund’s evaluations, including:

• Opaque methodology: Without further details in the public domain, the methodology used by The Global Fund in calculating the lives it has saved cannot be verified or reproduced by external researchers. More transparency with regard to its modelling methodology, the authors argue, would build confidence in the organisation’s claims among funders and external researchers. Since the draft of the analysis was shared, The Global Fund has revised its methodology statement on its website, but the authors call for further improvements.

• Distinguishing The Global Fund’s institutional impact from the impact of the broader partnership: The authors’ analysis suggests The Global Fund results report conflates the impact of The Global Fund “partnership”—encompassing other donors, NGOs, and country governments—with the impact of The Global Fund as a standalone institution with a roughly $4bn per year annual budget. The authors argue that more clearly breaking out the activities of The Global Fund from activities carried out by partner organisations would make its impact much clearer. For example, domestic policies in large countries such as India and South Africa have led to significant investments to tackle tuberculosis and HIV. In addition, economic growth is estimated to be responsible for a significant reduction in maternal and child deaths from 1990-2010. The impact of both of these external factors is not distinguished in The Global Fund’s claim of saving 27 million lives.

• Unreliable data: Much of the data comes from countries where statistical systems are weak or non-existent. The authors acknowledge the difficulties of working with data in these scenarios—it would be unrealistic, for example, to produce reliable and up-to-date data in these countries for the number of people who died from each of the diseases covered by The Global Fund—but reiterate their call for data collected in rigorous performance evaluations to supplement the results of modelling exercises.

“There is no doubt that organisations like The Global Fund do great work but to ensure continuous donor investment they need to be more open and honest with their reporting,” said Dr. Rocco Friebel, Assistant Professor of Health Policy at LSE. “The methods and underlying data of the modelling exercise conducted by The Global Fund and others should be released and subject to public scrutiny. The organisation should be clear about its methodology, share relevant data and open itself up to peer review. Taking these steps toward openness will instil confidence in partners and lead to more sustainable fundraising for aid relief.”

“Funders have tough choices to make, and a more grounded and evidence-based assessment of The Global Fund’s actual impacts would help the organisation make its case in the difficult replenishment cycle ahead,” said Amanda Glassman, chief operating officer at the Center for Global Development.

“Our concerns are not new,” said Kalipso Chalkidou, director of global health policy at the Center for Global Development. “For years, we’ve called for The Global Fund to stand on firmer ground when estimating the lives its work has saved. The Global Fund does important work, and in order to make sure everyone has confidence in that work, it’s time for the organisation to take our calls for transparency seriously.”

“Global health partnerships, including The Global Fund, hold a position of immense responsibility as the stewards of scarce aid dollars,” said Rachel Silverman, policy fellow at the Center for Global Development. “To best serve the populations in need of that assistance—and to attract additional resources in the upcoming replenishment cycle—we hope that The Global Fund and other funders will embrace the highest standards of evidence and accountability.”

To read the analysis, visit the Lancet’s site here.

###