As Global Health Players Pivot To COVID-19 Responses, We Need Coordinated, Real-Time, Formative Evaluations

An estimated $15.9 trillion has been mobilised to respond to the health and economic impact of the COVID-19 pandemic. Some of this has gone to multilateral and bilateral funders to support low- and middle-income countries’ (LMIC) governments. But how is this money being spent, and how is it impacting country budgets and expenditure more broadly?

Of the total $15.9 trillion mobilised for COVID-19, multilateral funders⁠—including the IMF, UN, EU, Gavi, and the Global Fund⁠—have committed $4.2 trillion, with a further $600 million from bilateral funders. Clear standards for tracking, reporting, and evaluating the uses of new financial commitments is critical. Real-time, formative impact evaluation can provide timely information on what is being funded and paint a picture of “what works” to inform policy approaches in real time and avoid duplication, improving the overall efficiency of response funding. Furthermore, these evaluations can and should play a pivotal role in informing the response to future pandemics and crises.

In this blog, we set out three big challenges for programming, monitoring, and evaluation during a pandemic, highlight ways in which these challenges can be overcome, and explore the long-term humanitarian consequences that will be left in the wake of funding decisions being made now.

Challenge 1: Impact of Covidisation on health systems and policies

Health systems in LMICs are being influenced by “Covidisation”—the increased focusing of global policy and research on COVID-19. Health-focused multilaterals, bilaterals, NGOs, and philanthropies are pivoting parts or all of their funding and programming towards COVID-19, and at the same time LMIC policy makers are leveraging high-income country modelling estimates (e.g. Imperial and IHME) to inform their COVID-19 response. The pivot of global health-focused agencies towards COVID-19 has a clear influence on the way in which recipient countries address COVID-19, and the indirect impacts of this influence can have short- and long-term ramifications. For example, influence can “crowd out” other health and research priorities in favour of COVID-19 in the short term. In the long term, it will likely place a greater focus on global health security priorities above other endemic diseases or health challenges.

Furthermore, existing models which have been developed for high-income settings are being used to inform some low-and middle-income country policy decisions, despite significant data and contextual limitations in making them relevant to LMICs. However, due to weak national statistics and misaligned incentives, some countries do not have timely, complete, and accurate data to inform their own policy choices. Understanding the impact of Covidisation on health systems and policy making must be underpinned by locally relevant data and models.

Challenge 2: Precedent set for lack of coordination in crisis

As health-focused agencies pivot some of their spending to COVID-19, we should also be mindful of questions around their institutional effectiveness during crisis. Previous humanitarian disasters provide valuable lessons for COVID-19. For example, after Typhoon Haiyan in 2013, international agencies reportedly bypassed local actors and authorities, leading to divergence, duplication, and poor targeting of efforts. The widely acknowledged lack of coordination among the big players in the 2010 Haiti earthquake response compromised agencies’ ability to mobilise quickly around the devastating cholera outbreak which followed. Despite Haiti being the top-funded UN appeal in 2010, 80 percent of patients received in the outbreak’s first three months were taken in by actors outside of the response framework. Official mechanisms appeared inoperable.

Challenge 3: Historical lack of emphasis on attribution and impact assessments makes monitoring effectiveness and value of the COVID-19 pivot harder

Various evaluations and reviews⁠—including Itad’s evaluations of UNICEF, the Global Fund, and UNHCR, and broader reviews such as DFID’s Bilateral Aid Review⁠—have attempted to provide evidence on donors’ contributions to overarching development priorities and goals such as the SDGs. Yet unpacking these contributions in a robust way across sprawling organisations remains difficult⁠—a challenge reflected in analysis of the limitations of estimating lives saved. This sparse and variable picture of aid effectiveness is now being compounded by COVID-19, where in the melee to fund and mobilise an urgent response, little thought has gone to what success looks like in terms of agencies’ relative contributions. The baseline of how funders make a difference⁠—particularly at the country level⁠—is unclear, making the monitoring of COVID-19 pivots’ impact by multiple development partners all the more challenging. Who is making a difference, and how they are achieving impact, is a critical area for analysis that can help to improve future responses.

How to do better

There is, of course, a limit to the opportunity for reflexivity in the midst of a crisis such as COVID-19, where the response is unfolding in real time. This, however, is where independent and dedicated evaluators with the time and focus can step in to provide rapid analysis and learning for course correction, supporting agencies to prioritise funding and action where it is most needed. As agencies once again go down a complex response road with no end or clear markers of success in sight, impact evaluation plays a more critical role than ever. Now is the time for:  

• Transparency of data, including cash flows and procurement data: To support a collective global response to COVID-19, more transparency and information in the public domain on what COVID-19 commitments are purchasing is necessary.
• More local data and data collection systems, less reliance on internationally modelled estimates of mortality and morbidity: Countries that have relied heavily on modelling estimates from elsewhere should shift towards a focus on improving local data including civil registration and vital statistics systems, which can inform decision-making and timely impact evaluations.
• Define “success” for COVID-19 funding⁠—formative evaluation efforts of COVID-19 responses: Bilaterals, multilaterals, NGOs, and recipient countries should collectively agree on what success looks like in terms of their respective contributions to the COVID-19 response, which sheds light on the health system trade-offs likely to be made as a result, and supports value for money for agencies’ financial investments.
• Coordinated and independently funded monitoring, evaluation, and learning (MEL): As unprecedented funding pledges file down to country level, where there is arguably less visibility from international lenses, dedicated oversight and monitoring is urgently required. Organisations should carry out coordinated and collaborative MEL as well as fund local and global independent watchdogs to ensure transparency, allow for course correction, and drive real impact.

Funding in the post-COVID reality

If ensuring the immediate response is sensitive to evidence and the dynamic shifts in the COVID-19 crisis is one thing, sorting out the “messy” aftermath is another. Pre-COVID-19, the UN’s existing humanitarian “bill” exceeded $28 billion, with only 13 percent of this total funded. On top of this, COVID-19’s impact on other development and humanitarian priority areas will be profound. Famine and drought are looming threats, with more than 50 million people estimated to suffer from hunger in West Africa alone, as the toll of acute food insecurity globally jumps from 135 to 265 million. Life-saving immunisation campaigns targeting diseases such as measles have been suspended in 23 countries, cumulatively targeting 78 million children. The impact on livelihoods in the Global South is projected to be severe, with nearly half of all jobs in Africa lost and income losses to exceed $220 billion in developing countries as a whole.

The long-term devastation of COVID-19 in terms of humanitarian needs will require renewed and ambitious resourcing once the immediate crisis has waned. The breadth of impacts and the funding commitments they necessitate are set against a bleak macroeconomic background, with the world economy projected to shrink by 3.2 percent in 2020.

At present, seemingly infinite resources are being thrown at the COVID-19 crisis, but the tightening of budgets will soon come down the track. Tough decisions on future allocations by donor countries and agencies will have to be made. With independent and robust evaluation based on real data, we can help inform value-for-money assessments from the perspective of both high-income country taxpayers and recipient governments⁠⁠. These inputs will be critical for global and domestic allocation decisions.

This blog has been written in collaboration with Itad, a global organisation that provides strategy, monitoring, and evaluation services to make international development more effective.