An Expanded Tool to Estimate the Net Health Impact of COVID-19 Policies

Since the beginning of the COVID-19 pandemic, reports about the indirect health impacts of COVID have been published in nearly all countries. In the five months since we created our open-access inventory of resources, we have already logged in 200 resources and publications on the topic, illustrating the magnitude of the impact felt across the world. A recent survey by the WHO found that many countries are seeking guidance on monitoring the disruptions and developing mitigation strategies to avoid adding to the outbreak’s growing death toll.

Earlier this year, we released a tool to estimate the net health impacts of COVID-19 policies. Its objective is to support decision-makers in estimating the impact of different strategies to tackle COVID-19 (e.g., social distancing, suppression) using a whole-of-health approach, by not only looking at COVID deaths but also considering non-COVID (i.e., indirect, collateral) deaths. Since the tool’s release, we have received many questions and comments about its use.

Today we release a new version of the tool, which includes a modest update from the initial release: the addition of COVID mortality estimates from models developed by the London School of Hygiene and Tropical Medicine (LSHTM). The LSHTM estimates add to those from Imperial College London (ICL) and WHO’s Regional Office for Africa (WHO AFRO) that were included in the first version of the tool. The updated tool will allow 80+ countries (figures have not been released for all low- and middle-income countries (LMICs)) to consider mortality estimates from 11 new scenarios, including shielding, public health measures, and school closure. The rest of the tool remains unchanged.

Objective of the tool

This new version of the tool allows users to compare mortality estimates from three main models (ICL, WHO Afro, and now LSHTM), with estimates of indirect health impacts (generated by end users in the tool) using Global Burden of Disease (GBD) data. By doing so, users can make an educated guesstimate of the net health impacts. We provide an example of net health impacts estimation for the Sahel Region in our first blog.

As we stated in our previous blog, we define the net health impact as the difference in COVID-19 deaths averted between two scenarios, minus the difference in non-COVID-19 deaths between the same two scenarios. Where this number is positive, it suggests that the policy being examined overall does more good than harm compared to the comparator scenario. Conversely, where the net health number is negative, it suggests that the policy causes more harm than good.

The tool also allows users to calculate indirect health impacts, without calculating the net health impacts. Since the release of the first version of the tool, end users (including within our teams) have primarily used the tool for this purpose, mainly because the idea of “net health impacts” is complex to apply in practice, especially as it requires assumptions about what would have happened under counterfactual policy scenarios (see our discussion on the unmitigated scenario)—which is difficult given the uncertainty around this outbreak.

COVID mortality

A growing number of epidemiological models attempt to estimate the mortality associated with COVID under different policy scenarios. We selected data from three main models (ICL, WHO AFRO, and LSHTM) because the country estimates for those models are available in an accessible format to many LMICs (and, in the case of WHO AFRO, for the region). It is important to note that those estimates are neither predictions nor forecasts: they are not plotted against observed or collected data. Together, those models provide estimates for 17 policy scenarios (five scenarios from ICL, one from WHO AFRO, and 11 from LSHTM). Because of differences in approaches, methods, data, and assumptions, it is worth noting that the estimates vary significantly from source to source, as we have already discussed in a previous blog.  Where estimates exist from local models, we also encourage users to add those to the tool.

Mortality can be queried using two outcomes: the number of deaths, and years of life lost (YLLs). YLLs are a summary measure of premature mortality, by taking into account the age at which deaths occur, overall giving more weight to deaths occurring at a younger age. Calculating YLL for LMICs was tricky, an explanation is included in the user guide.

Non-COVID mortality

Global Burden of Disease is one of the most comprehensive efforts to date to map out the sources of morbidity and mortality across countries, time, age, and sex. For each country, the tool features mortality information along 22 causes/categories (referred to as level two classification in GBD) for the latest year available (2017).

Users estimate non-COVID deaths using assumptions on two parameters for each of the 22 causes of mortality: size of impact and duration of impact. For instance, users could estimate an increase in mortality attributed to HIV/AIDS and sexually transmitted infections, assuming the lockdown would increase mortality by 10 percent for eight months in their country. Those impact and duration estimates are entered for each cause separately because the impact will differ from one cause to another. For instance, the impact on mortality from road traffic accidents and injuries is likely to be negative due to restrictions of movement. On the other hand, it has now been widely documented that the impact of COVID has led to an increase in the mortality attributed to maternal and child health or nutrition.

Assumptions about the impact and duration of impact can be informed by the already growing number of studies and estimates produced (including from global models—such as those for HIV, TB, and malaria, or maternal and child health), data collected at the local level, “educated guesses,” or estimates from consulting experts or health professionals working on the ground. It is worth noting that we are working with five countries (Kenya, Philippines, South Africa, Thailand, and Uganda) to produce a range of estimates of non-COVID mortality for selected diseases based on surveillance data, routine health systems data (including electronic medical records and claims data), or existing literature as part of our work on indirect health impacts. We also worked with colleagues in Ethiopia.

Other use cases

While the tool’s primary objective is to calculate net health impacts, it is worth noting that it also allows users to (i) review and compare what estimates are available for their country from different sources and for different scenarios (this is, to our knowledge, the only tool available to do this) and (ii) compare the mortality estimates from COVID models to the overall burden of disease in their countries.

Unaddressed queries

We hope that the addition of the LSHTM estimates in this new version of our tool can further support countries by including new scenarios and estimates from a different approach. We want to highlight that we have received various comments which we could not (yet) address in this update. First, several users pointed to the need to use measures of morbidity (such as QALYs or DALYs) rather than deaths to include all health impacts, not just mortality. The available data do not allow us to do this; for instance, epidemiological models currently only estimate deaths. Second, we are not able at this point to represent uncertainty in the net health impact calculation, especially as the model uncertainty around COVID deaths is not shared in all sources. Finally, we have addressed some questions about the unmitigated scenarios in a separate blogpost.

A new version of the tool and updated guide can be found here.

We would love to hear your feedback about ways in which we can improve this tool to make it as useful as possible. Share your thoughts with Damian Walker (damiangwalker@gmail.com) or comment below.