A Tool to Estimate the Net Health Impact of COVID-19 Policies

This is part of a series of blogs on the non-COVID-19 excess mortality caused by the response to COVID-19. See here for the first in the series. Since we published this blog post, we’ve received lots of helpful feedback, including some concerns about our assumptions for the unmitigated scenario. To clarify, we’ve written a partner blog and encourage readers to take a look at it as well.

Since publishing, we have now developed an updated version of the tool. See here for the latest version.

Many groups have begun estimating the number of what some commentators call “lockdown victims” of the response to COVID-19. In the UK where data are available, excess non-COVID-19 deaths make up about a third of all excess deaths reported thus far; this is only likely to increase given the time lag in mortality in chronic conditions. In low- and middle-income countries (LMICs), where such official statistics are unavailable or less reliable, social media reports and evidence from health management information system are beginning to confirm some of the devastating predictions. While we agree with WHO chief scientist Dr. Soumya Swaminathan that “it is urgent for health systems everywhere … to ensure that routine health services…are not interrupted,” we can’t ignore that health services are, and will continue to be, interrupted—not only by the outbreak itself, by the response to it from governments, the global community, and individuals.

Without strong vital statistics systems, we must rely on models to predict excess mortality, both COVID-19 and non-COVID-19. Because there is no comprehensive modeling platform at this time able to take a “whole of health” perspective, we have developed a simple tool that enables users to estimate these non-COVID-19 health effects for their own context, using their own data or assumptions. In this blog, we explain how the COVID-19 Net Health Impact Calculator works and demonstrate its use by providing some preliminary estimates for the Sahel region. We encourage you to explore the calculator and read more about its use in its user guide.

An overview of the COVID-19 Net Health Impact Calculator

We built the COVID-19 Net Health Impact Calculator to be a practical tool. It is designed to help decision-makers planning the COVID-19 response in any given country to take a “whole of health” perspective that fully considers the indirect health effects of different policy responses to the outbreak.

Defining net health and choosing scenarios

We define the net health impact as the difference in COVID-19 deaths averted between two scenarios, minus the difference in non-COVID-19 deaths between the same two scenarios. Where this number is positive, it suggests that the policy being examined does more good than harm compared to the comparator scenario. Conversely, where the net health number is negative, it suggests that the policy causes more harm than good.

COVID-19 deaths

We use the predicted estimates from Imperial College London’s report on COVID-19 deaths in five scenarios:

1. unmitigated

2. mitigation including social distancing

3. mitigation including enhanced social distancing of the elderly

4. suppression “0.2 deaths per 100,000 per week trigger”

5. suppression “1.6 deaths per 100,000 per week trigger”

The Imperial predictions were published on 26 March 2020. Since then, there has been an enormous increase in data from all parts of the world. The Imperial team has not yet published a revised set of long-term predictions, but short-term forecasts of infections and deaths are now available for low- and middle-income countries. There are other country-specific models, including from the London School of Hygiene & Tropical Medicine, which also considers and unmitigated epidemic and interventions combining social distancing and shielding, and from the WHO’s Regional Office for Africa (WHO AFRO), which reports estimates for each of the 47 countries in the region, but does not report estimates with and without containment measures (we think it reports directly the difference between these two scenarios). In the current version of the calculator we include the Imperial and WHO’s AFRO estimates; we will be integrating the London School of Hygiene & Tropical Medicine estimates in the next version. We also provide the user with the option of inputting the results of local models for their country or countries of interest where they exist or as part of a “what if” scenario.

Non-COVID-19 deaths

We use the most recent (2017 estimates published in 2019) Global Burden of Disease (GBD) estimates produced by the Institute for Health Metrics and Evaluation to model potential excess deaths due to disruption in services from the COVID-19 response. The 2017 GBD effort classified causes of death or disability into four levels. We use level 2, which includes 22 disease and injury categories. (For additional details on our use of the GBD estimates, please read the calculator user guide, available here.)

Users can consider possible combinations of intensity (how much disruption is caused), and duration (how long the disruption lasts). In terms of duration, similarly, users can select a value across all causes or vary it by cause. Given that pandemic models such as Imperial’s assume that suppression strategies will need to be maintained in some manner until vaccines or effective treatments become available to avoid the risk of later outbreaks (possibly 18 months), in the best case scenario, we suggest users always include as one scenario a duration of at least 18 months when using the Imperial estimates of COVID-19 deaths. By contrast, estimates by WHO’s AFRO and the London School of Hygiene & Tropical Medicine only go as long at 12 months. When using these estimates, it’s important to ensure that the calculation of non-COVID-19 deaths uses the same time frame. The calculator also allows users to examine changes over time. For example, one might imagine that the disruption dissipates over time as efforts to mitigate it are put in place. Thus, users could assume that there’s a 20 percent increase in mortality, gradually decreasing back to pre-COVID-19 levels over 18 months.

The calculation of excess mortality is then: GBD estimates x intensity x duration

The COVID-19 response will also result in less mortality for certain causes. Specifically, of the level 2 GBD causes, transport injuries and unintentional injuries, will see reductions. In a similar fashion to excess mortality, we allow users to consider different combinations of duration and intensity. The calculation of a reduction in mortality is therefore the same as for excess mortality.

Unit of analysis

The Imperial model considered 202 countries. The 2017 GBD includes 195 countries and territories. Between the two sources, there are 188 overlapping countries and territories which we include in the tool. We also use the World Bank geographic regions and income categories, and have also added other regions, such as the Commonwealth and Sahel (defined as Senegal, Mauritania, Mali, Burkina Faso, Niger, Nigeria, Chad, Sudan and Eritrea). Please let us know if you would like to see more country groupings made available. (We plan to include sub-national burden data where available).

Preliminary estimates for the Sahel region

The Sahel region, defined for our purposes as the list of countries in table 1, includes some of the poorest countries in the world, with the worst health outcomes. Countries across Africa took early and decisive action against COVID-19, and the Sahel countries were no exception (see map below).

Here, we consider two scenarios: unmitigated and early suppression triggered at a rate of 0.2 deaths per 100,000 per week to reflect the early response seen in these countries—we assume that the WHO’s AFRO estimates are most similar to this comparison. We assume the effect of the policies introduced in these countries is the same, but table 1 shows that the stringency index varies between 61.61 (Niger) and 92.72 (Burkina Faso) (and data are not available for Eritrea).

As mentioned above, one might hypothesize that more stringent measures will be more disruptive to routine health services, leading to deeper and longer effects, but to keep the example relatively simple, we do not consider that here. Because the unmitigated scenario does not consider spontaneous social distancing (which would reduce COVID-19 deaths), we do not consider the potential effects of such behavioral changes (e.g. individuals not seeking care for fear of contacting the virus) on non-COVID-19 mortality either. Nor do we consider that an unmitigated outbreak across LMICs would result in excess non-COVID-19 deaths by overwhelming the limited number of secondary and tertiary care facilities. Hospital capacity in most LMICs is already very low, and most preventable mortality is addressed not at hospitals but through the primary healthcare system.

Using Imperial’s predictions, we assume mortality increases of 20 percent, gradually returning to pre-COVID-19 levels over 18 months. For transport and unintentional injuries, we assume a 50 percent reduction over 3 months. Using WHO’s AFRO predictions we assume the same but over a 12-month period to be consistent.

As we made clear in an earlier blog, we do not believe the widely modelled choice of do nothing versus complete shutdown is particularly helpful to policymakers. However, in this simple example, the conclusion depends on the source of COVID-19 projections—choose the Imperial figures and the policy of suppression triggered by deaths of 0.2 per 100,000 per week does more good averting more than 280,000 deaths. Choose WHO’s AFRO figures, and the same policy does more harm, causing 200,000 more deaths. We can also use the calculator to determine the thresholds at which a policy would be equivocal—that is to say, the COVID-19 deaths averted would be offset by the number of non-COVID-19 deaths caused by the policy. In this example, using the Imperial numbers, an increase of mortality of 35 percent, gradually returning to pre-COVID-19 levels over 18 months, leads to policy equipoise. Using the WHO’s AFRO numbers, just a 4 percent increase in non-COVID-19 mortality leads to policy equipoise.

Conclusions

Decisions to lockdown nations should have been informed by consideration of the unintended, yet entirely predictable, health consequences. But to date, there has been no effective tool for modeling those predictions.

Real data is emerging throughout the world about the number of COVID-19 deaths and also the disruption to routine health services caused by global and local responses to the pandemic. However, it is too early to know how deep and long-lasting these disruptions will be. Furthermore, global modelling efforts continue to be disease and population focused with the same groups publishing separately on COVID-19; on HIV, TB, and malaria; and on vaccine-preventable deaths.

Our calculator offers a simple heuristic framework for taking and examining a broad health systems perspective on the COVID-19 response. It can help quantify the net health impact of responses using a “whole of health” approach.

The appropriate use of the calculator and its outputs can contribute to effective policymaking. But like any tool, its misuse or misinterpretation of the outputs can mislead decision-making. Needless to say, the better the data that goes into the calculator, the better the “answers” it offers.

Please check out the calculator and its user guide here. We are currently adding features such as health- or quality-adjustments, discounting of deaths, and years of life lost. We would love to hear your feedback about ways in which we can improve this tool to make it as useful as possible. Share your thoughts with Damian Walker (damiangwalker@gmail.com) or comment below.