When Numbers Don’t Speak for Themselves: COVID-19 and Thoughts on How to Measure a Country’s Performance

Please note this blog is part of a short series on COVID-19 and mortality. Please see here and here for our previous blogs, and here for a link to a recent event.

There are multiple data and metrics used to assess a country’s performance in responding to the threat of COVID-19. As seen in a recent interview with President Trump of the United States, the choice of metric can influence whether a country’s response is considered a relative success or failure. In this case the metrics leading to different narratives were per capita death rate–the US looks relatively bad–and case fatality ratio–the US looks much better. In this blog we share our thoughts on how to measure a country’s performance in confronting the pandemic.

Measures of mortality relevant to assessing performance

Table 1 shows the five key mortality estimates required to understand the impact of COVID-19. Our central question is this:

Can existing sources of data, when combined with judicious use of modeling, tell us how well a country’s COVID-19 control measures have performed?

Table 1. Five mortality estimates, before, during, and after COVID-19

Pre-COVID-19 Mortality – A

We label deaths that occurred before the outbreak of COVID-19 “A.” Data on these deaths are produced by national governments as part of civil registration systems. However, in most low- and many middle-income countries (LMICs), the data are limited by low completeness and lack of timely reporting and data processing. If countries invest today in the use of the World Health Organization (WHO) international standard death certificate, (ICD-11), and in boosting the civil registration of deaths, they will quickly cease to need to rely on modeled estimates of these deaths. However, for many years the Institute of Health Metrics and Evaluation has modeled the number, and cause, of deaths (by age and sex) in 195 countries to fill this chronic data gap.

Excess Mortality during and after the COVID-19 Pandemic – B and C

The next broad category of deaths (both COVID-19 and non-COVID-19) to examine are those that are occurring now and in the future. These signify how policies have affected the level of infection and mortality. These latter deaths may be measured in the future but by necessity will be modeled today before they have occurred.

The deaths in cell B (COVID-19 deaths to date) are reported by national governments and summarized by groups such as the European Center for Disease Prevention and Control (the source used by Our World in Data) or the WHO. We also have predictions of future COVID-19 deaths for a range of policies available to governments to help slow the spread of the virus. Options here include the estimates published by Imperial College London, London School of Hygiene & Tropical Medicine, and (ideally) “local” models where available. Note that some of the early predictions of overall mortality burden of the pandemic have not been updated to include the actual policies adopted, which have evolved over time as countries course-correct their responses. Therefore, the original predictions are of limited utility today.

In cell C we have the non-COVID mortality occurring during and after the pandemic, which is challenging to measure where routine systems have not historically provided timely cause-specific or total mortality data, i.e., in most LMICs. We increasingly have modeled program/cause-specific estimates of the potential non-COVID-19 deaths caused by actions taken by governments (that will produce intended but also unintended consequences) and changes in the behavior of individuals (due both to policies but also the fear of the virus). However, unless countries specifically track the total mortality occurring and distinguish it from confirmed and suspected COVID-19 mortality, they will lack comprehensive estimates of these non-COVID-19 deaths. At present there are at least 13 countries, primarily supported by Bloomberg Philanthropies, seeking to address this question through the application of a technical package for rapid mortality surveillance.

Of course, part of the problem is that there will be a lag in the reporting of many of these deaths as routine health services disrupted by COVID-19 often prevent future mortality; hence our reliance on models. Similarly, the health consequences of the economic impact of COVID-19 will occur in the future. Nevertheless, we encourage more efforts to attempt to model (and then start and keep tracking) these deaths in order to provide a more complete picture of performance.  Without considering the indirect health effects, we do not see how the performance of a country’s response can be fairly assessed (even though most debates on this topic still tend to exclusively focus on the COVID cases and deaths count).

It is possible to associate the trajectory of total and cause-specific excess mortality with the introduction of public health and social measures. For some this may be taken as “plausible” evidence of the impact (or lack) of policies. What may never be well understood, however, is whether the virus or the control measures caused excess deaths.

COVID-19 and non-COVID 19 mortality absent control measures – D and E

Attributing excess deaths to either the virus or the unintended consequences of policy requires modeling the counterfactual of what would have happened in the absence of control measures. Let’s call the counterfactual of how many COVID-19 deaths would have happened in the absence of any control measures “D.” Note that the Imperial and London School models failed to consider any spontaneous social distancing in their unmitigated scenarios, and are therefore overestimates of COVID-19 mortality.

And finally, how many additional non-COVID-19 cases and deaths would there have been due to behavior changes, and the overcrowding of health systems, and the ensuing disruption to routine health services? Let’s call these deaths “E.” We are not aware of any published estimates of E. Note that these should not be assumed to be A (i.e. zero excess non-COVID-19 deaths).

Uses and limitations of mortality measures

These different estimates can be analyzed in the following ways:

In conclusion

At the beginning of this blog, we asked “Can existing sources of data, when combined with judicious use of modeling, provide summary measures capable of quantifying how well a country’s COVID-19 control measures have performed?” The short answer is yes, but we should proceed with caution in doing so.

As the COVID-19 pandemic passes its eight-month mark, it’s clear that there has been mixed success with controlling its spread. Countries such as Fiji, South Korea, Thailand, Vietnam, Germany, and New Zealand, in addition to most of the African continent, have been comparatively successful in controlling the outbreaks within their borders. Countries such as the United States, Brazil, and South Africa have yet to “bend the curve;” new confirmed cases are still increasing day after day. Most countries sit somewhere between these two groups of countries.

Of course, without the benefit of hindsight, we cannot definitively declare successes and failures. However, it matters how we define success (and failure). Has a country succeeded if it has stopped the spread of the virus leading to very few deaths? What if this “success” has been achieved at the expense of other non-COVID-19 deaths? What, in addition to excess mortality, must we measure or model?

As some countries begin to relax measures adopted at the peak of the pandemic several months ago, while yet others consider reintroducing lockdowns to whole or subpopulations, we believe that a better understanding of what drives COVID-19 mortality rates across time and space, as well as considering metrics beyond excess mortality, can help decision-makers who are planning the ongoing COVID-19 response.

Mortality measurement is part of the picture. It is necessary but not sufficient for answering relevant policy questions pertaining to the health impact of the virus and the response to it. A true focus on the “excess burden” would necessitate the use of disability-adjusted life-years (DALYs) or some other summary measure of population health capable of accounting for all the non-fatal disability caused by the virus. Likewise, for policymakers, health impact data in the absence of information about the cost of policy choices and other core indicators, only tells part of the story.

Finally, when listening to policymakers and reading their reports, our major recommendation is caveat lector–reader beware! Each and every metric has its own limitation, and selective use of one over others can present a distorted—or at best incomplete—picture of performance.