How Should We Evaluate Lockdowns? Disentangling Effectiveness, Context, and Politics

With thanks to Kalipso Chalkidou for valuable feedback

How should we evaluate measures taken in response to COVID-19? An early precautionary stance was to act first against an immediate threat, and later work out the evidence-base. Even now, as the UK government considers the range of options for tackling the ongoing outbreak, its Scientific Advisory Group for Emergencies (better known as SAGE) published a summary of the effectiveness and harms of different non-pharmaceutical interventions (NPIs), and concluded that “The evidence base into the effectiveness and harms of these interventions is generally weak. However, the urgency of the situation is such that we cannot wait for better quality evidence before making decisions.” However, many different NPIs—actions apart from taking medicines—have now been tried in a range of global settings, and thus we have the evidential means to begin evaluating them. In this blog we discuss some recent evidence from Africa, and we present some thoughts on how lockdowns can be evaluated, using some of the Bradford Hill “criteria.” While our examples are from Africa, we believe this blog is relevant to ongoing debates globally.

What works?

In the early stages of the pandemic, there was significant pressure and urgency to respond, and across the globe we witnessed the widespread adoption of stringent lockdown measures. At the time, decisionmakers had to operate amidst a dearth of evidence while models projected the worst in terms of COVID-19 cases and fatalities, with dire implications for eclipsing health systems’ capacity. Now, several months into the response, we are better equipped to assess the effectiveness and necessity of lockdowns, and to bring evidence to bear in determining the most appropriate package of interventions for different contexts in ongoing efforts to combat COVID-19. This includes assessing both the benefits and the harms of lockdowns (for different durations and levels of stringency) as well as their effectiveness compared to other options.

Questions about lockdowns will continue as many countries see increasing case counts and new spikes. Even after the first generation vaccines are approved for use, there are concerns about global availability and uptake in the face of vaccine hesitancy. With vaccines only forming part of the response strategy, we must critically evaluate other public health tools at our disposal. Below we present some emerging evidence on lockdowns across different settings and considerations that should inform future adoption of lockdown measures.

Context matters

Similar regulations mean different things from both an epidemiological and a human perspective in different places, and for different people in the same place. Thus, a lockdown requiring people to stay at home might reduce social contact in a suburb, but might actually increase social contact in a crowded informal settlement (as appears to have been the case in Accra). This means that intensity scales measuring the stringency of a government’s policy response to COVID–like the one developed by the University of Oxford–cannot be used as a proxy for degree of reduction in social contact, which is the key variable in most models, including the influential Imperial College London model. Preregulation levels of social contact may vary widely too, as may compliance with regulations. All of this means that a physical distancing recommendation in Stockholm might yield a greater reduction in social contact than a hard lockdown in the Kibera neighborhood of Nairobi, meaning the effectiveness of a regulatory intervention cannot be established independently of the local context.

Context also matters for the indirect health effects of lockdown. Wealthier nations and wealthier people are better able to withstand the economic shock of lockdown. In sub-Saharan Africa, the informal sector accounts for about two-thirds of total employment, according to some estimates. GDPs in the region are among the lowest in the world, and governments may not be any more able to protect citizens from the ill-effects of loss of livelihood than are individuals themselves. Malnutrition, disruption to treatment programs for other diseases such as HIV and TB, the long-term health consequences of disrupted education, and outbreaks of other diseases are probable consequences of lockdown in many parts of the developing world.

Recent evidence from Africa and the Bradford Hill “criteria”

Given that lockdown carries a more serious health risk in poorer regions and for poorer people in those regions, the evidence for its effectiveness in those contexts is especially important for decision-makers. Two recent papers look for correlations between regulatory regime and COVID-19 epidemic trends in Africa. In the first, the authors (including some of us) fail to identify any changes in the trajectory of the epidemic in South Africa that might be attributed to “lockdown,” (see Figure 1 below).

The earliest measures introduced on 17^th March by the South African government included restrictions on large gatherings, on restaurants and bars, and similarly “mild” measures. South Africa moved into a very restrictive lockdown from 27^th March, coinciding with a decline in the trajectory of new cases. Deaths follow the same pattern some time later. This can be nothing more than coincidence since it takes some days for regulatory measures to have an effect on detected cases. South Africa remained in “Level 5” lockdown, as it became known, until the end of April, and “Level 4”—which was almost as hard—until the end of May. Cases, followed by deaths, continued on a constant, although exponential, trajectory during this period, and remained on the same trajectory until the latter part of July, when the curve flattened out. (South Africa carried out widespread random testing.) There was no regulatory change in July, and when regulations were further eased to “Level 2” in August, there was no change in the—by then, falling—rate of infection, either.

Figure 1. COVID-19 caseload in South Africa, overlaying the dates of the different “levels” of the lockdown

One cannot simply “read off” causality from a line graph. However, we can look for features that might indicate causality, such as the nine “criteria” (strength, consistency, specificity, temporality, biological gradient, plausibility, coherence, experiment, and analogy) famously proposed by Austin Bradford Hill many years ago to provide epidemiological evidence of a causal relationship between a presumed cause and an observed effect. Bradford Hill’s “criteria” are still widely used by epidemiologists today (note that Bradford Hill was clear that these are not criteria in the sense of sine qua nons for causality, but rather intended them as viewpoints or perspectives from which to consider whether an observed association might be causal). One of these is biological gradient (or dose-response relationship), meaning the greater the exposure (the “dose”) to the presumed cause, the greater the response. However, there appears not to be a dose-response relationship in South Africa between lockdown level and reduction of transmission. This corroborates the point made earlier, that scales measuring policy stringency may not be a proxy for any epidemiologically significant variable.

Another Bradford Hill criterion for establishing a causal relationship is strength (or effect size). South Africa’s lockdown measures in mid-March correlated with a reduction in the rate of transmission, but there was no correlation between further regulatory changes and other changes in the epidemic’s trajectory. Similarly, a second recent study examines the nature of regulatory regimes in nine African countries, and looks for patterns in the trajectory of the epidemic in these countries. There is no single definition of “lockdown,” the authors argue, with different countries enacting regimes that differed in important details. The authors offer a helpful two-by-two matrix to characterize lockdowns two-dimensionally (an improvement on one-dimensional intensity scales). They also point out that, in overcrowded contexts, regulations with the effect of keeping people at home could plausibly have led to an increase in infection rates. They find a strong relationship between testing and cases, predictably, but observe “no obvious pattern” in the different lockdown measures imposed in different countries and the shape of the respective national epidemics, even though they reason that there must have been some effect of measures designed to reduce social contact.

Lack of plausibility– when lockdown measures achieve the opposite of what they intended

Another of Bradford Hill’s criteria for assessing causality is plausibility, i.e. whether a plausible biological mechanism exists for the proposed causal link. Bradford Hill recommended taking extra care here, since biological plausibility is relative to current biological knowledge, which might tail behind the causal knowledge derived from observations. The mechanism by which lockdown regulations are supposed to work is generally clear, although details are sometimes not. Following the blanket ban on outdoor exercise during lockdown Level 5, for example, the South African government decided to permit exercise only between 6am and 9am. This time restriction meant that all citizens wishing to exercise did so at roughly the same time, increasing the degree of social contact due to exercise. (Similar remarks might be made about the closing of pubs in the UK at 10pm, reducing what might otherwise be a staggering effect.)

Restricting certain activities is intended to lead to a reduction in the number of contacts between new individuals over short time periods (e.g., daily contacts). However, this mechanism is not plausible where people live in crowded conditions, share communal ablutions, and queue for social grants and food. Thus, the biological plausibility of strict lockdowns having a significant causal effect on the COVID-19 pandemic in such conditions is limited, and these conditions are widespread in the regions covered by both the studies we have described.

Perhaps one important lesson from both these papers is that the distinction between means and ends must be kept in mind when devising public health interventions. Severe restrictions on freedom of movement and activity have been characterized as “suppression” measures, but “suppression” has also been characterized as the reduction of social contact by a certain percentage (e.g., 75 percent). Yet in some situations, restricting movement outside the home for nonessential purposes may not yield reduction of social contact. Suppression is an end, lockdown a means, and lockdown is only a “suppression strategy” to the extent that it is appropriate to the end. In many parts of the world, lockdown is not a suppression strategy at all, because it does not reduce social contact to the level necessary to bring the reproduction number below 1; indeed, in some overcrowded contexts, it may even increase social contact.

The politicization of lockdowns

Debates about lockdown in the US and UK are becoming more political, not less. But as some of us argued previously, the way that opinions on lockdown have lined up along the interests of the liberal-conservative axis of American politics or the left-right dimensions of Britain politics, is largely irrelevant to the interests of the poorest in the world. This may be part of the WHO’s reasoning in recently stating: “We in the World Health Organization do not advocate lockdowns as the primary means of control of this virus.” Regulations that might look the same from a legal or political perspective might have very different epidemiological significance. Their assessment cannot, therefore, be in terms of political allegiance. It has to be empirically based, and localized to a region.

Politicization can also cause confusion between quite distinct public health measures. The term “lockdown” may even be confused for any exercise of state power to compel in the name of preventing COVID-19, even where what is compelled could be part of an alternative strategy to lockdown: contact tracing or mask-wearing. It is easy for all public health measures to become equated because they are compatible and imposed by a single authority, even when they are separable and justified by different rationales. Contact tracing formed the basis of some nations’ responses, rather than “lockdown,” and should be seen as a potential alternative strategy on a different dimension altogether from any “intensity scale” of lockdown. Political polarization makes it harder to see such alternatives.

We hope to see more assessment of the different strategies countries have used to combat COVID-19 based on the large quantity of data now available. Managing COVID-19 should no longer be a stab in the dark.