As noted by a few commentators, it is true that the Millions Saved selection criteria – which is an attributable health impact generated by a program at scale – excludes many large health programs. The aim of Millions Saved is to demonstrate that scaled health programs can have a direct impact on saving lives using the best evidence available. We have been able to identify cases that meet these criteria in a number of areas including hepatitis B, tobacco control, and sanitation. Identifying a suitable malaria prevention and control program has been a challenge.
While we recognize the large number of small-scale studies of the efficacy of specific malaria interventions (as reviewed in the Cochrane Library here), we are making a distinction between the efficacy of an intervention and its effectiveness in delivery at scale. Indeed, our research has identified several cases where an efficacious intervention has not had the expected health impact when rolled out at scale (for example see here and here). For this reason, we would hope for direct attribution of effect in lieu of a plausibility argument in at least a few settings.
Still, as commenters on the original blog noted, attribution requires a counterfactual – whether simulated or otherwise. There is a lot going on in countries combating a heavy malaria burden besides the scale-up of malaria interventions. In addition to many other child health interventions (immunization, iron supplementation, sanitation, better drinking water), there has been rapid economic growth, increasing urbanization, improved housing, and growing educational attainment, all of which are associated with changes in all-cause child mortality. Randomized controlled trials are not the only valid way to prove attribution. Among others, differences-in-differences, regression discontinuity designs, and other strategies also work. The book Impact Evaluation in Practice provides an overview to these approaches, all of which have in common a careful consideration of the counterfactual.
Absent a counterfactual, we are also “giving points” to cases that provide more rigorous analysis of possible confounders. For instance, a paper by Demombynes and Trommelerova on the impact of ITN ownership on infant mortality in Western Kenya does not help with estimation of a direct causal effect of ITN or other interventions on child mortality. However, it does use decomposition analysis to at least explore these other factors.
Other commenters noted an important ethical argument - that “in a situation where the resources exist to scale up more broadly, it would be unethical to withhold life-saving interventions for the purposes of an evaluation design.” To this point, one might also point out that scaling resources without evidence of their effect at scale would be similarly unethical—particularly if interventions are not as effective as predicted when applied under these circumstances and funding might have been more effective under a differing service delivery mechanism. This seems important given the relatively low correlation between net ownership and net use in DHS data (see here).
Finally, the blog was useful in bringing to our attention some additional studies we had not identified previously, such as one from Kenya and a few large RCT studies. For example, a study in Zambia by Richard Sedlmayr and colleagues uses a RCT with 81,600 farmers to demonstrate the health benefits of large-scale ITN distribution undertaken in collaboration with the country’s largest cotton buyer. This large-scale trial represents strong evidence and could be complemented by the work undertaken by IHME on all-cause child mortality reduction in Zambia, among other sources.
On the other hand, the result we highlight from Sedlmayr et al. reflects only 6 months of implementation. Do we feel confident, therefore, that these are durable effects?
As a next step, we’ll go back to our Advisory Group and DCPN editors and see what they advise. Thanks again for all of the comments and suggestions, and we look forward to continuing the discussion on Zambia’s program and others.
Disclaimer
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.
