New Lancet Article about GAVI Shows Need for Improved Monitoring of Immunization Coverage

Tracking progress towards universal childhood immunizations and the impact of global initiatives, a study by researchers at the Institute of Health Metrics and Evaluation at the University of Washington published in the December 13 edition of the Lancet, concludes that estimates of immunization coverage in developing countries have been overstated, and that in recent years many countries seem to have inflated their coverage figures in response to performance-based incentives from GAVI (see report in the Washington Post). Given the prominence of GAVI's novel use of performance awards, and growing interest by developing country governments and donors alike in results-based financing approaches, this study will no doubt be circulated widely -- and potentially will be misread as an indictment of the concept of performance incentives. There are better lessons to learn from it.The study analyzed the number of children receiving the full three doses of diphtheria, tetanus, and pertussis (DTP3) immunizations in 193 countries from 1986 to 2006, comparing estimates from multiple household surveys with figures officially reported to international agencies by Ministries of Health. The researchers looked at whether the increases in immunization coverage reported by governments, which have served as the basis upon which GAVI has provided a reward of $20 per additional fully immunized child, are exaggerated, presumably because of that material incentive. They find that of 51 countries eligible to receive performance incentives from GAVI since 1999, six countries overestimated the additional number of children immunized by four times, compared to coverage levels based on household surveys; 10 countries overestimated their increase in coverage by more than double; 23 countries overestimated by less than double; and eight countries underestimated their increase in the number of children immunized. Extrapolating, the authors calculate that about 6.2 million children reported to have been immunized in countries eligible for performance rewards may not have been, and that GAVI may have provided $140 million more in incentive payments than would have been justified on the basis of increases in coverage reported through surveys. I'm guessing that the conclusion some draw from this is that this is yet another case of corrupt behavior by malign developing country officials, and that doling out aid for achievement of particular targets is foolish.Let's start first with what isn't news about the study. The authors demonstrate that estimates of immunization coverage based on survey data -- that is, reports from mothers or other caretakers about whether their children have gotten their shots, and which ones -- are systematically lower than estimates based on the government's administrative data, made up of reports from health centers and other providers. This has been observed for many years, and is found in high- and low-income countries alike. Common sense alone suggests some of the reasons this might be so: Survey-based estimates might be lower than the true coverage rate because of omissions in reporting; not all moms will remember accurately whether a child has received all three rounds of DTP vaccine. At the same time, estimates based on provider reporting are likely to be higher than the true coverage rate because health workers may find it easier to tick the box for "immunization" than to actually give the shots, they may vaccinate the same child multiple times (as often occurs during mass immunization campaigns), or officials in the national immunization program or higher in the Ministry of Health may fudge the numbers to look better. In most cases, we've got to assume that the truth lies somewhere in the middle between the survey and the administrative data. That said, it's surely the case that the new study is right in its conclusion that official statistics paint a prettier picture than is warranted.The second thing that isn't news is that when incentives for good performance are offered, the integrity of information about performance can be eroded. We've seen that in the U.S. when funding for schools has been tied to test results; we've seen it in the U.K. when rewards have been provided for improvements in the quality of medical care; and we're seeing it here, with GAVI's performance-based incentive.What is news is the magnitude of the overreporting (again, at least compared to estimates from household survey data). This strongly suggests that GAVI needs much better ways to double-check reports of performance, that WHO and UNICEF have to step up their efforts to help governments improve their information systems, and that GAVI might want to consider introducing penalties for persistent gaming. In any performance-based incentive approach, there are four fundamental elements: setting appropriate targets; calibrating incentives right; making the deal between funder and recipient crystal clear, through a contract or other means; and having a means of verification of results that is credible to both sides. Each of these elements can be put in place at the outset, as GAVI tried to do, and then has to be checked on and adapted over time as new information arises. In this case, GAVI is getting valuable information about the need to improve its verification process. And that's a point that the Chris Murray, founding director of the Institute of Health Metrics and Evaluation and co-author of the study, who has long argued for development of independent monitoring of health sector performance, is bringing to light: "An incentive to over-report progress, either intentionally or unintentionally will always exist with performance-based payments," he says. "To counteract this problem requires not only independent monitoring, but also a system that is based on rigorous, empirical measurements using the best scientific methods available."