Improving Development Policy through Impact Evaluations: Then, Now, and Next?

In 2006, CGD published a working group report that addressed the insufficient number of rigorous impact evaluations of social programs in low- and middle-income countries. Last week —marking 10 years since the report’s release—CGD and J-PAL co-hosted the event, “Improving Development Policy through Impact Evaluation,” which echoed three key messages of the 2006 report: 1) conduct more and better evaluations; 2) connect evaluators and policymakers; and 3) recognize that impact evaluations are an important global public good that requires more unconstrained funding. Participants celebrated progress, but agreed that there’s more work to be done.

Here are our three takeaways:

1. Conduct more and better impact evaluations

Even if you’re not an avid follower of impact evaluation news, you may have read blogs or papers criticizing “Randomistas”—researchers who conduct randomized control trials (RCTs)—for taking over development research, lacking external validity, ignoring complexity, and displacing more important topics. The number of impact evaluations over the past 10 years has indeed increased markedly. But the presentations at the event put this trend into context. First, RCTs are not taking over; while they are driving the growth in impact evaluations, other methods (quasi-experimental studies) are increasing too. Moreover, fewer than 10 percent of evaluations conducted directly by major development agencies are impact evaluations and fewer than half of those are RCTs. In the case of USAID, for example, a 2013 study found that only three percent of the agency’s evaluations conducted between 2009 and 2012 were impact evaluations that use a counterfactual to attribute impact to a particular intervention.

More importantly, impact evaluations are influencing policy. Abhijeet Banerjee, J-PAL director and professor of economics at MIT, discussed how evidence from different contexts—rather than just a single, large RCT—has answered old questions and shifted policy debates. He highlighted three such cases: preventive health products like bed nets (for which charging even a small fee has a big effect on demand); microcredit (it doesn’t have a transformative impact on raising incomes); and improving learning (teaching at the right level can produce large improvements). Reflecting on these examples, Rachel Glennerster, executive director of J-PAL, suggested that the benefits from this kind of evidence in just a few areas are large enough to justify the spending that has gone into the entire portfolio of impact evaluations.

2. Connect evaluators and policymakers

Many of today’s impact evaluations engage evaluators and policymakers in jointly defining questions, designing programs, and identifying policy implications. Traditionally, evaluators insisted on independence to assure objectivity. But this separation often results in evaluators being brought in after a project is underway and when the opportunity to collect baseline data and test hypotheses is lost. Glennerster noted that researchers today rely on their methodology and transparency to assure objectivity, allowing them to engage in defining questions and collecting useful data.

The event included both policymakers and researchers who reflected on their partnerships. Bambang Widianto, executive secretary in the office of the vice president of Indonesia, discussed how collaboration with external researchers helped his government’s decision to expand an identification card system that sharply cut waste in the national rice subsidy program. Rema Hanna, scientific director for J-PAL Southeast Asia and one of the researchers who conducted the evaluation, explained how timely funding from the Australian Government permitted the research team to turn around results in a matter of six months to meet the government’s budget deadline. Banerjee used this case to reflect on the value to policymakers of having quick access to technical support from researchers.

3. Greater investment is necessary

A key message of “When Will We Ever Learn?” in 2006 was the need for greater investment in rigorous impact evaluations because the knowledge they create is a global public good. While most agencies and foundations have increased their own spending on impact evaluations, only a few, like the British Department for International Development (DFID) and the Bill and Melinda Gates Foundation, have provided substantial sums to organizations that contract studies with an eye on the collective benefit they provide. At the event, one of us (Savedoff) argued that agencies and foundations need to stop being free-riders and contribute resources to organizations that that can apply these funds flexibly and strategically, so that impact evaluations are not only good quality but also widely relevant to policy. Specifically, agencies could dedicate 0.1 percent of their annual disbursements to the International Initiative for Impact Evaluation (3ie) or other similar funds that conduct impact evaluations to build collective knowledge. Speakers also highlighted funds like USAID’s Development Innovation Ventures and the Global Innovation Fund that solicit a wide range of innovative proposals which are then rigorously evaluated as small-scale pilots or tested at scale.

Looking ahead…

Despite progress, impact evaluations have only addressed a few of the myriad unanswered questions about programs being implemented around the world. Moreover, data on cost effectiveness—an important complement to impact studies and a critical component for decision-making—are hard to come by. While conducting research for Millions Saved: New Cases of Proven Success in Global Health, Amanda Glassman and colleagues identified 50 evaluations of at-scale health programs that used rigorous methods to attribute impact (out of a total of 250 evaluations), but only three of them estimated cost effectiveness. Recapping this experience at the event, Glassman emphasized that cost analyses using standard methodologies can help policymakers draw useful comparisons and enhance value for money.

Summing up the road ahead, Santhosh Mathew, joint secretary of India’s ministry of rural development, highlighted the need for trust, resources, and sustained commitments from donors, researchers, implementers, and policymakers to continue closing the evaluation gap. And many questions still remain: how to assure external validity? how to prioritize when and where to conduct rigorous impact evaluations? While progress over the past decade is surely cause for optimism, it is definitely no time for complacency.