An Homage to the Randomistas on the Occasion of the J-PAL 10th Anniversary: Development as a Faith-Based Activity

Speak to us only with the killer’s tongue,
The animal madness of the fierce and young

—Conrad Aiken, “Sursum Corda”

Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has.

—Margaret Mead

This last week in my MPA/ID class Andrés Velasco gave a really great presentation. Andrés began as a stellar academic, with appointments at Columbia and Harvard Kennedy School, but has also been continually engaged in policy and political practice. From 2006 to 2010, he was minister of finance of Chile, and last summer he was a candidate for president of Chile in his party’s primary (he finished second to now president-elect Michelle Bachelet). He talked to the class about what he had learned by moving from academia to being a senior policymaker with real responsibility (not just an “adviser”) and then a political candidate.

He emphasized two things he had learned from his experience: passion and communication. He noted that a politician without passion is not really a politician at all. To sustain the level of effort and energy to win campaigns—and to inspire others to work for you—you need to be passionate about what you believe will make the world a better place.

The second thing he emphasized was that, while the academic emphasis on thinking clearly is valuable, being able to communicate ideas to people with compelling narratives is equally valuable. He noted that while academics use language to persuade, politicians use language to tell stories and establish connections.

Interestingly, one thing he did not mention at all was “rigorous evidence”—neither the word “evidence” not the word “rigorous” was used, neither separately or jointly.

One might think the rise of the randomista movement in development economics contradicts Andrés and shows the importance of rigorous evidence for policy success. But, ironically enough, the success of the randomista movement proves Andrés right. The enormous success of J-PAL and Innovations for Poverty Action (IPA), and other organizations like them, in promoting the generation of more “rigorous evidence” as a means to improve development practice is evidence of the ability of faith, passion, and communication to attract attention, resources, and acolytes to a cause—independent of any evidence about impact at all (much less “rigorous” evidence).

On December 7, 2013, I attended the J-PAL 10th anniversary extravaganza. One could not help but be overwhelmed with the impressive accomplishments of a small group of “thoughtful, committed citizens.” The rise in the number of people using rigorous methods—including randomized control trials—to examine development interventions is just astonishing.

The delightfully quirky aspect of the success of the randomista movement is that it was, and remains, entirely faith-based. That is, the central claim of the movement was not just that more inputs (intellectual and monetary) into RCTs would write better (and more publishable) papers, which is, in a development logframe sense, just an output. The claim that attracted resources and support from development organizations and attention from the press was the claim that “rigorous” evidence from these RCTs could, should, and would produce better development projects and policies and, hence, ultimately better outcomes for human beings.

Behind this latter claim—that RCTs would improve development outcomes—there was a great deal of commitment by key actors, a great deal of passion, a great deal of the “animal madness of the fierce and young,” a great deal of wishing normative were positive (e.g., since people should rely on good evidence we hope they will rely on good evidence).

There was also a great narrative: “old development bureaucracies are protecting their organizational self-interest and hence are not using the rigorous techniques available to evaluate the outcomes from their projects and hence are wasting taxpayer money and not doing good for the poor, but independent academics could be tough about what really works.”

What the randomistas did have was good evidence and arguments about ongoing disciplinary debates about the difficulties of empirically identifying causal impacts of policies and programs with non-experimental data. But there was never any theory or evidence that a key, or even important, constraint on development practice was the lack of rigorous evidence about causal impacts, or that the production of such evidence would change practices. This was to be taken on faith.

The argument that RCTs would be more than a tiny component of the overall process of improving development outcomes seems, even now, 10 years in, at best not provable and at worst not very likely—as it is at odds with some basic known facts about development and about policy formulation and implementation.

First of all, the argument that RCTs had, until recently, been used sparingly, if at all, and yet are important in achieving good outcomes sits in kind of embarrassing counterpoint with the obvious fact that lots of countries have really good outcomes. That is whether one uses the Human Development Index or the OECD Better Life Index or any social indicator—from poverty to education to health to life satisfaction—there is a similar set of countries near the top. (In the HDI the top five are Norway, Australia, USA, Netherlands, and Germany. In the OECD Better Life Index they are Australia, Sweden, Canada, Norway, and Switzerland.) No one has ever made the arguments that these countries are developed and prosperous because they used rigorous evidence—much less RCTs—in formulating policy and programs. While one might have faith that RCTs can help along the path to development, RCTs didn’t help for those that are there now.

Second, at about the time (early 2000s) that the randomista movement, which often claimed to be about reducing poverty, was gaining steam, several countries were experiencing or had experienced the rapidest reductions in low-bar absolute poverty in the history of man. Indonesia, China, Vietnam had all seen dollar-a-day–like poverty measures fall from well over half of their population to under 20 percent in less than 30 years. In Vietnam the World Bank figures show dollar-a-day poverty falling from over 40 percent to 16.9 percent just between 2002 and 2008. If these countries did this completely without any use of RCTs (and they did), then from where exactly does the argument that using RCTs will accelerate poverty reduction come? Is there any evidence that countries that used more RCTs had more rapid poverty reduction than those that didn’t? No. While it might be the case that RCTs could accelerate poverty reduction this was (and is) a faith-based, not evidence-based, claim.

Third, there was the recent historical experience with RCTs in social policy in the USA. The randomistas were not proposing new methods or techniques but rather broader adoption into the field of development methods that already had a long history. There was a big fad toward the use of experiments in a variety of social policy domains in the USA in the 1970s. The Rand Health Insurance Experiment, carried out from 1971 to 1986, is still the largest health-policy experiment ever done. Income maintenance (negative income tax) experiments were carried out between 1968 and 1982 in four sites around the USA. A 1970 act of Congress authorized experiments such as the Housing Allowance Demand Experiment. The Kansas City Police Preventive Patrol experiment was carried out in 1972–73. This created ample capacity in the USA for doing social experiments inside organizations like Rand, Abt Associates, and Mathematica Policy Research, among others. At the founding of J-PAL there was 35 years of experience with RCTs in the USA on which to draw evidence of their efficacy (or not) as tools for improving policy and hence human outcomes. You would have thought that if the widespread use of RCTs had strongly impacted policy in the USA, this would have been put forward as evidence. Strangely, whether or not decades of social policy RCTs actually did have impact on policies and outcomes in the USA just kind of never came up in arguing that they would in developing countries.

The success of J-PAL and IPA (and other new organizations) in promoting the use of RCTs is just another example that development is nearly always a faith-based activity. People with the ideas, passion, and capability set about the make the world a better place in ways that are empirically plausible and are emotionally or otherwise appealing to them, and then these people use persuasion, often embedded in narratives, to attract resources and converts to their causes. Kudos are due to the randomistas for their faith and their passionate efforts which have garnered them so many resources to use in pursuit of their agenda. As Margaret Mead would ask: has anything else ever really changed the world?

If you are a development innovator should you do as they did and move ahead on faith and passion or do as they say and produce rigorous evidence?