First Randomized Trial of Microcredit

I have blogged about the power and limitations of randomized controlled trials (RCTs). Overall, I am a believer. I think the question researchers impertinently ask practitioners—can we show statistically that microfinance is helping?—is worth asking. And non-randomized methods have largely failed to answer it with credibility. So in my view it was for decades essentially correct to say that we have zero solid studies of whether microfinance makes clients better off on average.But it is no longer true that we have zero studies. Now we have (drum roll)...one.Seriously, the new working paper by Abhijit Banerjee, Esther Duflo, Rachel Glennerster, and Cynthia Kinnan, along with an imminent one by Dean Karlan and Jonathan Zinman, is historic. (And in fact so is the earlier Karlan and Zinman RCT of “cash loans” in South Africa, which are not usually thought of as microfinance.) But once one gets over the excitement and carefully reads the new paper, the realization sets in that it contributes only incrementally to knowledge. Each RCT tells us a bit about what happened to particular groups of people at particular places and times. Banerjee et al. ask a big question—does microcredit “work”?—but do not pretend to offer a big answer. The PSD blog’s pronouncement, “The verdict is in on microfinance...And it’s not pretty,” is too sweeping.The researchers from the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT and the Indian Centre for Micro Finance worked with Spandana to randomize the roll-out of its microcredit operations in Hyderabad, India’s fifth-largest city. Spandana chose 104 areas of the city to expand into eventually, rejecting some districts as having too many construction workers, who come and go and might take Spandana’s money with them. In 2006–-07 Spandana started lending in a randomly chosen 52 of the 104. Researchers followed up by surveying more than 6,000 households between August 2007 and April 2008, restricting their visits to families that seemed more likely to borrow: ones that had lived in the area at least three years and had at least one working-age woman. The surveyors made sure not to visit an area until Spandana had been there at least a year. They surveyed in “treatment” areas (ones where Spandana worked) and control ones (where it did not yet).The researchers did not compare borrowers to non-borrowers. Rather, because they randomized on areas, they compared all those surveyed in treatment areas, including many who did not borrow, with all those surveyed in control areas, including some who borrowed from other microcreditors. 27.0% of households in Spandana-served areas reported taking microcredit, two-thirds of them from Spandana. In control areas, 18.7% took microcredit. Results: The researchers found no impacts on these bottom-line indicators: total household spending/person; household businesses owned/person; whether women have say in household spending decisions; health spending/person; children’s major illnesses; school enrollment; and school spending.But with regard to running businesses, credit helped those who were most able to help themselves, and who were, it turns out, slightly richer than average. Households in Spandana areas were 1.7% more likely to have opened a business in the last year. As a fraction of those who actually borrowed, the number opening businesses might be closer to 5%. And households with more propensity to open their first business—as indicated by having more land, more working-age or literate women, and of course no business—were indeed more likely to do so if they were in a Spandana-served area. They also spent more on “durables” such as sewing machines and cut back on “temptation goods” such as snacks and cigarettes. Meanwhile, existing business owners increased profits.A few points of perspective:

• Benefits (or harm) might kick in after the single year studied. One hopes for long-term impacts. However, these may be hard to study because Spandana is expanding into the control areas. Five years from now, we will not be able to benchmark people who had ready access to Spandana microcredit for five years against people who had none. Rather, we will only be able to benchmark against people who had microcredit for four years. If anything, contrasts will wane.
• The modest difference between the 27.0% of people in Spandana areas who took microcredit and the 18.7% in non-Spandana areas also weakens the ability to detect impacts. If offering a service barely increases how many people use it, it’s pretty hard to find any impacts on outcomes such as household spending. On the other hand, the study had enough power to detect differences in business start-ups and profits...
• While RCTs are often called the gold standard for economic evidence, RCT-based “subgroup analyses” such as those comparing families that did or did not own businesses, did or did not seem likely to start them, are held in somewhat less esteem. This is especially the case when the questions the subgroup analyses answer may have been articulated only after the data were gathered. The fear is that the authors quietly tried breaking out the groups surveyed all sort of ways until they found a pattern unlikely to occur just by chance—like flipping a coin repeatedly until you get five heads in a row and then only reporting the improbable five heads. Angus Deaton has become an acerbic skeptic of RCTs, and writes:
  

  In drug trials, the FDA rules require that analytical plans be submitted prior to trial....In large-scale, expensive, trials, a zero or very small result is unlikely to be welcome, and there is likely to be overwhelming pressure to search for some subpopulation that gives a more palatable result.

  In the case at hand, the original project summary lists only these questions:

  Does microcredit lead to an expansion of existing businesses and increase the number of small businesses or wealth generating programs in poor communities? What is the impact of microcredit on the incomes, health, and education of poor families?

  ...which, read literally, get only a minority of the ink in the paper.That said, I am not so worried about data mining here because the authors report a set of results that fit a single, plausible story. It would be another matter if they found that microcredit makes children sick unless they were born on a Thursday. (For well-aimed criticism along this line, see Easterly on Collier.) Still the process by which they decided how to divide the sample seems unfortunately opaque.
• Some of the asterisks need asterisks on them. I believe the statistical significance of the subgroup analysis results is measured incorrectly. That is, their assessment of how easily pure luck could generate their results contains a technical flaw. I discussed the issue with Esther Duflo and it sounds like it will be fixed. Probably it will not materially affect the results. (To get technical: the standard errors are bootstrapped in a way that does not account for oversampling based on an endogenous variable, namely the decision to borrow from Spandana.)Update: They fixed it and it makes little difference.

A good way to put such a study in perspective is to state its findings tightly: In 2007-–08, one year after Spandana began operating in some areas of Hyderabad, among households that had lived in their area for at least three years and had at least one working-age woman, those in Spandana areas saw no changes in empowerment, health, education, and total spending, on average, that were so large as to defy attribution to pure chance, compared to those in areas Spandana would soon expand into (as distinct from areas Spandana avoided for having too many geographically transient workers).Got that? It is a mouthful of qualifiers. Each study affords a glimpse of reality. As more microfinance RCTs are done in other contexts, we will gain a richer sense of the impacts.