The Other Shoe Drops: 2nd Randomized Microcredit Study

In February I enjoyed a preview by CGD non-resident fellow Dean Karlan of the results of his randomized experiment with microcredit in Manila. (Dean has other barely noteworthy affiliations.) Dean and coauthor Jonathan Zinman applied a randomization method they first used to good effect in South Africa. They persuaded the First Macro Bank in Manila to adopt a credit scoring system---a computer program that automatically approves or rejects loan applicants based on factors such as age, wealth, and years in business---and to tweak the program to randomly "unreject" some borderline applications. Karlan and Zinman posted the results of their experiment in the last few days.This is the second randomized controlled trial (RCT) of what is usually considered microcredit, the other being the J-PAL Hyderabad paper that came out in May. The lender in Karlan and Zinman's earlier South Africa study made four-month loans mostly to people who could show pay stubs---what is more commonly labeled consumer credit than microcredit. Then there's the new experiment with microsavings conducted by Pascaline Dupas and Jonathan Robinson. All in all, 2009 is a pivotal year in the study of microfinance's impacts.So...did microcredit "work" in Manila? Mostly not, as far as the evidence goes. Rossi's Stainless Steel Law has held: the better the study, the weaker the effect found.But before coming to how lives were affected in this experiment, it's hugely important to understand whose lives were affected. To merit consideration for a loan, thus inclusion in the study, applicants had to be "18-60 years old; in business for at least one year; in residence for at least one year if owner or at least three years if renter; and [have a] daily income of at least 750 pesos [$15]." Since the subjects were already in business (most commonly, running corner stores), the study could not check whether credit helps people become entrepreneurs. Also, the income floor put the study population well above the Philippine average. This excerpt from Table 1 shows that an impressive 25+37=62% of borderline applicants whom the computer allowed an 80% chance of approval, a group that dominates the sample, had some college education. The 80%-ers' education stats best those of the United States! Not surprisingly, the Manila and Philippines averages are lower, at 32% and 18%. Meanwhile, average income for the 80%-ers is 65,979 pesos/month, or $15,835/year. CGD's in-house Manilan Paolo Abarcar says this income is equivalent to having two gainfully employed college graduates in the household. [See update.]That K&Z stray so far from the $1--2/day poor we usually imagine as the target for microcredit is not a flaw, but it generates a caution for interpretation. And it reflects a limitation of randomized credit scoring: scoring entails too much time-consuming data collection to be economical when making the littlest loans to the poorest people. This is why J-PAL reached a poorer demographic than K&Z have in either of their impact studies. Karlan and Zinman do however limit their sample to borrowers of less than 50,000 pesos, $1,000, which falls within usual definitions of individual microcredit.After setting their RCT top to spinning, K&Z dispatched surveyors to the homes of accepted and rejected applicants alike. Loan applications did not all at happen at the same time, nor did survey visits, so the space between the two events ranged between 11 and 22 months. The survey teams ultimately tracked down about 70% of those rejected and those accepted. They asked many questions, observed the quality of borrowers' houses, and even administered psychological tests.The heart of the paper is a large set of tables that examine whether the computer's approve recommendations affected people's lives on average. Some 55 variables are checked, from total borrowing from informal sources, to whether someone in the household is working overseas, to self-reported trust in acquaintances. For each variable, K&Z check for a treatment-control difference in the full sample, within the female and male subsamples, and within the above- and below-income-median (50th percentile) samples. So that's 5×55=275 statistical tests. (I'm leaving out a few variables almost automatically affected by winning a loan, such as total borrowing from formal sources.) (Interestingly, borrowers lied to the surveyors almost half the time about whether they had a loan---another cautionary tale about survey data.)Now remember how econometrics works: for each test, an average difference is computed, such as that computer-blessed male applicants were 18.5% less likely to have health insurance than rejected men; then an estimate is made of the probability that you could get a result that big in your sample by pure chance if the true average effect of credit on having health insurance is actually nil. If that probability is low, say under 5%, then the finding of less health insurance is "significant at the 5% level." But sometimes the improbable happens, so some "significant" results are flukes. Implication: if you perform 275 tests and in fact the loans have no effect on anything, you would still expect 10% (27.5) of your results to be significant at 10%, of which 5% of the 275 (13.75) would be significant at 5%, of which 1% of the 275 (2.75) would be significant at 1%. Against 27.5, 13.75, and 2.75 are my tallies for the actual K&Z results: 37, 17, and 5.What is remarkable about this study is how little seems to be affected by credit. This is so much the case that a majority of the "significant" results may be flukes. If you include in my tallies the variables almost automatically affected by getting a loan, which do show up with statistical significance, the picture looks a bit better. But I'd argue it's appropriate to look at such proximate consequences separately. Another reason to discount: the small male sample (about 165) may be more easily distorted by a few outliers, which may explain why it generates more significant results. K&Z do drop outliers at times, showing the issue to be real.Clever econometricians have developed adjustments for interpreting such multiple hypothesis tests, as discussant David McKenzie pointed out at the February seminar. I don't know enough about the techniques to endorse them. In their absence, you need to make your own mental adjustments: don't take every significant result literally, but look for consistent and plausible patterns. This K&Z largely do, though I would discount the suggestion that borrowing men invested more in the education of their children (table 6), and perhaps the one that male borrowers increased profits.I am persuaded that access to First Macro Bank's loans on average led households to:

• increase total formal borrowing;
• cut back on employees, house renovation, and perhaps health insurance (I can imagine belt-tightening at the time of a big new loan-enabled investment, especially if such investments are lumpy);
• trust more in one's neighbors (after experiencing that others are there to help);
• more easily borrow from family and friends (having won a bank's seal of approval).

Also, I am persuaded that the effects tended to be stronger for higher-income households.I should emphasize the difference between statistical and scientific significance: statistically insignificant results, such as the lack of effects on household earnings, spending, and nutrition are scientifically significant. Thus we can tentatively conclude that among Manila's middle class, individual microcredit doesn't seem to affect material well-being 1--2 years out.In contrast, K&Z found strong benefits in South Africa. In February, I asked Dean why. He offered this story: in South Africa, many people have to pay for on-the-job-training a few months after starting work. This they often do by walking into a "cash loan" store with their initial pay stubs. New hires who cannot get such loans lose their jobs, and become poorer and hungrier. The key is that credit is enmeshed in a system of employment. The story is fascinating in itself. (I think it's not in the paper.) It also illustrates how the idiosyncrasies of local circumstances mediate the impacts of finance. As more solid studies like these appear, we'll come to better understand the richness of finance's effects on people's lives.