New Impact Studies of Indian Self-help Groups

Manuel Bueno's post on the NextBillion blog alerted me to two new World Bank evaluations of self-help groups (SHGs) in Andhra Pradesh, India. Both are by Klaus Deininger and Yanyan Liu.SHGs can be seen as the Indian government's response to that government-bypassing brand of microcredit from the upstart nation of Bangladesh. A non-governmental group (NGO) organizes 10--20 women into a group. They begin to save, putting their pooled deposits in a bank, thus "linking" to the bank. Once they have saved enough, they can also borrow from the bank as a group, then distribute the credit among themselves. Meanwhile, government regulations push the banks to meet targets for numbers of SHGs linked each year. The recruiting NGO often works with the women on things beyond finance, trying to build their self-esteem as women and their capacity as a group to make change in their communities---in a word, to empower them. (See this.) The Bangladeshi model is inferior, the minister for SHGs of the Indian state of West Bengal helpfully explained, because “the government does not support Yunus.”The World Bank--backed program in Andhra Pradesh state appears to layer new functions on the SHGs in keeping with the Community-Driven Development trend/fad. The government provides grants that the SHGs decide how to invest. Subsidized rice is lent in-kind. I don't know all the details. The bottom line from the point of view of evaluating microfinance is that the variety practiced here includes several complex and subsidized elements that go beyond credit and savings. By and large, the package is evaluated as a whole.Deininger and Liu write:

As a phased random roll-out originally envisaged...was not implemented, our identification strategy has to rely on weaker assumptions.

Hidden in that drab, technical sentence, I suspect, is a tale of woe that goes something like this: The World Bank researchers wanted Andhra Pradesh to randomize which parts of the state got the program first, because they knew that randomized trials are more reliable for studying impacts. (So above I think they mean stronger assumptions, making for a weaker study.) But randomizing went against the grain of the government because of moral qualms (about not prioritizing the poorest) or politics (influential politicians from certain districts wanted the program first). The World Bank researchers hoped for cooperation since the Bank pumped a quarter-billion dollars into the program. But the Indian officials bridled at the naive meddling of these foreign researchers. And at the end of the day, the Bank needed India more than India needed the Bank.Or something like that.Deininger and Liu thoughtfully make the most of their unrandomized situation. They use propensity score (PS) matching. The idea is something like this. Suppose high-definition TV is introduced sooner in some parts of the United States than others in a non-random way---the coasts before the heartland. You want to know how the introduction affects television viewing habits. So you survey people who have it, in L.A., and people who don't, in Podunk, and compare their viewing patterns. Of course, people differ in many ways between the two places, so you can't plausibly attribute any differences you find in TV viewing just to staggered HDTV introduction. Propensity score matching tries to make your samples more comparable by matching, say, a family with four people, one dog, and two cars and income of $70,000/year in L.A. with one with the same stats in Podunk---and doing the same for every other household in the study. The hope is that if given families in L.A. and Podunk look the same going by number of cars, etc., then they are the same in all respects that matter for the study. Except of course that the L.A. family can watch HDTV. So if the L.A. family does watch more TV than its Podunk match, that's because of HDTV.This exemplifies the logical structure of causal studies that I've written about. You assume something to conclude something. As Deininger and Liu make clear, to believe their findings you must assume that people who got the SHG program early and and people who got it late---and who look the same by the numbers---are in fact the same in all respects that matter for the study even though they live in districts that differ systematically. (Poorer, more remote districts got the program first.) This assumption is plausible but untestable. And as I wrote, it's worth pondering whether such a key assumption is easier to believe than the conclusions it makes possible, such as that distributing subsidized rice improves nutrition. Evaluations are useful when the assumptions on which they rest are easier to believe that the assumptions they test.A famous 1986 study applied propensity score matching non-experimental methods to data from a randomized experiment, for which no such cleverness was actually needed. The methods gave the wrong answers, underestimating some impacts and overestimating others. On the other hand, a 2005 revisit found that PS matching did better when applied to "differences in differences," which is what Deininger and Liu do. (But see here and here for generally cautious conclusions about the efficacy on non-experimental methods, based on literature reviews.)In their first study, focusing on impacts within a couple of years, Deininger and Liu find strong increases in empowerment. That resonates with my review of the qualitative evidence. They also find improvements in nutrition and spending, but not income or assets.The second study looks at impacts after 2.5--3 years. It finds gains in nutrition, especially among the poorest people, in non-financial assets such as goats, and perhaps in overall spending. It does not look at empowerment.Again, the work is thoughtfully done. But the lack of randomization and the prominence of non-financial aspects in the SHG program mean that if you're looking for proof that microcredit cuts poverty, you'll need to keep looking.