Microfinance Groups, Feeling Misunderstood, Misunderstand Research

In Meeting of the Minds? Researchers, Microfinance Leaders at CGD, I told you about a meeting I co-hosted at the Center for Global Development with Elisabeth Rhyne:

Personally, I see the meeting as responding to the rupture created by the release last year of randomized studies of the impact of microfinance. Putting the question of whether microfinance reduces poverty on average to such a rigorous test was healthy---and impertinent. The studies found no clear effect on bottom-line indicators of household welfare, and the press unsurprisingly accentuated the negative. "Perhaps microfinance isn’t such a big deal after all" and "it looks like ‘microlending’ doesn’t actually do much to fight poverty," ran real headlines. They posed a public relations challenge for microfinance groups, or at least for the network organizations based in the west. And I think they also forced some tough thinking within the groups about what we know about impacts and even how we should define success. The events also raised questions about whether researchers can write and talk about their work in ways that will reduce misleading coverage.

Yesterday, six of those network organizations, including four represented at the meeting, went public with another response to the public relations challenge from the new studies: a four-page document making the case that microfinance, while not a cure for poverty, is still a very good thing. Me not blogging it would be like the Podunk Daily News not reporting the election of a U.S. president, except that the Podunk Daily News is something I just made up.Seriously, substantial parts of this document elicit inward "Amen!"s from me. Here are significant and reasonable statements from the American microfinance community about the role of microfinance in economic development. The document emphasizes that microfinance is about more than microcredit. It lists seven ways that poor people use financial services, thereby making a clean break from the tradition of emphasizing just one, microcredit for microenterprise. And the closing shows a laudable humility:

We need to be reasonable and measured in our claims for what microfinance can accomplish. To thrive, microfinance requires a host of favorable conditions, including a sympathetic regulatory environment, political stability, mission-driven practitioners, cost-effective delivery, capital adequacy and---most of all---clients ready and able to leverage its services.Nor can microfinance be reduced to a sound bite. A host of critical and complex issues must be consistently considered and thoughtfully debated, including consumer protection, fair lending, transparency, acceptable profits, measureable impact, and capital-market investment, to name but a few. ‘Financial access’ and ‘building inclusive societies’ are more forthright terms to describe what microfinance delivers, without risking either simplification or exaggeration.Microfinance takes effort to understand, and patience to exercise. It is but one mechanism in the toolkit of global poverty alleviation---although one that has clearly demonstrated scale, sustainability, results, and enormous further potential.

My major concern is about the way the document handles the uncomfortable challenge from researchers, more fundamentally, the way it manages the tricky relationship between evidence and knowledge. I see that while minds did meet at our February meeting---and I do believe it was a constructive afternoon---there was no true meeting of the minds. This document does not really understand what good research is, and how good researchers must think in order to do it. (Perhaps if my work made me more of a practitioner, I would detect a similar lack of understanding in the other direction?) Though the document speaks of researchers politely, it subtly betrays an inability to understand, therefore respect, their professional competence. As I write in chapter 1:

Some practitioners who work daily with appreciative customers roll their eyes at the researchers who insist we cannot be sure if microfinance reduces poverty on average. But pursuit of empirical truth is what researchers are trained for, and just as they should hesitate to tell practitioners how to do their jobs, so should practitioners acknowledge researchers’ competence in measuring microfinance’s social return.

The document responds first to the challenge from academics by citing those academics to the effect that reporters have mischaracterized their work. Fair enough.Next the document frames the new studies:

...these studies, conducted over a short timeframe and with small sample sizes, are part of a much larger body of research conducted over the last 20 years that have explored how microfinance affects the lives and well-being of clients.

Implied here: These aren't the only impact studies of microfinance; the older ones reflected better on microfinance and deserve weight. Unacknowledged is that the new studies are qualitatively more credible, just like the ones that upended previous literature and cut the U.S. breast cancer rate. "Short timeframe" is fair since the new studies look at impacts over just 15--18 months. But "small sample sizes" is right only if interpreted liberally. The studies' samples, covering thousands of people, are not too small to support their conclusions about the short-term impacts of individual lending in Manila and group lending in Hyderabad, India. But certainly the samples are not representative of all poor people in the world, so we don't know about impacts in other contexts. Perhaps that is what is meant by "small sample sizes."Then come the stories---one from each of the six signatories. Going by word count, this is the groups' biggest counter-volley. "Numerous first-hand accounts of how microfinance can improve lives are available on our websites." Indeed. Considering the number of clients these six groups represent, this may be the most filtered, unrepresentative collection of microfinance stories ever. (And surely it is a "small sample" next to the Hyderabad study's 6,800 subjects?) For counterpoint, here are a couple of stories of the kind you won't find on those web sites.From BusinessWeek:

Hernández, 29, the daughter of a small farmer, says her mother purchased several knitting machines in 1992, but lacked cash for yarn. The equipment remained idle for years. A loan, Hernández thought, would enable her to buy nylon and more machines. She aimed to lure home her brother and two sisters, who she says are undocumented workers with restaurant and hotel jobs in the U.S. Over four years, beginning in 2001, she, her mother, and a sister took out a series of loans ranging from $200 to $1,800, at an [annualized interest rate] of 105%. They rolled one into the next and used the money to increase their weekly output from 800 dozen pairs of socks to 1,500 dozen. At their peak, they say they brought in $800 a week, more than enough to sustain an extended family of six. Then things unraveled. Wholesale customers fell behind on payments. Compartamos’ steep inter-est rates took an unremitting toll, as Hernández and her relatives each missed several $130 payments to the lender. That was a lot for the rest of the 23-member borrowing circle to make up. Resentment sur-faced. Soon after Compartamos trumpeted her story in 2005, Hernández and her family were banished from the group.Lacking capital, she has seen her production and earnings plunge to 500 dozen pairs of socks and $270 a week. Her siblings remain north of the border. Stoic about her tarnished accomplishments, she is uncertain about the future. “It’s been a huge effort,” she says, “and we’re barely afloat now.”

From Lamia Karim, "Demystifying Micro-Credit: The Grameen Bank, NGOs, and Neoliberalism," Cultural Dynamics 20(5): 5--29 (2008), copying from an earlier post:

I saw that credit-related strife amongst members and their families were routine occurrences. Women would march off together to scold the defaulting woman, shame her or her husband in a public place, and when she could not pay the full amount of the installment, go through her possessions and take away whatever they could sell off to recover the defaulted sum. In circumstances when the woman failed to pay the sum, which happened several times a month in the NGOs I studied, the group members would repossess the capital that the woman had built with her loans. This ranged from taking away her gold nose-ring (a symbol of marital status for rural women, and removing it symbolically marks the 'divorcing/widowing' of a woman) to cows and chicks to trees that had been planted to be sold as timber to collecting rice and grains that the family had accumulated as food, very often leaving the family with no food whatsoever. The women who committed these acts did so at the exhortations of NGO officers, but they also considered these acts to be 'protecting their investments', and the defaulting woman as someone who had 'broken faith with the community'. These acts were committed with the full knowledge of NGO officers, but the officers did not participate in these collective acts of aggression. Instead, they threatened to withhold future loans unless the defaulted money was recovered.In instances where everything had been repossessed because of a large default, members would sell off the defaulting member's house. This is known as house-breaking (ghar bhanga) and has a long history in rural society.

Of the recitation of success stories in the document, I see two possible readings. One is as evidence that microfinance does work. But by the same logic, my stories prove that microfinance doesn't work. The other reading is as evidence that microfinance can work. But we can deduce that from common sense: surely microcredit in particular has made some people better off and some worse off. Credit always does that. I don't think that microfinance groups can rest their case at proving opportunities if the opportunities sometimes harm. The question stands: what is the overall pattern of impacts?Think about how you might answer that question with stories. How would you go about collecting the stories by which to judge? Probably you'd start to think about how to build a representative sample; how big you could practically make the sample; and how to distill the core elements of each story into a few common terms in order to allow summary statements, such as the percentage that are success stories. This kind of thinking---not fancy statistical arguments---is what these new microfinance impact studies are really about. It's hard to see how to avoid it if you want to understand the impacts of microfinance. Yet this document does avoid it. Telling a few success stories not only hides that question of overall impacts; it denies the importance of generalizing carefully from instances. It denies, in other words, the value of serious research. The signatories tack a final contentious caveat onto the academic research: being quantitative, it cannot "capture and analyze all the benefits of microfinance" (emphasis in original). Yes, life cannot be reduced to numbers. But social scientists have ways of measuring many things: happiness, trust in neighbors, role in household decision-making, income, spending. The methods are highly imperfect, but more reliable than story-filtering. Surely most defining aspects of poverty---earnings, type of roof, whether kids are in school---can be reasonably represented in numbers. Check out the Grameen Foundation's Progress out of Poverty Index, whose web page contains no apology for quantifying.Quantitative research seeks statistical power and objectivity by abstracting from particulars and studying many instances. That is both its strength and its weakness. It actually can measure benefits---outcomes---pretty well. Where it struggles---and where the disciplined story collecting we call qualitative research can shine---is in elucidating causal chains, the subtle and unguessed pathways along which life moves. That said, qualitative research like that of Lamia Karim quoted above can help us understand outcomes too. Helen Todd's book is a great example; but even it does some serious quantitative analysis (on a small sample). I strongly doubt that the drafters of this document have access to insight about the impacts of microfinance on clients that is beyond quantification.Having in a sense judged the statement on my terms, by scrutinizing its fidelity to the spirit of genuine inquiry, I should also try to judge it on its own terms. Will it serve the apparent goal of changing how reporters, donors, and the general public perceive microfinance? It is hard to know. More randomized studies are coming. If the next one also shows average poverty impacts statistically indistinguishable from zero, then maybe this document and message-honing exercise will prevent simplistic headlines about the "failure of microfinance." The document contains good statements about the way microfinance enhances freedom and builds industries. Or maybe no one will pay attention to the third randomized study of microcredit anyway.Arguably, what bothers me about the document will undermine the drafters' goals. The American microfinance industry has not shown in this document that it has learned one lesson from last year's publicity black eye: the danger of getting ahead of the academic evidence. While calling for more evaluations, the industry seems uncomfortable with the ones just completed, grudgingly dwelling on their limitations rather than what they teach, seeking to marginalize them with 20 years of inferior studies and cherry-picked stories. It seems to me that to the extent the industry substitutes gut beliefs ("truthiness") for critical thinking, it risks further ugly surprises. These could take the form of more bad publicity or real people getting hurt by programs implemented and funded too much on faith.In a post last month on CGD's main blog I gave the last word to Stephen Colbert. I'll do it again. Turns out he put "truthiness" into currency on the very first Colbert Report. In doing so, he made a serious point about the dangers of anti-intellectualism in policymaking:

The Colbert Report	Mon - Thurs 11:30pm / 10:30c
The Word - Truthiness
www.colbertnation.com
Colbert Report Full Episodes Political Humor Health Care reform