"Impact Stories" from Freedom from Hunger

As I mentioned before, I appeared on a panel at the Microfinance USA conference in New York a couple of weeks ago, alongside Jonathan Morduch, Dean Karlan, and Chris Dunford, who is the president of Freedom from Hunger. Chris spoke last, as the practitioner who could bring our discussion back to the real world (in case it left). Freedom from Hunger is distinguished among U.S.-based microfinance groups by its "credit with education" method, which Chris helped develop, and which bundles credit with basic education in accounting or nutrition. It is also distinguished by its commitment to rigorous evaluation. Chris said, and Dean confirmed, that FfH has done more randomized trials than any other microfinance group.Chris closed his talk by unveiling a new publication:

Now I’ll leave you with an intriguing question. Can we find less rigorous, expensive and time-consuming alternatives to RCTs that nonetheless overcome most of the biases and limitations of our human cognition but also allow us to be more intuitive or holistic in our understanding of microfinance? Rather than seek alternatives to RCTs, we would do better to find methods that can complement and enrich what we learn from the few RCTs we can reasonably expect to do. Toward this end, my organization, Freedom from Hunger, is engaged in a long-term project to develop and test a method we call “impact stories”---open-ended story-collecting from randomly selected microfinance clients, not unlike the financial diaries that informed Portfolios of the Poor, but broader in scope. We’ve just posted...the first interim report---called Human Faces of Microfinance Impact: What We Can Learn from Freedom from Hunger’s Impact Stories Methodology.

(You can watch the panel here. The quote above comes at 49:00. And read this great post about the panel by Tom Murphy at A View from the Cave.)I think the report is a fine piece of work. It summarizes results from 274 questionnaire-driven interviews with randomly selected clients of FfH-affiliated programs in Benin, Bolivia, Burkina Faso, Ecuador, India, Mali, Mexico, and Peru. (The questionnaire is an appendix.) It conveys vivid impressions of their lives...well they are glimpses, but vivid compared to a table of regression coefficients. There are Beatriz in Ecuador and Kotine and Mali, who say their lives have improved somewhat since they began with microfinance. Beatriz participated in Credit with Education while Kotine did something called Saving for Change (see post on VSLAs). And there are Maya Luisa in Mexico and Julie in Benin who say---much less typically, according to the report---that their lives have gotten worse. The overall impression I get from the stories is that in reflecting on the quality of their lives what these very poor microfinance clients worry about most is keeping themselves and their families fed and healthy; and that it is unrealistic to expect microfinance to bring radical improvements on these fronts. The report works hard not to over-interpret its evidence. It recognizes the strengths of randomized studies---ability to rigorously check impacts---and the strength of such qualitative work---ability to generate stories with which the human mind can understand the world:

“Knowing” starts with the stories of individual, recognizable people, especially when we have heard them tell the stories themselves. Human beings respond to stories about human beings; this is part of our DNA. Statistics from large numbers of people tell us how common or rare are the stories we have heard, but it is the story of a flesh-and-blood person with a name, like Beatriz, that brings the statistics to life. We “know” much more deeply that microfinance has helped Beatriz if we have held her hand as she shared with us how her family has experienced much hunger, but for the first time in her life, she has savings that she uses to improve her business and feed her family. Human life stories help us “know” what that impact looks and feels like.But is her microfinance participation what causes the positive change in Beatriz’s life? What would have happened if Beatriz had not been a microfinance client? While she was saving and investing in her business, were there other events in her life that were the real cause of her business improving and her family eating better? We cannot truly attribute positive (or negative) change to microfinance unless we have studied large numbers of people like Beatriz in “randomized control trials.”

The report is short and clear, so there is not much point in my trying to distill it further. Just read it.I do have a few quibbles. It's not clear to me how this work is a "breakthrough." Qualitative research from random samples is not new... But it is not done enough in microfinance, or I wouldn't be writing this post. Perhaps the breakthrough is the novelty of integrating such research into program operation.Another concern is that a reader could miss importance of the distinction between random sampling (which the study does) and random treatment (which it doesn't). If you sort a bunch of people in a room by sex and then sample on the side where the women are in order to estimate the gender composition of the entire room, it doesn't matter how randomly you sample that subgroup---you'll still get the wrong answer of 100% female. Just so, when the offer of microfinance is not randomized, no matter how disciplined and rigorous your collection and analysis of data, what is causing what will be uncertain. Nothing in the report contradicts this point. But the frequent use of "random" could obscure it. So could the use of the word "impact" in the term "impact stories," since, as the report says, we cannot be sure that the improvements found are actually impacts of microfinance.Finally, one rationale given for this qualitative work is that the alternative, randomized experimentation, is expensive. So I wonder how much this study, which is part of ongoing research, is costing. I would guess the costs for both types are in six figures.In addition to the way it captures the richness of people's lives, a strength of this work, relative to the randomized-trial studies released so far, is its ability to follow people over longer periods. So far, the randomized studies have followed up with subjects after a typical 18 months of access to microcredit. One criticism, especially from promoters of microfinance, is that 18 months is too short a time for the full benefits of microcredit to appear. I hope that as Freedom from Hunger's research continues, it will give insights into the potential trajectories of impact over time.