March 31, 2011
[Note: A full response to Mark Pitt is now available.]On Saturday night, Jonathan Morduch and I learned second-hand that Brown University economist Mark Pitt had circulated a paper via blast e-mail that challenges our replication of Pitt and Khandker, which was for a decade the leading study of the impact of microcredit on poverty. Here's the abstract of the new paper:
This response to Roodman and Morduch seeks to correct the substantial damage that their claims have caused to the reputation of microfinance as a means of alleviating poverty by providing a detailed explanation of why their replication of Pitt and Khandker (1998) is incorrect. Using the dataset constructed by Pitt and Khandker, as well as the data set Roodman and Morduch constructed themselves, the Pitt and Khandker results standup extremely well, indeed are strengthened, when estimated with Roodman’s cmp program, after correcting for the Roodman and Morduch errors.History has repeated itself. Back in 1999, Pitt wrote a similar response to Jonathan's original attempt to understand Pitt and Khandker.The timing is terrible for me as I am in the midst of a big push to finalize the book. So here is a quick response, pending the investment of time needed for a more thorough reply. Take these thoughts as preliminary.Pitt's response has exposed an important mistake in our work. With his fixes, we now match their key results extremely well. This is good news. But in truth we did not dwell on the sign mismatch---Pitt and Khandker found a positive association between borrowing and household spending where we found a negative one---but on whether cause and effect had been proved. In announcing the study, for instance, I wrote, "Seemingly, lending to women makes families poorer…but I just told you how much credence we put on such claims about cause and effect." And our statical analysis on the causality claims has not changed (though we have more to do).As a result, Pitt's response:
- Validates our open approach to research, in which we, in contrast to Pitt and Khandker, have freely shared our data and code.
- Strengthens our main conclusion, which is about lack of proof of causality.
A couple of years ago I spent time scrutinizing what was then the leading academic study finding that microcredit reduces poverty. To decide whether I believed this crucial study, I replicated it—rerunning all the original math on the original data. The math and computer programming were really complex. In time, with my coauthor Jonathan Morduch, I would conclude that the study does not stack up. We’re not saying microcredit doesn’t help people, just that you cannot judge the matter with this data.......we have little solid evidence that microcredit, the dominant form of microfinance, reduces poverty.This is not about our inability to match the sign. It is about whether we can interpret the sign as an effect of borrowing on poverty. (It could be an effect of poverty on borrowing, with better-off people capable of borrowing more.) This is the standard problem with inferring causality from non-experimental data, and is one reason that the randomized approach has caught on. More to the point, when we fix our regressions, they continue to fail tests of the assumptions needed to infer causality. So improving the match to the original greatly strengthens our conclusion that this study does not convincingly demonstrate an impact of microcredit on poverty. (Non-econometricians can read Roodman's Law of Instrumental Value to understand the underlying issues.)Now that we are able to replicate the original much better, Jonathan and I will spend some time studying it. (And you can too: just add the nontar dummy to the second stage.) For now, here are a couple of preliminary tables (updated April 4) . In the first, under "WESML-LIML-FE" you'll see that the "PK" and "new" columns now match well. That's Pitt's main point. But at the bottom of the last column is the result of a statistical test, a "0.022." (For experts, this is a Hansen test on a parallel 2SLS regression, as explained in our paper.) That says that if the assumptions Pitt and Khandker make in order to infer causality from correlations are correct, then there is only a 2.2% chance that a certain test statistic would be as large as it actually is. And this updates Table 4 of our working paper. This shows the parallel 2SLS regressions still failing Sargan/Hansen tests, and (in the right half) excluded instruments having clear explanatory power when included:One of the questions we will investigate in coming weeks is why the 2SLS regressions produce such weak results compared to the LIML ones. In theory 2SLS is less efficient, but has the advantage of robustness to heteroskedasticity. (The LIML regressions model the Tobit censoring of credit and so require homoskedasticity.)More technical notes: The regressions reported above incorporate two other changes from Pitt. One is his distinction between what I call the censoring threshold of log(1000) and the censoring value of log(1). cmp can actually handle this fine. The other is that in the instrumenting stage, observations for all three survey rounds now use the first round's data. In addition to homoskedasticity, the Pitt-Khandker LIML regressions assume no cross-household error correlation; thus the only deviation from sphericity that they allow is serial correlation. This makes errors i.i.d. within survey rounds and makes the Sargan test valid for 2SLS regressions restricted to individual rounds. The Hansen test is required for regression that pool all three rounds. [Update: this is not quite right. The regressions are weighted for sampling---"pweights" in Stata parlance---which effectively introduces heteroskedasticity and invalidates the Sargan test even in cross-sections. But Hansen, now shown for all columns, remains valid and corroborates Sargan.]Code and data are here.
Disclaimer
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.