Taking the Con Out of Econometrics

I love to quote the early-1980s talk by economist Edward Leamer, Let's Take the Con Out of Econometrics. Ack, I can't resist:

The econometric art as it is practiced at the computer terminal involves fitting many, perhaps thousands, of statistical models. One or several that the researcher finds pleasing are selected for reporting purposes. This search for a model is often well intentioned, but there can be no doubt that such a specification search invalidates the traditional theories of inference….[A]ll the concepts of traditional theory…utterly lose their meaning by the time an applied researcher pulls from the bramble of computer output the one thorn of a model he likes best, the one he chooses to portray as a rose.…This is a sad and decidedly unscientific state of affairs we find ourselves in. Hardly anyone takes data analyses seriously. Or perhaps more accurately, hardly anyone takes anyone else’s data analyses seriously. Like elaborately plumed birds who have long since lost the ability to procreate but not the desire, we preen and strut and display our t-values [which measure statistical significance].

Through a new column by Tim Harford, with whom I enjoyed cupcakes in London on Tuesday, I learned about a paper by Joshua Angrist and Jörn-Steffen Pischke celebrating what can be seen as a response to Leamer's manifesto. I say "can be seen as" because it is better than what Leamer envisioned, which was systematic sensitivity analysis. How is the con being taken out of econometrics? Through more credible research designs. Foremost among these is the randomized trial, already blogged here to exhaustion. Another is the "regression discontinuity design." Imagine if the Grameen Bank had mercilessly enforced the rule that people owning more than half an acre of land could not borrow. Presumably people with 0.499 acres would statistically resemble those owning 0.501, so any differences that arose between the two groups could have been attributed to differential access to microcredit. (Alas for science, Grameen was not so strict.)Tracing the history of this movement, Angrist and Pischke mention the landmark 1986 paper by Robert LaLonde. To a randomized experiment, he applied the sorts of fancy math usually reserved for non-randomized situations, in which adjudicating between competing causal stories is tricky business. (Did microcredit increase prosperity or did prosperous people borrow more?) Thanks to the randomization, LaLonde knew the right answers pretty precisely. The fancy math gave the wrong answers.Angrist and Pischke's first example of valuable, credible research in the new mode is the randomized evaluation of the Progresa program in Mexico, which demonstrated the benefits of giving cash to poor families conditional on them putting their kids in school (among other actions). The second is an ongoing U.S. government experiment that offered poor families the opportunity to move, and has revealed the complex effects of moving to better neighborhood. The program is run by Todd Richardson, whose 8-year-old son is in my basement debating with my boys over the finer points of the latest Legos.All this---well, except for the cupcakes and Legos---is directly relevant to the question of what we know about the impacts of microfinance on its users. In the comments on Meeting of the Minds?, I have debated with Grameen Foundation President and CEO Alex Counts about whether we should toss out 20 years of non-randomized quantitative research just because a couple randomized trials have been aired. I say we should, basically. But I do think Alex is on-target with this tough question: if David is right, how could the economics profession have gone so wrong for so long? You can read the comments for my answer. The point here is to cite Leamer's original and the new paper to bolster my case that the old research is fundamentally suspect and the new much better (though hardly perfect). The fancy math in what was once the leading study of microcredit's impacts is, though beautiful, typical of the old generation in its propensity to obscure rather than resolve the fundamental barriers to identifying cause and effect.Unfortunately, these credible research methods are harder to apply to macroeconomics because its hard to experiment on countries, and there are a lot fewer countries than people so sample sizes are small.I don't quite share Tim's pessimism that lay people will struggle to understand the new-wave studies, which I think should be more credible to experts and non-experts for the same reason: their simplicity. You understand how randomized drug tests work, right?Then again, a majority of Americans reject the theory of evolution, believing god created humans as we are.OK, maybe British lay people will believe the new studies.Update: Edward Leamer has replied to Angrist and Pischke with the flair for which he is known. (HT Bhagwan Chowdhry.)