Is Obama's Budget Director an Epistemological Nihilist?

The Institute of Medicine's Roundtable on Evidence-Based Medicine has the weighty job of strategizing about how the American health care behemoth should learn---that is, how doctors, scientists, and other players should generate knowledge about what works and how they should incorporate it into practice. This graph is from the Roundtable's 2008 annual report:Time runs from left to right. So the graph says that before a drug or device is introduced into use it should be tested with clinical trials, preferably randomized ones (RCTs). (Notice how the dark gray "experimental studies" area is the tallest segment at the far left). Once the intervention is approved, data on how patients are doing should be collected into national databases or "registries." (That's the light gray "recorded clinical outcomes" area that grows to the right.) Researchers with computers can mine the data in real time to detect patterns of concern, such as higher rates of heart attack among users, then inform doctors quickly to facilitate continuous, national learning. My wife Mai explains that "efficacy" = "works in clinical trials" and "effectiveness" = "works in the real world."In the Chronicle of Philanthropy, Lisbeth Schorr cites this report as showing that even in medicine, conventional wisdom is stripping randomized trials of their gold standard status. (See also her Education Week article, this chart, and this working paper. Hat tip to Paul Isenman.) Schorr appears to be reacting against a new U.S. government policy, articulated by Office of Management and Budget director Peter Orszag on a White House blog, to factor rigorous evaluation into social program spending decisions. Orszag:

…we're using a…two-tiered approach. First, we're providing more money to programs that generate results backed up by strong evidence. That's the top tier. Then, for an additional group of programs, with some supportive evidence but not as much, we've said: Let's try those too, but rigorously evaluate them and see whether they work.

Orszag cites the Coalition for Evidence-Based Policy, which cites the National Academy of Sciences, which says that "rigorous" pretty much means "randomized." (Ironically, before Obama tapped him, Orszag contributed to the IOM report Schorr invokes against him.)Schorr retorts:

The definition most aggressively promoted today holds that approaches to solving social problems should be considered evidence-based only when they have been found effective by research methods involving random assignment of participants to experimental and control groups. This narrow definition is an understandable reaction to the era of letting a thousand flowers bloom and allowing good intentions to substitute for sound reasoning about how activities will lead to results. But it is an overreaction.We have reached the point that the late MIT organizational theorist Donald Schon described as "epistemological nihilism in public affairs," the view that nothing can be known because the certainty we demand is unattainable. And we have done so at a time when richer, more inclusive ways of determining what works are available.…Unless we stop ranking possible solutions to problems by their evaluation methodology and find ways to judge how well they accomplish important goals, we will be left with a seriously impoverished tool kit. If government agencies and private grant makers, afraid of being considered not rigorous, unscientific, or wasteful, choose to support only those efforts that meet the randomized-trial test, we will be robbed of:

• Good programs that do not lend themselves to random-assignment evaluations.
• Reforms that are deeper and wider than individual programs.
• Innovations of all kinds.

One weakness of RCTs is that they miss synergies:

The Obama administration is about to embark on an exciting new Promise Neighborhoods program, inspired by the Harlem Children's Zone.Applicants for planning support will have no chance to meet this federal effort's ambitious goals if they rely only on programs that fit the narrow definition of "evidence-based."Harlem Children's Zone itself has been described as an endeavor that "meshes educational and social services into an interlocking web, and then it drops that web over an entire neighborhood."We won't find interlocking webs or web drops in the directories of evidence-based programs, now or ever. Nor is the problem solved by evaluating the impact of each discrete program, because the entire point of efforts like the Harlem Children's Zone is that we expect the whole to have a far greater impact than the sum of its parts.

Schorr advocates a "results framework." I am sure this is a serious idea but I don't quite understand it from her short article: hasn't every social program ever funded avowedly focused on results?I doubt that any of the randomistas would question Schorr's call for methodological pluralism. But I can certainly see the need to guard against a government agency, hamstrung by a well-intended edict from above, rigidly pursuing RCT purism. Whether the OMB is such a juggernaut, I do not know.To return to the graph, it raises two questions in my mind: 1) Will this continuous learning paradigm, which deemphasizes RCTs over time, work for medicine? And 2) will it work for social policy, in particular microfinance?I'm not a health care policy professional and don't play one on this blog so I am not qualified to answer question 1. But Mai is one and the case she pointed out in March shows the complexity of the question. A randomized trial of a well-established treatment, hormone replacement therapy for post-menopausal women, exposed serious dangers; by reducing HRT use it may now be preventing thousands of breast cancers a year. No study of "recorded clinical outcomes" picked up the problem.The IOM experts must know that example and have a thoughtful reply. So grant for the sake of argument that national registries of outcomes data can supplant RCTs once interventions become common. Would that work for social policy too? This is in part a question about whether the human body is an apt metaphor for human society when it comes to studying causality. If we learn that microcredit borrowers are earning higher profits, can we be as confident that the noted intervention is causing the noted outcome as we can if we learn that Americans who take aspirin have fewer heart attacks? Maybe not. It seems easier to imagine reverse causation for microfinance (those with better business prospects anyway borrowing more) than medicine (the heart attack-prone deliberately taking less aspirin?). Anyway, even the IOM calls for predominant reliance on RCTs for incipient interventions. So I'm not sure how far Schorr's invocation of the IOM gets her.More than almost anyone in the grand conversation about RCTs, I have replicated and scrutinized, with a mathematician's eye, noted non-experimental studies of the effects of social policy. It has been my peculiar obsession. Based on my admittedly small sample (a ~dozen studies of aggregate foreign aid and microfinance), I have concluded with especial conviction that non-randomized quantitative studies are usually useless for studying the effects of social policy. This conclusion is based not merely on my discovery (which is no discovery at all) of the inherent inability of non-randomized methods to adjudicate between competing causal stories. I am not merely insinuating. Rather, I have repeatedly found demonstrable mathematical flaws, most of which are hard to explain in a non-technical blog. This is why I believe that it would be hopeless to rely on "recorded clinical outcomes" to evaluate microfinance. Perhaps if microfinance had gone through the whole process shown in the graph---being carefully, randomly tested before the global rollout---the story would be different.Does that make me an "epistemological nihilist"? I don't think so. I say we should first figure out what we know, then act as best we can on that knowledge. We must use wise theories to extrapolate from the inevitably fragmentary data available to us. For example, in my own efforts to understand the impacts of microfinance, I have leaned heavily on non-randomized qualitative studies: you have to take seriously someone who lived among clients for a year and demonstrates acuity on paper. And I am using the big-picture thinking about development of Amartya Sen and Joseph Schumpeter. Meanwhile, those with the means and motive should invest in learning more: with one RCT of microcredit for the truly poor in 30 years, microfinance is hardly over-RCT'd.