If Medical Science Is Biased, What Does It Mean for Development…?

In “Lies, Damned Lies and Medical Science,” David H. Freedman profiles the work of Dr. John Ioannidis who studies the quality of medical research and finds it to be full of findings that range from misleading and overhyped to completely false. As concerned patients, this raises the big question of whether medical professionals can be trusted to give us good advice when the evidence base is so flawed. But  the dynamic that Ioannidis reveals in medical research is also a clear problem for anyone working in health policy research.For me, the most troubling issue revealed by Ioannidis’ research occurs when he moves from identifying the specific sources of research errors to asking questions about why such errors are so widespread and systemic. He concludes that the entire research process is full of bias:

“At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

This reminded me of another article by De Long and Lang (“Are All Economic Hypotheses False?” JPE 1992) which analyzes published economic research to look for publication bias. They showed that if the published literature were unbiased, it would contain far more negative findings than actually occur (i.e. studies that fail to reject the null hypothesis). The result is a literature that over-represents positive findings and is therefore misleading. Publication bias also figured in the CGD Evaluation Gap Working Group’s report which discussed the tendency of development institutions to disseminate positive findings and either soften or hide studies that are less flattering.The pressure to show that a particular medical treatment works is obvious when financial interests are at stake, but it is no less insidious when professional reputations or academic tenure are on the table. For health projects in developing countries, the pressure to claim success is furthered by the fear that any hint of failure or even modest progress will discourage the public and private donations that make development work possible.It may be difficult to stand against these pressures, but we can and should. As Freedman quotes Ioannidis:

“If the drugs don’t work and we’re not sure how to treat something, why should we claim differently? Some fear that there may be less funding because we stop claiming we can prove we have miraculous treatments. But if we can’t really provide those miracles, how long will we be able to fool the public anyway? The scientific enterprise is probably the most fantastic achievement in human history, but that doesn’t mean we have a right to overstate what we’re accomplishing.”