AIDS “Truths” Can Also Wear Off

In his December article in the New Yorker (tweeted here by Nancy Birdsall), Jonah Lehrer contends that some so-called “truths” in the scientific literature are transient; they “wear off” over time.  At first his analysis seems to apply most directly to psychology, where the possibility that the researcher can inadvertently influence the subjects to change the outcome of a trial can only increase the proportion of statistically  significant, but false, results. However, the AIDS research arena has seen several initiatives that have looked good at first, only to be attenuated in subsequent trials, and then ultimately diluted to insignificance.Economists and epidemiologists can sometimes be justifiably accused of “data-mining,” the process of torturing the data until it relents and coughs up a significant t-statistic.  Similarly, pharmaceutical researchers may sometimes be guilty of “molecule-mining,” running tests on so many different molecules that preliminary results must, by the laws of chance, be statistically significant at the 5% significance level for 5% of them, even if none are truly effective.  Molecules which at first seemed to offer promise, at least to those who first tried them, include the false cures for AIDS (Kemron, Pearl Omega and MM1 are the most notorious).  Pharmaceutical firms are all too familiar with molecules that seem promising in early trials, only for their effects fizzle out in the course of more rigorous investigation.In the domain of HIV prevention, the rigorously designed and analyzed “Mwanza trial,” whose results were published in the Lancet in 1995 here,  seemed to establish that providing a community with access to superior treatment for the curable sexually transmitted infections would cause a 40% reduction in new HIV infections in those communities.  However, subsequent equally rigorous randomly controlled trials in the nearby communities of Masaka and Rakai were unable to find this benefit for treatment of the curable sexually transmitted infections.  Had the treatment “worn off?”In contrast to these failures to replicate, male circumcision has now proved successful in several trials and the statistical significance of the results seems to be increasing rather than “wearing off.”  Other recent successes in HIV prevention that have been flagged on this site have not yet been replicated.  These include partially successful vaccine trials from East Asia, the Caprisa trial of female-controlled microbicides, and the use of incentives to reward sexually active young people for remaining free of sexually transmitted infections or for attending school.  The weight of evidence in all of these cases may be strong enough to justify initial scale-up in some places – but only if scale-up is designed to include rigorous evaluation which will measure whether the intervention is “wearing off” – or detect the field conditions under which it is most effective.