Rossi's Rules

A friend of mine who runs the evaluation division of the U.S. government's Department of Housing and Urban Development once pointed me to a clever 1987 article by sociologist Peter H. Rossi, who distilled years of experience evaluating social programs into a few simple "metallic laws":

A dramatic but slightly overdrawn view of two decades of evaluation efforts can be stated as a set of "laws," each summarizing some strong tendency that can be discerned in that body of materials. Following a 19th Century practice that has fallen into disuse in social science, these laws are named after substances of varying durability, roughly indexing each law's robustness.The Iron Law of Evaluation: The expected value of any net impact assessment of any large scale social program is zero.The Iron Law arises from the experience that few impact assessments of large scale social programs have found that the programs in question had any net impact. The law also means that, based on the evaluation efforts of the last twenty years, the best a priori estimate of the net impact assessment of any program is zero, i.e., that the program will have no effect.The Stainless Steel Law of Evaluation: The better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero.This law means that the more technically rigorous the net impact assessment, the more likely are its results to be zero---or no effect. Specifically, this law implies that estimating net impacts through randomized controlled experiments, the avowedly best approach to estimating net impacts, is more likely to show zero effects than other less rigorous approaches.The Brass Law of Evaluation: The more social programs are designed to change individuals, the more likely the net impact of the program will be zero.This law means that social programs designed to rehabilitate individuals by changing them in some way or another are more likely to fail. The Brass Law may appear to be redundant since all programs, including those designed to deal with individuals, are covered by the Iron Law. This redundancy is intended to emphasize the especially difficult task faced in designing and implementing effective programs that are designed to rehabilitate individuals.The Zinc Law of Evaluation: Only those programs that are likely to fail are evaluated.Of the several metallic laws of evaluation, the zinc law has the most optimistic slant since it implies that there are effective programs but that such effective programs are never evaluated. It also implies that if a social program is effective, that characteristic is obvious enough and hence policy makers and others who sponsor and fund evaluations decide against evaluation.

Sociology, of course, is not physics. It has no, well, ironclad truths. In fact, Rossi wrote that the Iron Law is "somewhat spongy and therefore easily, although not frequently, broken." But the Stainless Steel law in particular accords with my own experience examining the literature on the impacts of microfinance (which is mostly about microcredit). The better done the study, the more muted the effects found. This is why I doubt most microfinance impact studies.Rossi wrote this article before the Web, so my research assistant could only obtain it by visiting the Library of Congress. Copyright law forbids me from sharing the whole thing with you. The full citation is: Peter H. Rossi, The Iron Law of Evaluation and Other Metallic Rules. Research in Social Problems and Public Policy 4 (1987): 3--20.Do not confuse Rossi's 4 rules with Ross's Reel #4.