Live from New York: Impact Evaluations and Policymaking

A recent New York Times article gives a real time lesson in the relationship between good impact evaluations (which often provide a lot of useful nuanced information) and policymaking (which thrives on big messages while trying to accommodate political, social and cultural pressures).The focus of this experience, unfolding right now, is New York City’s Opportunity NYC Family Rewards – a conditional cash transfer program that was inspired by Mexico’s Oportunidades program (which, incidentally was brought to Mayor Bloomberg’s attention by someone who read about Oportunidades in the Center’s book Millions Saved).After 3 years, the evaluation of Opportunity NYC showed mixed results. The program gave participating families a 16% chance of moving out of poverty, improved high school performance for some students, and increased preventive dental care; but it didn’t affect elementary and middle school performance as was hoped. The experiment was privately funded at a cost of about $6,000 per family and requires another $14 million in operational funds to continue for another year.But, for now, the Bloomberg administration has no plans to continue the program. Bloomberg’s reaction to the study, according to the NYT:

“You always hope that you’ll come across a magic silver bullet, and you never do,” he said. “If there were simple solutions, somebody would have found them a long time ago. And you make progress incrementally, particularly if you’re trying to focus on some of society’s biggest problems.”

While the Center has promoted initiatives to close the Evaluation Gap by encouraging the production of more and better impact evaluations, we’re also keenly aware that the link between evidence and policymaking is not a simple one. In fact, as part of our ongoing work, we’ll be co-hosting a workshop with the International Initiative for Impact Evaluation (3ie) on May 4^th to address the question of how evidence and policymaking interact while reviewing a series of new studies on conditional cash transfer programs around the world. (Details on the event will be posted later this month).On balance, the impact evaluation by MDRC showed this was a worthwhile experiment and Deputy Mayor Linda Biggs reaction was: “Big lesson for the future? Got to make it a lot more simple.”I hope  that the Mayor will treat the evaluation results as a source of feedback and an opportunity to learn. Then he could live up to his pragmatic reputation by continuing the Opportunity NYC Family Rewards program, modifying it based on the recent findings, and giving it another 3 years to further prove itself.