The Challenge of Scaling Up Proven Interventions -- Justin Sandefur

My guest on this week’s Wonkcast is Justin Sandefur, a research fellow at CGD whose recent work has focused on education in Kenya. One study examines the returns of private schooling, while another looks at the effects of contract teachers on student test scores. The results of these studies highlight shortcomings in public education, including failures of accountability and a dense bureaucracy.Public vs Private SchoolsJustin tells me he first became involved in Africa while working in Tanzania on a household survey measuring poverty and agricultural statistics, as a government employee. His insights into how government works—or fails to work—have since shaped his research on schooling in Kenya. His CGD Working Paper, The High Return to Private Schooling in a Low-Income Country (with coauthors Tessa Bold from Goethe University Frankfurt, Mwangi Kimenyi from the Brookings Institutions, and Germano Mwabu from the University of Nairobi) found that private schools in Kenya not only cost less than public schools, but produce better results.“We focused our time estimating the impact of private schooling on test scores and proving to ourselves that going to a private school increases your test scores,” says Justin. “When we shopped that idea around in Kenya no one was surprised. But people had a hard time believing is that these private schools really are cheaper, operating on about half the budget of the government public schools.”I ask Justin why private schools in Kenya get better results with less money.“That is the 64,000 dollar question,” he says. “If we knew the answer to that I think we’d have the agenda for education reform in the developing world laid out for us. The private schools are a black box and the point of this paper is to really point out how big that black box is.”Justin tells me that parents are more involved in private schools because their money goes towards schools fees. This element of bottom-up accountability, which is typically absent in public schools, could contribute to the success of private schools in Kenya, he says.Another advantage of private schools is simply that they have found a cheap technology for teaching -- employing teachers on short-term contracts at salaries far below civil service wages.Can Government Learn From The Private Sector?Justin and I discuss a forthcoming working paper which looks at whether the Kenyan government can adopt this technology of contract teachers and successfully scale it up nationwide in government schools.Justin and his coauthors organized a randomized controlled trial (RCT) in Kenya which draws on prior work by Esther Duflo, Michael Kramer, and Pascaline Dupas. These earlier studies, also in Kenya,  showed employing contract teachers in public schools had a statistically significant effect on improving test scores. Based on these findings, the Ministry of Education agreed to scale up the intervention across the country. But in their new paper, Justin and his co-authors have found it matters a lot which entity is responsible for running the program. When the contract teachers are hired by an international NGO (as was also the case in the earlier pilots) the impact on test scores is significant and positive. But when the same intervention is executed by the Ministry of Education, there is no impact on test scores.Why?“The question of why is really difficult,” explains Justin. “There are a number of hypotheses on the table that are mostly related to the government’s ability to implement the programs in the districts.”I end the Wonkcast by pushing Justin to discuss whether there are broader lessons from the new study. If scaling-up an intervention proven effective in an RCT means shifting from an NGO to a much larger government bureaucracy, can policymakers assume that a proven intervention will still work?“There are some frightening implications here,” says Justin. The problem, he says, is not with RCTs, which are rightly considered the gold standard for evaluation.   But as policy makers look to turn RCTs into national programs, more attention needs to be paid to the potential differences in the implementing organizations.I’d like to thank Alexandra Gordon for serving as producer and recording engineer, and for helping to draft this post.