The Biggest Experiment in Evaluation: MCC and Lessons for Farmer Training

In my first blog on recently published impact evaluations by the MCC, I argued that MCC has established a new standard for systematic learning about development programs. In my second blog, I addressed questions related to the design and use of impact evaluations. In this blog, I’m asking what the MCC evaluations tell us about farmer training programs themselves.

The five impact evaluation studies report on the implementation of farmer training programs that represented a small share of the overall grants (about 13 percent of the total budget and 2 percent of MCC’s portfolio of compacts). They found that all five of the programs were successful in completing their activities and training farmers, largely as planned. In fact, the number of farmers using improved cultivation techniques actually exceeded targets in El Salvador (by 64%), Ghana (by 29%) and Armenia (by 17%). Farm incomes appear to have increased in El Salvador (for dairy farmers), northern Ghana and Nicaragua while no impact was found in Armenia or central Ghana. The biggest puzzle is that none of the programs demonstrated reductions in poverty (as measured by household consumption) even when farm incomes rose. The MCC largely attributes this disconnect to failures in program logic, study implementation and the difficulties of measuring rural incomes. While these caveats are important, it is at least as important to begin questioning some of the fundamental tenets behind the claim that farmer training can reduce poverty.  My guess is that these MCC studies will be actively cited for years to come, perhaps contributing to a revolution in thinking about farmer training programs comparable to the shakeups experienced by job training programs decades ago and microfinance today.Why do I predict such a big effect on farmer training programs? Because the impeccable logic of farmer training programs – from increased productivity to increased farm income to reduced household poverty – has rarely been tested. Taken together, these five studies (combined with three earlier studies from other institutions) suggest that some farmer training programs increase farm income. They also suggest that the impact on household incomes may be negligible.This result should not be a surprise. Farm income is influenced by many things beyond productivity. Think about the way commodity prices fluctuate; consider how increased local supplies could even drive prices down. Household consumption is influenced by even more things –tradeoffs between children working or spending time in school, allocation of time between remunerated and unremunerated work, sharing across households. In his peer review for MCC, Will Masters concluded:

“... successful program designs that target productive public investment will – like this excellent impact evaluation – be aimed at beneficiaries’ full income from multiple activities, taking account of heterogeneity and risk.  To raise full income for a whole population, program targets should be defined in terms of the market and policy failures that they remedy and the productive inputs to be supplied, with success measured in terms of final consumption from both farm and nonfarm enterprises.  The Nicaragua RBD evaluation shows clearly how programs that promote specific businesses can meet their targets, and yet fall short of their larger objective.”

It will take years to distinguish whether the limited impact on poverty is due to poorly designed interventions or to some failure in the strategy as a whole. But that discussion can move faster and more deliberately as a result of these additions to the knowledge base.As I wrote in part one, MCC seems to be taking the right approach to these evaluations. I particularly congratulated them for their public commitment to keep doing rigorous evaluations and publish their results. In part two, I emphasized the lessons for doing better studies in the future. The next step for the substantive discussion raised by these studies is to ask whether farmer training is a “good buy” for development efforts. What standard will future farmer training proposals get when they are presented to decision making bodies at the MCC… or at USAID, the World Bank, SIDA, the Gates Foundation? Until now, in our relative ignorance, it was plausible to think that successfully implemented farmer training programs significantly reduced poverty. What these studies tell us is that the standard of proof is shifting. The fault may lie in the logic of farmer training programs or something else. But these studies are a wakeup call. We know something about the links between training and increased productivity, though we could do much better. We know very little about whether such programs can reduce poverty. If MCC and others take these lessons to heart, future programs will have to be designed very differently (… and tested!).