MCC Threshold Program is on the Threshold

This is a joint post with Casey Dunning.When the Millennium Challenge Corporation (MCC) board of directors meets tomorrow, they’ll be hearing more about proposed changes to the MCC threshold program. The threshold program was designed to help countries become eligible for MCC compacts and its current review sends signals the MCC is learning from its work and making mid-course corrections accordingly. This is good. But some of the proposed changes to the program beg answers to bigger questions for the MCC’s future.What is the threshold program?The MCC’s threshold program grew out of a concern that too few countries would pass MCA eligibility selection criteria and that the MCC would operate as a stand-alone agency, uncoordinated with USAID. To allay these fears, the MCC’s legislation authorizes up to ten percent of total funds “to provide assistance to a candidate country…for the purpose of assisting such country to become an eligible country.” It also allows this assistance to be provided through USAID.What’s the story so far?MCC compact eligibility is largely based on passing the majority of indicators in categories for ruling justly, investing in people, and economic freedom. For this reason, the MCC focused the threshold program almost exclusively on improving performance in failed MCC selection indicator areas. For example, Kenya is failing the control of corruption indicator and so its threshold program aims to control corruption and improve that indicator.The conceptual model of the threshold program is: select countries that don’t quite pass the selection indicators, implement a threshold program, see countries pass the indicators, and countries will be selected as compact-eligible. The design sounds logical, but six years, 23 threshold agreements and $495 million later, the MCC has recognized some flaws in this approach:

• Limitations of the indicators, data lags, and attribution issues. While MCC indicators are useful proxies for policy performance and good for comparing countries against one another, they are imprecise measures with methodological limitations, the most important of which is the time lag. The time lag is twofold: first, the data measures past performance (for example, all six ruling justly indicators are on a two-year delay) and second, it takes four to twelve months to prepare a threshold plan before it can even be implemented. Problems arise when threshold programs focus on improving indicator performance that may already be passing in real time, but does not yet show up in the data because of the time lag. For example, four of the eleven countries that signed threshold agreements in 2005 and 2006 – Indonesia, Malawi, Moldova, and Tanzania – had already passed at least one of the indicators targeted in the threshold program by the time the programs became operational.  In other words, they became eligible not as a result of threshold programs, but merely by waiting long enough for the data to catch up. It is also extremely difficult to attribute two- or three-year threshold program results to improvements in the indicators, not just because of the time lags, but also because of the focused nature of threshold interventions and the broad scope measured by the policy indicators. The eligibility indicators alone are ineffective measures of programmatic impact.
• Little demonstrable impact on either indicator performance or compact eligibility. A solid majority of the 23 threshold countries have not yet become eligible for compacts. This could be due to time lags with the data so it is simply too early to tell.  But of the ten threshold countries that have become eligible, only two – Malawi and Zambia – appear to have had a threshold program in place long enough to feasibly impact the selection indicators.
• Undermines the "MCC effect." The MCC drives home the message that countries themselves are responsible for their own policy performance and compact eligibility. The incentive created for countries to adopt legal, policy, regulatory, and institutional reforms related to MCC eligibility criteria has been dubbed the "MCC effect." The ability to encourage policy reform beyond (and before) MCC compact interventions is regarded as one of the MCC's core achievements. In contrast, the threshold program directly funds efforts to improve indicator performance, blurring responsibility for policy performance between country and donor (MCC).

Other concerns with the threshold program include:

• Inconsistent threshold program selection criteria. The MCC board has not used consistent criteria when selecting threshold countries. The result is a wide range of countries in the program from those who fall just short of MCC eligibility (such as Burkina Faso and Tanzania when selected), to those who are doing the right things but not necessarily close to meeting the indicators (Liberia) and those that are far from qualifying (the Kyrgyz Republic). This led Senators Kerry and Lugar (chair and ranking members of the Senate Foreign Relations Committee) to ask whether the threshold program is "meant to graduate countries which are at the 'tipping point' of eligibility" or whether it is supposed to be "a longer-term program that will over time improve policy performance." They also flagged significant concern over the selection of Yemen, The Gambia, and Niger for the threshold program – countries which have since been suspended.
• It doesn't help prepare countries for successful compacts. The threshold program, as currently envisioned, does little to address one of the MCC’s core challenges: building country capacity for a successful compact. This has become a bigger challenge for the MCC than having enough eligible countries. (The proposed technical fix to increase the number of low income countries in the candidate pool further increases the number of potential candidates with which the MCC might develop compacts.)
• Too much like USAID technical assistance. There is also a perception that the current threshold program is too similar to USAID technical assistance and that threshold programs could be managed entirely by USAID.

These are among the reasons that Senators Kerry and Lugar urged incoming MCC CEO Daniel Yohannes to “rethink the threshold program” and address its muddled mandates, diffuse results, and unclear selection process. They said they were “open to fairly wide changes that would modify the threshold program’s mandate and implementation.” Yohannes said he shared many of their concerns and that the MCC had initiated a thorough review of the program aiming to create more incentives for ongoing policy reforms in candidate countries.Where is it going?The MCC launched the review of the threshold program in March and is currently refining a new approach. Among the options under review were to: get rid of it, reform it but maintain a focus on the indicators, refocus the program around a broader understanding of eligibility criteria and policy reform, or repurpose it around policy performance but de-link it from compact eligibility. Despite the flaws in the original threshold approach, the MCC also recognizes some of the positive aspects of the program, including:

• Getting to know each other. Individual threshold programs have familiarized countries with the MCC process, indicators, and staff. In turn, the programs give the MCC board and staff insight into the countries' commitment to reform, compact readiness, and whether they might make good MCC partners.
• Individual positive effects. While difficult to attribute significant policy reform to the threshold programs, they have had positive results in select areas – girls' primary education in Burkina Faso, government transparency and accountability in Jordan, and control of corruption in Malawi. (Burkina Faso's threshold program, however, exemplifies the successes and shortcomings of the program. Selected in FY2005, Burkina Faso's threshold program improved enrollment of children in school by roughly 20 percent, according an independant impact evaluation. But the threshold program still had no effect on Burkina Faso's compact eligibility as it was selected as eligible in FY2006, before the threshold program had even begun.)
• Constructive relationship with USAID. The threshold program has helped the MCC to work constructively with USAID, something that was perceived to be lacking in the MCC's early years.

Consequently, the MCC is refocusing the threshold program in an effort to address the design flaws while maintaining the perceived benefits. The new threshold design aims to de-link threshold programs from improvement in the indicators and instead focus on broad policy reform; include clearer selection criteria; use threshold experiences to inform board decisions on eligibility; extend the program to three years; and strengthen engagement with USAID on design, implementation, and evaluation of threshold programs. The major shift in approach centers on a broader definition of what constitutes compact eligibility. Rather than focusing on the indicators alone, the new threshold approach recognizes that eligibility is based on a combination of 1) selection indicator performance, 2) supplementary data and information and 3) board discretion on a country's perceived commitment to reform among, other things.What questions remain?The MCC wisely recognizes a non-result (no impact on indicator scores) not as failure, but as important feedback on what is working and what isn't. The mid-course correction signals the MCC is learning and adapting from its experience, which is exactly what it should be doing. And the willingness to be transparent about the feedback—especially when the results are mixed—is a huge step in the right direction. Would that all development agencies shared as much! (More on the lack of incentives to share impact evaluation here.)As the MCC refines the new threshold approach, I expect both staff and board will be looking for specifics on selection criteria and process; the relationship with country representatives as well as USAID staff on program design, implementation and evaluation; sequencing of threshold versus full compacts; and weighing whether the new approach adequately addresses the shortcomings of the original program. The kicker is that answering these specifics raises a host of tough questions for the future of the MCC, not just the threshold program. For example:

• What are the measures of success for the threshold program? Maintaining the MCC incentive effect presents a conceptual "Catch-22" with the threshold program. The MCC incentive effect only works if eligibility is not guaranteed and countries are still motivated to improve policy before MCC compacts, but that also means we should not expect every threshold program to result in eligibility even though eligibility is the name of the threshold game. Knowing this, how will threshold success be defined?
• How does the threshold program fit into the mission and mandate of the MCC? One of the MCC's strengths is its clarity of mission, purpose, selection of countries, and model. The threshold program has been criticized for its diffuse nature and unclear results. How will the new approach stick closer to the MCC model and avoid brand confusion?
• Should the MCC go deeper in fewer countries, or broader across more countries? One aspect of old and new threshold programs is that it allows the MCC to engage a broader range of countries. Given the MCC's relatively small funds of late, should it focus on bigger impact in fewer countries, or take its unique development aproach to a greater number of countries? This also requires some assessment of how long the MCC should ideally operate in a country. Most agree five years is too short (hence the possibility of second compacts) but is twenty too long? (These are, incidentally, among the questions the Presidential Study Directive on U.S. Global Development Policy is trying to tackle for all of U.S. development engagement.)

The onus is also on the research community to improve the data and measurement of policy performance areas such as those used in the MCC's selection indicators. Accurate, timely, and comprehensible data goes a long way in helping communicate the goals, objectives, selection, and impact of the MCC and other development investments. The MCC uses the best of what's available, but we know there is much room for improvement that could in turn help tackle some of the challenges faced by the threshold program.When the MCC board meets on Wednesday, let’s hope there is a robust understanding of why the threshold program is being revised and an effort to answer the tough questions that go along with defining a new approach. Let's also hope the board has the rest of the public board seats filled soon so that there are more minds working on these issues...and so the MCC has the quorum it need to operate.Tell us what you think of the new approach!