Focus on Results: MCC’s Model in Practice (brief)

When MCC was founded, there was widespread skepticism of the effectiveness of foreign assistance. Many people, including those working in development institutions, agreed that too much aid money was being spent on poor projects in service of poorly defined objectives with little understanding of what these funds were achieving. As a result, when MCC was being created between 2002 and 2004, there was substantial bipartisan support for the new agency to focus on achieving (and measuring) results. In practice, the broad idea of “measurable results” quickly became operationalized by MCC as a commitment to funding only projects that would raise local incomes in a cost effective way.

MCC’s “Framework for Results” has four key components:

1. A constraints analysis that helps focus the investment package on the sectors that most constrain the private investment that will drive poverty-reducing economic growth.
2. Economic rate of return (or cost-benefit) analyses that estimate the expected increase in local income (compared to the cost of implementation) for almost every proposed investment.
3. Project monitoring during the lifespan of the compact that tracks the extent to which project implementation is proceeding compared to expectations.
4. Independent ex post evaluation that seeks to determine whether MCC investments caused increases in income for local beneficiaries that would not have occurred without the compact (and whether these attributable increases in income exceeded the value of resources expended on them).

While MCC’s “Framework for Results” remains the gold standard among development institutions, MCC’s practices have sometimes been inconsistent with its guidelines.

Constraints Analyses Focus Investments on Growth

MCC expects its partner countries to take the lead in identifying the investments MCC will fund. One of the first steps is a “Constraints Analysis” (CA), an analysis of economic data to identify the most pressing factors that limit private investment and economic growth. [1] Since its formal adoption in 2006, the CA has served to shape expectations of what MCC will fund (e.g., investments in a small number of sectors that will contribute to economic growth) and what it will not (e.g., low-return activities reflecting local political interests), leading to better proposals more focused on growth.

MCC's use of the CA has enabled the agency to mitigate the politicization of the program development process, shifting focus away from the "laundry lists" sometimes found in existing country plans and specific interests with powerful advocates toward a small number of key challenges that need to be addressed first. The CA has not totally eliminated these other pressures, but its use in the program development process to identify a small number of priorities clearly distinguishes MCC from other donors.

The CA process has occasionally been challenged by two key factors: the scarcity of economists with specific training or expertise in growth diagnostics; and government responses in some MCC partner countries to politically undesirable findings. Because the final CA is a public document, some governments have been reluctant to publicly acknowledge findings that suggest policy and institutional failures are constraining growth or that fail to identify problems that the governments have already determined are political priorities for donor support. While several governments have allowed politically sensitive CA findings to be published with little or no interference, some country officials have requested that language and conclusions be modified either to limit discussion of controversial policy issues or to better conform to their preexisting development strategies. [2] In every case, however, the technical analysis provided important information that helped partner countries identify growth-related priorities and exclude some nonpriority program areas.

This tension between economic evidence and local political priorities is a natural one, but MCC has managed it by taking steps to ensure that: (1) country CA teams have high-level political support to identify sensitive issues; (2) country CA teams are willing to let evidence inform their positions; (3) MCC teams are given political backing to confront controversial themes; and (4) a mechanism to resolve differences exists (e.g., the attachment of dissenting opinions or disclaimers).

When MCC adopted the CA in 2006, it was the only donor to embrace the use of growth diagnostics as a part of its normal operations. The practice is now more common, but no other agency requires a growth diagnostic as part of a strategic prioritization process.

Recommendations

1. MCC should protect the role of evidence in the final CA report by discouraging countries from suppressing data when the results appear unfavorable to the partner country government or do not fully conform to the country’s national strategy.
2. MCC should devote more technical resources up front for the initial analytical work by increasing the number of staff with training in growth diagnostics.
3. MCC should include economists from other donors in the conduct of CAs to raise the profile of the exercise and reduce the likelihood of multiple (and often contradictory) analyses being conducted at the same time.
4. MCC should publish completed constraints analyses for countries while they are still developing compacts to increase accountability for how well-targeted the proposed compact investments are.

Cost-Benefit Analysis Focuses MCC Investments on Value-For-Money Results

When a foreign assistance critic complains that aid projects do not work, they rarely mean that the programs are not helping anybody. Most aid programs can find anecdotal evidence that somebody was helped (and thereby claim “success”). The criticism is that the gains to the beneficiaries were too small to justify the expenditure. In this context, MCC committed to funding only projects whose benefits exceed their costs. To identify such investments, MCC asks partner countries to undertake cost-benefit analysis (CBA) for each proposed project. This exercise compares all project costs (including any fees or investments made by beneficiaries themselves and any cofinancing by the partner government and other donors) with expected increases in local incomes. MCC then calculates the economic rate of return (ERR), or the interest rate at which the net benefits would equal zero. [3] MCC requires that estimated ERRs be above a minimum “hurdle” rate (currently 10 percent) for the project to be considered economically justified. This decision process is similar to businesses only investing in activities whose internal rate of return exceeds the interest rate on money they borrow.

MCC’s use of CBA for project selection sets MCC apart from other donors, even when activities might look very similar. For example, while another donor might spend $5 million to train 5,000 local farmers and measure success by counting whether those 5,000 farmers were trained, MCC would approve such a proposal only if there were a reasonable expectation from economic theory or empirical evidence that the newly trained farmers would adopt new practices and raise their incomes by $5 million or more (in discounted terms) over the program’s lifetime. MCC is not the only donor to use cost-benefit analysis, but it is the only donor to use it for the vast majority of its portfolio.

The ERR is not the only factor MCC considers when selecting projects for approval. In fact, MCC might choose not to approve a project with a high ERR since MCC also considers noneconomic factors such as the level of local support for the project and the level of conformity with other MCC policies (e.g., social and environmental safeguards). Funding a project with an ERR below 10 percent, however, would be inconsistent with MCC’s growth focus, as it would reflect an activity whose costs exceed its benefits. Some proposed projects do not lend themselves well to ERR analysis, such as policy-reform activities and innovative, experimental proposals that have not been tested in other contexts. Such activities, however, usually can be expected to be small relative to an overall MCC country program; large expenditures by MCC on proposals with no solid basis for estimating impact can be seen as inconsistent with its fundamental focus on results.

How consistently has MCC based its investment decisions on ERR analysis? According to available information, MCC has invested the vast majority of its portfolio (over 90% of compact funds) in projects expected to deliver benefits in excess of their costs. This record reflects the seriousness with which MCC managers take their institutional commitment to results, and it should be recognized as a tremendous accomplishment that no other donor can claim.

However, according to public data, MCC’s investment decision-making record also includes around $800 million worth of major investments (around 9 percent of the agency’s total compact portfolio) that did not demonstrate cost-effectiveness at the time they were approved for funding. [4] These inconsistencies have taken two main forms: (1) approving projects whose costs exceed their benefits (table 1); or (2) deciding to fund projects in the absence of cost-benefit analysis (table 2).

Table 1: MCC has approved over $300 million worth of investments known not to be cost-effective

Table 2: MCC has approved around $500 million for projects or activities whose cost-effectiveness was unknown

Why did MCC agree to spend its scarce funds on investments with low or unknown value? Several reasons were likely at play, including the following. First, the partner government may have expressed strong preferences for the investments, and MCC may have used another pillar of its model, country ownership, as rationale for proceeding. Second, some MCC staff likely had various reasons to support the investment, including (a) the interests of program staff to design and support larger projects in their areas of interest; (b) the view that larger projects would be more likely to create sufficient incentives for the partner government to undertake related policy and institutional reforms; and (c) a desire to obligate enough of MCC’s balances to demonstrate to Congress the agency’s need for full funding the following year.

Initially, ERRs were just a tool to guide initial investment decisions; there were no plans to revise them after initial project selection. However, as the first set of compacts approached their midpoints, MCC found that almost every one needed substantial restructuring or “rescoping.” Because of implementation delays, higher-than-expected costs, early evidence of program failure, and other circumstances, MCC found country partners asking to shift money between activities. In some early cases, MCC calculated and used revised ERRs to guide those decisions, but not always. Since 2012, MCC policy has required considering revised ERRs during rescoping, but decisions are not required to align with the results. Because MCC is not transparent about its rescoping decisions, it is hard for stakeholders to hold MCC accountable for its midcourse decisions which may continue (or even increase) funds for low-return activities.

Recommendations

1. MCC should reinforce its commitment to investments with adequate economic rates of return.
2. Congress and other external stakeholders should inquire about the economic rates of return of MCC’s compact investments—both at project selection and during rescoping exercises—and encourage the agency to preserve its focus on value-for-money results.
3. MCC should use only a small portion of its compact funds to test innovative concepts for which no ERR can be constructed; these experiments should always be accompanied by a rigorous evaluation.
4. MCC should publish rescoping decisions in a systematic and timely way and publish the data—including revised ERRs—that inform the decisions.

Project Monitoring Tracks Progress and Allows Regular Assessment of Cost-Effectiveness

Project monitoring during implementation—tracking progress toward pre-set targets—is a relatively standard practice among donors, but MCC practices are distinctive in two key ways. First, it is the only US donor agency to link its performance targets to the expected results generated by cost-benefit analysis. This allows MCC not only to see how projects are progressing but also to determine whether deviations from expected progress undermine the cost-effectiveness of the investment. Second, compared to most other donors, MCC is far more transparent with its monitoring data, posting regular, relatively comprehensive information on projects’ progress toward their targets.

There are, however, limitations to MCC’s transparency in tracking project progress. Compact performance data are sometimes posted with some delay. [5] Also, MCC does not always publish the results of midterm evaluations conducted to determine how a project is performing or whether pilot activities should be brought to scale. Without this information, it is hard for external stakeholders to understand the basis for MCC decisions to continue or scale up programming.

Further, MCC does not provide a systematic report on partner countries’ fulfillment of conditions precedent (CPs). CPs are policy changes or other actions agreed to by the partner country and written into the compact. In general, the projected impact of a compact’s projects—as well as the cost-effectiveness of these projects—are highly contingent upon the partner country fulfilling these obligations. Many CPs entail politically difficult regulatory or administrative changes, and getting timely compliance with the agreed-upon conditions has sometimes been challenging. [6] MCC tracks countries’ progress fulfilling CPs, but it does not systematically make this information public.

Recommendations

1. MCC should publish decisions on project scale-up in a systematic and timely way and publish the data that informed the decision, such as midterm evaluation results.
2. MCC should increase resources dedicated to disseminating monitoring and evaluation information. [7]
3. MCC should audit countries’ compliance with past conditions precedents and work with them to regularly publish progress on current and future conditions precedent.

MCC’s Robust Evaluation Strategy Covers the Vast Majority of Its Portfolio including Notable Use of Rigorous Impact Evaluation Methods

In addition to tracking progress during implementation, MCC funds independent evaluations for most of its investments. Since the agency’s goal is to increase local incomes, these evaluations help assess the extent to which the evaluated projects have actually done so, allowing for both learning and accountability. MCC’s evaluation policy follows several aspects of best practice including the following:

• Sufficient coverage: Almost 85 percent of the value of the portfolio is or will be covered by an evaluation following the program’s completion.
• Up-front planning: Evaluations are incorporated into project design from the beginning.
• Rigor: MCC is committed to using the most rigorous methods feasible and appropriate, and recognizes that impact evaluation methods are the most precise and credible—roughly half of planned evaluations use experimental or quasi-experimental methods.
• Independence: Evaluations are conducted by independent evaluators.
• Peer review: Evaluation reports are subject to independent peer review.
• Publication: MCC is committed to posting all evaluation reports as well as datasets.

However, there are some limitations to MCC’s evaluation policy in practice, particularly with respect to public dissemination.

• Timeliness: MCC has posted only seven final impact evaluations and approximately 10 more performance evaluations, though there are many others in process that are long overdue for publication. Of course, impact evaluations do take time to complete and results should not be expected immediately upon program closure. In fact, data collection, analysis, and reporting may reasonably take two to three years after the program ends. However, even with this in mind, MCC has delayed the review and release of some evaluation reports; such delays devalue their usefulness for accountability and learning.
• Comprehensiveness: An evaluation strategy must be assessed not only on the basis of what is evaluated but also by what is not evaluated. MCC acknowledges that there are gaps in its current evaluation plans (i.e., there are activities that are not covered by an evaluation), but it is hard for external observers to identify those gaps and understand why activities were not evaluated.
• Clarity: MCC has a mixed record in terms of reporting the findings of the evaluations released to date. MCC was praised for acknowledging and discussing lower-than-expected results for the first set of five evaluations, but the framing of the discussion drew some attention away from the main conclusion that none of the five evaluations found significant evidence that beneficiary incomes had increased by including discussion of other, noncore findings that suggested more success. Two subsequent impact evaluations, also showing weak or no impact on local incomes, have been released very quietly.

MCC is neither the first nor the only US government foreign assistance agency to conduct evaluations. What stands out when comparing evaluation policies and practices among US foreign assistance agencies is that MCC’s process is more institutionalized, covers more of the portfolio, and has a greater emphasis on rigorous methods.

Recommendations

1. MCC should post country evaluation plans before implementation begins.
2. MCC should provide an organization-wide evaluation strategy that details how the agency will prioritize impact evaluations versus less rigorous performance evaluations in terms of spending allocation, countries, sectors, and activities.
3. MCC should assign responsibility for public statements about the results of impact evaluations to a specific MCC official to provide more accountability for how results—both positive and negative—are discussed.
4. MCC should, as more ex post evaluations are finalized, demonstrate how it is applying lessons learned from evaluations to current programming.

Additional Resources

In the MCC at 10 series, Sarah Rose and Franck Wiebe take an in-depth look at three main pillars of MCC’s model: the agency’s focus on policy performance, results, and country ownership. To what extent has MCC’s model governed its operations in practice? How should MCC strengthen and expand its model and operations during the next 10 years? How is MCC different than other modes of US foreign assistance? Find the papers and more briefs like this one at CGDev.org/page/mcc-ten.




---

[1]" MCC’s CA follows the growth diagnostic approach developed in R. Hausmann, D. Rodrik, and A. Velasco, “Growth Diagnostics,” Faculty Research Working Paper Series (RWP06-030), John F. Kennedy School of Government, Harvard University, Cambridge, MA (2006).

[2] A notable example of an independent CA occurred in Moldova, where the government allowed frank discussion of public corruption as a binding constraints to growth. In contrast, the CAs in three Partnership for Growth countries (El Salvador, the Philippines, and Tanzania) were among those influenced by local political pressures. The author’s (Wiebe) own experience with the CA process under the Partnership for Growth, reinforced by the views of several economists working directly on country analyses, suggested that the CAs in these countries were more significantly shaped by local preferences and prior assumptions or those held by other USG agencies, with language and final conclusions affected in the process.

[3] This analysis is consistent with the idea that benefits this year are worth more than the same amount of benefits delivered ten or twenty years later

[4] These estimates are based on ERR calculations posted by MCC on its website and references to original ERR values included in online monitoring and evaluation reports. Some M&E reports refer to ERRs that are not posted, suggesting that the public set of information is incomplete. Thus, the value of investments known to be not cost-effective could be higher. Indeed, when MCC reports ERRs for activities, these often include subactivities that are known or could be expected to have ERRs below 10 percent. In addition, more than $100 million was invested in projects with low ERRs in three early compacts (Cape Verde, Nicaragua, and El Salvador), approved by MCC before the current minimum 10 percent ERR threshold was put into place. While the approved activities were fully consistent with MCC’s requirements at the time, MCC arguably should have realized that the country-specific hurdle rates (based on recent GDP growth rates) used for these countries at the time (1.4 percent for El Salvador, 4.8 percent for Cape Verde, and 2.7 percent for Nicaragua) were too low to meaningfully indicate cost-effectiveness.

[5] For instance, in November 2014, of the eight compacts currently being implemented, recent (4th quarter 2014) performance data were available for only five. For two countries posted results data were over a year old; one country had no results data posted.

[6] In fact, MCC is moving to front-load more conditions (i.e., before the compact enters into force) to avoid the implementation delays that have sometimes befallen compacts in countries slow to complete a required condition.

[7] MCC has plans to release a new, revamped website, probably sometime in 2015. Hopefully it will address some existing limitations like hard-to-find and/or out-of-date information.

Rights & Permissions

You may use and disseminate CGD’s publications under these conditions.

Topics