This is a joint post with Christina Droggitis.
This week’ NONIE conference in Paris is focusing on lessons learned in impact evaluations. At the two-day conference, the executive director of the International Initiative for Impact Evaluation (3ie), Howard White, will present his organization’s Policy Window – a grant piloted this past year that seeks to improve the relevance of impact evaluations by first soliciting proposals from developing country government agencies and NGOs regarding the policy questions that they wish to have rigorously evaluated. To coincide with the conference, 3ie released its annual report which highlights the success of the Policy Window pilot, along with the other achievements of 3ie’s second operational year.
A proper evaluation of 3ie’s progress requires looking at the quality of the studies that it finances, the relevance of findings that it disseminates and its contribution to raising the general standards for evidence. With only 2 years of operation it is too early for such a complete assessment; after all, most of the studies financed in its first round of grants are still being implemented. Nevertheless, we can assess whether 3ie seems to be on the right track, and the balance looks good.
A first cheer goes to 3ie for progress on its commitment to assure that the research it finances is not dominated exclusively by researchers from rich countries. 3ie reported that over 40% of the principle investigators for the studies it is funding are from developing country institutions and almost all of the studies with principal investigators from wealthy countries involve partnerships with local researchers or institutions. India formally created an independent evaluation organization this year, partly with assistance from 3ie, including a prominent workshop in 2009. 3ie’s outreach efforts have also included hundreds of policymakers and researchers who have attended its training sessions and workshops; and its bi-monthly e-newsletter reaches over 5,000 subscribers around the world.
A second cheer goes to 3ie’s members for making substantial commitments to funding. Members have committed over $28 million to fund 72 new impact evaluations of which $8 million has already been disbursed. 3ie gained two new full members last year, Irish Aid and USAID. 3ie’s governance structure sets minimum contribution levels for members but also adopted the principal that members should provide funding in relation to their scale of operations. Thus far, the bulk of resources are provided by three key supporters – the Gates Foundation, DfID and the Hewlett Foundation – who contributed $5 million, $3 million and $2 million, respectively, in 2010. CIDA and Irish Aid were new financial contributors last year and AusAid, the Netherlands, NORA, the MCC, and Save the Children all contributed additional grants in 2010.
A third cheer goes to 3ie’s management and board for simply getting 3ie up and running. At the beginning of 2009, 3ie had one employee – the executive director. It was only beginning to get its financial administration in order, to develop a relationship with the Global Development Network that hosts it in Delhi, and to construct a rigorous and transparent grant review system. Setting up any new organization is difficult but to do so under the public magnifying glass and with international scope is no mean feat. And with all this internal organizational effort, 3ie has continued to reach outside itself to connect with a new partner and 22 more associate members (mainly research centers) who join a growing network of more than 100 organizations.
Looking ahead, how will we measure progress next year? To its credit, 3ie “walks the walk” by presenting an explicit theory of change and identifying how it will measure its performance (right up front on page 6!). My own interests in 3ie’s progress revolve around just a few measures in the coming year:
- First, we should be able to better assess the quality and relevance of completed impact evaluations funded by 3ie’s first round of grants and see if the review process needs improvement.
- Second, the involvement of developing country researchers, NGOs and governments in 3ie grants and activities will continue to be an important proxy for the relevance and sustainability of impact evaluation in these countries (this is one of the indicators highlighted by 3ie).
- Third, we can look for progress on key services like establishing a prospective registry for impact evaluations and assuring public access to data from 3ie-funded studies.
- Finally, we can ask whether bilateral agencies and foundations (other than Gates, DFID and Hewlett) have stepped up their financial contributions in relation to the scale of their operations. In particular, will the World Bank finally recognize that it should join 3ie and contribute resources to this independent external mechanism for building evidence on policy in developing countries?
Many organizations and individuals have contributed in recent years to the expanding evidence base from good quality impact evaluations. 3ie has a part to play in this movement and, so far, has come through admirably. So, three cheers for 3ie and its supporters. Keep up the good work.