How USAID Could Score a Double-Win for Learning and Transparency

USAID has made sweeping new commitments to two important elements of foreign assistance reform: improving learning and increasing transparency.  A commitment to publish datasets associated with USAID-funded evaluations would be a further win for both.USAID’s evaluation policy seeks to increase the quantity and quality of evaluations to improve accountability and learning.  As part of this, and as part of USAID’s commitment to greater transparency, all evaluation reports must be published to the Development Experience Clearinghouse (DEC).  This is hugely important, but USAID should go further by also publishing the underlying data (at least for impact evaluations, which, by definition, should be based on rigorous data analysis).  The Millennium Challenge Corporation (MCC), arguably a USG leader in foreign assistance evaluation, has committed to do just this, has already published three evaluation datasets, and has publicized its timeline to post additional data (by the end of 2014, pending the finalization of a privacy protection plan that will provide for proper anonymization, safe storage of sensitive data, etc.).Sharing underlying data is not just good practice for transparency, it’s also a research best practice (CGD adopted its own research data and code disclosure policy in 2011).  Data sharing enables others to try to replicate (or potentially challenge) the evaluator’s results, provide feedback that can inform future evaluations, or attempt to answer new research questions, all of which offer opportunity for greater learning—one of USAID’s key goals for its evaluation efforts.USAID has some policies in place that get it part of the way there.  USAID staff are supposed to ensure datasets are among an evaluation contractor’s set of deliverables, and USAID missions are required to warehouse all quantitative evaluation data.  But the fate of these datasets after they are warehoused in individual country offices is not clear.  It appears that while datasets collected with USAID funds for the purposes of an evaluation must be stored in a central database (according to operational policy), USAID is still “working to determine how best to meet [this requirement]” (according to the first annual report on the evaluation policy). This seems to suggest that managing underlying evaluation data—a necessary precursor to publishing it—is at least on the radar but has not, to date, been at the top of the “to do” list.  I understand that USAID has very recently dedicated funding to pursue this activity, however, which is an encouraging sign for forward progress.To achieve a double-win for learning and transparency, my recommendations for USAID are to:

1. Increase momentum around the development of a central management system for evaluation data.
2. Develop (and publish) policies and plans that would govern the publication of the data, including provisions for privacy protection where datasets include evaluation subjects’ personal information.
3. Commit to publishing evaluation datasets, either to the DEC or another public website.  Whatever the location, USAID should ensure a link between the evaluation reports available on the DEC and any associated published datasets to facilitate attempts to replicate results.

USAID appears to be moving in this direction.  While its Open Government Plan doesn’t exactly reference an intent to publish evaluation data, it does say—in reference to the DEC—that it is moving toward “collecting and publishing underlying data in the most accessible forms and formats as called for in the Open Government Directive….With this additional content USAID will be able to share the supporting data from its projects and analysis directly with the public in ways not previously possible.”  This is promising but indirect, unspecific, and not clearly applicable to evaluation data if the DEC doesn’t end up being the most appropriate place to host these datasets.  It would be great to see USAID state more clearly a commitment to publishing evaluation data, including a timeline and the steps it is taking to make it happen.  I’ll be keeping my eye out for this, perhaps in a future iteration of the Open Government Plan, the upcoming progress report on USAID Forward reforms, or a future report on the Evaluation Policy.