Evaluation, Evaluation Everywhere: IOM Progress on Evaluating the Impact of PEPFAR

Within the beltway and across newspaper headlines, PEPFAR is widely perceived as a very successful US foreign assistance program. Without fail, mentions of PEPFAR’s success all reference the same single measure: currently PEPFAR supports 2.4 million people on ARV treatment.Interestingly, these broad perceptions of success exist despite the fact that, or maybe because, there has never been a systematic evaluation of PEPFAR’s impact. However, that is in the process of changing.In the 2008 re-authorization of PEPFAR, the US Institute of Medicine (IOM) was mandated to evaluate the impact of PEPFAR. This is a monumental task. PEPFAR will have spent around US$60 billion over 10 years, through multiple US government agencies and hundreds of partnering organizations, and across dozens of countries and equally many program areas. The IOM has been tasked with figuring out how this immense initiative has changed the world.The IOM approach has two phases. In November 2009 it started a planning committee, made up of 20 members with a wide range of expertise, which is currently designing an evaluation plan. By fall 2010, after five meetings, information gathering sessions, and discussion, they hope to have an evaluation plan. After that an evaluation committee will be formed, with some overlapping membership from the planning committee, and asked to produced an impact evaluation report by fall 2012.So far the planning committee has met three times (out of the five they have scheduled): a November meeting in Washington, a December meeting in Geneva, and most recently a meeting last week again in Washington. Last Thursday I attended the “public information gathering session”, which was the second day of the three day meeting and open to the public. At this meeting the planning committee received presentations from staff responsible for monitoring and evaluation (M&E) at various levels of the PEPFAR, including: staff in the strategic information department at OGAC headquarters, M&E managers within CDC and US AID, and program managers and funding recipients based in developing countries. Surprisingly, six hours of these presentations continued to capture my interest. It struck me that this process, perhaps even more than the final report, offers great opportunities for learning and building understanding of what PEPFAR is and how it has performed. It also struck me that the room was half empty, despite being open to the public.There have been literally hundreds or perhaps even thousands of evaluation studies of PEPFAR programs. While not all of these studies investigated impact, they do indicate an environment potentially rich with program data and analysis. At the committee meeting I was disappointed to learn that OGAC does not receive or even track all of these evaluations, receiving only very general aggregated data for their own analysis upon which to base decisions.A further concern is that there is very little sharing of data outside the programmatic silos. In particular, national stakeholders are often left out of the loop, neither benefiting from the M&E learning nor being armed with the evidence for effective decision-making about programs. Country ownership and accountability to national stakeholders will always be limited if they are not involved in the design, implementation and learning that comes from monitoring and evaluation.Given the critical importance and opportunity of this activity, I hope that we start to see far more engagement around this process. Perhaps more could be accomplished if the committees share their intermediate products as they take stock of the PEPFAR program in its entirety and try to figure out how to evaluate its impact. For instance, it would be invaluable if the committee would conduct and publish a systematic synthetic review of the evaluation studies that have already been done of PEPFAR programs and projects. Such a review would benefit both the committee as they plan their evaluation and everyone concerned with the effectiveness of PEPFAR programs.It could help:

• Provide a clear picture of which PEPFAR programs have and have not been evaluated and how programs have been evaluated;
• Map what has already been learned about impact, even if existing results are limited in scale; and
• Identify gaps and opportunities for evaluating PEPFAR on a broader scale.