For U.S. Aid in Pakistan, Absence of Evidence Is Not Evidence of Absence

This week, a quarterly report from the three offices charged with monitoring U.S. non-military aid programs in Pakistan set off a flurry of media coverage, all focusing on this stark assessment of the progress of the Pakistan program: “One year after the launch of the civilian assistance strategy in Pakistan, USAID has not been able to demonstrate measurable progress.”Let’s be clear here. The real story is not that USAID’s programs have all failed to make progress. It’s that USAID and the other agencies involved (State, Agriculture, Energy, Commerce, Labor, and more…) haven’t yet managed to collect data in a systematic way to prove whether or not they have. As my boyhood hero Carl Sagan once said, “Absence of evidence is not evidence of absence.” But as carefully as the report’s authors chose their words, that’s the sort of nuanced conclusion that can easily get distorted in the harsh light of news reporting or on Capitol Hill. (Among the news outlets that reported the story, some correctly represented the nuance of the report’s findings, others missed it.)If you read through the report, what you’ll find are dozens of sensible measures of aid outputs: the number of poor women receiving U.S.-funded income support grants, the number of irrigation channels rehabilitated with U.S. aid, the number of children vaccinated with U.S. aid and more. There are even some good outcome indicators: percent change in power blackouts, the number of students finishing school, the degree of confidence Pakistani citizens have in government institutions and the like. However, next to every single one of those output and outcome indicators is the same descriptor: “Target and result: no data.”There are plenty of “examples of success” sprinkled through the report, descriptions of individual projects that produced real benefits. But clearly, USAID’s auditors and its funders in Congress aren’t going to be satisfied with anecdotal evidence alone—nor should they be. Despite the difficulties involved, including the need to hire adequate staff, dangerous working conditions, and a new aid model that stresses working with new, local partners, USAID needs to get moving on putting some real data next to all of those good indicators. But the bottom line is that this report does not mean that aid in Pakistan has done no good.With that understood, here are three suggestions for those in charge of the aid program—not just in USAID but across the dozen-plus U.S. agencies delivering assistance in Pakistan—for how they can fill the current information vacuum with information that’s actually useful for policymakers and the public. Doing so is the best way to avoid this sort of bad publicity in the future:

• Make it much easier to understand the yardstick you’re measuring yourself against. And, even better, do it in collaboration with your Pakistani partners. We have proposed that the United States and Pakistan sit down together and pick just a few indicators of overall development that will be closely tracked and reported on to the Pakistani and the American people. If this is truly, as leaders of the two countries keep calling it, a “long-term partnership,” then measuring success should also be done in concert. Carrying out this exercise will require coordinating the efforts of all of the U.S. agencies involved in aid delivery in Pakistan—but that’s a positive thing, not a reason for inaction. In fact, since Pakistan is not the only difficult country where multiple U.S. agencies carry out different parts of a complex assistance strategy, perhaps the government needs an evaluation framework for measuring overall progress in these situations. (USAID’s own brand-new evaluation policy is not a bad place to start.)
• Tell us where the money is going! Even this report, which contains the most comprehensive information available on how aid to Pakistan is being programmed, has zero information on how much money has actually been spent. Digest that one more time—there is nowhere online where a Pakistani or American citizen could find out how much aid money has actually been disbursed (or to whom). The longer this vacuum of information persists, the more it will be filled with misleading or flat out wrong reporting.
• Please get your web geeks talking to your data people! For a long time now, we have been calling for a website that would present the U.S. aid program in Pakistan in a way that is useful and understandable. The sort of information in this report, while by no means complete, is too good to have trapped in a PDF file hiding in the bowels of some impossible website. (Just try to find the report on USAID Pakistan’s site—we’re not sure it’s even there…). USAID as a whole has made progress in the form of the Foreign Assistance Dashboard, but the information on that site doesn’t yet include everything that’s publicly available (supposedly, it will be expanded in the future—I for one will jump for joy the day project level disbursement data is posted).

The good news and the bad news for the U.S. aid program is that these auditor’s reports come out every three months—one of the terms embedded in the legislation that authorized the current $7.5 billion aid package for Pakistan. That means there will be plenty of opportunities to fill in the blanks with good hard data and make the next round of news stories positive ones. Or, looking at it another way, if the “no data” labels are still there in three or six months, there will be plenty of fresh opportunities for those in charge of the aid program to end up with egg on their faces.