Measuring the Impact of Open Contracting

Last week, the Open Government Partnership held its annual Global Summit in Ottawa. As well as celebrating the unprecedented level of transparency Canada has achieved regarding the pattern on Prime Minister Justin Trudeau’s socks, the Summit highlighted the immense progress made by the open contracting movement over the past few years—with numerous sessions reporting out successes and discussing extensions and improvements. Open contracting champions the publication of documents and data around the whole government procurement process from tendering through completion reports and it has become a major focus of government commitments around improving transparency at the Open Government Partnership.

In a new CGD working paper, Artur Kovalchuk, Mallika Snyder and I try to measure the impact of one of the most significant open contracting reforms worldwide, which took place in Ukraine in the last few years. Since 2015, the country has overhauled its procurement system, including the introduction of an e-procurement platform called ProZorro. Beyond digitizing the procurement process and introducing e-auction as the default tendering type for all major procurements in all government entities, ProZorro and underlying legislative reform introduced considerably greater transparency by adopting the Open Contracting Data Standard and ensuring information was available for small (below threshold) procurements which had previously been unrecorded.

The ‘unrecorded’ bit turned out to be a first important problem in our effort to measure the impact of the reforms. We wanted to compare similar tenders before and after the reform to see how outcomes like number of bidders and final prices compared to initial cost estimates changed. But, even with a lot of work we could only get data on larger, ‘above threshold’ tenders for the pre-ProZorro period that were issued competitively, so we had to limit our analysis to these larger ‘always competitive’ tenders. That misses a big effect of the reforms: the proportion of Ukraine's government procurement budget that was awarded competitively increased from 25 percent of procurement value in 2015 to 70 percent in 2017. Given evidence that more competitive tender procedures tend to produce better outcomes, the impact of that change alone will likely have been considerable—but we couldn’t evaluate it.

(My co-author Artur Kovalchuk spent many hours dealing with the fact that, before ProZorro was introduced, even data on above-threshold procurements was only available in PDFs. He merged data sources and scraped files to produce a large and comparable dataset of competitive procurements from the pre-ProZorro period to match with ProZorro data. We’ll publish the data soon so that others can run their own analysis—Artur’s work is a real public good.)

A second significant problem we faced with the analysis involved the resulting dataset’s suitability for the analysis we wanted to run. (My other co-author Mallika Snyder took the lead on data analysis and uncovered the problem.) We tried to use an econometric approach (regression discontinuity) that would allow for making stronger casual statements about the impact of the reforms—sadly the data did not cooperate. We were left with averages and regression results that illustrate changes between the pre-ProZorro period and the period after its introduction for ‘above threshold’ contracts that we hope are still informative, but not as conclusive as we had wanted.

Our results suggest that Ukraine’s reforms weren’t associated with an increase in the number of firms bidding on contracts that were large enough to be competitively bid both pre- and post-ProZorro. But the post-reform period did see a wider range of firms winning at least one bid, suggesting ProZorro may have helped open up contracting opportunities to more companies. We also found that the Post-ProZorro period saw the gap between estimated costs and final contract prices increase by more than three percent—suggesting cost savings—and procurement being completed 5 to 6 days more rapidly than under the old system.

These results highlight a third problem, however: one of attribution. Was it the impact of ProZorro or other elements of reform that played a role in driving a wedge between cost estimates and prices as well as shortening procurement times? The reform legislation banned contracts where final prices exceeded estimated price, and when we only looked at cases where final prices were less than estimated prices in both periods, the wedge disappeared. Again, the procurement reforms tightened time limits for procurement processes.

It is frustrating that the Ukrainian government didn’t do a better job of collecting data on smaller procurements before reforms, carrying out procurement during reforms in a way that allowed for a regression discontinuity test, and staggering reform introduction so that we could study the impact of legal changes and transparent e-procurement. Shockingly, they clearly weren’t prioritizing the needs of researchers when they rolled out a massive and rapid institutional change covering an activity worth 15 percent of GDP in the midst of a war.

But perhaps as a result, the findings of this paper haven’t shifted my priors regarding e-procurement and open contracting: transparency can help improve outcomes as part of procurement reform. In earlier work, Ben Crisman, and I found that transparency is associated with more competition in World Bank funded procurement, for example. That paper and this one also point to previous literature suggesting better outcomes from transparency. And so I hope next year’s Open Government Partnership Summit in Buenos Aires sees even more commitments to open contracting.