Measuring Corruption: Still Hard after All These Years

Last month, I spent an enjoyable if fractious couple of days in Vienna, taking part in a meeting of the UN Office of Drugs and Crime 2nd Task Force on the Measurement of Corruption. The task force is meant to help the office “define and refine methodologies for measuring corruption and to formulate proposals to establish a comprehensive, scientifically rigorous and objective framework to assist States Parties in measuring corruption,” for approval by UN member states. Despite the best efforts of document drafters and participants, and this being the second meeting of the second task force, I’m not sure we managed that. But for me the session was a fascinating refresher on debates over corruption measurement, which left me thinking the problem might not be the task force but the terms of reference.

I’m definitely with the terms of reference of the task force that you need a framework when thinking about measures of corruption, in that one country-level measure will never capture its infinite variety. I’ve gone on ad nauseam about why I think “expert opinion” and “mashup” indicators are pretty useless, and won’t belabor the point here, but one reason is that corruption can’t be measured by a single indicator because it involves so many very different forms of activity: taking bribes for anything from ignoring crimes through awarding contracts and licenses, extortion using your official power, profiting on inside information obtained from your job, doing favors for a company in hopes of future employment, collecting payments for ghost employees, the list is long and varied. And, at least where we can sort-of-measure, it appears that there is a lot of variation across countries in which bits of government administration are more corrupt when it comes to surveyed bribe levels.

The limitations of survey measures of corruption

What indicators could you put in a framework, then? For direct measures of corruption that can be regularly collected and might be cross-country comparable, you are pretty much limited to surveys and legal investigations or convictions.

Surveys have their problems: they’re best at providing measures of bribe direction, frequency, and volume and not so good at capturing much of the rest—in particular, they can’t easily catch the kinds of grand corruption that involve very few players. There’s a very weak correlation between people saying they think corruption is a problem in their country and the proportion of people who report paying bribes in that country, suggesting the person in the street thinks there’s a lot more to corruption than bribery, and I agree with the person on the street. There is also a reasonable level of noise even in better surveys, especially when it comes to the size of bribe payments.

The much bigger limitations of investigation and convictions measures….

But I’d argue surveys are still a much, much, much better measure of the level of corruption than legal investigations or convictions. What might a comparatively high level of corruption convictions represent? It can be about a high level of underlying corruption, a high level of effort to bring corrupt people to justice, a high level of capacity to investigate and prosecute, and/or a high level of effort to use corruption laws to hound political enemies. I’d hazard it is more often about a combination of the latter three than the first. And convictions are an utterly dire measure when it comes to being “actionable”: if you want to look good on it, simply shut down corruption prosecutions. 

In the best of cases, we know that investigations and prosecutions will probably capture a small minority of corruption, with considerable variation in how much they miss. Take World Bank financed projects as an example. If you look at investigative cases that have uncovered corruption, they involve about 0.1 percent of lending volumes. But if you survey firms in the countries where the World Bank works, the proportion of firms reporting they are expected to bribe to win a government contract is 21 percent. Maybe World Bank procurement oversight is that good (it really is pretty good), maybe firms are over-reporting, but maybe also some considerable proportion of procurement-related corruption is going undetected.

…And the even bigger problems with “indirect” measures

What about indirect measures, sometimes known as “red flags”? Things like non-merit-based hiring, or low transparency, or single bidding on contracts? The problem is that it isn’t always possible to tell if they are good indirect measures, because we lack direct measures to test them against. Even when we do, simple correlation is hardly enough to demonstrate their quality: that takes sweeping the issue of causality under the table (which we shouldn’t), and ignoring that how good they are and where they are good likely depends on a wider set of circumstances (which we shouldn’t, either). And if convictions are almost certainly a significant undercount of actual corrupt acts, indirect measures often face the opposite problem. A while ago I Looked at a sample of World Bank-financed contracts, some of which had been referred for investigation on the grounds of corruption, and found procurement “red flags” were ubiquitous, but not more common in the contracts there was reason to believe were corrupt.

If I had to choose one single indirect measure of the impact of large-scale corruption in a country, it would be the income share of the top one percent. Considerable income inequality is a sign of governments not intervening where they should to fix market failures (think: allowing Standard Oil in the US to develop a monopoly, massively enriching the Rockefellers) or intervening where they shouldn’t to create market failures with concentrated benefits (think: massive sole-sourced contracts to defense companies or the Mickey Mouse Protection Act). A lot of the resulting wealth is perfectly legal, but it still involves misusing the power of government to benefit a few well-connected people at the expense of the many, which I’d argue isn’t a bad definition of what most people think of when they think about corruption. 

The share of the top one percent isn’t as strongly correlated with GDP per capita as most corruption measures, but maybe that’s a feature not a bug. For what it is worth, out of the 118 countries with 2015 data on income shares, the Netherlands, Slovenia, North Macedonia, Iceland, and Belgium come out looking the best while Chile, the Dominican Republic, the Central African Republic, Mozambique, and Mexico look the worst. Colombia and the US are neighbors in the rankings.

I think the share of the one percent is probably a better indirect measure of large-scale corruption than days taken to obtain an operating license or number of convictions of senior officials, as it might be, but because we don’t have good measures of the extent of large-scale corruption (or even an agreed definition detailed enough to design a measure around), my evidence for that is extremely weak. I’d bet trying to include it as a preferred indirect corruption measure at the UN would raise plenty of complaints, with a number of very legitimate arguments against it. That’s going to be—indeed probably should be—a problem for any measure that isn’t specifically addressing the extent of a particular corrupt act.

Image

Data Source: Our World in Data, from underlying sources the World Inequality Database and the Penn World Tables (2015 data). Income share is pretax.

Embracing the known unknowns

But the bigger point is that we have to accept  there sometimes simply aren’t even somewhat adequate direct measures of corruption that can be regularly collected and compared across countries. And because of that we have to accept we don’t know a lot about what works where, and so we should be very cautious about assuming particular “indirect measures” or “red flags” will be reliably associated with particular forms of corruption (even if improving them might be a good thing regardless). So, a “comprehensive, scientifically rigorous and objective framework to assist States Parties in measuring corruption” would by necessity repeat that “objectively, there’s no good measure” across a range of types of corrupt activity: listing gaps as much as it lists measures, and admitting evidence for links between indicators is weak rather than assuming it is strong. In consolation to the member states who approved the task force terms of reference, understanding better what we don’t know is still surely a sign of science progressing.