How to Avoid Indicator Scandals: Three Ways to Fix the Doing Business Index

On Friday, the World Bank’s chief economist, Paul Romer, told the Wall Street Journal that the Bank unfairly influenced its own competitiveness rankings. He highlighted the case of Chile which suffered lower rankings on the Doing Business index during the Bachelet administration versus the Piñera years, and recalculated these rankings on his personal blog. Today, he issued a clarification of his views.  

In a statement, the World Bank announced that it would carry out an independent assessment of the Doing Business indicators. 

This led to a flurry of tweets about the shortcomings of the Doing Business index. Former World Banker Martin Ravallion writes:

Whether or not there was deliberate manipulation of its Doing Business Indicators, the Bank should better acknowledge the huge uncertainty about this index especially in the middle 80% of country rankings. Only top and bottom seem reliable.https://t.co/ylJbt1jkaO https://t.co/1clq4Y3HY1

— Martin Ravallion (@MartinRavallion) January 14, 2018

Brian Levy responds:

The narrowness of the questions also reflects normative bias. As we are painfully learning, a 'good' business environment is dependent on broad social legitimacy - which in turn often incorporates "restrictive" regulation.

— Brian Levy (@Brianlevy387) January 14, 2018

And Dani Rodrik says:

I never put much importance on WB doing business rankings and found governments’ focus on them misplaced. Same with WEF competitiveness rankings. https://t.co/iWLfSN6qqd

— Dani Rodrik (@rodrikdani) January 13, 2018

We are not overwhelmed by Romer’s observations. Everyone knows, or should know, that there is a substantial degree of imprecision on all governance or economic-management-related ratings and that even small differences in ratings can sometimes correspond to big rankings differences because of clustering of scores. The biggest differences in rankings for Chile (by Romer’s calculation) are around 10 places, which is not a big deal even for a fixed methodology. If the difference had been say a change from a ranking of 34 to, say, one of 90, we would have been more impressed! The error range for World Bank’s Worldwide Governance Indicators (WGI)—one of the few to publish error estimates—is revealing. For a mid-level country like Ecuador, the error ranges for the individual indicators in various years correspond to rankings differences of between 20 and 40 countries!

However, this latest kerfuffle does present an opportunity to improve the Doing Business index. Here, we pose three fixes to avoid future scandals and improve the index:

First, the World Bank should acknowledge that any system of ratings will inevitably embody a range of uncertainty—the Doing Business indicators can be improved by estimating standard errors for each index subcomponent and for the overall index. This is difficult to do for expert-based indicators because it requires several independent assessments, but it is not impossible. For example, Gelb, Ngo, and Ye (2004) estimated that the standard error for the World Bank’s Country Policy and Institutional Assessment (CPIA) ratings was around 0.25 for measurements of policy reforms on a scale of 1 to 6. A related point is that Doing Business has changed its methodology over the years but has not documented these changes very well (for example, in area of labor markets where it was accused of a right-leaning agenda by its anti-union stance or on tax where it rewarded countries with lower corporate tax rates). An assessment carried out in 2008 by the World Bank’s Independent Evaluation Group revealed changes to many countries indices and rankings, some far greater than Romer's estimates for Chile. Clear explanations with links to changes in underlying methodology will likely lead to fewer accusations of data manipulation.

Second, the Bank needs to ask itself whether the Doing Business scores are unduly influenced by the degree of World Bank Group engagement in the private sector agenda. This could be because there is more known about the business climate or because the Bank defines certain countries as “good guys” as they engage—or appear to engage—on private sector reforms. A rating can be skewed, perhaps unconsciously, by individuals with particularly favorable or negative views on a country or its current government. We have no evidence to support this for any particular case, but there has long been sensitivity to this possibility for the CPIA, which is why the process of estimating the CPIA embodies layers of independent checks by units that are not part of the operating units of the World Bank Group. 

Third, the Bank must consider the value of multiple independent estimates, even though this is hard to do for expert assessments. The CPIA exercise mentioned above was only possible because the ratings of each organization (World Bank, African Development Bank, etc.) were confidential at the time, so that the assessments were truly independent of each other. Independent assessments would be hard to make for the Doing Business indicators, but perhaps some pilots could be done on a few countries. One possibility is to use other indicators to spark questions (comparing Doing Business to Enterprise Surveys to ask—is Country X really that good? What are the differences between de facto and de jure estimates?)This can help to flag flagrant outliners.  

In the end the World Bank has to rely on its own judgment to produce a more robust and more clearly communicated index of the business climate. Last week’s events present an opportunity to do just that.