It’s no surprise that rich countries outperform poor countries on standardized tests. But if you compare kids with similar household wealth across countries, that gap disappears.
Every three years, the Program for International Student Assessment, or PISA, releases a new batch of standardized test scores comparing 15-year-olds around the world on reading, math, and science. In countries that underperform expectations, such as the United States, the rankings inevitably provoke calls for education reform to imitate the school systems of high performers. (“We should be more like Finland!” “No, Singapore!”) Jon Stewart even mocked this ritual on the Daily Show:
I always feel bad for whatever country is just above America on these lists, because invariably that country is used as a standard for just how far we have fallen as a people. Thirty-sixth, beneath the Slovak Republic. I mean, those f***ing people eat their own vomit.
Setting aside the Slovakian-American math comparison, richer countries generally do better on PISA. A lot better. Indonesia was the poorest country to administer the test in 2012, and Norway was the richest. Their performance gap is huge: Indonesian students who score extremely well, with math scores at the 90th percentile locally, would still be in the bottom half of Norwegian students. Zero Indonesian students scored high enough on reading to clear the 90th percentile in Norway.
So does that mean poor countries such as Indonesia, Peru, or Vietnam should be looking to rich countries like Norway and the United States for education policy advice? The answer depends, in part, on how much of the test-score gap between rich and poor countries you think reflects superior school systems in rich countries and how much is due to the simple fact that American and Norwegian kids are a lot richer and have innumerable advantages both in and out of school due to that wealth.
What if you compared Indonesian kids with similar wealth levels? Would Indonesia still underperform? This is the question my CGD colleagues Amanda Beatty, Lant Pritchett, and I are currently exploring in a new paper. We started off by computing the relationship between wealth and test scores within each country, as seen in the graph below. [1]
There’s a lot of heterogeneity across countries. In the United States, the wealth gradient in test performance is quite steep, while in Norway it is very shallow (in line with your preconceptions of America and Scandinavia). And some countries, like Vietnam, are simply off the charts with spectacular performance given its relative poverty. But overall, it turns out that on average rich and poor countries are roughly on the same upward sloping line relating household wealth to PISA scores. What does this mean? Crudely put, Indonesian and Peruvian students score about where you’d predict them to score in the United States given their household wealth. Poor countries don’t do any worse on PISA than most OECD countries once you adjust for their socio-economic demographics, and some poor countries like Vietnam do considerably better.
(Note that we’re careful not to give any causal interpretation to the relationship between wealth and scores within countries here. Our argument doesn’t require that. It’s entirely possible — indeed very likely — household wealth proxies for lots of other factors: including household factors like nutrition and parental involvement, as well as within-country variation in school quality. The point is simply to ask how a child with a given wealth level is expected to score in country X, given all the advantages and disadvantages they’re likely to have at home and at school.)
To make this a little clearer, we decided to rank countries not by their average score, but by the predicted score of kids with the same wealth level in each country. (Basically, draw a vertical line through Figure 1 at a wealth score of 50, and see where countries intersect that line.) Comparing apples and apples, which countries do best?
Here the results are even more striking: If you compare students at the global median of household wealth, the test-score gap between rich and poor countries essentially disappears. There’s no correlation between a country’s average wealth and the test performance of students in that country who are at the global median.
There’s an interesting analog here to work on global income inequality. In both cases, after controlling for everything we can control for, the gaps between countries are still huge, as Michael Clemens, Claudio Montenegro, and Lant Pritchett documented in an earlier CGD paper on earnings. Likewise, Branko Milanovic has demonstrated that the vast bulk of global inequality is between countries — it’s where you’re born that largely determines your income. Similarly here, the gaps between countries remain huge even when comparing children with similar socio-economic backgrounds.
The enormous difference is that in contrast to income, you wouldn’t necessarily maximize your test scores by picking to go to school in the richest countries. A student with global median wealth in Turkey performs much better on reading tests than an equivalent student in Norway. And Vietnam beats out the best performers in the OECD such as Japan and Canada.
There is one tempting interpretation of the first graph above that we would caution against. If Indonesia, Peru, and the United States are all on the same line relating wealth and test scores, you could argue that means economic growth is secret to better education in poor countries. As they move up the wealth gradient, scores will rise. As Ludger Woessman and coauthors have shown for OECD countries, wealth has been associated with higher test scores within countries for a long time. But as OECD countries have gotten richer, scores haven’t gone up. We should be careful when converting cross-sectional correlations into time-series forecasts.
So why then do some systems deliver so much more learning than others? We have very little idea. At CGD, we’re getting ready to launch a new research program on what makes for an effective education system, and how to reform ineffective ones. Most of the countries we’ll be focused on — low- and lower-middle-income countries in Africa and South Asia — don’t appear in the PISA sample at all. As we start this research program, it’s daunting to realize how little we know not only about how to make reform happen, but even which systems perform well and which ones don’t. One thing is clear though: we shouldn’t assume rich countries hold all the answers.
[1] Technical footnote: The biggest challenge in this exercise was the potential for a high degree in measurement error when asking schoolchildren about their household wealth. This turns out to be crucial. Measurement error will tend to underestimate the slope of the relationship between wealth and test scores and, by failing to fully account for the wealth-score gradient within countries, exaggerate the cross-country relationship between wealth and scores. The regressions underlying all the results shown here an instrument variables approach to minimize this problem. More details forthcoming soon in the full paper. If you’re curious in the meantime, see the Stata code posted here.
Disclaimer
CGD blog posts reflect the views of the authors, drawing on prior research and experience in their areas of expertise. CGD is a nonpartisan, independent organization and does not take institutional positions.
