Do Mobile Phone Surveys Work in Poor Countries?

Update: This blog was updated on 3/11/2015 from the original version.

The days of pushing priorities, pet projects, or expat consultants on countries are coming to a close. Connected and increasingly empowered individuals are demanding a greater say in setting priorities, designing and implementing programs, and assessing whether projects have achieved their desired results. For those agencies that recognize this trend, the question is how to meaningfully and cost effectively engage citizens in real time. This is where the explosion of mobile phone technology comes in.

By illustration, nearly 300,000 Ugandans get a brief survey on their mobile phone every week. The questions typically ask about people’s access to public services or options for expanding economic opportunities, such as buying agricultural land. There are now countless tools like UReport all across the developing world that gather input from thousands of people in a matter of hours. 

However, many question whether mobile phone surveys are truly reliable tools and can accurately reflect the broader population. After all, aren’t they just reaching the urban elite and failing to reach poor people? There have been few studies that have rigorously grappled with this completely legitimate question. 

Along with our World Bank collaborators (Tiago Peixoto, Jon Mellon, and Steve Davenport), we sought to rigorously test whether mobile phone surveys can reliably produce nationally representative samples across a range of country contexts. We focused on four poor countries (Afghanistan, Ethiopia, Mozambique, and Zimbabwe) that have very different levels of mobile phone penetration. We deliberately picked countries where we thought it might work and where it probably wouldn’t; and then, dispassionately assessed the findings.

Here are a few key takeaways of what we found. We will be publishing a working paper soon that will include much more detail, both on the findings and the underlying methodology so please stay tuned for that.

• Including the Poor:  The survey performed well at reaching poor inhabitants. We used several asset ownership questions as proxies for respondents’ income level. In Afghanistan and Zimbabwe, our samples mirrored the underlying population in terms of these measures. While less precise in Ethiopia and Mozambique, they were still quite close. This gives us comfort that we were reaching the bottom of the pyramid, in contrast to common criticisms and truthfully, what we expected to find.
• Reaching Rural Women:  Reaching rural women was a consistent challenge, especially in Afghanistan and Ethiopia. Some of this was due to cultural norms as well as mobile phone ownership and usage patterns. At the same time, we made a few regrettable methodological choices, such as using a male voice for the surveys. We plan to adjust these techniques going forward and test new approaches that may improve our ability to capture more rural female respondents.
• Sample Precision:  After normal weighting methods, sample accuracy was a challenge in the two lower feasibility countries (Ethiopia and Mozambique), with a sampling error of +/- 5 to 7 percent. While these figures are relatively high, the data collected was still meaningful for a range of purposes. On the other hand, the sample was much more precise in Zimbabwe (+/- 2.8 percent). We will explore all of this in much greater detail in a forthcoming post.

Mobile Penetration Rate versus Sample and Population Difference 

Image

Our findings only start to rigorously assess the reliability of mobile surveys. Yet, they suggest that these tools may be a promising way to promote representative engagement with citizens in many places. Not to mention that they are extremely cheap (roughly $10-15k per country). Going forward, we plan to expand this work to other countries, including some additional tough places. Plus, we hope to test new techniques for better reaching under-represented groups, such as rural women. These new country cases and lessons learned will further deepen our understanding of where and how mobile phones can be reliably deployed. But for now, we hope that this work adds to the body of knowledge about these innovative techniques for deepening citizen engagement in poor countries.

*We are thankful to Voto Mobile, particularly Levi Goertz, for their commitment to and unending patience on this research project.