For Innovations for Poverty Action, Alex Bartik blogs enjoyably about why collecting good survey data is really hard:
For the Informative Advertising and Spillovers project we have some math questions designed as a very rough measure of cognitive ability. To my surprise, many people enjoy these questions. However, they enjoy them so much that often times people’s neighbors or family members will crowd around us while doing the survey and try to help with the math questions. We appreciate the enthusiasm, but we want to know whether the person we’re talking to can answer the questions, not if they have a neighbor who can do so.
That reminds me of something Helen Todd wrote after living and working for a year amid two villages in Bangladesh. At the start of the year, Todd, her husband, and their research team surveyed in conventional fashion the 62 women whose lives they would soon follow with periodic in-depth interviews. For the survey, they asked each woman such questions as whether she had an equal say in decisions about purchase of land and schooling of children, then recorded the answers on forms with tick marks. “Looking back at them a year later, the ticks bore only limited relation to the reality we had come to know. If they had any usefulness it was to demonstrate a cultural idea—the way our respondent felt families ought to behave, rather than the way they actually did.” (Women at the Center, p. 87)And it reminds me of a technique that Mehnaz Rabbani told me about last year when she was a researcher for BRAC in Bangladesh. If she asked a woman for a survey whether the woman borrows from a microcredit group in addition to BRAC, the woman would probably say “no” regardless of the truth, since belonging to more than one group is frowned upon. Mehnaz's colleagues taught her to ask the children, perhaps out of earshot of their mother: “Does your mommy go to the Grameen meetings too?” The little ones became informants, and not merely in the surveyor's technical sense.All of which amounts to yet another caution in interpreting quantitative studies of the impact of microfinance. Maybe borrowers, for example, exaggerate their business success. Then a finding that microcredit boosts profits would really be one that microcredit boosts claimed profits. This caution goes especially for non-randomized studies. If a randomized trial turns up a systematic difference between treatment and control groups in how people answer a particular question, then whatever the difference means, there can be little doubt that the luck of being treated caused the difference rather than the other way around. In non-randomized studies, on the other hand, weird survey answers are yet another potential source of patterns that look like microcredit borrowing causing something when really microcredit borrowing is being caused by something. And all of which is a reason I like qualitative research (when it's high quality).I bet the world's most famous survey statistician agrees with me.
