Simanaitis Says
On cars, old, new and future; science & technology; vintage airplanes, computer flight simulation of them; Sherlockiana; our English language; travel; and other stuff
ARE SURVEYS LEGIT?
THE PHRASE “According to a Pew Survey…” typically instills confidence in the conclusions that follow. However, two ex-Pew researchers have reported on controversial analyses suggesting that some broad-range surveys may not be legit.
The brouhaha is discussed in “Many Surveys, About One in Five, May Contain Fraudulent Data,” by John Bohannon, Science online, February 24, 2016, also in the March 4, 2016, issue of Science magazine, published by the American Association for the Advancement of Science. Bohannon might be remembered for another Science tidbit reported here: his spoof paper used to assess the referee process of pay-to-publish journals.
Survey data collection can be challenging anywhere in the world. Image from Science, March 4, 2016.
A challenge with any survey is the legitimacy of its data collection. At a Washington, D.C., meeting on this topic in February, Michael Robbins, University of Michigan at Ann Arbor, Princeton University, and Noble Kuriakose, SurveyMonkey, presented results indicating that perhaps more than a quarter of international surveys are marred by data fabrication.
The problems, especially in developing countries, can be manifold. Communications might be primitive. Face-to-face encounters may involve dangerous environments. To avoid risk, save time or just out of less than full initiative, interviewers may be tempted to “curbstone,” sit on the curb and simply invent data by duplicating previous responses.
Sniffing out this practice is the point of the Robbins/Kuriakose methodology, by identifying highly similar responses from multiple respondents.
What’s “highly similar”? The researchers adopted 85 percent as a cutoff. Bohannon offers an example of genuine data: “In a 100-question survey of 100 people, fewer than five would be expected to have identical answers on any 85 of the questions.”
Robbins and Kuriakose concede that the 85-percent rule need not apply to all surveys. Some questioning, by its nature, tends to produce more uniform responses. However, for large-scale assessments, Kuriakose notes, “this is exactly the appropriate method for detecting fabrication.”
Image from Science online, February 24, 2016.
Applying this methodology to more than 1000 public data sets from international surveys, the researchers identified about 20 percent with fabricated data; that is, one in five surveys. See “Don’t Get Duped: Fraud through Duplication in Public Opinion Surveys.”
Not surprisingly, the results were dismissed by Pew officials, who claimed the technique is prone to false positives (that is, non-existent flaws). The Robbins/Kuriakose paper was accepted for publication in December 2015, but Pew asked them to withdraw it.
The pair refused. Indeed, they refocused their test on 309 Pew international studies and found a higher failure rate, 30 percent.
Pew posted its own analysis online. Among alleged shortcomings addressed were not accounting for the number of survey questions or the number of respondents.
Observes Bohannon, “Now that both sides have hashed it out publicly, the community is digging into the issue.”
Who says science is dull and boring? ds
© Dennis Simanaitis, SimanaitisSays.com, 2016
Simanaitis Says 