I’m lucky enough to be married to a researcher who grapples with polls and academic polling mechanics daily. When I showed her the particulars of an Ipsos Poll conducted for Thomson Reuters on video games, the results of which Reuters is touting as “Sony’s PS4 tops Xbox One as gamers’ holiday choice,” her takeaway was something else entirely: that the vast majority of people (note Reuters’ assumption that these were all “gamers” per its story title is erroneous) are interested in neither.
The Ipsos Poll was conducted between September 23-27 and involved a survey sample of 1,297 Americans, aged 18 or older, and interviewed online. In it, Ipsos asked half a dozen questions, like which gaming device(s) did the interviewee use to play games. (Response: Wii 23%, Android devices 21%, Xbox 360 20%, PC/Mac 20%, PS3 18%, iOS devices 12% and so forth.) Or how much time in a week did these people play games. (Response: 2 hours 17%, 3-5 hours 15%, etc.)
And then this question: “Which of the following, if any, would you be interested in purchasing?”
26% PlayStation 4
15% Xbox One
5% Nintendo 2DS
3% Steam Box
…and last but not least in that list, “None of these,” which tallied 64%. That — not this notable but comparably nominal difference between PS4 and Xbox One interest — seems like the takeaway to me.
On the other hand, it’s important to be mindful of the survey methodology. This was an online, opt-in survey, for starters. Its precision was measured using something called a “credibility interval.” In this instance, the credibility interval was plus or minus 3.1 percentage points, which sounds pretty darned precise, except that credibility intervals — as Ipsos itself points out in the survey — are not the same as statistical margins of error.
Without delving into the byzantine and confusing world of survey mechanics and math, the rule of thumb for opt-in polls, according to the American Association for Public Opinion Research, is to tag such polls as potentially “subject to multiple sources of error, including, but not limited to sampling error, coverage error, and measurement error.” (Ipsos, to its credit, includes AAPOR’s guidance and precisely those words in the preface to its survey.)
What’s this so-called “credibility interval” worth, then? Does it tell us anything about the validity of the results above? Perhaps, but in Ipsos’ own words, we’re in “Brave New World” territory. The “credibility interval” is a modern means of attempting to reduce error in something like an opt-in, online survey. Given time and further refinement, it may prove more reliable. But for now, with apologies for the cliché, take the results above with the requisite grain of salt.