A widely-reported CDC study found that 4% of Americans had drunk diluted bleach to prevent Covid-19. This was based on an opt-in survey of 502 adults.

 

As Courtney Kennedy, Director of Survey Research, Pew Research Center, tells us in a thought-provoking presentation at the AMEC Virtual Global Summit on Measurement, it quickly became obvious that there was more to this horrifying statistic than meets the eye.

 

It had come about due to bogus respondents – the mischievous, inattentive, and careless that are often present in an online opt-in sample.

 

The bleach statistic is representative of a much broader problem with opt-in surveys. They  suffer from a whole range of potential problems: people from outside the required territory taking part; survey bots; click farms; and trolls.

 

So Pew Research Center’s methodologists examined whether variability in methods of recruiting online respondents produces any consequences for data quality. They aimed to find out: How much do bogus respondents impact the quality of survey data? What share of the sample can we expect to be bogus? And, if there are bogus cases, are they answering at random, or are they systematically biasing the estimates in favour of a certain viewpoint?

 

In a study in Spring 2019, Pew administered a common questionnaire via six different online survey sources: three widely used opt-in survey panels, one crowdsourced platform; and two online probability-based panels.

 

They obtained around 10,000 completed interviews from each source, from which Pew manually coded over 375,000 open-ended answers into four categories: responsive answers; non sequiturs; gibberish; or blank/don’t know/ Refused.

 

The results revealed that across the online opt-in samples, an average of 5% of the responses were bogus. Rates went as high as 7% from the crowdsourced website, and as low as 1% from those recruited offline.

 

But are these bogus respondents simply adding noise or are they aiming to bias estimates in a certain direction? Courtney reveals that it is the former – adding a positivity bias about whatever subject they touch on, rather than providing cohesive viewpoints that make sense.

 

So, could these bogus respondents seriously affect your topline estimates? Sadly, yes – they could produce a 5% difference to your survey results, particularly in terms of inflating approval ratings.

 

Courtney adds one final warning: “Do common data checks work? No, the vendor checks in places are not fully effective, so I highly recommend you do your own checking for data quality.”