A Question of Trust

The golden age of the online survey is well and truly over. A mixture of fraudulent  responses and the shedding of senior expertise within agencies, means that it's too much of a risk to make business decisions based solely on survey data.

Bot networks exploiting panel based survey research is a well-known channel of fraudulent data but real panellist giving made up answers is a serious problem that the industry is only just waking up to.

Is size everything?

How has the industry got into this situation? For far too long, research providers have been concerned more about the size of their panel rather than its quality. It's a simple numbers game - the more respondents you have, the more surveys you can field and the more data you have on people, including their likes, dislikes, lifestyles and  attitudes. And this had driven acquisitions and panel building with the message that size really does matter when it comes to panel.

But actually size isn't everything. Quality is. Providers are very happy to tell you how many people are on their panel but less happy to tell you about how many they can actually verify as real people. If you can't prove that panellists are who they say they are, then how can you be sure that their answers are true?

Fraudulent panellists are not  solely motivated by money - let's face it, nobody is going to get rich filling in online surveys. No, what really motivates them is creating a false narrative around their identity and lifestyle that is reflected in how they answer questionnaires. Sometimes their inconsistencies can catch them out but more often than not they go under the radar because agency quality control systems are geared to catching speedsters and straightlining rather than unconsidered and unreal responses.

And because these people sign up to multiple panels, the problem is amplified within the industry. 

Dilution of expertise

At the same time many market research agencies have dispensed with senior staff in order to reduce costs. This has compounded the problem. Senior staff have the experience to spot anomalies in the data, to question whether the findings makes sense and to push back on things that don't look right. This expertise is being lost and consequently agencies are becoming too junior in composition to spot fraudulent data.

The need for verification

So what can be done about it? Obviously, you can set traps within surveys to expose inconsistencies but the most effective thing to do is to stop these people joining in the first place by making it much harder to join.

A system of verification such as showing government ID (passport, driving licence or voter ID), plus an in-home sign up interview (including the collection of key demographics) would simultaneously deter fraudsters and make panels more believable. We will end up with panels being a lot smaller and the sign up process being more time consuming but at least they would be believable.

Ultimately, clients have to be able to trust the data. At the moment making business decisions based on survey data is just too much of a risk.