40 Years On : The Emergence of Big Data

When I started my career in market research 40 years ago, things were much simpler – or at least the data was. Back then, our toolkit consisted of surveys, focus groups, and perhaps the occasional in-depth interview. We asked people questions, they gave us answers and we analysed the data. We could still generate valuable insights but the scope of information we had to manage was limited. It was structured, often linear and relatively easy to process with the methods available at the time.

Fast forward four decades and we now find ourselves in the age of Big Data – and what a transformation it has been! Data has become not just a by-product of research but a constant stream flowing from multiple sources. We no longer rely solely on what consumers tell us in surveys; now we observe their actions and behaviours in real time, across countless touchpoints. From social media activity to transactional data, online behaviour and even in home on our smart devices, we’re surrounded by data. And it's not going away anytime soon.

But here’s the thing: having more data isn’t the same as having better data. The real challenge isn’t just collecting it – it’s making sense of it all. It’s about finding meaning and connections in what can sometimes feel like an overwhelming amount of information. And this, in my view, is where the role of the insights professional has evolved the most.

In the early years, we worked with one or two data sets, usually survey-based, to develop insights for our clients. Today, insights professionals are often tasked with blending five or six data streams, sometimes in real time. The art is in integrating these streams and uncovering the bigger picture without getting lost in the weeds. It’s easy to be seduced by the volume of data available but more data doesn’t always lead to clearer insights. In fact, it can muddy the waters if not handled with care.

One of the most significant shifts I’ve seen in my career is the rise of data visualisation tools. Back in the day, we would present our findings in dense reports, filled with tables, graphs and pages of text. These reports could be highly informative but they weren’t always accessible or engaging for a broader audience. PowerPoint came along and became the default word for long boring 100 slide presentations that lost the audience immediately.

Today, the ability to tell a story with data is as crucial as the research itself. Tools like Tableau, Power BI and customised dashboards have transformed how we communicate insights, allowing us to simplify complex data and present it in a way that’s intuitive and easy to understand.

No longer is it acceptable to rely on a series of static charts or pages of numbers. Stakeholders want insights that are clear, concise and – above all – actionable. Data visualisation helps us do that by focusing attention on the key takeaways and enabling decision-makers to see the story behind the numbers.

And let’s not forget the role of data science. When I first started, coding, algorithms and machine learning felt far removed from our world of consumer research.  Today, they sit at the very heart of it. The lines between data science and traditional market research have blurred, with many insights professionals now required to have a working knowledge of programming languages like Python or R. It’s no longer enough to rely on statistical analysis alone – being able to leverage machine learning, AI, and predictive models has become a vital part of what we do.

The rise of data integration is another key development. Where we once relied on structured, survey-based data alone, today’s insights often involve pulling together a variety of different sources, from customer databases to social media sentiment analysis, and even unstructured data like video or audio. This gives us a much more holistic view of consumer behaviour but it also requires a broader skill set. Modern insights professionals need to be as comfortable working with raw data as they are developing strategic recommendations.

But with all this change, one thing has stayed constant: it’s still about people. At the end of the day, regardless of how much data we have or how advanced our tools have become, the goal remains the same – to understand what makes consumers tick. The technology may have evolved but human behaviour is still at the heart of what we do. In fact, with more data than ever, it’s even more important to focus on the human stories behind the numbers.

So, to all the insights professionals out there, particularly those working with brands: don’t be overwhelmed by the explosion of data. Yes, the landscape has changed dramatically but at its core, our job remains the same. The tools and techniques we use may have evolved, but our mission – understanding people and helping brands make better decisions – is as relevant as ever. It’s been a fascinating journey, and I, for one, can’t wait to see where it goes next.

Key Takeaways:

• Big Data: The challenge isn’t collecting it, but making sense of it.
• Integration: Insights today involve pulling together multiple data sources to see the bigger picture.
• Data Visualisation: Storytelling with data is now an essential skill.
• Data Science: Knowing how to use AI, machine learning, and predictive models is becoming part of the insights toolkit.
• Human Focus: Despite all the changes, insights remain about understanding people.