Video open ends



Conversational Chat

Thought Leadership


Campaign Assessment

Concept Testing

Our Community

The Angus Reid Forum

A proprietary opinion and insight community we've built from the ground up.

About Us

Learn about the Angus Reid Group


Work with Angus Reid

Our Team

Learn about our leadership team


Here's what we're thinking

Media & Press

News coverage and press releases

Latest Blog

The Science and Art of Insights

This blog post originally ran in QCRA Views. You can read the unabridged version, with...

Is your data reliable?

Trusting your data sources is the first step.

Sector Expertise

We're experts in your business.

Case Study

SID LEE: The Belong Effect

Tapping into Gen-Z and Millennials to Understand the Impact of Community


Three questions that determine the value of your research

You can protect yourself by asking a few basic questions of any potential sample provider

Asking good questions is an art. But getting answers you can trust is all science.

Science rests on getting the same answer when you ask the same question. If you ask the same question of three different but identical samples and you get three different answers, what have you learned? You’ve learned you have a problem.

Unfortunately, lack of reliability is a major concern in the insight world these days. In the old days we had fraudsters, click farms and bored people, all vying to earn a couple of quarters per survey. That was bad. But now with Artificial Intelligence (AI) driven bots joining the sample exchange party, it’s much, much worse—it’s a free-for-all. And data reliability is suffering drastically.

Fortunately, you can protect yourself by asking a few basic questions of any potential sample provider:

  • Where did the people in the sample come from?
  • What is the evidence of the reliability of data from this sample source?
  • What is being done to detect and eliminate AI-driven bots and other bad actors?

Where do they all come from?

The source of the people taking your survey is the first question because it is foundational. Get the wrong people and you’ll have a misleading answer.

Is the sample recruited from people who are part of a loyalty card scheme—and therefore biased in where and how they shop? Or is the sample coming from unknown people who didn’t qualify for other surveys and got bounced to yours via a sample exchange? Or are they even people at all? Did someone send bots to your survey so that they could scoop up the honoraria?

The more you know about a respondent and how they were recruited into a panel, the more certain you can be that they are a real person whose answers you can count on. Ask hard questions the sample source and don’t settle for half answers. The quality of your work hangs in the balance.

At the Angus Reid Group (ARG) we run carefully targeted advertising to attract people to join the Angus Reid Forum. They are carefully profiled and invited to take a series of engagement surveys—which we scrutinize carefully—before they are accepted as a full-fledged member of the community.

Show me the evidence  

Ask potential suppliers for evidence of the reliability of their sample. Ask them to share data that shows that, if you keep asking the same question, you keep getting the same answer.

At the Angus Reid Group we run omnibus surveys on an ongoing basis. We rotate through a battery of simple questions over time so that we end up with repeat measures for each question. You can see from this graph that the answers are consistent—with any variation being within what you’d expect with nationally representative samples of 1500 adult Canadians. Those straight lines are boring, but they are boringly reliable.

Ask any potential supplier to provide similar evidence of reliability. If they don’t have it, ask yourself why not?

As the old proverb says: “Trust, but verify.”

Battling bots

Speeders and straight liners—people who answer the same way over and over—are easy enough to detect, as are people who answer open-ended questions with “asdfg” and “yuio.” But AI-driven bots have changed the game. Their patterns of answering need not be so simplistic, and that makes them hard to identify.

Ask potential suppliers what tools they are using to detect bots, and how they are changing as bots evolve too.

At ARG we’ve developed tools that employ advanced pattern detection algorithms to identify bots, as well as knowledge traps that bots can’t identify, hidden questions that bots can “see” but humans can’t, time stamps to identify speeders, analysis of open ends that includes AI-driven lookups for duplications and web-based sources, tools to identify duplicate IP addresses, and an ever-evolving set of approaches that allow us to take a multi-dimensional look at whose answers are to be trusted, and whose are to be deleted.

Research rests on reproducibility

The quality of your sample is what determines the usefulness of the research you do. With so much hanging in the balance, you can’t afford not to ask potential sample providers these three questions:

  • Where did the people in the sample come from?
  • What is the evidence of the reliability of data from this sample source?
  • What is being done to detect and eliminate AI-driven bots and other bad actors?

With good answers to these inquiries, you can be sure the answers to your questions will be reliable too. Trust, but verify.

Suggested read

Empowerment Through AI

Empowerment Through AI

A quiet revolution is reshaping market research, driven by the silent engines of Artificial Intelligence (AI). This transformation isn't just...

read more