Trick or Truth? Five Survey Mistakes That Can Fool Your Audience

A dramatic survey statistic can be a little like a Halloween costume. From a distance, it makes an impression. Get closer, and you may discover that things are not quite what they appeared to be.

That is a good reason to look behind a headline before sharing it. Who answered the questions? What were they asked? Does the statement match the evidence?

For Halloween, here are five checks to make when survey findings are about to become public. Keep the questionnaire, response bases and proposed copy together as you review them.

1. Publishing a result without checking the question

Before approving a headline, put the original question beside it. Does the headline describe what people were actually asked, including the time period and any conditions?

Imagine a question asking whether someone would consider a service if it were free. A headline saying consumers want to buy the service has dropped a condition that changes the meaning.

This check belongs in the publishing process even when the questionnaire was carefully designed. Interpretation can drift as a finding moves into a presentation, press release or graphic. Our research methodology provides context for reviewing the evidence behind a public statement.

2. Letting the audience disappear from the headline

Write the audience description next to the proposed claim. A result from recent users should not quietly become a statement about all consumers. Findings from professionals in one country should retain that geographic scope.

Check the chart label, caption and surrounding copy as well as the main headline. A qualification in a report's appendix may be lost when a graphic travels on its own.

I would give the communications team approved wording for the audience and study period. That makes the finding easier to reuse accurately and gives reviewers something consistent to check across the campaign's different materials.

3. Sharing a percentage without its base

Seventy percent can sound impressive. It could describe seven people out of ten or 700 out of 1,000. Those findings require very different context.

The relevant number is the group answering that question, not necessarily the total study sample. A survey of 1,000 people might contain a small subgroup with experience of a particular issue.

Keep the response base visible when developing charts and headlines. If a question permits multiple answers, explain that too. Readers should not have to guess why the percentages add up to more than 100.

4. Giving the headline more confidence than the question

Someone saying they would consider trying a product is expressing an intention. That does not establish that they will purchase it.

Similarly, a survey finding that two experiences occur together does not establish that one caused the other. Review the proposed headline against the actual wording and the audience who answered.

My background at Nielsen and Edelman makes this an important part of the work for me. I want findings to be interesting enough to discuss and clear enough that someone can understand exactly what they mean.

For public campaigns, our PR survey research connects the communications question with study design and documented findings.

5. Treating a small difference as a big discovery

One group's percentage may be higher than another's in the sample. Before building a story around the gap, check the group sizes and whether the research supports that comparison.

Plan important comparisons before recruitment. Be cautious about repeatedly dividing the results until an eye-catching difference appears. Some patterns deserve further investigation rather than an immediate headline.

A careful review considers the study design and uncertainty, along with whether the difference is practically useful. It does not assume every pair of different percentages tells a meaningful story.

---

Halloween is a good time for tricks. A research report should make its evidence easier to understand. If you are planning a survey campaign, talk with CITE about the question you want to investigate and how the findings will be used.

---

Andrew Edwards is the founder and CEO of CITE Research. His background includes survey research at Nielsen and public relations at Edelman. He helps organisations use original research for business decisions, PR, marketing and thought leadership.