I love macarons. When made correctly, macarons have a glistening candy shell and a nougat-like, chewy center. They’re a favorite of young and old around the world thanks to their fun colors and beautiful contrast of textures and flavors. The only downside is that macarons are extremely difficult to make. Believe me, I’ve tried. Many times.
A survey is like a macaron — a delicate process with a low threshold for error. Unless you are trained to look for weaknesses within a study, everything can appear to be deceivingly in order. Poorly designed surveys are often difficult to identify without proper scrutiny and, unfortunately, a lot of poorly-made data finds its way into the public conversation, news and social media.
As PR and marketing professionals who use surveys for brand storytelling, creative content and thought leadership, we have to make sure that we are being delicate with data. But in a sea of mismade macarons, how do you know which data to trust? And how do you know when you’re interpreting it correctly? I’m here to give you my recipe for handling data so that your next survey turns out like a sweet treat and not a hollow mess.
1. Your Audience Matters
My first step is always to look at who was surveyed. I learned from a mentor once that credible data comes from credible sources. Pay attention to these factors when building out your audience to ensure you’re getting the richest data possible:
Who?
Ask the right people. For example, don’t survey renters when you’re looking for data from homeowners. You’d be surprised how often I see this seemingly obvious step go overlooked, and how often the research has to be done over again.
How many?
In general, larger sample sizes are always better, and if you’re ever wondering about what size a certain sample should be, there are a plethora of online tools available. Qualtrics and Pollfish are some of my favorites.
Where?
This is one where many folks trip up. Once you’ve decided on your “who” (audience) and “how many” (sample size), look at your demographics — are they balanced? I’ve seen audiences that were supposed to be representative of a large demographic, only to find out that the respondents were 90% female, or all baby boomers, or from only one city in the target country. Look for the words “census targeting” or “representative,” and keep an eye out for age, race, ethnicity, geography and identifying gender in a sample.
2. Better subgroups, better data
We love to look at subgroups within a population because differences between them often hold the best insights. It’s important to remember to cite those more specific audiences as clearly as possible when you’re narrowing your focus. For example, “unemployed women who quit their job” is a different group than “unemployed women.” Keep a close eye on how subgroups are labeled to avoid confusion.
One other key point is to remember the “how many” rule: when you are honing in on a subgroup, it’s still important to make sure that your audience size doesn’t get too small.
3. Don’t push a narrative; pull from the data
We all love a good story, but sometimes in our eagerness we force a narrative rather than letting the data speak for itself. Watch out for these storytelling traps:
Question bias
There are multiple ways in which a survey can include biases, such as leading questions, assumptive questions, overly complicated questions, or double-barreled questions. In general, simple questions with a wide variety of answer choices are best. Take a look at the following examples of bias-ridden questions in a survey on baking:
Leading question: “Do you enjoy eating delicious macarons?” Avoid using subjective adjectives.
Assumptive questions: “When you bake macarons, do you enjoy doing so?” Not everyone bakes macarons, also, avoid yes/no binary answer choices.
Double-barreled: “How much do you enjoy eating macarons and baking them?” This is two questions in one, never a good idea.
Inferring is your enemy
If 92% of people in a survey say they love eating macarons, that must mean that around 92% also love almond flavoring (the most common flavor of macaron), right? Though that could be the case, it wasn’t asked whether people liked almond flavoring, so it would be imposing further than what the data shows. When in doubt, stick to the exact wording of the question and answer choices for data statements. Let interpretations and opinions follow but remain separate from the factual source.
Happy baking
The ingredients that go into a macaron don’t differ much between recipes, but the process is where things can go wrong. In the same way, research is an invaluable tool for PR and marketing professionals with similar basic ingredients that require the proper attention and care. Data, and the valuable insights we can pull from it, will always be helpful in informing company decisions, shaping brand strategies and providing fuel for content. The above are by no means an exhaustive list, but can help make sure that the data you’re using is credible and reliable.
