Image by Editor | Midjourney
Surveys can be powerful tools as long as you’re asking the right questions. But a poorly crafted question can skew your data, lead to ambiguous results, or even drive respondents away.
Whether you’re just starting or you’re a seasoned data scientist looking to refine your craft, understanding these pitfalls will help you create surveys that truly capture the voice of your audience. With that in mind, here are the types of questions you should never ask in your surveys.
1. Leading or Biased Questions
One of the most common survey pitfalls is the inclusion of leading or biased questions. These questions nudge respondents toward a particular answer by using suggestive or loaded language. A question like “Don’t you agree that our customer service is excellent?” can lead to a 15-20% inflation in positive responses since it’s structured to push respondents into agreeing, rather than allowing them to express their genuine opinions.
Another way of introducing bias into questions is using overly emotive language. Using emotive language in a question can cause participants to give more over-the-top answers. There’s a famous psychological study by Loftus and Palmer that shows the impact (no pun intended).
150 participants were shown a video of a car crash. One group was asked how fast the cars ‘hit’ each other, another was asked how fast the cars were travelling when they ‘smashed’ into each other. The ‘hit’ group, on average, estimated the cars to be travelling at 40mph, while the ‘smashed’ group gave an average of 55mph.
When survey questions are phrased like this, the resulting data might not accurately reflect the respondents’ true sentiments. Instead, the survey becomes a tool for reinforcing your preconceptions rather than gathering unbiased insights. When designing a survey, you have to keep your phrasing neutral so your respondents feel free to express their honest opinions without any external influence.
2. Loaded Questions
Loaded questions are similar to leading questions but come with an extra layer of complexity—they embed assumptions or emotionally charged language that can put pressure on respondents. Consider a question such as, “How much do you love our product?” This question implies that the respondent should already hold positive feelings about the product, even if that isn’t the case.
A study by Erčulj and Šulc (2024) found the phrasing of questions significantly influences respondents’ answers. Their research indicated that when questions are loaded, about 65% of respondents (combining those who are “suggestible” and “contra-suggestible”) shift their opinions in response to the assumed premise, whether that’s towards agreement or disagreement.
These types of questions rely on your assumptions and don’t allow your respondents to fully express their true feelings. Instead of making assumptions, a well-designed question should allow for a range of responses. When in doubt, rephrase your questions in a way that minimizes any implied sentiment, ensuring your respondents have the freedom to answer as accurately as possible.
3. Double-Barreled Questions
Double-barreled questions are another common survey design error that occurs when two or more questions are combined into one. Asking “How satisfied are you with our customer service and product quality?” forces the respondent to provide a single answer that might not adequately reflect their views on both topics. They might be satisfied with one aspect while dissatisfied with the other, but you’ve forced them to provide a single, undifferentiated response.
A study has found that these types of questions take longer to answer as they require respondents to process multiple, unrelated concepts at once. The same study found that the longer it takes for a survey question to be answered, the more distorted and sub-optimal the responses can be.
Clarity is crucial when designing survey questions. Breaking down complex queries into separate, focused questions allows for more precise and actionable insights. It also reduces the cognitive load on respondents, ensuring that each part of your survey targets a specific area of feedback without causing confusion.
4. Ambiguous or Vague Questions
Ambiguity in survey questions can lead to a host of problems, from misinterpretation to inconsistent responses. When a question contains undefined terms or lacks clarity, respondents may interpret it in different ways, resulting in data that is difficult to analyze. This type of inconsistency can make it challenging to draw clear conclusions from your survey results.
Vague questions create a barrier between your survey’s intent and the actual responses received. To avoid this, ensure every question in your survey is precise and leaves little room for interpretation.
5. Overly Complex or Jargon-Laden Questions
Simplicity is key when it comes to survey design. Overly complex questions that use technical jargon or convoluted sentence structures can confuse your respondents, especially if your audience spans a range of backgrounds and expertise levels. When a question is too wordy or filled with industry-specific language, respondents may end up misunderstanding the query, leading to inaccurate or incomplete responses.
This is about knowing your audience. If you’re asking cloud experts about their cloud automation strategies, then you can probably include a lot of technical jargon. If it’s a survey for the general public on what their favorite fruits and vegetables are, you probably don’t need to get into the scientific definition of what a fruit or vegetable is.
Clear, simple language is vital to successful survey design. By avoiding unnecessary complexity, you ensure that respondents from all walks of life can understand and engage with your survey. This approach not only improves the quality of the data collected but also enhances the overall user experience, making respondents more willing to complete your survey.
6. Negatively Phrased or Double Negative Questions
Negatively phrased questions or those containing double negatives can create significant confusion. When respondents are faced with a question that is framed in a negative light, or that contains multiple negatives, it becomes challenging to decipher the intended meaning. This often leads to unreliable responses that do not accurately reflect the respondent’s true opinions.
An example of this is the use of a double negative in a survey conducted in 1992 that led to massively distorted results. The double negative shifted the responses by roughly 20% compared to responses made to the question when it was asked in a clearer, less ambiguous manner.
These types of questions should be avoided in favor of clear, positively framed language. When constructing your questions, aim for straightforward phrasing that eliminates any ambiguity. This not only streamlines the response process but also ensures that your survey data remains consistent and reliable across all respondents.
7. Assumptive or Presumptive Questions
Similar to loaded questions, assumptive or presumptive questions operate under the false assumption that all respondents share a common experience or belief. Asking “What made you stop using our service?” presupposes that the respondent has indeed stopped using the service, which may not be the case for all participants. These assumptions can lead to frustration and alienation, as respondents feel that the survey does not accurately reflect their situation.
It’s crucial to frame questions in a way that is inclusive and neutral. By avoiding assumptions, you open the door for a wider range of responses, ensuring that all participants feel their perspective is valid and valued. This practice ultimately leads to richer, more actionable insights that can inform better decision-making.
8. Sensitive or Invasive Personal Questions
While gathering detailed insights is important, there is a fine line between what’s useful and what’s intrusive. Sensitive or invasive personal questions can make respondents uncomfortable, especially if they’re unexpected, potentially causing them to abandon the survey altogether or provide dishonest answers.
Even if your survey requires asking sensitive questions, you run the risk of getting distorted results. The psychological concept of social-desirability bias means that people will attempt to give results that bring them closer to what they believe is socially acceptable. For example, respondents to a survey on alcohol consumption might attempt to downplay the amount they drink due to negative societal perceptions of people who drink too much.
Instead, consider whether the information you’re seeking is absolutely necessary for your analysis. If it is, be sure to provide clear context for why the information is needed and ensure that respondents feel safe and respected when sharing their experiences.
9. Forced Choice Questions with Inadequate Options
Forced choice questions can be particularly problematic when the answer options provided are too narrow or unbalanced. When respondents are given a limited set of choices that do not fully capture their perspective, they are forced to select an answer that may not accurately represent the truth. This can introduce bias into your data and result in conclusions that do not fully reflect the nuances of your audience’s opinions.
The most obvious example of this in action is known as “the ceiling effect”, where the upper limit of options fails to recognize variations of anything above it. Let’s say, as part of a survey, you need to ask about annual income, and you use a multi-choice question.
You might put the topmost option as something like “\$500,000 or more.” But there’s a big difference between \$500,000 and \$5 million, and by grouping respondents who chose $500,000 or more, you’ll get distorted data due to how broad and different the pool is.
It’s critical to think carefully about the range of possible answers and to allow for open-ended responses where appropriate. If you must use multi-choice, something as simple as having an “Other” option where respondents can give a specific answer can help a long way.
10. Irrelevant or Off-Topic Questions
Every question in your survey should serve a clear purpose. Including questions that don’t align with the main objectives of your survey can confuse respondents and dilute the quality of your findings. When questions stray too far from the central topic, they can create distractions and reduce the overall effectiveness of your survey.
Keeping questions relevant is essential to maintaining respondent engagement and ensuring that the data you collect is focused and actionable. Before finalizing your survey, review each question carefully to confirm that it aligns with your overall research goals. This extra step can help eliminate unnecessary or off-topic questions that might otherwise compromise the integrity of your data.
Conclusion
Even with the best intentions, it can be challenging to pinpoint exactly when a question is off the mark. One of the best ways to gauge whether you’re including problematic questions is to conduct a pilot survey with a small segment of your target audience. Look for signs of confusion, incomplete responses, or a high drop-off rate.
Another useful strategy is to ask colleagues or peers for feedback on your survey design; a fresh pair of eyes can often spot issues that you might have overlooked.
