What Makes A Good Questionnaire

Creating a good questionnaire is more than just stringing together a set of questions. It's about crafting a tool that not only gathers the data you need but also engages respondents, respects their time, and provides actionable insights. In the digital age, where online questionnaires are prevalent, the art of designing them effectively is more crucial than ever. So, what makes a good questionnaire? Let's dive into some principles and practices that can help you design questionnaires that work.

Basic considerations

Start with the end in mind

One of the biggest mistakes in questionnaire design is jumping straight into writing questions without considering the end goal. It's like setting off on a journey without knowing your destination. Start by asking yourself: what do you want to achieve with this survey? What insights are you hoping to gather? By having a clear understanding of your objectives, you can reverse-engineer the questionnaire to ensure every question aligns with these goals. This not only makes the survey more focused but also ensures that the data you collect will be useful and relevant to your needs.

Timing is everything

Timing can make or break the effectiveness of your questionnaire. The closer to the experience of the product or service the questionnaire is delivered, the fresher the respondent's memory and the more accurate the feedback. Avoid sending questionnaires at arbitrary times; instead, consider the customer journey and identify the optimal moments for reaching out. For example, sending a follow-up survey shortly after a purchase or after a customer service interaction can yield more detailed and actionable feedback than a survey sent weeks or months later.

Competitive comparison matters

If you're using a questionnaire to gauge customer satisfaction, it's worth considering how your customers see you compared to your competitors. Asking questions that allow respondents to compare your offerings with those of your competitors can provide a wealth of information on where you stand in the market. It can highlight areas where you excel, as well as areas where you may be falling behind. This type of competitive benchmarking can be invaluable in refining your products, services, and overall customer experience.

Pre-testing and pilot surveys

Before launching your questionnaire to a wider audience, it's a good idea to conduct a pre-test or pilot survey with a small group of respondents. This can help you identify any issues with question clarity, survey length, or logic flow. Pre-testing allows you to make adjustments based on feedback, ensuring that the final questionnaire is as effective as possible. It's a crucial step that can save time and improve the quality of the data collected, as it helps to catch potential problems early.

Questionnaire distribution strategies

How you distribute your questionnaire can have a big impact on who responds and the quality of the data you collect. There are several distribution methods to choose from, each with its own advantages and disadvantages. Email surveys are easy to distribute and track but may be overlooked in busy inboxes. Social media can reach a large audience quickly but might not target your ideal respondents. Embedding surveys on your website can capture engaged users but may miss those who visit infrequently. Consider your target audience and choose the distribution method that best suits your goals.

Questionnaire design considerations

Great questionnaire design isn't just about asking the right questions; it's also about how those questions are structured and presented. There are several best practices that can help you create a more effective questionnaire. Start by ensuring that your questions flow logically, leading respondents naturally from one topic to the next. Avoid grouping too many similar questions together, as this can create fatigue or boredom. Mix in different types of questions to keep respondents engaged and to capture different dimensions of their experience.

Keep it short and sweet

In today's fast-paced world, attention spans are shorter than ever. Long, drawn-out questionnaires are a sure-fire way to drive respondents away. A good questionnaire respects the respondent's time by being as concise as possible. Aim to include only the most essential questions, and keep them direct and to the point. This doesn't mean cutting corners on important data, but rather being mindful of what's truly necessary. Shorter questionnaires often yield higher response rates and more accurate data, as respondents are less likely to rush through or abandon the survey halfway.

Simplicity is key

In the quest to sound professional or thorough, it's easy to fall into the trap of using jargon, acronyms, or complex sentences. However, simplicity is key to effective communication. A good questionnaire is easy to understand, free from technical language that could confuse respondents. Aim to write questions that are clear and straightforward, using everyday language that your audience will understand. This not only makes the questionnaire more accessible but also reduces the likelihood of misinterpretation, leading to more accurate responses.

Accessibility considerations

A good questionnaire is one that can be completed by as many people as possible, including those with disabilities. Ensuring your survey is accessible means considering factors like screen reader compatibility, providing alternative text for images, and using high-contrast colour schemes. It also means writing questions in a way that is clear and easy to understand for all respondents. Following accessibility guidelines not only helps you reach a wider audience but also demonstrates a commitment to inclusivity.

Types of questions to include

Choosing the right survey questions is critical for capturing the data you need. Here's a more detailed look at different question types you might consider, along with when and why to use each one:

Multiple-choice questions

These are the most common type of survey questions, where respondents are given a list of options and asked to select one or more answers. They are useful for collecting quantitative data that's easy to analyse. Multiple-choice questions are ideal when you want to limit responses to a specific set of options, making it easier to compare results across respondents. To get the most out of multiple-choice questions:

• Ensure the options are mutually exclusive and cover all possible answers
• Consider including an “Other” option with a text box for respondents to provide an answer not listed

Likert scale questions

Likert scale questions are used to measure attitudes, opinions, or perceptions. Respondents are asked to rate their level of agreement or satisfaction on a symmetric agree-disagree scale, often ranging from "Strongly Agree" to "Strongly Disagree". These questions are particularly useful when you want to gauge the intensity of respondents' feelings about a particular statement. They help in capturing the nuance in opinions and can be used to track changes over time. When designing Likert scale questions:

• Use an odd number of points (e.g., 5 or 7) to provide a neutral option
• Phrase statements clearly and ensure they are focused on a single issue

Rating scale questions

Similar to Likert scales, rating scale questions ask respondents to rate something on a scale, such as from 1 to 10 or from “Very Poor” to “Excellent”. These are particularly effective for measuring the quality or importance of something. They offer a quick way to get a snapshot of how respondents feel about specific aspects of a product or service. To make the most of rating scale questions:

• Ensure the scale is balanced, with equal positive and negative options
• Define the endpoints of the scale clearly to avoid ambiguity

Matrix questions

Matrix questions allow you to group several Likert scale or rating scale questions together in a grid format, where respondents can evaluate multiple items using the same set of response options. This format is efficient, saving space and time, and is useful for comparing different variables under the same metric. However, matrix questions can also lead to respondent fatigue if overused. When using matrix questions:

• Keep the number of items manageable, ideally under ten
• Use matrix questions only when the items are closely related to each other

Open-ended questions

These questions allow respondents to answer in their own words, providing rich, qualitative data. They are particularly useful for exploring complex issues, gathering detailed feedback, or capturing ideas and opinions that structured questions might miss. Open-ended questions can uncover insights that you might not have anticipated, but they require more effort to analyse. To get the most out of open-ended questions:

• Use them sparingly, typically at the beginning or end of the survey
• Ask clear, concise questions that guide respondents on the type of information you're seeking

Demographic questions

Demographic questions collect information about respondents' characteristics, such as age, gender, education level, or employment status. Demographic data is crucial for segmenting responses and understanding how different groups might view your product or service differently. These questions are often placed at the beginning or end of a survey and should be worded sensitively to ensure respondent comfort. When designing demographic questions:

• Offer inclusive options (e.g., gender identity, income brackets)
• Make these questions optional unless they are absolutely necessary for your analysis

Ranking questions

These questions ask respondents to order items according to their preference or importance. Ranking questions are great for understanding priorities or preferences within a given set of options. However, they can be more cognitively demanding for respondents, so it's best to limit the number of items they need to rank. To design effective ranking questions:

• Limit the number of items to rank to between 5 and 7
• Ensure that respondents can rank items easily, using drag-and-drop interfaces if possible

Dichotomous questions

Dichotomous questions are simple yes/no or true/false questions. They are easy for respondents to answer and are useful for filtering respondents based on eligibility criteria or understanding binary choices. However, they don't capture nuance, so use them when you need a straightforward, clear-cut answer. For example:

• Ask whether respondents have used a product before delving into their experience with it
• Use dichotomous questions to quickly gauge whether a condition applies to the respondent

Image-based questions

These questions use images instead of text to gather feedback. They can be particularly useful in product design or marketing research, where respondents might be asked to choose between different designs, logos, or packaging options. Image-based questions engage respondents visually and can lead to more intuitive responses. When designing image-based questions:

• Ensure images are clear, high-quality, and load quickly
• Provide a brief description or context for each image to guide the respondent

Slider questions

Slider questions allow respondents to choose a value along a continuum, rather than selecting a fixed option. They are interactive and can be more engaging than traditional scales. Slider questions are particularly effective when you want to measure the degree of feeling or opinion across a spectrum, such as satisfaction or likelihood to recommend. To use slider questions effectively:

• Clearly label the endpoints of the slider to define what each extreme represents
• Consider the scale carefully, ensuring that it is appropriate for the question being asked

Participant considerations

Anonymity and confidentiality

Anonymity and confidentiality are critical in building trust with your respondents. If participants believe their responses will be shared or used against them, they may not provide honest answers. Clearly communicate how you will protect respondent data and what measures you have in place to ensure confidentiality. In some cases, offering an anonymous survey can encourage more candid feedback, particularly on sensitive topics. Be transparent about your data handling policies to reassure respondents that their privacy is respected.

Focus on personal experience

One of the keys to collecting reliable data is to ask respondents about their own experiences. Questions that are too general or that ask for opinions on matters outside the respondent's direct experience can lead to guesswork or inaccurate answers. By focusing on personal experiences, you ensure that the data collected is grounded in reality. For instance, if you're surveying customers about a product, ask them about their specific interactions with that product rather than their thoughts on industry trends or what they think others might feel.

Don't forget the emotional element

Numbers and facts are crucial, but emotions drive behaviour. Incorporating questions that explore how respondents feel about your product, service, or company can provide valuable insights that raw data might miss. Asking follow up questions can be particularly effective in this regard. Understanding the emotional response of your customers can help you identify pain points, areas of delight, and opportunities for improvement. It's these emotional insights that often lead to the most powerful changes in business strategy.

Personalisation and adaptive questioning

Personalisation can greatly enhance the respondent experience and lead to more accurate data. Tailoring questions based on the respondent's previous answers, known as adaptive questioning, can make the questionnaire feel more relevant and engaging. For example, if a respondent indicates that they've used a specific product, subsequent questions can delve deeper into their experience with that product. This approach not only improves the quality of the data but also shows respondents that you value their individual input.

Avoiding common pitfalls

Designing a good questionnaire is as much about avoiding mistakes as it is about following best practices. Common pitfalls include leading questions that push respondents towards a certain answer, double-barrelled questions that ask about two things at once, and questions that are too complex or confusing. To avoid these issues, always review your questionnaire with a critical eye and consider getting feedback from colleagues or a small test group. Aim for clarity, simplicity, and neutrality in your questions to ensure you're collecting reliable and unbiased data.

Ensure your data is representative

When designing a questionnaire, it's important to consider who your respondents are and whether they represent your target audience. If the data collected isn't representative, it can skew your results and lead to incorrect conclusions. This might involve using screening questions to ensure that respondents meet certain criteria, or distributing the survey in a way that reaches a diverse and representative sample of your audience. In cases where you identify underrepresented groups, consider methods such as oversampling to balance the dataset.

Incentivising responses

Getting people to participate in your survey can be challenging, especially if they see no immediate benefit to themselves. Offering incentives can be an effective way to increase response rates. These could be tangible rewards like discounts, gift cards, or entry into a prize draw, or they could be intangible, such as sharing the survey results or offering exclusive insights. When using incentives, it's important to ensure they don't bias the responses and that they are appropriately matched to the effort required to complete the questionnaire.

Ethical considerations in survey research

Ethics are a fundamental aspect of survey research. This involves ensuring informed consent, where respondents are fully aware of the purpose of the survey and how their data will be used. It also means being honest about the nature of the survey and avoiding any deceptive practices. Respondents should have the right to withdraw at any time, and their data should be handled with care, following all relevant data protection laws. Ethical considerations are not just about legal compliance – they're about building trust and conducting research in a responsible manner.

Handling non-response bias

Non-response bias occurs when the people who choose not to respond to your survey differ significantly from those who do, potentially skewing your results. To minimise this bias, it's important to encourage a broad range of respondents to participate. This might involve sending reminders to non-respondents, offering incentives, or using different distribution methods to reach those who didn't respond initially. Analysing the characteristics of respondents versus non-respondents can also help you understand and adjust for any bias that might be present in your data.

Ongoing strategies for effective questionnaires

Data analysis and interpretation

Once you've collected your data, the next step is to analyse and interpret it. This involves more than just tallying up responses; it's about identifying trends, patterns, and insights that can inform your decisions. Statistical analysis tools and survey dashboards can help you make sense of the data, whether you're looking at simple frequency counts or more complex analyses like regression or factor analysis. It's also important to consider the context of the data – what external factors might be influencing the responses? How does this data compare to past surveys? Thoughtful analysis will help you extract maximum value from your questionnaire.

Follow up with respondents

Asking for feedback through a questionnaire sets an expectation that something will be done with the information collected. A good questionnaire is not just about collecting data; it's also about fostering a relationship with your respondents. Following up with participants to let them know how their feedback is being used, or the changes being made as a result, can enhance the value of the questionnaire and increase trust. This follow-up can be as simple as sending a thank-you email with a brief summary of the survey findings or outlining any actions you plan to take based on the responses.

Integrate feedback with other data sources

Feedback from a questionnaire is valuable, but it's even more powerful when integrated with other data sources such as operational or financial data. This holistic view allows you to see the bigger picture and understand how customer perceptions align with actual performance metrics. For instance, customer satisfaction scores from a questionnaire could be correlated with sales data or churn rates to identify patterns and areas for improvement. This integration helps ensure that the insights gained are actionable and directly linked to your business outcomes. See also: survey integrations.

Iterative improvement

Creating a good questionnaire is not a one-time task – it's an iterative process. After each survey, take the time to review the results and the questionnaire itself. What worked well? What could be improved? Use this feedback to refine and improve your future questionnaires. This process of continuous improvement will help you create better surveys over time, leading to more reliable data and more actionable insights. It's also a good idea to stay up to date with the latest research and best practices in survey design, as the field is constantly evolving.

Author's note: this blog was originally published October 2012 and updated August 2024