Pilot Testing Questionnaires

by
Philip Cleave
on
December 24, 2021
Putting together pilot test questionnaire

It doesn’t matter whether you’ve previously built lots of questionnaires, or you’re working on your very first one. You’ll want to maximise its engagement with respondents and the quality of insight you can obtain from it. But you’re very unlikely to be able to achieve that with your first attempt, especially if you’re new to surveys.

The key reason for this is that even if you’re an expert on the topic of your questions, without prior input from a sample of your target audience, it’s unlikely you will know exactly how they will respond to your questions.

This is where the value of questionnaire pretesting and piloting at the survey planning stage, can really pay dividends.

Pretesting vs pilot testing

When it comes to survey testing, some people have the misconception that it takes too long to do. Meaning they end up running their survey without any testing at all. This is a huge mistake. Even testing it with a couple of people could throw up issues, that if not spotted and solved could cause real problems with your data later on. So, any testing is better than none at all.

Pretesting

Having completed the first draft of your survey questionnaire, there are a number of steps you can take to improve it.

Question critique

Re-examine the wording of your questions to see if they contain anything potentially problematic for the reader. This could include double-barrelled questions (essentially two questions in one), negative questions (those containing negative words) and overly long questions.

You also want to assess the flow of your questionnaire. Check that your question order and any transition statements make sense.

Informal testing

Try interviewing yourself, where you put yourself in the position of the respondent and try to answer each question. You might also consider doing a mock interview of the questions with a close colleague. Both methods can enable you to identify potential difficulties in your question-answer process.

Enlist further expertise support

While you may be reasonably knowledgeable on the subject area of your written questions, there could be others in your business with more detailed expertise. They could also have expertise in specific areas such as questions design and testing, field issues and the cultural perspectives of the survey.

Whatever beneficial knowledge they have, try getting a small panel of them together to review your questionnaire. This can provide some really useful initial feedback for you regarding any potential problems with your survey questionnaire.

Systemic review of the questionnaire

If your organisation has been creating questionnaires for some time already, you may have a checklist in place to assess whether you’re on the right track. This can be very useful in quickly identifying any flaws that might exist with your questions.

These pre-test steps offer an inexpensive way of quickly uncovering a range of potential issues you need to fix. However, while they are dependent on the capabilities and availability of experts involved, there are some limitations. Most notably due to the lack of feedback from the audience you’re looking to target. That’s where primary research methods such as pilot testing can add real value to your planning and testing.

Pilot testing

Pilot testing enables you to assess your whole questionnaire under survey conditions. The primary benefit of pilot testing is to identify problems before implementing the full survey. Pilot testing looks to examine the validity of each question. It’s concerned with whether the question is capturing the information it’s intended to measure. Not only to meet the goals and objectives of the study, but analysing various aspects of the whole questionnaire, such as:

  • Analysing the question flow. Essentially, assessing the arrangement of questions, and whether they support the flow of the piece.
  • Assessing how well it meets expectations, in terms of the response rate at first contact.
  • The average time it takes respondents to complete each question.
  • The length of the questionnaire and how quickly respondents are completing it against the time you originally estimated for it.
  • Checking for any instances of confusion. Respondent trouble with understanding questions could result in many incorrect answers, respondents taking too long reply or skipping questions altogether.
  • Checking the question types used and whether you need to tweak them, in order to improve clarity and response time.

The pilot test of your questionnaire which should be carried out under survey conditions, should be tested on a sample size that is representative of the target population for your research. Besides completing your survey, your respondents should offer up their feedback about their experiences of filling it out.

Ideally you want to be able to get all your respondents together to observe their behaviour while they take your survey. Making notes throughout, look for places where they hesitate or make mistakes. You can also record the overall time it takes each participant to complete your survey. Much of what you find could indicate that your survey questions and layout are not clear enough and need improving.

For anyone struggling to find enough respondents to carry out a test from their existing contacts, a consumer panels service is a good alternative. With their ability to provide instant access to millions of respondents worldwide, consumer panels are great for reaching the exact demographic or niche audience you require.

Smarter online survey software

Capture data and opinions from the people that matter most – your customers, your staff or your students – with the use of smart, flexible online survey software. Free and paid plans for every budget.

The advantages of pilot testing questionnaires

While we have already touched on how pilot testing can help with a number of aspects concerning your questionnaire. Here are some more key advantages to pilot testing your questionnaire.

Maximise meaningful results data

When you consider that insightful data is the main reason why businesses run surveys in the first place.  Then piloting your survey can be extremely valuable in highlighting any questions that are currently not returning the information you need.

For example, you may find that the flexibility of a free text question is great for your respondents, but you’re getting the same answer reported in so many different ways that it makes analysis incredibly difficult.  Equally, you may find a selection list question doesn’t offer the answer options respondents need, meaning they constantly select “Other”.

However, it’s much better to identify this at the pilot stage and fix it, rather than gather thousands of genuine responses that you can’t usefully analyse.

Ensure surveys display correctly for all respondents

While your survey may look fine on your screen, respondents are likely to be using a range of mobile devices. They may also use assistive technology such as screen readers. So, it’s prudent to ensure you have a responsive design, and your questionnaire’s accessible for all users too.

Ensure respondents understand your questions

While you may be familiar with your research topic, someone without your background may not necessarily understand the terms you have used. So be mindful of this when you’re creating your questions, making sure they can be easily understood by everyone in your target audience.

Eliminate any difficult to identify typos

Typos can be difficult to spot. So, having many other people look through your survey helps ensure you spot and resolve as many errors as possible.

Ensure your survey complies with your ethics requirements

If you’re trying to create an anonymous or confidential questionnaire, and you’re allowing participants to withdraw their responses at any time, running a pilot will enable you to confirm that your survey has been set up with the appropriate safeguards.  For example, you can check that you have effective processes in place that can ensure you’re able to honour any request to withdraw a response.

Identify and fix any practical problems

From discovering if your questionnaire takes too long to complete, or if there are any firewall-related issues that prevent your respondents from accessing your survey, to any problems with your email client, disabling the links to your survey invitation emails. There are many practical problems you can identify and overcome with a pilot survey.

Is pilot testing always necessary?

It really depends on the size and audience involved. For smaller, less complex surveys, pretesting may be sufficient in revealing any necessary improvements and corrections required. Similarly, pretesting may be ample before running an HR survey to assess staff contentment, particularly for a smaller company.

However, for surveys where you will need to reach out to a much larger audience such as a customer survey questionnaire, then pilot testing is a very sensible idea. Without this you risk issuing too many questions that don’t make sense to participants, or a questionnaire that results in biased answers.

Important considerations for your pilot test

Finally, there are different pilot approaches from an organisational and respondent point of view that you need to consider. It’s important to be aware of these differences before you proceed with your own pilot study, as each offers their own distinct advantages in different situations.

Internal or external pilot study?

Looking at pilot studies from the organisational standpoint first, there are two main types of pilot study, internal and external.

With the internal pilot study, respondents in the pilot are also considered as the first participants in the main survey. In contrast, with an external pilot survey the questionnaire will be administered to a small group of target participants who are not included in the main survey.

Internal pilot studies are generally more efficient, if you’re only looking to change feasibility aspects such as participant recruitment and not the trial design. While external pilot studies offer greater flexibility to change the trial design for the main study once the pilot is complete.

Undeclared or participatory?

It’s also important to consider the approach you want to take when it comes to respondent participation.

There are two main types of respondent approaches to consider – undeclared and participatory.

With an undeclared pilot, the participants are not aware that they are part of a pilot group. And you issue this survey as if it were real. This type of pilot is useful for reviewing the results you’ve received and ensuring there’s no issues with survey completion.

In contrast, with a participatory pilot participants are fully aware of their participation and clearly informed about what they need to do. This pilot type is useful when you require feedback from the target audience about your survey content, processes and messages.

Get started and create your first survey

If you would like more information then please get in touch.