Guidelines for Authors of ENT UK Questionnaires

by Jim Fairley, ENT UK Survey Guardian. Updated: 3 January 2012 email: surveys@entuk.org

Aims

• To improve the quality and scientific value of questionnaires distributed to our membership.
• To improve validity of questionnaires.
• To increase the response rate for questionnaires which pass quality standards.

Background

The first decade of the 21st Century saw a steady increase in questionnaires received by UK ENT specialists. The advent of electronic distribution reduced the cost, and made it easier to survey the ENT community. While recognizing legitimate interest in establishing national practice and opinion in the speciality, there was also the lure of a relatively easy publication to adorn the CV of the researcher.

The quality of design and implementation of research surveys varied. Too many fell below basic standards. Natural selection operated to some extent. The worst offenders would gravitate to their rightful place - the recycle bin. But not before they caused nuisance and annoyance. This prejudiced future, more worthy questionnaires, resulting in poor response rates. Low response rates had a negative impact on validity. Bad apples spoiled the good.

In 2008, ENT UK resolved that it was neither efficient nor ethical to waste the time and expertise voluntarily donated by respondents. It was no longer acceptable to burden our members with poorly thought-out, poorly presented questionnaires masquerading as research. We therefore decided to exercise quality control before allowing the use of our mailing lists for questionnaires. In 2009, there was a moratorium on distributing questionnaires. By 2010, we had introduced a system of quality control, vetting of questionnaires by the newly-appointed Survey Guardian. We also surveyed our members and sought their preferences on how they would like to have questionnaires distributed in future.

The Guidelines

Identify a proper research question

It is essential to have a legitimate research question, that is capable of being answered by the survey methodology. You must state clearly

• What question you are trying to answer
• Why it is important
• What is known / not known already
• Why surveying the members of ENT UK Expert Panel is the best way to answer your question

Broad categories of suitable questions include

• What is the range of opinion and practice on [Controversial Subject]
• What is the level of awareness amongst ENT Specialists on [New or Developing Subject]
• Rarely encountered conditions: Pooling experience on [Rare Subject]

Ask the question properly

If a research question is worth asking, it is worth asking properly. Bear in mind that, on current response rates, even if ENT UK accept your questionnaire, many of your potential respondents will not complete it. Unless you can engage their interest, and maintain their good will, for the duration of the questionnaire, your response rate will be poor and your results invalid.

Keep it short and focused

Short, focused questionnaires do better than long, rambling ones.

The title should be the research question.

The title is crucial. Unless it raises interest, most will ignore everything beyond it. The title should, ideally, be the research question. It should be brief, to the point, and invite further investment of the respondent's time.

How long will it take to complete?

Be upfront about the length of time it will take to complete. Base this on pilot studies. Under ten seconds would be great. Half a minute is probably OK. Anything over a minute is a long time. Five minutes is an eternity, and highly unlikely to be completed without serious incentives.

The check-list

The following check-list, whilst not guaranteeing acceptance, should help avoid the commonest errors resulting in rejection.

1. Objectives must be clearly stated
• State your objectives
• Be precise, and concise, and explain why it is important
• One or two sentences should be sufficient.
• Avoid the "Life, the Universe and Everything" trap (answer = 42, but What is the Question?)
2. Ask only appropriate questions
• Ensure your questions are appropriate.
• At the design stage, be ruthless.
• Limit the subject of your questions to those directly related to the stated objectives.
• Acknowledge and deal with your tendency to ramble and drift off topic.
• Do not kid yourself that, by asking more and more arcane questions, you are fully exploring this fascinating subject.
• If there is one, overall most important question that you would like answered, put it up on its own, at the beginning. Then, once the respondent has answered, bring on the more detailed questions - if you must.
3. All questions must be intelligible
• Ensure your questions are intelligible to your target audience.
• This will mean piloting them.
• If your target audience includes a range of levels, your pilot should cover the same range.
• Please provide evidence that you have piloted your questions. The best evidence is the draft vs the final version. (The final version should be better).
4. All questions must be unambiguous
• Ensure your questions are unambiguous, ie mean the same thing to different people.
• This is best achieved by piloting across the target range.
5. Response type must be appropriate
• Check that the type of response is appropriate.
• For qualitative research, open rather than closed questions are likely to be needed.
6. Use only appropriate options / rating scales
• For closed questions, ensure that the options are appropriate -yes / no, single best option from a list, a rating scale. Rating scales can be categorical (Likert scale) or continuous.
• It is often reasonable to have more than one type of response within a questionnaire.
• If you aren't sure which response is best, take advice from someone with experience.
7. Don't shoehorn respondents into categories that don't fit
• You should know how you are going to analyse the responses.
• Analysis is easier with closed questions, but do not let the easier analysis cause you to force respondents into categories which may not fit all eventualities.
• A response category of "Other" or "Don't Know" is often useful in maintaining the engagement of your respondent, even though it may reduce the power of your analysis.
• Unless you are very experienced in this area, take statistical advice at the design stage.
8. Let respondents have their say
• Allow a free text box at the end, for anything that the respondent feels is relevant and has not been covered. You may be surprised at what people say.
9. Please and thank you
• Be polite. Always thank your respondent for completing the survey.

Further reading

Survey Monkey guide to smart survey design

Technical Instructions for Survey Authors