Designing a questionnaire for research purposes: The good, the bad & not so ugly By Dr Ted Brown Associate Professor, First Year Undergraduate Program Convenor, & Postgraduate Coordinator Department of Occupational Therapy Monash University Peninsula Campus
What are surveys? Surveys are designed to obtain information about the prevalence, distribution and interrelations of variables within a population (Polit & Beck, 2004, p. 234)
What are surveys? Surveys gather information regarding beliefs, activities, preferences & attitudes, usually by asking a sample of direct questions Surveys are also designed to investigate links or associations between variables, known as correlational research
Uses of surveys Part of research study Part of a service evaluation Part of an audit For quality assurance activities For marketing studies in business Used as a research tool in many contexts included health, business & government
Advantages of survey Can be applied to different topics (e.g., attitudes towards people with disability, consumer spending habits, etc) Can be applied to different populations or groups of participants (e.g., citizens of Frankston, Monash University students, etc) Can be broad or narrow in scope & breadth of topic covered
Advantages of survey Are an economical (inexpensive) means of collecting data (e.g., cost of postage & printing) Is a means of generating a summary snapshot of an issue (e.g., government poll on raising taxes, fan support for local football club, student views on quality of teaching, etc)
Qualities of good survey research Measurement accuracy (reliable, valid, sensitive, responsive) Reproducibility Generalisability
Qualities of good survey research Accessibility Readability / understandable Comprehensive (covers a topic in a thorough manner)
Self-completion questionnaire Are completed in writing by the respondent Most common mode of completion is delivery & return through the mail Can also be completed in the presence of the researcher (who can provide assistance and check questionnaire for completeness)
Self-completion questionnaire Offer low cost relative to other types of surveys (cost of postage & printing) Potential coverage of a widely dispersed population (e.g., throughout Victoria) Has the potential for low response rates
Self-completion questionnaire Requires a certain literacy level for respondents to complete survey Are well suited for less complex topics Should be shorter than an interview and use more close-ended questions (e.g., true/false, yes/no, agree/disagree)
Questionnaires Used to collect information on facts, attitudes, knowledge, beliefs, opinions, attitudes, perceptions, expectations, experiences and the behaviour of individuals (Parahoo, 1997) Questionnaires can be used with large samples either by direction distribution, through the mail, or online via the internet
Questionnaires When measuring psychological variables, it can be difficult to ensure the instruments are precise due to the abstract nature of the construct being measured (e.g., attitudes towards gambling) Questionnaires must be carefully planned & constructed to ensure they are gathering the right data
Questionnaires Questionnaires can be standardised in that the same questions are asked in the same order of all the participants Questionnaires can be used in combination with other methods of gathering data
Questionnaires It is worth noting that although questionnaires may appear to be deceptively easy to construct, a welldesigned instrument requires considerable effort (Coates, 2004, p. 249) Constructing a questionnaire and wording the questions are not simple tasks (Meadows, 2003, p. 563)
Questionnaires Questionnaires should be designed for the use they were intended for Should be designed to minimise respondent errors in the understanding of questions & recording of answers Need to maintain interest and cooperation of the respondent
Constructing a questionnaire Constructing a questionnaire requires skill and an understanding of the key issues Adequate time is required as well One pitfall when designing a questionnaire is formulating questions that are difficult to ask or answer Example: Tell me about your opinions about the evolution of the English language )
Constructing a questionnaire Questions that include unfamiliar words or phrases or that are too vague or unspecific should be avoided Example: How have you been feeling?
Constructing a questionnaire Should avoid double-barrelled and catch-all questions Example: How satisfied are you with the medication you are taking and the staff who look after you? Example: How satisfied are you with your hair colour, body weight, IQ, the current state premier, cost of tram tickets, & the New Zealand cricket team performance in 2008?
Constructing a questionnaire Should avoid use of leading questions as well Example: Tell me how much you like the St. Kilda Football team since they are best team in the AFL? Example: How much do you like chocolate since it is supposed to be a great aphrodisiac?
Constructing a questionnaire Should avoid overly long questions that are difficult to answer Example: Over the past year have you attended any hospital including your local hospital, but not including accident and emergency departments, to visit friends or family, and to fly on Virgin Airlines to the Sunshine Coast?
Types of questions posed Open-ended and close-ended questions have their use in survey design A closed question provides a choice of alternative answers from which the respondent is asked to select Closed questions can be attitudinal or factual
Quiz question: Tell me about yourself and how is the weather as a survey question is. 1. An opened-ended question 2. Is broad and vague 3. Is colourful and inviting 4. A double-barrel question 5. Is valid and sensitive 6. Would be appropriate for a telephone interview
Quiz question: A closedended question would be best when asking about? 1. Political views 2. Cultural values 3. Job satisfaction 4. Knowledge of OT 5. Income level 6. Gender
Quiz question: What is the most important feature of a survey question for respondents? 1. Verb tense 2. Wording 3. Length 4. Clarity 5. Interest
Close-ended questions enable comparisons across individuals or groups of respondents Requires less time to complete than open-ended questions Are easy to code and process
Close-ended questions Example: Why do students not buy prescribed texts for the units they are enrolled in? 1. Too expensive 2. Not relevant 3. Too heavy to carry 4. Colour of cover is not attractive 5. Required readings are not relevant
Open-ended questions Do not provide any predetermined answers Allows respondent to answer the question in his/her own words Allows exploration of respondents thoughts, feelings, views, perspectives, & experiences in more depth
Open-ended questions Requires more effort from the respondent Possibility of interviewer variability Are more time-consuming to complete hence may limit the number of questions that can be asked Answers are more difficult to score and analyse which makes comparisons challenging to complete
Question wording & sequencing: Factors to consider when designing survey questions Memory: avoid over-taxing the respondent s memory Motivation: ask questions that are relevant to the respondent
Question wording & sequencing: Factors to consider when designing survey questions Communication: ensure that what we are asking is clear & understandable Knowledge: only ask for information the respondent is likely to have or be familiar with
Quiz question: What are two factors best NOT avoided when constructing survey items? 1. double-barrel questions 2. leading questions 3. objective questions 4. loaded questions 5. vague wording 6. complex items 7. correct spelling
Quiz question: What is the following statement an example of? What is your view of cute, furry, huggable possums that eat your mother s roses? 1. double-barrel question 2. leading questions 3. double-negative question 4. loaded questions
Response categories The choice, wording and ordering of response categories is critical as the wording of the question itself Impacts on how respondents interpret and answer the question
Suggestions for success Ensure response categories for closed questions are mutually exclusive / do not overlap All potential options / categories are exhaustive and if necessary include the option other, please indicate/specify
Suggestions for success Be aware that response options can send a message about the type and range of ideas, in addition to concepts the respondent should be thinking about An open space should be provided in self-completion questionnaires for free comment which can improve response rates
Suggestions for success Use simple language Avoid use of technical terms, professional jargon, & abbreviations Examples: ADL, PADL, DADL, PROM, AROM, ICF, ABI, CVA, AHA, CAOT, AJOT, OTHC
Suggestions for success Avoid words that may have more than one meaning Avoid questions that are not sufficiently specific Example: Does your astrological sign impact on the amount of rain that falls on your garden?
Suggestions for success: Things to avoid ambiguity double-barrel questions double negatives leading questions use of loaded questions and concepts use of presuming questions
Question sequencing The position of a question may affect the way the respondent answers it Specific questions should follow from general ones Demographic questions (such as age, gender, education level) can be placed either at the beginning or end of a questionnaire
Question sequencing Relevant & salient questions should be placed close to the front of questionnaire Highly sensitive questions should not be among the first few that are asked best to lead the respondent in with a few easy, non-confronting questions at the beginning & then move on to more personal or sensitive questions
Question sequencing Where possible, questions should be arranged from easy to difficult Questions should be blocked / grouped together by topic (e.g., demographic questions grouped together, attitude questions about gender grouped together, opinions about OT profession should be blocked together, etc)
Formatting Questionnaire Design & layout of questionnaire is important Appearance & formatting of questionnaire can impact whether a respondent chooses to respond or not Appearance includes length, print details, pagination and instructions
Questionnaire length Questionnaire length can directly impact the response rate; usually inversely proportionate to each other Longer questionnaires can lead to respondent fatigue or carelessness If questionnaire topic is of interest to respondent, they will be likely to complete longer questionnaires
Question & response category format A consistent format should be used throughout the questionnaire Questions & response categories can be formatted vertically or horizontally
Question & response category format Most researchers suggest the use of vertical format since the question is clearly differentiated from the responses Vertical format also makes data entry easier & less prone to error
Example of vertical format: Are you finding this lecture: 1. Is boring me to tears 2. Way too long 3. No opinion on the subject 4. Just about the right length 5. More interesting than watching the Australian cricket team beat the British cricket team 6. More preferable than chocolate and a cold Fosters
Quiz question: Closed ended questions provide researchers with more objective data. 1. True 2. False
Quiz question: What is NOT a factor that impacts a person completing a survey? 1. Knowledge 2. Marital status 3. Memory 4. Motivation 5. Height 6. Sexual orientation 7. Footy club allegiance 8. Cognitive status
Print Details Font sizes less than 10 should be avoided Overuse of upper case letters, italics should & multiple types of Font should be avoided
Instructions Self-administered questionnaires will generally include brief introductory comments about the purpose of the survey, the type of questions to be asked, why the information is required and what to do when the questionnaire is completed
Questionnaire appearance Avoid splitting a question or its response options across two pages Do not use lines for responses to openended questions, instead leave sufficient white space between questions Consider the use of coloured paper to distinguish between different questionnaires
Questionnaire appearance Ensure the front cover contains the title of the survey and the name and address of the organisation carrying it out Provide a back cover with blank space for respondents to make open comments Print a thank you after the last question and provide details of where to return the questionnaire