Designing a questionnaire for research purposes: The good, the bad & not so ugly

Transcription

1 Designing a questionnaire for research purposes: The good, the bad & not so ugly By Dr Ted Brown Associate Professor, First Year Undergraduate Program Convenor, & Postgraduate Coordinator Department of Occupational Therapy Monash University Peninsula Campus

2 What are surveys? Surveys are designed to obtain information about the prevalence, distribution and interrelations of variables within a population (Polit & Beck, 2004, p. 234)

3 What are surveys? Surveys gather information regarding beliefs, activities, preferences & attitudes, usually by asking a sample of direct questions Surveys are also designed to investigate links or associations between variables, known as correlational research

4 Uses of surveys Part of research study Part of a service evaluation Part of an audit For quality assurance activities For marketing studies in business Used as a research tool in many contexts included health, business & government

5 Advantages of survey Can be applied to different topics (e.g., attitudes towards people with disability, consumer spending habits, etc) Can be applied to different populations or groups of participants (e.g., citizens of Frankston, Monash University students, etc) Can be broad or narrow in scope & breadth of topic covered

6 Advantages of survey Are an economical (inexpensive) means of collecting data (e.g., cost of postage & printing) Is a means of generating a summary snapshot of an issue (e.g., government poll on raising taxes, fan support for local football club, student views on quality of teaching, etc)

7 Qualities of good survey research Measurement accuracy (reliable, valid, sensitive, responsive) Reproducibility Generalisability

8 Qualities of good survey research Accessibility Readability / understandable Comprehensive (covers a topic in a thorough manner)

9 Self-completion questionnaire Are completed in writing by the respondent Most common mode of completion is delivery & return through the mail Can also be completed in the presence of the researcher (who can provide assistance and check questionnaire for completeness)

10 Self-completion questionnaire Offer low cost relative to other types of surveys (cost of postage & printing) Potential coverage of a widely dispersed population (e.g., throughout Victoria) Has the potential for low response rates

11 Self-completion questionnaire Requires a certain literacy level for respondents to complete survey Are well suited for less complex topics Should be shorter than an interview and use more close-ended questions (e.g., true/false, yes/no, agree/disagree)

12 Questionnaires Used to collect information on facts, attitudes, knowledge, beliefs, opinions, attitudes, perceptions, expectations, experiences and the behaviour of individuals (Parahoo, 1997) Questionnaires can be used with large samples either by direction distribution, through the mail, or online via the internet

13 Questionnaires When measuring psychological variables, it can be difficult to ensure the instruments are precise due to the abstract nature of the construct being measured (e.g., attitudes towards gambling) Questionnaires must be carefully planned & constructed to ensure they are gathering the right data

14 Questionnaires Questionnaires can be standardised in that the same questions are asked in the same order of all the participants Questionnaires can be used in combination with other methods of gathering data

15 Questionnaires It is worth noting that although questionnaires may appear to be deceptively easy to construct, a welldesigned instrument requires considerable effort (Coates, 2004, p. 249) Constructing a questionnaire and wording the questions are not simple tasks (Meadows, 2003, p. 563)

16 Questionnaires Questionnaires should be designed for the use they were intended for Should be designed to minimise respondent errors in the understanding of questions & recording of answers Need to maintain interest and cooperation of the respondent

17 Constructing a questionnaire Constructing a questionnaire requires skill and an understanding of the key issues Adequate time is required as well One pitfall when designing a questionnaire is formulating questions that are difficult to ask or answer Example: Tell me about your opinions about the evolution of the English language )

18 Constructing a questionnaire Questions that include unfamiliar words or phrases or that are too vague or unspecific should be avoided Example: How have you been feeling?

19 Constructing a questionnaire Should avoid double-barrelled and catch-all questions Example: How satisfied are you with the medication you are taking and the staff who look after you? Example: How satisfied are you with your hair colour, body weight, IQ, the current state premier, cost of tram tickets, & the New Zealand cricket team performance in 2008?

20 Constructing a questionnaire Should avoid use of leading questions as well Example: Tell me how much you like the St. Kilda Football team since they are best team in the AFL? Example: How much do you like chocolate since it is supposed to be a great aphrodisiac?

21 Constructing a questionnaire Should avoid overly long questions that are difficult to answer Example: Over the past year have you attended any hospital including your local hospital, but not including accident and emergency departments, to visit friends or family, and to fly on Virgin Airlines to the Sunshine Coast?

22 Types of questions posed Open-ended and close-ended questions have their use in survey design A closed question provides a choice of alternative answers from which the respondent is asked to select Closed questions can be attitudinal or factual

23 Quiz question: Tell me about yourself and how is the weather as a survey question is. 1. An opened-ended question 2. Is broad and vague 3. Is colourful and inviting 4. A double-barrel question 5. Is valid and sensitive 6. Would be appropriate for a telephone interview

24 Quiz question: A closedended question would be best when asking about? 1. Political views 2. Cultural values 3. Job satisfaction 4. Knowledge of OT 5. Income level 6. Gender

25 Quiz question: What is the most important feature of a survey question for respondents? 1. Verb tense 2. Wording 3. Length 4. Clarity 5. Interest

26 Close-ended questions enable comparisons across individuals or groups of respondents Requires less time to complete than open-ended questions Are easy to code and process

27 Close-ended questions Example: Why do students not buy prescribed texts for the units they are enrolled in? 1. Too expensive 2. Not relevant 3. Too heavy to carry 4. Colour of cover is not attractive 5. Required readings are not relevant

28 Open-ended questions Do not provide any predetermined answers Allows respondent to answer the question in his/her own words Allows exploration of respondents thoughts, feelings, views, perspectives, & experiences in more depth

29 Open-ended questions Requires more effort from the respondent Possibility of interviewer variability Are more time-consuming to complete hence may limit the number of questions that can be asked Answers are more difficult to score and analyse which makes comparisons challenging to complete

30 Question wording & sequencing: Factors to consider when designing survey questions Memory: avoid over-taxing the respondent s memory Motivation: ask questions that are relevant to the respondent

31 Question wording & sequencing: Factors to consider when designing survey questions Communication: ensure that what we are asking is clear & understandable Knowledge: only ask for information the respondent is likely to have or be familiar with

32 Quiz question: What are two factors best NOT avoided when constructing survey items? 1. double-barrel questions 2. leading questions 3. objective questions 4. loaded questions 5. vague wording 6. complex items 7. correct spelling

33 Quiz question: What is the following statement an example of? What is your view of cute, furry, huggable possums that eat your mother s roses? 1. double-barrel question 2. leading questions 3. double-negative question 4. loaded questions

34 Response categories The choice, wording and ordering of response categories is critical as the wording of the question itself Impacts on how respondents interpret and answer the question

35 Suggestions for success Ensure response categories for closed questions are mutually exclusive / do not overlap All potential options / categories are exhaustive and if necessary include the option other, please indicate/specify

36 Suggestions for success Be aware that response options can send a message about the type and range of ideas, in addition to concepts the respondent should be thinking about An open space should be provided in self-completion questionnaires for free comment which can improve response rates

37 Suggestions for success Use simple language Avoid use of technical terms, professional jargon, & abbreviations Examples: ADL, PADL, DADL, PROM, AROM, ICF, ABI, CVA, AHA, CAOT, AJOT, OTHC

38 Suggestions for success Avoid words that may have more than one meaning Avoid questions that are not sufficiently specific Example: Does your astrological sign impact on the amount of rain that falls on your garden?

39 Suggestions for success: Things to avoid ambiguity double-barrel questions double negatives leading questions use of loaded questions and concepts use of presuming questions

40 Question sequencing The position of a question may affect the way the respondent answers it Specific questions should follow from general ones Demographic questions (such as age, gender, education level) can be placed either at the beginning or end of a questionnaire

41 Question sequencing Relevant & salient questions should be placed close to the front of questionnaire Highly sensitive questions should not be among the first few that are asked best to lead the respondent in with a few easy, non-confronting questions at the beginning & then move on to more personal or sensitive questions

42 Question sequencing Where possible, questions should be arranged from easy to difficult Questions should be blocked / grouped together by topic (e.g., demographic questions grouped together, attitude questions about gender grouped together, opinions about OT profession should be blocked together, etc)

43 Formatting Questionnaire Design & layout of questionnaire is important Appearance & formatting of questionnaire can impact whether a respondent chooses to respond or not Appearance includes length, print details, pagination and instructions

44 Questionnaire length Questionnaire length can directly impact the response rate; usually inversely proportionate to each other Longer questionnaires can lead to respondent fatigue or carelessness If questionnaire topic is of interest to respondent, they will be likely to complete longer questionnaires

45 Question & response category format A consistent format should be used throughout the questionnaire Questions & response categories can be formatted vertically or horizontally

46 Question & response category format Most researchers suggest the use of vertical format since the question is clearly differentiated from the responses Vertical format also makes data entry easier & less prone to error

47 Example of vertical format: Are you finding this lecture: 1. Is boring me to tears 2. Way too long 3. No opinion on the subject 4. Just about the right length 5. More interesting than watching the Australian cricket team beat the British cricket team 6. More preferable than chocolate and a cold Fosters

48 Quiz question: Closed ended questions provide researchers with more objective data. 1. True 2. False

49 Quiz question: What is NOT a factor that impacts a person completing a survey? 1. Knowledge 2. Marital status 3. Memory 4. Motivation 5. Height 6. Sexual orientation 7. Footy club allegiance 8. Cognitive status

50 Print Details Font sizes less than 10 should be avoided Overuse of upper case letters, italics should & multiple types of Font should be avoided

51 Instructions Self-administered questionnaires will generally include brief introductory comments about the purpose of the survey, the type of questions to be asked, why the information is required and what to do when the questionnaire is completed

52 Questionnaire appearance Avoid splitting a question or its response options across two pages Do not use lines for responses to openended questions, instead leave sufficient white space between questions Consider the use of coloured paper to distinguish between different questionnaires

53 Questionnaire appearance Ensure the front cover contains the title of the survey and the name and address of the organisation carrying it out Provide a back cover with blank space for respondents to make open comments Print a thank you after the last question and provide details of where to return the questionnaire