Questionnaire-OFID 2012-NKheir

Transcription

1 A Tool Kit for Questionnaire Development Dr. Nadir Kheir PhD MPS FNZCP Assistant Professor & Coordinator of Continuing Professional Pharmacy Development College of Pharmacy, Qatar University, Doha, Qatar

2 The great popularity with questionnaires is they provide a quick fix for research methodology. No single method has been so abused. Gillham B. Developing a questionnaire (real world research). London: Continuum, A study is unethical if it is scientifically unsound, causes undue offence or trauma, breaches confidentiality, or wastes people s time or money Boynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328:

3 Learning Outcomes After the completion of this session, the participant will be able to: 1. Describe the main uses of survey questionnaires 2. list the major stages of questionnaire development process 3. discuss the known pitfalls that could affect questionnaire development 4. Explain what is meant by reliability and validity 5. Examine questionnaire face validity 6. Develop a short questionnaire with consideration to the main components of the development processes

4 Useful Resources A guide to good survey design. Statistics New Zealand Publishing & Media Services Division Research methods knowledge base. Survey Research. Available in: QuestionPro (Survey software). Available in:

5 Survey Research Can be a questionnaire OR an interview Important area of measurement in applied social research Considered branch of social science Any measurement procedures that involve asking questions Can be a short paper-and-pencil feedback form to an intensive oneon-one in-depth interview Study samples chosen from target population to discover inferences Study relations between behaviors and other sociological variables Different from a census (where the whole population is studied)

6 Purposes of Surveys: Description Making descriptive assertions about a population Answering the what questions: - what is the distribution of certain traits? Examples: -the attitude towards new service -the quality of life of diabetics -the skill level of vocational workers

7 Purposes of Surveys: Explanation Making explanatory assertions For example: -why some persons abuse pain killers? -why do students resort to cramming in exams? -what are the reasons for the increased visits to the staff clinic during 2010 compared to 2009? Explanatory objective needs multivariate analysis -Cramming could be explained by quality of teaching, number of assessments, class size - Abusing pan-killers can be associated with issues around access, employment status, relaxed laws - Increased visits to the clinic in 2010 could be explained by a big conference at the University, an endemic, etc

8 Purposes of Surveys: Exploration Also: pilot survey, Focus Group, qualitative research Before a large study, you need to refine your methodology through a dress-rehearsal survey Focus group to explore deep meaning and complex phenomena No attempt to select representative sample Might raise some questions and possibilities They identify logistical, sampling, cost, design issues In pilot study: a small sample size means its results might not useable as stand alone data

9 Types of Surveys 1. Interviews: face-to-face, telephone 2. Mail surveys, Groups, drop-off surveys, online surveys 3. The Focus Group All Utilize a Questionnaire or questions

10 Questionnaire Questionnaires are usually paper-and-pen or online instruments that the respondent completes Type of Questionnaire Mail survey questionnaires Group-administered questionnaire Household drop-off survey Characteristics Adv: inexpensive; can send the exact instrument to a wide number of people; allow the respondent to fill it out at their own convenience Disadv: Low response rate, not suitable for detailed responses, errors, missing data, inaccuracies Adv: convenience; high response rate; low errors Disadv: difficult to assemble, costly, inconvenience to participants Adv: the respondent can fill in private, when it's convenient; the interviewer makes personal contact with the respondent Disadv: response may have to be posted back or collected in person

11 Developing a Questionnaire A complex and iterative process This means the designer needs to go back and forth to resolve issues (change questions to suit respondents leading to reduce/change objectives, etc) Multiple evaluations during the developmental phases to reduce burden at the end

12 A Questionnaire: Offer an objective means of collecting information about people s knowledge, beliefs, attitudes, and behaviour Can be used as the sole research instrument (such as in a cross sectional survey) or within clinical trials If inappropriately developed, will lead to poor quality data, misleading conclusions, and woolly recommendations

13 Before Settling on a Questionnaire: Consider: is a questionnaire the right tool? Map the area for logistical problems, sampling issues, range of possible responses Seek information available in the literature If such information are not available, conduct a focus group or pilot study

14 When can an Existing Questionnaire be Used? If the questionnaire address the same objective If the questionnaire is validated in the same cultural context First and best option! BUT BE CAREFUL: Importing an existing questionnaire Translating an existing questionnaire Examples: Quality of Life Questionnaires

15 Why develop your own questionnaire if one is available? Boynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328:

16

17 Steps in Developing a Questionnaire: 1. The Background A thorough understanding of the problem through literature search Establish clear objectives, purpose, main question Conduct a scoping exercise to ensure that you clarify goals and if necessary reach an agreed compromise This will also flag up potential practical problems (e.g. how long the questionnaire will be and how it might be administered) Decide the process used to select the respondents Seek statistical advice if generalizability of results is required, differences measured, for comparing results

18 Steps in Developing a Questionnaire: The Background Consider all people who would use the questionnaire Participants: general public? Consider literacy Interviewers, coders, data entry staff, program writers, researchers who analyze Tasks might be shared, or several might be done by one individual (list tasks as reminder) Respondents: VIP group Why: cannot be trained, and they give their time and effort free of charge

19 Steps in Developing a Questionnaire: 2: Questionnaire Conceptualization Decide: what will the questionnaire measure (knowledge, attitudes, perceptions, opinions) Generate statements/questions for the questionnaire Start with broad statements Follow with a list of detailed information needs Looks like a list of tables in the final result section Involve other parties who are within the research team A link among the objectives of the study and their translation into content is established

20 Attitudes and Practices of Pharmacy Students, Pharmacists, and Academics on Pharmacokinetic Teaching and Application in Clinical Practice Hypothesis: PK is a knowledge-based service that is poorly utilized in practice by pharmacists Identify Populations and questions : Students Pharmacists Aim: To assess the attitudes and practices of undergraduate pharmacy students, practicing pharmacists, and pharmacy academics on the on PK teaching and learning as well as application in clinical practice Attitude: importance, relevance, etc Existing questionnaire on Barriers, challenges (exploration) students attitudes towards an undergraduate courses in pharmacy? Attitude: importance, relevance Need of validation exercise? Practice: application, satisfaction Barriers, challenges (exploration) Faculty/Teachers Practices: teaching strategies, satisfaction Attitudes: satisfaction with application, Questionnaire-OFID need for change 2012-NKheir

21 Steps in Developing a Questionnaire: 3: Format and Data Analysis Turning questions into a questionnaire The focus is on writing statements/questions Selection of appropriate scales of measurement Questionnaire layout Question ordering Number questions Add cover note or introduction, routing instructions, codes, page numbers In online survey, most done automatically (to the scoring and producing results)

22 Poorly Designed questions I frequently engage in exercise : strongly agree through to strongly disagree. frequently implies frequency, so a frequency based rating scale (with options such as at least once a day, twice a week, and so on) would be more appropriate. I Regularly, engage in exercise One person can regularly engage in exercise once a month whereas another person can regularly do so four times a week Other words to avoid in question stems include: commonly, usually, many, some, and hardly ever

23 Steps in Developing a Questionnaire: 4: Establishing Validity Development of a valid and reliable questionnaire is a must to reduce measurement error Measurement error is the "discrepancy between respondents' attributes and their survey responses " This requires pre-testing the questionnaire and correlating its results with other variables Poorly constructed questions and wrong response scaling lead to poor validity

24 Steps in Developing a Questionnaire: 4: Establishing Validity Validity is the amount of systematic or built-in error in measurement The following questions are addressed in Step 4: 1. Is the questionnaire valid? 2. Does it represent the content? 3. Is it appropriate for the sample/population? 4. Is the questionnaire comprehensive enough to collect all the information needed to address the purpose and goals of the study? 5. Does the instrument look like a questionnaire?

25 Steps in Developing a Questionnaire: 4: Establishing Validity Desk Evaluations -Looking for potential problems -The designer before user evaluation -Simplify, but careful -Conduct at each stage, after a change -Done by more than 1 person Content analysis, Social and Face Validity User Evaluations The designer knows too much about the questions -Other users also need to try the questionnaire -Self-Admin: Observing respondents fill a questionnaire -Also: check completed questionnaire and ask about errors, missing data, changes required -3rd person observes and take notes -Follow-up questions to check understanding and difficulties

26 Pitfalls with Questions Also see: 1. Hard to understand, badly written 2. Overly educated: Question: How often do you engage in exercise or use other nonpharmacological remedies? Place your comments here:

27 Pitfalls with Questions Also see: 3. Abbreviations that respondents may not be aware of: Q. What is your ID No. Q. Which type of diabetes do you have: T2DM T1DM 4. Use of undefined terms: Q. What was your income last year? Last year may mean 12 months ending today, financial year, calendar year. Income: poorly defined, before or after tax? etc

28 Pitfalls with Questions Also see: 5. Questions that do not fit the objectives: Irrelevance, undue effort and burden 6. Ambiguity: what exactly do you mean? Q. Was your performance better in your ability to do daily exercise or your sleep quality this year? 7. Questions that are too long: Includes questions with too many options that do not suit phone interviews or can be burdensome to respondents

29 Pitfalls with Questions Also see: 8. Double or triple questions: Q. During the past 2 weeks, did you have to see your physician or another healthcare professional? Yes No Q. Do you agree that insulin injections should be taken before food and stored in a fridge to avoid harmful effect of temperature? Yes No

30 Pitfalls with Questions Also see: 9. Unreasonable recall period: Q. In the last 12 months, how many times did you feel numbness in your left arm? 10.Double negative: Q. Do you agree or disagree that pharmacy graduates from the College of Pharmacy in Qatar University should not be asked to register in order that they practice Pharmacy in Qatar?

31 Pitfalls with Questions Also see: 11.Questions that are based on an assumption: Q. What is the your spouse s qualification: No education Primary school Secondary school College University graduate Post-graduate qualification Assumption: Respondent has a spouse Adding a Not Applicable is not enough (might not be read) Problematic: if no spouse and next questions about spouse A respondent might tick (No education) as an option when no spouse!

32 Pitfalls with Questions Also see: Social Desirability and other bias (leading, loaded, unbalanced): Q. In a week, how often do you miss taking your pills? Q. Physicians in this hospital are: Very caring Fairly caring Not very caring

33 Pitfalls with Questions Also see: Responses that are not mutually exclusive: Q. Which of these over-the-counter classes of medicine have you sold in the last two weeks from your pharmacy: Non-steroid anti-inflammatory Drugs Analgesics Laxatives Anti-hypertensives Drugs acting on the cardiovsacular system

34 Pitfalls with Questions Also see: Response options that are not exhaustive Q. To which of the following do you most refer your patients for further investigation: Hamad General Hospital Primary health care clinic Private health clinic There might be other referral sites. How to resolve this?

35 Pitfalls with Questions 1. Crowded, untidy (in typed questionnaire) 2. Hard to read (in typed questionnaire) 3. Inconsistent use of terms (medicine and drug) 4. Inappropriate titles: can be confusing, respondents may skip sections (avoid) 5. Routing instructions that are hard to follow

36 Pitfalls with Questions 6. Inconsistent lay out: Yes before No then changes 7. Lay out that is likely to cause mistakes: No Yes Don t know Formatting issues resolved in online surveys. Try:

37 Survey Errors Sampling Error -Resulting from collecting information from a fraction of the population Extent of error depends on: 1. Sample size 2. Variability of the characteristic : greater variation= greater error 3. Sample design: use of information known about the population reduces sampling error Non Sampling Error -These are hard to measure -May be larger than sampling errors can result in bias Examples: 1. Poor definition of target population 2. Including or excluding groups 3. Selecting a frame that does not match the target population 4. Poor design that does not give every respondent the same chance of being selected 5. Non-responses* 6. Poor definition of objectives 7. Time frame: seasonality factors 8. Quality of data collection 9. Inadequate interviewer training

38 A Checklist of Considerations From: Some important issues to be considered in designing a questionnaire: Start with easy, nonthreatening questions Put more difficult, threatening questions near end For historical demographics, follow chronological order Ask about one topic at a time When switching topics, use a transition Reduce response set (the tendency of respondent to just keep checking the same response) For filter or contingency questions, make a flowchart

39 Ethics of Questionnaire Survey From: Thank the respondent at the beginning for allowing you to conduct your study Keep your survey as short as possible -- only include what is absolutely necessary Be sensitive to the needs of the respondent Be alert for any sign that the respondent is uncomfortable Thank the respondent at the end for participating Assure the respondent that you will send a copy of the final results (if s/he desires, and ensure to fulfill your promise )

40 Finally An exercise Find the pitfalls (mistakes, errors, weaknesses) in online survey:

41 Questions?