Writing Good Survey Questions
Who We Are Are Capacity for Health at APIAHF, is a Capacity Building Assistance Program, with offices in San Francisco and Washington D.C.
Who We Are Our program is funded by the Centers for Disease Control and Prevention (CDC) to provide free trainings and oneon one technical assistance for community based organization (CBOs) on: Organizational Infrastructure and Program Sustainability Evidence Based Interventions (EBIs) and Public Health Strategies, and Monitoring and Evaluation Who We Are
Housekeeping All participants are automatically muted by webinar administrators. For discussion, please raise your hand. You will be called on and un muted so that you can ask your question live. If your audio is through a telephone, please make sure to enter your audio PIN, otherwise we are unable to unmute you. You may ask questions by typing them in using the questions feature. If anyone is watching the webinar with multiple people around one computer, please chat in your contact information so that we can include you on any follow up. Today s webinar will be recorded. The recording will be available on our website.
Kristin Fernandez, MPH Presenter Kristin has spent the majority of her career working with HIV and other health programs as a health educator, research associate and program manager. Currently, she consults with various health agencies as a grant writer and program planning and evaluation specialist. When not working, she is chasing around her 2 year old daughter.
Today s Agenda Today s Agenda Introduction Survey Objectives Means to Administer a Survey Types of Questions Problem Questions Question Order Q&A Closing
??? Any questions before we begin?
Learning Objectives Learning Objectives By the end of today s webinar you will: Understand the importance of having clear objectives for your survey Be able to recognize the different types of survey questions Be able to recognize problem questions Have confidence to develop precise and appropriate survey questions
What is a Survey? A survey is a system for collecting information to describe, compare or explain knowledge, attitudes, and practices or behavior. How to Ask Survey Questions (Fink, 1995)
Survey Objectives Survey Objectives An objective is a statement of the survey s hoped for outcomes. Survey Development and Key Informant Interviews Workshop. (Accion Mutua, Proposal Writing Preparedness Institute, January 2008)
Survey Objectives Survey Objectives Example 1: To identify what participants have learned from participating in a specific program. Example 2: To determine specific behavioral risks for HIV among a specific population we want to work with.
Survey Objectives Survey Objectives Clear objectives will help you answer the following questions: What information should the survey collect? What questions should be asked? Every question should refer back to the objectives Adapted from Survey Development and Key Informant Interviews Workshop (Accion Mutua, Proposal Writing Preparedness Institute, January 2008)
??? Any questions about objectives?
Means to Administer a Survey Means to administer Survey Mail Phone In Person Internet
QUICK POLL #1 Which types of surveys do you feel most comfortable using? (select as many as you like) MAIL PHONE IN PERSON INTERNET
Means to Administer a Survey Means to administer Survey MAIL PROS: Anonymity = Truth Length & Complexity CONS: Low Response Rate No On Site Assistance Cost
Means to Administer a Survey Means to administer Survey PHONE PROS: Higher Response Rate Higher Quality Data CONS: Time Limited Simplicity of Questions Questionable Honesty
Means to Administer a Survey Means to administer Survey IN PERSON Can be either : Self Administered or Interviewer Administered PROS: Higher Quality Data Better Response Rate CONS: Cost Questionable Honesty Interviewer Bias
Means to Administer a Survey Means to Administer Survey INTERNET PROS: Global Access Low Cost Anonymity Automated Data Collection CONS: Not all populations have access Uncertain Validity of Data Learning Curve
Mail Pros Truth * Length Cons Response Rate * Cost Phone/ In Person Pros Response Rate * Validity Cons Time * Cost * Truth Internet Pros Cost * Truth Cons Access * Validity
??? Any questions?
Survey Questions Survey Questions
Survey Questions 7 Basic Types of Questions Problem Questions Question Order
Survey Questions 7 Basic Types of Questions
Survey Questions 7 Basic Types of Questions 1)Open Response 2)Partially Open Response 3)Closed Response 4)Agreement Scale / Likert Scale 5)Rating Scale 6)Ranking Scale 7)Checklist
Survey Questions Open Response ex. What do you think keeps your friends from using condoms? Monitoring Outcomes of HIV Prevention Programs Question Bank www.utsouthwestern.edu/preventiontoolbox
Survey Questions Partially Open Response Ex#1. How did you hear about the homeless shelter in your neighborhood? Friend Relative Case Worker Other (specify)
Survey Questions Closed Response ex. Have you had more than 1 sexual partner in the last 30 days? Yes No ex. With whom are you most likely to talk about your problems? My best friend My mother My father Another adult in my family A teacher or school counselor A youth worker or staff member at a youth center My boyfriend/girlfriend ex. #2 UC Berkeley School of Public Health, PH 218B Evaluation of Health and Human Service Programs, Lyn Paleo, DrPH, Instructor. Session 7: From Indicators to Questions
Survey Questions OPEN ENDED questions PROS: Personal POV allows for richness in answers CONS: More variability Difficult to code Time and money Possible interviewer bias CLOSED questions PROS: Easier to code Less variability Greater uniformity CONS: Creates bias
??? Any questions?
Quick Poll #2 If you want to understand in detail why people don t talk to their partners about condom use, what type of question would you use?
Quick Poll #3 How would you ask about number of sexual partners for subgroups of your population in order to be able to easily code and compare answers?
Survey Questions 7 Basic Types of Questions 1)Open Response 2)Partially Open Response 3)Closed Response 4)Agreement Scale / Likert Scale
Survey Questions Agreement Scale / Likert Scale Example: It is easy to talk to my partner about safe sex. Strongly Agree Agree Neutral Disagree Strongly Disagree NOT: Strongly Agree Agree Strongly Disagree
Survey Questions Agreement Scale / Likert Scale Example: Use this scale to answer the following questions: Strongly Agree Agree Neutral Disagree Strongly Disagree 1 2 3 4 5 Circle only one: a) It is easy to talk to my partner about safe sex. 1 2 3 4 5 b) My teacher is knowledgeable about HIV/AIDS. 1 2 3 4 5
Survey Questions Rating Scales Example: When you decide to have sex with someone, how important are each of the criteria listed below? (Please select a number on the scale of 1 to 10, with 10 being very important and 1 being not at all important.) Age 1 2 3 4 5 6 7 8 9 10 Ethnicity 1 2 3 4 5 6 7 8 9 10 Good Looks 1 2 3 4 5 6 7 8 9 10
Survey Questions Ranking Scales ex. Of the behaviors below, number them in order of what you think is the LEAST likely way to contract HIV (1) to the MOST likely way to contract HIV (5) : french kissing anal intercourse vaginal intercourse shaking hands oral sex
??? Any questions about scales?
Survey Questions 7 Basic Types of Questions 1)Open Response 2)Partially Open Response 3)Closed Response 4)Agreement Scale / Likert Scale 5)Rating Scale 6)Ranking Scale 7)Checklist
Survey Questions Checklists Example: What substances have you used in the past 30 days? (check all that apply) Nicotine Alcohol Marijuana Hallucinogens Cocaine Heroin
??? Any questions about what we ve talked about so far?
Survey Questions Problem Questions
What do you think? What are common mistakes people make when writing survey questions? (Please write in using the questions feature)
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Double barreled ex. The shelter should give out needles and offer substance abuse counseling. Agree Disagree
Problem Questions Double barreled ex. The shelter should give out needles and offer substance abuse counseling. Agree Disagree The shelter should give out needles. Agree Disagree The shelter should offer substance abuse counseling. Agree Disagree
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Must be competent ex. What percent of the shelter s budget should be spent on: substance abuse counseling childcare career development building maintenance ex. How many times have you shared a needle in the past 30 days?
Problem Questions Must be competent ex. What percent of the shelter s budget should be spent on: substance abuse counseling childcare career development building maintenance ex. How many times have you shared a needle in the past 30 days? ex. How frequently do you share needles? Never rarely occasionally regularly I don t use needles
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Relevance ex. Have you ever had sexual intercourse? YES NO ex. What is your opinion about practicing safe sex?
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Length ex. If you are currently sexually active and have had more than 3 sexual partners in the past 30 days, we d like to know more about your sexual partners and/or relationships. Do you consider any of them to be your most important (or primary) sexual partner? If so, which one or ones? ex. Of the people you have had sex with in the last 30 days, is there one that you consider to be your most important (or primary) sexual partner?
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Negatives ex. Do you agree or disagree: People should not practice unsafe sex in order to not get a sexually transmitted disease? Strongly Agree Agree Neutral Disagree Strongly Disagree ex. Do you agree or disagree: People should practice safe sex to avoid getting a sexually transmitted disease? Strongly Agree Agree Neutral Disagree Strongly Disagree
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Bias ex. It has been suggested that abstinence is the only way to prevent HIV/AIDS. Does this seem like a good idea to you? ex. At what age did you start abusing drugs?
Problem Questions Bias ex. It has been suggested that abstinence is the only way to prevent HIV/AIDS. Does this seem like a good idea to you? ex. Is abstinence a method you would choose to prevent HIV/AIDS? YES NO ex. At what age did you start abusing drugs? ex. At what age did you start using drugs?
Problem Questions Bias ex. The President of the US has committed significant US funding to the international HIV/AIDS epidemic. Does this seem like a good idea to you? ex. Do you support committing significant US funding to the international HIV/AIDS epidemic? YES NO
Problem Questions Bias ex. Many people were not able to get all the education they would have liked. What was the last grade you completed in school?
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Ambiguity ex. What is your income? ex. What is your annual income? ex. Do you watch the news? YES NO ex. How often do you watch the news? Regularly Often Sometimes Rarely Never
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Overlapping Alternatives ex. Tell us about your condom use: (please check one) I always use condoms I mostly use condoms I use condoms with some people and not with others I forget to use condoms when I drink I use condoms but I don t like them
Problem Questions Overlapping Alternatives ex. Tell us about your condom use: I always use condoms I frequently use condoms I sometimes use condoms I rarely use condoms I never use condoms
Problem Questions 1)Double barreled 2)Must be competent to answer 3)Relevance 4)Length 5)Negatives 6)Bias 7)Ambiguity 8)Overlapping alternatives 9)Sensitive subjects
Problem Questions Sensitive Subjects ex. Have you ever had an STD? YES NO ex. There are a number of sexually transmitted diseases that can be spread through unprotected sex. As we discussed during this workshop, teens have higher rates of certain sexually transmitted diseases than other age groups. How serious a problem do you consider STDs to be for teens in your community? Serious problem Moderate problem Slight problem Not a problem at all Have you ever known anyone who has had an STD? Yes No How about yourself, have you ever had an STD? YES NO
Slide 67 S37 Explain the example more, this is really about NORMALIZING behavior that people might have trouble admitting to. When we ask a sensitve questions in a good way (embedding it in others, opening with leads like in the notes of the last slide) we make it easier for people to be honest with us. I would show more than one correct way to ask these questions. Also, I think this example is a bit too long, can we simplify it? Sonya, 4/19/2011
Problem Questions Sensitive Subjects ex. Have you ever had sex without a condom? YES NO
Slide 68 S59 Explain the example more, this is really about NORMALIZING behavior that people might have trouble admitting to. When we ask a sensitve questions in a good way (embedding it in others, opening with leads like in the notes of the last slide) we make it easier for people to be honest with us. I would show more than one correct way to ask these questions. Also, I think this example is a bit too long, can we simplify it? Sonya, 4/19/2011
Problem Questions Sensitive Subjects ex. Have you ever had sex without a condom? YES NO How many times have you had sex without a condom?
Slide 69 S60 Explain the example more, this is really about NORMALIZING behavior that people might have trouble admitting to. When we ask a sensitve questions in a good way (embedding it in others, opening with leads like in the notes of the last slide) we make it easier for people to be honest with us. I would show more than one correct way to ask these questions. Also, I think this example is a bit too long, can we simplify it? Sonya, 4/19/2011
Social Response Bias OVER reported UNDER reported Being a good citizen Illnesses and disabilities Being a well informed and cultured person Illegal or frowned upon behavior Fulfilling moral and social responsibilities Financial Status (Adapted from UC Berkeley School of Public Health, PH 218B Evaluation of Health and Human Service Programs, Lyn Paleo, DrPH, Instructor. Session 7: From Indicators to Questions).
Social Response Bias Checklist when asking about socially undesirable behavior : 1. Open Response 2. Long Questions 3. Familiar or Slang Words 4. Informants 5. Loaded Questions 6. Current Behavior 7. Embed into a List 8. Alternatives to Standard Questions 9. Pilot 10. Validate Adapted from Asking Questions, 1st ed. San Francisco: Jossey Bass, 1982.
Problem Questions Any questions?
Survey Questions Question Order
Question Order Questions at the beginning of a survey should be:
Question Order 1. Sensory/physical: What do you see when you walk into a clinic? Where do you go when you need STD testing? 2. Knowledge/feeling: How do you feel when you have unprotected sex? What is the window period for HIV? 3. Opinion/values: What is your opinion of sex education in schools? 4. Analytic/action: How long does it take to put on a condom properly? How would you prepare to have safe sex on a date?
Further Resources Slides, a recorded version of this webinar, and links to further resources will be available on our website at: www.capacity4health.org Contact us for individualized capacity-building assistance: Sonya Dublin, Capacity for Health 415-568-3335, sdublin@apiahf.org
Resources The content of this webinar was drawn from: UC Berkeley School of Public Health, PH 218B Evaluation of Health and Human Service Programs, Lyn Paleo, DrPH, Instructor. Session 7: From Indicators to Questions Introduction to Survey Research Design, Linda K. Owens, Assistant Director for Research Planning, Survey Research Laboratory. SRL. Fall 2002 Seminar Series. www.srl.uic.edu/seminars/intro/introsrm.pdf Planning a Survey. UT Southwestern, Capacity Building Assistance Center, Prevention Toolbox. http://www8.utsouthwestern.edu/vgn/images/portal/cit_56417/20/28/205319plannin g_a_survery.pdf Survey Development and Key Informant Interviews Workshop Accion Mutua, Proposal Writing Preparedness Institute, January 2008 How to Ask Survey Questions (Fink, 1995) Monitoring Outcomes of HIV Prevention Programs Question Bank www.utsouthwestern.edu/preventiontoolbox