SUCCESSFUL RESEARCHERS SPEND TIME CONSTRUCTING VALID INDICATORS As the most commonly used indicator is a survey/questionnaire, the following power point provides detailed instructions for construction of valid indicators using this data collection technique. Why is it important to construct valid indicators? Garbage in Garbage OUT! Your indicator may appear professional, but if it is not valid, you have garbage. You may have a large sample size, but if your indicator is not valid, you have garbage. You may use high powered statistics, but if your indicator is not valid, you have garbage. You may have a great program, but if your indicators are not valid, then your evaluation of the program is garbage. You may have a well-written journal article, but if you indicator is not valid, you have garbage.
STEPS TO CONSTRUCTING A SURVEY INSTRUMENT Step 1 Make Decisions about TYPES of Indicators Level of measurement Single vs. multiple indicators Close vs. open ended indicators Step 2 Consider Content of Questions Make it easy for respondents Make it easy for those who input data Construct indicators that yield reliable (consistent) and valid (accurate) results Step 3 Consider Format of Survey/Questionnaire Step 4 Know How to Evaluate Surveys Step 5 Construct a Codebook
STEP 1 MAKE DECISIONS ABOUT TYPES OF INDICATORS Level of measurement Measure at highest level possible. Determined by 1. How precise can respondents be? For example - Can they remember their exact income? 2. How precise are they willing to be? For example - Will they tell you how many STIs they have? Single vs. multiple indicators Multiple indicators should be used to measure different dimensions of same concept or ask about multidimensional concept. Examples 1. Scales used to measure marital happiness, depression, college readiness (i.e., SAT, ACT) etc. 2. Questions used to measure different dimensions of social class (e.g., income, occupation, education)
STEP 1 MAKE DECISIONS ABOUT TYPES OF INDICATORS Close vs. open-ended questions Two types of closed ended questions 1. Unordered - Ask about type of activity Example - Which of the following activities do you engage in before going to bed? 1. Reading 2. Eating 3. Both of the above 2. Ordered - Ask about degree or strength of reaction to statement Example - Indicate how satisfied you are with the following: VD DS S VS USU Extension Services 1 2 3 4 County Extension Agent 1 2 3 4
STEP 1 MAKE DECISIONS ABOUT TYPES OF INDICATORS Close vs. open-ended Open-ended questions What are the advantages? Important for exploratory research. For instance, use when you do not yet know WHAT questions to ask. Provide more in-depth information about topic. What are the disadvantages? Time consuming. Coding is required once data is collected Validity and reliability are dependent on the coders and their skills. Examples of open-ended questions commonly used in evaluation research What are the strengths of this program? How would you improve this program? What things have you learned from this presentation that you can share with others?
STEP 2 CONSIDER CONTENT OF QUESTIONS Make it easy for respondents Ask questions at the 6 th grade level 20 words or less Avoid the word NOT in negatively worded questions (i.e.,, I do NOT study every night.) Avoid Acronyms (i.e., USU, ARD) Avoid putting blanks in the middle (i.e., I read for pleasure times per week.) Avoid having respondent rank items (i.e., rank ten most serious crimes from high to low.) When using contingency questions mark them clearly with arrows. Match questions/statements with respondents in terms of 1. Interests Don t ask questions about something they have NO interest in. 2. Competency - Don t ask questions about things they know nothing about. 3. Relevance - Don t ask questions about things that aren t relevant to them and/or their situation.
STEP 2 EXAMPLES OF SUGGESTIONS ON SLIDE 5 Making it easy for respondent Write at 6 th grade reading level 20 words or less X Based on your annual net income for the previous decade, excluding any atypical pay periods or wages accrued by minors living in your household, what would be your best approximate estimate of your socio-economic status? X A better alternative might be X What is your social class? Avoid the word NOT in negatively worded questions I very seldom can t stop worrying about my grades. A better alternative would be I often worry about my grades. This professor is not a competent teacher. A better alternative would be This professor is an incompetent teacher Avoid Acronyms Do you think CEOs who graduated from USU should attend ARDs if their child has ADD or an STI? Avoid putting blanks in the middle I spend hours studying for exams. A better alternative would be On average, how many hours do you spend studying for exams during a typical week? hours
STEP 2 EXAMPLES OF SUGGESTIONS ON SLIDE 5 Making it easy for respondent Avoid having respondent rank items Rank the following leisure activities from 1 to 8 based on how much you enjoy them. Read books E-mail a friend Ride a bike Eat potato chips Swim Talk on my cell phone Watch T.V. Gossip X A better alternative might be Please indicate your two most favorite leisure time activities (1), (2) Use contingency questions marked clearly with arrows. Match questions/statements with respondents X Interests Don t ask teenagers about flat taxes or retirement pensions. X Competency Don t ask the yearly income of their great Uncle Fred. X Relevance Don t ask singles about their marital happiness.
STEP 2 CONSIDER CONTENT OF QUESTIONS Make it easy for those who code, input and /or interpret data. You cannot input letters or boxes into the computer. The computer only recognizes numbers. THEREFORE, have respondents circle numbers for their answers. Avoid letters and boxes. For answer categories make certain HIGH numbers correspond to POSITIVE answers. When doing analysis, it is much easier to interpret a number if a high number corresponds to a positive answer. That is the way we are socialized to think about numbers. Use open-ended questions only when necessary. When possible use a series of open ended questions to get more precise answers.
STEP 2 EXAMPLES OF SUGGESTIONS ON SLIDE 8 Making it easy for you Have respondent circle numbers for answer categories. Do you grow sorghum? Yes No A better alternative would be Do you grow corn? (1) Yes (2) No Have high numbers = positive answers What grade did you receive in this class last semester? (1) A (3) C (5) F (2) B (4) D A better alternative would be What grade did you received in this class last semester? (1) F (3) C (5) A (2) D (4) B Please indicate how satisfied you are with the information you received from this presenter. Possible answers range from (1) very satisfied to (4) very dissatisfied. VS S DS VD 1 2 3 4 A better alternative would be Please indicate how satisfied you are with the information you received from this presenter. Possible answers range from (1) very dissatisfied to (4) very satisfied.
STEP 2 EXAMPLES OF SUGGESTIONS ON SLIDE 8 Making it easy for you When possible use a series of open- ended questions to get more precise answers. This can make it much easier to code and interpret your findings. What do you do for a living? A better alternative might be to ask a series of questions Where do you work? If you work for a company or organization, what is the name of that company/organization? In a typical day what types of tasks do you complete?
STEP 2 CONSIDER CONTENT OF QUESTIONS Increase Validity/Reliability Wording of Question Be as Precise/specific as possible When asking about frequency, ask for a specified time (i.e., On average, how often, per week, if ever, do you telephone your mother? ). When asking for a number, ask for as precise a number as possible (i.e., How much time, if any, do you spend watching TV during an average day? hours minutes). When asking for a number, try to be exact (i.e., How old were you on your last birthday? years)
STEP 2 CONSIDER CONTENT OF QUESTIONS Increasing Validity/Reliability Wording of Question Avoid Slang hot cool Do you think it is cool to skip class? A better alternative might be Do you think students approve of skipping classes? Avoid Bias or Leading Questions Cancer is a devastating illness that kills millions of Americans every year. How much money do you think we should spend finding a cure for this illness? A better alternative might be How much money do you contribute to cancer research? dollars per year. Women who are raped never recover. They only survive. What penalty do you think rapists should receive? A better alternative might be Indicate the extent to which you agree with the following statement s Current penalties for convicted rapists are just. Penalties for convicted rapists should be more severe. Penalties for convicted rapists are too harsh.
STEP 2 CONSIDER CONTENT OF QUESTIONS Increasing Validity/Reliability Wording of Question For Indices Use statements not questions SD D NAND A SA Do you enjoy the fireworks 1 2 3 4 5 at the county fair? A better alternative would be SD D NAND A SA I enjoy the fireworks 1 2 3 4 5 at the county fair.
STEP 2 CONSIDER CONTENT OF QUESTIONS Increasing Validity/Reliability Wording of Question For Indices Avoid response set SD D A SA I loved this program. 1 2 3 4 The teachers were very friendly 1 2 3 4 I learned a great deal. 1 2 3 4 I want to do this again 1 2 3 4 A better alternative would be SD D A SA I loved this program 1 2 3 4 The teachers were unfriendly 1 2 3 4 I learned nothing new 1 2 3 4 I want to do this again 1 2 3 4
STEP 2 CONSIDER CONTENT OF QUESTIONS Increasing Validity/Reliability Wording of Question For Indices I did NOT like the Avoid using NOT in a negative statement SD D A SA program 1 2 3 4 The teachers were NOT friendly 1 2 3 4 I did NOT learn anything. 1 2 3 4 A better alternative would be SD D A SA I disliked this program 1 2 3 4 The teachers were unfriendly 1 2 3 4 I learned nothing new 1 2 3 4
STEP 2 CONSIDER CONTENT OF QUESTIONS Increasing Validity/Reliability Wording of Questions For indices Use multiple statements to measure multidimensional concepts (i.e., social distance) SD D NADA A SA I would live with an Asian 1 2 3 4 5 I would go to parties with Asians 1 2 3 4 5 I would feel uncomfortable sharing an office with an African American 1 2 3 4 5 My family would disapprove if 1 2 3 4 5 If I married a Hispanic
STEP 2 CONSIDER CONTENT OF QUESTIONS Increasing Validity/Reliability Wording of Questions For Indices Avoid double barreled/double bind statements Men and women are SD D A SA poor communicators 1 2 3 4 I like the staff and the 1 2 3 4 teachers. My health has changed 1 2 3 4 since last year. A better alternative might be SD D A SA Men are poor communicators. 1 2 3 4 Women are poor communicators 1 2 3 4 I liked the staff. 1 2 3 4 I liked the teachers 1 2 3 4 My health is better 1 2 3 4 My health is worse 1 2 3 4
STEP 2 CONSIDER CONTENT OF QUESTIONS Wording of Answer Categories Exhaustive Cover all possible answer Word questions so all answers are possible. Example How often, IF EVER, do you attend football games? times per semester. For instance Use the answer category Other (specify) whenever it is appropriate. Word answer categories so all possible answers are acceptable. For instance, Hitler was a 1. Very negative role model 2. Negative role model A better alternative would be Hitler was a 1. Very negative role model 3. Positive role model 2. Negative role model 4. Very positive role model
STEP 2 CONSIDER CONTENT OF QUESTIONS Wording of Answer Categories Use ALWAYS and NEVER with great caution Please indicate the extent to which you do the following. Possible responses are (1) Never, (2) Seldom, (3) Sometimes, (4) Often, and (5) Always N S ST O A I read the newsletter mailed 1 2 3 4 5 out by the extension agent. A better alternative would be to replace NEVER and ALWAYS with ALMOST NEVER and ALMOST ALWAYS. AN S ST O AA I read the newsletter mailed 1 2 3 4 5 out by the extension agent.
STEP 2 CONSIDER CONTENT OF QUESTIONS Wording of Answer Categories Mutually exclusive NEVER have two possible right answers For instance - How much money did you make last year? (1) $10,000 to $20,000 (2) $20,000 to $40,000 (3) $40,000 to $60,000 A better alternative would be (1) $10,000 to $19,999 (2) $20,000 to $39,999 (3) $40,000 to $59,999
STEP 2 CONSIDER CONTENT OF QUESTIONS Wording of Answer Categories Avoid offensive/biased answer categories For instance Use overweight NOT obese Use pro-life NOT anti-abortion Use pro-choice NOT pro-abortion Use thin NOT skinny Use government assistance NOT welfare For instance What do you do when you don t go to class? (Circle all that apply) 1. Act Like a Lazy Slob 2. Waste my time A better alternative would be How do you spent your time when you don t attend class?
STEP 2 CONSIDER CONTENT OF QUESTIONS Wording of Answer Categories Be as precise/specific as possible Use YES/NO sparingly as answer categories Disadvantages Advantages Doesn t give you much variation. Is not precise enough. Sometimes it is the only plausible answer. Should you use Don t Know as an answer category? Disadvantages Sometimes, it is NOT a plausible answer. Can result in response set, as respondents may see it as the easier, though not most accurate answer. Advantages Can add important variation to possible answers, and REMEMBER you need/ want variation when you start doing analysis.
STEP 3 CONSIDER FORMAT OF SURVEY General Guidelines Using Numbers For single items number from top to bottom in columns. What grade did you receive? 1. F 4. B 2. D 5. A 3. C For index : (1) put numbers under symbols, (2) number from left to right (3) number from lower to higher SD D NAD A SA 1 2 3 4 5 1 2 3 4 5 Professional Appearance Spacing even and balanced Uncluttered appearance No spelling or grammar errors Can use shading etc.
STEP 3 CONSIDER FORMAT Required Sections OF SURVEY Textbox Should contain the following information General statement about the purpose of survey Statement indicating the time survey will take Statement that indicates it is anonymous or confidential Statement that indicates it is voluntary Contact information for those wanting more information Basic sociodemographic questions Questions measuring indicators/variables Contingency questions All parts should be on the same page Indices/Scales The entire index/scales should be on same page if possible. If you put it on two pages, then repeat headings.
PLACEMENT OF SECTIONS Section Placement Textbox Sociodemographic Questions measuring indicators Sensitive questions Independent variable Dependent variable Open-ended questions Contingency Questions Index (ices) Top of first page of survey First page of survey Bottom of last page of survey First page of survey Second page of survey End of survey Throughout the survey Generally middle or end of survey
STEP 4 - CRITERIA FOR EVALUATION - QUESTIONS YOU MUST ASK Appearance Does it look professional? Is the text balanced? Did you use numbers? Are they in correct order? Statement at Top Does it include relevant/complete information? Is it in a textbox? Questions Format for index or scale Does it have a textbox at top with instructions? Is it numbered from left to right lower to higher? Did you use statements not questions? Are statements logically related to each other? Wording Did you use a spelling and grammar check? Did you avoid biased statements/words, slang, abbreviations, double-barreled questions/statements etc.? Answer categories must be: -measured at highest level possible? -exhaustive/mutually exclusive? Did you test your hypotheses/measure your indicators?
AVOID COMMON MISTAKES Use numbers not letters Measure indicators/variables at the highest level possible (i.e., the ratio level, whenever possible) Ask only relevant questions Don t ask college students if they graduated from high school Use statements rather than questions in indices. You can t agree with a question! Avoid bias. If you can predict what answer the researcher wants then it is a biased statement or question. Make wording across answer categories consistent For instance, IF you use pro-life, then use pro-choice For instance, IF you use African American, then use Asian American.
EVALUATING YOUR INDICATORS You can use the following chart or one like it to evaluate the validity of the indicators used in your survey/questionnaire Concept Instrument Face Validity (Y/N) For each indicator, ask yourself the following question? 1. Can you match a question or questions on your survey with each indicator you are using to measure impact? 2. Are you measuring attitudes or behavior? Does the question/statement in the survey match? 2. If you are measuring a multidimensional concept do you have more than one question/statement? Then determine if your concept has face validity (i.e., it logically appears to measure the concept).
EVALUATING YOUR INDICATORS EXAMPLE You can use the following chart or one like it to evaluate the validity of the indicators used in your survey/questionnaire Concept Instrument Face Validity (Y/N) Impact of program How often do you recycle Y on recycling behavior your drink containers? Impact on attitudes toward To what extent do you agree Y water conservation that Utah residents must learn to conserve water or face shortages. For each indicator, ask yourself the following question: 1. Can you match a question or questions on your survey with each indicator you are using to measure impact? 2. Are you measuring attitudes or behavior? Does the question/statement in the survey match? 2. If you are measuring a multidimensional concept do you have more than one question/statement? Then determine if your concept has face validity (i.e., it logically appears to measure the concept).
EVALUATING YOUR INDICATORS You can use the following chart to evaluate the reliability of each of your indicators. Take each concept, and then rate the indicator on the following characteristics. Generally, the higher the score, the more reliable the indicator. Indicator 1 Precise Unbiased Not Double barreled Exhaustive Exclusive Not offensive Reliability (+/-) For each of your indicators, ask yourself if you think your indicators would yield consistent results across time. Answer the following questions: 1. Are questions precise and unambiguous? 2. Are there any words that may be interpreted differently by different people? (slang) 3. Are there biased questions that may influence some people s answers? 4. Are any questions double barreled? 5. Are answers exhaustive? (Did you use Other please specify as an answer category?) 6. Are answers mutually exclusive? 7. Are there questions that may offend someone?
EVALUATING YOUR INDICATORS AN EXAMPLE Use the following chart to evaluate the reliability of your indicators. Take each concept, and rate the indicator.. Generally, the higher the score, the more reliable the indicator. Indicator 1 Reliability How would you rate the presentation you just attended? 1. Fair 2. Good 3. Excellent (+/-) Precise - Unbiased - Not Double barreled + Exhaustive - Exclusive + Not offensive + Only 3 of 6. This indicator needs to be reworded For each of your indicators, ask yourself if you think your indicators would yield consistent results across time. Answer the following questions: 1. Are questions precise and unambiguous? 2. Are there any words that may be interpreted differently by different people? (slang) 3. Are there biased questions that may influence some people s answers? 4. Are any questions double barreled? 5. Are answers exhaustive? (Did you use Other please specify as an answer category?) 6. Are answers mutually exclusive? 7. Are there questions that may offend someone?
STEP 5 CONSTRUCTING A CODEBOOK General Guidelines Identify a name for your codebook. It should start with a letter (not a number), and should contain no slashes, hyphens etc. It should be about 8 characters. Put an identification number on each of your surveys. This identification number should be the first variable in your codebook. Construct a name to identify each statement/question used in your survey to measure each indicator. Do not use slashes/dashes/spaces. This name will appear on your statistical outputs when analyzing this question/statement. Compose a BRIEF description for each variable/indicator. Construct a code that corresponds to each variable/indicator. The code should consist of numbers representing each possible response/answer.
STEP 5 CONSTRUCTING A CODEBOOK FORMAT OF CODEBOOK Q # Var# Var Name Description Code 1 ID Identification # 1-100 1 2 Sex Sex of respondent 1=male 2=female 3 Age Age of Respondent 18-99 years 4 Fear I fear my future 1=strongly disagree has few 2=disagree 3=undecided 4=agree 5=strongly agree 5 Hope I have great same hope for my future
EXAMPLES OF SURVEYS Following is an Example of a Survey. Please note the following: There is no textbox at the beginning. The information often provided in the textbox was provided in a cover letter. There are a number of different indices You don t have to always use Strongly Disagree to Strongly Agree. Some of the questions indicate that you can circle all that apply This ensures that possible answer categories are exhaustive. Some questions have an other specify category. This ensures possible answer categories are exhaustive. Not all of the questions/statements are written correctly
This should be from 1. Mostly F s to 5. Mostly A s.
IMPORTANT FINAL COMMENTS ALWAYS DO Have others proofread your survey and give you suggestions Pre-test your survey using respondents similar to those in your sample Submit your survey to IRB for approval Let respondents know you are associated with USU NEVER DO Send our surveys without IRB approval
FOR MORE INFORMATION Contact Dr. Carol Albrecht Assessment Specialist USU Extension Phone 979-777-2421 E-mail carol.albrecht@usu.edu